Project

General

Profile

Feature #138

Try out tcmalloc

Added by Greg Farnum almost 14 years ago. Updated over 13 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

The cosd daemon seems to eat up fragmented memory or something, since heap size stays fairly consistent but top memory usage keeps growing. johnl in the channel suggested we look at tcmalloc (http://goog-perftools.sourceforge.net/doc/tcmalloc.html) and the description certainly sounds promising.
I'm concerned by a sentence under the caveats section, though: "TCMalloc currently does not return any memory to the system."

But http://google-perftools.googlecode.com/svn/trunk/doc/tcmalloc.html seems to be more up-to-date and omits that one, so maybe we're good.

Anyway, it should be looked into!

History

#1 Updated by Sage Weil almost 14 years ago

  • Priority changed from Normal to Low

#2 Updated by Greg Farnum almost 14 years ago

  • Assignee set to Greg Farnum

#3 Updated by Greg Farnum almost 14 years ago

  • Status changed from New to In Progress

Okay, this is definitely something we need to look into more. I tried running with tcmalloc and the standard malloc, and ran
"./rados -p data bench 3600 write -t 20 -b 10485760" (1 hour run, 10MB writes, 20 in-flight)
The standard malloc resulted in the OSD using 3.1GB resident and 3.6GB virtual memory at the end of the hour (peaking at 3.3GB and 3.7GB).
tcmalloc resulted in the OSD using 703.7MB resident and 1.0GB virtual at the end of the hour (that was the peak, but it did clean up at itself several times over the run).

#4 Updated by Greg Farnum almost 14 years ago

  • Priority changed from Low to Normal

Going to implement this as a compile-time option (based on available libraries) once I've audited the mds and osd for memory leaks so we can see if ptmalloc holds up better once those are toned down.

#5 Updated by Greg Farnum almost 14 years ago

All right, this is on hold while we work through some of the bugs that have been reported recently.

#6 Updated by Sage Weil over 13 years ago

  • Target version set to v0.22

#7 Updated by Sage Weil over 13 years ago

let's turn this on for cmds and cosd.
and update configure.ac to detect it.
and set debian/control and ceph.spec.in dependencies.

#8 Updated by Greg Farnum over 13 years ago

Dunno how to set up the packaging stuff, but the configure.ac/Makefile stuff wasn't too complicated and is pushed in the tcmalloc branch.

#9 Updated by Wido den Hollander over 13 years ago

I can confirm that tcmalloc() works fine, seeing about 70% memory reduction on my OSD and MDS, great!

Tested it on Ubuntu 10.04 AMD64, but there was a problem with the libgoogle* packages from Ubuntu are not correct: https://bugs.launchpad.net/ubuntu/+source/google-perftools/+bug/359736

Installing libgoogle* libtcmalloc and libunwind from Debian Sid works.

#10 Updated by Greg Farnum over 13 years ago

  • Assignee changed from Greg Farnum to Sage Weil

I guess Sage is going to handle the packaging once the kernel rbd split is accomplished.

#11 Updated by Sage Weil over 13 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF