Project

General

Profile

Bug #3797

osd takes 100% cpu after upgrading from 0.48.2argonaut to the latest 0.48.3argonaut

Added by Corin Langosch about 11 years ago. Updated about 11 years ago.

Status:
Duplicate
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I just upgraded one of my production servers (2 osds) from 0.48.2argonaut to the latest 0.48.3argonaut and now of the osds is taking 100% for over 12 minutes now. I never had this with the previous argonaut before. Memory usage is only 30mb rss (the other osd seems to be running normally, taking almost no cpu and around 300 mb rss). "ceph -w" reports the cluster is healty and everything is up, but it seems the one osd is hanging and not really doing anything (because memory usage stays that low)?

Here's the logfile of the osd: http://pastie.org/pastes/5687380/text
Output of strace is here (not really much): http://pastie.org/5687401/text

Restarting the osd doesn't help - it shuts down cleanly, but after starting it it takes again 100% cpu. What can I do?

ceph-osd.8.log View (4.86 MB) Corin Langosch, 01/27/2013 05:03 AM

ceph.png View (25.6 KB) Corin Langosch, 01/27/2013 05:06 AM

ceph-osd.8.log View (7.23 MB) Corin Langosch, 01/27/2013 10:25 AM

gdb.txt View (130 KB) Corin Langosch, 01/27/2013 10:56 AM


Related issues

Related to Ceph - Feature #3376: use external leveldb package for default builds Duplicate

History

#1 Updated by Corin Langosch about 11 years ago

I just noticed the second osd is now consuming 100% cpu too. Before it was properly running for around 15 minutes. Guess I'll need to downgrade now... :(

#2 Updated by Corin Langosch about 11 years ago

Here's the output of dstat http://pastie.org/5687470.text

I'm not sure why it is writing so much now, before the update there was almost no disk activity.

Output of "iostat -xdm 5 /dev/sd*" is here http://pastie.org/pastes/5687479/text

osd.8 is on /dev/sda1 (the one taking 100% cpu right after starting it)
osd.9 is on /dev/sdb1 (the one taking 100% cpu around 15 minutes after starting it)

#3 Updated by Corin Langosch about 11 years ago

I just downgraded to 0.48.2argonaut and everything seems to be running normally again now:

Before downgrade:
ii ceph 0.48.3argonaut-1quantal amd64 distributed storage and file system
ii ceph-common 0.48.3argonaut-1quantal amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fs-common 0.48.3argonaut-1quantal amd64 common utilities to mount and interact with a ceph file system
ii ceph-fuse 0.48.3argonaut-1quantal amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 0.48.3argonaut-1quantal amd64 metadata server for the ceph distributed file system
ii libcephfs1 0.48.3argonaut-1quantal amd64 Ceph distributed file system client library
ii librados2 0.48.3argonaut-1quantal amd64 RADOS distributed object store client library
ii librbd1 0.48.3argonaut-1quantal amd64 RADOS block device client library
ii python-ceph 0.48.3argonaut-1quantal amd64 Python libraries for the Ceph distributed filesystem

After downgrade:
ii ceph 0.48.2-0ubuntu2 amd64 distributed storage and file system
ii ceph-common 0.48.2-0ubuntu2 amd64 common utilities to mount and interact with a ceph filesystem
ii ceph-fs-common 0.48.2-0ubuntu2 amd64 common utilities to mount and interact with a ceph filesystem
ii ceph-mds 0.48.2-0ubuntu2 amd64 distributed filesystem service
ii libcephfs1 0.48.2-0ubuntu2 amd64 Ceph distributed file system client library
ii librados2 0.48.2-0ubuntu2 amd64 RADOS distributed object store client library
ii librbd1 0.48.2-0ubuntu2 amd64 RADOS block device client library
ii python-ceph 0.48.2-0ubuntu2 amd64 Python libraries for the Ceph distributed filesystem

#4 Updated by Sage Weil about 11 years ago

Can you try reupgrading one of the nodes and start it with debug file store = 20? That will tell is what it is writing.

#5 Updated by Ian Colle about 11 years ago

  • Status changed from New to Need More Info

#6 Updated by Sage Weil about 11 years ago

  • Priority changed from Immediate to Urgent

#7 Updated by Sage Weil about 11 years ago

Corin-

Have you tried 0.48.3 again since then? I'd like to get to the bottom of this, if possible... :)

#8 Updated by Corin Langosch about 11 years ago

Hi Sage,

sorry for the delay. I just shutdown the osd, upgraded it and started it again. It's again using almost 100% of one core and performing 11MB/s write.

ps aux
root 27045 71.9 0.3 1268848 59392 ? Ssl 12:54 2:00 /usr/bin/ceph-osd -i 8 --pid-file /var/run/ceph/osd.8.pid -c /etc/ceph/ceph.conf

iotop
27131 be/4 root 0.00 B/s 10.46 M/s 0.00 % 16.01 % ceph-osd -i 8 --pid-file /var/run/ceph/osd.8.pid -c /etc/ceph/ceph.conf

The log file of the last few minutes is attached. Hope it helps :)

Corin

#9 Updated by Corin Langosch about 11 years ago

Here's a nice graph to see the difference before/ after upgrade of disk activity....

The cluster is clean, no recovery running.

#10 Updated by Corin Langosch about 11 years ago

Just a small update - nothing changed so far. The cluster is still healthy, but the osd is still using 100% of one core and writing 11MB/s.

27045 root 20 0 1245m 57m 4188 S 76.6 0.4 40:12.77 ceph-osd

#11 Updated by Corin Langosch about 11 years ago

Just another small update - nothing changed so far. The cluster is still healthy, but the osd is still using 100% of one core and writing 11MB/s.

root 27045 72.4 0.3 1290560 57896 ? Ssl 12:54 156:28 /usr/bin/ceph-osd -i 8 --pid-file /var/run/ceph/osd.8.pid -c /etc/ceph/ceph.conf

Do you need more information? Do you think the bug will really soon be fixed and a new version released? Otherwise I'd downgrad to 0.48.2 within the next 1-2 days.

#12 Updated by Sage Weil about 11 years ago

Hi Corin-

Can you enable 'debug osd = 20' for a bit and attach that log? I think this is related to 830b8ffa2e53fde37b74de12a053e037ff3f15b0 which fixed a bug in snap trimming, but i'm not quite sure. Thanks!

#13 Updated by Corin Langosch about 11 years ago

Hi Sage, here we go. Is it enough data or do you need more? I didn't disable the logging yet...

#14 Updated by Corin Langosch about 11 years ago

Output of gdb /usr/bin/ceph-osd $pid, then 'thread apply all bt'

#15 Updated by Sage Weil about 11 years ago

looks like levedb spinning on background compaction.

his .2 package is quantals, which is leveldb 1.5.. newer than argonaut .3 but older than what we were about to merge in for master. pushing wip-argonaut-leveldb for him to test (that has 1.6).

#16 Updated by Sage Weil about 11 years ago

that fixed it it seems. we could

- update argonaut and bobtail to newer leveldb :/
- link dynamically for quantal builds only
- link dynamically for all deb builds :/

#17 Updated by Sage Weil about 11 years ago

  • Status changed from Need More Info to Fix Under Review

#18 Updated by Corin Langosch about 11 years ago

Ok, after around 10 minutes of runtime everything seems normal. Thanks for the fast and great help! :-)

ceph version 0.48.3argonaut-14-g5df37c2 (commit:5df37c228a7992e7878c6acc0f45b1ad1808481c)

root 4281 5.1 0.3 1243308 50288 ? Ssl 19:51 0:28 /usr/bin/ceph-osd -i 8 --pid-file /var/run/ceph/osd.8.pid -c /etc/ceph/ceph.conf

#19 Updated by Corin Langosch about 11 years ago

Ok, everything still looks good :). Last question: should I upgrade my whole cluster to this version or will a new argonaut release made the following days? :)

#20 Updated by Sage Weil about 11 years ago

The conclusion:

- quantal had a newer libleveldb than we built statically into our debs
- downgrading made compaction spin
- a test branch that staticall linked a newer version that quantals' (the version that gary prepared for master) fixed it.

Corin: for now just upgrade the others to the branch i pushed. we're going to redo the way we link to leveldb to avoid situations like this, but it is a non-trivial change.

thanks!

#21 Updated by Sage Weil about 11 years ago

  • Status changed from Fix Under Review to Duplicate
  • Source changed from Development to Community (user)

see #3376

Also available in: Atom PDF