Bug #15558: ceph-osd crashed when starting. - Ceph - Ceph

Actions

#1

Updated by zhou wei about 8 years ago

My os is centos 7.
Just now, I upgraded it to v10.1.2 with yum from v10.1.1.
OSD crushed when I restarted it.

Here is the log:

2016-04-21 14:40:58.603908 7f956f6cd800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-04-21 14:40:58.603936 7f956f6cd800  0 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 17305
2016-04-21 14:40:58.604005 7f956f6cd800 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

2016-04-21 14:40:58.605484 7f956f6cd800  0 pidfile_write: ignore empty --pid-file
2016-04-21 14:40:58.609536 7f956f6cd800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-04-21 14:40:58.627786 7f956f6cd800  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2016-04-21 14:40:58.627855 7f956f6cd800  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-04-21 14:40:58.628734 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-04-21 14:40:58.628842 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 2000293010944 (1862 GB) block_size 4096 (4096 B)
2016-04-21 14:40:58.629356 7f956f6cd800  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2016-04-21 14:40:58.630161 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2016-04-21 14:40:58.630256 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 2000293010944 (1862 GB) block_size 4096 (4096 B)
2016-04-21 14:40:58.630270 7f956f6cd800  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 1862 GB
2016-04-21 14:40:58.630309 7f956f6cd800  1 bluefs mount
2016-04-21 14:40:58.635855 7f956f6cd800 -1 *** Caught signal (Segmentation fault) **
 in thread 7f956f6cd800 thread_name:ceph-osd

 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x9126aa) [0x7f956f0786aa]
 2: (()+0xf130) [0x7f956d0b9130]
 3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x48b) [0x7f956eecb5fb]
 4: (BlueFS::_replay()+0x29c) [0x7f956eed989c]
 5: (BlueFS::mount()+0x1d2) [0x7f956eedc852]
 6: (BlueStore::_open_db(bool)+0xd04) [0x7f956ede3a94]
 7: (BlueStore::mount()+0x49a) [0x7f956edeafaa]
 8: (OSD::init()+0x27d) [0x7f956eb0885d]
 9: (main()+0x2c55) [0x7f956ea6e835]
 10: (__libc_start_main()+0xf5) [0x7f956b66baf5]
 11: (()+0x352c59) [0x7f956eab8c59]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -54> 2016-04-21 14:40:58.599021 7f956f6cd800  5 asok(0x7f9579328280) register_command perfcounters_dump hook 0x7f95792d8040
   -53> 2016-04-21 14:40:58.599057 7f956f6cd800  5 asok(0x7f9579328280) register_command 1 hook 0x7f95792d8040
   -52> 2016-04-21 14:40:58.599066 7f956f6cd800  5 asok(0x7f9579328280) register_command perf dump hook 0x7f95792d8040
   -51> 2016-04-21 14:40:58.599077 7f956f6cd800  5 asok(0x7f9579328280) register_command perfcounters_schema hook 0x7f95792d8040
   -50> 2016-04-21 14:40:58.599088 7f956f6cd800  5 asok(0x7f9579328280) register_command 2 hook 0x7f95792d8040
   -49> 2016-04-21 14:40:58.599092 7f956f6cd800  5 asok(0x7f9579328280) register_command perf schema hook 0x7f95792d8040
   -48> 2016-04-21 14:40:58.599100 7f956f6cd800  5 asok(0x7f9579328280) register_command perf reset hook 0x7f95792d8040
   -47> 2016-04-21 14:40:58.599106 7f956f6cd800  5 asok(0x7f9579328280) register_command config show hook 0x7f95792d8040
   -46> 2016-04-21 14:40:58.599114 7f956f6cd800  5 asok(0x7f9579328280) register_command config set hook 0x7f95792d8040
   -45> 2016-04-21 14:40:58.599123 7f956f6cd800  5 asok(0x7f9579328280) register_command config get hook 0x7f95792d8040
   -44> 2016-04-21 14:40:58.599132 7f956f6cd800  5 asok(0x7f9579328280) register_command config diff hook 0x7f95792d8040
   -43> 2016-04-21 14:40:58.599140 7f956f6cd800  5 asok(0x7f9579328280) register_command log flush hook 0x7f95792d8040
   -42> 2016-04-21 14:40:58.599144 7f956f6cd800  5 asok(0x7f9579328280) register_command log dump hook 0x7f95792d8040
   -41> 2016-04-21 14:40:58.599152 7f956f6cd800  5 asok(0x7f9579328280) register_command log reopen hook 0x7f95792d8040
   -40> 2016-04-21 14:40:58.603716 7f956f6cd800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -39> 2016-04-21 14:40:58.603908 7f956f6cd800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -38> 2016-04-21 14:40:58.603936 7f956f6cd800  0 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4), process ceph-osd, pid 17305
   -37> 2016-04-21 14:40:58.603979 7f956f6cd800  5 object store type is bluestore
   -36> 2016-04-21 14:40:58.604005 7f956f6cd800 -1 WARNING: experimental feature 'bluestore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster.  Do not use
feature with important data.

   -35> 2016-04-21 14:40:58.604639 7f956f6cd800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/17305 need_addr=1
   -34> 2016-04-21 14:40:58.604677 7f956f6cd800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/17305 need_addr=1
   -33> 2016-04-21 14:40:58.604696 7f956f6cd800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/17305 need_addr=1
   -32> 2016-04-21 14:40:58.604714 7f956f6cd800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/17305 need_addr=1
   -31> 2016-04-21 14:40:58.605484 7f956f6cd800  0 pidfile_write: ignore empty --pid-file
   -30> 2016-04-21 14:40:58.605657 7f956f6cd800  1 finished global_init_daemonize
   -29> 2016-04-21 14:40:58.609536 7f956f6cd800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -28> 2016-04-21 14:40:58.609553 7f956f6cd800  5 asok(0x7f9579328280) init /var/run/ceph/ceph-osd.0.asok
   -27> 2016-04-21 14:40:58.609571 7f956f6cd800  5 asok(0x7f9579328280) bind_and_listen /var/run/ceph/ceph-osd.0.asok
   -26> 2016-04-21 14:40:58.609736 7f956f6cd800  5 asok(0x7f9579328280) register_command 0 hook 0x7f95792d00a8
   -25> 2016-04-21 14:40:58.609759 7f956f6cd800  5 asok(0x7f9579328280) register_command version hook 0x7f95792d00a8
   -24> 2016-04-21 14:40:58.609767 7f956f6cd800  5 asok(0x7f9579328280) register_command git_version hook 0x7f95792d00a8
   -23> 2016-04-21 14:40:58.609780 7f956f6cd800  5 asok(0x7f9579328280) register_command help hook 0x7f95792d8130
   -22> 2016-04-21 14:40:58.609793 7f956f6cd800  5 asok(0x7f9579328280) register_command get_command_descriptions hook 0x7f95792d8120
   -21> 2016-04-21 14:40:58.609869 7f9568a47700  5 asok(0x7f9579328280) entry start
   -20> 2016-04-21 14:40:58.609938 7f956f6cd800 10 monclient(hunting): build_initial_monmap
   -19> 2016-04-21 14:40:58.626936 7f956f6cd800  5 adding auth protocol: cephx
   -18> 2016-04-21 14:40:58.626953 7f956f6cd800  5 adding auth protocol: cephx
   -17> 2016-04-21 14:40:58.627292 7f956f6cd800  5 asok(0x7f9579328280) register_command objecter_requests hook 0x7f95792d8220
   -16> 2016-04-21 14:40:58.627424 7f956f6cd800  1 -- 0.0.0.0:6800/17305 messenger.start
   -15> 2016-04-21 14:40:58.627498 7f956f6cd800  1 -- :/0 messenger.start
   -14> 2016-04-21 14:40:58.627540 7f956f6cd800  1 -- 0.0.0.0:6803/17305 messenger.start
   -13> 2016-04-21 14:40:58.627579 7f956f6cd800  1 -- 0.0.0.0:6802/17305 messenger.start
   -12> 2016-04-21 14:40:58.627619 7f956f6cd800  1 -- 0.0.0.0:6801/17305 messenger.start
   -11> 2016-04-21 14:40:58.627657 7f956f6cd800  1 -- :/0 messenger.start
   -10> 2016-04-21 14:40:58.627771 7f956f6cd800  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
    -9> 2016-04-21 14:40:58.627786 7f956f6cd800  1 bluestore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
    -8> 2016-04-21 14:40:58.627855 7f956f6cd800  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -7> 2016-04-21 14:40:58.628734 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -6> 2016-04-21 14:40:58.628842 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 2000293010944 (1862 GB) block_size 4096 (4096 B)
    -5> 2016-04-21 14:40:58.629356 7f956f6cd800  1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
    -4> 2016-04-21 14:40:58.630161 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
    -3> 2016-04-21 14:40:58.630256 7f956f6cd800  1 bdev(/var/lib/ceph/osd/ceph-0/block) open size 2000293010944 (1862 GB) block_size 4096 (4096 B)
    -2> 2016-04-21 14:40:58.630270 7f956f6cd800  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block size 1862 GB
    -1> 2016-04-21 14:40:58.630309 7f956f6cd800  1 bluefs mount
     0> 2016-04-21 14:40:58.635855 7f956f6cd800 -1 *** Caught signal (Segmentation fault) **
 in thread 7f956f6cd800 thread_name:ceph-osd

 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x9126aa) [0x7f956f0786aa]
 2: (()+0xf130) [0x7f956d0b9130]
 3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::list*, char*)+0x48b) [0x7f956eecb5fb]
 4: (BlueFS::_replay()+0x29c) [0x7f956eed989c]
 5: (BlueFS::mount()+0x1d2) [0x7f956eedc852]
 6: (BlueStore::_open_db(bool)+0xd04) [0x7f956ede3a94]
 7: (BlueStore::mount()+0x49a) [0x7f956edeafaa]
 8: (OSD::init()+0x27d) [0x7f956eb0885d]
 9: (main()+0x2c55) [0x7f956ea6e835]
 10: (__libc_start_main()+0xf5) [0x7f956b66baf5]
 11: (()+0x352c59) [0x7f956eab8c59]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---

Actions

Copy link

#2

Updated by zhou wei about 8 years ago

mon crushed too:

2016-04-21 15:20:20.282296 7f0c6bbd74c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-04-21 15:20:20.282502 7f0c6bbd74c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-04-21 15:20:20.289002 7f0c6bbd74c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7f0c6bbd74c0 thread_name:ceph-mon
 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x506c8a) [0x7f0c6b5f9c8a]
 2: (()+0xf130) [0x7f0c69e85130]
 3: (gsignal()+0x39) [0x7f0c686875c9]
 4: (abort()+0x148) [0x7f0c68688cd8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f0c68f8b9d5]
 6: (()+0x5e946) [0x7f0c68f89946]
 7: (()+0x5e973) [0x7f0c68f89973]
 8: (()+0x5eb9f) [0x7f0c68f89b9f]
 9: (()+0x637075) [0x7f0c6b72a075]
 10: (FSMap::decode(ceph::buffer::list::iterator&)+0x103f) [0x7f0c6b63f1bf]
 11: (MDSMonitor::update_from_paxos(bool*)+0x190) [0x7f0c6b456c10]
 12: (PaxosService::refresh(bool*)+0x1a5) [0x7f0c6b3c7d55]
 13: (Monitor::refresh_from_paxos(bool*)+0x15b) [0x7f0c6b36136b]
 14: (Monitor::init_paxos()+0x95) [0x7f0c6b361805]
 15: (Monitor::preinit()+0x949) [0x7f0c6b371a49]
 16: (main()+0x23e5) [0x7f0c6b2fdd45]
 17: (__libc_start_main()+0xf5) [0x7f0c68673af5]
 18: (()+0x25e2b1) [0x7f0c6b3512b1]
2016-04-21 15:20:20.406313 7f0c6bbd74c0 -1 *** Caught signal (Aborted) **
 in thread 7f0c6bbd74c0 thread_name:ceph-mon

 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x506c8a) [0x7f0c6b5f9c8a]
 2: (()+0xf130) [0x7f0c69e85130]
 3: (gsignal()+0x39) [0x7f0c686875c9]
 4: (abort()+0x148) [0x7f0c68688cd8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f0c68f8b9d5]
 6: (()+0x5e946) [0x7f0c68f89946]
 7: (()+0x5e973) [0x7f0c68f89973]
 8: (()+0x5eb9f) [0x7f0c68f89b9f]
 9: (()+0x637075) [0x7f0c6b72a075]
 10: (FSMap::decode(ceph::buffer::list::iterator&)+0x103f) [0x7f0c6b63f1bf]
 11: (MDSMonitor::update_from_paxos(bool*)+0x190) [0x7f0c6b456c10]
 12: (PaxosService::refresh(bool*)+0x1a5) [0x7f0c6b3c7d55]
 13: (Monitor::refresh_from_paxos(bool*)+0x15b) [0x7f0c6b36136b]
 14: (Monitor::init_paxos()+0x95) [0x7f0c6b361805]
 15: (Monitor::preinit()+0x949) [0x7f0c6b371a49]
 16: (main()+0x23e5) [0x7f0c6b2fdd45]
 17: (__libc_start_main()+0xf5) [0x7f0c68673af5]
 18: (()+0x25e2b1) [0x7f0c6b3512b1]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

   -28> 2016-04-21 15:20:20.282296 7f0c6bbd74c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -27> 2016-04-21 15:20:20.282502 7f0c6bbd74c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
   -24> 2016-04-21 15:20:20.289002 7f0c6bbd74c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
     0> 2016-04-21 15:20:20.406313 7f0c6bbd74c0 -1 *** Caught signal (Aborted) **
 in thread 7f0c6bbd74c0 thread_name:ceph-mon

 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x506c8a) [0x7f0c6b5f9c8a]
 2: (()+0xf130) [0x7f0c69e85130]
 3: (gsignal()+0x39) [0x7f0c686875c9]
 4: (abort()+0x148) [0x7f0c68688cd8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f0c68f8b9d5]
 6: (()+0x5e946) [0x7f0c68f89946]
 7: (()+0x5e973) [0x7f0c68f89973]
 8: (()+0x5eb9f) [0x7f0c68f89b9f]
 9: (()+0x637075) [0x7f0c6b72a075]
 10: (FSMap::decode(ceph::buffer::list::iterator&)+0x103f) [0x7f0c6b63f1bf]
 11: (MDSMonitor::update_from_paxos(bool*)+0x190) [0x7f0c6b456c10]
 12: (PaxosService::refresh(bool*)+0x1a5) [0x7f0c6b3c7d55]
 13: (Monitor::refresh_from_paxos(bool*)+0x15b) [0x7f0c6b36136b]
 14: (Monitor::init_paxos()+0x95) [0x7f0c6b361805]
 15: (Monitor::preinit()+0x949) [0x7f0c6b371a49]
 16: (main()+0x23e5) [0x7f0c6b2fdd45]
 17: (__libc_start_main()+0xf5) [0x7f0c68673af5]
 18: (()+0x25e2b1) [0x7f0c6b3512b1]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

[106558]: (33) Numerical argument out of domain

Actions

Copy link

#3