Ceph » bluestore

- After hitting this crash the systemd restarted the OSD container pod and after the restart, OSD is running fine!

The same cluster OSD.95 had also hit the same assert and got restarted by systemd and after that running fine!

2022-01-17T22:56:40.077+0000 7f03e4a29700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f03e4a29700 time 2022-01-17T22:56:40.066674+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: 4148: FAILED ceph_assert(cur >= p.length)

 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x561ec5505b8e]
 2: /usr/bin/ceph-osd(+0x5d5daf) [0x561ec5505daf]
 3: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x16a) [0x561ec5bc9cca]
 4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x561ec5c67b45]
 5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x561ec5c68009]
 6: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x561ec5c836de]
 7: (BlueRocksWritableFile::Sync()+0x18) [0x561ec5c93fb8]
 8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x561ec61b774f]
 9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x561ec62e7f22]
 10: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x561ec62e98e8]
 11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x561ec61ce701]
 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x561ec61d6454]
 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x561ec61d65a1]
 14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x561ec6170f84]
 15: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x561ec617198a]
 16: (BlueStore::_kv_sync_thread()+0x3530) [0x561ec5bc8390]
 17: (BlueStore::KVSyncThread::entry()+0x11) [0x561ec5bf70b1]
 18: /lib64/libpthread.so.0(+0x817a) [0x7f03fb44b17a]
 19: clone()

2022-01-17T22:57:03.972+0000 7f33a0448240  0 set uid:gid to 167:167 (ceph:ceph)
2022-01-17T22:57:03.972+0000 7f33a0448240  0 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev), process ceph-osd, pid 8
2022-01-17T22:57:03.972+0000 7f33a0448240  0 pidfile_write: ignore empty --pid-file
2022-01-17T22:57:03.974+0000 7f33a0448240  1 bdev(0x563d0b97d400 /var/lib/ceph/osd/ceph-95/block) open path /var/lib/ceph/osd/ceph-95/block
2022-01-17T22:57:03.975+0000 7f33a0448240  1 bdev(0x563d0b97d400 /var/lib/ceph/osd/ceph-95/block) open size 1999839952896 (0x1d19fc00000, 1.8 TiB) block_size 4096 (4 KiB) rotational discard not supported

- OSD.0 also had similar crash

2022-01-17T23:40:22.136+0000 7f32423b0700 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7f32423b0700 time 2022-01-17T23:40:22.040838+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10229-g7e035110/rpm/el8/BUILD/ceph-17.0.0-10229-g7e035110/src/os/bluestore/BlueStore.h: 4148: FAILED ceph_assert(cur >= p.length)

 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55ffca66cb8e]
 2: /usr/bin/ceph-osd(+0x5d5daf) [0x55ffca66cdaf]
 3: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x16a) [0x55ffcad30cca]
 4: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x55ffcadceb45]
 5: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x55ffcadcf009]
 6: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x55ffcadea6de]
 7: (BlueRocksWritableFile::Sync()+0x18) [0x55ffcadfafb8]
 8: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55ffcb31e74f]
 9: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x55ffcb44ef22]
 10: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x55ffcb4508e8]
 11: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x55ffcb335701]
 12: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x55ffcb33d454]
 13: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x55ffcb33d5a1]
 14: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x55ffcb2d7f84]
 15: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x55ffcb2d898a]
 16: (BlueStore::_kv_sync_thread()+0x3530) [0x55ffcad2f390]
 17: (BlueStore::KVSyncThread::entry()+0x11) [0x55ffcad5e0b1]
 18: /lib64/libpthread.so.0(+0x817a) [0x7f3258dd217a]
 19: clone()

2022-01-17T23:40:22.230+0000 7f32423b0700 -1 *** Caught signal (Aborted) **
 in thread 7f32423b0700 thread_name:bstore_kv_sync

 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f3258ddcc20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x55ffca66cbec]
 5: /usr/bin/ceph-osd(+0x5d5daf) [0x55ffca66cdaf]
 6: (RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x16a) [0x55ffcad30cca]
 7: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x735) [0x55ffcadceb45]
 8: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xa9) [0x55ffcadcf009]
 9: (BlueFS::fsync(BlueFS::FileWriter*)+0x18e) [0x55ffcadea6de]
 10: (BlueRocksWritableFile::Sync()+0x18) [0x55ffcadfafb8]
 11: (rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55ffcb31e74f]
 12: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x662) [0x55ffcb44ef22]
 13: (rocksdb::WritableFileWriter::Sync(bool)+0xf8) [0x55ffcb4508e8]
 14: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x341) [0x55ffcb335701]
 15: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1c04) [0x55ffcb33d454]
 16: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x55ffcb33d5a1]
 17: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x55ffcb2d7f84]
 18: (RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x55ffcb2d898a]
 19: (BlueStore::_kv_sync_thread()+0x3530) [0x55ffcad2f390]
 20: (BlueStore::KVSyncThread::entry()+0x11) [0x55ffcad5e0b1]
 21: /lib64/libpthread.so.0(+0x817a) [0x7f3258dd217a]
 22: clone()

2022-01-17T23:40:42.537+0000 7fa18eaa0240  0 set uid:gid to 167:167 (ceph:ceph)
2022-01-17T23:40:42.537+0000 7fa18eaa0240  0 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev), process ceph-osd, pid 7
2022-01-17T23:40:42.537+0000 7fa18eaa0240  0 pidfile_write: ignore empty --pid-file
2022-01-17T23:40:42.555+0000 7fa18eaa0240  1 bdev(0x559239489400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2022-01-17T23:40:42.556+0000 7fa18eaa0240  1 bdev(0x559239489400 /var/lib/ceph/osd/ceph-0/block) open size 1999839952896 (0x1d19fc00000, 1.8 TiB) block_size 4096 (4 KiB) rotational discard not supported
2022-01-17T23:40:42.556+0000 7fa18eaa0240  1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06
2022-01-17T23:40:42.556+0000 7fa18eaa0240  1 bdev(0x559239488c00 /var/lib/ceph/osd/ceph-0/block.db) open path /var/lib/ceph/osd/ceph-0/block.db

Hello.

we just upgraded our ceph from 16.2.13 to 16.2.14 and saw a osd crash with the same error message that this bug report states.

ceph-16.2.14/src/os/bluestore/BlueStore.h: 3870: FAILED ceph_assert(cur >= p.length)

The osd comes up without any problems after that.
We are not sure if this is related to the minor upgrade or not because before update 16.2.14 we had failing osds everyday due to another bug.

Should we create a new bug report regarding the ceph_assert error or could it be related to the same issue stated here?

@Maximilian @Zakhar Kirpichenko - the issue to be fixed in the next Pacific minor release. Relevant backport PR is https://github.com/ceph/ceph/pull/53587

Please also see more details on how to workaround this in my post at https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4HXF4XWB6IOZ2RKXX7EQCEIY/

We also ran into the same issue, after upgrading from 16.2.13 to 16.2.14

2023-10-15T20:08:11.719196Z_d95e3d5d-3cfe-4a53-9025-daf1745f345d  osd.6    *
2023-10-15T22:27:03.691912Z_1b30c7d8-98f7-448c-a4d1-035c03ee79e7  osd.17   *
2023-10-16T01:38:59.282064Z_0ce2a23a-fe39-4aae-aceb-6cb17864f593  osd.4    *
2023-10-16T04:49:14.343657Z_912bda6e-dc46-4dbb-bc7d-697a709f626c  osd.0    *
2023-10-16T04:56:47.076272Z_bbf9634b-c2ba-40ea-bd20-9deca41c60fd  osd.10   *
2023-10-16T10:21:53.195344Z_610d7e04-a997-4562-89e5-ac86bf78f9e5  osd.23   *
2023-10-16T13:40:57.835138Z_c16c472d-9699-49b1-87d3-d7e2177c2912  osd.9    *
2023-10-16T13:40:57.840230Z_78e52ea5-396a-46ae-bb91-1cece6f3473b  osd.9    *
2023-10-16T20:00:21.100773Z_83aba303-ddec-4ffa-aa95-c5eea6466165  osd.11   *
2023-10-16T21:58:17.103333Z_2d5f5025-95f5-44a6-bbc9-ea60fa9381ee  osd.7    *
2023-10-17T01:19:32.163215Z_8b418d76-3925-4818-8de5-3f5181b930be  osd.6    *
2023-10-17T01:19:32.178271Z_ef39c70c-cdef-4f7a-90ca-54938165b5df  osd.6    *
2023-10-17T01:40:06.571108Z_800aedb2-6092-4f01-89d9-cb214a9ec6e4  osd.21   *

{
    "assert_condition": "cur2 >= p.length",
    "assert_file": "/root/rpmbuild/BUILD/ceph/src/os/bluestore/BlueStore.h",
    "assert_func": "virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)",
    "assert_line": 3875,
    "assert_msg": "/root/rpmbuild/BUILD/ceph/src/os/bluestore/BlueStore.h: In function 'virtual void RocksDBBlueFSVolumeSelector::sub_usage(void*, const bluefs_fnode_t&)' thread 7fba288fb700 time 2023-10-17T01:40:06.473145+0000\n/root/rpmbuild/BUILD/ceph/src/os/bluestore/BlueStore.h: 3875: FAILED ceph_assert(cur2 >= p.length)\n",
    "assert_thread_name": "bstore_kv_sync",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12c20) [0x7fba3d320c20]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x55fc33a4d199]",
        "ceph-osd(+0x540362) [0x55fc33a4d362]",
        "(RocksDBBlueFSVolumeSelector::sub_usage(void*, bluefs_fnode_t const&)+0x15e) [0x55fc340fb6ee]",
        "(BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x74d) [0x55fc3418e5cd]",
        "(BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0x90) [0x55fc3418ea70]",
        "(BlueFS::fsync(BlueFS::FileWriter*)+0x181) [0x55fc341aa711]",
        "(BlueRocksWritableFile::Sync()+0x18) [0x55fc341bbb98]",
        "(rocksdb::LegacyWritableFileWrapper::Sync(rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x55fc3468537f]",
        "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x402) [0x55fc347a2e92]",
        "(rocksdb::WritableFileWriter::Sync(bool)+0x88) [0x55fc347a4558]",
        "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x30b) [0x55fc34696a1b]",
        "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x2687) [0x55fc3469f747]",
        "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21) [0x55fc3469f941]",
        "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x84) [0x55fc3463e304]",
        "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x9a) [0x55fc3463ed0a]",
        "(BlueStore::_kv_sync_thread()+0x2f78) [0x55fc340f9c38]",
        "(BlueStore::KVSyncThread::entry()+0x11) [0x55fc34121ab1]",
        "(Thread::entry_wrapper()+0x53) [0x55fc3421f4e3]",
        "/lib64/libpthread.so.0(+0x817a) [0x7fba3d31617a]",
        "clone()" 
    ],
    "ceph_version": "16.2.14-5.0.1",
    "crash_id": "2023-10-17T01:40:06.571108Z_800aedb2-6092-4f01-89d9-cb214a9ec6e4",
    "entity_name": "osd.21",
    "os_id": "centos",
    "os_name": "CentOS Linux",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-osd",
    "stack_sig": "d4133fd81fc283e10ef4edc13fc01336f8bd22579ecab0ea9c0653bda7f3ffca",
    "timestamp": "2023-10-17T01:40:06.571108Z",
    "utsname_hostname": "storage-001",
    "utsname_machine": "x86_64",
    "utsname_release": "5.4.61-050461-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#202008260931 SMP Wed Aug 26 09:34:29 UTC 2020" 
}

Just wanted to confirm that since we upgraded to 16.2.15 we dont see crashing osd's anymore.

Great, thanks for the update!