Project

General

Profile

Bug #2595

filestore: error creating filestore during mkcephfs

Added by Sage Weil almost 12 years ago. Updated almost 12 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

from ML:

From:     Stefan Priebe - Profihost AG <s.priebe@profihost.ag>

=== osd.32 ===
pushing conf and monmap to ssdstor002:/tmp/mkfs.ceph.4885
2012-06-14 14:02:09.333309 7f815cd26780 -1 journal FileJournal::_open: 
unable to open journal: open() failed: (2) No such file or directory
2012-06-14 14:02:09.690097 7f815cd26780 -1 filestore(/srv/osd.32) could 
not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or 
directory
2012-06-14 14:02:09.758456 7f815cd26780 -1 created object store 
/srv/osd.32 journal /journal/osd.32.journal for osd.32 fsid 
308b2ada-5c2a-4b54-b094-b7035413d247
creating private key for osd.32 keyring /etc/ceph/osd.32.keyring
creating /etc/ceph/osd.32.keyring
collecting osd.32 key

History

#1 Updated by Anonymous almost 12 years ago

FYI, I saw this once when I was working on the OSD hotplug code paths. Mine might have been caused by a missing "osd journal size", at the time I didn't want to get sidetracked and my memory of the event is already fuzzy.

#2 Updated by Stefan Priebe almost 12 years ago

Can you reproduce with 'debug filestore = 20' and attach the log to this
bug?

Log:

2012-06-15 22:51:13.217367 7fe0a4c1b780  1 filestore(/srv/osd.10) mkfs in /srv/osd.10
2012-06-15 22:51:13.217434 7fe0a4c1b780  1 filestore(/srv/osd.10) mkfs generated fsid c2505fd1-de2e-407c-a3bf-692b98e224b4
2012-06-15 22:51:13.220021 7fe0a4c1b780 10 filestore(/srv/osd.10) mkfs fsid is c2505fd1-de2e-407c-a3bf-692b98e224b4
2012-06-15 22:51:13.220091 7fe0a4c1b780  2 filestore(/srv/osd.10)  BTRFS_IOC_SUBVOL_CREATE ioctl failed, trying mkdir /srv/osd.10/current
2012-06-15 22:51:13.223389 7fe0a4c1b780  1 filestore(/srv/osd.10) leveldb db exists/created
2012-06-15 22:51:13.223458 7fe0a4c1b780 10 filestore(/srv/osd.10) open_journal at /journal/osd.10.journal
2012-06-15 22:51:13.224448 7fe0a4c1b780 -1 journal FileJournal::_open: unable to open journal: open() failed: (2) No such file or directory
2012-06-15 22:51:13.791658 7fe0a4c1b780  1 journal _open /journal/osd.10.journal fd 9: 2097152000 bytes, block size 4096 bytes, directio = 0, aio = 0
2012-06-15 22:51:13.791716 7fe0a4c1b780  0 filestore(/srv/osd.10) mkjournal created journal on /journal/osd.10.journal
2012-06-15 22:51:13.791729 7fe0a4c1b780  1 filestore(/srv/osd.10) mkfs done in /srv/osd.10
2012-06-15 22:51:13.791737 7fe0a4c1b780  5 filestore(/srv/osd.10) basedir /srv/osd.10 journal /journal/osd.10.journal
2012-06-15 22:51:13.791754 7fe0a4c1b780 10 filestore(/srv/osd.10) mount fsid is c2505fd1-de2e-407c-a3bf-692b98e224b4
2012-06-15 22:51:13.822836 7fe0a4c1b780  0 filestore(/srv/osd.10) mount FIEMAP ioctl is supported and appears to work
2012-06-15 22:51:13.822844 7fe0a4c1b780  0 filestore(/srv/osd.10) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2012-06-15 22:51:13.823005 7fe0a4c1b780  0 filestore(/srv/osd.10) mount did NOT detect btrfs
2012-06-15 22:51:13.863831 7fe0a4c1b780  0 filestore(/srv/osd.10) mount syncfs(2) syscall fully supported (by glibc and kernel)
2012-06-15 22:51:13.863908 7fe0a4c1b780  0 filestore(/srv/osd.10) mount found snaps <>
2012-06-15 22:51:13.863927 7fe0a4c1b780  5 filestore(/srv/osd.10) mount op_seq is 1
2012-06-15 22:51:13.865884 7fe0a4c1b780 20 filestore (init)dbobjectmap: seq is 1
2012-06-15 22:51:13.865892 7fe0a4c1b780 10 filestore(/srv/osd.10) open_journal at /journal/osd.10.journal
2012-06-15 22:51:13.865900 7fe0a4c1b780  0 filestore(/srv/osd.10) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2012-06-15 22:51:13.865904 7fe0a4c1b780 10 filestore(/srv/osd.10) list_collections
2012-06-15 22:51:13.865954 7fe0a4c1b780  1 journal _open /journal/osd.10.journal fd 17: 2097152000 bytes, block size 4096 bytes, directio = 0, aio = 0
2012-06-15 22:51:13.865957 7fe0a0995700 20 filestore(/srv/osd.10) sync_entry waiting for max_interval 5.000000
2012-06-15 22:51:13.865979 7fe0a4c1b780  1 journal _open /journal/osd.10.journal fd 17: 2097152000 bytes, block size 4096 bytes, directio = 0, aio = 0
2012-06-15 22:51:13.866053 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry start
2012-06-15 22:51:13.866058 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry sleeping
2012-06-15 22:51:13.866091 7fe0a4c1b780 15 filestore(/srv/osd.10) read meta/23c2fcde/osd_superblock/0//-1 0~0
2012-06-15 22:51:13.866106 7fe0a4c1b780 -1 filestore(/srv/osd.10) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2012-06-15 22:51:13.866114 7fe0a4c1b780 10 filestore(/srv/osd.10) FileStore::read(meta/23c2fcde/osd_superblock/0//-1) open error: (2) No such file or directory
2012-06-15 22:51:13.866126 7fe0a4c1b780 10 filestore(/srv/osd.10) apply queued
2012-06-15 22:51:13.866129 7fe0a4c1b780  5 filestore(/srv/osd.10) queue_transactions new osr(default 0x2654678)/0x2654678
2012-06-15 22:51:13.866137 7fe0a4c1b780  5 filestore(/srv/osd.10) queue_transactions (writeahead) 2 0x7fff7de89ba0
2012-06-15 22:51:13.866229 7fe09f993700  5 filestore(/srv/osd.10) _journaled_ahead 0x263d190 seq 2 osr(default 0x2654678) 0x7fff7de89ba0
2012-06-15 22:51:13.866260 7fe09f993700  5 filestore(/srv/osd.10) queue_op 0x263d190 seq 2 osr(default 0x2654678) 238 bytes   (queue has 1 ops and 238 bytes)
2012-06-15 22:51:13.866267 7fe09f192700  5 filestore(/srv/osd.10) _do_op 0x263d190 seq 2 osr(default 0x2654678)/0x2654678 start
2012-06-15 22:51:13.866273 7fe09f192700 10 filestore(/srv/osd.10) _do_transaction on 0x7fff7de89ba0
2012-06-15 22:51:13.866279 7fe09f192700 15 filestore(/srv/osd.10) create_collection /srv/osd.10/current/meta
2012-06-15 22:51:13.866306 7fe09f192700 10 filestore(/srv/osd.10) create_collection /srv/osd.10/current/meta = 0
2012-06-15 22:51:13.866388 7fe09f192700 15 filestore(/srv/osd.10) write meta/23c2fcde/osd_superblock/0//-1 0~106
2012-06-15 22:51:13.866487 7fe09f192700 10 filestore(/srv/osd.10) queue_flusher ep 0 fd 18 0~106 qlen 1
2012-06-15 22:51:13.866511 7fe09f192700 10 filestore(/srv/osd.10) write meta/23c2fcde/osd_superblock/0//-1 0~106 = 106
2012-06-15 22:51:13.866522 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry awoke
2012-06-15 22:51:13.866527 7fe09f192700 10 filestore(/srv/osd.10) _do_op 0x263d190 seq 2 r = 0, finisher 0x264fe10 0
2012-06-15 22:51:13.866533 7fe09e190700 10 filestore(/srv/osd.10) flusher_entry flushing+closing 18 ep 0
2012-06-15 22:51:13.866534 7fe09f192700 10 filestore(/srv/osd.10) _finish_op 0x263d190 seq 2 osr(default 0x2654678)/0x2654678
2012-06-15 22:51:13.866591 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry sleeping
2012-06-15 22:51:13.866596 7fe0a4c1b780 10 filestore(/srv/osd.10) apply done r = 0
2012-06-15 22:51:13.866598 7fe0a4c1b780 10 filestore(/srv/osd.10) sync_and_flush
2012-06-15 22:51:13.866600 7fe0a4c1b780 10 filestore(/srv/osd.10) _flush_op_queue draining op tp
2012-06-15 22:51:13.866601 7fe0a4c1b780 10 filestore(/srv/osd.10) _flush_op_queue waiting for apply finisher
2012-06-15 22:51:13.866602 7fe0a4c1b780 10 filestore(/srv/osd.10) sync_and_flush done
2012-06-15 22:51:13.932680 7fe0a4c1b780  5 filestore(/srv/osd.10) umount /srv/osd.10
2012-06-15 22:51:13.932717 7fe0a0995700 20 filestore(/srv/osd.10) sync_entry force_sync set
2012-06-15 22:51:13.932739 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry awoke
2012-06-15 22:51:13.932746 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry finish
2012-06-15 22:51:13.932752 7fe0a0995700 15 filestore(/srv/osd.10) sync_entry committing 2 sync_epoch 1
2012-06-15 22:51:13.932778 7fe0a0995700 15 filestore(/srv/osd.10) sync_entry doing a full sync (syncfs(2) if possible)
2012-06-15 22:51:13.933625 7fe0a0995700 10 filestore(/srv/osd.10) sync_entry commit took 0.000887, interval was 0.067667
2012-06-15 22:51:13.933653 7fe0a0995700 15 filestore(/srv/osd.10) sync_entry committed to op_seq 2
2012-06-15 22:51:13.933797 7fe0a4c1b780  1 journal close /journal/osd.10.journal
2012-06-15 22:51:13.934117 7fe0a4c1b780 -1 created object store /srv/osd.10 journal /journal/osd.10.journal for osd.10 fsid 72571270-b891-4dca-ad4d-19e4996dc509

#3 Updated by Sage Weil almost 12 years ago

Oh, it looks like it's just noise from checking the journal. The mkcephfs succeeded, right?

#4 Updated by Stefan Priebe almost 12 years ago

Yes

#6 Updated by Sage Weil almost 12 years ago

  • Status changed from 12 to Resolved

#7 Updated by Stefan Priebe almost 12 years ago

2012-06-18 17:42:16.232924 7f54292fb780 -1 filestore(/srv/osd.20) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory

Also available in: Atom PDF