2012-10-11 13:43:43.128929 7fa306f68780 0 ceph version 0.52 (commit:e48859474c4944d4ff201ddc9f5fd400e8898173), process ceph-osd, pid 1098 2012-10-11 13:43:43.177300 7fa306f68780 0 filestore(/data/osd.1) mount FIEMAP ioctl is NOT supported 2012-10-11 13:43:43.177314 7fa306f68780 0 filestore(/data/osd.1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2012-10-11 13:43:43.177431 7fa306f68780 0 filestore(/data/osd.1) mount did NOT detect btrfs 2012-10-11 13:43:43.185669 7fa306f68780 0 filestore(/data/osd.1) mount syncfs(2) syscall fully supported (by glibc and kernel) 2012-10-11 13:43:43.185779 7fa306f68780 0 filestore(/data/osd.1) mount found snaps <> 2012-10-11 13:43:43.205379 7fa306f68780 0 filestore(/data/osd.1) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2012-10-11 13:43:43.205625 7fa306f68780 1 journal _open /data/osd.1/journal fd 20: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:43.205865 7fa306f68780 1 journal _open /data/osd.1/journal fd 20: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:43.207890 7fa306f68780 1 journal close /data/osd.1/journal 2012-10-11 13:43:43.236392 7fa306f68780 0 filestore(/data/osd.1) mount FIEMAP ioctl is NOT supported 2012-10-11 13:43:43.236406 7fa306f68780 0 filestore(/data/osd.1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2012-10-11 13:43:43.236513 7fa306f68780 0 filestore(/data/osd.1) mount did NOT detect btrfs 2012-10-11 13:43:43.245203 7fa306f68780 0 filestore(/data/osd.1) mount syncfs(2) syscall fully supported (by glibc and kernel) 2012-10-11 13:43:43.245301 7fa306f68780 0 filestore(/data/osd.1) mount found snaps <> 2012-10-11 13:43:43.263736 7fa306f68780 0 filestore(/data/osd.1) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2012-10-11 13:43:43.263882 7fa306f68780 1 journal _open /data/osd.1/journal fd 29: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:43.264035 7fa306f68780 1 journal _open /data/osd.1/journal fd 29: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:54.233320 7fa2f2605700 0 -- 149.165.228.10:6805/1098 >> 149.165.228.10:6802/338 pipe(0x2c65d80 sd=31 :58515 pgs=1 cs=1 l=0).fault with nothing to send, going to standby 2012-10-11 13:43:58.479440 7fa2f2f07700 0 -- 149.165.228.10:0/1098 submit_message osd_ping(ping e8 stamp 2012-10-11 13:43:58.479426) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x2e24c40 2012-10-11 13:44:01.279837 7fa2f2f07700 0 -- 149.165.228.10:0/1098 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:01.279821) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x2eeae00 2012-10-11 13:44:01.780191 7fa2f2f07700 0 -- 149.165.228.10:0/1098 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:01.780177) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x34a71c0 2012-10-11 13:44:09.980528 7fa2f2f07700 0 -- 149.165.228.10:0/1098 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:09.980511) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x2cbf1c0 2012-10-11 13:44:11.523128 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:51.523127) 2012-10-11 13:44:12.280672 7fa2f2f07700 0 -- 149.165.228.10:0/1098 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:12.280656) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x2eb71c0 2012-10-11 13:44:12.280698 7fa2f2f07700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:52.280697) 2012-10-11 13:44:12.523310 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:52.523309) 2012-10-11 13:44:13.523501 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:53.523500) 2012-10-11 13:44:14.523618 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:54.523617) 2012-10-11 13:44:15.523761 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:55.523761) 2012-10-11 13:44:16.523902 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:56.523901) 2012-10-11 13:44:17.524142 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:57.524141) 2012-10-11 13:44:18.180837 7fa2f2f07700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:58.180837) 2012-10-11 13:44:18.524262 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:58.524261) 2012-10-11 13:44:19.280972 7fa2f2f07700 0 -- 149.165.228.10:0/1098 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:19.280958) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x2e24700 2012-10-11 13:44:19.280991 7fa2f2f07700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:59.280990) 2012-10-11 13:44:19.524380 7fa300022700 -1 osd.1 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:50.879138 (cutoff 2012-10-11 13:43:59.524379) 2012-10-11 13:44:24.526092 7fa300022700 0 log [WRN] : 4 slow requests, 4 included below; oldest blocked for > 30.289687 secs 2012-10-11 13:44:24.526159 7fa300022700 0 log [WRN] : slow request 30.289687 seconds old, received at 2012-10-11 13:43:54.236279: osd_op(mds.0.1:26 mds_snaptable [writefull 0~41] 1.d90270ad RETRY) v4 currently reached pg 2012-10-11 13:44:24.526167 7fa300022700 0 log [WRN] : slow request 30.289475 seconds old, received at 2012-10-11 13:43:54.236491: osd_op(mds.0.1:11 600.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.c2e541b0 RETRY) v4 currently reached pg 2012-10-11 13:44:24.526173 7fa300022700 0 log [WRN] : slow request 30.289452 seconds old, received at 2012-10-11 13:43:54.236514: osd_op(mds.0.1:13 602.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.2a8b6bd0 RETRY) v4 currently delayed 2012-10-11 13:44:24.526178 7fa300022700 0 log [WRN] : slow request 30.289435 seconds old, received at 2012-10-11 13:43:54.236531: osd_op(mds.0.1:14 603.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.7a4d91b0 RETRY) v4 currently reached pg 2012-10-11 13:44:54.534754 7fa300022700 0 log [WRN] : 4 slow requests, 4 included below; oldest blocked for > 60.298406 secs 2012-10-11 13:44:54.534765 7fa300022700 0 log [WRN] : slow request 60.298406 seconds old, received at 2012-10-11 13:43:54.236279: osd_op(mds.0.1:26 mds_snaptable [writefull 0~41] 1.d90270ad RETRY) v4 currently reached pg 2012-10-11 13:44:54.534772 7fa300022700 0 log [WRN] : slow request 60.298194 seconds old, received at 2012-10-11 13:43:54.236491: osd_op(mds.0.1:11 600.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.c2e541b0 RETRY) v4 currently reached pg 2012-10-11 13:44:54.534776 7fa300022700 0 log [WRN] : slow request 60.298171 seconds old, received at 2012-10-11 13:43:54.236514: osd_op(mds.0.1:13 602.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.2a8b6bd0 RETRY) v4 currently delayed 2012-10-11 13:44:54.534781 7fa300022700 0 log [WRN] : slow request 60.298154 seconds old, received at 2012-10-11 13:43:54.236531: osd_op(mds.0.1:14 603.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.7a4d91b0 RETRY) v4 currently reached pg 2012-10-11 13:45:54.550543 7fa300022700 0 log [WRN] : 4 slow requests, 4 included below; oldest blocked for > 120.314194 secs 2012-10-11 13:45:54.550553 7fa300022700 0 log [WRN] : slow request 120.314194 seconds old, received at 2012-10-11 13:43:54.236279: osd_op(mds.0.1:26 mds_snaptable [writefull 0~41] 1.d90270ad RETRY) v4 currently reached pg 2012-10-11 13:45:54.550560 7fa300022700 0 log [WRN] : slow request 120.313982 seconds old, received at 2012-10-11 13:43:54.236491: osd_op(mds.0.1:11 600.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.c2e541b0 RETRY) v4 currently reached pg 2012-10-11 13:45:54.550566 7fa300022700 0 log [WRN] : slow request 120.313959 seconds old, received at 2012-10-11 13:43:54.236514: osd_op(mds.0.1:13 602.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.2a8b6bd0 RETRY) v4 currently delayed 2012-10-11 13:45:54.550573 7fa300022700 0 log [WRN] : slow request 120.313942 seconds old, received at 2012-10-11 13:43:54.236531: osd_op(mds.0.1:14 603.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.7a4d91b0 RETRY) v4 currently reached pg