* Kernel RBD hang on OSD Failure
@ 2015-12-07 20:56 Matt Conner
[not found] ` <CAKeiORSsNQ-LOWQ0heLZQkekN_mwYQ81XSXfdJe-R1eP1Osz1w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-08 10:59 ` Ilya Dryomov
0 siblings, 2 replies; 10+ messages in thread
From: Matt Conner @ 2015-12-07 20:56 UTC (permalink / raw)
To: ceph-devel
Hi,
We have a Ceph cluster in which we have been having issues with RBD
clients hanging when an OSD failure occurs. We are using a NAS gateway
server which maps RBD images to filesystems and serves the filesystems
out via NFS. The gateway server has close to 180 NFS clients and
almost every time even 1 OSD goes down during heavy load, the NFS
exports lock up and the clients are unable to access the NAS share via
NFS. When the OSD fails, Ceph recovers without issue, but the gateway
kernel RBD module appears to get stuck waiting on the now failed OSD.
Note that this works correctly when under lighter loads.
From what we have been able to determine, the NFS server daemon hangs
waiting for I/O from the OSD that went out and never recovers.
Similarly, attempting to access files from the exported FS locally on
the gateway server will result in a similar hang. We also noticed that
Ceph health details will continue to report blocked I/O on the now
down OSD until either the OSD is recovered or the gateway server is
rebooted. Based on a few kernel logs from NFS and PVS, we were able
to trace the problem to the RBD kernel module.
Unfortunately, the only way we have been able to recover our gateway
is by hard rebooting the server.
Has anyone else encountered this issue and/or have a possible solution?
Are there suggestions for getting more detailed debugging information
from the RBD kernel module?
Few notes on our setup:
We are using Kernel RBD on a gateway server that exports filesystems via NFS
The exported filesystems are XFS on LVMs which are each composed of 16
striped images (NFS->LVM->XFS->PVS->RBD)
There are currently 176 mapped RBD images on the server (11
filesystems, 16 mapped RBD images per FS)
Gateway Kernel: 3.18.6
Ceph version: 0.80.9
Note - We've tried using different kernels all the way up to 4.3.0 but
the problem persists.
Thanks,
Matt Conner
Keeper Technology
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
[not found] ` <CAKeiORSsNQ-LOWQ0heLZQkekN_mwYQ81XSXfdJe-R1eP1Osz1w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-07 21:55 ` Blair Bethwaite
[not found] ` <CA+z5DswarP0KHomWJocCxT29W1-kyzV2RhDOg7tc-0_DV0D1Kg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 10+ messages in thread
From: Blair Bethwaite @ 2015-12-07 21:55 UTC (permalink / raw)
To: Matt Conner
Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA, ceph-users-idqoXFIVOFJgJs9I8MT0rw
Hi Matt,
(CC'ing in ceph-users too - similar reports there:
http://www.spinics.net/lists/ceph-users/msg23037.html)
We've seen something similar for KVM [lib]RBD clients acting as NFS
gateways within our OpenStack cloud, the NFS services were locking up
and causing client timeouts whenever we started doing Ceph
maintenance. We eventually realised we'd somehow set the pool min_size
== size, so any single OSD outage was blocking client IO - *oops*.
Your issue sounds like something different, but NFS does seem to be
very touchy and lacking any graceful recovery from issues with the
underlying FS.
On 8 December 2015 at 07:56, Matt Conner <matt.conner-yfee5r9JHcEqDJ6do+/SaQ@public.gmane.org> wrote:
> Hi,
>
> We have a Ceph cluster in which we have been having issues with RBD
> clients hanging when an OSD failure occurs. We are using a NAS gateway
> server which maps RBD images to filesystems and serves the filesystems
> out via NFS. The gateway server has close to 180 NFS clients and
> almost every time even 1 OSD goes down during heavy load, the NFS
> exports lock up and the clients are unable to access the NAS share via
> NFS. When the OSD fails, Ceph recovers without issue, but the gateway
> kernel RBD module appears to get stuck waiting on the now failed OSD.
> Note that this works correctly when under lighter loads.
>
> From what we have been able to determine, the NFS server daemon hangs
> waiting for I/O from the OSD that went out and never recovers.
> Similarly, attempting to access files from the exported FS locally on
> the gateway server will result in a similar hang. We also noticed that
> Ceph health details will continue to report blocked I/O on the now
> down OSD until either the OSD is recovered or the gateway server is
> rebooted. Based on a few kernel logs from NFS and PVS, we were able
> to trace the problem to the RBD kernel module.
>
> Unfortunately, the only way we have been able to recover our gateway
> is by hard rebooting the server.
>
> Has anyone else encountered this issue and/or have a possible solution?
> Are there suggestions for getting more detailed debugging information
> from the RBD kernel module?
>
>
> Few notes on our setup:
> We are using Kernel RBD on a gateway server that exports filesystems via NFS
> The exported filesystems are XFS on LVMs which are each composed of 16
> striped images (NFS->LVM->XFS->PVS->RBD)
> There are currently 176 mapped RBD images on the server (11
> filesystems, 16 mapped RBD images per FS)
> Gateway Kernel: 3.18.6
> Ceph version: 0.80.9
> Note - We've tried using different kernels all the way up to 4.3.0 but
> the problem persists.
>
> Thanks,
> Matt Conner
> Keeper Technology
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Cheers,
~Blairo
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
[not found] ` <CA+z5DswarP0KHomWJocCxT29W1-kyzV2RhDOg7tc-0_DV0D1Kg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-08 9:57 ` Tom Christensen
2015-12-08 10:35 ` [ceph-users] " Ilya Dryomov
0 siblings, 1 reply; 10+ messages in thread
From: Tom Christensen @ 2015-12-08 9:57 UTC (permalink / raw)
To: ceph-users-idqoXFIVOFJgJs9I8MT0rw; +Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1.1: Type: text/plain, Size: 4619 bytes --]
We aren't running NFS, but regularly use the kernel driver to map RBDs and
mount filesystems in same. We see very similar behavior across nearly all
kernel versions we've tried. In my experience only very few versions of
the kernel driver survive any sort of crush map change/update while
something is mapped. In fact in the last 2 years I think I've only seen
this work on 1 kernel version unfortunately its badly out of date and we
can't run it in our environment anymore, I think it was a 3.0 kernel
version running on ubuntu 12.04. We have just recently started trying to
find a kernel that will survive OSD outages or changes to the cluster.
We're on ubuntu 14.04, and have tried 3.16, 3.19.0-25, 4.3, and 4.2 without
success in the last week. We only map 1-3 RBDs per client machine at a
time but we regularly will get processes stuck in D state which are
accessing the filesystem inside the RBD and will have to hard reboot the
RBD client machine. This is always associated with a cluster change in
some way, reweighting OSDs, rebooting an OSD host, restarting an individual
OSD, adding OSDs, and removing OSDs all cause the kernel client to hang.
If no change is made to the cluster, the kernel client will be happy for
weeks.
On Mon, Dec 7, 2015 at 2:55 PM, Blair Bethwaite <blair.bethwaite-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> Hi Matt,
>
> (CC'ing in ceph-users too - similar reports there:
> http://www.spinics.net/lists/ceph-users/msg23037.html)
>
> We've seen something similar for KVM [lib]RBD clients acting as NFS
> gateways within our OpenStack cloud, the NFS services were locking up
> and causing client timeouts whenever we started doing Ceph
> maintenance. We eventually realised we'd somehow set the pool min_size
> == size, so any single OSD outage was blocking client IO - *oops*.
> Your issue sounds like something different, but NFS does seem to be
> very touchy and lacking any graceful recovery from issues with the
> underlying FS.
>
>
> On 8 December 2015 at 07:56, Matt Conner <matt.conner-yfee5r9JHcEqDJ6do+/SaQ@public.gmane.org>
> wrote:
> > Hi,
> >
> > We have a Ceph cluster in which we have been having issues with RBD
> > clients hanging when an OSD failure occurs. We are using a NAS gateway
> > server which maps RBD images to filesystems and serves the filesystems
> > out via NFS. The gateway server has close to 180 NFS clients and
> > almost every time even 1 OSD goes down during heavy load, the NFS
> > exports lock up and the clients are unable to access the NAS share via
> > NFS. When the OSD fails, Ceph recovers without issue, but the gateway
> > kernel RBD module appears to get stuck waiting on the now failed OSD.
> > Note that this works correctly when under lighter loads.
> >
> > From what we have been able to determine, the NFS server daemon hangs
> > waiting for I/O from the OSD that went out and never recovers.
> > Similarly, attempting to access files from the exported FS locally on
> > the gateway server will result in a similar hang. We also noticed that
> > Ceph health details will continue to report blocked I/O on the now
> > down OSD until either the OSD is recovered or the gateway server is
> > rebooted. Based on a few kernel logs from NFS and PVS, we were able
> > to trace the problem to the RBD kernel module.
> >
> > Unfortunately, the only way we have been able to recover our gateway
> > is by hard rebooting the server.
> >
> > Has anyone else encountered this issue and/or have a possible solution?
> > Are there suggestions for getting more detailed debugging information
> > from the RBD kernel module?
> >
> >
> > Few notes on our setup:
> > We are using Kernel RBD on a gateway server that exports filesystems via
> NFS
> > The exported filesystems are XFS on LVMs which are each composed of 16
> > striped images (NFS->LVM->XFS->PVS->RBD)
> > There are currently 176 mapped RBD images on the server (11
> > filesystems, 16 mapped RBD images per FS)
> > Gateway Kernel: 3.18.6
> > Ceph version: 0.80.9
> > Note - We've tried using different kernels all the way up to 4.3.0 but
> > the problem persists.
> >
> > Thanks,
> > Matt Conner
> > Keeper Technology
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Cheers,
> ~Blairo
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
[-- Attachment #1.2: Type: text/html, Size: 5927 bytes --]
[-- Attachment #2: Type: text/plain, Size: 178 bytes --]
_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [ceph-users] Kernel RBD hang on OSD Failure
2015-12-08 9:57 ` Tom Christensen
@ 2015-12-08 10:35 ` Ilya Dryomov
[not found] ` <CAOi1vP-Aw4oWV6eC2h7ZkW87FycnBm0XjMMo0T2updLDjMNULQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 10+ messages in thread
From: Ilya Dryomov @ 2015-12-08 10:35 UTC (permalink / raw)
To: Tom Christensen; +Cc: ceph-users, ceph-devel
On Tue, Dec 8, 2015 at 10:57 AM, Tom Christensen <pavera@gmail.com> wrote:
> We aren't running NFS, but regularly use the kernel driver to map RBDs and
> mount filesystems in same. We see very similar behavior across nearly all
> kernel versions we've tried. In my experience only very few versions of the
> kernel driver survive any sort of crush map change/update while something is
> mapped. In fact in the last 2 years I think I've only seen this work on 1
> kernel version unfortunately its badly out of date and we can't run it in
> our environment anymore, I think it was a 3.0 kernel version running on
> ubuntu 12.04. We have just recently started trying to find a kernel that
> will survive OSD outages or changes to the cluster. We're on ubuntu 14.04,
> and have tried 3.16, 3.19.0-25, 4.3, and 4.2 without success in the last
> week. We only map 1-3 RBDs per client machine at a time but we regularly
> will get processes stuck in D state which are accessing the filesystem
> inside the RBD and will have to hard reboot the RBD client machine. This is
> always associated with a cluster change in some way, reweighting OSDs,
> rebooting an OSD host, restarting an individual OSD, adding OSDs, and
> removing OSDs all cause the kernel client to hang. If no change is made to
> the cluster, the kernel client will be happy for weeks.
There are a couple of known bugs in the remap/resubmit area, but those
are supposedly corner cases (like *all* the OSDs going down and then
back up, etc). I had no idea it was that severe and goes that back.
Apparently triggering it requires a heavier load, as we've never seen
anything like that in our tests.
For unrelated reasons, remap/resubmit code is getting entirely
rewritten for kernel 4.5, so, if you've been dealing with this issue
for the last two years (I don't remember seeing any tickets listing
that many kernel versions and not mentioning NFS), I'm afraid the best
course of action for you would be to wait for 4.5 to come out and try
it. If you'd be willing to test out an early version on one of more of
your client boxes, I can ping you when it's ready.
I'll take a look at 3.0 vs 3.16 with an eye on remap code. Did you
happen to try 3.10?
It sounds like you can reproduce this pretty easily. Can you get it to
lock up and do:
# cat /sys/kernel/debug/ceph/*/osdmap
# cat /sys/kernel/debug/ceph/*/osdc
$ ceph status
and bunch of times? I have a hunch that kernel client simply fails to
request enough of new osdmaps after the cluster topology changes under
load.
Thanks,
Ilya
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
2015-12-07 20:56 Kernel RBD hang on OSD Failure Matt Conner
[not found] ` <CAKeiORSsNQ-LOWQ0heLZQkekN_mwYQ81XSXfdJe-R1eP1Osz1w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-08 10:59 ` Ilya Dryomov
2015-12-10 23:16 ` Matt Conner
1 sibling, 1 reply; 10+ messages in thread
From: Ilya Dryomov @ 2015-12-08 10:59 UTC (permalink / raw)
To: Matt Conner; +Cc: ceph-devel
On Mon, Dec 7, 2015 at 9:56 PM, Matt Conner <matt.conner@keepertech.com> wrote:
> Hi,
>
> We have a Ceph cluster in which we have been having issues with RBD
> clients hanging when an OSD failure occurs. We are using a NAS gateway
> server which maps RBD images to filesystems and serves the filesystems
> out via NFS. The gateway server has close to 180 NFS clients and
> almost every time even 1 OSD goes down during heavy load, the NFS
> exports lock up and the clients are unable to access the NAS share via
> NFS. When the OSD fails, Ceph recovers without issue, but the gateway
> kernel RBD module appears to get stuck waiting on the now failed OSD.
> Note that this works correctly when under lighter loads.
>
> From what we have been able to determine, the NFS server daemon hangs
> waiting for I/O from the OSD that went out and never recovers.
> Similarly, attempting to access files from the exported FS locally on
> the gateway server will result in a similar hang. We also noticed that
> Ceph health details will continue to report blocked I/O on the now
> down OSD until either the OSD is recovered or the gateway server is
> rebooted. Based on a few kernel logs from NFS and PVS, we were able
> to trace the problem to the RBD kernel module.
>
> Unfortunately, the only way we have been able to recover our gateway
> is by hard rebooting the server.
>
> Has anyone else encountered this issue and/or have a possible solution?
> Are there suggestions for getting more detailed debugging information
> from the RBD kernel module?
Dumping osdc, osdmap and ceph status when it gets stuck would be
a start:
# cat /sys/kernel/debug/ceph/*/osdmap
# cat /sys/kernel/debug/ceph/*/osdc
$ ceph status
Thanks,
Ilya
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
2015-12-08 10:59 ` Ilya Dryomov
@ 2015-12-10 23:16 ` Matt Conner
2015-12-10 23:58 ` Ilya Dryomov
0 siblings, 1 reply; 10+ messages in thread
From: Matt Conner @ 2015-12-10 23:16 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: ceph-devel
Hi Ilya,
Took me a little time to reproduce, but I'm once again got into the
state. In the past I had reproduced the issue with a single OSD
failure but in this case I failed an entire server. Ignore the "noout"
flag as I set it to prevent rebalance during my attempts to reproduce
the issue.
osdmap:
epoch 6191
flags
pool 0 pg_num 64 (63) read_tier -1 write_tier -1
pool 1 pg_num 64 (63) read_tier -1 write_tier -1
pool 2 pg_num 64 (63) read_tier -1 write_tier -1
pool 3 pg_num 2048 (2047) read_tier -1 write_tier -1
pool 5 pg_num 2048 (2047) read_tier -1 write_tier -1
pool 6 pg_num 2048 (2047) read_tier -1 write_tier -1
pool 7 pg_num 2048 (2047) read_tier -1 write_tier -1
osd0 10.10.30.55:6830 100% (exists, up) 100%
osd1 10.10.30.53:6809 100% (exists, up) 100%
osd2 10.10.30.54:6839 100% (exists, up) 100%
osd3 10.10.30.56:6833 100% (exists, up) 100%
osd4 10.10.30.55:6848 100% (exists, up) 100%
osd5 10.10.30.53:6854 100% (exists, up) 100%
osd6 10.10.30.56:6818 100% (exists, up) 100%
osd7 10.10.30.54:6830 100% (exists, up) 100%
osd8 10.10.30.55:6839 100% (exists, up) 100%
osd9 10.10.30.56:6845 100% (exists, up) 100%
osd10 10.10.30.53:6860 100% (exists, up) 100%
osd11 10.10.30.54:6842 100% (exists, up) 100%
osd12 10.10.30.55:6836 100% (exists, up) 100%
osd13 10.10.30.56:6863 100% (exists, up) 100%
osd14 10.10.30.53:6839 100% (exists, up) 100%
osd15 10.10.30.54:6866 100% (exists, up) 100%
osd16 10.10.30.55:6869 100% (exists, up) 100%
osd17 10.10.30.56:6812 100% (exists, up) 100%
osd18 10.10.30.53:6824 100% (exists, up) 100%
osd19 10.10.30.54:6818 100% (exists, up) 100%
osd20 10.10.30.55:6833 100% (exists, up) 100%
osd21 10.10.30.56:6869 100% (exists, up) 100%
osd22 10.10.30.53:6857 100% (exists, up) 100%
osd23 10.10.30.54:6833 100% (exists, up) 100%
osd24 10.10.30.55:6803 100% (exists, up) 100%
osd25 10.10.30.56:6821 100% (exists, up) 100%
osd26 10.10.30.54:6848 100% (exists, up) 100%
osd27 10.10.30.53:6848 100% (exists, up) 100%
osd28 10.10.30.55:6863 100% (exists, up) 100%
osd29 10.10.30.56:6827 100% (exists, up) 100%
osd30 10.10.30.54:6836 100% (exists, up) 100%
osd31 10.10.30.53:6863 100% (exists, up) 100%
osd32 10.10.30.55:6845 100% (exists, up) 100%
osd33 10.10.30.56:6839 100% (exists, up) 100%
osd34 10.10.30.54:6851 100% (exists, up) 100%
osd35 10.10.30.53:6845 100% (exists, up) 100%
osd36 10.10.30.55:6854 100% (exists, up) 100%
osd37 10.10.30.56:6851 100% (exists, up) 100%
osd38 10.10.30.54:6806 100% (exists, up) 100%
osd39 10.10.30.53:6824 0% (doesn't exist) 100%
osd40 10.10.30.55:6815 100% (exists, up) 100%
osd41 10.10.30.56:6842 100% (exists, up) 100%
osd42 10.10.30.54:6863 100% (exists, up) 100%
osd43 10.10.30.53:6806 100% (exists, up) 100%
osd44 10.10.30.55:6821 100% (exists, up) 100%
osd45 10.10.30.56:6803 100% (exists, up) 100%
osd46 10.10.30.54:6812 100% (exists, up) 100%
osd47 10.10.30.53:6866 100% (exists, up) 100%
osd48 10.10.30.55:6812 100% (exists, up) 100%
osd49 10.10.30.56:6815 100% (exists, up) 100%
osd50 10.10.30.53:6842 100% (exists, up) 100%
osd51 10.10.30.54:6854 100% (exists, up) 100%
osd52 10.10.30.55:6842 100% (exists, up) 100%
osd53 10.10.30.56:6836 100% (exists, up) 100%
osd54 10.10.30.53:6803 100% (exists, up) 100%
osd55 10.10.30.54:6869 100% (exists, up) 100%
osd56 10.10.30.55:6866 100% (exists, up) 100%
osd57 10.10.30.56:6860 100% (exists, up) 100%
osd58 10.10.30.53:6800 100% (exists, up) 100%
osd59 10.10.30.54:6803 100% (exists, up) 100%
osd60 10.10.30.55:6824 100% (exists, up) 100%
osd61 10.10.30.56:6804 100% (exists, up) 100%
osd62 10.10.30.53:6815 100% (exists, up) 100%
osd63 10.10.30.54:6857 100% (exists, up) 100%
osd64 10.10.30.55:6800 100% (exists, up) 100%
osd65 10.10.30.56:6866 100% (exists, up) 100%
osd66 10.10.30.53:6836 100% (exists, up) 100%
osd67 10.10.30.54:6845 100% (exists, up) 100%
osd68 10.10.30.55:6860 100% (exists, up) 100%
osd69 10.10.30.56:6824 100% (exists, up) 100%
osd70 10.10.30.53:6821 100% (exists, up) 100%
osd71 10.10.30.54:6809 100% (exists, up) 100%
osd72 10.10.30.55:6806 100% (exists, up) 100%
osd73 10.10.30.56:6809 100% (exists, up) 100%
osd74 10.10.30.53:6818 100% (exists, up) 100%
osd75 10.10.30.54:6815 100% (exists, up) 100%
osd76 10.10.30.55:6827 100% (exists, up) 100%
osd77 10.10.30.56:6830 100% (exists, up) 100%
osd78 10.10.30.53:6833 100% (exists, up) 100%
osd79 10.10.30.54:6800 100% (exists, up) 100%
osd80 10.10.30.55:6851 100% (exists, up) 100%
osd81 10.10.30.56:6800 100% (exists, up) 100%
osd82 10.10.30.53:6812 100% (exists, up) 100%
osd83 10.10.30.54:6821 100% (exists, up) 100%
osd84 10.10.30.55:6857 100% (exists, up) 100%
osd85 10.10.30.56:6848 100% (exists, up) 100%
osd86 10.10.30.53:6830 100% (exists, up) 100%
osd87 10.10.30.54:6827 100% (exists, up) 100%
osd88 10.10.30.55:6809 100% (exists, up) 100%
osd89 10.10.30.56:6854 100% (exists, up) 100%
osd90 10.10.30.53:6827 100% (exists, up) 100%
osd91 10.10.30.54:6860 100% (exists, up) 100%
osd92 10.10.30.55:6818 100% (exists, up) 100%
osd93 10.10.30.56:6857 100% (exists, up) 100%
osd94 10.10.30.53:6851 100% (exists, up) 100%
osd95 10.10.30.54:6824 100% (exists, up) 100%
osd96 10.10.30.52:6863 100% (exists, up) 100%
osd97 10.10.30.51:6841 100% (exists) 100%
osd98 10.10.30.52:6854 100% (exists, up) 100%
osd99 10.10.30.51:6844 100% (exists) 100%
osd100 10.10.30.52:6809 100% (exists, up) 100%
osd101 10.10.30.51:6853 100% (exists) 100%
osd102 10.10.30.52:6857 100% (exists, up) 100%
osd103 10.10.30.51:6807 100% (exists) 100%
osd104 10.10.30.51:6810 100% (exists) 100%
osd105 10.10.30.52:6827 100% (exists, up) 100%
osd106 10.10.30.52:6881 100% (exists, up) 100%
osd107 10.10.30.51:6801 100% (exists) 100%
osd108 10.10.30.52:6803 100% (exists, up) 100%
osd109 10.10.30.51:6835 100% (exists) 100%
osd110 10.10.30.52:6878 100% (exists, up) 100%
osd111 10.10.30.51:6825 100% (exists) 100%
osd112 10.10.30.51:6804 100% (exists) 100%
osd113 10.10.30.52:6860 100% (exists, up) 100%
osd114 10.10.30.51:6838 100% (exists) 100%
osd115 10.10.30.52:6845 100% (exists, up) 100%
osd116 10.10.30.52:6872 100% (exists, up) 100%
osd117 10.10.30.51:6862 100% (exists) 100%
osd118 10.10.30.52:6806 100% (exists, up) 100%
osd119 10.10.30.51:6832 100% (exists) 100%
osd120 10.10.30.52:6842 100% (exists, up) 100%
osd121 10.10.30.51:6816 100% (exists) 100%
osd122 10.10.30.52:6824 100% (exists, up) 100%
osd123 10.10.30.51:6859 100% (exists) 100%
osd124 10.10.30.52:6839 100% (exists, up) 100%
osd125 10.10.30.51:6868 100% (exists) 100%
osd126 10.10.30.51:6829 100% (exists) 100%
osd127 10.10.30.52:6848 100% (exists, up) 100%
osd128 10.10.30.51:6883 100% (exists) 100%
osd129 10.10.30.52:6836 100% (exists, up) 100%
osd130 10.10.30.52:6812 100% (exists, up) 100%
osd131 10.10.30.51:6819 100% (exists) 100%
osd132 10.10.30.52:6821 100% (exists, up) 100%
osd133 10.10.30.51:6874 100% (exists) 100%
osd134 10.10.30.52:6828 100% (exists, up) 100%
osd135 10.10.30.51:6871 100% (exists) 100%
osd136 10.10.30.52:6833 100% (exists, up) 100%
osd137 10.10.30.51:6813 100% (exists) 100%
osd138 10.10.30.52:6869 100% (exists, up) 100%
osd139 10.10.30.51:6865 100% (exists) 100%
osd140 10.10.30.52:6875 100% (exists, up) 100%
osd141 10.10.30.51:6822 100% (exists) 100%
osd142 10.10.30.52:6800 100% (exists, up) 100%
osd143 10.10.30.51:6880 100% (exists) 100%
osd144 10.10.30.52:6851 100% (exists, up) 100%
osd145 10.10.30.51:6877 100% (exists) 100%
osd146 10.10.30.52:6815 100% (exists, up) 100%
osd147 10.10.30.51:6850 100% (exists) 100%
osd148 10.10.30.52:6866 100% (exists, up) 100%
osd149 10.10.30.51:6800 100% (exists) 100%
osd150 10.10.30.52:6818 100% (exists, up) 100%
osd151 10.10.30.51:6856 100% (exists) 100%
osdc:
2064277 osd136 3.79a01658 rbd_data.f659f6ca5381e.0000000000003e0d read
2066619 osd29 3.a54ffb5 rbd_data.f54f37207fb18.0000000000000013 read
2067420 osd29 3.8ca62e20 rbd_data.f54f921268b0f.0000000000001a0f read
2068773 osd136 3.6145463c rbd_data.f65ad4ab53d2.0000000000002c0c read
2069250 osd29 3.a54ffb5 rbd_data.f54f37207fb18.0000000000000013
set-alloc-hint,write
2069272 osd136 3.e9f670ce rbd_data.12621d18fb34.00000000000011c7 read
2069838 osd136 3.5c533eb5 rbd_data.f65adcf2a53f.0000000000000000 read
2069872 osd136 3.5c533eb5 rbd_data.f65adcf2a53f.0000000000000000 read
2069880 osd136 3.b4d593da rbd_data.f663471f4e3a4.0000000000000035
set-alloc-hint,write
2069994 osd43 3.485e015f rbd_header.f42e85cd8f5d2 192438'26551487823872 watch
2069995 osd23 3.967f0ac rbd_header.f54f943ff3510 271821'26551487823872 watch
2069996 osd22 3.1679d3fd rbd_header.f54f346170e75 248651'26551487823872 watch
2069997 osd132 3.4c72415a rbd_header.f659f6ca5381e 212921'26551487823872 watch
2069998 osd40 3.dcb615cd rbd_header.f659f3b6599ce 281696'26551487823872 watch
2069999 osd4 3.2c5dc7d9 rbd_header.f659f7b8275f8 265762'26551487823872 watch
2070000 osd29 3.49c607ca rbd_header.f66343de35900 230429'26551487823872 watch
2070001 osd140 3.a030d1d8 rbd_header.f66341d1ccf35 290629'26551487823872 watch
2070002 osd45 3.ce94805a rbd_header.f66345f368639 295864'26551487823872 watch
2070003 osd52 3.854e4dfc rbd_header.593b6491ec9f 280116'26551487823872 watch
2070004 osd38 3.5cce84bc rbd_header.f42e822939526 163677'26551487823872 watch
2070005 osd47 3.13094d0 rbd_header.f42e869ead798 165850'26551487823872 watch
2070006 osd92 3.b72d8628 rbd_header.f42e8534aeb08 220526'26551487823872 watch
2070007 osd41 3.a44fd23b rbd_header.1262487bcab5 170538'26551487823872 watch
2070008 osd0 3.184871e7 rbd_header.12622ffb8aff 221268'26551487823872 watch
2070009 osd73 3.41d03b49 rbd_header.12628c3de5a 188527'26551487823872 watch
2070010 osd38 3.b63d7b6c rbd_header.126253d67973 116592'26551487823872 watch
2070011 osd115 3.e6b3a4e rbd_header.1262614d8810 231173'26551487823872 watch
2070012 osd72 3.d9ede1c3 rbd_header.1262115f0cf 140197'26551487823872 watch
2070013 osd12 3.6642c157 rbd_header.12621e7fc115 187593'26551487823872 watch
2070014 osd91 3.b3705118 rbd_header.593b7323f25f 243419'26551487823872 watch
2070015 osd57 3.abcf9bf6 rbd_header.593b363a4411 209538'26551487823872 watch
2070016 osd53 3.72c67cfe rbd_header.593b6072b1c7 291406'26551487823872 watch
2070017 osd70 3.10504835 rbd_header.593b35e9ba2b 205415'26551487823872 watch
2070018 osd45 3.8b381aaa rbd_header.593b418e8823 272511'26551487823872 watch
2070019 osd142 3.2b5318b7 rbd_header.593b46f1aa31 210594'26551487823872 watch
2070020 osd68 3.25c6ad00 rbd_header.4e449416a8478 72833'26551487823872 watch
2070021 osd57 3.3df2d8ae rbd_header.4e44968299a31 74608'26551487823872 watch
2070022 osd13 3.5102bd35 rbd_header.4e449723d5105 87452'26551487823872 watch
2070023 osd46 3.d0ad9250 rbd_header.4e449104e5eea 75590'26551487823872 watch
2070024 osd148 3.7fb7b2bd rbd_header.4e4491321fb8d 68317'26551487823872 watch
2070025 osd18 3.f82976fd rbd_header.4e449402e564a 57095'26551487823872 watch
2070026 osd10 3.39b4da2e rbd_header.4e44914acd83 60123'26551487823872 watch
2070027 osd2 3.8fe8ba4d rbd_header.e6ec56f5dcb51 200753'26551487823872 watch
2070028 osd6 3.66d6ca30 rbd_header.e6ec57072b44a 235673'26551487823872 watch
2070029 osd80 3.25b25299 rbd_header.e6ec57e4a0ef9 188214'26551487823872 watch
2070030 osd46 3.58e2250 rbd_header.e6ec550078b66 190030'26551487823872 watch
2070031 osd116 3.b29b37d3 rbd_header.e6ec521c83559 180643'26551487823872 watch
2070032 osd80 3.eca20f5e rbd_header.e6ec570293241 160701'26551487823872 watch
2070033 osd80 3.b20df8a9 rbd_header.e6ec53b631e5a 144384'26551487823872 watch
2070034 osd40 3.6cb419d5 rbd_header.e6ec54ba29f38 200692'26551487823872 watch
2070035 osd102 3.7a15b5b6 rbd_header.f42e823bececa 182482'26551487823872 watch
2070036 osd96 3.2ceb4f46 rbd_header.f42e867b637bd 175132'26551487823872 watch
2070037 osd57 3.6daeb5a2 rbd_header.f42e82ecebca5 156055'26551487823872 watch
2070038 osd37 3.a133da1c rbd_header.f42e816513fa7 196719'26551487823872 watch
2070039 osd43 3.c0125aba rbd_header.f42e84c075b37 192587'26551487823872 watch
2070040 osd51 3.20cb2f53 rbd_header.f42e81f94a5be 134631'26551487823872 watch
2070041 osd69 3.a3527e5a rbd_header.f65a150a4c7f4 223891'26551487823872 watch
2070042 osd146 3.ee1e56f3 rbd_header.f65a15677948f 200523'26551487823872 watch
2070043 osd21 3.672cde13 rbd_header.f65a143514fd 198930'26551487823872 watch
2070044 osd36 3.97cc7470 rbd_header.f65a11185e747 201523'26551487823872 watch
2070045 osd70 3.18fcd272 rbd_header.f65a1158abf02 197611'26551487823872 watch
2070046 osd82 3.8208b836 rbd_header.f65a12c47fed6 287608'26551487823872 watch
2070047 osd78 3.9ce9d2d8 rbd_header.f54f3f3f78ae 216654'26551487823872 watch
2070048 osd16 3.99d5db06 rbd_header.f54f32ea892e4 250640'26551487823872 watch
2070049 osd68 3.9f1c071a rbd_header.f54f35d7af2cf 223450'26551487823872 watch
2070050 osd90 3.c8100cf5 rbd_header.f54f331b392f7 303745'26551487823872 watch
2070051 osd60 3.c41f015d rbd_header.f659f6f6c01c3 174666'26551487823872 watch
2070052 osd24 3.22b6834c rbd_header.f659f7570a216 252853'26551487823872 watch
2070053 osd100 3.4355d651 rbd_header.f659f4963d64c 245170'26551487823872 watch
2070054 osd80 3.133cf6eb rbd_header.f659f4da10e75 245743'26551487823872 watch
2070055 osd1 3.eb339cb3 rbd_header.f659f7eb5f195 245316'26551487823872 watch
2070056 osd82 3.918d4e00 rbd_header.f659f7c4753bc 220074'26551487823872 watch
2070057 osd42 3.ff1718f5 rbd_header.f659f2b730b02 230028'26551487823872 watch
2070058 osd92 3.e4e19983 rbd_header.f659f314a46f 269470'26551487823872 watch
2070059 osd56 3.25c96e8a rbd_header.f659f119f370d 266582'26551487823872 watch
2070060 osd115 3.642e224e rbd_header.f659f584de7fd 291576'26551487823872 watch
2070061 osd144 3.45280ccc rbd_header.f54f91f78d63 225918'26551487823872 watch
2070062 osd118 3.50873837 rbd_header.f54f921268b0f 245740'26551487823872 watch
2070063 osd55 3.dcad1c74 rbd_header.f54f91e0c08fc 216768'26551487823872 watch
2070064 osd7 3.877d0e06 rbd_header.f54f9253d6b46 230906'26551487823872 watch
2070065 osd63 3.c87ca915 rbd_header.f54f94f111989 299124'26551487823872 watch
2070066 osd8 3.7031dc84 rbd_header.f54f967833a1e 179952'26551487823872 watch
2070067 osd98 3.9fa8711e rbd_header.f54f9412b530 228333'26551487823872 watch
2070068 osd54 3.a56ea23f rbd_header.f65ad31c25f40 304518'26551487823872 watch
2070069 osd80 3.13e40a42 rbd_header.f65ad65e773b1 250650'26551487823872 watch
2070070 osd55 3.38db7474 rbd_header.f65adcf2a53f 216765'26551487823872 watch
2070071 osd54 3.1f5c8bf4 rbd_header.f65ad16be944c 255781'26551487823872 watch
2070072 osd3 3.73a827db rbd_header.f65ad4ab53d2 219699'26551487823872 watch
2070073 osd16 3.d4745306 rbd_header.f66347b1b9a6e 250637'26551487823872 watch
2070074 osd69 3.e8130397 rbd_header.f6634622f70bc 207323'26551487823872 watch
2070075 osd74 3.7f33b105 rbd_header.f66347871909c 214234'26551487823872 watch
2070076 osd44 3.27439970 rbd_header.f663436a83873 221219'26551487823872 watch
2070077 osd110 3.e75d1953 rbd_header.f663475df588b 283149'26551487823872 watch
2070078 osd4 3.5396c89b rbd_header.f6634a9aaa27 208564'26551487823872 watch
2070079 osd14 3.cbdfe123 rbd_header.f663455329544 215780'26551487823872 watch
2070080 osd129 3.c80e056e rbd_header.f54f9422698a8 279482'26551487823872 watch
2070081 osd48 3.7adea980 rbd_header.4e4495a579b90 73658'26551487823872 watch
2070082 osd138 3.42b95d27 rbd_header.f42e83bfec56b 154232'26551487823872 watch
2070083 osd124 3.5af1eb91 rbd_header.f54f97996dd85 230331'26551487823872 watch
2070084 osd35 3.658c8f8 rbd_header.f54f319d0e468 252143'26551487823872 watch
2070085 osd35 3.fe6122f9 rbd_header.f65ad565c0a5d 216244'26551487823872 watch
2070086 osd93 3.71d7a587 rbd_header.593b5906fecb 248271'26551487823872 watch
2070087 osd2 3.8a1702f8 rbd_header.f65a1f9017a4 229537'26551487823872 watch
2070088 osd67 3.264a59c8 rbd_header.f65ad2e962255 234483'26551487823872 watch
2070089 osd138 3.776060c2 rbd_header.f65a11e3967ed 209128'26551487823872 watch
2070090 osd138 3.5d6d3b8 rbd_header.f65a132381bb4 173801'26551487823872 watch
2070103 osd29 3.50cfcfca rbd_data.f66344509891b.0000000000003402 read
ceph status:
cluster f1885784-0159-11e4-be58-a0d3c1f1936c
health HEALTH_WARN 4594 pgs degraded; 4594 pgs stuck unclean; 635
requests are blocked > 32 sec; recovery 4052687/22088253 objects
degraded (18.348%); 28/151 in osds are down; noout flag(s) set
monmap e6: 3 mons at
{ic2mon01=10.10.30.115:6789/0,ic2mon02=10.10.30.116:6789/0,ic2mon03=10.10.30.117:6789/0},
election epoch 74, quorum 0,1,2 ic2mon01,ic2mon02,ic2mon03
osdmap e6191: 151 osds: 123 up, 151 in
flags noout
pgmap v2318683: 8384 pgs, 7 pools, 28006 GB data, 7190 kobjects
36799 GB used, 511 TB / 547 TB avail
4052687/22088253 objects degraded (18.348%)
4594 active+degraded
1 active+clean+scrubbing+deep
3789 active+clean
Matt Conner
Keeper Technology
On Tue, Dec 8, 2015 at 5:59 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
> On Mon, Dec 7, 2015 at 9:56 PM, Matt Conner <matt.conner@keepertech.com> wrote:
>> Hi,
>>
>> We have a Ceph cluster in which we have been having issues with RBD
>> clients hanging when an OSD failure occurs. We are using a NAS gateway
>> server which maps RBD images to filesystems and serves the filesystems
>> out via NFS. The gateway server has close to 180 NFS clients and
>> almost every time even 1 OSD goes down during heavy load, the NFS
>> exports lock up and the clients are unable to access the NAS share via
>> NFS. When the OSD fails, Ceph recovers without issue, but the gateway
>> kernel RBD module appears to get stuck waiting on the now failed OSD.
>> Note that this works correctly when under lighter loads.
>>
>> From what we have been able to determine, the NFS server daemon hangs
>> waiting for I/O from the OSD that went out and never recovers.
>> Similarly, attempting to access files from the exported FS locally on
>> the gateway server will result in a similar hang. We also noticed that
>> Ceph health details will continue to report blocked I/O on the now
>> down OSD until either the OSD is recovered or the gateway server is
>> rebooted. Based on a few kernel logs from NFS and PVS, we were able
>> to trace the problem to the RBD kernel module.
>>
>> Unfortunately, the only way we have been able to recover our gateway
>> is by hard rebooting the server.
>>
>> Has anyone else encountered this issue and/or have a possible solution?
>> Are there suggestions for getting more detailed debugging information
>> from the RBD kernel module?
>
> Dumping osdc, osdmap and ceph status when it gets stuck would be
> a start:
>
> # cat /sys/kernel/debug/ceph/*/osdmap
> # cat /sys/kernel/debug/ceph/*/osdc
> $ ceph status
>
> Thanks,
>
> Ilya
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
2015-12-10 23:16 ` Matt Conner
@ 2015-12-10 23:58 ` Ilya Dryomov
[not found] ` <CAKeiORRYycaoC6vXZ8TOpf9_oU3it3yrGO=nJen6Fnhiwn=Okw@mail.gmail.com>
0 siblings, 1 reply; 10+ messages in thread
From: Ilya Dryomov @ 2015-12-10 23:58 UTC (permalink / raw)
To: Matt Conner; +Cc: ceph-devel
On Fri, Dec 11, 2015 at 12:16 AM, Matt Conner
<matt.conner@keepertech.com> wrote:
> Hi Ilya,
>
> Took me a little time to reproduce, but I'm once again got into the
> state. In the past I had reproduced the issue with a single OSD
> failure but in this case I failed an entire server. Ignore the "noout"
> flag as I set it to prevent rebalance during my attempts to reproduce
> the issue.
>
> osdmap:
>
> epoch 6191
> flags
> pool 0 pg_num 64 (63) read_tier -1 write_tier -1
> pool 1 pg_num 64 (63) read_tier -1 write_tier -1
> pool 2 pg_num 64 (63) read_tier -1 write_tier -1
> pool 3 pg_num 2048 (2047) read_tier -1 write_tier -1
> pool 5 pg_num 2048 (2047) read_tier -1 write_tier -1
> pool 6 pg_num 2048 (2047) read_tier -1 write_tier -1
> pool 7 pg_num 2048 (2047) read_tier -1 write_tier -1
> osd0 10.10.30.55:6830 100% (exists, up) 100%
> osd1 10.10.30.53:6809 100% (exists, up) 100%
> osd2 10.10.30.54:6839 100% (exists, up) 100%
> osd3 10.10.30.56:6833 100% (exists, up) 100%
> osd4 10.10.30.55:6848 100% (exists, up) 100%
> osd5 10.10.30.53:6854 100% (exists, up) 100%
> osd6 10.10.30.56:6818 100% (exists, up) 100%
> osd7 10.10.30.54:6830 100% (exists, up) 100%
> osd8 10.10.30.55:6839 100% (exists, up) 100%
> osd9 10.10.30.56:6845 100% (exists, up) 100%
> osd10 10.10.30.53:6860 100% (exists, up) 100%
> osd11 10.10.30.54:6842 100% (exists, up) 100%
> osd12 10.10.30.55:6836 100% (exists, up) 100%
> osd13 10.10.30.56:6863 100% (exists, up) 100%
> osd14 10.10.30.53:6839 100% (exists, up) 100%
> osd15 10.10.30.54:6866 100% (exists, up) 100%
> osd16 10.10.30.55:6869 100% (exists, up) 100%
> osd17 10.10.30.56:6812 100% (exists, up) 100%
> osd18 10.10.30.53:6824 100% (exists, up) 100%
> osd19 10.10.30.54:6818 100% (exists, up) 100%
> osd20 10.10.30.55:6833 100% (exists, up) 100%
> osd21 10.10.30.56:6869 100% (exists, up) 100%
> osd22 10.10.30.53:6857 100% (exists, up) 100%
> osd23 10.10.30.54:6833 100% (exists, up) 100%
> osd24 10.10.30.55:6803 100% (exists, up) 100%
> osd25 10.10.30.56:6821 100% (exists, up) 100%
> osd26 10.10.30.54:6848 100% (exists, up) 100%
> osd27 10.10.30.53:6848 100% (exists, up) 100%
> osd28 10.10.30.55:6863 100% (exists, up) 100%
> osd29 10.10.30.56:6827 100% (exists, up) 100%
> osd30 10.10.30.54:6836 100% (exists, up) 100%
> osd31 10.10.30.53:6863 100% (exists, up) 100%
> osd32 10.10.30.55:6845 100% (exists, up) 100%
> osd33 10.10.30.56:6839 100% (exists, up) 100%
> osd34 10.10.30.54:6851 100% (exists, up) 100%
> osd35 10.10.30.53:6845 100% (exists, up) 100%
> osd36 10.10.30.55:6854 100% (exists, up) 100%
> osd37 10.10.30.56:6851 100% (exists, up) 100%
> osd38 10.10.30.54:6806 100% (exists, up) 100%
> osd39 10.10.30.53:6824 0% (doesn't exist) 100%
> osd40 10.10.30.55:6815 100% (exists, up) 100%
> osd41 10.10.30.56:6842 100% (exists, up) 100%
> osd42 10.10.30.54:6863 100% (exists, up) 100%
> osd43 10.10.30.53:6806 100% (exists, up) 100%
> osd44 10.10.30.55:6821 100% (exists, up) 100%
> osd45 10.10.30.56:6803 100% (exists, up) 100%
> osd46 10.10.30.54:6812 100% (exists, up) 100%
> osd47 10.10.30.53:6866 100% (exists, up) 100%
> osd48 10.10.30.55:6812 100% (exists, up) 100%
> osd49 10.10.30.56:6815 100% (exists, up) 100%
> osd50 10.10.30.53:6842 100% (exists, up) 100%
> osd51 10.10.30.54:6854 100% (exists, up) 100%
> osd52 10.10.30.55:6842 100% (exists, up) 100%
> osd53 10.10.30.56:6836 100% (exists, up) 100%
> osd54 10.10.30.53:6803 100% (exists, up) 100%
> osd55 10.10.30.54:6869 100% (exists, up) 100%
> osd56 10.10.30.55:6866 100% (exists, up) 100%
> osd57 10.10.30.56:6860 100% (exists, up) 100%
> osd58 10.10.30.53:6800 100% (exists, up) 100%
> osd59 10.10.30.54:6803 100% (exists, up) 100%
> osd60 10.10.30.55:6824 100% (exists, up) 100%
> osd61 10.10.30.56:6804 100% (exists, up) 100%
> osd62 10.10.30.53:6815 100% (exists, up) 100%
> osd63 10.10.30.54:6857 100% (exists, up) 100%
> osd64 10.10.30.55:6800 100% (exists, up) 100%
> osd65 10.10.30.56:6866 100% (exists, up) 100%
> osd66 10.10.30.53:6836 100% (exists, up) 100%
> osd67 10.10.30.54:6845 100% (exists, up) 100%
> osd68 10.10.30.55:6860 100% (exists, up) 100%
> osd69 10.10.30.56:6824 100% (exists, up) 100%
> osd70 10.10.30.53:6821 100% (exists, up) 100%
> osd71 10.10.30.54:6809 100% (exists, up) 100%
> osd72 10.10.30.55:6806 100% (exists, up) 100%
> osd73 10.10.30.56:6809 100% (exists, up) 100%
> osd74 10.10.30.53:6818 100% (exists, up) 100%
> osd75 10.10.30.54:6815 100% (exists, up) 100%
> osd76 10.10.30.55:6827 100% (exists, up) 100%
> osd77 10.10.30.56:6830 100% (exists, up) 100%
> osd78 10.10.30.53:6833 100% (exists, up) 100%
> osd79 10.10.30.54:6800 100% (exists, up) 100%
> osd80 10.10.30.55:6851 100% (exists, up) 100%
> osd81 10.10.30.56:6800 100% (exists, up) 100%
> osd82 10.10.30.53:6812 100% (exists, up) 100%
> osd83 10.10.30.54:6821 100% (exists, up) 100%
> osd84 10.10.30.55:6857 100% (exists, up) 100%
> osd85 10.10.30.56:6848 100% (exists, up) 100%
> osd86 10.10.30.53:6830 100% (exists, up) 100%
> osd87 10.10.30.54:6827 100% (exists, up) 100%
> osd88 10.10.30.55:6809 100% (exists, up) 100%
> osd89 10.10.30.56:6854 100% (exists, up) 100%
> osd90 10.10.30.53:6827 100% (exists, up) 100%
> osd91 10.10.30.54:6860 100% (exists, up) 100%
> osd92 10.10.30.55:6818 100% (exists, up) 100%
> osd93 10.10.30.56:6857 100% (exists, up) 100%
> osd94 10.10.30.53:6851 100% (exists, up) 100%
> osd95 10.10.30.54:6824 100% (exists, up) 100%
> osd96 10.10.30.52:6863 100% (exists, up) 100%
> osd97 10.10.30.51:6841 100% (exists) 100%
> osd98 10.10.30.52:6854 100% (exists, up) 100%
> osd99 10.10.30.51:6844 100% (exists) 100%
> osd100 10.10.30.52:6809 100% (exists, up) 100%
> osd101 10.10.30.51:6853 100% (exists) 100%
> osd102 10.10.30.52:6857 100% (exists, up) 100%
> osd103 10.10.30.51:6807 100% (exists) 100%
> osd104 10.10.30.51:6810 100% (exists) 100%
> osd105 10.10.30.52:6827 100% (exists, up) 100%
> osd106 10.10.30.52:6881 100% (exists, up) 100%
> osd107 10.10.30.51:6801 100% (exists) 100%
> osd108 10.10.30.52:6803 100% (exists, up) 100%
> osd109 10.10.30.51:6835 100% (exists) 100%
> osd110 10.10.30.52:6878 100% (exists, up) 100%
> osd111 10.10.30.51:6825 100% (exists) 100%
> osd112 10.10.30.51:6804 100% (exists) 100%
> osd113 10.10.30.52:6860 100% (exists, up) 100%
> osd114 10.10.30.51:6838 100% (exists) 100%
> osd115 10.10.30.52:6845 100% (exists, up) 100%
> osd116 10.10.30.52:6872 100% (exists, up) 100%
> osd117 10.10.30.51:6862 100% (exists) 100%
> osd118 10.10.30.52:6806 100% (exists, up) 100%
> osd119 10.10.30.51:6832 100% (exists) 100%
> osd120 10.10.30.52:6842 100% (exists, up) 100%
> osd121 10.10.30.51:6816 100% (exists) 100%
> osd122 10.10.30.52:6824 100% (exists, up) 100%
> osd123 10.10.30.51:6859 100% (exists) 100%
> osd124 10.10.30.52:6839 100% (exists, up) 100%
> osd125 10.10.30.51:6868 100% (exists) 100%
> osd126 10.10.30.51:6829 100% (exists) 100%
> osd127 10.10.30.52:6848 100% (exists, up) 100%
> osd128 10.10.30.51:6883 100% (exists) 100%
> osd129 10.10.30.52:6836 100% (exists, up) 100%
> osd130 10.10.30.52:6812 100% (exists, up) 100%
> osd131 10.10.30.51:6819 100% (exists) 100%
> osd132 10.10.30.52:6821 100% (exists, up) 100%
> osd133 10.10.30.51:6874 100% (exists) 100%
> osd134 10.10.30.52:6828 100% (exists, up) 100%
> osd135 10.10.30.51:6871 100% (exists) 100%
> osd136 10.10.30.52:6833 100% (exists, up) 100%
> osd137 10.10.30.51:6813 100% (exists) 100%
> osd138 10.10.30.52:6869 100% (exists, up) 100%
> osd139 10.10.30.51:6865 100% (exists) 100%
> osd140 10.10.30.52:6875 100% (exists, up) 100%
> osd141 10.10.30.51:6822 100% (exists) 100%
> osd142 10.10.30.52:6800 100% (exists, up) 100%
> osd143 10.10.30.51:6880 100% (exists) 100%
> osd144 10.10.30.52:6851 100% (exists, up) 100%
> osd145 10.10.30.51:6877 100% (exists) 100%
> osd146 10.10.30.52:6815 100% (exists, up) 100%
> osd147 10.10.30.51:6850 100% (exists) 100%
> osd148 10.10.30.52:6866 100% (exists, up) 100%
> osd149 10.10.30.51:6800 100% (exists) 100%
> osd150 10.10.30.52:6818 100% (exists, up) 100%
> osd151 10.10.30.51:6856 100% (exists) 100%
>
> osdc:
>
> 2064277 osd136 3.79a01658 rbd_data.f659f6ca5381e.0000000000003e0d read
> 2066619 osd29 3.a54ffb5 rbd_data.f54f37207fb18.0000000000000013 read
> 2067420 osd29 3.8ca62e20 rbd_data.f54f921268b0f.0000000000001a0f read
> 2068773 osd136 3.6145463c rbd_data.f65ad4ab53d2.0000000000002c0c read
> 2069250 osd29 3.a54ffb5 rbd_data.f54f37207fb18.0000000000000013
> set-alloc-hint,write
> 2069272 osd136 3.e9f670ce rbd_data.12621d18fb34.00000000000011c7 read
> 2069838 osd136 3.5c533eb5 rbd_data.f65adcf2a53f.0000000000000000 read
> 2069872 osd136 3.5c533eb5 rbd_data.f65adcf2a53f.0000000000000000 read
> 2069880 osd136 3.b4d593da rbd_data.f663471f4e3a4.0000000000000035
> set-alloc-hint,write
> 2069994 osd43 3.485e015f rbd_header.f42e85cd8f5d2 192438'26551487823872 watch
> 2069995 osd23 3.967f0ac rbd_header.f54f943ff3510 271821'26551487823872 watch
> 2069996 osd22 3.1679d3fd rbd_header.f54f346170e75 248651'26551487823872 watch
> 2069997 osd132 3.4c72415a rbd_header.f659f6ca5381e 212921'26551487823872 watch
> 2069998 osd40 3.dcb615cd rbd_header.f659f3b6599ce 281696'26551487823872 watch
> 2069999 osd4 3.2c5dc7d9 rbd_header.f659f7b8275f8 265762'26551487823872 watch
> 2070000 osd29 3.49c607ca rbd_header.f66343de35900 230429'26551487823872 watch
> 2070001 osd140 3.a030d1d8 rbd_header.f66341d1ccf35 290629'26551487823872 watch
> 2070002 osd45 3.ce94805a rbd_header.f66345f368639 295864'26551487823872 watch
> 2070003 osd52 3.854e4dfc rbd_header.593b6491ec9f 280116'26551487823872 watch
> 2070004 osd38 3.5cce84bc rbd_header.f42e822939526 163677'26551487823872 watch
> 2070005 osd47 3.13094d0 rbd_header.f42e869ead798 165850'26551487823872 watch
> 2070006 osd92 3.b72d8628 rbd_header.f42e8534aeb08 220526'26551487823872 watch
> 2070007 osd41 3.a44fd23b rbd_header.1262487bcab5 170538'26551487823872 watch
> 2070008 osd0 3.184871e7 rbd_header.12622ffb8aff 221268'26551487823872 watch
> 2070009 osd73 3.41d03b49 rbd_header.12628c3de5a 188527'26551487823872 watch
> 2070010 osd38 3.b63d7b6c rbd_header.126253d67973 116592'26551487823872 watch
> 2070011 osd115 3.e6b3a4e rbd_header.1262614d8810 231173'26551487823872 watch
> 2070012 osd72 3.d9ede1c3 rbd_header.1262115f0cf 140197'26551487823872 watch
> 2070013 osd12 3.6642c157 rbd_header.12621e7fc115 187593'26551487823872 watch
> 2070014 osd91 3.b3705118 rbd_header.593b7323f25f 243419'26551487823872 watch
> 2070015 osd57 3.abcf9bf6 rbd_header.593b363a4411 209538'26551487823872 watch
> 2070016 osd53 3.72c67cfe rbd_header.593b6072b1c7 291406'26551487823872 watch
> 2070017 osd70 3.10504835 rbd_header.593b35e9ba2b 205415'26551487823872 watch
> 2070018 osd45 3.8b381aaa rbd_header.593b418e8823 272511'26551487823872 watch
> 2070019 osd142 3.2b5318b7 rbd_header.593b46f1aa31 210594'26551487823872 watch
> 2070020 osd68 3.25c6ad00 rbd_header.4e449416a8478 72833'26551487823872 watch
> 2070021 osd57 3.3df2d8ae rbd_header.4e44968299a31 74608'26551487823872 watch
> 2070022 osd13 3.5102bd35 rbd_header.4e449723d5105 87452'26551487823872 watch
> 2070023 osd46 3.d0ad9250 rbd_header.4e449104e5eea 75590'26551487823872 watch
> 2070024 osd148 3.7fb7b2bd rbd_header.4e4491321fb8d 68317'26551487823872 watch
> 2070025 osd18 3.f82976fd rbd_header.4e449402e564a 57095'26551487823872 watch
> 2070026 osd10 3.39b4da2e rbd_header.4e44914acd83 60123'26551487823872 watch
> 2070027 osd2 3.8fe8ba4d rbd_header.e6ec56f5dcb51 200753'26551487823872 watch
> 2070028 osd6 3.66d6ca30 rbd_header.e6ec57072b44a 235673'26551487823872 watch
> 2070029 osd80 3.25b25299 rbd_header.e6ec57e4a0ef9 188214'26551487823872 watch
> 2070030 osd46 3.58e2250 rbd_header.e6ec550078b66 190030'26551487823872 watch
> 2070031 osd116 3.b29b37d3 rbd_header.e6ec521c83559 180643'26551487823872 watch
> 2070032 osd80 3.eca20f5e rbd_header.e6ec570293241 160701'26551487823872 watch
> 2070033 osd80 3.b20df8a9 rbd_header.e6ec53b631e5a 144384'26551487823872 watch
> 2070034 osd40 3.6cb419d5 rbd_header.e6ec54ba29f38 200692'26551487823872 watch
> 2070035 osd102 3.7a15b5b6 rbd_header.f42e823bececa 182482'26551487823872 watch
> 2070036 osd96 3.2ceb4f46 rbd_header.f42e867b637bd 175132'26551487823872 watch
> 2070037 osd57 3.6daeb5a2 rbd_header.f42e82ecebca5 156055'26551487823872 watch
> 2070038 osd37 3.a133da1c rbd_header.f42e816513fa7 196719'26551487823872 watch
> 2070039 osd43 3.c0125aba rbd_header.f42e84c075b37 192587'26551487823872 watch
> 2070040 osd51 3.20cb2f53 rbd_header.f42e81f94a5be 134631'26551487823872 watch
> 2070041 osd69 3.a3527e5a rbd_header.f65a150a4c7f4 223891'26551487823872 watch
> 2070042 osd146 3.ee1e56f3 rbd_header.f65a15677948f 200523'26551487823872 watch
> 2070043 osd21 3.672cde13 rbd_header.f65a143514fd 198930'26551487823872 watch
> 2070044 osd36 3.97cc7470 rbd_header.f65a11185e747 201523'26551487823872 watch
> 2070045 osd70 3.18fcd272 rbd_header.f65a1158abf02 197611'26551487823872 watch
> 2070046 osd82 3.8208b836 rbd_header.f65a12c47fed6 287608'26551487823872 watch
> 2070047 osd78 3.9ce9d2d8 rbd_header.f54f3f3f78ae 216654'26551487823872 watch
> 2070048 osd16 3.99d5db06 rbd_header.f54f32ea892e4 250640'26551487823872 watch
> 2070049 osd68 3.9f1c071a rbd_header.f54f35d7af2cf 223450'26551487823872 watch
> 2070050 osd90 3.c8100cf5 rbd_header.f54f331b392f7 303745'26551487823872 watch
> 2070051 osd60 3.c41f015d rbd_header.f659f6f6c01c3 174666'26551487823872 watch
> 2070052 osd24 3.22b6834c rbd_header.f659f7570a216 252853'26551487823872 watch
> 2070053 osd100 3.4355d651 rbd_header.f659f4963d64c 245170'26551487823872 watch
> 2070054 osd80 3.133cf6eb rbd_header.f659f4da10e75 245743'26551487823872 watch
> 2070055 osd1 3.eb339cb3 rbd_header.f659f7eb5f195 245316'26551487823872 watch
> 2070056 osd82 3.918d4e00 rbd_header.f659f7c4753bc 220074'26551487823872 watch
> 2070057 osd42 3.ff1718f5 rbd_header.f659f2b730b02 230028'26551487823872 watch
> 2070058 osd92 3.e4e19983 rbd_header.f659f314a46f 269470'26551487823872 watch
> 2070059 osd56 3.25c96e8a rbd_header.f659f119f370d 266582'26551487823872 watch
> 2070060 osd115 3.642e224e rbd_header.f659f584de7fd 291576'26551487823872 watch
> 2070061 osd144 3.45280ccc rbd_header.f54f91f78d63 225918'26551487823872 watch
> 2070062 osd118 3.50873837 rbd_header.f54f921268b0f 245740'26551487823872 watch
> 2070063 osd55 3.dcad1c74 rbd_header.f54f91e0c08fc 216768'26551487823872 watch
> 2070064 osd7 3.877d0e06 rbd_header.f54f9253d6b46 230906'26551487823872 watch
> 2070065 osd63 3.c87ca915 rbd_header.f54f94f111989 299124'26551487823872 watch
> 2070066 osd8 3.7031dc84 rbd_header.f54f967833a1e 179952'26551487823872 watch
> 2070067 osd98 3.9fa8711e rbd_header.f54f9412b530 228333'26551487823872 watch
> 2070068 osd54 3.a56ea23f rbd_header.f65ad31c25f40 304518'26551487823872 watch
> 2070069 osd80 3.13e40a42 rbd_header.f65ad65e773b1 250650'26551487823872 watch
> 2070070 osd55 3.38db7474 rbd_header.f65adcf2a53f 216765'26551487823872 watch
> 2070071 osd54 3.1f5c8bf4 rbd_header.f65ad16be944c 255781'26551487823872 watch
> 2070072 osd3 3.73a827db rbd_header.f65ad4ab53d2 219699'26551487823872 watch
> 2070073 osd16 3.d4745306 rbd_header.f66347b1b9a6e 250637'26551487823872 watch
> 2070074 osd69 3.e8130397 rbd_header.f6634622f70bc 207323'26551487823872 watch
> 2070075 osd74 3.7f33b105 rbd_header.f66347871909c 214234'26551487823872 watch
> 2070076 osd44 3.27439970 rbd_header.f663436a83873 221219'26551487823872 watch
> 2070077 osd110 3.e75d1953 rbd_header.f663475df588b 283149'26551487823872 watch
> 2070078 osd4 3.5396c89b rbd_header.f6634a9aaa27 208564'26551487823872 watch
> 2070079 osd14 3.cbdfe123 rbd_header.f663455329544 215780'26551487823872 watch
> 2070080 osd129 3.c80e056e rbd_header.f54f9422698a8 279482'26551487823872 watch
> 2070081 osd48 3.7adea980 rbd_header.4e4495a579b90 73658'26551487823872 watch
> 2070082 osd138 3.42b95d27 rbd_header.f42e83bfec56b 154232'26551487823872 watch
> 2070083 osd124 3.5af1eb91 rbd_header.f54f97996dd85 230331'26551487823872 watch
> 2070084 osd35 3.658c8f8 rbd_header.f54f319d0e468 252143'26551487823872 watch
> 2070085 osd35 3.fe6122f9 rbd_header.f65ad565c0a5d 216244'26551487823872 watch
> 2070086 osd93 3.71d7a587 rbd_header.593b5906fecb 248271'26551487823872 watch
> 2070087 osd2 3.8a1702f8 rbd_header.f65a1f9017a4 229537'26551487823872 watch
> 2070088 osd67 3.264a59c8 rbd_header.f65ad2e962255 234483'26551487823872 watch
> 2070089 osd138 3.776060c2 rbd_header.f65a11e3967ed 209128'26551487823872 watch
> 2070090 osd138 3.5d6d3b8 rbd_header.f65a132381bb4 173801'26551487823872 watch
> 2070103 osd29 3.50cfcfca rbd_data.f66344509891b.0000000000003402 read
>
>
> ceph status:
>
> cluster f1885784-0159-11e4-be58-a0d3c1f1936c
> health HEALTH_WARN 4594 pgs degraded; 4594 pgs stuck unclean; 635
> requests are blocked > 32 sec; recovery 4052687/22088253 objects
> degraded (18.348%); 28/151 in osds are down; noout flag(s) set
> monmap e6: 3 mons at
> {ic2mon01=10.10.30.115:6789/0,ic2mon02=10.10.30.116:6789/0,ic2mon03=10.10.30.117:6789/0},
> election epoch 74, quorum 0,1,2 ic2mon01,ic2mon02,ic2mon03
> osdmap e6191: 151 osds: 123 up, 151 in
> flags noout
> pgmap v2318683: 8384 pgs, 7 pools, 28006 GB data, 7190 kobjects
> 36799 GB used, 511 TB / 547 TB avail
> 4052687/22088253 objects degraded (18.348%)
> 4594 active+degraded
> 1 active+clean+scrubbing+deep
> 3789 active+clean
That doesn't look like a simple kernel client bug...
What is the output of (substitute $POOL with the name of the pool with
id 3):
$ ceph osd pool ls detail
$ ceph osd map $POOL rbd_data.f659f6ca5381e.0000000000003e0d
$ ceph osd getmap -o /tmp/osdmap.bin
$ osdmaptool --test-map-pgs-dump /tmp/osdmap.bin
What happens if you do:
$ rados -p $POOL get rbd_data.f659f6ca5381e.0000000000003e0d /tmp/object.bin
Thanks,
Ilya
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
[not found] ` <CAOi1vP-Aw4oWV6eC2h7ZkW87FycnBm0XjMMo0T2updLDjMNULQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-11 0:39 ` Matt Conner
0 siblings, 0 replies; 10+ messages in thread
From: Matt Conner @ 2015-12-11 0:39 UTC (permalink / raw)
To: Ilya Dryomov
Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw, ceph-devel-u79uwXL29TY76Z2rM5mHXA
Hi Ilya,
I had already recovered but I managed to recreate the problem again. I
ran the commands against rbd_data.f54f9422698a8.0000000000000000 which
was one of those listed in osdc this time. We have 2048 PGs in the
pool so the list is long.
As for when I fetched the object using rados, it grabbed it without issue.
osd map:
osdmap e6247 pool 'NAS-ic2gw01' (3) object
'rbd_data.f54f9422698a8.0000000000000000' -> pg 3.cac46c43 (3.443) ->
up ([33,56], p33) acting ([33,56], p33)
osdmaptool:
pool 3 pg_num 2048
3.0 [23,138] 23
3.1 [96,76] 96
3.2 [16,62] 16
3.3 [75,150,57] 75
3.4 [85,30] 85
3.5 [138,93,55] 138
3.6 [134,91,64] 134
3.7 [129,56] 129
3.8 [150,63] 150
3.9 [129,84] 129
3.a [62,40,102] 62
3.b [75,105,43] 75
3.c [31,65,132] 31
3.d [122,62,69] 122
3.e [26,144] 26
3.f [87,4] 87
3.10 [82,134,24] 82
3.11 [134,38] 134
3.12 [45,118] 45
3.13 [144,43,20] 144
3.14 [66,40,144] 66
3.15 [67,102,43] 67
3.16 [50,124,53] 50
3.17 [122,51,84] 122
3.18 [61,142] 61
3.19 [9,0,31] 9
3.1a [93,72] 93
3.1b [35,46] 35
3.1c [5,48,17] 5
3.1d [14,134] 14
3.1e [136,72] 136
3.1f [138,59,93] 138
3.20 [34,20] 34
3.21 [148,55] 148
3.22 [24,63] 24
3.23 [68,33,58] 68
3.24 [25,19] 25
3.25 [32,108] 32
3.26 [122,38] 122
3.27 [138,72] 138
3.28 [89,60] 89
3.29 [61,11] 61
3.2a [1,60,122] 1
3.2b [42,64] 42
3.2c [15,88] 15
3.2d [6,100] 6
3.2e [21,8] 21
3.2f [87,0] 87
3.30 [74,3] 74
3.31 [76,58,120] 76
3.32 [108,90] 108
3.33 [48,94,41] 48
3.34 [4,124,91] 4
3.35 [70,83] 70
3.36 [82,142] 82
3.37 [118,25] 118
3.38 [93,70] 93
3.39 [27,115,24] 27
3.3a [88,61,138] 88
3.3b [85,148] 85
3.3c [2,32] 2
3.3d [20,55,41] 20
3.3e [122,86] 122
3.3f [0,81] 0
3.40 [113,29,18] 113
3.41 [75,52] 75
3.42 [21,150,92] 21
3.43 [16,17,86] 16
3.44 [38,140] 38
3.45 [146,67] 146
3.46 [129,10] 129
3.47 [35,36,37] 35
3.48 [127,65] 127
3.49 [77,108] 77
3.4a [134,50] 134
3.4b [59,9,10] 59
3.4c [7,24,82] 7
3.4d [34,27] 34
3.4e [78,49] 78
3.4f [84,37,110] 84
3.50 [108,60] 108
3.51 [51,0,93] 51
3.52 [82,84] 82
3.53 [138,33] 138
3.54 [102,36] 102
3.55 [55,129,32] 55
3.56 [38,68,106] 38
3.57 [8,49] 8
3.58 [40,110,42] 40
3.59 [144,0,54] 144
3.5a [45,24] 45
3.5b [127,30] 127
3.5c [64,148] 64
3.5d [83,43] 83
3.5e [35,23,108] 35
3.5f [65,4] 65
3.60 [63,61,18] 63
3.61 [122,47,42] 122
3.62 [134,8,1] 134
3.63 [18,49,150] 18
3.64 [37,14] 37
3.65 [110,70] 110
3.66 [74,32,9] 74
3.67 [66,26] 66
3.68 [13,86,115] 13
3.69 [40,150,94] 40
3.6a [86,55] 86
3.6b [14,13,105] 14
3.6c [130,45,5] 130
3.6d [47,41] 47
3.6e [50,34,138] 50
3.6f [13,20] 13
3.70 [31,108,42] 31
3.71 [82,52,110] 82
3.72 [49,54] 49
3.73 [115,10] 115
3.74 [110,88,69] 110
3.75 [96,20] 96
3.76 [16,43,2] 16
3.77 [24,65,7] 24
3.78 [64,129] 64
3.79 [105,53] 105
3.7a [56,144,38] 56
3.7b [148,84] 148
3.7c [38,50] 38
3.7d [25,12] 25
3.7e [16,95] 16
3.7f [85,130] 85
3.80 [78,53] 78
3.81 [38,146,4] 38
3.82 [65,22] 65
3.83 [69,12] 69
3.84 [19,43,4] 19
3.85 [89,122] 89
3.86 [120,28,57] 120
3.87 [150,25,71] 150
3.88 [142,10] 142
3.89 [120,26] 120
3.8a [20,132,11] 20
3.8b [14,102,85] 14
3.8c [16,150,79] 16
3.8d [120,29,54] 120
3.8e [53,138] 53
3.8f [30,54,52] 30
3.90 [87,56,132] 87
3.91 [46,28,6] 46
3.92 [86,84] 86
3.93 [37,124,1] 37
3.94 [7,20] 7
3.95 [130,6,50] 130
3.96 [11,4,142] 11
3.97 [83,14] 83
3.98 [61,50] 61
3.99 [78,81,36] 78
3.9a [2,10,33] 2
3.9b [4,148] 4
3.9c [42,74,32] 42
3.9d [80,148] 80
3.9e [140,73] 140
3.9f [77,105] 77
3.a0 [29,102] 29
3.a1 [24,134] 24
3.a2 [23,102,43] 23
3.a3 [8,25] 8
3.a4 [62,122,60] 62
3.a5 [134,93,7] 134
3.a6 [48,7] 48
3.a7 [47,56] 47
3.a8 [32,46] 32
3.a9 [80,85] 80
3.aa [8,70] 8
3.ab [144,78,63] 144
3.ac [23,1] 23
3.ad [56,58] 56
3.ae [57,22] 57
3.af [70,6] 70
3.b0 [105,63,86] 105
3.b1 [127,87,52] 127
3.b2 [136,67,77] 136
3.b3 [3,2] 3
3.b4 [28,120] 28
3.b5 [48,27] 48
3.b6 [100,77] 100
3.b7 [142,20] 142
3.b8 [92,86] 92
3.b9 [83,136,62] 83
3.ba [37,48,150] 37
3.bb [63,96,72] 63
3.bc [50,26,93] 50
3.bd [150,35] 150
3.be [100,80,22] 100
3.bf [142,3,10] 142
3.c0 [83,44,100] 83
3.c1 [146,26] 146
3.c2 [138,86] 138
3.c3 [122,34] 122
3.c4 [72,54] 72
3.c5 [120,93] 120
3.c6 [60,23,120] 60
3.c7 [105,84,25] 105
3.c8 [69,50,92] 69
3.c9 [61,48] 61
3.ca [92,90] 92
3.cb [102,7] 102
3.cc [65,40] 65
3.cd [108,83] 108
3.ce [136,83,31] 136
3.cf [113,24,38] 113
3.d0 [127,55] 127
3.d1 [80,14,106] 80
3.d2 [150,36,67] 150
3.d3 [127,16] 127
3.d4 [51,120] 51
3.d5 [69,24] 69
3.d6 [56,100] 56
3.d7 [14,142] 14
3.d8 [88,15] 88
3.d9 [15,76,77] 15
3.da [98,70] 98
3.db [122,77,79] 122
3.dc [32,3] 32
3.dd [55,54] 55
3.de [120,47,95] 120
3.df [22,108] 22
3.e0 [70,71] 70
3.e1 [40,10,136] 40
3.e2 [30,9] 30
3.e3 [115,83,14] 115
3.e4 [4,37] 4
3.e5 [1,34] 1
3.e6 [52,86] 52
3.e7 [87,80,14] 87
3.e8 [84,19,43] 84
3.e9 [132,93] 132
3.ea [52,130] 52
3.eb [70,84,53] 70
3.ec [59,0,144] 59
3.ed [26,106,27] 26
3.ee [45,70] 45
3.ef [34,48] 34
3.f0 [4,120] 4
3.f1 [5,75] 5
3.f2 [52,146,51] 52
3.f3 [81,55] 81
3.f4 [31,130] 31
3.f5 [42,66] 42
3.f6 [67,68,61] 67
3.f7 [26,6] 26
3.f8 [35,30] 35
3.f9 [81,66] 81
3.fa [87,41,134] 87
3.fb [18,21] 18
3.fc [118,67] 118
3.fd [66,3] 66
3.fe [73,5] 73
3.ff [17,31] 17
3.100 [19,35,85] 19
3.101 [11,8,150] 11
3.102 [17,120,90] 17
3.103 [37,127,50] 37
3.104 [86,57] 86
3.105 [74,8] 74
3.106 [93,75] 93
3.107 [59,100] 59
3.108 [79,18] 79
3.109 [74,127,88] 74
3.10a [106,73] 106
3.10b [78,44] 78
3.10c [11,85] 11
3.10d [80,23] 80
3.10e [22,7] 22
3.10f [50,113] 50
3.110 [23,36,54] 23
3.111 [47,71,130] 47
3.112 [75,49] 75
3.113 [80,45,124] 80
3.114 [140,34] 140
3.115 [63,122] 63
3.116 [50,15,57] 50
3.117 [94,127] 94
3.118 [91,21] 91
3.119 [113,15] 113
3.11a [79,44] 79
3.11b [22,102,16] 22
3.11c [87,53,76] 87
3.11d [30,88] 30
3.11e [98,20] 98
3.11f [29,136] 29
3.120 [43,3,34] 43
3.121 [44,11,98] 44
3.122 [113,47,6] 113
3.123 [14,51] 14
3.124 [50,148] 50
3.125 [11,68] 11
3.126 [87,116] 87
3.127 [100,25,46] 100
3.128 [56,45] 56
3.129 [68,21] 68
3.12a [51,88] 51
3.12b [150,83,50] 150
3.12c [98,40,2] 98
3.12d [24,29,95] 24
3.12e [144,1] 144
3.12f [3,98] 3
3.130 [150,55] 150
3.131 [62,146,15] 62
3.132 [146,49] 146
3.133 [86,38,53] 86
3.134 [20,26] 20
3.135 [146,77] 146
3.136 [136,53,70] 136
3.137 [17,76,78] 17
3.138 [142,21,71] 142
3.139 [80,148,75] 80
3.13a [122,95,14] 122
3.13b [87,25] 87
3.13c [84,122] 84
3.13d [21,87,129] 21
3.13e [58,45,108] 58
3.13f [21,0,140] 21
3.140 [94,55] 94
3.141 [42,3,5] 42
3.142 [62,120] 62
3.143 [18,9,118] 18
3.144 [35,29] 35
3.145 [105,7,64] 105
3.146 [150,38,88] 150
3.147 [42,70] 42
3.148 [18,56,122] 18
3.149 [58,132,29] 58
3.14a [95,36] 95
3.14b [35,116,72] 35
3.14c [16,77] 16
3.14d [75,47] 75
3.14e [54,13,46] 54
3.14f [102,33,88] 102
3.150 [118,35] 118
3.151 [76,77,38] 76
3.152 [74,124,95] 74
3.153 [110,26] 110
3.154 [18,53,0] 18
3.155 [102,1,32] 102
3.156 [118,57,8] 118
3.157 [12,136] 12
3.158 [87,148,80] 87
3.159 [115,76] 115
3.15a [132,28] 132
3.15b [72,19] 72
3.15c [113,45,87] 113
3.15d [60,69] 60
3.15e [144,30] 144
3.15f [43,38] 43
3.160 [138,62,13] 138
3.161 [65,72] 65
3.162 [30,65] 30
3.163 [26,32,86] 26
3.164 [88,94] 88
3.165 [84,115,31] 84
3.166 [2,41] 2
3.167 [6,118,86] 6
3.168 [89,26] 89
3.169 [66,116] 66
3.16a [41,68,7] 41
3.16b [134,38,77] 134
3.16c [65,86] 65
3.16d [74,91,144] 74
3.16e [58,13,113] 58
3.16f [14,20] 14
3.170 [44,9] 44
3.171 [100,17] 100
3.172 [15,4] 15
3.173 [82,60] 82
3.174 [24,70] 24
3.175 [15,52,89] 15
3.176 [80,2] 80
3.177 [7,78] 7
3.178 [9,136] 9
3.179 [13,59] 13
3.17a [52,63,89] 52
3.17b [85,0] 85
3.17c [138,9] 138
3.17d [31,138] 31
3.17e [98,65] 98
3.17f [53,144] 53
3.180 [48,41] 48
3.181 [23,84,54] 23
3.182 [69,108] 69
3.183 [92,108] 92
3.184 [73,43] 73
3.185 [6,83,70] 6
3.186 [3,108,88] 3
3.187 [98,54,30] 98
3.188 [113,29] 113
3.189 [72,106,85] 72
3.18a [50,21,26] 50
3.18b [90,59] 90
3.18c [89,70,134] 89
3.18d [69,110] 69
3.18e [129,9] 129
3.18f [108,91] 108
3.190 [20,89,86] 20
3.191 [64,120] 64
3.192 [67,68] 67
3.193 [45,98] 45
3.194 [84,94,98] 84
3.195 [70,24,122] 70
3.196 [124,27,30] 124
3.197 [35,150] 35
3.198 [15,50,52] 15
3.199 [124,66,16] 124
3.19a [61,105] 61
3.19b [9,55] 9
3.19c [37,60] 37
3.19d [9,23,100] 9
3.19e [28,74,115] 28
3.19f [67,49] 67
3.1a0 [88,124] 88
3.1a1 [60,106] 60
3.1a2 [28,29] 28
3.1a3 [71,52] 71
3.1a4 [20,57] 20
3.1a5 [40,3,18] 40
3.1a6 [72,100,33] 72
3.1a7 [59,66] 59
3.1a8 [54,28] 54
3.1a9 [140,16] 140
3.1aa [26,94] 26
3.1ab [136,21,12] 136
3.1ac [94,26,96] 94
3.1ad [95,18,118] 95
3.1ae [16,22,85] 16
3.1af [88,120] 88
3.1b0 [31,67] 31
3.1b1 [130,29] 130
3.1b2 [32,29] 32
3.1b3 [140,79,3] 140
3.1b4 [113,88] 113
3.1b5 [64,96] 64
3.1b6 [20,18,98] 20
3.1b7 [85,18] 85
3.1b8 [74,53,100] 74
3.1b9 [20,70] 20
3.1ba [78,13,68] 78
3.1bb [8,106,71] 8
3.1bc [82,87] 82
3.1bd [53,136] 53
3.1be [75,90,100] 75
3.1bf [78,45] 78
3.1c0 [46,21,102] 46
3.1c1 [84,54,37] 84
3.1c2 [36,51] 36
3.1c3 [72,22] 72
3.1c4 [108,69,58] 108
3.1c5 [136,80] 136
3.1c6 [10,23,136] 10
3.1c7 [34,16,53] 34
3.1c8 [67,0] 67
3.1c9 [7,24] 7
3.1ca [35,134] 35
3.1cb [27,73] 27
3.1cc [23,98,86] 23
3.1cd [72,5] 72
3.1ce [80,136,94] 80
3.1cf [35,60] 35
3.1d0 [58,95,102] 58
3.1d1 [47,11,124] 47
3.1d2 [148,18] 148
3.1d3 [53,22] 53
3.1d4 [34,64,82] 34
3.1d5 [40,98] 40
3.1d6 [129,95] 129
3.1d7 [29,92] 29
3.1d8 [140,58] 140
3.1d9 [46,1] 46
3.1da [31,89] 31
3.1db [144,7] 144
3.1dc [19,10,88] 19
3.1dd [52,100] 52
3.1de [3,23,100] 3
3.1df [17,67,70] 17
3.1e0 [45,0] 45
3.1e1 [77,122] 77
3.1e2 [81,54] 81
3.1e3 [140,60] 140
3.1e4 [0,45,132] 0
3.1e5 [25,146] 25
3.1e6 [9,60] 9
3.1e7 [0,108] 0
3.1e8 [40,29] 40
3.1e9 [78,12] 78
3.1ea [82,20] 82
3.1eb [72,82] 72
3.1ec [61,102,64] 61
3.1ed [52,65] 52
3.1ee [105,9] 105
3.1ef [62,11] 62
3.1f0 [72,116] 72
3.1f1 [127,42,68] 127
3.1f2 [134,44,73] 134
3.1f3 [140,4] 140
3.1f4 [124,8,41] 124
3.1f5 [98,22,44] 98
3.1f6 [10,129] 10
3.1f7 [85,42,12] 85
3.1f8 [64,66] 64
3.1f9 [59,150,61] 59
3.1fa [96,45,31] 96
3.1fb [14,132,89] 14
3.1fc [115,41,24] 115
3.1fd [132,51,18] 132
3.1fe [2,35] 2
3.1ff [84,132] 84
3.200 [41,71] 41
3.201 [96,14] 96
3.202 [86,113,81] 86
3.203 [80,27] 80
3.204 [31,11] 31
3.205 [132,72,18] 132
3.206 [54,46,96] 54
3.207 [144,68] 144
3.208 [30,56,10] 30
3.209 [132,2] 132
3.20a [67,80,21] 67
3.20b [73,34] 73
3.20c [30,0] 30
3.20d [148,44] 148
3.20e [29,116,11] 29
3.20f [5,23] 5
3.210 [24,46,120] 24
3.211 [113,55,60] 113
3.212 [30,35] 30
3.213 [52,89,18] 52
3.214 [74,75] 74
3.215 [16,26,98] 16
3.216 [48,47] 48
3.217 [10,98] 10
3.218 [43,75] 43
3.219 [108,48] 108
3.21a [85,31] 85
3.21b [16,38] 16
3.21c [37,34] 37
3.21d [25,4] 25
3.21e [94,12,122] 94
3.21f [36,18] 36
3.220 [108,77] 108
3.221 [38,49,74] 38
3.222 [50,130,61] 50
3.223 [7,148] 7
3.224 [17,82,36] 17
3.225 [129,28] 129
3.226 [0,129] 0
3.227 [127,73,7] 127
3.228 [85,44] 85
3.229 [68,17] 68
3.22a [43,76,142] 43
3.22b [12,34,6] 12
3.22c [53,0] 53
3.22d [105,85] 105
3.22e [10,77] 10
3.22f [65,75,5] 65
3.230 [6,70] 6
3.231 [77,78,0] 77
3.232 [16,79] 16
3.233 [62,89] 62
3.234 [62,140,89] 62
3.235 [11,140,85] 11
3.236 [62,130,33] 62
3.237 [105,3] 105
3.238 [5,30] 5
3.239 [83,72] 83
3.23a [78,115,72] 78
3.23b [41,0] 41
3.23c [58,83,17] 58
3.23d [130,94] 130
3.23e [41,110] 41
3.23f [54,38] 54
3.240 [148,26,69] 148
3.241 [41,60] 41
3.242 [80,132] 80
3.243 [102,95] 102
3.244 [11,108] 11
3.245 [32,45,79] 32
3.246 [68,2] 68
3.247 [58,146,57] 58
3.248 [37,144] 37
3.249 [55,85,140] 55
3.24a [150,15] 150
3.24b [17,48] 17
3.24c [74,33] 74
3.24d [2,22] 2
3.24e [115,86] 115
3.24f [57,127] 57
3.250 [46,12] 46
3.251 [40,11,31] 40
3.252 [130,68] 130
3.253 [122,64,93] 122
3.254 [95,65] 95
3.255 [38,118] 38
3.256 [46,127,37] 46
3.257 [42,58] 42
3.258 [24,124,11] 24
3.259 [124,86] 124
3.25a [19,31,32] 19
3.25b [0,21] 0
3.25c [1,116,6] 1
3.25d [52,91] 52
3.25e [92,79] 92
3.25f [85,27,44] 85
3.260 [61,110,44] 61
3.261 [73,19,127] 73
3.262 [92,37,91] 92
3.263 [71,98,25] 71
3.264 [60,14,7] 60
3.265 [130,3] 130
3.266 [98,35] 98
3.267 [2,36] 2
3.268 [25,106,63] 25
3.269 [44,29] 44
3.26a [148,94] 148
3.26b [92,34,86] 92
3.26c [67,62,52] 67
3.26d [88,42,105] 88
3.26e [115,94,60] 115
3.26f [96,11] 96
3.270 [21,14] 21
3.271 [108,90] 108
3.272 [70,127] 70
3.273 [74,129,32] 74
3.274 [56,81] 56
3.275 [86,72] 86
3.276 [19,22,120] 19
3.277 [3,82] 3
3.278 [38,110,47] 38
3.279 [55,94] 55
3.27a [33,26,8] 33
3.27b [9,30] 9
3.27c [98,57] 98
3.27d [73,52] 73
3.27e [91,57] 91
3.27f [22,37,32] 22
3.280 [55,100] 55
3.281 [27,77] 27
3.282 [136,37,82] 136
3.283 [8,25,91] 8
3.284 [106,67,21] 106
3.285 [116,26,65] 116
3.286 [148,71] 148
3.287 [140,51,13] 140
3.288 [140,47] 140
3.289 [96,8,78] 96
3.28a [116,50] 116
3.28b [58,98] 58
3.28c [100,79,76] 100
3.28d [63,69,124] 63
3.28e [13,150,55] 13
3.28f [44,37] 44
3.290 [136,89,32] 136
3.291 [77,90] 77
3.292 [49,150,46] 49
3.293 [59,33] 59
3.294 [18,71] 18
3.295 [83,1,110] 83
3.296 [110,34,20] 110
3.297 [48,41,106] 48
3.298 [142,64] 142
3.299 [80,54] 80
3.29a [82,55,61] 82
3.29b [21,79,66] 21
3.29c [27,21] 27
3.29d [58,25] 58
3.29e [120,82,87] 120
3.29f [41,84] 41
3.2a0 [92,10,61] 92
3.2a1 [85,110] 85
3.2a2 [13,2,52] 13
3.2a3 [3,122] 3
3.2a4 [88,87,66] 88
3.2a5 [14,88,129] 14
3.2a6 [144,73,2] 144
3.2a7 [33,4,59] 33
3.2a8 [113,44,66] 113
3.2a9 [92,30,31] 92
3.2aa [45,74] 45
3.2ab [132,20,14] 132
3.2ac [7,98,76] 7
3.2ad [18,87,25] 18
3.2ae [77,94] 77
3.2af [24,82] 24
3.2b0 [63,122,70] 63
3.2b1 [22,91] 22
3.2b2 [5,144] 5
3.2b3 [14,32,15] 14
3.2b4 [146,29,78] 146
3.2b5 [51,72] 51
3.2b6 [93,130,10] 93
3.2b7 [64,115,1] 64
3.2b8 [25,66] 25
3.2b9 [34,31,127] 34
3.2ba [43,41] 43
3.2bb [10,144,57] 10
3.2bc [14,23,65] 14
3.2bd [148,94] 148
3.2be [60,110] 60
3.2bf [46,9] 46
3.2c0 [87,64,98] 87
3.2c1 [38,29,146] 38
3.2c2 [62,127,71] 62
3.2c3 [78,75,118] 78
3.2c4 [150,94,93] 150
3.2c5 [74,138,83] 74
3.2c6 [118,94] 118
3.2c7 [10,61] 10
3.2c8 [65,100] 65
3.2c9 [18,57] 18
3.2ca [70,69] 70
3.2cb [31,144,13] 31
3.2cc [33,58] 33
3.2cd [118,41] 118
3.2ce [25,132] 25
3.2cf [102,3,80] 102
3.2d0 [61,106,32] 61
3.2d1 [59,10,4] 59
3.2d2 [74,12,83] 74
3.2d3 [8,25,132] 8
3.2d4 [91,122,6] 91
3.2d5 [132,17,54] 132
3.2d6 [90,77] 90
3.2d7 [49,32] 49
3.2d8 [78,100] 78
3.2d9 [47,124] 47
3.2da [64,25,71] 64
3.2db [15,106] 15
3.2dc [7,74] 7
3.2dd [55,148] 55
3.2de [79,70] 79
3.2df [80,110] 80
3.2e0 [65,118] 65
3.2e1 [47,40,142] 47
3.2e2 [148,11] 148
3.2e3 [132,40,95] 132
3.2e4 [45,51] 45
3.2e5 [69,42] 69
3.2e6 [7,140] 7
3.2e7 [18,61] 18
3.2e8 [76,70,67] 76
3.2e9 [24,130] 24
3.2ea [116,21] 116
3.2eb [42,93,94] 42
3.2ec [12,70] 12
3.2ed [115,69] 115
3.2ee [38,32] 38
3.2ef [56,6,50] 56
3.2f0 [105,93] 105
3.2f1 [61,108] 61
3.2f2 [7,9] 7
3.2f3 [89,30] 89
3.2f4 [36,25] 36
3.2f5 [93,142,8] 93
3.2f6 [21,2,88] 21
3.2f7 [48,6,75] 48
3.2f8 [2,6] 2
3.2f9 [35,106] 35
3.2fa [73,94] 73
3.2fb [48,140,27] 48
3.2fc [52,100,21] 52
3.2fd [66,2] 66
3.2fe [78,95] 78
3.2ff [1,148] 1
3.300 [90,20] 90
3.301 [93,15,20] 93
3.302 [72,118] 72
3.303 [34,3] 34
3.304 [27,91,48] 27
3.305 [52,18] 52
3.306 [16,81] 16
3.307 [83,78,65] 83
3.308 [41,50,132] 41
3.309 [28,122] 28
3.30a [74,146] 74
3.30b [65,51] 65
3.30c [136,52,66] 136
3.30d [20,11] 20
3.30e [82,29,15] 82
3.30f [23,47,12] 23
3.310 [41,12] 41
3.311 [79,148] 79
3.312 [26,49] 26
3.313 [63,57,54] 63
3.314 [43,83] 43
3.315 [95,85,47] 95
3.316 [129,86,84] 129
3.317 [94,40,34] 94
3.318 [105,46] 105
3.319 [140,76] 140
3.31a [36,73] 36
3.31b [75,102] 75
3.31c [64,110] 64
3.31d [1,144,63] 1
3.31e [30,68,134] 30
3.31f [60,58] 60
3.320 [47,61] 47
3.321 [6,72,132] 6
3.322 [110,60] 110
3.323 [18,130] 18
3.324 [85,19,136] 85
3.325 [83,113,92] 83
3.326 [54,53,68] 54
3.327 [26,120] 26
3.328 [6,5,19] 6
3.329 [10,7] 10
3.32a [47,142,88] 47
3.32b [2,60] 2
3.32c [98,70] 98
3.32d [55,40] 55
3.32e [53,58,87] 53
3.32f [30,68,94] 30
3.330 [118,50] 118
3.331 [67,47] 67
3.332 [58,61,127] 58
3.333 [88,13,5] 88
3.334 [113,86] 113
3.335 [18,150] 18
3.336 [23,60] 23
3.337 [57,115,76] 57
3.338 [110,86,51] 110
3.339 [8,70,29] 8
3.33a [38,49] 38
3.33b [23,93] 23
3.33c [21,58,40] 21
3.33d [62,55,116] 62
3.33e [129,64,1] 129
3.33f [48,132,21] 48
3.340 [70,144,33] 70
3.341 [15,68] 15
3.342 [102,48,66] 102
3.343 [122,90,88] 122
3.344 [37,48,118] 37
3.345 [1,40,46] 1
3.346 [6,87] 6
3.347 [132,51,85] 132
3.348 [58,4,38] 58
3.349 [73,1] 73
3.34a [38,31] 38
3.34b [23,142] 23
3.34c [24,91] 24
3.34d [132,15,13] 132
3.34e [44,148] 44
3.34f [102,94,61] 102
3.350 [72,127] 72
3.351 [15,144] 15
3.352 [31,75] 31
3.353 [53,87] 53
3.354 [118,49] 118
3.355 [86,77,144] 86
3.356 [146,15] 146
3.357 [33,132] 33
3.358 [94,33] 94
3.359 [127,63] 127
3.35a [108,7,90] 108
3.35b [105,70] 105
3.35c [84,79,100] 84
3.35d [80,9] 80
3.35e [65,75,1] 65
3.35f [89,102,1] 89
3.360 [130,38,40] 130
3.361 [18,60] 18
3.362 [19,28,17] 19
3.363 [24,63,90] 24
3.364 [130,49] 130
3.365 [81,136] 81
3.366 [8,51] 8
3.367 [106,67] 106
3.368 [47,7] 47
3.369 [36,50] 36
3.36a [16,31,11] 16
3.36b [72,50] 72
3.36c [38,24] 38
3.36d [100,86,51] 100
3.36e [4,130] 4
3.36f [45,136] 45
3.370 [40,105] 40
3.371 [55,70,148] 55
3.372 [44,21,46] 44
3.373 [144,24,21] 144
3.374 [90,49,75] 90
3.375 [62,2] 62
3.376 [29,124] 29
3.377 [10,21] 10
3.378 [24,134,51] 24
3.379 [36,105,81] 36
3.37a [73,74] 73
3.37b [54,79,88] 54
3.37c [29,115,16] 29
3.37d [84,57] 84
3.37e [91,94] 91
3.37f [23,24,129] 23
3.380 [6,48,54] 6
3.381 [129,30] 129
3.382 [110,24] 110
3.383 [45,130] 45
3.384 [96,13,90] 96
3.385 [82,21,124] 82
3.386 [12,50] 12
3.387 [18,42] 18
3.388 [78,132] 78
3.389 [78,115,42] 78
3.38a [52,96] 52
3.38b [43,8] 43
3.38c [14,8] 14
3.38d [136,63,68] 136
3.38e [22,102] 22
3.38f [120,2] 120
3.390 [68,9] 68
3.391 [124,45] 124
3.392 [73,79,8] 73
3.393 [1,65] 1
3.394 [20,45] 20
3.395 [45,47] 45
3.396 [102,80] 102
3.397 [69,51] 69
3.398 [140,95] 140
3.399 [14,116,91] 14
3.39a [2,82] 2
3.39b [13,91,8] 13
3.39c [81,27] 81
3.39d [65,58,118] 65
3.39e [93,74] 93
3.39f [21,83,52] 21
3.3a0 [23,47] 23
3.3a1 [89,118] 89
3.3a2 [92,5] 92
3.3a3 [10,98] 10
3.3a4 [32,130] 32
3.3a5 [79,58] 79
3.3a6 [15,28] 15
3.3a7 [56,33,43] 56
3.3a8 [68,142,2] 68
3.3a9 [76,142,21] 76
3.3aa [63,64] 63
3.3ab [19,118,25] 19
3.3ac [64,38,144] 64
3.3ad [59,142,41] 59
3.3ae [3,58] 3
3.3af [105,27] 105
3.3b0 [15,44] 15
3.3b1 [82,71,118] 82
3.3b2 [49,31,67] 49
3.3b3 [35,76,21] 35
3.3b4 [7,58,76] 7
3.3b5 [102,23] 102
3.3b6 [50,142] 50
3.3b7 [34,122] 34
3.3b8 [138,34,81] 138
3.3b9 [0,142] 0
3.3ba [57,4] 57
3.3bb [122,11,62] 122
3.3bc [120,6] 120
3.3bd [21,59] 21
3.3be [105,1,13] 105
3.3bf [72,77,82] 72
3.3c0 [8,66,120] 8
3.3c1 [43,140,48] 43
3.3c2 [88,31] 88
3.3c3 [94,83,52] 94
3.3c4 [45,130,30] 45
3.3c5 [25,90] 25
3.3c6 [115,53,23] 115
3.3c7 [71,150] 71
3.3c8 [53,1,67] 53
3.3c9 [37,100] 37
3.3ca [52,45,30] 52
3.3cb [68,71,90] 68
3.3cc [92,118,73] 92
3.3cd [5,56,110] 5
3.3ce [15,146] 15
3.3cf [74,44,29] 74
3.3d0 [13,76] 13
3.3d1 [5,38,142] 5
3.3d2 [7,148,93] 7
3.3d3 [35,130] 35
3.3d4 [20,58,91] 20
3.3d5 [62,146,44] 62
3.3d6 [50,118,48] 50
3.3d7 [79,130] 79
3.3d8 [86,24,115] 86
3.3d9 [6,120,2] 6
3.3da [136,63] 136
3.3db [146,7] 146
3.3dc [22,64,25] 22
3.3dd [44,120,42] 44
3.3de [85,124] 85
3.3df [0,132] 0
3.3e0 [78,72] 78
3.3e1 [32,30] 32
3.3e2 [59,61] 59
3.3e3 [74,98,4] 74
3.3e4 [54,83,25] 54
3.3e5 [0,93,5] 0
3.3e6 [87,27] 87
3.3e7 [46,48] 46
3.3e8 [134,77] 134
3.3e9 [27,24] 27
3.3ea [108,27] 108
3.3eb [127,47,19] 127
3.3ec [68,37,102] 68
3.3ed [5,20,59] 5
3.3ee [150,81] 150
3.3ef [22,28] 22
3.3f0 [85,148] 85
3.3f1 [76,94] 76
3.3f2 [28,132,27] 28
3.3f3 [40,118,54] 40
3.3f4 [54,129] 54
3.3f5 [127,14,84] 127
3.3f6 [57,70] 57
3.3f7 [96,35] 96
3.3f8 [74,150,13] 74
3.3f9 [67,90,85] 67
3.3fa [79,3] 79
3.3fb [47,138,57] 47
3.3fc [82,81,0] 82
3.3fd [22,0] 22
3.3fe [59,76] 59
3.3ff [24,10] 24
3.400 [54,28] 54
3.401 [76,113] 76
3.402 [98,46,29] 98
3.403 [49,55,94] 49
3.404 [80,35] 80
3.405 [34,90,108] 34
3.406 [22,46] 22
3.407 [7,56] 7
3.408 [56,82,6] 56
3.409 [89,124] 89
3.40a [20,50,83] 20
3.40b [31,146] 31
3.40c [3,30,144] 3
3.40d [68,96,10] 68
3.40e [94,19,118] 94
3.40f [0,54] 0
3.410 [138,31] 138
3.411 [25,44,140] 25
3.412 [41,113] 41
3.413 [53,51,118] 53
3.414 [37,18,100] 37
3.415 [24,85] 24
3.416 [115,58,55] 115
3.417 [122,78] 122
3.418 [38,62,84] 38
3.419 [5,106,67] 5
3.41a [44,42,108] 44
3.41b [2,1,118] 2
3.41c [22,59] 22
3.41d [148,48] 148
3.41e [47,142] 47
3.41f [142,44,27] 142
3.420 [13,150,82] 13
3.421 [59,9,40] 59
3.422 [80,116,9] 80
3.423 [19,92] 19
3.424 [26,94] 26
3.425 [1,113] 1
3.426 [15,4] 15
3.427 [148,71,80] 148
3.428 [35,53] 35
3.429 [80,6] 80
3.42a [110,54,26] 110
3.42b [82,65,16] 82
3.42c [48,132,6] 48
3.42d [2,138] 2
3.42e [47,95,92] 47
3.42f [134,52,82] 134
3.430 [25,24,22] 25
3.431 [8,10,116] 8
3.432 [52,86,77] 52
3.433 [78,20] 78
3.434 [73,23] 73
3.435 [84,53] 84
3.436 [42,94,68] 42
3.437 [53,72] 53
3.438 [65,134] 65
3.439 [6,86,132] 6
3.43a [144,85,11] 144
3.43b [80,18] 80
3.43c [66,100,21] 66
3.43d [93,86] 93
3.43e [105,30] 105
3.43f [1,44] 1
3.440 [148,37,23] 148
3.441 [146,49] 146
3.442 [79,32,82] 79
3.443 [33,56] 33
3.444 [49,88,146] 49
3.445 [22,85,19] 22
3.446 [146,86] 146
3.447 [92,95,129] 92
3.448 [51,66,113] 51
3.449 [93,87] 93
3.44a [59,1,102] 59
3.44b [63,57,47] 63
3.44c [127,40] 127
3.44d [58,61,72] 58
3.44e [61,79,134] 61
3.44f [23,70] 23
3.450 [81,36,34] 81
3.451 [54,134,15] 54
3.452 [118,64,18] 118
3.453 [129,72] 129
3.454 [75,130,84] 75
3.455 [46,47] 46
3.456 [51,80,18] 51
3.457 [17,98,68] 17
3.458 [37,71] 37
3.459 [27,116,85] 27
3.45a [84,49] 84
3.45b [8,71] 8
3.45c [46,132] 46
3.45d [22,38,76] 22
3.45e [5,13] 5
3.45f [24,129,75] 24
3.460 [98,8] 98
3.461 [60,96] 60
3.462 [0,42] 0
3.463 [130,28,50] 130
3.464 [2,69,118] 2
3.465 [32,130] 32
3.466 [15,68,10] 15
3.467 [34,89] 34
3.468 [105,32,25] 105
3.469 [40,11,57] 40
3.46a [59,41] 59
3.46b [113,51,53] 113
3.46c [9,95] 9
3.46d [22,85,36] 22
3.46e [100,80] 100
3.46f [42,102] 42
3.470 [36,47] 36
3.471 [92,69,136] 92
3.472 [42,60] 42
3.473 [38,33] 38
3.474 [55,14] 55
3.475 [35,17,96] 35
3.476 [11,52,1] 11
3.477 [60,41] 60
3.478 [17,118,42] 17
3.479 [74,11] 74
3.47a [130,95,14] 130
3.47b [19,27] 19
3.47c [50,0] 50
3.47d [74,33,106] 74
3.47e [10,84] 10
3.47f [94,132] 94
3.480 [74,93,113] 74
3.481 [73,38] 73
3.482 [51,68,81] 51
3.483 [1,148,4] 1
3.484 [8,19] 8
3.485 [67,28] 67
3.486 [84,93] 84
3.487 [22,92] 22
3.488 [7,37,116] 7
3.489 [59,45,74] 59
3.48a [19,54] 19
3.48b [144,29,52] 144
3.48c [83,24] 83
3.48d [48,83] 48
3.48e [140,76,61] 140
3.48f [82,118] 82
3.490 [53,72] 53
3.491 [10,73] 10
3.492 [28,13] 28
3.493 [68,110] 68
3.494 [8,63] 8
3.495 [130,50] 130
3.496 [11,12] 11
3.497 [17,86] 17
3.498 [38,124] 38
3.499 [1,77,118] 1
3.49a [17,142,18] 17
3.49b [75,20] 75
3.49c [36,140,6] 36
3.49d [56,30] 56
3.49e [33,92,140] 33
3.49f [6,68] 6
3.4a0 [5,3,108] 5
3.4a1 [129,71] 129
3.4a2 [67,136] 67
3.4a3 [24,94,69] 24
3.4a4 [113,44,49] 113
3.4a5 [28,110,71] 28
3.4a6 [68,74] 68
3.4a7 [20,150] 20
3.4a8 [24,110] 24
3.4a9 [69,102] 69
3.4aa [50,95,144] 50
3.4ab [89,15] 89
3.4ac [91,16,127] 91
3.4ad [41,11] 41
3.4ae [57,70,106] 57
3.4af [4,42,35] 4
3.4b0 [32,14,3] 32
3.4b1 [1,75] 1
3.4b2 [124,29,66] 124
3.4b3 [1,26] 1
3.4b4 [18,51] 18
3.4b5 [146,91,48] 146
3.4b6 [47,138] 47
3.4b7 [8,102] 8
3.4b8 [150,36] 150
3.4b9 [71,127,76] 71
3.4ba [37,47,96] 37
3.4bb [90,40] 90
3.4bc [38,6] 38
3.4bd [46,52,102] 46
3.4be [30,88,29] 30
3.4bf [21,58,108] 21
3.4c0 [17,82] 17
3.4c1 [63,93,76] 63
3.4c2 [29,10,87] 29
3.4c3 [7,18] 7
3.4c4 [84,85] 84
3.4c5 [9,51] 9
3.4c6 [23,9] 23
3.4c7 [146,7,32] 146
3.4c8 [50,142] 50
3.4c9 [124,13] 124
3.4ca [50,32,45] 50
3.4cb [88,65,27] 88
3.4cc [144,26] 144
3.4cd [106,14,60] 106
3.4ce [66,38] 66
3.4cf [37,86,84] 37
3.4d0 [47,138] 47
3.4d1 [42,60] 42
3.4d2 [0,33,98] 0
3.4d3 [46,124] 46
3.4d4 [64,78] 64
3.4d5 [146,16,71] 146
3.4d6 [61,11] 61
3.4d7 [5,63] 5
3.4d8 [20,90] 20
3.4d9 [36,45] 36
3.4da [9,40] 9
3.4db [95,17] 95
3.4dc [0,120,10] 0
3.4dd [17,4,83] 17
3.4de [94,113,7] 94
3.4df [120,62,75] 120
3.4e0 [59,54,142] 59
3.4e1 [93,130] 93
3.4e2 [116,23] 116
3.4e3 [127,91] 127
3.4e4 [90,32,130] 90
3.4e5 [28,71,82] 28
3.4e6 [134,6] 134
3.4e7 [73,122] 73
3.4e8 [96,91,10] 96
3.4e9 [6,43] 6
3.4ea [122,91] 122
3.4eb [58,129,76] 58
3.4ec [94,115,92] 94
3.4ed [144,44,37] 144
3.4ee [124,7] 124
3.4ef [40,87] 40
3.4f0 [13,24,63] 13
3.4f1 [3,118,94] 3
3.4f2 [64,108] 64
3.4f3 [146,93] 146
3.4f4 [142,57] 142
3.4f5 [90,120] 90
3.4f6 [83,105,93] 83
3.4f7 [20,129] 20
3.4f8 [5,11] 5
3.4f9 [115,91] 115
3.4fa [24,110] 24
3.4fb [49,58] 49
3.4fc [7,58] 7
3.4fd [20,124] 20
3.4fe [53,94] 53
3.4ff [24,116] 24
3.500 [68,47] 68
3.501 [118,90] 118
3.502 [44,150] 44
3.503 [57,47,36] 57
3.504 [7,48,148] 7
3.505 [55,110] 55
3.506 [56,138,1] 56
3.507 [16,73] 16
3.508 [86,17,91] 86
3.509 [35,6] 35
3.50a [1,129,87] 1
3.50b [1,41] 1
3.50c [59,17] 59
3.50d [86,25] 86
3.50e [96,1] 96
3.50f [87,17] 87
3.510 [27,30] 27
3.511 [33,105,72] 33
3.512 [4,95,57] 4
3.513 [64,23,5] 64
3.514 [122,52] 122
3.515 [92,53] 92
3.516 [115,6,56] 115
3.517 [63,142] 63
3.518 [102,30,20] 102
3.519 [16,41,138] 16
3.51a [64,13] 64
3.51b [87,17,18] 87
3.51c [34,41,52] 34
3.51d [17,75] 17
3.51e [136,95,12] 136
3.51f [49,78,102] 49
3.520 [75,144,70] 75
3.521 [27,17] 27
3.522 [8,43,110] 8
3.523 [37,67] 37
3.524 [46,20,62] 46
3.525 [49,95] 49
3.526 [49,116,78] 49
3.527 [138,12] 138
3.528 [87,65,144] 87
3.529 [78,46,124] 78
3.52a [40,134,47] 40
3.52b [12,35,146] 12
3.52c [43,67] 43
3.52d [89,19] 89
3.52e [90,69] 90
3.52f [110,71,61] 110
3.530 [1,77,108] 1
3.531 [91,68,57] 91
3.532 [6,136,83] 6
3.533 [118,0] 118
3.534 [89,2] 89
3.535 [13,62] 13
3.536 [9,140] 9
3.537 [12,34,136] 12
3.538 [6,110,15] 6
3.539 [71,36] 71
3.53a [54,83] 54
3.53b [26,113,78] 26
3.53c [71,129,50] 71
3.53d [5,3,19] 5
3.53e [29,67,47] 29
3.53f [9,100,67] 9
3.540 [70,36,61] 70
3.541 [10,9] 10
3.542 [26,142] 26
3.543 [113,73] 113
3.544 [18,49] 18
3.545 [136,95] 136
3.546 [113,0] 113
3.547 [67,72] 67
3.548 [82,34] 82
3.549 [57,122] 57
3.54a [83,8,89] 83
3.54b [94,120] 94
3.54c [82,57] 82
3.54d [106,90] 106
3.54e [25,48,129] 25
3.54f [18,4] 18
3.550 [86,20] 86
3.551 [37,86,16] 37
3.552 [28,95,13] 28
3.553 [84,45] 84
3.554 [34,20] 34
3.555 [8,42,50] 8
3.556 [66,34] 66
3.557 [36,77] 36
3.558 [9,63,80] 9
3.559 [98,81] 98
3.55a [76,43,57] 76
3.55b [26,73] 26
3.55c [25,8,142] 25
3.55d [26,86] 26
3.55e [89,113] 89
3.55f [63,86] 63
3.560 [130,73,11] 130
3.561 [53,90] 53
3.562 [71,115,43] 71
3.563 [17,115,8] 17
3.564 [17,62,146] 17
3.565 [127,32] 127
3.566 [59,44] 59
3.567 [95,4,118] 95
3.568 [150,77] 150
3.569 [76,67] 76
3.56a [26,62] 26
3.56b [6,74,0] 6
3.56c [136,7,78] 136
3.56d [1,124] 1
3.56e [129,20] 129
3.56f [98,12,95] 98
3.570 [105,13,14] 105
3.571 [56,35] 56
3.572 [55,0,124] 55
3.573 [113,50,60] 113
3.574 [8,77,132] 8
3.575 [75,10,81] 75
3.576 [94,60,63] 94
3.577 [32,19] 32
3.578 [108,32,21] 108
3.579 [64,83,93] 64
3.57a [86,132] 86
3.57b [146,92,86] 146
3.57c [24,50,122] 24
3.57d [27,24,136] 27
3.57e [91,136,73] 91
3.57f [28,83] 28
3.580 [23,54,33] 23
3.581 [1,93,130] 1
3.582 [31,46,150] 31
3.583 [26,132,0] 26
3.584 [72,108,11] 72
3.585 [18,98] 18
3.586 [96,94] 96
3.587 [93,82] 93
3.588 [124,86,45] 124
3.589 [40,66,13] 40
3.58a [89,30] 89
3.58b [73,87] 73
3.58c [140,4,19] 140
3.58d [13,66] 13
3.58e [72,89] 72
3.58f [96,42] 96
3.590 [142,14] 142
3.591 [11,29] 11
3.592 [106,9,47] 106
3.593 [47,44] 47
3.594 [64,5,67] 64
3.595 [82,83,57] 82
3.596 [130,38] 130
3.597 [62,91,4] 62
3.598 [79,35] 79
3.599 [5,53,67] 5
3.59a [95,61,72] 95
3.59b [108,89] 108
3.59c [130,20] 130
3.59d [91,62,37] 91
3.59e [15,102,10] 15
3.59f [26,22] 26
3.5a0 [14,130,41] 14
3.5a1 [31,142,15] 31
3.5a2 [57,10] 57
3.5a3 [77,102] 77
3.5a4 [59,33,102] 59
3.5a5 [88,2,102] 88
3.5a6 [65,16,11] 65
3.5a7 [116,49,55] 116
3.5a8 [132,6,34] 132
3.5a9 [67,88,58] 67
3.5aa [38,85] 38
3.5ab [2,29,1] 2
3.5ac [0,93] 0
3.5ad [134,24] 134
3.5ae [49,96,64] 49
3.5af [95,66,6] 95
3.5b0 [122,11,56] 122
3.5b1 [122,56,11] 122
3.5b2 [75,36,74] 75
3.5b3 [52,21] 52
3.5b4 [105,38] 105
3.5b5 [129,94] 129
3.5b6 [102,86] 102
3.5b7 [138,25,1] 138
3.5b8 [79,31] 79
3.5b9 [11,62] 11
3.5ba [14,130,60] 14
3.5bb [110,60] 110
3.5bc [76,18,37] 76
3.5bd [56,27] 56
3.5be [130,16,75] 130
3.5bf [105,72,77] 105
3.5c0 [11,93,58] 11
3.5c1 [90,55,105] 90
3.5c2 [49,98,50] 49
3.5c3 [28,79,132] 28
3.5c4 [45,76] 45
3.5c5 [105,58] 105
3.5c6 [58,100] 58
3.5c7 [27,73] 27
3.5c8 [6,46] 6
3.5c9 [13,0,59] 13
3.5ca [118,95] 118
3.5cb [11,37,64] 11
3.5cc [57,4] 57
3.5cd [40,47] 40
3.5ce [31,24] 31
3.5cf [1,8,25] 1
3.5d0 [116,31] 116
3.5d1 [83,74,129] 83
3.5d2 [31,32] 31
3.5d3 [69,70] 69
3.5d4 [8,15] 8
3.5d5 [17,31] 17
3.5d6 [61,16] 61
3.5d7 [10,132] 10
3.5d8 [43,136] 43
3.5d9 [34,106,77] 34
3.5da [3,20,58] 3
3.5db [9,12] 9
3.5dc [129,29] 129
3.5dd [85,96] 85
3.5de [95,1,25] 95
3.5df [124,11,37] 124
3.5e0 [24,34] 24
3.5e1 [77,11] 77
3.5e2 [100,83,69] 100
3.5e3 [127,48,58] 127
3.5e4 [86,60,124] 86
3.5e5 [41,74] 41
3.5e6 [30,40,93] 30
3.5e7 [94,17,84] 94
3.5e8 [23,1,37] 23
3.5e9 [54,60] 54
3.5ea [124,0] 124
3.5eb [142,24,1] 142
3.5ec [17,134,79] 17
3.5ed [74,148,52] 74
3.5ee [38,36,9] 38
3.5ef [82,17] 82
3.5f0 [43,105] 43
3.5f1 [68,82] 68
3.5f2 [96,63,54] 96
3.5f3 [83,144,18] 83
3.5f4 [41,56,30] 41
3.5f5 [58,71,29] 58
3.5f6 [59,105] 59
3.5f7 [73,134] 73
3.5f8 [52,18,21] 52
3.5f9 [18,64,118] 18
3.5fa [129,46] 129
3.5fb [92,31] 92
3.5fc [52,86] 52
3.5fd [2,43,60] 2
3.5fe [3,42] 3
3.5ff [40,148,82] 40
3.600 [82,110] 82
3.601 [95,1] 95
3.602 [68,43,69] 68
3.603 [24,83,90] 24
3.604 [56,146,67] 56
3.605 [69,148] 69
3.606 [7,92] 7
3.607 [42,13] 42
3.608 [91,94] 91
3.609 [116,49] 116
3.60a [71,27,102] 71
3.60b [31,4,41] 31
3.60c [38,52] 38
3.60d [102,76,77] 102
3.60e [3,95,144] 3
3.60f [113,92,19] 113
3.610 [25,66,23] 25
3.611 [57,142] 57
3.612 [38,68] 38
3.613 [21,67] 21
3.614 [50,48,124] 50
3.615 [53,132] 53
3.616 [60,59,122] 60
3.617 [14,108,7] 14
3.618 [13,34,129] 13
3.619 [47,142,29] 47
3.61a [122,90] 122
3.61b [150,31,44] 150
3.61c [47,87,92] 47
3.61d [90,60,53] 90
3.61e [2,134] 2
3.61f [47,127,15] 47
3.620 [29,59] 29
3.621 [110,11] 110
3.622 [57,18,102] 57
3.623 [80,45] 80
3.624 [55,80] 55
3.625 [16,78,55] 16
3.626 [15,33] 15
3.627 [6,79] 6
3.628 [92,53] 92
3.629 [7,43] 7
3.62a [48,6] 48
3.62b [110,6] 110
3.62c [25,120,86] 25
3.62d [130,70,42] 130
3.62e [37,43,110] 37
3.62f [28,93] 28
3.630 [132,40,55] 132
3.631 [49,38] 49
3.632 [146,64] 146
3.633 [23,33] 23
3.634 [71,77,70] 71
3.635 [25,70] 25
3.636 [36,14] 36
3.637 [93,0] 93
3.638 [27,88] 27
3.639 [87,45] 87
3.63a [79,22,140] 79
3.63b [32,93,55] 32
3.63c [136,93,66] 136
3.63d [78,26] 78
3.63e [81,120,23] 81
3.63f [66,49,4] 66
3.640 [25,64,18] 25
3.641 [79,52,89] 79
3.642 [44,82] 44
3.643 [7,44,35] 7
3.644 [30,106,6] 30
3.645 [94,20] 94
3.646 [87,106,80] 87
3.647 [11,108,17] 11
3.648 [16,27] 16
3.649 [140,91] 140
3.64a [118,69] 118
3.64b [10,63] 10
3.64c [74,72,63] 74
3.64d [140,70,23] 140
3.64e [51,24,74] 51
3.64f [78,40] 78
3.650 [62,21,91] 62
3.651 [100,23] 100
3.652 [75,100,41] 75
3.653 [57,91,122] 57
3.654 [106,62] 106
3.655 [42,77,32] 42
3.656 [9,113,10] 9
3.657 [10,2,110] 10
3.658 [136,79] 136
3.659 [82,129,59] 82
3.65a [69,44] 69
3.65b [130,56] 130
3.65c [4,96,38] 4
3.65d [7,25] 7
3.65e [89,68,1] 89
3.65f [26,74,136] 26
3.660 [36,69] 36
3.661 [100,74] 100
3.662 [36,89] 36
3.663 [9,59,102] 9
3.664 [22,19] 22
3.665 [70,150] 70
3.666 [105,57,26] 105
3.667 [77,1,68] 77
3.668 [16,144] 16
3.669 [130,34,8] 130
3.66a [81,14] 81
3.66b [50,76] 50
3.66c [113,58] 113
3.66d [116,60,38] 116
3.66e [78,79] 78
3.66f [5,28,15] 5
3.670 [89,30,31] 89
3.671 [52,122] 52
3.672 [132,53,28] 132
3.673 [90,77,108] 90
3.674 [122,23,62] 122
3.675 [48,54,37] 48
3.676 [146,29] 146
3.677 [72,19] 72
3.678 [18,146,15] 18
3.679 [82,116,80] 82
3.67a [75,64,57] 75
3.67b [9,82] 9
3.67c [46,66,138] 46
3.67d [2,73,98] 2
3.67e [25,116,95] 25
3.67f [76,30] 76
3.680 [150,61,27] 150
3.681 [37,110,11] 37
3.682 [7,124] 7
3.683 [74,142] 74
3.684 [17,95] 17
3.685 [72,86,136] 72
3.686 [108,77,74] 108
3.687 [23,105] 23
3.688 [51,124] 51
3.689 [106,83,56] 106
3.68a [56,69] 56
3.68b [58,105] 58
3.68c [8,41] 8
3.68d [94,67] 94
3.68e [49,63] 49
3.68f [144,42] 144
3.690 [96,1,24] 96
3.691 [68,108,2] 68
3.692 [110,68] 110
3.693 [40,66] 40
3.694 [0,69,46] 0
3.695 [71,136,13] 71
3.696 [54,83] 54
3.697 [25,100] 25
3.698 [87,142] 87
3.699 [45,94] 45
3.69a [127,4,55] 127
3.69b [48,1,26] 48
3.69c [43,124,59] 43
3.69d [76,2,124] 76
3.69e [100,48] 100
3.69f [94,3] 94
3.6a0 [41,84] 41
3.6a1 [11,88] 11
3.6a2 [122,43] 122
3.6a3 [1,130,2] 1
3.6a4 [15,80,50] 15
3.6a5 [9,83] 9
3.6a6 [5,44,134] 5
3.6a7 [22,28,69] 22
3.6a8 [13,122,62] 13
3.6a9 [54,24] 54
3.6aa [13,79] 13
3.6ab [102,3,26] 102
3.6ac [34,100] 34
3.6ad [86,68,138] 86
3.6ae [96,62] 96
3.6af [75,56] 75
3.6b0 [102,60] 102
3.6b1 [43,89,32] 43
3.6b2 [124,30] 124
3.6b3 [74,3] 74
3.6b4 [102,51,60] 102
3.6b5 [136,16] 136
3.6b6 [30,35] 30
3.6b7 [20,127,51] 20
3.6b8 [30,4,54] 30
3.6b9 [127,36] 127
3.6ba [56,106] 56
3.6bb [92,67] 92
3.6bc [7,124] 7
3.6bd [47,148,77] 47
3.6be [86,80,129] 86
3.6bf [88,69] 88
3.6c0 [80,89] 80
3.6c1 [49,36,134] 49
3.6c2 [105,71] 105
3.6c3 [63,50] 63
3.6c4 [2,61,28] 2
3.6c5 [12,90] 12
3.6c6 [138,23] 138
3.6c7 [17,59] 17
3.6c8 [34,100,27] 34
3.6c9 [2,102] 2
3.6ca [130,71,25] 130
3.6cb [20,116] 20
3.6cc [73,8] 73
3.6cd [5,53,88] 5
3.6ce [84,86] 84
3.6cf [35,57] 35
3.6d0 [108,92,43] 108
3.6d1 [83,132] 83
3.6d2 [41,1,2] 41
3.6d3 [98,26] 98
3.6d4 [140,32] 140
3.6d5 [129,94] 129
3.6d6 [18,84,41] 18
3.6d7 [118,29,62] 118
3.6d8 [14,118] 14
3.6d9 [53,115] 53
3.6da [140,8] 140
3.6db [116,81] 116
3.6dc [81,83,132] 81
3.6dd [49,63,127] 49
3.6de [13,120,7] 13
3.6df [51,21,28] 51
3.6e0 [52,50] 52
3.6e1 [50,49,60] 50
3.6e2 [62,7] 62
3.6e3 [48,49,118] 48
3.6e4 [91,43] 91
3.6e5 [10,15] 10
3.6e6 [45,1] 45
3.6e7 [59,35,49] 59
3.6e8 [69,27] 69
3.6e9 [122,66] 122
3.6ea [48,25] 48
3.6eb [80,10] 80
3.6ec [8,49,148] 8
3.6ed [43,24] 43
3.6ee [130,81,36] 130
3.6ef [4,23] 4
3.6f0 [41,7] 41
3.6f1 [42,52,89] 42
3.6f2 [148,29] 148
3.6f3 [146,50] 146
3.6f4 [82,34,64] 82
3.6f5 [124,1] 124
3.6f6 [7,5] 7
3.6f7 [102,55] 102
3.6f8 [115,92] 115
3.6f9 [14,13] 14
3.6fa [136,23] 136
3.6fb [92,26,33] 92
3.6fc [140,6] 140
3.6fd [18,72] 18
3.6fe [102,51,93] 102
3.6ff [60,130,82] 60
3.700 [91,17] 91
3.701 [41,56] 41
3.702 [62,64,148] 62
3.703 [118,2,85] 118
3.704 [3,62] 3
3.705 [150,69,31] 150
3.706 [16,95] 16
3.707 [127,94,28] 127
3.708 [18,110] 18
3.709 [19,16,90] 19
3.70a [124,10] 124
3.70b [76,61] 76
3.70c [130,5] 130
3.70d [79,8] 79
3.70e [8,25,130] 8
3.70f [5,96] 5
3.710 [20,82,75] 20
3.711 [146,77,47] 146
3.712 [13,58] 13
3.713 [45,2] 45
3.714 [48,46,124] 48
3.715 [120,63] 120
3.716 [31,68] 31
3.717 [130,80,79] 130
3.718 [61,56,78] 61
3.719 [88,122,50] 88
3.71a [68,142] 68
3.71b [55,150] 55
3.71c [146,26,80] 146
3.71d [110,45,20] 110
3.71e [6,51,92] 6
3.71f [144,46,56] 144
3.720 [78,46,64] 78
3.721 [74,75,28] 74
3.722 [4,41] 4
3.723 [56,110,74] 56
3.724 [96,73,11] 96
3.725 [30,9,50] 30
3.726 [52,1,85] 52
3.727 [45,87,54] 45
3.728 [43,57] 43
3.729 [67,96,43] 67
3.72a [65,102,26] 65
3.72b [63,132,35] 63
3.72c [25,59,68] 25
3.72d [110,94,51] 110
3.72e [118,81,14] 118
3.72f [77,43] 77
3.730 [63,20] 63
3.731 [90,44,75] 90
3.732 [89,72] 89
3.733 [127,12,35] 127
3.734 [108,54] 108
3.735 [86,37,75] 86
3.736 [38,8,142] 38
3.737 [11,62,24] 11
3.738 [88,86,19] 88
3.739 [7,41,54] 7
3.73a [15,88,124] 15
3.73b [55,102] 55
3.73c [64,53,30] 64
3.73d [64,127] 64
3.73e [110,73,26] 110
3.73f [87,94] 87
3.740 [28,90] 28
3.741 [118,43] 118
3.742 [86,83] 86
3.743 [94,100,8] 94
3.744 [17,95] 17
3.745 [78,93] 78
3.746 [96,62] 96
3.747 [61,118] 61
3.748 [84,30] 84
3.749 [124,14] 124
3.74a [45,86] 45
3.74b [67,76,120] 67
3.74c [110,0,42] 110
3.74d [146,11] 146
3.74e [21,118] 21
3.74f [46,9] 46
3.750 [80,146] 80
3.751 [85,24,79] 85
3.752 [51,14,61] 51
3.753 [51,115] 51
3.754 [3,90] 3
3.755 [67,86,89] 67
3.756 [36,50,9] 36
3.757 [37,90] 37
3.758 [55,17] 55
3.759 [144,37] 144
3.75a [12,25] 12
3.75b [73,71] 73
3.75c [54,71,140] 54
3.75d [140,2] 140
3.75e [80,27] 80
3.75f [110,21] 110
3.760 [110,18] 110
3.761 [25,127,62] 25
3.762 [43,98] 43
3.763 [77,40] 77
3.764 [108,22] 108
3.765 [50,148] 50
3.766 [56,74,116] 56
3.767 [132,92] 132
3.768 [24,26] 24
3.769 [102,69,66] 102
3.76a [19,127,4] 19
3.76b [95,94,129] 95
3.76c [54,85,150] 54
3.76d [106,1,71] 106
3.76e [106,46,20] 106
3.76f [89,138,74] 89
3.770 [142,25] 142
3.771 [58,77,142] 58
3.772 [4,14] 4
3.773 [82,4,67] 82
3.774 [11,85] 11
3.775 [91,100,21] 91
3.776 [28,65] 28
3.777 [21,50] 21
3.778 [68,132] 68
3.779 [148,36] 148
3.77a [49,30,94] 49
3.77b [54,88,73] 54
3.77c [91,105,76] 91
3.77d [56,87] 56
3.77e [61,31,16] 61
3.77f [93,59] 93
3.780 [84,113] 84
3.781 [79,9] 79
3.782 [91,80] 91
3.783 [93,15,144] 93
3.784 [25,2] 25
3.785 [84,98,29] 84
3.786 [120,82,6] 120
3.787 [140,79] 140
3.788 [102,65,32] 102
3.789 [86,56,34] 86
3.78a [81,1,12] 81
3.78b [15,49] 15
3.78c [150,72] 150
3.78d [102,80,26] 102
3.78e [33,32] 33
3.78f [138,29,38] 138
3.790 [30,36] 30
3.791 [120,36] 120
3.792 [49,88] 49
3.793 [113,31,48] 113
3.794 [47,21,38] 47
3.795 [90,42] 90
3.796 [75,49] 75
3.797 [7,70] 7
3.798 [65,19] 65
3.799 [65,52] 65
3.79a [17,146] 17
3.79b [86,26] 86
3.79c [138,33] 138
3.79d [44,116] 44
3.79e [68,10,75] 68
3.79f [72,26,120] 72
3.7a0 [46,52,105] 46
3.7a1 [47,89,12] 47
3.7a2 [4,15,74] 4
3.7a3 [86,134] 86
3.7a4 [106,46,57] 106
3.7a5 [75,89] 75
3.7a6 [21,138] 21
3.7a7 [58,88] 58
3.7a8 [11,113] 11
3.7a9 [42,49] 42
3.7aa [150,78,4] 150
3.7ab [69,36,71] 69
3.7ac [106,53,28] 106
3.7ad [92,110,15] 92
3.7ae [48,57,106] 48
3.7af [95,96,45] 95
3.7b0 [93,23,127] 93
3.7b1 [8,108] 8
3.7b2 [124,79] 124
3.7b3 [22,130,48] 22
3.7b4 [129,8,5] 129
3.7b5 [29,43] 29
3.7b6 [22,21,28] 22
3.7b7 [69,67,70] 69
3.7b8 [56,27] 56
3.7b9 [92,150,57] 92
3.7ba [36,18,73] 36
3.7bb [44,50] 44
3.7bc [90,144,83] 90
3.7bd [63,148] 63
3.7be [9,132,94] 9
3.7bf [31,34,37] 31
3.7c0 [84,98,27] 84
3.7c1 [146,75] 146
3.7c2 [49,68,14] 49
3.7c3 [138,69] 138
3.7c4 [124,74,4] 124
3.7c5 [67,29] 67
3.7c6 [86,148,84] 86
3.7c7 [134,94] 134
3.7c8 [0,94,130] 0
3.7c9 [108,1] 108
3.7ca [29,150] 29
3.7cb [102,20] 102
3.7cc [53,110,10] 53
3.7cd [55,115] 55
3.7ce [0,2] 0
3.7cf [22,127] 22
3.7d0 [19,27,6] 19
3.7d1 [87,134,86] 87
3.7d2 [62,6] 62
3.7d3 [116,33] 116
3.7d4 [19,85] 19
3.7d5 [27,120,91] 27
3.7d6 [25,50,71] 25
3.7d7 [1,63] 1
3.7d8 [84,45,59] 84
3.7d9 [4,86] 4
3.7da [26,0] 26
3.7db [3,70] 3
3.7dc [14,124] 14
3.7dd [59,66] 59
3.7de [81,76] 81
3.7df [46,8] 46
3.7e0 [106,37] 106
3.7e1 [52,79,41] 52
3.7e2 [47,69] 47
3.7e3 [92,74,63] 92
3.7e4 [19,73,62] 19
3.7e5 [90,89,115] 90
3.7e6 [87,102] 87
3.7e7 [15,25,76] 15
3.7e8 [55,80,89] 55
3.7e9 [144,81] 144
3.7ea [29,124] 29
3.7eb [132,11,68] 132
3.7ec [82,80] 82
3.7ed [83,25] 83
3.7ee [61,62] 61
3.7ef [4,57] 4
3.7f0 [144,70,28] 144
3.7f1 [7,52] 7
3.7f2 [12,29] 12
3.7f3 [57,72] 57
3.7f4 [43,56,113] 43
3.7f5 [71,17,110] 71
3.7f6 [144,12] 144
3.7f7 [124,72] 124
3.7f8 [150,81,54] 150
3.7f9 [74,136,57] 74
3.7fa [118,90] 118
3.7fb [13,66] 13
3.7fc [89,144,24] 89
3.7fd [6,98] 6
3.7fe [34,22,49] 34
3.7ff [47,38] 47
#osd count first primary c wt wt
osd.0 43 16 16 3.62999 1
osd.1 57 21 21 3.62999 1
osd.2 44 18 18 3.62999 1
osd.3 34 15 15 3.62999 1
osd.4 43 14 14 3.62999 1
osd.5 33 19 19 3.62999 1
osd.6 48 19 19 3.62999 1
osd.7 48 26 26 3.62999 1
osd.8 48 22 22 3.62999 1
osd.9 40 18 18 3.62999 1
osd.10 43 16 16 3.62999 1
osd.11 49 17 17 3.62999 1
osd.12 28 9 9 3.62999 1
osd.13 40 18 18 3.62999 1
osd.14 42 18 18 3.62999 1
osd.15 41 19 19 3.62999 1
osd.16 34 18 18 3.62999 1
osd.17 41 21 21 3.62999 1
osd.18 49 22 22 3.62999 1
osd.19 34 15 15 3.62999 1
osd.20 46 18 18 3.62999 1
osd.21 45 15 15 3.62999 1
osd.22 34 20 20 3.62999 1
osd.23 41 18 18 3.62999 1
osd.24 52 23 23 3.62999 1
osd.25 51 22 22 3.62999 1
osd.26 46 17 17 3.62999 1
osd.27 38 13 13 3.62999 1
osd.28 35 14 14 3.62999 1
osd.29 44 12 12 3.62999 1
osd.30 43 16 16 3.62999 1
osd.31 41 17 17 3.62999 1
osd.32 38 11 11 3.62999 1
osd.33 29 8 8 3.62999 1
osd.34 39 16 16 3.62999 1
osd.35 33 16 16 3.62999 1
osd.36 38 15 15 3.62999 1
osd.37 41 18 18 3.62999 1
osd.38 47 21 21 3.62999 1
osd.40 35 16 16 3.62999 1
osd.41 41 16 16 3.62999 1
osd.42 36 15 15 3.62999 1
osd.43 42 17 17 3.62999 1
osd.44 37 12 12 3.62999 1
osd.45 39 17 17 3.62999 1
osd.46 38 16 16 3.62999 1
osd.47 47 23 23 3.62999 1
osd.48 40 18 18 3.62999 1
osd.49 46 19 19 3.62999 1
osd.50 51 18 18 3.62999 1
osd.51 35 12 12 3.62999 1
osd.52 46 20 20 3.62999 1
osd.53 39 16 16 3.62999 1
osd.54 45 17 17 3.62999 1
osd.55 39 17 17 3.62999 1
osd.56 39 19 19 3.62999 1
osd.57 45 14 14 3.62999 1
osd.58 43 17 17 3.62999 1
osd.59 38 21 21 3.62999 1
osd.60 41 10 10 3.62999 1
osd.61 38 16 16 3.62999 1
osd.62 42 17 17 3.62999 1
osd.63 39 15 15 3.62999 1
osd.64 38 17 17 3.62999 1
osd.65 32 16 16 3.62999 1
osd.66 34 9 9 3.62999 1
osd.67 42 17 17 3.62999 1
osd.68 47 18 18 3.62999 1
osd.69 38 14 14 3.62999 1
osd.70 44 10 10 3.62999 1
osd.71 39 11 11 3.62999 1
osd.72 44 17 17 3.62999 1
osd.73 37 16 16 3.62999 1
osd.74 48 25 25 3.62999 1
osd.75 39 17 17 3.62999 1
osd.76 38 12 12 3.62999 1
osd.77 42 11 11 3.62999 1
osd.78 37 21 21 3.62999 1
osd.79 37 14 14 3.62999 1
osd.80 44 22 22 3.62999 1
osd.81 28 11 11 3.62999 1
osd.82 48 24 24 3.62999 1
osd.83 44 15 15 3.62999 1
osd.84 39 20 20 3.62999 1
osd.85 39 15 15 3.62999 1
osd.86 61 21 21 3.62999 1
osd.87 38 20 20 3.62999 1
osd.88 39 14 14 3.62999 1
osd.89 41 18 18 3.62999 1
osd.90 41 15 15 3.62999 1
osd.91 38 14 14 3.62999 1
osd.92 37 20 20 3.62999 1
osd.93 44 15 15 3.62999 1
osd.94 57 18 18 3.62999 1
osd.95 40 12 12 3.62999 1
osd.96 32 17 17 3.62999 1
osd.97 0 0 0 3.62999 1
osd.98 38 14 14 3.62999 1
osd.99 0 0 0 3.62999 1
osd.100 37 11 11 3.62999 1
osd.101 0 0 0 3.62999 1
osd.102 52 22 22 3.62999 1
osd.103 0 0 0 3.62999 1
osd.104 0 0 0 3.62999 1
osd.105 36 20 20 3.62999 1
osd.106 32 13 13 3.62999 1
osd.107 0 0 0 3.62999 1
osd.108 41 17 17 3.62999 1
osd.109 0 0 0 3.62999 1
osd.110 48 19 19 3.62999 1
osd.111 0 0 0 3.62999 1
osd.112 0 0 0 3.62999 1
osd.113 34 18 18 3.62999 1
osd.114 0 0 0 3.62999 1
osd.115 29 12 12 3.62999 1
osd.116 29 10 10 3.62999 1
osd.117 0 0 0 3.62999 1
osd.118 48 18 18 3.62999 1
osd.119 0 0 0 3.62999 1
osd.120 36 12 12 3.62999 1
osd.121 0 0 0 3.62999 1
osd.122 42 20 20 3.62999 1
osd.123 0 0 0 3.62999 1
osd.124 49 18 18 3.62999 1
osd.125 0 0 0 3.62999 1
osd.126 0 0 0 3.62999 1
osd.127 39 18 18 3.62999 1
osd.128 0 0 0 3.62999 1
osd.129 38 17 17 3.62999 1
osd.130 49 22 22 3.62999 1
osd.131 0 0 0 3.62999 1
osd.132 47 15 15 3.62999 1
osd.133 0 0 0 3.62999 1
osd.134 31 12 12 3.62999 1
osd.135 0 0 0 3.62999 1
osd.136 40 18 18 3.62999 1
osd.137 0 0 0 3.62999 1
osd.138 31 15 15 3.62999 1
osd.139 0 0 0 3.62999 1
osd.140 34 20 20 3.62999 1
osd.141 0 0 0 3.62999 1
osd.142 40 10 10 3.62999 1
osd.143 0 0 0 3.62999 1
osd.144 44 19 19 3.62999 1
osd.145 0 0 0 3.62999 1
osd.146 38 21 21 3.62999 1
osd.147 0 0 0 3.62999 1
osd.148 40 14 14 3.62999 1
osd.149 0 0 0 3.62999 1
osd.150 41 18 18 3.62999 1
osd.151 0 0 0 3.62999 1
in 151
avg 33 stddev 16.7417 (0.507324x) (expected 5.741 0.17397x))
min osd.12 28
max osd.86 61
size 0 0
size 1 0
size 2 1134
size 3 914
Matt Conner
Keeper Technology
On Tue, Dec 8, 2015 at 5:35 AM, Ilya Dryomov <idryomov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> On Tue, Dec 8, 2015 at 10:57 AM, Tom Christensen <pavera-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > We aren't running NFS, but regularly use the kernel driver to map RBDs and
> > mount filesystems in same. We see very similar behavior across nearly all
> > kernel versions we've tried. In my experience only very few versions of the
> > kernel driver survive any sort of crush map change/update while something is
> > mapped. In fact in the last 2 years I think I've only seen this work on 1
> > kernel version unfortunately its badly out of date and we can't run it in
> > our environment anymore, I think it was a 3.0 kernel version running on
> > ubuntu 12.04. We have just recently started trying to find a kernel that
> > will survive OSD outages or changes to the cluster. We're on ubuntu 14.04,
> > and have tried 3.16, 3.19.0-25, 4.3, and 4.2 without success in the last
> > week. We only map 1-3 RBDs per client machine at a time but we regularly
> > will get processes stuck in D state which are accessing the filesystem
> > inside the RBD and will have to hard reboot the RBD client machine. This is
> > always associated with a cluster change in some way, reweighting OSDs,
> > rebooting an OSD host, restarting an individual OSD, adding OSDs, and
> > removing OSDs all cause the kernel client to hang. If no change is made to
> > the cluster, the kernel client will be happy for weeks.
>
> There are a couple of known bugs in the remap/resubmit area, but those
> are supposedly corner cases (like *all* the OSDs going down and then
> back up, etc). I had no idea it was that severe and goes that back.
> Apparently triggering it requires a heavier load, as we've never seen
> anything like that in our tests.
>
> For unrelated reasons, remap/resubmit code is getting entirely
> rewritten for kernel 4.5, so, if you've been dealing with this issue
> for the last two years (I don't remember seeing any tickets listing
> that many kernel versions and not mentioning NFS), I'm afraid the best
> course of action for you would be to wait for 4.5 to come out and try
> it. If you'd be willing to test out an early version on one of more of
> your client boxes, I can ping you when it's ready.
>
> I'll take a look at 3.0 vs 3.16 with an eye on remap code. Did you
> happen to try 3.10?
>
> It sounds like you can reproduce this pretty easily. Can you get it to
> lock up and do:
>
> # cat /sys/kernel/debug/ceph/*/osdmap
> # cat /sys/kernel/debug/ceph/*/osdc
> $ ceph status
>
> and bunch of times? I have a hunch that kernel client simply fails to
> request enough of new osdmaps after the cluster topology changes under
> load.
>
> Thanks,
>
> Ilya
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
[not found] ` <CAKeiORRYycaoC6vXZ8TOpf9_oU3it3yrGO=nJen6Fnhiwn=Okw@mail.gmail.com>
@ 2015-12-11 8:26 ` Ilya Dryomov
2015-12-11 13:32 ` Matt Conner
0 siblings, 1 reply; 10+ messages in thread
From: Ilya Dryomov @ 2015-12-11 8:26 UTC (permalink / raw)
To: Matt Conner; +Cc: ceph-devel
On Fri, Dec 11, 2015 at 1:37 AM, Matt Conner <matt.conner@keepertech.com> wrote:
> Hi Ilya,
>
> I had already recovered but I managed to recreate the problem again. I ran
How did you recover?
> the commands against rbd_data.f54f9422698a8.0000000000000000 which was one
> of those listed in osdc this time. We have 2048 PGs in the pool so the list
> is long.
That's not going to work - I need it in a consistent "stuck" state so
I can match the outputs. I understand you can't keep it that way for
a long time, so can you please reproduce it and email me off list with
osdmap, osdc and ceph -s right away?
You have a lot of OSDs but it also doesn't seem to take a long time to
recreate. Bump debug ms to 1 on all OSDs.
>
> As for when I fetched the object using rados, it grabbed it without issue.
>
> osd map:
> osdmap e6247 pool 'NAS-ic2gw01' (3) object
> 'rbd_data.f54f9422698a8.0000000000000000' -> pg 3.cac46c43 (3.443) -> up
> ([33,56], p33) acting ([33,56], p33)
>
> osdmaptool:
> pool 3 pg_num 2048
> 3.0 [23,138] 23
If it's long, better to compress and attach.
Thanks,
Ilya
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Kernel RBD hang on OSD Failure
2015-12-11 8:26 ` Ilya Dryomov
@ 2015-12-11 13:32 ` Matt Conner
0 siblings, 0 replies; 10+ messages in thread
From: Matt Conner @ 2015-12-11 13:32 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: ceph-devel
Hi Ilya,
The only recovery method is to hard reboot the client. Unfortunately
it will not gracefully reboot in this state. I managed to leave the
system in this state overnight and I've placed the requested output
below. Note that rbd_data.f54f9422698a8.0000000000000000 still appears
in the list.
osdmap:
epoch 6255
flags
pool 3 pg_num 2048 (2047) read_tier -1 write_tier -1
pool 8 pg_num 2048 (2047) read_tier -1 write_tier -1
osd0 10.10.30.55:6830 100% (exists, up) 100%
osd1 10.10.30.53:6809 100% (exists, up) 100%
osd2 10.10.30.54:6839 100% (exists, up) 100%
osd3 10.10.30.56:6833 100% (exists, up) 100%
osd4 10.10.30.55:6848 100% (exists, up) 100%
osd5 10.10.30.53:6854 100% (exists, up) 100%
osd6 10.10.30.56:6818 100% (exists, up) 100%
osd7 10.10.30.54:6830 100% (exists, up) 100%
osd8 10.10.30.55:6839 100% (exists, up) 100%
osd9 10.10.30.56:6845 100% (exists, up) 100%
osd10 10.10.30.53:6860 100% (exists, up) 100%
osd11 10.10.30.54:6842 100% (exists, up) 100%
osd12 10.10.30.55:6836 100% (exists, up) 100%
osd13 10.10.30.56:6863 100% (exists, up) 100%
osd14 10.10.30.53:6839 100% (exists, up) 100%
osd15 10.10.30.54:6866 100% (exists, up) 100%
osd16 10.10.30.55:6869 100% (exists, up) 100%
osd17 10.10.30.56:6812 100% (exists, up) 100%
osd18 10.10.30.53:6824 100% (exists, up) 100%
osd19 10.10.30.54:6818 100% (exists, up) 100%
osd20 10.10.30.55:6833 100% (exists, up) 100%
osd21 10.10.30.56:6869 100% (exists, up) 100%
osd22 10.10.30.53:6857 100% (exists, up) 100%
osd23 10.10.30.54:6833 100% (exists, up) 100%
osd24 10.10.30.55:6803 100% (exists, up) 100%
osd25 10.10.30.56:6821 100% (exists, up) 100%
osd26 10.10.30.54:6848 100% (exists, up) 100%
osd27 10.10.30.53:6848 100% (exists, up) 100%
osd28 10.10.30.55:6863 100% (exists, up) 100%
osd29 10.10.30.56:6827 100% (exists, up) 100%
osd30 10.10.30.54:6836 100% (exists, up) 100%
osd31 10.10.30.53:6863 100% (exists, up) 100%
osd32 10.10.30.55:6845 100% (exists, up) 100%
osd33 10.10.30.56:6839 100% (exists, up) 100%
osd34 10.10.30.54:6851 100% (exists, up) 100%
osd35 10.10.30.53:6845 100% (exists, up) 100%
osd36 10.10.30.55:6854 100% (exists, up) 100%
osd37 10.10.30.56:6851 100% (exists, up) 100%
osd38 10.10.30.54:6806 100% (exists, up) 100%
osd39 10.10.30.53:6824 0% (doesn't exist) 100%
osd40 10.10.30.55:6815 100% (exists, up) 100%
osd41 10.10.30.56:6842 100% (exists, up) 100%
osd42 10.10.30.54:6863 100% (exists, up) 100%
osd43 10.10.30.53:6806 100% (exists, up) 100%
osd44 10.10.30.55:6821 100% (exists, up) 100%
osd45 10.10.30.56:6803 100% (exists, up) 100%
osd46 10.10.30.54:6812 100% (exists, up) 100%
osd47 10.10.30.53:6866 100% (exists, up) 100%
osd48 10.10.30.55:6812 100% (exists, up) 100%
osd49 10.10.30.56:6815 100% (exists, up) 100%
osd50 10.10.30.53:6842 100% (exists, up) 100%
osd51 10.10.30.54:6854 100% (exists, up) 100%
osd52 10.10.30.55:6842 100% (exists, up) 100%
osd53 10.10.30.56:6836 100% (exists, up) 100%
osd54 10.10.30.53:6803 100% (exists, up) 100%
osd55 10.10.30.54:6869 100% (exists, up) 100%
osd56 10.10.30.55:6866 100% (exists, up) 100%
osd57 10.10.30.56:6860 100% (exists, up) 100%
osd58 10.10.30.53:6800 100% (exists, up) 100%
osd59 10.10.30.54:6803 100% (exists, up) 100%
osd60 10.10.30.55:6824 100% (exists, up) 100%
osd61 10.10.30.56:6804 100% (exists, up) 100%
osd62 10.10.30.53:6815 100% (exists, up) 100%
osd63 10.10.30.54:6857 100% (exists, up) 100%
osd64 10.10.30.55:6800 100% (exists, up) 100%
osd65 10.10.30.56:6866 100% (exists, up) 100%
osd66 10.10.30.53:6836 100% (exists, up) 100%
osd67 10.10.30.54:6845 100% (exists, up) 100%
osd68 10.10.30.55:6860 100% (exists, up) 100%
osd69 10.10.30.56:6824 100% (exists, up) 100%
osd70 10.10.30.53:6821 100% (exists, up) 100%
osd71 10.10.30.54:6809 100% (exists, up) 100%
osd72 10.10.30.55:6806 100% (exists, up) 100%
osd73 10.10.30.56:6809 100% (exists, up) 100%
osd74 10.10.30.53:6818 100% (exists, up) 100%
osd75 10.10.30.54:6815 100% (exists, up) 100%
osd76 10.10.30.55:6827 100% (exists, up) 100%
osd77 10.10.30.56:6830 100% (exists, up) 100%
osd78 10.10.30.53:6833 100% (exists, up) 100%
osd79 10.10.30.54:6800 100% (exists, up) 100%
osd80 10.10.30.55:6851 100% (exists, up) 100%
osd81 10.10.30.56:6800 100% (exists, up) 100%
osd82 10.10.30.53:6812 100% (exists, up) 100%
osd83 10.10.30.54:6821 100% (exists, up) 100%
osd84 10.10.30.55:6857 100% (exists, up) 100%
osd85 10.10.30.56:6848 100% (exists, up) 100%
osd86 10.10.30.53:6830 100% (exists, up) 100%
osd87 10.10.30.54:6827 100% (exists, up) 100%
osd88 10.10.30.55:6809 100% (exists, up) 100%
osd89 10.10.30.56:6854 100% (exists, up) 100%
osd90 10.10.30.53:6827 100% (exists, up) 100%
osd91 10.10.30.54:6860 100% (exists, up) 100%
osd92 10.10.30.55:6818 100% (exists, up) 100%
osd93 10.10.30.56:6857 100% (exists, up) 100%
osd94 10.10.30.53:6851 100% (exists, up) 100%
osd95 10.10.30.54:6824 100% (exists, up) 100%
osd96 10.10.30.52:6863 100% (exists, up) 100%
osd97 10.10.30.51:6839 100% (exists) 100%
osd98 10.10.30.52:6854 100% (exists, up) 100%
osd99 10.10.30.51:6842 100% (exists) 100%
osd100 10.10.30.52:6809 100% (exists, up) 100%
osd101 10.10.30.51:6851 100% (exists) 100%
osd102 10.10.30.52:6857 100% (exists, up) 100%
osd103 10.10.30.51:6806 100% (exists) 100%
osd104 10.10.30.51:6809 100% (exists) 100%
osd105 10.10.30.52:6827 100% (exists, up) 100%
osd106 10.10.30.52:6881 100% (exists, up) 100%
osd107 10.10.30.51:6845 100% (exists) 100%
osd108 10.10.30.52:6803 100% (exists, up) 100%
osd109 10.10.30.51:6833 100% (exists) 100%
osd110 10.10.30.52:6878 100% (exists, up) 100%
osd111 10.10.30.51:6824 100% (exists) 100%
osd112 10.10.30.51:6803 100% (exists) 100%
osd113 10.10.30.52:6860 100% (exists, up) 100%
osd114 10.10.30.51:6836 100% (exists) 100%
osd115 10.10.30.52:6845 100% (exists, up) 100%
osd116 10.10.30.52:6872 100% (exists, up) 100%
osd117 10.10.30.51:6860 100% (exists) 100%
osd118 10.10.30.52:6806 100% (exists, up) 100%
osd119 10.10.30.51:6830 100% (exists) 100%
osd120 10.10.30.52:6842 100% (exists, up) 100%
osd121 10.10.30.51:6815 100% (exists) 100%
osd122 10.10.30.52:6824 100% (exists, up) 100%
osd123 10.10.30.51:6857 100% (exists) 100%
osd124 10.10.30.52:6839 100% (exists, up) 100%
osd125 10.10.30.51:6866 100% (exists) 100%
osd126 10.10.30.51:6827 100% (exists) 100%
osd127 10.10.30.52:6848 100% (exists, up) 100%
osd128 10.10.30.51:6881 100% (exists) 100%
osd129 10.10.30.52:6836 100% (exists, up) 100%
osd130 10.10.30.52:6812 100% (exists, up) 100%
osd131 10.10.30.51:6818 100% (exists) 100%
osd132 10.10.30.52:6821 100% (exists, up) 100%
osd133 10.10.30.51:6872 100% (exists) 100%
osd134 10.10.30.52:6828 100% (exists, up) 100%
osd135 10.10.30.51:6869 100% (exists) 100%
osd136 10.10.30.52:6833 100% (exists, up) 100%
osd137 10.10.30.51:6812 100% (exists) 100%
osd138 10.10.30.52:6869 100% (exists, up) 100%
osd139 10.10.30.51:6863 100% (exists) 100%
osd140 10.10.30.52:6875 100% (exists, up) 100%
osd141 10.10.30.51:6821 100% (exists) 100%
osd142 10.10.30.52:6800 100% (exists, up) 100%
osd143 10.10.30.51:6878 100% (exists) 100%
osd144 10.10.30.52:6851 100% (exists, up) 100%
osd145 10.10.30.51:6875 100% (exists) 100%
osd146 10.10.30.52:6815 100% (exists, up) 100%
osd147 10.10.30.51:6848 100% (exists) 100%
osd148 10.10.30.52:6866 100% (exists, up) 100%
osd149 10.10.30.51:6800 100% (exists) 100%
osd150 10.10.30.52:6818 100% (exists, up) 100%
osd151 10.10.30.51:6854 100% (exists) 100%
osdc:
2565262 osd33 3.4f7c1f8e rbd_data.f659f7b8275f8.0000000000003c00 read
2565924 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2565925 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2566603 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2566631 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2567779 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2567878 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2568494 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2568512 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2569177 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2569187 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2569548 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2569864 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2570180 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2570496 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2570857 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2571173 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2571534 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2571850 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2572166 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2572482 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2572843 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2573159 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2573475 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2573836 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2574152 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2574468 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2574829 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2575145 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2575461 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2575822 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2576138 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2576454 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2576815 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2577131 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2577447 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2577808 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2578124 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2578440 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2578801 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2579117 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2579433 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2579794 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2580110 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2580426 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2580787 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2581103 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2581419 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2581780 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2582096 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2582412 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2582773 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2583089 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2583405 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2583766 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2584082 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2584398 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2584759 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2585075 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2585391 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2585752 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2586068 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2586384 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2586745 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2587061 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2587377 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2587738 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2588054 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2588370 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2588731 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2589047 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2589363 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2589724 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2590040 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2590356 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2590717 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2591033 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2591349 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2591710 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2592026 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2592342 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2592703 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2593019 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2593335 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2593696 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2593792 osd43 3.485e015f rbd_header.f42e85cd8f5d2 192438'26800595927040 watch
2593793 osd22 3.1679d3fd rbd_header.f54f346170e75 248651'26800595927040 watch
2593794 osd23 3.967f0ac rbd_header.f54f943ff3510 271821'26800595927040 watch
2593795 osd73 3.41d03b49 rbd_header.12628c3de5a 188527'26800595927040 watch
2593796 osd91 3.b3705118 rbd_header.593b7323f25f 243419'26800595927040 watch
2593797 osd148 3.7fb7b2bd rbd_header.4e4491321fb8d 68317'26800595927040 watch
2593798 osd18 3.f82976fd rbd_header.4e449402e564a 57095'26800595927040 watch
2593799 osd138 3.776060c2 rbd_header.f65a11e3967ed 209128'26800595927040 watch
2593800 osd78 3.9ce9d2d8 rbd_header.f54f3f3f78ae 216654'26800595927040 watch
2593801 osd35 3.658c8f8 rbd_header.f54f319d0e468 252143'26800595927040 watch
2593802 osd132 3.4c72415a rbd_header.f659f6ca5381e 212921'26800595927040 watch
2593803 osd40 3.dcb615cd rbd_header.f659f3b6599ce 281696'26800595927040 watch
2593804 osd4 3.2c5dc7d9 rbd_header.f659f7b8275f8 265762'26800595927040 watch
2593805 osd42 3.ff1718f5 rbd_header.f659f2b730b02 230028'26800595927040 watch
2593806 osd144 3.45280ccc rbd_header.f54f91f78d63 225918'26800595927040 watch
2593807 osd55 3.dcad1c74 rbd_header.f54f91e0c08fc 216768'26800595927040 watch
2593808 osd55 3.38db7474 rbd_header.f65adcf2a53f 216765'26800595927040 watch
2593809 osd45 3.ce94805a rbd_header.f66345f368639 295864'26800595927040 watch
2593810 osd140 3.a030d1d8 rbd_header.f66341d1ccf35 290629'26800595927040 watch
2593811 osd57 3.3df2d8ae rbd_header.4e44968299a31 74608'26800595927040 watch
2593812 osd29 3.49c607ca rbd_header.f66343de35900 230429'26800595927040 watch
2593813 osd41 3.a44fd23b rbd_header.1262487bcab5 170538'26800595927040 watch
2593814 osd0 3.184871e7 rbd_header.12622ffb8aff 221268'26800595927040 watch
2593815 osd38 3.b63d7b6c rbd_header.126253d67973 116592'26800595927040 watch
2593816 osd115 3.e6b3a4e rbd_header.1262614d8810 231173'26800595927040 watch
2593817 osd72 3.d9ede1c3 rbd_header.1262115f0cf 140197'26800595927040 watch
2593818 osd12 3.6642c157 rbd_header.12621e7fc115 187593'26800595927040 watch
2593819 osd52 3.854e4dfc rbd_header.593b6491ec9f 280116'26800595927040 watch
2593820 osd93 3.71d7a587 rbd_header.593b5906fecb 248271'26800595927040 watch
2593821 osd57 3.abcf9bf6 rbd_header.593b363a4411 209538'26800595927040 watch
2593822 osd53 3.72c67cfe rbd_header.593b6072b1c7 291406'26800595927040 watch
2593823 osd70 3.10504835 rbd_header.593b35e9ba2b 205415'26800595927040 watch
2593824 osd45 3.8b381aaa rbd_header.593b418e8823 272511'26800595927040 watch
2593825 osd142 3.2b5318b7 rbd_header.593b46f1aa31 210594'26800595927040 watch
2593826 osd68 3.25c6ad00 rbd_header.4e449416a8478 72833'26800595927040 watch
2593827 osd13 3.5102bd35 rbd_header.4e449723d5105 87452'26800595927040 watch
2593828 osd46 3.d0ad9250 rbd_header.4e449104e5eea 75590'26800595927040 watch
2593829 osd48 3.7adea980 rbd_header.4e4495a579b90 73658'26800595927040 watch
2593830 osd10 3.39b4da2e rbd_header.4e44914acd83 60123'26800595927040 watch
2593831 osd2 3.8fe8ba4d rbd_header.e6ec56f5dcb51 200753'26800595927040 watch
2593832 osd6 3.66d6ca30 rbd_header.e6ec57072b44a 235673'26800595927040 watch
2593833 osd80 3.25b25299 rbd_header.e6ec57e4a0ef9 188214'26800595927040 watch
2593834 osd46 3.58e2250 rbd_header.e6ec550078b66 190030'26800595927040 watch
2593835 osd116 3.b29b37d3 rbd_header.e6ec521c83559 180643'26800595927040 watch
2593836 osd80 3.eca20f5e rbd_header.e6ec570293241 160701'26800595927040 watch
2593837 osd80 3.b20df8a9 rbd_header.e6ec53b631e5a 144384'26800595927040 watch
2593838 osd40 3.6cb419d5 rbd_header.e6ec54ba29f38 200692'26800595927040 watch
2593839 osd38 3.5cce84bc rbd_header.f42e822939526 163677'26800595927040 watch
2593840 osd47 3.13094d0 rbd_header.f42e869ead798 165850'26800595927040 watch
2593841 osd102 3.7a15b5b6 rbd_header.f42e823bececa 182482'26800595927040 watch
2593842 osd96 3.2ceb4f46 rbd_header.f42e867b637bd 175132'26800595927040 watch
2593843 osd92 3.b72d8628 rbd_header.f42e8534aeb08 220526'26800595927040 watch
2593844 osd138 3.42b95d27 rbd_header.f42e83bfec56b 154232'26800595927040 watch
2593845 osd57 3.6daeb5a2 rbd_header.f42e82ecebca5 156055'26800595927040 watch
2593846 osd37 3.a133da1c rbd_header.f42e816513fa7 196719'26800595927040 watch
2593847 osd43 3.c0125aba rbd_header.f42e84c075b37 192587'26800595927040 watch
2593848 osd69 3.a3527e5a rbd_header.f65a150a4c7f4 223891'26800595927040 watch
2593849 osd2 3.8a1702f8 rbd_header.f65a1f9017a4 229537'26800595927040 watch
2593850 osd146 3.ee1e56f3 rbd_header.f65a15677948f 200523'26800595927040 watch
2593851 osd21 3.672cde13 rbd_header.f65a143514fd 198930'26800595927040 watch
2593852 osd36 3.97cc7470 rbd_header.f65a11185e747 201523'26800595927040 watch
2593853 osd70 3.18fcd272 rbd_header.f65a1158abf02 197611'26800595927040 watch
2593854 osd82 3.8208b836 rbd_header.f65a12c47fed6 287608'26800595927040 watch
2593855 osd16 3.99d5db06 rbd_header.f54f32ea892e4 250640'26800595927040 watch
2593856 osd90 3.c8100cf5 rbd_header.f54f331b392f7 303745'26800595927040 watch
2593857 osd60 3.c41f015d rbd_header.f659f6f6c01c3 174666'26800595927040 watch
2593858 osd24 3.22b6834c rbd_header.f659f7570a216 252853'26800595927040 watch
2593859 osd100 3.4355d651 rbd_header.f659f4963d64c 245170'26800595927040 watch
2593860 osd80 3.133cf6eb rbd_header.f659f4da10e75 245743'26800595927040 watch
2593861 osd1 3.eb339cb3 rbd_header.f659f7eb5f195 245316'26800595927040 watch
2593862 osd82 3.918d4e00 rbd_header.f659f7c4753bc 220074'26800595927040 watch
2593863 osd92 3.e4e19983 rbd_header.f659f314a46f 269470'26800595927040 watch
2593864 osd115 3.642e224e rbd_header.f659f584de7fd 291576'26800595927040 watch
2593865 osd118 3.50873837 rbd_header.f54f921268b0f 245740'26800595927040 watch
2593866 osd7 3.877d0e06 rbd_header.f54f9253d6b46 230906'26800595927040 watch
2593867 osd63 3.c87ca915 rbd_header.f54f94f111989 299124'26800595927040 watch
2593868 osd124 3.5af1eb91 rbd_header.f54f97996dd85 230331'26800595927040 watch
2593869 osd8 3.7031dc84 rbd_header.f54f967833a1e 179952'26800595927040 watch
2593870 osd129 3.c80e056e rbd_header.f54f9422698a8 279482'26800595927040 watch
2593871 osd98 3.9fa8711e rbd_header.f54f9412b530 228333'26800595927040 watch
2593872 osd67 3.264a59c8 rbd_header.f65ad2e962255 234483'26800595927040 watch
2593873 osd54 3.a56ea23f rbd_header.f65ad31c25f40 304518'26800595927040 watch
2593874 osd80 3.13e40a42 rbd_header.f65ad65e773b1 250650'26800595927040 watch
2593875 osd54 3.1f5c8bf4 rbd_header.f65ad16be944c 255781'26800595927040 watch
2593876 osd3 3.73a827db rbd_header.f65ad4ab53d2 219699'26800595927040 watch
2593877 osd16 3.d4745306 rbd_header.f66347b1b9a6e 250637'26800595927040 watch
2593878 osd69 3.e8130397 rbd_header.f6634622f70bc 207323'26800595927040 watch
2593879 osd74 3.7f33b105 rbd_header.f66347871909c 214234'26800595927040 watch
2593880 osd14 3.cbdfe123 rbd_header.f663455329544 215780'26800595927040 watch
2593881 osd44 3.27439970 rbd_header.f663436a83873 221219'26800595927040 watch
2593882 osd110 3.e75d1953 rbd_header.f663475df588b 283149'26800595927040 watch
2593883 osd4 3.5396c89b rbd_header.f6634a9aaa27 208564'26800595927040 watch
2593884 osd51 3.20cb2f53 rbd_header.f42e81f94a5be 134631'26800595927040 watch
2593885 osd68 3.9f1c071a rbd_header.f54f35d7af2cf 223450'26800595927040 watch
2593886 osd35 3.fe6122f9 rbd_header.f65ad565c0a5d 216244'26800595927040 watch
2594202 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2594518 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2594879 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2595195 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2595511 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2595872 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2596188 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2596504 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2596865 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2597181 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2597497 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2597858 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2598174 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2598490 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2598851 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2599167 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2599483 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2599844 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2600160 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2600476 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2600837 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2601153 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2601469 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2601830 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2602146 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2602462 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2602823 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2603139 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2603455 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2603816 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2604132 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2604448 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2604809 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2605125 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2605441 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2605802 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2606118 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2606434 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2606795 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2607111 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2607427 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2607788 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2608104 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
2608420 osd33 3.cac46c43 rbd_data.f54f9422698a8.0000000000000000 read
ceph status:
cluster f1885784-0159-11e4-be58-a0d3c1f1936c
health HEALTH_WARN 2247 pgs degraded; 2247 pgs stuck unclean; 54
requests are blocked > 32 sec; recovery 1804769/9770742 objects
degraded (18.471%); 28/151 in osds are down; noout flag(s) set
monmap e6: 3 mons at
{ic2mon01=10.10.30.115:6789/0,ic2mon02=10.10.30.116:6789/0,ic2mon03=10.10.30.117:6789/0},
election epoch 74, quorum 0,1,2 ic2mon01,ic2mon02,ic2mon03
osdmap e6255: 151 osds: 123 up, 151 in
flags noout
pgmap v2328636: 4096 pgs, 2 pools, 12709 GB data, 3180 kobjects
35486 GB used, 513 TB / 547 TB avail
1804769/9770742 objects degraded (18.471%)
2247 active+degraded
1849 active+clean
Thanks,
Matt Conner
Keeper Technology
On Fri, Dec 11, 2015 at 3:26 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
> On Fri, Dec 11, 2015 at 1:37 AM, Matt Conner <matt.conner@keepertech.com> wrote:
>> Hi Ilya,
>>
>> I had already recovered but I managed to recreate the problem again. I ran
>
> How did you recover?
>
>> the commands against rbd_data.f54f9422698a8.0000000000000000 which was one
>> of those listed in osdc this time. We have 2048 PGs in the pool so the list
>> is long.
>
> That's not going to work - I need it in a consistent "stuck" state so
> I can match the outputs. I understand you can't keep it that way for
> a long time, so can you please reproduce it and email me off list with
> osdmap, osdc and ceph -s right away?
>
> You have a lot of OSDs but it also doesn't seem to take a long time to
> recreate. Bump debug ms to 1 on all OSDs.
>
>>
>> As for when I fetched the object using rados, it grabbed it without issue.
>>
>> osd map:
>> osdmap e6247 pool 'NAS-ic2gw01' (3) object
>> 'rbd_data.f54f9422698a8.0000000000000000' -> pg 3.cac46c43 (3.443) -> up
>> ([33,56], p33) acting ([33,56], p33)
>>
>> osdmaptool:
>> pool 3 pg_num 2048
>> 3.0 [23,138] 23
>
> If it's long, better to compress and attach.
>
> Thanks,
>
> Ilya
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-12-11 13:32 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-07 20:56 Kernel RBD hang on OSD Failure Matt Conner
[not found] ` <CAKeiORSsNQ-LOWQ0heLZQkekN_mwYQ81XSXfdJe-R1eP1Osz1w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-07 21:55 ` Blair Bethwaite
[not found] ` <CA+z5DswarP0KHomWJocCxT29W1-kyzV2RhDOg7tc-0_DV0D1Kg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-08 9:57 ` Tom Christensen
2015-12-08 10:35 ` [ceph-users] " Ilya Dryomov
[not found] ` <CAOi1vP-Aw4oWV6eC2h7ZkW87FycnBm0XjMMo0T2updLDjMNULQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-11 0:39 ` Matt Conner
2015-12-08 10:59 ` Ilya Dryomov
2015-12-10 23:16 ` Matt Conner
2015-12-10 23:58 ` Ilya Dryomov
[not found] ` <CAKeiORRYycaoC6vXZ8TOpf9_oU3it3yrGO=nJen6Fnhiwn=Okw@mail.gmail.com>
2015-12-11 8:26 ` Ilya Dryomov
2015-12-11 13:32 ` Matt Conner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.