* Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
@ 2012-01-14 14:40 Martin Mailand
2012-01-15 2:45 ` Sage Weil
0 siblings, 1 reply; 6+ messages in thread
From: Martin Mailand @ 2012-01-14 14:40 UTC (permalink / raw)
To: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 287 bytes --]
Hi
one of four OSD died during the update to v0.40 with an Assertion
os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
Even after a complete shutdown of the cluster an a new start with all
OSD at the same version, this osd did not start.
The OSD Log it attached.
-martin
[-- Attachment #2: osd.0.log.bz2 --]
[-- Type: application/x-bzip, Size: 23701 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
2012-01-14 14:40 Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error") Martin Mailand
@ 2012-01-15 2:45 ` Sage Weil
2012-01-15 5:52 ` Sage Weil
2012-01-15 11:39 ` Martin Mailand
0 siblings, 2 replies; 6+ messages in thread
From: Sage Weil @ 2012-01-15 2:45 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel
Hi Martin-
On Sat, 14 Jan 2012, Martin Mailand wrote:
> Hi
> one of four OSD died during the update to v0.40 with an Assertion
> os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
> Even after a complete shutdown of the cluster an a new start with all OSD at
> the same version, this osd did not start.
>
> The OSD Log it attached.
It's trying to replay a transaction that appears to be invalid because the
.2 clone is smaller than it thinks. Is this the first time the OSD
crashed, or did it crash once, and you cranked up logs and generated
this one? If you have the previous log, that would be helpful... it
should have a similar tranasction dump but a different stack trace.
Also, are any of the 6 patches on top of 0.40 related to the filestore or
osd?
Thanks!
sage
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
2012-01-15 2:45 ` Sage Weil
@ 2012-01-15 5:52 ` Sage Weil
2012-01-15 11:41 ` Martin Mailand
2012-01-15 11:39 ` Martin Mailand
1 sibling, 1 reply; 6+ messages in thread
From: Sage Weil @ 2012-01-15 5:52 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel
Hi Martin-
On Sat, 14 Jan 2012, Sage Weil wrote:
> Hi Martin-
>
> On Sat, 14 Jan 2012, Martin Mailand wrote:
>
> > Hi
> > one of four OSD died during the update to v0.40 with an Assertion
> > os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
> > Even after a complete shutdown of the cluster an a new start with all OSD at
> > the same version, this osd did not start.
> >
> > The OSD Log it attached.
>
> It's trying to replay a transaction that appears to be invalid because the
> .2 clone is smaller than it thinks. Is this the first time the OSD
> crashed, or did it crash once, and you cranked up logs and generated
> this one? If you have the previous log, that would be helpful... it
> should have a similar tranasction dump but a different stack trace.
I pushed a wip-osd-dump-journal branch to git that will make
ceph-osd -i <whatever> --dump-journal > /tmp/foo.txt
dump the contents of your entire osd journal (sans data) to a text file.
Do you mind sending that along as well? I'd like to see what is in the
journal _after_ the event that is failing (if anything).
Thanks!
sage
>
> Also, are any of the 6 patches on top of 0.40 related to the filestore or
> osd?
>
> Thanks!
> sage
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
2012-01-15 2:45 ` Sage Weil
2012-01-15 5:52 ` Sage Weil
@ 2012-01-15 11:39 ` Martin Mailand
2012-01-16 6:14 ` Sage Weil
1 sibling, 1 reply; 6+ messages in thread
From: Martin Mailand @ 2012-01-15 11:39 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
Hi Sage,
that's exactly what I did, the first two crashes are in this log,
unfortunately there was no debug level set.
http://85.214.49.87/ceph/osd.0.full.log.bz2
-martin
Am 15.01.2012 03:45, schrieb Sage Weil:
> Hi Martin-
>
> On Sat, 14 Jan 2012, Martin Mailand wrote:
>
>> Hi
>> one of four OSD died during the update to v0.40 with an Assertion
>> os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
>> Even after a complete shutdown of the cluster an a new start with all OSD at
>> the same version, this osd did not start.
>>
>> The OSD Log it attached.
>
> It's trying to replay a transaction that appears to be invalid because the
> .2 clone is smaller than it thinks. Is this the first time the OSD
> crashed, or did it crash once, and you cranked up logs and generated
> this one? If you have the previous log, that would be helpful... it
> should have a similar tranasction dump but a different stack trace.
>
> Also, are any of the 6 patches on top of 0.40 related to the filestore or
> osd?
>
> Thanks!
> sage
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
2012-01-15 5:52 ` Sage Weil
@ 2012-01-15 11:41 ` Martin Mailand
0 siblings, 0 replies; 6+ messages in thread
From: Martin Mailand @ 2012-01-15 11:41 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
Hi Sage,
here is the requested dump file.
http://85.214.49.87/ceph/foo.txt.bz2
-martin
Am 15.01.2012 06:52, schrieb Sage Weil:
> Hi Martin-
>
> On Sat, 14 Jan 2012, Sage Weil wrote:
>> Hi Martin-
>>
>> On Sat, 14 Jan 2012, Martin Mailand wrote:
>>
>>> Hi
>>> one of four OSD died during the update to v0.40 with an Assertion
>>> os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
>>> Even after a complete shutdown of the cluster an a new start with all OSD at
>>> the same version, this osd did not start.
>>>
>>> The OSD Log it attached.
>>
>> It's trying to replay a transaction that appears to be invalid because the
>> .2 clone is smaller than it thinks. Is this the first time the OSD
>> crashed, or did it crash once, and you cranked up logs and generated
>> this one? If you have the previous log, that would be helpful... it
>> should have a similar tranasction dump but a different stack trace.
>
> I pushed a wip-osd-dump-journal branch to git that will make
>
> ceph-osd -i<whatever> --dump-journal> /tmp/foo.txt
>
> dump the contents of your entire osd journal (sans data) to a text file.
> Do you mind sending that along as well? I'd like to see what is in the
> journal _after_ the event that is failing (if anything).
>
> Thanks!
> sage
>
>
>>
>> Also, are any of the 6 patches on top of 0.40 related to the filestore or
>> osd?
>>
>> Thanks!
>> sage
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
2012-01-15 11:39 ` Martin Mailand
@ 2012-01-16 6:14 ` Sage Weil
0 siblings, 0 replies; 6+ messages in thread
From: Sage Weil @ 2012-01-16 6:14 UTC (permalink / raw)
To: Martin Mailand; +Cc: ceph-devel
On Sun, 15 Jan 2012, Martin Mailand wrote:
> Hi Sage,
> that's exactly what I did, the first two crashes are in this log,
> unfortunately there was no debug level set.
Whoops, right.. the (old) replay messages above confused me.
There are a couple possibilities here. One is that the recovery code went
in the wrong order. I'm a bit skeptical, though, and even if it did, this
was mostly just rewritten in wip-backfill for 0.41, so I don't think it's
worth debugging. We have some tools to hammer on then snapshot +
recovery code, but they aren't in the regular qa rotation yet.
More likely is that the SnapSet notion of clone_overlap got out of sync
with the actual clones. To check that, we need a dump of the xattrs on
the _head object. File sizes and attrs for the clones would help too.
Are the _2 clone objects on other replicas 4MB or 23 bytes?
Is this keeping your cluster down? If so, you can work around the problem
by making the _2 clone object 4MB so that reply will succeed. Just be
aware that that rbd image's content will be corrupted. :/
sage
>
> http://85.214.49.87/ceph/osd.0.full.log.bz2
>
> -martin
>
>
>
> Am 15.01.2012 03:45, schrieb Sage Weil:
> > Hi Martin-
> >
> > On Sat, 14 Jan 2012, Martin Mailand wrote:
> >
> > > Hi
> > > one of four OSD died during the update to v0.40 with an Assertion
> > > os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error")
> > > Even after a complete shutdown of the cluster an a new start with all OSD
> > > at
> > > the same version, this osd did not start.
> > >
> > > The OSD Log it attached.
> >
> > It's trying to replay a transaction that appears to be invalid because the
> > .2 clone is smaller than it thinks. Is this the first time the OSD
> > crashed, or did it crash once, and you cranked up logs and generated
> > this one? If you have the previous log, that would be helpful... it
> > should have a similar tranasction dump but a different stack trace.
> >
> > Also, are any of the 6 patches on top of 0.40 related to the filestore or
> > osd?
> >
> > Thanks!
> > sage
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-01-16 6:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-01-14 14:40 Assertion in v0.40 - os/FileStore.cc: 2438: FAILED assert(0 == "unexpected error") Martin Mailand
2012-01-15 2:45 ` Sage Weil
2012-01-15 5:52 ` Sage Weil
2012-01-15 11:41 ` Martin Mailand
2012-01-15 11:39 ` Martin Mailand
2012-01-16 6:14 ` Sage Weil
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.