Latest bobtail branch still crashing KVM VMs in bh_write_commit()

Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Travis Rhoden]

Hey folks,

We were hoping this one was fixed.  I upgraded all my nodes to the
latest bobtail branch, but still hit this today:

osdc/ObjectCacher.cc: In function 'void
ObjectCacher::bh_write_commit(int64_t, sobject_t, loff_t, uint64_t,
tid_t, int)' thread 7f650e62f700 time 2013-03-20 19:34:39.952616
osdc/ObjectCacher.cc: 834: FAILED assert(ob->last_commit_tid < tid)
 ceph version 0.56.3-42-ga30903c (a30903c6adaa023587d3147179d6038ad37ca520)
 1: (ObjectCacher::bh_write_commit(long, sobject_t, long, unsigned
long, unsigned long, int)+0xd68) [0x7f651d0ada48]
 2: (ObjectCacher::C_WriteCommit::finish(int)+0x6b) [0x7f651d0b460b]
 3: (Context::complete(int)+0xa) [0x7f651d06c9fa]
 4: (librbd::C_Request::finish(int)+0x85) [0x7f651d09c315]
 5: (Context::complete(int)+0xa) [0x7f651d06c9fa]
 6: (librbd::rados_req_cb(void*, void*)+0x47) [0x7f651d081387]
 7: (librados::C_AioSafe::finish(int)+0x1d) [0x7f651c43163d]
 8: (Finisher::finisher_thread_entry()+0x1c0) [0x7f651c49c920]
 9: (()+0x7e9a) [0x7f6519cffe9a]
 10: (clone()+0x6d) [0x7f6519a2bcbd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

Is this occuring in librbd caching?  If so, I could disable it for the
time being.

First saw this mentioned on-list here:
http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/13577

Will be happy to provide anything I can for this one -- definitely
critical for my use case.  It happens with about 10% of the VMs I
create.  Always within the first 60 seconds of the VM booting and
being network accessible.

 - Travis

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Campbell, Bill]

Travis, are you using format 1 or 2 images?  I've seen the same behavior on format 2 images using cloned snapshots, but haven't run into this issue on any normal format 2 images.

----- Original Message -----
From: "Travis Rhoden" <trhoden@gmail.com>
To: "ceph-devel" <ceph-devel@vger.kernel.org>
Sent: Wednesday, March 20, 2013 3:49:23 PM
Subject: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

Hey folks,

We were hoping this one was fixed.  I upgraded all my nodes to the
latest bobtail branch, but still hit this today:

osdc/ObjectCacher.cc: In function 'void
ObjectCacher::bh_write_commit(int64_t, sobject_t, loff_t, uint64_t,
tid_t, int)' thread 7f650e62f700 time 2013-03-20 19:34:39.952616
osdc/ObjectCacher.cc: 834: FAILED assert(ob->last_commit_tid < tid)
 ceph version 0.56.3-42-ga30903c (a30903c6adaa023587d3147179d6038ad37ca520)
 1: (ObjectCacher::bh_write_commit(long, sobject_t, long, unsigned
long, unsigned long, int)+0xd68) [0x7f651d0ada48]
 2: (ObjectCacher::C_WriteCommit::finish(int)+0x6b) [0x7f651d0b460b]
 3: (Context::complete(int)+0xa) [0x7f651d06c9fa]
 4: (librbd::C_Request::finish(int)+0x85) [0x7f651d09c315]
 5: (Context::complete(int)+0xa) [0x7f651d06c9fa]
 6: (librbd::rados_req_cb(void*, void*)+0x47) [0x7f651d081387]
 7: (librados::C_AioSafe::finish(int)+0x1d) [0x7f651c43163d]
 8: (Finisher::finisher_thread_entry()+0x1c0) [0x7f651c49c920]
 9: (()+0x7e9a) [0x7f6519cffe9a]
 10: (clone()+0x6d) [0x7f6519a2bcbd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

Is this occuring in librbd caching?  If so, I could disable it for the
time being.

First saw this mentioned on-list here:
http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/13577

Will be happy to provide anything I can for this one -- definitely
critical for my use case.  It happens with about 10% of the VMs I
create.  Always within the first 60 seconds of the VM booting and
being network accessible.

 - Travis
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message in error, immediately notify the sender and destroy all copies.

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Stefan Priebe]

Hi,

strange i've never seen this. Which qemu version?

Stefan
Am 20.03.2013 20:49, schrieb Travis Rhoden:
> Hey folks,
>
> We were hoping this one was fixed.  I upgraded all my nodes to the
> latest bobtail branch, but still hit this today:
>
> osdc/ObjectCacher.cc: In function 'void
> ObjectCacher::bh_write_commit(int64_t, sobject_t, loff_t, uint64_t,
> tid_t, int)' thread 7f650e62f700 time 2013-03-20 19:34:39.952616
> osdc/ObjectCacher.cc: 834: FAILED assert(ob->last_commit_tid < tid)
>   ceph version 0.56.3-42-ga30903c (a30903c6adaa023587d3147179d6038ad37ca520)
>   1: (ObjectCacher::bh_write_commit(long, sobject_t, long, unsigned
> long, unsigned long, int)+0xd68) [0x7f651d0ada48]
>   2: (ObjectCacher::C_WriteCommit::finish(int)+0x6b) [0x7f651d0b460b]
>   3: (Context::complete(int)+0xa) [0x7f651d06c9fa]
>   4: (librbd::C_Request::finish(int)+0x85) [0x7f651d09c315]
>   5: (Context::complete(int)+0xa) [0x7f651d06c9fa]
>   6: (librbd::rados_req_cb(void*, void*)+0x47) [0x7f651d081387]
>   7: (librados::C_AioSafe::finish(int)+0x1d) [0x7f651c43163d]
>   8: (Finisher::finisher_thread_entry()+0x1c0) [0x7f651c49c920]
>   9: (()+0x7e9a) [0x7f6519cffe9a]
>   10: (clone()+0x6d) [0x7f6519a2bcbd]
>   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> Is this occuring in librbd caching?  If so, I could disable it for the
> time being.
>
> First saw this mentioned on-list here:
> http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/13577
>
> Will be happy to provide anything I can for this one -- definitely
> critical for my use case.  It happens with about 10% of the VMs I
> create.  Always within the first 60 seconds of the VM booting and
> being network accessible.
>
>   - Travis
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Travis Rhoden]

Hello.

> Travis, are you using format 1 or 2 images?  I've seen the same behavior on format 2 images using cloned snapshots, but haven't run into this issue on any normal format 2 images.

In this case, they are format 2. And they are from cloned snapshots.
Exactly like the following:

# rbd ls -l -p volumes
NAME                                                         SIZE
PARENT                                           FMT PROT LOCK
volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2

I'm doing an OpenStack boot-from-volume setup.

> strange i've never seen this. Which qemu version?

# qemu-x86_64 -version
qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard

that's coming from Ubuntu 12.04 apt repos.

 - Travis

On Wed, Mar 20, 2013 at 3:53 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi,
>
> strange i've never seen this. Which qemu version?
>
> Stefan
> Am 20.03.2013 20:49, schrieb Travis Rhoden:
>>
>> Hey folks,
>>
>> We were hoping this one was fixed.  I upgraded all my nodes to the
>> latest bobtail branch, but still hit this today:
>>
>> osdc/ObjectCacher.cc: In function 'void
>> ObjectCacher::bh_write_commit(int64_t, sobject_t, loff_t, uint64_t,
>> tid_t, int)' thread 7f650e62f700 time 2013-03-20 19:34:39.952616
>> osdc/ObjectCacher.cc: 834: FAILED assert(ob->last_commit_tid < tid)
>>   ceph version 0.56.3-42-ga30903c
>> (a30903c6adaa023587d3147179d6038ad37ca520)
>>   1: (ObjectCacher::bh_write_commit(long, sobject_t, long, unsigned
>> long, unsigned long, int)+0xd68) [0x7f651d0ada48]
>>   2: (ObjectCacher::C_WriteCommit::finish(int)+0x6b) [0x7f651d0b460b]
>>   3: (Context::complete(int)+0xa) [0x7f651d06c9fa]
>>   4: (librbd::C_Request::finish(int)+0x85) [0x7f651d09c315]
>>   5: (Context::complete(int)+0xa) [0x7f651d06c9fa]
>>   6: (librbd::rados_req_cb(void*, void*)+0x47) [0x7f651d081387]
>>   7: (librados::C_AioSafe::finish(int)+0x1d) [0x7f651c43163d]
>>   8: (Finisher::finisher_thread_entry()+0x1c0) [0x7f651c49c920]
>>   9: (()+0x7e9a) [0x7f6519cffe9a]
>>   10: (clone()+0x6d) [0x7f6519a2bcbd]
>>   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> Is this occuring in librbd caching?  If so, I could disable it for the
>> time being.
>>
>> First saw this mentioned on-list here:
>> http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/13577
>>
>> Will be happy to provide anything I can for this one -- definitely
>> critical for my use case.  It happens with about 10% of the VMs I
>> create.  Always within the first 60 seconds of the VM booting and
>> being network accessible.
>>
>>   - Travis
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Stefan Priebe]

Hi,

> In this case, they are format 2. And they are from cloned snapshots.
> Exactly like the following:
>
> # rbd ls -l -p volumes
> NAME                                                         SIZE
> PARENT                                           FMT PROT LOCK
> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>
> I'm doing an OpenStack boot-from-volume setup.

OK i've never used cloned snapshots so maybe this is the reason.

>> strange i've never seen this. Which qemu version?
>
> # qemu-x86_64 -version
> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard
>
> that's coming from Ubuntu 12.04 apt repos.

maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does 
not exist anymore it was merged into qemu with 1.3 or 1.4.

Stefan

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Travis Rhoden]

On Wed, Mar 20, 2013 at 4:14 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi,
>
>
>> In this case, they are format 2. And they are from cloned snapshots.
>> Exactly like the following:
>>
>> # rbd ls -l -p volumes
>> NAME                                                         SIZE
>> PARENT                                           FMT PROT LOCK
>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>
>> I'm doing an OpenStack boot-from-volume setup.
>
>
> OK i've never used cloned snapshots so maybe this is the reason.
>
>
>>> strange i've never seen this. Which qemu version?
>>
>>
>> # qemu-x86_64 -version
>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice
>> Bellard
>>
>> that's coming from Ubuntu 12.04 apt repos.
>
>
> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does not
> exist anymore it was merged into qemu with 1.3 or 1.4.

Since the crash is in librbd, would an update of qemu help anything?
>
> Stefan

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Josh Durgin]

On 03/20/2013 01:14 PM, Stefan Priebe wrote:
> Hi,
>
>> In this case, they are format 2. And they are from cloned snapshots.
>> Exactly like the following:
>>
>> # rbd ls -l -p volumes
>> NAME                                                         SIZE
>> PARENT                                           FMT PROT LOCK
>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>
>> I'm doing an OpenStack boot-from-volume setup.
>
> OK i've never used cloned snapshots so maybe this is the reason.
>
>>> strange i've never seen this. Which qemu version?
>>
>> # qemu-x86_64 -version
>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>> Fabrice Bellard
>>
>> that's coming from Ubuntu 12.04 apt repos.
>
> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does
> not exist anymore it was merged into qemu with 1.3 or 1.4.

This particular problem won't be solved by upgrading qemu. It's a ceph
bug. Disabling caching would work around the issue.

Travis, could you get a log from qemu of this happening with:

debug ms = 20
debug objectcacher = 20
debug rbd = 20
log file = /path/writeable/by/qemu

 From those we can tell whether the issue is on the client side at least,
and hopefully what's causing it.

Thanks!
Josh

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Josh Durgin]

On 03/20/2013 01:19 PM, Josh Durgin wrote:
> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>> Hi,
>>
>>> In this case, they are format 2. And they are from cloned snapshots.
>>> Exactly like the following:
>>>
>>> # rbd ls -l -p volumes
>>> NAME                                                         SIZE
>>> PARENT                                           FMT PROT LOCK
>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>
>>> I'm doing an OpenStack boot-from-volume setup.
>>
>> OK i've never used cloned snapshots so maybe this is the reason.
>>
>>>> strange i've never seen this. Which qemu version?
>>>
>>> # qemu-x86_64 -version
>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>> Fabrice Bellard
>>>
>>> that's coming from Ubuntu 12.04 apt repos.
>>
>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does
>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>
> This particular problem won't be solved by upgrading qemu. It's a ceph
> bug. Disabling caching would work around the issue.
>
> Travis, could you get a log from qemu of this happening with:
>
> debug ms = 20
> debug objectcacher = 20
> debug rbd = 20
> log file = /path/writeable/by/qemu

If it doesn't reproduce with those settings, try changing debug ms to 1
instead of 20.

>  From those we can tell whether the issue is on the client side at least,
> and hopefully what's causing it.
>
> Thanks!
> Josh

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Travis Rhoden]

Thanks Josh.  I will respond when I have something useful!

On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>
>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>
>>> Hi,
>>>
>>>> In this case, they are format 2. And they are from cloned snapshots.
>>>> Exactly like the following:
>>>>
>>>> # rbd ls -l -p volumes
>>>> NAME                                                         SIZE
>>>> PARENT                                           FMT PROT LOCK
>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>
>>>> I'm doing an OpenStack boot-from-volume setup.
>>>
>>>
>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>
>>>>> strange i've never seen this. Which qemu version?
>>>>
>>>>
>>>> # qemu-x86_64 -version
>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>> Fabrice Bellard
>>>>
>>>> that's coming from Ubuntu 12.04 apt repos.
>>>
>>>
>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does
>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>
>>
>> This particular problem won't be solved by upgrading qemu. It's a ceph
>> bug. Disabling caching would work around the issue.
>>
>> Travis, could you get a log from qemu of this happening with:
>>
>> debug ms = 20
>> debug objectcacher = 20
>> debug rbd = 20
>> log file = /path/writeable/by/qemu
>
>
> If it doesn't reproduce with those settings, try changing debug ms to 1
> instead of 20.
>
>
>>  From those we can tell whether the issue is on the client side at least,
>> and hopefully what's causing it.
>>
>> Thanks!
>> Josh
>
>

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Travis Rhoden]

Didn't take long to re-create with the detailed debugging (ms =  20).
I'm sending Josh a link to the gzip'd log off-list, I"m not sure if
the log will contain any CephX keys or anything like that.

On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@gmail.com> wrote:
> Thanks Josh.  I will respond when I have something useful!
>
> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
>> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>>
>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>>
>>>> Hi,
>>>>
>>>>> In this case, they are format 2. And they are from cloned snapshots.
>>>>> Exactly like the following:
>>>>>
>>>>> # rbd ls -l -p volumes
>>>>> NAME                                                         SIZE
>>>>> PARENT                                           FMT PROT LOCK
>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>>
>>>>> I'm doing an OpenStack boot-from-volume setup.
>>>>
>>>>
>>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>>
>>>>>> strange i've never seen this. Which qemu version?
>>>>>
>>>>>
>>>>> # qemu-x86_64 -version
>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>>> Fabrice Bellard
>>>>>
>>>>> that's coming from Ubuntu 12.04 apt repos.
>>>>
>>>>
>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does
>>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>>
>>>
>>> This particular problem won't be solved by upgrading qemu. It's a ceph
>>> bug. Disabling caching would work around the issue.
>>>
>>> Travis, could you get a log from qemu of this happening with:
>>>
>>> debug ms = 20
>>> debug objectcacher = 20
>>> debug rbd = 20
>>> log file = /path/writeable/by/qemu
>>
>>
>> If it doesn't reproduce with those settings, try changing debug ms to 1
>> instead of 20.
>>
>>
>>>  From those we can tell whether the issue is on the client side at least,
>>> and hopefully what's causing it.
>>>
>>> Thanks!
>>> Josh
>>
>>

RE: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Jacky.He]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Stefan Priebe
> Sent: Thursday, March 21, 2013 4:14 AM
> To: Travis Rhoden
> Cc: bcampbell@axcess-financial.com; ceph-devel
> Subject: Re: Latest bobtail branch still crashing KVM VMs in
bh_write_commit()
> 
> Hi,
> 
> > In this case, they are format 2. And they are from cloned snapshots.
> > Exactly like the following:
> >
> > # rbd ls -l -p volumes
> > NAME                                                         SIZE
> > PARENT                                           FMT PROT LOCK
> > volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
> > images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
> >
> > I'm doing an OpenStack boot-from-volume setup.
> 
> OK i've never used cloned snapshots so maybe this is the reason.
> 
> >> strange i've never seen this. Which qemu version?
> >
> > # qemu-x86_64 -version
> > qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
> > Fabrice Bellard
> >
> > that's coming from Ubuntu 12.04 apt repos.
> 
> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does
not
> exist anymore it was merged into qemu with 1.3 or 1.4.
> 
[jacky_he] I also encountered the same issue, ceph version is 0.56.3.
I have tried Qemu 1.3.1 and Qemu 1.4.0, KVM VM with format 2 cloned image
crashs.
My host OS is ubuntu 12.04, guest OS are CentOS 6.3 and Windows XP/Windows 7

> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of
> a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
> 
> 
> __________ Information from ESET NOD32 Antivirus, version of virus
signature
> database 8141 (20130320) __________
> 
> The message was checked by ESET NOD32 Antivirus.
> 
> http://www.eset.com
> 

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Josh Durgin]

I think I found the root cause based on your logs:

http://tracker.ceph.com/issues/4531

Josh

On 03/20/2013 02:47 PM, Travis Rhoden wrote:
> Didn't take long to re-create with the detailed debugging (ms =  20).
> I'm sending Josh a link to the gzip'd log off-list, I"m not sure if
> the log will contain any CephX keys or anything like that.
>
> On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@gmail.com> wrote:
>> Thanks Josh.  I will respond when I have something useful!
>>
>> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
>>> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>>>
>>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>> In this case, they are format 2. And they are from cloned snapshots.
>>>>>> Exactly like the following:
>>>>>>
>>>>>> # rbd ls -l -p volumes
>>>>>> NAME                                                         SIZE
>>>>>> PARENT                                           FMT PROT LOCK
>>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>>>
>>>>>> I'm doing an OpenStack boot-from-volume setup.
>>>>>
>>>>>
>>>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>>>
>>>>>>> strange i've never seen this. Which qemu version?
>>>>>>
>>>>>>
>>>>>> # qemu-x86_64 -version
>>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>>>> Fabrice Bellard
>>>>>>
>>>>>> that's coming from Ubuntu 12.04 apt repos.
>>>>>
>>>>>
>>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does
>>>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>>>
>>>>
>>>> This particular problem won't be solved by upgrading qemu. It's a ceph
>>>> bug. Disabling caching would work around the issue.
>>>>
>>>> Travis, could you get a log from qemu of this happening with:
>>>>
>>>> debug ms = 20
>>>> debug objectcacher = 20
>>>> debug rbd = 20
>>>> log file = /path/writeable/by/qemu
>>>
>>>
>>> If it doesn't reproduce with those settings, try changing debug ms to 1
>>> instead of 20.
>>>
>>>
>>>>   From those we can tell whether the issue is on the client side at least,
>>>> and hopefully what's causing it.
>>>>
>>>> Thanks!
>>>> Josh
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Travis Rhoden]

That's awesome Josh.  Thanks for looking into it.  Good luck with the fix!

 - Travis

On Fri, Mar 22, 2013 at 1:11 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> I think I found the root cause based on your logs:
>
> http://tracker.ceph.com/issues/4531
>
> Josh
>
>
> On 03/20/2013 02:47 PM, Travis Rhoden wrote:
>>
>> Didn't take long to re-create with the detailed debugging (ms =  20).
>> I'm sending Josh a link to the gzip'd log off-list, I"m not sure if
>> the log will contain any CephX keys or anything like that.
>>
>> On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@gmail.com> wrote:
>>>
>>> Thanks Josh.  I will respond when I have something useful!
>>>
>>> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com>
>>> wrote:
>>>>
>>>> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>>>>
>>>>>
>>>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>> In this case, they are format 2. And they are from cloned snapshots.
>>>>>>> Exactly like the following:
>>>>>>>
>>>>>>> # rbd ls -l -p volumes
>>>>>>> NAME                                                         SIZE
>>>>>>> PARENT                                           FMT PROT LOCK
>>>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>>>>
>>>>>>> I'm doing an OpenStack boot-from-volume setup.
>>>>>>
>>>>>>
>>>>>>
>>>>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>>>>
>>>>>>>> strange i've never seen this. Which qemu version?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # qemu-x86_64 -version
>>>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>>>>> Fabrice Bellard
>>>>>>>
>>>>>>> that's coming from Ubuntu 12.04 apt repos.
>>>>>>
>>>>>>
>>>>>>
>>>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm
>>>>>> does
>>>>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>>>>
>>>>>
>>>>>
>>>>> This particular problem won't be solved by upgrading qemu. It's a ceph
>>>>> bug. Disabling caching would work around the issue.
>>>>>
>>>>> Travis, could you get a log from qemu of this happening with:
>>>>>
>>>>> debug ms = 20
>>>>> debug objectcacher = 20
>>>>> debug rbd = 20
>>>>> log file = /path/writeable/by/qemu
>>>>
>>>>
>>>>
>>>> If it doesn't reproduce with those settings, try changing debug ms to 1
>>>> instead of 20.
>>>>
>>>>
>>>>>   From those we can tell whether the issue is on the client side at
>>>>> least,
>>>>> and hopefully what's causing it.
>>>>>
>>>>> Thanks!
>>>>> Josh
>>>>
>>>>
>>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Josh Durgin]

Finally got some time to fix this (hopefully).
Could you try librbd from the wip-objectcacher-handler-ordered branch?
Just librbd on the host running qemu needs to be updated.

Thanks,
Josh

On 03/22/2013 11:30 AM, Travis Rhoden wrote:
> That's awesome Josh.  Thanks for looking into it.  Good luck with the fix!
>
>   - Travis
>
> On Fri, Mar 22, 2013 at 1:11 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
>> I think I found the root cause based on your logs:
>>
>> http://tracker.ceph.com/issues/4531
>>
>> Josh
>>
>>
>> On 03/20/2013 02:47 PM, Travis Rhoden wrote:
>>>
>>> Didn't take long to re-create with the detailed debugging (ms =  20).
>>> I'm sending Josh a link to the gzip'd log off-list, I"m not sure if
>>> the log will contain any CephX keys or anything like that.
>>>
>>> On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@gmail.com> wrote:
>>>>
>>>> Thanks Josh.  I will respond when I have something useful!
>>>>
>>>> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com>
>>>> wrote:
>>>>>
>>>>> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>>>>>
>>>>>>
>>>>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>> In this case, they are format 2. And they are from cloned snapshots.
>>>>>>>> Exactly like the following:
>>>>>>>>
>>>>>>>> # rbd ls -l -p volumes
>>>>>>>> NAME                                                         SIZE
>>>>>>>> PARENT                                           FMT PROT LOCK
>>>>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>>>>>
>>>>>>>> I'm doing an OpenStack boot-from-volume setup.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>>>>>
>>>>>>>>> strange i've never seen this. Which qemu version?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # qemu-x86_64 -version
>>>>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>>>>>> Fabrice Bellard
>>>>>>>>
>>>>>>>> that's coming from Ubuntu 12.04 apt repos.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm
>>>>>>> does
>>>>>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>>>>>
>>>>>>
>>>>>>
>>>>>> This particular problem won't be solved by upgrading qemu. It's a ceph
>>>>>> bug. Disabling caching would work around the issue.
>>>>>>
>>>>>> Travis, could you get a log from qemu of this happening with:
>>>>>>
>>>>>> debug ms = 20
>>>>>> debug objectcacher = 20
>>>>>> debug rbd = 20
>>>>>> log file = /path/writeable/by/qemu
>>>>>
>>>>>
>>>>>
>>>>> If it doesn't reproduce with those settings, try changing debug ms to 1
>>>>> instead of 20.
>>>>>
>>>>>
>>>>>>    From those we can tell whether the issue is on the client side at
>>>>>> least,
>>>>>> and hopefully what's causing it.
>>>>>>
>>>>>> Thanks!
>>>>>> Josh
>>>>>
>>>>>
>>>>>

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Travis Rhoden]

Hi Josh,

Thanks for the heads up.  I've been testing the fix all morning, and
haven't run into a single crash yet!  I turned on the RBD logging
during a couple of VM startups just to look and make sure I saw a
bunch of objectcacher traffic (to know I was really doing caching).

I'll keep the new version installed for now and see how things play
out through the day.  So far things are looking very promising.

A couple of obligatory questions:

Any idea when the fixes will be backported to bobtail?

I"m running the latest bobtail packages everywhere else.  I now have
0.60+ for librbd, librados, and ceph-common on my host running qemu
(all that host does is run virtual machiens with librbd).  Do you know
of anything that would make this mixed environment a cause for
concern?  Once the backport is done, I will revert these packages to
the bobtail version.

Thanks so much for the good work.

 - Travis

On Wed, Apr 10, 2013 at 8:53 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> Finally got some time to fix this (hopefully).
> Could you try librbd from the wip-objectcacher-handler-ordered branch?
> Just librbd on the host running qemu needs to be updated.
>
> Thanks,
> Josh
>
>
> On 03/22/2013 11:30 AM, Travis Rhoden wrote:
>>
>> That's awesome Josh.  Thanks for looking into it.  Good luck with the fix!
>>
>>   - Travis
>>
>> On Fri, Mar 22, 2013 at 1:11 PM, Josh Durgin <josh.durgin@inktank.com>
>> wrote:
>>>
>>> I think I found the root cause based on your logs:
>>>
>>> http://tracker.ceph.com/issues/4531
>>>
>>> Josh
>>>
>>>
>>> On 03/20/2013 02:47 PM, Travis Rhoden wrote:
>>>>
>>>>
>>>> Didn't take long to re-create with the detailed debugging (ms =  20).
>>>> I'm sending Josh a link to the gzip'd log off-list, I"m not sure if
>>>> the log will contain any CephX keys or anything like that.
>>>>
>>>> On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> Thanks Josh.  I will respond when I have something useful!
>>>>>
>>>>> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>> In this case, they are format 2. And they are from cloned
>>>>>>>>> snapshots.
>>>>>>>>> Exactly like the following:
>>>>>>>>>
>>>>>>>>> # rbd ls -l -p volumes
>>>>>>>>> NAME                                                         SIZE
>>>>>>>>> PARENT                                           FMT PROT LOCK
>>>>>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>>>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>>>>>>
>>>>>>>>> I'm doing an OpenStack boot-from-volume setup.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>>>>>>
>>>>>>>>>> strange i've never seen this. Which qemu version?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # qemu-x86_64 -version
>>>>>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>>>>>>> Fabrice Bellard
>>>>>>>>>
>>>>>>>>> that's coming from Ubuntu 12.04 apt repos.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm
>>>>>>>> does
>>>>>>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This particular problem won't be solved by upgrading qemu. It's a
>>>>>>> ceph
>>>>>>> bug. Disabling caching would work around the issue.
>>>>>>>
>>>>>>> Travis, could you get a log from qemu of this happening with:
>>>>>>>
>>>>>>> debug ms = 20
>>>>>>> debug objectcacher = 20
>>>>>>> debug rbd = 20
>>>>>>> log file = /path/writeable/by/qemu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> If it doesn't reproduce with those settings, try changing debug ms to
>>>>>> 1
>>>>>> instead of 20.
>>>>>>
>>>>>>
>>>>>>>    From those we can tell whether the issue is on the client side at
>>>>>>> least,
>>>>>>> and hopefully what's causing it.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Josh
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>

Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Josh Durgin]

On 04/11/2013 08:41 AM, Travis Rhoden wrote:
> Hi Josh,
>
> Thanks for the heads up.  I've been testing the fix all morning, and
> haven't run into a single crash yet!  I turned on the RBD logging
> during a couple of VM startups just to look and make sure I saw a
> bunch of objectcacher traffic (to know I was really doing caching).
>
> I'll keep the new version installed for now and see how things play
> out through the day.  So far things are looking very promising.

Great!

> A couple of obligatory questions:
>
> Any idea when the fixes will be backported to bobtail?

Hopefully tomorrow. There are a couple other bugs I'd like to fix, and
then I'll backport several recent fixes at once so I can test the
backports all together.

> I"m running the latest bobtail packages everywhere else.  I now have
> 0.60+ for librbd, librados, and ceph-common on my host running qemu
> (all that host does is run virtual machiens with librbd).  Do you know
> of anything that would make this mixed environment a cause for
> concern?  Once the backport is done, I will revert these packages to
> the bobtail version.

I'm not aware of anything that would cause problems with upgraded
client-side packages.

> Thanks so much for the good work.

Thanks for helping track down these bugs!
Josh

>   - Travis
>
> On Wed, Apr 10, 2013 at 8:53 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
>> Finally got some time to fix this (hopefully).
>> Could you try librbd from the wip-objectcacher-handler-ordered branch?
>> Just librbd on the host running qemu needs to be updated.
>>
>> Thanks,
>> Josh
>>
>>
>> On 03/22/2013 11:30 AM, Travis Rhoden wrote:
>>>
>>> That's awesome Josh.  Thanks for looking into it.  Good luck with the fix!
>>>
>>>    - Travis
>>>
>>> On Fri, Mar 22, 2013 at 1:11 PM, Josh Durgin <josh.durgin@inktank.com>
>>> wrote:
>>>>
>>>> I think I found the root cause based on your logs:
>>>>
>>>> http://tracker.ceph.com/issues/4531
>>>>
>>>> Josh
>>>>
>>>>
>>>> On 03/20/2013 02:47 PM, Travis Rhoden wrote:
>>>>>
>>>>>
>>>>> Didn't take long to re-create with the detailed debugging (ms =  20).
>>>>> I'm sending Josh a link to the gzip'd log off-list, I"m not sure if
>>>>> the log will contain any CephX keys or anything like that.
>>>>>
>>>>> On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Thanks Josh.  I will respond when I have something useful!
>>>>>>
>>>>>> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>> In this case, they are format 2. And they are from cloned
>>>>>>>>>> snapshots.
>>>>>>>>>> Exactly like the following:
>>>>>>>>>>
>>>>>>>>>> # rbd ls -l -p volumes
>>>>>>>>>> NAME                                                         SIZE
>>>>>>>>>> PARENT                                           FMT PROT LOCK
>>>>>>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>>>>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>>>>>>>
>>>>>>>>>> I'm doing an OpenStack boot-from-volume setup.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>>>>>>>
>>>>>>>>>>> strange i've never seen this. Which qemu version?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # qemu-x86_64 -version
>>>>>>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>>>>>>>> Fabrice Bellard
>>>>>>>>>>
>>>>>>>>>> that's coming from Ubuntu 12.04 apt repos.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm
>>>>>>>>> does
>>>>>>>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This particular problem won't be solved by upgrading qemu. It's a
>>>>>>>> ceph
>>>>>>>> bug. Disabling caching would work around the issue.
>>>>>>>>
>>>>>>>> Travis, could you get a log from qemu of this happening with:
>>>>>>>>
>>>>>>>> debug ms = 20
>>>>>>>> debug objectcacher = 20
>>>>>>>> debug rbd = 20
>>>>>>>> log file = /path/writeable/by/qemu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If it doesn't reproduce with those settings, try changing debug ms to
>>>>>>> 1
>>>>>>> instead of 20.
>>>>>>>
>>>>>>>
>>>>>>>>     From those we can tell whether the issue is on the client side at
>>>>>>>> least,
>>>>>>>> and hopefully what's causing it.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> Josh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>