Xen 3.4.2 kernel / lvcreate crash

Xen 3.4.2 kernel / lvcreate crash

[Valtteri Kiviniemi]

Hi,

I am using Xen 3.4.2 with kernel 2.6.18.8 and Debian Lenny. I am having
some problems with LVM-snapshots. I have crontab which makes once a
night a snapshot from domU's LV.

The problem is that maybe once in a month the lvcreate command stucks
and after that I have to reboot the machine to solve the problem.

So is this a problem on the Xen kernel or with Debian?

dmesg shows the following:

Unable to handle kernel NULL pointer dereference at 0000000000000030 RIP:
  [<ffffffff804218a2>] __map_bio+0x45/0x9b
PGD 38057067 PUD 10657067 PMD 0
Oops: 0000 [1] SMP
CPU 0
Pid: 17350, comm: lvcreate Not tainted 2.6.18.8-dom0 #1
RIP: e030:[<ffffffff804218a2>]  [<ffffffff804218a2>] __map_bio+0x45/0x9b
RSP: e02b:ffff880028693d68  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
RDX: ffff8800311bfaf0 RSI: ffff8800219c38c0 RDI: ffffc200000f30d8
RBP: ffff8800311bfae0 R08: 0000000000000001 R09: 001322b628693ce8
R10: 0000000000000001 R11: 00000000006cc4a0 R12: ffff8800219c38c0
R13: fffffffff3800008 R14: ffff88003c56c5d8 R15: ffff8800311bfae0
FS:  00002b35f3f7ac20(0000) GS:ffffffff805ca000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process lvcreate (pid: 17350, threadinfo ffff880028692000, task
ffff88003cc510c0)
Stack:  ffff88003f5a0440 0000000000000001 ffff88003c56c5c0 ffff88003c565400
  fffffffff3800008 ffffffff804221b5 ffff88003c56c5d8 ffff88003f5a0440
  ffff88003c56c5c0 ffff88003c565400 ffff8800252b1f10 ffffffff8025e099
Call Trace:
  [<ffffffff804221b5>] __split_bio+0x1a9/0x3f7
  [<ffffffff8025e099>] __down_write_nested+0x12/0xf1
  [<ffffffff8022fe0e>] __up_write+0x1d/0x115
  [<ffffffff804225b9>] dm_resume+0xa1/0x137
  [<ffffffff80425158>] dev_suspend+0x0/0x171
  [<ffffffff804252a3>] dev_suspend+0x14b/0x171
  [<ffffffff80425bb6>] ctl_ioctl+0x1c6/0x20e
  [<ffffffff8024012a>] do_ioctl+0x56/0x6c
  [<ffffffff8022fddb>] vfs_ioctl+0x23a/0x250
  [<ffffffff8024a42f>] sys_ioctl+0x3d/0x5c
  [<ffffffff8025a21c>] system_call+0x68/0x6d
  [<ffffffff8025a1b4>] system_call+0x0/0x6d


Code: ff 50 30 41 89 c5 83 f8 00 7e 10 4c 89 e7 41 59 5b 5d 41 5c
RIP  [<ffffffff804218a2>] __map_bio+0x45/0x9b
  RSP <ffff880028693d68>
CR2: 0000000000000030

Re: Xen 3.4.2 kernel / lvcreate crash

[Jeremy Fitzhardinge]

On 03/31/2010 05:34 PM, Valtteri Kiviniemi wrote:
> I am using Xen 3.4.2 with kernel 2.6.18.8 and Debian Lenny. I am having
> some problems with LVM-snapshots. I have crontab which makes once a
> night a snapshot from domU's LV.
>
> The problem is that maybe once in a month the lvcreate command stucks
> and after that I have to reboot the machine to solve the problem.
>
> So is this a problem on the Xen kernel or with Debian?

Someone reported a very similar crash with a Debian Xen kernel.  I would 
suspect a bug in the Debian Xen kernel.

     J

>
> dmesg shows the following:
>
> Unable to handle kernel NULL pointer dereference at 0000000000000030 RIP:
>  [<ffffffff804218a2>] __map_bio+0x45/0x9b
> PGD 38057067 PUD 10657067 PMD 0
> Oops: 0000 [1] SMP
> CPU 0
> Pid: 17350, comm: lvcreate Not tainted 2.6.18.8-dom0 #1
> RIP: e030:[<ffffffff804218a2>]  [<ffffffff804218a2>] __map_bio+0x45/0x9b
> RSP: e02b:ffff880028693d68  EFLAGS: 00010202
> RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
> RDX: ffff8800311bfaf0 RSI: ffff8800219c38c0 RDI: ffffc200000f30d8
> RBP: ffff8800311bfae0 R08: 0000000000000001 R09: 001322b628693ce8
> R10: 0000000000000001 R11: 00000000006cc4a0 R12: ffff8800219c38c0
> R13: fffffffff3800008 R14: ffff88003c56c5d8 R15: ffff8800311bfae0
> FS:  00002b35f3f7ac20(0000) GS:ffffffff805ca000(0000) 
> knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000
> Process lvcreate (pid: 17350, threadinfo ffff880028692000, task
> ffff88003cc510c0)
> Stack:  ffff88003f5a0440 0000000000000001 ffff88003c56c5c0 
> ffff88003c565400
>  fffffffff3800008 ffffffff804221b5 ffff88003c56c5d8 ffff88003f5a0440
>  ffff88003c56c5c0 ffff88003c565400 ffff8800252b1f10 ffffffff8025e099
> Call Trace:
>  [<ffffffff804221b5>] __split_bio+0x1a9/0x3f7
>  [<ffffffff8025e099>] __down_write_nested+0x12/0xf1
>  [<ffffffff8022fe0e>] __up_write+0x1d/0x115
>  [<ffffffff804225b9>] dm_resume+0xa1/0x137
>  [<ffffffff80425158>] dev_suspend+0x0/0x171
>  [<ffffffff804252a3>] dev_suspend+0x14b/0x171
>  [<ffffffff80425bb6>] ctl_ioctl+0x1c6/0x20e
>  [<ffffffff8024012a>] do_ioctl+0x56/0x6c
>  [<ffffffff8022fddb>] vfs_ioctl+0x23a/0x250
>  [<ffffffff8024a42f>] sys_ioctl+0x3d/0x5c
>  [<ffffffff8025a21c>] system_call+0x68/0x6d
>  [<ffffffff8025a1b4>] system_call+0x0/0x6d
>
>
> Code: ff 50 30 41 89 c5 83 f8 00 7e 10 4c 89 e7 41 59 5b 5d 41 5c
> RIP  [<ffffffff804218a2>] __map_bio+0x45/0x9b
>  RSP <ffff880028693d68>
> CR2: 0000000000000030
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Xen 3.4.2 kernel / lvcreate crash

[Valtteri Kiviniemi]

Hi,

I am not using Debian Xen kernel, I have always downloaded the latest 
release from xen.org and compiled it myself.

- Valtteri

Jeremy Fitzhardinge kirjoitti:
> On 03/31/2010 05:34 PM, Valtteri Kiviniemi wrote:
>> I am using Xen 3.4.2 with kernel 2.6.18.8 and Debian Lenny. I am having
>> some problems with LVM-snapshots. I have crontab which makes once a
>> night a snapshot from domU's LV.
>>
>> The problem is that maybe once in a month the lvcreate command stucks
>> and after that I have to reboot the machine to solve the problem.
>>
>> So is this a problem on the Xen kernel or with Debian?
> 
> Someone reported a very similar crash with a Debian Xen kernel.  I would 
> suspect a bug in the Debian Xen kernel.
> 
>     J
> 
>>
>> dmesg shows the following:
>>
>> Unable to handle kernel NULL pointer dereference at 0000000000000030 RIP:
>>  [<ffffffff804218a2>] __map_bio+0x45/0x9b
>> PGD 38057067 PUD 10657067 PMD 0
>> Oops: 0000 [1] SMP
>> CPU 0
>> Pid: 17350, comm: lvcreate Not tainted 2.6.18.8-dom0 #1
>> RIP: e030:[<ffffffff804218a2>]  [<ffffffff804218a2>] __map_bio+0x45/0x9b
>> RSP: e02b:ffff880028693d68  EFLAGS: 00010202
>> RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
>> RDX: ffff8800311bfaf0 RSI: ffff8800219c38c0 RDI: ffffc200000f30d8
>> RBP: ffff8800311bfae0 R08: 0000000000000001 R09: 001322b628693ce8
>> R10: 0000000000000001 R11: 00000000006cc4a0 R12: ffff8800219c38c0
>> R13: fffffffff3800008 R14: ffff88003c56c5d8 R15: ffff8800311bfae0
>> FS:  00002b35f3f7ac20(0000) GS:ffffffff805ca000(0000) 
>> knlGS:0000000000000000
>> CS:  e033 DS: 0000 ES: 0000
>> Process lvcreate (pid: 17350, threadinfo ffff880028692000, task
>> ffff88003cc510c0)
>> Stack:  ffff88003f5a0440 0000000000000001 ffff88003c56c5c0 
>> ffff88003c565400
>>  fffffffff3800008 ffffffff804221b5 ffff88003c56c5d8 ffff88003f5a0440
>>  ffff88003c56c5c0 ffff88003c565400 ffff8800252b1f10 ffffffff8025e099
>> Call Trace:
>>  [<ffffffff804221b5>] __split_bio+0x1a9/0x3f7
>>  [<ffffffff8025e099>] __down_write_nested+0x12/0xf1
>>  [<ffffffff8022fe0e>] __up_write+0x1d/0x115
>>  [<ffffffff804225b9>] dm_resume+0xa1/0x137
>>  [<ffffffff80425158>] dev_suspend+0x0/0x171
>>  [<ffffffff804252a3>] dev_suspend+0x14b/0x171
>>  [<ffffffff80425bb6>] ctl_ioctl+0x1c6/0x20e
>>  [<ffffffff8024012a>] do_ioctl+0x56/0x6c
>>  [<ffffffff8022fddb>] vfs_ioctl+0x23a/0x250
>>  [<ffffffff8024a42f>] sys_ioctl+0x3d/0x5c
>>  [<ffffffff8025a21c>] system_call+0x68/0x6d
>>  [<ffffffff8025a1b4>] system_call+0x0/0x6d
>>
>>
>> Code: ff 50 30 41 89 c5 83 f8 00 7e 10 4c 89 e7 41 59 5b 5d 41 5c
>> RIP  [<ffffffff804218a2>] __map_bio+0x45/0x9b
>>  RSP <ffff880028693d68>
>> CR2: 0000000000000030
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen 3.4.2 kernel / lvcreate crash

[Pasi Kärkkäinen]

On Thu, Apr 01, 2010 at 10:33:57AM +0300, Valtteri Kiviniemi wrote:
> Hi,
>
> I am not using Debian Xen kernel, I have always downloaded the latest  
> release from xen.org and compiled it myself.
>

You might want to take a look at the RHEL5 kernel-xen 2.6.18 sources/patches,
I think their kernel has stable snapshots.

And yeah, RHEL5 kernel has huge amount of patches, so it might take
some time to go through all of them and figure out which ones are relevant..

-- Pasi

> - Valtteri
>
> Jeremy Fitzhardinge kirjoitti:
>> On 03/31/2010 05:34 PM, Valtteri Kiviniemi wrote:
>>> I am using Xen 3.4.2 with kernel 2.6.18.8 and Debian Lenny. I am having
>>> some problems with LVM-snapshots. I have crontab which makes once a
>>> night a snapshot from domU's LV.
>>>
>>> The problem is that maybe once in a month the lvcreate command stucks
>>> and after that I have to reboot the machine to solve the problem.
>>>
>>> So is this a problem on the Xen kernel or with Debian?
>>
>> Someone reported a very similar crash with a Debian Xen kernel.  I 
>> would suspect a bug in the Debian Xen kernel.
>>
>>     J
>>
>>>
>>> dmesg shows the following:
>>>
>>> Unable to handle kernel NULL pointer dereference at 0000000000000030 RIP:
>>>  [<ffffffff804218a2>] __map_bio+0x45/0x9b
>>> PGD 38057067 PUD 10657067 PMD 0
>>> Oops: 0000 [1] SMP
>>> CPU 0
>>> Pid: 17350, comm: lvcreate Not tainted 2.6.18.8-dom0 #1
>>> RIP: e030:[<ffffffff804218a2>]  [<ffffffff804218a2>] __map_bio+0x45/0x9b
>>> RSP: e02b:ffff880028693d68  EFLAGS: 00010202
>>> RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
>>> RDX: ffff8800311bfaf0 RSI: ffff8800219c38c0 RDI: ffffc200000f30d8
>>> RBP: ffff8800311bfae0 R08: 0000000000000001 R09: 001322b628693ce8
>>> R10: 0000000000000001 R11: 00000000006cc4a0 R12: ffff8800219c38c0
>>> R13: fffffffff3800008 R14: ffff88003c56c5d8 R15: ffff8800311bfae0
>>> FS:  00002b35f3f7ac20(0000) GS:ffffffff805ca000(0000)  
>>> knlGS:0000000000000000
>>> CS:  e033 DS: 0000 ES: 0000
>>> Process lvcreate (pid: 17350, threadinfo ffff880028692000, task
>>> ffff88003cc510c0)
>>> Stack:  ffff88003f5a0440 0000000000000001 ffff88003c56c5c0  
>>> ffff88003c565400
>>>  fffffffff3800008 ffffffff804221b5 ffff88003c56c5d8 ffff88003f5a0440
>>>  ffff88003c56c5c0 ffff88003c565400 ffff8800252b1f10 ffffffff8025e099
>>> Call Trace:
>>>  [<ffffffff804221b5>] __split_bio+0x1a9/0x3f7
>>>  [<ffffffff8025e099>] __down_write_nested+0x12/0xf1
>>>  [<ffffffff8022fe0e>] __up_write+0x1d/0x115
>>>  [<ffffffff804225b9>] dm_resume+0xa1/0x137
>>>  [<ffffffff80425158>] dev_suspend+0x0/0x171
>>>  [<ffffffff804252a3>] dev_suspend+0x14b/0x171
>>>  [<ffffffff80425bb6>] ctl_ioctl+0x1c6/0x20e
>>>  [<ffffffff8024012a>] do_ioctl+0x56/0x6c
>>>  [<ffffffff8022fddb>] vfs_ioctl+0x23a/0x250
>>>  [<ffffffff8024a42f>] sys_ioctl+0x3d/0x5c
>>>  [<ffffffff8025a21c>] system_call+0x68/0x6d
>>>  [<ffffffff8025a1b4>] system_call+0x0/0x6d
>>>
>>>
>>> Code: ff 50 30 41 89 c5 83 f8 00 7e 10 4c 89 e7 41 59 5b 5d 41 5c
>>> RIP  [<ffffffff804218a2>] __map_bio+0x45/0x9b
>>>  RSP <ffff880028693d68>
>>> CR2: 0000000000000030
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen 3.4.2 kernel / lvcreate crash

[Jeremy Fitzhardinge]

On 04/01/2010 12:33 AM, Valtteri Kiviniemi wrote:
> I am not using Debian Xen kernel, I have always downloaded the latest 
> release from xen.org and compiled it myself.

Ah.  It would be interesting to see if the pvops-based kernels work any 
better for you.

Thanks,
     J

Re: Xen 3.4.2 kernel / lvcreate crash

[Valtteri Kiviniemi]

Hi,

Of coure I will be upgrading to Xen 4.0 as soon as it released :)

Jeremy Fitzhardinge kirjoitti:
> On 04/01/2010 12:33 AM, Valtteri Kiviniemi wrote:
>> I am not using Debian Xen kernel, I have always downloaded the latest 
>> release from xen.org and compiled it myself.
> 
> Ah.  It would be interesting to see if the pvops-based kernels work any 
> better for you.
> 
> Thanks,
>     J
> 

Re: Xen 3.4.2 kernel / lvcreate crash

[Jeremy Fitzhardinge]

On 04/02/2010 02:26 PM, Valtteri Kiviniemi wrote:
> Of coure I will be upgrading to Xen 4.0 as soon as it released :)

There's nothing stopping you from trying it now, if you want.  The most 
recent 2.6.32 and .33-based kernels require Xen 4.0 or 3.4.3, but you 
can try the 2.6.31-based pvops dom0 kernel.

     J

Re: Xen 3.4.2 kernel / lvcreate crash

[Valtteri Kiviniemi]

Hi,

I have now been using Xen 4 and I have not seen this problem with the 
newer kernel. However, I have still several Xen 3.4 servers with 2.6.18 
kernel that I cant upgrade just yet and I would like to get a fix to 
this problem.

Is there any known solution for the lvcreate crash for 2.6.18 kernel?

Jeremy Fitzhardinge kirjoitti:
> On 04/02/2010 02:26 PM, Valtteri Kiviniemi wrote:
>> Of coure I will be upgrading to Xen 4.0 as soon as it released :)
> 
> There's nothing stopping you from trying it now, if you want.  The most 
> recent 2.6.32 and .33-based kernels require Xen 4.0 or 3.4.3, but you 
> can try the 2.6.31-based pvops dom0 kernel.
> 
>     J
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Xen 3.4.2 kernel / lvcreate crash

[Valtteri Kiviniemi]

Hi,

I am using Xen 3.4.2 with kernel 2.6.18.8 and Debian Lenny. I am having 
some problems with LVM-snapshots. I have crontab which makes once a 
night a snapshot from domU's LV.

The problem is that maybe once in a month the lvcreate command stucks 
and after that I have to reboot the machine to solve the problem.

So is this a problem on the Xen kernel or with Debian?

dmesg shows the following:

Unable to handle kernel NULL pointer dereference at 0000000000000030 RIP:
  [<ffffffff804218a2>] __map_bio+0x45/0x9b
PGD 38057067 PUD 10657067 PMD 0
Oops: 0000 [1] SMP
CPU 0
Pid: 17350, comm: lvcreate Not tainted 2.6.18.8-dom0 #1
RIP: e030:[<ffffffff804218a2>]  [<ffffffff804218a2>] __map_bio+0x45/0x9b
RSP: e02b:ffff880028693d68  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
RDX: ffff8800311bfaf0 RSI: ffff8800219c38c0 RDI: ffffc200000f30d8
RBP: ffff8800311bfae0 R08: 0000000000000001 R09: 001322b628693ce8
R10: 0000000000000001 R11: 00000000006cc4a0 R12: ffff8800219c38c0
R13: fffffffff3800008 R14: ffff88003c56c5d8 R15: ffff8800311bfae0
FS:  00002b35f3f7ac20(0000) GS:ffffffff805ca000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process lvcreate (pid: 17350, threadinfo ffff880028692000, task 
ffff88003cc510c0)
Stack:  ffff88003f5a0440 0000000000000001 ffff88003c56c5c0 ffff88003c565400
  fffffffff3800008 ffffffff804221b5 ffff88003c56c5d8 ffff88003f5a0440
  ffff88003c56c5c0 ffff88003c565400 ffff8800252b1f10 ffffffff8025e099
Call Trace:
  [<ffffffff804221b5>] __split_bio+0x1a9/0x3f7
  [<ffffffff8025e099>] __down_write_nested+0x12/0xf1
  [<ffffffff8022fe0e>] __up_write+0x1d/0x115
  [<ffffffff804225b9>] dm_resume+0xa1/0x137
  [<ffffffff80425158>] dev_suspend+0x0/0x171
  [<ffffffff804252a3>] dev_suspend+0x14b/0x171
  [<ffffffff80425bb6>] ctl_ioctl+0x1c6/0x20e
  [<ffffffff8024012a>] do_ioctl+0x56/0x6c
  [<ffffffff8022fddb>] vfs_ioctl+0x23a/0x250
  [<ffffffff8024a42f>] sys_ioctl+0x3d/0x5c
  [<ffffffff8025a21c>] system_call+0x68/0x6d
  [<ffffffff8025a1b4>] system_call+0x0/0x6d


Code: ff 50 30 41 89 c5 83 f8 00 7e 10 4c 89 e7 41 59 5b 5d 41 5c
RIP  [<ffffffff804218a2>] __map_bio+0x45/0x9b
  RSP <ffff880028693d68>
CR2: 0000000000000030