[linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Alexander Lyakas]

Greetings all,
on stock ubuntu natty, using liblvm 2.02.66, sometimes
lvm_vg_create_lv_linear () gets stuck in the following stack.

#0  0x00007f3b9636a0f7 in semop () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3b9606863c in dm_udev_wait () from /lib/libdevmapper.so.1.02.1
#2  0x00007f3b973df41b in ?? () from /lib/liblvm2app.so.2.2
#3  0x00007f3b973e012e in ?? () from /lib/liblvm2app.so.2.2
#4  0x00007f3b97399aec in ?? () from /lib/liblvm2app.so.2.2
#5  0x00007f3b9739b02b in ?? () from /lib/liblvm2app.so.2.2
#6  0x00007f3b9739bb7e in ?? () from /lib/liblvm2app.so.2.2
#7  0x00007f3b973e4c51 in ?? () from /lib/liblvm2app.so.2.2
#8  0x00007f3b973b9238 in ?? () from /lib/liblvm2app.so.2.2
#9  0x00007f3b973b9e36 in ?? () from /lib/liblvm2app.so.2.2
#10 0x00007f3b973c2118 in ?? () from /lib/liblvm2app.so.2.2
#11 0x00007f3b973989ba in lvm_vg_create_lv_linear () from /lib/liblvm2app.so.2.2
...

I realize that somehow it is related to udev synchronization. In my
application, I have a udev monitor for "block" subsystem type, but
currently it does nothing with the events it receives (it calls
udev_monitor_receive_device(), udev_device_get_action() and then
udev_device_unref() for every event. I plan in the future to use it
for waking up other threads.

Can somebody pls advise on how to investigate this issue. It does not
happen always.

Thanks,
   Alex.

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Zdenek Kabelac]

Dne 21.8.2011 20:14, Alexander Lyakas napsal(a):
> Greetings all,
> on stock ubuntu natty, using liblvm 2.02.66, sometimes
> lvm_vg_create_lv_linear () gets stuck in the following stack.
> 
> #0  0x00007f3b9636a0f7 in semop () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007f3b9606863c in dm_udev_wait () from /lib/libdevmapper.so.1.02.1
> #2  0x00007f3b973df41b in ?? () from /lib/liblvm2app.so.2.2
>

dmsetup udevcomplete_all

this should finish the execution of this command.

> 
> I realize that somehow it is related to udev synchronization. In my
> application, I have a udev monitor for "block" subsystem type, but
> currently it does nothing with the events it receives (it calls
> udev_monitor_receive_device(), udev_device_get_action() and then
> udev_device_unref() for every event. I plan in the future to use it
> for waking up other threads.
> 
> Can somebody pls advise on how to investigate this issue. It does not
> happen always.


My assumption is - you Debian system is using non-upstream Debian deviation of
udev rules and those are broken (I guess some related discussion might be googled)

Another issue which is currently being explored is the namespace usage - but I
think this is not related to the problem you are seeing.

Anyway I'd strongly suggest to use newer lvm package (either build form CVS)
or at least  Debian unstable version - as 2.02.66 is very old one.

Zdenek

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Alexander Lyakas]

Zdenek,
thank you for your reply.

I am using stock ubuntu natty:
2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64
x86_64 x86_64 GNU/Linux
and 2.02.66 is what comes with natty.

I will check the udev rules part further.

If I want to try the new LVM package, say LVM2.2.02.88.tgz, do I need
to do any update to the kernel, or will it work with my existing
kernel 2.6.38-8-server.

Thanks,
  Alex.

On Tue, Aug 23, 2011 at 2:44 PM, Zdenek Kabelac
<zdenek.kabelac@gmail.com> wrote:
> Dne 21.8.2011 20:14, Alexander Lyakas napsal(a):
>> Greetings all,
>> on stock ubuntu natty, using liblvm 2.02.66, sometimes
>> lvm_vg_create_lv_linear () gets stuck in the following stack.
>>
>> #0 �0x00007f3b9636a0f7 in semop () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 �0x00007f3b9606863c in dm_udev_wait () from /lib/libdevmapper.so.1.02.1
>> #2 �0x00007f3b973df41b in ?? () from /lib/liblvm2app.so.2.2
>>
>
> dmsetup udevcomplete_all
>
> this should finish the execution of this command.
>
>>
>> I realize that somehow it is related to udev synchronization. In my
>> application, I have a udev monitor for "block" subsystem type, but
>> currently it does nothing with the events it receives (it calls
>> udev_monitor_receive_device(), udev_device_get_action() and then
>> udev_device_unref() for every event. I plan in the future to use it
>> for waking up other threads.
>>
>> Can somebody pls advise on how to investigate this issue. It does not
>> happen always.
>
>
> My assumption is - you Debian system is using non-upstream Debian deviation of
> udev rules and those are broken (I guess some related discussion might be googled)
>
> Another issue which is currently being explored is the namespace usage - but I
> think this is not related to the problem you are seeing.
>
> Anyway I'd strongly suggest to use newer lvm package (either build form CVS)
> or at least �Debian unstable version - as 2.02.66 is very old one.
>
> Zdenek
>
>

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Zdenek Kabelac]

2011/8/24 Alexander Lyakas <alex.bolshoy@gmail.com>:
> Zdenek,
> thank you for your reply.
>
> I am using stock ubuntu natty:
> 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64
> x86_64 x86_64 GNU/Linux
> and 2.02.66 is what comes with natty.
>
> I will check the udev rules part further.

Best would be to use the latest package with it's own rule - but it
might be not so easy if you are not udev experienced user to properly
fix udev rules in Ubuntu. Eventually you may try this package:

http://packages.debian.org/unstable/admin/lvm2

It's not the latest - but there should be bigger chance of more
correct udev behaviour.

>
> If I want to try the new LVM package, say LVM2.2.02.88.tgz, do I need
> to do any update to the kernel, or will it work with my existing
> kernel 2.6.38-8-server.
>

It should work out-of-the-box - if you will notice any problem - feel
free to report.

Zdenek

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Alexander Lyakas]

Hello Zdenek,
I did some investigations, and I think the problem is the following:

When an LV is created, udev event comes in. It looks like the problem
happens only when this first udev event has the following properties:
ID_FS_UUID=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else)
ID_FS_UUID_ENC=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else)
ID_FS_VERSION=LVM2\x20001
ID_FS_TYPE=LVM2_member
ID_FS_USAGE=raid

Usually, I see these properties on PVs (which are MD arrays) and not on LVs.
I don't really know why these properties sometimes appear on a new LV
(some leftover on disk???), but when it happens, then the following
ubuntu udev rule kicks in:

---------- code ----------
# This file causes block devices with LVM signatures to be automatically
# added to their volume group.
# See udev(8) for syntax
SUBSYSTEM=="block", ACTION=="add|change",
ENV{ID_FS_TYPE}=="lvm*|LVM*", RUN+="watershed sh -c '/sbin/lvm vgscan;
/sbin/lvm vgchange -a y'"
------ end code -------

This rule is described here:
https://wiki.ubuntu.com/UdevLvm

So vgscan is issued and it gets stuck waiting for the LVM lock,
because currently my thread is stuck in lv_create_linear(), waiting
for udev event, and my thread holds this lock.

At this point the system is stuck. What I did then, I killed the
vgscan and vgchange processes, but still additional udev events did
not come. Then I issued 'udevadm settle' and next udev event arrived.
This next event did not contain the problematic ID_FS_XXX fields. And
then LV creation completed smoothly. Also, in all the cases, in which
LV creation went smoothly, I did not see the ID_FS_XXX fields.

So I have several doubts:
1) I understand why vgscan was stuck, but do not understand why udev
events stopped coming. Does udev need the previous event processing
rules to complete, before sending new events?
2) Why these problematic ID_FS_XXX properties appeared, and how can we
prevent this?
3) Or otherwise, do you think that ubuntu's rule is problematic? In my
case, I don't need this rule, because I activate all the LVs in my
code anyways.

Thanks,
  Alex.


On Wed, Aug 24, 2011 at 12:41 PM, Zdenek Kabelac
<zdenek.kabelac@gmail.com> wrote:
> 2011/8/24 Alexander Lyakas <alex.bolshoy@gmail.com>:
>> Zdenek,
>> thank you for your reply.
>>
>> I am using stock ubuntu natty:
>> 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64
>> x86_64 x86_64 GNU/Linux
>> and 2.02.66 is what comes with natty.
>>
>> I will check the udev rules part further.
>
> Best would be to use the latest package with it's own rule - but it
> might be not so easy if you are not udev experienced user to properly
> fix udev rules in Ubuntu. Eventually you may try this package:
>
> http://packages.debian.org/unstable/admin/lvm2
>
> It's not the latest - but there should be bigger chance of more
> correct udev behaviour.
>
>>
>> If I want to try the new LVM package, say LVM2.2.02.88.tgz, do I need
>> to do any update to the kernel, or will it work with my existing
>> kernel 2.6.38-8-server.
>>
>
> It should work out-of-the-box - if you will notice any problem - feel
> free to report.
>
> Zdenek
>

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Zdenek Kabelac]

Dne 29.8.2011 17:47, Alexander Lyakas napsal(a):
> Hello Zdenek,
> I did some investigations, and I think the problem is the following:
> 
> When an LV is created, udev event comes in. It looks like the problem
> happens only when this first udev event has the following properties:
> ID_FS_UUID=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else)
> ID_FS_UUID_ENC=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else)
> ID_FS_VERSION=LVM2\x20001
> ID_FS_TYPE=LVM2_member
> ID_FS_USAGE=raid
> 
> Usually, I see these properties on PVs (which are MD arrays) and not on LVs.
> I don't really know why these properties sometimes appear on a new LV
> (some leftover on disk???), but when it happens, then the following
> ubuntu udev rule kicks in:
> 
> ---------- code ----------
> # This file causes block devices with LVM signatures to be automatically
> # added to their volume group.
> # See udev(8) for syntax
> SUBSYSTEM=="block", ACTION=="add|change",
> ENV{ID_FS_TYPE}=="lvm*|LVM*", RUN+="watershed sh -c '/sbin/lvm vgscan;
> /sbin/lvm vgchange -a y'"
> ------ end code -------
> 
> This rule is described here:
> https://wiki.ubuntu.com/UdevLvm
> 
> So vgscan is issued and it gets stuck waiting for the LVM lock,
> because currently my thread is stuck in lv_create_linear(), waiting
> for udev event, and my thread holds this lock.
> 
> At this point the system is stuck. What I did then, I killed the
> vgscan and vgchange processes, but still additional udev events did
> not come. Then I issued 'udevadm settle' and next udev event arrived.
> This next event did not contain the problematic ID_FS_XXX fields. And
> then LV creation completed smoothly. Also, in all the cases, in which
> LV creation went smoothly, I did not see the ID_FS_XXX fields.
> 
> So I have several doubts:
> 1) I understand why vgscan was stuck, but do not understand why udev
> events stopped coming. Does udev need the previous event processing
> rules to complete, before sending new events?
> 2) Why these problematic ID_FS_XXX properties appeared, and how can we
> prevent this?
> 3) Or otherwise, do you think that ubuntu's rule is problematic? In my
> case, I don't need this rule, because I activate all the LVs in my
> code anyways.

I assume udev vars are being set from blkid execution on block device.

I really think version 2.02.66 is way to obsolete especially if all
surrounding apps (like udev) have been upgraded many times.

Vgscan should not be executed by any udev rules.

lvm2 package is bundled with its own  udev rules - which are tested and
verified to work. If a distribution decides to ignore them and bundle its own
version I afraid you need to ask for help the package maintainer in your
distro for such support.

So I'd suggest to to install latest version from upstream tarball otherwise
you will be losing your time fighting with more then a year old bugs.

Zdenek

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Alexander Lyakas]

Zdenek,
thank you.
I want to try the newer version.
Can you pls point me to the "latest version from upstream tarball",
because those Linux terms are still quite confusing to me.

Thanks,
  Alex.



On Tue, Aug 30, 2011 at 11:03 AM, Zdenek Kabelac
<zdenek.kabelac@gmail.com> wrote:
> Dne 29.8.2011 17:47, Alexander Lyakas napsal(a):
>> Hello Zdenek,
>> I did some investigations, and I think the problem is the following:
>>
>> When an LV is created, udev event comes in. It looks like the problem
>> happens only when this first udev event has the following properties:
>> ID_FS_UUID=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else)
>> ID_FS_UUID_ENC=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else)
>> ID_FS_VERSION=LVM2\x20001
>> ID_FS_TYPE=LVM2_member
>> ID_FS_USAGE=raid
>>
>> Usually, I see these properties on PVs (which are MD arrays) and not on LVs.
>> I don't really know why these properties sometimes appear on a new LV
>> (some leftover on disk???), but when it happens, then the following
>> ubuntu udev rule kicks in:
>>
>> ---------- code ----------
>> # This file causes block devices with LVM signatures to be automatically
>> # added to their volume group.
>> # See udev(8) for syntax
>> SUBSYSTEM=="block", ACTION=="add|change",
>> ENV{ID_FS_TYPE}=="lvm*|LVM*", RUN+="watershed sh -c '/sbin/lvm vgscan;
>> /sbin/lvm vgchange -a y'"
>> ------ end code -------
>>
>> This rule is described here:
>> https://wiki.ubuntu.com/UdevLvm
>>
>> So vgscan is issued and it gets stuck waiting for the LVM lock,
>> because currently my thread is stuck in lv_create_linear(), waiting
>> for udev event, and my thread holds this lock.
>>
>> At this point the system is stuck. What I did then, I killed the
>> vgscan and vgchange processes, but still additional udev events did
>> not come. Then I issued 'udevadm settle' and next udev event arrived.
>> This next event did not contain the problematic ID_FS_XXX fields. And
>> then LV creation completed smoothly. Also, in all the cases, in which
>> LV creation went smoothly, I did not see the ID_FS_XXX fields.
>>
>> So I have several doubts:
>> 1) I understand why vgscan was stuck, but do not understand why udev
>> events stopped coming. Does udev need the previous event processing
>> rules to complete, before sending new events?
>> 2) Why these problematic ID_FS_XXX properties appeared, and how can we
>> prevent this?
>> 3) Or otherwise, do you think that ubuntu's rule is problematic? In my
>> case, I don't need this rule, because I activate all the LVs in my
>> code anyways.
>
> I assume udev vars are being set from blkid execution on block device.
>
> I really think version 2.02.66 is way to obsolete especially if all
> surrounding apps (like udev) have been upgraded many times.
>
> Vgscan should not be executed by any udev rules.
>
> lvm2 package is bundled with its own �udev rules - which are tested and
> verified to work. If a distribution decides to ignore them and bundle its own
> version I afraid you need to ask for help the package maintainer in your
> distro for such support.
>
> So I'd suggest to to install latest version from upstream tarball otherwise
> you will be losing your time fighting with more then a year old bugs.
>
> Zdenek
>

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Zdenek Kabelac]

2011/8/30 Alexander Lyakas <alex.bolshoy@gmail.com>:
> Zdenek,
> thank you.
> I want to try the newer version.
> Can you pls point me to the "latest version from upstream tarball",
> because those Linux terms are still quite confusing to me.

In this case it might not be so easy as you might need to be familiar
with configure and installation of local binary files.

Anyway here are some links:

http://sourceware.org/lvm2/

ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.88.tgz

Zdenek

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Alexander Lyakas]

Hello Zdenek,
I just wanted to update you:

- ubuntu confirmed that this rule is problematic, and causes also other issues:
https://bugs.launchpad.net/ubuntu/+source/udev/+bug/833891

- we have uninstalled the ubuntu lvm package, and using now the 88
version that you pointed at.

At this point, this are going pretty well, although sometimes we have
conflicts with other ubuntu packages, which try to install lvm/dmsetup
from ubuntu, so we are preventing them from doing that.

We had to add a code to manually activate the LVs once the VG shows up
(it was done by the ubuntu rule), but this is perfectly fine for us.

A question that we have: building the official LVM package has a lot
of configuration options. We have picked some that make sense,
especially the udev synchronization options. But perhaps we missed
some important ones. Do you think you can recommend a set of conf
options, to make the package most "friendly"?

Thanks for your help,
  Alex.


On Tue, Aug 30, 2011 at 12:15 PM, Zdenek Kabelac
<zdenek.kabelac@gmail.com> wrote:
>
> 2011/8/30 Alexander Lyakas <alex.bolshoy@gmail.com>:
> > Zdenek,
> > thank you.
> > I want to try the newer version.
> > Can you pls point me to the "latest version from upstream tarball",
> > because those Linux terms are still quite confusing to me.
>
> In this case it might not be so easy as you might need to be familiar
> with configure and installation of local binary files.
>
> Anyway here are some links:
>
> http://sourceware.org/lvm2/
>
> ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.88.tgz
>
> Zdenek

Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait()

[Zdenek Kabelac]

2011/9/23 Alexander Lyakas <alex.bolshoy@gmail.com>:
> Hello Zdenek,
> I just wanted to update you:
>
> - ubuntu confirmed that this rule is problematic, and causes also other issues:
> https://bugs.launchpad.net/ubuntu/+source/udev/+bug/833891
>
> - we have uninstalled the ubuntu lvm package, and using now the 88
> version that you pointed at.
>
> At this point, this are going pretty well, although sometimes we have
> conflicts with other ubuntu packages, which try to install lvm/dmsetup
> from ubuntu, so we are preventing them from doing that.
>
> We had to add a code to manually activate the LVs once the VG shows up
> (it was done by the ubuntu rule), but this is perfectly fine for us.
>
> A question that we have: building the official LVM package has a lot
> of configuration options. We have picked some that make sense,
> especially the udev synchronization options. But perhaps we missed
> some important ones. Do you think you can recommend a set of conf
> options, to make the package most "friendly"?
>

It really depends on the type of use - i.e. you may check Fedora
packaging configure options construction in its rpm.spec.

Though some of them are now either defaults or not really usable.

I think the key ones are:  --enable-applib --enable-cmdlib
--enable-dmeventd -enable-udev_sync
and some cluster ones if you need them
--with-clvmd=cman,openais,corosync --enable-cmirrord

Zdenek