* [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() @ 2011-08-21 18:14 Alexander Lyakas 2011-08-23 11:44 ` Zdenek Kabelac 0 siblings, 1 reply; 10+ messages in thread From: Alexander Lyakas @ 2011-08-21 18:14 UTC (permalink / raw) To: LVM general discussion and development Greetings all, on stock ubuntu natty, using liblvm 2.02.66, sometimes lvm_vg_create_lv_linear () gets stuck in the following stack. #0 0x00007f3b9636a0f7 in semop () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3b9606863c in dm_udev_wait () from /lib/libdevmapper.so.1.02.1 #2 0x00007f3b973df41b in ?? () from /lib/liblvm2app.so.2.2 #3 0x00007f3b973e012e in ?? () from /lib/liblvm2app.so.2.2 #4 0x00007f3b97399aec in ?? () from /lib/liblvm2app.so.2.2 #5 0x00007f3b9739b02b in ?? () from /lib/liblvm2app.so.2.2 #6 0x00007f3b9739bb7e in ?? () from /lib/liblvm2app.so.2.2 #7 0x00007f3b973e4c51 in ?? () from /lib/liblvm2app.so.2.2 #8 0x00007f3b973b9238 in ?? () from /lib/liblvm2app.so.2.2 #9 0x00007f3b973b9e36 in ?? () from /lib/liblvm2app.so.2.2 #10 0x00007f3b973c2118 in ?? () from /lib/liblvm2app.so.2.2 #11 0x00007f3b973989ba in lvm_vg_create_lv_linear () from /lib/liblvm2app.so.2.2 ... I realize that somehow it is related to udev synchronization. In my application, I have a udev monitor for "block" subsystem type, but currently it does nothing with the events it receives (it calls udev_monitor_receive_device(), udev_device_get_action() and then udev_device_unref() for every event. I plan in the future to use it for waking up other threads. Can somebody pls advise on how to investigate this issue. It does not happen always. Thanks, Alex. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-21 18:14 [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() Alexander Lyakas @ 2011-08-23 11:44 ` Zdenek Kabelac 2011-08-24 9:14 ` Alexander Lyakas 0 siblings, 1 reply; 10+ messages in thread From: Zdenek Kabelac @ 2011-08-23 11:44 UTC (permalink / raw) To: LVM general discussion and development; +Cc: Alexander Lyakas Dne 21.8.2011 20:14, Alexander Lyakas napsal(a): > Greetings all, > on stock ubuntu natty, using liblvm 2.02.66, sometimes > lvm_vg_create_lv_linear () gets stuck in the following stack. > > #0 0x00007f3b9636a0f7 in semop () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007f3b9606863c in dm_udev_wait () from /lib/libdevmapper.so.1.02.1 > #2 0x00007f3b973df41b in ?? () from /lib/liblvm2app.so.2.2 > dmsetup udevcomplete_all this should finish the execution of this command. > > I realize that somehow it is related to udev synchronization. In my > application, I have a udev monitor for "block" subsystem type, but > currently it does nothing with the events it receives (it calls > udev_monitor_receive_device(), udev_device_get_action() and then > udev_device_unref() for every event. I plan in the future to use it > for waking up other threads. > > Can somebody pls advise on how to investigate this issue. It does not > happen always. My assumption is - you Debian system is using non-upstream Debian deviation of udev rules and those are broken (I guess some related discussion might be googled) Another issue which is currently being explored is the namespace usage - but I think this is not related to the problem you are seeing. Anyway I'd strongly suggest to use newer lvm package (either build form CVS) or at least Debian unstable version - as 2.02.66 is very old one. Zdenek ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-23 11:44 ` Zdenek Kabelac @ 2011-08-24 9:14 ` Alexander Lyakas 2011-08-24 9:41 ` Zdenek Kabelac 0 siblings, 1 reply; 10+ messages in thread From: Alexander Lyakas @ 2011-08-24 9:14 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: LVM general discussion and development Zdenek, thank you for your reply. I am using stock ubuntu natty: 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux and 2.02.66 is what comes with natty. I will check the udev rules part further. If I want to try the new LVM package, say LVM2.2.02.88.tgz, do I need to do any update to the kernel, or will it work with my existing kernel 2.6.38-8-server. Thanks, Alex. On Tue, Aug 23, 2011 at 2:44 PM, Zdenek Kabelac <zdenek.kabelac@gmail.com> wrote: > Dne 21.8.2011 20:14, Alexander Lyakas napsal(a): >> Greetings all, >> on stock ubuntu natty, using liblvm 2.02.66, sometimes >> lvm_vg_create_lv_linear () gets stuck in the following stack. >> >> #0 �0x00007f3b9636a0f7 in semop () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 �0x00007f3b9606863c in dm_udev_wait () from /lib/libdevmapper.so.1.02.1 >> #2 �0x00007f3b973df41b in ?? () from /lib/liblvm2app.so.2.2 >> > > dmsetup udevcomplete_all > > this should finish the execution of this command. > >> >> I realize that somehow it is related to udev synchronization. In my >> application, I have a udev monitor for "block" subsystem type, but >> currently it does nothing with the events it receives (it calls >> udev_monitor_receive_device(), udev_device_get_action() and then >> udev_device_unref() for every event. I plan in the future to use it >> for waking up other threads. >> >> Can somebody pls advise on how to investigate this issue. It does not >> happen always. > > > My assumption is - you Debian system is using non-upstream Debian deviation of > udev rules and those are broken (I guess some related discussion might be googled) > > Another issue which is currently being explored is the namespace usage - but I > think this is not related to the problem you are seeing. > > Anyway I'd strongly suggest to use newer lvm package (either build form CVS) > or at least �Debian unstable version - as 2.02.66 is very old one. > > Zdenek > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-24 9:14 ` Alexander Lyakas @ 2011-08-24 9:41 ` Zdenek Kabelac 2011-08-29 15:47 ` Alexander Lyakas 0 siblings, 1 reply; 10+ messages in thread From: Zdenek Kabelac @ 2011-08-24 9:41 UTC (permalink / raw) To: Alexander Lyakas; +Cc: LVM general discussion and development 2011/8/24 Alexander Lyakas <alex.bolshoy@gmail.com>: > Zdenek, > thank you for your reply. > > I am using stock ubuntu natty: > 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64 > x86_64 x86_64 GNU/Linux > and 2.02.66 is what comes with natty. > > I will check the udev rules part further. Best would be to use the latest package with it's own rule - but it might be not so easy if you are not udev experienced user to properly fix udev rules in Ubuntu. Eventually you may try this package: http://packages.debian.org/unstable/admin/lvm2 It's not the latest - but there should be bigger chance of more correct udev behaviour. > > If I want to try the new LVM package, say LVM2.2.02.88.tgz, do I need > to do any update to the kernel, or will it work with my existing > kernel 2.6.38-8-server. > It should work out-of-the-box - if you will notice any problem - feel free to report. Zdenek ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-24 9:41 ` Zdenek Kabelac @ 2011-08-29 15:47 ` Alexander Lyakas 2011-08-30 8:03 ` Zdenek Kabelac 0 siblings, 1 reply; 10+ messages in thread From: Alexander Lyakas @ 2011-08-29 15:47 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: LVM general discussion and development Hello Zdenek, I did some investigations, and I think the problem is the following: When an LV is created, udev event comes in. It looks like the problem happens only when this first udev event has the following properties: ID_FS_UUID=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else) ID_FS_UUID_ENC=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else) ID_FS_VERSION=LVM2\x20001 ID_FS_TYPE=LVM2_member ID_FS_USAGE=raid Usually, I see these properties on PVs (which are MD arrays) and not on LVs. I don't really know why these properties sometimes appear on a new LV (some leftover on disk???), but when it happens, then the following ubuntu udev rule kicks in: ---------- code ---------- # This file causes block devices with LVM signatures to be automatically # added to their volume group. # See udev(8) for syntax SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="lvm*|LVM*", RUN+="watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'" ------ end code ------- This rule is described here: https://wiki.ubuntu.com/UdevLvm So vgscan is issued and it gets stuck waiting for the LVM lock, because currently my thread is stuck in lv_create_linear(), waiting for udev event, and my thread holds this lock. At this point the system is stuck. What I did then, I killed the vgscan and vgchange processes, but still additional udev events did not come. Then I issued 'udevadm settle' and next udev event arrived. This next event did not contain the problematic ID_FS_XXX fields. And then LV creation completed smoothly. Also, in all the cases, in which LV creation went smoothly, I did not see the ID_FS_XXX fields. So I have several doubts: 1) I understand why vgscan was stuck, but do not understand why udev events stopped coming. Does udev need the previous event processing rules to complete, before sending new events? 2) Why these problematic ID_FS_XXX properties appeared, and how can we prevent this? 3) Or otherwise, do you think that ubuntu's rule is problematic? In my case, I don't need this rule, because I activate all the LVs in my code anyways. Thanks, Alex. On Wed, Aug 24, 2011 at 12:41 PM, Zdenek Kabelac <zdenek.kabelac@gmail.com> wrote: > 2011/8/24 Alexander Lyakas <alex.bolshoy@gmail.com>: >> Zdenek, >> thank you for your reply. >> >> I am using stock ubuntu natty: >> 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64 >> x86_64 x86_64 GNU/Linux >> and 2.02.66 is what comes with natty. >> >> I will check the udev rules part further. > > Best would be to use the latest package with it's own rule - but it > might be not so easy if you are not udev experienced user to properly > fix udev rules in Ubuntu. Eventually you may try this package: > > http://packages.debian.org/unstable/admin/lvm2 > > It's not the latest - but there should be bigger chance of more > correct udev behaviour. > >> >> If I want to try the new LVM package, say LVM2.2.02.88.tgz, do I need >> to do any update to the kernel, or will it work with my existing >> kernel 2.6.38-8-server. >> > > It should work out-of-the-box - if you will notice any problem - feel > free to report. > > Zdenek > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-29 15:47 ` Alexander Lyakas @ 2011-08-30 8:03 ` Zdenek Kabelac 2011-08-30 8:21 ` Alexander Lyakas 0 siblings, 1 reply; 10+ messages in thread From: Zdenek Kabelac @ 2011-08-30 8:03 UTC (permalink / raw) To: Alexander Lyakas; +Cc: LVM general discussion and development Dne 29.8.2011 17:47, Alexander Lyakas napsal(a): > Hello Zdenek, > I did some investigations, and I think the problem is the following: > > When an LV is created, udev event comes in. It looks like the problem > happens only when this first udev event has the following properties: > ID_FS_UUID=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else) > ID_FS_UUID_ENC=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else) > ID_FS_VERSION=LVM2\x20001 > ID_FS_TYPE=LVM2_member > ID_FS_USAGE=raid > > Usually, I see these properties on PVs (which are MD arrays) and not on LVs. > I don't really know why these properties sometimes appear on a new LV > (some leftover on disk???), but when it happens, then the following > ubuntu udev rule kicks in: > > ---------- code ---------- > # This file causes block devices with LVM signatures to be automatically > # added to their volume group. > # See udev(8) for syntax > SUBSYSTEM=="block", ACTION=="add|change", > ENV{ID_FS_TYPE}=="lvm*|LVM*", RUN+="watershed sh -c '/sbin/lvm vgscan; > /sbin/lvm vgchange -a y'" > ------ end code ------- > > This rule is described here: > https://wiki.ubuntu.com/UdevLvm > > So vgscan is issued and it gets stuck waiting for the LVM lock, > because currently my thread is stuck in lv_create_linear(), waiting > for udev event, and my thread holds this lock. > > At this point the system is stuck. What I did then, I killed the > vgscan and vgchange processes, but still additional udev events did > not come. Then I issued 'udevadm settle' and next udev event arrived. > This next event did not contain the problematic ID_FS_XXX fields. And > then LV creation completed smoothly. Also, in all the cases, in which > LV creation went smoothly, I did not see the ID_FS_XXX fields. > > So I have several doubts: > 1) I understand why vgscan was stuck, but do not understand why udev > events stopped coming. Does udev need the previous event processing > rules to complete, before sending new events? > 2) Why these problematic ID_FS_XXX properties appeared, and how can we > prevent this? > 3) Or otherwise, do you think that ubuntu's rule is problematic? In my > case, I don't need this rule, because I activate all the LVs in my > code anyways. I assume udev vars are being set from blkid execution on block device. I really think version 2.02.66 is way to obsolete especially if all surrounding apps (like udev) have been upgraded many times. Vgscan should not be executed by any udev rules. lvm2 package is bundled with its own udev rules - which are tested and verified to work. If a distribution decides to ignore them and bundle its own version I afraid you need to ask for help the package maintainer in your distro for such support. So I'd suggest to to install latest version from upstream tarball otherwise you will be losing your time fighting with more then a year old bugs. Zdenek ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-30 8:03 ` Zdenek Kabelac @ 2011-08-30 8:21 ` Alexander Lyakas 2011-08-30 9:15 ` Zdenek Kabelac 0 siblings, 1 reply; 10+ messages in thread From: Alexander Lyakas @ 2011-08-30 8:21 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: LVM general discussion and development Zdenek, thank you. I want to try the newer version. Can you pls point me to the "latest version from upstream tarball", because those Linux terms are still quite confusing to me. Thanks, Alex. On Tue, Aug 30, 2011 at 11:03 AM, Zdenek Kabelac <zdenek.kabelac@gmail.com> wrote: > Dne 29.8.2011 17:47, Alexander Lyakas napsal(a): >> Hello Zdenek, >> I did some investigations, and I think the problem is the following: >> >> When an LV is created, udev event comes in. It looks like the problem >> happens only when this first udev event has the following properties: >> ID_FS_UUID=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else) >> ID_FS_UUID_ENC=INtL1j-fpvA-Xx13-iXSL-XtDx-ekzV-8aGHg1 (or something else) >> ID_FS_VERSION=LVM2\x20001 >> ID_FS_TYPE=LVM2_member >> ID_FS_USAGE=raid >> >> Usually, I see these properties on PVs (which are MD arrays) and not on LVs. >> I don't really know why these properties sometimes appear on a new LV >> (some leftover on disk???), but when it happens, then the following >> ubuntu udev rule kicks in: >> >> ---------- code ---------- >> # This file causes block devices with LVM signatures to be automatically >> # added to their volume group. >> # See udev(8) for syntax >> SUBSYSTEM=="block", ACTION=="add|change", >> ENV{ID_FS_TYPE}=="lvm*|LVM*", RUN+="watershed sh -c '/sbin/lvm vgscan; >> /sbin/lvm vgchange -a y'" >> ------ end code ------- >> >> This rule is described here: >> https://wiki.ubuntu.com/UdevLvm >> >> So vgscan is issued and it gets stuck waiting for the LVM lock, >> because currently my thread is stuck in lv_create_linear(), waiting >> for udev event, and my thread holds this lock. >> >> At this point the system is stuck. What I did then, I killed the >> vgscan and vgchange processes, but still additional udev events did >> not come. Then I issued 'udevadm settle' and next udev event arrived. >> This next event did not contain the problematic ID_FS_XXX fields. And >> then LV creation completed smoothly. Also, in all the cases, in which >> LV creation went smoothly, I did not see the ID_FS_XXX fields. >> >> So I have several doubts: >> 1) I understand why vgscan was stuck, but do not understand why udev >> events stopped coming. Does udev need the previous event processing >> rules to complete, before sending new events? >> 2) Why these problematic ID_FS_XXX properties appeared, and how can we >> prevent this? >> 3) Or otherwise, do you think that ubuntu's rule is problematic? In my >> case, I don't need this rule, because I activate all the LVs in my >> code anyways. > > I assume udev vars are being set from blkid execution on block device. > > I really think version 2.02.66 is way to obsolete especially if all > surrounding apps (like udev) have been upgraded many times. > > Vgscan should not be executed by any udev rules. > > lvm2 package is bundled with its own �udev rules - which are tested and > verified to work. If a distribution decides to ignore them and bundle its own > version I afraid you need to ask for help the package maintainer in your > distro for such support. > > So I'd suggest to to install latest version from upstream tarball otherwise > you will be losing your time fighting with more then a year old bugs. > > Zdenek > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-30 8:21 ` Alexander Lyakas @ 2011-08-30 9:15 ` Zdenek Kabelac 2011-09-23 19:40 ` Alexander Lyakas 0 siblings, 1 reply; 10+ messages in thread From: Zdenek Kabelac @ 2011-08-30 9:15 UTC (permalink / raw) To: Alexander Lyakas; +Cc: LVM general discussion and development 2011/8/30 Alexander Lyakas <alex.bolshoy@gmail.com>: > Zdenek, > thank you. > I want to try the newer version. > Can you pls point me to the "latest version from upstream tarball", > because those Linux terms are still quite confusing to me. In this case it might not be so easy as you might need to be familiar with configure and installation of local binary files. Anyway here are some links: http://sourceware.org/lvm2/ ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.88.tgz Zdenek ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-08-30 9:15 ` Zdenek Kabelac @ 2011-09-23 19:40 ` Alexander Lyakas 2011-09-29 10:40 ` Zdenek Kabelac 0 siblings, 1 reply; 10+ messages in thread From: Alexander Lyakas @ 2011-09-23 19:40 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: LVM general discussion and development Hello Zdenek, I just wanted to update you: - ubuntu confirmed that this rule is problematic, and causes also other issues: https://bugs.launchpad.net/ubuntu/+source/udev/+bug/833891 - we have uninstalled the ubuntu lvm package, and using now the 88 version that you pointed at. At this point, this are going pretty well, although sometimes we have conflicts with other ubuntu packages, which try to install lvm/dmsetup from ubuntu, so we are preventing them from doing that. We had to add a code to manually activate the LVs once the VG shows up (it was done by the ubuntu rule), but this is perfectly fine for us. A question that we have: building the official LVM package has a lot of configuration options. We have picked some that make sense, especially the udev synchronization options. But perhaps we missed some important ones. Do you think you can recommend a set of conf options, to make the package most "friendly"? Thanks for your help, Alex. On Tue, Aug 30, 2011 at 12:15 PM, Zdenek Kabelac <zdenek.kabelac@gmail.com> wrote: > > 2011/8/30 Alexander Lyakas <alex.bolshoy@gmail.com>: > > Zdenek, > > thank you. > > I want to try the newer version. > > Can you pls point me to the "latest version from upstream tarball", > > because those Linux terms are still quite confusing to me. > > In this case it might not be so easy as you might need to be familiar > with configure and installation of local binary files. > > Anyway here are some links: > > http://sourceware.org/lvm2/ > > ftp://sources.redhat.com/pub/lvm2/LVM2.2.02.88.tgz > > Zdenek ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() 2011-09-23 19:40 ` Alexander Lyakas @ 2011-09-29 10:40 ` Zdenek Kabelac 0 siblings, 0 replies; 10+ messages in thread From: Zdenek Kabelac @ 2011-09-29 10:40 UTC (permalink / raw) To: Alexander Lyakas; +Cc: LVM general discussion and development 2011/9/23 Alexander Lyakas <alex.bolshoy@gmail.com>: > Hello Zdenek, > I just wanted to update you: > > - ubuntu confirmed that this rule is problematic, and causes also other issues: > https://bugs.launchpad.net/ubuntu/+source/udev/+bug/833891 > > - we have uninstalled the ubuntu lvm package, and using now the 88 > version that you pointed at. > > At this point, this are going pretty well, although sometimes we have > conflicts with other ubuntu packages, which try to install lvm/dmsetup > from ubuntu, so we are preventing them from doing that. > > We had to add a code to manually activate the LVs once the VG shows up > (it was done by the ubuntu rule), but this is perfectly fine for us. > > A question that we have: building the official LVM package has a lot > of configuration options. We have picked some that make sense, > especially the udev synchronization options. But perhaps we missed > some important ones. Do you think you can recommend a set of conf > options, to make the package most "friendly"? > It really depends on the type of use - i.e. you may check Fedora packaging configure options construction in its rpm.spec. Though some of them are now either defaults or not really usable. I think the key ones are: --enable-applib --enable-cmdlib --enable-dmeventd -enable-udev_sync and some cluster ones if you need them --with-clvmd=cman,openais,corosync --enable-cmirrord Zdenek ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2011-09-29 10:40 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-08-21 18:14 [linux-lvm] lvm_vg_create_lv_linear() stuck in dm_udev_wait() Alexander Lyakas 2011-08-23 11:44 ` Zdenek Kabelac 2011-08-24 9:14 ` Alexander Lyakas 2011-08-24 9:41 ` Zdenek Kabelac 2011-08-29 15:47 ` Alexander Lyakas 2011-08-30 8:03 ` Zdenek Kabelac 2011-08-30 8:21 ` Alexander Lyakas 2011-08-30 9:15 ` Zdenek Kabelac 2011-09-23 19:40 ` Alexander Lyakas 2011-09-29 10:40 ` Zdenek Kabelac
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.