[linux-lvm] lvm operation hangs at semop()

[linux-lvm] lvm operation hangs at semop()

[Neutron Sharc]

Hi all,

I'm running very latest lvm2 code and I frequently see lvm command
hangs. gdb shows the code is blocked waiting for semop to complete.

Is there some reference about udev rules I should use?

Below is an example of backtrace of "lvcreate" hanging:
(gdb) bt
#0  0x00007fdd91e64057 in semop () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007fdd925829b9 in _udev_wait (cookie=223166712,
nowait=nowait@entry=0x7fff92fd9884) at libdm-common.c:2611
#2  0x00007fdd92584087 in dm_udev_wait (cookie=<optimized out>) at
libdm-common.c:2630
#3  0x0000560b22a2ca65 in fs_unlock () at activate/fs.c:491
#4  0x0000560b22a36545 in _file_lock_resource (cmd=<optimized out>,
resource=<optimized out>, flags=320, lv=<optimized out>) at
locking/file_locking.c:64
#5  0x0000560b229b9e88 in _lock_vol (cmd=cmd@entry=0x560b249ae000,
resource=<optimized out>, resource@entry=0x7fff92fda950 "#sync_names",
flags=flags@entry=320, lv_op=lv_op@entry=LV_NOOP,
    lv=lv@entry=0x0) at locking/locking.c:275
#6  0x0000560b229ba7f3 in lock_vol (cmd=0x560b249ae000, vol=<optimized
out>, vol@entry=0x560b22a82839 "#sync_names", flags=flags@entry=320,
lv=lv@entry=0x0) at locking/locking.c:355
#7  0x0000560b229bb490 in sync_local_dev_names (cmd=<optimized out>)
at locking/locking.c:529
#8  0x0000560b229d1c7a in wipe_lv (lv=lv@entry=0x560b24a7f4d0, wp=...)
at metadata/lv_manip.c:7064
#9  0x0000560b229d6437 in _lv_create_an_lv
(vg=vg@entry=0x560b24a7d490, lp=lp@entry=0x7fff92fdbfe0,
new_lv_name=<optimized out>) at metadata/lv_manip.c:7842
#10 0x0000560b229d722f in lv_create_single
(vg=vg@entry=0x560b24a7d490, lp=lp@entry=0x7fff92fdbfe0) at
metadata/lv_manip.c:8018
#11 0x0000560b22953b44 in _lvcreate_single
(cmd=cmd@entry=0x560b249ae000, vg_name=vg_name@entry=0x560b249f6a40
"vol3name", vg=vg@entry=0x560b24a7d490,
handle=handle@entry=0x560b249f6998)
    at lvcreate.c:1608
#12 0x0000560b22972530 in _process_vgnameid_list
(process_single_vg=0x560b22952ff0 <_lvcreate_single>,
handle=0x560b249f6998, arg_tags=0x7fff92fdbe40,
arg_vgnames=0x7fff92fdbe50,
    vgnameids_to_process=0x7fff92fdbe70, read_flags=<optimized out>,
cmd=0x560b249ae000) at toollib.c:1964
#13 process_each_vg (cmd=cmd@entry=0x560b249ae000, argc=argc@entry=0,
argv=argv@entry=0x0, one_vgname=<optimized out>,
use_vgnames=use_vgnames@entry=0x0, read_flags=<optimized out>,
    read_flags@entry=1048576, include_internal=0,
handle=0x560b249f6998, process_single_vg=0x560b22952ff0
<_lvcreate_single>) at toollib.c:2277
#14 0x0000560b22956628 in lvcreate (cmd=0x560b249ae000,
argc=<optimized out>, argv=<optimized out>) at lvcreate.c:1647
#15 0x0000560b2295d9d1 in lvm_run_command
(cmd=cmd@entry=0x560b249ae000, argc=1, argc@entry=10,
argv=0x7fff92fdc490, argv@entry=0x7fff92fdc448) at lvmcmdline.c:2880
#16 0x0000560b2295e4d1 in lvm2_main (argc=10, argv=0x7fff92fdc448) at
lvmcmdline.c:3408
#17 0x00007fdd91d7c830 in __libc_start_main (main=0x560b2293ea10
<main>, argc=10, argv=0x7fff92fdc448, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fff92fdc438) at ../csu/libc-start.c:291
#18 0x0000560b2293ea49 in _start ()


I'm using (almost) latest lvm2 code from Mar 16:
 73d0280 [origin/master] lvmetad: fix bug in snprintf of disable reason

Re: [linux-lvm] lvm operation hangs at semop()

[Alasdair G Kergon]

On Tue, Mar 21, 2017 at 12:07:59PM -0700, Neutron Sharc wrote:
> Is there some reference about udev rules I should use?
 
It should all be handled consistently by configure and make - be sure to
build and install everything in a consistent way.

If you're using a linux distribution, you'll want to build your
replacement packages using the same lvm2 configuration options / patches
that that distribution uses.

Alasdair

Re: [linux-lvm] lvm operation hangs at semop()

[Alasdair G Kergon]

On Wed, Mar 22, 2017 at 11:25:12PM +0000, Alasdair G Kergon wrote:
> As I said before, if you're using a standard distro, you must find out what
> patches and config options were used to build the installed packages and use
> the same ones when you configure your own lvm build.
 
I've added an upstream patch so that - once it has found its way into
the distributions - you'll be able to get quickly the configuration line
used by the build by running 'lvm version'.

Alasdair

Re: [linux-lvm] lvm operation hangs at semop()

[Alasdair G Kergon]

Most likely your lvm and udev rules are incompatible.

As I said before, if you're using a standard distro, you must find out what
patches and config options were used to build the installed packages and use
the same ones when you configure your own lvm build.

Alasdair

Re: [linux-lvm] lvm operation hangs at semop()

[Neutron Sharc]

If I launch lvm command manually everything works fine.

However if I run several lvm commands back to back (to create multiple
volume groups etc) in a script,  very frequently I see  some lvm
commands hang.

Does this mean I should make some delay inbetween back-to-back lvm commands?