[linux-lvm] LVM autoactivation and udev

[linux-lvm] LVM autoactivation and udev

[Martin Wilck]

Hi David, hi Peter,

we have recently updated LVM2 on openSUSE Factory, and are using the
autoactivation feature now. We also use
'external_device_info_source="udev"' by default, because according to
our experience it is the only way to make LVM device detection work
reliably with multipath devices.

With these settings, we see lots of "Udev database has incomplete
information about device ..." messages:

> [   12.377563] apollon systemd-udevd[810]: sda5: Processing device (SEQNUM=2787, ACTION=add)
> [   12.412723] apollon systemd-udevd[810]: sda5: /usr/lib/udev/rules.d/69-dm-lvm.rules:82 Importing properties from results of '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/sda5'
> [   12.412767] apollon systemd-udevd[810]: sda5: Starting '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/sda5'
> [   12.413041] apollon systemd-udevd[810]: Successfully forked off '(spawn)' as PID 882.
> [   12.419458] apollon lvm[882]: Udev database has incomplete information about device /dev/sda5.

This is no surprise, because 69-dm-lvm.rules contains

IMPORT{program}="/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output $env{DEVNAME}"
ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="/usr/bin/systemd-run --no-block --property DefaultDependencies=no --unit lvm-activate-$env{LVM_VG_NAME_COMPLETE} (LVM_EXEC)/lvm vgchange -aay --autoactivation event $env{LVM_VG_NAME_COMPLETE}"

These rules are executed by udev while it is processing an "add" event
for a PV. At that time, the udev data base doesn't contain an entry for
this PV yet, because the entry is added only after the uevent is fully
processed.

Shouldn't "pvscan" be run in a RUN+= statement instead? Obviously if we
do this, the lvm-activate-$VG unit must be started in some other way
(e.g. by calling systemd-run directly from pvscan).

In the cases we have observed so far, the VGs were assembled correctly
despite these warnings. We aren't certain if this was by luck only. In
any case, the messages irritate users.

Or are we getting this completely wrong, and the new autoactivation
feature should not be used with external_device_info_source="udev" at
all? If that's the case, how is multipath component detection supposed
to work?

Regards,
Martin

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[David Teigland]

On Wed, Mar 09, 2022 at 04:29:27PM +0100, Martin Wilck wrote:
> Hi David, hi Peter,
> 
> we have recently updated LVM2 on openSUSE Factory, and are using the
> autoactivation feature now. We also use
> 'external_device_info_source="udev"' by default, because according to
> our experience it is the only way to make LVM device detection work
> reliably with multipath devices.
> 
> With these settings, we see lots of "Udev database has incomplete
> information about device ..." messages:
> 
> > [   12.377563] apollon systemd-udevd[810]: sda5: Processing device (SEQNUM=2787, ACTION=add)
> > [   12.412723] apollon systemd-udevd[810]: sda5: /usr/lib/udev/rules.d/69-dm-lvm.rules:82 Importing properties from results of '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/sda5'
> > [   12.412767] apollon systemd-udevd[810]: sda5: Starting '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/sda5'
> > [   12.413041] apollon systemd-udevd[810]: Successfully forked off '(spawn)' as PID 882.
> > [   12.419458] apollon lvm[882]: Udev database has incomplete information about device /dev/sda5.
> 
> This is no surprise, because 69-dm-lvm.rules contains
> 
> IMPORT{program}="/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output $env{DEVNAME}"
> ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="/usr/bin/systemd-run --no-block --property DefaultDependencies=no --unit lvm-activate-$env{LVM_VG_NAME_COMPLETE} (LVM_EXEC)/lvm vgchange -aay --autoactivation event $env{LVM_VG_NAME_COMPLETE}"
> 
> These rules are executed by udev while it is processing an "add" event
> for a PV. At that time, the udev data base doesn't contain an entry for
> this PV yet, because the entry is added only after the uevent is fully
> processed.

Right, this pvscan needs to avoid getting info from udev, regardless of
what's set in lvm.conf.  We don't use udev for external_device_info_source
here so I missed that and will add a fix.

> Shouldn't "pvscan" be run in a RUN+= statement instead? Obviously if we
> do this, the lvm-activate-$VG unit must be started in some other way
> (e.g. by calling systemd-run directly from pvscan).

IMPORT is needed to import LVM_VG_NAME_COMPLETE from the pvscan output
into the udev rule so we know which VG to activate.

> In the cases we have observed so far, the VGs were assembled correctly
> despite these warnings. We aren't certain if this was by luck only. In
> any case, the messages irritate users.
> 
> Or are we getting this completely wrong, and the new autoactivation
> feature should not be used with external_device_info_source="udev" at
> all? 

right

> If that's the case, how is multipath component detection supposed
> to work?

There are multiple ways that it's avoided, some apply in different
situations:

- 69-dm-lvm.rules will often not even be called on a multipath component
  device because udev has already figured out it's a component (I'd need
  some reminding about exactly when/how this happens.)

- if 69-dm-lvm.rules is called on a multipath component, that device will
  not exist in the lvm devices file, so pvscan will ignore it.

- if 69-dm-lvm.rules is called on a multipath component, and there's no
  devices file, then filter-mpath checking sysfs holder info will
  often detect and ignore it.

- if all three of those don't catch it, then filter-mpath will also
  check if the component wwid is listed in /etc/multipath/wwids and
  ignore the device if it is.

If all four of those methods fail to exclude a multipath component, then
an LV could be activated using the component.  While this isn't good, it
can be corrected by running lvchange --refresh.  I'd like to get any
details of that happening to see if we can improve it.

Dave

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[Martin Wilck]

On Wed, 2022-03-09 at 10:27 -0600, David Teigland wrote:
> 
> Right, this pvscan needs to avoid getting info from udev, regardless
> of
> what's set in lvm.conf.  We don't use udev for
> external_device_info_source
> here so I missed that and will add a fix.
> 
> > Shouldn't "pvscan" be run in a RUN+= statement instead? Obviously
> > if we
> > do this, the lvm-activate-$VG unit must be started in some other
> > way
> > (e.g. by calling systemd-run directly from pvscan).
> 
> IMPORT is needed to import LVM_VG_NAME_COMPLETE from the pvscan
> output
> into the udev rule so we know which VG to activate.

I see. That has the (IMO strange) side effect that the "udev property"
LVM_VG_NAME_COMPLETE is set on just one of multiple PVs, and will
disappear when that PV receives another uevent.

If we started pvscan later, in RUN, and a VG became complete, instead
of printing the VG name to stdout, it could run the "systemd-run"
command for lvm-activate-${VG}, which is currently called in 69-dm-
lvm.rules, directly instead, by fork()ing and exec()ing "systemd-run".
That was what I meant. Just a thought, not sure if it really works.

> > In the cases we have observed so far, the VGs were assembled
> > correctly
> > despite these warnings. We aren't certain if this was by luck only.
> > In
> > any case, the messages irritate users.
> > 
> > Or are we getting this completely wrong, and the new autoactivation
> > feature should not be used with external_device_info_source="udev"
> > at
> > all? 
> 
> right
> 
> > If that's the case, how is multipath component detection supposed
> > to work?
> 
> There are multiple ways that it's avoided, some apply in different
> situations:
> 
> - 69-dm-lvm.rules will often not even be called on a multipath
> component
>   device because udev has already figured out it's a component (I'd
> need
>   some reminding about exactly when/how this happens.)

Right: the rules are skipped if ENV{DM_MULTIPATH_DEVICE_PATH}=="1",
which is fine.

> - if 69-dm-lvm.rules is called on a multipath component, that device
> will
>   not exist in the lvm devices file, so pvscan will ignore it.

I need some reminding about how the devices file works :-)


In the past (with the previous event activation scheme), we'd see
effects like this: Suppose we have a dm device dm-1 consisting of 2
SCSI devices sda, sdb. sda is processed by udev, and multipathd sets up
the dm device with just that one member. sdb is present in the system,
but not yet processed by udev and not present in the udev db, thus not
seen by multipathd either. When pvscan was run on the dm device (dm-1,
say), it scanned sysfs (where sdb was present) for other devices. sdb
had no "holders" yet, so pvscan with external_device_info_source="none"
would consider it, find "duplicate devices" dm-1 and sdb, and fail.

Am I understanding correctly that with the new scheme, the devices file
would prevent this from happening?

> - if 69-dm-lvm.rules is called on a multipath component, and there's
> no
>   devices file, then filter-mpath checking sysfs holder info will
>   often detect and ignore it.
> 
> - if all three of those don't catch it, then filter-mpath will also
>   check if the component wwid is listed in /etc/multipath/wwids and
>   ignore the device if it is.

Off-topic: I have seen that, and I'm not particularly happy about it,
because the wwids file doesn't always represent multipathd's view of
the world. It depends on the find_multipaths setting in multipath.conf.
Only if it's set to "strict" (the RHEL default) you can be sure that a
device that isn't in the wwids file will not be grabbed by multipathd
later.

Regards
Martin

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[Roger Heflin]

On Wed, Mar 9, 2022 at 10:35 AM David Teigland <teigland@redhat.com> wrote:
>

>
> - if all three of those don't catch it, then filter-mpath will also
>   check if the component wwid is listed in /etc/multipath/wwids and
>   ignore the device if it is.
>
> If all four of those methods fail to exclude a multipath component, then
> an LV could be activated using the component.  While this isn't good, it
> can be corrected by running lvchange --refresh.  I'd like to get any
> details of that happening to see if we can improve it.
>

I have also had luck with making sure multipath is in the initramfs,
and adding rd.lvm.vg=<rootvgnameneededforboot> on the kernel boot line
so that in the initramfs only the VG needed for boot up is started up
early inside the initramfs.      That has typically let multipathd
inside the initrd grab any other devices and configure them for
multipath before LVM  activates them.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[Martin Wilck]

On Wed, 2022-03-09 at 11:07 -0600, Roger Heflin wrote:
> 
> I have also had luck with making sure multipath is in the initramfs,
> and adding rd.lvm.vg=<rootvgnameneededforboot> on the kernel boot
> line
> so that in the initramfs only the VG needed for boot up is started up
> early inside the initramfs.      That has typically let multipathd
> inside the initrd grab any other devices and configure them for
> multipath before LVM  activates them.
> 

Agreed, tricks like this are often helpful.

My goal is (and has to be, as the maintainer of the multipath package
on a distro) to make this work out of the box, without any
configuration effort on the part of the user, in arbitrary environments
and storage setups. If this stuff fails, users end up in emergency
mode, and with no clue how to recover. 

Recent changes in multipath-tools have modified multipathd startup to
occur earlier in the boot sequence, which is a good thing in general,
but tends to increase the visibilty of shortcomings in other layers of
the storage stack.

Regards,
Martin

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[David Teigland]

On Wed, Mar 09, 2022 at 06:04:07PM +0100, Martin Wilck wrote:
> On Wed, 2022-03-09 at 10:27 -0600, David Teigland wrote:
> > 
> > Right, this pvscan needs to avoid getting info from udev, regardless
> > of
> > what's set in lvm.conf.  We don't use udev for
> > external_device_info_source
> > here so I missed that and will add a fix.
> > 
> > > Shouldn't "pvscan" be run in a RUN+= statement instead? Obviously
> > > if we
> > > do this, the lvm-activate-$VG unit must be started in some other
> > > way
> > > (e.g. by calling systemd-run directly from pvscan).
> > 
> > IMPORT is needed to import LVM_VG_NAME_COMPLETE from the pvscan
> > output
> > into the udev rule so we know which VG to activate.
> 
> I see. That has the (IMO strange) side effect that the "udev property"
> LVM_VG_NAME_COMPLETE is set on just one of multiple PVs, and will
> disappear when that PV receives another uevent.
> 
> If we started pvscan later, in RUN, and a VG became complete, instead
> of printing the VG name to stdout, it could run the "systemd-run"
> command for lvm-activate-${VG}, which is currently called in 69-dm-
> lvm.rules, directly instead, by fork()ing and exec()ing "systemd-run".
> That was what I meant. Just a thought, not sure if it really works.

Having pvscan fork systemd-run vgchange -aay doesn't sound nice at all.
The point of this new design is clean up and simplify things, separating
the scanning from the activation:  pvscan just scans the device, and
vgchange activates the VG.

> > There are multiple ways that it's avoided, some apply in different
> > situations:
> > 
> > - 69-dm-lvm.rules will often not even be called on a multipath
> > component device because udev has already figured out it's a component (I'd
> > need some reminding about exactly when/how this happens.)
> 
> Right: the rules are skipped if ENV{DM_MULTIPATH_DEVICE_PATH}=="1",
> which is fine.
> 
> > - if 69-dm-lvm.rules is called on a multipath component, that device
> > will not exist in the lvm devices file, so pvscan will ignore it.
> 
> I need some reminding about how the devices file works :-)

/etc/lvm/devices/system.devices lists the devices that lvm will use,
and lvm won't look at any other devices.  More details in lvmdevices(8)
https://man7.org/linux/man-pages/man8/lvmdevices.8.html

> In the past (with the previous event activation scheme), we'd see
> effects like this: Suppose we have a dm device dm-1 consisting of 2
> SCSI devices sda, sdb. sda is processed by udev, and multipathd sets up
> the dm device with just that one member. sdb is present in the system,
> but not yet processed by udev and not present in the udev db, thus not
> seen by multipathd either. When pvscan was run on the dm device (dm-1,
> say), it scanned sysfs (where sdb was present) for other devices. sdb
> had no "holders" yet, so pvscan with external_device_info_source="none"
> would consider it, find "duplicate devices" dm-1 and sdb, and fail.
> 
> Am I understanding correctly that with the new scheme, the devices file
> would prevent this from happening?

Right, lvm will never look at sda or sdb because they won't be listed in
system.devices.

> > - if 69-dm-lvm.rules is called on a multipath component, and there's
> > no devices file, then filter-mpath checking sysfs holder info will
> > often detect and ignore it.
> > 
> > - if all three of those don't catch it, then filter-mpath will also
> >   check if the component wwid is listed in /etc/multipath/wwids and
> >   ignore the device if it is.
> 
> Off-topic: I have seen that, and I'm not particularly happy about it,
> because the wwids file doesn't always represent multipathd's view of
> the world. It depends on the find_multipaths setting in multipath.conf.
> Only if it's set to "strict" (the RHEL default) you can be sure that a
> device that isn't in the wwids file will not be grabbed by multipathd
> later.

Yes, it's not perfect, but it may help in some cases, and there are
multiple methods that will usually exclude a component before this.

Dave
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[Martin Wilck]

On Wed, 2022-03-09 at 11:26 -0600, David Teigland wrote:
> > 
> > If we started pvscan later, in RUN, and a VG became complete,
> > instead
> > of printing the VG name to stdout, it could run the "systemd-run"
> > command for lvm-activate-${VG}, which is currently called in 69-dm-
> > lvm.rules, directly instead, by fork()ing and exec()ing
> > "systemd-run".
> > That was what I meant. Just a thought, not sure if it really works.
> 
> Having pvscan fork systemd-run vgchange -aay doesn't sound nice at
> all.
> The point of this new design is clean up and simplify things,
> separating
> the scanning from the activation:  pvscan just scans the device, and
> vgchange activates the VG.

It was just a thought, trying to figure out a way to move this logic in
to RUN completely, so that pvscan would be able to access udev
properties.

I'll wait for the fix for the external_device_info_source=udev case,
which you mentioned in your previous post.

Thanks,
Martin

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[David Teigland]

> I'll wait for the fix for the external_device_info_source=udev case,
> which you mentioned in your previous post.

I only ran a few tests with this but it's pretty straight forward
https://sourceware.org/git/?p=lvm2.git;a=commit;h=c7a5b5cca067496f5912bf992dcfedb6771dd083

Dave
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[heming.zhao]

On 3/9/22 23:29, Martin Wilck wrote:
> Hi David, hi Peter,
> 
> we have recently updated LVM2 on openSUSE Factory, and are using the
> autoactivation feature now. We also use
> 'external_device_info_source="udev"' by default, because according to
> our experience it is the only way to make LVM device detection work
> reliably with multipath devices.
> 
> With these settings, we see lots of "Udev database has incomplete
> information about device ..." messages:
> 
>> [   12.377563] apollon systemd-udevd[810]: sda5: Processing device (SEQNUM=2787, ACTION=add)
>> [   12.412723] apollon systemd-udevd[810]: sda5: /usr/lib/udev/rules.d/69-dm-lvm.rules:82 Importing properties from results of '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/sda5'
>> [   12.412767] apollon systemd-udevd[810]: sda5: Starting '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/sda5'
>> [   12.413041] apollon systemd-udevd[810]: Successfully forked off '(spawn)' as PID 882.
>> [   12.419458] apollon lvm[882]: Udev database has incomplete information about device /dev/sda5.
> 
> This is no surprise, because 69-dm-lvm.rules contains
> 
> IMPORT{program}="/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output $env{DEVNAME}"
> ENV{LVM_VG_NAME_COMPLETE}=="?*", RUN+="/usr/bin/systemd-run --no-block --property DefaultDependencies=no --unit lvm-activate-$env{LVM_VG_NAME_COMPLETE} (LVM_EXEC)/lvm vgchange -aay --autoactivation event $env{LVM_VG_NAME_COMPLETE}"
> 
> These rules are executed by udev while it is processing an "add" event
> for a PV. At that time, the udev data base doesn't contain an entry for
> this PV yet, because the entry is added only after the uevent is fully
> processed.
> 
> Shouldn't "pvscan" be run in a RUN+= statement instead? Obviously if we
> do this, the lvm-activate-$VG unit must be started in some other way
> (e.g. by calling systemd-run directly from pvscan).
> 
> In the cases we have observed so far, the VGs were assembled correctly
> despite these warnings. We aren't certain if this was by luck only. In
> any case, the messages irritate users.
> 
> Or are we getting this completely wrong, and the new autoactivation
> feature should not be used with external_device_info_source="udev" at
> all? If that's the case, how is multipath component detection supposed
> to work?
> 
> Regards,
> Martin
> 

I make a branch for this topic.
Original topic focused on pvscan & udev in booting phase.
We (Martin & I) watched another issue, lvm2-monitor.service reported similar libudev error.
(the lvm2 version is latest 2.03.15+). the monitor issue was still related with libudev
not ready. log:

```
Mar 10 10:27:54 Mobile-PC systemd[1]: Stopped target Local File Systems.
Mar 10 10:27:55 Mobile-PC systemd[1]: Reached target Local File Systems.
Mar 10 10:27:55 Mobile-PC lvm[658]:   Udev database has incomplete information about device /dev/nvme0n1.
Mar 10 10:27:55 Mobile-PC lvm[658]:   /dev/nvme0n1: Failed to get external handle [udev].
Mar 10 10:27:55 Mobile-PC lvm[658]:   Udev database has incomplete information about device /dev/sda.
Mar 10 10:27:55 Mobile-PC lvm[658]:   /dev/sda: Failed to get external handle [udev].
```

In my understand, lvm2-monitor.service does the "clean up" job, which will complete the
monitor job for thin/mirror/others LVs, which was created during initrd phase. (because
lvm_scan doesn't have ability to start monitoring.)
on current lvm2-monitor.service, the dependency asks this service start as early as possible:
```
After=dm-event.socket dm-event.service
```
It makes monitor service start too early, and triggers libudev not ready issue.


To fix above issue, we find a workaround:
```
- After=dm-event.socket dm-event.service
+ After=dm-event.socket dm-event.service sysinit.target
```

And maybe there is another workaround (not verify):
```
-ExecStart=@SBINDIR@/lvm vgchange --monitor y

+ExecStart=@SBINDIR@/lvm vgchange --config 'devices { external_device_info_source="none" \
obtain_device_list_from_udev=0}' --monitor y
```

@David,
What's your opinion about the lvm2-monitor.service udev problem?
Could you mind to give some explain why lvm2-monitor.service is started so early?

Thanks,

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[David Teigland]

On Mon, Mar 21, 2022 at 04:57:09PM +0800, heming.zhao@suse.com wrote:
> ```
> Mar 10 10:27:54 Mobile-PC systemd[1]: Stopped target Local File Systems.
> Mar 10 10:27:55 Mobile-PC systemd[1]: Reached target Local File Systems.
> Mar 10 10:27:55 Mobile-PC lvm[658]:   Udev database has incomplete information about device /dev/nvme0n1.
> Mar 10 10:27:55 Mobile-PC lvm[658]:   /dev/nvme0n1: Failed to get external handle [udev].
> Mar 10 10:27:55 Mobile-PC lvm[658]:   Udev database has incomplete information about device /dev/sda.
> Mar 10 10:27:55 Mobile-PC lvm[658]:   /dev/sda: Failed to get external handle [udev].
> ```

Since vgchange --monitor runs during startup, it should also override any
udev settings to disable them, like pvscan does:

  init_obtain_device_list_from_udev(0);
  init_external_device_info_source(DEV_EXT_NONE);

(Those are the default settings we use, so we don't see this.)


> In my understand, lvm2-monitor.service does the "clean up" job, which will complete the
> monitor job for thin/mirror/others LVs, which was created during initrd phase. (because
> lvm_scan doesn't have ability to start monitoring.)
> on current lvm2-monitor.service, the dependency asks this service start as early as possible:
> ```
> After=dm-event.socket dm-event.service
> ```
> It makes monitor service start too early, and triggers libudev not ready issue.
> 
> 
> To fix above issue, we find a workaround:
> ```
> - After=dm-event.socket dm-event.service
> + After=dm-event.socket dm-event.service sysinit.target
> ```

Yes, I think starting it later would be better.  I don't know enough about
these systemd targets to comment on how well that change would work.  I've
not given much attention to lvm2-monitor, but I've also wished it would
start later.

> And maybe there is another workaround (not verify):
> ```
> -ExecStart=@SBINDIR@/lvm vgchange --monitor y
> 
> +ExecStart=@SBINDIR@/lvm vgchange --config 'devices { external_device_info_source="none" \
> obtain_device_list_from_udev=0}' --monitor y
> ```

Let's patch vgchange monitor as mentioned above.

Dave
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[heming.zhao]

On 3/22/22 00:44, David Teigland wrote:
> On Mon, Mar 21, 2022 at 04:57:09PM +0800, heming.zhao@suse.com wrote:
>> ```
>> Mar 10 10:27:54 Mobile-PC systemd[1]: Stopped target Local File Systems.
>> Mar 10 10:27:55 Mobile-PC systemd[1]: Reached target Local File Systems.
>> Mar 10 10:27:55 Mobile-PC lvm[658]:   Udev database has incomplete information about device /dev/nvme0n1.
>> Mar 10 10:27:55 Mobile-PC lvm[658]:   /dev/nvme0n1: Failed to get external handle [udev].
>> Mar 10 10:27:55 Mobile-PC lvm[658]:   Udev database has incomplete information about device /dev/sda.
>> Mar 10 10:27:55 Mobile-PC lvm[658]:   /dev/sda: Failed to get external handle [udev].
>> ```
> 
> Since vgchange --monitor runs during startup, it should also override any
> udev settings to disable them, like pvscan does:
> 
>    init_obtain_device_list_from_udev(0);
>    init_external_device_info_source(DEV_EXT_NONE);
> 
> (Those are the default settings we use, so we don't see this.)
> 

you mind me, I forgot SUSE chose to use udev by default.
So this is a SUSE special issue.

> 
>> In my understand, lvm2-monitor.service does the "clean up" job, which will complete the
>> monitor job for thin/mirror/others LVs, which was created during initrd phase. (because
>> lvm_scan doesn't have ability to start monitoring.)
>> on current lvm2-monitor.service, the dependency asks this service start as early as possible:
>> ```
>> After=dm-event.socket dm-event.service
>> ```
>> It makes monitor service start too early, and triggers libudev not ready issue.
>>
>>
>> To fix above issue, we find a workaround:
>> ```
>> - After=dm-event.socket dm-event.service
>> + After=dm-event.socket dm-event.service sysinit.target
>> ```
> 
> Yes, I think starting it later would be better.  I don't know enough about
> these systemd targets to comment on how well that change would work.  I've
> not given much attention to lvm2-monitor, but I've also wished it would
> start later> 
>> And maybe there is another workaround (not verify):
>> ```
>> -ExecStart=@SBINDIR@/lvm vgchange --monitor y
>>
>> +ExecStart=@SBINDIR@/lvm vgchange --config 'devices { external_device_info_source="none" \
>> obtain_device_list_from_udev=0}' --monitor y
>> ```
> 
> Let's patch vgchange monitor as mentioned above.
> 
> Dave
> 

I inclined to use the "--config" option to avoid booting warning.
(or write additional codes for vgchange "monitor_ARG")

I have two reasons:

1>
Martin & I also found it is a difficult to find the best time to start lvm2-monitor.service
So modification "After=" dependency will still failed with some cases.

2>
the lvm2-monitor.service helps to finish monitoring job for lvm_scan. So it's not necessary
to ask this service to handle the VG/LV which starting after switch rootfs. (These VG/LV
should be monitored by "pvscan --cache".)

So starting lvm2-monitor.service as early as possible is accepted.

Thanks,

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[Martin Wilck]

On Wed, 2022-03-23 at 15:33 +0800, heming.zhao@suse.com wrote:
> 
> I inclined to use the "--config" option to avoid booting warning.
> (or write additional codes for vgchange "monitor_ARG")
> 
> I have two reasons:
> 
> 1>
> Martin & I also found it is a difficult to find the best time to
> start lvm2-monitor.service
> So modification "After=" dependency will still failed with some
> cases.

Yes. The reason is that even after "sysinit.target" is reached, "udev
settle" for coldplug hasn't necessarily finished these days (as
systemd-udev-settle.service is deprecated, and often not activated any
more). Thus even if started after sysinit.target, lvm2-monitor.service
may encounter devices that haven't been fully processed by udev yet.

> 2>
> the lvm2-monitor.service helps to finish monitoring job for lvm_scan.
> So it's not necessary
> to ask this service to handle the VG/LV which starting after switch
> rootfs. (These VG/LV
> should be monitored by "pvscan --cache".)
> 
> So starting lvm2-monitor.service as early as possible is accepted.

Please forgive me if I am (out of ignorance about dmeventd) totally on
the wrong track here, but:

I still have my doubts. I can see that the warnings about missing udev
information have gone if we use  --config 'devices {
external_device_info_source="none" }'. But that doesn't mean much by
itself. vgchange will instead rely on native device detection, which,
as we know very well, will lead to wrong results more often than not,
in particular if multipath devices are present. IOW, it will probably
ask dmeventd to monitor SCSI devices that are path of multipath maps,
rather than the map devices themselves. I don't know dmeventd well
enough to judge whether that would be fatal, but past experience makes
me wary about it.

I suppose to test that we'd need to setup systems with root FS on
"monitored" LVM volumes (e.g. thin or mirror) on multipath (with recent
multipath releases, 0.8.8 or newer). I for one haven't tested this so
far.

What I would love to see wrt lvm2 monitoring is true event-based
activation - i.e. activate monitoring of devices one by one as they
actually appear in the system, in a manner that's consistent with udev.
Thus something similar to David's late approach with the "devices file"
for PV detection. That would avoid both the need to pass config options
and the need to figure out when to safely start the monitoring.

Regards,
Martin

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[heming.zhao]

On 3/23/22 15:51, Martin Wilck wrote:
> On Wed, 2022-03-23 at 15:33 +0800, heming.zhao@suse.com wrote:
>>
>> I inclined to use the "--config" option to avoid booting warning.
>> (or write additional codes for vgchange "monitor_ARG")
>>
>> I have two reasons:
>>
>> 1>
>> Martin & I also found it is a difficult to find the best time to
>> start lvm2-monitor.service
>> So modification "After=" dependency will still failed with some
>> cases.
> 
> Yes. The reason is that even after "sysinit.target" is reached, "udev
> settle" for coldplug hasn't necessarily finished these days (as
> systemd-udev-settle.service is deprecated, and often not activated any
> more). Thus even if started after sysinit.target, lvm2-monitor.service
> may encounter devices that haven't been fully processed by udev yet.
> 
>> 2>
>> the lvm2-monitor.service helps to finish monitoring job for lvm_scan.
>> So it's not necessary
>> to ask this service to handle the VG/LV which starting after switch
>> rootfs. (These VG/LV
>> should be monitored by "pvscan --cache".)
>>
>> So starting lvm2-monitor.service as early as possible is accepted.
> 
> Please forgive me if I am (out of ignorance about dmeventd) totally on
> the wrong track here, but:
> 
> I still have my doubts. I can see that the warnings about missing udev
> information have gone if we use  --config 'devices {
> external_device_info_source="none" }'. But that doesn't mean much by
> itself. vgchange will instead rely on native device detection, which,
> as we know very well, will lead to wrong results more often than not,
> in particular if multipath devices are present. IOW, it will probably
> ask dmeventd to monitor SCSI devices that are path of multipath maps,
> rather than the map devices themselves. I don't know dmeventd well
> enough to judge whether that would be fatal, but past experience makes
> me wary about it.
> 
> I suppose to test that we'd need to setup systems with root FS on
> "monitored" LVM volumes (e.g. thin or mirror) on multipath (with recent
> multipath releases, 0.8.8 or newer). I for one haven't tested this so
> far.
> 
> What I would love to see wrt lvm2 monitoring is true event-based
> activation - i.e. activate monitoring of devices one by one as they
> actually appear in the system, in a manner that's consistent with udev.
> Thus something similar to David's late approach with the "devices file"
> for PV detection. That would avoid both the need to pass config options
> and the need to figure out when to safely start the monitoring.
> 

Your concern is may right, "vgchange --monirtor y" will call lvmcache_label_scan() to search
active VGs , meanwhile libudev hasn't done on some devs. And then vgchange will output
some warning which can be ignored but may irritate users.

In theory, if "vgchange --monitor y" only handles the VGs which created during initrd phase
can avoid this problem.

- Heming

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[Martin Wilck]

On Thu, 2022-03-24 at 16:26 +0800, heming.zhao@suse.com wrote:
> 
> Your concern is may right, "vgchange --monirtor y" will call
> lvmcache_label_scan() to search
> active VGs , meanwhile libudev hasn't done on some devs. And then
> vgchange will output
> some warning which can be ignored but may irritate users.

Again, I'm not so much concerned about the warning. More about what
happens if "vgchange --monitor y" ignores information from udev.

> In theory, if "vgchange --monitor y" only handles the VGs which
> created during initrd phase
> can avoid this problem.

Yes, but that's not how it currently behaves, right?

Martin

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM autoactivation and udev

[David Teigland]

On Thu, Mar 24, 2022 at 04:26:33PM +0800, heming.zhao@suse.com wrote:
> Your concern is may right, "vgchange --monirtor y" will call lvmcache_label_scan() to search
> active VGs , meanwhile libudev hasn't done on some devs. And then vgchange will output
> some warning which can be ignored but may irritate users.
> 
> In theory, if "vgchange --monitor y" only handles the VGs which created during initrd phase
> can avoid this problem.

I'm adding this to the _vgchange_monitoring() function:
+ init_obtain_device_list_from_udev(0);
+ init_external_device_info_source(DEV_EXT_NONE);
so we won't need to add --config... to vgchange in lvm2-monitor.

This means that the vgchange monitor will not use udev, avoiding the
warnings and stalls caused when udev is slow and uninitialized.

Concerns about lvm commands using native detection, instead of udev info,
should be a thing of the past, assuming we're talking about the most
recent upstream code.  If you still see problems with that please let me
know.

It shouldn't matter when vgchange --monitor runs, or if it tries to
monitor all visible VGs.  It would be nice to do it more efficiently,
looking only at VGs that are unmonitored, or better looking only at VGs
for LVs that use monitoring (it's a wasteful step for many if not most
users.)  I don't see a good way of doing that at the moment.  It would
also be nice start the monitoring after most autoactivations are done, to
avoid any interference, but again I don't see a good way to do it right
now.

Dave
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/