Re: [linux-lvm] Discussion: performance issue on event activation mode

[Martin Wilck <martin.wilck@suse.com>]

On Mo, 2021-06-07 at 16:30 -0500, David Teigland wrote:
> On Mon, Jun 07, 2021 at 10:27:20AM +0000, Martin Wilck wrote:
> > Most importantly, this was about LVM2 scanning of physical volumes.
> > The
> > number of udev workers has very little influence on PV scanning,
> > because the udev rules only activate systemd service. The actual
> > scanning takes place in lvm2-pvscan@.service. And unlike udev,
> > there's
> > no limit for the number of instances of a given systemd service
> > template that can run at any given time.
> 
> Excessive device scanning has been the historical problem in this area,
> but Heming mentioned dev_cache_scan() specifically as a problem.  That
> was
> surprising to me since it doesn't scan/read devices, it just creates a
> list of device names on the system (either readdir in /dev or udev
> listing.)  If there are still problems with excessive
> scannning/reading,
> we'll need some more diagnosis of what's happening, there could be some
> cases we've missed.

Heming didn't include his measurement results in the initial post.
Here's a small summary. Heming will be able to provide more details.
You'll see that the effects are quite drastic, factors 3-4 between
every step below, factor >60 between best and worst. I'd say these
results are typical for what we observe also on real-world systems.

kvm-qemu, 6 vcpu, 20G memory, 1258 scsi disks, 1015 vg/lv
Shown is "systemd-analyze blame" output.

 1) lvm2 2.03.05 (SUSE SLE15-SP2),
    obtain_device_list_from_udev=1 & event_activation=1
        9min 51.782s lvm2-pvscan@253:2.service
        9min 51.626s lvm2-pvscan@65:96.service
    (many other lvm2-pvscan@ services follow)
 2) lvm2 latest master
    obtain_device_list_from_udev=1 & event_activation=1
        2min 6.736s lvm2-pvscan@70:384.service         
        2min 6.628s lvm2-pvscan@70:400.service
 3) lvm2 latest master
    obtain_device_list_from_udev=0 & event_activation=1
            40.589s lvm2-pvscan@131:976.service
            40.589s lvm2-pvscan@131:928.service
 4) lvm2 latest master
    obtain_device_list_from_udev=0 & event_activation=0,
            21.034s dracut-initqueue.service
             8.674s lvm2-activation-early.service

IIUC, 2) is the effect of _pvscan_aa_quick(). 3) is surprising;
apparently libudev's device detection causes a factor 3 slowdown.
While 40s is not bad, you can see that event based activation still
performs far worse than "serial" device detection lvm2-activation-
early.service.

Personally, I'm sort of wary about obtain_device_list_from_udev=0
because I'm uncertain whether it might break multipath/MD detection.
Perhaps you can clarify that.

Regards
Martin



_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/