Re: [linux-lvm] Discussion: performance issue on event activation mode

["heming.zhao@suse.com" <heming.zhao@suse.com>]

On 6/8/21 4:26 PM, Martin Wilck wrote:
> On Mo, 2021-06-07 at 16:30 -0500, David Teigland wrote:
>> On Mon, Jun 07, 2021 at 10:27:20AM +0000, Martin Wilck wrote:
>>> Most importantly, this was about LVM2 scanning of physical volumes.
>>> The
>>> number of udev workers has very little influence on PV scanning,
>>> because the udev rules only activate systemd service. The actual
>>> scanning takes place in lvm2-pvscan@.service. And unlike udev,
>>> there's
>>> no limit for the number of instances of a given systemd service
>>> template that can run at any given time.
>>
>> Excessive device scanning has been the historical problem in this area,
>> but Heming mentioned dev_cache_scan() specifically as a problem.  That
>> was
>> surprising to me since it doesn't scan/read devices, it just creates a
>> list of device names on the system (either readdir in /dev or udev
>> listing.)  If there are still problems with excessive
>> scannning/reading,
>> we'll need some more diagnosis of what's happening, there could be some
>> cases we've missed.
> 
> Heming didn't include his measurement results in the initial post.
> Here's a small summary. Heming will be able to provide more details.
> You'll see that the effects are quite drastic, factors 3-4 between
> every step below, factor >60 between best and worst. I'd say these
> results are typical for what we observe also on real-world systems.
> 
> kvm-qemu, 6 vcpu, 20G memory, 1258 scsi disks, 1015 vg/lv
> Shown is "systemd-analyze blame" output.
> 
>   1) lvm2 2.03.05 (SUSE SLE15-SP2),
>      obtain_device_list_from_udev=1 & event_activation=1
>          9min 51.782s lvm2-pvscan@253:2.service
>          9min 51.626s lvm2-pvscan@65:96.service
>      (many other lvm2-pvscan@ services follow)
>   2) lvm2 latest master
>      obtain_device_list_from_udev=1 & event_activation=1
>          2min 6.736s lvm2-pvscan@70:384.service
>          2min 6.628s lvm2-pvscan@70:400.service
>   3) lvm2 latest master
>      obtain_device_list_from_udev=0 & event_activation=1
>              40.589s lvm2-pvscan@131:976.service
>              40.589s lvm2-pvscan@131:928.service
>   4) lvm2 latest master
>      obtain_device_list_from_udev=0 & event_activation=0,
>              21.034s dracut-initqueue.service
>               8.674s lvm2-activation-early.service
> 
> IIUC, 2) is the effect of _pvscan_aa_quick(). 3) is surprising;
> apparently libudev's device detection causes a factor 3 slowdown.
> While 40s is not bad, you can see that event based activation still
> performs far worse than "serial" device detection lvm2-activation-
> early.service.
> 
> Personally, I'm sort of wary about obtain_device_list_from_udev=0
> because I'm uncertain whether it might break multipath/MD detection.
> Perhaps you can clarify that.
> 
> Regards
> Martin
> 
> 

my latest test results. there combines 3 cfg items:
devices/obtain_device_list_from_udev
global/event_activation
activation/udev_sync

<0> is under lvm2-2.03.05+
<1> ~ <8> is under lvm2-2.03.12+

all results are from "systemd-analyze blame", and I only
post top n services.


0>
with suse 15sp2 lvm2 version: lvm2-2.03.05+
"systemd-analyze blame" show the top serives:

devices/obtain_device_list_from_udev=1
global/event_activation=1
activation/udev_sync=1

     9min 51.782s lvm2-pvscan@253:2.service <===
     9min 51.626s lvm2-pvscan@65:96.service
     9min 51.625s lvm2-pvscan@65:208.service
     9min 51.624s lvm2-pvscan@65:16.service
     9min 51.622s lvm2-pvscan@8:176.service
     9min 51.614s lvm2-pvscan@65:144.service

1>
devices/obtain_device_list_from_udev=1
global/event_activation=0
activation/udev_sync=0

          18.307s dracut-initqueue.service
           6.168s btrfsmaintenance-refresh.service
           4.327s systemd-udev-settle.service
           3.633s wicked.service
           2.976s lvm2-activation-early.service  <===
           1.560s lvm2-pvscan@135:832.service
           1.559s lvm2-pvscan@135:816.service
           1.558s lvm2-pvscan@135:784.service
           1.558s lvm2-pvscan@134:976.service
           1.557s lvm2-pvscan@134:832.service
           1.556s dev-system-swap.swap
           1.554s lvm2-pvscan@134:992.service
           1.553s lvm2-pvscan@134:1008.service

2>
devices/obtain_device_list_from_udev=0
global/event_activation=0
activation/udev_sync=0

          17.164s dracut-initqueue.service
          10.420s wicked.service
           7.109s btrfsmaintenance-refresh.service
           4.471s systemd-udev-settle.service
           3.415s lvm2-activation-early.service <===
           1.679s lvm2-pvscan@135:816.service
           1.678s lvm2-pvscan@135:832.service
           1.677s lvm2-pvscan@134:992.service
           1.675s lvm2-pvscan@135:784.service
           1.674s lvm2-pvscan@134:928.service
           1.673s lvm2-pvscan@134:896.service
           1.673s dev-system-swap.swap
           1.672s lvm2-pvscan@134:1008.service


3>
devices/obtain_device_list_from_udev=1
global/event_activation=0
activation/udev_sync=1

          17.552s dracut-initqueue.service
           7.401s lvm2-activation-early.service <====
           6.519s btrfsmaintenance-refresh.service
           5.375s systemd-udev-settle.service
           3.588s wicked.service
           1.723s wickedd-nanny.service
           1.686s wickedd.service
           1.655s lvm2-pvscan@129:992.service
           1.654s lvm2-pvscan@129:960.service
           1.653s lvm2-pvscan@129:896.service
           1.652s lvm2-pvscan@130:784.service
           1.651s lvm2-pvscan@130:768.service


4>
devices/obtain_device_list_from_udev=0
global/event_activation=0
activation/udev_sync=1

          17.975s dracut-initqueue.service
          10.162s wicked.service
           8.238s lvm2-activation-early.service  <===
           6.955s btrfsmaintenance-refresh.service
           4.444s systemd-udev-settle.service
           1.800s rsyslog.service
           1.768s wickedd.service
           1.751s kbdsettings.service
           1.751s kdump-early.service
           1.602s lvm2-pvscan@135:832.service
           1.601s lvm2-pvscan@135:816.service
           1.601s lvm2-pvscan@135:784.service
           1.600s lvm2-pvscan@134:1008.service
           1.599s dev-system-swap.swap
           1.598s lvm2-pvscan@134:832.service

5>
devices/obtain_device_list_from_udev=0
global/event_activation=1
activation/udev_sync=1

          34.908s dracut-initqueue.service
          25.440s systemd-udev-settle.service
          23.335s lvm2-pvscan@66:832.service  <===
          23.335s lvm2-pvscan@65:976.service
          23.335s lvm2-pvscan@66:784.service
          23.335s lvm2-pvscan@65:816.service
          23.335s lvm2-pvscan@8:976.service
          23.327s lvm2-pvscan@66:864.service
          23.323s lvm2-pvscan@66:848.service
          23.316s lvm2-pvscan@65:800.service

6>
devices/obtain_device_list_from_udev=0
global/event_activation=1
activation/udev_sync=0

          36.222s lvm2-pvscan@134:912.service <===
          36.222s lvm2-pvscan@134:816.service
          36.222s lvm2-pvscan@134:784.service
          36.221s lvm2-pvscan@133:816.service
          36.221s lvm2-pvscan@133:848.service
          36.220s lvm2-pvscan@133:928.service
          36.220s lvm2-pvscan@133:768.service
          36.219s lvm2-pvscan@133:992.service
          36.218s lvm2-pvscan@133:784.service
          36.218s lvm2-pvscan@134:800.service
          36.218s lvm2-pvscan@133:864.service
          36.217s lvm2-pvscan@133:896.service
          36.209s lvm2-pvscan@133:960.service
          36.197s lvm2-pvscan@134:1008.service


7>
devices/obtain_device_list_from_udev=1
global/event_activation=1
activation/udev_sync=1

      2min 6.736s lvm2-pvscan@70:384.service <===
      2min 6.628s lvm2-pvscan@70:400.service
      2min 6.554s lvm2-pvscan@69:432.service
      2min 6.518s lvm2-pvscan@69:480.service
      2min 6.478s lvm2-pvscan@69:416.service
      2min 6.277s lvm2-pvscan@69:464.service
      2min 5.791s lvm2-pvscan@69:544.service


8>
devices/obtain_device_list_from_udev=1
global/event_activation=1
activation/udev_sync=0

     2min 27.091s lvm2-pvscan@129:944.service <===
     2min 26.952s lvm2-pvscan@129:912.service
     2min 26.950s lvm2-pvscan@129:880.service
     2min 26.947s lvm2-pvscan@129:960.service
     2min 26.947s lvm2-pvscan@129:928.service
     2min 26.947s lvm2-pvscan@129:832.service
     2min 26.938s lvm2-pvscan@129:848.service
     2min 26.733s lvm2-pvscan@129:864.service
     2min 16.241s lvm2-pvscan@66:976.service
     2min 15.166s lvm2-pvscan@66:992.service


_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/