From: Tom Yan <tom.ty89@gmail.com>
To: LVM general discussion and development <linux-lvm@redhat.com>
Cc: "zkabelac@redhat.com" <zkabelac@redhat.com>,
David Teigland <teigland@redhat.com>,
Martin Wilck <martin.wilck@suse.com>
Subject: Re: [linux-lvm] Discussion: performance issue on event activation mode
Date: Thu, 8 Jul 2021 18:10:56 +0800 [thread overview]
Message-ID: <CAGnHSEnOUfYSs8UiOnTYFRG2Xv9EsF4fpyq+OspgUe-d-EkrXw@mail.gmail.com> (raw)
In-Reply-To: <50b3e710-a6b7-2cc5-04fc-63525f0d720c@suse.com>
I wonder if using a shell wrapper to detect whether pvscan actually
exits way earlier than the unit does is a more accurate approach.
On Sat, 3 Jul 2021 at 19:50, heming.zhao@suse.com <heming.zhao@suse.com> wrote:
>
> On 7/3/21 6:02 AM, David Teigland wrote:
> > On Fri, Jul 02, 2021 at 09:22:03PM +0000, Martin Wilck wrote:
> >> On Fr, 2021-07-02 at 16:09 -0500, David Teigland wrote:
> >>> On Sun, Jun 06, 2021 at 02:15:23PM +0800, heming.zhao@suse.com wrote:
> >>>> dev_cache_scan //order: O(n^2)
> >>>> + _insert_dirs //O(n)
> >>>> | if obtain_device_list_from_udev() true
> >>>> | _insert_udev_dir //O(n)
> >>>> |
> >>>> + dev_cache_index_devs //O(n)
> >>>
> >>> I've been running some experiments and trying some patches to improve
> >>> this. By setting obtain_device_list_from_udev=0, and using the
> >>> attached
> >>> patch to disable dev_cache_index_devs, the pvscan is much better.
> >>>
> >>> systemctl status lvm2-pvscan appears to show that the pvscan command
> >>> itself runs for only 2-4 seconds, while the service as a whole takes
> >>> around 15 seconds. See the 16 sec gap below from the end of pvscan
> >>> to the systemd Started message. If that's accurate, the remaining
> >>> delay
> >>> would lie outside lvm.
> >>>
> >>> Jul 02 15:27:57 localhost.localdomain systemd[1]: Starting LVM event
> >>> activation on device 253:1710...
> >>> Jul 02 15:28:00 localhost.localdomain lvm[65620]: pvscan[65620] PV
> >>> /dev/mapper/mpathalz online, VG 1ed02c7d-0019-43c4-91b5-f220f3521ba9
> >>> is complete.
> >>> Jul 02 15:28:00 localhost.localdomain lvm[65620]: pvscan[65620] VG
> >>> 1ed02c7d-0019-43c4-91b5-f220f3521ba9 run autoactivation.
> >>> Jul 02 15:28:00 localhost.localdomain lvm[65620]: 1 logical
> >>> volume(s) in volume group "1ed02c7d-0019-43c4-91b5-f220f3521ba9" now
> >>> active
> >>
> >> Printing this message is really the last thing that pvscan does?
> >
> > I've not seen anything measurable after that message. However, digging
> > through the command init/exit paths I did find libudev setup/destroy calls
> > that can also be skipped when the command is not accessing libudev info.
> > A quick check seemed to show some further improvement from dropping those.
> > (That will be part of the larger patch isolating libudev usage.)
> >
> > I'm still seeing significant delay between the apparent command exit and
> > the systemd "Started" message, but this will require a little more work to
> > prove.
> >
>
> I applied the patch and got the very similar result.
>
>
> ```with patch
> # systemd-analyze blame
> 27.679s lvm2-pvscan@134:896.service
> 27.664s lvm2-pvscan@134:800.service
> 27.648s lvm2-pvscan@134:960.service
> 27.645s lvm2-pvscan@132:816.service
> 27.589s lvm2-pvscan@134:816.service
> 27.582s lvm2-pvscan@133:992.service
> ... ...
>
> # systemctl status lvm2-pvscan@134:896.service
> ... ...
> Jul 03 19:43:02 sle15sp2-base40g systemd[1]: Starting LVM event activation on device 134:896...
> Jul 03 19:43:03 sle15sp2-base40g lvm[24817]: pvscan[24817] PV /dev/sdalm online, VG vg_sdalm is complet>
> Jul 03 19:43:03 sle15sp2-base40g lvm[24817]: pvscan[24817] VG vg_sdalm run autoactivation.
> Jul 03 19:43:03 sle15sp2-base40g lvm[24817]: 1 logical volume(s) in volume group "vg_sdalm" now active
> Jul 03 19:43:30 sle15sp2-base40g systemd[1]: Started LVM event activation on device 134:896.
> ```
>
> the 27.679s get from 19:43:30 minus 19:43:02.
>
> and 27s is 10s quick than the lvm without patch (by today's test result)
> ```without patch
> # systemd-analyze blame
> 37.650s lvm2-pvscan@133:992.service
> 37.650s lvm2-pvscan@133:1008.service
> 37.649s lvm2-pvscan@133:896.service
> 37.649s lvm2-pvscan@134:960.service
> 37.612s lvm2-pvscan@133:880.service
> 37.612s lvm2-pvscan@133:864.servic
> ... ...
>
> # systemctl status lvm2-pvscan@133:992.service
> ... ...
> Jul 03 19:31:28 sle15sp2-base40g systemd[1]: Starting LVM event activation on device 133:992...
> Jul 03 19:31:30 sle15sp2-base40g lvm[24243]: pvscan[24243] PV /dev/sdalc online, VG vg_sdalc is complet>
> Jul 03 19:31:30 sle15sp2-base40g lvm[24243]: pvscan[24243] VG vg_sdalc run autoactivation.
> Jul 03 19:31:30 sle15sp2-base40g lvm[24243]: 1 logical volume(s) in volume group "vg_sdalc" now active
> Jul 03 19:32:05 sle15sp2-base40g systemd[1]: Started LVM event activation on device 133:992.
> ```
>
> I added log in lvm2 code, the time of log "1 logical volume(s) in volume
> group "vg_sdalm" now active" is the real pvscan finished time. Martin
> comment in previous mail is may right, systemd is may busy and delayed to
> detect/see the pvscan service completion.
>
> Thanks,
> heming
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
next prev parent reply other threads:[~2021-07-08 10:11 UTC|newest]
Thread overview: 86+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-06 6:15 [linux-lvm] Discussion: performance issue on event activation mode heming.zhao
2021-06-06 16:35 ` Roger Heflin
2021-06-07 10:27 ` Martin Wilck
2021-06-07 15:30 ` heming.zhao
2021-06-07 15:45 ` Martin Wilck
2021-06-07 20:52 ` Roger Heflin
2021-06-07 21:30 ` David Teigland
2021-06-08 8:26 ` Martin Wilck
2021-06-08 15:39 ` David Teigland
2021-06-08 15:47 ` Martin Wilck
2021-06-08 16:02 ` Zdenek Kabelac
2021-06-08 16:05 ` Martin Wilck
2021-06-08 16:03 ` David Teigland
2021-06-08 16:07 ` Martin Wilck
2021-06-15 17:03 ` David Teigland
2021-06-15 18:21 ` Zdenek Kabelac
2021-06-16 16:18 ` heming.zhao
2021-06-16 16:38 ` David Teigland
2021-06-17 3:46 ` heming.zhao
2021-06-17 15:27 ` David Teigland
2021-06-08 16:49 ` heming.zhao
2021-06-08 16:18 ` heming.zhao
2021-06-09 4:01 ` heming.zhao
2021-06-09 5:37 ` Heming Zhao
2021-06-09 18:59 ` David Teigland
2021-06-10 17:23 ` heming.zhao
2021-06-07 15:48 ` Martin Wilck
2021-06-07 16:31 ` Zdenek Kabelac
2021-06-07 21:48 ` David Teigland
2021-06-08 12:29 ` Peter Rajnoha
2021-06-08 13:23 ` Martin Wilck
2021-06-08 13:41 ` Peter Rajnoha
2021-06-08 13:46 ` Zdenek Kabelac
2021-06-08 13:56 ` Peter Rajnoha
2021-06-08 14:23 ` Zdenek Kabelac
2021-06-08 14:48 ` Martin Wilck
2021-06-08 15:19 ` Peter Rajnoha
2021-06-08 15:39 ` Martin Wilck
2021-09-09 19:44 ` David Teigland
2021-09-10 17:38 ` Martin Wilck
2021-09-12 16:51 ` heming.zhao
2021-09-27 10:00 ` Peter Rajnoha
2021-09-27 15:38 ` David Teigland
2021-09-28 6:34 ` Martin Wilck
2021-09-28 14:42 ` David Teigland
2021-09-28 15:16 ` Martin Wilck
2021-09-28 15:31 ` Martin Wilck
2021-09-28 15:56 ` David Teigland
2021-09-28 18:03 ` Benjamin Marzinski
2021-09-28 17:42 ` Benjamin Marzinski
2021-09-28 19:15 ` Martin Wilck
2021-09-29 22:06 ` Peter Rajnoha
2021-09-30 7:51 ` Martin Wilck
2021-09-30 8:07 ` heming.zhao
2021-09-30 9:31 ` Martin Wilck
2021-09-30 11:41 ` Peter Rajnoha
2021-09-30 15:32 ` heming.zhao
2021-10-01 7:41 ` Martin Wilck
2021-10-01 8:08 ` Peter Rajnoha
2021-09-30 11:29 ` Peter Rajnoha
2021-09-30 16:04 ` David Teigland
2021-09-30 14:41 ` Benjamin Marzinski
2021-10-01 7:42 ` Martin Wilck
2021-09-29 21:53 ` Peter Rajnoha
2021-09-30 7:45 ` Martin Wilck
2021-09-29 21:39 ` Peter Rajnoha
2021-09-30 7:22 ` Martin Wilck
2021-09-30 14:26 ` David Teigland
2021-09-30 15:55 ` David Teigland
2021-10-01 8:00 ` Peter Rajnoha
2021-10-18 6:24 ` Martin Wilck
2021-10-18 15:04 ` David Teigland
2021-10-18 16:56 ` heming.zhao
2021-10-18 21:51 ` Zdenek Kabelac
2021-10-19 17:18 ` David Teigland
2021-10-20 14:40 ` Martin Wilck
2021-10-20 14:50 ` David Teigland
2021-10-20 14:54 ` Martin Wilck
2021-10-20 15:12 ` David Teigland
2021-06-07 16:40 ` David Teigland
2021-07-02 21:09 ` David Teigland
2021-07-02 21:22 ` Martin Wilck
2021-07-02 22:02 ` David Teigland
2021-07-03 11:49 ` heming.zhao
2021-07-08 10:10 ` Tom Yan [this message]
2021-07-02 21:31 ` Tom Yan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAGnHSEnOUfYSs8UiOnTYFRG2Xv9EsF4fpyq+OspgUe-d-EkrXw@mail.gmail.com \
--to=tom.ty89@gmail.com \
--cc=linux-lvm@redhat.com \
--cc=martin.wilck@suse.com \
--cc=teigland@redhat.com \
--cc=zkabelac@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).