Re: [linux-lvm] Discussion: performance issue on event activation mode

[David Teigland <teigland@redhat.com>]

On Tue, Sep 28, 2021 at 06:34:06AM +0000, Martin Wilck wrote:
> Hello David and Peter,
> 
> On Mon, 2021-09-27 at 10:38 -0500, David Teigland wrote:
> > On Mon, Sep 27, 2021 at 12:00:32PM +0200, Peter Rajnoha wrote:
> > > > - We could use the new lvm-activate-* services to replace the
> > > > activation
> > > > generator when lvm.conf event_activation=0.  This would be done by
> > > > simply
> > > > not creating the event-activation-on file when event_activation=0.
> > > 
> > > ...the issue I see here is around the systemd-udev-settle:
> > 
> > Thanks, I have a couple questions about the udev-settle to understand
> > that
> > better, although it seems we may not need it.
> > 
> > >   - the setup where lvm-activate-vgs*.service are always there (not
> > >     generated only on event_activation=0 as it was before with the
> > >     original lvm2-activation-*.service) practically means we always
> > >     make a dependency on systemd-udev-settle.service, which we
> > > shouldn't
> > >     do in case we have event_activation=1.
> > 
> > Why wouldn't the event_activation=1 case want a dependency on udev-
> > settle?
> 
> You said it should wait for multipathd, which in turn waits for udev
> settle. And indeed it makes some sense. After all: the idea was to
> avoid locking issues or general resource starvation during uevent
> storms, which typically occur in the coldplug phase, and for which the
> completion of "udev settle" is the best available indicator.

Hi Martin, thanks, you have some interesting details here.

Right, the idea is for lvm-activate-vgs-last to wait for other services
like multipath (or anything else that a PV would typically sit on), so
that it will be able to activate as many VGs as it can that are present at
startup.  And we avoid responding to individual coldplug events for PVs,
saving time/effort/etc.

> I'm arguing against it (perhaps you want to join in :-), but odds are
> that it'll disappear sooner or later. Fot the time being, I don't see a
> good alternative.

multipath has more complex udev dependencies, I'll be interested to see
how you manage to reduce those, since I've been reducing/isolating our
udev usage also.

> The dependency type you have to use depends on what you need. Do you
> really only depend on udev settle because of multipathd? I don't think
> so; even without multipath, thousands of PVs being probed
> simultaneously can bring the performance of parallel pvscans down. That
> was the original motivation for this discussion, after all. If this is
> so, you should use both "Wants" and "After". Otherwise, using only
> "After" might be sufficient.

I don't think we really need the settle.  If device nodes for PVs are
present, then vgchange -aay from lvm-activate-vgs* will see them and
activate VGs from them, regardless of what udev has or hasn't done with
them yet.

> > - Reading the udev db: with the default
> > external_device_info_source=none
> > we no longer ask the udev db for any info about devs.  (We now
> > follow that setting strictly, and only ask udev when source=udev.)
> 
> This is a different discussion, but if you don't ask udev, how do you
> determine (reliably, and consistently with other services) whether a
> given device will be part of a multipath device or a MD Raid member?

Firstly, with the new devices file, only the actual md/mpath device will
be in the devices file, the components will not be, so lvm will never
attempt to look at an md or mpath component device.

Otherwise, when the devices file is not used,
md: from reading the md headers from the disk
mpath: from reading sysfs links and /etc/multipath/wwids

> In the past, there were issues with either pvscan or blkid (or
> multipath) failing to open a device while another process had opened it
> exclusively. I've never understood all the subtleties. See systemd
> commit 3ebdb81 ("udev: serialize/synchronize block device event
> handling with file locks").

Those locks look like a fine solution if a problem comes up like that.
I suspect the old issues may have been caused by a program using an
exclusive open when it shouldn't.

> After=udev-settle will make sure that you're past a coldplug uevent
> storm during boot. IMO this is the most important part of the equation.
> I'd be happy to find a solution for this that doesn't rely on udev
> settle, but I don't see any.

I don't think multipathd is listening to uevents directly?  If it were,
you might use a heuristic to detect a change in uevents (e.g. the volume)
and conclude coldplug is finished.

Dave

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/