[linux-lvm] Unusually long boot times with LVM Snapshots

[linux-lvm] Unusually long boot times with LVM Snapshots

[Sreyan Chakravarty]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

Hi,

I use LVM with full disk encryption. I also use LVM snapshots, currently I
have a 70GB snapshot.

Now the only problem is that the boot times when LVM snapshots are present
are extremely long.

I boot my machine, enter my encryption passphrase and then wait for about 3
minutes. It is only after that long wait does my machine boot.

This only happens if snapshots are present, if no snapshots are present,
boot is almost instantaneous after I enter my passphrase.

Will adding --skip-mappings help ?

Is there any way this can be solved ?

-- 
Regards,
Sreyan Chakravarty

[-- Attachment #2: Type: text/html, Size: 870 bytes --]

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Bryn M. Reeves]

On Thu, Nov 19, 2020 at 07:46:21PM +0530, Sreyan Chakravarty wrote:
> I use LVM with full disk encryption. I also use LVM snapshots, currently I
> have a 70GB snapshot.

What type of snapshot are you using? LVM2 allows either "classic" CoW snaps,
or the newer thin provisioned snapshots using the dm-thinp target.

Classic snapshots are known to have very poor IO performance when multiple
snapshots of the same volume exist simultaneously (especially for write-
heavy workloads).

Thin provisioned snapshots are not normally activated at boot time unless
they are explicitly requested (via dracut's rd.lvm.lv options) since they
have the skip activation flag set by default.

> I boot my machine, enter my encryption passphrase and then wait for about 3
> minutes. It is only after that long wait does my machine boot.
> 
> This only happens if snapshots are present, if no snapshots are present,
> boot is almost instantaneous after I enter my passphrase.

Is there any indication in the log of what's happening during the delay?
Look through the journalctl output to see if there are any messages logged
while the delay happens.

Another option is to use systemd-analyze to look into where the time is
going during boot. It has various commands including "plot" which will
generate an SVG plot of the boot timings on stdout. You can then compare
that with a regular boot to try to understand the difference.

Regards,
Bryn.

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Sreyan Chakravarty]

[-- Attachment #1: Type: text/plain, Size: 2767 bytes --]

On Fri, Nov 20, 2020 at 4:45 PM Bryn M. Reeves <bmr@redhat.com> wrote:

> What type of snapshot are you using? LVM2 allows either "classic" CoW
> snaps,
> or the newer thin provisioned snapshots using the dm-thinp target.
>
> Classic snapshots are known to have very poor IO performance when multiple
> snapshots of the same volume exist simultaneously (especially for write-
> heavy workloads).
>
> Thin provisioned snapshots are not normally activated at boot time unless
> they are explicitly requested (via dracut's rd.lvm.lv options) since they
> have the skip activation flag set by default.
>
>
How can I check which type of snapshot I am using ?

I am very interested in knowing more about these newer snapshots.

I think I am using the older COW snapshots.

I created it using:

              sudo lvcreate -L 70GB -s -n test_snapshot
/dev/mapper/vgfedora-fedora


Is there any indication in the log of what's happening during the delay?
> Look through the journalctl output to see if there are any messages logged
> while the delay happens.
>
>
Yes, I have found the reason for the 3 minute delay:

Nov 20 21:14:15  dracut-initqueue[902]: Scanning devices dm-0  for LVM
logical volumes vgfedora/fedora
Nov 20 21:14:16  dracut-initqueue[927]: inactive Original
'/dev/vgfedora/fedora' [700.00 GiB] inherit
Nov 20 21:14:16  dracut-initqueue[927]: inactive Snapshot
'/dev/vgfedora/pre_kde_Nov_9' [70.00 GiB] inherit
Nov 20 21:17:27  systemd[1]: Found device /dev/mapper/vgfedora-fedora.
Nov 20 21:17:27  systemd[1]: Found device
/dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e.
Nov 20 21:17:27  systemd[1]: Reached target Initrd Root Device.

As you can see it initializing the snapshots.

This is my entire boot log if you need to delve deeper.
https://pastebin.com/raw/275JPvZB


> Another option is to use systemd-analyze to look into where the time is
> going during boot. It has various commands including "plot" which will
> generate an SVG plot of the boot timings on stdout. You can then compare
> that with a regular boot to try to understand the difference.
>


Well I don't know about "plot" but this is the output from "systemd-analyze
blame"

3min 30.737s dracut-initqueue.service

     31.399s udisks2.service

     26.650s lvm2-monitor.service

     25.500s systemd-journal-flush.service

     20.289s ModemManager.service

     18.242s abrtd.service

     18.233s avahi-daemon.service

     18.227s bluetooth.service

     17.725s systemd-cryptsetup@luks\x2d2ec7f1ae\x2d6f9b\x2d4896\x2da7b2\x2dbe7809e9d2f4.service

     17.604s firewalld.service


As you can see the 3min delay matches with the above LVM entry from the
boot log.


Again, I am very interested in knowing more about these newer snapshots.
-- 
Regards,
Sreyan Chakravarty

[-- Attachment #2: Type: text/html, Size: 5016 bytes --]

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Bryn M. Reeves]

On Fri, Nov 20, 2020 at 09:48:10PM +0530, Sreyan Chakravarty wrote:
> How can I check which type of snapshot I am using ?
> 
> I am very interested in knowing more about these newer snapshots.
> 
> I think I am using the older COW snapshots.
> 
> I created it using:
> 
>               sudo lvcreate -L 70GB -s -n test_snapshot
> /dev/mapper/vgfedora-fedora

This is a classic (aka CoW/copy-on-write or "thick") snapshot since it
was created with a size (-L 70GB). When creating a thin snapshot you do
not specify a size for the snapshot LV - it consumes space from the
associated thin pool as needed.
 
Generally you can find information about the LVs on your system using
the "lvs" command. The "Attr" field displays lv attributes including the
volume type - see the NOTES section of 'man lvs' for the meaning of the
values. The lvs report can also show you whether a snapshot is using CoW
or thinp and the amount of data it is consuming.

Regards,
Bryn.

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Sreyan Chakravarty]

[-- Attachment #1: Type: text/plain, Size: 1998 bytes --]

On Mon, Nov 23, 2020 at 6:43 PM Bryn M. Reeves <bmr@redhat.com> wrote:

> This is a classic (aka CoW/copy-on-write or "thick") snapshot since it
> was created with a size (-L 70GB). When creating a thin snapshot you do
> not specify a size for the snapshot LV - it consumes space from the
> associated thin pool as needed.
>
> Generally you can find information about the LVs on your system using
> the "lvs" command. The "Attr" field displays lv attributes including the
> volume type - see the NOTES section of 'man lvs' for the meaning of the
> values. The lvs report can also show you whether a snapshot is using CoW
> or thinp and the amount of data it is consuming.
>

I have now migrated to thin snapshots:

These are my logical volumes:

#  lvs

 LV     VG       Attr       LSize   Pool   Origin Data%  Meta%  Move Log
Cpy%Sync Convert
  fedora vgfedora Vwi-a-tz-- 700.00g pool00        16.61

  pool00 vgfedora twi-aotz-- 929.75g               12.50  16.61



But I am unable to boot now due to these errors:

Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning
devices dm-0  for LVM logical volumes vgfedora/fedora
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/pool00' [929.75 GiB] inherit
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/fedora' [700.00 GiB] inherit
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe:
FATAL: Module dm-thin-pool not found in directory
/lib/modules/5.8.18-200.fc32.x86_64
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]:
/usr/sbin/modprobe failed: 1
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process
LV vgfedora/fedora: thin target support missing from kernel?

I have tried to mark the volumes as active by :

                lvchange -kn vgfedora/pool00
                lvchange -kn vgfedora/fedora

But no change.



How do I add the dm-thin-pool module ? I can't even boot.

-- 
Regards,
Sreyan Chakravarty

[-- Attachment #2: Type: text/html, Size: 2955 bytes --]

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Sreyan Chakravarty]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

On Tue, Nov 24, 2020 at 5:27 PM Sreyan Chakravarty <sreyan32@gmail.com>
wrote:

>
> But I am unable to boot now due to these errors:
>
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning
> devices dm-0  for LVM logical volumes vgfedora/fedora
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
> '/dev/vgfedora/pool00' [929.75 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
> '/dev/vgfedora/fedora' [700.00 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe:
> FATAL: Module dm-thin-pool not found in directory
> /lib/modules/5.8.18-200.fc32.x86_64
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]:
> /usr/sbin/modprobe failed: 1
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process
> LV vgfedora/fedora: thin target support missing from kernel?
>
> I have tried to mark the volumes as active by :
>
>                 lvchange -kn vgfedora/pool00
>                 lvchange -kn vgfedora/fedora
>
> But no change.
>
>
>
> How do I add the dm-thin-pool module ? I can't even boot.
>
>
Never mind, I have solved it by regenerating my initramfs.
-- 
Regards,
Sreyan Chakravarty

[-- Attachment #2: Type: text/html, Size: 1897 bytes --]

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Bryn M. Reeves]

On Tue, Nov 24, 2020 at 05:27:21PM +0530, Sreyan Chakravarty wrote:
> But I am unable to boot now due to these errors:
> 
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning
> devices dm-0  for LVM logical volumes vgfedora/fedora
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
> '/dev/vgfedora/pool00' [929.75 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
> '/dev/vgfedora/fedora' [700.00 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe:
> FATAL: Module dm-thin-pool not found in directory
> /lib/modules/5.8.18-200.fc32.x86_64
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]:
> /usr/sbin/modprobe failed: 1
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process
> LV vgfedora/fedora: thin target support missing from kernel?
> 
> I have tried to mark the volumes as active by :
> 
>                 lvchange -kn vgfedora/pool00
>                 lvchange -kn vgfedora/fedora
> 
> But no change.
> 
> 
> 
> How do I add the dm-thin-pool module ? I can't even boot.

Most likely you are just missing the dm-thin-pool module in the dracut
initramfs image.

You can re-build it with the following command:

  # dracut -f /boot/initramfs-$(uname -r).img $(uname -r)

If the running kernel matches the initramfs you want to rebuild then
you can leave out the file and kernel version:

  # dracut -f

To build an initramfs with only the drivers that the running host
uses add the -H switch (the dm-thin-pool module must be loaded at
the time you run dracut for this to work):

  # dracut -fH

See the dracut man page for more details.

Regards,
Bryn.

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Sreyan Chakravarty]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

On Wed, Nov 25, 2020 at 8:50 PM Bryn M. Reeves <bmr@redhat.com> wrote:

> Most likely you are just missing the dm-thin-pool module in the dracut
> initramfs image.
>
> You can re-build it with the following command:
>
>   # dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
>
> If the running kernel matches the initramfs you want to rebuild then
> you can leave out the file and kernel version:
>
>   # dracut -f
>
> To build an initramfs with only the drivers that the running host
> uses add the -H switch (the dm-thin-pool module must be loaded at
> the time you run dracut for this to work):
>
>   # dracut -fH
>
> See the dracut man page for more details.
>

Yes, I did do the above thanks.
-- 
Regards,
Sreyan Chakravarty

[-- Attachment #2: Type: text/html, Size: 1201 bytes --]

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Sreyan Chakravarty]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

On Fri, Nov 20, 2020 at 4:45 PM Bryn M. Reeves <bmr@redhat.com> wrote:

> What type of snapshot are you using? LVM2 allows either "classic" CoW
> snaps,
> or the newer thin provisioned snapshots using the dm-thinp target.
>
> Classic snapshots are known to have very poor IO performance when multiple
> snapshots of the same volume exist simultaneously (especially for write-
> heavy workloads).
>
> Thin provisioned snapshots are not normally activated at boot time unless
> they are explicitly requested (via dracut's rd.lvm.lv options) since they
> have the skip activation flag set by default.
>

I tried thin snapshots, but what use are they if I can't use my system all
together.

Once my root logical volume was marked read only I could not revert it back
to writable, after the snapshot was created.

-- 
Regards,
Sreyan Chakravarty

[-- Attachment #2: Type: text/html, Size: 1392 bytes --]

Re: [linux-lvm] Unusually long boot times with LVM Snapshots

[Bryn M. Reeves]

On Sat, Nov 21, 2020 at 01:51:42PM +0530, Sreyan Chakravarty wrote:
> On Fri, Nov 20, 2020 at 4:45 PM Bryn M. Reeves <bmr@redhat.com> wrote:
> 
> > What type of snapshot are you using? LVM2 allows either "classic" CoW
> > snaps,
> > or the newer thin provisioned snapshots using the dm-thinp target.
> >
> > Classic snapshots are known to have very poor IO performance when multiple
> > snapshots of the same volume exist simultaneously (especially for write-
> > heavy workloads).
> >
> > Thin provisioned snapshots are not normally activated at boot time unless
> > they are explicitly requested (via dracut's rd.lvm.lv options) since they
> > have the skip activation flag set by default.
> >
> 
> I tried thin snapshots, but what use are they if I can't use my system all
> together.

I'm not following this statement. Can you explain what happened when you
tried thin snapshots?
 
> Once my root logical volume was marked read only I could not revert it back
> to writable, after the snapshot was created.

That's not a normal outcome of creating a snapshot - it sounds like you
ran into some problem with the snaphot or the volume group, but without
more details of the situation and what actually happened it's very hard
to give you guidance to help avoid it in future.

Regards,
Bryn.