Questions about Backup and Restore Strategy for LVM2 on LUKS2 and for LUKS1 on boot partition

Questions about Backup and Restore Strategy for LVM2 on LUKS2 and for LUKS1 on boot partition

[Lamy Geier]

Hello!

This is my first ever post in any mailing list, please be a bit 
forgiving. First of all do I need to subscribe to 
cryptsetup+subscribe@lists.linux.dev to receive a reply? Or replies to 
my email would be delivered to me automatically?

I have read the entire [FrequentlyAskedQuestions · Wiki · cryptsetup / 
cryptsetup · 
GitLab](https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions), 
but I am a beginner and inexperiences and I would be happy to receive 
any suggestions.

## Operating System Installation with LUKS and LVM

I have installed

     - LUKS1 on ext4 `/boot` partition.

     - LVM2 on LUKS2 in a separate partition.

     - In LVM2, I have one physical volume (my NVME SSD), one volume 
group consisting of 3 logical volume all formatted with ext4: `/`, 
`/home`, `swap`.

     - I followed the installation of Ubuntu 22.04 instruction from 
Ubuntu Community Wiki [Full_Disk_Encryption_Howto_2019 - Community Help 
Wiki](https://help.ubuntu.com/community/Full_Disk_Encryption_Howto_2019).

## Backup Startegy

On an external drive,

     - I want to use rsync based [linuxmint/timeshift: System restore 
tool for Linux](https://github.com/linuxmint/timeshift) to backup `/` 
logical volume.

     - Further, I want to use rsync based [bit-team/backintime: Back In 
Time - A simple backup tool for 
Linux](https://github.com/bit-team/backintime) solution to backup my 
`/home` logical volume.

     - I am going to backup the LUKS header of both of the above 
partitions (one of which is `/boot` and the other is the LVM on LUKS 
partition).

## Questions

1. It was mentioned in the FAQ to avoid the LVM if possible, but 
unfortunately I read the FAQ only after installing my operating system. 
Could you please tell me if my backup strategy is enough to mitigate any 
disaster (expect that I am not keeping a third backup as per 3-2-1 
strategy)?

2. What exactly I need to  be careful about while using LVM on LUKS?

3. If the pre-boot authentication fails in case of disaster, I am 
wondering how will I mitigate it. Should I be using a LIVE USB and do 
LUKS1 Header restore for boot partition and then open it using 
`cryptsetup open` and inspect if the data got decrypted? And the same 
process for the LVM on LUKS partition to inspect if the root directory 
is decrypted?

4. Regarding timeshift:

4a. Do you think I would be able to restore my system?

4b. Also, do you think I also need to backup the `/boot` partition with 
timeshift?

4c. I am not exactly sure how the system restore with timeshift work. If 
the LUKS header have been restored as mentioned in step 3, will the 
timeshift be able to restore the system from the external drive after 
the pre-boot authentication. Because the LVM on LUKS partition is 
unlocked during the bootinf and not right after pre boot authentication.

5. To restore  the  home logical volume I can do it after the operating 
system is loaded. I was just not sure when and how will I be able to 
restore the  root  logical volume and the boot partition .

6. Anything else I need to be careful about LVM on LUKS?

Thanks and Regards

Lamy

Re: Questions about Backup and Restore Strategy for LVM2 on LUKS2 and for LUKS1 on boot partition

[Arno Wagner]

Just one comment on LVM:

LVM makes partitiopn set-up easy for distribution installers,
but it makes things much more complicated. LVM is one of these
tools that are incredibly powerful, but you should only use them
if you have a really good reason to, because it also makes
fixing problems incredibly complex. Unfortunately, there are 
a lot of lazy people out there that will gladly accept a simpler
job now for a much, much harder job in case of problems in 
the future. This is going on all over the software world, 
unfortunately, because the process of engineering software
is not mature at all yet.

So, to be clear, LVM will make it basically impossible to 
recover your LUKS partition if anything goes wrong, because
if it is used, you need to not only understand the technical
details of LUKS (relatively sinmple) but also the technical
details of LVM (pretty complex) to fix anything. 

And that is why I recommend not mixing LUKS and LVM and generally
not using LVM unless there is a very good reason to use it. 
More convenience in writing an installer is not even a regular
good reason. Classical partitions work just fine.

That said, if you have good backup, the danger is a lot smaller.

Regards,
Arno



On Thu, Nov 03, 2022 at 21:59:54 CET, Lamy Geier wrote:
> Hello!
> 
> This is my first ever post in any mailing list, please be a bit forgiving.
> First of all do I need to subscribe to cryptsetup+subscribe@lists.linux.dev
> to receive a reply? Or replies to my email would be delivered to me
> automatically?
> 
> I have read the entire [FrequentlyAskedQuestions · Wiki · cryptsetup /
> cryptsetup · GitLab](https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions),
> but I am a beginner and inexperiences and I would be happy to receive any
> suggestions.
> 
> ## Operating System Installation with LUKS and LVM
> 
> I have installed
> 
>     - LUKS1 on ext4 `/boot` partition.
> 
>     - LVM2 on LUKS2 in a separate partition.
> 
>     - In LVM2, I have one physical volume (my NVME SSD), one volume group
> consisting of 3 logical volume all formatted with ext4: `/`, `/home`,
> `swap`.
> 
>     - I followed the installation of Ubuntu 22.04 instruction from Ubuntu
> Community Wiki [Full_Disk_Encryption_Howto_2019 - Community Help
> Wiki](https://help.ubuntu.com/community/Full_Disk_Encryption_Howto_2019).
> 
> ## Backup Startegy
> 
> On an external drive,
> 
>     - I want to use rsync based [linuxmint/timeshift: System restore tool
> for Linux](https://github.com/linuxmint/timeshift) to backup `/` logical
> volume.
> 
>     - Further, I want to use rsync based [bit-team/backintime: Back In Time
> - A simple backup tool for Linux](https://github.com/bit-team/backintime)
> solution to backup my `/home` logical volume.
> 
>     - I am going to backup the LUKS header of both of the above partitions
> (one of which is `/boot` and the other is the LVM on LUKS partition).
> 
> ## Questions
> 
> 1. It was mentioned in the FAQ to avoid the LVM if possible, but
> unfortunately I read the FAQ only after installing my operating system.
> Could you please tell me if my backup strategy is enough to mitigate any
> disaster (expect that I am not keeping a third backup as per 3-2-1
> strategy)?
> 
> 2. What exactly I need to  be careful about while using LVM on LUKS?
> 
> 3. If the pre-boot authentication fails in case of disaster, I am wondering
> how will I mitigate it. Should I be using a LIVE USB and do LUKS1 Header
> restore for boot partition and then open it using `cryptsetup open` and
> inspect if the data got decrypted? And the same process for the LVM on LUKS
> partition to inspect if the root directory is decrypted?
> 
> 4. Regarding timeshift:
> 
> 4a. Do you think I would be able to restore my system?
> 
> 4b. Also, do you think I also need to backup the `/boot` partition with
> timeshift?
> 
> 4c. I am not exactly sure how the system restore with timeshift work. If the
> LUKS header have been restored as mentioned in step 3, will the timeshift be
> able to restore the system from the external drive after the pre-boot
> authentication. Because the LVM on LUKS partition is unlocked during the
> bootinf and not right after pre boot authentication.
> 
> 5. To restore  the  home logical volume I can do it after the operating
> system is loaded. I was just not sure when and how will I be able to restore
> the  root  logical volume and the boot partition .
> 
> 6. Anything else I need to be careful about LVM on LUKS?
> 
> Thanks and Regards
> 
> Lamy
> 

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier

Re: Questions about Backup and Restore Strategy for LVM2 on LUKS2 and for LUKS1 on boot partition

[Milan Broz]

On 11/4/22 08:39, Arno Wagner wrote:
> Just one comment on LVM:
> 
> LVM makes partitiopn set-up easy for distribution installers,
> but it makes things much more complicated. LVM is one of these
> tools that are incredibly powerful, but you should only use them
> if you have a really good reason to, because it also makes
> fixing problems incredibly complex. Unfortunately, there are
> a lot of lazy people out there that will gladly accept a simpler
> job now for a much, much harder job in case of problems in
> the future. This is going on all over the software world,
> unfortunately, because the process of engineering software
> is not mature at all yet.
> 
> So, to be clear, LVM will make it basically impossible to
> recover your LUKS partition if anything goes wrong, because
> if it is used, you need to not only understand the technical
> details of LUKS (relatively sinmple) but also the technical
> details of LVM (pretty complex) to fix anything.
> 
> And that is why I recommend not mixing LUKS and LVM and generally
> not using LVM unless there is a very good reason to use it.
> More convenience in writing an installer is not even a regular
> good reason. Classical partitions work just fine.

While I do not like some LVM concepts, it works very well in simple
scenarios and recovery is sometimes even easier than on regular partitions
(there are metadata backups, revert/restore is possible almost always
if data are still on-disk - not discarded by TRIM command or so).

Many distributions use LUKS/LVM combination by default.

For example, if you need to add space to a partition, resizing
can be really complicated if there is no free space directly next to it.
(Here I would say it is the parted tool that is overengineered and dangerous.)

With LVM you can simple add another segment anywhere on disk to logical volume
with much simpler and safer way (it will even resize fs, if you tell it to do).

I think we should focus how to help with these simple scenarios
(as documentation is often obsolete) than to just keep suggesting that
LVM is useless and recovery is complicated (it really is not if it is just
simple linear mapped volume).

Milan

Re: Questions about Backup and Restore Strategy for LVM2 on LUKS2 and for LUKS1 on boot partition

[Michael Kjörling]

On 3 Nov 2022 21:59 +0100, from lamyergeier@gmail.com (Lamy Geier):
> This is my first ever post in any mailing list, please be a bit forgiving.
> First of all do I need to subscribe to cryptsetup+subscribe@lists.linux.dev
> to receive a reply? Or replies to my email would be delivered to me
> automatically?

To receive replies that are sent only through the list, you need to
subscribe to the list. Since this list is publicly archived, you can
also check the archives at <https://lore.kernel.org/cryptsetup/>.


> I have installed
> 
>     - LUKS1 on ext4 `/boot` partition.
> 
>     - LVM2 on LUKS2 in a separate partition.
> 
>     - In LVM2, I have one physical volume (my NVME SSD), one volume group
> consisting of 3 logical volume all formatted with ext4: `/`, `/home`,
> `swap`.

Minor detail perhaps, but the order of nesting matters for clarity.
I'm assuming that the logical layout is something much like:

    storage device
        partition 1
            LUKS1
                ext4 /boot
        partition 2
            LUKS2
                LVM2
                    single-PV VG
                        LV
                            ext4 /
                        LV
                            ext4 /home
                        LV
                            swap


> ## Backup Startegy
> 
> On an external drive,
> 
>     - I want to use rsync based [linuxmint/timeshift: System restore tool
> for Linux](https://github.com/linuxmint/timeshift) to backup `/` logical
> volume.
> 
>     - Further, I want to use rsync based [bit-team/backintime: Back In Time
> - A simple backup tool for Linux](https://github.com/bit-team/backintime)
> solution to backup my `/home` logical volume.
> 
>     - I am going to backup the LUKS header of both of the above partitions
> (one of which is `/boot` and the other is the LVM on LUKS partition).
> 
> ## Questions
> 
> 1. It was mentioned in the FAQ to avoid the LVM if possible, but
> unfortunately I read the FAQ only after installing my operating system.
> Could you please tell me if my backup strategy is enough to mitigate any
> disaster (expect that I am not keeping a third backup as per 3-2-1
> strategy)?

Generally, exactly how you lay out your partitions doesn't make much
difference to how you back them up. That said, LVM _does_ add a fair
bit of complexity here.

Since it sounds like you have fairly recently installed your system, I
would suggest at least considering redoing the installation without
LVM but instead use just plain partitions. Having a separate /home is
convenient if you ever want to completely wipe your system and
reinstall from scratch but keep the user data, but for a single-user
system, you can definitely get away with keeping /home within the root
file system. My preferred setup would be more along the lines of three
partitions, each with a LUKS container; within the first two, /boot
and / respectively; within the last, swap (with a random key). Swap on
Linux has historically been to a dedicated partition, but I understand
that swap files are becoming increasingly common.


> 3. If the pre-boot authentication fails in case of disaster, I am wondering
> how will I mitigate it. Should I be using a LIVE USB and do LUKS1 Header
> restore for boot partition and then open it using `cryptsetup open` and
> inspect if the data got decrypted? And the same process for the LVM on LUKS
> partition to inspect if the root directory is decrypted?

If you use LUKS, then opening the container will either succeed or
fail. (If you use plain dm-crypt, there's no way to tell except by
examining the decrypted data, such as by trying to mount the file
system through the dm-crypt mapping and see if that works.) If you
have a LUKS header backup, you can give that directly to `cryptsetup
open` to bypass the on-disk header completely if the header has been
corrupted. Note that you need to guard the header backup carefully,
since an adversary who gets hold of a copy of the header can do an
offline attack on your passphrase. The latter can be mitigated by
using a longer passphrase or a higher iteration count; since GRUB's
password hashing seems to not be as optimized as that of cryptsetup,
the former may be preferable.


> 4. Regarding timeshift:
> 
> 4a. Do you think I would be able to restore my system?
> 
> 4b. Also, do you think I also need to backup the `/boot` partition with
> timeshift?

I'm not familiar with either timeshift or backintime, so I can't
really comment on either of those or on how you would do a bare-metal
restore based on what either has done. However, for backups, my advice
would be: keep it simple!

Personally, I use rsnapshot, which is simply a convenient front-end to
rsync for keeping multiple backups in such a way that only the
difference between the backups need to be stored separately (the rest
is hardlinked); you can do that with rsync and a few standard tools
directly, but rsnapshot makes it more convenient. Restoring would
involve simply copying the most recent valid copy back onto freshly
partitioned media, maybe updating some file system references in
boot-relevant configuration files such as /etc/fstab and
/etc/crypttab, and reinstalling the boot loader. I have also written
two helper scripts around it, one to perform a backup onto an external
drive (itself holding a LUKS container) which is normally kept soft
powered down along with a few other housekeeping tasks, and one to
purge old backups when it starts filling up. Neither of the those
scripts is strictly _necessary_; they are just tools of convenience.


> 5. To restore  the  home logical volume I can do it after the operating
> system is loaded. I was just not sure when and how will I be able to restore
> the  root  logical volume and the boot partition .

The easiest way to do that would probably be to use some live media to
partition and format new media as needed, in your case set up a LVM
hierarchy identical to the old storage, mount the file systems, and
copy everything. Since identifiers such as /dev/disk/by-id/* will
likely change, you will also need to update those and, of course,
reinstall the boot loader.

-- 
🪶 Michael Kjörling                  🏡 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”