[dm-crypt] Kdump with full-disk LUKS encryption

[dm-crypt] Kdump with full-disk LUKS encryption

[Kairui Song]

Hi all,

I'm currently trying to add kdump support for systemd with full-disk
LUKS encryption. vmcores contain sensitive data so they should also be
protected, and network dumps sometimes are not available. So kdump has
to open the LUKS encrypted device in the kdump environment.

I'm using systemd/dracut, my work machine is running Fedora 34, and
there are several problems I'm trying to solve:
1. Users have to input the password in the kdump kernel environment.
But users often don't have shell access to the kdump environment.
(headless server, graphic card not working after kexec, both are very
common)
2. LUKS2 prefers Argon2 as the key derivation function, designed to
use a lot of memory. kdump is expected to use a minimal amount of
memory. Users will have to reserve a huge amount of memory for kdump
to work (eg. 1G reserve for kdump with 4G total memory which is not
reasonable).

To fix these problems, I tried to pass the master key to the second
kernel directly via initramfs. Kdump will modify the initramfs in
ramfs to include the key, kexec_load it, and never write to any actual
back storage. This worked with old LUKS configurations.

But LUKS2/cryptsetup now stores the key in the kernel keyring by
default. The key is accessible from userspace.

Users can enter the password to start kdump manually and then it will
work, but usually people expect kdump service to start automatically.

(WIP patch series:
https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/RTYJEQ4BV25JYVYJHTTPOK476FUEJOWW/)

I've several ideas about how to improve it but not sure which one is
better, and if this is a good idea after all:
1. Simply introduce a config to let systemd-cryptsetup disable kernel
keyring on setup, there is currently no such option.

2. If we can let the key stay in userspace for a little longer, eg.
for systems booted with dracut/systemd, when
systemd-cryptsetup@%s.service opens the crypt device, keep the key in
dm-crypt. And later when services like kdump have finished loading,
cryptsetup can refresh the device and store the key in the kernel
keyring again.

3. Let kdump use some custom helper/service to load all needed
resources in the early initrd boot stage, prior to
systemd-cryptsetup@%s.service. It will ask the password / parse the
keyfile and load kdump, then provide info for systemd-cryptsetup or
just do the setup. Or maybe let systemd-cryptsetup support some kind
of "plugins" so other tools can use it.

4. A better and safer solution seems to keep a consistent key ring
between kexec boots but also more complex and difficult as different
arch implements kexec differently.

Any suggestions are welcomed!

--
Best Regards,
Kairui Song

_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Antw: [EXT] [systemd-devel] Kdump with full-disk LUKS encryption

[Ulrich Windl]

>>> Kairui Song <kasong@redhat.com> schrieb am 19.04.2021 um 12:00 in
Nachricht
<CACPcB9e0=KYNc_-Bz5EnoHntKKXpurmXzu4e60J1sADQkizvsg@mail.gmail.com>:
> Hi all,
> 
> I'm currently trying to add kdump support for systemd with full‑disk
> LUKS encryption. vmcores contain sensitive data so they should also be
> protected, and network dumps sometimes are not available. So kdump has
> to open the LUKS encrypted device in the kdump environment.
> 
> I'm using systemd/dracut, my work machine is running Fedora 34, and
> there are several problems I'm trying to solve:
> 1. Users have to input the password in the kdump kernel environment.
> But users often don't have shell access to the kdump environment.
> (headless server, graphic card not working after kexec, both are very
> common)
> 2. LUKS2 prefers Argon2 as the key derivation function, designed to
> use a lot of memory. kdump is expected to use a minimal amount of
> memory. Users will have to reserve a huge amount of memory for kdump
> to work (eg. 1G reserve for kdump with 4G total memory which is not
> reasonable).

I'm not a LUKS specialist, but can't you use different KDFs in a different key
slot?
That may weaken the security of course.

> 
> To fix these problems, I tried to pass the master key to the second
> kernel directly via initramfs. Kdump will modify the initramfs in
> ramfs to include the key, kexec_load it, and never write to any actual
> back storage. This worked with old LUKS configurations.
> 
> But LUKS2/cryptsetup now stores the key in the kernel keyring by
> default. The key is accessible from userspace.
> 
> Users can enter the password to start kdump manually and then it will
> work, but usually people expect kdump service to start automatically.
> 
> (WIP patch series:
> https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/

> thread/RTYJEQ4BV25JYVYJHTTPOK476FUEJOWW/)
> 
> I've several ideas about how to improve it but not sure which one is
> better, and if this is a good idea after all:
> 1. Simply introduce a config to let systemd‑cryptsetup disable kernel
> keyring on setup, there is currently no such option.
> 
> 2. If we can let the key stay in userspace for a little longer, eg.
> for systems booted with dracut/systemd, when
> systemd‑cryptsetup@%s.service opens the crypt device, keep the key in
> dm‑crypt. And later when services like kdump have finished loading,
> cryptsetup can refresh the device and store the key in the kernel
> keyring again.
> 
> 3. Let kdump use some custom helper/service to load all needed
> resources in the early initrd boot stage, prior to
> systemd‑cryptsetup@%s.service. It will ask the password / parse the
> keyfile and load kdump, then provide info for systemd‑cryptsetup or
> just do the setup. Or maybe let systemd‑cryptsetup support some kind
> of "plugins" so other tools can use it.
> 
> 4. A better and safer solution seems to keep a consistent key ring
> between kexec boots but also more complex and difficult as different
> arch implements kexec differently.
> 
> Any suggestions are welcomed!
> 
> ‑‑
> Best Regards,
> Kairui Song
> 
> _______________________________________________
> systemd‑devel mailing list
> systemd‑devel@lists.freedesktop.org 
> https://lists.freedesktop.org/mailman/listinfo/systemd‑devel 



_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Kdump with full-disk LUKS encryption

[Milan Broz]

Hi,

TL;DR what you are trying to do is to actually reverse many security measures
we added. It is perhaps acceptable for debugging but hardly for real generic system.

- using memory-hard function increases cost of dictionary and brute-force 
attacks
You can always decrease amount of memory needed, but you should do it only
if you know that security margin is ok (like password is randomly generated
with enough entropy).

- key is in keyring to remove possibility for normal userspace to receive
the key from kernel. Moreover, there is no need to retain kernel in keyring once
dm-crypt device is activated. (It is still in kernel memory but only in crypto
functions context). (Systemd also uses keyring to cache passphrase but that's
different thing.)

You can still use old way for activation with --disable-keyring activation,
but then you disable this possibility.

More comments below.

On 19/04/2021 12:00, Kairui Song wrote:
> Hi all,
> 
> I'm currently trying to add kdump support for systemd with full-disk
> LUKS encryption. vmcores contain sensitive data so they should also be
> protected, and network dumps sometimes are not available. So kdump has
> to open the LUKS encrypted device in the kdump environment.
> 
> I'm using systemd/dracut, my work machine is running Fedora 34, and
> there are several problems I'm trying to solve:
> 1. Users have to input the password in the kdump kernel environment.
> But users often don't have shell access to the kdump environment.
> (headless server, graphic card not working after kexec, both are very
> common)
> 2. LUKS2 prefers Argon2 as the key derivation function, designed to
> use a lot of memory. kdump is expected to use a minimal amount of
> memory. Users will have to reserve a huge amount of memory for kdump
> to work (eg. 1G reserve for kdump with 4G total memory which is not
> reasonable).

When I added Argon2 to LUKS2, I actually expected such issues. Despite
some people beats me that they cannot use arbitrary amount of memory,
we have some hard limits that were selected that it should work on most recent
systems. Maybe kdump can live with it.

 - maximum memory cost limit is 4GB, no LUKS2 device can use more for Argon2
 - we never use more than half of available physical memory
   (measured on the host where the device was formatted)
 - required amount of memory is visible in LUKS2 metadata (luksDump)
   for the particular keyslot  (Memory: the value is in kB)
 - we use benchmark to calculate memory cost with prefered unlocking
   time 2 seconds (again, on the device where LUKS was formatted)
   Small systems (like RPi2) the uses much smaller acceptable values.
  You can configure all costs (time, memory, threads) during format
  or even set them to predefined values.

I am sorry, but there is really no way around this - and the requeired
memory must be physical memory (otherwise it slows down extremely).
This is a feature, not a bug :-)


> To fix these problems, I tried to pass the master key to the second
> kernel directly via initramfs. Kdump will modify the initramfs in
> ramfs to include the key, kexec_load it, and never write to any actual
> back storage. This worked with old LUKS configurations.

Well, passing volume key this way is quite insecure, but perhaps
acceptable for debugging.

> 
> But LUKS2/cryptsetup now stores the key in the kernel keyring by
> default. The key is accessible from userspace.

If you are talking about volume key (not passsphrase), it is not
available from userspace. Only reference to it. But you can use
this reference to construct in-kernel dm-crypt device.
Please read https://gitlab.com/cryptsetup/cryptsetup/-/blob/master/docs/Keyring.txt
 
> Users can enter the password to start kdump manually and then it will
> work, but usually people expect kdump service to start automatically.
> 
> (WIP patch series:
> https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/RTYJEQ4BV25JYVYJHTTPOK476FUEJOWW/)
> 
> I've several ideas about how to improve it but not sure which one is
> better, and if this is a good idea after all:
> 1. Simply introduce a config to let systemd-cryptsetup disable kernel
> keyring on setup, there is currently no such option.

Well, that option could be useful anyway and we have support for it
in cryptsetup (--disable-keyring CLI option) and libcryptsetup, so why not.
Just this should not be a default option.

This is should be patch for systemd-cryptsetup only as libcryptsetup supports it.

...

> 2. If we can let the key stay in userspace for a little longer, eg.
> for systems booted with dracut/systemd, when
> systemd-cryptsetup@%s.service opens the crypt device, keep the key in
> dm-crypt. And later when services like kdump have finished loading,
> cryptsetup can refresh the device and store the key in the kernel
> keyring again.

We invalidate volume key in keyring after libceyposetup  operation
is finished (and kernel removes the reference once keyring garbage collection
is run).

I can imagine to add some option to keep key inside keyring even after
call is finished, but as said above, this removes some security margin
we intentionally introduced here.

...

Milan
_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Kdump with full-disk LUKS encryption

[Kairui Song]

Hi,

Thanks a lot, these info are very helpful.
Better to keep it for debugging for now, and ask users to use it very carefully.

On Tue, Apr 20, 2021 at 3:54 PM Milan Broz <gmazyland@gmail.com> wrote:
>
> Hi,
>
> TL;DR what you are trying to do is to actually reverse many security measures
> we added. It is perhaps acceptable for debugging but hardly for real generic system.
>
> - using memory-hard function increases cost of dictionary and brute-force
> attacks
> You can always decrease amount of memory needed, but you should do it only
> if you know that security margin is ok (like password is randomly generated
> with enough entropy).
>
> - key is in keyring to remove possibility for normal userspace to receive
> the key from kernel. Moreover, there is no need to retain kernel in keyring once
> dm-crypt device is activated. (It is still in kernel memory but only in crypto
> functions context). (Systemd also uses keyring to cache passphrase but that's
> different thing.)
>
> You can still use old way for activation with --disable-keyring activation,
> but then you disable this possibility.
>
> More comments below.
>
> On 19/04/2021 12:00, Kairui Song wrote:
> > Hi all,
> >
> > I'm currently trying to add kdump support for systemd with full-disk
> > LUKS encryption. vmcores contain sensitive data so they should also be
> > protected, and network dumps sometimes are not available. So kdump has
> > to open the LUKS encrypted device in the kdump environment.
> >
> > I'm using systemd/dracut, my work machine is running Fedora 34, and
> > there are several problems I'm trying to solve:
> > 1. Users have to input the password in the kdump kernel environment.
> > But users often don't have shell access to the kdump environment.
> > (headless server, graphic card not working after kexec, both are very
> > common)
> > 2. LUKS2 prefers Argon2 as the key derivation function, designed to
> > use a lot of memory. kdump is expected to use a minimal amount of
> > memory. Users will have to reserve a huge amount of memory for kdump
> > to work (eg. 1G reserve for kdump with 4G total memory which is not
> > reasonable).
>
> When I added Argon2 to LUKS2, I actually expected such issues. Despite
> some people beats me that they cannot use arbitrary amount of memory,
> we have some hard limits that were selected that it should work on most recent
> systems. Maybe kdump can live with it.
>
>  - maximum memory cost limit is 4GB, no LUKS2 device can use more for Argon2
>  - we never use more than half of available physical memory
>    (measured on the host where the device was formatted)
>  - required amount of memory is visible in LUKS2 metadata (luksDump)
>    for the particular keyslot  (Memory: the value is in kB)
>  - we use benchmark to calculate memory cost with prefered unlocking
>    time 2 seconds (again, on the device where LUKS was formatted)
>    Small systems (like RPi2) the uses much smaller acceptable values.
>   You can configure all costs (time, memory, threads) during format
>   or even set them to predefined values.
>
> I am sorry, but there is really no way around this - and the requeired
> memory must be physical memory (otherwise it slows down extremely).
> This is a feature, not a bug :-)
>
>
> > To fix these problems, I tried to pass the master key to the second
> > kernel directly via initramfs. Kdump will modify the initramfs in
> > ramfs to include the key, kexec_load it, and never write to any actual
> > back storage. This worked with old LUKS configurations.
>
> Well, passing volume key this way is quite insecure, but perhaps
> acceptable for debugging.
>
> >
> > But LUKS2/cryptsetup now stores the key in the kernel keyring by
> > default. The key is accessible from userspace.
>
> If you are talking about volume key (not passsphrase), it is not
> available from userspace. Only reference to it. But you can use
> this reference to construct in-kernel dm-crypt device.
> Please read https://gitlab.com/cryptsetup/cryptsetup/-/blob/master/docs/Keyring.txt
>
> > Users can enter the password to start kdump manually and then it will
> > work, but usually people expect kdump service to start automatically.
> >
> > (WIP patch series:
> > https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/RTYJEQ4BV25JYVYJHTTPOK476FUEJOWW/)
> >
> > I've several ideas about how to improve it but not sure which one is
> > better, and if this is a good idea after all:
> > 1. Simply introduce a config to let systemd-cryptsetup disable kernel
> > keyring on setup, there is currently no such option.
>
> Well, that option could be useful anyway and we have support for it
> in cryptsetup (--disable-keyring CLI option) and libcryptsetup, so why not.
> Just this should not be a default option.
>
> This is should be patch for systemd-cryptsetup only as libcryptsetup supports it.
>
> ...
>
> > 2. If we can let the key stay in userspace for a little longer, eg.
> > for systems booted with dracut/systemd, when
> > systemd-cryptsetup@%s.service opens the crypt device, keep the key in
> > dm-crypt. And later when services like kdump have finished loading,
> > cryptsetup can refresh the device and store the key in the kernel
> > keyring again.
>
> We invalidate volume key in keyring after libceyposetup  operation
> is finished (and kernel removes the reference once keyring garbage collection
> is run).
>
> I can imagine to add some option to keep key inside keyring even after
> call is finished, but as said above, this removes some security margin
> we intentionally introduced here.

I agree with your comments, thanks! These two approaches seem not a
good idea now.

>
> ...
>
> Milan
>

How about plan 3 and 4?

> 3. Let kdump use some custom helper/service to load all needed
> resources in the early initrd boot stage, prior to
> systemd-cryptsetup@%s.service. It will ask the password / parse the
> keyfile and load kdump, then provide info for systemd-cryptsetup or
> just do the setup. Or maybe let systemd-cryptsetup support some kind
> of "plugins" so other tools can use it.

Some details could be changed/improved, but
systemd-cryptsetup@%s.service will prompt for a password or use a
keyfile anyway.
So I think at this point, loading kdump with the volume key should be
safe? At least long as the kdump kernel/environment itself isn't
compromised. Loaded kdump resources can be restricted to be only
accessible from the kernel side.
After panic, kernel kexec jumps to kdump kernel, and that's an
minimized emergency environment that only lives for a very short
period.

> 4. A better and safer solution seems to keep a consistent key ring
> between kexec boots but also more complex and difficult as different
> arch implements kexec differently.

Maybe plan 4 will be a good idea if doable? Since that keeps the key
consistent in the kernel between kexec boots, and cryptsetup can just
reuse it.

--
Best Regards,
Kairui Song

_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de

[dm-crypt] Re: Antw: [EXT] [systemd-devel] Kdump with full-disk LUKS encryption

[Milan Broz]

On 20/04/2021 08:05, Ulrich Windl wrote:
>>>> Kairui Song <kasong@redhat.com> schrieb am 19.04.2021 um 12:00 in
> Nachricht
> <CACPcB9e0=KYNc_-Bz5EnoHntKKXpurmXzu4e60J1sADQkizvsg@mail.gmail.com>:
>> Hi all,
>>
>> I'm currently trying to add kdump support for systemd with full‑disk
>> LUKS encryption. vmcores contain sensitive data so they should also be
>> protected, and network dumps sometimes are not available. So kdump has
>> to open the LUKS encrypted device in the kdump environment.
>>
>> I'm using systemd/dracut, my work machine is running Fedora 34, and
>> there are several problems I'm trying to solve:
>> 1. Users have to input the password in the kdump kernel environment.
>> But users often don't have shell access to the kdump environment.
>> (headless server, graphic card not working after kexec, both are very
>> common)
>> 2. LUKS2 prefers Argon2 as the key derivation function, designed to
>> use a lot of memory. kdump is expected to use a minimal amount of
>> memory. Users will have to reserve a huge amount of memory for kdump
>> to work (eg. 1G reserve for kdump with 4G total memory which is not
>> reasonable).
> 
> I'm not a LUKS specialist, but can't you use different KDFs in a different key
> slot?

Yes, you can (for LUKS2). There are also priorities, so you can configure "admin"
keyslot that is never used unless explicitly specified.
It can use different PBKDF and/or cost parameters.

But this is not a solution for the mentioned problem - they have to work
with arbitrary devices.

Milan

p.s.
Some lists on cc rejects replies without subscription, so do not
be surprised if you see only some replies.
_______________________________________________
dm-crypt mailing list -- dm-crypt@saout.de
To unsubscribe send an email to dm-crypt-leave@saout.de