[dm-crypt] LUKS2 SSD swap and system freeze

[dm-crypt] LUKS2 SSD swap and system freeze

[Germano Massullo]

Good day. I am investigating about a weird behavior of my system that
leads to frequent freezes.
The main components of my system are: Fedora 33 (kernel
5.9.13-200.fc33.x86_64), cryptsetup 2.3.4, an AMD Ryzen 5 3600, 16 GB
RAM 3200 MHz, a Samsung 860 PRO used for 32 GB swap and LVM Cache, that
is encrypted with following criteria:

# cryptsetup status ssd_cache
/dev/mapper/ssd_cache is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/sdc
  sector size:  512
  offset:  32768 sectors
  size:    500085424 sectors
  mode:    read/write
  flags:   discards

During write bursts of swap (for example when booting some virtual
machines), the system completely freezes, I cannot even move the mouse
pointer. The system may remain in stuck state even for 10 minutes. When
the (host) system turns back to normal, I can see for example in virtual
machines, the following error in console
========
Message from syslogd@localhost at Dec 15 01:14:32 ...
 kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 124s!
[0x564a0d4baad0-:1924]
========

To check if the complete stuck of the system was caused by encryption, I
tried to disable SSD swap, and create a unencrypted swap on a regular
HDD, and then when I had write bursts, the system slowed, but it did not
freeze.
How could I proceed in my investigation to find out if it is a matter of
system configuration, a bug, etc.?
Thank you

[dm-crypt] LUKS2 SSD swap and system freeze

[Germano Massullo]

Good day. I am investigating about a weird behavior of my system that
leads to frequent freezes.
The main components of my system are: Fedora 33 (kernel
5.9.13-200.fc33.x86_64), cryptsetup 2.3.4, an AMD Ryzen 5 3600, 16 GB
RAM 3200 MHz, a Samsung 860 PRO used for 32 GB swap and LVM Cache, that
is encrypted with following criteria:

# cryptsetup status ssd_cache
/dev/mapper/ssd_cache is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/sdc
  sector size:  512
  offset:  32768 sectors
  size:    500085424 sectors
  mode:    read/write
  flags:   discards

During write bursts of swap (for example when booting some virtual
machines), the system completely freezes, I cannot even move the mouse
pointer. The system may remain in stuck state even for 10 minutes. When
the (host) system turns back to normal, I can see for example in virtual
machines, the following error in console
========
Message from syslogd@localhost at Dec 15 01:14:32 ...
 kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 124s!
[0x564a0d4baad0-:1924]
========

To check if the complete stuck of the system was caused by encryption, I
tried to disable SSD swap, and create a unencrypted swap on a regular
HDD, and then when I had write bursts, the system slowed, but it did not
freeze.
How could I proceed in my investigation to find out if it is a matter of
system configuration, a bug, etc.?
Thank you

_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] LUKS2 SSD swap and system freeze

[Michael Kjörling]

On 15 Dec 2020 11:42 +0100, from germano.massullo@gmail.com (Germano Massullo):
> To check if the complete stuck of the system was caused by encryption, I
> tried to disable SSD swap, and create a unencrypted swap on a regular
> HDD, and then when I had write bursts, the system slowed, but it did not
> freeze.

Note that by doing so, you actually changed at least two things at
once: absence/presence of encryption, and SSD/HDD. There is also a
possibility that you changed the properties of the storage
interconnect. You've shown that by changing all at once, you can
eliminate or introduce the problem, but you haven't unequivocally
shown which specific change made the difference.

To eliminate those factors and further narrow down the problem, can
you change just _one_ of those at a time? For example, do you still
see the same issue with similarly encrypted swap on the HDD, or with
unencrypted swap on the SSD? The answer would help at least somewhat
to lay the blame on either the storage device or the cryptography.

Also, have you checked the SMART data for the SSD?

Another thing that could be worth testing is to access that system
over the network. Assuming your firewall settings allow ICMP, does it
respond to pings while appearing frozen, for example?

-- 
Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
 “Remember when, on the Internet, nobody cared that you were a dog?”

Re: [dm-crypt] LUKS2 SSD swap and system freeze

[Michael Kjörling]

On 15 Dec 2020 11:42 +0100, from germano.massullo@gmail.com (Germano Massullo):
> To check if the complete stuck of the system was caused by encryption, I
> tried to disable SSD swap, and create a unencrypted swap on a regular
> HDD, and then when I had write bursts, the system slowed, but it did not
> freeze.

Note that by doing so, you actually changed at least two things at
once: absence/presence of encryption, and SSD/HDD. There is also a
possibility that you changed the properties of the storage
interconnect. You've shown that by changing all at once, you can
eliminate or introduce the problem, but you haven't unequivocally
shown which specific change made the difference.

To eliminate those factors and further narrow down the problem, can
you change just _one_ of those at a time? For example, do you still
see the same issue with similarly encrypted swap on the HDD, or with
unencrypted swap on the SSD? The answer would help at least somewhat
to lay the blame on either the storage device or the cryptography.

Also, have you checked the SMART data for the SSD?

Another thing that could be worth testing is to access that system
over the network. Assuming your firewall settings allow ICMP, does it
respond to pings while appearing frozen, for example?

-- 
Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
 “Remember when, on the Internet, nobody cared that you were a dog?”

_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] LUKS2 SSD swap and system freeze

[Milan Broz]

On 15/12/2020 11:42, Germano Massullo wrote:
> Good day. I am investigating about a weird behavior of my system that
> leads to frequent freezes.
> The main components of my system are: Fedora 33 (kernel
> 5.9.13-200.fc33.x86_64), cryptsetup 2.3.4, an AMD Ryzen 5 3600, 16 GB
> RAM 3200 MHz, a Samsung 860 PRO used for 32 GB swap and LVM Cache, that
> is encrypted with following criteria:
> 
> # cryptsetup status ssd_cache
> /dev/mapper/ssd_cache is active and is in use.
>   type:    LUKS2
>   cipher:  aes-xts-plain64
>   keysize: 512 bits
>   key location: keyring
>   device:  /dev/sdc
>   sector size:  512
>   offset:  32768 sectors
>   size:    500085424 sectors
>   mode:    read/write
>   flags:   discards
> 
> During write bursts of swap (for example when booting some virtual
> machines), the system completely freezes, I cannot even move the mouse
> pointer. The system may remain in stuck state even for 10 minutes. When
> the (host) system turns back to normal, I can see for example in virtual
> machines, the following error in console
> ========
> Message from syslogd@localhost at Dec 15 01:14:32 ...
>  kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 124s!
> [0x564a0d4baad0-:1924]
> ========
> 
> To check if the complete stuck of the system was caused by encryption, I
> tried to disable SSD swap, and create a unencrypted swap on a regular
> HDD, and then when I had write bursts, the system slowed, but it did not
> freeze.
> How could I proceed in my investigation to find out if it is a matter of
> system configuration, a bug, etc.?

First, this is kernel issue, not a cryptsetup userspace issue
(cryptsetup only configures kernel dm-crypt).

With some encryption drivers that allocates memory during encryption
there was a possible deadlock if swap is encrypted and system is out of memory.

There should be a workaround in latest stable kernel that does not allow
to use such crypto drivers in dm-crypt.
For dm-crypt it is commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/md/dm-crypt.c?id=cd74693870fb748d812867ba49af733d689a3604
(but it requires also patches for crypto subsystem for setting this flag in drivers.)

The long-term solution is to fix crypto drivers...

Anyway, please try the latest stable kernel (ideally 5.10.x) - if you still
see the problem, please report it to dm-devel@redhat.com mailing list.

(There is another patch that limits IOs in flight in dm-crypt, but this was
not yet merged upstream - that could help too).

Thanks,
Milan
_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt