How to resume decryption on LUKS1 in-place

How to resume decryption on LUKS1 in-place

[xndr]

Hello 1&All,

Please help us complete an in-place decryption of a LUKS1 device:-

We followed the steps here on the Arch Linux wiki by converting from
LUKS2 to LUKS1:-

https://wiki.archlinux.org/title/Removing_system_encryption#Decrypting_LUKS2_devices_in-place

Unfortunately the device suffered an IO error at about 85% after issuing
the final command:-

cryptsetup-reencrypt --decrypt /dev/sda3

 From the man pages it might seem that the process can be resumed by
using the command:-

cryptsetup reencrypt --resume-only --header
luks2-hdr-file /dev/encrypted_device

However it seems qualified by being a LUKS2 only facility. Should this
command still work on LUKS1 or is the process (if there's one at all)
different.

Available are copies of the headers (LUKS 1 & 2) in text and binary form
and what appears to be a log generated during decryption.

Any counsel greatly appreciated - thank you.

Re: How to resume decryption on LUKS1 in-place

[Ondrej Kozina]

Hi,

On 21/02/2024 18:22, xndr wrote:
> 
> We followed the steps here on the Arch Linux wiki by converting from
> LUKS2 to LUKS1:-
> 
> https://wiki.archlinux.org/title/Removing_system_encryption#Decrypting_LUKS2_devices_in-place

While I greatly appreciate what Archlinux community does for 
documentation effort this particular hint is quite unfortunate. The hint 
has mistaken the cause and the effect of a dangerous experimentation and 
on top of all, it is outdated (as advertised under section 1.3 heading).

1) Up to (including) 2.4.3 LUKS2 clearly did not support decryption with 
LUKS2 header put in the beginning of data device (regular LUKS2 device). 
Someone tried to force
cryptsetup utility to bypass the limitation by making header device and 
data device the same via cmd line parameters. There was a bug that could 
not detect such dangerous case and initiated decryption anyway. We have 
patched the bug out ASAP. It definitely did not destroyed data but 
decrypted data in-palce at the original data offset (which is not user 
friendly and that's why it was not support from cli).

2) Since 2.5.0 you can decrypt LUKS2 device by specifying a file where 
LUKS2 header is migrated for the decryption to be possible. See man page 
cryptsetup-reencrypt, LUKS2 decryption examples section.

I'll suggest edit to the wiki mentioned above.

> 
> Unfortunately the device suffered an IO error at about 85% after issuing
> the final command:-

If you have suffered IO error due to HW issue the best thing you can do 
now is copy all data to a new device first (by dd utility for example) 
and try to finish the operation on the healthy device.

> 
> cryptsetup-reencrypt --decrypt /dev/sda3

If you have downconverted the device to LUKS1 before the decryption
you'd need to pass the original device UUID via --uuid parameter. LUKS1 
decryption uses files LUKS-<UUID>.[|log|org|new] to track the progress 
and to safe tempory LUKS1 headers. Look for those file in work directory 
from what you have initiated the operation. See cryptsetup-reencrypt man 
page, --uuid option.

But again, migrate your data to a healthy device first, provided the IO 
error had originated in a HW issue.

Also I'd recommend to update the cryptsetup package. The 
cryptsetup-reencrypt binary was replaced fully by "cryptsetup reencrypt" 
command (also for LUKS1) in 2.5.0 release in Jul '22.

> 
>   From the man pages it might seem that the process can be resumed by
> using the command:-
> 
> cryptsetup reencrypt --resume-only --header
> luks2-hdr-file /dev/encrypted_device

That's for LUKS2 only.

> 
> However it seems qualified by being a LUKS2 only facility. Should this
> command still work on LUKS1 or is the process (if there's one at all)
> different.
> 
> Available are copies of the headers (LUKS 1 & 2) in text and binary form
> and what appears to be a log generated during decryption.

If you have converted LUKS2 to LUKS1 before the decryption as the wiki 
suggests there should be only LUKS1 related logs and header backups.

> 
> Any counsel greatly appreciated - thank you.
> 
> 

Kind regards
O.

Re: How to resume decryption on LUKS1 in-place

[xndr]

Thank you for the assist.

We've taken the advice to migrate the data to a backup and after
reattempted using the command:-

cryptsetup reencrypt --decrypt
--uuid=LUKS-aaecbfc4-459a-4cf9-9a34-fd4af43b8be0  /dev/sdd

This returns:-

Device /dev/sdd is not a valid LUKS device.

The same is true when omitting "LUKS" from the UUID.

Both the source and clone disks are readable 'as they are' and there's a
recovery of 263GB of 500GB - ostensibly it seems the original decryption
recovered the active data available.

But we cannot be sure and are somewhat confused now because the original
decrypt operation definitely tanked at ~85%.

Is there anything left we can do before to be sure?

Thanks

On 22/02/2024 11:04, Ondrej Kozina wrote:
> Hi,
>
> On 21/02/2024 18:22, xndr wrote:
>>
>> We followed the steps here on the Arch Linux wiki by converting from
>> LUKS2 to LUKS1:-
>>
>> https://wiki.archlinux.org/title/Removing_system_encryption#Decrypting_LUKS2_devices_in-place
>>
>
> While I greatly appreciate what Archlinux community does for
> documentation effort this particular hint is quite unfortunate. The
> hint has mistaken the cause and the effect of a dangerous
> experimentation and on top of all, it is outdated (as advertised under
> section 1.3 heading).
>
> 1) Up to (including) 2.4.3 LUKS2 clearly did not support decryption
> with LUKS2 header put in the beginning of data device (regular LUKS2
> device). Someone tried to force
> cryptsetup utility to bypass the limitation by making header device
> and data device the same via cmd line parameters. There was a bug that
> could not detect such dangerous case and initiated decryption anyway.
> We have patched the bug out ASAP. It definitely did not destroyed data
> but decrypted data in-palce at the original data offset (which is not
> user friendly and that's why it was not support from cli).
>
> 2) Since 2.5.0 you can decrypt LUKS2 device by specifying a file where
> LUKS2 header is migrated for the decryption to be possible. See man
> page cryptsetup-reencrypt, LUKS2 decryption examples section.
>
> I'll suggest edit to the wiki mentioned above.
>
>>
>> Unfortunately the device suffered an IO error at about 85% after issuing
>> the final command:-
>
> If you have suffered IO error due to HW issue the best thing you can
> do now is copy all data to a new device first (by dd utility for
> example) and try to finish the operation on the healthy device.
>
>>
>> cryptsetup-reencrypt --decrypt /dev/sda3
>
> If you have downconverted the device to LUKS1 before the decryption
> you'd need to pass the original device UUID via --uuid parameter.
> LUKS1 decryption uses files LUKS-<UUID>.[|log|org|new] to track the
> progress and to safe tempory LUKS1 headers. Look for those file in
> work directory from what you have initiated the operation. See
> cryptsetup-reencrypt man page, --uuid option.
>
> But again, migrate your data to a healthy device first, provided the
> IO error had originated in a HW issue.
>
> Also I'd recommend to update the cryptsetup package. The
> cryptsetup-reencrypt binary was replaced fully by "cryptsetup
> reencrypt" command (also for LUKS1) in 2.5.0 release in Jul '22.
>
>>
>>   From the man pages it might seem that the process can be resumed by
>> using the command:-
>>
>> cryptsetup reencrypt --resume-only --header
>> luks2-hdr-file /dev/encrypted_device
>
> That's for LUKS2 only.
>
>>
>> However it seems qualified by being a LUKS2 only facility. Should this
>> command still work on LUKS1 or is the process (if there's one at all)
>> different.
>>
>> Available are copies of the headers (LUKS 1 & 2) in text and binary form
>> and what appears to be a log generated during decryption.
>
> If you have converted LUKS2 to LUKS1 before the decryption as the wiki
> suggests there should be only LUKS1 related logs and header backups.
>
>>
>> Any counsel greatly appreciated - thank you.
>>
>>
>
> Kind regards
> O.
>
>

Re: How to resume decryption on LUKS1 in-place

[Ondrej Kozina]

On 01/03/2024 14:12, xndr wrote:
> Thank you for the assist.
> 
> We've taken the advice to migrate the data to a backup and after
> reattempted using the command:-
> 
> cryptsetup reencrypt --decrypt
> --uuid=LUKS-aaecbfc4-459a-4cf9-9a34-fd4af43b8be0  /dev/sdd
> 
> This returns:-
> 
> Device /dev/sdd is not a valid LUKS device.

Ok, this is a bug (regression in fact) in cryptsetup v2.5.0 which is not 
yet fixed.

The most straightforward workaround would be to install cryptsetup 
v2.4.3 and run following command using legacy cryptsetup-reencrypt utility:

"cryptsetup-reencrypt --decrypt --uuid 
aaecbfc4-459a-4cf9-9a34-fd4af43b8be0  /dev/sdd"

(no need to add 'LUKS-' prefix in --uuid parameter).

I'll figure out a fix for it (feel free to open issue on 
https://gitlab.com/cryptsetup/cryptsetup if interested).

Thank you for the feedback!
Ondrej

Re: How to resume decryption on LUKS1 in-place

[xndr]

For the record:-

     $: cryptsetup -V
     cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI

     $: apt show cryptsetup
     Package: cryptsetup
     Version: 2:2.6.1-4~deb12u2

We see the latest stable version on GitLab is now 2.7.0; is it still
recommended to rollback to 2.4.3?

As for raising a bug, we'd do so willingly but unfortunately this is not
possible as GitLab no longer allow us to use our  GitHub profile.

:-(

Thanks again though.

On 04/03/2024 10:59, Ondrej Kozina wrote:
> On 01/03/2024 14:12, xndr wrote:
>> Thank you for the assist.
>>
>> We've taken the advice to migrate the data to a backup and after
>> reattempted using the command:-
>>
>> cryptsetup reencrypt --decrypt
>> --uuid=LUKS-aaecbfc4-459a-4cf9-9a34-fd4af43b8be0  /dev/sdd
>>
>> This returns:-
>>
>> Device /dev/sdd is not a valid LUKS device.
>
> Ok, this is a bug (regression in fact) in cryptsetup v2.5.0 which is
> not yet fixed.
>
> The most straightforward workaround would be to install cryptsetup
> v2.4.3 and run following command using legacy cryptsetup-reencrypt
> utility:
>
> "cryptsetup-reencrypt --decrypt --uuid
> aaecbfc4-459a-4cf9-9a34-fd4af43b8be0  /dev/sdd"
>
> (no need to add 'LUKS-' prefix in --uuid parameter).
>
> I'll figure out a fix for it (feel free to open issue on
> https://gitlab.com/cryptsetup/cryptsetup if interested).
>
> Thank you for the feedback!
> Ondrej
>
>

Re: How to resume decryption on LUKS1 in-place

[Ondrej Kozina]

On 04/03/2024 15:51, xndr wrote:
> For the record:-
> 
>       $: cryptsetup -V
>       cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI
> 
>       $: apt show cryptsetup
>       Package: cryptsetup
>       Version: 2:2.6.1-4~deb12u2
> 
> We see the latest stable version on GitLab is now 2.7.0; is it still
> recommended to rollback to 2.4.3?

In general? No I don't recommend to downgrade to 2.4.3 for normal 
operation. It's just currently the best option to workaround the bug 
introduced in 2.5.0 regarding resuming LUKS1 decryption operation.

You'd need the old version only for this particular case. Nothing else.

Anyway, I'll send patch to upstream soon so there will be a proper fix 
for 2.7.0 and later releases.

Kind regards
O.

Re: How to resume decryption on LUKS1 in-place

[xndr]

> In general? No I don't recommend to downgrade to 2.4.3 for normal
> operation. It's just currently the best option to workaround the bug
> introduced in 2.5.0 regarding resuming LUKS1 decryption operation.
>
Apologies - my bad; this is really what we meant.

In the meantime we'll update this thread with the result.

Ta lots,

xander

Re: How to resume decryption on LUKS1 in-place

[Ondrej Kozina]

On 04/03/2024 17:34, xndr wrote:
> 
>> In general? No I don't recommend to downgrade to 2.4.3 for normal
>> operation. It's just currently the best option to workaround the bug
>> introduced in 2.5.0 regarding resuming LUKS1 decryption operation.
>
> Apologies - my bad; this is really what we meant.
No problem, the issue is already fixed upstream[1]. Will be released 
within cryptsetup 2.7.1.

[1]: https://gitlab.com/cryptsetup/cryptsetup/-/merge_requests/616

Kind regards
O.

Re: How to resume decryption on LUKS1 in-place

[xndr]

Hello again,

We managed to follow the instructions and resume the decryption from
where it left-off (and thus the data we needed).

Thanks again for helping us on this - much appreciated.

On 07/03/2024 12:14, Ondrej Kozina wrote:
> On 04/03/2024 17:34, xndr wrote:
>>
>>> In general? No I don't recommend to downgrade to 2.4.3 for normal
>>> operation. It's just currently the best option to workaround the bug
>>> introduced in 2.5.0 regarding resuming LUKS1 decryption operation.
>>
>> Apologies - my bad; this is really what we meant.
> No problem, the issue is already fixed upstream[1]. Will be released
> within cryptsetup 2.7.1.
>
> [1]: https://gitlab.com/cryptsetup/cryptsetup/-/merge_requests/616
>
> Kind regards
> O.
>
>