Error while unlocking encrypted BCacheFS: Required key not available

Error while unlocking encrypted BCacheFS: Required key not available

[Martin Steigerwald]

Hi!

Kernel 6.7.0-rc8 with BCacheFS new year fixes. Compliled with Debian gcc 
13.2.0-9.

BCacheFS tools 1.3.3 – according to bcachefs version – from Debian package 
bcachefs-tools 24+really1.3.4-2.

Linux kernel keyutils from Debian package keyutils 1.6.3-2+b2. (Not sure 
whether really required.)

Created BCacheFS on external 4 TB SSD:

% mkfs.bcachefs -L […] --data_checksum xxhash --metadata_checksum xxhash 
--compression=lz4 --encrypted /dev/sda1

(also tried without xxhash, no difference)


Unlock attempt with incorrect passphrase:

% bcachefs unlock /dev/sda1
Enter passphrase: 
incorrect passphrase

Unlock attempt with correct passphrase does not yield error message 
"incorrect passphrase". Key seems to be available:

% grep bcachefs /proc/keys
1b9e7153 I--Q---  1 perm 3f010000  0  0 user   bcachefs:[… UUID …]: 32

UUID matches filesystem.


Still I get:

% LANG=en mount /dev/sda1 /mnt/zeit
mount: /mnt/zeit: mount(2) system call failed: Required key not available.
       dmesg(1) may have more information after failed mount system call.

% dmesg | tail -1
[105441.695035] bcachefs ([…]): error requesting encryption key: ENOKEY


Why? And how to fix it?

I found

error requesting encryption key #93 

https://github.com/koverstreet/bcachefs/issues/93

But I am not sure whether it applies to my situation.

I use Devuan with elogind. Do I need that pam related configuration change 
from comment

https://github.com/koverstreet/bcachefs/issues/93#issuecomment-609430340

?

I do not like to do it in case it is not required.

Best,
-- 
Martin

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Martin Steigerwald]

Martin Steigerwald - 07.01.24, 12:22:53 CET:
> Hi!
> 
> Kernel 6.7.0-rc8 with BCacheFS new year fixes. Compliled with Debian gcc
> 13.2.0-9.
> 
> BCacheFS tools 1.3.3 – according to bcachefs version – from Debian
> package bcachefs-tools 24+really1.3.4-2.
> 
> Linux kernel keyutils from Debian package keyutils 1.6.3-2+b2. (Not sure
> whether really required.)
> 
> Created BCacheFS on external 4 TB SSD:
> 
> % mkfs.bcachefs -L […] --data_checksum xxhash --metadata_checksum xxhash
> --compression=lz4 --encrypted /dev/sda1
> 
> (also tried without xxhash, no difference)
> 
> 
> Unlock attempt with incorrect passphrase:
> 
> % bcachefs unlock /dev/sda1
> Enter passphrase:
> incorrect passphrase
> 
> Unlock attempt with correct passphrase does not yield error message
> "incorrect passphrase". Key seems to be available:
> 
> % grep bcachefs /proc/keys
> 1b9e7153 I--Q---  1 perm 3f010000  0  0 user   bcachefs:[… UUID …]: 32

Also keyctl sees the key in root user keyring:

% keyctl list @u
1 key in keyring:
463368531: --alswrv     0     0 user: bcachefs:[… UUID …]

In case this is an issue with Debian packaging of bcachefs-tools I can 
report there.

> UUID matches filesystem.
> 
> 
> Still I get:
> 
> % LANG=en mount /dev/sda1 /mnt/zeit
> mount: /mnt/zeit: mount(2) system call failed: Required key not
> available. dmesg(1) may have more information after failed mount system
> call.
> 
> % dmesg | tail -1
> [105441.695035] bcachefs ([…]): error requesting encryption key: ENOKEY
> 
> 
> Why? And how to fix it?
> 
> I found
> 
> error requesting encryption key #93
> 
> https://github.com/koverstreet/bcachefs/issues/93
> 
> But I am not sure whether it applies to my situation.
> 
> I use Devuan with elogind. Do I need that pam related configuration
> change from comment
> 
> https://github.com/koverstreet/bcachefs/issues/93#issuecomment-609430340
> 
> ?
> 
> I do not like to do it in case it is not required.
> 
> Best,


-- 
Martin

Re: Error while unlocking encrypted BCacheFS: Required key not available

[AP]

On Sun, Jan 07, 2024 at 12:27:29PM +0100, Martin Steigerwald wrote:
> In case this is an issue with Debian packaging of bcachefs-tools I can 
> report there.

What version of bcachefs-tools are you using? The default one in bookworm
is v0.24 and is >1 year old and that may be the issue.

I'm starting to switch from btrfs to bcachefs and I backported v1.3.4 from
sid.

AP

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Martin Steigerwald]

AP - 10.01.24, 03:13:01 CET:
> On Sun, Jan 07, 2024 at 12:27:29PM +0100, Martin Steigerwald wrote:
> > In case this is an issue with Debian packaging of bcachefs-tools I can
> > report there.
> 
> What version of bcachefs-tools are you using? The default one in
> bookworm is v0.24 and is >1 year old and that may be the issue.

As I wrote:

"BCacheFS tools 1.3.3 – according to bcachefs version – from Debian 
package bcachefs-tools 24+really1.3.4-2.".

> I'm starting to switch from btrfs to bcachefs and I backported v1.3.4
> from sid.

That is where above package is from.

Best,
-- 
Martin

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Kent Overstreet]

On Wed, Jan 10, 2024 at 03:18:55PM +0100, Martin Steigerwald wrote:
> AP - 10.01.24, 03:13:01 CET:
> > On Sun, Jan 07, 2024 at 12:27:29PM +0100, Martin Steigerwald wrote:
> > > In case this is an issue with Debian packaging of bcachefs-tools I can
> > > report there.
> > 
> > What version of bcachefs-tools are you using? The default one in
> > bookworm is v0.24 and is >1 year old and that may be the issue.
> 
> As I wrote:
> 
> "BCacheFS tools 1.3.3 – according to bcachefs version – from Debian 
> package bcachefs-tools 24+really1.3.4-2.".
> 
> > I'm starting to switch from btrfs to bcachefs and I backported v1.3.4
> > from sid.
> 
> That is where above package is from.

The keyring stuff has been a perpetual utter headache.

I've been debating rewriting that stuff to just pass a memfd handle as a
mount option and rip out keyring usage...

alternately - now that we're pretty much always mounting via the mount
helper, perhaps it would be a little bit less fragile if the mount
helper was adding the key to the keyring - that might be worth checking.

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Martin Steigerwald]

Hi!

Kent Overstreet - 10.01.24, 20:13:59 CET:
> On Wed, Jan 10, 2024 at 03:18:55PM +0100, Martin Steigerwald wrote:
> > AP - 10.01.24, 03:13:01 CET:
> > > On Sun, Jan 07, 2024 at 12:27:29PM +0100, Martin Steigerwald wrote:
> > > > In case this is an issue with Debian packaging of bcachefs-tools I
> > > > can
> > > > report there.
> > > 
> > > What version of bcachefs-tools are you using? The default one in
> > > bookworm is v0.24 and is >1 year old and that may be the issue.
> > 
> > As I wrote:
> > 
> > "BCacheFS tools 1.3.3 – according to bcachefs version – from Debian
> > package bcachefs-tools 24+really1.3.4-2.".
> > 
> > > I'm starting to switch from btrfs to bcachefs and I backported
> > > v1.3.4
> > > from sid.
> > 
> > That is where above package is from.
> 
> The keyring stuff has been a perpetual utter headache.
> 
> I've been debating rewriting that stuff to just pass a memfd handle as a
> mount option and rip out keyring usage...
> 
> alternately - now that we're pretty much always mounting via the mount
> helper, perhaps it would be a little bit less fragile if the mount
> helper was adding the key to the keyring - that might be worth checking.

So no suggestion what to try to make it work except for putting BCacheFS 
unto LUKS? I wanted to get rid of LUKS for removable media. Often enough I 
have "cryptsetup luksClose" complaining about still in use while I the 
filesystem on top of it clearly got unmounted already. In these case BTRFS 
still.

Ciao,
-- 
Martin

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Kent Overstreet]

On Thu, Jan 11, 2024 at 12:58:26PM +0100, Martin Steigerwald wrote:
> Hi!
> 
> Kent Overstreet - 10.01.24, 20:13:59 CET:
> > On Wed, Jan 10, 2024 at 03:18:55PM +0100, Martin Steigerwald wrote:
> > > AP - 10.01.24, 03:13:01 CET:
> > > > On Sun, Jan 07, 2024 at 12:27:29PM +0100, Martin Steigerwald wrote:
> > > > > In case this is an issue with Debian packaging of bcachefs-tools I
> > > > > can
> > > > > report there.
> > > > 
> > > > What version of bcachefs-tools are you using? The default one in
> > > > bookworm is v0.24 and is >1 year old and that may be the issue.
> > > 
> > > As I wrote:
> > > 
> > > "BCacheFS tools 1.3.3 – according to bcachefs version – from Debian
> > > package bcachefs-tools 24+really1.3.4-2.".
> > > 
> > > > I'm starting to switch from btrfs to bcachefs and I backported
> > > > v1.3.4
> > > > from sid.
> > > 
> > > That is where above package is from.
> > 
> > The keyring stuff has been a perpetual utter headache.
> > 
> > I've been debating rewriting that stuff to just pass a memfd handle as a
> > mount option and rip out keyring usage...
> > 
> > alternately - now that we're pretty much always mounting via the mount
> > helper, perhaps it would be a little bit less fragile if the mount
> > helper was adding the key to the keyring - that might be worth checking.
> 
> So no suggestion what to try to make it work except for putting BCacheFS 
> unto LUKS? I wanted to get rid of LUKS for removable media. Often enough I 
> have "cryptsetup luksClose" complaining about still in use while I the 
> filesystem on top of it clearly got unmounted already. In these case BTRFS 
> still.

I just made a couple suggestions. I'm sorry if they weren't exactly what
you were looking for?

If you or someone else wants to help out by writing some code, I just
laid out what needs to happen next - but I'm not your free helpdesk
here, and you're expecting an imediate fix that's not how this works :)

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Martin Steigerwald]

Kent Overstreet - 11.01.24, 17:35:10 CET:
> I just made a couple suggestions. I'm sorry if they weren't exactly what
> you were looking for?
> 
> If you or someone else wants to help out by writing some code, I just
> laid out what needs to happen next - but I'm not your free helpdesk
> here, and you're expecting an imediate fix that's not how this works :)

That is a misunderstanding. I am not expecting an immediate fix or a fix at 
all. I did not write I do either. So please do not read something into 
what I wrote that simply is not there.

My programming experience in C is more than 20 years old, certainly not 
kernel related and not even on the Linux platform and I thought there 
might be something I as an user might try to at least pin-point the cause 
of this issue a bit more. Apparently there is not and that is okay with 
me.

Nothing of what I do with BCacheFS has immediate urgency at the moment.

-- 
Martin

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Martin Steigerwald]

Hi George, hi Kent, hi,

George Hilliard - 16.01.24, 18:59:08 CET:
> The workaround for users is to run:
> 
>     keyctl link @u @s
> 
> just before running `bcachefs mount`.

That workaround works for me. Thanks.

Copying several TB to a new external 4 TB SSD with BCacheFS currently :)

I will try the other workaround from the other thread as well.

Thanks,
-- 
Martin

Re: Error while unlocking encrypted BCacheFS: Required key not available

[Martin Steigerwald]

George Hilliard - 16.01.24, 18:59:08 CET:
> Some other folks have found similar problems with other uses of keyctl,
> see [1]. It appears systemd segments each system service into its own
> kernel keyring. Presumably the one bcachefs-tools is writing into, is
> not the one the kernel is reading during mount.

I use Devuan with Runit, so no Systemd involved. However elogind is in 
use. Maybe a function of systemd-logind / elogind?

> The workaround for users is to run:
> 
>     keyctl link @u @s
> 
> just before running `bcachefs mount`.

I will be trying this out and report back. Thanks!

> I am not enough of an expert with kernel keyrings to know whether the
> kernel code, systemd, Arch's packaging, or something else is at fault
> here.

From what I gathered so far there is some complexity involved here, as 
seems to be usual when it comes to encryption.

Best,
-- 
Martin

Re: Error while unlocking encrypted BCacheFS: Required key not available

[George Hilliard]

> The keyring stuff has been a perpetual utter headache.
>
> I've been debating rewriting that stuff to just pass a memfd handle as a
> mount option and rip out keyring usage...
>
> alternately - now that we're pretty much always mounting via the mount
> helper, perhaps it would be a little bit less fragile if the mount
> helper was adding the key to the keyring - that might be worth checking.

I am hitting this exact issue with the same exact baffling behavior (bcachefs
format, keyctl list, bcachefs mount -> fails). I'm on Arch with Linux
6.7.0-arch3-1
and bcachefs-tools 3:1.4.1-1.

Some other folks have found similar problems with other uses of keyctl, see [1].
It appears systemd segments each system service into its own kernel keyring.
Presumably the one bcachefs-tools is writing into, is not the one the kernel is
reading during mount.

The workaround for users is to run:

    keyctl link @u @s

just before running `bcachefs mount`.

I am not enough of an expert with kernel keyrings to know whether the kernel
code, systemd, Arch's packaging, or something else is at fault here.

- George

[1]: https://github.com/NixOS/nixpkgs/issues/32279