multiuser/cifscreds not functioning on newer Ubuntu releases

multiuser/cifscreds not functioning on newer Ubuntu releases

[Nate Collins]

I initially posted this to the Samba mailing list, where it didn't 
receive much attention, but it might be more appropriate for a
CIFS-specific mailing list. Per the subject, this may be an
Ubuntu-specific bug, but I don't have the familiarity with CIFS to
claim this is so and that nothing is wrong with my setup.

---

This is a problem that's plagued us for a while that we haven't been
able to resolve due to the lack of familiarity with cifscreds and keyring
debugging, so I thought I'd ask here to see if there's anything obviously
wrong with our setup, or if there's any cifscreds/keyring debugging
advice that could help us.

We currently use multiuser CIFS mounts on a handful of domain-joined
Ubuntu 16.04 servers to permit users to access CIFS shares with their AD
credentials. The CIFS shares in question are mounted by a separate service
account. Everything has been working as expected on the 16.04 servers:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"

$ uname -a
Linux host1 4.4.0-206-generic #238-Ubuntu SMP Tue Mar 16 07:52:37 UTC
2021 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg --list | grep cifs
ii  cifs-utils                            2:6.4-1ubuntu1.1
amd64        Common Internet File System utilities

$ keyctl show
Session Keyring
 [key] --alswrv      0     0  keyring: _ses
 [key] ---lswrv      0 65534   \_ keyring: _uid.0

$ cifscreds add share
Password:

$ ls /mnt/share/
Content
...

$ keyctl show
Session Keyring
 [key] --alswrv      0     0  keyring: _ses
 [key] ---lswrv      0 65534   \_ keyring: _uid.0
 [key] ----sw-v  [uid] [gid]   \_ logon: cifs:a:[share IP]

$ mount | grep multiuser
//share/share on /mnt/share type cifs
(rw,relatime,vers=3.0,sec=ntlmsspi,cache=strict,multiuser,domain=[domain],
uid=0,noforceuid,gid=0,noforcegid,addr=[share IP],file_mode=0755,
dir_mode=0755,nounix,serverino,mapposix,noperm,rsize=1048576,
wsize=1048576,echo_interval=60,actimeo=1)

$ grep cifs /etc/pam.d/*
/etc/pam.d/common-auth:auth    optional            pam_cifscreds.so
/etc/pam.d/common-session:session optional    pam_cifscreds.so
host=[domain controller]

$

However, with the exact same setup on an 18.04 and 20.04 server, I am
unable to access the CIFS mount after running cifscreds add share:

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

$ uname -a
Linux host2 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg --list | grep cifs
ii  cifs-utils                            2:6.8-1ubuntu1.1
amd64        Common Internet File System utilities

$ keyctl show
Session Keyring
 [key] --alswrv      0     0  keyring: _ses
 [key] ---lswrv      0 65534   \_ keyring: _uid.0

$ cifscreds add share
Password:

$ ls /mnt/share
ls: cannot access '/mnt/share': Permission denied

$ keyctl show
Session Keyring
 [key] --alswrv      0     0  keyring: _ses
 [key] ---lswrv      0 65534   \_ keyring: _uid.0
 [key] ----sw-v  [uid] [gid]   \_ logon: cifs:a:[share IP]

$ mount | grep multiuser
//share/share on /mnt/share type cifs
(rw,relatime,vers=3.0,sec=ntlmsspi,cache=strict,multiuser,domain=[domain],
uid=0,noforceuid,gid=0,noforcegid,addr=[share IP],file_mode=0755,
dir_mode=0755,soft,nounix,serverino,mapposix,noperm,rsize=1048576,
wsize=1048576,echo_interval=60,actimeo=1)

$

The following error appears in dmesg whenever I try to interact with
/mnt/share:

[   44.660510] CIFS VFS: signing requested but authenticated as guest
[   44.663053] CIFS VFS: SMB signature verification returned error = -13
[   44.663172] CIFS VFS: failed to connect to IPC (rc=-13)
[   44.664501] CIFS VFS: SMB signature verification returned error = -13
[   44.665361] CIFS VFS: SMB signature verification returned error = -13
[   44.665442] CIFS VFS: cifs_put_smb_ses: Session Logoff failure rc=-13

I can't tell if these mssages are significant or not -- I recall reading
that this was just a change in logging between releases. I tried using
ntlm through ntlmssp for the mount, but none of those protocols resulted
in the mountpoint being accessible by my user (they did however get rid of
the dmesg message). I've tried adjusting other mount options to no avail.
I can recreate this issue on multiple shares across multiple servers.

I found this bug report:

https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1900856

along with this one:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986168

which seem to be similar issues, except with Kerberos authentication. From
the first report, I enabled some more verbose debugging:

$ echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
$ echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
$ echo 7 > /proc/fs/cifs/cifsFYI
$ echo 1 > /sys/module/dns_resolver/parameters/debug

Then, when attempting to ls /mnt/share:

/build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2maperror.c: Mapping SMB2
status code 0xc0000022 to POSIX err -13
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/misc.c: Null buffer passed
to cifs_small_buf_release
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/connect.c: CIFS VFS: leaving
cifs_setup_ipc (xid = 109) rc = -13
CIFS VFS: failed to connect to IPC (rc=-13)
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/connect.c: CIFS VFS: in
cifs_get_tcon as Xid: 110 with uid: 1001
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2pdu.c: TCON
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/transport.c: Sending smb:
smb_len=100
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/connect.c: RFC1002 header 0x49
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2misc.c: smb2_check_message
length: 0x4d, smb_buf_length: 0x49
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2misc.c: SMB2 len 77
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/transport.c:
cifs_sync_mid_result: cmd=3 mid=41 state=4
CIFS VFS: SMB signature verification returned error = -13
Status code returned 0xc0000022 STATUS_ACCESS_DENIED
/build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2maperror.c: Mapping SMB2
status code 0xc0000022 to POSIX err -13

Not sure if these messages are meaningful, but I thought I'd include them.

There's been other cifscreds strangeness that we've noticed, like access
to mounts not being immediately revoked when `cifscreds clear[all]`
is ran, but this is the main, easily reproducible issue we've been seeing
that's preventing us from using multiuser on CIFS mounts.

We use Samba DCs on version 4.12.3-8, and the CIFS share hosts are
up-to-date TrueNAS servers.

Re: multiuser/cifscreds not functioning on newer Ubuntu releases

[Aurélien Aptel]

Nate Collins <ncollins@xes-inc.com> writes:
> I initially posted this to the Samba mailing list, where it didn't 
> receive much attention, but it might be more appropriate for a
> CIFS-specific mailing list. Per the subject, this may be an
> Ubuntu-specific bug, but I don't have the familiarity with CIFS to
> claim this is so and that nothing is wrong with my setup.

You can try to look at what happens in Wireshark. If you are getting
STATUS_ACCESS_DENIED during session setup it could be a signing problem
or kerberos issue. I would be interesting to see to which request are
you getting that error back from anyway.

Ubuntu 18 is from 2018 right? There have been multiple regressions and
fixes related to signing since then. After a quick scan I see these
fixes (from most recent to oldest):

05946d4b7a73 cifs: Fix preauth hash corruption
edad734c74a4 smb3: use SMB2_SIGNATURE_SIZE define
cc95b6772790 cifs: fix channel signing
ff6b6f3f9160 cifs: Always update signing key of first channel
46f17d17687e smb3: fix signing verification of large reads
8c11a607d1d9 SMB3: Fix SMB3.1.1 guest mounts to Samba
e71ab2aa06f7 cifs: allow guest mounts to work for smb3.11
a5c62f4833c2 CIFS: fix uninitialized ptr deref in smb2 signing
a12d0c590cc7 cifs: Make sure all data pages are signed correctly
8de8c4608fe9 cifs: Fix validation of signed data in smb2
27c32b49c3db cifs: Fix validation of signed data in smb3+
83ffdeadb46b cifs: Fix invalid check in __cifs_calc_signature()
...

For debugging keyrings you can write a shell script wrapper for all the
cifs keys defined in /etc/request-keys.conf. This allows you to log
calls, strace the bins, etc. See Sashin's blog:

http://sprabhu.blogspot.com/2014/12/debugging-calls-to-cifsupcall.html

If you can recompile the cifs-utils upcall bins you can add calls to log
PID and sleep so you can attach gdb and step into it, can be really
useful as well.

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: multiuser/cifscreds not functioning on newer Ubuntu releases

[Nate Collins]

Thanks for the wireshark tip, I found that the domain wasn't getting
included in the NTLMSSP_AUTH packet.

cifscreds add -u user@domain.com share

allows me to access the mutiuser mount as my user. My question now is,
where should that domain be specified so that it's not needed when
running cifscreds? The mount already has domain=domain.com set, and
in the pam_cifscreds.co module's line in /etc/pam.d/common-session,
I tried replacing host=dc.domain.com with domain=domain.com, but still
the NTLMSSP_AUTH packet didn't have the domain set.

> Ubuntu 18 is from 2018 right? There have been multiple regressions and
> fixes related to signing since then. After a quick scan I see these
> fixes (from most recent to oldest):

Yeah, 18.04 is from 2018, but 20.04 (the most recent LTS release) has
the same change in functionality. Should've made that clearer in my
initial post.

Re: multiuser/cifscreds not functioning on newer Ubuntu releases

[Shyam Prasad N]

Hi Nate,

From the log messages that you provided, the server has responded with
STATUS_ACCESS_DENIED. So the server does not believe that the user has
the necessary credentials to access the share.
Assuming that you provided the correct password to cifscreds, does the
local user (who's running the cifscreds add share) have the
credentials to access the share?

If you believe that all these are setup properly, you could provide us
with the packet trace capture from the time the share was mounted to
when the access was denied.
That could help you give a better insight into what was requested (and
as what user?). If the traffic is encrypted, you may have to provide
the keys for decrypting:
https://wiki.samba.org/index.php/Wireshark_Decryption

Regards,
Shyam

On Tue, Apr 6, 2021 at 3:08 PM Nate Collins <ncollins@xes-inc.com> wrote:
>
> I initially posted this to the Samba mailing list, where it didn't
> receive much attention, but it might be more appropriate for a
> CIFS-specific mailing list. Per the subject, this may be an
> Ubuntu-specific bug, but I don't have the familiarity with CIFS to
> claim this is so and that nothing is wrong with my setup.
>
> ---
>
> This is a problem that's plagued us for a while that we haven't been
> able to resolve due to the lack of familiarity with cifscreds and keyring
> debugging, so I thought I'd ask here to see if there's anything obviously
> wrong with our setup, or if there's any cifscreds/keyring debugging
> advice that could help us.
>
> We currently use multiuser CIFS mounts on a handful of domain-joined
> Ubuntu 16.04 servers to permit users to access CIFS shares with their AD
> credentials. The CIFS shares in question are mounted by a separate service
> account. Everything has been working as expected on the 16.04 servers:
>
> $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"
>
> $ uname -a
> Linux host1 4.4.0-206-generic #238-Ubuntu SMP Tue Mar 16 07:52:37 UTC
> 2021 x86_64 x86_64 x86_64 GNU/Linux
>
> $ dpkg --list | grep cifs
> ii  cifs-utils                            2:6.4-1ubuntu1.1
> amd64        Common Internet File System utilities
>
> $ keyctl show
> Session Keyring
>  [key] --alswrv      0     0  keyring: _ses
>  [key] ---lswrv      0 65534   \_ keyring: _uid.0
>
> $ cifscreds add share
> Password:
>
> $ ls /mnt/share/
> Content
> ...
>
> $ keyctl show
> Session Keyring
>  [key] --alswrv      0     0  keyring: _ses
>  [key] ---lswrv      0 65534   \_ keyring: _uid.0
>  [key] ----sw-v  [uid] [gid]   \_ logon: cifs:a:[share IP]
>
> $ mount | grep multiuser
> //share/share on /mnt/share type cifs
> (rw,relatime,vers=3.0,sec=ntlmsspi,cache=strict,multiuser,domain=[domain],
> uid=0,noforceuid,gid=0,noforcegid,addr=[share IP],file_mode=0755,
> dir_mode=0755,nounix,serverino,mapposix,noperm,rsize=1048576,
> wsize=1048576,echo_interval=60,actimeo=1)
>
> $ grep cifs /etc/pam.d/*
> /etc/pam.d/common-auth:auth    optional            pam_cifscreds.so
> /etc/pam.d/common-session:session optional    pam_cifscreds.so
> host=[domain controller]
>
> $
>
> However, with the exact same setup on an 18.04 and 20.04 server, I am
> unable to access the CIFS mount after running cifscreds add share:
>
> $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=18.04
> DISTRIB_CODENAME=bionic
> DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
>
> $ uname -a
> Linux host2 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> $ dpkg --list | grep cifs
> ii  cifs-utils                            2:6.8-1ubuntu1.1
> amd64        Common Internet File System utilities
>
> $ keyctl show
> Session Keyring
>  [key] --alswrv      0     0  keyring: _ses
>  [key] ---lswrv      0 65534   \_ keyring: _uid.0
>
> $ cifscreds add share
> Password:
>
> $ ls /mnt/share
> ls: cannot access '/mnt/share': Permission denied
>
> $ keyctl show
> Session Keyring
>  [key] --alswrv      0     0  keyring: _ses
>  [key] ---lswrv      0 65534   \_ keyring: _uid.0
>  [key] ----sw-v  [uid] [gid]   \_ logon: cifs:a:[share IP]
>
> $ mount | grep multiuser
> //share/share on /mnt/share type cifs
> (rw,relatime,vers=3.0,sec=ntlmsspi,cache=strict,multiuser,domain=[domain],
> uid=0,noforceuid,gid=0,noforcegid,addr=[share IP],file_mode=0755,
> dir_mode=0755,soft,nounix,serverino,mapposix,noperm,rsize=1048576,
> wsize=1048576,echo_interval=60,actimeo=1)
>
> $
>
> The following error appears in dmesg whenever I try to interact with
> /mnt/share:
>
> [   44.660510] CIFS VFS: signing requested but authenticated as guest
> [   44.663053] CIFS VFS: SMB signature verification returned error = -13
> [   44.663172] CIFS VFS: failed to connect to IPC (rc=-13)
> [   44.664501] CIFS VFS: SMB signature verification returned error = -13
> [   44.665361] CIFS VFS: SMB signature verification returned error = -13
> [   44.665442] CIFS VFS: cifs_put_smb_ses: Session Logoff failure rc=-13
>
> I can't tell if these mssages are significant or not -- I recall reading
> that this was just a change in logging between releases. I tried using
> ntlm through ntlmssp for the mount, but none of those protocols resulted
> in the mountpoint being accessible by my user (they did however get rid of
> the dmesg message). I've tried adjusting other mount options to no avail.
> I can recreate this issue on multiple shares across multiple servers.
>
> I found this bug report:
>
> https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1900856
>
> along with this one:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986168
>
> which seem to be similar issues, except with Kerberos authentication. From
> the first report, I enabled some more verbose debugging:
>
> $ echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
> $ echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
> $ echo 7 > /proc/fs/cifs/cifsFYI
> $ echo 1 > /sys/module/dns_resolver/parameters/debug
>
> Then, when attempting to ls /mnt/share:
>
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2maperror.c: Mapping SMB2
> status code 0xc0000022 to POSIX err -13
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/misc.c: Null buffer passed
> to cifs_small_buf_release
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/connect.c: CIFS VFS: leaving
> cifs_setup_ipc (xid = 109) rc = -13
> CIFS VFS: failed to connect to IPC (rc=-13)
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/connect.c: CIFS VFS: in
> cifs_get_tcon as Xid: 110 with uid: 1001
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2pdu.c: TCON
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/transport.c: Sending smb:
> smb_len=100
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/connect.c: RFC1002 header 0x49
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2misc.c: smb2_check_message
> length: 0x4d, smb_buf_length: 0x49
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2misc.c: SMB2 len 77
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/transport.c:
> cifs_sync_mid_result: cmd=3 mid=41 state=4
> CIFS VFS: SMB signature verification returned error = -13
> Status code returned 0xc0000022 STATUS_ACCESS_DENIED
> /build/linux-Y38gIP/linux-4.15.0/fs/cifs/smb2maperror.c: Mapping SMB2
> status code 0xc0000022 to POSIX err -13
>
> Not sure if these messages are meaningful, but I thought I'd include them.
>
> There's been other cifscreds strangeness that we've noticed, like access
> to mounts not being immediately revoked when `cifscreds clear[all]`
> is ran, but this is the main, easily reproducible issue we've been seeing
> that's preventing us from using multiuser on CIFS mounts.
>
> We use Samba DCs on version 4.12.3-8, and the CIFS share hosts are
> up-to-date TrueNAS servers.



-- 
Regards,
Shyam