Kernels 3.7 and newer break rpc.gssd -n

Kernels 3.7 and newer break rpc.gssd -n

[Veli-Matti Lintu]

I've been using kerberized nfs4 mounts without machine credentials for 
quite some time by running rpc.gssd with the -n option. This has 
resulted rpc.gssd in using ccache in /tmp/krb5cc_0 when doing the mount 
instead of machine credentials. This functionality seems to break when 
using kernel 3.7 or newer. 3.6.11 and earlier work like expected.

The use case for this is diskless workstations that do not have machine 
credentials stored on them as they have no secure storage medium. When a 
user logs in, the home directory is mounted using the user's credentials 
only.

Steps to reproduce the problem:

# kinit user (this creates /tmp/krb5cc_0)
# rpc.gssd -f -n -vvvv
# mount -t nfs4 -o sec=krb5 server.example.org:/home /mnt

The mount works when using kernel 3.6.11 or earlier and fails on 3.7-rc1 
or later. Testing was done on Ubuntu 12.04 and 12.10 using Ubuntu kernels
and kernel.org kernels (up to 3.8-rc7) with similar results.

nfs-utils versions 1.2.5 and the latest version from git master head 
(git://linux-nfs.org/nfs-utils) behave the same way.


rpc.gssd outputs the following when the mount command is given:

3.6.11:

# rpc.gssd -f -n -vvvv
beginning poll
dir_notify_handler: sig 37 si 0xbfb2c4ec data 0xbfb2c56c
dir_notify_handler: sig 37 si 0xbfb2c4ec data 0xbfb2c56c
dir_notify_handler: sig 37 si 0xbfb2c4ec data 0xbfb2c56c
dir_notify_handler: sig 37 si 0xbfb26bac data 0xbfb26c2c
dir_notify_handler: sig 37 si 0xbfb2c4ec data 0xbfb2c56c
handling gssd upcall (/run/rpc_pipefs/nfs/clnt0)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/run/rpc_pipefs/nfs/clnt0)
process_krb5_upcall: service is '<null>'
getting credentials for client with uid 0 for server server.example.org
...

3.7-rc1:

# rpc.gssd -f -n -vvvv
beginning poll
dir_notify_handler: sig 37 si 0xbf9e2d8c data 0xbf9e2e0c
dir_notify_handler: sig 37 si 0xbf9de3cc data 0xbf9de44c
handling gssd upcall (/run/rpc_pipefs/nfs/clnt1a)
handle_gssd_upcall: 'mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/run/rpc_pipefs/nfs/clnt1a)
process_krb5_upcall: service is '*'
...


It seems like the kernel now asks rpc.gssd to fetch credentials for 
service '*' instead of NULL like with 3.6 and earlier.

This leads to method process_krb5_upcall() in 
nfs-utils/utils/gssd/gssd_proc.c where there's a comment saying:

        /*
         * If "service" is specified, then the kernel is indicating that
         * we must use machine credentials for this request.  (Regardless
         * of the uid value or the setting of root_uses_machine_creds.)
         * If the service value is "*", then any service name can be used.
         * Otherwise, it specifies the service name that should be used.
         * (For now, the values of service will only be "*" or "nfs".)
         *
         * Restricting gssd to use "nfs" service name is needed for when
         * the NFS server is doing a callback to the NFS client.  In this
         * case, the NFS server has to authenticate itself as "nfs" --
         * even if there are other service keys such as "host" or "root"
         * in the keytab.
         *
         * Another case when the kernel may specify the service attribute
         * is when gssd is being asked to create the context for a
         * SETCLIENT_ID operation.  In this case, machine credentials
         * must be used for the authentication.  However, the service name
         * used for this case is not important.
         *
         */

It looks like rpc.gssd does not want to load the user credentials 
because the service name is * instead of null like before. When I 
patched gssd_proc.c in gssd sources to ignore the check for 
service_name==NULL, rpc.gssd returned the user credentials and the nfs 
mount succeeded.

Is this something that should be still be supported? Any pointers where I 
should start looking for a fix?

Veli-Matti

Re: Kernels 3.7 and newer break rpc.gssd -n

[Veli-Matti Lintu]

> I've been using kerberized nfs4 mounts without machine credentials for
> quite some time by running rpc.gssd with the -n option. This has
> resulted rpc.gssd in using ccache in /tmp/krb5cc_0 when doing the mount
> instead of machine credentials. This functionality seems to break when
> using kernel 3.7 or newer. 3.6.11 and earlier work like expected.
> 
> The use case for this is diskless workstations that do not have machine
> credentials stored on them as they have no secure storage medium. When a
> user logs in, the home directory is mounted using the user's credentials
> only.
> 
> Steps to reproduce the problem:
> 
> # kinit user (this creates /tmp/krb5cc_0)
> # rpc.gssd -f -n -vvvv
> # mount -t nfs4 -o sec=krb5 server.example.org:/home /mnt
> 
> The mount works when using kernel 3.6.11 or earlier and fails on 3.7-rc1
> or later. Testing was done on Ubuntu 12.04 and 12.10 using Ubuntu kernels
> and kernel.org kernels (up to 3.8-rc7) with similar results.
> 
> nfs-utils versions 1.2.5 and the latest version from git master head
> (git://linux-nfs.org/nfs-utils) behave the same way.

...

> It seems like the kernel now asks rpc.gssd to fetch credentials for
> service '*' instead of NULL like with 3.6 and earlier.

I got some bisecting help from Tuomas Räsänen who managed to find the 
commit that introduces the change. Here's what he found. I haven't 
yet figured out what in the commit causes the change in behaviour. 
Before that patch the kernel asks rpc.gssd for service <null>, but 
with the patch applied, it requests for service '*' when doing the 
initial mount with sec=krb5.

Veli-Matti

--------------------------------------------------------------

I bisected between v3.6 (good) and v3.7-rc1 (bad) and it seems
that the first commit which introduces the change Veli-Matti
described in his mail is:

  commit 05f4c350ee02e9461c6ae3a880ea326a06835e37
  Author: Chuck Lever <chuck.lever@oracle.com>
  Date:   Fri Sep 14 17:24:32 2012 -0400
  
      NFS: Discover NFSv4 server trunking when mounting

That very same commit introduces also a compilation error, which is
fixed in the merge commit a bit later in the tree
(9f62387d6e26532bcbfb15606956074192ee526a). Therefore the bisect log
below results in multiple skips. I picked the patch from
9f62387d6e26532bcbfb15606956074192ee526a and applied it to
05f4c350ee02e9461c6ae3a880ea326a06835e37, and then bisected the rest.

git bisect start
# bad: [ddffeb8c4d0331609ef2581d84de4d763607bd37] Linux 3.7-rc1
git bisect bad ddffeb8c4d0331609ef2581d84de4d763607bd37
# good: [a0d271cbfed1dd50278c6b06bead3d00ba0a88f9] Linux 3.6
git bisect good a0d271cbfed1dd50278c6b06bead3d00ba0a88f9
# good: [24d7b40a60cf19008334bcbcbd98da374d4d9c64] ARM: OMAP2+: PM: MPU DVFS: use generic CPU device for MPU-SS
git bisect good 24d7b40a60cf19008334bcbcbd98da374d4d9c64
# good: [21c8715f0a1f4df8bfa2bd6f3915e5e33c1c2e6e] ARM: integrator: use __iomem pointers for MMIO, part 2
git bisect good 21c8715f0a1f4df8bfa2bd6f3915e5e33c1c2e6e
# good: [23d5385f382a7c7d8b6bf19b0c2cfb3acbb12d31] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc
git bisect good 23d5385f382a7c7d8b6bf19b0c2cfb3acbb12d31
# bad: [35e9a274fdc9c8feb763e4970a32d7089f51393c] Merge branch 'kconfig' of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
git bisect bad 35e9a274fdc9c8feb763e4970a32d7089f51393c
# bad: [ba0a5a36f60e4c1152af3a2ae2813251974405bf] Merge tag 'firewire-fix' of git://git.kernel.org/pub/scm/linux/kernel/git/ieee1394/linux1394
git bisect bad ba0a5a36f60e4c1152af3a2ae2813251974405bf
# good: [a188e7e93a36627fb3f0013f41857ab54f076d04] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending
git bisect good a188e7e93a36627fb3f0013f41857ab54f076d04
# bad: [ca57ccc48f6a9a3ec655f87acebab82bf01088e7] nfs: include NFSv4 header in netns.h
git bisect bad ca57ccc48f6a9a3ec655f87acebab82bf01088e7
# skip: [fcb6d9c6b719b633e9b98d26d8a7937209e8bf21] NFS: Always use the open stateid when checking for expired opens
git bisect skip fcb6d9c6b719b633e9b98d26d8a7937209e8bf21
# skip: [78e4e05c643768af170e5a4b21712d9a7a26cce5] NFSv4.1: Replace get_device_info() with filelayout_get_device_info()
git bisect skip 78e4e05c643768af170e5a4b21712d9a7a26cce5
# skip: [c8ceb4124b53a439edfe3fe89a646be1e067ef17] NFS: pass net to nfs_callback_down()
git bisect skip c8ceb4124b53a439edfe3fe89a646be1e067ef17
# skip: [e984a55a7418f777407c7edbb2bdf5eb9559b5e2] NFS: Use the same nfs_client_id4 for every server
git bisect skip e984a55a7418f777407c7edbb2bdf5eb9559b5e2
# skip: [65857d5768f7716da539933c2075d384b117812d] NFSv4.1: _pnfs_return_layout() shouldn't invalidate the layout on failure
git bisect skip 65857d5768f7716da539933c2075d384b117812d
# good: [896526174ce2b6a773e187ebe5a047b68230e2c4] NFS: Introduce "migration" mount option
git bisect good 896526174ce2b6a773e187ebe5a047b68230e2c4
# skip: [4e266229dbb0782d91b75633322edd632794b86d] pnfsblock: use list_move_tail instead of list_del/list_add_tail
git bisect skip 4e266229dbb0782d91b75633322edd632794b86d
# skip: [758201e2c94b7d26ea0ac64e55cab1d53742780a] NFSv4: Fix the minor version callback channel startup
git bisect skip 758201e2c94b7d26ea0ac64e55cab1d53742780a
# skip: [7297cb682acb506ada2e01fbc9b447b04d69936c] nfs: replace strict_strto* with kstrto*
git bisect skip 7297cb682acb506ada2e01fbc9b447b04d69936c
# skip: [6f2ea7f2a3ff3cd342bface43f8b4bf5e431cf36] NFS: Add nfs4_unique_id boot parameter
git bisect skip 6f2ea7f2a3ff3cd342bface43f8b4bf5e431cf36
# skip: [96c9eae638765c2bf2ca4f5a6325484f9bb69aa7] pnfsblock: fix non-aligned DIO write
git bisect skip 96c9eae638765c2bf2ca4f5a6325484f9bb69aa7
# good: [0cac120233305b614cfe3ad419f3655876066017] NFSv4: Ensure that idmap_pipe_downcall sanity-checks the downcall data
git bisect good 0cac120233305b614cfe3ad419f3655876066017
# skip: [f742dc4a32587bff50b13dde9d8894b96851951a] pnfsblock: fix non-aligned DIO read
git bisect skip f742dc4a32587bff50b13dde9d8894b96851951a
# skip: [7acdb026818455638543b04b68d4a580c367fba8] NFSv41: fix DIO write_io calculation
git bisect skip 7acdb026818455638543b04b68d4a580c367fba8
# skip: [ee34e13620d0678d420ce50101aaef94ab81fc74] NFS: Remove unnecessary semicolons (fs/nfs/client.c)
git bisect skip ee34e13620d0678d420ce50101aaef94ab81fc74
# skip: [dc182549d439f60c332bf74d7f220a1bccf37da6] NFS41: fix error of setting blocklayoutdriver
git bisect skip dc182549d439f60c332bf74d7f220a1bccf37da6
# skip: [fe6e1e8d9fad86873eb74a26e80a8f91f9e870b5] pnfsblock: fix partial page buffer wirte
git bisect skip fe6e1e8d9fad86873eb74a26e80a8f91f9e870b5
# skip: [05f4c350ee02e9461c6ae3a880ea326a06835e37] NFS: Discover NFSv4 server trunking when mounting
git bisect skip 05f4c350ee02e9461c6ae3a880ea326a06835e37
# skip: [5d0e3a004f02bffab51f542fa1d5b2e2854d8545] Revert "pnfsblock: bail out partial page IO"
git bisect skip 5d0e3a004f02bffab51f542fa1d5b2e2854d8545
# bad: [9f62387d6e26532bcbfb15606956074192ee526a] NFSv4: Fix up a merge conflict between migration and container changes
git bisect bad 9f62387d6e26532bcbfb15606956074192ee526a
# skip: [2afdfa5a846246de50e1881f71ba5c0aac0b415f] Merge branch 'bugfixes' into nfs-for-next
git bisect skip 2afdfa5a846246de50e1881f71ba5c0aac0b415f

-- 
Tuomas

--------------------------------------------------------------

Re: Kernels 3.7 and newer break rpc.gssd -n

[Chuck Lever]

On Feb 14, 2013, at 9:24 AM, Veli-Matti Lintu <veli-matti.lintu@opinsys.fi> wrote:

> 
>> I've been using kerberized nfs4 mounts without machine credentials for
>> quite some time by running rpc.gssd with the -n option. This has
>> resulted rpc.gssd in using ccache in /tmp/krb5cc_0 when doing the mount
>> instead of machine credentials. This functionality seems to break when
>> using kernel 3.7 or newer. 3.6.11 and earlier work like expected.
>> 
>> The use case for this is diskless workstations that do not have machine
>> credentials stored on them as they have no secure storage medium. When a
>> user logs in, the home directory is mounted using the user's credentials
>> only.
>> 
>> Steps to reproduce the problem:
>> 
>> # kinit user (this creates /tmp/krb5cc_0)
>> # rpc.gssd -f -n -vvvv
>> # mount -t nfs4 -o sec=krb5 server.example.org:/home /mnt
>> 
>> The mount works when using kernel 3.6.11 or earlier and fails on 3.7-rc1
>> or later. Testing was done on Ubuntu 12.04 and 12.10 using Ubuntu kernels
>> and kernel.org kernels (up to 3.8-rc7) with similar results.
>> 
>> nfs-utils versions 1.2.5 and the latest version from git master head
>> (git://linux-nfs.org/nfs-utils) behave the same way.
> 
> ...
> 
>> It seems like the kernel now asks rpc.gssd to fetch credentials for
>> service '*' instead of NULL like with 3.6 and earlier.
> 
> I got some bisecting help from Tuomas Räsänen who managed to find the 
> commit that introduces the change. Here's what he found. I haven't 
> yet figured out what in the commit causes the change in behaviour. 
> Before that patch the kernel asks rpc.gssd for service <null>, but 
> with the patch applied, it requests for service '*' when doing the 
> initial mount with sec=krb5.

Have you tried a kernel at commit 05f4c350 but with the compilation fix applied, then one with 05f4c350 removed, to confirm that this indeed is the commit that introduces the problem?

When the mount operation fails, is it the first time this client attempts to mount a share on server.example.org, or does the client already have mounts of server.example.org, possibly using other security flavors?

I will attempt to reproduce this.

> Veli-Matti
> 
> --------------------------------------------------------------
> 
> I bisected between v3.6 (good) and v3.7-rc1 (bad) and it seems
> that the first commit which introduces the change Veli-Matti
> described in his mail is:
> 
>  commit 05f4c350ee02e9461c6ae3a880ea326a06835e37
>  Author: Chuck Lever <chuck.lever@oracle.com>
>  Date:   Fri Sep 14 17:24:32 2012 -0400
> 
>      NFS: Discover NFSv4 server trunking when mounting
> 
> That very same commit introduces also a compilation error, which is
> fixed in the merge commit a bit later in the tree
> (9f62387d6e26532bcbfb15606956074192ee526a). Therefore the bisect log
> below results in multiple skips. I picked the patch from
> 9f62387d6e26532bcbfb15606956074192ee526a and applied it to
> 05f4c350ee02e9461c6ae3a880ea326a06835e37, and then bisected the rest.
> 
> git bisect start
> # bad: [ddffeb8c4d0331609ef2581d84de4d763607bd37] Linux 3.7-rc1
> git bisect bad ddffeb8c4d0331609ef2581d84de4d763607bd37
> # good: [a0d271cbfed1dd50278c6b06bead3d00ba0a88f9] Linux 3.6
> git bisect good a0d271cbfed1dd50278c6b06bead3d00ba0a88f9
> # good: [24d7b40a60cf19008334bcbcbd98da374d4d9c64] ARM: OMAP2+: PM: MPU DVFS: use generic CPU device for MPU-SS
> git bisect good 24d7b40a60cf19008334bcbcbd98da374d4d9c64
> # good: [21c8715f0a1f4df8bfa2bd6f3915e5e33c1c2e6e] ARM: integrator: use __iomem pointers for MMIO, part 2
> git bisect good 21c8715f0a1f4df8bfa2bd6f3915e5e33c1c2e6e
> # good: [23d5385f382a7c7d8b6bf19b0c2cfb3acbb12d31] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc
> git bisect good 23d5385f382a7c7d8b6bf19b0c2cfb3acbb12d31
> # bad: [35e9a274fdc9c8feb763e4970a32d7089f51393c] Merge branch 'kconfig' of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild
> git bisect bad 35e9a274fdc9c8feb763e4970a32d7089f51393c
> # bad: [ba0a5a36f60e4c1152af3a2ae2813251974405bf] Merge tag 'firewire-fix' of git://git.kernel.org/pub/scm/linux/kernel/git/ieee1394/linux1394
> git bisect bad ba0a5a36f60e4c1152af3a2ae2813251974405bf
> # good: [a188e7e93a36627fb3f0013f41857ab54f076d04] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending
> git bisect good a188e7e93a36627fb3f0013f41857ab54f076d04
> # bad: [ca57ccc48f6a9a3ec655f87acebab82bf01088e7] nfs: include NFSv4 header in netns.h
> git bisect bad ca57ccc48f6a9a3ec655f87acebab82bf01088e7
> # skip: [fcb6d9c6b719b633e9b98d26d8a7937209e8bf21] NFS: Always use the open stateid when checking for expired opens
> git bisect skip fcb6d9c6b719b633e9b98d26d8a7937209e8bf21
> # skip: [78e4e05c643768af170e5a4b21712d9a7a26cce5] NFSv4.1: Replace get_device_info() with filelayout_get_device_info()
> git bisect skip 78e4e05c643768af170e5a4b21712d9a7a26cce5
> # skip: [c8ceb4124b53a439edfe3fe89a646be1e067ef17] NFS: pass net to nfs_callback_down()
> git bisect skip c8ceb4124b53a439edfe3fe89a646be1e067ef17
> # skip: [e984a55a7418f777407c7edbb2bdf5eb9559b5e2] NFS: Use the same nfs_client_id4 for every server
> git bisect skip e984a55a7418f777407c7edbb2bdf5eb9559b5e2
> # skip: [65857d5768f7716da539933c2075d384b117812d] NFSv4.1: _pnfs_return_layout() shouldn't invalidate the layout on failure
> git bisect skip 65857d5768f7716da539933c2075d384b117812d
> # good: [896526174ce2b6a773e187ebe5a047b68230e2c4] NFS: Introduce "migration" mount option
> git bisect good 896526174ce2b6a773e187ebe5a047b68230e2c4
> # skip: [4e266229dbb0782d91b75633322edd632794b86d] pnfsblock: use list_move_tail instead of list_del/list_add_tail
> git bisect skip 4e266229dbb0782d91b75633322edd632794b86d
> # skip: [758201e2c94b7d26ea0ac64e55cab1d53742780a] NFSv4: Fix the minor version callback channel startup
> git bisect skip 758201e2c94b7d26ea0ac64e55cab1d53742780a
> # skip: [7297cb682acb506ada2e01fbc9b447b04d69936c] nfs: replace strict_strto* with kstrto*
> git bisect skip 7297cb682acb506ada2e01fbc9b447b04d69936c
> # skip: [6f2ea7f2a3ff3cd342bface43f8b4bf5e431cf36] NFS: Add nfs4_unique_id boot parameter
> git bisect skip 6f2ea7f2a3ff3cd342bface43f8b4bf5e431cf36
> # skip: [96c9eae638765c2bf2ca4f5a6325484f9bb69aa7] pnfsblock: fix non-aligned DIO write
> git bisect skip 96c9eae638765c2bf2ca4f5a6325484f9bb69aa7
> # good: [0cac120233305b614cfe3ad419f3655876066017] NFSv4: Ensure that idmap_pipe_downcall sanity-checks the downcall data
> git bisect good 0cac120233305b614cfe3ad419f3655876066017
> # skip: [f742dc4a32587bff50b13dde9d8894b96851951a] pnfsblock: fix non-aligned DIO read
> git bisect skip f742dc4a32587bff50b13dde9d8894b96851951a
> # skip: [7acdb026818455638543b04b68d4a580c367fba8] NFSv41: fix DIO write_io calculation
> git bisect skip 7acdb026818455638543b04b68d4a580c367fba8
> # skip: [ee34e13620d0678d420ce50101aaef94ab81fc74] NFS: Remove unnecessary semicolons (fs/nfs/client.c)
> git bisect skip ee34e13620d0678d420ce50101aaef94ab81fc74
> # skip: [dc182549d439f60c332bf74d7f220a1bccf37da6] NFS41: fix error of setting blocklayoutdriver
> git bisect skip dc182549d439f60c332bf74d7f220a1bccf37da6
> # skip: [fe6e1e8d9fad86873eb74a26e80a8f91f9e870b5] pnfsblock: fix partial page buffer wirte
> git bisect skip fe6e1e8d9fad86873eb74a26e80a8f91f9e870b5
> # skip: [05f4c350ee02e9461c6ae3a880ea326a06835e37] NFS: Discover NFSv4 server trunking when mounting
> git bisect skip 05f4c350ee02e9461c6ae3a880ea326a06835e37
> # skip: [5d0e3a004f02bffab51f542fa1d5b2e2854d8545] Revert "pnfsblock: bail out partial page IO"
> git bisect skip 5d0e3a004f02bffab51f542fa1d5b2e2854d8545
> # bad: [9f62387d6e26532bcbfb15606956074192ee526a] NFSv4: Fix up a merge conflict between migration and container changes
> git bisect bad 9f62387d6e26532bcbfb15606956074192ee526a
> # skip: [2afdfa5a846246de50e1881f71ba5c0aac0b415f] Merge branch 'bugfixes' into nfs-for-next
> git bisect skip 2afdfa5a846246de50e1881f71ba5c0aac0b415f
> 
> -- 
> Tuomas
> 
> --------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Kernels 3.7 and newer break rpc.gssd -n

[Tuomas Räsänen]

----- Original Message -----
> From: "Chuck Lever" <chuck.lever@oracle.com>
> 
> Have you tried a kernel at commit 05f4c350 but with the compilation fix
> applied, then one with 05f4c350 removed, to confirm that this indeed is the
> commit that introduces the problem?

Yes. With commit 05f4c350 + the compilation fix, the problem occurs, but without
05f4c350, the parent (6f2ea7f) works as expected, requesting for service <null>.

> 
> When the mount operation fails, is it the first time this client attempts to
> mount a share on server.example.org, or does the client already have mounts
> of server.example.org, possibly using other security flavors?

Yes, the problem occurs on the very first mount attempt.

-- 
Tuomas

Re: Kernels 3.7 and newer break rpc.gssd -n

[Chuck Lever]

On Feb 15, 2013, at 7:00 AM, Tuomas Räsänen <tuomasjjrasanen@opinsys.fi> wrote:

> ----- Original Message -----
>> From: "Chuck Lever" <chuck.lever@oracle.com>
>> 
>> Have you tried a kernel at commit 05f4c350 but with the compilation fix
>> applied, then one with 05f4c350 removed, to confirm that this indeed is the
>> commit that introduces the problem?
> 
> Yes. With commit 05f4c350 + the compilation fix, the problem occurs, but without
> 05f4c350, the parent (6f2ea7f) works as expected, requesting for service <null>.
> 
>> 
>> When the mount operation fails, is it the first time this client attempts to
>> mount a share on server.example.org, or does the client already have mounts
>> of server.example.org, possibly using other security flavors?
> 
> Yes, the problem occurs on the very first mount attempt.

Thanks for confirming.  Did you try updating nfs-utils on your NFS client to 1.2.8-rc1 or later (specifically to replace rpc.gssd)?

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Kernels 3.7 and newer break rpc.gssd -n

[Veli-Matti Lintu]

> > Yes. With commit 05f4c350 + the compilation fix, the problem occurs, but
> > without
> > 05f4c350, the parent (6f2ea7f) works as expected, requesting for service
> > <null>.

> >> When the mount operation fails, is it the first time this client attempts
> >> to
> >> mount a share on server.example.org, or does the client already have
> >> mounts
> >> of server.example.org, possibly using other security flavors?

> > Yes, the problem occurs on the very first mount attempt.
> 
> Thanks for confirming.  Did you try updating nfs-utils on your NFS client to
> 1.2.8-rc1 or later (specifically to replace rpc.gssd)?

Yes, gssd from nfs-utils 1.2.8-rc3 behaves the same way and fails when service 
name is not null.

The code in gssd that checks for the service name seems to be the same if I
understand the code correctly.

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=utils/gssd/gssd_proc.c;h=c17ab3bf914526f433fb6c76ace1daa63c10d921;hb=HEAD#l985

---------------------------------------------------------------------------------
        if (uid != 0 || (uid == 0 && root_uses_machine_creds == 0 &&
                                service == NULL)) {
                /* Tell krb5 gss which credentials cache to use */
                for (dirname = ccachesearch; *dirname != NULL; dirname++) {
                        err = gssd_setup_krb5_user_gss_ccache(uid, clp->servername, *dirname);
                        if (err == -EKEYEXPIRED)
                                downcall_err = -EKEYEXPIRED;
                        else if (!err)
                                create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
                                                             AUTHTYPE_KRB5);
                        if (create_resp == 0)
                                break;
                }
        }
---------------------------------------------------------------------------------

Veli-Matti

Re: Kernels 3.7 and newer break rpc.gssd -n

[Chuck Lever]

On Feb 18, 2013, at 3:23 AM, Veli-Matti Lintu <veli-matti.lintu@opinsys.fi> wrote:

>>> Yes. With commit 05f4c350 + the compilation fix, the problem occurs, but
>>> without
>>> 05f4c350, the parent (6f2ea7f) works as expected, requesting for service
>>> <null>.
> 
>>>> When the mount operation fails, is it the first time this client attempts
>>>> to
>>>> mount a share on server.example.org, or does the client already have
>>>> mounts
>>>> of server.example.org, possibly using other security flavors?
> 
>>> Yes, the problem occurs on the very first mount attempt.
>> 
>> Thanks for confirming.  Did you try updating nfs-utils on your NFS client to
>> 1.2.8-rc1 or later (specifically to replace rpc.gssd)?
> 
> Yes, gssd from nfs-utils 1.2.8-rc3 behaves the same way and fails when service 
> name is not null.

Thanks for confirming.  My KDC is now back online so I will try to reproduce this today.


> The code in gssd that checks for the service name seems to be the same if I
> understand the code correctly.
> 
> http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=utils/gssd/gssd_proc.c;h=c17ab3bf914526f433fb6c76ace1daa63c10d921;hb=HEAD#l985
> 
> ---------------------------------------------------------------------------------
>        if (uid != 0 || (uid == 0 && root_uses_machine_creds == 0 &&
>                                service == NULL)) {
>                /* Tell krb5 gss which credentials cache to use */
>                for (dirname = ccachesearch; *dirname != NULL; dirname++) {
>                        err = gssd_setup_krb5_user_gss_ccache(uid, clp->servername, *dirname);
>                        if (err == -EKEYEXPIRED)
>                                downcall_err = -EKEYEXPIRED;
>                        else if (!err)
>                                create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
>                                                             AUTHTYPE_KRB5);
>                        if (create_resp == 0)
>                                break;
>                }
>        }
> ---------------------------------------------------------------------------------
> 
> Veli-Matti
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Kernels 3.7 and newer break rpc.gssd -n

[Chuck Lever]

On Feb 14, 2013, at 9:24 AM, Veli-Matti Lintu <veli-matti.lintu@opinsys.fi> wrote:

> 
>> I've been using kerberized nfs4 mounts without machine credentials for
>> quite some time by running rpc.gssd with the -n option. This has
>> resulted rpc.gssd in using ccache in /tmp/krb5cc_0 when doing the mount
>> instead of machine credentials. This functionality seems to break when
>> using kernel 3.7 or newer. 3.6.11 and earlier work like expected.
>> 
>> The use case for this is diskless workstations that do not have machine
>> credentials stored on them as they have no secure storage medium. When a
>> user logs in, the home directory is mounted using the user's credentials
>> only.
>> 
>> Steps to reproduce the problem:
>> 
>> # kinit user (this creates /tmp/krb5cc_0)
>> # rpc.gssd -f -n -vvvv
>> # mount -t nfs4 -o sec=krb5 server.example.org:/home /mnt
>> 
>> The mount works when using kernel 3.6.11 or earlier and fails on 3.7-rc1
>> or later. Testing was done on Ubuntu 12.04 and 12.10 using Ubuntu kernels
>> and kernel.org kernels (up to 3.8-rc7) with similar results.
>> 
>> nfs-utils versions 1.2.5 and the latest version from git master head
>> (git://linux-nfs.org/nfs-utils) behave the same way.
> 
> ...
> 
>> It seems like the kernel now asks rpc.gssd to fetch credentials for
>> service '*' instead of NULL like with 3.6 and earlier.
> 
> I got some bisecting help from Tuomas Räsänen who managed to find the 
> commit that introduces the change. Here's what he found. I haven't 
> yet figured out what in the commit causes the change in behaviour. 
> Before that patch the kernel asks rpc.gssd for service <null>, but 
> with the patch applied, it requests for service '*' when doing the 
> initial mount with sec=krb5.

While I am trying to reproduce this, have a look at commit a56989b6 in nfs-utils 1.2.8-rc1.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Kernels 3.7 and newer break rpc.gssd -n

[Chuck Lever]

On Feb 13, 2013, at 10:29 AM, Veli-Matti Lintu <veli-matti.lintu@opinsys.fi> wrote:

> I've been using kerberized nfs4 mounts without machine credentials for 
> quite some time by running rpc.gssd with the -n option. This has 
> resulted rpc.gssd in using ccache in /tmp/krb5cc_0 when doing the mount 
> instead of machine credentials. This functionality seems to break when 
> using kernel 3.7 or newer. 3.6.11 and earlier work like expected.
> 
> The use case for this is diskless workstations that do not have machine 
> credentials stored on them as they have no secure storage medium. When a 
> user logs in, the home directory is mounted using the user's credentials 
> only.
> 
> Steps to reproduce the problem:
> 
> # kinit user (this creates /tmp/krb5cc_0)
> # rpc.gssd -f -n -vvvv
> # mount -t nfs4 -o sec=krb5 server.example.org:/home /mnt
> 
> The mount works when using kernel 3.6.11 or earlier and fails on 3.7-rc1 
> or later. Testing was done on Ubuntu 12.04 and 12.10 using Ubuntu kernels
> and kernel.org kernels (up to 3.8-rc7) with similar results.
> 
> nfs-utils versions 1.2.5 and the latest version from git master head 
> (git://linux-nfs.org/nfs-utils) behave the same way.

Reproduced.  Not clear yet if this is a kernel regression or a latent gssd bug.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Kernels 3.7 and newer break rpc.gssd -n

[Chuck Lever]

On Feb 18, 2013, at 4:16 PM, Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Feb 13, 2013, at 10:29 AM, Veli-Matti Lintu <veli-matti.lintu@opinsys.fi> wrote:
> 
>> I've been using kerberized nfs4 mounts without machine credentials for 
>> quite some time by running rpc.gssd with the -n option. This has 
>> resulted rpc.gssd in using ccache in /tmp/krb5cc_0 when doing the mount 
>> instead of machine credentials. This functionality seems to break when 
>> using kernel 3.7 or newer. 3.6.11 and earlier work like expected.
>> 
>> The use case for this is diskless workstations that do not have machine 
>> credentials stored on them as they have no secure storage medium. When a 
>> user logs in, the home directory is mounted using the user's credentials 
>> only.
>> 
>> Steps to reproduce the problem:
>> 
>> # kinit user (this creates /tmp/krb5cc_0)
>> # rpc.gssd -f -n -vvvv
>> # mount -t nfs4 -o sec=krb5 server.example.org:/home /mnt
>> 
>> The mount works when using kernel 3.6.11 or earlier and fails on 3.7-rc1 
>> or later. Testing was done on Ubuntu 12.04 and 12.10 using Ubuntu kernels
>> and kernel.org kernels (up to 3.8-rc7) with similar results.
>> 
>> nfs-utils versions 1.2.5 and the latest version from git master head 
>> (git://linux-nfs.org/nfs-utils) behave the same way.
> 
> Reproduced.  Not clear yet if this is a kernel regression or a latent gssd bug.

With pre-3.7 kernels, kernel upcalls with "service=<null>" because the first operation is a LOOKUP_ROOT, and a machine credential is not needed.  With 3.7, kernel is upcalling to get a context to perform SETCLIENTID, and it's specifying "service='*'" as a way to get machine credentials.

In the case where there is no keytab and rpc.gssd is invoked with "-n", gssd is trying to use /etc/krb5.keytab anyway, and failing.

"man rpc.gssd" says this about "-n":

> By default, rpc.gssd treats accesses by the user with UID 0  specially,
> and  uses  "machine  credentials"  for  all accesses by that user which
> require Kerberos authentication.  With the -n option, "machine  creden‐
> tials"  will  not  be used for accesses by UID 0.  Instead, credentials
> must be obtained manually like all other users.   Use  of  this  option
> means  that  "root"  must  manually  obtain Kerberos credentials before
> attempting to mount an nfs filesystem  requiring  Kerberos  authentica‐
> tion.

It seems to me that if the kernel asks for "service='*'" and gssd was invoked with "-n", gssd should retrieve the manually-obtained Kerberos credentials in that case.

I'm staring at process_krb5_upcall(), just past that big block comment around line 962.  I think that logic is probably wrong to check for "service_name==NULL" but should rather check that the service_name is not "nfs".  This a fix that Veli-Matti hinted at earlier.

Thoughts?

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: Kernels 3.7 and newer break rpc.gssd -n

[Myklebust, Trond]

T24gTW9uLCAyMDEzLTAyLTE4IGF0IDE3OjQ4IC0wNTAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
T24gRmViIDE4LCAyMDEzLCBhdCA0OjE2IFBNLCBDaHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3Jh
Y2xlLmNvbT4gd3JvdGU6DQo+IA0KPiA+IA0KPiA+IE9uIEZlYiAxMywgMjAxMywgYXQgMTA6Mjkg
QU0sIFZlbGktTWF0dGkgTGludHUgPHZlbGktbWF0dGkubGludHVAb3BpbnN5cy5maT4gd3JvdGU6
DQo+ID4gDQo+ID4+IEkndmUgYmVlbiB1c2luZyBrZXJiZXJpemVkIG5mczQgbW91bnRzIHdpdGhv
dXQgbWFjaGluZSBjcmVkZW50aWFscyBmb3IgDQo+ID4+IHF1aXRlIHNvbWUgdGltZSBieSBydW5u
aW5nIHJwYy5nc3NkIHdpdGggdGhlIC1uIG9wdGlvbi4gVGhpcyBoYXMgDQo+ID4+IHJlc3VsdGVk
IHJwYy5nc3NkIGluIHVzaW5nIGNjYWNoZSBpbiAvdG1wL2tyYjVjY18wIHdoZW4gZG9pbmcgdGhl
IG1vdW50IA0KPiA+PiBpbnN0ZWFkIG9mIG1hY2hpbmUgY3JlZGVudGlhbHMuIFRoaXMgZnVuY3Rp
b25hbGl0eSBzZWVtcyB0byBicmVhayB3aGVuIA0KPiA+PiB1c2luZyBrZXJuZWwgMy43IG9yIG5l
d2VyLiAzLjYuMTEgYW5kIGVhcmxpZXIgd29yayBsaWtlIGV4cGVjdGVkLg0KPiA+PiANCj4gPj4g
VGhlIHVzZSBjYXNlIGZvciB0aGlzIGlzIGRpc2tsZXNzIHdvcmtzdGF0aW9ucyB0aGF0IGRvIG5v
dCBoYXZlIG1hY2hpbmUgDQo+ID4+IGNyZWRlbnRpYWxzIHN0b3JlZCBvbiB0aGVtIGFzIHRoZXkg
aGF2ZSBubyBzZWN1cmUgc3RvcmFnZSBtZWRpdW0uIFdoZW4gYSANCj4gPj4gdXNlciBsb2dzIGlu
LCB0aGUgaG9tZSBkaXJlY3RvcnkgaXMgbW91bnRlZCB1c2luZyB0aGUgdXNlcidzIGNyZWRlbnRp
YWxzIA0KPiA+PiBvbmx5Lg0KPiA+PiANCj4gPj4gU3RlcHMgdG8gcmVwcm9kdWNlIHRoZSBwcm9i
bGVtOg0KPiA+PiANCj4gPj4gIyBraW5pdCB1c2VyICh0aGlzIGNyZWF0ZXMgL3RtcC9rcmI1Y2Nf
MCkNCj4gPj4gIyBycGMuZ3NzZCAtZiAtbiAtdnZ2dg0KPiA+PiAjIG1vdW50IC10IG5mczQgLW8g
c2VjPWtyYjUgc2VydmVyLmV4YW1wbGUub3JnOi9ob21lIC9tbnQNCj4gPj4gDQo+ID4+IFRoZSBt
b3VudCB3b3JrcyB3aGVuIHVzaW5nIGtlcm5lbCAzLjYuMTEgb3IgZWFybGllciBhbmQgZmFpbHMg
b24gMy43LXJjMSANCj4gPj4gb3IgbGF0ZXIuIFRlc3Rpbmcgd2FzIGRvbmUgb24gVWJ1bnR1IDEy
LjA0IGFuZCAxMi4xMCB1c2luZyBVYnVudHUga2VybmVscw0KPiA+PiBhbmQga2VybmVsLm9yZyBr
ZXJuZWxzICh1cCB0byAzLjgtcmM3KSB3aXRoIHNpbWlsYXIgcmVzdWx0cy4NCj4gPj4gDQo+ID4+
IG5mcy11dGlscyB2ZXJzaW9ucyAxLjIuNSBhbmQgdGhlIGxhdGVzdCB2ZXJzaW9uIGZyb20gZ2l0
IG1hc3RlciBoZWFkIA0KPiA+PiAoZ2l0Oi8vbGludXgtbmZzLm9yZy9uZnMtdXRpbHMpIGJlaGF2
ZSB0aGUgc2FtZSB3YXkuDQo+ID4gDQo+ID4gUmVwcm9kdWNlZC4gIE5vdCBjbGVhciB5ZXQgaWYg
dGhpcyBpcyBhIGtlcm5lbCByZWdyZXNzaW9uIG9yIGEgbGF0ZW50IGdzc2QgYnVnLg0KPiANCj4g
V2l0aCBwcmUtMy43IGtlcm5lbHMsIGtlcm5lbCB1cGNhbGxzIHdpdGggInNlcnZpY2U9PG51bGw+
IiBiZWNhdXNlIHRoZSBmaXJzdCBvcGVyYXRpb24gaXMgYSBMT09LVVBfUk9PVCwgYW5kIGEgbWFj
aGluZSBjcmVkZW50aWFsIGlzIG5vdCBuZWVkZWQuICBXaXRoIDMuNywga2VybmVsIGlzIHVwY2Fs
bGluZyB0byBnZXQgYSBjb250ZXh0IHRvIHBlcmZvcm0gU0VUQ0xJRU5USUQsIGFuZCBpdCdzIHNw
ZWNpZnlpbmcgInNlcnZpY2U9JyonIiBhcyBhIHdheSB0byBnZXQgbWFjaGluZSBjcmVkZW50aWFs
cy4NCj4gDQo+IEluIHRoZSBjYXNlIHdoZXJlIHRoZXJlIGlzIG5vIGtleXRhYiBhbmQgcnBjLmdz
c2QgaXMgaW52b2tlZCB3aXRoICItbiIsIGdzc2QgaXMgdHJ5aW5nIHRvIHVzZSAvZXRjL2tyYjUu
a2V5dGFiIGFueXdheSwgYW5kIGZhaWxpbmcuDQo+IA0KPiAibWFuIHJwYy5nc3NkIiBzYXlzIHRo
aXMgYWJvdXQgIi1uIjoNCj4gDQo+ID4gQnkgZGVmYXVsdCwgcnBjLmdzc2QgdHJlYXRzIGFjY2Vz
c2VzIGJ5IHRoZSB1c2VyIHdpdGggVUlEIDAgIHNwZWNpYWxseSwNCj4gPiBhbmQgIHVzZXMgICJt
YWNoaW5lICBjcmVkZW50aWFscyIgIGZvciAgYWxsIGFjY2Vzc2VzIGJ5IHRoYXQgdXNlciB3aGlj
aA0KPiA+IHJlcXVpcmUgS2VyYmVyb3MgYXV0aGVudGljYXRpb24uICBXaXRoIHRoZSAtbiBvcHRp
b24sICJtYWNoaW5lICBjcmVkZW7igJANCj4gPiB0aWFscyIgIHdpbGwgIG5vdCAgYmUgdXNlZCBm
b3IgYWNjZXNzZXMgYnkgVUlEIDAuICBJbnN0ZWFkLCBjcmVkZW50aWFscw0KPiA+IG11c3QgYmUg
b2J0YWluZWQgbWFudWFsbHkgbGlrZSBhbGwgb3RoZXIgdXNlcnMuICAgVXNlICBvZiAgdGhpcyAg
b3B0aW9uDQo+ID4gbWVhbnMgIHRoYXQgICJyb290IiAgbXVzdCAgbWFudWFsbHkgIG9idGFpbiBL
ZXJiZXJvcyBjcmVkZW50aWFscyBiZWZvcmUNCj4gPiBhdHRlbXB0aW5nIHRvIG1vdW50IGFuIG5m
cyBmaWxlc3lzdGVtICByZXF1aXJpbmcgIEtlcmJlcm9zICBhdXRoZW50aWNh4oCQDQo+ID4gdGlv
bi4NCj4gDQo+IEl0IHNlZW1zIHRvIG1lIHRoYXQgaWYgdGhlIGtlcm5lbCBhc2tzIGZvciAic2Vy
dmljZT0nKiciIGFuZCBnc3NkIHdhcyBpbnZva2VkIHdpdGggIi1uIiwgZ3NzZCBzaG91bGQgcmV0
cmlldmUgdGhlIG1hbnVhbGx5LW9idGFpbmVkIEtlcmJlcm9zIGNyZWRlbnRpYWxzIGluIHRoYXQg
Y2FzZS4NCg0KV2h5PyBUaGF0IGlzIG5vdCB3aGF0IHRoZSBkb2N1bWVudGF0aW9uIGFib3ZlIHNh
eXMuIEFzIEkgcmVhZCBpdCwgdGhlDQpkb2N1bWVudGF0aW9uIHN0YXRlcyB0aGF0IGFjY2Vzc2Vz
IGJ5IHVpZCBfMF8gd2lsbCBubyBsb25nZXIgdXNlIG1hY2hpbmUNCmNyZWRzLiBJdCBzYXlzIG5v
dGhpbmcgYWJvdXQgYWNjZXNzZXMgdGhhdCBhY3R1YWxseSByZXF1ZXN0IHRoZSB1c2Ugb2YNCm1h
Y2hpbmUgY3JlZHMsIHN1Y2ggYXMgTkZTdjQgc3RhdGUgb3duZXJzaGlwLi4uDQoNCj4gSSdtIHN0
YXJpbmcgYXQgcHJvY2Vzc19rcmI1X3VwY2FsbCgpLCBqdXN0IHBhc3QgdGhhdCBiaWcgYmxvY2sg
Y29tbWVudCBhcm91bmQgbGluZSA5NjIuICBJIHRoaW5rIHRoYXQgbG9naWMgaXMgcHJvYmFibHkg
d3JvbmcgdG8gY2hlY2sgZm9yICJzZXJ2aWNlX25hbWU9PU5VTEwiIGJ1dCBzaG91bGQgcmF0aGVy
IGNoZWNrIHRoYXQgdGhlIHNlcnZpY2VfbmFtZSBpcyBub3QgIm5mcyIuICBUaGlzIGEgZml4IHRo
YXQgVmVsaS1NYXR0aSBoaW50ZWQgYXQgZWFybGllci4NCj4gDQo+IFRob3VnaHRzPw0KDQpKdXN0
IGJsaW5kbHkgY2hhbmdpbmcgdGhlIG1lYW5pbmcgb2YgJy1uJyB3b3VsZCBiZSB3cm9uZyBJTU8u
IFRoZSB3aG9sZQ0KcG9pbnQgb2YgYWRkaW5nIHRoZSBuZXcgInYxIiB1cGNhbGwgd2FzIHRvIGVu
YWJsZSB0aGUga2VybmVsIHRvIGJlDQpzcGVjaWZpYyBhYm91dCB3aGVuIGl0IHJlcXVpcmVzIG1h
Y2hpbmUgY3JlZGVudGlhbHMgc28gdGhhdCB3ZSBjb3VsZA0KYXZvaWQgdGhlIHdob2xlIGF1dGgo
bWFjaGluZSk9PWF1dGgocm9vdCkgYnJhaW4tZGFtYWdlLg0KDQpOb3RlIHRoYXQgdGhlIG9sZCBi
ZWhhdmlvdXIgY2FuIGJlIHJlLWVuYWJsZWQgYnkgdXNpbmcgYSB2ZXJzaW9uIG9mDQpycGMuZ3Nz
ZCB0aGF0IG9ubHkgc3VwcG9ydHMgdGhlICd2MCcgdXBjYWxsLiBTbyBvbmUgYWx0ZXJuYXRpdmUg
bWlnaHQgYmUNCnRvIHNpbXBseSBhZGQgYW4gb3B0aW9uIHRvIHJwYy5nc3NkIHRoYXQgdHVybnMg
b2ZmICd2MScgdXBjYWxsIHN1cHBvcnQuDQoNCkFub3RoZXIsIG1vcmUgZnV0dXJlLXByb29mIGFs
dGVybmF0aXZlLCBtYXkgYmUgdG8gaGF2ZSBycGMuZ3NzZA0Kc3Vic3RpdHV0ZSBhIHVzZXIgY3Jl
ZCBpZiB0aGVyZSBhcmUgbm8gbWFjaGluZSBjcmVkcywgYW5kIF9pZl8gdGhlIGFkbWluDQplbmFi
bGVzIGFuIG9wdGlvbiB0aGF0IHNwZWNpZmllcyBleGFjdGx5IHdoaWNoIGNyZWRlbnRpYWwgZmls
ZSBzaG91bGQgYmUNCnVzZWQuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xp
ZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3
Lm5ldGFwcC5jb20NCg==

Re: Kernels 3.7 and newer break rpc.gssd -n

[Chuck Lever]

On Feb 18, 2013, at 6:14 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

> On Mon, 2013-02-18 at 17:48 -0500, Chuck Lever wrote:
>> On Feb 18, 2013, at 4:16 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>>> 
>>> On Feb 13, 2013, at 10:29 AM, Veli-Matti Lintu <veli-matti.lintu@opinsys.fi> wrote:
>>> 
>>>> I've been using kerberized nfs4 mounts without machine credentials for 
>>>> quite some time by running rpc.gssd with the -n option. This has 
>>>> resulted rpc.gssd in using ccache in /tmp/krb5cc_0 when doing the mount 
>>>> instead of machine credentials. This functionality seems to break when 
>>>> using kernel 3.7 or newer. 3.6.11 and earlier work like expected.
>>>> 
>>>> The use case for this is diskless workstations that do not have machine 
>>>> credentials stored on them as they have no secure storage medium. When a 
>>>> user logs in, the home directory is mounted using the user's credentials 
>>>> only.
>>>> 
>>>> Steps to reproduce the problem:
>>>> 
>>>> # kinit user (this creates /tmp/krb5cc_0)
>>>> # rpc.gssd -f -n -vvvv
>>>> # mount -t nfs4 -o sec=krb5 server.example.org:/home /mnt
>>>> 
>>>> The mount works when using kernel 3.6.11 or earlier and fails on 3.7-rc1 
>>>> or later. Testing was done on Ubuntu 12.04 and 12.10 using Ubuntu kernels
>>>> and kernel.org kernels (up to 3.8-rc7) with similar results.
>>>> 
>>>> nfs-utils versions 1.2.5 and the latest version from git master head 
>>>> (git://linux-nfs.org/nfs-utils) behave the same way.
>>> 
>>> Reproduced.  Not clear yet if this is a kernel regression or a latent gssd bug.
>> 
>> With pre-3.7 kernels, kernel upcalls with "service=<null>" because the first operation is a LOOKUP_ROOT, and a machine credential is not needed.  With 3.7, kernel is upcalling to get a context to perform SETCLIENTID, and it's specifying "service='*'" as a way to get machine credentials.
>> 
>> In the case where there is no keytab and rpc.gssd is invoked with "-n", gssd is trying to use /etc/krb5.keytab anyway, and failing.
>> 
>> "man rpc.gssd" says this about "-n":
>> 
>>> By default, rpc.gssd treats accesses by the user with UID 0  specially,
>>> and  uses  "machine  credentials"  for  all accesses by that user which
>>> require Kerberos authentication.  With the -n option, "machine  creden‐
>>> tials"  will  not  be used for accesses by UID 0.  Instead, credentials
>>> must be obtained manually like all other users.   Use  of  this  option
>>> means  that  "root"  must  manually  obtain Kerberos credentials before
>>> attempting to mount an nfs filesystem  requiring  Kerberos  authentica‐
>>> tion.
>> 
>> It seems to me that if the kernel asks for "service='*'" and gssd was invoked with "-n", gssd should retrieve the manually-obtained Kerberos credentials in that case.
> 
> Why? That is not what the documentation above says. As I read it, the
> documentation states that accesses by uid _0_ will no longer use machine
> creds. It says nothing about accesses that actually request the use of
> machine creds, such as NFSv4 state ownership...

True, it doesn't, and the use of passive voice makes it easy for a reader to conflate the use of machine cred and UID 0.  But this:

> "root" must manually obtain Kerberos credentials before attempting to mount an nfs filesystem requiring Kerberos authentication.


Sure sounds to me like the use case that Veli-Matti presented is supposed to work by becoming root, kinit'ing, then mounting.

We should make the man page more clear.  It would be especially helpful to describe what "machine credentials" are, and how to set up a machine that doesn't have a host key.  Or if there already is such an explanation, a citation in the man page would be nice.

> 
>> I'm staring at process_krb5_upcall(), just past that big block comment around line 962.  I think that logic is probably wrong to check for "service_name==NULL" but should rather check that the service_name is not "nfs".  This a fix that Veli-Matti hinted at earlier.
>> 
>> Thoughts?
> 
> Just blindly changing the meaning of '-n' would be wrong IMO. The whole
> point of adding the new "v1" upcall was to enable the kernel to be
> specific about when it requires machine credentials so that we could
> avoid the whole auth(machine)==auth(root) brain-damage.

So, we agree that the 3.7 kernel is operating as expected, and that gssd should be behaving differently in this case.  Maybe not an outright bug, but a change is needed.

> Note that the old behaviour can be re-enabled by using a version of
> rpc.gssd that only supports the 'v0' upcall. So one alternative might be
> to simply add an option to rpc.gssd that turns off 'v1' upcall support.
> 
> Another, more future-proof alternative, may be to have rpc.gssd
> substitute a user cred if there are no machine creds, and _if_ the admin
> enables an option that specifies exactly which credential file should be
> used.

gssd already has "-k" to specify a different keytab.  "-k /tmp/krb5cc_0" seems easy enough.

Should we simply add another command line option that specifies an additional entry in the machine credentials search order?

root/<hostname>@<REALM> is already in that list, for example.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com