Linux NFSv4 security issue: client presents wrong user's credentials to NFS server

Linux NFSv4 security issue: client presents wrong user's credentials to NFS server

[Andrew J. Romero]

Hi

It is common in certain cases for
a Linux workstation administrator to 
create a shared account with a 
corresponding .k5login file.

Kerberos principal strings for users
are added to the .k5login and the corresponding
users can logon as the local shared account using
their Kerberos credentials.

Once logged on to the workstation, the user's
local identity (local to the workstation) is
the UID of the local shared account; however,
when accessing kerberized resources external
to the workstation (for example NFSv4 volumes),
the user's identity is expected to be the 
individual user's kerberos identity.

I am seeing the following serious security issue:

If N users (user1@REALM , user2@REALM ...userN@REALM)
(who are listed  sharusr's .k5login) all logon to 
workstation1 as sharusr and are all running sessions
simultaneously, then, the GSS Context / kerberos credentials 
stored in and used by the NFS-client kernel code when
processing NFS requests on behalf of a user,
will be correct for 1 user and incorrect for N-1 users.

Each of the N-1 users will not be able to 
access NFS server resources that only they have been 
granted access to; but, they will be able
to access all NFS resources that the 1  user
has been granted access to. (even if their
kerberos identity has not been granted access).

Even after each of the N-1 users destroys their credentials
they still retain access to all NFS resources 
that the 1 user has been granted access to.
Access will not be denied until the 1 user also destroys
his / her credentials.

It is my understanding that the RPC GSS kernel module 
on an NFS client system, keeps a GSS security context 
(copy of a user's credentials) that it uses when it 
communicates with the NFS server on the user's behalf. 

I believe that the kernel security context
relates back to the user mode process solely by UID number.

Would adding session ID to the relation resolve the issue ?
Each Session for each user would then have a unique tuple
(UID,SessionID) that would map to the *correct* GSS Security
context in the kernel.

Thanks

Andy Romero
romero@fnal.gov
NAS / SAN Administrator, Fermilab

Re: Linux NFSv4 security issue: client presents wrong user's credentials to NFS server

[Benjamin Coddington]

On Tue, 7 Oct 2014, Andrew J. Romero wrote:

> Hi
>
> It is common in certain cases for
> a Linux workstation administrator to
> create a shared account with a
> corresponding .k5login file.
>
> Kerberos principal strings for users
> are added to the .k5login and the corresponding
> users can logon as the local shared account using
> their Kerberos credentials.
>
> Once logged on to the workstation, the user's
> local identity (local to the workstation) is
> the UID of the local shared account; however,
> when accessing kerberized resources external
> to the workstation (for example NFSv4 volumes),
> the user's identity is expected to be the
> individual user's kerberos identity.
>
> I am seeing the following serious security issue:
>
> If N users (user1@REALM , user2@REALM ...userN@REALM)
> (who are listed  sharusr's .k5login) all logon to
> workstation1 as sharusr and are all running sessions
> simultaneously, then, the GSS Context / kerberos credentials
> stored in and used by the NFS-client kernel code when
> processing NFS requests on behalf of a user,
> will be correct for 1 user and incorrect for N-1 users.
>
> Each of the N-1 users will not be able to
> access NFS server resources that only they have been
> granted access to; but, they will be able
> to access all NFS resources that the 1  user
> has been granted access to. (even if their
> kerberos identity has not been granted access).
>
> Even after each of the N-1 users destroys their credentials
> they still retain access to all NFS resources
> that the 1 user has been granted access to.
> Access will not be denied until the 1 user also destroys
> his / her credentials.
>
> It is my understanding that the RPC GSS kernel module
> on an NFS client system, keeps a GSS security context
> (copy of a user's credentials) that it uses when it
> communicates with the NFS server on the user's behalf.
>
> I believe that the kernel security context
> relates back to the user mode process solely by UID number.

That is correct.

> Would adding session ID to the relation resolve the issue ?
> Each Session for each user would then have a unique tuple
> (UID,SessionID) that would map to the *correct* GSS Security
> context in the kernel.

I think what could be done is to hang gss contexts off either session
or process keyrings - then each process group would be able to own an
identity independently of other processes running with the same UID.

To do this the gssd upcall could be moved to a request-key mechanism, as
there would need to be a way to pass along which krb5 credentials ought
to be used to establish the context.  I suppose that if the process
already has krb5 credentials in a keyring ccache, those could be used -
but I don't think that we could count on always having a keyring ccache
available, and the fallback would be to try to guess which ccache to use
based on UID as things are today.  There would additionally be the
situation where keyring ccaches cannot be used, and the main problem
becomes how to have a process tell gssd which credential cache to use.

Maybe instead of establishing contexts at file access time, they could
be established deliberately beforehand with a userspace command that
would specify which krb5 credential to use to create the context, and
then the context would hang off the process keyrings.  The client could
then check to see if there's an existing keyring context, and if so use
that, and if not do the upcall.

Ben

Re: Linux NFSv4 security issue: client presents wrong user's credentials to NFS server

[Trond Myklebust]

On Tue, Oct 7, 2014 at 9:10 AM, Benjamin Coddington <bcodding@redhat.com> wrote:
> On Tue, 7 Oct 2014, Andrew J. Romero wrote:
>
>> Hi
>>
>> It is common in certain cases for
>> a Linux workstation administrator to
>> create a shared account with a
>> corresponding .k5login file.
>>
>> Kerberos principal strings for users
>> are added to the .k5login and the corresponding
>> users can logon as the local shared account using
>> their Kerberos credentials.
>>
>> Once logged on to the workstation, the user's
>> local identity (local to the workstation) is
>> the UID of the local shared account; however,
>> when accessing kerberized resources external
>> to the workstation (for example NFSv4 volumes),
>> the user's identity is expected to be the
>> individual user's kerberos identity.
>>
>> I am seeing the following serious security issue:
>>
>> If N users (user1@REALM , user2@REALM ...userN@REALM)
>> (who are listed  sharusr's .k5login) all logon to
>> workstation1 as sharusr and are all running sessions
>> simultaneously, then, the GSS Context / kerberos credentials
>> stored in and used by the NFS-client kernel code when
>> processing NFS requests on behalf of a user,
>> will be correct for 1 user and incorrect for N-1 users.
>>
>> Each of the N-1 users will not be able to
>> access NFS server resources that only they have been
>> granted access to; but, they will be able
>> to access all NFS resources that the 1  user
>> has been granted access to. (even if their
>> kerberos identity has not been granted access).
>>
>> Even after each of the N-1 users destroys their credentials
>> they still retain access to all NFS resources
>> that the 1 user has been granted access to.
>> Access will not be denied until the 1 user also destroys
>> his / her credentials.
>>
>> It is my understanding that the RPC GSS kernel module
>> on an NFS client system, keeps a GSS security context
>> (copy of a user's credentials) that it uses when it
>> communicates with the NFS server on the user's behalf.
>>
>> I believe that the kernel security context
>> relates back to the user mode process solely by UID number.
>
>
> That is correct.
>
>> Would adding session ID to the relation resolve the issue ?
>> Each Session for each user would then have a unique tuple
>> (UID,SessionID) that would map to the *correct* GSS Security
>> context in the kernel.
>
>
> I think what could be done is to hang gss contexts off either session
> or process keyrings - then each process group would be able to own an
> identity independently of other processes running with the same UID.
>
> To do this the gssd upcall could be moved to a request-key mechanism, as
> there would need to be a way to pass along which krb5 credentials ought
> to be used to establish the context.  I suppose that if the process
> already has krb5 credentials in a keyring ccache, those could be used -
> but I don't think that we could count on always having a keyring ccache
> available, and the fallback would be to try to guess which ccache to use
> based on UID as things are today.  There would additionally be the
> situation where keyring ccaches cannot be used, and the main problem
> becomes how to have a process tell gssd which credential cache to use.

The problem with the request key interface is that it is completely
broken when applied to containers, since it only runs in the global
init namespace context. Fixing that is a non-trivial exercise; you'd
have to not only carry a full namespace context with the RPC
credential, but also somehow apply it to the upcall thread.

The rpc.gssd model is much easier to manage, since the daemon itself
is started from the container's local init process.

> Maybe instead of establishing contexts at file access time, they could
> be established deliberately beforehand with a userspace command that
> would specify which krb5 credential to use to create the context, and
> then the context would hang off the process keyrings.  The client could
> then check to see if there's an existing keyring context, and if so use
> that, and if not do the upcall.
>

The server is free to drop any GSS session at any time, depending on
internal resource management requirements. A client must be able to
recover from that scenario or risk being unable to write back data.

Cheers
  Trond

-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com

Re: Linux NFSv4 security issue: client presents wrong user's credentials to NFS server

[J. Bruce Fields]

On Tue, Oct 07, 2014 at 09:36:12AM -0400, Trond Myklebust wrote:
> The problem with the request key interface is that it is completely
> broken when applied to containers, since it only runs in the global
> init namespace context. Fixing that is a non-trivial exercise; you'd
> have to not only carry a full namespace context with the RPC
> credential, but also somehow apply it to the upcall thread.

This one's really keeping me up at night.  Mainly because of the server
reboot recovery stuff.  The client v4 idmapping has this problem too,
doesn't it?

How are we going to decide if the user-helper containerization is doable
or if it's just a hopeless case?

Jeff seems optimistic about fixing it, and that'd be great, but if it's
not going to happen then we need to give up and use daemons for this
stuff.

--b.

Re: Linux NFSv4 security issue: client presents wrong user's credentials to NFS server

[Benjamin Coddington]

On Tue, 7 Oct 2014, Trond Myklebust wrote:

> On Tue, Oct 7, 2014 at 9:10 AM, Benjamin Coddington <bcodding@redhat.com> wrote:
>> On Tue, 7 Oct 2014, Andrew J. Romero wrote:
>>
>>> Hi
>>>
>>> It is common in certain cases for
>>> a Linux workstation administrator to
>>> create a shared account with a
>>> corresponding .k5login file.
>>>
>>> Kerberos principal strings for users
>>> are added to the .k5login and the corresponding
>>> users can logon as the local shared account using
>>> their Kerberos credentials.
>>>
>>> Once logged on to the workstation, the user's
>>> local identity (local to the workstation) is
>>> the UID of the local shared account; however,
>>> when accessing kerberized resources external
>>> to the workstation (for example NFSv4 volumes),
>>> the user's identity is expected to be the
>>> individual user's kerberos identity.
>>>
>>> I am seeing the following serious security issue:
>>>
>>> If N users (user1@REALM , user2@REALM ...userN@REALM)
>>> (who are listed  sharusr's .k5login) all logon to
>>> workstation1 as sharusr and are all running sessions
>>> simultaneously, then, the GSS Context / kerberos credentials
>>> stored in and used by the NFS-client kernel code when
>>> processing NFS requests on behalf of a user,
>>> will be correct for 1 user and incorrect for N-1 users.
>>>
>>> Each of the N-1 users will not be able to
>>> access NFS server resources that only they have been
>>> granted access to; but, they will be able
>>> to access all NFS resources that the 1  user
>>> has been granted access to. (even if their
>>> kerberos identity has not been granted access).
>>>
>>> Even after each of the N-1 users destroys their credentials
>>> they still retain access to all NFS resources
>>> that the 1 user has been granted access to.
>>> Access will not be denied until the 1 user also destroys
>>> his / her credentials.
>>>
>>> It is my understanding that the RPC GSS kernel module
>>> on an NFS client system, keeps a GSS security context
>>> (copy of a user's credentials) that it uses when it
>>> communicates with the NFS server on the user's behalf.
>>>
>>> I believe that the kernel security context
>>> relates back to the user mode process solely by UID number.
>>
>>
>> That is correct.
>>
>>> Would adding session ID to the relation resolve the issue ?
>>> Each Session for each user would then have a unique tuple
>>> (UID,SessionID) that would map to the *correct* GSS Security
>>> context in the kernel.
>>
>>
>> I think what could be done is to hang gss contexts off either session
>> or process keyrings - then each process group would be able to own an
>> identity independently of other processes running with the same UID.
>>
>> To do this the gssd upcall could be moved to a request-key mechanism, as
>> there would need to be a way to pass along which krb5 credentials ought
>> to be used to establish the context.  I suppose that if the process
>> already has krb5 credentials in a keyring ccache, those could be used -
>> but I don't think that we could count on always having a keyring ccache
>> available, and the fallback would be to try to guess which ccache to use
>> based on UID as things are today.  There would additionally be the
>> situation where keyring ccaches cannot be used, and the main problem
>> becomes how to have a process tell gssd which credential cache to use.
>
> The problem with the request key interface is that it is completely
> broken when applied to containers, since it only runs in the global
> init namespace context. Fixing that is a non-trivial exercise; you'd
> have to not only carry a full namespace context with the RPC
> credential, but also somehow apply it to the upcall thread.
>
> The rpc.gssd model is much easier to manage, since the daemon itself
> is started from the container's local init process.

That seems like something that might be fixed in the keys stuff instead
of dragging the namespace along on the rpc cred, because it seems like a
more general problem that request-key upcalls can execute outside their
container..  but maybe that's exactly what request-key is supposed to
do.  I'll ask David what he thinks about it.

Alternatively, the contexts could still hang off session or process
keyrings but still use the existing upcall...

How did the idmap upcall get moved over to the request-key interface
without taking the container problem into account?

>> Maybe instead of establishing contexts at file access time, they could
>> be established deliberately beforehand with a userspace command that
>> would specify which krb5 credential to use to create the context, and
>> then the context would hang off the process keyrings.  The client could
>> then check to see if there's an existing keyring context, and if so use
>> that, and if not do the upcall.
>>
>
> The server is free to drop any GSS session at any time, depending on
> internal resource management requirements. A client must be able to
> recover from that scenario or risk being unable to write back data.

Ah, right.  Thank you for pointing that out.

Ben

Re: Linux NFSv4 security issue: client presents wrong user's credentials to NFS server

[Trond Myklebust]

On Wed, Oct 8, 2014 at 3:39 PM, Benjamin Coddington <bcodding@redhat.com> wrote:
>
> On Tue, 7 Oct 2014, Trond Myklebust wrote:
>
>> On Tue, Oct 7, 2014 at 9:10 AM, Benjamin Coddington <bcodding@redhat.com>
>> wrote:
>>>
>>> On Tue, 7 Oct 2014, Andrew J. Romero wrote:
>>>
>>>> Hi
>>>>
>>>> It is common in certain cases for
>>>> a Linux workstation administrator to
>>>> create a shared account with a
>>>> corresponding .k5login file.
>>>>
>>>> Kerberos principal strings for users
>>>> are added to the .k5login and the corresponding
>>>> users can logon as the local shared account using
>>>> their Kerberos credentials.
>>>>
>>>> Once logged on to the workstation, the user's
>>>> local identity (local to the workstation) is
>>>> the UID of the local shared account; however,
>>>> when accessing kerberized resources external
>>>> to the workstation (for example NFSv4 volumes),
>>>> the user's identity is expected to be the
>>>> individual user's kerberos identity.
>>>>
>>>> I am seeing the following serious security issue:
>>>>
>>>> If N users (user1@REALM , user2@REALM ...userN@REALM)
>>>> (who are listed  sharusr's .k5login) all logon to
>>>> workstation1 as sharusr and are all running sessions
>>>> simultaneously, then, the GSS Context / kerberos credentials
>>>> stored in and used by the NFS-client kernel code when
>>>> processing NFS requests on behalf of a user,
>>>> will be correct for 1 user and incorrect for N-1 users.
>>>>
>>>> Each of the N-1 users will not be able to
>>>> access NFS server resources that only they have been
>>>> granted access to; but, they will be able
>>>> to access all NFS resources that the 1  user
>>>> has been granted access to. (even if their
>>>> kerberos identity has not been granted access).
>>>>
>>>> Even after each of the N-1 users destroys their credentials
>>>> they still retain access to all NFS resources
>>>> that the 1 user has been granted access to.
>>>> Access will not be denied until the 1 user also destroys
>>>> his / her credentials.
>>>>
>>>> It is my understanding that the RPC GSS kernel module
>>>> on an NFS client system, keeps a GSS security context
>>>> (copy of a user's credentials) that it uses when it
>>>> communicates with the NFS server on the user's behalf.
>>>>
>>>> I believe that the kernel security context
>>>> relates back to the user mode process solely by UID number.
>>>
>>>
>>>
>>> That is correct.
>>>
>>>> Would adding session ID to the relation resolve the issue ?
>>>> Each Session for each user would then have a unique tuple
>>>> (UID,SessionID) that would map to the *correct* GSS Security
>>>> context in the kernel.
>>>
>>>
>>>
>>> I think what could be done is to hang gss contexts off either session
>>> or process keyrings - then each process group would be able to own an
>>> identity independently of other processes running with the same UID.
>>>
>>> To do this the gssd upcall could be moved to a request-key mechanism, as
>>> there would need to be a way to pass along which krb5 credentials ought
>>> to be used to establish the context.  I suppose that if the process
>>> already has krb5 credentials in a keyring ccache, those could be used -
>>> but I don't think that we could count on always having a keyring ccache
>>> available, and the fallback would be to try to guess which ccache to use
>>> based on UID as things are today.  There would additionally be the
>>> situation where keyring ccaches cannot be used, and the main problem
>>> becomes how to have a process tell gssd which credential cache to use.
>>
>>
>> The problem with the request key interface is that it is completely
>> broken when applied to containers, since it only runs in the global
>> init namespace context. Fixing that is a non-trivial exercise; you'd
>> have to not only carry a full namespace context with the RPC
>> credential, but also somehow apply it to the upcall thread.
>>
>> The rpc.gssd model is much easier to manage, since the daemon itself
>> is started from the container's local init process.
>
>
> That seems like something that might be fixed in the keys stuff instead
> of dragging the namespace along on the rpc cred, because it seems like a
> more general problem that request-key upcalls can execute outside their
> container..  but maybe that's exactly what request-key is supposed to
> do.  I'll ask David what he thinks about it.
>
> Alternatively, the contexts could still hang off session or process
> keyrings but still use the existing upcall...
>
> How did the idmap upcall get moved over to the request-key interface
> without taking the container problem into account?

Timing is everything.

We converted idmapper in 2010 just before the Parallels people came
and wanted to do net namespaces in 2011. However you will note that we
still do support the rpc.idmapd daemon, and that the interfaces it
uses have been converted to support net namespace.

>>> Maybe instead of establishing contexts at file access time, they could
>>> be established deliberately beforehand with a userspace command that
>>> would specify which krb5 credential to use to create the context, and
>>> then the context would hang off the process keyrings.  The client could
>>> then check to see if there's an existing keyring context, and if so use
>>> that, and if not do the upcall.
>>>
>>
>> The server is free to drop any GSS session at any time, depending on
>> internal resource management requirements. A client must be able to
>> recover from that scenario or risk being unable to write back data.
>
>
> Ah, right.  Thank you for pointing that out.
>
> Ben



-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com