How to use NFS with multiple principals in different realms?

How to use NFS with multiple principals in different realms?

[Cedric Blancher]

How can I use NFS with kerberos krb5p auth when I want to use NFS
filesystems which come from different realms?

I know klist -A can show all tickets I got from all realms I kinit to
- but how can NFS use them?

OS is RHEL7

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
Institute Pasteur

Re: How to use NFS with multiple principals in different realms?

[Jurjen Bokma]

You use cross realm authentication, so that your NFS client may obtain
tickets for servers that are not in its own realm.

To allow for cross-realm authentication from your own MYREALM.NET (where
te client is) to your neighbour's HISREALM.ORG (the realm of the
server), you basically create *in*each*realm* a principal
krbtgt/MYREALM.NET@HISREALM.ORG.
Both principals must be identical. They have to have the same name, the
same encryption types, and the same kvno.
After that, it's a matter of configuring the krb5.conf right: tell the
Kerberos servers which machine is in which realm if DNS doesn't do that.
Debian bug #571244 may also come into play on RHEL?

I've got step by step documentation for an Ubuntu NFS client with
Kerberos served by Heimdal under Debian and Windows AD respectively. But
I haven't anonymized the text yet, so I can't put it online immediately.
Would it help you if I anonymize it soon, or can you work it our from
just the above hints?

Regards
Jurjen


On 09/04/2014 11:04 AM, Cedric Blancher wrote:
> How can I use NFS with kerberos krb5p auth when I want to use NFS
> filesystems which come from different realms?
> 
> I know klist -A can show all tickets I got from all realms I kinit to
> - but how can NFS use them?
> 
> OS is RHEL7
> 
> Ced
> 

Re: How to use NFS with multiple principals in different realms?

[Cedric Blancher]

On 4 September 2014 11:33, Jurjen Bokma <j.bokma@rug.nl> wrote:
> You use cross realm authentication, so that your NFS client may obtain
> tickets for servers that are not in its own realm.

What if I cannot use cross realm authentication? For example if both
realms do not like each other?
What if I really have to kinit into multiple realms? Kerberos since
1.10 can do that and klist now has a new flag -A to list all entries
if KRB5CCNAME points to a directory, e.g.
KRB5CCNAME=DIR:/tmp/krbcc$UID/

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
Institute Pasteur

Re: How to use NFS with multiple principals in different realms?

[Jurjen Bokma]

On 09/04/2014 01:25 PM, Cedric Blancher wrote:
> On 4 September 2014 11:33, Jurjen Bokma <j.bokma@rug.nl> wrote:
>> You use cross realm authentication, so that your NFS client may obtain
>> tickets for servers that are not in its own realm.
> 
> What if I cannot use cross realm authentication? For example if both
> realms do not like each other?
> What if I really have to kinit into multiple realms? Kerberos since
> 1.10 can do that and klist now has a new flag -A to list all entries
> if KRB5CCNAME points to a directory, e.g.
> KRB5CCNAME=DIR:/tmp/krbcc$UID/
> 
> Ced
> 
I tried that about a year ago, and failed to make it work.
As far as I know, gssd always picks the same key to authenticate with. I
did offer a patch on this list a couple of weeks ago that uses a
krb5.conf appdefaults option to configure *which* key, but that one
still doesn't make it possible to pick a different key for different shares.

Sorry
Jurjen

Re: How to use NFS with multiple principals in different realms?

[Simo Sorce]

On Thu, 2014-09-04 at 14:32 +0200, Jurjen Bokma wrote:
> On 09/04/2014 01:25 PM, Cedric Blancher wrote:
> > On 4 September 2014 11:33, Jurjen Bokma <j.bokma@rug.nl> wrote:
> >> You use cross realm authentication, so that your NFS client may obtain
> >> tickets for servers that are not in its own realm.
> > 
> > What if I cannot use cross realm authentication? For example if both
> > realms do not like each other?
> > What if I really have to kinit into multiple realms? Kerberos since
> > 1.10 can do that and klist now has a new flag -A to list all entries
> > if KRB5CCNAME points to a directory, e.g.
> > KRB5CCNAME=DIR:/tmp/krbcc$UID/
> > 
> > Ced
> > 
> I tried that about a year ago, and failed to make it work.

The problem is that the server can only have one set of credentials from
the POV of the client, and that's: nfs@fqdn (a GSSAPI name), that gets
converted into a principal of the form nfs/fqdn@REALM (where REALM is
determined by a mapping of the form domain_name->REALM in the client
usually).

> As far as I know, gssd always picks the same key to authenticate with. I

When rpc.gssd (potentially interposed by gss-proxy) then uses GSSAPI to
obtain a ticket for the server it will choose the credentials that match
the same REALM in preference, even if you have multiple credentials.

The client has no way to know that you want to use a different set of
credentials, because it doesn't even know (IIRC) what you are trying to
access on the server when this call is made.

> did offer a patch on this list a couple of weeks ago that uses a
> krb5.conf appdefaults option to configure *which* key, but that one
> still doesn't make it possible to pick a different key for different shares.

It allows you to choose a different key for the *machine* principal, but
does nothing for authenticating users using different credentials.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

Re: How to use NFS with multiple principals in different realms?

[Cedric Blancher]

On 4 September 2014 20:35, Simo Sorce <simo@redhat.com> wrote:
> On Thu, 2014-09-04 at 14:32 +0200, Jurjen Bokma wrote:
>> On 09/04/2014 01:25 PM, Cedric Blancher wrote:
>> > On 4 September 2014 11:33, Jurjen Bokma <j.bokma@rug.nl> wrote:
>> >> You use cross realm authentication, so that your NFS client may obtain
>> >> tickets for servers that are not in its own realm.
>> >
>> > What if I cannot use cross realm authentication? For example if both
>> > realms do not like each other?
>> > What if I really have to kinit into multiple realms? Kerberos since
>> > 1.10 can do that and klist now has a new flag -A to list all entries
>> > if KRB5CCNAME points to a directory, e.g.
>> > KRB5CCNAME=DIR:/tmp/krbcc$UID/
>> >
>> > Ced
>> >
>> I tried that about a year ago, and failed to make it work.
>
> The problem is that the server can only have one set of credentials from
> the POV of the client, and that's: nfs@fqdn (a GSSAPI name), that gets
> converted into a principal of the form nfs/fqdn@REALM (where REALM is
> determined by a mapping of the form domain_name->REALM in the client
> usually).

Per Oracle support this is not quite correct: if you have multiple
tickets in a DIR: then the NFS client is either required to negotiate
with the server (RFC 3530) or try the credentials in order until one
works.

>> As far as I know, gssd always picks the same key to authenticate with. I
>
> When rpc.gssd (potentially interposed by gss-proxy) then uses GSSAPI to
> obtain a ticket for the server it will choose the credentials that match
> the same REALM in preference, even if you have multiple credentials.

But that can't be right if you have tickets originating from more than
one realm in a DIR:, can it?

>
> The client has no way to know that you want to use a different set of
> credentials, because it doesn't even know (IIRC) what you are trying to
> access on the server when this call is made.

Still it has to try all options, i.e. negotiate. This is what the
reference implementation for NFS (Solaris) does.

>
>> did offer a patch on this list a couple of weeks ago that uses a
>> krb5.conf appdefaults option to configure *which* key, but that one
>> still doesn't make it possible to pick a different key for different shares.
>
> It allows you to choose a different key for the *machine* principal, but
> does nothing for authenticating users using different credentials.
>
> Simo.
>
> --
> Simo Sorce * Red Hat, Inc * New York
>

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
Institute Pasteur

RE: How to use NFS with multiple principals in different realms?

[Nordgren, Bryce L -FS]

> Per Oracle support this is not quite correct: if you have multiple tickets in a
> DIR: then the NFS client is either required to negotiate with the server (RFC
> 3530) or try the credentials in order until one works.

"negotiate" appears to mean select a security mechanism, such as Kerberos vs. something else. Is there actually an "intra-mechanism" requirement?

> Still it has to try all options, i.e. negotiate. This is what the reference
> implementation for NFS (Solaris) does.


What are "all the options"? It could try nfs/fqdn@REALM for each REALM present in your ticket cache. It could maintain a separate list of realms not in your cache to be tried. It could let the user specify a principal to try. This has nothing to do with client/server communication and more with probing the Kerberos trust network to see if there is a path to walk from one of the credentials in your cache to one of the potential service principals.

I think what you're saying is that after the negotiation has settled on Kerberos, you want more aggressive probing.

Alternatively, in lieu of aggressive probing (which would yield shared Kerberos user principal trustable by both client and server), you want the id mapper on the NFS client and on the NFS server to map the same NFS id to different Kerberos principals. And then the client system should also know what Kerberos principal the server system has mapped to the relevant NFS ID (server-wide? For a particular mount point?). And both client-recognized and server-recognized Kerberos principals must be in the user's cache. And trust that the two principals are the same ultimately stems from the fact that they're in the same user's Kerberos cache, which is presumably under the control of one human. But I think there still has to be a trust path from NFS client to NFS server. Or no?

Do NFS ID mappers talk to one another?  How would they negotiate? I think I need a picture to nail down all the moving parts.

Bryce





This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.

Re: How to use NFS with multiple principals in different realms?

[Trond Myklebust]

On Tue, Sep 9, 2014 at 7:18 PM, Nordgren, Bryce L -FS
<bnordgren@fs.fed.us> wrote:
>
>> Per Oracle support this is not quite correct: if you have multiple tickets in a
>> DIR: then the NFS client is either required to negotiate with the server (RFC
>> 3530) or try the credentials in order until one works.
>
> "negotiate" appears to mean select a security mechanism, such as Kerberos vs. something else. Is there actually an "intra-mechanism" requirement?
>
>> Still it has to try all options, i.e. negotiate. This is what the reference
>> implementation for NFS (Solaris) does.

Is it time to party like it's 1999? Who listed Solaris as the
functional reference for the Linux rpc.gssd daemon while I wasn't
looking?

rpc.gssd and its intended user functionality is listed in its manpage.

Feel free to put in a feature request if you think that new
functionality is needed to support a good use case that you may have,
but please don't try to pass it off as a protocol requirement that
we're not fulfilling unless you are prepared to point to chapter and
verse of the relevant RFC. A quick grep will show you that there is
nothing in RFC3530 that even mentions kerberos tickets, let alone
describes how they are to be managed if you have more than one.
Neither does RFC2623.

> What are "all the options"? It could try nfs/fqdn@REALM for each REALM present in your ticket cache. It could maintain a separate list of realms not in your cache to be tried. It could let the user specify a principal to try. This has nothing to do with client/server communication and more with probing the Kerberos trust network to see if there is a path to walk from one of the credentials in your cache to one of the potential service principals.
>
> I think what you're saying is that after the negotiation has settled on Kerberos, you want more aggressive probing.
>
> Alternatively, in lieu of aggressive probing (which would yield shared Kerberos user principal trustable by both client and server), you want the id mapper on the NFS client and on the NFS server to map the same NFS id to different Kerberos principals. And then the client system should also know what Kerberos principal the server system has mapped to the relevant NFS ID (server-wide? For a particular mount point?). And both client-recognized and server-recognized Kerberos principals must be in the user's cache. And trust that the two principals are the same ultimately stems from the fact that they're in the same user's Kerberos cache, which is presumably under the control of one human. But I think there still has to be a trust path from NFS client to NFS server. Or no?
>
> Do NFS ID mappers talk to one another?  How would they negotiate? I think I need a picture to nail down all the moving parts.

RFC3530 does not describe any form of NFS id mapper negotiation. That
lies entirely outside the scope of the protocol, and is usually
ensured by deploying third party solutions such as LDAP.

In the NFSv4 spec,  the client just asserts a "owner@dns_domain" or
"owner_group@dns_domain" string as part of a SETATTR command to set an
ACL or a file owner/owner_group, and either the server recognises it
or it doesn't. If the server doesn't recognise the string, then it
returns an error, and the acl/ownership change is not allowed.

On the other hand, the client accepts whatever owner@domain or
owner_group@domain strings the server returns as part of a GETATTR or
READDIR command to retrieve an ACL or file owner/owner_group. If the
client doesn't have a mapping, then it will usually try to represent
the string as an anonymous user/owner so that it can at least list
something that the application can recognise.

Either way, the client isn't allowed to use any of that idmapping
information for file access or other security enforcement tasks. Such
enforcement decisions are rather done through RPC calls directly to
the server, and that may be authenticated by an RPCSEC_GSS session.
E.g. if the client is trying to decide if it should allow a process
access to the cached data for a given file, then it should send an
ACCESS rpc call to the server. It will never try to read the ACL
directly and interpret that. If the client wants to know if a process
is allowed to create a directory, it sends an RPC call that attempts
the operation.

IOW: the NFSv4 protocol limits the scope of the problem by trying to
be robust against relying on idmapping for security purposes, but
limiting the functionality available to applications if your client
and server don't have a way to share idmapping information (ability to
set/read/modify acls, or to change file ownership).

-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com

Re: How to use NFS with multiple principals in different realms?

[Simo Sorce]

----- Original Message -----
> From: "Cedric Blancher" <cedric.blancher@gmail.com>
> To: "Simo Sorce" <simo@redhat.com>
> Cc: "Jurjen Bokma" <j.bokma@rug.nl>, "<kerberos@mit.edu>" <kerberos@mit.edu>, "Linux NFS Mailing List"
> <linux-nfs@vger.kernel.org>, "Steve Dickson" <steved@redhat.com>
> Sent: Tuesday, September 9, 2014 8:31:00 PM
> Subject: Re: How to use NFS with multiple principals in different realms?
> 
> On 4 September 2014 20:35, Simo Sorce <simo@redhat.com> wrote:
> > On Thu, 2014-09-04 at 14:32 +0200, Jurjen Bokma wrote:
> >> On 09/04/2014 01:25 PM, Cedric Blancher wrote:
> >> > On 4 September 2014 11:33, Jurjen Bokma <j.bokma@rug.nl> wrote:
> >> >> You use cross realm authentication, so that your NFS client may obtain
> >> >> tickets for servers that are not in its own realm.
> >> >
> >> > What if I cannot use cross realm authentication? For example if both
> >> > realms do not like each other?
> >> > What if I really have to kinit into multiple realms? Kerberos since
> >> > 1.10 can do that and klist now has a new flag -A to list all entries
> >> > if KRB5CCNAME points to a directory, e.g.
> >> > KRB5CCNAME=DIR:/tmp/krbcc$UID/
> >> >
> >> > Ced
> >> >
> >> I tried that about a year ago, and failed to make it work.
> >
> > The problem is that the server can only have one set of credentials from
> > the POV of the client, and that's: nfs@fqdn (a GSSAPI name), that gets
> > converted into a principal of the form nfs/fqdn@REALM (where REALM is
> > determined by a mapping of the form domain_name->REALM in the client
> > usually).
> 
> Per Oracle support this is not quite correct: if you have multiple
> tickets in a DIR: then the NFS client is either required to negotiate
> with the server (RFC 3530) or try the credentials in order until one
> works.

I have the impression you are confusing client and server credentials, what
section do you refer to exactly ?

> >> As far as I know, gssd always picks the same key to authenticate with. I
> >
> > When rpc.gssd (potentially interposed by gss-proxy) then uses GSSAPI to
> > obtain a ticket for the server it will choose the credentials that match
> > the same REALM in preference, even if you have multiple credentials.
> 
> But that can't be right if you have tickets originating from more than
> one realm in a DIR:, can it?

It doesn't matter what tickets you have, you can have only one fqdn->REALM
mapping (at least with current libraries), so the server REALM is determined
first, then the most appropriate set of credentials available to the client
is selected.
>From memory the order is:
1. matching Realm if any credential in that Realm is available
2. default Realm as defined in krb5.conf if credentials in that Realm are
   available, this will attempt to get a cross-realm TGT to the target
   Realm first
3. the first credential available in (1) and (2) do not match

> > The client has no way to know that you want to use a different set of
> > credentials, because it doesn't even know (IIRC) what you are trying to
> > access on the server when this call is made.
> 
> Still it has to try all options, i.e. negotiate. This is what the
> reference implementation for NFS (Solaris) does.

Not the Linux clients, afair.

Simo.

Re: How to use NFS with multiple principals in different realms?

[Cedric Blancher]

On 10 September 2014 15:06, Simo Sorce <simo@redhat.com> wrote:
>
>
> ----- Original Message -----
>> From: "Cedric Blancher" <cedric.blancher@gmail.com>
>> To: "Simo Sorce" <simo@redhat.com>
>> Cc: "Jurjen Bokma" <j.bokma@rug.nl>, "<kerberos@mit.edu>" <kerberos@mit.edu>, "Linux NFS Mailing List"
>> <linux-nfs@vger.kernel.org>, "Steve Dickson" <steved@redhat.com>
>> Sent: Tuesday, September 9, 2014 8:31:00 PM
>> Subject: Re: How to use NFS with multiple principals in different realms?
>>
>> On 4 September 2014 20:35, Simo Sorce <simo@redhat.com> wrote:
>> > On Thu, 2014-09-04 at 14:32 +0200, Jurjen Bokma wrote:
>> >> On 09/04/2014 01:25 PM, Cedric Blancher wrote:
>> >> > On 4 September 2014 11:33, Jurjen Bokma <j.bokma@rug.nl> wrote:
>> >> >> You use cross realm authentication, so that your NFS client may obtain
>> >> >> tickets for servers that are not in its own realm.
>> >> >
>> >> > What if I cannot use cross realm authentication? For example if both
>> >> > realms do not like each other?
>> >> > What if I really have to kinit into multiple realms? Kerberos since
>> >> > 1.10 can do that and klist now has a new flag -A to list all entries
>> >> > if KRB5CCNAME points to a directory, e.g.
>> >> > KRB5CCNAME=DIR:/tmp/krbcc$UID/
>> >> >
>> >> > Ced
>> >> >
>> >> I tried that about a year ago, and failed to make it work.
>> >
>> > The problem is that the server can only have one set of credentials from
>> > the POV of the client, and that's: nfs@fqdn (a GSSAPI name), that gets
>> > converted into a principal of the form nfs/fqdn@REALM (where REALM is
>> > determined by a mapping of the form domain_name->REALM in the client
>> > usually).
>>
>> Per Oracle support this is not quite correct: if you have multiple
>> tickets in a DIR: then the NFS client is either required to negotiate
>> with the server (RFC 3530) or try the credentials in order until one
>> works.
>
> I have the impression you are confusing client and server credentials, what
> section do you refer to exactly ?

The tickets on the side calling mount(8) - nfs client side

>
>> >> As far as I know, gssd always picks the same key to authenticate with. I
>> >
>> > When rpc.gssd (potentially interposed by gss-proxy) then uses GSSAPI to
>> > obtain a ticket for the server it will choose the credentials that match
>> > the same REALM in preference, even if you have multiple credentials.
>>
>> But that can't be right if you have tickets originating from more than
>> one realm in a DIR:, can it?
>
> It doesn't matter what tickets you have, you can have only one fqdn->REALM
> mapping (at least with current libraries), so the server REALM is determined
> first, then the most appropriate set of credentials available to the client
> is selected.
> From memory the order is:
> 1. matching Realm if any credential in that Realm is available
> 2. default Realm as defined in krb5.conf if credentials in that Realm are
>    available, this will attempt to get a cross-realm TGT to the target
>    Realm first
> 3. the first credential available in (1) and (2) do not match

What happens if there is no relation between KRB Realm names and
FQDN/DNS? Can the NFS client find out which KRB Realm is used by the
server?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
Institute Pasteur

Re: How to use NFS with multiple principals in different realms?

[Simo Sorce]

On Wed, 17 Sep 2014 13:20:19 +0200
Cedric Blancher <cedric.blancher@gmail.com> wrote:

> What happens if there is no relation between KRB Realm names and
> FQDN/DNS? Can the NFS client find out which KRB Realm is used by the
> server?

Depending on the environment you may have 1 or 2 ways.

1. add domain to realm mapping in the appropriate section in krb5.conf
on the client.
2. allow the KDC to send back a referral (but not all clients will ask
their own KDC, some can do only 1).

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

Re: How to use NFS with multiple principals in different realms?

[Cedric Blancher]

On 17 September 2014 17:05, Simo Sorce <simo@redhat.com> wrote:
> On Wed, 17 Sep 2014 13:20:19 +0200
> Cedric Blancher <cedric.blancher@gmail.com> wrote:
>
>> What happens if there is no relation between KRB Realm names and
>> FQDN/DNS? Can the NFS client find out which KRB Realm is used by the
>> server?
>
> Depending on the environment you may have 1 or 2 ways.
>
> 1. add domain to realm mapping in the appropriate section in krb5.conf
> on the client.
> 2. allow the KDC to send back a referral (but not all clients will ask
> their own KDC, some can do only 1).

But how can 1. help? Sure I can have my own krb5.conf but AFAIK
rpc.gssd only looks at he system /etc/krb5.conf and not at any custom
user defined location. Basically mount(8) would have to pass the
location of the custom krb5.conf file to rpc.gssd to facilitate the
mount, right?

I *think* we have a bigger problem here: Kerberos5 support in NFS
appears to be designed around the philosophy of one realm per machine
(one-to-rule-them'-all) and not that a single user or machine has
mounts from many different realms, right?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
Institute Pasteur

Re: How to use NFS with multiple principals in different realms?

[Simo Sorce]

On Wed, 17 Sep 2014 22:30:29 +0200
Cedric Blancher <cedric.blancher@gmail.com> wrote:

> On 17 September 2014 17:05, Simo Sorce <simo@redhat.com> wrote:
> > On Wed, 17 Sep 2014 13:20:19 +0200
> > Cedric Blancher <cedric.blancher@gmail.com> wrote:
> >
> >> What happens if there is no relation between KRB Realm names and
> >> FQDN/DNS? Can the NFS client find out which KRB Realm is used by
> >> the server?
> >
> > Depending on the environment you may have 1 or 2 ways.
> >
> > 1. add domain to realm mapping in the appropriate section in
> > krb5.conf on the client.
> > 2. allow the KDC to send back a referral (but not all clients will
> > ask their own KDC, some can do only 1).
> 
> But how can 1. help? Sure I can have my own krb5.conf but AFAIK
> rpc.gssd only looks at he system /etc/krb5.conf and not at any custom
> user defined location. Basically mount(8) would have to pass the
> location of the custom krb5.conf file to rpc.gssd to facilitate the
> mount, right?

A mount operation is a system-wide operation and requires privileges,
the system krb5.conf is what is used. Trusting a user provided
krb5.conf file for system level operations is not possible.

> I *think* we have a bigger problem here: Kerberos5 support in NFS
> appears to be designed around the philosophy of one realm per machine
> (one-to-rule-them'-all) and not that a single user or machine has
> mounts from many different realms, right?

wrong, the machine just need to 'know' about multiple realms and that
is done via domain_realm mappings, of course you can only have one
realm per dns domain.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York