mount hangs in NFS4+Kerberos setup

mount hangs in NFS4+Kerberos setup

[whats_up]

Hi, 

I want to setup a file server with NFS4+Kerberos and Debian squeeze for
clients running Ubuntu 11.10.

What is already working:
1) Mount NFS4 on client without krb5 option works. Users are able to
access files and uids/gids are correct. 2) KDC works. Access from
client, get tickets, user authentication/change password through pam is
ok.

Now I want to mount with sec=krb5 but this time the command hangs and
does not return to shell. See also logs below.

Any hints to fix the issue or to get more helpful debug information are
welcome.

regards
  knut




=== server status ===

Debian Linux squeeze

# uname -a
Linux tm 2.6.32-5-686 #1 SMP Mon Jan 16 16:04:25 UTC 2012 i686 GNU/Linux

# cat /etc/exports 
/srv        *(rw,sync,no_subtree_check,fsid=0,crossmnt)
/srv/opt    *(sec=sys:krb5,rw,sync,no_subtree_check)

# ps -ef|grep -e rpc -e portmap -e nfs
daemon    1019     1  0 10:26 ?        00:00:00 /sbin/portmap
root      1043     2  0 10:26 ?        00:00:00 [rpciod/0]
root      1044     2  0 10:26 ?        00:00:00 [rpciod/1]
root      1055     2  0 10:26 ?        00:00:00 [nfsiod]
root      1079     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.idmapd -vvv
root      1083     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.gssd -vvv
root      1309     2  0 10:26 ?        00:00:00 [nfsd4]
root      1310     2  0 10:26 ?        00:00:00 [nfsd]
...
root      1336     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.svcgssd -vvv
root      1338     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.mountd --manage-gids

# rpcdebug -m nfsd -s all


=== client status ===

Ubuntu Linux 10.11

# uname -a
Linux orest 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20 15:59:53 UTC 2012 i686 i686 i386 GNU/Linux

# ps -ef |grep rpc
root       702     1  0 13:16 ?        00:00:00 rpcbind -w
root       712     2  0 13:16 ?        00:00:00 [rpciod]
root       805     1  0 13:16 ?        00:00:00 rpc.gssd -vvv
root       890     1  0 13:16 ?        00:00:00 rpc.idmapd -vvv


=== client log ===

# mount -t nfs4 <server-fqdn>:/opt /opt
# umount /opt

# mount -t nfs4 -osec=krb5 <server-fqdn>:/opt /opt
Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:45:17 orest rpc.idmapd[728]: New client: 0
Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7102c data 0xbfe710ac
Feb 10 14:45:17 orest rpc.idmapd[728]: Opened /var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap
Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7102c data 0xbfe710ac
Feb 10 14:45:17 orest rpc.idmapd[728]: New client: 1
Feb 10 14:45:17 orest rpc.gssd[883]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
Feb 10 14:45:17 orest rpc.gssd[883]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
Feb 10 14:45:17 orest rpc.gssd[883]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
Feb 10 14:45:17 orest rpc.gssd[883]: process_krb5_upcall: service is '<null>'
Feb 10 14:45:17 orest rpc.gssd[883]: Full hostname for '<server-fqdn>' is '<server-fqdn>'
Feb 10 14:45:17 orest rpc.gssd[883]: Full hostname for '<client-fqdn>' is '<client-fqdn>'
Feb 10 14:45:17 orest rpc.gssd[883]: No key table entry found for <CLIENT-HOSTNAME>$@<MYREALM> while getting keytab entry for '<CLIENT-HOSTNAME>$@<MYREALM>'
Feb 10 14:45:17 orest rpc.gssd[883]: Success getting keytab entry for 'root/<client-fqdn>@<MYREALM>'
Feb 10 14:45:17 orest rpc.gssd[883]: Successfully obtained machine credentials for principal 'root/<client-fqdn>@<MYREALM>' stored in ccache 'FILE:/tmp/krb5cc_machine_<MYREALM>'
Feb 10 14:45:17 orest rpc.gssd[883]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_<MYREALM>' are good until 1328967917
Feb 10 14:45:17 orest rpc.gssd[883]: using FILE:/tmp/krb5cc_machine_<MYREALM> as credentials cache for machine creds
Feb 10 14:45:17 orest rpc.gssd[883]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_<MYREALM>
Feb 10 14:45:17 orest rpc.gssd[883]: creating context using fsuid 0 (save_uid 0)
Feb 10 14:45:17 orest rpc.gssd[883]: creating tcp client for server <server-fqdn>
Feb 10 14:45:17 orest rpc.gssd[883]: DEBUG: port already set to 2049
Feb 10 14:45:17 orest rpc.gssd[883]: creating context with server nfs@<server-fqdn>
Feb 10 14:45:17 orest rpc.gssd[883]: DEBUG: serialize_krb5_ctx: lucid version!
Feb 10 14:45:17 orest rpc.gssd[883]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
Feb 10 14:45:17 orest rpc.gssd[883]: doing downcall

(waiting...)

Feb 10 14:48:17 orest kernel: [  249.312058] nfs: server <server-fqdn> not responding, still trying

(press Ctrl-C, >10 min later)

Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:59:40 orest rpc.gssd[883]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt1
Feb 10 14:59:40 orest rpc.idmapd[728]: Stale client: 1
Feb 10 14:59:40 orest rpc.idmapd[728]: #011-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt1/idmap
Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:59:40  rpc.gssd[883]: last message repeated 4 times
Feb 10 14:59:40 orest rpc.idmapd[728]: Stale client: 0
Feb 10 14:59:40 orest rpc.idmapd[728]: #011-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap
Feb 10 14:59:40 orest rpc.gssd[883]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt0



=== server log ===

The following messages appeared while initiating mount command on client:

==> auth.log <==
Feb 10 14:45:17 tm krb5kdc[1429]: AS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: NEEDED_PREAUTH: root/<client-fqdn>@<MYREALM> for krbtgt/<MYREALM>@<MYREALM>, Additional pre-authentication required
Feb 10 14:45:17 tm krb5kdc[1429]: AS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: ISSUE: authtime 1328881517, etypes {rep=1 tkt=18 ses=18}, root/<client-fqdn>@<MYREALM> for krbtgt/<MYREALM>@<MYREALM>
Feb 10 14:45:17 tm krb5kdc[1429]: TGS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: ISSUE: authtime 1328881517, etypes {rep=18 tkt=1 ses=1}, root/<client-fqdn>@<MYREALM> for nfs/<server-fqdn>@<MYREALM>

==> syslog <==
Feb 10 14:45:17 tm rpc.svcgssd[1335]: leaving poll
Feb 10 14:45:17 tm rpc.svcgssd[1335]: handling null request
Feb 10 14:45:17 tm rpc.svcgssd[1335]: sname = root/<client-fqdn>@<MYREALM>
Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: using (default) domain: cs.uni-potsdam.de
Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/nsswitch.so for method nsswitch
Feb 10 14:45:17 tm rpc.svcgssd[1335]: nss_gss_princ_to_ids: Local-Realm '<MYREALM>': NOT FOUND
Feb 10 14:45:17 tm rpc.svcgssd[1335]: DEBUG: serialize_krb5_ctx: lucid version!
Feb 10 14:45:17 tm rpc.svcgssd[1335]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
Feb 10 14:45:17 tm rpc.svcgssd[1335]: doing downcall
Feb 10 14:45:17 tm rpc.svcgssd[1335]: mech: krb5, hndl len: 4, ctx len 85, timeout: 1328967917 (86400 from now), clnt: root@<client-fqdn>, uid: -1, gid: -1, num aux grps: 0:
Feb 10 14:45:17 tm rpc.svcgssd[1335]: sending null reply
Feb 10 14:45:17 tm rpc.svcgssd[1335]: writing message: \x \x..<lots of digits> 1328881577 0 0 \x01000000 \x607006092a864886f71201020202006f61305fa003020105a10302010fa2533051a003020101a24a0448d85b3308847004f703039fd5f315f81fcbb287c6a53821eb2ea8660169c687bbb0f1d7fbdecb9016284f3adf11b122949b98488862d956d352c6c14e6fd2beb565ba1de7afc99d1c 
Feb 10 14:45:17 tm rpc.svcgssd[1335]: finished handling null request
Feb 10 14:45:17 tm rpc.svcgssd[1335]: entering poll

(waiting...)

Re: mount hangs in NFS4+Kerberos setup

[Sven Geggus]

whats_up@gmx.net wrote:

> Any hints to fix the issue or to get more helpful debug information are
> welcome.

I would at least recomend using a recent kernel and backports of nfs-common,
rpcbind and libtirpc (compiling them is very straight forward).

Sven

-- 
"Ich fürchte mich nicht vor der Rückkehr der Faschisten in der Maske der
Faschisten, sondern vor der Rückkehr der Faschisten in der Maske der
Demokraten" (Theodor W. Adorno)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web

Re: mount hangs in NFS4+Kerberos setup

[Andy Adamson]

Hi

It appears that the RPCSEC_GSS Kerberos calls were successful, but
that the Kerberos principal to id mapping failed.

> Feb 10 14:45:17 tm rpc.svcgssd[1335]: sname = root/<client-fqdn>@<MYREALM>
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: using (default) domain: cs.uni-potsdam.de
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/nsswitch.so for method nsswitch
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: nss_gss_princ_to_ids: Local-Realm '<MYREALM>': NOT FOUND

What kernel is the server running?
What nfs-utils version is the server using?
What libnfsidmap version is the server using?

Thanks

-->Andy




On Fri, Feb 10, 2012 at 9:45 AM,  <whats_up@gmx.net> wrote:
>
> Hi,
>
> I want to setup a file server with NFS4+Kerberos and Debian squeeze for
> clients running Ubuntu 11.10.
>
> What is already working:
> 1) Mount NFS4 on client without krb5 option works. Users are able to
> access files and uids/gids are correct. 2) KDC works. Access from
> client, get tickets, user authentication/change password through pam is
> ok.
>
> Now I want to mount with sec=krb5 but this time the command hangs and
> does not return to shell. See also logs below.
>
> Any hints to fix the issue or to get more helpful debug information are
> welcome.
>
> regards
>  knut
>
>
>
>
> === server status ===
>
> Debian Linux squeeze
>
> # uname -a
> Linux tm 2.6.32-5-686 #1 SMP Mon Jan 16 16:04:25 UTC 2012 i686 GNU/Linux
>
> # cat /etc/exports
> /srv        *(rw,sync,no_subtree_check,fsid=0,crossmnt)
> /srv/opt    *(sec=sys:krb5,rw,sync,no_subtree_check)
>
> # ps -ef|grep -e rpc -e portmap -e nfs
> daemon    1019     1  0 10:26 ?        00:00:00 /sbin/portmap
> root      1043     2  0 10:26 ?        00:00:00 [rpciod/0]
> root      1044     2  0 10:26 ?        00:00:00 [rpciod/1]
> root      1055     2  0 10:26 ?        00:00:00 [nfsiod]
> root      1079     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.idmapd -vvv
> root      1083     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.gssd -vvv
> root      1309     2  0 10:26 ?        00:00:00 [nfsd4]
> root      1310     2  0 10:26 ?        00:00:00 [nfsd]
> ...
> root      1336     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.svcgssd -vvv
> root      1338     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.mountd --manage-gids
>
> # rpcdebug -m nfsd -s all
>
>
> === client status ===
>
> Ubuntu Linux 10.11
>
> # uname -a
> Linux orest 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20 15:59:53 UTC 2012 i686 i686 i386 GNU/Linux
>
> # ps -ef |grep rpc
> root       702     1  0 13:16 ?        00:00:00 rpcbind -w
> root       712     2  0 13:16 ?        00:00:00 [rpciod]
> root       805     1  0 13:16 ?        00:00:00 rpc.gssd -vvv
> root       890     1  0 13:16 ?        00:00:00 rpc.idmapd -vvv
>
>
> === client log ===
>
> # mount -t nfs4 <server-fqdn>:/opt /opt
> # umount /opt
>
> # mount -t nfs4 -osec=krb5 <server-fqdn>:/opt /opt
> Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
> Feb 10 14:45:17 orest rpc.idmapd[728]: New client: 0
> Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7102c data 0xbfe710ac
> Feb 10 14:45:17 orest rpc.idmapd[728]: Opened /var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap
> Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7102c data 0xbfe710ac
> Feb 10 14:45:17 orest rpc.idmapd[728]: New client: 1
> Feb 10 14:45:17 orest rpc.gssd[883]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
> Feb 10 14:45:17 orest rpc.gssd[883]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
> Feb 10 14:45:17 orest rpc.gssd[883]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
> Feb 10 14:45:17 orest rpc.gssd[883]: process_krb5_upcall: service is '<null>'
> Feb 10 14:45:17 orest rpc.gssd[883]: Full hostname for '<server-fqdn>' is '<server-fqdn>'
> Feb 10 14:45:17 orest rpc.gssd[883]: Full hostname for '<client-fqdn>' is '<client-fqdn>'
> Feb 10 14:45:17 orest rpc.gssd[883]: No key table entry found for <CLIENT-HOSTNAME>$@<MYREALM> while getting keytab entry for '<CLIENT-HOSTNAME>$@<MYREALM>'
> Feb 10 14:45:17 orest rpc.gssd[883]: Success getting keytab entry for 'root/<client-fqdn>@<MYREALM>'
> Feb 10 14:45:17 orest rpc.gssd[883]: Successfully obtained machine credentials for principal 'root/<client-fqdn>@<MYREALM>' stored in ccache 'FILE:/tmp/krb5cc_machine_<MYREALM>'
> Feb 10 14:45:17 orest rpc.gssd[883]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_<MYREALM>' are good until 1328967917
> Feb 10 14:45:17 orest rpc.gssd[883]: using FILE:/tmp/krb5cc_machine_<MYREALM> as credentials cache for machine creds
> Feb 10 14:45:17 orest rpc.gssd[883]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_<MYREALM>
> Feb 10 14:45:17 orest rpc.gssd[883]: creating context using fsuid 0 (save_uid 0)
> Feb 10 14:45:17 orest rpc.gssd[883]: creating tcp client for server <server-fqdn>
> Feb 10 14:45:17 orest rpc.gssd[883]: DEBUG: port already set to 2049
> Feb 10 14:45:17 orest rpc.gssd[883]: creating context with server nfs@<server-fqdn>
> Feb 10 14:45:17 orest rpc.gssd[883]: DEBUG: serialize_krb5_ctx: lucid version!
> Feb 10 14:45:17 orest rpc.gssd[883]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
> Feb 10 14:45:17 orest rpc.gssd[883]: doing downcall
>
> (waiting...)
>
> Feb 10 14:48:17 orest kernel: [  249.312058] nfs: server <server-fqdn> not responding, still trying
>
> (press Ctrl-C, >10 min later)
>
> Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
> Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
> Feb 10 14:59:40 orest rpc.gssd[883]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt1
> Feb 10 14:59:40 orest rpc.idmapd[728]: Stale client: 1
> Feb 10 14:59:40 orest rpc.idmapd[728]: #011-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt1/idmap
> Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
> Feb 10 14:59:40  rpc.gssd[883]: last message repeated 4 times
> Feb 10 14:59:40 orest rpc.idmapd[728]: Stale client: 0
> Feb 10 14:59:40 orest rpc.idmapd[728]: #011-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap
> Feb 10 14:59:40 orest rpc.gssd[883]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt0
>
>
>
> === server log ===
>
> The following messages appeared while initiating mount command on client:
>
> ==> auth.log <==
> Feb 10 14:45:17 tm krb5kdc[1429]: AS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: NEEDED_PREAUTH: root/<client-fqdn>@<MYREALM> for krbtgt/<MYREALM>@<MYREALM>, Additional pre-authentication required
> Feb 10 14:45:17 tm krb5kdc[1429]: AS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: ISSUE: authtime 1328881517, etypes {rep=1 tkt=18 ses=18}, root/<client-fqdn>@<MYREALM> for krbtgt/<MYREALM>@<MYREALM>
> Feb 10 14:45:17 tm krb5kdc[1429]: TGS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: ISSUE: authtime 1328881517, etypes {rep=18 tkt=1 ses=1}, root/<client-fqdn>@<MYREALM> for nfs/<server-fqdn>@<MYREALM>
>
> ==> syslog <==
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: leaving poll
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: handling null request
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: sname = root/<client-fqdn>@<MYREALM>
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: using (default) domain: cs.uni-potsdam.de
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/nsswitch.so for method nsswitch
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: nss_gss_princ_to_ids: Local-Realm '<MYREALM>': NOT FOUND
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: DEBUG: serialize_krb5_ctx: lucid version!
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: doing downcall
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: mech: krb5, hndl len: 4, ctx len 85, timeout: 1328967917 (86400 from now), clnt: root@<client-fqdn>, uid: -1, gid: -1, num aux grps: 0:
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: sending null reply
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: writing message: \x \x..<lots of digits> 1328881577 0 0 \x01000000 \x607006092a864886f71201020202006f61305fa003020105a10302010fa2533051a003020101a24a0448d85b3308847004f703039fd5f315f81fcbb287c6a53821eb2ea8660169c687bbb0f1d7fbdecb9016284f3adf11b122949b98488862d956d352c6c14e6fd2beb565ba1de7afc99d1c
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: finished handling null request
> Feb 10 14:45:17 tm rpc.svcgssd[1335]: entering poll
>
> (waiting...)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mount hangs in NFS4+Kerberos setup

[steve]

On 02/10/2012 03:45 PM, whats_up@gmx.net wrote:
> Hi,
>
> I want to setup a file server with NFS4+Kerberos and Debian squeeze for
> clients running Ubuntu 11.10.
>
> What is already working:
> 1) Mount NFS4 on client without krb5 option works. Users are able to
> access files and uids/gids are correct. 2) KDC works. Access from
> client, get tickets, user authentication/change password through pam is
> ok.
>
> Now I want to mount with sec=krb5 but this time the command hangs and
> does not return to shell. See also logs below.
>
> Any hints to fix the issue or to get more helpful debug information are
> welcome.
>
> regards
>    knut
>
>
>
>
> === server status ===
>
> Debian Linux squeeze
>
> # uname -a
> Linux tm 2.6.32-5-686 #1 SMP Mon Jan 16 16:04:25 UTC 2012 i686 GNU/Linux
Ubuntu 11.10
uname -r
3.0.0-15-generic

Some older kernels do not support strong keys. Try adding:
allow_weak_crypto = true
to the
  [libdefaults]
in /etc/krb5.conf

Here it is using the machine principal with arcfour:

Kerberos: AS-REQ nfs/hh3.hh3.site@HH3.SITE from ipv4:192.168.1.3:49650 
for krbtgt/HH3.SITE@HH3.SITE
Kerberos: UNKNOWN -- nfs/hh3.hh3.site@HH3.SITE: no such entry found in hdb
Kerberos: AS-REQ HH3$@HH3.SITE from ipv4:192.168.1.3:43041 for 
krbtgt/HH3.SITE@HH3.SITE
Kerberos: Client sent patypes: 149
Kerberos: Looking for PKINIT pa-data -- HH3$@HH3.SITE
Kerberos: Looking for ENC-TS pa-data -- HH3$@HH3.SITE
Kerberos: No preauth found, returning PREAUTH-REQUIRED -- HH3$@HH3.SITE
Kerberos: AS-REQ HH3$@HH3.SITE from ipv4:192.168.1.3:32850 for 
krbtgt/HH3.SITE@HH3.SITE
Kerberos: Client sent patypes: encrypted-timestamp, 149
Kerberos: Looking for PKINIT pa-data -- HH3$@HH3.SITE
Kerberos: Looking for ENC-TS pa-data -- HH3$@HH3.SITE
Kerberos: ENC-TS Pre-authentication succeeded -- HH3$@HH3.SITE using 
arcfour-hmac-md5
Kerberos: AS-REQ authtime: 2012-02-10T18:00:16 starttime: unset endtime: 
2012-02-11T04:00:16 renew till: 2012-02-11T18:00:15
Kerberos: Client supported enctypes: aes256-cts-hmac-sha1-96, 
aes128-cts-hmac-sha1-96, des3-cbc-sha1, arcfour-hmac-md5, using 
arcfour-hmac-md5/arcfour-hmac-md5
Kerberos: Requested flags: renewable-ok
Kerberos: TGS-REQ HH3$@HH3.SITE from ipv4:192.168.1.3:41288 for 
nfs/hh3.hh3.site@HH3.SITE [canonicalize, renewable]
Kerberos: TGS-REQ authtime: 2012-02-10T18:00:16 starttime: 
2012-02-10T18:00:16 endtime: 2012-02-11T04:00:16 renew till: 
2012-02-11T18:00:15

Also it's not recommended to use the pseudo-root fsid=0 method for nfs 
exports under Linux:
  http://wiki.linux-nfs.org/wiki/index.php/Nfsv4_configuration

HTH,
Steve

Re: mount hangs in NFS4+Kerberos setup

[whats_up]

> Some older kernels do not support strong keys. Try adding:
> allow_weak_crypto = true
> to the
>   [libdefaults]
> in /etc/krb5.conf

yes. I painfully (mount only says access denied) found out this already
and I use allow_weak_crypto to limit to DES. More encryption
types have been introduced with kernel 2.6.39...

I tried to use kernel 3.2 from squeeze-backports but this introduced new
errors, thus I decided to try with 2.6 first.


> Also it's not recommended to use the pseudo-root fsid=0 method for
> nfs exports under Linux:
>   http://wiki.linux-nfs.org/wiki/index.php/Nfsv4_configuration

hmm, as far as I have understood I have to:
- export the root folder /exports explicitly beside the "real"
  exports p.ex. /exports/opt
- use fsid=0 for the root folder to force version 4 of NFS

What's your suggestion to improve/secure my configuration?

regards
  knut

Re: mount hangs in NFS4+Kerberos setup

[steve]

On 02/10/2012 06:41 PM, whats_up@gmx.net wrote:
>
>> Some older kernels do not support strong keys. Try adding:
>> allow_weak_crypto = true
>> to the
>>    [libdefaults]
>> in /etc/krb5.conf
> yes. I painfully (mount only says access denied) found out this already
> and I use allow_weak_crypto to limit to DES. More encryption
> types have been introduced with kernel 2.6.39...
>
> I tried to use kernel 3.2 from squeeze-backports but this introduced new
> errors, thus I decided to try with 2.6 first.
>
>
>> Also it's not recommended to use the pseudo-root fsid=0 method for
>> nfs exports under Linux:
>>    http://wiki.linux-nfs.org/wiki/index.php/Nfsv4_configuration
> hmm, as far as I have understood I have to:
> - export the root folder /exports explicitly beside the "real"
>    exports p.ex. /exports/opt
> - use fsid=0 for the root folder to force version 4 of NFS
>
> What's your suggestion to improve/secure my configuration?
>
> regards
>    knut
Officially, you should not export from a pseudo root. Please see the 
last few lines in the link I sent.
man rpc.gssd(8) adds:
<quote>
Previous versions of
rpc.gssd used only "nfs/*" keys found within the keytab. To be more 
consistent with other implementations, we now look for specific keytab 
entries. The search order for keytabs to be used for "machine 
credentials" is now:
<HOSTNAME>$@<REALM>
root/<hostname>@<REALM>

nfs/<hostname>@<REALM>

host/<hostname>@<REALM>

root/<anyname>@<REALM>

nfs/<anyname>@<REALM>

host/<anyname>@<REALM>
</quote>

I see your setup uses the root principal. If you still get access 
denied, create another keytab with just the machine$ and host/fqdn keys. 
I can remember having to fiddle with nfs-utils and keytabs on openSUSE 
at some stage last year.

If none of this works you can either stick with the old kernel and 
accept he security, get an up to date nfs-utils and see if hat fixes it 
with the DES keys or grab an up to date distro where all this stuff will 
work out of the box.
Cheers,
Steve

Re: mount hangs in NFS4+Kerberos setup

[Andy Adamson]

On Fri, Feb 10, 2012 at 11:25 AM,  <whats_up@gmx.net> wrote:
>
>> Hi
>>
>> It appears that the RPCSEC_GSS Kerberos calls were successful, but
>> that the Kerberos principal to id mapping failed.
>
> Is this influenced by /etc/idmapd.conf?

Yes. libnfsidmapd.023 nss_gss_princ_to_id checks the Kerberos REALM
passed in the sname against the configured REALMS in /etc/idmapd.conf
as explained:




[General]
#Verbosity = 0
# The following should be set to the local NFSv4 domain name
# The default is the host's DNS domain name.
#Domain = local.domain.edu

# The following is a comma-separated list of Kerberos realm
# names that should be considered to be equivalent to the
# local realm, such that <user>@REALM.A can be assumed to
# be the same user as <user>@REALM.B
# If not specified, the default local realm is the domain name,
# which defaults to the host's DNS domain name,
# translated to upper-case.
# Note that if this value is specified, the local realm name
# must be included in the list!
#Local-Realms =


I believe you have not set the Local-Realms, so libnfsidmapd.023 uses
the default of the upper-case of the local domain-name. Thus the
nss_gss_print_ids error message:

> Feb 10 14:45:17 tm rpc.svcgssd[1335]: nss_gss_princ_to_ids: Local-Realm '<MYREALM>': NOT FOUND

Try setting the Local-Realms =  in /etc/idmapd.conf.

-->Andy


>  I played with "Domain" and
> "Local-Realm" but I didn't understand the exact meaning. Server and
> client aren't in the same subdomain:
> server:  hostname.subdomain.domain.tld
> client:  hostname.subdomain.subdomain.domain.tld
> Is this a problem?
>
>> What kernel is the server running?
>> What nfs-utils version is the server using?
>> What libnfsidmap version is the server using?
>
> I'm using Debian squeeze with updates.
>
> $ uname -a
> Linux tm 2.6.32-5-686 #1 SMP Mon Jan 16 16:04:25 UTC 2012 i686 GNU/Linux
>
> $ dpkg -l nfs-\*
> un  nfs-client                             <none>                                 (no description available)
> ii  nfs-common                             1:1.2.2-4squeeze2                      NFS support files common to client and server
> ii  nfs-kernel-server                      1:1.2.2-4squeeze2                      support for NFS kernel server
> un  nfs-server                             <none>                                 (no description available)
>
> $ dpkg -l libnfsidmap\*
> un  libnfsidmap1                           <none>                                 (no description available)
> ii  libnfsidmap2                           0.23-2                                 An nfs idmapping library
>
>
> regards
>  knut

Re: mount hangs in NFS4+Kerberos setup

[Daniel Kahn Gillmor]

On 02/10/2012 01:07 PM, steve wrote:
> Officially, you should not export from a pseudo root. Please see the
> last few lines in the link I sent.

That link offers the same assertion but no explanation for it.  Can you
explain why that's the recommendation?

If an admin were to remove fsid=0 from their exports line, they'd also
need to adjust the paths in all the connecting clients, right?

	--dkg

Re: mount hangs in NFS4+Kerberos setup

[J. Bruce Fields]

On Fri, Feb 10, 2012 at 01:21:46PM -0500, Daniel Kahn Gillmor wrote:
> On 02/10/2012 01:07 PM, steve wrote:
> > Officially, you should not export from a pseudo root. Please see the
> > last few lines in the link I sent.
> 
> That link offers the same assertion but no explanation for it.  Can you
> explain why that's the recommendation?
> 
> If an admin were to remove fsid=0 from their exports line, they'd also
> need to adjust the paths in all the connecting clients, right?

Yep.

The main problem with the fsid=0 trick is that your v3 and v4 clients
end up with different paths.

--b.

Re: mount hangs in NFS4+Kerberos setup

[steve]

On 02/10/2012 07:21 PM, Daniel Kahn Gillmor wrote:
> On 02/10/2012 01:07 PM, steve wrote:
>> Officially, you should not export from a pseudo root. Please see the
>> last few lines in the link I sent.
> That link offers the same assertion but no explanation for it.  Can you
> explain why that's the recommendation?
Sorry, no. I can however explain why we changed: We were having problems 
with idmapd uid:gid mappings from the bind mount and just as an 
experiment we tried the old way. Problem solved.
>
> If an admin were to remove fsid=0 from their exports line, they'd also
> need to adjust the paths in all the connecting clients, right?
>
> 	--dkg
No. If you are exporting /export fsid-0 and crossmnt-ing /export/opt, 
you mount server:/opt client/somewhere. The only file you change is 
/etc/exports on the server, to contain the old style mount spec.
/opt yourIPs(rw,sec=none:sys:krb5. . .)
The client sees exactly he same path.

HTH
Steve

Re: mount hangs in NFS4+Kerberos setup

[J. Bruce Fields]

On Fri, Feb 10, 2012 at 08:06:31PM +0100, steve wrote:
> On 02/10/2012 07:21 PM, Daniel Kahn Gillmor wrote:
> >On 02/10/2012 01:07 PM, steve wrote:
> >>Officially, you should not export from a pseudo root. Please see the
> >>last few lines in the link I sent.
> >That link offers the same assertion but no explanation for it.  Can you
> >explain why that's the recommendation?
> Sorry, no. I can however explain why we changed: We were having
> problems with idmapd uid:gid mappings from the bind mount and just
> as an experiment we tried the old way. Problem solved.

That almost had to be a coincidence--there's no connection between the
two things.

--b.

> >If an admin were to remove fsid=0 from their exports line, they'd also
> >need to adjust the paths in all the connecting clients, right?
> >
> >	--dkg
> No. If you are exporting /export fsid-0 and crossmnt-ing
> /export/opt, you mount server:/opt client/somewhere. The only file
> you change is /etc/exports on the server, to contain the old style
> mount spec.
> /opt yourIPs(rw,sec=none:sys:krb5. . .)
> The client sees exactly he same path.
> 
> HTH
> Steve
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mount hangs in NFS4+Kerberos setup

[whats_up]

> I believe you have not set the Local-Realms, so libnfsidmapd.023 uses
> the default of the upper-case of the local domain-name. Thus the
> nss_gss_print_ids error message:
> 
> > Feb 10 14:45:17 tm rpc.svcgssd[1335]: nss_gss_princ_to_ids:
> > Local-Realm '<MYREALM>': NOT FOUND
> 
> Try setting the Local-Realms =  in /etc/idmapd.conf.

I'm a bit confused because <MYREALM> in the error message is the
correct realm.

I altered the idmapd.conf (server and client). Just be be sure we speak
about the same things:

<MYREALM>:      MYSERVERHOSTNAME.SUB1.DOMAIN.TLD
server:         myserverhostname.sub1.domain.tld
client:    myclienthostname.sub2.sub1.domain.tld

/etc/idmapd.conf  
Domain = myserverhostname.sub1.domain.tld
Local-Realm = MYSERVERHOSTNAME.SUB1.DOMAIN.TLD

> # The following should be set to the local NFSv4 domain name
> # The default is the host's DNS domain name.
> #Domain = local.domain.edu

I wondered if a "local domain name" should include the hostname or not,
thus I tried sub1.domain.tld and also myserverhostname.sub1.domain.tld
The later worked and let the error message disappear from the log. The
rest is the same and mount still hangs.

regards
 knut


server log:

Feb 13 10:23:29 tm rpc.svcgssd[18043]: leaving poll
Feb 13 10:23:29 tm rpc.svcgssd[18043]: handling null request
Feb 13 10:23:29 tm rpc.svcgssd[18043]: sname = root/<client-fqdn>@<MYREALM>
Feb 13 10:23:29 tm rpc.svcgssd[18043]: DEBUG: serialize_krb5_ctx: lucid version!
Feb 13 10:23:29 tm rpc.svcgssd[18043]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
Feb 13 10:23:29 tm rpc.svcgssd[18043]: doing downcall
Feb 13 10:23:29 tm rpc.svcgssd[18043]: mech: krb5, hndl len: 4, ctx len 85, timeout: 1329211409 (86400 from now), clnt: root@<client-fqdn>, uid: -1, gid: -1, num aux grps: 0:
Feb 13 10:23:29 tm rpc.svcgssd[18043]: sending null reply
Feb 13 10:23:29 tm rpc.svcgssd[18043]: writing message: \x... 1329125069 0 0 \x02000000 \x...
Feb 13 10:23:29 tm rpc.svcgssd[18043]: finished handling null request
Feb 13 10:23:29 tm rpc.svcgssd[18043]: entering poll

Re: mount hangs in NFS4+Kerberos setup

[whats_up]

> Officially, you should not export from a pseudo root. Please see the 
> last few lines in the link I sent.

I removed the pseudo root entry.
/etc/exports:
/srv/opt    *(sec=sys:krb5,rw,sync,no_subtree_check)

but now I get:
Feb 13 10:43:45 tm mountd[18045]: Kernel does not have pseudo root support.
Feb 13 10:43:45 tm mountd[18045]: NFS v4 mounts will be disabled unless fsid=0
Feb 13 10:43:45 tm mountd[18045]: is specfied in /etc/exports file.

Also I can't mount on client anymore:

# mount -t nfs4 <server-fqdn>:/opt /opt
mount.nfs4: mounting ...:/opt failed, reason given by server:
  No such file or directory

# mount -t nfs4 :/srv/opt /opt
mount.nfs4: mounting ...:/srv/opt failed, reason given by server:
  No such file or directory



> man rpc.gssd(8) adds:
> <quote>
> Previous versions of
> rpc.gssd used only "nfs/*" keys found within the keytab. To be more 
> consistent with other implementations, we now look for specific
> keytab entries. The search order for keytabs to be used for "machine 
> credentials" is now:
> <HOSTNAME>$@<REALM>
> root/<hostname>@<REALM>
> nfs/<hostname>@<REALM>
> host/<hostname>@<REALM>

> I see your setup uses the root principal. If you still get access 
> denied, create another keytab with just the machine$ and host/fqdn
> keys. I can remember having to fiddle with nfs-utils and keytabs on
> openSUSE at some stage last year.

I started with host/... and nfs/... principals and got an access denied
error while mounting. Thus I added the root principal also. 

Which of the four mentioned keys are necessary, resp. which combinations
are sufficient? Or do I always need host, nfs and root?

regards
  knut

Re: mount hangs in NFS4+Kerberos setup

[Sven Geggus]

whats_up@gmx.net wrote:

> What's your suggestion to improve/secure my configuration?

I already told you that you need a backport of the latest version of
libtirpc. The Version included in squeeze is broken.

Sven

-- 
If you can spend five minutes on the Internet and do not run Linux,
you're a genius. (Dirk Hohndel)

/me is giggls@ircnet, http://sven.gegg.us/ on the Web

Re: mount hangs in NFS4+Kerberos setup

[whats_up]

> > What's your suggestion to improve/secure my configuration?
> 
> I already told you that you need a backport of the latest version of
> libtirpc. The Version included in squeeze is broken.

You recommended to use backports. As you now say the lib is broken
things are different and I finally solved my problem.

For those who face similar issues I sum up my last steps:

- include squeeze-backports and upgrade nfs-common, nfs-kernel-server
  to version 1.2.4 and Linux kernel to 3.2
- replace portmap by rpcbind
- install version 0.2.2 of libtirpc from unstable (forced new libc6)
- remove pseudo root from /etc/exports
- use AES keys for Kerberos

Thanks to all for your helpful hints! :)

regards
  knut


------

# dpkg -l ...
ii  libnfsidmap2                                0.23-2                                      An nfs idmapping library
ii  nfs-common                                  1:1.2.4-1~bpo60+1                           NFS support files common to client and server
ii  nfs-kernel-server                           1:1.2.4-1~bpo60+1                           support for NFS kernel server

ii  libgssrpc4                                  1.8.3+dfsg-4squeeze5                        MIT Kerberos runtime libraries - GSS enabled ONCRPC
ii  librpcsecgss3                               0.19-2                                      allows secure rpc communication using the rpcsec_gss protocol
ii  libtirpc1                                   0.2.2-5                                     transport-independent RPC library
ii  rpcbind                                     0.2.0-4.1                                   converts RPC program numbers into universal addresses

ii  linux-image-3.2.0-0.bpo.1-686-pae           3.2.4-1~bpo60+1                             Linux 3.2 for modern PCs

Re: mount hangs in NFS4+Kerberos setup

[Daniel Kahn Gillmor]

On 02/13/2012 01:50 PM, whats_up@gmx.net wrote:
> 
> - include squeeze-backports and upgrade nfs-common, nfs-kernel-server
>   to version 1.2.4 and Linux kernel to 3.2
> - replace portmap by rpcbind
> - install version 0.2.2 of libtirpc from unstable (forced new libc6)

It sounds like you'd like to request a backport of libtirpc to
squeeze-backports.  is that right?  that is best done on the debian
backports mailing list (cc'ed here)

I've set reply-to debian-backports@lists.debian.org, so please follow up
there if you'd like to see this package backported.

Regards,

	--dkg

Re: mount hangs in NFS4+Kerberos setup

[Sven Geggus]

whats_up@gmx.net wrote:

> - install version 0.2.2 of libtirpc from unstable (forced new libc6)

Don't do this. Just backport yourself. This is trivial in that case (apt-get
source + dpkg-buildpackage)

Sven

-- 
Das Internet ist kein rechtsfreier Raum, das Internet ist aber auch
kein bürgerrechtsfreier Raum. (Wolfgang Wieland Bündnis 90/Die Grünen)

/me is giggls@ircnet, http://sven.gegg.us/ on the Web