Linux-NFS Archive on lore.kernel.org
 help / color / Atom feed
From: Simo Sorce <simo@redhat.com>
To: Chuck Lever <chuck.lever@oracle.com>
Cc: Jeff Layton <jlayton@redhat.com>,
	Bruce Fields <bfields@fieldses.org>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
	Robbie Harwood <rharwood@redhat.com>
Subject: Re: Fedora 32 rpc.gssd misbehavior
Date: Thu, 30 Jul 2020 14:55:18 -0400
Message-ID: <33e219ec909a7ef35589d4ee9c8f6c45c4dacc6f.camel@redhat.com> (raw)
In-Reply-To: <44BF8895-5018-4743-A4DD-C0734448BE15@oracle.com>

On Thu, 2020-07-30 at 14:29 -0400, Chuck Lever wrote:
> > On Jul 30, 2020, at 2:20 PM, Simo Sorce <simo@redhat.com> wrote:
> > 
> > On Thu, 2020-07-30 at 14:07 -0400, Chuck Lever wrote:
> > > > On Jul 30, 2020, at 1:57 PM, Simo Sorce <simo@redhat.com> wrote:
> > > > 
> > > > On Thu, 2020-07-30 at 13:09 -0400, Chuck Lever wrote:
> > > > > > On Jul 30, 2020, at 12:14 PM, Simo Sorce <simo@redhat.com> wrote:
> > > > > > 
> > > > > > On Wed, 2020-07-29 at 14:27 -0400, Chuck Lever wrote:
> > > > > > > > On Jul 29, 2020, at 1:19 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
> > > > > > > > 
> > > > > > > > Hi!
> > > > > > > > 
> > > > > > > > I recently updated my test systems from EL7 to Fedora 32, and
> > > > > > > > NFSv4.0 with Kerberos has stopped working.
> > > > > > > > 
> > > > > > > > I mount with "klimt.ib" as before. The client workload stops
> > > > > > > > dead when the server tries to perform its first CB_RECALL.
> > > > > > > > 
> > > > > > > > I added some client instrumentation:
> > > > > > > > 
> > > > > > > > kernel: NFSv4: Callback principal (nfs@klimt.ib.1015granger.net) does not match acceptor (nfs@klimt.ib).
> > > > > > > > kernel: NFS: NFSv4 callback contains invalid cred
> > > > > > > > 
> > > > > > > > I boosted gssd verbosity, and it says:
> > > > > > > > 
> > > > > > > > rpc.gssd[986]: doing downcall: lifetime_rec=72226 acceptor=nfs@klimt.ib
> > > > > > > > 
> > > > > > > > But it knows the full hostname for the server:
> > > > > > > > 
> > > > > > > > rpc.gssd[986]: Full hostname for 'klimt.ib' is 'klimt.ib.1015granger.net'
> > > > > > > > 
> > > > > > > > 
> > > > > > > > The acceptor appears to come from the Kerberos library. Shouldn't
> > > > > > > > it be canonicalized? If so, should the Kerberos library do it, or
> > > > > > > > should gssd? Since this behavior appeared after an upgrade, I
> > > > > > > > suspect a Kerberos library regression. But it could be config-
> > > > > > > > related, since both systems were re-imaged from the ground up.
> > > > > > > > 
> > > > > > > > Also noticing some other problems on the server (missing hostname
> > > > > > > > strings in debug messages, sssd_kcm infinite loops, and gssd
> > > > > > > > sending garbage to the client after the NULL request that
> > > > > > > > establishes the callback context).
> > > > > > > > 
> > > > > > > > But let's look at the client acceptor problem first.
> > > > > > > 
> > > > > > > I believe I found the problem.
> > > > > > > 
> > > > > > > 8bffe8c5ec1a ("gssd: add /etc/nfs.conf support") added a number of gssd config
> > > > > > > options to /etc/nfs.conf, including "avoid-dns". The default setting of avoid-
> > > > > > > dns is 1. When I set this option on my client system explicitly to 0, NFSv4.0
> > > > > > > with Kerberos works again.
> > > > > > > 
> > > > > > > Is there a reason the default setting is 1?
> > > > > > > 
> > > > > > 
> > > > > > Now that you mention DNS, this may be an interaction between a new
> > > > > > default in Fedora 32 and how your environment is setup re DNS.
> > > > > > 
> > > > > > In F32 we changed the option dns_canonicalize_hostname from 'true' to
> > > > > > 'fallback'.
> > > > > > This is a transitional state to eventually move it to 'false' at some
> > > > > > point in the future.
> > > > > > 
> > > > > > What it changes in practice is that it will first try the name passed
> > > > > > in *as is* and only as a fallback try a CNAME if the name passed is not
> > > > > > resolved as an A name. If you have principals in the KDC for both
> > > > > > names, but you do not have keys in the keytab for both, you can have
> > > > > > transitional issues.
> > > > > > 
> > > > > > Additionally we discovered a bug that causes non qualified names to
> > > > > > fail resolution with the 'fallback' option.
> > > > > > If your name in the principal is really not qualified it will try to
> > > > > > qualify it anyway, so if your principal is literally nfs/foo@FOO
> > > > > > libgssapi may try to use nfs/foo.my.domdain@FOO, where "my.domain" is
> > > > > > what is defined in resolv.conf search path.
> > > > > > 
> > > > > > We are trying to address this regression.
> > > > > > 
> > > > > > So try to set dns_canonicalize_hostname to true to see if that may
> > > > > > influence your issue. If so, please let me know, as we still need to
> > > > > > address this where possible.
> > > > > 
> > > > > I set avoid-dns to 1 and dns_canonicalize_hostname to true. The
> > > > > workload hang is not reproducible, and the acceptor is fully qualified.
> > > > > 
> > > > > rpc.gssd[965]: doing downcall: lifetime_rec=86338 acceptor=nfs@klimt.ib.1015granger.net
> > > > 
> > > > Chuck,
> > > > can you tell what does klimt.ib.1015granger.net resolve to (A names
> > > > CNAMEs, not really interested in IP address)?
> > > 
> > > [root@manet ~]# dig klimt.ib.1015granger.net
> > > 
> > > ; <<>> DiG 9.11.20-RedHat-9.11.20-1.fc32 <<>> klimt.ib.1015granger.net
> > > ;; global options: +cmd
> > > ;; Got answer:
> > > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55806
> > > ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
> > > 
> > > ;; OPT PSEUDOSECTION:
> > > ; EDNS: version: 0, flags:; udp: 4096
> > > ; COOKIE: 0a7a07d8b06eedeab886314f5f230a8c6f752fe4a24c2f97 (good)
> > > ;; QUESTION SECTION:
> > > ;klimt.ib.1015granger.net.	IN	A
> > > 
> > > ;; ANSWER SECTION:
> > > klimt.ib.1015granger.net. 10800	IN	A	192.168.2.55
> > > 
> > > ;; AUTHORITY SECTION:
> > > ib.1015granger.net.	10800	IN	NS	gateway.1015granger.net.
> > > 
> > > ;; ADDITIONAL SECTION:
> > > gateway.1015granger.net. 10800	IN	A	192.168.1.1
> > > 
> > > ;; Query time: 0 msec
> > > ;; SERVER: 192.168.1.1#53(192.168.1.1)
> > > ;; WHEN: Thu Jul 30 13:59:40 EDT 2020
> > > ;; MSG SIZE  rcvd: 135
> > > 
> > > [root@manet ~]#
> > 
> > so klimt is an A name, and it is a fqdn in the principal, so I am
> > puzzled why the krb5.conf option would make a difference, unless
> > 192.168.2.55 perhaps resolves to a different name and you have rdns =
> > true ?
> 
> rdns is false on my NFS client.
> 
> [cel@manet ~]$ dig -x 192.168.2.55
> 
> ; <<>> DiG 9.11.20-RedHat-9.11.20-1.fc32 <<>> -x 192.168.2.55
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22491
> ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
> 
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ; COOKIE: 868fe3a23c55fdee29cc812d5f231096a66a540a8064f2e8 (good)
> ;; QUESTION SECTION:
> ;55.2.168.192.in-addr.arpa.	IN	PTR
> 
> ;; ANSWER SECTION:
> 55.2.168.192.IN-ADDR.ARPA. 10800 IN	PTR	klimt.ib.1015granger.net.
> 
> ;; AUTHORITY SECTION:
> 2.168.192.IN-ADDR.ARPA.	10800	IN	NS	gateway.1015granger.net.
> 
> ;; ADDITIONAL SECTION:
> gateway.1015granger.net. 10800	IN	A	192.168.1.1
> 
> ;; Query time: 0 msec
> ;; SERVER: 192.168.1.1#53(192.168.1.1)
> ;; WHEN: Thu Jul 30 14:25:26 EDT 2020
> ;; MSG SIZE  rcvd: 183
> 
> [cel@manet ~]$
> 
> 
> > > > Also what ticket do you ultimately get in the ccache when this request
> > > > is made ?
> > > 
> > > I'm not exactly sure what you're asking, but:
> > > 
> > > [root@manet ~]# klist FILE:/tmp/krb5ccmachine_1015GRANGER.NET
> > > Ticket cache: FILE:/tmp/krb5ccmachine_1015GRANGER.NET
> > > Default principal: host/manet.1015granger.net@1015GRANGER.NET
> > > 
> > > Valid starting       Expires              Service principal
> > > 07/30/2020 13:45:38  07/31/2020 13:45:38  krbtgt/1015GRANGER.NET@1015GRANGER.NET
> > > 	renew until 08/06/2020 13:45:38
> > > [root@manet ~]#
> > 
> > I was asking what ticket you get to use to talk to klimt, this is the
> > machine ccache but it only has the krbtgt for the machine host key.
> 
> I'm mounting with sec=sys, so the machine host key is the only
> key in play. The client uses that key for NFSv4.0 lease management
> and callbacks.
> 
> It is the callback context that is the problem. The client is
> required to match the server's principal to the client's
> acceptor when authenticating a callback operation.
> 
> 
> > And now that I reread the thread better I see:
> > 
> > Callback principal (nfs@klimt.ib.1015granger.net) does not match
> > acceptor (nfs@klimt.ib)
> > 
> > do you use klimt.ib explicitly somewhere instad of the fqdn and rely on
> > name expansion ?
> 
> Yes, I mount with "mount -o vers=4.0,sec=sys klimt.ib:/export /mnt" .
> 
> When dns_canonicalize_hostname = true or avoid-dns = 0, gssd
> properly canonicalizes the acceptor so that it matches the
> server's callback principal.

Ok this seems to be the "issue", try restoring all the defaults but set
`qualify_shortname = ""` and see if that also make it work.

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc





      reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-29 17:19 Chuck Lever
2020-07-29 18:27 ` Chuck Lever
2020-07-30 14:43   ` Steve Dickson
2020-07-30 16:14   ` Simo Sorce
2020-07-30 17:08     ` Robbie Harwood
2020-07-30 17:59       ` Chuck Lever
2020-07-30 19:10         ` Simo Sorce
2020-07-30 19:39           ` Chuck Lever
2020-07-30 17:09     ` Chuck Lever
2020-07-30 17:57       ` Simo Sorce
2020-07-30 18:07         ` Chuck Lever
2020-07-30 18:20           ` Simo Sorce
2020-07-30 18:29             ` Chuck Lever
2020-07-30 18:55               ` Simo Sorce [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=33e219ec909a7ef35589d4ee9c8f6c45c4dacc6f.camel@redhat.com \
    --to=simo@redhat.com \
    --cc=bfields@fieldses.org \
    --cc=chuck.lever@oracle.com \
    --cc=jlayton@redhat.com \
    --cc=linux-nfs@vger.kernel.org \
    --cc=rharwood@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nfs/0 linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ https://lore.kernel.org/linux-nfs \
		linux-nfs@vger.kernel.org
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-nfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git