Linux-NFS Archive on
 help / color / Atom feed
From: Chuck Lever <>
To: Simo Sorce <>
Cc: Jeff Layton <>,
	Bruce Fields <>,
	Linux NFS Mailing List <>,
	Robbie Harwood <>
Subject: Re: Fedora 32 rpc.gssd misbehavior
Date: Thu, 30 Jul 2020 13:09:14 -0400
Message-ID: <> (raw)
In-Reply-To: <>

> On Jul 30, 2020, at 12:14 PM, Simo Sorce <> wrote:
> On Wed, 2020-07-29 at 14:27 -0400, Chuck Lever wrote:
>>> On Jul 29, 2020, at 1:19 PM, Chuck Lever <> wrote:
>>> Hi!
>>> I recently updated my test systems from EL7 to Fedora 32, and
>>> NFSv4.0 with Kerberos has stopped working.
>>> I mount with "klimt.ib" as before. The client workload stops
>>> dead when the server tries to perform its first CB_RECALL.
>>> I added some client instrumentation:
>>>  kernel: NFSv4: Callback principal ( does not match acceptor (nfs@klimt.ib).
>>>  kernel: NFS: NFSv4 callback contains invalid cred
>>> I boosted gssd verbosity, and it says:
>>>  rpc.gssd[986]: doing downcall: lifetime_rec=72226 acceptor=nfs@klimt.ib
>>> But it knows the full hostname for the server:
>>>  rpc.gssd[986]: Full hostname for 'klimt.ib' is ''
>>> The acceptor appears to come from the Kerberos library. Shouldn't
>>> it be canonicalized? If so, should the Kerberos library do it, or
>>> should gssd? Since this behavior appeared after an upgrade, I
>>> suspect a Kerberos library regression. But it could be config-
>>> related, since both systems were re-imaged from the ground up.
>>> Also noticing some other problems on the server (missing hostname
>>> strings in debug messages, sssd_kcm infinite loops, and gssd
>>> sending garbage to the client after the NULL request that
>>> establishes the callback context).
>>> But let's look at the client acceptor problem first.
>> I believe I found the problem.
>> 8bffe8c5ec1a ("gssd: add /etc/nfs.conf support") added a number of gssd config
>> options to /etc/nfs.conf, including "avoid-dns". The default setting of avoid-
>> dns is 1. When I set this option on my client system explicitly to 0, NFSv4.0
>> with Kerberos works again.
>> Is there a reason the default setting is 1?
> Now that you mention DNS, this may be an interaction between a new
> default in Fedora 32 and how your environment is setup re DNS.
> In F32 we changed the option dns_canonicalize_hostname from 'true' to
> 'fallback'.
> This is a transitional state to eventually move it to 'false' at some
> point in the future.
> What it changes in practice is that it will first try the name passed
> in *as is* and only as a fallback try a CNAME if the name passed is not
> resolved as an A name. If you have principals in the KDC for both
> names, but you do not have keys in the keytab for both, you can have
> transitional issues.
> Additionally we discovered a bug that causes non qualified names to
> fail resolution with the 'fallback' option.
> If your name in the principal is really not qualified it will try to
> qualify it anyway, so if your principal is literally nfs/foo@FOO
> libgssapi may try to use nfs/, where "my.domain" is
> what is defined in resolv.conf search path.
> We are trying to address this regression.
> So try to set dns_canonicalize_hostname to true to see if that may
> influence your issue. If so, please let me know, as we still need to
> address this where possible.

I set avoid-dns to 1 and dns_canonicalize_hostname to true. The
workload hang is not reproducible, and the acceptor is fully qualified.

rpc.gssd[965]: doing downcall: lifetime_rec=86338

Chuck Lever

  parent reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-29 17:19 Chuck Lever
2020-07-29 18:27 ` Chuck Lever
2020-07-30 14:43   ` Steve Dickson
2020-07-30 16:14   ` Simo Sorce
2020-07-30 17:08     ` Robbie Harwood
2020-07-30 17:59       ` Chuck Lever
2020-07-30 19:10         ` Simo Sorce
2020-07-30 19:39           ` Chuck Lever
2020-07-30 17:09     ` Chuck Lever [this message]
2020-07-30 17:57       ` Simo Sorce
2020-07-30 18:07         ` Chuck Lever
2020-07-30 18:20           ` Simo Sorce
2020-07-30 18:29             ` Chuck Lever
2020-07-30 18:55               ` Simo Sorce

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on

Archives are clonable:
	git clone --mirror linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ \
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone