[PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Trond Myklebust]

According to RFC5661, if any of the SEQUENCE status bits
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, SEQ4_STATUS_ADMIN_STATE_REVOKED,
or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need to use
TEST_STATEID to figure out which stateids have been revoked, so we
can acknowledge the loss of state using FREE_STATEID.

While we already do this for open and lock state, we have not been doing
so for all the delegations.

v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired too
v3: Now with added lock revoke fixes and close/delegreturn/locku fixes
v4: Close a bunch of corner cases
v5: Report revoked delegations as invalid in nfs_have_delegation()
    Fix an infinite loop in nfs_reap_expired_delegations.
    Fixes for other looping behaviour

Trond Myklebust (25):
  NFSv4.1: Don't deadlock the state manager on the SEQUENCE status flags
  NFSv4: Don't report revoked delegations as valid in
    nfs_have_delegation()
  NFSv4.1: Don't check delegations that are already marked as revoked
  NFSv4.1: Allow test_stateid to handle session errors without waiting
  NFSv4.1: Add a helper function to deal with expired stateids
  NFSv4.x: Allow callers of nfs_remove_bad_delegation() to specify a
    stateid
  NFSv4.1: Test delegation stateids when server declares "some state
    revoked"
  NFSv4.1: Deal with server reboots during delegation expiration
    recovery
  NFSv4.1: Don't recheck delegations that have already been checked
  NFSv4.1: Allow revoked stateids to skip the call to TEST_STATEID
  NFSv4.1: Ensure we always run TEST/FREE_STATEID on locks
  NFSv4.1: FREE_STATEID can be asynchronous
  NFSv4.1: Ensure we call FREE_STATEID if needed on
    close/delegreturn/locku
  NFSv4: Ensure we don't re-test revoked and freed stateids
  NFSv4: nfs_inode_find_delegation_state_and_recover() should check all
    stateids
  NFSv4: nfs4_do_handle_exception() handle revoke/expiry of a single
    stateid
  NFSv4: nfs4_handle_delegation_recall_error() handle expiration as
    revoke case
  NFSv4: nfs4_handle_setlk_error() handle expiration as revoke case
  NFSv4.1: nfs4_layoutget_handle_exception handle revoked state
  NFSv4: Pass the stateid to the exception handler in
    nfs4_read/write_done_cb
  NFSv4: Fix a race in nfs_inode_reclaim_delegation()
  NFSv4: Fix a race when updating an open_stateid
  NFS: Always call nfs_inode_find_state_and_recover() when revoking a
    delegation
  NFSv4: Don't test open_stateid unless it is set
  NFSv4: Mark the lock and open stateids as invalid after freeing them

 fs/nfs/delegation.c                    | 197 +++++++++++++++--
 fs/nfs/delegation.h                    |   8 +-
 fs/nfs/flexfilelayout/flexfilelayout.c |   2 +-
 fs/nfs/nfs4_fs.h                       |   5 +-
 fs/nfs/nfs4proc.c                      | 371 +++++++++++++++++++++++----------
 fs/nfs/nfs4session.h                   |   1 +
 fs/nfs/nfs4state.c                     |  73 +++++--
 include/linux/nfs4.h                   |   1 +
 8 files changed, 508 insertions(+), 150 deletions(-)

-- 
2.7.4

[PATCH v5 01/25] NFSv4.1: Don't deadlock the state manager on the SEQUENCE status flags

[Trond Myklebust]

As described in RFC5661, section 18.46, some of the status flags exist
in order to tell the client when it needs to acknowledge the existence of
revoked state on the server and/or to recover state.
Those flags will then remain set until the recovery procedure is done.

In order to avoid looping, the client therefore needs to ignore
those particular flags while recovering.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4_fs.h     |  2 +-
 fs/nfs/nfs4proc.c    |  5 ++++-
 fs/nfs/nfs4session.h |  1 +
 fs/nfs/nfs4state.c   | 12 +++++++++++-
 4 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index f230aa62ca59..4390d73a92e5 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -439,7 +439,7 @@ extern void nfs4_schedule_path_down_recovery(struct nfs_client *clp);
 extern int nfs4_schedule_stateid_recovery(const struct nfs_server *, struct nfs4_state *);
 extern int nfs4_schedule_migration_recovery(const struct nfs_server *);
 extern void nfs4_schedule_lease_moved_recovery(struct nfs_client *);
-extern void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags);
+extern void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags, bool);
 extern void nfs41_handle_server_scope(struct nfs_client *,
 				      struct nfs41_server_scope **);
 extern void nfs4_put_lock_state(struct nfs4_lock_state *lsp);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 251e48e7ba16..6b700c59eede 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -616,6 +616,7 @@ int nfs40_setup_sequence(struct nfs4_slot_table *tbl,
 	}
 	spin_unlock(&tbl->slot_tbl_lock);
 
+	slot->privileged = args->sa_privileged ? 1 : 0;
 	args->sa_slot = slot;
 	res->sr_slot = slot;
 
@@ -728,7 +729,8 @@ static int nfs41_sequence_process(struct rpc_task *task,
 		clp = session->clp;
 		do_renew_lease(clp, res->sr_timestamp);
 		/* Check sequence flags */
-		nfs41_handle_sequence_flag_errors(clp, res->sr_status_flags);
+		nfs41_handle_sequence_flag_errors(clp, res->sr_status_flags,
+				!!slot->privileged);
 		nfs41_update_target_slotid(slot->table, slot, res);
 		break;
 	case 1:
@@ -875,6 +877,7 @@ int nfs41_setup_sequence(struct nfs4_session *session,
 	}
 	spin_unlock(&tbl->slot_tbl_lock);
 
+	slot->privileged = args->sa_privileged ? 1 : 0;
 	args->sa_slot = slot;
 
 	dprintk("<-- %s slotid=%u seqid=%u\n", __func__,
diff --git a/fs/nfs/nfs4session.h b/fs/nfs/nfs4session.h
index 3bb6af70973c..dae385500005 100644
--- a/fs/nfs/nfs4session.h
+++ b/fs/nfs/nfs4session.h
@@ -23,6 +23,7 @@ struct nfs4_slot {
 	u32			slot_nr;
 	u32		 	seq_nr;
 	unsigned int		interrupted : 1,
+				privileged : 1,
 				seq_done : 1;
 };
 
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index cada00aa5096..9801b5bb5fac 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -2227,13 +2227,22 @@ static void nfs41_handle_cb_path_down(struct nfs_client *clp)
 		nfs4_schedule_state_manager(clp);
 }
 
-void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags)
+void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags,
+		bool recovery)
 {
 	if (!flags)
 		return;
 
 	dprintk("%s: \"%s\" (client ID %llx) flags=0x%08x\n",
 		__func__, clp->cl_hostname, clp->cl_clientid, flags);
+	/*
+	 * If we're called from the state manager thread, then assume we're
+	 * already handling the RECLAIM_NEEDED and/or STATE_REVOKED.
+	 * Those flags are expected to remain set until we're done
+	 * recovering (see RFC5661, section 18.46.3).
+	 */
+	if (recovery)
+		goto out_recovery;
 
 	if (flags & SEQ4_STATUS_RESTART_RECLAIM_NEEDED)
 		nfs41_handle_server_reboot(clp);
@@ -2246,6 +2255,7 @@ void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags)
 		nfs4_schedule_lease_moved_recovery(clp);
 	if (flags & SEQ4_STATUS_RECALLABLE_STATE_REVOKED)
 		nfs41_handle_recallable_state_revoked(clp);
+out_recovery:
 	if (flags & SEQ4_STATUS_BACKCHANNEL_FAULT)
 		nfs41_handle_backchannel_fault(clp);
 	else if (flags & (SEQ4_STATUS_CB_PATH_DOWN |
-- 
2.7.4

[PATCH v5 02/25] NFSv4: Don't report revoked delegations as valid in nfs_have_delegation()

[Trond Myklebust]

If the delegation is revoked, then it can't be used for caching.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 322c2585bc34..86d2c748140b 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -51,6 +51,7 @@ nfs4_do_check_delegation(struct inode *inode, fmode_t flags, bool mark)
 	rcu_read_lock();
 	delegation = rcu_dereference(NFS_I(inode)->delegation);
 	if (delegation != NULL && (delegation->type & flags) == flags &&
+	    !test_bit(NFS_DELEGATION_REVOKED, &delegation->flags) &&
 	    !test_bit(NFS_DELEGATION_RETURNING, &delegation->flags)) {
 		if (mark)
 			nfs_mark_delegation_referenced(delegation);
-- 
2.7.4

[PATCH v5 03/25] NFSv4.1: Don't check delegations that are already marked as revoked

[Trond Myklebust]

If the delegation has been marked as revoked, we don't have to test
it, because we should already have called FREE_STATEID on it.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 6b700c59eede..a026156434ae 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2423,6 +2423,11 @@ static void nfs41_check_delegation_stateid(struct nfs4_state *state)
 		rcu_read_unlock();
 		return;
 	}
+	if (test_bit(NFS_DELEGATION_REVOKED, &delegation->flags)) {
+		rcu_read_unlock();
+		nfs_finish_clear_delegation_stateid(state);
+		return;
+	}
 
 	nfs4_stateid_copy(&stateid, &delegation->stateid);
 	cred = get_rpccred(delegation->cred);
-- 
2.7.4

[PATCH v5 04/25] NFSv4.1: Allow test_stateid to handle session errors without waiting

[Trond Myklebust]

If the server crashes while we're testing stateids for validity, then
we want to initiate session recovery. Usually, we will be calling from
a state manager thread, though, so we don't really want to wait.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index a026156434ae..2b569d5fb3e2 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -8598,6 +8598,23 @@ static int _nfs41_test_stateid(struct nfs_server *server,
 	return -res.status;
 }
 
+static void nfs4_handle_delay_or_session_error(struct nfs_server *server,
+		int err, struct nfs4_exception *exception)
+{
+	exception->retry = 0;
+	switch(err) {
+	case -NFS4ERR_DELAY:
+		nfs4_handle_exception(server, err, exception);
+		break;
+	case -NFS4ERR_BADSESSION:
+	case -NFS4ERR_BADSLOT:
+	case -NFS4ERR_BAD_HIGH_SLOT:
+	case -NFS4ERR_CONN_NOT_BOUND_TO_SESSION:
+	case -NFS4ERR_DEADSESSION:
+		nfs4_do_handle_exception(server, err, exception);
+	}
+}
+
 /**
  * nfs41_test_stateid - perform a TEST_STATEID operation
  *
@@ -8617,9 +8634,7 @@ static int nfs41_test_stateid(struct nfs_server *server,
 	int err;
 	do {
 		err = _nfs41_test_stateid(server, stateid, cred);
-		if (err != -NFS4ERR_DELAY)
-			break;
-		nfs4_handle_exception(server, err, &exception);
+		nfs4_handle_delay_or_session_error(server, err, &exception);
 	} while (exception.retry);
 	return err;
 }
-- 
2.7.4

[PATCH v5 05/25] NFSv4.1: Add a helper function to deal with expired stateids

[Trond Myklebust]

In NFSv4.1 and newer, if the server decides to revoke some or all of
the protocol state, the client is required to iterate through all the
stateids that it holds and call TEST_STATEID to determine which stateids
still correspond to valid state, and then call FREE_STATEID on the
others.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 45 ++++++++++++++++++++++++---------------------
 1 file changed, 24 insertions(+), 21 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 2b569d5fb3e2..0ade81441ac2 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2408,6 +2408,26 @@ static int nfs40_open_expired(struct nfs4_state_owner *sp, struct nfs4_state *st
 }
 
 #if defined(CONFIG_NFS_V4_1)
+static int nfs41_test_and_free_expired_stateid(struct nfs_server *server,
+		nfs4_stateid *stateid,
+		struct rpc_cred *cred)
+{
+	int status;
+
+	status = nfs41_test_stateid(server, stateid, cred);
+
+	switch (status) {
+	case -NFS4ERR_EXPIRED:
+	case -NFS4ERR_ADMIN_REVOKED:
+	case -NFS4ERR_DELEG_REVOKED:
+		/* Ack the revoked state to the server */
+		nfs41_free_stateid(server, stateid, cred);
+	case -NFS4ERR_BAD_STATEID:
+		return status;
+	}
+	return NFS_OK;
+}
+
 static void nfs41_check_delegation_stateid(struct nfs4_state *state)
 {
 	struct nfs_server *server = NFS_SERVER(state->inode);
@@ -2432,16 +2452,10 @@ static void nfs41_check_delegation_stateid(struct nfs4_state *state)
 	nfs4_stateid_copy(&stateid, &delegation->stateid);
 	cred = get_rpccred(delegation->cred);
 	rcu_read_unlock();
-	status = nfs41_test_stateid(server, &stateid, cred);
+	status = nfs41_test_and_free_expired_stateid(server, &stateid, cred);
 	trace_nfs4_test_delegation_stateid(state, NULL, status);
-
-	if (status != NFS_OK) {
-		/* Free the stateid unless the server explicitly
-		 * informs us the stateid is unrecognized. */
-		if (status != -NFS4ERR_BAD_STATEID)
-			nfs41_free_stateid(server, &stateid, cred);
+	if (status != NFS_OK)
 		nfs_finish_clear_delegation_stateid(state);
-	}
 
 	put_rpccred(cred);
 }
@@ -2467,14 +2481,9 @@ static int nfs41_check_open_stateid(struct nfs4_state *state)
 	    (test_bit(NFS_O_RDWR_STATE, &state->flags) == 0))
 		return -NFS4ERR_BAD_STATEID;
 
-	status = nfs41_test_stateid(server, stateid, cred);
+	status = nfs41_test_and_free_expired_stateid(server, stateid, cred);
 	trace_nfs4_test_open_stateid(state, NULL, status);
 	if (status != NFS_OK) {
-		/* Free the stateid unless the server explicitly
-		 * informs us the stateid is unrecognized. */
-		if (status != -NFS4ERR_BAD_STATEID)
-			nfs41_free_stateid(server, stateid, cred);
-
 		clear_bit(NFS_O_RDONLY_STATE, &state->flags);
 		clear_bit(NFS_O_WRONLY_STATE, &state->flags);
 		clear_bit(NFS_O_RDWR_STATE, &state->flags);
@@ -6109,17 +6118,11 @@ static int nfs41_check_expired_locks(struct nfs4_state *state)
 		if (test_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags)) {
 			struct rpc_cred *cred = lsp->ls_state->owner->so_cred;
 
-			status = nfs41_test_stateid(server,
+			status = nfs41_test_and_free_expired_stateid(server,
 					&lsp->ls_stateid,
 					cred);
 			trace_nfs4_test_lock_stateid(state, lsp, status);
 			if (status != NFS_OK) {
-				/* Free the stateid unless the server
-				 * informs us the stateid is unrecognized. */
-				if (status != -NFS4ERR_BAD_STATEID)
-					nfs41_free_stateid(server,
-							&lsp->ls_stateid,
-							cred);
 				clear_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags);
 				ret = status;
 			}
-- 
2.7.4

[PATCH v5 06/25] NFSv4.x: Allow callers of nfs_remove_bad_delegation() to specify a stateid

[Trond Myklebust]

Allow the callers of nfs_remove_bad_delegation() to specify the stateid
that needs to be marked as bad.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c                    | 30 +++++++++++++++++++++++-------
 fs/nfs/delegation.h                    |  2 +-
 fs/nfs/flexfilelayout/flexfilelayout.c |  2 +-
 fs/nfs/nfs4proc.c                      | 14 ++++++++------
 4 files changed, 33 insertions(+), 15 deletions(-)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 86d2c748140b..32fff8985a29 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -643,23 +643,39 @@ static void nfs_client_mark_return_unused_delegation_types(struct nfs_client *cl
 	rcu_read_unlock();
 }
 
-static void nfs_revoke_delegation(struct inode *inode)
+static void nfs_mark_delegation_revoked(struct nfs_server *server,
+		struct nfs_delegation *delegation)
+{
+	set_bit(NFS_DELEGATION_REVOKED, &delegation->flags);
+	nfs_mark_return_delegation(server, delegation);
+}
+
+static bool nfs_revoke_delegation(struct inode *inode,
+		const nfs4_stateid *stateid)
 {
 	struct nfs_delegation *delegation;
+	bool ret = false;
+
 	rcu_read_lock();
 	delegation = rcu_dereference(NFS_I(inode)->delegation);
-	if (delegation != NULL) {
-		set_bit(NFS_DELEGATION_REVOKED, &delegation->flags);
-		nfs_mark_return_delegation(NFS_SERVER(inode), delegation);
-	}
+	if (delegation == NULL)
+		goto out;
+	if (stateid && !nfs4_stateid_match(stateid, &delegation->stateid))
+		goto out;
+	nfs_mark_delegation_revoked(NFS_SERVER(inode), delegation);
+	ret = true;
+out:
 	rcu_read_unlock();
+	return ret;
 }
 
-void nfs_remove_bad_delegation(struct inode *inode)
+void nfs_remove_bad_delegation(struct inode *inode,
+		const nfs4_stateid *stateid)
 {
 	struct nfs_delegation *delegation;
 
-	nfs_revoke_delegation(inode);
+	if (!nfs_revoke_delegation(inode, stateid))
+		return;
 	delegation = nfs_inode_detach_delegation(inode);
 	if (delegation) {
 		nfs_inode_find_state_and_recover(inode, &delegation->stateid);
diff --git a/fs/nfs/delegation.h b/fs/nfs/delegation.h
index 64724d252a79..d40827af5913 100644
--- a/fs/nfs/delegation.h
+++ b/fs/nfs/delegation.h
@@ -47,7 +47,7 @@ void nfs_expire_unused_delegation_types(struct nfs_client *clp, fmode_t flags);
 void nfs_expire_unreferenced_delegations(struct nfs_client *clp);
 int nfs_client_return_marked_delegations(struct nfs_client *clp);
 int nfs_delegations_present(struct nfs_client *clp);
-void nfs_remove_bad_delegation(struct inode *inode);
+void nfs_remove_bad_delegation(struct inode *inode, const nfs4_stateid *stateid);
 
 void nfs_delegation_mark_reclaim(struct nfs_client *clp);
 void nfs_delegation_reap_unclaimed(struct nfs_client *clp);
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 51b51369704c..98ace127bf86 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -1080,7 +1080,7 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
 	case -NFS4ERR_BAD_STATEID:
 		if (state == NULL)
 			break;
-		nfs_remove_bad_delegation(state->inode);
+		nfs_remove_bad_delegation(state->inode, NULL);
 	case -NFS4ERR_OPENMODE:
 		if (state == NULL)
 			break;
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 0ade81441ac2..9c203c221876 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2385,9 +2385,10 @@ static int nfs4_open_expired(struct nfs4_state_owner *sp, struct nfs4_state *sta
 	return ret;
 }
 
-static void nfs_finish_clear_delegation_stateid(struct nfs4_state *state)
+static void nfs_finish_clear_delegation_stateid(struct nfs4_state *state,
+		const nfs4_stateid *stateid)
 {
-	nfs_remove_bad_delegation(state->inode);
+	nfs_remove_bad_delegation(state->inode, stateid);
 	write_seqlock(&state->seqlock);
 	nfs4_stateid_copy(&state->stateid, &state->open_stateid);
 	write_sequnlock(&state->seqlock);
@@ -2397,7 +2398,7 @@ static void nfs_finish_clear_delegation_stateid(struct nfs4_state *state)
 static void nfs40_clear_delegation_stateid(struct nfs4_state *state)
 {
 	if (rcu_access_pointer(NFS_I(state->inode)->delegation) != NULL)
-		nfs_finish_clear_delegation_stateid(state);
+		nfs_finish_clear_delegation_stateid(state, NULL);
 }
 
 static int nfs40_open_expired(struct nfs4_state_owner *sp, struct nfs4_state *state)
@@ -2443,19 +2444,20 @@ static void nfs41_check_delegation_stateid(struct nfs4_state *state)
 		rcu_read_unlock();
 		return;
 	}
+
+	nfs4_stateid_copy(&stateid, &delegation->stateid);
 	if (test_bit(NFS_DELEGATION_REVOKED, &delegation->flags)) {
 		rcu_read_unlock();
-		nfs_finish_clear_delegation_stateid(state);
+		nfs_finish_clear_delegation_stateid(state, &stateid);
 		return;
 	}
 
-	nfs4_stateid_copy(&stateid, &delegation->stateid);
 	cred = get_rpccred(delegation->cred);
 	rcu_read_unlock();
 	status = nfs41_test_and_free_expired_stateid(server, &stateid, cred);
 	trace_nfs4_test_delegation_stateid(state, NULL, status);
 	if (status != NFS_OK)
-		nfs_finish_clear_delegation_stateid(state);
+		nfs_finish_clear_delegation_stateid(state, &stateid);
 
 	put_rpccred(cred);
 }
-- 
2.7.4

[PATCH v5 07/25] NFSv4.1: Test delegation stateids when server declares "some state revoked"

[Trond Myklebust]

According to RFC5661, if any of the SEQUENCE status bits
SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, SEQ4_STATUS_ADMIN_STATE_REVOKED,
or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need to use
TEST_STATEID to figure out which stateids have been revoked, so we
can acknowledge the loss of state using FREE_STATEID.

While we already do this for open and lock state, we have not been doing
so for all the delegations.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 fs/nfs/delegation.h |  4 +++
 fs/nfs/nfs4_fs.h    |  3 ++
 fs/nfs/nfs4proc.c   | 10 ++++++
 fs/nfs/nfs4state.c  | 17 +++++-----
 5 files changed, 122 insertions(+), 9 deletions(-)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 32fff8985a29..f0765968a819 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -803,8 +803,15 @@ static void nfs_delegation_mark_reclaim_server(struct nfs_server *server)
 {
 	struct nfs_delegation *delegation;
 
-	list_for_each_entry_rcu(delegation, &server->delegations, super_list)
+	list_for_each_entry_rcu(delegation, &server->delegations, super_list) {
+		/*
+		 * If the delegation may have been admin revoked, then we
+		 * cannot reclaim it.
+		 */
+		if (test_bit(NFS_DELEGATION_TEST_EXPIRED, &delegation->flags))
+			continue;
 		set_bit(NFS_DELEGATION_NEED_RECLAIM, &delegation->flags);
+	}
 }
 
 /**
@@ -868,6 +875,94 @@ restart:
 	rcu_read_unlock();
 }
 
+static void nfs_mark_test_expired_delegation(struct nfs_server *server,
+	    struct nfs_delegation *delegation)
+{
+	clear_bit(NFS_DELEGATION_NEED_RECLAIM, &delegation->flags);
+	set_bit(NFS_DELEGATION_TEST_EXPIRED, &delegation->flags);
+	set_bit(NFS4CLNT_DELEGATION_EXPIRED, &server->nfs_client->cl_state);
+}
+
+static void nfs_delegation_mark_test_expired_server(struct nfs_server *server)
+{
+	struct nfs_delegation *delegation;
+
+	list_for_each_entry_rcu(delegation, &server->delegations, super_list)
+		nfs_mark_test_expired_delegation(server, delegation);
+}
+
+/**
+ * nfs_mark_test_expired_all_delegations - mark all delegations for testing
+ * @clp: nfs_client to process
+ *
+ * Iterates through all the delegations associated with this server and
+ * marks them as needing to be checked for validity.
+ */
+void nfs_mark_test_expired_all_delegations(struct nfs_client *clp)
+{
+	struct nfs_server *server;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(server, &clp->cl_superblocks, client_link)
+		nfs_delegation_mark_test_expired_server(server);
+	rcu_read_unlock();
+}
+
+/**
+ * nfs_reap_expired_delegations - reap expired delegations
+ * @clp: nfs_client to process
+ *
+ * Iterates through all the delegations associated with this server and
+ * checks if they have may have been revoked. This function is usually
+ * expected to be called in cases where the server may have lost its
+ * lease.
+ */
+void nfs_reap_expired_delegations(struct nfs_client *clp)
+{
+	const struct nfs4_minor_version_ops *ops = clp->cl_mvops;
+	struct nfs_delegation *delegation;
+	struct nfs_server *server;
+	struct inode *inode;
+	struct rpc_cred *cred;
+	nfs4_stateid stateid;
+
+restart:
+	rcu_read_lock();
+	list_for_each_entry_rcu(server, &clp->cl_superblocks, client_link) {
+		list_for_each_entry_rcu(delegation, &server->delegations,
+								super_list) {
+			if (test_bit(NFS_DELEGATION_RETURNING,
+						&delegation->flags))
+				continue;
+			if (test_bit(NFS_DELEGATION_TEST_EXPIRED,
+						&delegation->flags) == 0)
+				continue;
+			if (!nfs_sb_active(server->super))
+				continue;
+			inode = nfs_delegation_grab_inode(delegation);
+			if (inode == NULL) {
+				rcu_read_unlock();
+				nfs_sb_deactive(server->super);
+				goto restart;
+			}
+			cred = get_rpccred_rcu(delegation->cred);
+			nfs4_stateid_copy(&stateid, &delegation->stateid);
+			clear_bit(NFS_DELEGATION_TEST_EXPIRED, &delegation->flags);
+			rcu_read_unlock();
+			if (cred != NULL &&
+			    ops->test_and_free_expired(server, &stateid, cred) < 0) {
+				nfs_revoke_delegation(inode, &stateid);
+				nfs_inode_find_state_and_recover(inode, &stateid);
+			}
+			put_rpccred(cred);
+			iput(inode);
+			nfs_sb_deactive(server->super);
+			goto restart;
+		}
+	}
+	rcu_read_unlock();
+}
+
 /**
  * nfs_delegations_present - check for existence of delegations
  * @clp: client state handle
diff --git a/fs/nfs/delegation.h b/fs/nfs/delegation.h
index d40827af5913..1442e3b1521d 100644
--- a/fs/nfs/delegation.h
+++ b/fs/nfs/delegation.h
@@ -32,6 +32,7 @@ enum {
 	NFS_DELEGATION_REFERENCED,
 	NFS_DELEGATION_RETURNING,
 	NFS_DELEGATION_REVOKED,
+	NFS_DELEGATION_TEST_EXPIRED,
 };
 
 int nfs_inode_set_delegation(struct inode *inode, struct rpc_cred *cred, struct nfs_openres *res);
@@ -52,6 +53,9 @@ void nfs_remove_bad_delegation(struct inode *inode, const nfs4_stateid *stateid)
 void nfs_delegation_mark_reclaim(struct nfs_client *clp);
 void nfs_delegation_reap_unclaimed(struct nfs_client *clp);
 
+void nfs_mark_test_expired_all_delegations(struct nfs_client *clp);
+void nfs_reap_expired_delegations(struct nfs_client *clp);
+
 /* NFSv4 delegation-related procedures */
 int nfs4_proc_delegreturn(struct inode *inode, struct rpc_cred *cred, const nfs4_stateid *stateid, int issync);
 int nfs4_open_delegation_recall(struct nfs_open_context *ctx, struct nfs4_state *state, const nfs4_stateid *stateid, fmode_t type);
diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 4390d73a92e5..b9083a6cefd6 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -39,6 +39,7 @@ enum nfs4_client_state {
 	NFS4CLNT_BIND_CONN_TO_SESSION,
 	NFS4CLNT_MOVED,
 	NFS4CLNT_LEASE_MOVED,
+	NFS4CLNT_DELEGATION_EXPIRED,
 };
 
 #define NFS4_RENEW_TIMEOUT		0x01
@@ -57,6 +58,8 @@ struct nfs4_minor_version_ops {
 			struct nfs_fsinfo *);
 	void	(*free_lock_state)(struct nfs_server *,
 			struct nfs4_lock_state *);
+	int	(*test_and_free_expired)(struct nfs_server *,
+			nfs4_stateid *, struct rpc_cred *);
 	struct nfs_seqid *
 		(*alloc_seqid)(struct nfs_seqid_counter *, gfp_t);
 	const struct rpc_call_ops *call_sync_ops;
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 9c203c221876..3552f3a4ceb0 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2408,6 +2408,13 @@ static int nfs40_open_expired(struct nfs4_state_owner *sp, struct nfs4_state *st
 	return nfs4_open_expired(sp, state);
 }
 
+static int nfs40_test_and_free_expired_stateid(struct nfs_server *server,
+		nfs4_stateid *stateid,
+		struct rpc_cred *cred)
+{
+	return -NFS4ERR_BAD_STATEID;
+}
+
 #if defined(CONFIG_NFS_V4_1)
 static int nfs41_test_and_free_expired_stateid(struct nfs_server *server,
 		nfs4_stateid *stateid,
@@ -8862,6 +8869,7 @@ static const struct nfs4_minor_version_ops nfs_v4_0_minor_ops = {
 	.match_stateid = nfs4_match_stateid,
 	.find_root_sec = nfs4_find_root_sec,
 	.free_lock_state = nfs4_release_lockowner,
+	.test_and_free_expired = nfs40_test_and_free_expired_stateid,
 	.alloc_seqid = nfs_alloc_seqid,
 	.call_sync_ops = &nfs40_call_sync_ops,
 	.reboot_recovery_ops = &nfs40_reboot_recovery_ops,
@@ -8889,6 +8897,7 @@ static const struct nfs4_minor_version_ops nfs_v4_1_minor_ops = {
 	.match_stateid = nfs41_match_stateid,
 	.find_root_sec = nfs41_find_root_sec,
 	.free_lock_state = nfs41_free_lock_state,
+	.test_and_free_expired = nfs41_test_and_free_expired_stateid,
 	.alloc_seqid = nfs_alloc_no_seqid,
 	.call_sync_ops = &nfs41_call_sync_ops,
 	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
@@ -8918,6 +8927,7 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = {
 	.find_root_sec = nfs41_find_root_sec,
 	.free_lock_state = nfs41_free_lock_state,
 	.call_sync_ops = &nfs41_call_sync_ops,
+	.test_and_free_expired = nfs41_test_and_free_expired_stateid,
 	.alloc_seqid = nfs_alloc_no_seqid,
 	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
 	.nograce_recovery_ops = &nfs41_nograce_recovery_ops,
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 9801b5bb5fac..63da0411e2af 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1656,15 +1656,9 @@ static void nfs4_state_end_reclaim_reboot(struct nfs_client *clp)
 	put_rpccred(cred);
 }
 
-static void nfs_delegation_clear_all(struct nfs_client *clp)
-{
-	nfs_delegation_mark_reclaim(clp);
-	nfs_delegation_reap_unclaimed(clp);
-}
-
 static void nfs4_state_start_reclaim_nograce(struct nfs_client *clp)
 {
-	nfs_delegation_clear_all(clp);
+	nfs_mark_test_expired_all_delegations(clp);
 	nfs4_state_mark_reclaim_helper(clp, nfs4_state_mark_reclaim_nograce);
 }
 
@@ -2195,7 +2189,7 @@ static void nfs41_handle_all_state_revoked(struct nfs_client *clp)
 
 static void nfs41_handle_some_state_revoked(struct nfs_client *clp)
 {
-	nfs4_state_mark_reclaim_helper(clp, nfs4_state_mark_reclaim_nograce);
+	nfs4_state_start_reclaim_nograce(clp);
 	nfs4_schedule_state_manager(clp);
 
 	dprintk("%s: state revoked on server %s\n", __func__, clp->cl_hostname);
@@ -2420,6 +2414,13 @@ static void nfs4_state_manager(struct nfs_client *clp)
 			nfs4_state_end_reclaim_reboot(clp);
 		}
 
+		/* Detect expired delegations... */
+		if (test_and_clear_bit(NFS4CLNT_DELEGATION_EXPIRED, &clp->cl_state)) {
+			section = "detect expired delegations";
+			nfs_reap_expired_delegations(clp);
+			continue;
+		}
+
 		/* Now recover expired state... */
 		if (test_and_clear_bit(NFS4CLNT_RECLAIM_NOGRACE, &clp->cl_state)) {
 			section = "reclaim nograce";
-- 
2.7.4

[PATCH v5 08/25] NFSv4.1: Deal with server reboots during delegation expiration recovery

[Trond Myklebust]

Ensure that if the server reboots while we're testing and recovering
from revoked delegations, we exit to allow the state manager to
handle matters.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index f0765968a819..f431fde13e90 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -875,6 +875,13 @@ restart:
 	rcu_read_unlock();
 }
 
+static inline bool nfs4_server_rebooted(const struct nfs_client *clp)
+{
+	return (clp->cl_state & (BIT(NFS4CLNT_CHECK_LEASE) |
+				BIT(NFS4CLNT_LEASE_EXPIRED) |
+				BIT(NFS4CLNT_SESSION_RESET))) != 0;
+}
+
 static void nfs_mark_test_expired_delegation(struct nfs_server *server,
 	    struct nfs_delegation *delegation)
 {
@@ -883,6 +890,19 @@ static void nfs_mark_test_expired_delegation(struct nfs_server *server,
 	set_bit(NFS4CLNT_DELEGATION_EXPIRED, &server->nfs_client->cl_state);
 }
 
+static void nfs_inode_mark_test_expired_delegation(struct nfs_server *server,
+		struct inode *inode)
+{
+	struct nfs_delegation *delegation;
+
+	rcu_read_lock();
+	delegation = rcu_dereference(NFS_I(inode)->delegation);
+	if (delegation)
+		nfs_mark_test_expired_delegation(server, delegation);
+	rcu_read_unlock();
+
+}
+
 static void nfs_delegation_mark_test_expired_server(struct nfs_server *server)
 {
 	struct nfs_delegation *delegation;
@@ -955,6 +975,12 @@ restart:
 				nfs_inode_find_state_and_recover(inode, &stateid);
 			}
 			put_rpccred(cred);
+			if (nfs4_server_rebooted(clp)) {
+				nfs_inode_mark_test_expired_delegation(server,inode);
+				iput(inode);
+				nfs_sb_deactive(server->super);
+				return;
+			}
 			iput(inode);
 			nfs_sb_deactive(server->super);
 			goto restart;
-- 
2.7.4

[PATCH v5 09/25] NFSv4.1: Don't recheck delegations that have already been checked

[Trond Myklebust]

Ensure we don't spam the server with test_stateid() calls for
delegations that have already been checked.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 3552f3a4ceb0..97a614370604 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2459,6 +2459,11 @@ static void nfs41_check_delegation_stateid(struct nfs4_state *state)
 		return;
 	}
 
+	if (!test_and_clear_bit(NFS_DELEGATION_TEST_EXPIRED, &delegation->flags)) {
+		rcu_read_unlock();
+		return;
+	}
+
 	cred = get_rpccred(delegation->cred);
 	rcu_read_unlock();
 	status = nfs41_test_and_free_expired_stateid(server, &stateid, cred);
-- 
2.7.4

[PATCH v5 10/25] NFSv4.1: Allow revoked stateids to skip the call to TEST_STATEID

[Trond Myklebust]

In some cases (e.g. when the SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED sequence
flag is set) we may already know that the stateid was revoked and that the
only valid operation we can call is FREE_STATEID. In those cases, allow
the stateid to carry the information in the type field, so that we skip
the redundant call to TEST_STATEID.
---
 fs/nfs/nfs4proc.c    | 32 +++++++++++++++++++++++---------
 include/linux/nfs4.h |  1 +
 2 files changed, 24 insertions(+), 9 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 97a614370604..3c1b8cb7dd95 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2422,18 +2422,29 @@ static int nfs41_test_and_free_expired_stateid(struct nfs_server *server,
 {
 	int status;
 
-	status = nfs41_test_stateid(server, stateid, cred);
+	switch (stateid->type) {
+	default:
+		break;
+	case NFS4_INVALID_STATEID_TYPE:
+	case NFS4_SPECIAL_STATEID_TYPE:
+		return -NFS4ERR_BAD_STATEID;
+	case NFS4_REVOKED_STATEID_TYPE:
+		goto out_free;
+	}
 
+	status = nfs41_test_stateid(server, stateid, cred);
 	switch (status) {
 	case -NFS4ERR_EXPIRED:
 	case -NFS4ERR_ADMIN_REVOKED:
 	case -NFS4ERR_DELEG_REVOKED:
-		/* Ack the revoked state to the server */
-		nfs41_free_stateid(server, stateid, cred);
-	case -NFS4ERR_BAD_STATEID:
+		break;
+	default:
 		return status;
 	}
-	return NFS_OK;
+out_free:
+	/* Ack the revoked state to the server */
+	nfs41_free_stateid(server, stateid, cred);
+	return -NFS4ERR_EXPIRED;
 }
 
 static void nfs41_check_delegation_stateid(struct nfs4_state *state)
@@ -2468,7 +2479,7 @@ static void nfs41_check_delegation_stateid(struct nfs4_state *state)
 	rcu_read_unlock();
 	status = nfs41_test_and_free_expired_stateid(server, &stateid, cred);
 	trace_nfs4_test_delegation_stateid(state, NULL, status);
-	if (status != NFS_OK)
+	if (status == -NFS4ERR_EXPIRED || status == -NFS4ERR_BAD_STATEID)
 		nfs_finish_clear_delegation_stateid(state, &stateid);
 
 	put_rpccred(cred);
@@ -2497,7 +2508,7 @@ static int nfs41_check_open_stateid(struct nfs4_state *state)
 
 	status = nfs41_test_and_free_expired_stateid(server, stateid, cred);
 	trace_nfs4_test_open_stateid(state, NULL, status);
-	if (status != NFS_OK) {
+	if (status == -NFS4ERR_EXPIRED || status == -NFS4ERR_BAD_STATEID) {
 		clear_bit(NFS_O_RDONLY_STATE, &state->flags);
 		clear_bit(NFS_O_WRONLY_STATE, &state->flags);
 		clear_bit(NFS_O_RDWR_STATE, &state->flags);
@@ -6124,7 +6135,7 @@ out:
  */
 static int nfs41_check_expired_locks(struct nfs4_state *state)
 {
-	int status, ret = -NFS4ERR_BAD_STATEID;
+	int status, ret = NFS_OK;
 	struct nfs4_lock_state *lsp;
 	struct nfs_server *server = NFS_SERVER(state->inode);
 
@@ -6136,9 +6147,12 @@ static int nfs41_check_expired_locks(struct nfs4_state *state)
 					&lsp->ls_stateid,
 					cred);
 			trace_nfs4_test_lock_stateid(state, lsp, status);
-			if (status != NFS_OK) {
+			if (status == -NFS4ERR_EXPIRED ||
+			    status == -NFS4ERR_BAD_STATEID)
 				clear_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags);
+			else if (status != NFS_OK) {
 				ret = status;
+				break;
 			}
 		}
 	};
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index c6564ada9beb..9094faf0699d 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -67,6 +67,7 @@ struct nfs4_stateid_struct {
 		NFS4_DELEGATION_STATEID_TYPE,
 		NFS4_LAYOUT_STATEID_TYPE,
 		NFS4_PNFS_DS_STATEID_TYPE,
+		NFS4_REVOKED_STATEID_TYPE,
 	} type;
 };
 
-- 
2.7.4

[PATCH v5 11/25] NFSv4.1: Ensure we always run TEST/FREE_STATEID on locks

[Trond Myklebust]

Right now, we're only running TEST/FREE_STATEID on the locks if
the open stateid recovery succeeds. The protocol requires us to
always do so.
The fix would be to move the call to TEST/FREE_STATEID and do it
before we attempt open recovery.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 92 +++++++++++++++++++++++++++++++------------------------
 1 file changed, 52 insertions(+), 40 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 3c1b8cb7dd95..33ca6d768bd2 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2486,6 +2486,45 @@ static void nfs41_check_delegation_stateid(struct nfs4_state *state)
 }
 
 /**
+ * nfs41_check_expired_locks - possibly free a lock stateid
+ *
+ * @state: NFSv4 state for an inode
+ *
+ * Returns NFS_OK if recovery for this stateid is now finished.
+ * Otherwise a negative NFS4ERR value is returned.
+ */
+static int nfs41_check_expired_locks(struct nfs4_state *state)
+{
+	int status, ret = NFS_OK;
+	struct nfs4_lock_state *lsp;
+	struct nfs_server *server = NFS_SERVER(state->inode);
+
+	if (!test_bit(LK_STATE_IN_USE, &state->flags))
+		goto out;
+	list_for_each_entry(lsp, &state->lock_states, ls_locks) {
+		if (test_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags)) {
+			struct rpc_cred *cred = lsp->ls_state->owner->so_cred;
+
+			status = nfs41_test_and_free_expired_stateid(server,
+					&lsp->ls_stateid,
+					cred);
+			trace_nfs4_test_lock_stateid(state, lsp, status);
+			if (status == -NFS4ERR_EXPIRED ||
+			    status == -NFS4ERR_BAD_STATEID) {
+				clear_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags);
+				if (!recover_lost_locks)
+					set_bit(NFS_LOCK_LOST, &lsp->ls_flags);
+			} else if (status != NFS_OK) {
+				ret = status;
+				break;
+			}
+		}
+	};
+out:
+	return ret;
+}
+
+/**
  * nfs41_check_open_stateid - possibly free an open stateid
  *
  * @state: NFSv4 state for an inode
@@ -2522,6 +2561,9 @@ static int nfs41_open_expired(struct nfs4_state_owner *sp, struct nfs4_state *st
 	int status;
 
 	nfs41_check_delegation_stateid(state);
+	status = nfs41_check_expired_locks(state);
+	if (status != NFS_OK)
+		return status;
 	status = nfs41_check_open_stateid(state);
 	if (status != NFS_OK)
 		status = nfs4_open_expired(sp, state);
@@ -6125,49 +6167,19 @@ out:
 }
 
 #if defined(CONFIG_NFS_V4_1)
-/**
- * nfs41_check_expired_locks - possibly free a lock stateid
- *
- * @state: NFSv4 state for an inode
- *
- * Returns NFS_OK if recovery for this stateid is now finished.
- * Otherwise a negative NFS4ERR value is returned.
- */
-static int nfs41_check_expired_locks(struct nfs4_state *state)
-{
-	int status, ret = NFS_OK;
-	struct nfs4_lock_state *lsp;
-	struct nfs_server *server = NFS_SERVER(state->inode);
-
-	list_for_each_entry(lsp, &state->lock_states, ls_locks) {
-		if (test_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags)) {
-			struct rpc_cred *cred = lsp->ls_state->owner->so_cred;
-
-			status = nfs41_test_and_free_expired_stateid(server,
-					&lsp->ls_stateid,
-					cred);
-			trace_nfs4_test_lock_stateid(state, lsp, status);
-			if (status == -NFS4ERR_EXPIRED ||
-			    status == -NFS4ERR_BAD_STATEID)
-				clear_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags);
-			else if (status != NFS_OK) {
-				ret = status;
-				break;
-			}
-		}
-	};
-
-	return ret;
-}
-
 static int nfs41_lock_expired(struct nfs4_state *state, struct file_lock *request)
 {
-	int status = NFS_OK;
+	struct nfs4_lock_state *lsp;
+	int status;
 
-	if (test_bit(LK_STATE_IN_USE, &state->flags))
-		status = nfs41_check_expired_locks(state);
-	if (status != NFS_OK)
-		status = nfs4_lock_expired(state, request);
+	status = nfs4_set_lock_state(state, request);
+	if (status != 0)
+		return status;
+	lsp = request->fl_u.nfs4_fl.owner;
+	if (test_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags) ||
+	    test_bit(NFS_LOCK_LOST, &lsp->ls_flags))
+		return 0;
+	status = nfs4_lock_expired(state, request);
 	return status;
 }
 #endif
-- 
2.7.4

[PATCH v5 12/25] NFSv4.1: FREE_STATEID can be asynchronous

[Trond Myklebust]

Nothing should need to be serialised with FREE_STATEID on the client,
so let's make the RPC call always asynchronous. Also constify the
stateid argument.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 33ca6d768bd2..a53caeb73f75 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -99,8 +99,8 @@ static int nfs4_do_setattr(struct inode *inode, struct rpc_cred *cred,
 #ifdef CONFIG_NFS_V4_1
 static int nfs41_test_stateid(struct nfs_server *, nfs4_stateid *,
 		struct rpc_cred *);
-static int nfs41_free_stateid(struct nfs_server *, nfs4_stateid *,
-		struct rpc_cred *);
+static int nfs41_free_stateid(struct nfs_server *, const nfs4_stateid *,
+		struct rpc_cred *, bool);
 #endif
 
 #ifdef CONFIG_NFS_V4_SECURITY_LABEL
@@ -2443,7 +2443,7 @@ static int nfs41_test_and_free_expired_stateid(struct nfs_server *server,
 	}
 out_free:
 	/* Ack the revoked state to the server */
-	nfs41_free_stateid(server, stateid, cred);
+	nfs41_free_stateid(server, stateid, cred, true);
 	return -NFS4ERR_EXPIRED;
 }
 
@@ -8722,7 +8722,7 @@ static const struct rpc_call_ops nfs41_free_stateid_ops = {
 };
 
 static struct rpc_task *_nfs41_free_stateid(struct nfs_server *server,
-		nfs4_stateid *stateid,
+		const nfs4_stateid *stateid,
 		struct rpc_cred *cred,
 		bool privileged)
 {
@@ -8765,38 +8765,31 @@ static struct rpc_task *_nfs41_free_stateid(struct nfs_server *server,
  * @server: server / transport on which to perform the operation
  * @stateid: state ID to release
  * @cred: credential
+ * @is_recovery: set to true if this call needs to be privileged
  *
- * Returns NFS_OK if the server freed "stateid".  Otherwise a
- * negative NFS4ERR value is returned.
+ * Note: this function is always asynchronous.
  */
 static int nfs41_free_stateid(struct nfs_server *server,
-		nfs4_stateid *stateid,
-		struct rpc_cred *cred)
+		const nfs4_stateid *stateid,
+		struct rpc_cred *cred,
+		bool is_recovery)
 {
 	struct rpc_task *task;
-	int ret;
 
-	task = _nfs41_free_stateid(server, stateid, cred, true);
+	task = _nfs41_free_stateid(server, stateid, cred, is_recovery);
 	if (IS_ERR(task))
 		return PTR_ERR(task);
-	ret = rpc_wait_for_completion_task(task);
-	if (!ret)
-		ret = task->tk_status;
 	rpc_put_task(task);
-	return ret;
+	return 0;
 }
 
 static void
 nfs41_free_lock_state(struct nfs_server *server, struct nfs4_lock_state *lsp)
 {
-	struct rpc_task *task;
 	struct rpc_cred *cred = lsp->ls_state->owner->so_cred;
 
-	task = _nfs41_free_stateid(server, &lsp->ls_stateid, cred, false);
+	nfs41_free_stateid(server, &lsp->ls_stateid, cred, false);
 	nfs4_free_lock_state(server, lsp);
-	if (IS_ERR(task))
-		return;
-	rpc_put_task(task);
 }
 
 static bool nfs41_match_stateid(const nfs4_stateid *s1,
-- 
2.7.4

[PATCH v5 13/25] NFSv4.1: Ensure we call FREE_STATEID if needed on close/delegreturn/locku

[Trond Myklebust]

If a server returns NFS4ERR_ADMIN_REVOKED, NFS4ERR_DELEG_REVOKED
or NFS4ERR_EXPIRED on a call to close, open_downgrade, delegreturn, or
locku, we should call FREE_STATEID before attempting to recover.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 43 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 40 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index a53caeb73f75..f79d4f55d83f 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -328,6 +328,33 @@ static void nfs4_setup_readdir(u64 cookie, __be32 *verifier, struct dentry *dent
 	kunmap_atomic(start);
 }
 
+static void nfs4_test_and_free_stateid(struct nfs_server *server,
+		nfs4_stateid *stateid,
+		struct rpc_cred *cred)
+{
+	const struct nfs4_minor_version_ops *ops = server->nfs_client->cl_mvops;
+
+	ops->test_and_free_expired(server, stateid, cred);
+}
+
+static void __nfs4_free_revoked_stateid(struct nfs_server *server,
+		nfs4_stateid *stateid,
+		struct rpc_cred *cred)
+{
+	stateid->type = NFS4_REVOKED_STATEID_TYPE;
+	nfs4_test_and_free_stateid(server, stateid, cred);
+}
+
+static void nfs4_free_revoked_stateid(struct nfs_server *server,
+		const nfs4_stateid *stateid,
+		struct rpc_cred *cred)
+{
+	nfs4_stateid tmp;
+
+	nfs4_stateid_copy(&tmp, stateid);
+	__nfs4_free_revoked_stateid(server, &tmp, cred);
+}
+
 static long nfs4_update_delay(long *timeout)
 {
 	long ret;
@@ -2983,9 +3010,12 @@ static void nfs4_close_done(struct rpc_task *task, void *data)
 			break;
 		case -NFS4ERR_ADMIN_REVOKED:
 		case -NFS4ERR_STALE_STATEID:
+		case -NFS4ERR_EXPIRED:
+			nfs4_free_revoked_stateid(server,
+					&calldata->arg.stateid,
+					task->tk_msg.rpc_cred);
 		case -NFS4ERR_OLD_STATEID:
 		case -NFS4ERR_BAD_STATEID:
-		case -NFS4ERR_EXPIRED:
 			if (!nfs4_stateid_match(&calldata->arg.stateid,
 						&state->open_stateid)) {
 				rpc_restart_call_prepare(task);
@@ -5477,10 +5507,13 @@ static void nfs4_delegreturn_done(struct rpc_task *task, void *calldata)
 		renew_lease(data->res.server, data->timestamp);
 	case -NFS4ERR_ADMIN_REVOKED:
 	case -NFS4ERR_DELEG_REVOKED:
+	case -NFS4ERR_EXPIRED:
+		nfs4_free_revoked_stateid(data->res.server,
+				data->args.stateid,
+				task->tk_msg.rpc_cred);
 	case -NFS4ERR_BAD_STATEID:
 	case -NFS4ERR_OLD_STATEID:
 	case -NFS4ERR_STALE_STATEID:
-	case -NFS4ERR_EXPIRED:
 		task->tk_status = 0;
 		if (data->roc)
 			pnfs_roc_set_barrier(data->inode, data->roc_barrier);
@@ -5745,10 +5778,14 @@ static void nfs4_locku_done(struct rpc_task *task, void *data)
 			if (nfs4_update_lock_stateid(calldata->lsp,
 					&calldata->res.stateid))
 				break;
+		case -NFS4ERR_ADMIN_REVOKED:
+		case -NFS4ERR_EXPIRED:
+			nfs4_free_revoked_stateid(calldata->server,
+					&calldata->arg.stateid,
+					task->tk_msg.rpc_cred);
 		case -NFS4ERR_BAD_STATEID:
 		case -NFS4ERR_OLD_STATEID:
 		case -NFS4ERR_STALE_STATEID:
-		case -NFS4ERR_EXPIRED:
 			if (!nfs4_stateid_match(&calldata->arg.stateid,
 						&calldata->lsp->ls_stateid))
 				rpc_restart_call_prepare(task);
-- 
2.7.4

[PATCH v5 14/25] NFSv4: Ensure we don't re-test revoked and freed stateids

[Trond Myklebust]

This fixes a potential infinite loop in nfs_reap_expired_delegations.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index f431fde13e90..34aa5dfbe99e 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -647,6 +647,7 @@ static void nfs_mark_delegation_revoked(struct nfs_server *server,
 		struct nfs_delegation *delegation)
 {
 	set_bit(NFS_DELEGATION_REVOKED, &delegation->flags);
+	delegation->stateid.type = NFS4_INVALID_STATEID_TYPE;
 	nfs_mark_return_delegation(server, delegation);
 }
 
@@ -885,6 +886,8 @@ static inline bool nfs4_server_rebooted(const struct nfs_client *clp)
 static void nfs_mark_test_expired_delegation(struct nfs_server *server,
 	    struct nfs_delegation *delegation)
 {
+	if (delegation->stateid.type == NFS4_INVALID_STATEID_TYPE)
+		return;
 	clear_bit(NFS_DELEGATION_NEED_RECLAIM, &delegation->flags);
 	set_bit(NFS_DELEGATION_TEST_EXPIRED, &delegation->flags);
 	set_bit(NFS4CLNT_DELEGATION_EXPIRED, &server->nfs_client->cl_state);
-- 
2.7.4

[PATCH v5 15/25] NFSv4: nfs_inode_find_delegation_state_and_recover() should check all stateids

[Trond Myklebust]

Modify the helper nfs_inode_find_delegation_state_and_recover() so that it
can check all stateids for whether or not they need recovery.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c | 18 ++++++++++++++++++
 fs/nfs/delegation.h |  2 ++
 fs/nfs/nfs4state.c  | 44 +++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 59 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 34aa5dfbe99e..11d49e1aa41b 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -992,6 +992,24 @@ restart:
 	rcu_read_unlock();
 }
 
+void nfs_inode_find_delegation_state_and_recover(struct inode *inode,
+		const nfs4_stateid *stateid)
+{
+	struct nfs_client *clp = NFS_SERVER(inode)->nfs_client;
+	struct nfs_delegation *delegation;
+	bool found = false;
+
+	rcu_read_lock();
+	delegation = rcu_dereference(NFS_I(inode)->delegation);
+	if (delegation && nfs4_stateid_match(&delegation->stateid, stateid)) {
+		nfs_mark_test_expired_delegation(NFS_SERVER(inode), delegation);
+		found = true;
+	}
+	rcu_read_unlock();
+	if (found)
+		nfs4_schedule_state_manager(clp);
+}
+
 /**
  * nfs_delegations_present - check for existence of delegations
  * @clp: client state handle
diff --git a/fs/nfs/delegation.h b/fs/nfs/delegation.h
index 1442e3b1521d..e9d555796873 100644
--- a/fs/nfs/delegation.h
+++ b/fs/nfs/delegation.h
@@ -66,6 +66,8 @@ void nfs_mark_delegation_referenced(struct nfs_delegation *delegation);
 int nfs4_have_delegation(struct inode *inode, fmode_t flags);
 int nfs4_check_delegation(struct inode *inode, fmode_t flags);
 bool nfs4_delegation_flush_on_close(const struct inode *inode);
+void nfs_inode_find_delegation_state_and_recover(struct inode *inode,
+		const nfs4_stateid *stateid);
 
 #endif
 
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 63da0411e2af..609643f31e7a 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1337,6 +1337,35 @@ int nfs4_schedule_stateid_recovery(const struct nfs_server *server, struct nfs4_
 }
 EXPORT_SYMBOL_GPL(nfs4_schedule_stateid_recovery);
 
+static struct nfs4_lock_state *
+nfs_state_find_lock_state_by_stateid(struct nfs4_state *state,
+		const nfs4_stateid *stateid)
+{
+	struct nfs4_lock_state *pos;
+
+	list_for_each_entry(pos, &state->lock_states, ls_locks) {
+		if (!test_bit(NFS_LOCK_INITIALIZED, &pos->ls_flags))
+			continue;
+		if (nfs4_stateid_match(&pos->ls_stateid, stateid))
+			return pos;
+	}
+	return NULL;
+}
+
+static bool nfs_state_lock_state_matches_stateid(struct nfs4_state *state,
+		const nfs4_stateid *stateid)
+{
+	bool found = false;
+
+	if (test_bit(LK_STATE_IN_USE, &state->flags)) {
+		spin_lock(&state->state_lock);
+		if (nfs_state_find_lock_state_by_stateid(state, stateid))
+			found = true;
+		spin_unlock(&state->state_lock);
+	}
+	return found;
+}
+
 void nfs_inode_find_state_and_recover(struct inode *inode,
 		const nfs4_stateid *stateid)
 {
@@ -1351,14 +1380,19 @@ void nfs_inode_find_state_and_recover(struct inode *inode,
 		state = ctx->state;
 		if (state == NULL)
 			continue;
-		if (!test_bit(NFS_DELEGATED_STATE, &state->flags))
-			continue;
-		if (!nfs4_stateid_match(&state->stateid, stateid))
+		if (nfs4_stateid_match(&state->stateid, stateid)) {
+			nfs4_state_mark_reclaim_nograce(clp, state);
+			found = true;
 			continue;
-		nfs4_state_mark_reclaim_nograce(clp, state);
-		found = true;
+		}
+		if (nfs_state_lock_state_matches_stateid(state, stateid)) {
+			nfs4_state_mark_reclaim_nograce(clp, state);
+			found = true;
+		}
 	}
 	spin_unlock(&inode->i_lock);
+
+	nfs_inode_find_delegation_state_and_recover(inode, stateid);
 	if (found)
 		nfs4_schedule_state_manager(clp);
 }
-- 
2.7.4

[PATCH v5 16/25] NFSv4: nfs4_do_handle_exception() handle revoke/expiry of a single stateid

[Trond Myklebust]

If we're not yet sure that all state has expired or been revoked, we
should try to do a minimal recovery on just the one stateid.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index f79d4f55d83f..84ee836a976c 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -400,10 +400,16 @@ static int nfs4_do_handle_exception(struct nfs_server *server,
 	switch(errorcode) {
 		case 0:
 			return 0;
-		case -NFS4ERR_OPENMODE:
 		case -NFS4ERR_DELEG_REVOKED:
 		case -NFS4ERR_ADMIN_REVOKED:
+		case -NFS4ERR_EXPIRED:
 		case -NFS4ERR_BAD_STATEID:
+			if (inode != NULL && stateid != NULL) {
+				nfs_inode_find_delegation_state_and_recover(inode,
+						stateid);
+				goto wait_on_recovery;
+			}
+		case -NFS4ERR_OPENMODE:
 			if (inode) {
 				int err;
 
@@ -422,12 +428,6 @@ static int nfs4_do_handle_exception(struct nfs_server *server,
 			if (ret < 0)
 				break;
 			goto wait_on_recovery;
-		case -NFS4ERR_EXPIRED:
-			if (state != NULL) {
-				ret = nfs4_schedule_stateid_recovery(server, state);
-				if (ret < 0)
-					break;
-			}
 		case -NFS4ERR_STALE_STATEID:
 		case -NFS4ERR_STALE_CLIENTID:
 			nfs4_schedule_lease_recovery(clp);
-- 
2.7.4

[PATCH v5 17/25] NFSv4: nfs4_handle_delegation_recall_error() handle expiration as revoke case

[Trond Myklebust]

If the server tells us our stateid has expired, then handle that as if
it was revoked.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 84ee836a976c..75edf90455b4 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -1919,7 +1919,6 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct
 		case -NFS4ERR_STALE_CLIENTID:
 		case -NFS4ERR_STALE_STATEID:
 			set_bit(NFS_DELEGATED_STATE, &state->flags);
-		case -NFS4ERR_EXPIRED:
 			/* Don't recall a delegation if it was lost */
 			nfs4_schedule_lease_recovery(server->nfs_client);
 			return -EAGAIN;
@@ -1931,6 +1930,7 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct
 			return -EAGAIN;
 		case -NFS4ERR_DELEG_REVOKED:
 		case -NFS4ERR_ADMIN_REVOKED:
+		case -NFS4ERR_EXPIRED:
 		case -NFS4ERR_BAD_STATEID:
 		case -NFS4ERR_OPENMODE:
 			nfs_inode_find_state_and_recover(state->inode,
-- 
2.7.4

[PATCH v5 18/25] NFSv4: nfs4_handle_setlk_error() handle expiration as revoke case

[Trond Myklebust]

If the server tells us our stateid has expired, then handle that as if
it was revoked.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 75edf90455b4..0c050e932c7e 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -6086,6 +6086,7 @@ static void nfs4_handle_setlk_error(struct nfs_server *server, struct nfs4_lock_
 {
 	switch (error) {
 	case -NFS4ERR_ADMIN_REVOKED:
+	case -NFS4ERR_EXPIRED:
 	case -NFS4ERR_BAD_STATEID:
 		lsp->ls_seqid.flags &= ~NFS_SEQID_CONFIRMED;
 		if (new_lock_owner != 0 ||
@@ -6094,7 +6095,6 @@ static void nfs4_handle_setlk_error(struct nfs_server *server, struct nfs4_lock_
 		break;
 	case -NFS4ERR_STALE_STATEID:
 		lsp->ls_seqid.flags &= ~NFS_SEQID_CONFIRMED;
-	case -NFS4ERR_EXPIRED:
 		nfs4_schedule_lease_recovery(server->nfs_client);
 	};
 }
-- 
2.7.4

[PATCH v5 19/25] NFSv4.1: nfs4_layoutget_handle_exception handle revoked state

[Trond Myklebust]

Handle revoked open/lock/delegation stateids when LAYOUTGET tells us
the state was revoked.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 0c050e932c7e..e2ab6f665dba 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -8069,6 +8069,8 @@ nfs4_layoutget_handle_exception(struct rpc_task *task,
 	case -NFS4ERR_RECALLCONFLICT:
 		status = -ERECALLCONFLICT;
 		break;
+	case -NFS4ERR_DELEG_REVOKED:
+	case -NFS4ERR_ADMIN_REVOKED:
 	case -NFS4ERR_EXPIRED:
 	case -NFS4ERR_BAD_STATEID:
 		exception->timeout = 0;
@@ -8080,6 +8082,7 @@ nfs4_layoutget_handle_exception(struct rpc_task *task,
 					&lgp->args.ctx->state->stateid)) {
 			spin_unlock(&inode->i_lock);
 			exception->state = lgp->args.ctx->state;
+			exception->stateid = &lgp->args.stateid;
 			break;
 		}
 
-- 
2.7.4

[PATCH v5 20/25] NFSv4: Pass the stateid to the exception handler in nfs4_read/write_done_cb

[Trond Myklebust]

The actual stateid used in the READ or WRITE can represent a delegation,
a lock or a stateid, so it is useful to pass it as an argument to the
exception handler when an expired/revoked response is received from the
server. It also ensures that we don't re-label the state as needing
recovery if that has already occurred.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 35 +++++++++++++++++++++++++----------
 1 file changed, 25 insertions(+), 10 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index e2ab6f665dba..82449e578608 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -4523,11 +4523,18 @@ static int nfs4_read_done_cb(struct rpc_task *task, struct nfs_pgio_header *hdr)
 	struct nfs_server *server = NFS_SERVER(hdr->inode);
 
 	trace_nfs4_read(hdr, task->tk_status);
-	if (nfs4_async_handle_error(task, server,
-				    hdr->args.context->state,
-				    NULL) == -EAGAIN) {
-		rpc_restart_call_prepare(task);
-		return -EAGAIN;
+	if (task->tk_status < 0) {
+		struct nfs4_exception exception = {
+			.inode = hdr->inode,
+			.state = hdr->args.context->state,
+			.stateid = &hdr->args.stateid,
+		};
+		task->tk_status = nfs4_async_handle_exception(task,
+				server, task->tk_status, &exception);
+		if (exception.retry) {
+			rpc_restart_call_prepare(task);
+			return -EAGAIN;
+		}
 	}
 
 	__nfs4_read_done_cb(hdr);
@@ -4596,11 +4603,19 @@ static int nfs4_write_done_cb(struct rpc_task *task,
 	struct inode *inode = hdr->inode;
 
 	trace_nfs4_write(hdr, task->tk_status);
-	if (nfs4_async_handle_error(task, NFS_SERVER(inode),
-				    hdr->args.context->state,
-				    NULL) == -EAGAIN) {
-		rpc_restart_call_prepare(task);
-		return -EAGAIN;
+	if (task->tk_status < 0) {
+		struct nfs4_exception exception = {
+			.inode = hdr->inode,
+			.state = hdr->args.context->state,
+			.stateid = &hdr->args.stateid,
+		};
+		task->tk_status = nfs4_async_handle_exception(task,
+				NFS_SERVER(inode), task->tk_status,
+				&exception);
+		if (exception.retry) {
+			rpc_restart_call_prepare(task);
+			return -EAGAIN;
+		}
 	}
 	if (task->tk_status >= 0) {
 		renew_lease(NFS_SERVER(inode), hdr->timestamp);
-- 
2.7.4

[PATCH v5 21/25] NFSv4: Fix a race in nfs_inode_reclaim_delegation()

[Trond Myklebust]

If we race with a delegreturn before taking the spin lock, we
currently end up dropping the delegation stateid.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 11d49e1aa41b..45b97cf4bab1 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -186,15 +186,13 @@ void nfs_inode_reclaim_delegation(struct inode *inode, struct rpc_cred *cred,
 			rcu_read_unlock();
 			put_rpccred(oldcred);
 			trace_nfs4_reclaim_delegation(inode, res->delegation_type);
-		} else {
-			/* We appear to have raced with a delegation return. */
-			spin_unlock(&delegation->lock);
-			rcu_read_unlock();
-			nfs_inode_set_delegation(inode, cred, res);
+			return;
 		}
-	} else {
-		rcu_read_unlock();
+		/* We appear to have raced with a delegation return. */
+		spin_unlock(&delegation->lock);
 	}
+	rcu_read_unlock();
+	nfs_inode_set_delegation(inode, cred, res);
 }
 
 static int nfs_do_return_delegation(struct inode *inode, struct nfs_delegation *delegation, int issync)
-- 
2.7.4

[PATCH v5 22/25] NFSv4: Fix a race when updating an open_stateid

[Trond Myklebust]

If we're replacing an old stateid which has a different 'other' field,
then we probably need to free the old stateid.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 35 ++++++++++++++++++++++++++---------
 1 file changed, 26 insertions(+), 9 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 82449e578608..08c6e46c5b95 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -1399,11 +1399,12 @@ static void nfs_test_and_clear_all_open_stateid(struct nfs4_state *state)
 }
 
 static bool nfs_need_update_open_stateid(struct nfs4_state *state,
-		nfs4_stateid *stateid)
+		const nfs4_stateid *stateid, nfs4_stateid *freeme)
 {
 	if (test_and_set_bit(NFS_OPEN_STATE, &state->flags) == 0)
 		return true;
 	if (!nfs4_stateid_match_other(stateid, &state->open_stateid)) {
+		nfs4_stateid_copy(freeme, &state->open_stateid);
 		nfs_test_and_clear_all_open_stateid(state);
 		return true;
 	}
@@ -1467,7 +1468,9 @@ static void nfs_clear_open_stateid(struct nfs4_state *state,
 		nfs4_schedule_state_manager(state->owner->so_server->nfs_client);
 }
 
-static void nfs_set_open_stateid_locked(struct nfs4_state *state, nfs4_stateid *stateid, fmode_t fmode)
+static void nfs_set_open_stateid_locked(struct nfs4_state *state,
+		const nfs4_stateid *stateid, fmode_t fmode,
+		nfs4_stateid *freeme)
 {
 	switch (fmode) {
 		case FMODE_READ:
@@ -1479,14 +1482,18 @@ static void nfs_set_open_stateid_locked(struct nfs4_state *state, nfs4_stateid *
 		case FMODE_READ|FMODE_WRITE:
 			set_bit(NFS_O_RDWR_STATE, &state->flags);
 	}
-	if (!nfs_need_update_open_stateid(state, stateid))
+	if (!nfs_need_update_open_stateid(state, stateid, freeme))
 		return;
 	if (test_bit(NFS_DELEGATED_STATE, &state->flags) == 0)
 		nfs4_stateid_copy(&state->stateid, stateid);
 	nfs4_stateid_copy(&state->open_stateid, stateid);
 }
 
-static void __update_open_stateid(struct nfs4_state *state, nfs4_stateid *open_stateid, const nfs4_stateid *deleg_stateid, fmode_t fmode)
+static void __update_open_stateid(struct nfs4_state *state,
+		const nfs4_stateid *open_stateid,
+		const nfs4_stateid *deleg_stateid,
+		fmode_t fmode,
+		nfs4_stateid *freeme)
 {
 	/*
 	 * Protect the call to nfs4_state_set_mode_locked and
@@ -1499,16 +1506,22 @@ static void __update_open_stateid(struct nfs4_state *state, nfs4_stateid *open_s
 		set_bit(NFS_DELEGATED_STATE, &state->flags);
 	}
 	if (open_stateid != NULL)
-		nfs_set_open_stateid_locked(state, open_stateid, fmode);
+		nfs_set_open_stateid_locked(state, open_stateid, fmode, freeme);
 	write_sequnlock(&state->seqlock);
 	update_open_stateflags(state, fmode);
 	spin_unlock(&state->owner->so_lock);
 }
 
-static int update_open_stateid(struct nfs4_state *state, nfs4_stateid *open_stateid, nfs4_stateid *delegation, fmode_t fmode)
+static int update_open_stateid(struct nfs4_state *state,
+		const nfs4_stateid *open_stateid,
+		const nfs4_stateid *delegation,
+		fmode_t fmode)
 {
+	struct nfs_server *server = NFS_SERVER(state->inode);
+	struct nfs_client *clp = server->nfs_client;
 	struct nfs_inode *nfsi = NFS_I(state->inode);
 	struct nfs_delegation *deleg_cur;
+	nfs4_stateid freeme = {0};
 	int ret = 0;
 
 	fmode &= (FMODE_READ|FMODE_WRITE);
@@ -1530,7 +1543,8 @@ static int update_open_stateid(struct nfs4_state *state, nfs4_stateid *open_stat
 		goto no_delegation_unlock;
 
 	nfs_mark_delegation_referenced(deleg_cur);
-	__update_open_stateid(state, open_stateid, &deleg_cur->stateid, fmode);
+	__update_open_stateid(state, open_stateid, &deleg_cur->stateid,
+			fmode, &freeme);
 	ret = 1;
 no_delegation_unlock:
 	spin_unlock(&deleg_cur->lock);
@@ -1538,11 +1552,14 @@ no_delegation:
 	rcu_read_unlock();
 
 	if (!ret && open_stateid != NULL) {
-		__update_open_stateid(state, open_stateid, NULL, fmode);
+		__update_open_stateid(state, open_stateid, NULL, fmode, &freeme);
 		ret = 1;
 	}
 	if (test_bit(NFS_STATE_RECLAIM_NOGRACE, &state->flags))
-		nfs4_schedule_state_manager(state->owner->so_server->nfs_client);
+		nfs4_schedule_state_manager(clp);
+	if (freeme.type != 0)
+		nfs4_test_and_free_stateid(server, &freeme,
+				state->owner->so_cred);
 
 	return ret;
 }
-- 
2.7.4

[PATCH v5 23/25] NFS: Always call nfs_inode_find_state_and_recover() when revoking a delegation

[Trond Myklebust]

Don't rely on nfs_inode_detach_delegation() succeeding. That can race...

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/delegation.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 45b97cf4bab1..62b2215a30b9 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -653,18 +653,24 @@ static bool nfs_revoke_delegation(struct inode *inode,
 		const nfs4_stateid *stateid)
 {
 	struct nfs_delegation *delegation;
+	nfs4_stateid tmp;
 	bool ret = false;
 
 	rcu_read_lock();
 	delegation = rcu_dereference(NFS_I(inode)->delegation);
 	if (delegation == NULL)
 		goto out;
-	if (stateid && !nfs4_stateid_match(stateid, &delegation->stateid))
+	if (stateid == NULL) {
+		nfs4_stateid_copy(&tmp, &delegation->stateid);
+		stateid = &tmp;
+	} else if (!nfs4_stateid_match(stateid, &delegation->stateid))
 		goto out;
 	nfs_mark_delegation_revoked(NFS_SERVER(inode), delegation);
 	ret = true;
 out:
 	rcu_read_unlock();
+	if (ret)
+		nfs_inode_find_state_and_recover(inode, stateid);
 	return ret;
 }
 
@@ -676,10 +682,8 @@ void nfs_remove_bad_delegation(struct inode *inode,
 	if (!nfs_revoke_delegation(inode, stateid))
 		return;
 	delegation = nfs_inode_detach_delegation(inode);
-	if (delegation) {
-		nfs_inode_find_state_and_recover(inode, &delegation->stateid);
+	if (delegation)
 		nfs_free_delegation(delegation);
-	}
 }
 EXPORT_SYMBOL_GPL(nfs_remove_bad_delegation);
 
-- 
2.7.4

[PATCH v5 24/25] NFSv4: Don't test open_stateid unless it is set

[Trond Myklebust]

We need to test the NFS_OPEN_STATE flag for whether or not the
open_stateid is valid.

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 08c6e46c5b95..e4f2008cde7d 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2583,6 +2583,11 @@ static int nfs41_check_open_stateid(struct nfs4_state *state)
 	struct rpc_cred *cred = state->owner->so_cred;
 	int status;
 
+	if (test_bit(NFS_OPEN_STATE, &state->flags) == 0) {
+		if (test_bit(NFS_DELEGATED_STATE, &state->flags) == 0)
+			return NFS_OK;
+		return -NFS4ERR_BAD_STATEID;
+	}
 	/* If a state reset has been done, test_stateid is unneeded */
 	if ((test_bit(NFS_O_RDONLY_STATE, &state->flags) == 0) &&
 	    (test_bit(NFS_O_WRONLY_STATE, &state->flags) == 0) &&
-- 
2.7.4

[PATCH v5 25/25] NFSv4: Mark the lock and open stateids as invalid after freeing them

[Trond Myklebust]

Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
 fs/nfs/nfs4proc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index e4f2008cde7d..b44ea0b2ed23 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -2556,6 +2556,7 @@ static int nfs41_check_expired_locks(struct nfs4_state *state)
 			if (status == -NFS4ERR_EXPIRED ||
 			    status == -NFS4ERR_BAD_STATEID) {
 				clear_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags);
+				lsp->ls_stateid.type = NFS4_INVALID_STATEID_TYPE;
 				if (!recover_lost_locks)
 					set_bit(NFS_LOCK_LOST, &lsp->ls_flags);
 			} else if (status != NFS_OK) {
@@ -2601,6 +2602,7 @@ static int nfs41_check_open_stateid(struct nfs4_state *state)
 		clear_bit(NFS_O_WRONLY_STATE, &state->flags);
 		clear_bit(NFS_O_RDWR_STATE, &state->flags);
 		clear_bit(NFS_OPEN_STATE, &state->flags);
+		stateid->type = NFS4_INVALID_STATEID_TYPE;
 	}
 	return status;
 }
-- 
2.7.4

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Oleg Drokin]

On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:

> According to RFC5661, if any of the SEQUENCE status bits
> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, SEQ4_STATUS_ADMIN_STATE_REVOKED,
> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need to use
> TEST_STATEID to figure out which stateids have been revoked, so we
> can acknowledge the loss of state using FREE_STATEID.
> 
> While we already do this for open and lock state, we have not been doing
> so for all the delegations.
> 
> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired too
> v3: Now with added lock revoke fixes and close/delegreturn/locku fixes
> v4: Close a bunch of corner cases
> v5: Report revoked delegations as invalid in nfs_have_delegation()
>    Fix an infinite loop in nfs_reap_expired_delegations.
>    Fixes for other looping behaviour

This time around the loop seems to be more tight,
in userspace process:

[ 9197.256571] --> nfs41_call_sync_prepare data->seq_server ffff8800a73ce000
[ 9197.256572] --> nfs41_setup_sequence
[ 9197.256573] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[ 9197.256574] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[ 9197.256574] <-- nfs41_setup_sequence slotid=0 seqid=14013800
[ 9197.256582] encode_sequence: sessionid=1474126170:1:2:0 seqid=14013800 slotid=0 max_slotid=0 cache_this=1
[ 9197.256755] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[ 9197.256756] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
[ 9197.256758] nfs41_sequence_process: Error 0 free the slot 
[ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[ 9197.256779] --> nfs_put_client({2})
[ 9197.256799] --> nfs41_call_sync_prepare data->seq_server ffff8800a73ce000
[ 9197.256800] --> nfs41_setup_sequence
[ 9197.256801] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[ 9197.256802] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[ 9197.256803] <-- nfs41_setup_sequence slotid=0 seqid=14013801
[ 9197.256811] encode_sequence: sessionid=1474126170:1:2:0 seqid=14013801 slotid=0 max_slotid=0 cache_this=1
[ 9197.257005] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[ 9197.257006] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[ 9197.257008] nfs4_free_slot: slotid 1 highest_used_slotid 0
[ 9197.257008] nfs41_sequence_process: Error 0 free the slot 
[ 9197.257010] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[ 9197.257029] --> nfs_put_client({2})
[ 9197.257035] --> nfs41_call_sync_prepare data->seq_server ffff8800a73ce000
[ 9197.257035] --> nfs41_setup_sequence
[ 9197.257036] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[ 9197.257037] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[ 9197.257038] <-- nfs41_setup_sequence slotid=0 seqid=14013802
[ 9197.257063] encode_sequence: sessionid=1474126170:1:2:0 seqid=14013802 slotid=0 max_slotid=0 cache_this=1
[ 9197.257238] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[ 9197.257239] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[ 9197.257240] nfs4_free_slot: slotid 1 highest_used_slotid 0
[ 9197.257241] nfs41_sequence_process: Error 0 free the slot 
[ 9197.257242] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[ 9197.257262] --> nfs_put_client({2})
[ 9197.257267] --> nfs41_call_sync_prepare data->seq_server ffff8800a73ce000
[ 9197.257267] --> nfs41_setup_sequence
[ 9197.257268] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[ 9197.257269] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[ 9197.257270] <-- nfs41_setup_sequence slotid=0 seqid=14013803
[ 9197.257278] encode_sequence: sessionid=1474126170:1:2:0 seqid=14013803 slotid=0 max_slotid=0 cache_this=1
[ 9197.257451] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[ 9197.257452] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[ 9197.257454] nfs4_free_slot: slotid 1 highest_used_slotid 0
[ 9197.257454] nfs41_sequence_process: Error 0 free the slot 
[ 9197.257456] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[ 9197.257475] --> nfs_put_client({2})
[ 9197.257480] --> nfs41_call_sync_prepare data->seq_server ffff8800a73ce000
[ 9197.257480] --> nfs41_setup_sequence
[ 9197.257481] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[ 9197.257482] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[ 9197.257483] <-- nfs41_setup_sequence slotid=0 seqid=14013804
[ 9197.257491] encode_sequence: sessionid=1474126170:1:2:0 seqid=14013804 slotid=0 max_slotid=0 cache_this=1
[ 9197.257658] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[ 9197.257659] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[ 9197.257660] nfs4_free_slot: slotid 1 highest_used_slotid 0
[ 9197.257661] nfs41_sequence_process: Error 0 free the slot 
[ 9197.257662] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[ 9197.257681] --> nfs_put_client({2})

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Trond Myklebust]

> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru> wrote:
>=20
>=20
> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>=20
>> According to RFC5661, if any of the SEQUENCE status bits
>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, SEQ4_STATUS_ADMIN_STATE_REVOKED,
>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need to use
>> TEST_STATEID to figure out which stateids have been revoked, so we
>> can acknowledge the loss of state using FREE_STATEID.
>>=20
>> While we already do this for open and lock state, we have not been doing
>> so for all the delegations.
>>=20
>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired too
>> v3: Now with added lock revoke fixes and close/delegreturn/locku fixes
>> v4: Close a bunch of corner cases
>> v5: Report revoked delegations as invalid in nfs_have_delegation()
>>   Fix an infinite loop in nfs_reap_expired_delegations.
>>   Fixes for other looping behaviour
>=20
> This time around the loop seems to be more tight,
> in userspace process:
>=20
> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server ffff8800a73ce=
000
> [ 9197.256572] --> nfs41_setup_sequence
> [ 9197.256573] --> nfs4_alloc_slot used_slots=3D0000 highest_used=3D42949=
67295 max_slots=3D31
> [ 9197.256574] <-- nfs4_alloc_slot used_slots=3D0001 highest_used=3D0 slo=
tid=3D0
> [ 9197.256574] <-- nfs41_setup_sequence slotid=3D0 seqid=3D14013800
> [ 9197.256582] encode_sequence: sessionid=3D1474126170:1:2:0 seqid=3D1401=
3800 slotid=3D0 max_slotid=3D0 cache_this=3D1
> [ 9197.256755] --> nfs4_alloc_slot used_slots=3D0001 highest_used=3D0 max=
_slots=3D31
> [ 9197.256756] <-- nfs4_alloc_slot used_slots=3D0003 highest_used=3D1 slo=
tid=3D1
> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot=20
> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
> [ 9197.256779] --> nfs_put_client({2})

What operation is the userspace process hanging on? Do you have a stack tra=
ce for it?

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Oleg Drokin]

On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:

> 
>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru> wrote:
>> 
>> 
>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>> 
>>> According to RFC5661, if any of the SEQUENCE status bits
>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need to use
>>> TEST_STATEID to figure out which stateids have been revoked, so we
>>> can acknowledge the loss of state using FREE_STATEID.
>>> 
>>> While we already do this for open and lock state, we have not been doing
>>> so for all the delegations.
>>> 
>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired too
>>> v3: Now with added lock revoke fixes and close/delegreturn/locku fixes
>>> v4: Close a bunch of corner cases
>>> v5: Report revoked delegations as invalid in nfs_have_delegation()
>>>  Fix an infinite loop in nfs_reap_expired_delegations.
>>>  Fixes for other looping behaviour
>> 
>> This time around the loop seems to be more tight,
>> in userspace process:
>> 
>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server ffff8800a73ce000
>> [ 9197.256572] --> nfs41_setup_sequence
>> [ 9197.256573] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
>> [ 9197.256574] <-- nfs41_setup_sequence slotid=0 seqid=14013800
>> [ 9197.256582] encode_sequence: sessionid=1474126170:1:2:0 seqid=14013800 slotid=0 max_slotid=0 cache_this=1
>> [ 9197.256755] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot 
>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
>> [ 9197.256779] --> nfs_put_client({2})
> 
> What operation is the userspace process hanging on? Do you have a stack trace for it?

seems to be open_create->truncate->ssetattr coming from:
cp /bin/sleep /mnt/nfs2/racer/12

(gdb) bt
#0  nfs41_setup_sequence (session=0xffff88005a853800, args=0xffff8800a7253b80, 
    res=0xffff8800a7253b48, task=0xffff8800b0eb0f00)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
#1  0xffffffff813a751c in nfs41_call_sync_prepare (task=<optimized out>, 
    calldata=0xffff8800a7253b80)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
#2  0xffffffff8185c639 in rpc_prepare_task (task=<optimized out>)
    at /home/green/bk/linux-test/net/sunrpc/sched.c:683
#3  0xffffffff8185f12b in __rpc_execute (task=0xffff88005a853800)
    at /home/green/bk/linux-test/net/sunrpc/sched.c:775
#4  0xffffffff818617b4 in rpc_execute (task=0xffff88005a853800)
    at /home/green/bk/linux-test/net/sunrpc/sched.c:843
#5  0xffffffff818539b9 in rpc_run_task (task_setup_data=0xffff8800a7253a50)
    at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
#6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized out>, 
    server=<optimized out>, msg=<optimized out>, args=<optimized out>, 
    res=<optimized out>) at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1051
#7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=<optimized out>, 
    res=<optimized out>, args=<optimized out>, msg=<optimized out>, 
    server=<optimized out>, clnt=<optimized out>)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
#8  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
    res=<optimized out>, arg=<optimized out>, inode=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
#9  nfs4_do_setattr (inode=0xffff880079b152a8, cred=<optimized out>, 
    fattr=<optimized out>, sattr=<optimized out>, state=0xffff880060588e00, 
    ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
#10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=<optimized out>, 
    fattr=0xffff8800a7253b80, sattr=0xffff8800a7253b48)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
#11 0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800740c1000, 
    attr=0xffff8800a7253ce8) at /home/green/bk/linux-test/fs/nfs/inode.c:556
#12 0xffffffff81298d6b in notify_change (dentry=0xffff8800740c1000, 
    attr=0xffff8800a7253ce8, delegated_inode=<optimized out>)
    at /home/green/bk/linux-test/fs/attr.c:285
#13 0xffffffff812734e3 in do_truncate (dentry=0xffff88005a853800, 
    length=<optimized out>, time_attrs=<optimized out>, filp=<optimized out>)
    at /home/green/bk/linux-test/fs/open.c:63
#14 0xffffffff81287459 in handle_truncate (filp=<optimized out>)
    at /home/green/bk/linux-test/fs/namei.c:2960
#15 do_last (opened=<optimized out>, op=<optimized out>, file=<optimized out>, 
    nd=<optimized out>) at /home/green/bk/linux-test/fs/namei.c:3383
#16 path_openat (nd=0xffff8800a7253de0, op=<optimized out>, 
    flags=<optimized out>) at /home/green/bk/linux-test/fs/namei.c:3497
#17 0xffffffff81288b01 in do_filp_open (dfd=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    pathname=<optimized out>, op=0xffff8800a7253ef4)
    at /home/green/bk/linux-test/fs/namei.c:3532
#18 0xffffffff81274ab0 in do_sys_open (dfd=-100, filename=<optimized out>, 
    flags=<optimized out>, mode=<optimized out>)
    at /home/green/bk/linux-test/fs/open.c:1036
#19 0xffffffff81274bbe in SYSC_open (mode=<optimized out>, 
    flags=<optimized out>, filename=<optimized out>)
    at /home/green/bk/linux-test/fs/open.c:1054
#20 SyS_open (filename=<optimized out>, flags=<optimized out>, 
    mode=<optimized out>) at /home/green/bk/linux-test/fs/open.c:1049
#21 0xffffffff8189f03c in entry_SYSCALL_64_fastpath ()
    at /home/green/bk/linux-test/arch/x86/entry/entry_64.S:207

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Trond Myklebust]

T24gU2F0LCAyMDE2LTA5LTE3IGF0IDE1OjE2IC0wNDAwLCBPbGVnIERyb2tpbiB3cm90ZToNCj4g
T24gU2VwIDE3LCAyMDE2LCBhdCAyOjE4IFBNLCBUcm9uZCBNeWtsZWJ1c3Qgd3JvdGU6DQo+IA0K
PiA+IA0KPiA+IA0KPiA+ID4gDQo+ID4gPiBPbiBTZXAgMTcsIDIwMTYsIGF0IDE0OjA0LCBPbGVn
IERyb2tpbiA8Z3JlZW5AbGludXhoYWNrZXIucnU+DQo+ID4gPiB3cm90ZToNCj4gPiA+IA0KPiA+
ID4gDQo+ID4gPiBPbiBTZXAgMTcsIDIwMTYsIGF0IDE6MTMgQU0sIFRyb25kIE15a2xlYnVzdCB3
cm90ZToNCj4gPiA+IA0KPiA+ID4gPiANCj4gPiA+ID4gQWNjb3JkaW5nIHRvIFJGQzU2NjEsIGlm
IGFueSBvZiB0aGUgU0VRVUVOQ0Ugc3RhdHVzIGJpdHMNCj4gPiA+ID4gU0VRNF9TVEFUVVNfRVhQ
SVJFRF9BTExfU1RBVEVfUkVWT0tFRCwNCj4gPiA+ID4gU0VRNF9TVEFUVVNfRVhQSVJFRF9TT01F
X1NUQVRFX1JFVk9LRUQsDQo+ID4gPiA+IFNFUTRfU1RBVFVTX0FETUlOX1NUQVRFX1JFVk9LRUQs
DQo+ID4gPiA+IG9yIFNFUTRfU1RBVFVTX1JFQ0FMTEFCTEVfU1RBVEVfUkVWT0tFRCBhcmUgc2V0
LCB0aGVuIHdlIG5lZWQNCj4gPiA+ID4gdG8gdXNlDQo+ID4gPiA+IFRFU1RfU1RBVEVJRCB0byBm
aWd1cmUgb3V0IHdoaWNoIHN0YXRlaWRzIGhhdmUgYmVlbiByZXZva2VkLCBzbw0KPiA+ID4gPiB3
ZQ0KPiA+ID4gPiBjYW4gYWNrbm93bGVkZ2UgdGhlIGxvc3Mgb2Ygc3RhdGUgdXNpbmcgRlJFRV9T
VEFURUlELg0KPiA+ID4gPiANCj4gPiA+ID4gV2hpbGUgd2UgYWxyZWFkeSBkbyB0aGlzIGZvciBv
cGVuIGFuZCBsb2NrIHN0YXRlLCB3ZSBoYXZlIG5vdA0KPiA+ID4gPiBiZWVuIGRvaW5nDQo+ID4g
PiA+IHNvIGZvciBhbGwgdGhlIGRlbGVnYXRpb25zLg0KPiA+ID4gPiANCj4gPiA+ID4gdjI6IG5m
c192NF8yX21pbm9yX29wcyBuZWVkcyB0byBzZXQgLnRlc3RfYW5kX2ZyZWVfZXhwaXJlZCB0b28N
Cj4gPiA+ID4gdjM6IE5vdyB3aXRoIGFkZGVkIGxvY2sgcmV2b2tlIGZpeGVzIGFuZA0KPiA+ID4g
PiBjbG9zZS9kZWxlZ3JldHVybi9sb2NrdSBmaXhlcw0KPiA+ID4gPiB2NDogQ2xvc2UgYSBidW5j
aCBvZiBjb3JuZXIgY2FzZXMNCj4gPiA+ID4gdjU6IFJlcG9ydCByZXZva2VkIGRlbGVnYXRpb25z
IGFzIGludmFsaWQgaW4NCj4gPiA+ID4gbmZzX2hhdmVfZGVsZWdhdGlvbigpDQo+ID4gPiA+IMKg
Rml4IGFuIGluZmluaXRlIGxvb3AgaW4gbmZzX3JlYXBfZXhwaXJlZF9kZWxlZ2F0aW9ucy4NCj4g
PiA+ID4gwqBGaXhlcyBmb3Igb3RoZXIgbG9vcGluZyBiZWhhdmlvdXINCj4gPiA+IA0KPiA+ID4g
VGhpcyB0aW1lIGFyb3VuZCB0aGUgbG9vcCBzZWVtcyB0byBiZSBtb3JlIHRpZ2h0LA0KPiA+ID4g
aW4gdXNlcnNwYWNlIHByb2Nlc3M6DQo+ID4gPiANCj4gPiA+IFsgOTE5Ny4yNTY1NzFdIC0tPiBu
ZnM0MV9jYWxsX3N5bmNfcHJlcGFyZSBkYXRhLT5zZXFfc2VydmVyDQo+ID4gPiBmZmZmODgwMGE3
M2NlMDAwDQo+ID4gPiBbIDkxOTcuMjU2NTcyXSAtLT4gbmZzNDFfc2V0dXBfc2VxdWVuY2UNCj4g
PiA+IFsgOTE5Ny4yNTY1NzNdIC0tPiBuZnM0X2FsbG9jX3Nsb3QgdXNlZF9zbG90cz0wMDAwDQo+
ID4gPiBoaWdoZXN0X3VzZWQ9NDI5NDk2NzI5NSBtYXhfc2xvdHM9MzENCj4gPiA+IFsgOTE5Ny4y
NTY1NzRdIDwtLSBuZnM0X2FsbG9jX3Nsb3QgdXNlZF9zbG90cz0wMDAxIGhpZ2hlc3RfdXNlZD0w
DQo+ID4gPiBzbG90aWQ9MA0KPiA+ID4gWyA5MTk3LjI1NjU3NF0gPC0tIG5mczQxX3NldHVwX3Nl
cXVlbmNlIHNsb3RpZD0wIHNlcWlkPTE0MDEzODAwDQo+ID4gPiBbIDkxOTcuMjU2NTgyXSBlbmNv
ZGVfc2VxdWVuY2U6IHNlc3Npb25pZD0xNDc0MTI2MTcwOjE6MjowDQo+ID4gPiBzZXFpZD0xNDAx
MzgwMCBzbG90aWQ9MCBtYXhfc2xvdGlkPTAgY2FjaGVfdGhpcz0xDQo+ID4gPiBbIDkxOTcuMjU2
NzU1XSAtLT4gbmZzNF9hbGxvY19zbG90IHVzZWRfc2xvdHM9MDAwMSBoaWdoZXN0X3VzZWQ9MA0K
PiA+ID4gbWF4X3Nsb3RzPTMxDQo+ID4gPiBbIDkxOTcuMjU2NzU2XSA8LS0gbmZzNF9hbGxvY19z
bG90IHVzZWRfc2xvdHM9MDAwMyBoaWdoZXN0X3VzZWQ9MQ0KPiA+ID4gc2xvdGlkPTENCj4gPiA+
IFsgOTE5Ny4yNTY3NTddIG5mczRfZnJlZV9zbG90OiBzbG90aWQgMSBoaWdoZXN0X3VzZWRfc2xv
dGlkIDANCj4gPiA+IFsgOTE5Ny4yNTY3NThdIG5mczQxX3NlcXVlbmNlX3Byb2Nlc3M6IEVycm9y
IDAgZnJlZSB0aGUgc2xvdMKgDQo+ID4gPiBbIDkxOTcuMjU2NzYwXSBuZnM0X2ZyZWVfc2xvdDog
c2xvdGlkIDAgaGlnaGVzdF91c2VkX3Nsb3RpZA0KPiA+ID4gNDI5NDk2NzI5NQ0KPiA+ID4gWyA5
MTk3LjI1Njc3OV0gLS0+IG5mc19wdXRfY2xpZW50KHsyfSkNCj4gPiANCj4gPiBXaGF0IG9wZXJh
dGlvbiBpcyB0aGUgdXNlcnNwYWNlIHByb2Nlc3MgaGFuZ2luZyBvbj8gRG8geW91IGhhdmUgYQ0K
PiA+IHN0YWNrIHRyYWNlIGZvciBpdD8NCj4gDQo+IHNlZW1zIHRvIGJlIG9wZW5fY3JlYXRlLT50
cnVuY2F0ZS0+c3NldGF0dHIgY29taW5nIGZyb206DQo+IGNwIC9iaW4vc2xlZXAgL21udC9uZnMy
L3JhY2VyLzEyDQo+IA0KPiAoZ2RiKSBidA0KPiAjMMKgwqBuZnM0MV9zZXR1cF9zZXF1ZW5jZSAo
c2Vzc2lvbj0weGZmZmY4ODAwNWE4NTM4MDAsDQo+IGFyZ3M9MHhmZmZmODgwMGE3MjUzYjgwLMKg
DQo+IMKgwqDCoMKgcmVzPTB4ZmZmZjg4MDBhNzI1M2I0OCwgdGFzaz0weGZmZmY4ODAwYjBlYjBm
MDApDQo+IMKgwqDCoMKgYXQgL2hvbWUvZ3JlZW4vYmsvbGludXgtdGVzdC9mcy9uZnMvbmZzNHBy
b2MuYzo4NzYNCj4gIzHCoMKgMHhmZmZmZmZmZjgxM2E3NTFjIGluIG5mczQxX2NhbGxfc3luY19w
cmVwYXJlICh0YXNrPTxvcHRpbWl6ZWQNCj4gb3V0PizCoA0KPiDCoMKgwqDCoGNhbGxkYXRhPTB4
ZmZmZjg4MDBhNzI1M2I4MCkNCj4gwqDCoMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0
L2ZzL25mcy9uZnM0cHJvYy5jOjk2Ng0KPiAjMsKgwqAweGZmZmZmZmZmODE4NWM2MzkgaW4gcnBj
X3ByZXBhcmVfdGFzayAodGFzaz08b3B0aW1pemVkIG91dD4pDQo+IMKgwqDCoMKgYXQgL2hvbWUv
Z3JlZW4vYmsvbGludXgtdGVzdC9uZXQvc3VucnBjL3NjaGVkLmM6NjgzDQo+ICMzwqDCoDB4ZmZm
ZmZmZmY4MTg1ZjEyYiBpbiBfX3JwY19leGVjdXRlICh0YXNrPTB4ZmZmZjg4MDA1YTg1MzgwMCkN
Cj4gwqDCoMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0L25ldC9zdW5ycGMvc2NoZWQu
Yzo3NzUNCj4gIzTCoMKgMHhmZmZmZmZmZjgxODYxN2I0IGluIHJwY19leGVjdXRlICh0YXNrPTB4
ZmZmZjg4MDA1YTg1MzgwMCkNCj4gwqDCoMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0
L25ldC9zdW5ycGMvc2NoZWQuYzo4NDMNCj4gIzXCoMKgMHhmZmZmZmZmZjgxODUzOWI5IGluIHJw
Y19ydW5fdGFzaw0KPiAodGFza19zZXR1cF9kYXRhPTB4ZmZmZjg4MDBhNzI1M2E1MCkNCj4gwqDC
oMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0L25ldC9zdW5ycGMvY2xudC5jOjEwNTIN
Cj4gIzbCoMKgMHhmZmZmZmZmZjgxM2E3NWUzIGluIG5mczRfY2FsbF9zeW5jX3NlcXVlbmNlIChj
bG50PTxvcHRpbWl6ZWQNCj4gb3V0PizCoA0KPiDCoMKgwqDCoHNlcnZlcj08b3B0aW1pemVkIG91
dD4sIG1zZz08b3B0aW1pemVkIG91dD4sIGFyZ3M9PG9wdGltaXplZA0KPiBvdXQ+LMKgDQo+IMKg
wqDCoMKgcmVzPTxvcHRpbWl6ZWQgb3V0PikgYXQgL2hvbWUvZ3JlZW4vYmsvbGludXgtDQo+IHRl
c3QvZnMvbmZzL25mczRwcm9jLmM6MTA1MQ0KPiAjN8KgwqAweGZmZmZmZmZmODEzYjQ2NDUgaW4g
bmZzNF9jYWxsX3N5bmMgKGNhY2hlX3JlcGx5PTxvcHRpbWl6ZWQNCj4gb3V0PizCoA0KPiDCoMKg
wqDCoHJlcz08b3B0aW1pemVkIG91dD4sIGFyZ3M9PG9wdGltaXplZCBvdXQ+LCBtc2c9PG9wdGlt
aXplZCBvdXQ+LMKgDQo+IMKgwqDCoMKgc2VydmVyPTxvcHRpbWl6ZWQgb3V0PiwgY2xudD08b3B0
aW1pemVkIG91dD4pDQo+IMKgwqDCoMKgYXQgL2hvbWUvZ3JlZW4vYmsvbGludXgtdGVzdC9mcy9u
ZnMvbmZzNHByb2MuYzoxMDY5DQo+ICM4wqDCoF9uZnM0X2RvX3NldGF0dHIgKHN0YXRlPTxvcHRp
bWl6ZWQgb3V0PiwgY3JlZD08b3B0aW1pemVkIG91dD4swqANCj4gwqDCoMKgwqByZXM9PG9wdGlt
aXplZCBvdXQ+LCBhcmc9PG9wdGltaXplZCBvdXQ+LCBpbm9kZT08b3B0aW1pemVkIG91dD4pDQo+
IC0tLVR5cGUgPHJldHVybj4gdG8gY29udGludWUsIG9yIHEgPHJldHVybj4gdG8gcXVpdC0tLQ0K
PiDCoMKgwqDCoGF0IC9ob21lL2dyZWVuL2JrL2xpbnV4LXRlc3QvZnMvbmZzL25mczRwcm9jLmM6
MjkxNg0KPiAjOcKgwqBuZnM0X2RvX3NldGF0dHIgKGlub2RlPTB4ZmZmZjg4MDA3OWIxNTJhOCwg
Y3JlZD08b3B0aW1pemVkIG91dD4swqANCj4gwqDCoMKgwqBmYXR0cj08b3B0aW1pemVkIG91dD4s
IHNhdHRyPTxvcHRpbWl6ZWQgb3V0PiwNCj4gc3RhdGU9MHhmZmZmODgwMDYwNTg4ZTAwLMKgDQo+
IMKgwqDCoMKgaWxhYmVsPTxvcHRpbWl6ZWQgb3V0Piwgb2xhYmVsPTB4MCA8aXJxX3N0YWNrX3Vu
aW9uPikNCj4gwqDCoMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0L2ZzL25mcy9uZnM0
cHJvYy5jOjI5NTUNCj4gIzEwIDB4ZmZmZmZmZmY4MTNiNGExNiBpbiBuZnM0X3Byb2Nfc2V0YXR0
ciAoZGVudHJ5PTxvcHRpbWl6ZWQgb3V0PizCoA0KPiDCoMKgwqDCoGZhdHRyPTB4ZmZmZjg4MDBh
NzI1M2I4MCwgc2F0dHI9MHhmZmZmODgwMGE3MjUzYjQ4KQ0KPiDCoMKgwqDCoGF0IC9ob21lL2dy
ZWVuL2JrL2xpbnV4LXRlc3QvZnMvbmZzL25mczRwcm9jLmM6MzY4NA0KPiAjMTEgMHhmZmZmZmZm
ZjgxMzhmMWNiIGluIG5mc19zZXRhdHRyIChkZW50cnk9MHhmZmZmODgwMDc0MGMxMDAwLMKgDQo+
wqANCg0KQ29vbCEgRG9lcyB0aGUgZm9sbG93aW5nIGhlbHA/DQoNCjg8LS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpGcm9tIDk4ZGRm
MzJhOTljZmUwMGU5YWUxMDgwNDRlMmJlNjc1MjI5ODc1MTEgTW9uIFNlcCAxNyAwMDowMDowMCAy
MDAxDQpGcm9tOiBUcm9uZCBNeWtsZWJ1c3QgPHRyb25kLm15a2xlYnVzdEBwcmltYXJ5ZGF0YS5j
b20+DQpEYXRlOiBTYXQsIDE3IFNlcCAyMDE2IDE1OjI3OjEwIC0wNDAwDQpTdWJqZWN0OiBbUEFU
Q0hdIE5GUzogRG9uJ3QgYXNzdW1lIGEgc3RhdGVpZCByZXByZXNlbnRzIGEgZGVsZWdhdGlvbiBp
bg0KIG5mczRfZG9faGFuZGxlX2V4Y2VwdGlvbg0KDQpJZiB0aGUgc3RhdGVpZCBiZWluZyBwYXNz
ZWQgdG8gdGhlIGVycm9yIGhhbmRsZXIgaXMgbm90IGEgZGVsZWdhdGlvbg0Kc3RhdGVpZCwgd2Ug
d2FudCB0byBtYXJrIHRoZSBsb2Nrcy9vcGVuX3N0YXRlIGl0IGRvZXMgcmVwcmVzZW50IGZvcg0K
cmVjb3ZlcnkuDQoNClNpZ25lZC1vZmYtYnk6IFRyb25kIE15a2xlYnVzdCA8dHJvbmQubXlrbGVi
dXN0QHByaW1hcnlkYXRhLmNvbT4NCi0tLQ0KIGZzL25mcy9uZnM0cHJvYy5jIHwgNiArKysrKy0N
CiAxIGZpbGUgY2hhbmdlZCwgNSBpbnNlcnRpb25zKCspLCAxIGRlbGV0aW9uKC0pDQoNCmRpZmYg
LS1naXQgYS9mcy9uZnMvbmZzNHByb2MuYyBiL2ZzL25mcy9uZnM0cHJvYy5jDQppbmRleCA3Y2Vj
YjFkN2EyMTcuLmFjYzU3MmM1MTczNSAxMDA2NDQNCi0tLSBhL2ZzL25mcy9uZnM0cHJvYy5jDQor
KysgYi9mcy9uZnMvbmZzNHByb2MuYw0KQEAgLTM5Nyw2ICszOTcsMTAgQEAgc3RhdGljIGludCBu
ZnM0X2RvX2hhbmRsZV9leGNlcHRpb24oc3RydWN0IG5mc19zZXJ2ZXIgKnNlcnZlciwNCiAJZXhj
ZXB0aW9uLT5kZWxheSA9IDA7DQogCWV4Y2VwdGlvbi0+cmVjb3ZlcmluZyA9IDA7DQogCWV4Y2Vw
dGlvbi0+cmV0cnkgPSAwOw0KKw0KKwlpZiAoc3RhdGVpZCA9PSBOVUxMICYmIHN0YXRlICE9IE5V
TEwpDQorCQlzdGF0ZWlkID0gJnN0YXRlLT5zdGF0ZWlkOw0KKw0KIAlzd2l0Y2goZXJyb3Jjb2Rl
KSB7DQogCQljYXNlIDA6DQogCQkJcmV0dXJuIDA7DQpAQCAtNDA1LDcgKzQwOSw3IEBAIHN0YXRp
YyBpbnQgbmZzNF9kb19oYW5kbGVfZXhjZXB0aW9uKHN0cnVjdCBuZnNfc2VydmVyICpzZXJ2ZXIs
DQogCQljYXNlIC1ORlM0RVJSX0VYUElSRUQ6DQogCQljYXNlIC1ORlM0RVJSX0JBRF9TVEFURUlE
Og0KIAkJCWlmIChpbm9kZSAhPSBOVUxMICYmIHN0YXRlaWQgIT0gTlVMTCkgew0KLQkJCQluZnNf
aW5vZGVfZmluZF9kZWxlZ2F0aW9uX3N0YXRlX2FuZF9yZWNvdmVyKGlub2RlLA0KKwkJCQluZnNf
aW5vZGVfZmluZF9zdGF0ZV9hbmRfcmVjb3Zlcihpbm9kZSwNCiAJCQkJCQlzdGF0ZWlkKTsNCiAJ
CQkJZ290byB3YWl0X29uX3JlY292ZXJ5Ow0KIAkJCX0NCi0tIA0KMi43LjQ=

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Trond Myklebust]

T24gU2F0LCAyMDE2LTA5LTE3IGF0IDE1OjMyIC0wNDAwLCBUcm9uZCBNeWtsZWJ1c3Qgd3JvdGU6
DQo+IE9uIFNhdCwgMjAxNi0wOS0xNyBhdCAxNToxNiAtMDQwMCwgT2xlZyBEcm9raW4gd3JvdGU6
DQo+ID4gDQo+ID4gT24gU2VwIDE3LCAyMDE2LCBhdCAyOjE4IFBNLCBUcm9uZCBNeWtsZWJ1c3Qg
d3JvdGU6DQo+ID4gDQo+ID4gPiANCj4gPiA+IA0KPiA+ID4gDQo+ID4gPiA+IA0KPiA+ID4gPiAN
Cj4gPiA+ID4gT24gU2VwIDE3LCAyMDE2LCBhdCAxNDowNCwgT2xlZyBEcm9raW4gPGdyZWVuQGxp
bnV4aGFja2VyLnJ1Pg0KPiA+ID4gPiB3cm90ZToNCj4gPiA+ID4gDQo+ID4gPiA+IA0KPiA+ID4g
PiBPbiBTZXAgMTcsIDIwMTYsIGF0IDE6MTMgQU0sIFRyb25kIE15a2xlYnVzdCB3cm90ZToNCj4g
PiA+ID4gDQo+ID4gPiA+ID4gDQo+ID4gPiA+ID4gDQo+ID4gPiA+ID4gQWNjb3JkaW5nIHRvIFJG
QzU2NjEsIGlmIGFueSBvZiB0aGUgU0VRVUVOQ0Ugc3RhdHVzIGJpdHMNCj4gPiA+ID4gPiBTRVE0
X1NUQVRVU19FWFBJUkVEX0FMTF9TVEFURV9SRVZPS0VELA0KPiA+ID4gPiA+IFNFUTRfU1RBVFVT
X0VYUElSRURfU09NRV9TVEFURV9SRVZPS0VELA0KPiA+ID4gPiA+IFNFUTRfU1RBVFVTX0FETUlO
X1NUQVRFX1JFVk9LRUQsDQo+ID4gPiA+ID4gb3IgU0VRNF9TVEFUVVNfUkVDQUxMQUJMRV9TVEFU
RV9SRVZPS0VEIGFyZSBzZXQsIHRoZW4gd2UgbmVlZA0KPiA+ID4gPiA+IHRvIHVzZQ0KPiA+ID4g
PiA+IFRFU1RfU1RBVEVJRCB0byBmaWd1cmUgb3V0IHdoaWNoIHN0YXRlaWRzIGhhdmUgYmVlbiBy
ZXZva2VkLA0KPiA+ID4gPiA+IHNvDQo+ID4gPiA+ID4gd2UNCj4gPiA+ID4gPiBjYW4gYWNrbm93
bGVkZ2UgdGhlIGxvc3Mgb2Ygc3RhdGUgdXNpbmcgRlJFRV9TVEFURUlELg0KPiA+ID4gPiA+IA0K
PiA+ID4gPiA+IFdoaWxlIHdlIGFscmVhZHkgZG8gdGhpcyBmb3Igb3BlbiBhbmQgbG9jayBzdGF0
ZSwgd2UgaGF2ZSBub3QNCj4gPiA+ID4gPiBiZWVuIGRvaW5nDQo+ID4gPiA+ID4gc28gZm9yIGFs
bCB0aGUgZGVsZWdhdGlvbnMuDQo+ID4gPiA+ID4gDQo+ID4gPiA+ID4gdjI6IG5mc192NF8yX21p
bm9yX29wcyBuZWVkcyB0byBzZXQgLnRlc3RfYW5kX2ZyZWVfZXhwaXJlZA0KPiA+ID4gPiA+IHRv
bw0KPiA+ID4gPiA+IHYzOiBOb3cgd2l0aCBhZGRlZCBsb2NrIHJldm9rZSBmaXhlcyBhbmQNCj4g
PiA+ID4gPiBjbG9zZS9kZWxlZ3JldHVybi9sb2NrdSBmaXhlcw0KPiA+ID4gPiA+IHY0OiBDbG9z
ZSBhIGJ1bmNoIG9mIGNvcm5lciBjYXNlcw0KPiA+ID4gPiA+IHY1OiBSZXBvcnQgcmV2b2tlZCBk
ZWxlZ2F0aW9ucyBhcyBpbnZhbGlkIGluDQo+ID4gPiA+ID4gbmZzX2hhdmVfZGVsZWdhdGlvbigp
DQo+ID4gPiA+ID4gwqBGaXggYW4gaW5maW5pdGUgbG9vcCBpbiBuZnNfcmVhcF9leHBpcmVkX2Rl
bGVnYXRpb25zLg0KPiA+ID4gPiA+IMKgRml4ZXMgZm9yIG90aGVyIGxvb3BpbmcgYmVoYXZpb3Vy
DQo+ID4gPiA+IA0KPiA+ID4gPiBUaGlzIHRpbWUgYXJvdW5kIHRoZSBsb29wIHNlZW1zIHRvIGJl
IG1vcmUgdGlnaHQsDQo+ID4gPiA+IGluIHVzZXJzcGFjZSBwcm9jZXNzOg0KPiA+ID4gPiANCj4g
PiA+ID4gWyA5MTk3LjI1NjU3MV0gLS0+IG5mczQxX2NhbGxfc3luY19wcmVwYXJlIGRhdGEtPnNl
cV9zZXJ2ZXINCj4gPiA+ID4gZmZmZjg4MDBhNzNjZTAwMA0KPiA+ID4gPiBbIDkxOTcuMjU2NTcy
XSAtLT4gbmZzNDFfc2V0dXBfc2VxdWVuY2UNCj4gPiA+ID4gWyA5MTk3LjI1NjU3M10gLS0+IG5m
czRfYWxsb2Nfc2xvdCB1c2VkX3Nsb3RzPTAwMDANCj4gPiA+ID4gaGlnaGVzdF91c2VkPTQyOTQ5
NjcyOTUgbWF4X3Nsb3RzPTMxDQo+ID4gPiA+IFsgOTE5Ny4yNTY1NzRdIDwtLSBuZnM0X2FsbG9j
X3Nsb3QgdXNlZF9zbG90cz0wMDAxDQo+ID4gPiA+IGhpZ2hlc3RfdXNlZD0wDQo+ID4gPiA+IHNs
b3RpZD0wDQo+ID4gPiA+IFsgOTE5Ny4yNTY1NzRdIDwtLSBuZnM0MV9zZXR1cF9zZXF1ZW5jZSBz
bG90aWQ9MCBzZXFpZD0xNDAxMzgwMA0KPiA+ID4gPiBbIDkxOTcuMjU2NTgyXSBlbmNvZGVfc2Vx
dWVuY2U6IHNlc3Npb25pZD0xNDc0MTI2MTcwOjE6MjowDQo+ID4gPiA+IHNlcWlkPTE0MDEzODAw
IHNsb3RpZD0wIG1heF9zbG90aWQ9MCBjYWNoZV90aGlzPTENCj4gPiA+ID4gWyA5MTk3LjI1Njc1
NV0gLS0+IG5mczRfYWxsb2Nfc2xvdCB1c2VkX3Nsb3RzPTAwMDENCj4gPiA+ID4gaGlnaGVzdF91
c2VkPTANCj4gPiA+ID4gbWF4X3Nsb3RzPTMxDQo+ID4gPiA+IFsgOTE5Ny4yNTY3NTZdIDwtLSBu
ZnM0X2FsbG9jX3Nsb3QgdXNlZF9zbG90cz0wMDAzDQo+ID4gPiA+IGhpZ2hlc3RfdXNlZD0xDQo+
ID4gPiA+IHNsb3RpZD0xDQo+ID4gPiA+IFsgOTE5Ny4yNTY3NTddIG5mczRfZnJlZV9zbG90OiBz
bG90aWQgMSBoaWdoZXN0X3VzZWRfc2xvdGlkIDANCj4gPiA+ID4gWyA5MTk3LjI1Njc1OF0gbmZz
NDFfc2VxdWVuY2VfcHJvY2VzczogRXJyb3IgMCBmcmVlIHRoZSBzbG90wqANCj4gPiA+ID4gWyA5
MTk3LjI1Njc2MF0gbmZzNF9mcmVlX3Nsb3Q6IHNsb3RpZCAwIGhpZ2hlc3RfdXNlZF9zbG90aWQN
Cj4gPiA+ID4gNDI5NDk2NzI5NQ0KPiA+ID4gPiBbIDkxOTcuMjU2Nzc5XSAtLT4gbmZzX3B1dF9j
bGllbnQoezJ9KQ0KPiA+ID4gDQo+ID4gPiBXaGF0IG9wZXJhdGlvbiBpcyB0aGUgdXNlcnNwYWNl
IHByb2Nlc3MgaGFuZ2luZyBvbj8gRG8geW91IGhhdmUgYQ0KPiA+ID4gc3RhY2sgdHJhY2UgZm9y
IGl0Pw0KPiA+IA0KPiA+IHNlZW1zIHRvIGJlIG9wZW5fY3JlYXRlLT50cnVuY2F0ZS0+c3NldGF0
dHIgY29taW5nIGZyb206DQo+ID4gY3AgL2Jpbi9zbGVlcCAvbW50L25mczIvcmFjZXIvMTINCj4g
PiANCj4gPiAoZ2RiKSBidA0KPiA+ICMwwqDCoG5mczQxX3NldHVwX3NlcXVlbmNlIChzZXNzaW9u
PTB4ZmZmZjg4MDA1YTg1MzgwMCwNCj4gPiBhcmdzPTB4ZmZmZjg4MDBhNzI1M2I4MCzCoA0KPiA+
IMKgwqDCoMKgcmVzPTB4ZmZmZjg4MDBhNzI1M2I0OCwgdGFzaz0weGZmZmY4ODAwYjBlYjBmMDAp
DQo+ID4gwqDCoMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0L2ZzL25mcy9uZnM0cHJv
Yy5jOjg3Ng0KPiA+ICMxwqDCoDB4ZmZmZmZmZmY4MTNhNzUxYyBpbiBuZnM0MV9jYWxsX3N5bmNf
cHJlcGFyZSAodGFzaz08b3B0aW1pemVkDQo+ID4gb3V0PizCoA0KPiA+IMKgwqDCoMKgY2FsbGRh
dGE9MHhmZmZmODgwMGE3MjUzYjgwKQ0KPiA+IMKgwqDCoMKgYXQgL2hvbWUvZ3JlZW4vYmsvbGlu
dXgtdGVzdC9mcy9uZnMvbmZzNHByb2MuYzo5NjYNCj4gPiAjMsKgwqAweGZmZmZmZmZmODE4NWM2
MzkgaW4gcnBjX3ByZXBhcmVfdGFzayAodGFzaz08b3B0aW1pemVkIG91dD4pDQo+ID4gwqDCoMKg
wqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0L25ldC9zdW5ycGMvc2NoZWQuYzo2ODMNCj4g
PiAjM8KgwqAweGZmZmZmZmZmODE4NWYxMmIgaW4gX19ycGNfZXhlY3V0ZSAodGFzaz0weGZmZmY4
ODAwNWE4NTM4MDApDQo+ID4gwqDCoMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0L25l
dC9zdW5ycGMvc2NoZWQuYzo3NzUNCj4gPiAjNMKgwqAweGZmZmZmZmZmODE4NjE3YjQgaW4gcnBj
X2V4ZWN1dGUgKHRhc2s9MHhmZmZmODgwMDVhODUzODAwKQ0KPiA+IMKgwqDCoMKgYXQgL2hvbWUv
Z3JlZW4vYmsvbGludXgtdGVzdC9uZXQvc3VucnBjL3NjaGVkLmM6ODQzDQo+ID4gIzXCoMKgMHhm
ZmZmZmZmZjgxODUzOWI5IGluIHJwY19ydW5fdGFzaw0KPiA+ICh0YXNrX3NldHVwX2RhdGE9MHhm
ZmZmODgwMGE3MjUzYTUwKQ0KPiA+IMKgwqDCoMKgYXQgL2hvbWUvZ3JlZW4vYmsvbGludXgtdGVz
dC9uZXQvc3VucnBjL2NsbnQuYzoxMDUyDQo+ID4gIzbCoMKgMHhmZmZmZmZmZjgxM2E3NWUzIGlu
IG5mczRfY2FsbF9zeW5jX3NlcXVlbmNlIChjbG50PTxvcHRpbWl6ZWQNCj4gPiBvdXQ+LMKgDQo+
ID4gwqDCoMKgwqBzZXJ2ZXI9PG9wdGltaXplZCBvdXQ+LCBtc2c9PG9wdGltaXplZCBvdXQ+LCBh
cmdzPTxvcHRpbWl6ZWQNCj4gPiBvdXQ+LMKgDQo+ID4gwqDCoMKgwqByZXM9PG9wdGltaXplZCBv
dXQ+KSBhdCAvaG9tZS9ncmVlbi9iay9saW51eC0NCj4gPiB0ZXN0L2ZzL25mcy9uZnM0cHJvYy5j
OjEwNTENCj4gPiAjN8KgwqAweGZmZmZmZmZmODEzYjQ2NDUgaW4gbmZzNF9jYWxsX3N5bmMgKGNh
Y2hlX3JlcGx5PTxvcHRpbWl6ZWQNCj4gPiBvdXQ+LMKgDQo+ID4gwqDCoMKgwqByZXM9PG9wdGlt
aXplZCBvdXQ+LCBhcmdzPTxvcHRpbWl6ZWQgb3V0PiwgbXNnPTxvcHRpbWl6ZWQNCj4gPiBvdXQ+
LMKgDQo+ID4gwqDCoMKgwqBzZXJ2ZXI9PG9wdGltaXplZCBvdXQ+LCBjbG50PTxvcHRpbWl6ZWQg
b3V0PikNCj4gPiDCoMKgwqDCoGF0IC9ob21lL2dyZWVuL2JrL2xpbnV4LXRlc3QvZnMvbmZzL25m
czRwcm9jLmM6MTA2OQ0KPiA+ICM4wqDCoF9uZnM0X2RvX3NldGF0dHIgKHN0YXRlPTxvcHRpbWl6
ZWQgb3V0PiwgY3JlZD08b3B0aW1pemVkIG91dD4swqANCj4gPiDCoMKgwqDCoHJlcz08b3B0aW1p
emVkIG91dD4sIGFyZz08b3B0aW1pemVkIG91dD4sIGlub2RlPTxvcHRpbWl6ZWQNCj4gPiBvdXQ+
KQ0KPiA+IC0tLVR5cGUgPHJldHVybj4gdG8gY29udGludWUsIG9yIHEgPHJldHVybj4gdG8gcXVp
dC0tLQ0KPiA+IMKgwqDCoMKgYXQgL2hvbWUvZ3JlZW4vYmsvbGludXgtdGVzdC9mcy9uZnMvbmZz
NHByb2MuYzoyOTE2DQo+ID4gIznCoMKgbmZzNF9kb19zZXRhdHRyIChpbm9kZT0weGZmZmY4ODAw
NzliMTUyYTgsIGNyZWQ9PG9wdGltaXplZA0KPiA+IG91dD4swqANCj4gPiDCoMKgwqDCoGZhdHRy
PTxvcHRpbWl6ZWQgb3V0Piwgc2F0dHI9PG9wdGltaXplZCBvdXQ+LA0KPiA+IHN0YXRlPTB4ZmZm
Zjg4MDA2MDU4OGUwMCzCoA0KPiA+IMKgwqDCoMKgaWxhYmVsPTxvcHRpbWl6ZWQgb3V0Piwgb2xh
YmVsPTB4MCA8aXJxX3N0YWNrX3VuaW9uPikNCj4gPiDCoMKgwqDCoGF0IC9ob21lL2dyZWVuL2Jr
L2xpbnV4LXRlc3QvZnMvbmZzL25mczRwcm9jLmM6Mjk1NQ0KPiA+ICMxMCAweGZmZmZmZmZmODEz
YjRhMTYgaW4gbmZzNF9wcm9jX3NldGF0dHIgKGRlbnRyeT08b3B0aW1pemVkDQo+ID4gb3V0PizC
oA0KPiA+IMKgwqDCoMKgZmF0dHI9MHhmZmZmODgwMGE3MjUzYjgwLCBzYXR0cj0weGZmZmY4ODAw
YTcyNTNiNDgpDQo+ID4gwqDCoMKgwqBhdCAvaG9tZS9ncmVlbi9iay9saW51eC10ZXN0L2ZzL25m
cy9uZnM0cHJvYy5jOjM2ODQNCj4gPiAjMTEgMHhmZmZmZmZmZjgxMzhmMWNiIGluIG5mc19zZXRh
dHRyIChkZW50cnk9MHhmZmZmODgwMDc0MGMxMDAwLMKgDQo+ID4gwqANCj4gDQo+IENvb2whIERv
ZXMgdGhlIGZvbGxvd2luZyBoZWxwPw0KDQpHcnJyLi4uIFRoZXJlIGlzIGFub3RoZXIgYnVnIHRo
ZXJlLi4uDQoNCjg8LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0NCkZyb20gY2E1ZmQyZTA1NWNjMGZkNDlkOWE1ZDQ0ZjRkMDFj
OWNhMDFmYWQ5OCBNb24gU2VwIDE3IDAwOjAwOjAwIDIwMDENCkZyb206IFRyb25kIE15a2xlYnVz
dCA8dHJvbmQubXlrbGVidXN0QHByaW1hcnlkYXRhLmNvbT4NCkRhdGU6IFNhdCwgMTcgU2VwIDIw
MTYgMTc6Mzc6NDcgLTA0MDANClN1YmplY3Q6IFtQQVRDSF0gTkZTdjQ6IG5mczRfY29weV9kZWxl
Z2F0aW9uX3N0YXRlaWQoKSBtdXN0IGZhaWwgaWYgdGhlDQogZGVsZWdhdGlvbiBpcyBpbnZhbGlk
DQoNCldlIG11c3Qgbm90IGFsbG93IHRoZSB1c2Ugb2YgZGVsZWdhdGlvbnMgdGhhdCBoYXZlIGJl
ZW4gcmV2b2tlZCBvciBhcmUNCmJlaW5nIHJldHVybmVkLg0KDQpTaWduZWQtb2ZmLWJ5OiBUcm9u
ZCBNeWtsZWJ1c3QgPHRyb25kLm15a2xlYnVzdEBwcmltYXJ5ZGF0YS5jb20+DQotLS0NCiBmcy9u
ZnMvZGVsZWdhdGlvbi5jIHwgMTcgKysrKysrKysrKysrKy0tLS0NCiAxIGZpbGUgY2hhbmdlZCwg
MTMgaW5zZXJ0aW9ucygrKSwgNCBkZWxldGlvbnMoLSkNCg0KZGlmZiAtLWdpdCBhL2ZzL25mcy9k
ZWxlZ2F0aW9uLmMgYi9mcy9uZnMvZGVsZWdhdGlvbi5jDQppbmRleCA2MmIyMjE1YTMwYjkuLjAw
NzM4NDhiNWFkMyAxMDA2NDQNCi0tLSBhL2ZzL25mcy9kZWxlZ2F0aW9uLmMNCisrKyBiL2ZzL25m
cy9kZWxlZ2F0aW9uLmMNCkBAIC00MSw2ICs0MSwxNyBAQCB2b2lkIG5mc19tYXJrX2RlbGVnYXRp
b25fcmVmZXJlbmNlZChzdHJ1Y3QgbmZzX2RlbGVnYXRpb24gKmRlbGVnYXRpb24pDQogCXNldF9i
aXQoTkZTX0RFTEVHQVRJT05fUkVGRVJFTkNFRCwgJmRlbGVnYXRpb24tPmZsYWdzKTsNCiB9DQog
DQorc3RhdGljIGJvb2wNCituZnM0X2lzX3ZhbGlkX2RlbGVnYXRpb24oY29uc3Qgc3RydWN0IG5m
c19kZWxlZ2F0aW9uICpkZWxlZ2F0aW9uLA0KKwkJZm1vZGVfdCBmbGFncykNCit7DQorCWlmIChk
ZWxlZ2F0aW9uICE9IE5VTEwgJiYgKGRlbGVnYXRpb24tPnR5cGUgJiBmbGFncykgPT0gZmxhZ3Mg
JiYNCisJICAgICF0ZXN0X2JpdChORlNfREVMRUdBVElPTl9SRVZPS0VELCAmZGVsZWdhdGlvbi0+
ZmxhZ3MpICYmDQorCSAgICAhdGVzdF9iaXQoTkZTX0RFTEVHQVRJT05fUkVUVVJOSU5HLCAmZGVs
ZWdhdGlvbi0+ZmxhZ3MpKQ0KKwkJcmV0dXJuIHRydWU7DQorCXJldHVybiBmYWxzZTsNCit9DQor
DQogc3RhdGljIGludA0KIG5mczRfZG9fY2hlY2tfZGVsZWdhdGlvbihzdHJ1Y3QgaW5vZGUgKmlu
b2RlLCBmbW9kZV90IGZsYWdzLCBib29sIG1hcmspDQogew0KQEAgLTUwLDkgKzYxLDcgQEAgbmZz
NF9kb19jaGVja19kZWxlZ2F0aW9uKHN0cnVjdCBpbm9kZSAqaW5vZGUsIGZtb2RlX3QgZmxhZ3Ms
IGJvb2wgbWFyaykNCiAJZmxhZ3MgJj0gRk1PREVfUkVBRHxGTU9ERV9XUklURTsNCiAJcmN1X3Jl
YWRfbG9jaygpOw0KIAlkZWxlZ2F0aW9uID0gcmN1X2RlcmVmZXJlbmNlKE5GU19JKGlub2RlKS0+
ZGVsZWdhdGlvbik7DQotCWlmIChkZWxlZ2F0aW9uICE9IE5VTEwgJiYgKGRlbGVnYXRpb24tPnR5
cGUgJiBmbGFncykgPT0gZmxhZ3MgJiYNCi0JICAgICF0ZXN0X2JpdChORlNfREVMRUdBVElPTl9S
RVZPS0VELCAmZGVsZWdhdGlvbi0+ZmxhZ3MpICYmDQotCSAgICAhdGVzdF9iaXQoTkZTX0RFTEVH
QVRJT05fUkVUVVJOSU5HLCAmZGVsZWdhdGlvbi0+ZmxhZ3MpKSB7DQorCWlmIChuZnM0X2lzX3Zh
bGlkX2RlbGVnYXRpb24oZGVsZWdhdGlvbiwgZmxhZ3MpKSB7DQogCQlpZiAobWFyaykNCiAJCQlu
ZnNfbWFya19kZWxlZ2F0aW9uX3JlZmVyZW5jZWQoZGVsZWdhdGlvbik7DQogCQlyZXQgPSAxOw0K
QEAgLTEwNTQsNyArMTA2Myw3IEBAIGJvb2wgbmZzNF9jb3B5X2RlbGVnYXRpb25fc3RhdGVpZChz
dHJ1Y3QgaW5vZGUgKmlub2RlLCBmbW9kZV90IGZsYWdzLA0KIAlmbGFncyAmPSBGTU9ERV9SRUFE
fEZNT0RFX1dSSVRFOw0KIAlyY3VfcmVhZF9sb2NrKCk7DQogCWRlbGVnYXRpb24gPSByY3VfZGVy
ZWZlcmVuY2UobmZzaS0+ZGVsZWdhdGlvbik7DQotCXJldCA9IChkZWxlZ2F0aW9uICE9IE5VTEwg
JiYgKGRlbGVnYXRpb24tPnR5cGUgJiBmbGFncykgPT0gZmxhZ3MpOw0KKwlyZXQgPSBuZnM0X2lz
X3ZhbGlkX2RlbGVnYXRpb24oZGVsZWdhdGlvbiwgZmxhZ3MpOw0KIAlpZiAocmV0KSB7DQogCQlu
ZnM0X3N0YXRlaWRfY29weShkc3QsICZkZWxlZ2F0aW9uLT5zdGF0ZWlkKTsNCiAJCW5mc19tYXJr
X2RlbGVnYXRpb25fcmVmZXJlbmNlZChkZWxlZ2F0aW9uKTsNCi0tIA0KMi43LjQ=

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Oleg Drokin]

On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:

> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>> 
>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>> 
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>> wrote:
>>>>> 
>>>>> 
>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>> to use
>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>> so
>>>>>> we
>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>> 
>>>>>> While we already do this for open and lock state, we have not
>>>>>> been doing
>>>>>> so for all the delegations.
>>>>>> 
>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>> too
>>>>>> v3: Now with added lock revoke fixes and
>>>>>> close/delegreturn/locku fixes
>>>>>> v4: Close a bunch of corner cases
>>>>>> v5: Report revoked delegations as invalid in
>>>>>> nfs_have_delegation()
>>>>>>  Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>  Fixes for other looping behaviour
>>>>> 
>>>>> This time around the loop seems to be more tight,
>>>>> in userspace process:
>>>>> 
>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>> ffff8800a73ce000
>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=0000
>>>>> highest_used=4294967295 max_slots=31
>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=0001
>>>>> highest_used=0
>>>>> slotid=0
>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=0 seqid=14013800
>>>>> [ 9197.256582] encode_sequence: sessionid=1474126170:1:2:0
>>>>> seqid=14013800 slotid=0 max_slotid=0 cache_this=1
>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=0001
>>>>> highest_used=0
>>>>> max_slots=31
>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=0003
>>>>> highest_used=1
>>>>> slotid=1
>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot 
>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>> 4294967295
>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>> 
>>>> What operation is the userspace process hanging on? Do you have a
>>>> stack trace for it?
>>> 
>>> seems to be open_create->truncate->ssetattr coming from:
>>> cp /bin/sleep /mnt/nfs2/racer/12
>>> 
>>> (gdb) bt
>>> #0  nfs41_setup_sequence (session=0xffff88005a853800,
>>> args=0xffff8800a7253b80, 
>>>     res=0xffff8800a7253b48, task=0xffff8800b0eb0f00)
>>>     at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=<optimized
>>> out>, 
>>>     calldata=0xffff8800a7253b80)
>>>     at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=<optimized out>)
>>>     at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>> #3  0xffffffff8185f12b in __rpc_execute (task=0xffff88005a853800)
>>>     at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>> #4  0xffffffff818617b4 in rpc_execute (task=0xffff88005a853800)
>>>     at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>> #5  0xffffffff818539b9 in rpc_run_task
>>> (task_setup_data=0xffff8800a7253a50)
>>>     at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized
>>> out>, 
>>>     server=<optimized out>, msg=<optimized out>, args=<optimized
>>> out>, 
>>>     res=<optimized out>) at /home/green/bk/linux-
>>> test/fs/nfs/nfs4proc.c:1051
>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=<optimized
>>> out>, 
>>>     res=<optimized out>, args=<optimized out>, msg=<optimized
>>> out>, 
>>>     server=<optimized out>, clnt=<optimized out>)
>>>     at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>> #8  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
>>>     res=<optimized out>, arg=<optimized out>, inode=<optimized
>>> out>)
>>> ---Type <return> to continue, or q <return> to quit---
>>>     at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>> #9  nfs4_do_setattr (inode=0xffff880079b152a8, cred=<optimized
>>> out>, 
>>>     fattr=<optimized out>, sattr=<optimized out>,
>>> state=0xffff880060588e00, 
>>>     ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
>>>     at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=<optimized
>>> out>, 
>>>     fattr=0xffff8800a7253b80, sattr=0xffff8800a7253b48)
>>>     at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800740c1000, 
>>>  
>> 
>> Cool! Does the following help?
> 
> Grrr... There is another bug there…

Is this in addition to the previous patch or instead of?

> 
> 8<-----------------------------------------------------------------
>> From ca5fd2e055cc0fd49d9a5d44f4d01c9ca01fad98 Mon Sep 17 00:00:00 2001
> From: Trond Myklebust <trond.myklebust@primarydata.com>
> Date: Sat, 17 Sep 2016 17:37:47 -0400
> Subject: [PATCH] NFSv4: nfs4_copy_delegation_stateid() must fail if the
> delegation is invalid
> 
> We must not allow the use of delegations that have been revoked or are
> being returned.
> 
> Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
> ---
> fs/nfs/delegation.c | 17 +++++++++++++----
> 1 file changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> index 62b2215a30b9..0073848b5ad3 100644
> --- a/fs/nfs/delegation.c
> +++ b/fs/nfs/delegation.c
> @@ -41,6 +41,17 @@ void nfs_mark_delegation_referenced(struct nfs_delegation *delegation)
> 	set_bit(NFS_DELEGATION_REFERENCED, &delegation->flags);
> }
> 
> +static bool
> +nfs4_is_valid_delegation(const struct nfs_delegation *delegation,
> +		fmode_t flags)
> +{
> +	if (delegation != NULL && (delegation->type & flags) == flags &&
> +	    !test_bit(NFS_DELEGATION_REVOKED, &delegation->flags) &&
> +	    !test_bit(NFS_DELEGATION_RETURNING, &delegation->flags))
> +		return true;
> +	return false;
> +}
> +
> static int
> nfs4_do_check_delegation(struct inode *inode, fmode_t flags, bool mark)
> {
> @@ -50,9 +61,7 @@ nfs4_do_check_delegation(struct inode *inode, fmode_t flags, bool mark)
> 	flags &= FMODE_READ|FMODE_WRITE;
> 	rcu_read_lock();
> 	delegation = rcu_dereference(NFS_I(inode)->delegation);
> -	if (delegation != NULL && (delegation->type & flags) == flags &&
> -	    !test_bit(NFS_DELEGATION_REVOKED, &delegation->flags) &&
> -	    !test_bit(NFS_DELEGATION_RETURNING, &delegation->flags)) {
> +	if (nfs4_is_valid_delegation(delegation, flags)) {
> 		if (mark)
> 			nfs_mark_delegation_referenced(delegation);
> 		ret = 1;
> @@ -1054,7 +1063,7 @@ bool nfs4_copy_delegation_stateid(struct inode *inode, fmode_t flags,
> 	flags &= FMODE_READ|FMODE_WRITE;
> 	rcu_read_lock();
> 	delegation = rcu_dereference(nfsi->delegation);
> -	ret = (delegation != NULL && (delegation->type & flags) == flags);
> +	ret = nfs4_is_valid_delegation(delegation, flags);
> 	if (ret) {
> 		nfs4_stateid_copy(dst, &delegation->stateid);
> 		nfs_mark_delegation_referenced(delegation);
> -- 
> 2.7.4

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Trond Myklebust]

> On Sep 17, 2016, at 17:55, Oleg Drokin <green@linuxhacker.ru> wrote:
>=20
>=20
> On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:
>=20
>> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>>>=20
>>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>>>=20
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>>> wrote:
>>>>>>=20
>>>>>>=20
>>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>>> to use
>>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>>> so
>>>>>>> we
>>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>>>=20
>>>>>>> While we already do this for open and lock state, we have not
>>>>>>> been doing
>>>>>>> so for all the delegations.
>>>>>>>=20
>>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>>> too
>>>>>>> v3: Now with added lock revoke fixes and
>>>>>>> close/delegreturn/locku fixes
>>>>>>> v4: Close a bunch of corner cases
>>>>>>> v5: Report revoked delegations as invalid in
>>>>>>> nfs_have_delegation()
>>>>>>> Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>> Fixes for other looping behaviour
>>>>>>=20
>>>>>> This time around the loop seems to be more tight,
>>>>>> in userspace process:
>>>>>>=20
>>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>>> ffff8800a73ce000
>>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=3D0000
>>>>>> highest_used=3D4294967295 max_slots=3D31
>>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=3D0001
>>>>>> highest_used=3D0
>>>>>> slotid=3D0
>>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=3D0 seqid=3D14013800
>>>>>> [ 9197.256582] encode_sequence: sessionid=3D1474126170:1:2:0
>>>>>> seqid=3D14013800 slotid=3D0 max_slotid=3D0 cache_this=3D1
>>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=3D0001
>>>>>> highest_used=3D0
>>>>>> max_slots=3D31
>>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=3D0003
>>>>>> highest_used=3D1
>>>>>> slotid=3D1
>>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot=20
>>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>>> 4294967295
>>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>>>=20
>>>>> What operation is the userspace process hanging on? Do you have a
>>>>> stack trace for it?
>>>>=20
>>>> seems to be open_create->truncate->ssetattr coming from:
>>>> cp /bin/sleep /mnt/nfs2/racer/12
>>>>=20
>>>> (gdb) bt
>>>> #0  nfs41_setup_sequence (session=3D0xffff88005a853800,
>>>> args=3D0xffff8800a7253b80,=20
>>>>    res=3D0xffff8800a7253b48, task=3D0xffff8800b0eb0f00)
>>>>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=3D<optimized
>>>> out>,=20
>>>>    calldata=3D0xffff8800a7253b80)
>>>>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=3D<optimized out>)
>>>>    at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>>> #3  0xffffffff8185f12b in __rpc_execute (task=3D0xffff88005a853800)
>>>>    at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>>> #4  0xffffffff818617b4 in rpc_execute (task=3D0xffff88005a853800)
>>>>    at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>>> #5  0xffffffff818539b9 in rpc_run_task
>>>> (task_setup_data=3D0xffff8800a7253a50)
>>>>    at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=3D<optimized
>>>> out>,=20
>>>>    server=3D<optimized out>, msg=3D<optimized out>, args=3D<optimized
>>>> out>,=20
>>>>    res=3D<optimized out>) at /home/green/bk/linux-
>>>> test/fs/nfs/nfs4proc.c:1051
>>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=3D<optimized
>>>> out>,=20
>>>>    res=3D<optimized out>, args=3D<optimized out>, msg=3D<optimized
>>>> out>,=20
>>>>    server=3D<optimized out>, clnt=3D<optimized out>)
>>>>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>>> #8  _nfs4_do_setattr (state=3D<optimized out>, cred=3D<optimized out>,=
=20
>>>>    res=3D<optimized out>, arg=3D<optimized out>, inode=3D<optimized
>>>> out>)
>>>> ---Type <return> to continue, or q <return> to quit---
>>>>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>>> #9  nfs4_do_setattr (inode=3D0xffff880079b152a8, cred=3D<optimized
>>>> out>,=20
>>>>    fattr=3D<optimized out>, sattr=3D<optimized out>,
>>>> state=3D0xffff880060588e00,=20
>>>>    ilabel=3D<optimized out>, olabel=3D0x0 <irq_stack_union>)
>>>>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=3D<optimized
>>>> out>,=20
>>>>    fattr=3D0xffff8800a7253b80, sattr=3D0xffff8800a7253b48)
>>>>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=3D0xffff8800740c1000,=20
>>>>=20
>>>=20
>>> Cool! Does the following help?
>>=20
>> Grrr... There is another bug there=85
>=20
> Is this in addition to the previous patch or instead of?

It can apply on top of it. The two patches fix different bugs.

>=20
>>=20
>> 8<-----------------------------------------------------------------
>>> From ca5fd2e055cc0fd49d9a5d44f4d01c9ca01fad98 Mon Sep 17 00:00:00 2001
>> From: Trond Myklebust <trond.myklebust@primarydata.com>
>> Date: Sat, 17 Sep 2016 17:37:47 -0400
>> Subject: [PATCH] NFSv4: nfs4_copy_delegation_stateid() must fail if the
>> delegation is invalid
>>=20
>> We must not allow the use of delegations that have been revoked or are
>> being returned.
>>=20
>> Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
>> ---
>> fs/nfs/delegation.c | 17 +++++++++++++----
>> 1 file changed, 13 insertions(+), 4 deletions(-)
>>=20
>> diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
>> index 62b2215a30b9..0073848b5ad3 100644
>> --- a/fs/nfs/delegation.c
>> +++ b/fs/nfs/delegation.c
>> @@ -41,6 +41,17 @@ void nfs_mark_delegation_referenced(struct nfs_delega=
tion *delegation)
>> =09set_bit(NFS_DELEGATION_REFERENCED, &delegation->flags);
>> }
>>=20
>> +static bool
>> +nfs4_is_valid_delegation(const struct nfs_delegation *delegation,
>> +=09=09fmode_t flags)
>> +{
>> +=09if (delegation !=3D NULL && (delegation->type & flags) =3D=3D flags =
&&
>> +=09    !test_bit(NFS_DELEGATION_REVOKED, &delegation->flags) &&
>> +=09    !test_bit(NFS_DELEGATION_RETURNING, &delegation->flags))
>> +=09=09return true;
>> +=09return false;
>> +}
>> +
>> static int
>> nfs4_do_check_delegation(struct inode *inode, fmode_t flags, bool mark)
>> {
>> @@ -50,9 +61,7 @@ nfs4_do_check_delegation(struct inode *inode, fmode_t =
flags, bool mark)
>> =09flags &=3D FMODE_READ|FMODE_WRITE;
>> =09rcu_read_lock();
>> =09delegation =3D rcu_dereference(NFS_I(inode)->delegation);
>> -=09if (delegation !=3D NULL && (delegation->type & flags) =3D=3D flags =
&&
>> -=09    !test_bit(NFS_DELEGATION_REVOKED, &delegation->flags) &&
>> -=09    !test_bit(NFS_DELEGATION_RETURNING, &delegation->flags)) {
>> +=09if (nfs4_is_valid_delegation(delegation, flags)) {
>> =09=09if (mark)
>> =09=09=09nfs_mark_delegation_referenced(delegation);
>> =09=09ret =3D 1;
>> @@ -1054,7 +1063,7 @@ bool nfs4_copy_delegation_stateid(struct inode *in=
ode, fmode_t flags,
>> =09flags &=3D FMODE_READ|FMODE_WRITE;
>> =09rcu_read_lock();
>> =09delegation =3D rcu_dereference(nfsi->delegation);
>> -=09ret =3D (delegation !=3D NULL && (delegation->type & flags) =3D=3D f=
lags);
>> +=09ret =3D nfs4_is_valid_delegation(delegation, flags);
>> =09if (ret) {
>> =09=09nfs4_stateid_copy(dst, &delegation->stateid);
>> =09=09nfs_mark_delegation_referenced(delegation);
>> --=20
>> 2.7.4

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Oleg Drokin]

On Sep 17, 2016, at 5:59 PM, Trond Myklebust wrote:

> 
>> On Sep 17, 2016, at 17:55, Oleg Drokin <green@linuxhacker.ru> wrote:
>> 
>> 
>> On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:
>> 
>>> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>>>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>>>> 
>>>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>>>> to use
>>>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>>>> so
>>>>>>>> we
>>>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>>>> 
>>>>>>>> While we already do this for open and lock state, we have not
>>>>>>>> been doing
>>>>>>>> so for all the delegations.
>>>>>>>> 
>>>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>>>> too
>>>>>>>> v3: Now with added lock revoke fixes and
>>>>>>>> close/delegreturn/locku fixes
>>>>>>>> v4: Close a bunch of corner cases
>>>>>>>> v5: Report revoked delegations as invalid in
>>>>>>>> nfs_have_delegation()
>>>>>>>> Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>>> Fixes for other looping behaviour
>>>>>>> 
>>>>>>> This time around the loop seems to be more tight,
>>>>>>> in userspace process:
>>>>>>> 
>>>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>>>> ffff8800a73ce000
>>>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=0000
>>>>>>> highest_used=4294967295 max_slots=31
>>>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=0001
>>>>>>> highest_used=0
>>>>>>> slotid=0
>>>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=0 seqid=14013800
>>>>>>> [ 9197.256582] encode_sequence: sessionid=1474126170:1:2:0
>>>>>>> seqid=14013800 slotid=0 max_slotid=0 cache_this=1
>>>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=0001
>>>>>>> highest_used=0
>>>>>>> max_slots=31
>>>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=0003
>>>>>>> highest_used=1
>>>>>>> slotid=1
>>>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot 
>>>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>>>> 4294967295
>>>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>>>> 
>>>>>> What operation is the userspace process hanging on? Do you have a
>>>>>> stack trace for it?
>>>>> 
>>>>> seems to be open_create->truncate->ssetattr coming from:
>>>>> cp /bin/sleep /mnt/nfs2/racer/12
>>>>> 
>>>>> (gdb) bt
>>>>> #0  nfs41_setup_sequence (session=0xffff88005a853800,
>>>>> args=0xffff8800a7253b80, 
>>>>>   res=0xffff8800a7253b48, task=0xffff8800b0eb0f00)
>>>>>   at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=<optimized
>>>>> out>, 
>>>>>   calldata=0xffff8800a7253b80)
>>>>>   at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=<optimized out>)
>>>>>   at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>>>> #3  0xffffffff8185f12b in __rpc_execute (task=0xffff88005a853800)
>>>>>   at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>>>> #4  0xffffffff818617b4 in rpc_execute (task=0xffff88005a853800)
>>>>>   at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>>>> #5  0xffffffff818539b9 in rpc_run_task
>>>>> (task_setup_data=0xffff8800a7253a50)
>>>>>   at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized
>>>>> out>, 
>>>>>   server=<optimized out>, msg=<optimized out>, args=<optimized
>>>>> out>, 
>>>>>   res=<optimized out>) at /home/green/bk/linux-
>>>>> test/fs/nfs/nfs4proc.c:1051
>>>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=<optimized
>>>>> out>, 
>>>>>   res=<optimized out>, args=<optimized out>, msg=<optimized
>>>>> out>, 
>>>>>   server=<optimized out>, clnt=<optimized out>)
>>>>>   at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>>>> #8  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
>>>>>   res=<optimized out>, arg=<optimized out>, inode=<optimized
>>>>> out>)
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>   at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>>>> #9  nfs4_do_setattr (inode=0xffff880079b152a8, cred=<optimized
>>>>> out>, 
>>>>>   fattr=<optimized out>, sattr=<optimized out>,
>>>>> state=0xffff880060588e00, 
>>>>>   ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
>>>>>   at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=<optimized
>>>>> out>, 
>>>>>   fattr=0xffff8800a7253b80, sattr=0xffff8800a7253b48)
>>>>>   at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800740c1000, 
>>>>> 
>>>> 
>>>> Cool! Does the following help?
>>> 
>>> Grrr... There is another bug there…
>> 
>> Is this in addition to the previous patch or instead of?
> 
> It can apply on top of it. The two patches fix different bugs.

Ok, it does not seem to have helped:
[11397.552805] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
[11397.552806] --> nfs41_setup_sequence
[11397.552807] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[11397.552808] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[11397.552809] <-- nfs41_setup_sequence slotid=0 seqid=8829382
[11397.552817] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829382 slotid=0 max_slotid=0 cache_this=1
[11397.553032] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[11397.553033] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[11397.553034] nfs4_free_slot: slotid 1 highest_used_slotid 0
[11397.553035] nfs41_sequence_process: Error 0 free the slot 
[11397.553036] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[11397.553056] --> nfs_put_client({2})
[11397.553062] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
[11397.553063] --> nfs41_setup_sequence
[11397.553064] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[11397.553065] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[11397.553066] <-- nfs41_setup_sequence slotid=0 seqid=8829383
[11397.553074] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829383 slotid=0 max_slotid=0 cache_this=1
[11397.553290] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[11397.553291] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[11397.553292] nfs4_free_slot: slotid 1 highest_used_slotid 0
[11397.553293] nfs41_sequence_process: Error 0 free the slot 
[11397.553294] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[11397.553316] --> nfs_put_client({2})
[11397.553322] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
[11397.553322] --> nfs41_setup_sequence
[11397.553324] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[11397.553324] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[11397.553325] <-- nfs41_setup_sequence slotid=0 seqid=8829384
[11397.553333] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829384 slotid=0 max_slotid=0 cache_this=1
[11397.553575] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[11397.553576] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[11397.553577] nfs4_free_slot: slotid 1 highest_used_slotid 0
[11397.553578] nfs41_sequence_process: Error 0 free the slot 
[11397.553580] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[11397.553602] --> nfs_put_client({2})
[11397.553607] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
[11397.553608] --> nfs41_setup_sequence
[11397.553609] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[11397.553610] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[11397.553611] <-- nfs41_setup_sequence slotid=0 seqid=8829385
[11397.553619] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829385 slotid=0 max_slotid=0 cache_this=1
[11397.560057] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[11397.560059] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[11397.560061] nfs4_free_slot: slotid 1 highest_used_slotid 0
[11397.560062] nfs41_sequence_process: Error 0 free the slot 
[11397.560064] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[11397.560094] --> nfs_put_client({2})
[11397.560103] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
[11397.560103] --> nfs41_setup_sequence
[11397.560104] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[11397.560105] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[11397.560106] <-- nfs41_setup_sequence slotid=0 seqid=8829386
[11397.560120] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829386 slotid=0 max_slotid=0 cache_this=1
[11397.560388] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[11397.560389] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[11397.560391] nfs4_free_slot: slotid 1 highest_used_slotid 0
[11397.560391] nfs41_sequence_process: Error 0 free the slot 
[11397.560393] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[11397.560415] --> nfs_put_client({2})


bt looks about the same:
#0  xprt_get (xprt=0x0 <irq_stack_union>)
    at /home/green/bk/linux-test/net/sunrpc/xprt.c:1435
#1  0xffffffff8186196f in rpc_init_task (task_setup_data=<optimized out>, 
    task=<optimized out>) at /home/green/bk/linux-test/net/sunrpc/sched.c:936
#2  rpc_new_task (setup_data=0x0 <irq_stack_union>)
    at /home/green/bk/linux-test/net/sunrpc/sched.c:971
#3  0xffffffff818538c8 in rpc_run_task (task_setup_data=0xffff88007236ba50)
    at /home/green/bk/linux-test/net/sunrpc/clnt.c:1041
#4  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized out>, 
    server=<optimized out>, msg=<optimized out>, args=<optimized out>, 
    res=<optimized out>) at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1055
#5  0xffffffff813b4675 in nfs4_call_sync (cache_reply=<optimized out>, 
    res=<optimized out>, args=<optimized out>, msg=<optimized out>, 
    server=<optimized out>, clnt=<optimized out>)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1073
#6  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
    res=<optimized out>, arg=<optimized out>, inode=<optimized out>)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2920
#7  nfs4_do_setattr (inode=0xffff8800640f82a8, cred=<optimized out>, 
    fattr=<optimized out>, sattr=<optimized out>, state=0xffff88005a95be00, 
    ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2959
#8  0xffffffff813b4a46 in nfs4_proc_setattr (dentry=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    fattr=0xf8 <irq_stack_union+248>, sattr=0x1 <irq_stack_union+1>)
    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3688
#9  0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800119c0000, 
    attr=0xffff88007236bce8) at /home/green/bk/linux-test/fs/nfs/inode.c:556
#10 0xffffffff81298d6b in notify_change (dentry=0xffff8800119c0000, 
    attr=0xffff88007236bce8, delegated_inode=<optimized out>)
    at /home/green/bk/linux-test/fs/attr.c:285
#11 0xffffffff812734e3 in do_truncate (dentry=0x0 <irq_stack_union>, 
    length=<optimized out>, time_attrs=<optimized out>, filp=<optimized out>)
    at /home/green/bk/linux-test/fs/open.c:63
#12 0xffffffff81287459 in handle_truncate (filp=<optimized out>)
    at /home/green/bk/linux-test/fs/namei.c:2960
#13 do_last (opened=<optimized out>, op=<optimized out>, file=<optimized out>, 
    nd=<optimized out>) at /home/green/bk/linux-test/fs/namei.c:3383
#14 path_openat (nd=0xffff88007236bde0, op=<optimized out>, 
    flags=<optimized out>) at /home/green/bk/linux-test/fs/namei.c:3497
#15 0xffffffff81288b01 in do_filp_open (dfd=<optimized out>, 
    pathname=<optimized out>, op=0xffff88007236bef4)
    at /home/green/bk/linux-test/fs/namei.c:3532
#16 0xffffffff81274ab0 in do_sys_open (dfd=-100, filename=<optimized out>, 
    flags=<optimized out>, mode=<optimized out>)
    at /home/green/bk/linux-test/fs/open.c:1036
#17 0xffffffff81274bbe in SYSC_open (mode=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    flags=<optimized out>, filename=<optimized out>)
    at /home/green/bk/linux-test/fs/open.c:1054
#18 SyS_open (filename=<optimized out>, flags=<optimized out>, 
    mode=<optimized out>) at /home/green/bk/linux-test/fs/open.c:1049
#19 0xffffffff8189f03c in entry_SYSCALL_64_fastpath ()

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Oleg Drokin]

On Sep 17, 2016, at 9:19 PM, Oleg Drokin wrote:

> 
> On Sep 17, 2016, at 5:59 PM, Trond Myklebust wrote:
> 
>> 
>>> On Sep 17, 2016, at 17:55, Oleg Drokin <green@linuxhacker.ru> wrote:
>>> 
>>> 
>>> On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:
>>> 
>>>> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>>>>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>>>>> 
>>>>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>>>>> to use
>>>>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>>>>> so
>>>>>>>>> we
>>>>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>>>>> 
>>>>>>>>> While we already do this for open and lock state, we have not
>>>>>>>>> been doing
>>>>>>>>> so for all the delegations.
>>>>>>>>> 
>>>>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>>>>> too
>>>>>>>>> v3: Now with added lock revoke fixes and
>>>>>>>>> close/delegreturn/locku fixes
>>>>>>>>> v4: Close a bunch of corner cases
>>>>>>>>> v5: Report revoked delegations as invalid in
>>>>>>>>> nfs_have_delegation()
>>>>>>>>> Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>>>> Fixes for other looping behaviour
>>>>>>>> 
>>>>>>>> This time around the loop seems to be more tight,
>>>>>>>> in userspace process:
>>>>>>>> 
>>>>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>>>>> ffff8800a73ce000
>>>>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=0000
>>>>>>>> highest_used=4294967295 max_slots=31
>>>>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=0001
>>>>>>>> highest_used=0
>>>>>>>> slotid=0
>>>>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=0 seqid=14013800
>>>>>>>> [ 9197.256582] encode_sequence: sessionid=1474126170:1:2:0
>>>>>>>> seqid=14013800 slotid=0 max_slotid=0 cache_this=1
>>>>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=0001
>>>>>>>> highest_used=0
>>>>>>>> max_slots=31
>>>>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=0003
>>>>>>>> highest_used=1
>>>>>>>> slotid=1
>>>>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot 
>>>>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>>>>> 4294967295
>>>>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>>>>> 
>>>>>>> What operation is the userspace process hanging on? Do you have a
>>>>>>> stack trace for it?
>>>>>> 
>>>>>> seems to be open_create->truncate->ssetattr coming from:
>>>>>> cp /bin/sleep /mnt/nfs2/racer/12
>>>>>> 
>>>>>> (gdb) bt
>>>>>> #0  nfs41_setup_sequence (session=0xffff88005a853800,
>>>>>> args=0xffff8800a7253b80, 
>>>>>>  res=0xffff8800a7253b48, task=0xffff8800b0eb0f00)
>>>>>>  at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>>>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=<optimized
>>>>>> out>, 
>>>>>>  calldata=0xffff8800a7253b80)
>>>>>>  at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>>>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=<optimized out>)
>>>>>>  at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>>>>> #3  0xffffffff8185f12b in __rpc_execute (task=0xffff88005a853800)
>>>>>>  at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>>>>> #4  0xffffffff818617b4 in rpc_execute (task=0xffff88005a853800)
>>>>>>  at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>>>>> #5  0xffffffff818539b9 in rpc_run_task
>>>>>> (task_setup_data=0xffff8800a7253a50)
>>>>>>  at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>>>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized
>>>>>> out>, 
>>>>>>  server=<optimized out>, msg=<optimized out>, args=<optimized
>>>>>> out>, 
>>>>>>  res=<optimized out>) at /home/green/bk/linux-
>>>>>> test/fs/nfs/nfs4proc.c:1051
>>>>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=<optimized
>>>>>> out>, 
>>>>>>  res=<optimized out>, args=<optimized out>, msg=<optimized
>>>>>> out>, 
>>>>>>  server=<optimized out>, clnt=<optimized out>)
>>>>>>  at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>>>>> #8  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
>>>>>>  res=<optimized out>, arg=<optimized out>, inode=<optimized
>>>>>> out>)
>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>  at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>>>>> #9  nfs4_do_setattr (inode=0xffff880079b152a8, cred=<optimized
>>>>>> out>, 
>>>>>>  fattr=<optimized out>, sattr=<optimized out>,
>>>>>> state=0xffff880060588e00, 
>>>>>>  ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
>>>>>>  at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>>>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=<optimized
>>>>>> out>, 
>>>>>>  fattr=0xffff8800a7253b80, sattr=0xffff8800a7253b48)
>>>>>>  at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>>>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800740c1000, 
>>>>>> 
>>>>> 
>>>>> Cool! Does the following help?
>>>> 
>>>> Grrr... There is another bug there…
>>> 
>>> Is this in addition to the previous patch or instead of?
>> 
>> It can apply on top of it. The two patches fix different bugs.
> 
> Ok, it does not seem to have helped:

Of the 10 nodes where I run this, I had some more failures overnight,
some of them different.

The other one I got was (I have 3 nodes in this state,
only one in the one reported yesterday.):

[65712.698123] NFS call  test_stateid ffff8800b05b1f24
[65712.698126] --> nfs41_call_sync_prepare data->seq_server ffff8800aeb16000
[65712.698127] --> nfs41_setup_sequence
[65712.698128] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[65712.698129] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[65712.698129] <-- nfs41_setup_sequence slotid=0 seqid=27222672
[65712.698136] encode_sequence: sessionid=1474186002:21:22:0 seqid=27222672 slotid=0 max_slotid=0 cache_this=0
[65712.698278] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[65712.698279] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[65712.698280] nfs4_free_slot: slotid 1 highest_used_slotid 0
[65712.698281] nfs41_sequence_process: Error 0 free the slot 
[65712.698282] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[65712.698295] NFS reply test_stateid: succeeded, 0
[65712.698301] --> nfs_put_client({3})
[65712.698333] --> nfs_put_client({2})
[65712.698337] --> nfs41_call_sync_prepare data->seq_server ffff8800aeb16000
[65712.698338] --> nfs41_setup_sequence
[65712.698339] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[65712.698339] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[65712.698340] <-- nfs41_setup_sequence slotid=0 seqid=27222673
[65712.698347] encode_sequence: sessionid=1474186002:21:22:0 seqid=27222673 slotid=0 max_slotid=0 cache_this=1
[65712.698501] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[65712.698502] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[65712.698503] nfs4_free_slot: slotid 1 highest_used_slotid 0
[65712.698503] nfs41_sequence_process: Error 0 free the slot 
[65712.698504] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[65712.698520] nfs4_schedule_stateid_recovery: scheduling stateid recovery for server localhost
[65712.698636] NFS call  test_stateid ffff8800b05b1f24
[65712.698639] --> nfs41_call_sync_prepare data->seq_server ffff8800aeb16000
[65712.698640] --> nfs41_setup_sequence
[65712.698641] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[65712.698642] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[65712.698642] <-- nfs41_setup_sequence slotid=0 seqid=27222674
[65712.698649] encode_sequence: sessionid=1474186002:21:22:0 seqid=27222674 slotid=0 max_slotid=0 cache_this=0
[65712.698800] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[65712.698801] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[65712.698802] nfs4_free_slot: slotid 1 highest_used_slotid 0
[65712.698802] nfs41_sequence_process: Error 0 free the slot 
[65712.698803] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[65712.698817] NFS reply test_stateid: succeeded, 0
[65712.698822] --> nfs_put_client({3})
[65712.698855] --> nfs_put_client({2})
[65712.698859] --> nfs41_call_sync_prepare data->seq_server ffff8800aeb16000
[65712.698860] --> nfs41_setup_sequence
[65712.698861] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
[65712.698862] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[65712.698863] <-- nfs41_setup_sequence slotid=0 seqid=27222675
[65712.698869] encode_sequence: sessionid=1474186002:21:22:0 seqid=27222675 slotid=0 max_slotid=0 cache_this=1
[65712.699022] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
[65712.699023] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[65712.699024] nfs4_free_slot: slotid 1 highest_used_slotid 0
[65712.699025] nfs41_sequence_process: Error 0 free the slot 
[65712.699026] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[65712.699042] nfs4_schedule_stateid_recovery: scheduling stateid recovery for server localhost


> [11397.552805] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
> [11397.552806] --> nfs41_setup_sequence
> [11397.552807] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
> [11397.552808] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
> [11397.552809] <-- nfs41_setup_sequence slotid=0 seqid=8829382
> [11397.552817] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829382 slotid=0 max_slotid=0 cache_this=1
> [11397.553032] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
> [11397.553033] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
> [11397.553034] nfs4_free_slot: slotid 1 highest_used_slotid 0
> [11397.553035] nfs41_sequence_process: Error 0 free the slot 
> [11397.553036] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
> [11397.553056] --> nfs_put_client({2})
> [11397.553062] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
> [11397.553063] --> nfs41_setup_sequence
> [11397.553064] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
> [11397.553065] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
> [11397.553066] <-- nfs41_setup_sequence slotid=0 seqid=8829383
> [11397.553074] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829383 slotid=0 max_slotid=0 cache_this=1
> [11397.553290] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
> [11397.553291] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
> [11397.553292] nfs4_free_slot: slotid 1 highest_used_slotid 0
> [11397.553293] nfs41_sequence_process: Error 0 free the slot 
> [11397.553294] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
> [11397.553316] --> nfs_put_client({2})
> [11397.553322] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
> [11397.553322] --> nfs41_setup_sequence
> [11397.553324] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
> [11397.553324] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
> [11397.553325] <-- nfs41_setup_sequence slotid=0 seqid=8829384
> [11397.553333] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829384 slotid=0 max_slotid=0 cache_this=1
> [11397.553575] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
> [11397.553576] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
> [11397.553577] nfs4_free_slot: slotid 1 highest_used_slotid 0
> [11397.553578] nfs41_sequence_process: Error 0 free the slot 
> [11397.553580] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
> [11397.553602] --> nfs_put_client({2})
> [11397.553607] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
> [11397.553608] --> nfs41_setup_sequence
> [11397.553609] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
> [11397.553610] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
> [11397.553611] <-- nfs41_setup_sequence slotid=0 seqid=8829385
> [11397.553619] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829385 slotid=0 max_slotid=0 cache_this=1
> [11397.560057] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
> [11397.560059] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
> [11397.560061] nfs4_free_slot: slotid 1 highest_used_slotid 0
> [11397.560062] nfs41_sequence_process: Error 0 free the slot 
> [11397.560064] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
> [11397.560094] --> nfs_put_client({2})
> [11397.560103] --> nfs41_call_sync_prepare data->seq_server ffff880085b17000
> [11397.560103] --> nfs41_setup_sequence
> [11397.560104] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
> [11397.560105] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
> [11397.560106] <-- nfs41_setup_sequence slotid=0 seqid=8829386
> [11397.560120] encode_sequence: sessionid=1474153431:3:4:0 seqid=8829386 slotid=0 max_slotid=0 cache_this=1
> [11397.560388] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
> [11397.560389] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
> [11397.560391] nfs4_free_slot: slotid 1 highest_used_slotid 0
> [11397.560391] nfs41_sequence_process: Error 0 free the slot 
> [11397.560393] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
> [11397.560415] --> nfs_put_client({2})
> 
> 
> bt looks about the same:
> #0  xprt_get (xprt=0x0 <irq_stack_union>)
>    at /home/green/bk/linux-test/net/sunrpc/xprt.c:1435
> #1  0xffffffff8186196f in rpc_init_task (task_setup_data=<optimized out>, 
>    task=<optimized out>) at /home/green/bk/linux-test/net/sunrpc/sched.c:936
> #2  rpc_new_task (setup_data=0x0 <irq_stack_union>)
>    at /home/green/bk/linux-test/net/sunrpc/sched.c:971
> #3  0xffffffff818538c8 in rpc_run_task (task_setup_data=0xffff88007236ba50)
>    at /home/green/bk/linux-test/net/sunrpc/clnt.c:1041
> #4  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized out>, 
>    server=<optimized out>, msg=<optimized out>, args=<optimized out>, 
>    res=<optimized out>) at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1055
> #5  0xffffffff813b4675 in nfs4_call_sync (cache_reply=<optimized out>, 
>    res=<optimized out>, args=<optimized out>, msg=<optimized out>, 
>    server=<optimized out>, clnt=<optimized out>)
>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1073
> #6  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
>    res=<optimized out>, arg=<optimized out>, inode=<optimized out>)
>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2920
> #7  nfs4_do_setattr (inode=0xffff8800640f82a8, cred=<optimized out>, 
>    fattr=<optimized out>, sattr=<optimized out>, state=0xffff88005a95be00, 
>    ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2959
> #8  0xffffffff813b4a46 in nfs4_proc_setattr (dentry=<optimized out>, 
> ---Type <return> to continue, or q <return> to quit---
>    fattr=0xf8 <irq_stack_union+248>, sattr=0x1 <irq_stack_union+1>)
>    at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3688
> #9  0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800119c0000, 
>    attr=0xffff88007236bce8) at /home/green/bk/linux-test/fs/nfs/inode.c:556
> #10 0xffffffff81298d6b in notify_change (dentry=0xffff8800119c0000, 
>    attr=0xffff88007236bce8, delegated_inode=<optimized out>)
>    at /home/green/bk/linux-test/fs/attr.c:285
> #11 0xffffffff812734e3 in do_truncate (dentry=0x0 <irq_stack_union>, 
>    length=<optimized out>, time_attrs=<optimized out>, filp=<optimized out>)
>    at /home/green/bk/linux-test/fs/open.c:63
> #12 0xffffffff81287459 in handle_truncate (filp=<optimized out>)
>    at /home/green/bk/linux-test/fs/namei.c:2960
> #13 do_last (opened=<optimized out>, op=<optimized out>, file=<optimized out>, 
>    nd=<optimized out>) at /home/green/bk/linux-test/fs/namei.c:3383
> #14 path_openat (nd=0xffff88007236bde0, op=<optimized out>, 
>    flags=<optimized out>) at /home/green/bk/linux-test/fs/namei.c:3497
> #15 0xffffffff81288b01 in do_filp_open (dfd=<optimized out>, 
>    pathname=<optimized out>, op=0xffff88007236bef4)
>    at /home/green/bk/linux-test/fs/namei.c:3532
> #16 0xffffffff81274ab0 in do_sys_open (dfd=-100, filename=<optimized out>, 
>    flags=<optimized out>, mode=<optimized out>)
>    at /home/green/bk/linux-test/fs/open.c:1036
> #17 0xffffffff81274bbe in SYSC_open (mode=<optimized out>, 
> ---Type <return> to continue, or q <return> to quit---
>    flags=<optimized out>, filename=<optimized out>)
>    at /home/green/bk/linux-test/fs/open.c:1054
> #18 SyS_open (filename=<optimized out>, flags=<optimized out>, 
>    mode=<optimized out>) at /home/green/bk/linux-test/fs/open.c:1049
> #19 0xffffffff8189f03c in entry_SYSCALL_64_fastpath ()
> 
> 

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Trond Myklebust]

> On Sep 18, 2016, at 12:23, Oleg Drokin <green@linuxhacker.ru> wrote:
>=20
>=20
> On Sep 17, 2016, at 9:19 PM, Oleg Drokin wrote:
>=20
>>=20
>> On Sep 17, 2016, at 5:59 PM, Trond Myklebust wrote:
>>=20
>>>=20
>>>> On Sep 17, 2016, at 17:55, Oleg Drokin <green@linuxhacker.ru> wrote:
>>>>=20
>>>>=20
>>>> On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:
>>>>=20
>>>>> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>>>>>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>>>>>>=20
>>>>>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>>>>>> wrote:
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>>>>>> to use
>>>>>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>>>>>> so
>>>>>>>>>> we
>>>>>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>>>>>>=20
>>>>>>>>>> While we already do this for open and lock state, we have not
>>>>>>>>>> been doing
>>>>>>>>>> so for all the delegations.
>>>>>>>>>>=20
>>>>>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>>>>>> too
>>>>>>>>>> v3: Now with added lock revoke fixes and
>>>>>>>>>> close/delegreturn/locku fixes
>>>>>>>>>> v4: Close a bunch of corner cases
>>>>>>>>>> v5: Report revoked delegations as invalid in
>>>>>>>>>> nfs_have_delegation()
>>>>>>>>>> Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>>>>> Fixes for other looping behaviour
>>>>>>>>>=20
>>>>>>>>> This time around the loop seems to be more tight,
>>>>>>>>> in userspace process:
>>>>>>>>>=20
>>>>>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>>>>>> ffff8800a73ce000
>>>>>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=3D0000
>>>>>>>>> highest_used=3D4294967295 max_slots=3D31
>>>>>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=3D0001
>>>>>>>>> highest_used=3D0
>>>>>>>>> slotid=3D0
>>>>>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=3D0 seqid=3D140138=
00
>>>>>>>>> [ 9197.256582] encode_sequence: sessionid=3D1474126170:1:2:0
>>>>>>>>> seqid=3D14013800 slotid=3D0 max_slotid=3D0 cache_this=3D1
>>>>>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=3D0001
>>>>>>>>> highest_used=3D0
>>>>>>>>> max_slots=3D31
>>>>>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=3D0003
>>>>>>>>> highest_used=3D1
>>>>>>>>> slotid=3D1
>>>>>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot=20
>>>>>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>>>>>> 4294967295
>>>>>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>>>>>>=20
>>>>>>>> What operation is the userspace process hanging on? Do you have a
>>>>>>>> stack trace for it?
>>>>>>>=20
>>>>>>> seems to be open_create->truncate->ssetattr coming from:
>>>>>>> cp /bin/sleep /mnt/nfs2/racer/12
>>>>>>>=20
>>>>>>> (gdb) bt
>>>>>>> #0  nfs41_setup_sequence (session=3D0xffff88005a853800,
>>>>>>> args=3D0xffff8800a7253b80,=20
>>>>>>> res=3D0xffff8800a7253b48, task=3D0xffff8800b0eb0f00)
>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>>>>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=3D<optimize=
d
>>>>>>> out>,=20
>>>>>>> calldata=3D0xffff8800a7253b80)
>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>>>>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=3D<optimized out>)
>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>>>>>> #3  0xffffffff8185f12b in __rpc_execute (task=3D0xffff88005a853800)
>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>>>>>> #4  0xffffffff818617b4 in rpc_execute (task=3D0xffff88005a853800)
>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>>>>>> #5  0xffffffff818539b9 in rpc_run_task
>>>>>>> (task_setup_data=3D0xffff8800a7253a50)
>>>>>>> at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>>>>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=3D<optimize=
d
>>>>>>> out>,=20
>>>>>>> server=3D<optimized out>, msg=3D<optimized out>, args=3D<optimized
>>>>>>> out>,=20
>>>>>>> res=3D<optimized out>) at /home/green/bk/linux-
>>>>>>> test/fs/nfs/nfs4proc.c:1051
>>>>>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=3D<optimized
>>>>>>> out>,=20
>>>>>>> res=3D<optimized out>, args=3D<optimized out>, msg=3D<optimized
>>>>>>> out>,=20
>>>>>>> server=3D<optimized out>, clnt=3D<optimized out>)
>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>>>>>> #8  _nfs4_do_setattr (state=3D<optimized out>, cred=3D<optimized ou=
t>,=20
>>>>>>> res=3D<optimized out>, arg=3D<optimized out>, inode=3D<optimized
>>>>>>> out>)
>>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>>>>>> #9  nfs4_do_setattr (inode=3D0xffff880079b152a8, cred=3D<optimized
>>>>>>> out>,=20
>>>>>>> fattr=3D<optimized out>, sattr=3D<optimized out>,
>>>>>>> state=3D0xffff880060588e00,=20
>>>>>>> ilabel=3D<optimized out>, olabel=3D0x0 <irq_stack_union>)
>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>>>>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=3D<optimized
>>>>>>> out>,=20
>>>>>>> fattr=3D0xffff8800a7253b80, sattr=3D0xffff8800a7253b48)
>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>>>>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=3D0xffff8800740c1000,=
=20
>>>>>>>=20
>>>>>>=20
>>>>>> Cool! Does the following help?
>>>>>=20
>>>>> Grrr... There is another bug there=85
>>>>=20
>>>> Is this in addition to the previous patch or instead of?
>>>=20
>>> It can apply on top of it. The two patches fix different bugs.
>>=20
>> Ok, it does not seem to have helped:
>=20
> Of the 10 nodes where I run this, I had some more failures overnight,
> some of them different.
>=20
> The other one I got was (I have 3 nodes in this state,
> only one in the one reported yesterday.):


I=92ve been trying desperately to reproduce the issue, but I can=92t make k=
nfsd return or revoke the delegations with the =93racer=94 script that you =
pointed me to (I=92m using the stock knfsd from Linux 4.7.3). Is there anyt=
hing else special about your setup that I should know about?

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Oleg Drokin]

On Sep 18, 2016, at 6:38 PM, Trond Myklebust wrote:

> 
>> On Sep 18, 2016, at 12:23, Oleg Drokin <green@linuxhacker.ru> wrote:
>> 
>> 
>> On Sep 17, 2016, at 9:19 PM, Oleg Drokin wrote:
>> 
>>> 
>>> On Sep 17, 2016, at 5:59 PM, Trond Myklebust wrote:
>>> 
>>>> 
>>>>> On Sep 17, 2016, at 17:55, Oleg Drokin <green@linuxhacker.ru> wrote:
>>>>> 
>>>>> 
>>>>> On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:
>>>>> 
>>>>>> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>>>>>>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>>>>>>> 
>>>>>>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>>>>>>> to use
>>>>>>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>>>>>>> so
>>>>>>>>>>> we
>>>>>>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>>>>>>> 
>>>>>>>>>>> While we already do this for open and lock state, we have not
>>>>>>>>>>> been doing
>>>>>>>>>>> so for all the delegations.
>>>>>>>>>>> 
>>>>>>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>>>>>>> too
>>>>>>>>>>> v3: Now with added lock revoke fixes and
>>>>>>>>>>> close/delegreturn/locku fixes
>>>>>>>>>>> v4: Close a bunch of corner cases
>>>>>>>>>>> v5: Report revoked delegations as invalid in
>>>>>>>>>>> nfs_have_delegation()
>>>>>>>>>>> Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>>>>>> Fixes for other looping behaviour
>>>>>>>>>> 
>>>>>>>>>> This time around the loop seems to be more tight,
>>>>>>>>>> in userspace process:
>>>>>>>>>> 
>>>>>>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>>>>>>> ffff8800a73ce000
>>>>>>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>>>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=0000
>>>>>>>>>> highest_used=4294967295 max_slots=31
>>>>>>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=0001
>>>>>>>>>> highest_used=0
>>>>>>>>>> slotid=0
>>>>>>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=0 seqid=14013800
>>>>>>>>>> [ 9197.256582] encode_sequence: sessionid=1474126170:1:2:0
>>>>>>>>>> seqid=14013800 slotid=0 max_slotid=0 cache_this=1
>>>>>>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=0001
>>>>>>>>>> highest_used=0
>>>>>>>>>> max_slots=31
>>>>>>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=0003
>>>>>>>>>> highest_used=1
>>>>>>>>>> slotid=1
>>>>>>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>>>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot 
>>>>>>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>>>>>>> 4294967295
>>>>>>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>>>>>>> 
>>>>>>>>> What operation is the userspace process hanging on? Do you have a
>>>>>>>>> stack trace for it?
>>>>>>>> 
>>>>>>>> seems to be open_create->truncate->ssetattr coming from:
>>>>>>>> cp /bin/sleep /mnt/nfs2/racer/12
>>>>>>>> 
>>>>>>>> (gdb) bt
>>>>>>>> #0  nfs41_setup_sequence (session=0xffff88005a853800,
>>>>>>>> args=0xffff8800a7253b80, 
>>>>>>>> res=0xffff8800a7253b48, task=0xffff8800b0eb0f00)
>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>>>>>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=<optimized
>>>>>>>> out>, 
>>>>>>>> calldata=0xffff8800a7253b80)
>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>>>>>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=<optimized out>)
>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>>>>>>> #3  0xffffffff8185f12b in __rpc_execute (task=0xffff88005a853800)
>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>>>>>>> #4  0xffffffff818617b4 in rpc_execute (task=0xffff88005a853800)
>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>>>>>>> #5  0xffffffff818539b9 in rpc_run_task
>>>>>>>> (task_setup_data=0xffff8800a7253a50)
>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>>>>>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized
>>>>>>>> out>, 
>>>>>>>> server=<optimized out>, msg=<optimized out>, args=<optimized
>>>>>>>> out>, 
>>>>>>>> res=<optimized out>) at /home/green/bk/linux-
>>>>>>>> test/fs/nfs/nfs4proc.c:1051
>>>>>>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=<optimized
>>>>>>>> out>, 
>>>>>>>> res=<optimized out>, args=<optimized out>, msg=<optimized
>>>>>>>> out>, 
>>>>>>>> server=<optimized out>, clnt=<optimized out>)
>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>>>>>>> #8  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
>>>>>>>> res=<optimized out>, arg=<optimized out>, inode=<optimized
>>>>>>>> out>)
>>>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>>>>>>> #9  nfs4_do_setattr (inode=0xffff880079b152a8, cred=<optimized
>>>>>>>> out>, 
>>>>>>>> fattr=<optimized out>, sattr=<optimized out>,
>>>>>>>> state=0xffff880060588e00, 
>>>>>>>> ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>>>>>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=<optimized
>>>>>>>> out>, 
>>>>>>>> fattr=0xffff8800a7253b80, sattr=0xffff8800a7253b48)
>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>>>>>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800740c1000, 
>>>>>>>> 
>>>>>>> 
>>>>>>> Cool! Does the following help?
>>>>>> 
>>>>>> Grrr... There is another bug there…
>>>>> 
>>>>> Is this in addition to the previous patch or instead of?
>>>> 
>>>> It can apply on top of it. The two patches fix different bugs.
>>> 
>>> Ok, it does not seem to have helped:
>> 
>> Of the 10 nodes where I run this, I had some more failures overnight,
>> some of them different.
>> 
>> The other one I got was (I have 3 nodes in this state,
>> only one in the one reported yesterday.):
> 
> 
> I’ve been trying desperately to reproduce the issue, but I can’t make knfsd return or revoke the delegations with the “racer” script that you pointed me to (I’m using the stock knfsd from Linux 4.7.3). Is there anything else special about your setup that I should know about?


There is not really anything special here.
I have a full debug enabled kernel (I checking latest Linus' master before
applying your whole patchset, but it reproduces the same on earlier ones too).

In my particular case I run in VMs with overcommitted CPUs (that really helps to
promote races) and with HT enabled (that serves the same purpose).

I also boot my VMs from nfsroot, but I doubt it plays any role here.

Then I just ssh into the VMs and run the script and that is all.

I can show you my kernel config if you think it would help and I can also
provide you with access to my testbed if you think you want to take a closer
look at my setup.

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Trond Myklebust]

> On Sep 18, 2016, at 18:44, Oleg Drokin <green@linuxhacker.ru> wrote:
>=20
>=20
> On Sep 18, 2016, at 6:38 PM, Trond Myklebust wrote:
>=20
>>=20
>>> On Sep 18, 2016, at 12:23, Oleg Drokin <green@linuxhacker.ru> wrote:
>>>=20
>>>=20
>>> On Sep 17, 2016, at 9:19 PM, Oleg Drokin wrote:
>>>=20
>>>>=20
>>>> On Sep 17, 2016, at 5:59 PM, Trond Myklebust wrote:
>>>>=20
>>>>>=20
>>>>>> On Sep 17, 2016, at 17:55, Oleg Drokin <green@linuxhacker.ru> wrote:
>>>>>>=20
>>>>>>=20
>>>>>> On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:
>>>>>>=20
>>>>>>> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>>>>>>>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>>>>>>>>=20
>>>>>>>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>>>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>>>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>>>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>>>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>>>>>>>> to use
>>>>>>>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>>>>>>>> so
>>>>>>>>>>>> we
>>>>>>>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>>>>>>>>=20
>>>>>>>>>>>> While we already do this for open and lock state, we have not
>>>>>>>>>>>> been doing
>>>>>>>>>>>> so for all the delegations.
>>>>>>>>>>>>=20
>>>>>>>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>>>>>>>> too
>>>>>>>>>>>> v3: Now with added lock revoke fixes and
>>>>>>>>>>>> close/delegreturn/locku fixes
>>>>>>>>>>>> v4: Close a bunch of corner cases
>>>>>>>>>>>> v5: Report revoked delegations as invalid in
>>>>>>>>>>>> nfs_have_delegation()
>>>>>>>>>>>> Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>>>>>>> Fixes for other looping behaviour
>>>>>>>>>>>=20
>>>>>>>>>>> This time around the loop seems to be more tight,
>>>>>>>>>>> in userspace process:
>>>>>>>>>>>=20
>>>>>>>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>>>>>>>> ffff8800a73ce000
>>>>>>>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>>>>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=3D0000
>>>>>>>>>>> highest_used=3D4294967295 max_slots=3D31
>>>>>>>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=3D0001
>>>>>>>>>>> highest_used=3D0
>>>>>>>>>>> slotid=3D0
>>>>>>>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=3D0 seqid=3D1401=
3800
>>>>>>>>>>> [ 9197.256582] encode_sequence: sessionid=3D1474126170:1:2:0
>>>>>>>>>>> seqid=3D14013800 slotid=3D0 max_slotid=3D0 cache_this=3D1
>>>>>>>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=3D0001
>>>>>>>>>>> highest_used=3D0
>>>>>>>>>>> max_slots=3D31
>>>>>>>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=3D0003
>>>>>>>>>>> highest_used=3D1
>>>>>>>>>>> slotid=3D1
>>>>>>>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>>>>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot=20
>>>>>>>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>>>>>>>> 4294967295
>>>>>>>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>>>>>>>>=20
>>>>>>>>>> What operation is the userspace process hanging on? Do you have =
a
>>>>>>>>>> stack trace for it?
>>>>>>>>>=20
>>>>>>>>> seems to be open_create->truncate->ssetattr coming from:
>>>>>>>>> cp /bin/sleep /mnt/nfs2/racer/12
>>>>>>>>>=20
>>>>>>>>> (gdb) bt
>>>>>>>>> #0  nfs41_setup_sequence (session=3D0xffff88005a853800,
>>>>>>>>> args=3D0xffff8800a7253b80,=20
>>>>>>>>> res=3D0xffff8800a7253b48, task=3D0xffff8800b0eb0f00)
>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>>>>>>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=3D<optimi=
zed
>>>>>>>>> out>,=20
>>>>>>>>> calldata=3D0xffff8800a7253b80)
>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>>>>>>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=3D<optimized out=
>)
>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>>>>>>>> #3  0xffffffff8185f12b in __rpc_execute (task=3D0xffff88005a85380=
0)
>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>>>>>>>> #4  0xffffffff818617b4 in rpc_execute (task=3D0xffff88005a853800)
>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>>>>>>>> #5  0xffffffff818539b9 in rpc_run_task
>>>>>>>>> (task_setup_data=3D0xffff8800a7253a50)
>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>>>>>>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=3D<optimi=
zed
>>>>>>>>> out>,=20
>>>>>>>>> server=3D<optimized out>, msg=3D<optimized out>, args=3D<optimize=
d
>>>>>>>>> out>,=20
>>>>>>>>> res=3D<optimized out>) at /home/green/bk/linux-
>>>>>>>>> test/fs/nfs/nfs4proc.c:1051
>>>>>>>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=3D<optimize=
d
>>>>>>>>> out>,=20
>>>>>>>>> res=3D<optimized out>, args=3D<optimized out>, msg=3D<optimized
>>>>>>>>> out>,=20
>>>>>>>>> server=3D<optimized out>, clnt=3D<optimized out>)
>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>>>>>>>> #8  _nfs4_do_setattr (state=3D<optimized out>, cred=3D<optimized =
out>,=20
>>>>>>>>> res=3D<optimized out>, arg=3D<optimized out>, inode=3D<optimized
>>>>>>>>> out>)
>>>>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>>>>>>>> #9  nfs4_do_setattr (inode=3D0xffff880079b152a8, cred=3D<optimize=
d
>>>>>>>>> out>,=20
>>>>>>>>> fattr=3D<optimized out>, sattr=3D<optimized out>,
>>>>>>>>> state=3D0xffff880060588e00,=20
>>>>>>>>> ilabel=3D<optimized out>, olabel=3D0x0 <irq_stack_union>)
>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>>>>>>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=3D<optimized
>>>>>>>>> out>,=20
>>>>>>>>> fattr=3D0xffff8800a7253b80, sattr=3D0xffff8800a7253b48)
>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>>>>>>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=3D0xffff8800740c100=
0,=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> Cool! Does the following help?
>>>>>>>=20
>>>>>>> Grrr... There is another bug there=85
>>>>>>=20
>>>>>> Is this in addition to the previous patch or instead of?
>>>>>=20
>>>>> It can apply on top of it. The two patches fix different bugs.
>>>>=20
>>>> Ok, it does not seem to have helped:
>>>=20
>>> Of the 10 nodes where I run this, I had some more failures overnight,
>>> some of them different.
>>>=20
>>> The other one I got was (I have 3 nodes in this state,
>>> only one in the one reported yesterday.):
>>=20
>>=20
>> I=92ve been trying desperately to reproduce the issue, but I can=92t mak=
e knfsd return or revoke the delegations with the =93racer=94 script that y=
ou pointed me to (I=92m using the stock knfsd from Linux 4.7.3). Is there a=
nything else special about your setup that I should know about?
>=20
>=20
> There is not really anything special here.
> I have a full debug enabled kernel (I checking latest Linus' master befor=
e
> applying your whole patchset, but it reproduces the same on earlier ones =
too).
>=20
> In my particular case I run in VMs with overcommitted CPUs (that really h=
elps to
> promote races) and with HT enabled (that serves the same purpose).

Interesting. I can try that.

>=20
> I also boot my VMs from nfsroot, but I doubt it plays any role here.
>=20

Do they all have different hostnames (and/or are they set up with unique nf=
s.nfs4_unique_id module parameters) , or is there a chance that they might =
be using the same client-id and thus be stepping on each others state?

> Then I just ssh into the VMs and run the script and that is all.
>=20
> I can show you my kernel config if you think it would help and I can also
> provide you with access to my testbed if you think you want to take a clo=
ser
> look at my setup.

Let me see if I can come up with some extra debugging dprintk()s that might=
 help understand that loop. The current set are apparently not sufficient t=
o clue me in (sorry.. :-( )

Re: [PATCH v5 00/25] Fix delegation behaviour when server revokes some state

[Oleg Drokin]

On Sep 18, 2016, at 7:08 PM, Trond Myklebust wrote:

> 
>> On Sep 18, 2016, at 18:44, Oleg Drokin <green@linuxhacker.ru> wrote:
>> 
>> 
>> On Sep 18, 2016, at 6:38 PM, Trond Myklebust wrote:
>> 
>>> 
>>>> On Sep 18, 2016, at 12:23, Oleg Drokin <green@linuxhacker.ru> wrote:
>>>> 
>>>> 
>>>> On Sep 17, 2016, at 9:19 PM, Oleg Drokin wrote:
>>>> 
>>>>> 
>>>>> On Sep 17, 2016, at 5:59 PM, Trond Myklebust wrote:
>>>>> 
>>>>>> 
>>>>>>> On Sep 17, 2016, at 17:55, Oleg Drokin <green@linuxhacker.ru> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On Sep 17, 2016, at 5:45 PM, Trond Myklebust wrote:
>>>>>>> 
>>>>>>>> On Sat, 2016-09-17 at 15:32 -0400, Trond Myklebust wrote:
>>>>>>>>> On Sat, 2016-09-17 at 15:16 -0400, Oleg Drokin wrote:
>>>>>>>>>> 
>>>>>>>>>> On Sep 17, 2016, at 2:18 PM, Trond Myklebust wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 17, 2016, at 14:04, Oleg Drokin <green@linuxhacker.ru>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sep 17, 2016, at 1:13 AM, Trond Myklebust wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> According to RFC5661, if any of the SEQUENCE status bits
>>>>>>>>>>>>> SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED,
>>>>>>>>>>>>> SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED,
>>>>>>>>>>>>> SEQ4_STATUS_ADMIN_STATE_REVOKED,
>>>>>>>>>>>>> or SEQ4_STATUS_RECALLABLE_STATE_REVOKED are set, then we need
>>>>>>>>>>>>> to use
>>>>>>>>>>>>> TEST_STATEID to figure out which stateids have been revoked,
>>>>>>>>>>>>> so
>>>>>>>>>>>>> we
>>>>>>>>>>>>> can acknowledge the loss of state using FREE_STATEID.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> While we already do this for open and lock state, we have not
>>>>>>>>>>>>> been doing
>>>>>>>>>>>>> so for all the delegations.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> v2: nfs_v4_2_minor_ops needs to set .test_and_free_expired
>>>>>>>>>>>>> too
>>>>>>>>>>>>> v3: Now with added lock revoke fixes and
>>>>>>>>>>>>> close/delegreturn/locku fixes
>>>>>>>>>>>>> v4: Close a bunch of corner cases
>>>>>>>>>>>>> v5: Report revoked delegations as invalid in
>>>>>>>>>>>>> nfs_have_delegation()
>>>>>>>>>>>>> Fix an infinite loop in nfs_reap_expired_delegations.
>>>>>>>>>>>>> Fixes for other looping behaviour
>>>>>>>>>>>> 
>>>>>>>>>>>> This time around the loop seems to be more tight,
>>>>>>>>>>>> in userspace process:
>>>>>>>>>>>> 
>>>>>>>>>>>> [ 9197.256571] --> nfs41_call_sync_prepare data->seq_server
>>>>>>>>>>>> ffff8800a73ce000
>>>>>>>>>>>> [ 9197.256572] --> nfs41_setup_sequence
>>>>>>>>>>>> [ 9197.256573] --> nfs4_alloc_slot used_slots=0000
>>>>>>>>>>>> highest_used=4294967295 max_slots=31
>>>>>>>>>>>> [ 9197.256574] <-- nfs4_alloc_slot used_slots=0001
>>>>>>>>>>>> highest_used=0
>>>>>>>>>>>> slotid=0
>>>>>>>>>>>> [ 9197.256574] <-- nfs41_setup_sequence slotid=0 seqid=14013800
>>>>>>>>>>>> [ 9197.256582] encode_sequence: sessionid=1474126170:1:2:0
>>>>>>>>>>>> seqid=14013800 slotid=0 max_slotid=0 cache_this=1
>>>>>>>>>>>> [ 9197.256755] --> nfs4_alloc_slot used_slots=0001
>>>>>>>>>>>> highest_used=0
>>>>>>>>>>>> max_slots=31
>>>>>>>>>>>> [ 9197.256756] <-- nfs4_alloc_slot used_slots=0003
>>>>>>>>>>>> highest_used=1
>>>>>>>>>>>> slotid=1
>>>>>>>>>>>> [ 9197.256757] nfs4_free_slot: slotid 1 highest_used_slotid 0
>>>>>>>>>>>> [ 9197.256758] nfs41_sequence_process: Error 0 free the slot 
>>>>>>>>>>>> [ 9197.256760] nfs4_free_slot: slotid 0 highest_used_slotid
>>>>>>>>>>>> 4294967295
>>>>>>>>>>>> [ 9197.256779] --> nfs_put_client({2})
>>>>>>>>>>> 
>>>>>>>>>>> What operation is the userspace process hanging on? Do you have a
>>>>>>>>>>> stack trace for it?
>>>>>>>>>> 
>>>>>>>>>> seems to be open_create->truncate->ssetattr coming from:
>>>>>>>>>> cp /bin/sleep /mnt/nfs2/racer/12
>>>>>>>>>> 
>>>>>>>>>> (gdb) bt
>>>>>>>>>> #0  nfs41_setup_sequence (session=0xffff88005a853800,
>>>>>>>>>> args=0xffff8800a7253b80, 
>>>>>>>>>> res=0xffff8800a7253b48, task=0xffff8800b0eb0f00)
>>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:876
>>>>>>>>>> #1  0xffffffff813a751c in nfs41_call_sync_prepare (task=<optimized
>>>>>>>>>> out>, 
>>>>>>>>>> calldata=0xffff8800a7253b80)
>>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:966
>>>>>>>>>> #2  0xffffffff8185c639 in rpc_prepare_task (task=<optimized out>)
>>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:683
>>>>>>>>>> #3  0xffffffff8185f12b in __rpc_execute (task=0xffff88005a853800)
>>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:775
>>>>>>>>>> #4  0xffffffff818617b4 in rpc_execute (task=0xffff88005a853800)
>>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/sched.c:843
>>>>>>>>>> #5  0xffffffff818539b9 in rpc_run_task
>>>>>>>>>> (task_setup_data=0xffff8800a7253a50)
>>>>>>>>>> at /home/green/bk/linux-test/net/sunrpc/clnt.c:1052
>>>>>>>>>> #6  0xffffffff813a75e3 in nfs4_call_sync_sequence (clnt=<optimized
>>>>>>>>>> out>, 
>>>>>>>>>> server=<optimized out>, msg=<optimized out>, args=<optimized
>>>>>>>>>> out>, 
>>>>>>>>>> res=<optimized out>) at /home/green/bk/linux-
>>>>>>>>>> test/fs/nfs/nfs4proc.c:1051
>>>>>>>>>> #7  0xffffffff813b4645 in nfs4_call_sync (cache_reply=<optimized
>>>>>>>>>> out>, 
>>>>>>>>>> res=<optimized out>, args=<optimized out>, msg=<optimized
>>>>>>>>>> out>, 
>>>>>>>>>> server=<optimized out>, clnt=<optimized out>)
>>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:1069
>>>>>>>>>> #8  _nfs4_do_setattr (state=<optimized out>, cred=<optimized out>, 
>>>>>>>>>> res=<optimized out>, arg=<optimized out>, inode=<optimized
>>>>>>>>>> out>)
>>>>>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2916
>>>>>>>>>> #9  nfs4_do_setattr (inode=0xffff880079b152a8, cred=<optimized
>>>>>>>>>> out>, 
>>>>>>>>>> fattr=<optimized out>, sattr=<optimized out>,
>>>>>>>>>> state=0xffff880060588e00, 
>>>>>>>>>> ilabel=<optimized out>, olabel=0x0 <irq_stack_union>)
>>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:2955
>>>>>>>>>> #10 0xffffffff813b4a16 in nfs4_proc_setattr (dentry=<optimized
>>>>>>>>>> out>, 
>>>>>>>>>> fattr=0xffff8800a7253b80, sattr=0xffff8800a7253b48)
>>>>>>>>>> at /home/green/bk/linux-test/fs/nfs/nfs4proc.c:3684
>>>>>>>>>> #11 0xffffffff8138f1cb in nfs_setattr (dentry=0xffff8800740c1000, 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Cool! Does the following help?
>>>>>>>> 
>>>>>>>> Grrr... There is another bug there…
>>>>>>> 
>>>>>>> Is this in addition to the previous patch or instead of?
>>>>>> 
>>>>>> It can apply on top of it. The two patches fix different bugs.
>>>>> 
>>>>> Ok, it does not seem to have helped:
>>>> 
>>>> Of the 10 nodes where I run this, I had some more failures overnight,
>>>> some of them different.
>>>> 
>>>> The other one I got was (I have 3 nodes in this state,
>>>> only one in the one reported yesterday.):
>>> 
>>> 
>>> I’ve been trying desperately to reproduce the issue, but I can’t make knfsd return or revoke the delegations with the “racer” script that you pointed me to (I’m using the stock knfsd from Linux 4.7.3). Is there anything else special about your setup that I should know about?
>> 
>> 
>> There is not really anything special here.
>> I have a full debug enabled kernel (I checking latest Linus' master before
>> applying your whole patchset, but it reproduces the same on earlier ones too).
>> 
>> In my particular case I run in VMs with overcommitted CPUs (that really helps to
>> promote races) and with HT enabled (that serves the same purpose).
> 
> Interesting. I can try that.
> 
>> 
>> I also boot my VMs from nfsroot, but I doubt it plays any role here.
>> 
> 
> Do they all have different hostnames (and/or are they set up with unique nfs.nfs4_unique_id module parameters) , or is there a chance that they might be using the same client-id and thus be stepping on each others state?

They have uniqueue hostnames now, but that is not important for the reproducer,
and it's a relatively recent change.

If you remember my test script, I start a local ext4 on every client, then
export that with nfs and then mount it twice on the same node locally,
and then run 3 copies of racer - once per mountpoint.

>> Then I just ssh into the VMs and run the script and that is all.
>> 
>> I can show you my kernel config if you think it would help and I can also
>> provide you with access to my testbed if you think you want to take a closer
>> look at my setup.
> 
> Let me see if I can come up with some extra debugging dprintk()s that might help understand that loop. The current set are apparently not sufficient to clue me in (sorry.. :-( )

I can attach a gdb to fetch some more data with no debug printks and I have
several nodes sitting in this state now.

Just tell me what it is you want me to look for?