[PATCH v1 00/19] NFSv4 migration

[PATCH v1 00/19] NFSv4 migration

[Chuck Lever]

I hope to get basic NFSv4 migration support into 3.12.  This is the
patch series, still based on 3.10, posted here for review.  See also

  http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=commit;
	h=8aa9aa847fdd073abe24789c5acb28b853755616


- Patch roadmap -

The first two patches are bug fixes.

Patches "NFS: Refactor nfs4_call_data_sequence()" through "NFS:
Implement a transport blocking scheme for migration" introduce a
mechanism that can plug and uncork an NFS transport.  NFSv4.0
migration uses this mechanism to block forward NFS operations while
migration recovery is occurring.  NFSv4.1 and following will use
session draining to accomplish the same.

Patch "NFS: Add functions to swap transports during migration
recovery" moves an nfs_server from one nfs_client to another, and
swaps out the nfs_server's RPC transport at the same time.  This
mechanism probably still needs some work.

Patches "NFS: Add method to retrieve fs_locations during migration
recovery" through "NFS: Add migration recovery callouts in
nfs4proc.c" add the bulk of support for recovering from NFS4ERR_MOVED.

Patches "NFS: Add method to detect whether an FSID is still on the
server" through "NFS: Implement support for NFS4ERR_LEASE_MOVED" add
support for recovering from NFS4ERR_LEASE_MOVED.

Patch "NFS: Set EXCHGID4_FLAG_SUPP_MOVED_MIGR" advertises migration
support for NFSv4.1.


- Merge criteria -

I have three criteria for determining whether I believe this work is
ready to merge:

  1.  The series doesn't regress any non-migration related
      functionality

  2.  Everyone is happy with the architecture and APIs, such that
      subsequent patches will amount to bug fixes and enhancements
      rather than reworking anything

  3.  The series is ported to the tip of Linus' master (3.11-rc1
      as of this writing)

I'd like to merge sooner rather than later so others can start to
build on this functionality, and so I don't have to keep forward
porting for another two years.

In addition, this is something that distributions want yesterday.
Including it now and then fixing it makes it easier for them to
include it in their kernels.

I expect bug fixes and enhancements after merge.  Enhancements might
include:

  Optimize LEASE_MOVED recovery by performing the fsid_present
  checks asynchronously

  Introduce support for fs_locations_info

  Finish NFSv4.1 migration support

One thing that hasn't been tested yet is NFSv4.1 migration.  There
are no server implementations yet, and we believe there may be some
(IETF) protocol work to do to get NFSv4.1 migration working
reliably.

We could drop that patch, or we could change it to add a CONFIG
option, default N, marked EXPERIMENTAL so that the flag can be
enabled conveniently for testing.


- Test plan -

This implementation still needs a lot of testing.  I've done some of
this list, plan to do more.

   * Single filesystem migration with TSM
   * Single filesystem migration without TSM
   * Active filesystem A while B is migrated (lease-moved recovery)
   * Active filesystem A while active filesystem B is migrated
   * Migrate filesystem A, reboot destination server, migrate B
   * Migration during lock-intensive workload
   * What happens when migration recovery fails?
   * How about migration with a Kerberos 5 mount?


---

Chuck Lever (18):
      NFS: Never use user credentials for lease renewal
      NFS: Fix return type of nfs4_end_drain_session() stub
      NFS: Introduce a vector of migration recovery ops
      NFS: Refactor nfs4_call_data_sequence()
      NFS: Rename nfs41_call_sync_data as a common data structure
      NFS: Clean up nfs4_setup_sequence()
      NFS: Fix compiler warning in nfs_setup_sequence()
      NFS: Use RPC callouts to start NFSv4.0 synchronous requests
      NFS: Add a "struct nfs_server *" argument to nfs4_sequence_done()
      NFS: Implement a transport blocking scheme for migration
      NFS: Add a super_block backpointer to the nfs_server struct
      NFS: Add functions to swap transports during migration recovery
      NFS: Add method to retrieve fs_locations during migration recovery
      NFS: Add basic migration support to state manager thread
      NFS: Add migration recovery callouts in nfs4proc.c
      NFS: Add method to detect whether an FSID is still on the server
      NFS: Implement support for NFS4ERR_LEASE_MOVED
      NFS: Set EXCHGID4_FLAG_SUPP_MOVED_MIGR

Trond Myklebust (1):
      SUNRPC: Add a helper to switch the transport of an rpc_clnt


 fs/nfs/client.c             |    3 
 fs/nfs/internal.h           |    3 
 fs/nfs/nfs4_fs.h            |   36 +--
 fs/nfs/nfs4client.c         |   83 ++++++
 fs/nfs/nfs4namespace.c      |  114 +++++++++
 fs/nfs/nfs4proc.c           |  554 +++++++++++++++++++++++++++++++++++++------
 fs/nfs/nfs4session.c        |    2 
 fs/nfs/nfs4state.c          |  318 +++++++++++++++++++++----
 fs/nfs/nfs4xdr.c            |  115 ++++++++-
 fs/nfs/super.c              |    1 
 include/linux/nfs4.h        |    1 
 include/linux/nfs_fs_sb.h   |   11 +
 include/linux/nfs_xdr.h     |   17 +
 include/linux/sunrpc/clnt.h |    4 
 net/sunrpc/clnt.c           |   83 ++++++
 15 files changed, 1183 insertions(+), 162 deletions(-)

-- 
Chuck Lever

[PATCH v1 01/19] NFS: Never use user credentials for lease renewal

[Chuck Lever]

Don't try to use a non-UID-0 user credential for lease management,
as that credential can change out from under us.  The server will
block NFSv4 lease recovery with NFS4ERR_CLID_INUSE.

Replace minor version-specific callouts with a single function.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h     |    4 +---
 fs/nfs/nfs4proc.c    |    6 +----
 fs/nfs/nfs4session.c |    2 +-
 fs/nfs/nfs4state.c   |   55 ++++++--------------------------------------------
 4 files changed, 10 insertions(+), 57 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index a1dd768..3bc554e 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -193,7 +193,6 @@ struct nfs4_state_recovery_ops {
 	int (*recover_open)(struct nfs4_state_owner *, struct nfs4_state *);
 	int (*recover_lock)(struct nfs4_state *, struct file_lock *);
 	int (*establish_clid)(struct nfs_client *, struct rpc_cred *);
-	struct rpc_cred * (*get_clid_cred)(struct nfs_client *);
 	int (*reclaim_complete)(struct nfs_client *);
 	int (*detect_trunking)(struct nfs_client *, struct nfs_client **,
 		struct rpc_cred *);
@@ -319,7 +318,7 @@ extern void nfs4_kill_renewd(struct nfs_client *);
 extern void nfs4_renew_state(struct work_struct *);
 
 /* nfs4state.c */
-struct rpc_cred *nfs4_get_setclientid_cred(struct nfs_client *clp);
+struct rpc_cred *nfs4_get_clid_cred(struct nfs_client *clp);
 struct rpc_cred *nfs4_get_machine_cred_locked(struct nfs_client *clp);
 struct rpc_cred *nfs4_get_renew_cred_locked(struct nfs_client *clp);
 int nfs4_discover_server_trunking(struct nfs_client *clp,
@@ -327,7 +326,6 @@ int nfs4_discover_server_trunking(struct nfs_client *clp,
 int nfs40_discover_server_trunking(struct nfs_client *clp,
 			struct nfs_client **, struct rpc_cred *);
 #if defined(CONFIG_NFS_V4_1)
-struct rpc_cred *nfs4_get_exchange_id_cred(struct nfs_client *clp);
 int nfs41_discover_server_trunking(struct nfs_client *clp,
 			struct nfs_client **, struct rpc_cred *);
 extern void nfs4_schedule_session_recovery(struct nfs4_session *, int);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index d7ba561..d74a2b5 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5643,7 +5643,7 @@ int nfs4_destroy_clientid(struct nfs_client *clp)
 		goto out;
 	if (clp->cl_preserve_clid)
 		goto out;
-	cred = nfs4_get_exchange_id_cred(clp);
+	cred = nfs4_get_clid_cred(clp);
 	ret = nfs4_proc_destroy_clientid(clp, cred);
 	if (cred)
 		put_rpccred(cred);
@@ -6919,7 +6919,6 @@ static const struct nfs4_state_recovery_ops nfs40_reboot_recovery_ops = {
 	.recover_open	= nfs4_open_reclaim,
 	.recover_lock	= nfs4_lock_reclaim,
 	.establish_clid = nfs4_init_clientid,
-	.get_clid_cred	= nfs4_get_setclientid_cred,
 	.detect_trunking = nfs40_discover_server_trunking,
 };
 
@@ -6930,7 +6929,6 @@ static const struct nfs4_state_recovery_ops nfs41_reboot_recovery_ops = {
 	.recover_open	= nfs4_open_reclaim,
 	.recover_lock	= nfs4_lock_reclaim,
 	.establish_clid = nfs41_init_clientid,
-	.get_clid_cred	= nfs4_get_exchange_id_cred,
 	.reclaim_complete = nfs41_proc_reclaim_complete,
 	.detect_trunking = nfs41_discover_server_trunking,
 };
@@ -6942,7 +6940,6 @@ static const struct nfs4_state_recovery_ops nfs40_nograce_recovery_ops = {
 	.recover_open	= nfs4_open_expired,
 	.recover_lock	= nfs4_lock_expired,
 	.establish_clid = nfs4_init_clientid,
-	.get_clid_cred	= nfs4_get_setclientid_cred,
 };
 
 #if defined(CONFIG_NFS_V4_1)
@@ -6952,7 +6949,6 @@ static const struct nfs4_state_recovery_ops nfs41_nograce_recovery_ops = {
 	.recover_open	= nfs41_open_expired,
 	.recover_lock	= nfs41_lock_expired,
 	.establish_clid = nfs41_init_clientid,
-	.get_clid_cred	= nfs4_get_exchange_id_cred,
 };
 #endif /* CONFIG_NFS_V4_1 */
 
diff --git a/fs/nfs/nfs4session.c b/fs/nfs/nfs4session.c
index c4e225e..d35b816 100644
--- a/fs/nfs/nfs4session.c
+++ b/fs/nfs/nfs4session.c
@@ -441,7 +441,7 @@ void nfs4_destroy_session(struct nfs4_session *session)
 	struct rpc_xprt *xprt;
 	struct rpc_cred *cred;
 
-	cred = nfs4_get_exchange_id_cred(session->clp);
+	cred = nfs4_get_clid_cred(session->clp);
 	nfs4_proc_destroy_session(session, cred);
 	if (cred)
 		put_rpccred(cred);
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 6ceece7..15fde8c 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -353,62 +353,21 @@ int nfs41_discover_server_trunking(struct nfs_client *clp,
 	return nfs41_walk_client_list(clp, result, cred);
 }
 
-struct rpc_cred *nfs4_get_exchange_id_cred(struct nfs_client *clp)
-{
-	struct rpc_cred *cred;
-
-	spin_lock(&clp->cl_lock);
-	cred = nfs4_get_machine_cred_locked(clp);
-	spin_unlock(&clp->cl_lock);
-	return cred;
-}
-
 #endif /* CONFIG_NFS_V4_1 */
 
-static struct rpc_cred *
-nfs4_get_setclientid_cred_server(struct nfs_server *server)
-{
-	struct nfs_client *clp = server->nfs_client;
-	struct rpc_cred *cred = NULL;
-	struct nfs4_state_owner *sp;
-	struct rb_node *pos;
-
-	spin_lock(&clp->cl_lock);
-	pos = rb_first(&server->state_owners);
-	if (pos != NULL) {
-		sp = rb_entry(pos, struct nfs4_state_owner, so_server_node);
-		cred = get_rpccred(sp->so_cred);
-	}
-	spin_unlock(&clp->cl_lock);
-	return cred;
-}
-
 /**
- * nfs4_get_setclientid_cred - Acquire credential for a setclientid operation
+ * nfs4_get_clid_cred - Acquire credential for a setclientid operation
  * @clp: client state handle
  *
  * Returns an rpc_cred with reference count bumped, or NULL.
  */
-struct rpc_cred *nfs4_get_setclientid_cred(struct nfs_client *clp)
+struct rpc_cred *nfs4_get_clid_cred(struct nfs_client *clp)
 {
-	struct nfs_server *server;
 	struct rpc_cred *cred;
 
 	spin_lock(&clp->cl_lock);
 	cred = nfs4_get_machine_cred_locked(clp);
 	spin_unlock(&clp->cl_lock);
-	if (cred != NULL)
-		goto out;
-
-	rcu_read_lock();
-	list_for_each_entry_rcu(server, &clp->cl_superblocks, client_link) {
-		cred = nfs4_get_setclientid_cred_server(server);
-		if (cred != NULL)
-			break;
-	}
-	rcu_read_unlock();
-
-out:
 	return cred;
 }
 
@@ -1739,7 +1698,7 @@ static int nfs4_check_lease(struct nfs_client *clp)
 	cred = ops->get_state_renewal_cred_locked(clp);
 	spin_unlock(&clp->cl_lock);
 	if (cred == NULL) {
-		cred = nfs4_get_setclientid_cred(clp);
+		cred = nfs4_get_clid_cred(clp);
 		status = -ENOKEY;
 		if (cred == NULL)
 			goto out;
@@ -1811,7 +1770,7 @@ static int nfs4_establish_lease(struct nfs_client *clp)
 		clp->cl_mvops->reboot_recovery_ops;
 	int status;
 
-	cred = ops->get_clid_cred(clp);
+	cred = nfs4_get_clid_cred(clp);
 	if (cred == NULL)
 		return -ENOENT;
 	status = ops->establish_clid(clp, cred);
@@ -1885,7 +1844,7 @@ int nfs4_discover_server_trunking(struct nfs_client *clp,
 	mutex_lock(&nfs_clid_init_mutex);
 again:
 	status  = -ENOENT;
-	cred = ops->get_clid_cred(clp);
+	cred = nfs4_get_clid_cred(clp);
 	if (cred == NULL)
 		goto out_unlock;
 
@@ -2064,7 +2023,7 @@ static int nfs4_reset_session(struct nfs_client *clp)
 	if (!nfs4_has_session(clp))
 		return 0;
 	nfs4_begin_drain_session(clp);
-	cred = nfs4_get_exchange_id_cred(clp);
+	cred = nfs4_get_clid_cred(clp);
 	status = nfs4_proc_destroy_session(clp->cl_session, cred);
 	switch (status) {
 	case 0:
@@ -2107,7 +2066,7 @@ static int nfs4_bind_conn_to_session(struct nfs_client *clp)
 	if (!nfs4_has_session(clp))
 		return 0;
 	nfs4_begin_drain_session(clp);
-	cred = nfs4_get_exchange_id_cred(clp);
+	cred = nfs4_get_clid_cred(clp);
 	ret = nfs4_proc_bind_conn_to_session(clp, cred);
 	if (cred)
 		put_rpccred(cred);

[PATCH v1 02/19] NFS: Fix return type of nfs4_end_drain_session() stub

[Chuck Lever]

Clean up: when NFSv4.1 support is compiled out,
nfs4_end_drain_session() becomes a stub.  Make the synopsis of the
stub match the synopsis of the real version of the function.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4state.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 15fde8c..71fc85d 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -2087,7 +2087,7 @@ static int nfs4_bind_conn_to_session(struct nfs_client *clp)
 }
 #else /* CONFIG_NFS_V4_1 */
 static int nfs4_reset_session(struct nfs_client *clp) { return 0; }
-static int nfs4_end_drain_session(struct nfs_client *clp) { return 0; }
+static void nfs4_end_drain_session(struct nfs_client *clp) { }
 
 static int nfs4_bind_conn_to_session(struct nfs_client *clp)
 {

[PATCH v1 03/19] NFS: Introduce a vector of migration recovery ops

[Chuck Lever]

The differences between minor version 0 and minor version 1
migration will be abstracted by the addition of a set of migration
recovery ops.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h  |    4 ++++
 fs/nfs/nfs4proc.c |   10 ++++++++++
 2 files changed, 14 insertions(+)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 3bc554e..40457e2 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -52,6 +52,7 @@ struct nfs4_minor_version_ops {
 	const struct nfs4_state_recovery_ops *reboot_recovery_ops;
 	const struct nfs4_state_recovery_ops *nograce_recovery_ops;
 	const struct nfs4_state_maintenance_ops *state_renewal_ops;
+	const struct nfs4_mig_recovery_ops *mig_recovery_ops;
 };
 
 #define NFS_SEQID_CONFIRMED 1
@@ -204,6 +205,9 @@ struct nfs4_state_maintenance_ops {
 	int (*renew_lease)(struct nfs_client *, struct rpc_cred *);
 };
 
+struct nfs4_mig_recovery_ops {
+};
+
 extern const struct dentry_operations nfs4_dentry_operations;
 
 /* dir.c */
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index d74a2b5..2f6f3b9 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -6966,6 +6966,14 @@ static const struct nfs4_state_maintenance_ops nfs41_state_renewal_ops = {
 };
 #endif
 
+static const struct nfs4_mig_recovery_ops nfs40_mig_recovery_ops = {
+};
+
+#if defined(CONFIG_NFS_V4_1)
+static const struct nfs4_mig_recovery_ops nfs41_mig_recovery_ops = {
+};
+#endif	/* CONFIG_NFS_V4_1 */
+
 static const struct nfs4_minor_version_ops nfs_v4_0_minor_ops = {
 	.minor_version = 0,
 	.init_caps = NFS_CAP_READDIRPLUS
@@ -6979,6 +6987,7 @@ static const struct nfs4_minor_version_ops nfs_v4_0_minor_ops = {
 	.reboot_recovery_ops = &nfs40_reboot_recovery_ops,
 	.nograce_recovery_ops = &nfs40_nograce_recovery_ops,
 	.state_renewal_ops = &nfs40_state_renewal_ops,
+	.mig_recovery_ops = &nfs40_mig_recovery_ops,
 };
 
 #if defined(CONFIG_NFS_V4_1)
@@ -6997,6 +7006,7 @@ static const struct nfs4_minor_version_ops nfs_v4_1_minor_ops = {
 	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
 	.nograce_recovery_ops = &nfs41_nograce_recovery_ops,
 	.state_renewal_ops = &nfs41_state_renewal_ops,
+	.mig_recovery_ops = &nfs41_mig_recovery_ops,
 };
 #endif
 

[PATCH v1 04/19] NFS: Refactor nfs4_call_data_sequence()

[Chuck Lever]

Pull main logic out of nfs4_call_data_sequence() so it can
eventually be shared with the NFSv4.0 call_sync path.  The new
helper function is left behind the NFSv4.1 CONFIG switch for now to
keep compiler warning noise at a minimum.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4proc.c |   28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 2f6f3b9..e52d2b3 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -393,6 +393,23 @@ static void renew_lease(const struct nfs_server *server, unsigned long timestamp
 
 #if defined(CONFIG_NFS_V4_1)
 
+static int nfs4_run_rpc_task(struct rpc_clnt *clnt,
+			     struct rpc_message *msg,
+			     struct rpc_task_setup *task_setup)
+{
+	struct rpc_task *task;
+	int ret;
+
+	task = rpc_run_task(task_setup);
+	if (IS_ERR(task))
+		ret = PTR_ERR(task);
+	else {
+		ret = task->tk_status;
+		rpc_put_task(task);
+	}
+	return ret;
+}
+
 static void nfs41_sequence_free_slot(struct nfs4_sequence_res *res)
 {
 	struct nfs4_session *session;
@@ -674,8 +691,6 @@ static int nfs4_call_sync_sequence(struct rpc_clnt *clnt,
 				   struct nfs4_sequence_args *args,
 				   struct nfs4_sequence_res *res)
 {
-	int ret;
-	struct rpc_task *task;
 	struct nfs41_call_sync_data data = {
 		.seq_server = server,
 		.seq_args = args,
@@ -688,14 +703,7 @@ static int nfs4_call_sync_sequence(struct rpc_clnt *clnt,
 		.callback_data = &data
 	};
 
-	task = rpc_run_task(&task_setup);
-	if (IS_ERR(task))
-		ret = PTR_ERR(task);
-	else {
-		ret = task->tk_status;
-		rpc_put_task(task);
-	}
-	return ret;
+	return nfs4_run_rpc_task(clnt, msg, &task_setup);
 }
 
 #else

[PATCH v1 05/19] NFS: Rename nfs41_call_sync_data as a common data structure

[Chuck Lever]

Clean up: rename nfs41_call_sync_data for use as a data structure
common to all NFSv4 minor versions.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4proc.c |   18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index e52d2b3..d7b2c85 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -391,6 +391,12 @@ static void renew_lease(const struct nfs_server *server, unsigned long timestamp
 	do_renew_lease(server->nfs_client, timestamp);
 }
 
+struct nfs4_call_sync_data {
+	const struct nfs_server *seq_server;
+	struct nfs4_sequence_args *seq_args;
+	struct nfs4_sequence_res *seq_res;
+};
+
 #if defined(CONFIG_NFS_V4_1)
 
 static int nfs4_run_rpc_task(struct rpc_clnt *clnt,
@@ -657,15 +663,9 @@ out:
 	return ret;
 }
 
-struct nfs41_call_sync_data {
-	const struct nfs_server *seq_server;
-	struct nfs4_sequence_args *seq_args;
-	struct nfs4_sequence_res *seq_res;
-};
-
 static void nfs41_call_sync_prepare(struct rpc_task *task, void *calldata)
 {
-	struct nfs41_call_sync_data *data = calldata;
+	struct nfs4_call_sync_data *data = calldata;
 	struct nfs4_session *session = nfs4_get_session(data->seq_server);
 
 	dprintk("--> %s data->seq_server %p\n", __func__, data->seq_server);
@@ -675,7 +675,7 @@ static void nfs41_call_sync_prepare(struct rpc_task *task, void *calldata)
 
 static void nfs41_call_sync_done(struct rpc_task *task, void *calldata)
 {
-	struct nfs41_call_sync_data *data = calldata;
+	struct nfs4_call_sync_data *data = calldata;
 
 	nfs41_sequence_done(task, data->seq_res);
 }
@@ -691,7 +691,7 @@ static int nfs4_call_sync_sequence(struct rpc_clnt *clnt,
 				   struct nfs4_sequence_args *args,
 				   struct nfs4_sequence_res *res)
 {
-	struct nfs41_call_sync_data data = {
+	struct nfs4_call_sync_data data = {
 		.seq_server = server,
 		.seq_args = args,
 		.seq_res = res,

[PATCH v1 06/19] NFS: Clean up nfs4_setup_sequence()

[Chuck Lever]

The NFSv4.0-only version of nfs4_setup_sequence() is a static inline
function defined in a header file.  We're about to add a little more
to the NFSv4.0-only version, so move the NFSv4.0-only version into
nfs4proc.c next to the NFSv4.1 version.

There are no callsites outside of nfs4proc.c, so I removed the
header declarations.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h  |   11 -----------
 fs/nfs/nfs4proc.c |   16 ++++++++++++----
 2 files changed, 12 insertions(+), 15 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 40457e2..419fc87 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -251,9 +251,6 @@ static inline struct nfs4_session *nfs4_get_session(const struct nfs_server *ser
 	return server->nfs_client->cl_session;
 }
 
-extern int nfs4_setup_sequence(const struct nfs_server *server,
-		struct nfs4_sequence_args *args, struct nfs4_sequence_res *res,
-		struct rpc_task *task);
 extern int nfs41_setup_sequence(struct nfs4_session *session,
 		struct nfs4_sequence_args *args, struct nfs4_sequence_res *res,
 		struct rpc_task *task);
@@ -282,14 +279,6 @@ static inline struct nfs4_session *nfs4_get_session(const struct nfs_server *ser
 	return NULL;
 }
 
-static inline int nfs4_setup_sequence(const struct nfs_server *server,
-		struct nfs4_sequence_args *args, struct nfs4_sequence_res *res,
-		struct rpc_task *task)
-{
-	rpc_call_start(task);
-	return 0;
-}
-
 static inline bool
 is_ds_only_client(struct nfs_client *clp)
 {
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index d7b2c85..7b2e008 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -640,10 +640,10 @@ out_sleep:
 }
 EXPORT_SYMBOL_GPL(nfs41_setup_sequence);
 
-int nfs4_setup_sequence(const struct nfs_server *server,
-			struct nfs4_sequence_args *args,
-			struct nfs4_sequence_res *res,
-			struct rpc_task *task)
+static int nfs4_setup_sequence(const struct nfs_server *server,
+			       struct nfs4_sequence_args *args,
+			       struct nfs4_sequence_res *res,
+			       struct rpc_task *task)
 {
 	struct nfs4_session *session = nfs4_get_session(server);
 	int ret = 0;
@@ -717,6 +717,14 @@ static void nfs4_set_sequence_privileged(struct nfs4_sequence_args *args)
 {
 }
 
+static int nfs4_setup_sequence(const struct nfs_server *server,
+			       struct nfs4_sequence_args *args,
+			       struct nfs4_sequence_res *res,
+			       struct rpc_task *task)
+{
+	rpc_call_start(task);
+	return 0;
+}
 
 static int nfs4_sequence_done(struct rpc_task *task,
 			       struct nfs4_sequence_res *res)

[PATCH v1 07/19] NFS: Fix compiler warning in nfs_setup_sequence()

[Chuck Lever]

Clean up.  Squelch compiler warning:

linux/fs/nfs/nfs4proc.c: In function ‘nfs4_setup_sequence’:
linux/fs/nfs/nfs4proc.c:703:2: warning: signed and unsigned type in
	conditional expression [-Wsign-compare]

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4proc.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 7b2e008..5913e1d 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -655,7 +655,7 @@ static int nfs4_setup_sequence(const struct nfs_server *server,
 
 	dprintk("--> %s clp %p session %p sr_slot %d\n",
 		__func__, session->clp, session, res->sr_slot ?
-			res->sr_slot->slot_nr : -1);
+			(int)res->sr_slot->slot_nr : -1);
 
 	ret = nfs41_setup_sequence(session, args, res, task);
 out:

[PATCH v1 08/19] NFS: Use RPC callouts to start NFSv4.0 synchronous requests

[Chuck Lever]

In preparation for adding a transport blocking mechanism for
NFSv4.0, replace _nfs4_call_sync() with a common function that
handles synchronous RPC calls for all minor versions using RPC
callouts.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h  |    6 +-----
 fs/nfs/nfs4proc.c |   47 ++++++++++++++++++++++++++++++++++-------------
 2 files changed, 35 insertions(+), 18 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 419fc87..ae05978 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -38,17 +38,13 @@ struct nfs4_minor_version_ops {
 	u32	minor_version;
 	unsigned init_caps;
 
-	int	(*call_sync)(struct rpc_clnt *clnt,
-			struct nfs_server *server,
-			struct rpc_message *msg,
-			struct nfs4_sequence_args *args,
-			struct nfs4_sequence_res *res);
 	bool	(*match_stateid)(const nfs4_stateid *,
 			const nfs4_stateid *);
 	int	(*find_root_sec)(struct nfs_server *, struct nfs_fh *,
 			struct nfs_fsinfo *);
 	int	(*free_lock_state)(struct nfs_server *,
 			struct nfs4_lock_state *);
+	const struct rpc_call_ops *call_sync_ops;
 	const struct nfs4_state_recovery_ops *reboot_recovery_ops;
 	const struct nfs4_state_recovery_ops *nograce_recovery_ops;
 	const struct nfs4_state_maintenance_ops *state_renewal_ops;
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 5913e1d..0c1fa13 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -397,8 +397,6 @@ struct nfs4_call_sync_data {
 	struct nfs4_sequence_res *seq_res;
 };
 
-#if defined(CONFIG_NFS_V4_1)
-
 static int nfs4_run_rpc_task(struct rpc_clnt *clnt,
 			     struct rpc_message *msg,
 			     struct rpc_task_setup *task_setup)
@@ -416,6 +414,8 @@ static int nfs4_run_rpc_task(struct rpc_clnt *clnt,
 	return ret;
 }
 
+#if defined(CONFIG_NFS_V4_1)
+
 static void nfs41_sequence_free_slot(struct nfs4_sequence_res *res)
 {
 	struct nfs4_session *session;
@@ -733,16 +733,26 @@ static int nfs4_sequence_done(struct rpc_task *task,
 }
 #endif /* CONFIG_NFS_V4_1 */
 
-static
-int _nfs4_call_sync(struct rpc_clnt *clnt,
-		    struct nfs_server *server,
-		    struct rpc_message *msg,
-		    struct nfs4_sequence_args *args,
-		    struct nfs4_sequence_res *res)
+static void nfs40_call_sync_prepare(struct rpc_task *task, void *calldata)
+{
+	struct nfs4_call_sync_data *data = calldata;
+	const struct nfs_server *server = data->seq_server;
+
+	nfs4_setup_sequence(server, data->seq_args, data->seq_res, task);
+}
+
+static void nfs40_call_sync_done(struct rpc_task *task, void *calldata)
 {
-	return rpc_call_sync(clnt, msg, 0);
+	struct nfs4_call_sync_data *data = calldata;
+
+	nfs4_sequence_done(task, data->seq_res);
 }
 
+static const struct rpc_call_ops nfs40_call_sync_ops = {
+	.rpc_call_prepare = nfs40_call_sync_prepare,
+	.rpc_call_done = nfs40_call_sync_done,
+};
+
 static
 int nfs4_call_sync(struct rpc_clnt *clnt,
 		   struct nfs_server *server,
@@ -751,9 +761,20 @@ int nfs4_call_sync(struct rpc_clnt *clnt,
 		   struct nfs4_sequence_res *res,
 		   int cache_reply)
 {
+	struct nfs4_call_sync_data data = {
+		.seq_server	= server,
+		.seq_args	= args,
+		.seq_res	= res,
+	};
+	struct rpc_task_setup task_setup = {
+		.rpc_client	= clnt,
+		.rpc_message	= msg,
+		.callback_ops	= server->nfs_client->cl_mvops->call_sync_ops,
+		.callback_data	= &data,
+	};
+
 	nfs41_init_sequence(args, res, cache_reply);
-	return server->nfs_client->cl_mvops->call_sync(clnt, server, msg,
-						args, res);
+	return nfs4_run_rpc_task(clnt, msg, &task_setup);
 }
 
 static void update_changeattr(struct inode *dir, struct nfs4_change_info *cinfo)
@@ -6996,10 +7017,10 @@ static const struct nfs4_minor_version_ops nfs_v4_0_minor_ops = {
 		| NFS_CAP_ATOMIC_OPEN
 		| NFS_CAP_CHANGE_ATTR
 		| NFS_CAP_POSIX_LOCK,
-	.call_sync = _nfs4_call_sync,
 	.match_stateid = nfs4_match_stateid,
 	.find_root_sec = nfs4_find_root_sec,
 	.free_lock_state = nfs4_release_lockowner,
+	.call_sync_ops = &nfs40_call_sync_ops,
 	.reboot_recovery_ops = &nfs40_reboot_recovery_ops,
 	.nograce_recovery_ops = &nfs40_nograce_recovery_ops,
 	.state_renewal_ops = &nfs40_state_renewal_ops,
@@ -7015,10 +7036,10 @@ static const struct nfs4_minor_version_ops nfs_v4_1_minor_ops = {
 		| NFS_CAP_POSIX_LOCK
 		| NFS_CAP_STATEID_NFSV41
 		| NFS_CAP_ATOMIC_OPEN_V1,
-	.call_sync = nfs4_call_sync_sequence,
 	.match_stateid = nfs41_match_stateid,
 	.find_root_sec = nfs41_find_root_sec,
 	.free_lock_state = nfs41_free_lock_state,
+	.call_sync_ops = &nfs41_call_sync_ops,
 	.reboot_recovery_ops = &nfs41_reboot_recovery_ops,
 	.nograce_recovery_ops = &nfs41_nograce_recovery_ops,
 	.state_renewal_ops = &nfs41_state_renewal_ops,

[PATCH v1 09/19] NFS: Add a "struct nfs_server *" argument to nfs4_sequence_done()

[Chuck Lever]

For NFSv4.1, a session slot table reference is passed to
nfs4_sequence_done().

For NFSv4.0, transport state is managed in the nfs_client that owns
this nfs_server.  The nfs_server struct must be passed to
nfs4_sequence_done() to make it available to the functions dealing
with transport blocking.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4proc.c |   46 ++++++++++++++++++++++++++--------------------
 1 file changed, 26 insertions(+), 20 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 0c1fa13..20768e8 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -553,8 +553,9 @@ out_retry:
 	return 0;
 }
 
-static int nfs4_sequence_done(struct rpc_task *task,
-			       struct nfs4_sequence_res *res)
+static int nfs4_sequence_done(const struct nfs_server *server,
+			      struct rpc_task *task,
+			      struct nfs4_sequence_res *res)
 {
 	if (res->sr_slot == NULL)
 		return 1;
@@ -726,8 +727,9 @@ static int nfs4_setup_sequence(const struct nfs_server *server,
 	return 0;
 }
 
-static int nfs4_sequence_done(struct rpc_task *task,
-			       struct nfs4_sequence_res *res)
+static int nfs4_sequence_done(const struct nfs_server *server,
+			      struct rpc_task *task,
+			      struct nfs4_sequence_res *res)
 {
 	return 1;
 }
@@ -745,7 +747,7 @@ static void nfs40_call_sync_done(struct rpc_task *task, void *calldata)
 {
 	struct nfs4_call_sync_data *data = calldata;
 
-	nfs4_sequence_done(task, data->seq_res);
+	nfs4_sequence_done(data->seq_server, task, data->seq_res);
 }
 
 static const struct rpc_call_ops nfs40_call_sync_ops = {
@@ -1622,7 +1624,7 @@ unlock_no_action:
 out_no_action:
 	task->tk_action = NULL;
 out_wait:
-	nfs4_sequence_done(task, &data->o_res.seq_res);
+	nfs4_sequence_done(data->o_arg.server, task, &data->o_res.seq_res);
 }
 
 static void nfs4_open_done(struct rpc_task *task, void *calldata)
@@ -1631,7 +1633,7 @@ static void nfs4_open_done(struct rpc_task *task, void *calldata)
 
 	data->rpc_status = task->tk_status;
 
-	if (!nfs4_sequence_done(task, &data->o_res.seq_res))
+	if (!nfs4_sequence_done(data->o_arg.server, task, &data->o_res.seq_res))
 		return;
 
 	if (task->tk_status == 0) {
@@ -2294,7 +2296,7 @@ static void nfs4_close_done(struct rpc_task *task, void *data)
 	struct nfs_server *server = NFS_SERVER(calldata->inode);
 
 	dprintk("%s: begin!\n", __func__);
-	if (!nfs4_sequence_done(task, &calldata->res.seq_res))
+	if (!nfs4_sequence_done(server, task, &calldata->res.seq_res))
 		return;
         /* hmm. we are done with the inode, and in the process of freeing
 	 * the state_owner. we keep this around to process errors
@@ -2381,7 +2383,7 @@ static void nfs4_close_prepare(struct rpc_task *task, void *data)
 out_no_action:
 	task->tk_action = NULL;
 out_wait:
-	nfs4_sequence_done(task, &calldata->res.seq_res);
+	nfs4_sequence_done(NFS_SERVER(inode), task, &calldata->res.seq_res);
 }
 
 static const struct rpc_call_ops nfs4_close_ops = {
@@ -3149,7 +3151,7 @@ static int nfs4_proc_unlink_done(struct rpc_task *task, struct inode *dir)
 {
 	struct nfs_removeres *res = task->tk_msg.rpc_resp;
 
-	if (!nfs4_sequence_done(task, &res->seq_res))
+	if (!nfs4_sequence_done(res->server, task, &res->seq_res))
 		return 0;
 	if (nfs4_async_handle_error(task, res->server, NULL) == -EAGAIN)
 		return 0;
@@ -3181,7 +3183,7 @@ static int nfs4_proc_rename_done(struct rpc_task *task, struct inode *old_dir,
 {
 	struct nfs_renameres *res = task->tk_msg.rpc_resp;
 
-	if (!nfs4_sequence_done(task, &res->seq_res))
+	if (!nfs4_sequence_done(res->server, task, &res->seq_res))
 		return 0;
 	if (nfs4_async_handle_error(task, res->server, NULL) == -EAGAIN)
 		return 0;
@@ -3705,10 +3707,11 @@ static bool nfs4_read_stateid_changed(struct rpc_task *task,
 
 static int nfs4_read_done(struct rpc_task *task, struct nfs_read_data *data)
 {
+	struct nfs_server *server = NFS_SERVER(data->header->inode);
 
 	dprintk("--> %s\n", __func__);
 
-	if (!nfs4_sequence_done(task, &data->res.seq_res))
+	if (!nfs4_sequence_done(server, task, &data->res.seq_res))
 		return -EAGAIN;
 	if (nfs4_read_stateid_changed(task, &data->args))
 		return -EAGAIN;
@@ -3753,7 +3756,6 @@ static int nfs4_write_done_cb(struct rpc_task *task, struct nfs_write_data *data
 static bool nfs4_write_stateid_changed(struct rpc_task *task,
 		struct nfs_writeargs *args)
 {
-
 	if (!nfs4_error_stateid_expired(task->tk_status) ||
 		nfs4_stateid_is_current(&args->stateid,
 				args->context,
@@ -3766,7 +3768,9 @@ static bool nfs4_write_stateid_changed(struct rpc_task *task,
 
 static int nfs4_write_done(struct rpc_task *task, struct nfs_write_data *data)
 {
-	if (!nfs4_sequence_done(task, &data->res.seq_res))
+	struct nfs_server *server = NFS_SERVER(data->header->inode);
+
+	if (!nfs4_sequence_done(server, task, &data->res.seq_res))
 		return -EAGAIN;
 	if (nfs4_write_stateid_changed(task, &data->args))
 		return -EAGAIN;
@@ -3839,7 +3843,9 @@ static int nfs4_commit_done_cb(struct rpc_task *task, struct nfs_commit_data *da
 
 static int nfs4_commit_done(struct rpc_task *task, struct nfs_commit_data *data)
 {
-	if (!nfs4_sequence_done(task, &data->res.seq_res))
+	struct nfs_server *server = NFS_SERVER(data->inode);
+
+	if (!nfs4_sequence_done(server, task, &data->res.seq_res))
 		return -EAGAIN;
 	return data->commit_done_cb(task, data);
 }
@@ -4439,7 +4445,7 @@ static void nfs4_delegreturn_done(struct rpc_task *task, void *calldata)
 {
 	struct nfs4_delegreturndata *data = calldata;
 
-	if (!nfs4_sequence_done(task, &data->res.seq_res))
+	if (!nfs4_sequence_done(data->res.server, task, &data->res.seq_res))
 		return;
 
 	switch (task->tk_status) {
@@ -4690,7 +4696,7 @@ static void nfs4_locku_done(struct rpc_task *task, void *data)
 {
 	struct nfs4_unlockdata *calldata = data;
 
-	if (!nfs4_sequence_done(task, &calldata->res.seq_res))
+	if (!nfs4_sequence_done(calldata->server, task, &calldata->res.seq_res))
 		return;
 	switch (task->tk_status) {
 		case 0:
@@ -4730,7 +4736,7 @@ static void nfs4_locku_prepare(struct rpc_task *task, void *data)
 out_no_action:
 	task->tk_action = NULL;
 out_wait:
-	nfs4_sequence_done(task, &calldata->res.seq_res);
+	nfs4_sequence_done(calldata->server, task, &calldata->res.seq_res);
 }
 
 static const struct rpc_call_ops nfs4_locku_ops = {
@@ -4905,7 +4911,7 @@ out_release_open_seqid:
 out_release_lock_seqid:
 	nfs_release_seqid(data->arg.lock_seqid);
 out_wait:
-	nfs4_sequence_done(task, &data->res.seq_res);
+	nfs4_sequence_done(data->server, task, &data->res.seq_res);
 	dprintk("%s: done!, ret = %d\n", __func__, data->rpc_status);
 }
 
@@ -4915,7 +4921,7 @@ static void nfs4_lock_done(struct rpc_task *task, void *calldata)
 
 	dprintk("%s: begin!\n", __func__);
 
-	if (!nfs4_sequence_done(task, &data->res.seq_res))
+	if (!nfs4_sequence_done(data->server, task, &data->res.seq_res))
 		return;
 
 	data->rpc_status = task->tk_status;

[PATCH v1 10/19] NFS: Implement a transport blocking scheme for migration

[Chuck Lever]

When an NFSv4 migration event occurs, an NFS client must redirect
a mount point's underlying RPC transport to a different NFS server.

Before performing this redirection, the client must plug the RPC
transport to prevent the initiation of new requests, and allow
any pending requests or the migrating FSID to complete.

Only then is it safe to swap in a transport to the destination
server.  This is like NFSv4.1 session draining, so we adopt a
similar approach.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h          |    1 +
 fs/nfs/nfs4client.c       |    4 ++++
 fs/nfs/nfs4proc.c         |   43 ++++++++++++++++++++++++++++++++++++++-----
 fs/nfs/nfs4state.c        |   19 +++++++++++++++++++
 include/linux/nfs_fs_sb.h |    4 ++++
 5 files changed, 66 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index ae05978..b9c839e 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -29,6 +29,7 @@ enum nfs4_client_state {
 	NFS4CLNT_SERVER_SCOPE_MISMATCH,
 	NFS4CLNT_PURGE_STATE,
 	NFS4CLNT_BIND_CONN_TO_SESSION,
+	NFS4CLNT_BLOCK_XPRT,
 };
 
 #define NFS4_RENEW_TIMEOUT		0x01
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index 3aec2de..4d208e0 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -72,6 +72,10 @@ struct nfs_client *nfs4_alloc_client(const struct nfs_client_initdata *cl_init)
 	clp->cl_state = 1 << NFS4CLNT_LEASE_EXPIRED;
 	clp->cl_minorversion = cl_init->minorversion;
 	clp->cl_mvops = nfs_v4_minor_ops[cl_init->minorversion];
+	if (clp->cl_minorversion == 0) {
+		init_completion(&clp->cl_xpcomplete);
+		rpc_init_wait_queue(&clp->cl_xpwaitq, "NFSv4.0 xprt");
+	}
 	return clp;
 
 error:
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 20768e8..bb5100a 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -414,6 +414,40 @@ static int nfs4_run_rpc_task(struct rpc_clnt *clnt,
 	return ret;
 }
 
+/*
+ * Returns -EAGAIN if task must wait for migration recovery;
+ * otherwise, zero is returned.
+ */
+static int nfs4_transport_setup(const struct nfs_server *server,
+				struct rpc_task *task)
+{
+	struct nfs_client *clp = server->nfs_client;
+
+	spin_lock(&clp->cl_lock);
+	if (test_bit(NFS4CLNT_BLOCK_XPRT, &clp->cl_state)) {
+		rpc_sleep_on(&clp->cl_xpwaitq, task, NULL);
+		spin_unlock(&clp->cl_lock);
+		dprintk("--> %s transport blocked for server %s\n",
+			__func__, clp->cl_hostname);
+		return -EAGAIN;
+	}
+	atomic_inc(&clp->cl_xppending);
+	spin_unlock(&clp->cl_lock);
+
+	rpc_call_start(task);
+	return 0;
+}
+
+static int nfs4_transport_done(const struct nfs_server *server)
+{
+	struct nfs_client *clp = server->nfs_client;
+
+	if (atomic_dec_and_test(&clp->cl_xppending) &&
+	    test_bit(NFS4CLNT_BLOCK_XPRT, &clp->cl_state))
+		complete(&clp->cl_xpcomplete);
+	return 1;
+}
+
 #if defined(CONFIG_NFS_V4_1)
 
 static void nfs41_sequence_free_slot(struct nfs4_sequence_res *res)
@@ -558,7 +592,7 @@ static int nfs4_sequence_done(const struct nfs_server *server,
 			      struct nfs4_sequence_res *res)
 {
 	if (res->sr_slot == NULL)
-		return 1;
+		return nfs4_transport_done(server);
 	return nfs41_sequence_done(task, res);
 }
 
@@ -650,7 +684,7 @@ static int nfs4_setup_sequence(const struct nfs_server *server,
 	int ret = 0;
 
 	if (session == NULL) {
-		rpc_call_start(task);
+		ret = nfs4_transport_setup(server, task);
 		goto out;
 	}
 
@@ -723,15 +757,14 @@ static int nfs4_setup_sequence(const struct nfs_server *server,
 			       struct nfs4_sequence_res *res,
 			       struct rpc_task *task)
 {
-	rpc_call_start(task);
-	return 0;
+	return nfs4_transport_setup(server, task);
 }
 
 static int nfs4_sequence_done(const struct nfs_server *server,
 			      struct rpc_task *task,
 			      struct nfs4_sequence_res *res)
 {
-	return 1;
+	return nfs4_transport_done(server);
 }
 #endif /* CONFIG_NFS_V4_1 */
 
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 71fc85d..ae482b2 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -355,6 +355,25 @@ int nfs41_discover_server_trunking(struct nfs_client *clp,
 
 #endif /* CONFIG_NFS_V4_1 */
 
+static void nfs4_end_drain_xprt(struct nfs_client *clp)
+{
+	if (test_and_clear_bit(NFS4CLNT_BLOCK_XPRT, &clp->cl_state))
+		rpc_wake_up(&clp->cl_xpwaitq);
+}
+
+static int nfs4_begin_drain_xprt(struct nfs_client *clp)
+{
+	set_bit(NFS4CLNT_BLOCK_XPRT, &clp->cl_state);
+	spin_lock(&clp->cl_lock);
+	if (atomic_read(&clp->cl_xppending)) {
+		INIT_COMPLETION(clp->cl_xpcomplete);
+		spin_unlock(&clp->cl_lock);
+		return wait_for_completion_interruptible(&clp->cl_xpcomplete);
+	}
+	spin_unlock(&clp->cl_lock);
+	return 0;
+}
+
 /**
  * nfs4_get_clid_cred - Acquire credential for a setclientid operation
  * @clp: client state handle
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 3b7fa2a..ad47c90 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -78,6 +78,10 @@ struct nfs_client {
 	u32			cl_cb_ident;	/* v4.0 callback identifier */
 	const struct nfs4_minor_version_ops *cl_mvops;
 
+	atomic_t		cl_xppending;
+	struct completion	cl_xpcomplete;
+	struct rpc_wait_queue	cl_xpwaitq;
+
 	/* The sequence id to use for the next CREATE_SESSION */
 	u32			cl_seqid;
 	/* The flags used for obtaining the clientid during EXCHANGE_ID */

[PATCH v1 11/19] SUNRPC: Add a helper to switch the transport of an rpc_clnt

[Chuck Lever]

From: Trond Myklebust <Trond.Myklebust@netapp.com>

Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
[ cel: forward ported to 3.10 ]
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 include/linux/sunrpc/clnt.h |    4 ++
 net/sunrpc/clnt.c           |   83 ++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 81 insertions(+), 6 deletions(-)

diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
index bfe11be..29bb018 100644
--- a/include/linux/sunrpc/clnt.h
+++ b/include/linux/sunrpc/clnt.h
@@ -134,6 +134,10 @@ void rpc_task_reset_client(struct rpc_task *task, struct rpc_clnt *clnt);
 struct rpc_clnt *rpc_clone_client(struct rpc_clnt *);
 struct rpc_clnt *rpc_clone_client_set_auth(struct rpc_clnt *,
 				rpc_authflavor_t);
+int		rpc_switch_client_transport(struct rpc_clnt *,
+				struct xprt_create *,
+				const struct rpc_timeout *);
+
 void		rpc_shutdown_client(struct rpc_clnt *);
 void		rpc_release_client(struct rpc_clnt *);
 void		rpc_task_release_client(struct rpc_task *);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 5a750b9..e4e8ab0 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -25,6 +25,7 @@
 #include <linux/namei.h>
 #include <linux/mount.h>
 #include <linux/slab.h>
+#include <linux/rcupdate.h>
 #include <linux/utsname.h>
 #include <linux/workqueue.h>
 #include <linux/in.h>
@@ -289,12 +290,35 @@ static void rpc_clnt_set_nodename(struct rpc_clnt *clnt, const char *nodename)
 	memcpy(clnt->cl_nodename, nodename, clnt->cl_nodelen);
 }
 
+static void rpc_clnt_set_transport(struct rpc_clnt *clnt,
+		struct rpc_xprt *xprt,
+		const struct rpc_timeout *timeout)
+{
+	struct rpc_xprt *old;
+
+	spin_lock(&clnt->cl_lock);
+	old = clnt->cl_xprt;
+
+	if (!xprt_bound(xprt))
+		clnt->cl_autobind = 1;
+
+	clnt->cl_timeout = timeout;
+	rcu_assign_pointer(clnt->cl_xprt, xprt);
+	spin_unlock(&clnt->cl_lock);
+
+	if (old != NULL) {
+		synchronize_rcu();
+		xprt_put(old);
+	}
+}
+
 static struct rpc_clnt * rpc_new_client(const struct rpc_create_args *args, struct rpc_xprt *xprt)
 {
 	const struct rpc_program *program = args->program;
 	const struct rpc_version *version;
 	struct rpc_clnt		*clnt = NULL;
 	struct rpc_auth		*auth;
+	const struct rpc_timeout *timeout;
 	int err;
 
 	/* sanity check the name before trying to print it */
@@ -318,7 +342,6 @@ static struct rpc_clnt * rpc_new_client(const struct rpc_create_args *args, stru
 		goto out_err;
 	clnt->cl_parent = clnt;
 
-	rcu_assign_pointer(clnt->cl_xprt, xprt);
 	clnt->cl_procinfo = version->procs;
 	clnt->cl_maxproc  = version->nrprocs;
 	clnt->cl_protname = program->name;
@@ -333,16 +356,15 @@ static struct rpc_clnt * rpc_new_client(const struct rpc_create_args *args, stru
 	INIT_LIST_HEAD(&clnt->cl_tasks);
 	spin_lock_init(&clnt->cl_lock);
 
-	if (!xprt_bound(xprt))
-		clnt->cl_autobind = 1;
-
-	clnt->cl_timeout = xprt->timeout;
+	timeout = xprt->timeout;
 	if (args->timeout != NULL) {
 		memcpy(&clnt->cl_timeout_default, args->timeout,
 				sizeof(clnt->cl_timeout_default));
-		clnt->cl_timeout = &clnt->cl_timeout_default;
+		timeout = &clnt->cl_timeout_default;
 	}
 
+	rpc_clnt_set_transport(clnt, xprt, timeout);
+
 	clnt->cl_rtt = &clnt->cl_rtt_default;
 	rpc_init_rtt(&clnt->cl_rtt_default, clnt->cl_timeout->to_initval);
 	clnt->cl_principal = NULL;
@@ -573,6 +595,55 @@ rpc_clone_client_set_auth(struct rpc_clnt *clnt, rpc_authflavor_t flavor)
 }
 EXPORT_SYMBOL_GPL(rpc_clone_client_set_auth);
 
+/**
+ * rpc_switch_client_transport: switch the RPC transport on the fly
+ * @clnt: pointer to a struct rpc_clnt
+ * @args: pointer to the new transport arguments
+ * @timeout: pointer to the new timeout parameters
+ *
+ * This function allows the caller to switch the RPC transport for the
+ * rpc_clnt structure 'clnt' to allow it to connect to a mirrored NFS
+ * server, for instance.  It assumes that the caller has ensured that
+ * there are no active tasks by using some form of locking.
+ *
+ * Returns zero on success, otherwise a negative errno is returned.
+ */
+int rpc_switch_client_transport(struct rpc_clnt *clnt,
+		struct xprt_create *args,
+		const struct rpc_timeout *timeout)
+{
+	struct rpc_xprt *xprt;
+	struct rpc_auth *auth;
+	rpc_authflavor_t pseudoflavor;
+
+	xprt = xprt_create_transport(args);
+	if (IS_ERR(xprt)) {
+		dprintk("RPC:      failed to switch xprt for clnt %p\n", clnt);
+		/* XXX: this leaves the clnt in its previous state */
+		return PTR_ERR(xprt);
+	}
+
+	pseudoflavor = clnt->cl_auth->au_flavor;
+
+	rpc_clnt_set_transport(clnt, xprt, timeout);
+
+	/*
+	 * Note: we must always create a new rpc_auth cache
+	 * when switching to a different server!  GSS contexts
+	 * in particular are between a single client and
+	 * server.
+	 */
+	auth = rpcauth_create(pseudoflavor, clnt);
+	if (IS_ERR(auth)) {
+		printk(KERN_INFO "RPC: Couldn't create auth handle (flavor %u)\n",
+				pseudoflavor);
+		/* XXX: this leaves the clnt in a broken state */
+		return PTR_ERR(auth);
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(rpc_switch_client_transport);
+
 /*
  * Kill all tasks for the given client.
  * XXX: kill their descendants as well?

[PATCH v1 12/19] NFS: Add a super_block backpointer to the nfs_server struct

[Chuck Lever]

NFS_SB() gives the pointer to an nfs_server struct, given a pointer
to a super_block.  But we have no way to go back the other way.

Add a super_block backpointer pointer so that, given an nfs_server
struct, it is easy to get to the root dentry, or access the list of
inodes, associated with an nfs_server.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/super.c            |    1 +
 include/linux/nfs_fs_sb.h |    1 +
 2 files changed, 2 insertions(+)

diff --git a/fs/nfs/super.c b/fs/nfs/super.c
index 2d7525f..455d56a 100644
--- a/fs/nfs/super.c
+++ b/fs/nfs/super.c
@@ -2463,6 +2463,7 @@ struct dentry *nfs_fs_mount_common(struct nfs_server *server,
 			mntroot = ERR_PTR(error);
 			goto error_splat_bdi;
 		}
+		server->super = s;
 	}
 
 	if (!s->s_root) {
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index ad47c90..84d8f0c 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -138,6 +138,7 @@ struct nfs_server {
 	__u64			maxfilesize;	/* maximum file size */
 	struct timespec		time_delta;	/* smallest time granularity */
 	unsigned long		mount_time;	/* when this fs was mounted */
+	struct super_block	*super;		/* VFS super block */
 	dev_t			s_dev;		/* superblock dev numbers */
 
 #ifdef CONFIG_NFS_FSCACHE

[PATCH v1 13/19] NFS: Add functions to swap transports during migration recovery

[Chuck Lever]

Introduce functions that can walk through an array of returned
fs_locations information and connect a transport to one of the
destination servers listed therein.

Note that NFS minor version 1 introduces "fs_locations_info" which
extends the locations array sorting criteria available to clients.
This is not supported yet.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/client.c        |    3 +
 fs/nfs/internal.h      |    3 +
 fs/nfs/nfs4_fs.h       |    2 +
 fs/nfs/nfs4client.c    |   78 +++++++++++++++++++++++++++++++++
 fs/nfs/nfs4namespace.c |  114 ++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 199 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index dbb65fb..5b7c60a 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -945,7 +945,7 @@ void nfs_server_insert_lists(struct nfs_server *server)
 }
 EXPORT_SYMBOL_GPL(nfs_server_insert_lists);
 
-static void nfs_server_remove_lists(struct nfs_server *server)
+void nfs_server_remove_lists(struct nfs_server *server)
 {
 	struct nfs_client *clp = server->nfs_client;
 	struct nfs_net *nn;
@@ -962,6 +962,7 @@ static void nfs_server_remove_lists(struct nfs_server *server)
 
 	synchronize_rcu();
 }
+EXPORT_SYMBOL_GPL(nfs_server_remove_lists);
 
 /*
  * Allocate and initialise a server record
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 91e59a3..3de40da 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -153,6 +153,7 @@ struct nfs_client *nfs_get_client(const struct nfs_client_initdata *,
 				  rpc_authflavor_t);
 int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *, struct nfs_fattr *);
 void nfs_server_insert_lists(struct nfs_server *);
+void nfs_server_remove_lists(struct nfs_server *);
 void nfs_init_timeout_values(struct rpc_timeout *, int, unsigned int, unsigned int);
 int nfs_init_server_rpcclient(struct nfs_server *, const struct rpc_timeout *t,
 		rpc_authflavor_t);
@@ -173,6 +174,8 @@ extern struct nfs_server *nfs4_create_server(
 					struct nfs_subversion *);
 extern struct nfs_server *nfs4_create_referral_server(struct nfs_clone_mount *,
 						      struct nfs_fh *);
+extern int nfs4_update_server(struct nfs_server *server, const char *hostname,
+					struct sockaddr *sap, size_t salen);
 extern void nfs_free_server(struct nfs_server *server);
 extern struct nfs_server *nfs_clone_server(struct nfs_server *,
 					   struct nfs_fh *,
diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index b9c839e..4b83ffc 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -219,6 +219,8 @@ rpc_authflavor_t nfs_find_best_sec(struct nfs4_secinfo_flavors *);
 struct rpc_clnt *nfs4_create_sec_client(struct rpc_clnt *, struct inode *, struct qstr *);
 struct vfsmount *nfs4_submount(struct nfs_server *, struct dentry *,
 			       struct nfs_fh *, struct nfs_fattr *);
+int nfs4_replace_transport(struct nfs_server *server,
+				const struct nfs4_fs_locations *locations);
 
 /* nfs4proc.c */
 extern int nfs4_proc_setclientid(struct nfs_client *, u32, unsigned short, struct rpc_cred *, struct nfs4_setclientid_res *);
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index 4d208e0..dadd118 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -920,3 +920,81 @@ error:
 	dprintk("<-- nfs4_create_referral_server() = error %d\n", error);
 	return ERR_PTR(error);
 }
+
+/**
+ * nfs4_update_server - Replace transitioned nfs_server's transport
+ *
+ * @server: FSID whose transport will be replaced
+ * @hostname: new end-point's hostname
+ * @sap: new end-point's socket address
+ * @salen: size of "sap"
+ *
+ * Returns zero on success, or a negative errno value.
+ */
+int nfs4_update_server(struct nfs_server *server, const char *hostname,
+		       struct sockaddr *sap, size_t salen)
+{
+	struct nfs_client *clp = server->nfs_client;
+	struct rpc_clnt *clnt = server->client;
+	struct xprt_create xargs = {
+		.ident		= clp->cl_proto,
+		.net		= &init_net,
+		.dstaddr	= sap,
+		.addrlen	= salen,
+		.servername	= hostname,
+	};
+	char buf[INET6_ADDRSTRLEN + 1];
+	struct sockaddr_storage address;
+	struct sockaddr *localaddr = (struct sockaddr *)&address;
+	struct nfs_fh *mntfh;
+	int error;
+
+	dprintk("--> %s: move FSID %llx:%llx to \"%s\")\n", __func__,
+			(unsigned long long)server->fsid.major,
+			(unsigned long long)server->fsid.minor,
+			hostname);
+
+	error = rpc_switch_client_transport(clnt, &xargs, clnt->cl_timeout);
+	if (error != 0) {
+		dprintk("<-- %s(): rpc_switch_client_transport returned %d\n",
+			__func__, error);
+		goto out;
+	}
+
+	error = rpc_localaddr(clnt, localaddr, sizeof(address));
+	if (error != 0) {
+		dprintk("<-- %s(): rpc_localaddr returned %d\n",
+			__func__, error);
+		goto out;
+	}
+
+	error = -EAFNOSUPPORT;
+	if (rpc_ntop(localaddr, buf, sizeof(buf)) == 0) {
+		dprintk("<-- %s(): rpc_ntop returned %d\n",
+			__func__, error);
+		goto out;
+	}
+
+	nfs_server_remove_lists(server);
+	error = nfs4_set_client(server, hostname, sap, salen, buf,
+				clp->cl_rpcclient->cl_auth->au_flavor,
+				clp->cl_proto, clnt->cl_timeout,
+				clp->cl_minorversion, clp->cl_net);
+	if (error != 0) {
+		dprintk("<-- %s(): nfs4_set_client returned %d\n",
+			__func__, error);
+		goto out;
+	}
+
+	if (server->nfs_client->cl_hostname == NULL)
+		server->nfs_client->cl_hostname = kstrdup(hostname, GFP_KERNEL);
+
+	mntfh = NFS_FH(server->super->s_root->d_inode);
+	error = nfs4_server_common_setup(server, mntfh);
+	if (error < 0)
+		goto out;
+
+	dprintk("<-- %s() succeeded\n", __func__);
+out:
+	return error;
+}
diff --git a/fs/nfs/nfs4namespace.c b/fs/nfs/nfs4namespace.c
index cdb0b41..2459e60 100644
--- a/fs/nfs/nfs4namespace.c
+++ b/fs/nfs/nfs4namespace.c
@@ -387,3 +387,117 @@ struct vfsmount *nfs4_submount(struct nfs_server *server, struct dentry *dentry,
 	rpc_shutdown_client(client);
 	return mnt;
 }
+
+/*
+ * Try one location from the fs_locations array.
+ *
+ * Returns zero on success, or a negative errno value.
+ */
+static int nfs4_try_replacing_one_location(struct nfs_server *server,
+		char *page, char *page2,
+		const struct nfs4_fs_location *location)
+{
+	const size_t addr_bufsize = sizeof(struct sockaddr_storage);
+	struct sockaddr *sap;
+	unsigned int s;
+	size_t salen;
+	int error;
+
+	dprintk("--> %s(%llx:%llx)\n", __func__,
+			(unsigned long long)server->fsid.major,
+			(unsigned long long)server->fsid.minor);
+
+	error = -ENOMEM;
+	sap = kmalloc(addr_bufsize, GFP_KERNEL);
+	if (sap == NULL)
+		goto out;
+
+	error = -ENOENT;
+	for (s = 0; s < location->nservers; s++) {
+		const struct nfs4_string *buf = &location->servers[s];
+		char *hostname;
+
+		if (buf->len <= 0 || buf->len > PAGE_SIZE)
+			continue;
+
+		if (memchr(buf->data, IPV6_SCOPE_DELIMITER, buf->len) != NULL)
+			continue;
+
+		salen = nfs_parse_server_name(buf->data, buf->len,
+						sap, addr_bufsize, server);
+		if (salen == 0)
+			continue;
+		rpc_set_port(sap, NFS_PORT);
+
+		error = -ENOMEM;
+		hostname = kstrndup(buf->data, buf->len, GFP_KERNEL);
+		if (hostname == NULL)
+			break;
+
+		error = nfs4_update_server(server, hostname, sap, salen);
+		kfree(hostname);
+		if (error == 0)
+			break;
+	}
+
+	kfree(sap);
+out:
+	dprintk("<-- %s() = %d\n", __func__, error);
+	return error;
+}
+
+/**
+ * nfs4_replace_transport - set up transport to destination server
+ *
+ * @server: export being migrated
+ * @locations: fs_locations array
+ *
+ * Returns zero on success, or a negative errno value.
+ *
+ * The client tries all the entries in the "locations" array, in the
+ * order returned by the server, until one works or the end of the
+ * array is reached.
+ */
+int nfs4_replace_transport(struct nfs_server *server,
+			   const struct nfs4_fs_locations *locations)
+{
+	char *page = NULL, *page2 = NULL;
+	int loc, error;
+
+	dprintk("--> %s(%llx:%llx)\n", __func__,
+			(unsigned long long)server->fsid.major,
+			(unsigned long long)server->fsid.minor);
+
+	error = -ENOENT;
+	if (locations == NULL || locations->nlocations <= 0)
+		goto out;
+
+	error = -ENOMEM;
+	page = (char *) __get_free_page(GFP_USER);
+	if (!page)
+		goto out;
+	page2 = (char *) __get_free_page(GFP_USER);
+	if (!page2)
+		goto out;
+
+	for (loc = 0; loc < locations->nlocations; loc++) {
+		const struct nfs4_fs_location *location =
+						&locations->locations[loc];
+
+		if (location == NULL || location->nservers <= 0 ||
+		    location->rootpath.ncomponents == 0)
+			continue;
+
+		error = nfs4_try_replacing_one_location(server, page,
+							page2, location);
+		if (error == 0)
+			break;
+	}
+
+out:
+	free_page((unsigned long)page);
+	free_page((unsigned long)page2);
+
+	dprintk("<-- %s() = %d\n", __func__, error);
+	return error;
+}

[PATCH v1 14/19] NFS: Add method to retrieve fs_locations during migration recovery

[Chuck Lever]

The nfs4_proc_fs_locations() function is invoked during referral
processing to perform a GETATTR(fs_locations) on an object's parent
directory in order to discover the target of the referral.  It
performs a LOOKUP in the compound, so the client needs to know the
parent's file handle a priori.

Unfortunately this function is not adequate for handling migration
recovery.  We need to probe fs_locations information on an FSID's
root directory, but the "parent" of a root directory is not
available for a LOOKUP operation.  Fortunately we already know the
FSID's root FH, as it is now saved at mount time.

Another subtlety: recovering from NFS4ERR_LEASE_MOVED is a process
of walking over a list of known FSIDs that reside on the server, and
probing whether they have migrated.  Once the server has detected
that the client has probed all migrated file systems, it stops
returning NFS4ERR_LEASE_MOVED.

A minor version zero server needs to know what client ID is
requesting fs_locations information so it can clear the flag that
forces it to continue returning NFS4ERR_LEASE_MOVED.  This flag is
set per client ID and per FSID.  However, the client ID is not an
argument of either the PUTFH or GETATTR operations.  Later minor
versions have client ID information embedded in the compound's
SEQUENCE operation.

Therefore, by convention, minor version zero clients send a RENEW
operation in the same compound as the GETATTR(fs_locations), since
RENEW's one argument is a clientid4.  This allows a minor version
zero server to identify correctly the client that is probing for a
migration.

(Needing to send a RENEW in the mv0 case by itself might not be a
strong justification for separate get_locations methods for mv0 and
mv1.  But eventually, fs_locations_info support can be plumbed into
the mv1 method without disturbing the mv0 method).

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h        |    2 +
 fs/nfs/nfs4proc.c       |  147 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4xdr.c        |   46 +++++++++++----
 include/linux/nfs_xdr.h |    4 +
 4 files changed, 188 insertions(+), 11 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 4b83ffc..66c34c4 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -203,6 +203,7 @@ struct nfs4_state_maintenance_ops {
 };
 
 struct nfs4_mig_recovery_ops {
+	int (*get_locations)(struct inode *, struct nfs4_fs_locations *, struct page *);
 };
 
 extern const struct dentry_operations nfs4_dentry_operations;
@@ -235,6 +236,7 @@ extern int nfs4_do_close(struct nfs4_state *state, gfp_t gfp_mask, int wait);
 extern int nfs4_server_capabilities(struct nfs_server *server, struct nfs_fh *fhandle);
 extern int nfs4_proc_fs_locations(struct rpc_clnt *, struct inode *, const struct qstr *,
 				  struct nfs4_fs_locations *, struct page *);
+extern int nfs4_proc_get_locations(struct inode *, struct nfs4_fs_locations *, struct page *page);
 extern struct rpc_clnt *nfs4_proc_lookup_mountpoint(struct inode *, struct qstr *,
 			    struct nfs_fh *, struct nfs_fattr *);
 extern int nfs4_proc_secinfo(struct inode *, const struct qstr *, struct nfs4_secinfo_flavors *);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index bb5100a..c849721 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5445,6 +5445,151 @@ int nfs4_proc_fs_locations(struct rpc_clnt *client, struct inode *dir,
 	return err;
 }
 
+/*
+ * This operation also signals the server that this client is
+ * performing migration recovery.  The server can stop returning
+ * NFS4ERR_LEASE_MOVED to this client.  A RENEW operation is
+ * appended to this compound to identify the client ID which is
+ * performing recovery.
+ */
+static int _nfs40_proc_get_locations(struct inode *inode,
+				     struct nfs4_fs_locations *locations,
+				     struct page *page)
+{
+	struct nfs_server *server = NFS_SERVER(inode);
+	struct rpc_clnt *clnt = server->nfs_client->cl_rpcclient;
+	u32 bitmask[2] = {
+		[0] = FATTR4_WORD0_FSID | FATTR4_WORD0_FS_LOCATIONS,
+	};
+	struct nfs4_fs_locations_arg args = {
+		.clientid	= server->nfs_client->cl_clientid,
+		.fh		= NFS_FH(inode),
+		.page		= page,
+		.bitmask	= bitmask,
+		.migration	= 1,
+		.renew		= 1,
+	};
+	struct nfs4_fs_locations_res res = {
+		.fs_locations	= locations,
+		.migration	= 1,
+		.renew		= 1,
+	};
+	struct rpc_message msg = {
+		.rpc_proc	= &nfs4_procedures[NFSPROC4_CLNT_FS_LOCATIONS],
+		.rpc_argp	= &args,
+		.rpc_resp	= &res,
+	};
+	unsigned long now = jiffies;
+	int status;
+
+	nfs_fattr_init(&locations->fattr);
+	locations->server = server;
+	locations->nlocations = 0;
+
+	status = rpc_call_sync(clnt, &msg, 0);
+	if (status)
+		return status;
+
+	renew_lease(server, now);
+	return 0;
+}
+
+#ifdef CONFIG_NFS_V4_1
+/*
+ * This operation also signals the server that this client is
+ * performing migration recovery.  The server can stop asserting
+ * SEQ4_STATUS_LEASE_MOVED for this client.  The client ID
+ * performing this operation is identified in the SEQUENCE
+ * operation in this compound.
+ *
+ * When the client supports GETATTR(fs_locations_info), it can
+ * be plumbed in here.
+ */
+static int _nfs41_proc_get_locations(struct inode *inode,
+				     struct nfs4_fs_locations *locations,
+				     struct page *page)
+{
+	struct nfs_server *server = NFS_SERVER(inode);
+	struct rpc_clnt *clnt = server->nfs_client->cl_rpcclient;
+	u32 bitmask[2] = {
+		[0] = FATTR4_WORD0_FSID | FATTR4_WORD0_FS_LOCATIONS,
+	};
+	struct nfs4_fs_locations_arg args = {
+		.fh		= NFS_FH(inode),
+		.page		= page,
+		.bitmask	= bitmask,
+		.migration	= 1,
+	};
+	struct nfs4_fs_locations_res res = {
+		.fs_locations	= locations,
+		.migration	= 1,
+	};
+	struct rpc_message msg = {
+		.rpc_proc	= &nfs4_procedures[NFSPROC4_CLNT_FS_LOCATIONS],
+		.rpc_argp	= &args,
+		.rpc_resp	= &res,
+	};
+	int status;
+
+	nfs_fattr_init(&locations->fattr);
+	locations->server = server;
+	locations->nlocations = 0;
+
+	nfs41_init_sequence(&args.seq_args, &res.seq_res, 0);
+	nfs4_set_sequence_privileged(&args.seq_args);
+	status = nfs4_call_sync_sequence(clnt, server, &msg,
+					&args.seq_args, &res.seq_res);
+	if (status == NFS4_OK &&
+	    res.seq_res.sr_status_flags & SEQ4_STATUS_LEASE_MOVED)
+		status = -NFS4ERR_LEASE_MOVED;
+	return status;
+}
+#endif	/* CONFIG_NFS_V4_1 */
+
+/**
+ * nfs4_proc_get_locations - discover locations for a migrated FSID
+ * @inode: inode on FSID that is migrating
+ * @locations: result of query
+ * @page: buffer
+ *
+ * Returns NFS4_OK on success, a negative NFS4ERR status code if the
+ * operation failed, or a negative errno if a local error occurred.
+ *
+ * On success, "locations" is filled in, but if the server has
+ * no locations information, NFS_ATTR_FATTR_V4_LOCATIONS is not
+ * asserted.
+ *
+ * -NFS4ERR_LEASE_MOVED is returned if the server still has leases
+ * from this client that require migration recovery.
+ */
+int nfs4_proc_get_locations(struct inode *inode,
+			    struct nfs4_fs_locations *locations,
+			    struct page *page)
+{
+	struct nfs_server *server = NFS_SERVER(inode);
+	struct nfs_client *clp = server->nfs_client;
+	const struct nfs4_mig_recovery_ops *ops =
+					clp->cl_mvops->mig_recovery_ops;
+	struct nfs4_exception exception = { };
+	int status;
+
+	dprintk("--> %s: FSID %llx:%llx on \"%s\"\n", __func__,
+		(unsigned long long)server->fsid.major,
+		(unsigned long long)server->fsid.minor,
+		clp->cl_hostname);
+	nfs_display_fhandle(NFS_FH(inode), __func__);
+
+	do {
+		status = ops->get_locations(inode, locations, page);
+		if (status != -NFS4ERR_DELAY)
+			break;
+		nfs4_handle_exception(server, status, &exception);
+	} while (exception.retry);
+
+	dprintk("<-- %s status=%d\n", __func__, status);
+	return status;
+}
+
 static int _nfs4_proc_secinfo(struct inode *dir, const struct qstr *name, struct nfs4_secinfo_flavors *flavors)
 {
 	int status;
@@ -7043,10 +7188,12 @@ static const struct nfs4_state_maintenance_ops nfs41_state_renewal_ops = {
 #endif
 
 static const struct nfs4_mig_recovery_ops nfs40_mig_recovery_ops = {
+	.get_locations = _nfs40_proc_get_locations,
 };
 
 #if defined(CONFIG_NFS_V4_1)
 static const struct nfs4_mig_recovery_ops nfs41_mig_recovery_ops = {
+	.get_locations = _nfs41_proc_get_locations,
 };
 #endif	/* CONFIG_NFS_V4_1 */
 
diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index 4be8d13..c63f4b7 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -718,13 +718,15 @@ static int nfs4_stat_to_errno(int);
 				 encode_sequence_maxsz + \
 				 encode_putfh_maxsz + \
 				 encode_lookup_maxsz + \
-				 encode_fs_locations_maxsz)
+				 encode_fs_locations_maxsz + \
+				 encode_renew_maxsz)
 #define NFS4_dec_fs_locations_sz \
 				(compound_decode_hdr_maxsz + \
 				 decode_sequence_maxsz + \
 				 decode_putfh_maxsz + \
 				 decode_lookup_maxsz + \
-				 decode_fs_locations_maxsz)
+				 decode_fs_locations_maxsz + \
+				 decode_renew_maxsz)
 #define NFS4_enc_secinfo_sz 	(compound_encode_hdr_maxsz + \
 				encode_sequence_maxsz + \
 				encode_putfh_maxsz + \
@@ -2636,11 +2638,20 @@ static void nfs4_xdr_enc_fs_locations(struct rpc_rqst *req,
 
 	encode_compound_hdr(xdr, req, &hdr);
 	encode_sequence(xdr, &args->seq_args, &hdr);
-	encode_putfh(xdr, args->dir_fh, &hdr);
-	encode_lookup(xdr, args->name, &hdr);
-	replen = hdr.replen;	/* get the attribute into args->page */
-	encode_fs_locations(xdr, args->bitmask, &hdr);
+	if (args->migration) {
+		encode_putfh(xdr, args->fh, &hdr);
+		replen = hdr.replen;
+		encode_fs_locations(xdr, args->bitmask, &hdr);
+		if (args->renew)
+			encode_renew(xdr, args->clientid, &hdr);
+	} else {
+		encode_putfh(xdr, args->dir_fh, &hdr);
+		encode_lookup(xdr, args->name, &hdr);
+		replen = hdr.replen;
+		encode_fs_locations(xdr, args->bitmask, &hdr);
+	}
 
+	/* Set up reply kvec to capture returned fs_locations array. */
 	xdr_inline_pages(&req->rq_rcv_buf, replen << 2, &args->page,
 			0, PAGE_SIZE);
 	encode_nops(&hdr);
@@ -6690,13 +6701,26 @@ static int nfs4_xdr_dec_fs_locations(struct rpc_rqst *req,
 	status = decode_putfh(xdr);
 	if (status)
 		goto out;
-	status = decode_lookup(xdr);
-	if (status)
-		goto out;
-	xdr_enter_page(xdr, PAGE_SIZE);
-	status = decode_getfattr_generic(xdr, &res->fs_locations->fattr,
+	if (res->migration) {
+		xdr_enter_page(xdr, PAGE_SIZE);
+		status = decode_getfattr_generic(xdr,
+					&res->fs_locations->fattr,
 					 NULL, res->fs_locations,
 					 res->fs_locations->server);
+		if (status)
+			goto out;
+		if (res->renew)
+			status = decode_renew(xdr);
+	} else {
+		status = decode_lookup(xdr);
+		if (status)
+			goto out;
+		xdr_enter_page(xdr, PAGE_SIZE);
+		status = decode_getfattr_generic(xdr,
+					&res->fs_locations->fattr,
+					 NULL, res->fs_locations,
+					 res->fs_locations->server);
+	}
 out:
 	return status;
 }
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 104b62f..db5a1bc 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1040,14 +1040,18 @@ struct nfs4_fs_locations {
 struct nfs4_fs_locations_arg {
 	struct nfs4_sequence_args	seq_args;
 	const struct nfs_fh *dir_fh;
+	const struct nfs_fh *fh;
 	const struct qstr *name;
 	struct page *page;
 	const u32 *bitmask;
+	clientid4 clientid;
+	unsigned char migration:1, renew:1;
 };
 
 struct nfs4_fs_locations_res {
 	struct nfs4_sequence_res	seq_res;
 	struct nfs4_fs_locations       *fs_locations;
+	unsigned char			migration:1, renew:1;
 };
 
 struct nfs4_secinfo4 {

[PATCH v1 15/19] NFS: Add basic migration support to state manager thread

[Chuck Lever]

Migration recovery and state recovery must be serialized, so handle
both in the state manager thread.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h          |    2 
 fs/nfs/nfs4client.c       |    1 
 fs/nfs/nfs4state.c        |  186 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/nfs_fs_sb.h |    6 +
 4 files changed, 195 insertions(+)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 66c34c4..6391b94 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -30,6 +30,7 @@ enum nfs4_client_state {
 	NFS4CLNT_PURGE_STATE,
 	NFS4CLNT_BIND_CONN_TO_SESSION,
 	NFS4CLNT_BLOCK_XPRT,
+	NFS4CLNT_MOVED,
 };
 
 #define NFS4_RENEW_TIMEOUT		0x01
@@ -348,6 +349,7 @@ extern int nfs4_client_recover_expired_lease(struct nfs_client *clp);
 extern void nfs4_schedule_state_manager(struct nfs_client *);
 extern void nfs4_schedule_path_down_recovery(struct nfs_client *clp);
 extern int nfs4_schedule_stateid_recovery(const struct nfs_server *, struct nfs4_state *);
+extern int nfs4_schedule_migration_recovery(struct inode *);
 extern void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags);
 extern void nfs41_handle_server_scope(struct nfs_client *,
 				      struct nfs41_server_scope **);
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index dadd118..5050349 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -76,6 +76,7 @@ struct nfs_client *nfs4_alloc_client(const struct nfs_client_initdata *cl_init)
 		init_completion(&clp->cl_xpcomplete);
 		rpc_init_wait_queue(&clp->cl_xpwaitq, "NFSv4.0 xprt");
 	}
+	clp->cl_mig_gen = 1;
 	return clp;
 
 error:
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index ae482b2..4757188 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1210,6 +1210,54 @@ void nfs4_schedule_lease_recovery(struct nfs_client *clp)
 }
 EXPORT_SYMBOL_GPL(nfs4_schedule_lease_recovery);
 
+/**
+ * nfs4_schedule_migration_recovery - trigger migration recovery
+ *
+ * @inode: file residing on FSID that is migrating
+ *
+ * Returns zero if recovery has started, otherwise a negative NFS4ERR
+ * value is returned.
+ */
+int nfs4_schedule_migration_recovery(struct inode *inode)
+{
+	struct nfs_server *server;
+	struct nfs_client *clp;
+
+	if (inode == NULL) {
+		pr_err("%s: no inode for migration recovery\n",
+			__func__);
+		WARN_ON(1);
+		return -NFS4ERR_IO;
+	}
+ 	server = NFS_SERVER(inode);
+	clp = server->nfs_client;
+
+	if (test_bit(NFS_CS_MIGRATION, &clp->cl_flags) == 0) {
+		pr_err("NFS: migration not supported (server %s)\n",
+				clp->cl_hostname);
+		return -NFS4ERR_IO;
+	}
+
+	if (server->fh_expire_type != NFS4_FH_PERSISTENT) {
+		pr_err("NFS: volatile file handles not supported (server %s)\n",
+				clp->cl_hostname);
+		return -NFS4ERR_IO;
+	}
+
+	dprintk("%s: scheduling migration recovery for (%llx:%llx) on %s\n",
+			__func__,
+			(unsigned long long)server->fsid.major,
+			(unsigned long long)server->fsid.minor,
+			clp->cl_hostname);
+
+	set_bit(NFS_MIG_IN_TRANSITION, &server->mig_status);
+	set_bit(NFS4CLNT_MOVED, &clp->cl_state);
+
+	nfs4_schedule_state_manager(clp);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nfs4_schedule_migration_recovery);
+
 int nfs4_wait_clnt_recover(struct nfs_client *clp)
 {
 	int res;
@@ -1833,6 +1881,137 @@ static int nfs4_purge_lease(struct nfs_client *clp)
 	return 0;
 }
 
+#ifdef CONFIG_NFS_V4_1
+static int nfs4_begin_drain(struct nfs_client *clp)
+{
+	if (clp->cl_session == NULL)
+		return nfs4_begin_drain_xprt(clp);
+	return nfs4_begin_drain_session(clp);
+}
+
+static void nfs4_end_drain(struct nfs_client *clp)
+{
+	if (clp->cl_session == NULL)
+		nfs4_end_drain_xprt(clp);
+	else
+		nfs4_end_drain_session(clp);
+}
+#else	/* CONFIG_NFS_V4_1 */
+static int nfs4_begin_drain(struct nfs_client *clp)
+{
+	return nfs4_begin_drain_xprt(clp);
+}
+
+static void nfs4_end_drain(struct nfs_client *clp)
+{
+	nfs4_end_drain_xprt(clp);
+}
+#endif	/* CONFIG_NFS_V4_1 */
+
+/*
+ * Try remote migration of one FSID from a source server to a
+ * destination server.  The source server provides a list of
+ * potential destinations.
+ *
+ * Returns zero or a negative NFS4ERR status code.
+ */
+static int nfs4_try_migration(struct nfs_server *server)
+{
+	struct nfs_client *clp = server->nfs_client;
+	struct nfs4_fs_locations *locations = NULL;
+	struct inode *inode;
+	struct page *page;
+	int status, result;
+
+	dprintk("--> %s: FSID %llx:%llx on \"%s\"\n", __func__,
+			(unsigned long long)server->fsid.major,
+			(unsigned long long)server->fsid.minor,
+			clp->cl_hostname);
+
+	result = 0;
+	page = alloc_page(GFP_KERNEL);
+	locations = kmalloc(sizeof(struct nfs4_fs_locations), GFP_KERNEL);
+	if (page == NULL || locations == NULL) {
+		dprintk("<-- %s: no memory\n", __func__);
+		goto out_err;
+	}
+
+	inode = server->super->s_root->d_inode;
+	result = nfs4_proc_get_locations(inode, locations, page);
+	if (result != NFS4_OK) {
+		dprintk("<-- %s: failed to retrieve fs_locations: %d\n",
+			__func__, result);
+		goto out_err;
+	}
+	if (!(locations->fattr.valid & NFS_ATTR_FATTR_V4_LOCATIONS)) {
+		dprintk("<-- %s: No fs_locations data available, "
+			"migration skipped\n", __func__);
+		goto out_err;
+	}
+
+	status = nfs4_begin_drain(clp);
+	if (status != 0) {
+		dprintk("<-- %s: failed to drain transport: %d\n",
+			__func__, status);
+		nfs4_end_drain(clp);
+		goto out_err;
+	}
+
+	status = nfs4_replace_transport(server, locations);
+	nfs4_end_drain(clp);
+	if (status != 0) {
+		dprintk("<-- %s: failed to replace transport: %d\n",
+			__func__, status);
+		goto out_err;
+	}
+
+	dprintk("<-- %s: migration succeeded\n", __func__);
+
+out_err:
+	if (page != NULL)
+		__free_page(page);
+	kfree(locations);
+	return result;
+}
+
+/*
+ * Returns zero or a negative NFS4ERR status code.
+ */
+static int nfs4_handle_migration(struct nfs_client *clp)
+{
+	struct nfs_server *server;
+
+	dprintk("--> %s: migration reported on \"%s\"\n", __func__,
+			clp->cl_hostname);
+
+	clp->cl_mig_gen++;
+restart:
+	rcu_read_lock();
+	list_for_each_entry_rcu(server, &clp->cl_superblocks, client_link) {
+		int status;
+
+		if (server->mig_gen == clp->cl_mig_gen)
+			continue;
+		server->mig_gen = clp->cl_mig_gen;
+
+		if (!test_and_clear_bit(NFS_MIG_IN_TRANSITION,
+						&server->mig_status))
+			continue;
+
+		rcu_read_unlock();
+		status = nfs4_try_migration(server);
+		if (status < 0) {
+			dprintk("<-- %s: %d\n", __func__, status);
+			return status;
+		}
+		goto restart;
+	}
+	rcu_read_unlock();
+
+	dprintk("<-- %s\n", __func__);
+	return 0;
+}
+
 /**
  * nfs4_discover_server_trunking - Detect server IP address trunking
  *
@@ -2163,6 +2342,13 @@ static void nfs4_state_manager(struct nfs_client *clp)
 			status = nfs4_check_lease(clp);
 			if (status < 0)
 				goto out_error;
+		}
+
+		if (test_and_clear_bit(NFS4CLNT_MOVED, &clp->cl_state)) {
+			section = "migration";
+			status = nfs4_handle_migration(clp);
+			if (status < 0)
+				goto out_error;
 			continue;
 		}
 
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 84d8f0c..4cfc58a 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -77,6 +77,7 @@ struct nfs_client {
 	char			cl_ipaddr[48];
 	u32			cl_cb_ident;	/* v4.0 callback identifier */
 	const struct nfs4_minor_version_ops *cl_mvops;
+	unsigned long		cl_mig_gen;
 
 	atomic_t		cl_xppending;
 	struct completion	cl_xpcomplete;
@@ -174,6 +175,11 @@ struct nfs_server {
 	struct list_head	state_owners_lru;
 	struct list_head	layouts;
 	struct list_head	delegations;
+
+	unsigned long		mig_gen;
+	unsigned long		mig_status;
+#define NFS_MIG_IN_TRANSITION		(1)
+
 	void (*destroy)(struct nfs_server *);
 
 	atomic_t active; /* Keep trace of any activity to this server */

[PATCH v1 16/19] NFS: Add migration recovery callouts in nfs4proc.c

[Chuck Lever]

When the server returns NFS4ERR_MOVED, trigger the new migration
recovery logic in the state manager.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4proc.c |   56 +++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 45 insertions(+), 11 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index c849721..b75629a 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -275,10 +275,17 @@ static int nfs4_delay(struct rpc_clnt *clnt, long *timeout)
 	return res;
 }
 
-/* This is the error handling routine for processes that are allowed
- * to sleep.
+/**
+ * nfs4_handle_exception - Common error handling for callers allowed to sleep
+ *
+ * @server: local state context for the server
+ * @errorcode: NFS4ERR value returned from the server
+ * @exception: exception handling state
+ *
+ * Returns zero on success, or a negative errno value.
  */
-static int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_exception *exception)
+static int nfs4_handle_exception(struct nfs_server *server, int errorcode,
+				 struct nfs4_exception *exception)
 {
 	struct nfs_client *clp = server->nfs_client;
 	struct nfs4_state *state = exception->state;
@@ -338,6 +345,11 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, struc
 			nfs4_schedule_session_recovery(clp->cl_session, errorcode);
 			goto wait_on_recovery;
 #endif /* defined(CONFIG_NFS_V4_1) */
+		case -NFS4ERR_MOVED:
+			ret = nfs4_schedule_migration_recovery(inode);
+			if (ret < 0)
+				break;
+			goto wait_on_recovery;
 		case -NFS4ERR_FILE_OPEN:
 			if (exception->timeout > HZ) {
 				/* We have retried a decent amount, time to
@@ -2763,11 +2775,16 @@ static int nfs4_get_referral(struct rpc_clnt *client, struct inode *dir,
 	status = nfs4_proc_fs_locations(client, dir, name, locations, page);
 	if (status != 0)
 		goto out;
-	/* Make sure server returned a different fsid for the referral */
+
+	/*
+	 * If the fsid didn't change, this is a migration event, not a
+	 * referral.  Cause us to drop into the exception handler, which
+	 * will kick off migration recovery.
+	 */
 	if (nfs_fsid_equal(&NFS_SERVER(dir)->fsid, &locations->fattr.fsid)) {
 		dprintk("%s: server did not return a different fsid for"
 			" a referral at %s\n", __func__, name->name);
-		status = -EIO;
+		status = -NFS4ERR_MOVED;
 		goto out;
 	}
 	/* Fixup attributes for the nfs_lookup() call to nfs_fhget() */
@@ -4263,8 +4280,18 @@ static int nfs4_proc_set_acl(struct inode *inode, const void *buf, size_t buflen
 	return err;
 }
 
-static int
-nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server, struct nfs4_state *state)
+/**
+ * nfs4_async_handle_error - Common error handling for callers who cannot sleep
+ *
+ * @task: active RPC task
+ * @server: local state for the server
+ * @state: NFSv4 state for active operation
+ *
+ * Returns zero on success, or a negative errno value.
+ */
+static int nfs4_async_handle_error(struct rpc_task *task,
+				   const struct nfs_server *server,
+				   struct nfs4_state *state)
 {
 	struct nfs_client *clp = server->nfs_client;
 
@@ -4306,16 +4333,22 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
 			task->tk_status = 0;
 			return -EAGAIN;
 #endif /* CONFIG_NFS_V4_1 */
+		case -NFS4ERR_MOVED:
+			rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
+			if (nfs4_schedule_migration_recovery(state->inode) < 0)
+				goto stateid_invalid;
+			if (test_bit(NFS4CLNT_MANAGER_RUNNING,
+							&clp->cl_state) == 0)
+				rpc_wake_up_queued_task(&clp->cl_rpcwaitq,
+									task);
+			goto restart_call;
 		case -NFS4ERR_DELAY:
 			nfs_inc_server_stats(server, NFSIOS_DELAY);
 		case -NFS4ERR_GRACE:
 			rpc_delay(task, NFS4_POLL_RETRY_MAX);
-			task->tk_status = 0;
-			return -EAGAIN;
 		case -NFS4ERR_RETRY_UNCACHED_REP:
 		case -NFS4ERR_OLD_STATEID:
-			task->tk_status = 0;
-			return -EAGAIN;
+			goto restart_call;
 	}
 	task->tk_status = nfs4_map_errors(task->tk_status);
 	return 0;
@@ -4326,6 +4359,7 @@ wait_on_recovery:
 	rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
 	if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
 		rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
+restart_call:
 	task->tk_status = 0;
 	return -EAGAIN;
 }

[PATCH v1 17/19] NFS: Add method to detect whether an FSID is still on the server

[Chuck Lever]

Introduce a mechanism for probing a server to determine if an FSID
is present or absent.

The on-the-wire is different between minor version 0 and 1.  Minor
version 0 appends a RENEW operation to identify which client ID is
probing.  Minor version 1 has a SEQUENCE operation in the compound
which effectively does the same.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h        |    2 +
 fs/nfs/nfs4proc.c       |  117 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4xdr.c        |   69 +++++++++++++++++++++++++++-
 include/linux/nfs4.h    |    1 
 include/linux/nfs_xdr.h |   13 +++++
 5 files changed, 200 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 6391b94..77313db 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -205,6 +205,7 @@ struct nfs4_state_maintenance_ops {
 
 struct nfs4_mig_recovery_ops {
 	int (*get_locations)(struct inode *, struct nfs4_fs_locations *, struct page *);
+	int (*fsid_present)(struct inode *);
 };
 
 extern const struct dentry_operations nfs4_dentry_operations;
@@ -238,6 +239,7 @@ extern int nfs4_server_capabilities(struct nfs_server *server, struct nfs_fh *fh
 extern int nfs4_proc_fs_locations(struct rpc_clnt *, struct inode *, const struct qstr *,
 				  struct nfs4_fs_locations *, struct page *);
 extern int nfs4_proc_get_locations(struct inode *, struct nfs4_fs_locations *, struct page *page);
+extern int nfs4_proc_fsid_present(struct inode *);
 extern struct rpc_clnt *nfs4_proc_lookup_mountpoint(struct inode *, struct qstr *,
 			    struct nfs_fh *, struct nfs_fattr *);
 extern int nfs4_proc_secinfo(struct inode *, const struct qstr *, struct nfs4_secinfo_flavors *);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index b75629a..a870a59 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5624,6 +5624,121 @@ int nfs4_proc_get_locations(struct inode *inode,
 	return status;
 }
 
+/*
+ * This operation also signals the server that this client is
+ * performing "lease moved" recovery.  The server can stop
+ * returning NFS4ERR_LEASE_MOVED to this client.  A RENEW operation
+ * is appended to this compound to identify the client ID which is
+ * performing recovery.
+ */
+static int _nfs40_proc_fsid_present(struct inode *inode)
+{
+	struct nfs_client *clp = NFS_SERVER(inode)->nfs_client;
+	struct rpc_clnt *clnt = clp->cl_rpcclient;
+	struct nfs4_fsid_present_arg args = {
+		.fh		= NFS_FH(inode),
+		.clientid	= clp->cl_clientid,
+		.renew		= 1,
+	};
+	struct nfs4_fsid_present_res res = {
+		.renew		= 1,
+	};
+	struct rpc_message msg = {
+		.rpc_proc	= &nfs4_procedures[NFSPROC4_CLNT_FSID_PRESENT],
+		.rpc_argp	= &args,
+		.rpc_resp	= &res,
+	};
+	unsigned long now = jiffies;
+	int status;
+
+	res.fh = nfs_alloc_fhandle();
+	if (res.fh == NULL)
+		return -ENOMEM;
+
+	status = rpc_call_sync(clnt, &msg, 0);
+	nfs_free_fhandle(res.fh);
+	if (status)
+		return status;
+
+	do_renew_lease(clp, now);
+	return 0;
+}
+
+#ifdef CONFIG_NFS_V4_1
+/*
+ * This operation also signals the server that this client is
+ * performing "lease moved" recovery.  The server can stop asserting
+ * SEQ4_STATUS_LEASE_MOVED for this client.  The client ID performing
+ * this operation is identified in the SEQUENCE operation in this
+ * compound.
+ */
+static int _nfs41_proc_fsid_present(struct inode *inode)
+{
+	struct nfs_server *server = NFS_SERVER(inode);
+	struct rpc_clnt *clnt = server->nfs_client->cl_rpcclient;
+	struct nfs4_fsid_present_arg args = {
+		.fh		= NFS_FH(inode),
+	};
+	struct nfs4_fsid_present_res res = {
+	};
+	struct rpc_message msg = {
+		.rpc_proc	= &nfs4_procedures[NFSPROC4_CLNT_FSID_PRESENT],
+		.rpc_argp	= &args,
+		.rpc_resp	= &res,
+	};
+	int status;
+
+	res.fh = nfs_alloc_fhandle();
+	if (res.fh == NULL)
+		return -ENOMEM;
+
+	nfs41_init_sequence(&args.seq_args, &res.seq_res, 0);
+	nfs4_set_sequence_privileged(&args.seq_args);
+	status = nfs4_call_sync_sequence(clnt, server, &msg,
+						&args.seq_args, &res.seq_res);
+	nfs_free_fhandle(res.fh);
+	return status;
+}
+#endif	/* CONFIG_NFS_V4_1 */
+
+/**
+ * nfs4_proc_fsid_present - Is this FSID present or absent on server?
+ * @inode: inode on FSID to check
+ *
+ * Server indicates whether the FSID is present, moved, or not
+ * recognized.  This operation is necessary to clear a LEASE_MOVED
+ * condition for this client ID.
+ *
+ * Returns NFS4_OK if the FSID is present on this server,
+ * -NFS4ERR_MOVED if the FSID is no longer present, a negative
+ *  NFS4ERR code if some error occurred on the server, or a
+ *  negative errno if a local failure occurred.
+ */
+int nfs4_proc_fsid_present(struct inode *inode)
+{
+	struct nfs_server *server = NFS_SERVER(inode);
+	struct nfs_client *clp = server->nfs_client;
+	const struct nfs4_mig_recovery_ops *ops =
+					clp->cl_mvops->mig_recovery_ops;
+	struct nfs4_exception exception = { };
+	int status;
+
+	dprintk("--> %s: FSID %llx:%llx on \"%s\"\n", __func__,
+		(unsigned long long)server->fsid.major,
+		(unsigned long long)server->fsid.minor,
+		clp->cl_hostname);
+	nfs_display_fhandle(NFS_FH(inode), __func__);
+
+	do {
+		status = ops->fsid_present(inode);
+		if (status != -NFS4ERR_DELAY)
+			break;
+		nfs4_handle_exception(server, status, &exception);
+	} while (exception.retry);
+	dprintk("<-- %s status=%d\n", __func__, status);
+	return status;
+}
+
 static int _nfs4_proc_secinfo(struct inode *dir, const struct qstr *name, struct nfs4_secinfo_flavors *flavors)
 {
 	int status;
@@ -7223,11 +7338,13 @@ static const struct nfs4_state_maintenance_ops nfs41_state_renewal_ops = {
 
 static const struct nfs4_mig_recovery_ops nfs40_mig_recovery_ops = {
 	.get_locations = _nfs40_proc_get_locations,
+	.fsid_present = _nfs40_proc_fsid_present,
 };
 
 #if defined(CONFIG_NFS_V4_1)
 static const struct nfs4_mig_recovery_ops nfs41_mig_recovery_ops = {
 	.get_locations = _nfs41_proc_get_locations,
+	.fsid_present = _nfs41_proc_fsid_present,
 };
 #endif	/* CONFIG_NFS_V4_1 */
 
diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index c63f4b7..8143956 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -577,11 +577,13 @@ static int nfs4_stat_to_errno(int);
 #define NFS4_enc_getattr_sz	(compound_encode_hdr_maxsz + \
 				encode_sequence_maxsz + \
 				encode_putfh_maxsz + \
-				encode_getattr_maxsz)
+				encode_getattr_maxsz + \
+				encode_renew_maxsz)
 #define NFS4_dec_getattr_sz	(compound_decode_hdr_maxsz + \
 				decode_sequence_maxsz + \
 				decode_putfh_maxsz + \
-				decode_getattr_maxsz)
+				decode_getattr_maxsz + \
+				decode_renew_maxsz)
 #define NFS4_enc_lookup_sz	(compound_encode_hdr_maxsz + \
 				encode_sequence_maxsz + \
 				encode_putfh_maxsz + \
@@ -735,6 +737,18 @@ static int nfs4_stat_to_errno(int);
 				decode_sequence_maxsz + \
 				decode_putfh_maxsz + \
 				decode_secinfo_maxsz)
+#define NFS4_enc_fsid_present_sz \
+				(compound_encode_hdr_maxsz + \
+				 encode_sequence_maxsz + \
+				 encode_putfh_maxsz + \
+				 encode_getfh_maxsz + \
+				 encode_renew_maxsz)
+#define NFS4_dec_fsid_present_sz \
+				(compound_decode_hdr_maxsz + \
+				 decode_sequence_maxsz + \
+				 decode_putfh_maxsz + \
+				 decode_getfh_maxsz + \
+				 decode_renew_maxsz)
 #if defined(CONFIG_NFS_V4_1)
 #define NFS4_enc_bind_conn_to_session_sz \
 				(compound_encode_hdr_maxsz + \
@@ -2675,6 +2689,26 @@ static void nfs4_xdr_enc_secinfo(struct rpc_rqst *req,
 	encode_nops(&hdr);
 }
 
+/*
+ * Encode FSID_PRESENT request
+ */
+static void nfs4_xdr_enc_fsid_present(struct rpc_rqst *req,
+				      struct xdr_stream *xdr,
+				      struct nfs4_fsid_present_arg *args)
+{
+	struct compound_hdr hdr = {
+		.minorversion = nfs4_xdr_minorversion(&args->seq_args),
+	};
+
+	encode_compound_hdr(xdr, req, &hdr);
+	encode_sequence(xdr, &args->seq_args, &hdr);
+	encode_putfh(xdr, args->fh, &hdr);
+	encode_getfh(xdr, &hdr);
+	if (args->renew)
+		encode_renew(xdr, args->clientid, &hdr);
+	encode_nops(&hdr);
+}
+
 #if defined(CONFIG_NFS_V4_1)
 /*
  * BIND_CONN_TO_SESSION request
@@ -6109,6 +6143,8 @@ static int nfs4_xdr_dec_getattr(struct rpc_rqst *rqstp, struct xdr_stream *xdr,
 	if (status)
 		goto out;
 	status = decode_getfattr(xdr, res->fattr, res->server);
+	if (status)
+		goto out;
 out:
 	return status;
 }
@@ -6749,6 +6785,34 @@ out:
 	return status;
 }
 
+/*
+ * Decode FSID_PRESENT response
+ */
+static int nfs4_xdr_dec_fsid_present(struct rpc_rqst *rqstp,
+				     struct xdr_stream *xdr,
+				     struct nfs4_fsid_present_res *res)
+{
+	struct compound_hdr hdr;
+	int status;
+
+	status = decode_compound_hdr(xdr, &hdr);
+	if (status)
+		goto out;
+	status = decode_sequence(xdr, &res->seq_res, rqstp);
+	if (status)
+		goto out;
+	status = decode_putfh(xdr);
+	if (status)
+		goto out;
+	status = decode_getfh(xdr, res->fh);
+	if (status)
+		goto out;
+	if (res->renew)
+		status = decode_renew(xdr);
+out:
+	return status;
+}
+
 #if defined(CONFIG_NFS_V4_1)
 /*
  * Decode BIND_CONN_TO_SESSION response
@@ -7263,6 +7327,7 @@ struct rpc_procinfo	nfs4_procedures[] = {
 	PROC(FS_LOCATIONS,	enc_fs_locations,	dec_fs_locations),
 	PROC(RELEASE_LOCKOWNER,	enc_release_lockowner,	dec_release_lockowner),
 	PROC(SECINFO,		enc_secinfo,		dec_secinfo),
+	PROC(FSID_PRESENT,	enc_fsid_present,	dec_fsid_present),
 #if defined(CONFIG_NFS_V4_1)
 	PROC(EXCHANGE_ID,	enc_exchange_id,	dec_exchange_id),
 	PROC(CREATE_SESSION,	enc_create_session,	dec_create_session),
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index 7b8fc73..043d009 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -438,6 +438,7 @@ enum {
 	NFSPROC4_CLNT_FS_LOCATIONS,
 	NFSPROC4_CLNT_RELEASE_LOCKOWNER,
 	NFSPROC4_CLNT_SECINFO,
+	NFSPROC4_CLNT_FSID_PRESENT,
 
 	/* nfs41 */
 	NFSPROC4_CLNT_EXCHANGE_ID,
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index db5a1bc..9f1867b 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1075,6 +1075,19 @@ struct nfs4_secinfo_res {
 	struct nfs4_secinfo_flavors	*flavors;
 };
 
+struct nfs4_fsid_present_arg {
+	struct nfs4_sequence_args	seq_args;
+	const struct nfs_fh		*fh;
+	clientid4			clientid;
+	unsigned char			renew:1;
+};
+
+struct nfs4_fsid_present_res {
+	struct nfs4_sequence_res	seq_res;
+	struct nfs_fh			*fh;
+	unsigned char			renew:1;
+};
+
 #endif /* CONFIG_NFS_V4 */
 
 struct nfstime4 {

[PATCH v1 18/19] NFS: Implement support for NFS4ERR_LEASE_MOVED

[Chuck Lever]

To recover from NFS4ERR_LEASE_MOVED, walk the cl_superblocks list and
invoke nfs4_try_migration() on each server's root file handle if that
FSID is no longer present on the server.  The FSID presence check also
informs servers that this client has recognized the LEASE_MOVED
condition and therefore should no longer receive NFS4ERR_LEASE_MOVED.

Minor version 1 servers set a bit flag to signal LEASE_MOVED rather
than returning an NFS4ERR status code, as minor version 0 servers do.
Our client must recognize this flag and kick off migration recovery.
The flag is detected when processing the SEQUENCE reply in each mv1
compound.

Note also that with minor version 0, the asynchronous lease RENEW
heartbeat can return NFS4ERR_LEASE_MOVED.  Error recovery logic for
async RENEW is a separate code path from the generic NFS proc paths,
so it must be updated to handle LEASE_MOVED as well.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4_fs.h   |    2 +
 fs/nfs/nfs4proc.c  |   19 +++++++++++++-
 fs/nfs/nfs4state.c |   70 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 87 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 77313db..9f2fe35 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -31,6 +31,7 @@ enum nfs4_client_state {
 	NFS4CLNT_BIND_CONN_TO_SESSION,
 	NFS4CLNT_BLOCK_XPRT,
 	NFS4CLNT_MOVED,
+	NFS4CLNT_LEASE_MOVED,
 };
 
 #define NFS4_RENEW_TIMEOUT		0x01
@@ -352,6 +353,7 @@ extern void nfs4_schedule_state_manager(struct nfs_client *);
 extern void nfs4_schedule_path_down_recovery(struct nfs_client *clp);
 extern int nfs4_schedule_stateid_recovery(const struct nfs_server *, struct nfs4_state *);
 extern int nfs4_schedule_migration_recovery(struct inode *);
+extern void nfs4_schedule_lease_moved_recovery(struct nfs_client *);
 extern void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags);
 extern void nfs41_handle_server_scope(struct nfs_client *,
 				      struct nfs41_server_scope **);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index a870a59..80afb9f 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -350,6 +350,9 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode,
 			if (ret < 0)
 				break;
 			goto wait_on_recovery;
+		case -NFS4ERR_LEASE_MOVED:
+			nfs4_schedule_lease_moved_recovery(clp);
+			goto wait_on_recovery;
 		case -NFS4ERR_FILE_OPEN:
 			if (exception->timeout > HZ) {
 				/* We have retried a decent amount, time to
@@ -3937,7 +3940,13 @@ static void nfs4_renew_done(struct rpc_task *task, void *calldata)
 	struct nfs_client *clp = data->client;
 	unsigned long timestamp = data->timestamp;
 
-	if (task->tk_status < 0) {
+	switch (task->tk_status) {
+	case 0:
+		break;
+	case -NFS4ERR_LEASE_MOVED:
+		nfs4_schedule_lease_moved_recovery(clp);
+		break;
+	default:
 		/* Unless we're shutting down, schedule state recovery! */
 		if (test_bit(NFS_CS_RENEWD, &clp->cl_res_state) == 0)
 			return;
@@ -4342,6 +4351,14 @@ static int nfs4_async_handle_error(struct rpc_task *task,
 				rpc_wake_up_queued_task(&clp->cl_rpcwaitq,
 									task);
 			goto restart_call;
+		case -NFS4ERR_LEASE_MOVED:
+			rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
+			nfs4_schedule_lease_moved_recovery(clp);
+			if (test_bit(NFS4CLNT_MANAGER_RUNNING,
+							&clp->cl_state) == 0)
+				rpc_wake_up_queued_task(&clp->cl_rpcwaitq,
+									task);
+			goto restart_call;
 		case -NFS4ERR_DELAY:
 			nfs_inc_server_stats(server, NFSIOS_DELAY);
 		case -NFS4ERR_GRACE:
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 4757188..ce2b51d 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1258,6 +1258,22 @@ int nfs4_schedule_migration_recovery(struct inode *inode)
 }
 EXPORT_SYMBOL_GPL(nfs4_schedule_migration_recovery);
 
+/**
+ * nfs4_schedule_lease_moved_recovery - start lease-moved recovery
+ *
+ * @clp: server to check for moved leases
+ *
+ */
+void nfs4_schedule_lease_moved_recovery(struct nfs_client *clp)
+{
+	dprintk("%s: scheduling lease-moved recovery for client ID %llx on %s\n",
+		__func__, clp->cl_clientid, clp->cl_hostname);
+
+	set_bit(NFS4CLNT_LEASE_MOVED, &clp->cl_state);
+	nfs4_schedule_state_manager(clp);
+}
+EXPORT_SYMBOL_GPL(nfs4_schedule_lease_moved_recovery);
+
 int nfs4_wait_clnt_recover(struct nfs_client *clp)
 {
 	int res;
@@ -1680,7 +1696,6 @@ static int nfs4_recovery_handle_error(struct nfs_client *clp, int error)
 			nfs4_state_end_reclaim_reboot(clp);
 			break;
 		case -NFS4ERR_STALE_CLIENTID:
-		case -NFS4ERR_LEASE_MOVED:
 			set_bit(NFS4CLNT_LEASE_EXPIRED, &clp->cl_state);
 			nfs4_state_clear_reclaim_reboot(clp);
 			nfs4_state_start_reclaim_reboot(clp);
@@ -2007,7 +2022,46 @@ restart:
 		goto restart;
 	}
 	rcu_read_unlock();
+	dprintk("<-- %s\n", __func__);
+	return 0;
+}
+
+/*
+ * Test each nfs_server on the clp's cl_superblocks list to see
+ * if it's moved to another server.  Stop when the server no longer
+ * returns NFS4ERR_LEASE_MOVED.
+ */
+static int nfs4_handle_lease_moved(struct nfs_client *clp)
+{
+	struct nfs_server *server;
+
+	dprintk("--> %s: \"%s\" (client ID %llx)\n",
+		__func__, clp->cl_hostname, clp->cl_clientid);
+
+	clp->cl_mig_gen++;
+restart:
+	rcu_read_lock();
+	list_for_each_entry_rcu(server, &clp->cl_superblocks, client_link) {
+		struct inode *inode;
+		int status;
+
+		if (server->mig_gen == clp->cl_mig_gen)
+			continue;
+		server->mig_gen = clp->cl_mig_gen;
+
+		rcu_read_unlock();
+
+		inode = server->super->s_root->d_inode;
+		status = nfs4_proc_fsid_present(inode);
+		if (status != -NFS4ERR_MOVED)
+			goto restart;	/* wasn't this one */
+		if (nfs4_try_migration(server) == -NFS4ERR_LEASE_MOVED)
+			goto restart;	/* there are more */
+		goto out;
+	}
+	rcu_read_unlock();
 
+out:
 	dprintk("<-- %s\n", __func__);
 	return 0;
 }
@@ -2201,9 +2255,10 @@ void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags)
 		nfs41_handle_server_reboot(clp);
 	if (flags & (SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED |
 			    SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED |
-			    SEQ4_STATUS_ADMIN_STATE_REVOKED |
-			    SEQ4_STATUS_LEASE_MOVED))
+			    SEQ4_STATUS_ADMIN_STATE_REVOKED))
 		nfs41_handle_state_revoked(clp);
+	if (flags & SEQ4_STATUS_LEASE_MOVED)
+		nfs4_schedule_lease_moved_recovery(clp);
 	if (flags & SEQ4_STATUS_RECALLABLE_STATE_REVOKED)
 		nfs41_handle_recallable_state_revoked(clp);
 	if (flags & SEQ4_STATUS_BACKCHANNEL_FAULT)
@@ -2352,6 +2407,15 @@ static void nfs4_state_manager(struct nfs_client *clp)
 			continue;
 		}
 
+		if (test_bit(NFS4CLNT_LEASE_MOVED, &clp->cl_state)) {
+			section = "lease moved";
+			status = nfs4_handle_lease_moved(clp);
+			if (status < 0)
+				goto out_error;
+			clear_bit(NFS4CLNT_LEASE_MOVED, &clp->cl_state);
+			continue;
+		}
+
 		/* First recover reboot state... */
 		if (test_bit(NFS4CLNT_RECLAIM_REBOOT, &clp->cl_state)) {
 			section = "reclaim reboot";

[PATCH v1 19/19] NFS: Set EXCHGID4_FLAG_SUPP_MOVED_MIGR

[Chuck Lever]

We have all the pieces in place to support NFSv4 minor version 1
migration in our client.  The client should advertise its support
for migration.

Note that v4.1 migration is untested.  There are no servers in the
wild that support NFSv4.1 migration.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 fs/nfs/nfs4proc.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 80afb9f..8d0fab7 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -5876,8 +5876,10 @@ out:
 	return status;
 }
 
-/*
- * nfs4_proc_exchange_id()
+/**
+ * nfs4_proc_exchange_id - Instantiate client ID
+ * @clp: server state
+ * @cred: RPC credential to use for this call
  *
  * Returns zero, a negative errno, or a negative NFS4ERR status code.
  *
@@ -5892,7 +5894,8 @@ int nfs4_proc_exchange_id(struct nfs_client *clp, struct rpc_cred *cred)
 	struct nfs41_exchange_id_args args = {
 		.verifier = &verifier,
 		.client = clp,
-		.flags = EXCHGID4_FLAG_SUPP_MOVED_REFER,
+		.flags = EXCHGID4_FLAG_SUPP_MOVED_REFER |
+			 EXCHGID4_FLAG_SUPP_MOVED_MIGR,
 	};
 	struct nfs41_exchange_id_res res = {
 		0

Re: [PATCH v1 01/19] NFS: Never use user credentials for lease renewal

[Myklebust, Trond]

T24gRnJpLCAyMDEzLTA3LTEyIGF0IDEyOjMxIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
RG9uJ3QgdHJ5IHRvIHVzZSBhIG5vbi1VSUQtMCB1c2VyIGNyZWRlbnRpYWwgZm9yIGxlYXNlIG1h
bmFnZW1lbnQsDQo+IGFzIHRoYXQgY3JlZGVudGlhbCBjYW4gY2hhbmdlIG91dCBmcm9tIHVuZGVy
IHVzLiAgVGhlIHNlcnZlciB3aWxsDQo+IGJsb2NrIE5GU3Y0IGxlYXNlIHJlY292ZXJ5IHdpdGgg
TkZTNEVSUl9DTElEX0lOVVNFLg0KPiANCj4gUmVwbGFjZSBtaW5vciB2ZXJzaW9uLXNwZWNpZmlj
IGNhbGxvdXRzIHdpdGggYSBzaW5nbGUgZnVuY3Rpb24uDQo+IA0KPiBTaWduZWQtb2ZmLWJ5OiBD
aHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4NCj4gLS0tDQo+ICBmcy9uZnMvbmZz
NF9mcy5oICAgICB8ICAgIDQgKy0tLQ0KPiAgZnMvbmZzL25mczRwcm9jLmMgICAgfCAgICA2ICst
LS0tDQo+ICBmcy9uZnMvbmZzNHNlc3Npb24uYyB8ICAgIDIgKy0NCj4gIGZzL25mcy9uZnM0c3Rh
dGUuYyAgIHwgICA1NSArKysrKystLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLQ0KPiAgNCBmaWxlcyBjaGFuZ2VkLCAxMCBpbnNlcnRpb25zKCspLCA1NyBkZWxldGlv
bnMoLSkNCg0KVGhpcyB3b24ndCBhcHBseSB0byB1cHN0cmVhbSBrZXJuZWxzLg0KDQotLSANClRy
b25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJv
bmQuTXlrbGVidXN0QG5ldGFwcC5jb20NCnd3dy5uZXRhcHAuY29tDQo=

Re: [PATCH v1 01/19] NFS: Never use user credentials for lease renewal

[Chuck Lever]

On Jul 22, 2013, at 2:33 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

> On Fri, 2013-07-12 at 12:31 -0400, Chuck Lever wrote:
>> Don't try to use a non-UID-0 user credential for lease management,
>> as that credential can change out from under us.  The server will
>> block NFSv4 lease recovery with NFS4ERR_CLID_INUSE.
>> 
>> Replace minor version-specific callouts with a single function.
>> 
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>> fs/nfs/nfs4_fs.h     |    4 +---
>> fs/nfs/nfs4proc.c    |    6 +----
>> fs/nfs/nfs4session.c |    2 +-
>> fs/nfs/nfs4state.c   |   55 ++++++--------------------------------------------
>> 4 files changed, 10 insertions(+), 57 deletions(-)
> 
> This won't apply to upstream kernels.

Thanks for having a look... As the cover letter stated, this series applies to 3.10.

Now that 3.11-rc2 is available, I'm porting the series to the latest upstream, with appropriate adjustments for NFSv4.2 support.  I hope to post the new series tomorrow or Wednesday.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

RE: [PATCH v1 01/19] NFS: Never use user credentials for lease renewal

[Myklebust, Trond]

> -----Original Message-----
> From: Chuck Lever [mailto:chuck.lever@oracle.com]
> Sent: Monday, July 22, 2013 2:52 PM
> To: Myklebust, Trond
> Cc: linux-nfs@vger.kernel.org
> Subject: Re: [PATCH v1 01/19] NFS: Never use user credentials for lease
> renewal
> 
> 
> On Jul 22, 2013, at 2:33 PM, "Myklebust, Trond"
> <Trond.Myklebust@netapp.com> wrote:
> 
> > On Fri, 2013-07-12 at 12:31 -0400, Chuck Lever wrote:
> >> Don't try to use a non-UID-0 user credential for lease management, as
> >> that credential can change out from under us.  The server will block
> >> NFSv4 lease recovery with NFS4ERR_CLID_INUSE.
> >>
> >> Replace minor version-specific callouts with a single function.
> >>
> >> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> >> ---
> >> fs/nfs/nfs4_fs.h     |    4 +---
> >> fs/nfs/nfs4proc.c    |    6 +----
> >> fs/nfs/nfs4session.c |    2 +-
> >> fs/nfs/nfs4state.c   |   55 ++++++--------------------------------------------
> >> 4 files changed, 10 insertions(+), 57 deletions(-)
> >
> > This won't apply to upstream kernels.
> 
> Thanks for having a look... As the cover letter stated, this series applies to
> 3.10.
> 
> Now that 3.11-rc2 is available, I'm porting the series to the latest upstream,
> with appropriate adjustments for NFSv4.2 support.  I hope to post the new
> series tomorrow or Wednesday.
> 
OK. Please note that I've already applied 02/19 as that seems to be unrelated to anything else.

Thanks
  Trond

Re: [PATCH v1 01/19] NFS: Never use user credentials for lease renewal

[Chuck Lever]

On Jul 22, 2013, at 2:53 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

>> -----Original Message-----
>> From: Chuck Lever [mailto:chuck.lever@oracle.com]
>> Sent: Monday, July 22, 2013 2:52 PM
>> To: Myklebust, Trond
>> Cc: linux-nfs@vger.kernel.org
>> Subject: Re: [PATCH v1 01/19] NFS: Never use user credentials for lease
>> renewal
>> 
>> 
>> On Jul 22, 2013, at 2:33 PM, "Myklebust, Trond"
>> <Trond.Myklebust@netapp.com> wrote:
>> 
>>> On Fri, 2013-07-12 at 12:31 -0400, Chuck Lever wrote:
>>>> Don't try to use a non-UID-0 user credential for lease management, as
>>>> that credential can change out from under us.  The server will block
>>>> NFSv4 lease recovery with NFS4ERR_CLID_INUSE.
>>>> 
>>>> Replace minor version-specific callouts with a single function.
>>>> 
>>>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>>>> ---
>>>> fs/nfs/nfs4_fs.h     |    4 +---
>>>> fs/nfs/nfs4proc.c    |    6 +----
>>>> fs/nfs/nfs4session.c |    2 +-
>>>> fs/nfs/nfs4state.c   |   55 ++++++--------------------------------------------
>>>> 4 files changed, 10 insertions(+), 57 deletions(-)
>>> 
>>> This won't apply to upstream kernels.
>> 
>> Thanks for having a look... As the cover letter stated, this series applies to
>> 3.10.
>> 
>> Now that 3.11-rc2 is available, I'm porting the series to the latest upstream,
>> with appropriate adjustments for NFSv4.2 support.  I hope to post the new
>> series tomorrow or Wednesday.
>> 
> OK. Please note that I've already applied 02/19 as that seems to be unrelated to anything else.

I'll drop 02/19 from my resend.  Thanks.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH v1 07/19] NFS: Fix compiler warning in nfs_setup_sequence()

[Myklebust, Trond]

T24gRnJpLCAyMDEzLTA3LTEyIGF0IDEyOjMyIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
Q2xlYW4gdXAuICBTcXVlbGNoIGNvbXBpbGVyIHdhcm5pbmc6DQo+IA0KPiBsaW51eC9mcy9uZnMv
bmZzNHByb2MuYzogSW4gZnVuY3Rpb24g4oCYbmZzNF9zZXR1cF9zZXF1ZW5jZeKAmToNCj4gbGlu
dXgvZnMvbmZzL25mczRwcm9jLmM6NzAzOjI6IHdhcm5pbmc6IHNpZ25lZCBhbmQgdW5zaWduZWQg
dHlwZSBpbg0KPiAJY29uZGl0aW9uYWwgZXhwcmVzc2lvbiBbLVdzaWduLWNvbXBhcmVdDQo+IA0K
PiBTaWduZWQtb2ZmLWJ5OiBDaHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4NCj4g
LS0tDQo+ICBmcy9uZnMvbmZzNHByb2MuYyB8ICAgIDIgKy0NCj4gIDEgZmlsZSBjaGFuZ2VkLCAx
IGluc2VydGlvbigrKSwgMSBkZWxldGlvbigtKQ0KPiANCj4gZGlmZiAtLWdpdCBhL2ZzL25mcy9u
ZnM0cHJvYy5jIGIvZnMvbmZzL25mczRwcm9jLmMNCj4gaW5kZXggN2IyZTAwOC4uNTkxM2UxZCAx
MDA2NDQNCj4gLS0tIGEvZnMvbmZzL25mczRwcm9jLmMNCj4gKysrIGIvZnMvbmZzL25mczRwcm9j
LmMNCj4gQEAgLTY1NSw3ICs2NTUsNyBAQCBzdGF0aWMgaW50IG5mczRfc2V0dXBfc2VxdWVuY2Uo
Y29uc3Qgc3RydWN0IG5mc19zZXJ2ZXIgKnNlcnZlciwNCj4gIA0KPiAgCWRwcmludGsoIi0tPiAl
cyBjbHAgJXAgc2Vzc2lvbiAlcCBzcl9zbG90ICVkXG4iLA0KPiAgCQlfX2Z1bmNfXywgc2Vzc2lv
bi0+Y2xwLCBzZXNzaW9uLCByZXMtPnNyX3Nsb3QgPw0KPiAtCQkJcmVzLT5zcl9zbG90LT5zbG90
X25yIDogLTEpOw0KPiArCQkJKGludClyZXMtPnNyX3Nsb3QtPnNsb3RfbnIgOiAtMSk7DQo+ICAN
Cg0KUGxlYXNlIGNvbnZlcnQgdG8gbWFrZSBpdCB1bnNpZ25lZCwgYW5kIHRvIHVzZSBORlM0X05P
X1NMT1QgaW5zdGVhZCBvZg0KJy0xJy4NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5G
UyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29t
DQp3d3cubmV0YXBwLmNvbQ0K

Re: [PATCH v1 04/19] NFS: Refactor nfs4_call_data_sequence()

[Myklebust, Trond]

T24gRnJpLCAyMDEzLTA3LTEyIGF0IDEyOjMyIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
UHVsbCBtYWluIGxvZ2ljIG91dCBvZiBuZnM0X2NhbGxfZGF0YV9zZXF1ZW5jZSgpIHNvIGl0IGNh
bg0KPiBldmVudHVhbGx5IGJlIHNoYXJlZCB3aXRoIHRoZSBORlN2NC4wIGNhbGxfc3luYyBwYXRo
LiAgVGhlIG5ldw0KPiBoZWxwZXIgZnVuY3Rpb24gaXMgbGVmdCBiZWhpbmQgdGhlIE5GU3Y0LjEg
Q09ORklHIHN3aXRjaCBmb3Igbm93IHRvDQo+IGtlZXAgY29tcGlsZXIgd2FybmluZyBub2lzZSBh
dCBhIG1pbmltdW0uDQo+IA0KPiBTaWduZWQtb2ZmLWJ5OiBDaHVjayBMZXZlciA8Y2h1Y2subGV2
ZXJAb3JhY2xlLmNvbT4NCj4gLS0tDQo+ICBmcy9uZnMvbmZzNHByb2MuYyB8ICAgMjggKysrKysr
KysrKysrKysrKysrLS0tLS0tLS0tLQ0KPiAgMSBmaWxlIGNoYW5nZWQsIDE4IGluc2VydGlvbnMo
KyksIDEwIGRlbGV0aW9ucygtKQ0KPiANCj4gZGlmZiAtLWdpdCBhL2ZzL25mcy9uZnM0cHJvYy5j
IGIvZnMvbmZzL25mczRwcm9jLmMNCj4gaW5kZXggMmY2ZjNiOS4uZTUyZDJiMyAxMDA2NDQNCj4g
LS0tIGEvZnMvbmZzL25mczRwcm9jLmMNCj4gKysrIGIvZnMvbmZzL25mczRwcm9jLmMNCj4gQEAg
LTM5Myw2ICszOTMsMjMgQEAgc3RhdGljIHZvaWQgcmVuZXdfbGVhc2UoY29uc3Qgc3RydWN0IG5m
c19zZXJ2ZXIgKnNlcnZlciwgdW5zaWduZWQgbG9uZyB0aW1lc3RhbXANCj4gIA0KPiAgI2lmIGRl
ZmluZWQoQ09ORklHX05GU19WNF8xKQ0KPiAgDQo+ICtzdGF0aWMgaW50IG5mczRfcnVuX3JwY190
YXNrKHN0cnVjdCBycGNfY2xudCAqY2xudCwNCj4gKwkJCSAgICAgc3RydWN0IHJwY19tZXNzYWdl
ICptc2csDQo+ICsJCQkgICAgIHN0cnVjdCBycGNfdGFza19zZXR1cCAqdGFza19zZXR1cCkNCj4g
K3sNCj4gKwlzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2s7DQo+ICsJaW50IHJldDsNCj4gKw0KPiArCXRh
c2sgPSBycGNfcnVuX3Rhc2sodGFza19zZXR1cCk7DQo+ICsJaWYgKElTX0VSUih0YXNrKSkNCj4g
KwkJcmV0ID0gUFRSX0VSUih0YXNrKTsNCj4gKwllbHNlIHsNCj4gKwkJcmV0ID0gdGFzay0+dGtf
c3RhdHVzOw0KPiArCQlycGNfcHV0X3Rhc2sodGFzayk7DQo+ICsJfQ0KPiArCXJldHVybiByZXQ7
DQo+ICt9DQoNClBsZWFzZSBkb24ndCBzcGxpdCB0aGlzIG91dC4gRXZlbnR1YWxseSwgc29tZW9u
ZSBpcyBnb2luZyB0byBjYWxsIHRoaXMNCndpdGggdGhlIFJQQ19UQVNLX0FTWU5DIGZsYWcgc2V0
LCBhbmQgYWxsIGhlbGwgd2lsbCBicmVhayBsb29zZS4NCg0KPiArDQo+ICBzdGF0aWMgdm9pZCBu
ZnM0MV9zZXF1ZW5jZV9mcmVlX3Nsb3Qoc3RydWN0IG5mczRfc2VxdWVuY2VfcmVzICpyZXMpDQo+
ICB7DQo+ICAJc3RydWN0IG5mczRfc2Vzc2lvbiAqc2Vzc2lvbjsNCj4gQEAgLTY3NCw4ICs2OTEs
NiBAQCBzdGF0aWMgaW50IG5mczRfY2FsbF9zeW5jX3NlcXVlbmNlKHN0cnVjdCBycGNfY2xudCAq
Y2xudCwNCj4gIAkJCQkgICBzdHJ1Y3QgbmZzNF9zZXF1ZW5jZV9hcmdzICphcmdzLA0KPiAgCQkJ
CSAgIHN0cnVjdCBuZnM0X3NlcXVlbmNlX3JlcyAqcmVzKQ0KPiAgew0KPiAtCWludCByZXQ7DQo+
IC0Jc3RydWN0IHJwY190YXNrICp0YXNrOw0KPiAgCXN0cnVjdCBuZnM0MV9jYWxsX3N5bmNfZGF0
YSBkYXRhID0gew0KPiAgCQkuc2VxX3NlcnZlciA9IHNlcnZlciwNCj4gIAkJLnNlcV9hcmdzID0g
YXJncywNCj4gQEAgLTY4OCwxNCArNzAzLDcgQEAgc3RhdGljIGludCBuZnM0X2NhbGxfc3luY19z
ZXF1ZW5jZShzdHJ1Y3QgcnBjX2NsbnQgKmNsbnQsDQo+ICAJCS5jYWxsYmFja19kYXRhID0gJmRh
dGENCj4gIAl9Ow0KPiAgDQo+IC0JdGFzayA9IHJwY19ydW5fdGFzaygmdGFza19zZXR1cCk7DQo+
IC0JaWYgKElTX0VSUih0YXNrKSkNCj4gLQkJcmV0ID0gUFRSX0VSUih0YXNrKTsNCj4gLQllbHNl
IHsNCj4gLQkJcmV0ID0gdGFzay0+dGtfc3RhdHVzOw0KPiAtCQlycGNfcHV0X3Rhc2sodGFzayk7
DQo+IC0JfQ0KPiAtCXJldHVybiByZXQ7DQo+ICsJcmV0dXJuIG5mczRfcnVuX3JwY190YXNrKGNs
bnQsIG1zZywgJnRhc2tfc2V0dXApOw0KPiAgfQ0KPiAgDQo+ICAjZWxzZQ0KPiANCj4gLS0NCj4g
VG8gdW5zdWJzY3JpYmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJl
IGxpbnV4LW5mcyIgaW4NCj4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2Vy
Lmtlcm5lbC5vcmcNCj4gTW9yZSBtYWpvcmRvbW8gaW5mbyBhdCAgaHR0cDovL3ZnZXIua2VybmVs
Lm9yZy9tYWpvcmRvbW8taW5mby5odG1sDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBO
RlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNv
bQ0Kd3d3Lm5ldGFwcC5jb20NCg==

Re: [PATCH v1 09/19] NFS: Add a "struct nfs_server *" argument to nfs4_sequence_done()

[Myklebust, Trond]

T24gRnJpLCAyMDEzLTA3LTEyIGF0IDEyOjMyIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
Rm9yIE5GU3Y0LjEsIGEgc2Vzc2lvbiBzbG90IHRhYmxlIHJlZmVyZW5jZSBpcyBwYXNzZWQgdG8N
Cj4gbmZzNF9zZXF1ZW5jZV9kb25lKCkuDQo+IA0KPiBGb3IgTkZTdjQuMCwgdHJhbnNwb3J0IHN0
YXRlIGlzIG1hbmFnZWQgaW4gdGhlIG5mc19jbGllbnQgdGhhdCBvd25zDQo+IHRoaXMgbmZzX3Nl
cnZlci4gIFRoZSBuZnNfc2VydmVyIHN0cnVjdCBtdXN0IGJlIHBhc3NlZCB0bw0KPiBuZnM0X3Nl
cXVlbmNlX2RvbmUoKSB0byBtYWtlIGl0IGF2YWlsYWJsZSB0byB0aGUgZnVuY3Rpb25zIGRlYWxp
bmcNCj4gd2l0aCB0cmFuc3BvcnQgYmxvY2tpbmcuDQoNCldoeT8gQ2FuJ3QgNC4wIGp1c3QgcmV1
c2UgdGhlIGV4aXN0aW5nIHY0LjEgc2Vzc2lvbiBzbG90IHRhYmxlIGZvciB0aGlzPw0KDQotLSAN
ClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0K
VHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20NCnd3dy5uZXRhcHAuY29tDQo=

Re: [PATCH v1 07/19] NFS: Fix compiler warning in nfs_setup_sequence()

[Chuck Lever]

On Jul 22, 2013, at 3:03 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

> On Fri, 2013-07-12 at 12:32 -0400, Chuck Lever wrote:
>> Clean up.  Squelch compiler warning:
>> 
>> linux/fs/nfs/nfs4proc.c: In function ‘nfs4_setup_sequence’:
>> linux/fs/nfs/nfs4proc.c:703:2: warning: signed and unsigned type in
>> 	conditional expression [-Wsign-compare]
>> 
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>> fs/nfs/nfs4proc.c |    2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>> index 7b2e008..5913e1d 100644
>> --- a/fs/nfs/nfs4proc.c
>> +++ b/fs/nfs/nfs4proc.c
>> @@ -655,7 +655,7 @@ static int nfs4_setup_sequence(const struct nfs_server *server,
>> 
>> 	dprintk("--> %s clp %p session %p sr_slot %d\n",
>> 		__func__, session->clp, session, res->sr_slot ?
>> -			res->sr_slot->slot_nr : -1);
>> +			(int)res->sr_slot->slot_nr : -1);
>> 
> 
> Please convert to make it unsigned, and to use NFS4_NO_SLOT instead of
> '-1'.

NFS4_NO_SLOT makes sense, but I'm not clear on "make it unsigned".

Leaving the format as "%d" allows NFS4_NO_SLOT to be displayed as "-1".  So, something like the following one-liner will address the compiler warning, and leave behavior unchanged:

-			res->sr_slot->slot_nr : -1);
+			res->sr_slot->slot_nr : NFS4_NO_SLOT);

That is, unless you prefer NFS4_NO_SLOT displayed in some other way?

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH v1 07/19] NFS: Fix compiler warning in nfs_setup_sequence()

[Myklebust, Trond]

T24gTW9uLCAyMDEzLTA3LTIyIGF0IDE2OjE2IC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
T24gSnVsIDIyLCAyMDEzLCBhdCAzOjAzIFBNLCAiTXlrbGVidXN0LCBUcm9uZCIgPFRyb25kLk15
a2xlYnVzdEBuZXRhcHAuY29tPiB3cm90ZToNCj4gDQo+ID4gT24gRnJpLCAyMDEzLTA3LTEyIGF0
IDEyOjMyIC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4gPj4gQ2xlYW4gdXAuICBTcXVlbGNo
IGNvbXBpbGVyIHdhcm5pbmc6DQo+ID4+IA0KPiA+PiBsaW51eC9mcy9uZnMvbmZzNHByb2MuYzog
SW4gZnVuY3Rpb24g4oCYbmZzNF9zZXR1cF9zZXF1ZW5jZeKAmToNCj4gPj4gbGludXgvZnMvbmZz
L25mczRwcm9jLmM6NzAzOjI6IHdhcm5pbmc6IHNpZ25lZCBhbmQgdW5zaWduZWQgdHlwZSBpbg0K
PiA+PiAJY29uZGl0aW9uYWwgZXhwcmVzc2lvbiBbLVdzaWduLWNvbXBhcmVdDQo+ID4+IA0KPiA+
PiBTaWduZWQtb2ZmLWJ5OiBDaHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT4NCj4g
Pj4gLS0tDQo+ID4+IGZzL25mcy9uZnM0cHJvYy5jIHwgICAgMiArLQ0KPiA+PiAxIGZpbGUgY2hh
bmdlZCwgMSBpbnNlcnRpb24oKyksIDEgZGVsZXRpb24oLSkNCj4gPj4gDQo+ID4+IGRpZmYgLS1n
aXQgYS9mcy9uZnMvbmZzNHByb2MuYyBiL2ZzL25mcy9uZnM0cHJvYy5jDQo+ID4+IGluZGV4IDdi
MmUwMDguLjU5MTNlMWQgMTAwNjQ0DQo+ID4+IC0tLSBhL2ZzL25mcy9uZnM0cHJvYy5jDQo+ID4+
ICsrKyBiL2ZzL25mcy9uZnM0cHJvYy5jDQo+ID4+IEBAIC02NTUsNyArNjU1LDcgQEAgc3RhdGlj
IGludCBuZnM0X3NldHVwX3NlcXVlbmNlKGNvbnN0IHN0cnVjdCBuZnNfc2VydmVyICpzZXJ2ZXIs
DQo+ID4+IA0KPiA+PiAJZHByaW50aygiLS0+ICVzIGNscCAlcCBzZXNzaW9uICVwIHNyX3Nsb3Qg
JWRcbiIsDQo+ID4+IAkJX19mdW5jX18sIHNlc3Npb24tPmNscCwgc2Vzc2lvbiwgcmVzLT5zcl9z
bG90ID8NCj4gPj4gLQkJCXJlcy0+c3Jfc2xvdC0+c2xvdF9uciA6IC0xKTsNCj4gPj4gKwkJCShp
bnQpcmVzLT5zcl9zbG90LT5zbG90X25yIDogLTEpOw0KPiA+PiANCj4gPiANCj4gPiBQbGVhc2Ug
Y29udmVydCB0byBtYWtlIGl0IHVuc2lnbmVkLCBhbmQgdG8gdXNlIE5GUzRfTk9fU0xPVCBpbnN0
ZWFkIG9mDQo+ID4gJy0xJy4NCj4gDQo+IE5GUzRfTk9fU0xPVCBtYWtlcyBzZW5zZSwgYnV0IEkn
bSBub3QgY2xlYXIgb24gIm1ha2UgaXQgdW5zaWduZWQiLg0KPiANCj4gTGVhdmluZyB0aGUgZm9y
bWF0IGFzICIlZCIgYWxsb3dzIE5GUzRfTk9fU0xPVCB0byBiZSBkaXNwbGF5ZWQgYXMgIi0xIi4g
IFNvLCBzb21ldGhpbmcgbGlrZSB0aGUgZm9sbG93aW5nIG9uZS1saW5lciB3aWxsIGFkZHJlc3Mg
dGhlIGNvbXBpbGVyIHdhcm5pbmcsIGFuZCBsZWF2ZSBiZWhhdmlvciB1bmNoYW5nZWQ6DQo+IA0K
PiAtCQkJcmVzLT5zcl9zbG90LT5zbG90X25yIDogLTEpOw0KPiArCQkJcmVzLT5zcl9zbG90LT5z
bG90X25yIDogTkZTNF9OT19TTE9UKTsNCj4gDQo+IFRoYXQgaXMsIHVubGVzcyB5b3UgcHJlZmVy
IE5GUzRfTk9fU0xPVCBkaXNwbGF5ZWQgaW4gc29tZSBvdGhlciB3YXk/DQoNCkJvdGggdGhlIHNs
b3QgaWQgYW5kIE5GUzRfTk9fU0xPVCBhcmUgZGVmaW5lZCBhcyBiZWluZyBvZiB0eXBlIHUzMiwg
c28NCml0IG1ha2VzIG5vIHNlbnNlIHRvIGRpc3BsYXkgYSAiLTEiLiBKdXN0IGRpc3BsYXlpbmcg
aXQgYXMgMl4zMiBzaG91bGQNCmJlIGdvb2QgZW5vdWdoIHNpbmNlIHRoaXMgaXMgYW4gb2J2aW91
cyAiZGV2ZWxvcGVyIG9ubHkiIGRwcmludGsuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51
eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBw
LmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg==

Re: [PATCH v1 07/19] NFS: Fix compiler warning in nfs_setup_sequence()

[Chuck Lever]

On Jul 22, 2013, at 4:22 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

> On Mon, 2013-07-22 at 16:16 -0400, Chuck Lever wrote:
>> On Jul 22, 2013, at 3:03 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:
>> 
>>> On Fri, 2013-07-12 at 12:32 -0400, Chuck Lever wrote:
>>>> Clean up.  Squelch compiler warning:
>>>> 
>>>> linux/fs/nfs/nfs4proc.c: In function ‘nfs4_setup_sequence’:
>>>> linux/fs/nfs/nfs4proc.c:703:2: warning: signed and unsigned type in
>>>> 	conditional expression [-Wsign-compare]
>>>> 
>>>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>>>> ---
>>>> fs/nfs/nfs4proc.c |    2 +-
>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>> 
>>>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>>>> index 7b2e008..5913e1d 100644
>>>> --- a/fs/nfs/nfs4proc.c
>>>> +++ b/fs/nfs/nfs4proc.c
>>>> @@ -655,7 +655,7 @@ static int nfs4_setup_sequence(const struct nfs_server *server,
>>>> 
>>>> 	dprintk("--> %s clp %p session %p sr_slot %d\n",
>>>> 		__func__, session->clp, session, res->sr_slot ?
>>>> -			res->sr_slot->slot_nr : -1);
>>>> +			(int)res->sr_slot->slot_nr : -1);
>>>> 
>>> 
>>> Please convert to make it unsigned, and to use NFS4_NO_SLOT instead of
>>> '-1'.
>> 
>> NFS4_NO_SLOT makes sense, but I'm not clear on "make it unsigned".
>> 
>> Leaving the format as "%d" allows NFS4_NO_SLOT to be displayed as "-1".  So, something like the following one-liner will address the compiler warning, and leave behavior unchanged:
>> 
>> -			res->sr_slot->slot_nr : -1);
>> +			res->sr_slot->slot_nr : NFS4_NO_SLOT);
>> 
>> That is, unless you prefer NFS4_NO_SLOT displayed in some other way?
> 
> Both the slot id and NFS4_NO_SLOT are defined as being of type u32, so
> it makes no sense to display a "-1". Just displaying it as 2^32 should
> be good enough since this is an obvious "developer only" dprintk.

OK, clear.  Thanks.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH v1 09/19] NFS: Add a "struct nfs_server *" argument to nfs4_sequence_done()

[Chuck Lever]

On Jul 22, 2013, at 3:27 PM, "Myklebust, Trond" <Trond.Myklebust@netapp.com> wrote:

> On Fri, 2013-07-12 at 12:32 -0400, Chuck Lever wrote:
>> For NFSv4.1, a session slot table reference is passed to
>> nfs4_sequence_done().
>> 
>> For NFSv4.0, transport state is managed in the nfs_client that owns
>> this nfs_server.  The nfs_server struct must be passed to
>> nfs4_sequence_done() to make it available to the functions dealing
>> with transport blocking.
> 
> Why? Can't 4.0 just reuse the existing v4.1 session slot table for this?

Before commit 774d5f14 "NFSv4.1 Fix a pNFS session draining deadlock," Mon May 20 14:13:50 2013, the DRAINING bit was in nfs4_session, not in nfs4_slot_table.  Before that commit, I would have had to enable most of the NFSv4.1 sessions code to re-use it for NFSv4.0 transport blocking.

(And btw, thanks Andy!  774d5f14 is a very useful refactoring).

I spent the last couple of days re-implementing my NFSv4.0 transport blocking mechanism based on the slot table abstraction, rather than adding similar fields to the nfs_client.  At this point, it looks like a workable approach, but I won't get it finished and tested before I head to Berlin.  I may have some time to continue work while I'm there.

Since our town hall meeting is cancelled this week, let me report on progress with migration support.

I was hoping to publish my 3.11-rc port of the migration patches today for another round of review, but I'm not going to make it due to the complexity of rebasing the transport blocking mechanism directly on struct nfs4_slot_table.  It will be at the top of my list when I get back.

I was able to check off some of the test cases I mentioned in the cover letter for the v1 patch series:

Migration with no TSM:  identified and fixed a client bug, then was able to witness state recovery during a successful migration.  The server prototype currently reports NFS4ERR_EXPIRED, so state recovery in this case involves OPEN(CLAIM_NULL) - the server is not in a grace period, and it is possible for clients to lose their locks.

Migration during a lock-intensive workload:  the server prototype is still attempting to put off client requests without using NFS4ERR_DELAY, which causes the client workload to terminate; waiting for that to be addressed, then will try again.

Migration recovery failure:  identified the need for more infrastructure to communicate state manager failure back to forward NFS processes.  That infrastructure has been added, but not yet tested.

The idea is that all I/O on the mount should fail after a migration, but an admin should be able to unmount cleanly.  In other words, manual recovery by remounting the FSID on the destination server, which is what would happen in the pre-migration world.

Migration with Kerberos:  exploring KDC resources available for testing my client with the server prototypes that are located in our Austin lab.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com