[PATCH v3 00/42] NFS/NFSD support for async and inter COPY

[PATCH v3 00/42] NFS/NFSD support for async and inter COPY

[Olga Kornievskaia]

This patch series provides support for NFSv4.2 COPY featuring support
for asynchronous copy and inter SSC copy.

In this iteration including both client and server side in one series
because server side does depend on client side patches. 

Client side:
In case, of the "inter" SSC copy files reside on different servers and
thus under different superblocks and require that VFS removes the
restriction that src and dst files must be on the same superblock.

NFS's copy_file_range() determines if the copy is "intra" or "inter"
and for "inter" it sends the COPY_NOTIFY to the source server. Then,
it would send of an asynchronous COPY to the server (destination in
case of "inter"). If server errs with ERR_OFFLOAD_NOREQS the copy will
be re-sent as a synchronous COPY. If application cancels an in-flight
COPY, OFFLOAD_CANCEL is sent to the source server.

If server replies to the COPY with the copy stateid, client will go
wait on the CB_OFFLOAD. To fend off the race between CB_OFFLOAD and
COPY reply, we check the list of pending callbacks before going to
wait. Client adds the copy to the global list of copy stateids for the
callback to look thru and signal the waiting copy.

If application cancels async COPY after reply is received, wait will be
interrupted and client will send OFFLOAD_CANCEL to the source and
destination servers (sending it as an async RPC in the context of the
nfsiod_workqueue).

When the client receives reply from the CB_OFFLOAD with some bytes and
committed how is UNSTABLE, then COMMIT is sent to the server. The results
are propagated to the VFS and application. Assuming that application
will deal with a partial result and continue from the new offset if needed.

Handling reboot of the destination server when client is waiting on the
CB_OFFLOAD happens when SEQUENCE discovers that destination server rebooted.
The open state initially is marked to be NFS_CLNT_DST_SSC_COPY_STATE
during the COPY. Then during the recovery if state is marked as such,
then look thru the list of copies for the server and see if any are
associated with this recovering open, if so mark the copy rebooted and
wake up the waiting copy. Upon wake up the waiting copy, will restart the
copy from scratch.

If the source server is rebooted, the destination server will also know
about it and it will return the partial result via CB_OFFLOAD, then the
result will be propagated back to the application which will initiate
the new copy and new COPY_NOTIFY will be sent.

If CB_OFFLOAD returned an error and non negative value of partial copy
and error is not ENOSPC, then ignore the error and send the commit
and return partial result to the client to start the next copy.

On the destination server, it's now acting as a client and needs to do
a special "open" and "close". Since destination server doesn't do an open
on the wire, we "fake" create the needed data structures and that's done
in the new function nfs42_ssc_open(). To clean up this open but not trigger
the CLOSE on the wire, we have a new function nfs42_ssc_close() that
accomplishes that.

Server side:
This is server-side support for NFSv4.2 inter and async COPY which is
on top of existing intra sync COPY. It also depends on the NFS client
piece for NFSv4.2 to do client side of the destination server piece
in the inter SSC.

NFSD determines if COPY is intra or inter and if sync or async. For
inter, NSFD uses NFSv4.1 protocol and creates an internal mount point
(superblock).

To do asynchronous copies, NFSD creates a single threaded workqueue
and does not tie up an NFSD thread to complete the copy. Upon receiving
the COPY, it generates a unique copy stateid (stores a global list
for keeping track of state for OFFLOAD_STATUS to be queried by),
queues up a workqueue for the copy, and replies back to the client.
nfsd4_copy arguments that are allocated on the stack are copied for
the work item.

In the async copy handler, it calls into VFS copy_file_range() and
loops until it completes the requested copy size. If error is
encountered it's saved but also we save the amount of data copied
so far. Once done, the results are queued for the callback
workqueue and sent via CB_OFFLOAD. Also currently, choosing to clean
up the copy state information stored in the global list when cope is
done and not doing it when callback's release function (it could be
done there alternatively if needed it?).

On the source server, upon receiving a COPY_NOTIFY, it generate a
unique stateid that's kept in the global list. Upon receiving a READ
with a stateid, the code checks the normal list of open stateid and
now additionally, it'll check the copy state list as well before
deciding to either fail with BAD_STATEID or find one that matches.
The stored stateid is only valid to be used for the first time
with a choosen lease period (90s currently). When the source server
received an OFFLOAD_CANCEL, it will remove the stateid from the
global list. Otherwise, the copy stateid is removed upon the removal
of its "parent" stateid (open/lock/delegation stateid).

v3
Add support to delay the unmount of the source server on the destination
server for 90s to allow for consecutive COPY operations to reuse the 
session/clientid.  

Andy Adamson (9):
  NFS inter ssc open
  NFS add COPY_NOTIFY operation
  NFSD add ca_source_server<> to COPY
  NFSD generalize nfsd4_compound_state flag names
  NFSD add COPY_NOTIFY operation
  NFSD: allow inter server COPY to have a STALE source server fh
  NFSD add nfs4 inter ssc to nfsd4_copy
  NFSD return nfs4_stid in nfs4_preprocess_stateid_op
  NFSD Unique stateid_t for inter server to server COPY authentication

Anna Schumaker (1):
  fs: Don't copy beyond the end of the file

Olga Kornievskaia (32):
  VFS permit cross device vfs_copy_file_range
  NFS CB_OFFLOAD xdr
  NFS OFFLOAD_STATUS xdr
  NFS OFFLOAD_STATUS op
  NFS OFFLOAD_CANCEL xdr
  NFS COPY xdr handle async reply
  NFS add support for asynchronous COPY
  NFS handle COPY reply CB_OFFLOAD call race
  NFS export nfs4_async_handle_error
  NFS test for intra vs inter COPY
  NFS send OFFLOAD_CANCEL when COPY killed
  NFS handle COPY ERR_OFFLOAD_NO_REQS
  NFS if we got partial copy ignore errors
  NFS recover from destination server reboot for copies
  NFS NFSD defining nl4_servers structure needed by both
  NFS also send OFFLOAD_CANCEL to source server
  NFS skip recovery of copy open on dest server
  NFS also send OFFLOAD_CANCEL to source server
  NFS add a simple sync nfs4_proc_commit after async COPY
  NFSD CB_OFFLOAD xdr
  NFSD OFFLOAD_STATUS xdr
  NFSD OFFLOAD_CANCEL xdr
  NFSD xdr callback stateid in async COPY reply
  NFSD first draft of async copy
  NFSD handle OFFLOAD_CANCEL op
  NFSD stop queued async copies on client shutdown
  NFSD create new stateid for async copy
  NFSD define EBADF in nfserrno
  NFSD support OFFLOAD_STATUS
  NFSD remove copy stateid when vfs_copy_file_range completes
  NFSD delay the umount after COPY

 Documentation/filesystems/vfs.txt |   6 +
 fs/nfs/callback.h                 |  13 +
 fs/nfs/callback_proc.c            |  52 +++
 fs/nfs/callback_xdr.c             |  80 ++++-
 fs/nfs/client.c                   |   1 +
 fs/nfs/internal.h                 |  10 +
 fs/nfs/nfs42.h                    |  10 +-
 fs/nfs/nfs42proc.c                | 380 +++++++++++++++++++-
 fs/nfs/nfs42xdr.c                 | 388 +++++++++++++++++++-
 fs/nfs/nfs4_fs.h                  |  18 +-
 fs/nfs/nfs4client.c               |  15 +
 fs/nfs/nfs4file.c                 | 146 +++++++-
 fs/nfs/nfs4proc.c                 |  44 ++-
 fs/nfs/nfs4state.c                |  31 +-
 fs/nfs/nfs4xdr.c                  |   3 +
 fs/nfsd/Kconfig                   |  10 +
 fs/nfsd/netns.h                   |   8 +
 fs/nfsd/nfs4callback.c            |  95 +++++
 fs/nfsd/nfs4proc.c                | 727 ++++++++++++++++++++++++++++++++++++--
 fs/nfsd/nfs4state.c               | 149 +++++++-
 fs/nfsd/nfs4xdr.c                 | 266 +++++++++++++-
 fs/nfsd/nfsctl.c                  |   2 +
 fs/nfsd/nfsd.h                    |   2 +
 fs/nfsd/nfsproc.c                 |   1 +
 fs/nfsd/state.h                   |  34 +-
 fs/nfsd/xdr4.h                    |  61 +++-
 fs/nfsd/xdr4cb.h                  |  10 +
 fs/read_write.c                   |  16 +-
 include/linux/nfs4.h              |  37 ++
 include/linux/nfs_fs.h            |  11 +
 include/linux/nfs_fs_sb.h         |   5 +
 include/linux/nfs_xdr.h           |  33 ++
 32 files changed, 2563 insertions(+), 101 deletions(-)

-- 
1.8.3.1

[RFC v3 01/42] fs: Don't copy beyond the end of the file

[Olga Kornievskaia]

From: Anna Schumaker <Anna.Schumaker@Netapp.com>

Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 fs/read_write.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/read_write.c b/fs/read_write.c
index 19d4d88..cc2618b 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -1584,6 +1584,9 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	if (unlikely(ret))
 		return ret;
 
+	if (pos_in >= i_size_read(inode_in))
+		return -EINVAL;
+
 	if (!(file_in->f_mode & FMODE_READ) ||
 	    !(file_out->f_mode & FMODE_WRITE) ||
 	    (file_out->f_flags & O_APPEND))
-- 
1.8.3.1

[RFC v3 02/42] VFS permit cross device vfs_copy_file_range

[Olga Kornievskaia]

Allow copy_file_range to copy between different superblocks but only
of the same file system types.

This feature is needed by NFSv4.2 to perform file copy operation on
the same server or file copy between different NFSv4.2 servers.

If a file system's fileoperations copy_file_range operation prohibits
cross-device copies, fall back to do_splice_direct. This would be
needed for the NFS (destination) server side implementation of the
file copy and currently for CIFS.

Besides NFS, there is only 1 implementor of the copy_file_range FS
operation -- CIFS. CIFS assumes incoming file descriptors are both
CIFS but it will check if they are coming from different servers and
return error code to fall back to do_splice_direct.

NFS will allow for copies between different NFS servers.

Adding to the vfs.txt documentation to explicitly warn about allowing
for different superblocks of the same file type to be passed into the
copy_file_range for the future users of copy_file_range method.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 Documentation/filesystems/vfs.txt |  6 ++++++
 fs/read_write.c                   | 13 ++++++-------
 2 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt
index f42b906..fd6b105 100644
--- a/Documentation/filesystems/vfs.txt
+++ b/Documentation/filesystems/vfs.txt
@@ -845,6 +845,8 @@ struct file_operations {
 #ifndef CONFIG_MMU
 	unsigned (*mmap_capabilities)(struct file *);
 #endif
+	ssize_t (*copy_file_range)(struct file *, loff_t, struct file *,
+				   loff_t, size_t, unsigned int);
 };
 
 Again, all methods are called without any locks being held, unless
@@ -913,6 +915,10 @@ otherwise noted.
 
   fallocate: called by the VFS to preallocate blocks or punch a hole.
 
+  copy_file_range: called by copy_file_range(2) system call. This method
+		   works on two file descriptors that might reside on
+		   different superblocks of the same type of file system.
+
 Note that the file operations are implemented by the specific
 filesystem in which the inode resides. When opening a device node
 (character or block special) most filesystems will call special
diff --git a/fs/read_write.c b/fs/read_write.c
index cc2618b..406d1c6 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -1592,10 +1592,6 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	    (file_out->f_flags & O_APPEND))
 		return -EBADF;
 
-	/* this could be relaxed once a method supports cross-fs copies */
-	if (inode_in->i_sb != inode_out->i_sb)
-		return -EXDEV;
-
 	if (len == 0)
 		return 0;
 
@@ -1605,7 +1601,8 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 	 * Try cloning first, this is supported by more file systems, and
 	 * more efficient if both clone and copy are supported (e.g. NFS).
 	 */
-	if (file_in->f_op->clone_file_range) {
+	if (inode_in->i_sb == inode_out->i_sb &&
+			file_in->f_op->clone_file_range) {
 		ret = file_in->f_op->clone_file_range(file_in, pos_in,
 				file_out, pos_out, len);
 		if (ret == 0) {
@@ -1614,10 +1611,12 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
 		}
 	}
 
-	if (file_out->f_op->copy_file_range) {
+	if (file_out->f_op->copy_file_range &&
+			(file_in->f_op->copy_file_range ==
+				file_out->f_op->copy_file_range)) {
 		ret = file_out->f_op->copy_file_range(file_in, pos_in, file_out,
 						      pos_out, len, flags);
-		if (ret != -EOPNOTSUPP)
+		if (ret != -EOPNOTSUPP && ret != -EXDEV)
 			goto done;
 	}
 
-- 
1.8.3.1

[RFC v3 03/42] NFS CB_OFFLOAD xdr

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/callback.h      | 13 ++++++++
 fs/nfs/callback_proc.c |  7 +++++
 fs/nfs/callback_xdr.c  | 80 +++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 99 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/callback.h b/fs/nfs/callback.h
index c701c30..e4ab65d 100644
--- a/fs/nfs/callback.h
+++ b/fs/nfs/callback.h
@@ -189,6 +189,19 @@ extern __be32 nfs4_callback_notify_lock(struct cb_notify_lock_args *args,
 					 void *dummy,
 					 struct cb_process_state *cps);
 #endif /* CONFIG_NFS_V4_1 */
+#ifdef CONFIG_NFS_V4_2
+struct cb_offloadargs {
+	struct nfs_fh		coa_fh;
+	nfs4_stateid		coa_stateid;
+	uint32_t		error;
+	uint64_t		wr_count;
+	struct nfs_writeverf	wr_writeverf;
+};
+
+extern __be32 nfs4_callback_offload(
+	struct cb_offloadargs *args,
+	void *dummy, struct cb_process_state *cps);
+#endif /* CONFIG_NFS_V4_2 */
 extern int check_gss_callback_principal(struct nfs_client *, struct svc_rqst *);
 extern __be32 nfs4_callback_getattr(struct cb_getattrargs *args,
 				    struct cb_getattrres *res,
diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
index 52479f1..5396751 100644
--- a/fs/nfs/callback_proc.c
+++ b/fs/nfs/callback_proc.c
@@ -647,3 +647,10 @@ __be32 nfs4_callback_notify_lock(struct cb_notify_lock_args *args, void *dummy,
 	return htonl(NFS4_OK);
 }
 #endif /* CONFIG_NFS_V4_1 */
+#ifdef CONFIG_NFS_V4_2
+__be32 nfs4_callback_offload(struct cb_offloadargs *args, void *dummy,
+				struct cb_process_state *cps)
+{
+	return 0;
+}
+#endif /* CONFIG_NFS_V4_2 */
diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c
index 390ac9c..be5ffa1 100644
--- a/fs/nfs/callback_xdr.c
+++ b/fs/nfs/callback_xdr.c
@@ -37,6 +37,9 @@
 #define CB_OP_RECALLSLOT_RES_MAXSZ	(CB_OP_HDR_RES_MAXSZ)
 #define CB_OP_NOTIFY_LOCK_RES_MAXSZ	(CB_OP_HDR_RES_MAXSZ)
 #endif /* CONFIG_NFS_V4_1 */
+#ifdef CONFIG_NFS_V4_2
+#define CB_OP_OFFLOAD_RES_MAXSZ		(CB_OP_HDR_RES_MAXSZ)
+#endif /* CONFIG_NFS_V4_2 */
 
 #define NFSDBG_FACILITY NFSDBG_CALLBACK
 
@@ -521,7 +524,72 @@ static __be32 decode_notify_lock_args(struct svc_rqst *rqstp, struct xdr_stream
 }
 
 #endif /* CONFIG_NFS_V4_1 */
+#ifdef CONFIG_NFS_V4_2
+static __be32 decode_write_response(struct xdr_stream *xdr,
+					struct cb_offloadargs *args)
+{
+	__be32 *p;
+	__be32 dummy;
+
+	/* skip the always zero field */
+	p = read_buf(xdr, 4);
+	if (unlikely(!p))
+		goto out;
+	dummy = ntohl(*p++);
+
+	/* decode count, stable_how, verifier */
+	p = xdr_inline_decode(xdr, 8 + 4);
+	if (unlikely(!p))
+		goto out;
+	p = xdr_decode_hyper(p, &args->wr_count);
+	args->wr_writeverf.committed = be32_to_cpup(p);
+	p = xdr_inline_decode(xdr, NFS4_VERIFIER_SIZE);
+	if (likely(p)) {
+		memcpy(&args->wr_writeverf.verifier.data[0], p,
+			NFS4_VERIFIER_SIZE);
+		return 0;
+	}
+out:
+	return htonl(NFS4ERR_RESOURCE);
+}
+
+static __be32 decode_offload_args(struct svc_rqst *rqstp,
+					struct xdr_stream *xdr,
+					struct cb_offloadargs *args)
+{
+	__be32 *p;
+	__be32 status;
+
+	/* decode fh */
+	status = decode_fh(xdr, &args->coa_fh);
+	if (unlikely(status != 0))
+		return status;
+
+	/* decode stateid */
+	status = decode_stateid(xdr, &args->coa_stateid);
+	if (unlikely(status != 0))
+		return status;
 
+	/* decode status */
+	p = read_buf(xdr, 4);
+	if (unlikely(!p))
+		goto out;
+	args->error = ntohl(*p++);
+	if (!args->error) {
+		status = decode_write_response(xdr, args);
+		if (unlikely(status != 0))
+			return status;
+	} else {
+		p = xdr_inline_decode(xdr, 8);
+		if (unlikely(!p))
+			goto out;
+		p = xdr_decode_hyper(p, &args->wr_count);
+	}
+	return 0;
+out:
+	return htonl(NFS4ERR_RESOURCE);
+}
+#endif /* CONFIG_NFS_V4_2 */
 static __be32 encode_string(struct xdr_stream *xdr, unsigned int len, const char *str)
 {
 	if (unlikely(xdr_stream_encode_opaque(xdr, str, len) < 0))
@@ -785,7 +853,10 @@ static void nfs4_cb_free_slot(struct cb_process_state *cps)
 	if (status != htonl(NFS4ERR_OP_ILLEGAL))
 		return status;
 
-	if (op_nr == OP_CB_OFFLOAD)
+	if (op_nr == OP_CB_OFFLOAD) {
+		*op = &callback_ops[op_nr];
+		return htonl(NFS_OK);
+	} else
 		return htonl(NFS4ERR_NOTSUPP);
 	return htonl(NFS4ERR_OP_ILLEGAL);
 }
@@ -982,6 +1053,13 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
 		.res_maxsize = CB_OP_NOTIFY_LOCK_RES_MAXSZ,
 	},
 #endif /* CONFIG_NFS_V4_1 */
+#ifdef CONFIG_NFS_V4_2
+	[OP_CB_OFFLOAD] = {
+		.process_op = (callback_process_op_t)nfs4_callback_offload,
+		.decode_args = (callback_decode_arg_t)decode_offload_args,
+		.res_maxsize = CB_OP_OFFLOAD_RES_MAXSZ,
+	},
+#endif /* CONFIG_NFS_V4_2 */
 };
 
 /*
-- 
1.8.3.1

[RFC v3 04/42] NFS OFFLOAD_STATUS xdr

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42xdr.c         | 87 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4proc.c         |  1 +
 fs/nfs/nfs4xdr.c          |  1 +
 include/linux/nfs4.h      |  1 +
 include/linux/nfs_fs_sb.h |  1 +
 include/linux/nfs_xdr.h   | 12 +++++++
 6 files changed, 103 insertions(+)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 528362f..e369de1 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -25,6 +25,11 @@
 					 NFS42_WRITE_RES_SIZE + \
 					 1 /* cr_consecutive */ + \
 					 1 /* cr_synchronous */)
+#define encode_offload_status_maxsz	(op_encode_hdr_maxsz + \
+					 XDR_QUADLEN(NFS4_STATEID_SIZE))
+#define decode_offload_status_maxsz	(op_decode_hdr_maxsz + \
+					 2 /* osr_count */ + \
+					 1 + 1 /* osr_complete<1> */)
 #define encode_deallocate_maxsz		(op_encode_hdr_maxsz + \
 					 encode_fallocate_maxsz)
 #define decode_deallocate_maxsz		(op_decode_hdr_maxsz)
@@ -74,6 +79,12 @@
 					 decode_putfh_maxsz + \
 					 decode_copy_maxsz + \
 					 decode_commit_maxsz)
+#define NFS4_enc_offload_status_sz	(compound_encode_hdr_maxsz + \
+					 encode_putfh_maxsz + \
+					 encode_offload_status_maxsz)
+#define NFS4_dec_offload_status_sz	(compound_decode_hdr_maxsz + \
+					 decode_putfh_maxsz + \
+					 decode_offload_status_maxsz)
 #define NFS4_enc_deallocate_sz		(compound_encode_hdr_maxsz + \
 					 encode_putfh_maxsz + \
 					 encode_deallocate_maxsz + \
@@ -144,6 +155,14 @@ static void encode_copy(struct xdr_stream *xdr,
 	encode_uint32(xdr, 0); /* src server list */
 }
 
+static void encode_offload_status(struct xdr_stream *xdr,
+				  struct nfs42_offload_status_args *args,
+				  struct compound_hdr *hdr)
+{
+	encode_op_hdr(xdr, OP_OFFLOAD_STATUS, decode_offload_status_maxsz, hdr);
+	encode_nfs4_stateid(xdr, &args->osa_stateid);
+}
+
 static void encode_deallocate(struct xdr_stream *xdr,
 			      struct nfs42_falloc_args *args,
 			      struct compound_hdr *hdr)
@@ -258,6 +277,24 @@ static void nfs4_xdr_enc_copy(struct rpc_rqst *req,
 }
 
 /*
+ * Encode OFFLOAD_STATUS request
+ */
+static void nfs4_xdr_enc_offload_status(struct rpc_rqst *req,
+					struct xdr_stream *xdr,
+					struct nfs42_offload_status_args *args)
+{
+	struct compound_hdr hdr = {
+		.minorversion = nfs4_xdr_minorversion(&args->osa_seq_args),
+	};
+
+	encode_compound_hdr(xdr, req, &hdr);
+	encode_sequence(xdr, &args->osa_seq_args, &hdr);
+	encode_putfh(xdr, args->osa_src_fh, &hdr);
+	encode_offload_status(xdr, args, &hdr);
+	encode_nops(&hdr);
+}
+
+/*
  * Encode DEALLOCATE request
  */
 static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,
@@ -406,6 +443,31 @@ static int decode_copy(struct xdr_stream *xdr, struct nfs42_copy_res *res)
 	return decode_copy_requirements(xdr, res);
 }
 
+static int decode_offload_status(struct xdr_stream *xdr,
+				 struct nfs42_offload_status_res *res)
+{
+	__be32 *p;
+	uint32_t count;
+	int status;
+
+	status = decode_op_hdr(xdr, OP_OFFLOAD_STATUS);
+	if (status)
+		return status;
+	p = xdr_inline_decode(xdr, 8 + 4);
+	if (unlikely(!p))
+		goto out_overflow;
+	p = xdr_decode_hyper(p, &res->osr_count);
+	count = be32_to_cpup(p++);
+	if (count) {
+		p = xdr_inline_decode(xdr, 4);
+		res->osr_status = nfs4_stat_to_errno(be32_to_cpup(p));
+	}
+	return 0;
+out_overflow:
+	print_overflow_msg(__func__, xdr);
+	return -EIO;
+}
+
 static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *res)
 {
 	return decode_op_hdr(xdr, OP_DEALLOCATE);
@@ -504,6 +566,31 @@ static int nfs4_xdr_dec_copy(struct rpc_rqst *rqstp,
 }
 
 /*
+ * Decode OFFLOAD_STATUS response
+ */
+static int nfs4_xdr_dec_offload_status(struct rpc_rqst *rqstp,
+				       struct xdr_stream *xdr,
+				       struct nfs42_offload_status_res *res)
+{
+	struct compound_hdr hdr;
+	int status;
+
+	status = decode_compound_hdr(xdr, &hdr);
+	if (status)
+		goto out;
+	status = decode_sequence(xdr, &res->osr_seq_res, rqstp);
+	if (status)
+		goto out;
+	status = decode_putfh(xdr);
+	if (status)
+		goto out;
+	status = decode_offload_status(xdr, res);
+
+out:
+	return status;
+}
+
+/*
  * Decode DEALLOCATE request
  */
 static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index dbfa189..90be5c1 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -9229,6 +9229,7 @@ static bool nfs4_match_stateid(const nfs4_stateid *s1,
 		| NFS_CAP_ATOMIC_OPEN_V1
 		| NFS_CAP_ALLOCATE
 		| NFS_CAP_COPY
+		| NFS_CAP_OFFLOAD_STATUS
 		| NFS_CAP_DEALLOCATE
 		| NFS_CAP_SEEK
 		| NFS_CAP_LAYOUTSTATS
diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index 3aebfdc..8b41f43 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -7561,6 +7561,7 @@ struct rpc_procinfo	nfs4_procedures[] = {
 	PROC(LAYOUTSTATS,	enc_layoutstats,	dec_layoutstats),
 	PROC(CLONE,		enc_clone,		dec_clone),
 	PROC(COPY,		enc_copy,		dec_copy),
+	PROC(OFFLOAD_STATUS,	enc_offload_status,	dec_offload_status),
 #endif /* CONFIG_NFS_V4_2 */
 };
 
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index 1b1ca04..d67c350 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -523,6 +523,7 @@ enum {
 	NFSPROC4_CLNT_LAYOUTSTATS,
 	NFSPROC4_CLNT_CLONE,
 	NFSPROC4_CLNT_COPY,
+	NFSPROC4_CLNT_OFFLOAD_STATUS,
 };
 
 /* nfs41 types */
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index e418a10..36535ad 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -250,5 +250,6 @@ struct nfs_server {
 #define NFS_CAP_LAYOUTSTATS	(1U << 22)
 #define NFS_CAP_CLONE		(1U << 23)
 #define NFS_CAP_COPY		(1U << 24)
+#define NFS_CAP_OFFLOAD_STATUS	(1U << 25)
 
 #endif
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index b28c834..eaaf540 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1386,6 +1386,18 @@ struct nfs42_copy_res {
 	struct nfs_commitres		commit_res;
 };
 
+struct nfs42_offload_status_args {
+	struct nfs4_sequence_args	osa_seq_args;
+	struct nfs_fh			*osa_src_fh;
+	nfs4_stateid			osa_stateid;
+};
+
+struct nfs42_offload_status_res {
+	struct nfs4_sequence_res	osr_seq_res;
+	uint64_t			osr_count;
+	int				osr_status;
+};
+
 struct nfs42_seek_args {
 	struct nfs4_sequence_args	seq_args;
 
-- 
1.8.3.1

[RFC v3 05/42] NFS OFFLOAD_STATUS op

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42.h     |  5 ++++-
 fs/nfs/nfs42proc.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs42.h b/fs/nfs/nfs42.h
index b6cd153..54b9941 100644
--- a/fs/nfs/nfs42.h
+++ b/fs/nfs/nfs42.h
@@ -12,6 +12,7 @@
 #define PNFS_LAYOUTSTATS_MAXDEV (4)
 
 /* nfs4.2proc.c */
+#if defined(CONFIG_NFS_V4_2)
 int nfs42_proc_allocate(struct file *, loff_t, loff_t);
 ssize_t nfs42_proc_copy(struct file *, loff_t, struct file *, loff_t, size_t);
 int nfs42_proc_deallocate(struct file *, loff_t, loff_t);
@@ -19,5 +20,7 @@
 int nfs42_proc_layoutstats_generic(struct nfs_server *,
 				   struct nfs42_layoutstat_data *);
 int nfs42_proc_clone(struct file *, struct file *, loff_t, loff_t, loff_t);
-
+int nfs42_proc_offload_status(struct file *, nfs4_stateid *,
+			      struct nfs42_offload_status_res *);
+#endif /* CONFIG_NFS_V4_2) */
 #endif /* __LINUX_FS_NFS_NFS4_2_H */
diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index 6c2db51..5c42e71 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -263,6 +263,48 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
 	return err;
 }
 
+int _nfs42_proc_offload_status(struct file *dst, nfs4_stateid *stateid,
+				struct nfs42_offload_status_res *res)
+{
+	struct nfs42_offload_status_args args;
+	struct nfs_server *dst_server = NFS_SERVER(file_inode(dst));
+	struct rpc_message msg = {
+		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_OFFLOAD_STATUS],
+		.rpc_resp = res,
+		.rpc_argp = &args,
+	};
+	int status;
+
+	args.osa_src_fh = NFS_FH(file_inode(dst));
+	memcpy(&args.osa_stateid, stateid, sizeof(args.osa_stateid));
+	status = nfs4_call_sync(dst_server->client, dst_server, &msg,
+				&args.osa_seq_args, &res->osr_seq_res, 0);
+	if (status == -ENOTSUPP)
+		dst_server->caps &= ~NFS_CAP_OFFLOAD_STATUS;
+
+	return status;
+}
+
+int nfs42_proc_offload_status(struct file *dst, nfs4_stateid *stateid,
+				struct nfs42_offload_status_res *res)
+{
+	struct nfs_server *dst_server = NFS_SERVER(file_inode(dst));
+	struct nfs4_exception exception = { };
+	int status;
+
+	if (!(dst_server->caps & NFS_CAP_OFFLOAD_STATUS))
+		return -EOPNOTSUPP;
+
+	do {
+		status = _nfs42_proc_offload_status(dst, stateid, res);
+		if (status == -ENOTSUPP)
+			return -EOPNOTSUPP;
+		status = nfs4_handle_exception(dst_server, status, &exception);
+	} while (exception.retry);
+
+	return status;
+}
+
 static loff_t _nfs42_proc_llseek(struct file *filep,
 		struct nfs_lock_context *lock, loff_t offset, int whence)
 {
-- 
1.8.3.1

[RFC v3 06/42] NFS OFFLOAD_CANCEL xdr

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42xdr.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4proc.c         |  1 +
 fs/nfs/nfs4xdr.c          |  1 +
 include/linux/nfs4.h      |  1 +
 include/linux/nfs_fs_sb.h |  1 +
 5 files changed, 70 insertions(+)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index e369de1..3c6ca02 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -30,6 +30,9 @@
 #define decode_offload_status_maxsz	(op_decode_hdr_maxsz + \
 					 2 /* osr_count */ + \
 					 1 + 1 /* osr_complete<1> */)
+#define encode_offload_cancel_maxsz	(op_encode_hdr_maxsz + \
+					 XDR_QUADLEN(NFS4_STATEID_SIZE))
+#define decode_offload_cancel_maxsz	(op_decode_hdr_maxsz)
 #define encode_deallocate_maxsz		(op_encode_hdr_maxsz + \
 					 encode_fallocate_maxsz)
 #define decode_deallocate_maxsz		(op_decode_hdr_maxsz)
@@ -85,6 +88,12 @@
 #define NFS4_dec_offload_status_sz	(compound_decode_hdr_maxsz + \
 					 decode_putfh_maxsz + \
 					 decode_offload_status_maxsz)
+#define NFS4_enc_offload_cancel_sz	(compound_encode_hdr_maxsz + \
+					 encode_putfh_maxsz + \
+					 encode_offload_cancel_maxsz)
+#define NFS4_dec_offload_cancel_sz	(compound_decode_hdr_maxsz + \
+					 decode_putfh_maxsz + \
+					 decode_offload_cancel_maxsz)
 #define NFS4_enc_deallocate_sz		(compound_encode_hdr_maxsz + \
 					 encode_putfh_maxsz + \
 					 encode_deallocate_maxsz + \
@@ -163,6 +172,14 @@ static void encode_offload_status(struct xdr_stream *xdr,
 	encode_nfs4_stateid(xdr, &args->osa_stateid);
 }
 
+static void encode_offload_cancel(struct xdr_stream *xdr,
+				  struct nfs42_offload_status_args *args,
+				  struct compound_hdr *hdr)
+{
+	encode_op_hdr(xdr, OP_OFFLOAD_CANCEL, decode_offload_cancel_maxsz, hdr);
+	encode_nfs4_stateid(xdr, &args->osa_stateid);
+}
+
 static void encode_deallocate(struct xdr_stream *xdr,
 			      struct nfs42_falloc_args *args,
 			      struct compound_hdr *hdr)
@@ -295,6 +312,24 @@ static void nfs4_xdr_enc_offload_status(struct rpc_rqst *req,
 }
 
 /*
+ * Encode OFFLOAD_CANEL request
+ */
+static void nfs4_xdr_enc_offload_cancel(struct rpc_rqst *req,
+					struct xdr_stream *xdr,
+					struct nfs42_offload_status_args *args)
+{
+	struct compound_hdr hdr = {
+		.minorversion = nfs4_xdr_minorversion(&args->osa_seq_args),
+	};
+
+	encode_compound_hdr(xdr, req, &hdr);
+	encode_sequence(xdr, &args->osa_seq_args, &hdr);
+	encode_putfh(xdr, args->osa_src_fh, &hdr);
+	encode_offload_cancel(xdr, args, &hdr);
+	encode_nops(&hdr);
+}
+
+/*
  * Encode DEALLOCATE request
  */
 static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,
@@ -468,6 +503,12 @@ static int decode_offload_status(struct xdr_stream *xdr,
 	return -EIO;
 }
 
+static int decode_offload_cancel(struct xdr_stream *xdr,
+				 struct nfs42_offload_status_res *res)
+{
+	return decode_op_hdr(xdr, OP_OFFLOAD_CANCEL);
+}
+
 static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *res)
 {
 	return decode_op_hdr(xdr, OP_DEALLOCATE);
@@ -591,6 +632,31 @@ static int nfs4_xdr_dec_offload_status(struct rpc_rqst *rqstp,
 }
 
 /*
+ * Decode OFFLOAD_CANCEL response
+ */
+static int nfs4_xdr_dec_offload_cancel(struct rpc_rqst *rqstp,
+				       struct xdr_stream *xdr,
+				       struct nfs42_offload_status_res *res)
+{
+	struct compound_hdr hdr;
+	int status;
+
+	status = decode_compound_hdr(xdr, &hdr);
+	if (status)
+		goto out;
+	status = decode_sequence(xdr, &res->osr_seq_res, rqstp);
+	if (status)
+		goto out;
+	status = decode_putfh(xdr);
+	if (status)
+		goto out;
+	status = decode_offload_cancel(xdr, res);
+
+out:
+	return status;
+}
+
+/*
  * Decode DEALLOCATE request
  */
 static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 90be5c1..b37e528 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -9230,6 +9230,7 @@ static bool nfs4_match_stateid(const nfs4_stateid *s1,
 		| NFS_CAP_ALLOCATE
 		| NFS_CAP_COPY
 		| NFS_CAP_OFFLOAD_STATUS
+		| NFS_CAP_OFFLOAD_CANCEL
 		| NFS_CAP_DEALLOCATE
 		| NFS_CAP_SEEK
 		| NFS_CAP_LAYOUTSTATS
diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index 8b41f43..9b66053 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -7562,6 +7562,7 @@ struct rpc_procinfo	nfs4_procedures[] = {
 	PROC(CLONE,		enc_clone,		dec_clone),
 	PROC(COPY,		enc_copy,		dec_copy),
 	PROC(OFFLOAD_STATUS,	enc_offload_status,	dec_offload_status),
+	PROC(OFFLOAD_CANCEL,    enc_offload_cancel,     dec_offload_cancel),
 #endif /* CONFIG_NFS_V4_2 */
 };
 
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index d67c350..7262908 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -524,6 +524,7 @@ enum {
 	NFSPROC4_CLNT_CLONE,
 	NFSPROC4_CLNT_COPY,
 	NFSPROC4_CLNT_OFFLOAD_STATUS,
+	NFSPROC4_CLNT_OFFLOAD_CANCEL,
 };
 
 /* nfs41 types */
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 36535ad..41a2af5 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -251,5 +251,6 @@ struct nfs_server {
 #define NFS_CAP_CLONE		(1U << 23)
 #define NFS_CAP_COPY		(1U << 24)
 #define NFS_CAP_OFFLOAD_STATUS	(1U << 25)
+#define NFS_CAP_OFFLOAD_CANCEL	(1U << 26)
 
 #endif
-- 
1.8.3.1

[RFC v3 07/42] NFS COPY xdr handle async reply

[Olga Kornievskaia]

If server returns async reply, it must include a callback stateid,
wr_callback_id in the write_response4.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42xdr.c       | 22 ++++++++++++----------
 include/linux/nfs_xdr.h |  1 +
 2 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 3c6ca02..27da108 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -418,21 +418,23 @@ static int decode_write_response(struct xdr_stream *xdr,
 				 struct nfs42_write_res *res)
 {
 	__be32 *p;
+	int status, count;
 
-	p = xdr_inline_decode(xdr, 4 + 8 + 4);
+	p = xdr_inline_decode(xdr, 4);
 	if (unlikely(!p))
 		goto out_overflow;
-
-	/*
-	 * We never use asynchronous mode, so warn if a server returns
-	 * a stateid.
-	 */
-	if (unlikely(*p != 0)) {
-		pr_err_once("%s: server has set unrequested "
-				"asynchronous mode\n", __func__);
+	count = be32_to_cpup(p);
+	if (count > 1)
 		return -EREMOTEIO;
+	else if (count == 1) {
+		status = decode_opaque_fixed(xdr, &res->stateid,
+				NFS4_STATEID_SIZE);
+		if (unlikely(status))
+			goto out_overflow;
 	}
-	p++;
+	p = xdr_inline_decode(xdr, 8 + 4);
+	if (unlikely(!p))
+		goto out_overflow;
 	p = xdr_decode_hyper(p, &res->count);
 	res->verifier.committed = be32_to_cpup(p);
 	return decode_verifier(xdr, &res->verifier.verifier);
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index eaaf540..60eafa0 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1374,6 +1374,7 @@ struct nfs42_copy_args {
 };
 
 struct nfs42_write_res {
+	nfs4_stateid		stateid;
 	u64			count;
 	struct nfs_writeverf	verifier;
 };
-- 
1.8.3.1

[RFC v3 08/42] NFS add support for asynchronous COPY

[Olga Kornievskaia]

Change xdr to always send COPY asynchronously.

Keep the list copies send in a list under a server structure.
Once copy is sent, it waits on a completion structure that will
be signalled by the callback thread that receives CB_OFFLOAD.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/callback_proc.c    | 34 ++++++++++++++++++++++++++++
 fs/nfs/client.c           |  1 +
 fs/nfs/nfs42proc.c        | 56 ++++++++++++++++++++++++++++++++++++++++++-----
 fs/nfs/nfs42xdr.c         |  8 ++++---
 include/linux/nfs_fs.h    |  9 ++++++++
 include/linux/nfs_fs_sb.h |  1 +
 include/linux/nfs_xdr.h   |  1 +
 7 files changed, 102 insertions(+), 8 deletions(-)

diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
index 5396751..7e211ed 100644
--- a/fs/nfs/callback_proc.c
+++ b/fs/nfs/callback_proc.c
@@ -648,9 +648,43 @@ __be32 nfs4_callback_notify_lock(struct cb_notify_lock_args *args, void *dummy,
 }
 #endif /* CONFIG_NFS_V4_1 */
 #ifdef CONFIG_NFS_V4_2
+static void nfs4_copy_cb_args(struct nfs4_copy_state *cp_state,
+				struct cb_offloadargs *args)
+{
+	cp_state->count = args->wr_count;
+	cp_state->error = args->error;
+	if (!args->error) {
+		cp_state->verf.committed = args->wr_writeverf.committed;
+		memcpy(&cp_state->verf.verifier.data[0],
+			&args->wr_writeverf.verifier.data[0],
+			NFS4_VERIFIER_SIZE);
+	}
+}
+
 __be32 nfs4_callback_offload(struct cb_offloadargs *args, void *dummy,
 				struct cb_process_state *cps)
 {
+	struct nfs_server *server;
+	struct nfs4_copy_state *copy;
+
+	rcu_read_lock();
+	list_for_each_entry_rcu(server, &cps->clp->cl_superblocks,
+				client_link) {
+		spin_lock(&server->nfs_client->cl_lock);
+		list_for_each_entry(copy, &server->ss_copies, copies) {
+			if (memcmp(args->coa_stateid.other,
+					copy->stateid.other,
+					sizeof(args->coa_stateid.other)))
+				continue;
+			nfs4_copy_cb_args(copy, args);
+			complete(&copy->completion);
+			spin_unlock(&server->nfs_client->cl_lock);
+			goto out;
+		}
+		spin_unlock(&server->nfs_client->cl_lock);
+	}
+out:
+	rcu_read_unlock();
 	return 0;
 }
 #endif /* CONFIG_NFS_V4_2 */
diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index ee5ddbd..6bde98a 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -875,6 +875,7 @@ struct nfs_server *nfs_alloc_server(void)
 	INIT_LIST_HEAD(&server->delegations);
 	INIT_LIST_HEAD(&server->layouts);
 	INIT_LIST_HEAD(&server->state_owners_lru);
+	INIT_LIST_HEAD(&server->ss_copies);
 
 	atomic_set(&server->active, 0);
 
diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index 5c42e71..d92ce46 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -129,6 +129,38 @@ int nfs42_proc_deallocate(struct file *filep, loff_t offset, loff_t len)
 	return err;
 }
 
+static int handle_async_copy(struct nfs42_copy_res *res,
+			     struct nfs_server *server,
+			     struct file *src,
+			     struct file *dst,
+			     nfs4_stateid *src_stateid,
+			     uint64_t *ret_count)
+{
+	struct nfs4_copy_state *copy;
+	int status;
+
+	copy = kzalloc(sizeof(struct nfs4_copy_state), GFP_NOFS);
+	if (!copy)
+		return -ENOMEM;
+	memcpy(&copy->stateid, &res->write_res.stateid, NFS4_STATEID_SIZE);
+	init_completion(&copy->completion);
+
+	spin_lock(&server->nfs_client->cl_lock);
+	list_add_tail(&copy->copies, &server->ss_copies);
+	spin_unlock(&server->nfs_client->cl_lock);
+
+	wait_for_completion_interruptible(&copy->completion);
+	spin_lock(&server->nfs_client->cl_lock);
+	list_del_init(&copy->copies);
+	spin_unlock(&server->nfs_client->cl_lock);
+	*ret_count = copy->count;
+	memcpy(&res->write_res.verifier, &copy->verf, sizeof(copy->verf));
+	status = -copy->error;
+
+	kfree(copy);
+	return status;
+}
+
 static ssize_t _nfs42_proc_copy(struct file *src,
 				struct nfs_lock_context *src_lock,
 				struct file *dst,
@@ -167,9 +199,13 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 	if (status)
 		return status;
 
-	res->commit_res.verf = kzalloc(sizeof(struct nfs_writeverf), GFP_NOFS);
-	if (!res->commit_res.verf)
-		return -ENOMEM;
+	res->commit_res.verf = NULL;
+	if (args->sync) {
+		res->commit_res.verf =
+			kzalloc(sizeof(struct nfs_writeverf), GFP_NOFS);
+		if (!res->commit_res.verf)
+			return -ENOMEM;
+	}
 	status = nfs4_call_sync(server->client, server, &msg,
 				&args->seq_args, &res->seq_res, 0);
 	if (status == -ENOTSUPP)
@@ -177,18 +213,27 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 	if (status)
 		goto out;
 
-	if (nfs_write_verifier_cmp(&res->write_res.verifier.verifier,
+	if (args->sync &&
+		!nfs_write_verifier_cmp(&res->write_res.verifier.verifier,
 				    &res->commit_res.verf->verifier)) {
 		status = -EAGAIN;
 		goto out;
 	}
 
+	if (!res->synchronous) {
+		status = handle_async_copy(res, server, src, dst,
+				&args->src_stateid, &res->write_res.count);
+		if (status)
+			return status;
+	}
+
 	truncate_pagecache_range(dst_inode, pos_dst,
 				 pos_dst + res->write_res.count);
 
 	status = res->write_res.count;
 out:
-	kfree(res->commit_res.verf);
+	if (args->sync)
+		kfree(res->commit_res.verf);
 	return status;
 }
 
@@ -205,6 +250,7 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
 		.dst_fh		= NFS_FH(file_inode(dst)),
 		.dst_pos	= pos_dst,
 		.count		= count,
+		.sync		= false,
 	};
 	struct nfs42_copy_res res;
 	struct nfs4_exception src_exception = {
diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index 27da108..f507fe4 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -160,7 +160,7 @@ static void encode_copy(struct xdr_stream *xdr,
 	encode_uint64(xdr, args->count);
 
 	encode_uint32(xdr, 1); /* consecutive = true */
-	encode_uint32(xdr, 1); /* synchronous = true */
+	encode_uint32(xdr, args->sync);
 	encode_uint32(xdr, 0); /* src server list */
 }
 
@@ -289,7 +289,8 @@ static void nfs4_xdr_enc_copy(struct rpc_rqst *req,
 	encode_savefh(xdr, &hdr);
 	encode_putfh(xdr, args->dst_fh, &hdr);
 	encode_copy(xdr, args, &hdr);
-	encode_copy_commit(xdr, args, &hdr);
+	if (args->sync)
+		encode_copy_commit(xdr, args, &hdr);
 	encode_nops(&hdr);
 }
 
@@ -603,7 +604,8 @@ static int nfs4_xdr_dec_copy(struct rpc_rqst *rqstp,
 	status = decode_copy(xdr, res);
 	if (status)
 		goto out;
-	status = decode_commit(xdr, &res->commit_res);
+	if (res->commit_res.verf)
+		status = decode_commit(xdr, &res->commit_res);
 out:
 	return status;
 }
diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h
index bb0eb2c..8b0192f 100644
--- a/include/linux/nfs_fs.h
+++ b/include/linux/nfs_fs.h
@@ -183,6 +183,15 @@ struct nfs_inode {
 	struct inode		vfs_inode;
 };
 
+struct nfs4_copy_state {
+	struct list_head	copies;
+	nfs4_stateid		stateid;
+	struct completion	completion;
+	uint64_t		count;
+	struct nfs_writeverf	verf;
+	int			error;
+};
+
 /*
  * Cache validity bit flags
  */
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 41a2af5..d4a2385 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -205,6 +205,7 @@ struct nfs_server {
 	struct list_head	state_owners_lru;
 	struct list_head	layouts;
 	struct list_head	delegations;
+	struct list_head	ss_copies;
 
 	unsigned long		mig_gen;
 	unsigned long		mig_status;
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 60eafa0..4875928 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1371,6 +1371,7 @@ struct nfs42_copy_args {
 	u64				dst_pos;
 
 	u64				count;
+	bool				sync;
 };
 
 struct nfs42_write_res {
-- 
1.8.3.1

[RFC v3 09/42] NFS handle COPY reply CB_OFFLOAD call race

[Olga Kornievskaia]

It's possible that server replies back with CB_OFFLOAD call and
COPY reply at the same time such that client will process
CB_OFFLOAD before reply to COPY. For that keep a list of pending
callback stateids received and them before waiting on completion
check the pending list.

Cleanup any pending copies on the client shutdown.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/callback_proc.c    | 17 ++++++++++++++---
 fs/nfs/nfs42proc.c        | 24 +++++++++++++++++++++---
 fs/nfs/nfs4client.c       | 15 +++++++++++++++
 include/linux/nfs_fs_sb.h |  1 +
 4 files changed, 51 insertions(+), 6 deletions(-)

diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
index 7e211ed..bac6022 100644
--- a/fs/nfs/callback_proc.c
+++ b/fs/nfs/callback_proc.c
@@ -666,11 +666,12 @@ __be32 nfs4_callback_offload(struct cb_offloadargs *args, void *dummy,
 {
 	struct nfs_server *server;
 	struct nfs4_copy_state *copy;
+	bool found = false;
 
+	spin_lock(&cps->clp->cl_lock);
 	rcu_read_lock();
 	list_for_each_entry_rcu(server, &cps->clp->cl_superblocks,
 				client_link) {
-		spin_lock(&server->nfs_client->cl_lock);
 		list_for_each_entry(copy, &server->ss_copies, copies) {
 			if (memcmp(args->coa_stateid.other,
 					copy->stateid.other,
@@ -678,13 +679,23 @@ __be32 nfs4_callback_offload(struct cb_offloadargs *args, void *dummy,
 				continue;
 			nfs4_copy_cb_args(copy, args);
 			complete(&copy->completion);
-			spin_unlock(&server->nfs_client->cl_lock);
+			found = true;
 			goto out;
 		}
-		spin_unlock(&server->nfs_client->cl_lock);
 	}
 out:
 	rcu_read_unlock();
+	if (!found) {
+		copy = kzalloc(sizeof(struct nfs4_copy_state), GFP_NOFS);
+		if (!copy) {
+			spin_unlock(&cps->clp->cl_lock);
+			return -ENOMEM;
+		}
+		memcpy(&copy->stateid, &args->coa_stateid, NFS4_STATEID_SIZE);
+		nfs4_copy_cb_args(copy, args);
+		list_add_tail(&copy->copies, &cps->clp->pending_cb_stateids);
+	}
+	spin_unlock(&cps->clp->cl_lock);
 	return 0;
 }
 #endif /* CONFIG_NFS_V4_2 */
diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index d92ce46..9acbde5 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -137,15 +137,32 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 			     uint64_t *ret_count)
 {
 	struct nfs4_copy_state *copy;
-	int status;
+	int status = NFS4_OK;
+	bool found_pending = false;
+
+	spin_lock(&server->nfs_client->cl_lock);
+	list_for_each_entry(copy, &server->nfs_client->pending_cb_stateids,
+				copies) {
+		if (memcmp(&res->write_res.stateid, &copy->stateid,
+				NFS4_STATEID_SIZE))
+			continue;
+		found_pending = true;
+		list_del(&copy->copies);
+		break;
+	}
+	if (found_pending) {
+		spin_unlock(&server->nfs_client->cl_lock);
+		goto out;
+	}
 
 	copy = kzalloc(sizeof(struct nfs4_copy_state), GFP_NOFS);
-	if (!copy)
+	if (!copy) {
+		spin_unlock(&server->nfs_client->cl_lock);
 		return -ENOMEM;
+	}
 	memcpy(&copy->stateid, &res->write_res.stateid, NFS4_STATEID_SIZE);
 	init_completion(&copy->completion);
 
-	spin_lock(&server->nfs_client->cl_lock);
 	list_add_tail(&copy->copies, &server->ss_copies);
 	spin_unlock(&server->nfs_client->cl_lock);
 
@@ -153,6 +170,7 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 	spin_lock(&server->nfs_client->cl_lock);
 	list_del_init(&copy->copies);
 	spin_unlock(&server->nfs_client->cl_lock);
+out:
 	*ret_count = copy->count;
 	memcpy(&res->write_res.verifier, &copy->verf, sizeof(copy->verf));
 	status = -copy->error;
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index 66776f0..89b6414 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -156,9 +156,23 @@ struct rpc_clnt *
 	}
 }
 
+static void
+nfs4_cleanup_callback(struct nfs_client *clp)
+{
+	struct nfs4_copy_state *cp_state;
+
+	while (!list_empty(&clp->pending_cb_stateids)) {
+		cp_state = list_entry(clp->pending_cb_stateids.next,
+					struct nfs4_copy_state, copies);
+		list_del(&cp_state->copies);
+		kfree(cp_state);
+	}
+}
+
 void nfs41_shutdown_client(struct nfs_client *clp)
 {
 	if (nfs4_has_session(clp)) {
+		nfs4_cleanup_callback(clp);
 		nfs4_shutdown_ds_clients(clp);
 		nfs4_destroy_session(clp->cl_session);
 		nfs4_destroy_clientid(clp);
@@ -202,6 +216,7 @@ struct nfs_client *nfs4_alloc_client(const struct nfs_client_initdata *cl_init)
 #if IS_ENABLED(CONFIG_NFS_V4_1)
 	init_waitqueue_head(&clp->cl_lock_waitq);
 #endif
+	INIT_LIST_HEAD(&clp->pending_cb_stateids);
 	return clp;
 
 error:
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index d4a2385..2b0397d 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -118,6 +118,7 @@ struct nfs_client {
 #endif
 
 	struct net		*cl_net;
+	struct list_head	pending_cb_stateids;
 };
 
 /*
-- 
1.8.3.1

[RFC v3 10/42] NFS export nfs4_async_handle_error

[Olga Kornievskaia]

Make this function available to nfs42proc.c

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs4_fs.h  | 3 +++
 fs/nfs/nfs4proc.c | 3 ++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index af285cc..f5d014e 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -244,6 +244,9 @@ int nfs4_replace_transport(struct nfs_server *server,
 
 /* nfs4proc.c */
 extern int nfs4_handle_exception(struct nfs_server *, int, struct nfs4_exception *);
+extern int nfs4_async_handle_error(struct rpc_task *task,
+				   struct nfs_server *server,
+				   struct nfs4_state *state, long *timeout);
 extern int nfs4_call_sync(struct rpc_clnt *, struct nfs_server *,
 			  struct rpc_message *, struct nfs4_sequence_args *,
 			  struct nfs4_sequence_res *, int);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index b37e528..fdb6b76 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -548,7 +548,7 @@ int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_
 	return ret;
 }
 
-static int
+int
 nfs4_async_handle_error(struct rpc_task *task, struct nfs_server *server,
 			struct nfs4_state *state, long *timeout)
 {
@@ -569,6 +569,7 @@ int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_
 		return -EAGAIN;
 	return 0;
 }
+EXPORT_SYMBOL_GPL(nfs4_async_handle_error);
 
 /*
  * Return 'true' if 'clp' is using an rpc_client that is integrity protected
-- 
1.8.3.1

[RFC v3 11/42] NFS test for intra vs inter COPY

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/internal.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 3e24392..dcbcffb 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -7,6 +7,7 @@
 #include <linux/security.h>
 #include <linux/crc32.h>
 #include <linux/nfs_page.h>
+#include <linux/sunrpc/addr.h>
 
 #define NFS_MS_MASK (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_SYNCHRONOUS)
 
@@ -766,3 +767,12 @@ static inline bool nfs_error_is_fatal(int err)
 		return false;
 	}
 }
+
+static inline bool nfs42_files_from_same_server(struct file *in, struct file *out)
+{
+	struct nfs_client *c_in = (NFS_SERVER(file_inode(in)))->nfs_client;
+	struct nfs_client *c_out = (NFS_SERVER(file_inode(out)))->nfs_client;
+
+	return rpc_cmp_addr((struct sockaddr *)&c_in->cl_addr,
+				(struct sockaddr *)&c_out->cl_addr);
+}
-- 
1.8.3.1

[RFC v3 12/42] NFS send OFFLOAD_CANCEL when COPY killed

[Olga Kornievskaia]

When COPY is killed send OFFLOAD_CANCEL to both destination and
source servers.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42proc.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 97 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index 9acbde5..8223726 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -16,6 +16,7 @@
 #include "internal.h"
 
 #define NFSDBG_FACILITY NFSDBG_PROC
+static int nfs42_do_offload_cancel_async(struct file *dst, nfs4_stateid *std);
 
 static int _nfs42_proc_fallocate(struct rpc_message *msg, struct file *filep,
 		struct nfs_lock_context *lock, loff_t offset, loff_t len)
@@ -166,10 +167,15 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 	list_add_tail(&copy->copies, &server->ss_copies);
 	spin_unlock(&server->nfs_client->cl_lock);
 
-	wait_for_completion_interruptible(&copy->completion);
+	status = wait_for_completion_interruptible(&copy->completion);
 	spin_lock(&server->nfs_client->cl_lock);
 	list_del_init(&copy->copies);
 	spin_unlock(&server->nfs_client->cl_lock);
+	if (status == -ERESTARTSYS) {
+		nfs42_do_offload_cancel_async(dst, &copy->stateid);
+		kfree(copy);
+		return status;
+	}
 out:
 	*ret_count = copy->count;
 	memcpy(&res->write_res.verifier, &copy->verf, sizeof(copy->verf));
@@ -226,8 +232,6 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 	}
 	status = nfs4_call_sync(server->client, server, &msg,
 				&args->seq_args, &res->seq_res, 0);
-	if (status == -ENOTSUPP)
-		server->caps &= ~NFS_CAP_COPY;
 	if (status)
 		goto out;
 
@@ -304,6 +308,14 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
 				dst, dst_lock,
 				&args, &res);
 		inode_unlock(file_inode(dst));
+		switch (err) {
+		case -ENOTSUPP:
+			server->caps &= ~NFS_CAP_COPY;
+			if (nfs42_files_from_same_server(src, dst))
+				break;
+		case -ERESTARTSYS:
+			nfs42_do_offload_cancel_async(src, &args.src_stateid);
+		}
 
 		if (err >= 0)
 			break;
@@ -327,6 +339,88 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
 	return err;
 }
 
+struct nfs42_offloadcancel_data {
+	struct nfs_server *seq_server;
+	struct nfs42_offload_status_args args;
+	struct nfs42_offload_status_res res;
+};
+
+static void nfs42_offload_cancel_prepare(struct rpc_task *task, void *calldata)
+{
+	struct nfs42_offloadcancel_data *data = calldata;
+
+	nfs4_setup_sequence(data->seq_server->nfs_client,
+				&data->args.osa_seq_args,
+				&data->res.osr_seq_res, task);
+}
+
+static void nfs42_offload_cancel_done(struct rpc_task *task, void *calldata)
+{
+	struct nfs42_offloadcancel_data *data = calldata;
+
+	nfs41_sequence_done(task, &data->res.osr_seq_res);
+	if (task->tk_status &&
+		nfs4_async_handle_error(task, data->seq_server, NULL,
+			NULL) == -EAGAIN)
+		rpc_restart_call_prepare(task);
+}
+
+static void nfs42_free_offloadcancel_data(void *data)
+{
+	kfree(data);
+}
+
+static const struct rpc_call_ops nfs42_offload_cancel_ops = {
+	.rpc_call_prepare = nfs42_offload_cancel_prepare,
+	.rpc_call_done = nfs42_offload_cancel_done,
+	.rpc_release = nfs42_free_offloadcancel_data,
+};
+
+static int nfs42_do_offload_cancel_async(struct file *dst,
+				nfs4_stateid *stateid)
+{
+	struct nfs_server *dst_server = NFS_SERVER(file_inode(dst));
+	struct nfs42_offloadcancel_data *data = NULL;
+	struct nfs_open_context *ctx = nfs_file_open_context(dst);
+	struct rpc_task *task;
+	struct rpc_message msg = {
+		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_OFFLOAD_CANCEL],
+		.rpc_cred = ctx->cred,
+	};
+	struct rpc_task_setup task_setup_data = {
+		.rpc_client = dst_server->client,
+		.rpc_message = &msg,
+		.callback_ops = &nfs42_offload_cancel_ops,
+		.workqueue = nfsiod_workqueue,
+		.flags = RPC_TASK_ASYNC,
+	};
+	int status;
+
+	if (!(dst_server->caps & NFS_CAP_OFFLOAD_CANCEL))
+		return -EOPNOTSUPP;
+
+	data = kzalloc(sizeof(struct nfs42_offloadcancel_data), GFP_NOFS);
+	if (data == NULL)
+		return -ENOMEM;
+
+	data->seq_server = dst_server;
+	data->args.osa_src_fh = NFS_FH(file_inode(dst));
+	memcpy(&data->args.osa_stateid, stateid,
+		sizeof(data->args.osa_stateid));
+	msg.rpc_argp = &data->args;
+	msg.rpc_resp = &data->res;
+	task_setup_data.callback_data = data;
+	nfs4_init_sequence(&data->args.osa_seq_args, &data->res.osr_seq_res, 1);
+	task = rpc_run_task(&task_setup_data);
+	if (IS_ERR(task))
+		return PTR_ERR(task);
+	status = rpc_wait_for_completion_task(task);
+	if (status == -ENOTSUPP)
+		dst_server->caps &= ~NFS_CAP_OFFLOAD_CANCEL;
+	rpc_put_task(task);
+	return status;
+}
+
 int _nfs42_proc_offload_status(struct file *dst, nfs4_stateid *stateid,
 				struct nfs42_offload_status_res *res)
 {
-- 
1.8.3.1

[RFC v3 13/42] NFS handle COPY ERR_OFFLOAD_NO_REQS

[Olga Kornievskaia]

If client sent async COPY and server replied with
ERR_OFFLOAD_NO_REQS, client should retry with a synchronous copy.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42proc.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index 8223726..dba2b62 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -309,6 +309,13 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
 				&args, &res);
 		inode_unlock(file_inode(dst));
 		switch (err) {
+		case -NFS4ERR_OFFLOAD_NO_REQS:
+			if (!args.sync) {
+				args.sync = true;
+				dst_exception.retry = 1;
+				continue;
+			}
+			break;
 		case -ENOTSUPP:
 			server->caps &= ~NFS_CAP_COPY;
 			if (nfs42_files_from_same_server(src, dst))
-- 
1.8.3.1

[RFC v3 14/42] NFS if we got partial copy ignore errors

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42proc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index dba2b62..dfd43bc 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -179,7 +179,8 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 out:
 	*ret_count = copy->count;
 	memcpy(&res->write_res.verifier, &copy->verf, sizeof(copy->verf));
-	status = -copy->error;
+	if (copy->count <= 0)
+		status = -copy->error;
 
 	kfree(copy);
 	return status;
-- 
1.8.3.1

[RFC v3 15/42] NFS recover from destination server reboot for copies

[Olga Kornievskaia]

Mark the destination state to indicate a server-side copy is
happening. On detecting a reboot and recovering open state check
if any state is engaged in a server-side copy, if so, find the
copy and mark it and then signal the waiting thread. Upon wakeup,
if copy was marked then propage EAGAIN to the nfsd_copy_file_range
and restart the copy from scratch (no partial state is being
queried at this point).

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42proc.c     | 16 +++++++++++++---
 fs/nfs/nfs4_fs.h       |  4 ++++
 fs/nfs/nfs4file.c      |  9 +++++++--
 fs/nfs/nfs4state.c     | 15 +++++++++++++++
 include/linux/nfs_fs.h |  2 ++
 5 files changed, 41 insertions(+), 5 deletions(-)

diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index dfd43bc..e145479 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -140,6 +140,7 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 	struct nfs4_copy_state *copy;
 	int status = NFS4_OK;
 	bool found_pending = false;
+	struct nfs_open_context *ctx = nfs_file_open_context(dst);
 
 	spin_lock(&server->nfs_client->cl_lock);
 	list_for_each_entry(copy, &server->nfs_client->pending_cb_stateids,
@@ -163,6 +164,7 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 	}
 	memcpy(&copy->stateid, &res->write_res.stateid, NFS4_STATEID_SIZE);
 	init_completion(&copy->completion);
+	copy->parent_state = ctx->state;
 
 	list_add_tail(&copy->copies, &server->ss_copies);
 	spin_unlock(&server->nfs_client->cl_lock);
@@ -172,9 +174,10 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 	list_del_init(&copy->copies);
 	spin_unlock(&server->nfs_client->cl_lock);
 	if (status == -ERESTARTSYS) {
-		nfs42_do_offload_cancel_async(dst, &copy->stateid);
-		kfree(copy);
-		return status;
+		goto out_cancel;
+	} else if (copy->flags) {
+		status = -EAGAIN;
+		goto out_cancel;
 	}
 out:
 	*ret_count = copy->count;
@@ -184,6 +187,10 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 
 	kfree(copy);
 	return status;
+out_cancel:
+	nfs42_do_offload_cancel_async(dst, &copy->stateid);
+	kfree(copy);
+	return status;
 }
 
 static ssize_t _nfs42_proc_copy(struct file *src,
@@ -231,6 +238,9 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 		if (!res->commit_res.verf)
 			return -ENOMEM;
 	}
+	set_bit(NFS_CLNT_DST_SSC_COPY_STATE,
+		&dst_lock->open_context->state->flags);
+
 	status = nfs4_call_sync(server->client, server, &msg,
 				&args->seq_args, &res->seq_res, 0);
 	if (status)
diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index f5d014e..54e8778 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -161,6 +161,10 @@ enum {
 	NFS_STATE_POSIX_LOCKS,		/* Posix locks are supported */
 	NFS_STATE_RECOVERY_FAILED,	/* OPEN stateid state recovery failed */
 	NFS_STATE_MAY_NOTIFY_LOCK,	/* server may CB_NOTIFY_LOCK */
+#ifdef CONFIG_NFS_V4_2
+	NFS_CLNT_DST_SSC_COPY_STATE,	/* dst server open state on client*/
+#endif /* CONFIG_NFS_V4_2 */
+
 };
 
 struct nfs4_state {
diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c
index 0efba77..881b819 100644
--- a/fs/nfs/nfs4file.c
+++ b/fs/nfs/nfs4file.c
@@ -133,10 +133,15 @@ static ssize_t nfs4_copy_file_range(struct file *file_in, loff_t pos_in,
 				    struct file *file_out, loff_t pos_out,
 				    size_t count, unsigned int flags)
 {
+	ssize_t ret;
+
 	if (file_inode(file_in) == file_inode(file_out))
 		return -EINVAL;
-
-	return nfs42_proc_copy(file_in, pos_in, file_out, pos_out, count);
+retry:
+	ret = nfs42_proc_copy(file_in, pos_in, file_out, pos_out, count);
+	if (ret == -EAGAIN)
+		goto retry;
+	return ret;
 }
 
 static loff_t nfs4_file_llseek(struct file *filep, loff_t offset, int whence)
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index cbf82b0..81f9d52 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1540,6 +1540,21 @@ static int nfs4_reclaim_open_state(struct nfs4_state_owner *sp, const struct nfs
 					&state->flags);
 				nfs4_put_open_state(state);
 				spin_lock(&sp->so_lock);
+#ifdef CONFIG_NFS_V4_2
+				if (test_bit(NFS_CLNT_DST_SSC_COPY_STATE, &state->flags)) {
+					struct nfs4_copy_state *copy;
+
+					spin_lock(&sp->so_server->nfs_client->cl_lock);
+					list_for_each_entry(copy, &sp->so_server->ss_copies, copies) {
+						if (memcmp(&state->stateid.other, &copy->parent_state->stateid.other, NFS4_STATEID_SIZE))
+							continue;
+						copy->flags = 1;
+						complete(&copy->completion);
+						break;
+					}
+					spin_unlock(&sp->so_server->nfs_client->cl_lock);
+				}
+#endif /* CONFIG_NFS_V4_2 */
 				goto restart;
 			}
 		}
diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h
index 8b0192f..f26a35b 100644
--- a/include/linux/nfs_fs.h
+++ b/include/linux/nfs_fs.h
@@ -190,6 +190,8 @@ struct nfs4_copy_state {
 	uint64_t		count;
 	struct nfs_writeverf	verf;
 	int			error;
+	int			flags;
+	struct nfs4_state	*parent_state;
 };
 
 /*
-- 
1.8.3.1

[RFC v3 16/42] NFS NFSD defining nl4_servers structure needed by both

[Olga Kornievskaia]

These structures are needed by COPY_NOTIFY on the client and needed
by the nfsd as well

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 include/linux/nfs4.h | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index 7262908..4179c78 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -15,6 +15,7 @@
 #include <linux/list.h>
 #include <linux/uidgid.h>
 #include <uapi/linux/nfs4.h>
+#include <linux/sunrpc/msg_prot.h>
 
 enum nfs4_acl_whotype {
 	NFS4_ACL_WHO_NAMED = 0,
@@ -659,4 +660,36 @@ struct nfs4_op_map {
 	} u;
 };
 
+struct nfs42_netaddr {
+	unsigned int	na_netid_len;
+	char		na_netid[RPCBIND_MAXNETIDLEN + 1];
+	unsigned int	na_uaddr_len;
+	char		na_uaddr[RPCBIND_MAXUADDRLEN + 1];
+};
+
+enum netloc_type4 {
+	NL4_NAME		= 1,
+	NL4_URL			= 2,
+	NL4_NETADDR		= 3,
+};
+
+struct nl4_server {
+	enum netloc_type4	nl4_type;
+	union {
+		struct { /* NL4_NAME, NL4_URL */
+			int	nl4_str_sz;
+			char	nl4_str[NFS4_OPAQUE_LIMIT + 1];
+		};
+		struct nfs42_netaddr	nl4_addr; /* NL4_NETADDR */
+	} u;
+};
+
+/*  support 1 nl4_server for now */
+#define NFS42_MAX_SSC_SRC       1
+
+struct nl4_servers {
+	int			nl_nsvr;
+	struct nl4_server	*nl_svr;
+};
+
 #endif
-- 
1.8.3.1

[RFC v3 17/42] NFS add COPY_NOTIFY operation

[Olga Kornievskaia]

Try using the delegation stateid, then the open stateid.

Only NL4_NETATTR, No support for NL4_NAME and NL4_URL.
Allow only one source server address to be returned for now.

Signed-off-by: Andy Adamson <andros@netapp.com>
Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42.h            |   2 +
 fs/nfs/nfs42proc.c        |  91 +++++++++++++++++++++++
 fs/nfs/nfs42xdr.c         | 186 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4_fs.h          |   1 +
 fs/nfs/nfs4file.c         |  28 +++++++
 fs/nfs/nfs4proc.c         |   1 +
 fs/nfs/nfs4state.c        |   2 +-
 fs/nfs/nfs4xdr.c          |   1 +
 include/linux/nfs4.h      |   1 +
 include/linux/nfs_fs_sb.h |   1 +
 include/linux/nfs_xdr.h   |  18 +++++
 11 files changed, 331 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs42.h b/fs/nfs/nfs42.h
index 54b9941..a0d6d45 100644
--- a/fs/nfs/nfs42.h
+++ b/fs/nfs/nfs42.h
@@ -22,5 +22,7 @@ int nfs42_proc_layoutstats_generic(struct nfs_server *,
 int nfs42_proc_clone(struct file *, struct file *, loff_t, loff_t, loff_t);
 int nfs42_proc_offload_status(struct file *, nfs4_stateid *,
 			      struct nfs42_offload_status_res *);
+int nfs42_proc_copy_notify(struct file *, struct file *,
+			   struct nfs42_copy_notify_res *);
 #endif /* CONFIG_NFS_V4_2) */
 #endif /* __LINUX_FS_NFS_NFS4_2_H */
diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index e145479..d133533 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -2,6 +2,7 @@
  * Copyright (c) 2014 Anna Schumaker <Anna.Schumaker@Netapp.com>
  */
 #include <linux/fs.h>
+#include <linux/sunrpc/addr.h>
 #include <linux/sunrpc/sched.h>
 #include <linux/nfs.h>
 #include <linux/nfs3.h>
@@ -14,10 +15,30 @@
 #include "pnfs.h"
 #include "nfs4session.h"
 #include "internal.h"
+#include "delegation.h"
 
 #define NFSDBG_FACILITY NFSDBG_PROC
 static int nfs42_do_offload_cancel_async(struct file *dst, nfs4_stateid *std);
 
+static void nfs42_set_netaddr(struct file *filep, struct nfs42_netaddr *naddr)
+{
+	struct nfs_client *clp = (NFS_SERVER(file_inode(filep)))->nfs_client;
+	unsigned short port = 2049;
+
+	rcu_read_lock();
+	naddr->na_netid_len = scnprintf(naddr->na_netid,
+					sizeof(naddr->na_netid), "%s",
+					rpc_peeraddr2str(clp->cl_rpcclient,
+					RPC_DISPLAY_NETID));
+	naddr->na_uaddr_len = scnprintf(naddr->na_uaddr,
+					sizeof(naddr->na_uaddr),
+					"%s.%u.%u",
+					rpc_peeraddr2str(clp->cl_rpcclient,
+					RPC_DISPLAY_ADDR),
+					port >> 8, port & 255);
+	rcu_read_unlock();
+}
+
 static int _nfs42_proc_fallocate(struct rpc_message *msg, struct file *filep,
 		struct nfs_lock_context *lock, loff_t offset, loff_t len)
 {
@@ -481,6 +502,76 @@ int nfs42_proc_offload_status(struct file *dst, nfs4_stateid *stateid,
 	return status;
 }
 
+int _nfs42_proc_copy_notify(struct file *src, struct file *dst,
+			    struct nfs42_copy_notify_args *args,
+			    struct nfs42_copy_notify_res *res)
+{
+	struct nfs_server *src_server = NFS_SERVER(file_inode(src));
+	struct rpc_message msg = {
+		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_COPY_NOTIFY],
+		.rpc_argp = args,
+		.rpc_resp = res,
+	};
+	int status;
+	struct nfs_open_context *ctx;
+	struct nfs_lock_context *l_ctx;
+
+	ctx = get_nfs_open_context(nfs_file_open_context(src));
+	l_ctx = nfs_get_lock_context(ctx);
+	if (IS_ERR(l_ctx))
+		return PTR_ERR(l_ctx);
+
+	status = nfs4_set_rw_stateid(&args->cna_src_stateid, ctx, l_ctx,
+				     FMODE_READ);
+	nfs_put_lock_context(l_ctx);
+	if (status)
+		return status;
+
+	status = nfs4_call_sync(src_server->client, src_server, &msg,
+				&args->cna_seq_args, &res->cnr_seq_res, 0);
+	if (status == -ENOTSUPP)
+		src_server->caps &= ~NFS_CAP_COPY_NOTIFY;
+
+	put_nfs_open_context(nfs_file_open_context(src));
+	return status;
+}
+
+int nfs42_proc_copy_notify(struct file *src, struct file *dst,
+				struct nfs42_copy_notify_res *res)
+{
+	struct nfs_server *src_server = NFS_SERVER(file_inode(src));
+	struct nfs42_copy_notify_args *args;
+	struct nfs4_exception exception = {
+		.inode = file_inode(src),
+	};
+	int status;
+
+	if (!(src_server->caps & NFS_CAP_COPY_NOTIFY))
+		return -EOPNOTSUPP;
+
+	args = kzalloc(sizeof(struct nfs42_copy_notify_args), GFP_NOFS);
+	if (args == NULL)
+		return -ENOMEM;
+
+	args->cna_src_fh  = NFS_FH(file_inode(src)),
+	args->cna_dst.nl4_type = NL4_NETADDR;
+	nfs42_set_netaddr(dst, &args->cna_dst.u.nl4_addr);
+	exception.stateid = &args->cna_src_stateid;
+
+	do {
+		status = _nfs42_proc_copy_notify(src, dst, args, res);
+		if (status == -ENOTSUPP) {
+			status = -EOPNOTSUPP;
+			goto out;
+		}
+		status = nfs4_handle_exception(src_server, status, &exception);
+	} while (exception.retry);
+
+out:
+	kfree(args);
+	return status;
+}
+
 static loff_t _nfs42_proc_llseek(struct file *filep,
 		struct nfs_lock_context *lock, loff_t offset, int whence)
 {
diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index f507fe4..ae84bae 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -33,6 +33,16 @@
 #define encode_offload_cancel_maxsz	(op_encode_hdr_maxsz + \
 					 XDR_QUADLEN(NFS4_STATEID_SIZE))
 #define decode_offload_cancel_maxsz	(op_decode_hdr_maxsz)
+#define encode_copy_notify_maxsz	(op_encode_hdr_maxsz + \
+					 XDR_QUADLEN(NFS4_STATEID_SIZE) + \
+					 1 + /* nl4_type */ \
+					 1 + XDR_QUADLEN(NFS4_OPAQUE_LIMIT))
+#define decode_copy_notify_maxsz	(op_decode_hdr_maxsz + \
+					 3 + /* cnr_lease_time */\
+					 XDR_QUADLEN(NFS4_STATEID_SIZE) + \
+					 1 + /* Support 1 cnr_source_server */\
+					 1 + /* nl4_type */ \
+					 1 + XDR_QUADLEN(NFS4_OPAQUE_LIMIT))
 #define encode_deallocate_maxsz		(op_encode_hdr_maxsz + \
 					 encode_fallocate_maxsz)
 #define decode_deallocate_maxsz		(op_decode_hdr_maxsz)
@@ -94,6 +104,12 @@
 #define NFS4_dec_offload_cancel_sz	(compound_decode_hdr_maxsz + \
 					 decode_putfh_maxsz + \
 					 decode_offload_cancel_maxsz)
+#define NFS4_enc_copy_notify_sz		(compound_encode_hdr_maxsz + \
+					 encode_putfh_maxsz + \
+					 encode_copy_notify_maxsz)
+#define NFS4_dec_copy_notify_sz		(compound_decode_hdr_maxsz + \
+					 decode_putfh_maxsz + \
+					 decode_copy_notify_maxsz)
 #define NFS4_enc_deallocate_sz		(compound_encode_hdr_maxsz + \
 					 encode_putfh_maxsz + \
 					 encode_deallocate_maxsz + \
@@ -147,6 +163,25 @@ static void encode_allocate(struct xdr_stream *xdr,
 	encode_fallocate(xdr, args);
 }
 
+static void encode_nl4_server(struct xdr_stream *xdr, struct nl4_server *ns)
+{
+	encode_uint32(xdr, ns->nl4_type);
+	switch (ns->nl4_type) {
+	case NL4_NAME:
+	case NL4_URL:
+		encode_string(xdr, ns->u.nl4_str_sz, ns->u.nl4_str);
+		break;
+	case NL4_NETADDR:
+		encode_string(xdr, ns->u.nl4_addr.na_netid_len,
+			      ns->u.nl4_addr.na_netid);
+		encode_string(xdr, ns->u.nl4_addr.na_uaddr_len,
+			      ns->u.nl4_addr.na_uaddr);
+		break;
+	default:
+		WARN_ON_ONCE(1);
+	}
+}
+
 static void encode_copy(struct xdr_stream *xdr,
 			struct nfs42_copy_args *args,
 			struct compound_hdr *hdr)
@@ -180,6 +215,15 @@ static void encode_offload_cancel(struct xdr_stream *xdr,
 	encode_nfs4_stateid(xdr, &args->osa_stateid);
 }
 
+static void encode_copy_notify(struct xdr_stream *xdr,
+			       struct nfs42_copy_notify_args *args,
+			       struct compound_hdr *hdr)
+{
+	encode_op_hdr(xdr, OP_COPY_NOTIFY, decode_copy_notify_maxsz, hdr);
+	encode_nfs4_stateid(xdr, &args->cna_src_stateid);
+	encode_nl4_server(xdr, &args->cna_dst);
+}
+
 static void encode_deallocate(struct xdr_stream *xdr,
 			      struct nfs42_falloc_args *args,
 			      struct compound_hdr *hdr)
@@ -331,6 +375,24 @@ static void nfs4_xdr_enc_offload_cancel(struct rpc_rqst *req,
 }
 
 /*
+ * Encode COPY_NOTIFY request
+ */
+static void nfs4_xdr_enc_copy_notify(struct rpc_rqst *req,
+				     struct xdr_stream *xdr,
+				     struct nfs42_copy_notify_args *args)
+{
+	struct compound_hdr hdr = {
+		.minorversion = nfs4_xdr_minorversion(&args->cna_seq_args),
+	};
+
+	encode_compound_hdr(xdr, req, &hdr);
+	encode_sequence(xdr, &args->cna_seq_args, &hdr);
+	encode_putfh(xdr, args->cna_src_fh, &hdr);
+	encode_copy_notify(xdr, args, &hdr);
+	encode_nops(&hdr);
+}
+
+/*
  * Encode DEALLOCATE request
  */
 static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,
@@ -445,6 +507,58 @@ static int decode_write_response(struct xdr_stream *xdr,
 	return -EIO;
 }
 
+static int decode_nl4_server(struct xdr_stream *xdr, struct nl4_server *ns)
+{
+	struct nfs42_netaddr *naddr;
+	uint32_t dummy;
+	char *dummy_str;
+	__be32 *p;
+	int status;
+
+	/* nl_type */
+	p = xdr_inline_decode(xdr, 4);
+	if (unlikely(!p))
+		return -EIO;
+	ns->nl4_type = be32_to_cpup(p);
+	switch (ns->nl4_type) {
+	case NL4_NAME:
+	case NL4_URL:
+		status = decode_opaque_inline(xdr, &dummy, &dummy_str);
+		if (unlikely(status))
+			return status;
+		if (unlikely(dummy > NFS4_OPAQUE_LIMIT))
+			return -EIO;
+		memcpy(&ns->u.nl4_str, dummy_str, dummy);
+		ns->u.nl4_str_sz = dummy;
+		break;
+	case NL4_NETADDR:
+		naddr = &ns->u.nl4_addr;
+
+		/* netid string */
+		status = decode_opaque_inline(xdr, &dummy, &dummy_str);
+		if (unlikely(status))
+			return status;
+		if (unlikely(dummy > RPCBIND_MAXNETIDLEN))
+			return -EIO;
+		naddr->na_netid_len = dummy;
+		memcpy(naddr->na_netid, dummy_str, naddr->na_netid_len);
+
+		/* uaddr string */
+		status = decode_opaque_inline(xdr, &dummy, &dummy_str);
+		if (unlikely(status))
+			return status;
+		if (unlikely(dummy > RPCBIND_MAXUADDRLEN))
+			return -EIO;
+		naddr->na_uaddr_len = dummy;
+		memcpy(naddr->na_uaddr, dummy_str, naddr->na_uaddr_len);
+		break;
+	default:
+		WARN_ON_ONCE(1);
+		return -EIO;
+	}
+	return 0;
+}
+
 static int decode_copy_requirements(struct xdr_stream *xdr,
 				    struct nfs42_copy_res *res) {
 	__be32 *p;
@@ -512,6 +626,53 @@ static int decode_offload_cancel(struct xdr_stream *xdr,
 	return decode_op_hdr(xdr, OP_OFFLOAD_CANCEL);
 }
 
+static int decode_copy_notify(struct xdr_stream *xdr,
+			      struct nfs42_copy_notify_res *res)
+{
+	__be32 *p;
+	struct nl4_server *ns;
+	int status, i;
+
+	status = decode_op_hdr(xdr, OP_COPY_NOTIFY);
+	if (status)
+		return status;
+	/* cnr_lease_time */
+	p = xdr_inline_decode(xdr, 12);
+	if (unlikely(!p))
+		goto out_overflow;
+	p = xdr_decode_hyper(p, &res->cnr_lease_time.seconds);
+	res->cnr_lease_time.nseconds = be32_to_cpup(p);
+
+	status = decode_opaque_fixed(xdr, &res->cnr_stateid, NFS4_STATEID_SIZE);
+	if (unlikely(status))
+		goto out_overflow;
+
+	/* number of source addresses */
+	p = xdr_inline_decode(xdr, 4);
+	if (unlikely(!p))
+		goto out_overflow;
+
+	res->cnr_src.nl_nsvr = be32_to_cpup(p);
+	if (res->cnr_src.nl_nsvr > NFS42_MAX_SSC_SRC) {
+		pr_warn("NFS: %s: nsvr %d > Supported. Use first %d servers\n",
+			 __func__, res->cnr_src.nl_nsvr, NFS42_MAX_SSC_SRC);
+		res->cnr_src.nl_nsvr = NFS42_MAX_SSC_SRC;
+	}
+
+	ns = res->cnr_src.nl_svr;
+	for (i = 0; i < res->cnr_src.nl_nsvr; i++) {
+		status = decode_nl4_server(xdr, ns);
+		if (unlikely(status))
+			goto out_overflow;
+		ns++;
+	}
+	return 0;
+
+out_overflow:
+	print_overflow_msg(__func__, xdr);
+	return -EIO;
+}
+
 static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *res)
 {
 	return decode_op_hdr(xdr, OP_DEALLOCATE);
@@ -661,6 +822,31 @@ static int nfs4_xdr_dec_offload_cancel(struct rpc_rqst *rqstp,
 }
 
 /*
+ * Decode COPY_NOTIFY response
+ */
+static int nfs4_xdr_dec_copy_notify(struct rpc_rqst *rqstp,
+				    struct xdr_stream *xdr,
+				    struct nfs42_copy_notify_res *res)
+{
+	struct compound_hdr hdr;
+	int status;
+
+	status = decode_compound_hdr(xdr, &hdr);
+	if (status)
+		goto out;
+	status = decode_sequence(xdr, &res->cnr_seq_res, rqstp);
+	if (status)
+		goto out;
+	status = decode_putfh(xdr);
+	if (status)
+		goto out;
+	status = decode_copy_notify(xdr, res);
+
+out:
+	return status;
+}
+
+/*
  * Decode DEALLOCATE request
  */
 static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,
diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 54e8778..70fa0e4 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -453,6 +453,7 @@ extern void nfs41_handle_server_scope(struct nfs_client *,
 extern int nfs4_select_rw_stateid(struct nfs4_state *, fmode_t,
 		const struct nfs_lock_context *, nfs4_stateid *,
 		struct rpc_cred **);
+extern void nfs4_copy_open_stateid(nfs4_stateid *, struct nfs4_state *);
 
 extern struct nfs_seqid *nfs_alloc_seqid(struct nfs_seqid_counter *counter, gfp_t gfp_mask);
 extern int nfs_wait_on_sequence(struct nfs_seqid *seqid, struct rpc_task *task);
diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c
index 881b819..b7b8c47 100644
--- a/fs/nfs/nfs4file.c
+++ b/fs/nfs/nfs4file.c
@@ -133,12 +133,40 @@ static ssize_t nfs4_copy_file_range(struct file *file_in, loff_t pos_in,
 				    struct file *file_out, loff_t pos_out,
 				    size_t count, unsigned int flags)
 {
+	struct nfs42_copy_notify_res *cn_resp = NULL;
 	ssize_t ret;
 
 	if (file_inode(file_in) == file_inode(file_out))
 		return -EINVAL;
 retry:
+	if (nfs42_files_from_same_server(file_in, file_out)) {  /* Intra-ssc */
+		if (file_in->f_op != file_out->f_op)
+			return -EXDEV;
+	} else {  /* Inter-ssc */
+		cn_resp = kzalloc(sizeof(struct nfs42_copy_notify_res),
+				  GFP_NOFS);
+		if (unlikely(cn_resp == NULL))
+			return -ENOMEM;
+
+		cn_resp->cnr_src.nl_svr = kzalloc(NFS42_MAX_SSC_SRC *
+						sizeof(struct nl4_server),
+						GFP_NOFS);
+		if (unlikely(cn_resp->cnr_src.nl_svr == NULL)) {
+			kfree(cn_resp);
+			return -ENOMEM;
+		}
+
+		ret = nfs42_proc_copy_notify(file_in, file_out, cn_resp);
+		if (ret)
+			goto out;
+	}
 	ret = nfs42_proc_copy(file_in, pos_in, file_out, pos_out, count);
+
+out:
+	if (cn_resp) {
+		kfree(cn_resp->cnr_src.nl_svr);
+		kfree(cn_resp);
+	}
 	if (ret == -EAGAIN)
 		goto retry;
 	return ret;
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index fdb6b76..94913fd 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -9232,6 +9232,7 @@ static bool nfs4_match_stateid(const nfs4_stateid *s1,
 		| NFS_CAP_COPY
 		| NFS_CAP_OFFLOAD_STATUS
 		| NFS_CAP_OFFLOAD_CANCEL
+		| NFS_CAP_COPY_NOTIFY
 		| NFS_CAP_DEALLOCATE
 		| NFS_CAP_SEEK
 		| NFS_CAP_LAYOUTSTATS
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 81f9d52..f640050 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -979,7 +979,7 @@ static int nfs4_copy_lock_stateid(nfs4_stateid *dst,
 	return ret;
 }
 
-static void nfs4_copy_open_stateid(nfs4_stateid *dst, struct nfs4_state *state)
+void nfs4_copy_open_stateid(nfs4_stateid *dst, struct nfs4_state *state)
 {
 	const nfs4_stateid *src;
 	int seq;
diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
index 9b66053..8c72bb6 100644
--- a/fs/nfs/nfs4xdr.c
+++ b/fs/nfs/nfs4xdr.c
@@ -7563,6 +7563,7 @@ struct rpc_procinfo	nfs4_procedures[] = {
 	PROC(COPY,		enc_copy,		dec_copy),
 	PROC(OFFLOAD_STATUS,	enc_offload_status,	dec_offload_status),
 	PROC(OFFLOAD_CANCEL,    enc_offload_cancel,     dec_offload_cancel),
+	PROC(COPY_NOTIFY,	enc_copy_notify,	dec_copy_notify),
 #endif /* CONFIG_NFS_V4_2 */
 };
 
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index 4179c78..5aa36ac 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -526,6 +526,7 @@ enum {
 	NFSPROC4_CLNT_COPY,
 	NFSPROC4_CLNT_OFFLOAD_STATUS,
 	NFSPROC4_CLNT_OFFLOAD_CANCEL,
+	NFSPROC4_CLNT_COPY_NOTIFY,
 };
 
 /* nfs41 types */
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 2b0397d..7099855 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -254,5 +254,6 @@ struct nfs_server {
 #define NFS_CAP_COPY		(1U << 24)
 #define NFS_CAP_OFFLOAD_STATUS	(1U << 25)
 #define NFS_CAP_OFFLOAD_CANCEL	(1U << 26)
+#define NFS_CAP_COPY_NOTIFY	(1U << 27)
 
 #endif
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 4875928..300289b 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1400,6 +1400,24 @@ struct nfs42_offload_status_res {
 	int				osr_status;
 };
 
+struct nfs42_copy_notify_args {
+	struct nfs4_sequence_args	cna_seq_args;
+
+	struct nfs_fh		*cna_src_fh;
+	nfs4_stateid		cna_src_stateid;
+	/* cna_destiniation_server */
+	struct nl4_server	cna_dst;
+};
+
+struct nfs42_copy_notify_res {
+	struct nfs4_sequence_res	cnr_seq_res;
+
+	struct nfstime4		cnr_lease_time;
+	nfs4_stateid		cnr_stateid;
+	/* cnr_source_server, for now, always 1 */
+	struct nl4_servers	cnr_src;
+};
+
 struct nfs42_seek_args {
 	struct nfs4_sequence_args	seq_args;
 
-- 
1.8.3.1

[RFC v3 18/42] NFS add ca_source_server<> to COPY

[Olga Kornievskaia]

Support only one source server address: the same address that
the client and source server use.

Signed-off-by: Andy Adamson <andros@netapp.com>
Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42.h          |  3 ++-
 fs/nfs/nfs42proc.c      | 26 +++++++++++++++++---------
 fs/nfs/nfs42xdr.c       | 19 +++++++++++++++++--
 fs/nfs/nfs4file.c       |  7 ++++++-
 include/linux/nfs_xdr.h |  1 +
 5 files changed, 43 insertions(+), 13 deletions(-)

diff --git a/fs/nfs/nfs42.h b/fs/nfs/nfs42.h
index a0d6d45..6bf5775 100644
--- a/fs/nfs/nfs42.h
+++ b/fs/nfs/nfs42.h
@@ -14,7 +14,8 @@
 /* nfs4.2proc.c */
 #if defined(CONFIG_NFS_V4_2)
 int nfs42_proc_allocate(struct file *, loff_t, loff_t);
-ssize_t nfs42_proc_copy(struct file *, loff_t, struct file *, loff_t, size_t);
+ssize_t nfs42_proc_copy(struct file *, loff_t, struct file *, loff_t, size_t,
+			struct nl4_servers *, nfs4_stateid *);
 int nfs42_proc_deallocate(struct file *, loff_t, loff_t);
 loff_t nfs42_proc_llseek(struct file *, loff_t, int);
 int nfs42_proc_layoutstats_generic(struct nfs_server *,
diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index d133533..36b265a 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -219,7 +219,9 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 				struct file *dst,
 				struct nfs_lock_context *dst_lock,
 				struct nfs42_copy_args *args,
-				struct nfs42_copy_res *res)
+				struct nfs42_copy_res *res,
+				struct nl4_servers *nss,
+				nfs4_stateid *cnr_stateid)
 {
 	struct rpc_message msg = {
 		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_COPY],
@@ -233,11 +235,15 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 	size_t count = args->count;
 	ssize_t status;
 
-	status = nfs4_set_rw_stateid(&args->src_stateid, src_lock->open_context,
-				     src_lock, FMODE_READ);
-	if (status)
-		return status;
-
+	if (nss) {
+		args->cp_src = nss;
+		nfs4_stateid_copy(&args->src_stateid, cnr_stateid);
+	} else {
+		status = nfs4_set_rw_stateid(&args->src_stateid,
+				src_lock->open_context, src_lock, FMODE_READ);
+		if (status)
+			return status;
+	}
 	status = nfs_filemap_write_and_wait_range(file_inode(src)->i_mapping,
 			pos_src, pos_src + (loff_t)count - 1);
 	if (status)
@@ -292,8 +298,9 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 }
 
 ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
-			struct file *dst, loff_t pos_dst,
-			size_t count)
+			struct file *dst, loff_t pos_dst, size_t count,
+			struct nl4_servers *nss,
+			nfs4_stateid *cnr_stateid)
 {
 	struct nfs_server *server = NFS_SERVER(file_inode(dst));
 	struct nfs_lock_context *src_lock;
@@ -338,7 +345,8 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
 		inode_lock(file_inode(dst));
 		err = _nfs42_proc_copy(src, src_lock,
 				dst, dst_lock,
-				&args, &res);
+				&args, &res,
+				nss, cnr_stateid);
 		inode_unlock(file_inode(dst));
 		switch (err) {
 		case -NFS4ERR_OFFLOAD_NO_REQS:
diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
index ae84bae..68ba3cc 100644
--- a/fs/nfs/nfs42xdr.c
+++ b/fs/nfs/nfs42xdr.c
@@ -20,7 +20,10 @@
 #define encode_copy_maxsz		(op_encode_hdr_maxsz +          \
 					 XDR_QUADLEN(NFS4_STATEID_SIZE) + \
 					 XDR_QUADLEN(NFS4_STATEID_SIZE) + \
-					 2 + 2 + 2 + 1 + 1 + 1)
+					 2 + 2 + 2 + 1 + 1 + 1 +\
+					 1 + /* One cnr_source_server */\
+					 1 + /* nl4_type */ \
+					 1 + XDR_QUADLEN(NFS4_OPAQUE_LIMIT))
 #define decode_copy_maxsz		(op_decode_hdr_maxsz + \
 					 NFS42_WRITE_RES_SIZE + \
 					 1 /* cr_consecutive */ + \
@@ -186,6 +189,9 @@ static void encode_copy(struct xdr_stream *xdr,
 			struct nfs42_copy_args *args,
 			struct compound_hdr *hdr)
 {
+	struct nl4_server *ns;
+	int i;
+
 	encode_op_hdr(xdr, OP_COPY, decode_copy_maxsz, hdr);
 	encode_nfs4_stateid(xdr, &args->src_stateid);
 	encode_nfs4_stateid(xdr, &args->dst_stateid);
@@ -196,7 +202,16 @@ static void encode_copy(struct xdr_stream *xdr,
 
 	encode_uint32(xdr, 1); /* consecutive = true */
 	encode_uint32(xdr, args->sync);
-	encode_uint32(xdr, 0); /* src server list */
+	if (args->cp_src == NULL) { /* intra-ssc */
+		encode_uint32(xdr, 0); /* no src server list */
+		return;
+	}
+	encode_uint32(xdr, args->cp_src->nl_nsvr);
+	ns = args->cp_src->nl_svr;
+	for (i = 0; i < args->cp_src->nl_nsvr; i++) {
+		encode_nl4_server(xdr, args->cp_src->nl_svr);
+		ns++;
+	}
 }
 
 static void encode_offload_status(struct xdr_stream *xdr,
diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c
index b7b8c47..0e15a6d 100644
--- a/fs/nfs/nfs4file.c
+++ b/fs/nfs/nfs4file.c
@@ -134,6 +134,8 @@ static ssize_t nfs4_copy_file_range(struct file *file_in, loff_t pos_in,
 				    size_t count, unsigned int flags)
 {
 	struct nfs42_copy_notify_res *cn_resp = NULL;
+	struct nl4_servers *nss = NULL;
+	nfs4_stateid *cnrs = NULL;
 	ssize_t ret;
 
 	if (file_inode(file_in) == file_inode(file_out))
@@ -159,8 +161,11 @@ static ssize_t nfs4_copy_file_range(struct file *file_in, loff_t pos_in,
 		ret = nfs42_proc_copy_notify(file_in, file_out, cn_resp);
 		if (ret)
 			goto out;
+		nss = &cn_resp->cnr_src;
+		cnrs = &cn_resp->cnr_stateid;
 	}
-	ret = nfs42_proc_copy(file_in, pos_in, file_out, pos_out, count);
+	ret = nfs42_proc_copy(file_in, pos_in, file_out, pos_out, count, nss,
+				cnrs);
 
 out:
 	if (cn_resp) {
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index 300289b..658aaff 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -1372,6 +1372,7 @@ struct nfs42_copy_args {
 
 	u64				count;
 	bool				sync;
+	struct nl4_servers		*cp_src;
 };
 
 struct nfs42_write_res {
-- 
1.8.3.1

[RFC v3 19/42] NFS also send OFFLOAD_CANCEL to source server

[Olga Kornievskaia]

In case of copy is cancelled, also send OFFLOAD_CANCEL to the source
server.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42proc.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index 36b265a..b7c08a9 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -206,12 +206,14 @@ static int handle_async_copy(struct nfs42_copy_res *res,
 	if (copy->count <= 0)
 		status = -copy->error;
 
+out_free:
 	kfree(copy);
 	return status;
 out_cancel:
 	nfs42_do_offload_cancel_async(dst, &copy->stateid);
-	kfree(copy);
-	return status;
+	if (!nfs42_files_from_same_server(src, dst))
+		nfs42_do_offload_cancel_async(src, src_stateid);
+	goto out_free;
 }
 
 static ssize_t _nfs42_proc_copy(struct file *src,
-- 
1.8.3.1

[RFC v3 20/42] NFS inter ssc open

[Olga Kornievskaia]

NFSv4.2 inter server to server copy requires the destination server to
READ the data from the source server using the provided stateid and
file handle.

Given an NFSv4 stateid and filehandle from the COPY operaion, provide the
destination server with an NFS client function to create a struct file
suitable for the destiniation server to READ the data to be copied.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4_fs.h  |   7 ++++
 fs/nfs/nfs4file.c | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfs/nfs4proc.c |   5 ++-
 3 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 70fa0e4..822c4cb 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -278,6 +278,13 @@ extern int nfs4_set_rw_stateid(nfs4_stateid *stateid,
 		const struct nfs_open_context *ctx,
 		const struct nfs_lock_context *l_ctx,
 		fmode_t fmode);
+extern int nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
+				struct nfs_fattr *fattr,
+				struct nfs4_label *label);
+extern int update_open_stateid(struct nfs4_state *state,
+				const nfs4_stateid *open_stateid,
+				const nfs4_stateid *deleg_stateid,
+				fmode_t fmode);
 
 #if defined(CONFIG_NFS_V4_1)
 extern int nfs41_sequence_done(struct rpc_task *, struct nfs4_sequence_res *);
diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c
index 0e15a6d..fa6ce82 100644
--- a/fs/nfs/nfs4file.c
+++ b/fs/nfs/nfs4file.c
@@ -7,6 +7,7 @@
 #include <linux/file.h>
 #include <linux/falloc.h>
 #include <linux/nfs_fs.h>
+#include <linux/file.h>
 #include <uapi/linux/btrfs.h>	/* BTRFS_IOC_CLONE/BTRFS_IOC_CLONE_RANGE */
 #include "delegation.h"
 #include "internal.h"
@@ -274,6 +275,111 @@ static int nfs42_clone_file_range(struct file *src_file, loff_t src_off,
 out:
 	return ret;
 }
+
+static int read_name_gen = 1;
+#define SSC_READ_NAME_BODY "ssc_read_%d"
+
+struct file *
+nfs42_ssc_open(struct vfsmount *ss_mnt, struct nfs_fh *src_fh,
+		nfs4_stateid *stateid)
+{
+	struct nfs_fattr fattr;
+	struct path path = {
+		.dentry = NULL,
+	};
+	struct file *filep, *res;
+	struct nfs_server *server;
+	struct inode *r_ino = NULL;
+	struct nfs_open_context *ctx;
+	struct nfs4_state_owner *sp;
+	char *read_name;
+	int len, status = 0;
+
+	server = NFS_SERVER(ss_mnt->mnt_root->d_inode);
+
+	nfs_fattr_init(&fattr);
+
+	status = nfs4_proc_getattr(server, src_fh, &fattr, NULL);
+	if (status < 0) {
+		res = ERR_PTR(status);
+		goto out;
+	}
+
+	res = ERR_PTR(-ENOMEM);
+	len = strlen(SSC_READ_NAME_BODY) + 16;
+	read_name = kzalloc(len, GFP_NOFS);
+	if (read_name == NULL)
+		goto out;
+	snprintf(read_name, len, SSC_READ_NAME_BODY, read_name_gen++);
+	dprintk("%s read_name %s\n", __func__, read_name);
+
+	/* Just put the file under the mount point */
+	path.dentry = d_alloc_name(ss_mnt->mnt_root, read_name);
+	kfree(read_name);
+	if (path.dentry == NULL)
+		goto out;
+	path.mnt = ss_mnt;
+	r_ino = nfs_fhget(ss_mnt->mnt_root->d_inode->i_sb, src_fh, &fattr,
+			NULL);
+	if (IS_ERR(r_ino)) {
+		res = ERR_CAST(r_ino);
+		goto out_path;
+	}
+
+	d_add(path.dentry, r_ino);
+
+	filep = alloc_file(&path, FMODE_READ, r_ino->i_fop);
+	if (IS_ERR(filep)) {
+		res = ERR_CAST(filep);
+		goto out_path;
+	}
+
+	ctx = alloc_nfs_open_context(filep->f_path.dentry, filep->f_mode,
+					filep);
+	if (IS_ERR(ctx)) {
+		res = ERR_CAST(ctx);
+		goto out_filep;
+	}
+
+	res = ERR_PTR(-EINVAL);
+	sp = nfs4_get_state_owner(server, ctx->cred, GFP_KERNEL);
+	if (sp == NULL)
+		goto out_ctx;
+
+	ctx->state = nfs4_get_open_state(r_ino, sp);
+	if (ctx->state == NULL)
+		goto out_stateowner;
+
+	update_open_stateid(ctx->state, stateid, NULL, filep->f_mode);
+
+	nfs_file_set_open_context(filep, ctx);
+	put_nfs_open_context(ctx);
+
+	file_ra_state_init(&filep->f_ra, filep->f_mapping->host->i_mapping);
+	res = filep;
+out:
+	dprintk("<-- %s error %ld filep %p r_ino %p\n",
+		__func__, IS_ERR(res) ? PTR_ERR(res) : 0, res, r_ino);
+
+	return res;
+out_stateowner:
+	nfs4_put_state_owner(sp);
+out_ctx:
+	put_nfs_open_context(ctx);
+out_filep:
+	fput(filep);
+out_path:
+	path_put(&path);
+	goto out;
+}
+EXPORT_SYMBOL_GPL(nfs42_ssc_open);
+void nfs42_ssc_close(struct file *filep)
+{
+	struct nfs_open_context *ctx = nfs_file_open_context(filep);
+
+	ctx->state->flags = 0;
+}
+EXPORT_SYMBOL_GPL(nfs42_ssc_close);
 #endif /* CONFIG_NFS_V4_2 */
 
 const struct file_operations nfs4_file_operations = {
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 94913fd..ecaadb0 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -89,7 +89,6 @@
 static int _nfs4_recover_proc_open(struct nfs4_opendata *data);
 static int nfs4_do_fsinfo(struct nfs_server *, struct nfs_fh *, struct nfs_fsinfo *);
 static void nfs_fixup_referral_attributes(struct nfs_fattr *fattr);
-static int nfs4_proc_getattr(struct nfs_server *, struct nfs_fh *, struct nfs_fattr *, struct nfs4_label *label);
 static int _nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle, struct nfs_fattr *fattr, struct nfs4_label *label);
 static int nfs4_do_setattr(struct inode *inode, struct rpc_cred *cred,
 			    struct nfs_fattr *fattr, struct iattr *sattr,
@@ -1461,7 +1460,7 @@ static void __update_open_stateid(struct nfs4_state *state,
 	spin_unlock(&state->owner->so_lock);
 }
 
-static int update_open_stateid(struct nfs4_state *state,
+int update_open_stateid(struct nfs4_state *state,
 		const nfs4_stateid *open_stateid,
 		const nfs4_stateid *delegation,
 		fmode_t fmode)
@@ -3615,7 +3614,7 @@ static int _nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
 	return nfs4_call_sync(server->client, server, &msg, &args.seq_args, &res.seq_res, 0);
 }
 
-static int nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
+int nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
 				struct nfs_fattr *fattr, struct nfs4_label *label)
 {
 	struct nfs4_exception exception = { };
-- 
1.8.3.1

[RFC v3 21/42] NFS skip recovery of copy open on dest server

[Olga Kornievskaia]

Mark the open created for the source file on the destination
server. Then if this open is going thru a recovery, then fail
the recovery as we don't need to be recoving a "fake" open.
We need to fail the ongoing READs and vfs_copy_file_range().

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs4_fs.h   |  1 +
 fs/nfs/nfs4file.c  |  2 +-
 fs/nfs/nfs4state.c | 14 ++++++++++++++
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 822c4cb..4ca9657 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -163,6 +163,7 @@ enum {
 	NFS_STATE_MAY_NOTIFY_LOCK,	/* server may CB_NOTIFY_LOCK */
 #ifdef CONFIG_NFS_V4_2
 	NFS_CLNT_DST_SSC_COPY_STATE,	/* dst server open state on client*/
+	NFS_SRV_SSC_COPY_STATE,		/* ssc state on the dst server */
 #endif /* CONFIG_NFS_V4_2 */
 
 };
diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c
index fa6ce82..49b6d15 100644
--- a/fs/nfs/nfs4file.c
+++ b/fs/nfs/nfs4file.c
@@ -349,7 +349,7 @@ struct file *
 	ctx->state = nfs4_get_open_state(r_ino, sp);
 	if (ctx->state == NULL)
 		goto out_stateowner;
-
+	set_bit(NFS_SRV_SSC_COPY_STATE, &ctx->state->flags);
 	update_open_stateid(ctx->state, stateid, NULL, filep->f_mode);
 
 	nfs_file_set_open_context(filep, ctx);
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index f640050..2c8556e 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1501,6 +1501,9 @@ static int nfs4_reclaim_open_state(struct nfs4_state_owner *sp, const struct nfs
 	struct nfs4_state *state;
 	struct nfs4_lock_state *lock;
 	int status = 0;
+#ifdef CONFIG_NFS_V4_2
+	bool found_ssc_copy_state = false;
+#endif /* CONFIG_NFS_V4_2 */
 
 	/* Note: we rely on the sp->so_states list being ordered 
 	 * so that we always reclaim open(O_RDWR) and/or open(O_WRITE)
@@ -1520,6 +1523,13 @@ static int nfs4_reclaim_open_state(struct nfs4_state_owner *sp, const struct nfs
 			continue;
 		if (state->state == 0)
 			continue;
+#ifdef CONFIG_NFS_V4_2
+		if (test_bit(NFS_SRV_SSC_COPY_STATE, &state->flags)) {
+			nfs4_state_mark_recovery_failed(state, -EIO);
+			found_ssc_copy_state = true;
+			continue;
+		}
+#endif /* CONFIG_NFS_V4_2 */
 		atomic_inc(&state->count);
 		spin_unlock(&sp->so_lock);
 		status = ops->recover_open(sp, state);
@@ -1597,6 +1607,10 @@ static int nfs4_reclaim_open_state(struct nfs4_state_owner *sp, const struct nfs
 	}
 	raw_write_seqcount_end(&sp->so_reclaim_seqcount);
 	spin_unlock(&sp->so_lock);
+#ifdef CONFIG_NFS_V4_2
+	if (found_ssc_copy_state)
+		return -EIO;
+#endif /* CONFIG_NFS_V4_2 */
 	return 0;
 out_err:
 	nfs4_put_open_state(state);
-- 
1.8.3.1

[RFC v3 22/42] NFS for "inter" copy treat ESTALE as ENOTSUPP

[Olga Kornievskaia]

If the client sends an "inter" copy to the destination server but
it only supports "intra" copy, it can return ESTALE (since it
doesn't know anything about the file handle from the other server
and does not recognize the special case of "inter" copy). Translate
this error as ENOTSUPP and also send OFFLOAD_CANCEL to teh source
server so that it can clean up state.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42proc.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index b7c08a9..09f653d 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -358,6 +358,13 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
 				continue;
 			}
 			break;
+		case -ESTALE:
+			if (!nfs42_files_from_same_server(src, dst)) {
+				err = -ENOTSUPP;
+				nfs42_do_offload_cancel_async(src,
+					&args.src_stateid);
+			}
+			break;
 		case -ENOTSUPP:
 			server->caps &= ~NFS_CAP_COPY;
 			if (nfs42_files_from_same_server(src, dst))
-- 
1.8.3.1

[RFC v3 23/42] NFS add a simple sync nfs4_proc_commit after async COPY

[Olga Kornievskaia]

A COPY with unstable write data needs a simple commit that doesnt
deal with inodes

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfs/nfs42proc.c | 22 ++++++++++++++++++++++
 fs/nfs/nfs4_fs.h   |  2 +-
 fs/nfs/nfs4proc.c  | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 56 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
index 09f653d..590bd50 100644
--- a/fs/nfs/nfs42proc.c
+++ b/fs/nfs/nfs42proc.c
@@ -289,6 +289,28 @@ static ssize_t _nfs42_proc_copy(struct file *src,
 			return status;
 	}
 
+	if ((!res->synchronous || !args->sync) &&
+			res->write_res.verifier.committed != NFS_FILE_SYNC) {
+		struct nfs_commitres cres;
+
+		cres.verf = kzalloc(sizeof(struct nfs_writeverf), GFP_NOFS);
+		if (!cres.verf)
+			return -ENOMEM;
+
+		status = nfs4_proc_commit(dst, pos_dst, res->write_res.count,
+					  &cres);
+		if (status) {
+			kfree(cres.verf);
+			return status;
+		}
+		if (!nfs_write_verifier_cmp(&res->write_res.verifier.verifier,
+					    &cres.verf->verifier)) {
+			/* what are we suppose to do here ? */
+			dprintk("commit verf differs from copy verf\n");
+		}
+		kfree(cres.verf);
+	}
+
 	truncate_pagecache_range(dst_inode, pos_dst,
 				 pos_dst + res->write_res.count);
 
diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index 4ca9657..1b817b4 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -477,7 +477,7 @@ extern int nfs4_sequence_done(struct rpc_task *task,
 			      struct nfs4_sequence_res *res);
 
 extern void nfs4_free_lock_state(struct nfs_server *server, struct nfs4_lock_state *lsp);
-
+extern int nfs4_proc_commit(struct file *dst, __u64 offset, __u32 count, struct nfs_commitres *res);
 extern const nfs4_stateid zero_stateid;
 
 /* nfs4super.c */
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index ecaadb0..c477333 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -4749,6 +4749,39 @@ static void nfs4_proc_commit_setup(struct nfs_commit_data *data, struct rpc_mess
 	nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1);
 }
 
+static int _nfs4_proc_commit(struct file *dst, struct nfs_commitargs *args,
+				struct nfs_commitres *res)
+{
+	struct inode *dst_inode = file_inode(dst);
+	struct nfs_server *server = NFS_SERVER(dst_inode);
+	struct rpc_message msg = {
+		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_COMMIT],
+		.rpc_argp = args,
+		.rpc_resp = res,
+	};
+	return nfs4_call_sync(server->client, server, &msg,
+			&args->seq_args, &res->seq_res, 1);
+}
+
+int nfs4_proc_commit(struct file *dst, __u64 offset, __u32 count, struct nfs_commitres *res)
+{
+	struct nfs_commitargs args = {
+		.fh = NFS_FH(file_inode(dst)),
+		.offset = offset,
+		.count = count,
+	};
+	struct nfs_server *dst_server = NFS_SERVER(file_inode(dst));
+	struct nfs4_exception exception = { };
+	int status;
+
+	do {
+		status = _nfs4_proc_commit(dst, &args, res);
+		status = nfs4_handle_exception(dst_server, status, &exception);
+	} while (exception.retry);
+
+	return status;
+}
+
 struct nfs4_renewdata {
 	struct nfs_client	*client;
 	unsigned long		timestamp;
-- 
1.8.3.1

[RFC v3 24/42] NFSD add ca_source_server<> to COPY

[Olga Kornievskaia]

From: Andy Adamson <andros@netapp.com>

Note: followed conventions and have struct nfsd4_compoundargs pointer as a
parameter even though it is unused.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfsd/nfs4xdr.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 fs/nfsd/xdr4.h    |  4 +++
 2 files changed, 77 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 26780d5..031d06d 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -41,6 +41,7 @@
 #include <linux/utsname.h>
 #include <linux/pagemap.h>
 #include <linux/sunrpc/svcauth_gss.h>
+#include <linux/sunrpc/addr.h>
 
 #include "idmap.h"
 #include "acl.h"
@@ -1726,11 +1727,58 @@ static __be32 nfsd4_decode_reclaim_complete(struct nfsd4_compoundargs *argp, str
 	DECODE_TAIL;
 }
 
+static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
+				      struct nl4_server *ns)
+{
+	DECODE_HEAD;
+	struct nfs42_netaddr *naddr;
+
+	READ_BUF(4);
+	ns->nl4_type = be32_to_cpup(p++);
+
+	/* currently support for 1 inter-server source server */
+	switch (ns->nl4_type) {
+	case NL4_NAME:
+	case NL4_URL:
+		READ_BUF(4);
+		ns->u.nl4_str_sz = be32_to_cpup(p++);
+		if (ns->u.nl4_str_sz > NFS4_OPAQUE_LIMIT)
+			goto xdr_error;
+
+		READ_BUF(ns->u.nl4_str_sz);
+		COPYMEM(ns->u.nl4_str,
+			ns->u.nl4_str_sz);
+		break;
+	case NL4_NETADDR:
+		naddr = &ns->u.nl4_addr;
+
+		READ_BUF(4);
+		naddr->na_netid_len = be32_to_cpup(p++);
+		if (naddr->na_netid_len > RPCBIND_MAXNETIDLEN)
+			goto xdr_error;
+
+		READ_BUF(naddr->na_netid_len + 4); /* 4 for uaddr len */
+		COPYMEM(naddr->na_netid, naddr->na_netid_len);
+
+		naddr->na_uaddr_len = be32_to_cpup(p++);
+		if (naddr->na_uaddr_len > RPCBIND_MAXUADDRLEN)
+			goto xdr_error;
+
+		READ_BUF(naddr->na_uaddr_len);
+		COPYMEM(naddr->na_uaddr, naddr->na_uaddr_len);
+		break;
+	default:
+		goto xdr_error;
+	}
+	DECODE_TAIL;
+}
+
 static __be32
 nfsd4_decode_copy(struct nfsd4_compoundargs *argp, struct nfsd4_copy *copy)
 {
 	DECODE_HEAD;
-	unsigned int tmp;
+	struct nl4_server *ns;
+	int i;
 
 	status = nfsd4_decode_stateid(argp, &copy->cp_src_stateid);
 	if (status)
@@ -1745,8 +1793,29 @@ static __be32 nfsd4_decode_reclaim_complete(struct nfsd4_compoundargs *argp, str
 	p = xdr_decode_hyper(p, &copy->cp_count);
 	copy->cp_consecutive = be32_to_cpup(p++);
 	copy->cp_synchronous = be32_to_cpup(p++);
-	tmp = be32_to_cpup(p); /* Source server list not supported */
+	copy->cp_src.nl_nsvr = be32_to_cpup(p++);
 
+	if (copy->cp_src.nl_nsvr == 0) /* intra-server copy */
+		goto intra;
+
+	/** Support NFSD4_MAX_SSC_SRC number of source servers.
+	 * freed in nfsd4_encode_copy
+	 */
+	if (copy->cp_src.nl_nsvr > NFSD4_MAX_SSC_SRC)
+		copy->cp_src.nl_nsvr = NFSD4_MAX_SSC_SRC;
+	copy->cp_src.nl_svr = kmalloc(copy->cp_src.nl_nsvr *
+					sizeof(struct nl4_server), GFP_KERNEL);
+	if (copy->cp_src.nl_svr == NULL)
+		return nfserrno(-ENOMEM);
+
+	ns = copy->cp_src.nl_svr;
+	for (i = 0; i < copy->cp_src.nl_nsvr; i++) {
+		status = nfsd4_decode_nl4_server(argp, ns);
+		if (status)
+			return status;
+		ns++;
+	}
+intra:
 	DECODE_TAIL;
 }
 
@@ -4300,6 +4369,8 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 		*p++ = cpu_to_be32(copy->cp_consecutive);
 		*p++ = cpu_to_be32(copy->cp_synchronous);
 	}
+	/* allocated in nfsd4_decode_copy */
+	kfree(copy->cp_src.nl_svr);
 	return nfserr;
 }
 
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index 8fda4ab..6b1a61fc 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -509,6 +509,9 @@ struct nfsd42_write_res {
 	nfs4_verifier		wr_verifier;
 };
 
+/*  support 1 source server for now */
+#define NFSD4_MAX_SSC_SRC       1
+
 struct nfsd4_copy {
 	/* request */
 	stateid_t	cp_src_stateid;
@@ -516,6 +519,7 @@ struct nfsd4_copy {
 	u64		cp_src_pos;
 	u64		cp_dst_pos;
 	u64		cp_count;
+	struct nl4_servers cp_src;
 
 	/* both */
 	bool		cp_consecutive;
-- 
1.8.3.1

[RFC v3 25/42] NFSD add COPY_NOTIFY operation

[Olga Kornievskaia]

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfsd/nfs4proc.c |  98 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/nfs4xdr.c  | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 fs/nfsd/xdr4.h     |  13 +++++++
 3 files changed, 221 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index dadb3bf..ed6b9f2 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -35,6 +35,7 @@
 #include <linux/file.h>
 #include <linux/falloc.h>
 #include <linux/slab.h>
+#include <linux/sunrpc/addr.h>
 
 #include "idmap.h"
 #include "cache.h"
@@ -1087,6 +1088,82 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 }
 
 static __be32
+nfsd4_set_src_nl4_netaddr(struct svc_rqst *rqstp, struct nl4_servers *svrs)
+{
+	const struct sockaddr *addr = (struct sockaddr *)&rqstp->rq_daddr;
+	const struct sockaddr_in *sin = (const struct sockaddr_in *)addr;
+	const struct sockaddr_in6 *sin6 = (const struct sockaddr_in6 *)addr;
+	int uaddr_len = rqstp->rq_daddrlen + 4 + 1; /* port (4) and '\0' (1) */
+	struct nfs42_netaddr *naddr;
+	size_t ret;
+	unsigned short port;
+
+	/* freed in nfsd4_encode_copy_notify */
+	svrs->nl_svr = kmalloc_array(svrs->nl_nsvr, sizeof(struct nl4_server),
+				GFP_KERNEL);
+	if (svrs->nl_svr == NULL)
+		return nfserrno(-ENOMEM);
+
+	svrs->nl_svr->nl4_type = NL4_NETADDR;
+	naddr = &svrs->nl_svr->u.nl4_addr;
+
+	switch (addr->sa_family) {
+	case AF_INET:
+		port = ntohs(sin->sin_port);
+		sprintf(naddr->na_netid, "tcp");
+		naddr->na_netid_len = 3;
+		break;
+	case AF_INET6:
+		port = ntohs(sin6->sin6_port);
+		sprintf(naddr->na_netid, "tcp6");
+		naddr->na_netid_len = 4;
+		break;
+	default:
+		dprintk("NFSD  nfsd4_set_notify_src: unknown address type: %d",
+			addr->sa_family);
+		kfree(svrs->nl_svr);
+		return nfserrno(-EINVAL);
+	}
+	ret = rpc_ntop(addr, naddr->na_uaddr, sizeof(naddr->na_uaddr));
+	snprintf(naddr->na_uaddr + ret, uaddr_len, ".%u.%u",
+		 port >> 8, port & 255);
+	naddr->na_uaddr_len = strlen(naddr->na_uaddr);
+	return 0;
+}
+
+static __be32
+nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
+		  struct nfsd4_copy_notify *cn)
+{
+	__be32 status;
+	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
+
+	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
+					&cn->cpn_src_stateid, RD_STATE, NULL,
+					NULL);
+	if (status)
+		return status;
+
+	cn->cpn_sec = nn->nfsd4_lease;
+	cn->cpn_nsec = 0;
+
+
+	/** XXX Save cpn_src_statid, cpn_src, and any other returned source
+	 * server addresses on which the source server is williing to accept
+	 * connections from the destination e.g. what is returned in cpn_src,
+	 * to verify READ from dest server.
+	 */
+
+	/**
+	 * For now, only return one server address in cpn_src, the
+	 * address used by the client to connect to this server.
+	 */
+	cn->cpn_src.nl_nsvr = 1;
+
+	return nfsd4_set_src_nl4_netaddr(rqstp, &cn->cpn_src);
+}
+
+static __be32
 nfsd4_fallocate(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		struct nfsd4_fallocate *fallocate, int flags)
 {
@@ -2042,6 +2119,21 @@ static inline u32 nfsd4_copy_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
 		1 /* cr_synchronous */) * sizeof(__be32);
 }
 
+static inline u32 nfsd4_copy_notify_rsize(struct svc_rqst *rqstp,
+					struct nfsd4_op *op)
+{
+	return (op_encode_hdr_size +
+		3 /* cnr_lease_time */ +
+		1 /* We support one cnr_source_server */ +
+		1 /* cnr_stateid seq */ +
+		op_encode_stateid_maxsz /* cnr_stateid */ +
+		1 /* num cnr_source_server*/ +
+		1 /* nl4_type */ +
+		1 /* nl4 size */ +
+		XDR_QUADLEN(NFS4_OPAQUE_LIMIT) /*nl4_loc + nl4_loc_sz */)
+		* sizeof(__be32);
+}
+
 #ifdef CONFIG_NFSD_PNFS
 static inline u32 nfsd4_getdeviceinfo_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
 {
@@ -2446,6 +2538,12 @@ static inline u32 nfsd4_seek_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
 		.op_name = "OP_SEEK",
 		.op_rsize_bop = (nfsd4op_rsize)nfsd4_seek_rsize,
 	},
+	[OP_COPY_NOTIFY] = {
+		.op_func = (nfsd4op_func)nfsd4_copy_notify,
+		.op_flags = OP_MODIFIES_SOMETHING | OP_CACHEME,
+		.op_name = "OP_COPY_NOTIFY",
+		.op_rsize_bop = (nfsd4op_rsize)nfsd4_copy_notify_rsize,
+	},
 };
 
 /**
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 031d06d..3e08c15 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1820,6 +1820,22 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
 }
 
 static __be32
+nfsd4_decode_copy_notify(struct nfsd4_compoundargs *argp,
+			 struct nfsd4_copy_notify *cn)
+{
+	int status;
+
+	status = nfsd4_decode_stateid(argp, &cn->cpn_src_stateid);
+	if (status)
+		return status;
+	status = nfsd4_decode_nl4_server(argp, &cn->cpn_dst);
+	if (status)
+		return status;
+
+	return status;
+}
+
+static __be32
 nfsd4_decode_seek(struct nfsd4_compoundargs *argp, struct nfsd4_seek *seek)
 {
 	DECODE_HEAD;
@@ -1920,7 +1936,7 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
 	/* new operations for NFSv4.2 */
 	[OP_ALLOCATE]		= (nfsd4_dec)nfsd4_decode_fallocate,
 	[OP_COPY]		= (nfsd4_dec)nfsd4_decode_copy,
-	[OP_COPY_NOTIFY]	= (nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_COPY_NOTIFY]	= (nfsd4_dec)nfsd4_decode_copy_notify,
 	[OP_DEALLOCATE]		= (nfsd4_dec)nfsd4_decode_fallocate,
 	[OP_IO_ADVISE]		= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_LAYOUTERROR]	= (nfsd4_dec)nfsd4_decode_notsupp,
@@ -4355,6 +4371,52 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 }
 
 static __be32
+nfsd42_encode_nl4_server(struct nfsd4_compoundres *resp, struct nl4_server *ns)
+{
+	struct xdr_stream *xdr = &resp->xdr;
+	struct nfs42_netaddr *addr;
+	__be32 *p;
+
+	p = xdr_reserve_space(xdr, 4);
+	*p++ = cpu_to_be32(ns->nl4_type);
+
+	switch (ns->nl4_type) {
+	case NL4_NAME:
+	case NL4_URL:
+		p = xdr_reserve_space(xdr, 4 /* url or name len */ +
+				      (XDR_QUADLEN(ns->u.nl4_str_sz) * 4));
+		if (!p)
+			return nfserr_resource;
+		*p++ = cpu_to_be32(ns->u.nl4_str_sz);
+		p = xdr_encode_opaque_fixed(p, ns->u.nl4_str, ns->u.nl4_str_sz);
+			break;
+	case NL4_NETADDR:
+		addr = &ns->u.nl4_addr;
+
+		/** netid_len, netid, uaddr_len, uaddr (port included
+		 * in RPCBIND_MAXUADDRLEN)
+		 */
+		p = xdr_reserve_space(xdr,
+			4 /* netid len */ +
+			(XDR_QUADLEN(addr->na_netid_len) * 4) +
+			4 /* uaddr len */ +
+			(XDR_QUADLEN(addr->na_uaddr_len) * 4));
+		if (!p)
+			return nfserr_resource;
+
+		*p++ = cpu_to_be32(addr->na_netid_len);
+		p = xdr_encode_opaque_fixed(p, addr->na_netid,
+					    addr->na_netid_len);
+		*p++ = cpu_to_be32(addr->na_uaddr_len);
+		p = xdr_encode_opaque_fixed(p, addr->na_uaddr,
+					addr->na_uaddr_len);
+		break;
+	}
+
+	return 0;
+}
+
+static __be32
 nfsd4_encode_copy(struct nfsd4_compoundres *resp, __be32 nfserr,
 		  struct nfsd4_copy *copy)
 {
@@ -4375,6 +4437,52 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 }
 
 static __be32
+nfsd4_encode_copy_notify(struct nfsd4_compoundres *resp, __be32 nfserr,
+			 struct nfsd4_copy_notify *cn)
+{
+	struct xdr_stream *xdr = &resp->xdr;
+	struct nl4_server *ns;
+	__be32 *p;
+	int i;
+
+	if (nfserr)
+		return nfserr;
+
+	/* 8 sec, 4 nsec */
+	p = xdr_reserve_space(xdr, 12);
+	if (!p)
+		return nfserr_resource;
+
+	/* cnr_lease_time */
+	p = xdr_encode_hyper(p, cn->cpn_sec);
+	*p++ = cpu_to_be32(cn->cpn_nsec);
+
+	/* cnr_stateid */
+	nfserr = nfsd4_encode_stateid(xdr, &cn->cpn_src_stateid);
+	if (nfserr)
+		return nfserr;
+
+	/* cnr_src.nl_nsvr */
+	p = xdr_reserve_space(xdr, 4);
+	if (!p)
+		return nfserr_resource;
+
+	*p++ = cpu_to_be32(cn->cpn_src.nl_nsvr);
+
+	ns = cn->cpn_src.nl_svr;
+	for (i = 0; i < cn->cpn_src.nl_nsvr; i++) {
+		nfserr = nfsd42_encode_nl4_server(resp, ns);
+		if (nfserr)
+			return nfserr;
+		ns++;
+	}
+
+	/* allocated in nfsd4_copy_notify */
+	kfree(cn->cpn_src.nl_svr);
+	return nfserr;
+}
+
+static __be32
 nfsd4_encode_seek(struct nfsd4_compoundres *resp, __be32 nfserr,
 		  struct nfsd4_seek *seek)
 {
@@ -4474,7 +4582,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	/* NFSv4.2 operations */
 	[OP_ALLOCATE]		= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_COPY]		= (nfsd4_enc)nfsd4_encode_copy,
-	[OP_COPY_NOTIFY]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_COPY_NOTIFY]	= (nfsd4_enc)nfsd4_encode_copy_notify,
 	[OP_DEALLOCATE]		= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_IO_ADVISE]		= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_LAYOUTERROR]	= (nfsd4_enc)nfsd4_encode_noop,
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index 6b1a61fc..5db7cd8 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -540,6 +540,18 @@ struct nfsd4_seek {
 	loff_t		seek_pos;
 };
 
+struct nfsd4_copy_notify {
+	/* request */
+	stateid_t		cpn_src_stateid;
+	struct nl4_server	cpn_dst;
+
+	/* response */
+	/* Note: cpn_src_stateid is used for cnr_stateid */
+	u64			cpn_sec;
+	u32			cpn_nsec;
+	struct nl4_servers	cpn_src;
+};
+
 struct nfsd4_op {
 	int					opnum;
 	__be32					status;
@@ -595,6 +607,7 @@ struct nfsd4_op {
 		struct nfsd4_fallocate		deallocate;
 		struct nfsd4_clone		clone;
 		struct nfsd4_copy		copy;
+		struct nfsd4_copy_notify	copy_notify;
 		struct nfsd4_seek		seek;
 	} u;
 	struct nfs4_replay *			replay;
-- 
1.8.3.1

[RFC v3 26/42] NFSD generalize nfsd4_compound_state flag names

[Olga Kornievskaia]

From: Andy Adamson <andros@netapp.com>

Allow for sid_flag field non-stateid use.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfsd/nfs4proc.c  | 8 ++++----
 fs/nfsd/nfs4state.c | 7 ++++---
 fs/nfsd/xdr4.h      | 6 +++---
 3 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index ed6b9f2..b49ff31 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -522,9 +522,9 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
 		return nfserr_restorefh;
 
 	fh_dup2(&cstate->current_fh, &cstate->save_fh);
-	if (HAS_STATE_ID(cstate, SAVED_STATE_ID_FLAG)) {
+	if (HAS_CSTATE_FLAG(cstate, SAVED_STATE_ID_FLAG)) {
 		memcpy(&cstate->current_stateid, &cstate->save_stateid, sizeof(stateid_t));
-		SET_STATE_ID(cstate, CURRENT_STATE_ID_FLAG);
+		SET_CSTATE_FLAG(cstate, CURRENT_STATE_ID_FLAG);
 	}
 	return nfs_ok;
 }
@@ -537,9 +537,9 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
 		return nfserr_nofilehandle;
 
 	fh_dup2(&cstate->save_fh, &cstate->current_fh);
-	if (HAS_STATE_ID(cstate, CURRENT_STATE_ID_FLAG)) {
+	if (HAS_CSTATE_FLAG(cstate, CURRENT_STATE_ID_FLAG)) {
 		memcpy(&cstate->save_stateid, &cstate->current_stateid, sizeof(stateid_t));
-		SET_STATE_ID(cstate, SAVED_STATE_ID_FLAG);
+		SET_CSTATE_FLAG(cstate, SAVED_STATE_ID_FLAG);
 	}
 	return nfs_ok;
 }
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 22002fb..e7e1b9f 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -7080,7 +7080,8 @@ static int nfs4_state_create_net(struct net *net)
 static void
 get_stateid(struct nfsd4_compound_state *cstate, stateid_t *stateid)
 {
-	if (HAS_STATE_ID(cstate, CURRENT_STATE_ID_FLAG) && CURRENT_STATEID(stateid))
+	if (HAS_CSTATE_FLAG(cstate, CURRENT_STATE_ID_FLAG) &&
+	    CURRENT_STATEID(stateid))
 		memcpy(stateid, &cstate->current_stateid, sizeof(stateid_t));
 }
 
@@ -7089,14 +7090,14 @@ static int nfs4_state_create_net(struct net *net)
 {
 	if (cstate->minorversion) {
 		memcpy(&cstate->current_stateid, stateid, sizeof(stateid_t));
-		SET_STATE_ID(cstate, CURRENT_STATE_ID_FLAG);
+		SET_CSTATE_FLAG(cstate, CURRENT_STATE_ID_FLAG);
 	}
 }
 
 void
 clear_current_stateid(struct nfsd4_compound_state *cstate)
 {
-	CLEAR_STATE_ID(cstate, CURRENT_STATE_ID_FLAG);
+	CLEAR_CSTATE_FLAG(cstate, CURRENT_STATE_ID_FLAG);
 }
 
 /*
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index 5db7cd8..38fcb4f 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -46,9 +46,9 @@
 #define CURRENT_STATE_ID_FLAG (1<<0)
 #define SAVED_STATE_ID_FLAG (1<<1)
 
-#define SET_STATE_ID(c, f) ((c)->sid_flags |= (f))
-#define HAS_STATE_ID(c, f) ((c)->sid_flags & (f))
-#define CLEAR_STATE_ID(c, f) ((c)->sid_flags &= ~(f))
+#define SET_CSTATE_FLAG(c, f) ((c)->sid_flags |= (f))
+#define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f))
+#define CLEAR_CSTATE_FLAG(c, f) ((c)->sid_flags &= ~(f))
 
 struct nfsd4_compound_state {
 	struct svc_fh		current_fh;
-- 
1.8.3.1

[RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[Olga Kornievskaia]

The inter server to server COPY source server filehandle
is guaranteed to be stale as the COPY is sent to the destination
server.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
 fs/nfsd/nfs4xdr.c  | 26 +++++++++++++++++++++++++-
 fs/nfsd/nfsd.h     |  2 ++
 fs/nfsd/xdr4.h     |  4 ++++
 4 files changed, 77 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index b49ff31..ceee852 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -496,11 +496,19 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
 nfsd4_putfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	    struct nfsd4_putfh *putfh)
 {
+	__be32 ret;
+
 	fh_put(&cstate->current_fh);
 	cstate->current_fh.fh_handle.fh_size = putfh->pf_fhlen;
 	memcpy(&cstate->current_fh.fh_handle.fh_base, putfh->pf_fhval,
 	       putfh->pf_fhlen);
-	return fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
+	ret = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
+	if (ret == nfserr_stale && HAS_CSTATE_FLAG(cstate, NO_VERIFY_FH)) {
+		CLEAR_CSTATE_FLAG(cstate, NO_VERIFY_FH);
+		SET_CSTATE_FLAG(cstate, IS_STALE_FH);
+		ret = 0;
+	}
+	return ret;
 }
 
 static __be32
@@ -533,6 +541,16 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
 nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	     void *arg)
 {
+	/**
+	* This is either an inter COPY (most likely) or an intra COPY with a
+	* stale file handle. If the latter, nfsd4_copy will reset the PUTFH to
+	* return nfserr_stale. No fh_dentry, just copy the file handle
+	* to use with the inter COPY READ.
+	*/
+	if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
+		cstate->save_fh = cstate->current_fh;
+		return nfs_ok;
+	}
 	if (!cstate->current_fh.fh_dentry)
 		return nfserr_nofilehandle;
 
@@ -1067,6 +1085,13 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 	if (status)
 		goto out;
 
+	/* Intra copy source fh is stale. PUTFH will fail with ESTALE */
+	if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
+		CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
+		cstate->status = nfserr_copy_stalefh;
+		goto out_put;
+	}
+
 	bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
 			dst, copy->cp_dst_pos, copy->cp_count);
 
@@ -1081,6 +1106,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 		status = nfs_ok;
 	}
 
+out_put:
 	fput(src);
 	fput(dst);
 out:
@@ -1759,6 +1785,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
 	struct nfsd4_compound_state *cstate = &resp->cstate;
 	struct svc_fh *current_fh = &cstate->current_fh;
 	struct svc_fh *save_fh = &cstate->save_fh;
+	int		i;
 	__be32		status;
 
 	svcxdr_init_encode(rqstp, resp);
@@ -1791,6 +1818,12 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
 		goto encode_op;
 	}
 
+	/* NFSv4.2 COPY source file handle may be from a different server */
+	for (i = 0; i < args->opcnt; i++) {
+		op = &args->ops[i];
+		if (op->opnum == OP_COPY)
+			SET_CSTATE_FLAG(cstate, NO_VERIFY_FH);
+	}
 	while (!status && resp->opcnt < args->opcnt) {
 		op = &args->ops[resp->opcnt++];
 
@@ -1810,6 +1843,9 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
 
 		opdesc = OPDESC(op);
 
+		if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
+			goto call_op;
+
 		if (!current_fh->fh_dentry) {
 			if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
 				op->status = nfserr_nofilehandle;
@@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
 
 		if (opdesc->op_get_currentstateid)
 			opdesc->op_get_currentstateid(cstate, &op->u);
+call_op:
 		op->status = opdesc->op_func(rqstp, cstate, &op->u);
 
 		/* Only from SEQUENCE */
@@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
 			if (need_wrongsec_check(rqstp))
 				op->status = check_nfsd_access(current_fh->fh_export, rqstp);
 		}
+		/* Only from intra COPY */
+		if (cstate->status == nfserr_copy_stalefh) {
+			dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
+				__func__);
+			status = nfserr_stale;
+			nfsd4_adjust_encode(resp);
+			goto out;
+		}
 encode_op:
 		if (op->status == nfserr_replay_me) {
 			op->replay = &cstate->replay_owner->so_replay;
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 3e08c15..2896a11 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -4624,15 +4624,28 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
 	return nfserr_rep_too_big;
 }
 
+/** Rewind the encoding to return nfserr_stale on the PUTFH
+ * in this failed Intra COPY compound
+ */
+void
+nfsd4_adjust_encode(struct nfsd4_compoundres *resp)
+{
+	__be32 *p;
+
+	p = resp->cstate.putfh_errp;
+	*p++ = nfserr_stale;
+}
+
 void
 nfsd4_encode_operation(struct nfsd4_compoundres *resp, struct nfsd4_op *op)
 {
 	struct xdr_stream *xdr = &resp->xdr;
 	struct nfs4_stateowner *so = resp->cstate.replay_owner;
+	struct nfsd4_compound_state *cstate = &resp->cstate;
 	struct svc_rqst *rqstp = resp->rqstp;
 	int post_err_offset;
 	nfsd4_enc encoder;
-	__be32 *p;
+	__be32 *p, *statp;
 
 	p = xdr_reserve_space(xdr, 8);
 	if (!p) {
@@ -4641,9 +4654,20 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
 	}
 	*p++ = cpu_to_be32(op->opnum);
 	post_err_offset = xdr->buf->len;
+	statp = p;
 
 	if (op->opnum == OP_ILLEGAL)
 		goto status;
+
+	/** This is a COPY compound with a stale source server file handle.
+	 * If OP_COPY processing determines that this is an intra server to
+	 * server COPY, then this PUTFH should return nfserr_ stale so the
+	 * putfh_errp will be set to nfserr_stale. If this is an inter server
+	 * to server COPY, ignore the nfserr_stale.
+	 */
+	if (op->opnum == OP_PUTFH && HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
+		cstate->putfh_errp = statp;
+
 	BUG_ON(op->opnum < 0 || op->opnum >= ARRAY_SIZE(nfsd4_enc_ops) ||
 	       !nfsd4_enc_ops[op->opnum]);
 	encoder = nfsd4_enc_ops[op->opnum];
diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
index d966068..8d6fb0f 100644
--- a/fs/nfsd/nfsd.h
+++ b/fs/nfsd/nfsd.h
@@ -272,6 +272,8 @@ static inline bool nfsd4_spo_must_allow(struct svc_rqst *rqstp)
 #define	nfserr_replay_me	cpu_to_be32(11001)
 /* nfs41 replay detected */
 #define	nfserr_replay_cache	cpu_to_be32(11002)
+/* nfs42 intra copy failed with nfserr_stale */
+#define nfserr_copy_stalefh	cpu_to_be32(1103)
 
 /* Check for dir entries '.' and '..' */
 #define isdotent(n, l)	(l < 3 && n[0] == '.' && (l == 1 || n[1] == '.'))
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index 38fcb4f..aa94295 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -45,6 +45,8 @@
 
 #define CURRENT_STATE_ID_FLAG (1<<0)
 #define SAVED_STATE_ID_FLAG (1<<1)
+#define NO_VERIFY_FH (1<<2)
+#define IS_STALE_FH  (1<<3)
 
 #define SET_CSTATE_FLAG(c, f) ((c)->sid_flags |= (f))
 #define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f))
@@ -63,6 +65,7 @@ struct nfsd4_compound_state {
 	size_t			iovlen;
 	u32			minorversion;
 	__be32			status;
+	__be32			*putfh_errp;
 	stateid_t	current_stateid;
 	stateid_t	save_stateid;
 	/* to indicate current and saved state id presents */
@@ -705,6 +708,7 @@ int nfs4svc_decode_compoundargs(struct svc_rqst *, __be32 *,
 int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
 		struct nfsd4_compoundres *);
 __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32);
+void nfsd4_adjust_encode(struct nfsd4_compoundres *);
 void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
 void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op);
 __be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words,
-- 
1.8.3.1

[RFC v3 28/42] NFSD add nfs4 inter ssc to nfsd4_copy

[Olga Kornievskaia]

Given a universal address, mount the source server from the destination
server.  Use an internal mount. Call the NFS client nfs42_ssc_open to
obtain the NFS struct file suitable for nfsd_copy_range.

Add Kconfig dependencies for inter server to server copy

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfsd/Kconfig      |  10 ++
 fs/nfsd/nfs4proc.c   | 262 +++++++++++++++++++++++++++++++++++++++++++++++++--
 include/linux/nfs4.h |   1 +
 3 files changed, 263 insertions(+), 10 deletions(-)

diff --git a/fs/nfsd/Kconfig b/fs/nfsd/Kconfig
index 20b1c17..37ff3d5 100644
--- a/fs/nfsd/Kconfig
+++ b/fs/nfsd/Kconfig
@@ -131,6 +131,16 @@ config NFSD_FLEXFILELAYOUT
 
 	  If unsure, say N.
 
+config NFSD_V4_2_INTER_SSC
+	bool "NFSv4.2 inter server to server COPY"
+	depends on NFSD_V4 && NFS_V4_1 && NFS_V4_2
+	help
+	  This option enables support for NFSv4.2 inter server to
+	  server copy where the destination server calls the NFSv4.2
+	  client to read the data to copy from the source server.
+
+	  If unsure, say N.
+
 config NFSD_V4_SECURITY_LABEL
 	bool "Provide Security Label support for NFSv4 server"
 	depends on NFSD_V4 && SECURITY
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index ceee852..b1095e9 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1072,16 +1072,227 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 	return status;
 }
 
+#ifdef CONFIG_NFSD_V4_2_INTER_SSC
+
+extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
+				   struct nfs_fh *src_fh,
+				   nfs4_stateid *stateid);
+extern struct file *nfs42_ssc_close(struct file *filep);
+
+extern void nfs_sb_deactive(struct super_block *sb);
+
+#define NFSD42_INTERSSC_RAWDATA "minorversion=2,vers=4,addr=%s,clientaddr=%s"
+
+/**
+ * Support one copy source server for now.
+ */
+static struct vfsmount *
+nfsd4_interssc_connect(struct nl4_servers *nss, struct svc_rqst *rqstp)
+{
+	struct file_system_type *type;
+	struct vfsmount *ss_mnt;
+	struct nfs42_netaddr *naddr;
+	struct sockaddr_storage tmp_addr;
+	size_t tmp_addrlen, match_netid_len = 3;
+	char *startsep = "", *endsep = "", *match_netid = "tcp";
+	char *ipaddr, *ipaddr2, *raw_data;
+	int len, raw_len, status = -EINVAL;
+
+	/* Currently support for one NL4_NETADDR source server */
+	if (nss->nl_svr->nl4_type != NL4_NETADDR) {
+		WARN(nss->nl_svr->nl4_type != NL4_NETADDR,
+			"nfsd4_copy src server not NL4_NETADDR\n");
+		goto out_err;
+	}
+
+	naddr = &nss->nl_svr->u.nl4_addr;
+
+	tmp_addrlen = rpc_uaddr2sockaddr(SVC_NET(rqstp), naddr->na_uaddr,
+					naddr->na_uaddr_len,
+					(struct sockaddr *)&tmp_addr,
+					sizeof(tmp_addr));
+	if (tmp_addrlen == 0)
+		goto out_err;
+
+	if (tmp_addr.ss_family == AF_INET6) {
+		startsep = "[";
+		endsep = "]";
+		match_netid = "tcp6";
+		match_netid_len = 4;
+	}
+
+	if (naddr->na_netid_len != match_netid_len ||
+	    strncmp(naddr->na_netid, match_netid, naddr->na_netid_len))
+		goto out_err;
+
+	/* Construct the raw data for the vfs_kern_mount call */
+	len = RPC_MAX_ADDRBUFLEN + 1;
+	ipaddr = kzalloc(len, GFP_KERNEL);
+	if (!ipaddr)
+		goto out_err;
+
+	rpc_ntop((struct sockaddr *)&tmp_addr, ipaddr, len);
+
+	/* 2 for ipv6 endsep and startsep. 3 for ":/" and trailing '/0'*/
+	ipaddr2 = kzalloc(len + 5, GFP_KERNEL);
+	if (!ipaddr2)
+		goto out_free_ipaddr;
+
+	rpc_ntop((struct sockaddr *)&rqstp->rq_daddr, ipaddr2, len + 5);
+
+	raw_len = strlen(NFSD42_INTERSSC_RAWDATA) + strlen(ipaddr) +
+			strlen(ipaddr2);
+	raw_data = kzalloc(raw_len, GFP_KERNEL);
+	if (!raw_data)
+		goto out_free_ipaddr2;
+
+	snprintf(raw_data, raw_len, NFSD42_INTERSSC_RAWDATA, ipaddr,
+		 ipaddr2);
+
+	status = -ENODEV;
+	type = get_fs_type("nfs");
+	if (!type)
+		goto out_free_rawdata;
+
+	/* Set the server:<export> for the vfs_kerne_mount call */
+	memset(ipaddr2, 0, len + 5);
+	snprintf(ipaddr2, len + 5, "%s%s%s:/", startsep, ipaddr, endsep);
+
+	dprintk("%s  Raw mount data:  %s server:export %s\n", __func__,
+		raw_data, ipaddr2);
+
+	/* Use an 'internal' mount: MS_KERNMOUNT -> MNT_INTERNAL */
+	ss_mnt = vfs_kern_mount(type, MS_KERNMOUNT, ipaddr2, raw_data);
+	if (IS_ERR(ss_mnt)) {
+		status = PTR_ERR(ss_mnt);
+		goto out_free_rawdata;
+	}
+
+	kfree(raw_data);
+	kfree(ipaddr2);
+	kfree(ipaddr);
+
+	return ss_mnt;
+
+out_free_rawdata:
+	kfree(raw_data);
+out_free_ipaddr2:
+	kfree(ipaddr2);
+out_free_ipaddr:
+	kfree(ipaddr);
+out_err:
+	dprintk("--> %s ERROR %d\n", __func__, status);
+	return ERR_PTR(status);
+}
+
+static void
+nfsd4_interssc_disconnect(struct vfsmount *ss_mnt)
+{
+	mntput(ss_mnt);
+	nfs_sb_deactive(ss_mnt->mnt_sb);
+}
+
+/**
+ * nfsd4_setup_inter_ssc
+ *
+ * Verify COPY destination stateid.
+ * Connect to the source server with NFSv4.1.
+ * Create the source struct file for nfsd_copy_range.
+ * Called with COPY cstate:
+ *    SAVED_FH: source filehandle
+ *    CURRENT_FH: destination filehandle
+ *
+ * Returns errno (not nfserrxxx)
+ */
+static struct vfsmount *
+nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
+			struct nfsd4_compound_state *cstate,
+			struct nfsd4_copy *copy, struct file **src,
+			struct file **dst)
+{
+	struct svc_fh *s_fh = NULL;
+	stateid_t *s_stid = &copy->cp_src_stateid;
+	struct nfs_fh fh;
+	nfs4_stateid stateid;
+	struct file *filp;
+	struct vfsmount *ss_mnt;
+	__be32 status;
+
+	/* Verify the destination stateid and set dst struct file*/
+	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
+					&copy->cp_dst_stateid,
+					WR_STATE, dst, NULL);
+	if (status) {
+		ss_mnt = ERR_PTR(be32_to_cpu(status));
+		goto out;
+	}
+
+	/* Inter copy source fh is always stale */
+	CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
+
+	ss_mnt = nfsd4_interssc_connect(&copy->cp_src, rqstp);
+	if (IS_ERR(ss_mnt))
+		goto out;
+
+	s_fh = &cstate->save_fh;
+
+	fh.size = s_fh->fh_handle.fh_size;
+	memcpy(fh.data, &s_fh->fh_handle.fh_base, fh.size);
+	stateid.seqid = s_stid->si_generation;
+	memcpy(stateid.other, (void *)&s_stid->si_opaque,
+		sizeof(stateid_opaque_t));
+
+	filp =  nfs42_ssc_open(ss_mnt, &fh, &stateid);
+	if (IS_ERR(filp)) {
+		nfsd4_interssc_disconnect(ss_mnt);
+		return ERR_CAST(filp);
+	}
+	*src = filp;
+out:
+	return ss_mnt;
+}
+
+static void
+nfsd4_cleanup_inter_ssc(struct vfsmount *ss_mnt, struct file *src,
+			struct file *dst)
+{
+	nfs42_ssc_close(src);
+	fput(src);
+	fput(dst);
+
+	nfsd4_interssc_disconnect(ss_mnt);
+
+}
+
+#else /* CONFIG_NFSD_V4_2_INTER_SSC */
+
+static struct vfsmount *
+nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
+			struct nfsd4_compound_state *cstate,
+			struct nfsd4_copy *copy, struct file **src,
+			struct file **dst)
+{
+	return ERR_PTR(-EINVAL);
+}
+
+static void
+nfsd4_cleanup_inter_ssc(struct vfsmount *ss_mnt, struct file *src,
+			struct file *dst)
+{
+}
+
+#endif /* CONFIG_NFSD_V4_2_INTER_SSC */
+
 static __be32
-nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
-		struct nfsd4_copy *copy)
+nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
+		      struct nfsd4_compound_state *cstate,
+		      struct nfsd4_copy *copy, struct file **src,
+		      struct file **dst)
 {
-	struct file *src, *dst;
 	__be32 status;
-	ssize_t bytes;
 
-	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid, &src,
-				   &copy->cp_dst_stateid, &dst);
+	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid, src,
+				   &copy->cp_dst_stateid, dst);
 	if (status)
 		goto out;
 
@@ -1089,7 +1300,37 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 	if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
 		CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
 		cstate->status = nfserr_copy_stalefh;
-		goto out_put;
+	}
+out:
+	return status;
+}
+
+static void
+nfsd4_cleanup_intra_ssc(struct file *src, struct file *dst)
+{
+	fput(src);
+	fput(dst);
+}
+
+static __be32
+nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
+	struct nfsd4_copy *copy)
+{
+	struct vfsmount *ss_mnt = NULL;
+	struct file *src, *dst;
+	__be32 status;
+	ssize_t bytes;
+
+	if (copy->cp_src.nl_nsvr > 0) { /* Inter server SSC */
+		ss_mnt = nfsd4_setup_inter_ssc(rqstp, cstate, copy, &src, &dst);
+		if (IS_ERR(ss_mnt)) {
+			status = nfserrno(PTR_ERR(ss_mnt));
+			goto out;
+		}
+	} else {
+		status = nfsd4_setup_intra_ssc(rqstp, cstate, copy, &src, &dst);
+		if (status)
+			goto out;
 	}
 
 	bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
@@ -1106,9 +1347,10 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 		status = nfs_ok;
 	}
 
-out_put:
-	fput(src);
-	fput(dst);
+	if (copy->cp_src.nl_nsvr > 0)   /* Inter server SSC */
+		nfsd4_cleanup_inter_ssc(ss_mnt, src, dst);
+	else
+		nfsd4_cleanup_intra_ssc(src, dst);
 out:
 	return status;
 }
diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
index 5aa36ac..843443b 100644
--- a/include/linux/nfs4.h
+++ b/include/linux/nfs4.h
@@ -16,6 +16,7 @@
 #include <linux/uidgid.h>
 #include <uapi/linux/nfs4.h>
 #include <linux/sunrpc/msg_prot.h>
+#include <linux/nfs.h>
 
 enum nfs4_acl_whotype {
 	NFS4_ACL_WHO_NAMED = 0,
-- 
1.8.3.1

[RFC v3 29/42] NFSD return nfs4_stid in nfs4_preprocess_stateid_op

[Olga Kornievskaia]

Needed for nfsd4_copy_notify to add nfs4_cp_state to the nfs4_stid.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfsd/nfs4proc.c  | 22 +++++++++++++---------
 fs/nfsd/nfs4state.c |  8 ++++++--
 fs/nfsd/state.h     |  3 ++-
 3 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index b1095e9..c6edf01 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -780,7 +780,8 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)
 	/* check stateid */
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					&read->rd_stateid, RD_STATE,
-					&read->rd_filp, &read->rd_tmp_file);
+					&read->rd_filp, &read->rd_tmp_file,
+					NULL);
 	if (status) {
 		dprintk("NFSD: nfsd4_read: couldn't process stateid!\n");
 		goto out;
@@ -926,7 +927,7 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)
 	if (setattr->sa_iattr.ia_valid & ATTR_SIZE) {
 		status = nfs4_preprocess_stateid_op(rqstp, cstate,
 				&cstate->current_fh, &setattr->sa_stateid,
-				WR_STATE, NULL, NULL);
+				WR_STATE, NULL, NULL, NULL);
 		if (status) {
 			dprintk("NFSD: nfsd4_setattr: couldn't process stateid!\n");
 			return status;
@@ -991,7 +992,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 		return nfserr_inval;
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
-						stateid, WR_STATE, &filp, NULL);
+					stateid, WR_STATE, &filp, NULL, NULL);
 	if (status) {
 		dprintk("NFSD: nfsd4_write: couldn't process stateid!\n");
 		return status;
@@ -1022,14 +1023,16 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 	__be32 status;
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->save_fh,
-					    src_stateid, RD_STATE, src, NULL);
+					    src_stateid, RD_STATE, src, NULL,
+					    NULL);
 	if (status) {
 		dprintk("NFSD: %s: couldn't process src stateid!\n", __func__);
 		goto out;
 	}
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
-					    dst_stateid, WR_STATE, dst, NULL);
+					    dst_stateid, WR_STATE, dst, NULL,
+					    NULL);
 	if (status) {
 		dprintk("NFSD: %s: couldn't process dst stateid!\n", __func__);
 		goto out_put_src;
@@ -1221,7 +1224,7 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	/* Verify the destination stateid and set dst struct file*/
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					&copy->cp_dst_stateid,
-					WR_STATE, dst, NULL);
+					WR_STATE, dst, NULL, NULL);
 	if (status) {
 		ss_mnt = ERR_PTR(be32_to_cpu(status));
 		goto out;
@@ -1405,10 +1408,11 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 {
 	__be32 status;
 	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
+	struct nfs4_stid *stid;
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					&cn->cpn_src_stateid, RD_STATE, NULL,
-					NULL);
+					NULL, &stid);
 	if (status)
 		return status;
 
@@ -1440,7 +1444,7 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					    &fallocate->falloc_stateid,
-					    WR_STATE, &file, NULL);
+					    WR_STATE, &file, NULL, NULL);
 	if (status != nfs_ok) {
 		dprintk("NFSD: nfsd4_fallocate: couldn't process stateid!\n");
 		return status;
@@ -1479,7 +1483,7 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					    &seek->seek_stateid,
-					    RD_STATE, &file, NULL);
+					    RD_STATE, &file, NULL, NULL);
 	if (status) {
 		dprintk("NFSD: nfsd4_seek: couldn't process stateid!\n");
 		return status;
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index e7e1b9f..8425735 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -4932,7 +4932,8 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)
 __be32
 nfs4_preprocess_stateid_op(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *cstate, struct svc_fh *fhp,
-		stateid_t *stateid, int flags, struct file **filpp, bool *tmp_file)
+		stateid_t *stateid, int flags, struct file **filpp,
+		bool *tmp_file, struct nfs4_stid **cstid)
 {
 	struct inode *ino = d_inode(fhp->fh_dentry);
 	struct net *net = SVC_NET(rqstp);
@@ -4983,8 +4984,11 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)
 	if (!status && filpp)
 		status = nfs4_check_file(rqstp, fhp, s, filpp, tmp_file, flags);
 out:
-	if (s)
+	if (s) {
+		if (!status && cstid)
+			*cstid = s;
 		nfs4_put_stid(s);
+	}
 	return status;
 }
 
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 005c911..f5ab89a 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -599,7 +599,8 @@ struct nfsd4_blocked_lock {
 
 extern __be32 nfs4_preprocess_stateid_op(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *cstate, struct svc_fh *fhp,
-		stateid_t *stateid, int flags, struct file **filp, bool *tmp_file);
+		stateid_t *stateid, int flags, struct file **filp,
+		bool *tmp_file, struct nfs4_stid **cstid);
 __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 		     stateid_t *stateid, unsigned char typemask,
 		     struct nfs4_stid **s, struct nfsd_net *nn);
-- 
1.8.3.1

[RFC v3 30/42] NFSD Unique stateid_t for inter server to server COPY authentication

[Olga Kornievskaia]

A per COPY unique stateid_t is required to authenticate the READ from
the destination server (acting as a client) on the source server.

Multiple concurrent inter server to server copies of the same source file
are also supported.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfsd/netns.h     |   8 ++++
 fs/nfsd/nfs4proc.c  |  25 ++++++++----
 fs/nfsd/nfs4state.c | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/nfs4xdr.c   |   2 +-
 fs/nfsd/nfsctl.c    |   2 +
 fs/nfsd/state.h     |  18 +++++++++
 fs/nfsd/xdr4.h      |   2 +-
 7 files changed, 156 insertions(+), 9 deletions(-)

diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
index 3714231..2c88a95 100644
--- a/fs/nfsd/netns.h
+++ b/fs/nfsd/netns.h
@@ -119,6 +119,14 @@ struct nfsd_net {
 	u32 clverifier_counter;
 
 	struct svc_serv *nfsd_serv;
+
+	/*
+	 * clientid and stateid data for construction of net unique COPY
+	 * stateids.
+	 */
+	u32		s2s_cp_cl_id;
+	struct idr	s2s_cp_stateids;
+	spinlock_t	s2s_cp_lock;
 };
 
 /* Simple check to find out if a given net was properly initialized */
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index c6edf01..3d8ed3f 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1402,6 +1402,13 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	return 0;
 }
 
+/*
+ * Use a unique stateid_t as the cnr_stateid so that the source server
+ * can authenticate the inter server to server copy READ from the
+ * destination server.
+ *
+ * Set the cnr_leasetime to the nfsd4_lease.
+ */
 static __be32
 nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		  struct nfsd4_copy_notify *cn)
@@ -1409,6 +1416,7 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	__be32 status;
 	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
 	struct nfs4_stid *stid;
+	struct nfs4_cp_state *cps;
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					&cn->cpn_src_stateid, RD_STATE, NULL,
@@ -1419,12 +1427,11 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	cn->cpn_sec = nn->nfsd4_lease;
 	cn->cpn_nsec = 0;
 
-
-	/** XXX Save cpn_src_statid, cpn_src, and any other returned source
-	 * server addresses on which the source server is williing to accept
-	 * connections from the destination e.g. what is returned in cpn_src,
-	 * to verify READ from dest server.
-	 */
+	status = nfserrno(-ENOMEM);
+	cps = nfs4_alloc_init_cp_state(nn, nn->nfsd4_lease, stid);
+	if (!cps)
+		return status;
+	memcpy(&cn->cpn_cnr_stateid, &cps->cp_stateid, sizeof(stateid_t));
 
 	/**
 	 * For now, only return one server address in cpn_src, the
@@ -1432,7 +1439,11 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	 */
 	cn->cpn_src.nl_nsvr = 1;
 
-	return nfsd4_set_src_nl4_netaddr(rqstp, &cn->cpn_src);
+	status = nfsd4_set_src_nl4_netaddr(rqstp, &cn->cpn_src);
+	if (status != 0)
+		nfs4_free_cp_state(cps);
+
+	return status;
 }
 
 static __be32
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 8425735..b019fc8 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -658,6 +658,7 @@ struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *sla
 	/* Will be incremented before return to client: */
 	atomic_set(&stid->sc_count, 1);
 	spin_lock_init(&stid->sc_lock);
+	INIT_LIST_HEAD(&stid->sc_cp_list);
 
 	/*
 	 * It shouldn't be a problem to reuse an opaque stateid value.
@@ -674,6 +675,106 @@ struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *sla
 	return NULL;
 }
 
+/*
+ * Create a unique stateid_t to represent each COPY - returned by
+ * COPY_NOTIFY cnr_stateid and used in the copy READ from the
+ * destination server. Hang the copy stateids off the OPEN/LOCK/DELEG
+ * stateid from the client open of the source file. Bookeep other
+ * copy state such as the cnr_leasetime.
+ */
+struct nfs4_cp_state *nfs4_alloc_init_cp_state(struct nfsd_net *nn,
+						u64 cpn_sec,
+						struct nfs4_stid *p_stid)
+{
+	struct nfs4_cp_state *cps;
+	int new_id;
+
+	cps = kzalloc(sizeof(struct nfs4_cp_state), GFP_KERNEL);
+	if (!cps)
+		return NULL;
+	idr_preload(GFP_KERNEL);
+	spin_lock(&nn->s2s_cp_lock);
+	new_id = idr_alloc_cyclic(&nn->s2s_cp_stateids, cps, 0, 0, GFP_NOWAIT);
+	spin_unlock(&nn->s2s_cp_lock);
+	idr_preload_end();
+	if (new_id < 0)
+		goto out_free;
+	cps->cp_stateid.si_opaque.so_id = new_id;
+	cps->cp_stateid.si_opaque.so_clid.cl_boot = nn->boot_time;
+	cps->cp_stateid.si_opaque.so_clid.cl_id = nn->s2s_cp_cl_id;
+	cps->cp_p_stid = p_stid;
+	cps->cp_active = false;
+	INIT_LIST_HEAD(&cps->cp_list);
+	cps->cp_timeout = jiffies + (cpn_sec * HZ);
+	list_add(&cps->cp_list, &p_stid->sc_cp_list);
+
+	return cps;
+out_free:
+	kfree(cps);
+	return NULL;
+}
+
+void nfs4_free_cp_state(struct nfs4_cp_state *cps)
+{
+	struct nfsd_net *nn;
+
+	dprintk("--> %s freeing cp_state so_id %d\n", __func__,
+		cps->cp_stateid.si_opaque.so_id);
+
+	nn = net_generic(cps->cp_p_stid->sc_client->net, nfsd_net_id);
+	spin_lock(&nn->s2s_cp_lock);
+	idr_remove(&nn->s2s_cp_stateids, cps->cp_stateid.si_opaque.so_id);
+	spin_unlock(&nn->s2s_cp_lock);
+
+	kfree(cps);
+}
+
+static void nfs4_free_cp_statelist(struct nfs4_stid *stid)
+{
+	struct nfs4_cp_state *cps;
+
+	might_sleep();
+
+	while (!list_empty(&stid->sc_cp_list)) {
+		cps = list_first_entry(&stid->sc_cp_list, struct nfs4_cp_state,
+				       cp_list);
+		list_del(&cps->cp_list);
+		nfs4_free_cp_state(cps);
+	}
+}
+
+/*
+ * A READ from an inter server to server COPY will have a
+ * copy stateid. Return the parent nfs4_stid.
+ */
+static __be32 find_cp_state_parent(struct nfsd_net *nn, stateid_t *st,
+				   struct nfs4_stid **stid)
+{
+	struct nfs4_cp_state *cps = NULL;
+
+	if (st->si_opaque.so_clid.cl_id != nn->s2s_cp_cl_id)
+		return nfserr_bad_stateid;
+	spin_lock(&nn->s2s_cp_lock);
+	cps = idr_find(&nn->s2s_cp_stateids, st->si_opaque.so_id);
+	spin_unlock(&nn->s2s_cp_lock);
+	if (!cps) {
+		pr_info("NFSD: find_cp_state cl_id %d so_id %d NOT FOUND\n",
+			st->si_opaque.so_clid.cl_id, st->si_opaque.so_id);
+		return nfserr_bad_stateid;
+	}
+
+	/* Did the inter server to server copy start in time? */
+	if (cps->cp_active == false && !time_after(cps->cp_timeout, jiffies))
+		return nfserr_partner_no_auth;
+	else
+		cps->cp_active = true;
+
+	*stid = cps->cp_p_stid;
+	atomic_inc(&cps->cp_p_stid->sc_count);
+
+	return nfs_ok;
+}
+
 static struct nfs4_ol_stateid * nfs4_alloc_open_stateid(struct nfs4_client *clp)
 {
 	struct nfs4_stid *stid;
@@ -819,6 +920,9 @@ static void block_delegations(struct knfsd_fh *fh)
 	}
 	idr_remove(&clp->cl_stateids, s->sc_stateid.si_opaque.so_id);
 	spin_unlock(&clp->cl_lock);
+
+	nfs4_free_cp_statelist(s);
+
 	s->sc_free(s);
 	if (fp)
 		put_nfs4_file(fp);
@@ -4957,6 +5061,8 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)
 	status = nfsd4_lookup_stateid(cstate, stateid,
 				NFS4_DELEG_STID|NFS4_OPEN_STID|NFS4_LOCK_STID,
 				&s, nn);
+	if (status == nfserr_bad_stateid)
+		status = find_cp_state_parent(nn, stateid, &s);
 	if (status)
 		return status;
 	status = check_stateid_generation(stateid, &s->sc_stateid,
@@ -6930,6 +7036,8 @@ static int nfs4_state_create_net(struct net *net)
 	INIT_LIST_HEAD(&nn->close_lru);
 	INIT_LIST_HEAD(&nn->del_recall_lru);
 	spin_lock_init(&nn->client_lock);
+	spin_lock_init(&nn->s2s_cp_lock);
+	idr_init(&nn->s2s_cp_stateids);
 
 	spin_lock_init(&nn->blocked_locks_lock);
 	INIT_LIST_HEAD(&nn->blocked_locks_lru);
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 2896a11..c4438ac 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -4458,7 +4458,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	*p++ = cpu_to_be32(cn->cpn_nsec);
 
 	/* cnr_stateid */
-	nfserr = nfsd4_encode_stateid(xdr, &cn->cpn_src_stateid);
+	nfserr = nfsd4_encode_stateid(xdr, &cn->cpn_cnr_stateid);
 	if (nfserr)
 		return nfserr;
 
diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index 6493df6..6ef5790 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -1241,6 +1241,8 @@ static __net_init int nfsd_init_net(struct net *net)
 	nn->nfsd4_grace = 90;
 	nn->clverifier_counter = prandom_u32();
 	nn->clientid_counter = prandom_u32();
+	nn->s2s_cp_cl_id = nn->clientid_counter++;
+	pr_info("%s s2s_cp_cl_id %d\n", __func__, nn->s2s_cp_cl_id);
 	return 0;
 
 out_idmap_error:
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index f5ab89a..c084c57 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -93,6 +93,7 @@ struct nfs4_stid {
 #define NFS4_REVOKED_DELEG_STID 16
 #define NFS4_CLOSED_DELEG_STID 32
 #define NFS4_LAYOUT_STID 64
+	struct list_head	sc_cp_list;
 	unsigned char		sc_type;
 	stateid_t		sc_stateid;
 	spinlock_t		sc_lock;
@@ -102,6 +103,20 @@ struct nfs4_stid {
 };
 
 /*
+ * An inter server to server copy stateid that is unique per nfsd_net.
+ * cp_stateid is returned as the COPY_NOTIFY cnr_stateid and presented
+ * by the destination server to the source server as a COPY authenticator.
+ * Used to lookup the parent COPY_NOTIFY cna_src_stateid nfs4_stid.
+ */
+struct nfs4_cp_state {
+	stateid_t		cp_stateid;
+	struct list_head	cp_list;	/* per parent nfs4_stid */
+	struct nfs4_stid	*cp_p_stid;	/* pointer to parent */
+	bool			cp_active;	/* has the copy started */
+	unsigned long		cp_timeout;	/* copy timeout */
+};
+
+/*
  * Represents a delegation stateid. The nfs4_client holds references to these
  * and they are put when it is being destroyed or when the delegation is
  * returned by the client:
@@ -606,6 +621,9 @@ __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 		     struct nfs4_stid **s, struct nfsd_net *nn);
 struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *slab,
 				  void (*sc_free)(struct nfs4_stid *));
+struct nfs4_cp_state *nfs4_alloc_init_cp_state(struct nfsd_net *nn, u64 cpn_sec,
+					       struct nfs4_stid *p_stid);
+void nfs4_free_cp_state(struct nfs4_cp_state *cps);
 void nfs4_unhash_stid(struct nfs4_stid *s);
 void nfs4_put_stid(struct nfs4_stid *s);
 void nfs4_inc_and_copy_stateid(stateid_t *dst, struct nfs4_stid *stid);
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index aa94295..5b38f0a 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -549,7 +549,7 @@ struct nfsd4_copy_notify {
 	struct nl4_server	cpn_dst;
 
 	/* response */
-	/* Note: cpn_src_stateid is used for cnr_stateid */
+	stateid_t		cpn_cnr_stateid;
 	u64			cpn_sec;
 	u32			cpn_nsec;
 	struct nl4_servers	cpn_src;
-- 
1.8.3.1

[RFC v3 31/42] NFSD CB_OFFLOAD xdr

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4callback.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/state.h        |  1 +
 fs/nfsd/xdr4.h         |  6 ++++
 fs/nfsd/xdr4cb.h       | 10 ++++++
 4 files changed, 112 insertions(+)

diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index 0274db6..d36fc05 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -39,6 +39,7 @@
 #include "state.h"
 #include "netns.h"
 #include "xdr4cb.h"
+#include "xdr4.h"
 
 #define NFSDDBG_FACILITY                NFSDDBG_PROC
 
@@ -105,6 +106,7 @@ enum nfs_cb_opnum4 {
 	OP_CB_WANTS_CANCELLED		= 12,
 	OP_CB_NOTIFY_LOCK		= 13,
 	OP_CB_NOTIFY_DEVICEID		= 14,
+	OP_CB_OFFLOAD			= 15,
 	OP_CB_ILLEGAL			= 10044
 };
 
@@ -677,6 +679,98 @@ static int nfs4_xdr_dec_cb_notify_lock(struct rpc_rqst *rqstp,
 }
 
 /*
+ * struct write_response4 {
+ *	stateid4	wr_callback_id<1>;
+ *	length4		wr_count;
+ *	stable_how4	wr_committed;
+ *	verifier4	wr_writeverf;
+ * };
+ * union offload_info4 switch (nfsstat4 coa_status) {
+ *	case NFS4_OK:
+ *		write_response4	coa_resok4;
+ *	default:
+ *	length4		coa_bytes_copied;
+ * };
+ * struct CB_OFFLOAD4args {
+ *	nfs_fh4		coa_fh;
+ *	stateid4	coa_stateid;
+ *	offload_info4	coa_offload_info;
+ * };
+ */
+static void encode_offload_info4(struct xdr_stream *xdr,
+				 __be32 nfserr,
+				 const struct nfsd4_copy *cp)
+{
+	__be32 *p;
+
+	p = xdr_reserve_space(xdr, 4);
+	*p++ = nfserr;
+	if (!nfserr) {
+		p = xdr_reserve_space(xdr, 4 + 8 + 4 + NFS4_VERIFIER_SIZE);
+		p = xdr_encode_empty_array(p);
+		p = xdr_encode_hyper(p, cp->cp_res.wr_bytes_written);
+		*p++ = cpu_to_be32(cp->cp_res.wr_stable_how);
+		p = xdr_encode_opaque_fixed(p, cp->cp_res.wr_verifier.data,
+					    NFS4_VERIFIER_SIZE);
+	} else {
+		p = xdr_reserve_space(xdr, 8);
+		p = xdr_encode_hyper(p, cp->cp_res.wr_bytes_written);
+	}
+}
+
+static void encode_cb_offload4args(struct xdr_stream *xdr,
+				   __be32 nfserr,
+				   const struct knfsd_fh *fh,
+				   const struct nfsd4_copy *cp,
+				   struct nfs4_cb_compound_hdr *hdr)
+{
+	__be32 *p;
+
+	p = xdr_reserve_space(xdr, 4);
+	*p++ = cpu_to_be32(OP_CB_OFFLOAD);
+	encode_nfs_fh4(xdr, fh);
+	encode_stateid4(xdr, &cp->cp_res.cb_stateid);
+	encode_offload_info4(xdr, nfserr, cp);
+
+	hdr->nops++;
+}
+
+static void nfs4_xdr_enc_cb_offload(struct rpc_rqst *req,
+				    struct xdr_stream *xdr,
+				    const struct nfsd4_callback *cb)
+{
+	const struct nfsd4_copy *cp =
+		container_of(cb, struct nfsd4_copy, cp_cb);
+	struct nfs4_cb_compound_hdr hdr = {
+		.ident = 0,
+		.minorversion = cb->cb_clp->cl_minorversion,
+	};
+
+	encode_cb_compound4args(xdr, &hdr);
+	encode_cb_sequence4args(xdr, cb, &hdr);
+	encode_cb_offload4args(xdr, cp->nfserr, &cp->fh, cp, &hdr);
+	encode_cb_nops(&hdr);
+}
+
+static int nfs4_xdr_dec_cb_offload(struct rpc_rqst *rqstp,
+				   struct xdr_stream *xdr,
+				   struct nfsd4_callback *cb)
+{
+	struct nfs4_cb_compound_hdr hdr;
+	int status;
+
+	status = decode_cb_compound4res(xdr, &hdr);
+	if (unlikely(status))
+		return status;
+
+	if (cb) {
+		status = decode_cb_sequence4res(xdr, cb);
+		if (unlikely(status || cb->cb_seq_status))
+			return status;
+	}
+	return decode_cb_op_status(xdr, OP_CB_OFFLOAD, &cb->cb_status);
+}
+/*
  * RPC procedure tables
  */
 #define PROC(proc, call, argtype, restype)				\
@@ -697,6 +791,7 @@ static int nfs4_xdr_dec_cb_notify_lock(struct rpc_rqst *rqstp,
 	PROC(CB_LAYOUT,	COMPOUND,	cb_layout,	cb_layout),
 #endif
 	PROC(CB_NOTIFY_LOCK,	COMPOUND,	cb_notify_lock,	cb_notify_lock),
+	PROC(CB_OFFLOAD,	COMPOUND,	cb_offload,	cb_offload),
 };
 
 static struct rpc_version nfs_cb_version4 = {
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index c084c57..3b0da32 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -585,6 +585,7 @@ enum nfsd4_cb_op {
 	NFSPROC4_CLNT_CB_NULL = 0,
 	NFSPROC4_CLNT_CB_RECALL,
 	NFSPROC4_CLNT_CB_LAYOUT,
+	NFSPROC4_CLNT_CB_OFFLOAD,
 	NFSPROC4_CLNT_CB_SEQUENCE,
 	NFSPROC4_CLNT_CB_NOTIFY_LOCK,
 };
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index 5b38f0a..e6fa914 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -510,6 +510,7 @@ struct nfsd42_write_res {
 	u64			wr_bytes_written;
 	u32			wr_stable_how;
 	nfs4_verifier		wr_verifier;
+	stateid_t		cb_stateid;
 };
 
 /*  support 1 source server for now */
@@ -530,6 +531,11 @@ struct nfsd4_copy {
 
 	/* response */
 	struct nfsd42_write_res	cp_res;
+
+	/* for cb_offload */
+	struct nfsd4_callback	cp_cb;
+	__be32			nfserr;
+	struct knfsd_fh		fh;
 };
 
 struct nfsd4_seek {
diff --git a/fs/nfsd/xdr4cb.h b/fs/nfsd/xdr4cb.h
index 49b719d..7e39913 100644
--- a/fs/nfsd/xdr4cb.h
+++ b/fs/nfsd/xdr4cb.h
@@ -37,3 +37,13 @@
 #define NFS4_dec_cb_notify_lock_sz	(cb_compound_dec_hdr_sz  +      \
 					cb_sequence_dec_sz +            \
 					op_dec_sz)
+#define enc_cb_offload_info_sz		(1 + 1 + 2 + 1 +		\
+					XDR_QUADLEN(NFS4_VERIFIER_SIZE))
+#define NFS4_enc_cb_offload_sz		(cb_compound_enc_hdr_sz +       \
+					cb_sequence_enc_sz +            \
+					enc_nfs4_fh_sz +		\
+					enc_stateid_sz +		\
+					enc_cb_offload_info_sz)
+#define NFS4_dec_cb_offload_sz		(cb_compound_dec_hdr_sz  +      \
+					cb_sequence_dec_sz +            \
+					op_dec_sz)
-- 
1.8.3.1

[RFC v3 32/42] NFSD OFFLOAD_STATUS xdr

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c | 20 ++++++++++++++++++++
 fs/nfsd/nfs4xdr.c  | 30 ++++++++++++++++++++++++++++--
 fs/nfsd/xdr4.h     | 10 ++++++++++
 3 files changed, 58 insertions(+), 2 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 3d8ed3f..6d20ee1 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1468,6 +1468,13 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	fput(file);
 	return status;
 }
+static __be32
+nfsd4_offload_status(struct svc_rqst *rqstp,
+		     struct nfsd4_compound_state *cstate,
+		     struct nfsd4_offload_status *os)
+{
+	return nfserr_notsupp;
+}
 
 static __be32
 nfsd4_allocate(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
@@ -2436,6 +2443,14 @@ static inline u32 nfsd4_copy_notify_rsize(struct svc_rqst *rqstp,
 		* sizeof(__be32);
 }
 
+static inline u32 nfsd4_offload_status_rsize(struct svc_rqst *rqstp,
+					     struct nfsd4_op *op)
+{
+	return (op_encode_hdr_size +
+		2 /* osr_count */ +
+		1 /* osr_complete<1> optional 0 for now */) * sizeof(__be32);
+}
+
 #ifdef CONFIG_NFSD_PNFS
 static inline u32 nfsd4_getdeviceinfo_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
 {
@@ -2846,6 +2861,11 @@ static inline u32 nfsd4_seek_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
 		.op_name = "OP_COPY_NOTIFY",
 		.op_rsize_bop = (nfsd4op_rsize)nfsd4_copy_notify_rsize,
 	},
+	[OP_OFFLOAD_STATUS] = {
+		.op_func = (nfsd4op_func)nfsd4_offload_status,
+		.op_name = "OP_OFFLOAD_STATUS",
+		.op_rsize_bop = (nfsd4op_rsize)nfsd4_offload_status_rsize,
+	},
 };
 
 /**
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index c4438ac..f032178 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1836,6 +1836,13 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
 }
 
 static __be32
+nfsd4_decode_offload_status(struct nfsd4_compoundargs *argp,
+			    struct nfsd4_offload_status *os)
+{
+	return nfsd4_decode_stateid(argp, &os->stateid);
+}
+
+static __be32
 nfsd4_decode_seek(struct nfsd4_compoundargs *argp, struct nfsd4_seek *seek)
 {
 	DECODE_HEAD;
@@ -1942,7 +1949,7 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
 	[OP_LAYOUTERROR]	= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_LAYOUTSTATS]	= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_OFFLOAD_CANCEL]	= (nfsd4_dec)nfsd4_decode_notsupp,
-	[OP_OFFLOAD_STATUS]	= (nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_OFFLOAD_STATUS]	= (nfsd4_dec)nfsd4_decode_offload_status,
 	[OP_READ_PLUS]		= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_SEEK]		= (nfsd4_dec)nfsd4_decode_seek,
 	[OP_WRITE_SAME]		= (nfsd4_dec)nfsd4_decode_notsupp,
@@ -4483,6 +4490,25 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 }
 
 static __be32
+nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr,
+			    struct nfsd4_offload_status *os)
+{
+	struct xdr_stream *xdr = &resp->xdr;
+	__be32 *p;
+
+	if (nfserr)
+		return nfserr;
+
+	p = xdr_reserve_space(xdr, 8 + 4);
+	if (!p)
+		return nfserr_resource;
+	p = xdr_encode_hyper(p, os->count);
+	*p++ = cpu_to_be32(0);
+
+	return nfserr;
+}
+
+static __be32
 nfsd4_encode_seek(struct nfsd4_compoundres *resp, __be32 nfserr,
 		  struct nfsd4_seek *seek)
 {
@@ -4588,7 +4614,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	[OP_LAYOUTERROR]	= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_LAYOUTSTATS]	= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_OFFLOAD_CANCEL]	= (nfsd4_enc)nfsd4_encode_noop,
-	[OP_OFFLOAD_STATUS]	= (nfsd4_enc)nfsd4_encode_noop,
+	[OP_OFFLOAD_STATUS]	= (nfsd4_enc)nfsd4_encode_offload_status,
 	[OP_READ_PLUS]		= (nfsd4_enc)nfsd4_encode_noop,
 	[OP_SEEK]		= (nfsd4_enc)nfsd4_encode_seek,
 	[OP_WRITE_SAME]		= (nfsd4_enc)nfsd4_encode_noop,
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index e6fa914..c3e6907 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -561,6 +561,15 @@ struct nfsd4_copy_notify {
 	struct nl4_servers	cpn_src;
 };
 
+struct nfsd4_offload_status {
+	/* request */
+	stateid_t	stateid;
+
+	/* response */
+	u64		count;
+	u32		status;
+};
+
 struct nfsd4_op {
 	int					opnum;
 	__be32					status;
@@ -617,6 +626,7 @@ struct nfsd4_op {
 		struct nfsd4_clone		clone;
 		struct nfsd4_copy		copy;
 		struct nfsd4_copy_notify	copy_notify;
+		struct nfsd4_offload_status	offload_status;
 		struct nfsd4_seek		seek;
 	} u;
 	struct nfs4_replay *			replay;
-- 
1.8.3.1

[RFC v3 33/42] NFSD OFFLOAD_CANCEL xdr

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c | 13 +++++++++++++
 fs/nfsd/nfs4xdr.c  |  2 +-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 6d20ee1..5dd8278 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1447,6 +1447,14 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 }
 
 static __be32
+nfsd4_offload_cancel(struct svc_rqst *rqstp,
+		     struct nfsd4_compound_state *cstate,
+		     struct nfsd4_offload_status *os)
+{
+	return 0;
+}
+
+static __be32
 nfsd4_fallocate(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		struct nfsd4_fallocate *fallocate, int flags)
 {
@@ -2866,6 +2874,11 @@ static inline u32 nfsd4_seek_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
 		.op_name = "OP_OFFLOAD_STATUS",
 		.op_rsize_bop = (nfsd4op_rsize)nfsd4_offload_status_rsize,
 	},
+	[OP_OFFLOAD_CANCEL] = {
+		.op_func = (nfsd4op_func)nfsd4_offload_cancel,
+		.op_name = "OP_OFFLOAD_CANCEL",
+		.op_rsize_bop = (nfsd4op_rsize)nfsd4_only_status_rsize,
+	},
 };
 
 /**
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index f032178..745ad17 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1948,7 +1948,7 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
 	[OP_IO_ADVISE]		= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_LAYOUTERROR]	= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_LAYOUTSTATS]	= (nfsd4_dec)nfsd4_decode_notsupp,
-	[OP_OFFLOAD_CANCEL]	= (nfsd4_dec)nfsd4_decode_notsupp,
+	[OP_OFFLOAD_CANCEL]	= (nfsd4_dec)nfsd4_decode_offload_status,
 	[OP_OFFLOAD_STATUS]	= (nfsd4_dec)nfsd4_decode_offload_status,
 	[OP_READ_PLUS]		= (nfsd4_dec)nfsd4_decode_notsupp,
 	[OP_SEEK]		= (nfsd4_dec)nfsd4_decode_seek,
-- 
1.8.3.1

[RFC v3 34/42] NFSD xdr callback stateid in async COPY reply

[Olga Kornievskaia]

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4xdr.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 745ad17..3211838 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -4361,15 +4361,27 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 #endif /* CONFIG_NFSD_PNFS */
 
 static __be32
-nfsd42_encode_write_res(struct nfsd4_compoundres *resp, struct nfsd42_write_res *write)
+nfsd42_encode_write_res(struct nfsd4_compoundres *resp,
+		struct nfsd42_write_res *write, bool sync)
 {
 	__be32 *p;
+	p = xdr_reserve_space(&resp->xdr, 4);
+	if (!p)
+		return nfserr_resource;
 
-	p = xdr_reserve_space(&resp->xdr, 4 + 8 + 4 + NFS4_VERIFIER_SIZE);
+	if (sync)
+		*p++ = cpu_to_be32(0);
+	else {
+		__be32 nfserr;
+		*p++ = cpu_to_be32(1);
+		nfserr = nfsd4_encode_stateid(&resp->xdr, &write->cb_stateid);
+		if (nfserr)
+			return nfserr;
+	}
+	p = xdr_reserve_space(&resp->xdr, 8 + 4 + NFS4_VERIFIER_SIZE);
 	if (!p)
 		return nfserr_resource;
 
-	*p++ = cpu_to_be32(0);
 	p = xdr_encode_hyper(p, write->wr_bytes_written);
 	*p++ = cpu_to_be32(write->wr_stable_how);
 	p = xdr_encode_opaque_fixed(p, write->wr_verifier.data,
@@ -4430,7 +4442,8 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	__be32 *p;
 
 	if (!nfserr) {
-		nfserr = nfsd42_encode_write_res(resp, &copy->cp_res);
+		nfserr = nfsd42_encode_write_res(resp, &copy->cp_res,
+				copy->cp_synchronous);
 		if (nfserr)
 			return nfserr;
 
-- 
1.8.3.1

[RFC v3 35/42] NFSD first draft of async copy

[Olga Kornievskaia]

Upong receiving copy request for the inter copy, nfsd will
establish the mount to the source server.

Asynchronous copies are handled by a single threaded workqueue.
If we get asynchronous request, make sure to copy the needed
arguments/state from the stack before starting the copy. Then
queue work and reply back to the client indicating copy is
asynchronous.

nfsd_copy_file_range() will copy in 4MBchunk so do a loop over
the total number of bytes need to copy. In case a failure
happens in the middle, we can return an error as well as how
much we copied so far. Once done creating a workitem for the
callback workqueue and send CB_OFFLOAD with the results.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c  | 224 +++++++++++++++++++++++++++++++++++++++++-----------
 fs/nfsd/nfs4state.c |   9 ++-
 fs/nfsd/state.h     |   2 +
 fs/nfsd/xdr4.h      |  11 +++
 4 files changed, 197 insertions(+), 49 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 5dd8278..a977142 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -77,6 +77,21 @@
 { }
 #endif
 
+static struct workqueue_struct *copy_wq;
+
+int nfsd4_create_copy_queue(void)
+{
+	copy_wq = create_singlethread_workqueue("nfsd4_copy");
+	if (!copy_wq)
+		return -ENOMEM;
+	return 0;
+}
+
+void nfsd4_destroy_copy_queue(void)
+{
+	destroy_workqueue(copy_wq);
+}
+
 #define NFSDDBG_FACILITY		NFSDDBG_PROC
 
 static u32 nfsd_attrmask[] = {
@@ -1210,21 +1225,17 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 static struct vfsmount *
 nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
 			struct nfsd4_compound_state *cstate,
-			struct nfsd4_copy *copy, struct file **src,
-			struct file **dst)
+			struct nfsd4_copy *copy)
 {
 	struct svc_fh *s_fh = NULL;
 	stateid_t *s_stid = &copy->cp_src_stateid;
-	struct nfs_fh fh;
-	nfs4_stateid stateid;
-	struct file *filp;
 	struct vfsmount *ss_mnt;
 	__be32 status;
 
 	/* Verify the destination stateid and set dst struct file*/
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					&copy->cp_dst_stateid,
-					WR_STATE, dst, NULL, NULL);
+					WR_STATE, &copy->fh_dst, NULL, NULL);
 	if (status) {
 		ss_mnt = ERR_PTR(be32_to_cpu(status));
 		goto out;
@@ -1238,19 +1249,11 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 		goto out;
 
 	s_fh = &cstate->save_fh;
-
-	fh.size = s_fh->fh_handle.fh_size;
-	memcpy(fh.data, &s_fh->fh_handle.fh_base, fh.size);
-	stateid.seqid = s_stid->si_generation;
-	memcpy(stateid.other, (void *)&s_stid->si_opaque,
+	copy->c_fh.size = s_fh->fh_handle.fh_size;
+	memcpy(copy->c_fh.data, &s_fh->fh_handle.fh_base, copy->c_fh.size);
+	copy->stateid.seqid = s_stid->si_generation;
+	memcpy(copy->stateid.other, (void *)&s_stid->si_opaque,
 		sizeof(stateid_opaque_t));
-
-	filp =  nfs42_ssc_open(ss_mnt, &fh, &stateid);
-	if (IS_ERR(filp)) {
-		nfsd4_interssc_disconnect(ss_mnt);
-		return ERR_CAST(filp);
-	}
-	*src = filp;
 out:
 	return ss_mnt;
 }
@@ -1272,8 +1275,7 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 static struct vfsmount *
 nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
 			struct nfsd4_compound_state *cstate,
-			struct nfsd4_copy *copy, struct file **src,
-			struct file **dst)
+			struct nfsd4_copy *copy)
 {
 	return ERR_PTR(-EINVAL);
 }
@@ -1289,13 +1291,13 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 static __be32
 nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
 		      struct nfsd4_compound_state *cstate,
-		      struct nfsd4_copy *copy, struct file **src,
-		      struct file **dst)
+		      struct nfsd4_copy *copy)
 {
 	__be32 status;
 
-	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid, src,
-				   &copy->cp_dst_stateid, dst);
+	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid,
+				   &copy->fh_src, &copy->cp_dst_stateid,
+				   &copy->fh_dst);
 	if (status)
 		goto out;
 
@@ -1315,47 +1317,173 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	fput(dst);
 }
 
+static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
+{
+	struct nfsd4_copy *copy = container_of(cb, struct nfsd4_copy, cp_cb);
+
+	kfree(copy);
+}
+
+static int nfsd4_cb_offload_done(struct nfsd4_callback *cb,
+				 struct rpc_task *task)
+{
+	return 1;
+}
+
+static const struct nfsd4_callback_ops nfsd4_cb_offload_ops = {
+	.release = nfsd4_cb_offload_release,
+	.done = nfsd4_cb_offload_done
+};
+
+static int nfsd4_init_copy_res(struct nfsd4_copy *copy, bool sync)
+{
+	memcpy(&copy->cp_res.cb_stateid, &copy->cp_dst_stateid,
+		sizeof(copy->cp_dst_stateid));
+	copy->cp_res.wr_stable_how = NFS_UNSTABLE;
+	copy->cp_consecutive = 1;
+	copy->cp_synchronous = sync;
+	gen_boot_verifier(&copy->cp_res.wr_verifier, copy->net);
+
+	return nfs_ok;
+}
+
+static int _nfsd_copy_file_range(struct nfsd4_copy *copy)
+{
+	ssize_t bytes_copied = 0;
+	size_t bytes_total = copy->cp_count;
+	size_t bytes_to_copy;
+	u64 src_pos = copy->cp_src_pos;
+	u64 dst_pos = copy->cp_dst_pos;
+
+	do {
+		bytes_to_copy = min_t(u64, bytes_total, MAX_RW_COUNT);
+		bytes_copied = nfsd_copy_file_range(copy->fh_src, src_pos,
+				copy->fh_dst, dst_pos, bytes_to_copy);
+		if (bytes_copied <= 0)
+			break;
+		bytes_total -= bytes_copied;
+		copy->cp_res.wr_bytes_written += bytes_copied;
+		src_pos += bytes_copied;
+		dst_pos += bytes_copied;
+	} while (bytes_total > 0 && !copy->cp_synchronous);
+	return bytes_copied;
+}
+
+static int nfsd4_do_copy(struct nfsd4_copy *copy, bool sync)
+{
+	__be32 status;
+	ssize_t bytes;
+
+	bytes = _nfsd_copy_file_range(copy);
+	if (bytes < 0)
+		status = nfserrno(bytes);
+	else
+		status = nfsd4_init_copy_res(copy, sync);
+
+	if (copy->cp_src.nl_nsvr > 0)   /* Inter server SSC */
+		nfsd4_cleanup_inter_ssc(copy->ss_mnt, copy->fh_src,
+				copy->fh_dst);
+	else
+		nfsd4_cleanup_intra_ssc(copy->fh_src, copy->fh_dst);
+
+	return status;
+}
+
+static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
+{
+	memcpy(&dst->cp_src_stateid, &src->cp_src_stateid, sizeof(stateid_t));
+	memcpy(&dst->cp_dst_stateid, &src->cp_dst_stateid, sizeof(stateid_t));
+	dst->cp_src_pos = src->cp_src_pos;
+	dst->cp_dst_pos = src->cp_dst_pos;
+	dst->cp_count = src->cp_count;
+	memcpy(&dst->cp_src, &src->cp_src, sizeof(struct nl4_servers));
+	dst->cp_consecutive = src->cp_consecutive;
+	dst->cp_synchronous = src->cp_synchronous;
+	memcpy(&dst->cp_res, &src->cp_res, sizeof(src->cp_res));
+	memcpy(&dst->stateid, &src->stateid, sizeof(src->stateid));
+	memcpy(&dst->c_fh, &src->c_fh, sizeof(src->c_fh));
+	/* skipping nfsd4_callback */
+	memcpy(&dst->fh, &src->fh, sizeof(src->fh));
+	dst->fh = src->fh;
+	dst->cp_clp = src->cp_clp;
+	dst->fh_src = src->fh_src;
+	dst->fh_dst = src->fh_dst;
+	dst->ss_mnt = src->ss_mnt;
+	dst->net = src->net;
+}
+
+static void nfsd4_do_async_copy(struct work_struct *work)
+{
+	struct nfsd4_copy *copy =
+		container_of(work, struct nfsd4_copy, cp_work);
+	struct nfsd4_copy *cb_copy;
+
+	if (copy->cp_src.nl_nsvr > 0) { /* Inter server SSC */
+		copy->fh_src = nfs42_ssc_open(copy->ss_mnt, &copy->c_fh,
+						&copy->stateid);
+		if (IS_ERR(copy->fh_src)) {
+			nfsd4_interssc_disconnect(copy->ss_mnt);
+			return;
+		}
+	}
+	copy->nfserr = nfsd4_do_copy(copy, 0);
+	cb_copy = kzalloc(sizeof(struct nfsd4_copy), GFP_KERNEL);
+	if (!cb_copy)
+		goto out;
+	memcpy(&cb_copy->cp_res, &copy->cp_res, sizeof(copy->cp_res));
+	cb_copy->cp_clp = copy->cp_clp;
+	cb_copy->nfserr = copy->nfserr;
+	memcpy(&cb_copy->fh, &copy->fh, sizeof(copy->fh));
+	nfsd4_init_cb(&cb_copy->cp_cb, cb_copy->cp_clp,
+			&nfsd4_cb_offload_ops, NFSPROC4_CLNT_CB_OFFLOAD);
+	nfsd4_run_cb(&cb_copy->cp_cb);
+out:
+	kfree(copy);
+}
+
 static __be32
 nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfsd4_copy *copy)
 {
-	struct vfsmount *ss_mnt = NULL;
-	struct file *src, *dst;
 	__be32 status;
-	ssize_t bytes;
 
 	if (copy->cp_src.nl_nsvr > 0) { /* Inter server SSC */
-		ss_mnt = nfsd4_setup_inter_ssc(rqstp, cstate, copy, &src, &dst);
-		if (IS_ERR(ss_mnt)) {
-			status = nfserrno(PTR_ERR(ss_mnt));
+		copy->ss_mnt = nfsd4_setup_inter_ssc(rqstp, cstate, copy);
+		if (IS_ERR(copy->ss_mnt)) {
+			status = nfserrno(PTR_ERR(copy->ss_mnt));
 			goto out;
 		}
 	} else {
-		status = nfsd4_setup_intra_ssc(rqstp, cstate, copy, &src, &dst);
+		status = nfsd4_setup_intra_ssc(rqstp, cstate, copy);
 		if (status)
 			goto out;
 	}
 
-	bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
-			dst, copy->cp_dst_pos, copy->cp_count);
-
-	if (bytes < 0)
-		status = nfserrno(bytes);
-	else {
-		copy->cp_res.wr_bytes_written = bytes;
-		copy->cp_res.wr_stable_how = NFS_UNSTABLE;
-		copy->cp_consecutive = 1;
-		copy->cp_synchronous = 1;
-		gen_boot_verifier(&copy->cp_res.wr_verifier, SVC_NET(rqstp));
-		status = nfs_ok;
+	copy->cp_clp = cstate->clp;
+	memcpy(&copy->fh, &cstate->current_fh.fh_handle,
+		sizeof(struct knfsd_fh));
+	copy->net = SVC_NET(rqstp);
+	if (!copy->cp_synchronous) {
+		struct nfsd4_copy *async_copy;
+
+		status = nfsd4_init_copy_res(copy, 0);
+		async_copy = kzalloc(sizeof(struct nfsd4_copy), GFP_KERNEL);
+		if (!async_copy)
+			goto out_err;
+		dup_copy_fields(copy, async_copy);
+		memcpy(&copy->cp_res.cb_stateid, &copy->cp_dst_stateid,
+			sizeof(copy->cp_dst_stateid));
+		INIT_WORK(&async_copy->cp_work, nfsd4_do_async_copy);
+		queue_work(copy_wq, &async_copy->cp_work);
+	} else {
+		status = nfsd4_do_copy(copy, 1);
 	}
-
-	if (copy->cp_src.nl_nsvr > 0)   /* Inter server SSC */
-		nfsd4_cleanup_inter_ssc(ss_mnt, src, dst);
-	else
-		nfsd4_cleanup_intra_ssc(src, dst);
 out:
 	return status;
+out_err:
+	nfsd4_interssc_disconnect(copy->ss_mnt);
+	status = nfserrno(-ENOMEM);
+	goto out;
 }
 
 static __be32
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index b019fc8..d391ce5 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -7123,8 +7123,14 @@ static int nfs4_state_create_net(struct net *net)
 		goto out_free_laundry;
 
 	set_max_delegations();
-	return 0;
 
+	ret = nfsd4_create_copy_queue();
+	if (ret)
+		goto out_free_callback;
+
+	return 0;
+out_free_callback:
+	nfsd4_destroy_callback_queue();
 out_free_laundry:
 	destroy_workqueue(laundry_wq);
 out_cleanup_cred:
@@ -7187,6 +7193,7 @@ static int nfs4_state_create_net(struct net *net)
 	destroy_workqueue(laundry_wq);
 	nfsd4_destroy_callback_queue();
 	cleanup_callback_cred();
+	nfsd4_destroy_copy_queue();
 }
 
 static void
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 3b0da32..2acea23 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -649,6 +649,8 @@ extern void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
 extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(const char *name,
 							struct nfsd_net *nn);
 extern bool nfs4_has_reclaimed_state(const char *name, struct nfsd_net *nn);
+extern int nfsd4_create_copy_queue(void);
+extern void nfsd4_destroy_copy_queue(void);
 
 struct nfs4_file *find_file(struct knfsd_fh *fh);
 void put_nfs4_file(struct nfs4_file *fi);
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index c3e6907..d75e530 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -532,10 +532,21 @@ struct nfsd4_copy {
 	/* response */
 	struct nfsd42_write_res	cp_res;
 
+	struct nfs_fh           c_fh;
+	nfs4_stateid            stateid;
+
 	/* for cb_offload */
 	struct nfsd4_callback	cp_cb;
 	__be32			nfserr;
 	struct knfsd_fh		fh;
+
+	struct work_struct	cp_work;
+	struct nfs4_client	*cp_clp;
+
+	struct file		*fh_src;
+	struct file		*fh_dst;
+	struct vfsmount		*ss_mnt;
+	struct net		*net;
 };
 
 struct nfsd4_seek {
-- 
1.8.3.1

[RFC v3 36/42] NFSD handle OFFLOAD_CANCEL op

[Olga Kornievskaia]

Upon receiving OFFLOAD_CANCEL search the list of copy stateids,
if found remove it and it will lead to following READs failing
with ERR_PARTNER_NO_AUTH.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c  | 11 +++++++++++
 fs/nfsd/nfs4state.c | 30 ++++++++++++++++++++----------
 fs/nfsd/state.h     |  2 ++
 3 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index a977142..7e7b416 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1579,6 +1579,17 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 		     struct nfsd4_compound_state *cstate,
 		     struct nfsd4_offload_status *os)
 {
+
+	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
+	__be32 status;
+	struct nfs4_cp_state *state = NULL;
+
+	status = find_cp_state(nn, &os->stateid, &state);
+	if (!status) {
+		list_del(&state->cp_list);
+		nfs4_free_cp_state(state);
+	}
+
 	return 0;
 }
 
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index d391ce5..bcd87c9 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -743,6 +743,22 @@ static void nfs4_free_cp_statelist(struct nfs4_stid *stid)
 	}
 }
 
+__be32 find_cp_state(struct nfsd_net *nn, stateid_t *st,
+			    struct nfs4_cp_state **cps)
+{
+	struct nfs4_cp_state *state = NULL;
+
+	if (st->si_opaque.so_clid.cl_id != nn->s2s_cp_cl_id)
+		return nfserr_bad_stateid;
+	spin_lock(&nn->s2s_cp_lock);
+	state = idr_find(&nn->s2s_cp_stateids, st->si_opaque.so_id);
+	spin_unlock(&nn->s2s_cp_lock);
+	if (!state)
+		return nfserr_bad_stateid;
+	*cps = state;
+	return 0;
+}
+
 /*
  * A READ from an inter server to server COPY will have a
  * copy stateid. Return the parent nfs4_stid.
@@ -750,18 +766,12 @@ static void nfs4_free_cp_statelist(struct nfs4_stid *stid)
 static __be32 find_cp_state_parent(struct nfsd_net *nn, stateid_t *st,
 				   struct nfs4_stid **stid)
 {
+	__be32 status;
 	struct nfs4_cp_state *cps = NULL;
 
-	if (st->si_opaque.so_clid.cl_id != nn->s2s_cp_cl_id)
-		return nfserr_bad_stateid;
-	spin_lock(&nn->s2s_cp_lock);
-	cps = idr_find(&nn->s2s_cp_stateids, st->si_opaque.so_id);
-	spin_unlock(&nn->s2s_cp_lock);
-	if (!cps) {
-		pr_info("NFSD: find_cp_state cl_id %d so_id %d NOT FOUND\n",
-			st->si_opaque.so_clid.cl_id, st->si_opaque.so_id);
-		return nfserr_bad_stateid;
-	}
+	status = find_cp_state(nn, st, &cps);
+	if (status)
+		return status;
 
 	/* Did the inter server to server copy start in time? */
 	if (cps->cp_active == false && !time_after(cps->cp_timeout, jiffies))
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 2acea23..ebf968d 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -651,6 +651,8 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(const char *name,
 extern bool nfs4_has_reclaimed_state(const char *name, struct nfsd_net *nn);
 extern int nfsd4_create_copy_queue(void);
 extern void nfsd4_destroy_copy_queue(void);
+extern __be32 find_cp_state(struct nfsd_net *nn, stateid_t *st,
+			struct nfs4_cp_state **cps);
 
 struct nfs4_file *find_file(struct knfsd_fh *fh);
 void put_nfs4_file(struct nfs4_file *fi);
-- 
1.8.3.1

[RFC v3 37/42] NFSD stop queued async copies on client shutdown

[Olga Kornievskaia]

If client is shutting down and there are still async copies going
on, then stop queued async copies.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c  | 21 ++++++++++++++++-----
 fs/nfsd/nfs4state.c |  1 +
 fs/nfsd/state.h     |  2 ++
 3 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 7e7b416..b93713d 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -92,6 +92,12 @@ void nfsd4_destroy_copy_queue(void)
 	destroy_workqueue(copy_wq);
 }
 
+void nfsd4_shutdown_copy(struct nfs4_client *clp)
+{
+	set_bit(NFSD4_CLIENT_COPY_KILL, &clp->cl_flags);
+	flush_workqueue(copy_wq);
+}
+
 #define NFSDDBG_FACILITY		NFSDDBG_PROC
 
 static u32 nfsd_attrmask[] = {
@@ -1416,15 +1422,16 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 {
 	struct nfsd4_copy *copy =
 		container_of(work, struct nfsd4_copy, cp_work);
-	struct nfsd4_copy *cb_copy;
+	struct nfsd4_copy *cb_copy = NULL;
+
+	if (test_bit(NFSD4_CLIENT_COPY_KILL, &copy->cp_clp->cl_flags))
+		goto out_err;
 
 	if (copy->cp_src.nl_nsvr > 0) { /* Inter server SSC */
 		copy->fh_src = nfs42_ssc_open(copy->ss_mnt, &copy->c_fh,
 						&copy->stateid);
-		if (IS_ERR(copy->fh_src)) {
-			nfsd4_interssc_disconnect(copy->ss_mnt);
-			return;
-		}
+		if (IS_ERR(copy->fh_src))
+			goto out_err;
 	}
 	copy->nfserr = nfsd4_do_copy(copy, 0);
 	cb_copy = kzalloc(sizeof(struct nfsd4_copy), GFP_KERNEL);
@@ -1439,6 +1446,10 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 	nfsd4_run_cb(&cb_copy->cp_cb);
 out:
 	kfree(copy);
+	return;
+out_err:
+	nfsd4_interssc_disconnect(copy->ss_mnt);
+	goto out;
 }
 
 static __be32
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index bcd87c9..7727ce53 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -1995,6 +1995,7 @@ static __be32 mark_client_expired_locked(struct nfs4_client *clp)
 	}
 	nfsd4_return_all_client_layouts(clp);
 	nfsd4_shutdown_callback(clp);
+	nfsd4_shutdown_copy(clp);
 	if (clp->cl_cb_conn.cb_xprt)
 		svc_xprt_put(clp->cl_cb_conn.cb_xprt);
 	free_client(clp);
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index ebf968d..fa749763 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -336,6 +336,7 @@ struct nfs4_client {
 #define NFSD4_CLIENT_RECLAIM_COMPLETE	(3)	/* reclaim_complete done */
 #define NFSD4_CLIENT_CONFIRMED		(4)	/* client is confirmed */
 #define NFSD4_CLIENT_UPCALL_LOCK	(5)	/* upcall serialization */
+#define NFSD4_CLIENT_COPY_KILL		(6)	/* stop copy workqueue */
 #define NFSD4_CLIENT_CB_FLAG_MASK	(1 << NFSD4_CLIENT_CB_UPDATE | \
 					 1 << NFSD4_CLIENT_CB_KILL)
 	unsigned long		cl_flags;
@@ -645,6 +646,7 @@ extern void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
 extern int nfsd4_create_callback_queue(void);
 extern void nfsd4_destroy_callback_queue(void);
 extern void nfsd4_shutdown_callback(struct nfs4_client *);
+extern void nfsd4_shutdown_copy(struct nfs4_client *clp);
 extern void nfsd4_prepare_cb_recall(struct nfs4_delegation *dp);
 extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(const char *name,
 							struct nfsd_net *nn);
-- 
1.8.3.1

[RFC v3 38/42] NFSD create new stateid for async copy

[Olga Kornievskaia]

Previously dst copy stateid was used as reply to the asynchronous
copy. Instead, generate a new stateid and the destination server
will keep a list of the stateids. If it receives a cancel, it can
decide to forego sending the CB_OFFLOAD.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c | 56 +++++++++++++++++++++++++++++++++++++++++++-----------
 fs/nfsd/state.h    |  3 +++
 fs/nfsd/xdr4.h     |  2 ++
 3 files changed, 50 insertions(+), 11 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index b93713d..c411887 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1039,7 +1039,8 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 static __be32
 nfsd4_verify_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 		  stateid_t *src_stateid, struct file **src,
-		  stateid_t *dst_stateid, struct file **dst)
+		  stateid_t *dst_stateid, struct file **dst,
+		  struct nfs4_stid **stid)
 {
 	__be32 status;
 
@@ -1053,7 +1054,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					    dst_stateid, WR_STATE, dst, NULL,
-					    NULL);
+					    stid);
 	if (status) {
 		dprintk("NFSD: %s: couldn't process dst stateid!\n", __func__);
 		goto out_put_src;
@@ -1083,7 +1084,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
 	__be32 status;
 
 	status = nfsd4_verify_copy(rqstp, cstate, &clone->cl_src_stateid, &src,
-				   &clone->cl_dst_stateid, &dst);
+				   &clone->cl_dst_stateid, &dst, NULL);
 	if (status)
 		goto out;
 
@@ -1241,7 +1242,8 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 	/* Verify the destination stateid and set dst struct file*/
 	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
 					&copy->cp_dst_stateid,
-					WR_STATE, &copy->fh_dst, NULL, NULL);
+					WR_STATE, &copy->fh_dst, NULL,
+					&copy->stid);
 	if (status) {
 		ss_mnt = ERR_PTR(be32_to_cpu(status));
 		goto out;
@@ -1303,7 +1305,7 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 
 	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid,
 				   &copy->fh_src, &copy->cp_dst_stateid,
-				   &copy->fh_dst);
+				   &copy->fh_dst, &copy->stid);
 	if (status)
 		goto out;
 
@@ -1343,8 +1345,6 @@ static int nfsd4_cb_offload_done(struct nfsd4_callback *cb,
 
 static int nfsd4_init_copy_res(struct nfsd4_copy *copy, bool sync)
 {
-	memcpy(&copy->cp_res.cb_stateid, &copy->cp_dst_stateid,
-		sizeof(copy->cp_dst_stateid));
 	copy->cp_res.wr_stable_how = NFS_UNSTABLE;
 	copy->cp_consecutive = 1;
 	copy->cp_synchronous = sync;
@@ -1360,6 +1360,7 @@ static int _nfsd_copy_file_range(struct nfsd4_copy *copy)
 	size_t bytes_to_copy;
 	u64 src_pos = copy->cp_src_pos;
 	u64 dst_pos = copy->cp_dst_pos;
+	bool cancelled = false;
 
 	do {
 		bytes_to_copy = min_t(u64, bytes_total, MAX_RW_COUNT);
@@ -1371,7 +1372,12 @@ static int _nfsd_copy_file_range(struct nfsd4_copy *copy)
 		copy->cp_res.wr_bytes_written += bytes_copied;
 		src_pos += bytes_copied;
 		dst_pos += bytes_copied;
-	} while (bytes_total > 0 && !copy->cp_synchronous);
+		if (!copy->cp_synchronous) {
+			spin_lock(&copy->cps->cp_lock);
+			cancelled = copy->cps->cp_cancelled;
+			spin_unlock(&copy->cps->cp_lock);
+		}
+	} while (bytes_total > 0 && !copy->cp_synchronous && !cancelled);
 	return bytes_copied;
 }
 
@@ -1416,6 +1422,8 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
 	dst->fh_dst = src->fh_dst;
 	dst->ss_mnt = src->ss_mnt;
 	dst->net = src->net;
+	dst->stid = src->stid;
+	dst->cps = src->cps;
 }
 
 static void nfsd4_do_async_copy(struct work_struct *work)
@@ -1434,6 +1442,10 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 			goto out_err;
 	}
 	copy->nfserr = nfsd4_do_copy(copy, 0);
+
+	if (copy->cps->cp_cancelled)
+		goto out;
+
 	cb_copy = kzalloc(sizeof(struct nfsd4_copy), GFP_KERNEL);
 	if (!cb_copy)
 		goto out;
@@ -1445,6 +1457,7 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 			&nfsd4_cb_offload_ops, NFSPROC4_CLNT_CB_OFFLOAD);
 	nfsd4_run_cb(&cb_copy->cp_cb);
 out:
+	nfs4_put_stid(copy->stid);
 	kfree(copy);
 	return;
 out_err:
@@ -1475,15 +1488,26 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 		sizeof(struct knfsd_fh));
 	copy->net = SVC_NET(rqstp);
 	if (!copy->cp_synchronous) {
+		struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
 		struct nfsd4_copy *async_copy;
 
 		status = nfsd4_init_copy_res(copy, 0);
 		async_copy = kzalloc(sizeof(struct nfsd4_copy), GFP_KERNEL);
 		if (!async_copy)
 			goto out_err;
+		copy->cps = nfs4_alloc_init_cp_state(nn, nn->nfsd4_lease,
+				copy->stid);
+		if (!copy->cps)
+			goto out_err;
+		/* take a reference on the parent stateid so it's not
+		 * not freed by the copy compound
+		 */
+		atomic_inc(&copy->stid->sc_count);
+		copy->cps->cp_dst_async = true;
+		spin_lock_init(&copy->cps->cp_lock);
+		memcpy(&copy->cp_res.cb_stateid, &copy->cps->cp_stateid,
+			sizeof(copy->cps->cp_stateid));
 		dup_copy_fields(copy, async_copy);
-		memcpy(&copy->cp_res.cb_stateid, &copy->cp_dst_stateid,
-			sizeof(copy->cp_dst_stateid));
 		INIT_WORK(&async_copy->cp_work, nfsd4_do_async_copy);
 		queue_work(copy_wq, &async_copy->cp_work);
 	} else {
@@ -1596,7 +1620,17 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 	struct nfs4_cp_state *state = NULL;
 
 	status = find_cp_state(nn, &os->stateid, &state);
-	if (!status) {
+	/* on the source server, remove stateid from list of acceptable
+	 * stateid to force reads to fail. on the destination server,
+	 * callback offload stateids shouldn't be removed and instead
+	 * mark the offload copy state to be cancelled.
+	 */
+	if (state) {
+		spin_lock(&state->cp_lock);
+		state->cp_cancelled = true;
+		spin_unlock(&state->cp_lock);
+	}
+	if (!status && !state->cp_dst_async) {
 		list_del(&state->cp_list);
 		nfs4_free_cp_state(state);
 	}
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index fa749763..70ee3fe 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -114,6 +114,9 @@ struct nfs4_cp_state {
 	struct nfs4_stid	*cp_p_stid;	/* pointer to parent */
 	bool			cp_active;	/* has the copy started */
 	unsigned long		cp_timeout;	/* copy timeout */
+	bool			cp_dst_async;	/* async copy on dst server */
+	bool			cp_cancelled;	/* copy cancelled */
+	spinlock_t		cp_lock;
 };
 
 /*
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index d75e530..ae5c6e3 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -547,6 +547,8 @@ struct nfsd4_copy {
 	struct file		*fh_dst;
 	struct vfsmount		*ss_mnt;
 	struct net		*net;
+	struct nfs4_stid	*stid;
+	struct nfs4_cp_state	*cps;
 };
 
 struct nfsd4_seek {
-- 
1.8.3.1

[RFC v3 39/42] NFSD define EBADF in nfserrno

[Olga Kornievskaia]

vfs_copy_file_range() can return EBADF which currently nfsd does
not recognize.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfsproc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/nfsd/nfsproc.c b/fs/nfsd/nfsproc.c
index 03a7e9d..5924926 100644
--- a/fs/nfsd/nfsproc.c
+++ b/fs/nfsd/nfsproc.c
@@ -787,6 +787,7 @@ struct svc_version	nfsd_version2 = {
 		{ nfserr_serverfault, -ENFILE },
 		{ nfserr_io, -EUCLEAN },
 		{ nfserr_perm, -ENOKEY },
+		{ nfserr_badhandle, -EBADF},
 	};
 	int	i;
 
-- 
1.8.3.1

[RFC v3 40/42] NFSD support OFFLOAD_STATUS

[Olga Kornievskaia]

Update number of bytes copied in the copy state and query that
value under lock if OFFLOAD_STATUS operation received.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c | 15 ++++++++++++++-
 fs/nfsd/state.h    |  1 +
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index c411887..1094286 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1375,6 +1375,8 @@ static int _nfsd_copy_file_range(struct nfsd4_copy *copy)
 		if (!copy->cp_synchronous) {
 			spin_lock(&copy->cps->cp_lock);
 			cancelled = copy->cps->cp_cancelled;
+			copy->cps->cp_bytes_copied =
+				copy->cp_res.wr_bytes_written;
 			spin_unlock(&copy->cps->cp_lock);
 		}
 	} while (bytes_total > 0 && !copy->cp_synchronous && !cancelled);
@@ -1665,7 +1667,18 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 		     struct nfsd4_compound_state *cstate,
 		     struct nfsd4_offload_status *os)
 {
-	return nfserr_notsupp;
+	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
+	__be32 status;
+	struct nfs4_cp_state *state = NULL;
+
+	status = find_cp_state(nn, &os->stateid, &state);
+
+	if (state) {
+		spin_lock(&state->cp_lock);
+		os->count = state->cp_bytes_copied;
+		spin_unlock(&state->cp_lock);
+	}
+	return status;
 }
 
 static __be32
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 70ee3fe..25c5d82 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -117,6 +117,7 @@ struct nfs4_cp_state {
 	bool			cp_dst_async;	/* async copy on dst server */
 	bool			cp_cancelled;	/* copy cancelled */
 	spinlock_t		cp_lock;
+	ssize_t			cp_bytes_copied;/* copy progress */
 };
 
 /*
-- 
1.8.3.1

[RFC v3 41/42] NFSD remove copy stateid when vfs_copy_file_range completes

[Olga Kornievskaia]

Before this patch, the copy state information sticks around until
destination file is closed. However, we can release the resources
earlier when copy is done.

In this patch, I choose to remove the stateid after vfs copy is
done but before CB_OFFLOAD is done. The reason is simple because
I don't need to keep track of the nfsd_net structure then when
doing the callback (alternatively, we also need to copy the
nfsd_net structure for the callback). The drawback is that time
copy state information is available for query by OFFLOAD_STATUS
is slightly less.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 1094286..86c20fa 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1459,6 +1459,8 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 			&nfsd4_cb_offload_ops, NFSPROC4_CLNT_CB_OFFLOAD);
 	nfsd4_run_cb(&cb_copy->cp_cb);
 out:
+	list_del(&copy->cps->cp_list);
+	nfs4_free_cp_state(copy->cps);
 	nfs4_put_stid(copy->stid);
 	kfree(copy);
 	return;
-- 
1.8.3.1

[RFC v3 42/42] NFSD delay the umount after COPY

[Olga Kornievskaia]

After COPY is done, we always unmounted right away. However, if
another copy arrives right after it (ie, large copy that's broken
down into pieces), then we want to save the time of re-establishing
the clientid/session with the source server.

Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
---
 fs/nfsd/nfs4proc.c  | 41 +++++++++++++++++++++++++++++++++++++----
 fs/nfsd/nfs4state.c |  6 ++++++
 fs/nfsd/state.h     |  2 ++
 fs/nfsd/xdr4.h      |  5 +++++
 4 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 86c20fa..7dbe8df 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -78,6 +78,7 @@
 #endif
 
 static struct workqueue_struct *copy_wq;
+static struct workqueue_struct *umount_wq;
 
 int nfsd4_create_copy_queue(void)
 {
@@ -98,6 +99,19 @@ void nfsd4_shutdown_copy(struct nfs4_client *clp)
 	flush_workqueue(copy_wq);
 }
 
+int nfsd4_create_umount_queue(void)
+{
+	umount_wq = create_singlethread_workqueue("nfsd4_umount");
+	if (!umount_wq)
+		return -ENOMEM;
+	return 0;
+}
+
+void nfsd4_destroy_umount_queue(void)
+{
+	destroy_workqueue(umount_wq);
+}
+
 #define NFSDDBG_FACILITY		NFSDDBG_PROC
 
 static u32 nfsd_attrmask[] = {
@@ -1211,12 +1225,24 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 }
 
 static void
-nfsd4_interssc_disconnect(struct vfsmount *ss_mnt)
+_nfsd4_interssc_disconnect(struct vfsmount *ss_mnt)
 {
 	mntput(ss_mnt);
 	nfs_sb_deactive(ss_mnt->mnt_sb);
 }
 
+static void
+nfsd4_interssc_disconnect(struct work_struct *work)
+{
+	struct delayed_work *dwork = to_delayed_work(work);
+	struct nfsd4_delayed_umount *mount =
+		container_of(dwork, struct nfsd4_delayed_umount, work);
+
+	_nfsd4_interssc_disconnect(mount->ss_mnt);
+	kfree(mount);
+}
+
+
 /**
  * nfsd4_setup_inter_ssc
  *
@@ -1270,11 +1296,18 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
 nfsd4_cleanup_inter_ssc(struct vfsmount *ss_mnt, struct file *src,
 			struct file *dst)
 {
+	struct nfsd4_delayed_umount *umount;
 	nfs42_ssc_close(src);
 	fput(src);
 	fput(dst);
 
-	nfsd4_interssc_disconnect(ss_mnt);
+	umount = kzalloc(sizeof(struct nfsd4_delayed_umount), GFP_KERNEL);
+	if (!umount)
+		return;
+	umount->ss_mnt = ss_mnt;
+	mntget(ss_mnt);
+	INIT_DELAYED_WORK(&umount->work, nfsd4_interssc_disconnect);
+	queue_delayed_work(umount_wq, &umount->work, 90 * HZ);
 
 }
 
@@ -1465,7 +1498,7 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 	kfree(copy);
 	return;
 out_err:
-	nfsd4_interssc_disconnect(copy->ss_mnt);
+	_nfsd4_interssc_disconnect(copy->ss_mnt);
 	goto out;
 }
 
@@ -1520,7 +1553,7 @@ static void nfsd4_do_async_copy(struct work_struct *work)
 out:
 	return status;
 out_err:
-	nfsd4_interssc_disconnect(copy->ss_mnt);
+	_nfsd4_interssc_disconnect(copy->ss_mnt);
 	status = nfserrno(-ENOMEM);
 	goto out;
 }
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 7727ce53..daee2cc 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -7138,8 +7138,13 @@ static int nfs4_state_create_net(struct net *net)
 	ret = nfsd4_create_copy_queue();
 	if (ret)
 		goto out_free_callback;
+	ret = nfsd4_create_umount_queue();
+	if (ret)
+		goto out_free_copy_queue;
 
 	return 0;
+out_free_copy_queue:
+	nfsd4_destroy_copy_queue();
 out_free_callback:
 	nfsd4_destroy_callback_queue();
 out_free_laundry:
@@ -7205,6 +7210,7 @@ static int nfs4_state_create_net(struct net *net)
 	nfsd4_destroy_callback_queue();
 	cleanup_callback_cred();
 	nfsd4_destroy_copy_queue();
+	nfsd4_destroy_umount_queue();
 }
 
 static void
diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
index 25c5d82..69db34d 100644
--- a/fs/nfsd/state.h
+++ b/fs/nfsd/state.h
@@ -657,6 +657,8 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(const char *name,
 extern bool nfs4_has_reclaimed_state(const char *name, struct nfsd_net *nn);
 extern int nfsd4_create_copy_queue(void);
 extern void nfsd4_destroy_copy_queue(void);
+extern int nfsd4_create_umount_queue(void);
+extern void nfsd4_destroy_umount_queue(void);
 extern __be32 find_cp_state(struct nfsd_net *nn, stateid_t *st,
 			struct nfs4_cp_state **cps);
 
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index ae5c6e3..3b2176f 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -551,6 +551,11 @@ struct nfsd4_copy {
 	struct nfs4_cp_state	*cps;
 };
 
+struct nfsd4_delayed_umount {
+	struct vfsmount		*ss_mnt;
+	struct delayed_work	work;
+};
+
 struct nfsd4_seek {
 	/* request */
 	stateid_t	seek_stateid;
-- 
1.8.3.1

Re: [RFC v3 01/42] fs: Don't copy beyond the end of the file

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> 
> Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>

Would it make sense to submit this patch to the VFS in the same series as the cross mountpoint patch, that way they both go in at once?  This one should really be able to go in now, rather than waiting for the rest of the series.

Thanks,
Anna

> ---
>  fs/read_write.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/fs/read_write.c b/fs/read_write.c
> index 19d4d88..cc2618b 100644
> --- a/fs/read_write.c
> +++ b/fs/read_write.c
> @@ -1584,6 +1584,9 @@ ssize_t vfs_copy_file_range(struct file *file_in, loff_t pos_in,
>  	if (unlikely(ret))
>  		return ret;
>  
> +	if (pos_in >= i_size_read(inode_in))
> +		return -EINVAL;
> +
>  	if (!(file_in->f_mode & FMODE_READ) ||
>  	    !(file_out->f_mode & FMODE_WRITE) ||
>  	    (file_out->f_flags & O_APPEND))
> 

Re: [RFC v3 03/42] NFS CB_OFFLOAD xdr

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfs/callback.h      | 13 ++++++++
>  fs/nfs/callback_proc.c |  7 +++++
>  fs/nfs/callback_xdr.c  | 80 +++++++++++++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 99 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/nfs/callback.h b/fs/nfs/callback.h
> index c701c30..e4ab65d 100644
> --- a/fs/nfs/callback.h
> +++ b/fs/nfs/callback.h
> @@ -189,6 +189,19 @@ extern __be32 nfs4_callback_notify_lock(struct cb_notify_lock_args *args,
>  					 void *dummy,
>  					 struct cb_process_state *cps);
>  #endif /* CONFIG_NFS_V4_1 */
> +#ifdef CONFIG_NFS_V4_2
> +struct cb_offloadargs {
> +	struct nfs_fh		coa_fh;
> +	nfs4_stateid		coa_stateid;
> +	uint32_t		error;
> +	uint64_t		wr_count;
> +	struct nfs_writeverf	wr_writeverf;
> +};
> +
> +extern __be32 nfs4_callback_offload(
> +	struct cb_offloadargs *args,
> +	void *dummy, struct cb_process_state *cps);
> +#endif /* CONFIG_NFS_V4_2 */
>  extern int check_gss_callback_principal(struct nfs_client *, struct svc_rqst *);
>  extern __be32 nfs4_callback_getattr(struct cb_getattrargs *args,
>  				    struct cb_getattrres *res,
> diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
> index 52479f1..5396751 100644
> --- a/fs/nfs/callback_proc.c
> +++ b/fs/nfs/callback_proc.c
> @@ -647,3 +647,10 @@ __be32 nfs4_callback_notify_lock(struct cb_notify_lock_args *args, void *dummy,
>  	return htonl(NFS4_OK);
>  }
>  #endif /* CONFIG_NFS_V4_1 */
> +#ifdef CONFIG_NFS_V4_2
> +__be32 nfs4_callback_offload(struct cb_offloadargs *args, void *dummy,
> +				struct cb_process_state *cps)
> +{
> +	return 0;
> +}
> +#endif /* CONFIG_NFS_V4_2 */
> diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c
> index 390ac9c..be5ffa1 100644
> --- a/fs/nfs/callback_xdr.c
> +++ b/fs/nfs/callback_xdr.c
> @@ -37,6 +37,9 @@
>  #define CB_OP_RECALLSLOT_RES_MAXSZ	(CB_OP_HDR_RES_MAXSZ)
>  #define CB_OP_NOTIFY_LOCK_RES_MAXSZ	(CB_OP_HDR_RES_MAXSZ)
>  #endif /* CONFIG_NFS_V4_1 */
> +#ifdef CONFIG_NFS_V4_2
> +#define CB_OP_OFFLOAD_RES_MAXSZ		(CB_OP_HDR_RES_MAXSZ)
> +#endif /* CONFIG_NFS_V4_2 */
>  
>  #define NFSDBG_FACILITY NFSDBG_CALLBACK
>  
> @@ -521,7 +524,72 @@ static __be32 decode_notify_lock_args(struct svc_rqst *rqstp, struct xdr_stream
>  }
>  
>  #endif /* CONFIG_NFS_V4_1 */
> +#ifdef CONFIG_NFS_V4_2
> +static __be32 decode_write_response(struct xdr_stream *xdr,
> +					struct cb_offloadargs *args)
> +{
> +	__be32 *p;
> +	__be32 dummy;
> +
> +	/* skip the always zero field */
> +	p = read_buf(xdr, 4);
> +	if (unlikely(!p))
> +		goto out;
> +	dummy = ntohl(*p++);
> +
> +	/* decode count, stable_how, verifier */
> +	p = xdr_inline_decode(xdr, 8 + 4);
> +	if (unlikely(!p))
> +		goto out;
> +	p = xdr_decode_hyper(p, &args->wr_count);
> +	args->wr_writeverf.committed = be32_to_cpup(p);
> +	p = xdr_inline_decode(xdr, NFS4_VERIFIER_SIZE);
> +	if (likely(p)) {
> +		memcpy(&args->wr_writeverf.verifier.data[0], p,
> +			NFS4_VERIFIER_SIZE);
> +		return 0;
> +	}
> +out:
> +	return htonl(NFS4ERR_RESOURCE);
> +}
> +
> +static __be32 decode_offload_args(struct svc_rqst *rqstp,
> +					struct xdr_stream *xdr,
> +					struct cb_offloadargs *args)
> +{
> +	__be32 *p;
> +	__be32 status;
> +
> +	/* decode fh */
> +	status = decode_fh(xdr, &args->coa_fh);
> +	if (unlikely(status != 0))
> +		return status;
> +
> +	/* decode stateid */
> +	status = decode_stateid(xdr, &args->coa_stateid);
> +	if (unlikely(status != 0))
> +		return status;
>  
> +	/* decode status */
> +	p = read_buf(xdr, 4);
> +	if (unlikely(!p))
> +		goto out;
> +	args->error = ntohl(*p++);
> +	if (!args->error) {
> +		status = decode_write_response(xdr, args);
> +		if (unlikely(status != 0))
> +			return status;
> +	} else {
> +		p = xdr_inline_decode(xdr, 8);
> +		if (unlikely(!p))
> +			goto out;
> +		p = xdr_decode_hyper(p, &args->wr_count);
> +	}
> +	return 0;
> +out:
> +	return htonl(NFS4ERR_RESOURCE);
> +}
> +#endif /* CONFIG_NFS_V4_2 */
>  static __be32 encode_string(struct xdr_stream *xdr, unsigned int len, const char *str)
>  {
>  	if (unlikely(xdr_stream_encode_opaque(xdr, str, len) < 0))
> @@ -785,7 +853,10 @@ static void nfs4_cb_free_slot(struct cb_process_state *cps)
>  	if (status != htonl(NFS4ERR_OP_ILLEGAL))
>  		return status;
>  
> -	if (op_nr == OP_CB_OFFLOAD)
> +	if (op_nr == OP_CB_OFFLOAD) {
> +		*op = &callback_ops[op_nr];
> +		return htonl(NFS_OK);
> +	} else
>  		return htonl(NFS4ERR_NOTSUPP);
>  	return htonl(NFS4ERR_OP_ILLEGAL);
>  }
> @@ -982,6 +1053,13 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
>  		.res_maxsize = CB_OP_NOTIFY_LOCK_RES_MAXSZ,
>  	},
>  #endif /* CONFIG_NFS_V4_1 */
> +#ifdef CONFIG_NFS_V4_2
> +	[OP_CB_OFFLOAD] = {
> +		.process_op = (callback_process_op_t)nfs4_callback_offload,
> +		.decode_args = (callback_decode_arg_t)decode_offload_args,
> +		.res_maxsize = CB_OP_OFFLOAD_RES_MAXSZ,
> +	},
> +#endif /* CONFIG_NFS_V4_2 */
Just a heads up that the function signatures change in 4.13 and the casting is removed.  The .process_op function now expects:
    __be32 func(void *argp, void *resp, struct cb_process_state *cps);

And .decode_args is now:
    __be32 func(struct svc_rqst *rqstp, struct xdr_stream *xdr, void *argp);

Thanks,
Anna

>  };
>  
>  /*
> 

Re: [RFC v3 04/42] NFS OFFLOAD_STATUS xdr

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfs/nfs42xdr.c         | 87 +++++++++++++++++++++++++++++++++++++++++++++++
>  fs/nfs/nfs4proc.c         |  1 +
>  fs/nfs/nfs4xdr.c          |  1 +
>  include/linux/nfs4.h      |  1 +
>  include/linux/nfs_fs_sb.h |  1 +
>  include/linux/nfs_xdr.h   | 12 +++++++
>  6 files changed, 103 insertions(+)
> 
> diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
> index 528362f..e369de1 100644
> --- a/fs/nfs/nfs42xdr.c
> +++ b/fs/nfs/nfs42xdr.c
> @@ -25,6 +25,11 @@
>  					 NFS42_WRITE_RES_SIZE + \
>  					 1 /* cr_consecutive */ + \
>  					 1 /* cr_synchronous */)
> +#define encode_offload_status_maxsz	(op_encode_hdr_maxsz + \
> +					 XDR_QUADLEN(NFS4_STATEID_SIZE))
> +#define decode_offload_status_maxsz	(op_decode_hdr_maxsz + \
> +					 2 /* osr_count */ + \
> +					 1 + 1 /* osr_complete<1> */)
>  #define encode_deallocate_maxsz		(op_encode_hdr_maxsz + \
>  					 encode_fallocate_maxsz)
>  #define decode_deallocate_maxsz		(op_decode_hdr_maxsz)
> @@ -74,6 +79,12 @@
>  					 decode_putfh_maxsz + \
>  					 decode_copy_maxsz + \
>  					 decode_commit_maxsz)
> +#define NFS4_enc_offload_status_sz	(compound_encode_hdr_maxsz + \
> +					 encode_putfh_maxsz + \
> +					 encode_offload_status_maxsz)
> +#define NFS4_dec_offload_status_sz	(compound_decode_hdr_maxsz + \
> +					 decode_putfh_maxsz + \
> +					 decode_offload_status_maxsz)
>  #define NFS4_enc_deallocate_sz		(compound_encode_hdr_maxsz + \
>  					 encode_putfh_maxsz + \
>  					 encode_deallocate_maxsz + \
> @@ -144,6 +155,14 @@ static void encode_copy(struct xdr_stream *xdr,
>  	encode_uint32(xdr, 0); /* src server list */
>  }
>  
> +static void encode_offload_status(struct xdr_stream *xdr,
> +				  struct nfs42_offload_status_args *args,
> +				  struct compound_hdr *hdr)
> +{
> +	encode_op_hdr(xdr, OP_OFFLOAD_STATUS, decode_offload_status_maxsz, hdr);
> +	encode_nfs4_stateid(xdr, &args->osa_stateid);
> +}
> +
>  static void encode_deallocate(struct xdr_stream *xdr,
>  			      struct nfs42_falloc_args *args,
>  			      struct compound_hdr *hdr)
> @@ -258,6 +277,24 @@ static void nfs4_xdr_enc_copy(struct rpc_rqst *req,
>  }
>  
>  /*
> + * Encode OFFLOAD_STATUS request
> + */
> +static void nfs4_xdr_enc_offload_status(struct rpc_rqst *req,
> +					struct xdr_stream *xdr,
> +					struct nfs42_offload_status_args *args)

This last argument should now be: "const void *data"

> +{
> +	struct compound_hdr hdr = {
> +		.minorversion = nfs4_xdr_minorversion(&args->osa_seq_args),
> +	};
> +
> +	encode_compound_hdr(xdr, req, &hdr);
> +	encode_sequence(xdr, &args->osa_seq_args, &hdr);
> +	encode_putfh(xdr, args->osa_src_fh, &hdr);
> +	encode_offload_status(xdr, args, &hdr);
> +	encode_nops(&hdr);
> +}
> +
> +/*
>   * Encode DEALLOCATE request
>   */
>  static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,
> @@ -406,6 +443,31 @@ static int decode_copy(struct xdr_stream *xdr, struct nfs42_copy_res *res)
>  	return decode_copy_requirements(xdr, res);
>  }
>  
> +static int decode_offload_status(struct xdr_stream *xdr,
> +				 struct nfs42_offload_status_res *res)
> +{
> +	__be32 *p;
> +	uint32_t count;
> +	int status;
> +
> +	status = decode_op_hdr(xdr, OP_OFFLOAD_STATUS);
> +	if (status)
> +		return status;
> +	p = xdr_inline_decode(xdr, 8 + 4);
> +	if (unlikely(!p))
> +		goto out_overflow;
> +	p = xdr_decode_hyper(p, &res->osr_count);
> +	count = be32_to_cpup(p++);
> +	if (count) {
> +		p = xdr_inline_decode(xdr, 4);
> +		res->osr_status = nfs4_stat_to_errno(be32_to_cpup(p));
> +	}
> +	return 0;
> +out_overflow:
> +	print_overflow_msg(__func__, xdr);
> +	return -EIO;
> +}
> +
>  static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *res)
>  {
>  	return decode_op_hdr(xdr, OP_DEALLOCATE);
> @@ -504,6 +566,31 @@ static int nfs4_xdr_dec_copy(struct rpc_rqst *rqstp,
>  }
>  
>  /*
> + * Decode OFFLOAD_STATUS response
> + */
> +static int nfs4_xdr_dec_offload_status(struct rpc_rqst *rqstp,
> +				       struct xdr_stream *xdr,
> +				       struct nfs42_offload_status_res *res)

And this one should be "void *data"

Thanks,
Anna

> +{
> +	struct compound_hdr hdr;
> +	int status;
> +
> +	status = decode_compound_hdr(xdr, &hdr);
> +	if (status)
> +		goto out;
> +	status = decode_sequence(xdr, &res->osr_seq_res, rqstp);
> +	if (status)
> +		goto out;
> +	status = decode_putfh(xdr);
> +	if (status)
> +		goto out;
> +	status = decode_offload_status(xdr, res);
> +
> +out:
> +	return status;
> +}
> +
> +/*
>   * Decode DEALLOCATE request
>   */
>  static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,
> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
> index dbfa189..90be5c1 100644
> --- a/fs/nfs/nfs4proc.c
> +++ b/fs/nfs/nfs4proc.c
> @@ -9229,6 +9229,7 @@ static bool nfs4_match_stateid(const nfs4_stateid *s1,
>  		| NFS_CAP_ATOMIC_OPEN_V1
>  		| NFS_CAP_ALLOCATE
>  		| NFS_CAP_COPY
> +		| NFS_CAP_OFFLOAD_STATUS
>  		| NFS_CAP_DEALLOCATE
>  		| NFS_CAP_SEEK
>  		| NFS_CAP_LAYOUTSTATS
> diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
> index 3aebfdc..8b41f43 100644
> --- a/fs/nfs/nfs4xdr.c
> +++ b/fs/nfs/nfs4xdr.c
> @@ -7561,6 +7561,7 @@ struct rpc_procinfo	nfs4_procedures[] = {
>  	PROC(LAYOUTSTATS,	enc_layoutstats,	dec_layoutstats),
>  	PROC(CLONE,		enc_clone,		dec_clone),
>  	PROC(COPY,		enc_copy,		dec_copy),
> +	PROC(OFFLOAD_STATUS,	enc_offload_status,	dec_offload_status),
>  #endif /* CONFIG_NFS_V4_2 */
>  };
>  
> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
> index 1b1ca04..d67c350 100644
> --- a/include/linux/nfs4.h
> +++ b/include/linux/nfs4.h
> @@ -523,6 +523,7 @@ enum {
>  	NFSPROC4_CLNT_LAYOUTSTATS,
>  	NFSPROC4_CLNT_CLONE,
>  	NFSPROC4_CLNT_COPY,
> +	NFSPROC4_CLNT_OFFLOAD_STATUS,
>  };
>  
>  /* nfs41 types */
> diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
> index e418a10..36535ad 100644
> --- a/include/linux/nfs_fs_sb.h
> +++ b/include/linux/nfs_fs_sb.h
> @@ -250,5 +250,6 @@ struct nfs_server {
>  #define NFS_CAP_LAYOUTSTATS	(1U << 22)
>  #define NFS_CAP_CLONE		(1U << 23)
>  #define NFS_CAP_COPY		(1U << 24)
> +#define NFS_CAP_OFFLOAD_STATUS	(1U << 25)
>  
>  #endif
> diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
> index b28c834..eaaf540 100644
> --- a/include/linux/nfs_xdr.h
> +++ b/include/linux/nfs_xdr.h
> @@ -1386,6 +1386,18 @@ struct nfs42_copy_res {
>  	struct nfs_commitres		commit_res;
>  };
>  
> +struct nfs42_offload_status_args {
> +	struct nfs4_sequence_args	osa_seq_args;
> +	struct nfs_fh			*osa_src_fh;
> +	nfs4_stateid			osa_stateid;
> +};
> +
> +struct nfs42_offload_status_res {
> +	struct nfs4_sequence_res	osr_seq_res;
> +	uint64_t			osr_count;
> +	int				osr_status;
> +};
> +
>  struct nfs42_seek_args {
>  	struct nfs4_sequence_args	seq_args;
>  
> 

Re: [RFC v3 05/42] NFS OFFLOAD_STATUS op

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfs/nfs42.h     |  5 ++++-
>  fs/nfs/nfs42proc.c | 42 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 46 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/nfs/nfs42.h b/fs/nfs/nfs42.h
> index b6cd153..54b9941 100644
> --- a/fs/nfs/nfs42.h
> +++ b/fs/nfs/nfs42.h
> @@ -12,6 +12,7 @@
>  #define PNFS_LAYOUTSTATS_MAXDEV (4)
>  
>  /* nfs4.2proc.c */
> +#if defined(CONFIG_NFS_V4_2)

This is just me being nitpicky.  Can you move the "#if defined()" check to before
the "/* nfs4.2proc.c */" comment?  I think it would make the code easier to follow,
especially if we ever add more v4.2 files.

Thanks,
Anna

>  int nfs42_proc_allocate(struct file *, loff_t, loff_t);
>  ssize_t nfs42_proc_copy(struct file *, loff_t, struct file *, loff_t, size_t);
>  int nfs42_proc_deallocate(struct file *, loff_t, loff_t);
> @@ -19,5 +20,7 @@
>  int nfs42_proc_layoutstats_generic(struct nfs_server *,
>  				   struct nfs42_layoutstat_data *);
>  int nfs42_proc_clone(struct file *, struct file *, loff_t, loff_t, loff_t);
> -
> +int nfs42_proc_offload_status(struct file *, nfs4_stateid *,
> +			      struct nfs42_offload_status_res *);
> +#endif /* CONFIG_NFS_V4_2) */
>  #endif /* __LINUX_FS_NFS_NFS4_2_H */
> diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
> index 6c2db51..5c42e71 100644
> --- a/fs/nfs/nfs42proc.c
> +++ b/fs/nfs/nfs42proc.c
> @@ -263,6 +263,48 @@ ssize_t nfs42_proc_copy(struct file *src, loff_t pos_src,
>  	return err;
>  }
>  
> +int _nfs42_proc_offload_status(struct file *dst, nfs4_stateid *stateid,
> +				struct nfs42_offload_status_res *res)
> +{
> +	struct nfs42_offload_status_args args;
> +	struct nfs_server *dst_server = NFS_SERVER(file_inode(dst));
> +	struct rpc_message msg = {
> +		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_OFFLOAD_STATUS],
> +		.rpc_resp = res,
> +		.rpc_argp = &args,
> +	};
> +	int status;
> +
> +	args.osa_src_fh = NFS_FH(file_inode(dst));
> +	memcpy(&args.osa_stateid, stateid, sizeof(args.osa_stateid));
> +	status = nfs4_call_sync(dst_server->client, dst_server, &msg,
> +				&args.osa_seq_args, &res->osr_seq_res, 0);
> +	if (status == -ENOTSUPP)
> +		dst_server->caps &= ~NFS_CAP_OFFLOAD_STATUS;
> +
> +	return status;
> +}
> +
> +int nfs42_proc_offload_status(struct file *dst, nfs4_stateid *stateid,
> +				struct nfs42_offload_status_res *res)
> +{
> +	struct nfs_server *dst_server = NFS_SERVER(file_inode(dst));
> +	struct nfs4_exception exception = { };
> +	int status;
> +
> +	if (!(dst_server->caps & NFS_CAP_OFFLOAD_STATUS))
> +		return -EOPNOTSUPP;
> +
> +	do {
> +		status = _nfs42_proc_offload_status(dst, stateid, res);
> +		if (status == -ENOTSUPP)
> +			return -EOPNOTSUPP;
> +		status = nfs4_handle_exception(dst_server, status, &exception);
> +	} while (exception.retry);
> +
> +	return status;
> +}
> +
>  static loff_t _nfs42_proc_llseek(struct file *filep,
>  		struct nfs_lock_context *lock, loff_t offset, int whence)
>  {
> 

Re: [RFC v3 10/42] NFS export nfs4_async_handle_error

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> Make this function available to nfs42proc.c

I'm confused about why this patch is needed.  Aren't NFS v4.2 and nfs4_async_handle_error()
both built into the NFSv4.ko module?

Anna

> 
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfs/nfs4_fs.h  | 3 +++
>  fs/nfs/nfs4proc.c | 3 ++-
>  2 files changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
> index af285cc..f5d014e 100644
> --- a/fs/nfs/nfs4_fs.h
> +++ b/fs/nfs/nfs4_fs.h
> @@ -244,6 +244,9 @@ int nfs4_replace_transport(struct nfs_server *server,
>  
>  /* nfs4proc.c */
>  extern int nfs4_handle_exception(struct nfs_server *, int, struct nfs4_exception *);
> +extern int nfs4_async_handle_error(struct rpc_task *task,
> +				   struct nfs_server *server,
> +				   struct nfs4_state *state, long *timeout);
>  extern int nfs4_call_sync(struct rpc_clnt *, struct nfs_server *,
>  			  struct rpc_message *, struct nfs4_sequence_args *,
>  			  struct nfs4_sequence_res *, int);
> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
> index b37e528..fdb6b76 100644
> --- a/fs/nfs/nfs4proc.c
> +++ b/fs/nfs/nfs4proc.c
> @@ -548,7 +548,7 @@ int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_
>  	return ret;
>  }
>  
> -static int
> +int
>  nfs4_async_handle_error(struct rpc_task *task, struct nfs_server *server,
>  			struct nfs4_state *state, long *timeout)
>  {
> @@ -569,6 +569,7 @@ int nfs4_handle_exception(struct nfs_server *server, int errorcode, struct nfs4_
>  		return -EAGAIN;
>  	return 0;
>  }
> +EXPORT_SYMBOL_GPL(nfs4_async_handle_error);
>  
>  /*
>   * Return 'true' if 'clp' is using an rpc_client that is integrity protected
> 

Re: [RFC v3 11/42] NFS test for intra vs inter COPY

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfs/internal.h | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
> index 3e24392..dcbcffb 100644
> --- a/fs/nfs/internal.h
> +++ b/fs/nfs/internal.h
> @@ -7,6 +7,7 @@
>  #include <linux/security.h>
>  #include <linux/crc32.h>
>  #include <linux/nfs_page.h>
> +#include <linux/sunrpc/addr.h>
>  
>  #define NFS_MS_MASK (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_SYNCHRONOUS)
>  
> @@ -766,3 +767,12 @@ static inline bool nfs_error_is_fatal(int err)
>  		return false;
>  	}
>  }
> +
> +static inline bool nfs42_files_from_same_server(struct file *in, struct file *out)
> +{
> +	struct nfs_client *c_in = (NFS_SERVER(file_inode(in)))->nfs_client;
> +	struct nfs_client *c_out = (NFS_SERVER(file_inode(out)))->nfs_client;
> +
> +	return rpc_cmp_addr((struct sockaddr *)&c_in->cl_addr,
> +				(struct sockaddr *)&c_out->cl_addr);
> +}

Can you move this to nfs42.h instead of internal.h?

Thanks,
Anna

> 

Re: [RFC v3 14/42] NFS if we got partial copy ignore errors

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfs/nfs42proc.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
> index dba2b62..dfd43bc 100644
> --- a/fs/nfs/nfs42proc.c
> +++ b/fs/nfs/nfs42proc.c
> @@ -179,7 +179,8 @@ static int handle_async_copy(struct nfs42_copy_res *res,
>  out:
>  	*ret_count = copy->count;
>  	memcpy(&res->write_res.verifier, &copy->verf, sizeof(copy->verf));
> -	status = -copy->error;
> +	if (copy->count <= 0)
> +		status = -copy->error;

This is a pretty small change.  Can you squash this patch into patch 8
("NFS add support for asynchronous COPY") that way the code is correct from
the time it's added?

Thanks,
Anna

>  
>  	kfree(copy);
>  	return status;
> 

Re: [RFC v3 23/42] NFS add a simple sync nfs4_proc_commit after async COPY

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
> A COPY with unstable write data needs a simple commit that doesnt
> deal with inodes
> 
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfs/nfs42proc.c | 22 ++++++++++++++++++++++
>  fs/nfs/nfs4_fs.h   |  2 +-
>  fs/nfs/nfs4proc.c  | 33 +++++++++++++++++++++++++++++++++
>  3 files changed, 56 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
> index 09f653d..590bd50 100644
> --- a/fs/nfs/nfs42proc.c
> +++ b/fs/nfs/nfs42proc.c
> @@ -289,6 +289,28 @@ static ssize_t _nfs42_proc_copy(struct file *src,
>  			return status;
>  	}
>  
> +	if ((!res->synchronous || !args->sync) &&
> +			res->write_res.verifier.committed != NFS_FILE_SYNC) {
> +		struct nfs_commitres cres;
> +
> +		cres.verf = kzalloc(sizeof(struct nfs_writeverf), GFP_NOFS);
> +		if (!cres.verf)
> +			return -ENOMEM;
> +
> +		status = nfs4_proc_commit(dst, pos_dst, res->write_res.count,
> +					  &cres);
> +		if (status) {
> +			kfree(cres.verf);
> +			return status;
> +		}
> +		if (!nfs_write_verifier_cmp(&res->write_res.verifier.verifier,
> +					    &cres.verf->verifier)) {
> +			/* what are we suppose to do here ? */
> +			dprintk("commit verf differs from copy verf\n");
> +		}
> +		kfree(cres.verf);
> +	}
> +
>  	truncate_pagecache_range(dst_inode, pos_dst,
>  				 pos_dst + res->write_res.count);
>  
> diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
> index 4ca9657..1b817b4 100644
> --- a/fs/nfs/nfs4_fs.h
> +++ b/fs/nfs/nfs4_fs.h
> @@ -477,7 +477,7 @@ extern int nfs4_sequence_done(struct rpc_task *task,
>  			      struct nfs4_sequence_res *res);
>  
>  extern void nfs4_free_lock_state(struct nfs_server *server, struct nfs4_lock_state *lsp);
> -
> +extern int nfs4_proc_commit(struct file *dst, __u64 offset, __u32 count, struct nfs_commitres *res);
>  extern const nfs4_stateid zero_stateid;
>  
>  /* nfs4super.c */
> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
> index ecaadb0..c477333 100644
> --- a/fs/nfs/nfs4proc.c
> +++ b/fs/nfs/nfs4proc.c
> @@ -4749,6 +4749,39 @@ static void nfs4_proc_commit_setup(struct nfs_commit_data *data, struct rpc_mess
>  	nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1);
>  }
>  
> +static int _nfs4_proc_commit(struct file *dst, struct nfs_commitargs *args,
> +				struct nfs_commitres *res)
> +{
> +	struct inode *dst_inode = file_inode(dst);
> +	struct nfs_server *server = NFS_SERVER(dst_inode);
> +	struct rpc_message msg = {
> +		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_COMMIT],
> +		.rpc_argp = args,
> +		.rpc_resp = res,
> +	};
> +	return nfs4_call_sync(server->client, server, &msg,
> +			&args->seq_args, &res->seq_res, 1);
> +}
> +
> +int nfs4_proc_commit(struct file *dst, __u64 offset, __u32 count, struct nfs_commitres *res)
> +{
> +	struct nfs_commitargs args = {
> +		.fh = NFS_FH(file_inode(dst)),
> +		.offset = offset,
> +		.count = count,
> +	};
> +	struct nfs_server *dst_server = NFS_SERVER(file_inode(dst));
> +	struct nfs4_exception exception = { };
> +	int status;
> +
> +	do {
> +		status = _nfs4_proc_commit(dst, &args, res);
                ^^^^^^
Do you want to check the status here before continuing on to nfs4_handle_exception()?

Thanks,
Anna

> +		status = nfs4_handle_exception(dst_server, status, &exception);
> +	} while (exception.retry);
> +
> +	return status;
> +}
> +
>  struct nfs4_renewdata {
>  	struct nfs_client	*client;
>  	unsigned long		timestamp;
> 

Re: [RFC v3 10/42] NFS export nfs4_async_handle_error

[Olga Kornievskaia]

> On Jul 12, 2017, at 9:56 AM, Anna Schumaker <schumaker.anna@gmail.com> =
wrote:
>=20
> Hi Olga,
>=20
> On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
>> Make this function available to nfs42proc.c
>=20
> I'm confused about why this patch is needed.  Aren't NFS v4.2 and =
nfs4_async_handle_error()
> both built into the NFSv4.ko module?
>=20

You are right I don=E2=80=99t need the EXPORT_SYMBOL chunk but I need =
the function to be callable from outside of nfs4proc.c
=20
> Anna
>=20
>>=20
>> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
>> ---
>> fs/nfs/nfs4_fs.h  | 3 +++
>> fs/nfs/nfs4proc.c | 3 ++-
>> 2 files changed, 5 insertions(+), 1 deletion(-)
>>=20
>> diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
>> index af285cc..f5d014e 100644
>> --- a/fs/nfs/nfs4_fs.h
>> +++ b/fs/nfs/nfs4_fs.h
>> @@ -244,6 +244,9 @@ int nfs4_replace_transport(struct nfs_server =
*server,
>>=20
>> /* nfs4proc.c */
>> extern int nfs4_handle_exception(struct nfs_server *, int, struct =
nfs4_exception *);
>> +extern int nfs4_async_handle_error(struct rpc_task *task,
>> +				   struct nfs_server *server,
>> +				   struct nfs4_state *state, long =
*timeout);
>> extern int nfs4_call_sync(struct rpc_clnt *, struct nfs_server *,
>> 			  struct rpc_message *, struct =
nfs4_sequence_args *,
>> 			  struct nfs4_sequence_res *, int);
>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>> index b37e528..fdb6b76 100644
>> --- a/fs/nfs/nfs4proc.c
>> +++ b/fs/nfs/nfs4proc.c
>> @@ -548,7 +548,7 @@ int nfs4_handle_exception(struct nfs_server =
*server, int errorcode, struct nfs4_
>> 	return ret;
>> }
>>=20
>> -static int
>> +int
>> nfs4_async_handle_error(struct rpc_task *task, struct nfs_server =
*server,
>> 			struct nfs4_state *state, long *timeout)
>> {
>> @@ -569,6 +569,7 @@ int nfs4_handle_exception(struct nfs_server =
*server, int errorcode, struct nfs4_
>> 		return -EAGAIN;
>> 	return 0;
>> }
>> +EXPORT_SYMBOL_GPL(nfs4_async_handle_error);
>>=20
>> /*
>>  * Return 'true' if 'clp' is using an rpc_client that is integrity =
protected
>>=20

Re: [RFC v3 23/42] NFS add a simple sync nfs4_proc_commit after async COPY

[Olga Kornievskaia]

> On Jul 12, 2017, at 1:13 PM, Anna Schumaker <schumaker.anna@gmail.com> =
wrote:
>=20
> Hi Olga,
>=20
> On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
>> A COPY with unstable write data needs a simple commit that doesnt
>> deal with inodes
>>=20
>> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
>> ---
>> fs/nfs/nfs42proc.c | 22 ++++++++++++++++++++++
>> fs/nfs/nfs4_fs.h   |  2 +-
>> fs/nfs/nfs4proc.c  | 33 +++++++++++++++++++++++++++++++++
>> 3 files changed, 56 insertions(+), 1 deletion(-)
>>=20
>> diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
>> index 09f653d..590bd50 100644
>> --- a/fs/nfs/nfs42proc.c
>> +++ b/fs/nfs/nfs42proc.c
>> @@ -289,6 +289,28 @@ static ssize_t _nfs42_proc_copy(struct file =
*src,
>> 			return status;
>> 	}
>>=20
>> +	if ((!res->synchronous || !args->sync) &&
>> +			res->write_res.verifier.committed !=3D =
NFS_FILE_SYNC) {
>> +		struct nfs_commitres cres;
>> +
>> +		cres.verf =3D kzalloc(sizeof(struct nfs_writeverf), =
GFP_NOFS);
>> +		if (!cres.verf)
>> +			return -ENOMEM;
>> +
>> +		status =3D nfs4_proc_commit(dst, pos_dst, =
res->write_res.count,
>> +					  &cres);
>> +		if (status) {
>> +			kfree(cres.verf);
>> +			return status;
>> +		}
>> +		if =
(!nfs_write_verifier_cmp(&res->write_res.verifier.verifier,
>> +					    &cres.verf->verifier)) {
>> +			/* what are we suppose to do here ? */
>> +			dprintk("commit verf differs from copy verf\n");
>> +		}
>> +		kfree(cres.verf);
>> +	}
>> +
>> 	truncate_pagecache_range(dst_inode, pos_dst,
>> 				 pos_dst + res->write_res.count);
>>=20
>> diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
>> index 4ca9657..1b817b4 100644
>> --- a/fs/nfs/nfs4_fs.h
>> +++ b/fs/nfs/nfs4_fs.h
>> @@ -477,7 +477,7 @@ extern int nfs4_sequence_done(struct rpc_task =
*task,
>> 			      struct nfs4_sequence_res *res);
>>=20
>> extern void nfs4_free_lock_state(struct nfs_server *server, struct =
nfs4_lock_state *lsp);
>> -
>> +extern int nfs4_proc_commit(struct file *dst, __u64 offset, __u32 =
count, struct nfs_commitres *res);
>> extern const nfs4_stateid zero_stateid;
>>=20
>> /* nfs4super.c */
>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>> index ecaadb0..c477333 100644
>> --- a/fs/nfs/nfs4proc.c
>> +++ b/fs/nfs/nfs4proc.c
>> @@ -4749,6 +4749,39 @@ static void nfs4_proc_commit_setup(struct =
nfs_commit_data *data, struct rpc_mess
>> 	nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1);
>> }
>>=20
>> +static int _nfs4_proc_commit(struct file *dst, struct nfs_commitargs =
*args,
>> +				struct nfs_commitres *res)
>> +{
>> +	struct inode *dst_inode =3D file_inode(dst);
>> +	struct nfs_server *server =3D NFS_SERVER(dst_inode);
>> +	struct rpc_message msg =3D {
>> +		.rpc_proc =3D &nfs4_procedures[NFSPROC4_CLNT_COMMIT],
>> +		.rpc_argp =3D args,
>> +		.rpc_resp =3D res,
>> +	};
>> +	return nfs4_call_sync(server->client, server, &msg,
>> +			&args->seq_args, &res->seq_res, 1);
>> +}
>> +
>> +int nfs4_proc_commit(struct file *dst, __u64 offset, __u32 count, =
struct nfs_commitres *res)
>> +{
>> +	struct nfs_commitargs args =3D {
>> +		.fh =3D NFS_FH(file_inode(dst)),
>> +		.offset =3D offset,
>> +		.count =3D count,
>> +	};
>> +	struct nfs_server *dst_server =3D NFS_SERVER(file_inode(dst));
>> +	struct nfs4_exception exception =3D { };
>> +	int status;
>> +
>> +	do {
>> +		status =3D _nfs4_proc_commit(dst, &args, res);
>                ^^^^^^
> Do you want to check the status here before continuing on to =
nfs4_handle_exception()?

No, the exception handler should be called with the status and deal with =
it and the loop exit will be determined by it.=20

>=20
> Thanks,
> Anna
>=20
>> +		status =3D nfs4_handle_exception(dst_server, status, =
&exception);
>> +	} while (exception.retry);
>> +
>> +	return status;
>> +}
>> +
>> struct nfs4_renewdata {
>> 	struct nfs_client	*client;
>> 	unsigned long		timestamp;
>>=20

Re: [RFC v3 14/42] NFS if we got partial copy ignore errors

[Olga Kornievskaia]

> On Jul 12, 2017, at 10:52 AM, Anna Schumaker =
<schumaker.anna@gmail.com> wrote:
>=20
> Hi Olga,
>=20
> On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
>> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
>> ---
>> fs/nfs/nfs42proc.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>=20
>> diff --git a/fs/nfs/nfs42proc.c b/fs/nfs/nfs42proc.c
>> index dba2b62..dfd43bc 100644
>> --- a/fs/nfs/nfs42proc.c
>> +++ b/fs/nfs/nfs42proc.c
>> @@ -179,7 +179,8 @@ static int handle_async_copy(struct =
nfs42_copy_res *res,
>> out:
>> 	*ret_count =3D copy->count;
>> 	memcpy(&res->write_res.verifier, &copy->verf, =
sizeof(copy->verf));
>> -	status =3D -copy->error;
>> +	if (copy->count <=3D 0)
>> +		status =3D -copy->error;
>=20
> This is a pretty small change.  Can you squash this patch into patch 8
> ("NFS add support for asynchronous COPY") that way the code is correct =
from
> the time it's added?

Ok.

>=20
> Thanks,
> Anna
>=20
>>=20
>> 	kfree(copy);
>> 	return status;
>>=20

Re: [RFC v3 11/42] NFS test for intra vs inter COPY

[Olga Kornievskaia]

> On Jul 12, 2017, at 10:06 AM, Anna Schumaker =
<schumaker.anna@gmail.com> wrote:
>=20
> Hi Olga,
>=20
> On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
>> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
>> ---
>> fs/nfs/internal.h | 10 ++++++++++
>> 1 file changed, 10 insertions(+)
>>=20
>> diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
>> index 3e24392..dcbcffb 100644
>> --- a/fs/nfs/internal.h
>> +++ b/fs/nfs/internal.h
>> @@ -7,6 +7,7 @@
>> #include <linux/security.h>
>> #include <linux/crc32.h>
>> #include <linux/nfs_page.h>
>> +#include <linux/sunrpc/addr.h>
>>=20
>> #define NFS_MS_MASK =
(MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_SYNCHRONOUS)
>>=20
>> @@ -766,3 +767,12 @@ static inline bool nfs_error_is_fatal(int err)
>> 		return false;
>> 	}
>> }
>> +
>> +static inline bool nfs42_files_from_same_server(struct file *in, =
struct file *out)
>> +{
>> +	struct nfs_client *c_in =3D =
(NFS_SERVER(file_inode(in)))->nfs_client;
>> +	struct nfs_client *c_out =3D =
(NFS_SERVER(file_inode(out)))->nfs_client;
>> +
>> +	return rpc_cmp_addr((struct sockaddr *)&c_in->cl_addr,
>> +				(struct sockaddr *)&c_out->cl_addr);
>> +}
>=20
> Can you move this to nfs42.h instead of internal.h?
>=20

Ok.

> Thanks,
> Anna
>=20
>>=20

Re: [RFC v3 04/42] NFS OFFLOAD_STATUS xdr

[Olga Kornievskaia]

Is there an ETA on when the casting patches will go in? If it=E2=80=99s =
soon, I would rebase then and fix the new calls.

> On Jul 11, 2017, at 5:01 PM, Anna Schumaker <schumaker.anna@gmail.com> =
wrote:
>=20
> Hi Olga,
>=20
> On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
>> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
>> ---
>> fs/nfs/nfs42xdr.c         | 87 =
+++++++++++++++++++++++++++++++++++++++++++++++
>> fs/nfs/nfs4proc.c         |  1 +
>> fs/nfs/nfs4xdr.c          |  1 +
>> include/linux/nfs4.h      |  1 +
>> include/linux/nfs_fs_sb.h |  1 +
>> include/linux/nfs_xdr.h   | 12 +++++++
>> 6 files changed, 103 insertions(+)
>>=20
>> diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
>> index 528362f..e369de1 100644
>> --- a/fs/nfs/nfs42xdr.c
>> +++ b/fs/nfs/nfs42xdr.c
>> @@ -25,6 +25,11 @@
>> 					 NFS42_WRITE_RES_SIZE + \
>> 					 1 /* cr_consecutive */ + \
>> 					 1 /* cr_synchronous */)
>> +#define encode_offload_status_maxsz	(op_encode_hdr_maxsz + \
>> +					 XDR_QUADLEN(NFS4_STATEID_SIZE))
>> +#define decode_offload_status_maxsz	(op_decode_hdr_maxsz + \
>> +					 2 /* osr_count */ + \
>> +					 1 + 1 /* osr_complete<1> */)
>> #define encode_deallocate_maxsz		(op_encode_hdr_maxsz + \
>> 					 encode_fallocate_maxsz)
>> #define decode_deallocate_maxsz		(op_decode_hdr_maxsz)
>> @@ -74,6 +79,12 @@
>> 					 decode_putfh_maxsz + \
>> 					 decode_copy_maxsz + \
>> 					 decode_commit_maxsz)
>> +#define NFS4_enc_offload_status_sz	(compound_encode_hdr_maxsz + \
>> +					 encode_putfh_maxsz + \
>> +					 encode_offload_status_maxsz)
>> +#define NFS4_dec_offload_status_sz	(compound_decode_hdr_maxsz + \
>> +					 decode_putfh_maxsz + \
>> +					 decode_offload_status_maxsz)
>> #define NFS4_enc_deallocate_sz		=
(compound_encode_hdr_maxsz + \
>> 					 encode_putfh_maxsz + \
>> 					 encode_deallocate_maxsz + \
>> @@ -144,6 +155,14 @@ static void encode_copy(struct xdr_stream *xdr,
>> 	encode_uint32(xdr, 0); /* src server list */
>> }
>>=20
>> +static void encode_offload_status(struct xdr_stream *xdr,
>> +				  struct nfs42_offload_status_args =
*args,
>> +				  struct compound_hdr *hdr)
>> +{
>> +	encode_op_hdr(xdr, OP_OFFLOAD_STATUS, =
decode_offload_status_maxsz, hdr);
>> +	encode_nfs4_stateid(xdr, &args->osa_stateid);
>> +}
>> +
>> static void encode_deallocate(struct xdr_stream *xdr,
>> 			      struct nfs42_falloc_args *args,
>> 			      struct compound_hdr *hdr)
>> @@ -258,6 +277,24 @@ static void nfs4_xdr_enc_copy(struct rpc_rqst =
*req,
>> }
>>=20
>> /*
>> + * Encode OFFLOAD_STATUS request
>> + */
>> +static void nfs4_xdr_enc_offload_status(struct rpc_rqst *req,
>> +					struct xdr_stream *xdr,
>> +					struct nfs42_offload_status_args =
*args)
>=20
> This last argument should now be: "const void *data"
>=20
>> +{
>> +	struct compound_hdr hdr =3D {
>> +		.minorversion =3D =
nfs4_xdr_minorversion(&args->osa_seq_args),
>> +	};
>> +
>> +	encode_compound_hdr(xdr, req, &hdr);
>> +	encode_sequence(xdr, &args->osa_seq_args, &hdr);
>> +	encode_putfh(xdr, args->osa_src_fh, &hdr);
>> +	encode_offload_status(xdr, args, &hdr);
>> +	encode_nops(&hdr);
>> +}
>> +
>> +/*
>>  * Encode DEALLOCATE request
>>  */
>> static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,
>> @@ -406,6 +443,31 @@ static int decode_copy(struct xdr_stream *xdr, =
struct nfs42_copy_res *res)
>> 	return decode_copy_requirements(xdr, res);
>> }
>>=20
>> +static int decode_offload_status(struct xdr_stream *xdr,
>> +				 struct nfs42_offload_status_res *res)
>> +{
>> +	__be32 *p;
>> +	uint32_t count;
>> +	int status;
>> +
>> +	status =3D decode_op_hdr(xdr, OP_OFFLOAD_STATUS);
>> +	if (status)
>> +		return status;
>> +	p =3D xdr_inline_decode(xdr, 8 + 4);
>> +	if (unlikely(!p))
>> +		goto out_overflow;
>> +	p =3D xdr_decode_hyper(p, &res->osr_count);
>> +	count =3D be32_to_cpup(p++);
>> +	if (count) {
>> +		p =3D xdr_inline_decode(xdr, 4);
>> +		res->osr_status =3D nfs4_stat_to_errno(be32_to_cpup(p));
>> +	}
>> +	return 0;
>> +out_overflow:
>> +	print_overflow_msg(__func__, xdr);
>> +	return -EIO;
>> +}
>> +
>> static int decode_deallocate(struct xdr_stream *xdr, struct =
nfs42_falloc_res *res)
>> {
>> 	return decode_op_hdr(xdr, OP_DEALLOCATE);
>> @@ -504,6 +566,31 @@ static int nfs4_xdr_dec_copy(struct rpc_rqst =
*rqstp,
>> }
>>=20
>> /*
>> + * Decode OFFLOAD_STATUS response
>> + */
>> +static int nfs4_xdr_dec_offload_status(struct rpc_rqst *rqstp,
>> +				       struct xdr_stream *xdr,
>> +				       struct nfs42_offload_status_res =
*res)
>=20
> And this one should be "void *data"
>=20
> Thanks,
> Anna
>=20
>> +{
>> +	struct compound_hdr hdr;
>> +	int status;
>> +
>> +	status =3D decode_compound_hdr(xdr, &hdr);
>> +	if (status)
>> +		goto out;
>> +	status =3D decode_sequence(xdr, &res->osr_seq_res, rqstp);
>> +	if (status)
>> +		goto out;
>> +	status =3D decode_putfh(xdr);
>> +	if (status)
>> +		goto out;
>> +	status =3D decode_offload_status(xdr, res);
>> +
>> +out:
>> +	return status;
>> +}
>> +
>> +/*
>>  * Decode DEALLOCATE request
>>  */
>> static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,
>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>> index dbfa189..90be5c1 100644
>> --- a/fs/nfs/nfs4proc.c
>> +++ b/fs/nfs/nfs4proc.c
>> @@ -9229,6 +9229,7 @@ static bool nfs4_match_stateid(const =
nfs4_stateid *s1,
>> 		| NFS_CAP_ATOMIC_OPEN_V1
>> 		| NFS_CAP_ALLOCATE
>> 		| NFS_CAP_COPY
>> +		| NFS_CAP_OFFLOAD_STATUS
>> 		| NFS_CAP_DEALLOCATE
>> 		| NFS_CAP_SEEK
>> 		| NFS_CAP_LAYOUTSTATS
>> diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
>> index 3aebfdc..8b41f43 100644
>> --- a/fs/nfs/nfs4xdr.c
>> +++ b/fs/nfs/nfs4xdr.c
>> @@ -7561,6 +7561,7 @@ struct rpc_procinfo	nfs4_procedures[] =3D {
>> 	PROC(LAYOUTSTATS,	enc_layoutstats,	=
dec_layoutstats),
>> 	PROC(CLONE,		enc_clone,		dec_clone),
>> 	PROC(COPY,		enc_copy,		dec_copy),
>> +	PROC(OFFLOAD_STATUS,	enc_offload_status,	=
dec_offload_status),
>> #endif /* CONFIG_NFS_V4_2 */
>> };
>>=20
>> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
>> index 1b1ca04..d67c350 100644
>> --- a/include/linux/nfs4.h
>> +++ b/include/linux/nfs4.h
>> @@ -523,6 +523,7 @@ enum {
>> 	NFSPROC4_CLNT_LAYOUTSTATS,
>> 	NFSPROC4_CLNT_CLONE,
>> 	NFSPROC4_CLNT_COPY,
>> +	NFSPROC4_CLNT_OFFLOAD_STATUS,
>> };
>>=20
>> /* nfs41 types */
>> diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
>> index e418a10..36535ad 100644
>> --- a/include/linux/nfs_fs_sb.h
>> +++ b/include/linux/nfs_fs_sb.h
>> @@ -250,5 +250,6 @@ struct nfs_server {
>> #define NFS_CAP_LAYOUTSTATS	(1U << 22)
>> #define NFS_CAP_CLONE		(1U << 23)
>> #define NFS_CAP_COPY		(1U << 24)
>> +#define NFS_CAP_OFFLOAD_STATUS	(1U << 25)
>>=20
>> #endif
>> diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
>> index b28c834..eaaf540 100644
>> --- a/include/linux/nfs_xdr.h
>> +++ b/include/linux/nfs_xdr.h
>> @@ -1386,6 +1386,18 @@ struct nfs42_copy_res {
>> 	struct nfs_commitres		commit_res;
>> };
>>=20
>> +struct nfs42_offload_status_args {
>> +	struct nfs4_sequence_args	osa_seq_args;
>> +	struct nfs_fh			*osa_src_fh;
>> +	nfs4_stateid			osa_stateid;
>> +};
>> +
>> +struct nfs42_offload_status_res {
>> +	struct nfs4_sequence_res	osr_seq_res;
>> +	uint64_t			osr_count;
>> +	int				osr_status;
>> +};
>> +
>> struct nfs42_seek_args {
>> 	struct nfs4_sequence_args	seq_args;
>>=20
>>=20

Re: [RFC v3 04/42] NFS OFFLOAD_STATUS xdr

[Anna Schumaker]

On 07/12/2017 01:23 PM, Olga Kornievskaia wrote:
> 
> Is there an ETA on when the casting patches will go in? If it’s soon, I would rebase then and fix the new calls.

It goes in as part of the current merge window, so hold off until 4.13-rc1 :)

> 
>> On Jul 11, 2017, at 5:01 PM, Anna Schumaker <schumaker.anna@gmail.com> wrote:
>>
>> Hi Olga,
>>
>> On 07/11/2017 12:43 PM, Olga Kornievskaia wrote:
>>> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
>>> ---
>>> fs/nfs/nfs42xdr.c         | 87 +++++++++++++++++++++++++++++++++++++++++++++++
>>> fs/nfs/nfs4proc.c         |  1 +
>>> fs/nfs/nfs4xdr.c          |  1 +
>>> include/linux/nfs4.h      |  1 +
>>> include/linux/nfs_fs_sb.h |  1 +
>>> include/linux/nfs_xdr.h   | 12 +++++++
>>> 6 files changed, 103 insertions(+)
>>>
>>> diff --git a/fs/nfs/nfs42xdr.c b/fs/nfs/nfs42xdr.c
>>> index 528362f..e369de1 100644
>>> --- a/fs/nfs/nfs42xdr.c
>>> +++ b/fs/nfs/nfs42xdr.c
>>> @@ -25,6 +25,11 @@
>>> 					 NFS42_WRITE_RES_SIZE + \
>>> 					 1 /* cr_consecutive */ + \
>>> 					 1 /* cr_synchronous */)
>>> +#define encode_offload_status_maxsz	(op_encode_hdr_maxsz + \
>>> +					 XDR_QUADLEN(NFS4_STATEID_SIZE))
>>> +#define decode_offload_status_maxsz	(op_decode_hdr_maxsz + \
>>> +					 2 /* osr_count */ + \
>>> +					 1 + 1 /* osr_complete<1> */)
>>> #define encode_deallocate_maxsz		(op_encode_hdr_maxsz + \
>>> 					 encode_fallocate_maxsz)
>>> #define decode_deallocate_maxsz		(op_decode_hdr_maxsz)
>>> @@ -74,6 +79,12 @@
>>> 					 decode_putfh_maxsz + \
>>> 					 decode_copy_maxsz + \
>>> 					 decode_commit_maxsz)
>>> +#define NFS4_enc_offload_status_sz	(compound_encode_hdr_maxsz + \
>>> +					 encode_putfh_maxsz + \
>>> +					 encode_offload_status_maxsz)
>>> +#define NFS4_dec_offload_status_sz	(compound_decode_hdr_maxsz + \
>>> +					 decode_putfh_maxsz + \
>>> +					 decode_offload_status_maxsz)
>>> #define NFS4_enc_deallocate_sz		(compound_encode_hdr_maxsz + \
>>> 					 encode_putfh_maxsz + \
>>> 					 encode_deallocate_maxsz + \
>>> @@ -144,6 +155,14 @@ static void encode_copy(struct xdr_stream *xdr,
>>> 	encode_uint32(xdr, 0); /* src server list */
>>> }
>>>
>>> +static void encode_offload_status(struct xdr_stream *xdr,
>>> +				  struct nfs42_offload_status_args *args,
>>> +				  struct compound_hdr *hdr)
>>> +{
>>> +	encode_op_hdr(xdr, OP_OFFLOAD_STATUS, decode_offload_status_maxsz, hdr);
>>> +	encode_nfs4_stateid(xdr, &args->osa_stateid);
>>> +}
>>> +
>>> static void encode_deallocate(struct xdr_stream *xdr,
>>> 			      struct nfs42_falloc_args *args,
>>> 			      struct compound_hdr *hdr)
>>> @@ -258,6 +277,24 @@ static void nfs4_xdr_enc_copy(struct rpc_rqst *req,
>>> }
>>>
>>> /*
>>> + * Encode OFFLOAD_STATUS request
>>> + */
>>> +static void nfs4_xdr_enc_offload_status(struct rpc_rqst *req,
>>> +					struct xdr_stream *xdr,
>>> +					struct nfs42_offload_status_args *args)
>>
>> This last argument should now be: "const void *data"
>>
>>> +{
>>> +	struct compound_hdr hdr = {
>>> +		.minorversion = nfs4_xdr_minorversion(&args->osa_seq_args),
>>> +	};
>>> +
>>> +	encode_compound_hdr(xdr, req, &hdr);
>>> +	encode_sequence(xdr, &args->osa_seq_args, &hdr);
>>> +	encode_putfh(xdr, args->osa_src_fh, &hdr);
>>> +	encode_offload_status(xdr, args, &hdr);
>>> +	encode_nops(&hdr);
>>> +}
>>> +
>>> +/*
>>>  * Encode DEALLOCATE request
>>>  */
>>> static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,
>>> @@ -406,6 +443,31 @@ static int decode_copy(struct xdr_stream *xdr, struct nfs42_copy_res *res)
>>> 	return decode_copy_requirements(xdr, res);
>>> }
>>>
>>> +static int decode_offload_status(struct xdr_stream *xdr,
>>> +				 struct nfs42_offload_status_res *res)
>>> +{
>>> +	__be32 *p;
>>> +	uint32_t count;
>>> +	int status;
>>> +
>>> +	status = decode_op_hdr(xdr, OP_OFFLOAD_STATUS);
>>> +	if (status)
>>> +		return status;
>>> +	p = xdr_inline_decode(xdr, 8 + 4);
>>> +	if (unlikely(!p))
>>> +		goto out_overflow;
>>> +	p = xdr_decode_hyper(p, &res->osr_count);
>>> +	count = be32_to_cpup(p++);
>>> +	if (count) {
>>> +		p = xdr_inline_decode(xdr, 4);
>>> +		res->osr_status = nfs4_stat_to_errno(be32_to_cpup(p));
>>> +	}
>>> +	return 0;
>>> +out_overflow:
>>> +	print_overflow_msg(__func__, xdr);
>>> +	return -EIO;
>>> +}
>>> +
>>> static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *res)
>>> {
>>> 	return decode_op_hdr(xdr, OP_DEALLOCATE);
>>> @@ -504,6 +566,31 @@ static int nfs4_xdr_dec_copy(struct rpc_rqst *rqstp,
>>> }
>>>
>>> /*
>>> + * Decode OFFLOAD_STATUS response
>>> + */
>>> +static int nfs4_xdr_dec_offload_status(struct rpc_rqst *rqstp,
>>> +				       struct xdr_stream *xdr,
>>> +				       struct nfs42_offload_status_res *res)
>>
>> And this one should be "void *data"
>>
>> Thanks,
>> Anna
>>
>>> +{
>>> +	struct compound_hdr hdr;
>>> +	int status;
>>> +
>>> +	status = decode_compound_hdr(xdr, &hdr);
>>> +	if (status)
>>> +		goto out;
>>> +	status = decode_sequence(xdr, &res->osr_seq_res, rqstp);
>>> +	if (status)
>>> +		goto out;
>>> +	status = decode_putfh(xdr);
>>> +	if (status)
>>> +		goto out;
>>> +	status = decode_offload_status(xdr, res);
>>> +
>>> +out:
>>> +	return status;
>>> +}
>>> +
>>> +/*
>>>  * Decode DEALLOCATE request
>>>  */
>>> static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,
>>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>>> index dbfa189..90be5c1 100644
>>> --- a/fs/nfs/nfs4proc.c
>>> +++ b/fs/nfs/nfs4proc.c
>>> @@ -9229,6 +9229,7 @@ static bool nfs4_match_stateid(const nfs4_stateid *s1,
>>> 		| NFS_CAP_ATOMIC_OPEN_V1
>>> 		| NFS_CAP_ALLOCATE
>>> 		| NFS_CAP_COPY
>>> +		| NFS_CAP_OFFLOAD_STATUS
>>> 		| NFS_CAP_DEALLOCATE
>>> 		| NFS_CAP_SEEK
>>> 		| NFS_CAP_LAYOUTSTATS
>>> diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c
>>> index 3aebfdc..8b41f43 100644
>>> --- a/fs/nfs/nfs4xdr.c
>>> +++ b/fs/nfs/nfs4xdr.c
>>> @@ -7561,6 +7561,7 @@ struct rpc_procinfo	nfs4_procedures[] = {
>>> 	PROC(LAYOUTSTATS,	enc_layoutstats,	dec_layoutstats),
>>> 	PROC(CLONE,		enc_clone,		dec_clone),
>>> 	PROC(COPY,		enc_copy,		dec_copy),
>>> +	PROC(OFFLOAD_STATUS,	enc_offload_status,	dec_offload_status),
>>> #endif /* CONFIG_NFS_V4_2 */
>>> };
>>>
>>> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
>>> index 1b1ca04..d67c350 100644
>>> --- a/include/linux/nfs4.h
>>> +++ b/include/linux/nfs4.h
>>> @@ -523,6 +523,7 @@ enum {
>>> 	NFSPROC4_CLNT_LAYOUTSTATS,
>>> 	NFSPROC4_CLNT_CLONE,
>>> 	NFSPROC4_CLNT_COPY,
>>> +	NFSPROC4_CLNT_OFFLOAD_STATUS,
>>> };
>>>
>>> /* nfs41 types */
>>> diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
>>> index e418a10..36535ad 100644
>>> --- a/include/linux/nfs_fs_sb.h
>>> +++ b/include/linux/nfs_fs_sb.h
>>> @@ -250,5 +250,6 @@ struct nfs_server {
>>> #define NFS_CAP_LAYOUTSTATS	(1U << 22)
>>> #define NFS_CAP_CLONE		(1U << 23)
>>> #define NFS_CAP_COPY		(1U << 24)
>>> +#define NFS_CAP_OFFLOAD_STATUS	(1U << 25)
>>>
>>> #endif
>>> diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
>>> index b28c834..eaaf540 100644
>>> --- a/include/linux/nfs_xdr.h
>>> +++ b/include/linux/nfs_xdr.h
>>> @@ -1386,6 +1386,18 @@ struct nfs42_copy_res {
>>> 	struct nfs_commitres		commit_res;
>>> };
>>>
>>> +struct nfs42_offload_status_args {
>>> +	struct nfs4_sequence_args	osa_seq_args;
>>> +	struct nfs_fh			*osa_src_fh;
>>> +	nfs4_stateid			osa_stateid;
>>> +};
>>> +
>>> +struct nfs42_offload_status_res {
>>> +	struct nfs4_sequence_res	osr_seq_res;
>>> +	uint64_t			osr_count;
>>> +	int				osr_status;
>>> +};
>>> +
>>> struct nfs42_seek_args {
>>> 	struct nfs4_sequence_args	seq_args;
>>>
>>>
> 

Re: [RFC v3 32/42] NFSD OFFLOAD_STATUS xdr

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:44 PM, Olga Kornievskaia wrote:
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfsd/nfs4proc.c | 20 ++++++++++++++++++++
>  fs/nfsd/nfs4xdr.c  | 30 ++++++++++++++++++++++++++++--
>  fs/nfsd/xdr4.h     | 10 ++++++++++
>  3 files changed, 58 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 3d8ed3f..6d20ee1 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -1468,6 +1468,13 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
>  	fput(file);
>  	return status;
>  }
> +static __be32
> +nfsd4_offload_status(struct svc_rqst *rqstp,
> +		     struct nfsd4_compound_state *cstate,
> +		     struct nfsd4_offload_status *os)
> +{
> +	return nfserr_notsupp;
> +}

One small comment on the placement of this function: I think it should go above nfsd4_fallocate() to keep fallocate / allocate / deallocate together.

Thanks!
Anna

>  
>  static __be32
>  nfsd4_allocate(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> @@ -2436,6 +2443,14 @@ static inline u32 nfsd4_copy_notify_rsize(struct svc_rqst *rqstp,
>  		* sizeof(__be32);
>  }
>  
> +static inline u32 nfsd4_offload_status_rsize(struct svc_rqst *rqstp,
> +					     struct nfsd4_op *op)
> +{
> +	return (op_encode_hdr_size +
> +		2 /* osr_count */ +
> +		1 /* osr_complete<1> optional 0 for now */) * sizeof(__be32);
> +}
> +
>  #ifdef CONFIG_NFSD_PNFS
>  static inline u32 nfsd4_getdeviceinfo_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>  {
> @@ -2846,6 +2861,11 @@ static inline u32 nfsd4_seek_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>  		.op_name = "OP_COPY_NOTIFY",
>  		.op_rsize_bop = (nfsd4op_rsize)nfsd4_copy_notify_rsize,
>  	},
> +	[OP_OFFLOAD_STATUS] = {
> +		.op_func = (nfsd4op_func)nfsd4_offload_status,
> +		.op_name = "OP_OFFLOAD_STATUS",
> +		.op_rsize_bop = (nfsd4op_rsize)nfsd4_offload_status_rsize,
> +	},
>  };
>  
>  /**
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index c4438ac..f032178 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1836,6 +1836,13 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
>  }
>  
>  static __be32
> +nfsd4_decode_offload_status(struct nfsd4_compoundargs *argp,
> +			    struct nfsd4_offload_status *os)
> +{
> +	return nfsd4_decode_stateid(argp, &os->stateid);
> +}
> +
> +static __be32
>  nfsd4_decode_seek(struct nfsd4_compoundargs *argp, struct nfsd4_seek *seek)
>  {
>  	DECODE_HEAD;
> @@ -1942,7 +1949,7 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
>  	[OP_LAYOUTERROR]	= (nfsd4_dec)nfsd4_decode_notsupp,
>  	[OP_LAYOUTSTATS]	= (nfsd4_dec)nfsd4_decode_notsupp,
>  	[OP_OFFLOAD_CANCEL]	= (nfsd4_dec)nfsd4_decode_notsupp,
> -	[OP_OFFLOAD_STATUS]	= (nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_OFFLOAD_STATUS]	= (nfsd4_dec)nfsd4_decode_offload_status,
>  	[OP_READ_PLUS]		= (nfsd4_dec)nfsd4_decode_notsupp,
>  	[OP_SEEK]		= (nfsd4_dec)nfsd4_decode_seek,
>  	[OP_WRITE_SAME]		= (nfsd4_dec)nfsd4_decode_notsupp,
> @@ -4483,6 +4490,25 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>  }
>  
>  static __be32
> +nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr,
> +			    struct nfsd4_offload_status *os)
> +{
> +	struct xdr_stream *xdr = &resp->xdr;
> +	__be32 *p;
> +
> +	if (nfserr)
> +		return nfserr;
> +
> +	p = xdr_reserve_space(xdr, 8 + 4);
> +	if (!p)
> +		return nfserr_resource;
> +	p = xdr_encode_hyper(p, os->count);
> +	*p++ = cpu_to_be32(0);
> +
> +	return nfserr;
> +}
> +
> +static __be32
>  nfsd4_encode_seek(struct nfsd4_compoundres *resp, __be32 nfserr,
>  		  struct nfsd4_seek *seek)
>  {
> @@ -4588,7 +4614,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>  	[OP_LAYOUTERROR]	= (nfsd4_enc)nfsd4_encode_noop,
>  	[OP_LAYOUTSTATS]	= (nfsd4_enc)nfsd4_encode_noop,
>  	[OP_OFFLOAD_CANCEL]	= (nfsd4_enc)nfsd4_encode_noop,
> -	[OP_OFFLOAD_STATUS]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_OFFLOAD_STATUS]	= (nfsd4_enc)nfsd4_encode_offload_status,
>  	[OP_READ_PLUS]		= (nfsd4_enc)nfsd4_encode_noop,
>  	[OP_SEEK]		= (nfsd4_enc)nfsd4_encode_seek,
>  	[OP_WRITE_SAME]		= (nfsd4_enc)nfsd4_encode_noop,
> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
> index e6fa914..c3e6907 100644
> --- a/fs/nfsd/xdr4.h
> +++ b/fs/nfsd/xdr4.h
> @@ -561,6 +561,15 @@ struct nfsd4_copy_notify {
>  	struct nl4_servers	cpn_src;
>  };
>  
> +struct nfsd4_offload_status {
> +	/* request */
> +	stateid_t	stateid;
> +
> +	/* response */
> +	u64		count;
> +	u32		status;
> +};
> +
>  struct nfsd4_op {
>  	int					opnum;
>  	__be32					status;
> @@ -617,6 +626,7 @@ struct nfsd4_op {
>  		struct nfsd4_clone		clone;
>  		struct nfsd4_copy		copy;
>  		struct nfsd4_copy_notify	copy_notify;
> +		struct nfsd4_offload_status	offload_status;
>  		struct nfsd4_seek		seek;
>  	} u;
>  	struct nfs4_replay *			replay;
> 

Re: [RFC v3 35/42] NFSD first draft of async copy

[Anna Schumaker]

Hi Olga,

On 07/11/2017 12:44 PM, Olga Kornievskaia wrote:
> Upong receiving copy request for the inter copy, nfsd will
> establish the mount to the source server.
> 
> Asynchronous copies are handled by a single threaded workqueue.
> If we get asynchronous request, make sure to copy the needed
> arguments/state from the stack before starting the copy. Then
> queue work and reply back to the client indicating copy is
> asynchronous.
> 
> nfsd_copy_file_range() will copy in 4MBchunk so do a loop over
> the total number of bytes need to copy. In case a failure
> happens in the middle, we can return an error as well as how
> much we copied so far. Once done creating a workitem for the
> callback workqueue and send CB_OFFLOAD with the results.

This doesn't compile if I set CONFIG_NFSD_V4_2_INTER_SSC=n:

fs/nfsd/nfs4proc.c: In function 'nfsd4_do_async_copy':           
fs/nfsd/nfs4proc.c:1439:18: error: implicit declaration of function 'nfs42_ssc_open'; did you mean 'nfsd4_open'? [-Werror=implicit-function-declaration]
   copy->fh_src = nfs42_ssc_open(copy->ss_mnt, &copy->c_fh,      
                  ^~~~~~~~~~~~~~                                 
                  nfsd4_open    
fs/nfsd/nfs4proc.c:1439:16: error: assignment makes pointer from integer without a cast [-Werror=int-conversion]                  
   copy->fh_src = nfs42_ssc_open(copy->ss_mnt, &copy->c_fh,      
                ^               
fs/nfsd/nfs4proc.c:1442:4: error: implicit declaration of function 'nfsd4_interssc_disconnect'; did you mean 'nfsd4_insert_layout'? [-Werror=implicit-function-declaration]
    nfsd4_interssc_disconnect(copy->ss_mnt);                     
    ^~~~~~~~~~~~~~~~~~~~~~~~~   
    nfsd4_insert_layout    

Thanks,
Anna

> 
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  fs/nfsd/nfs4proc.c  | 224 +++++++++++++++++++++++++++++++++++++++++-----------
>  fs/nfsd/nfs4state.c |   9 ++-
>  fs/nfsd/state.h     |   2 +
>  fs/nfsd/xdr4.h      |  11 +++
>  4 files changed, 197 insertions(+), 49 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 5dd8278..a977142 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -77,6 +77,21 @@
>  { }
>  #endif
>  
> +static struct workqueue_struct *copy_wq;
> +
> +int nfsd4_create_copy_queue(void)
> +{
> +	copy_wq = create_singlethread_workqueue("nfsd4_copy");
> +	if (!copy_wq)
> +		return -ENOMEM;
> +	return 0;
> +}
> +
> +void nfsd4_destroy_copy_queue(void)
> +{
> +	destroy_workqueue(copy_wq);
> +}
> +
>  #define NFSDDBG_FACILITY		NFSDDBG_PROC
>  
>  static u32 nfsd_attrmask[] = {
> @@ -1210,21 +1225,17 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
>  static struct vfsmount *
>  nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
>  			struct nfsd4_compound_state *cstate,
> -			struct nfsd4_copy *copy, struct file **src,
> -			struct file **dst)
> +			struct nfsd4_copy *copy)
>  {
>  	struct svc_fh *s_fh = NULL;
>  	stateid_t *s_stid = &copy->cp_src_stateid;
> -	struct nfs_fh fh;
> -	nfs4_stateid stateid;
> -	struct file *filp;
>  	struct vfsmount *ss_mnt;
>  	__be32 status;
>  
>  	/* Verify the destination stateid and set dst struct file*/
>  	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
>  					&copy->cp_dst_stateid,
> -					WR_STATE, dst, NULL, NULL);
> +					WR_STATE, &copy->fh_dst, NULL, NULL);
>  	if (status) {
>  		ss_mnt = ERR_PTR(be32_to_cpu(status));
>  		goto out;
> @@ -1238,19 +1249,11 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
>  		goto out;
>  
>  	s_fh = &cstate->save_fh;
> -
> -	fh.size = s_fh->fh_handle.fh_size;
> -	memcpy(fh.data, &s_fh->fh_handle.fh_base, fh.size);
> -	stateid.seqid = s_stid->si_generation;
> -	memcpy(stateid.other, (void *)&s_stid->si_opaque,
> +	copy->c_fh.size = s_fh->fh_handle.fh_size;
> +	memcpy(copy->c_fh.data, &s_fh->fh_handle.fh_base, copy->c_fh.size);
> +	copy->stateid.seqid = s_stid->si_generation;
> +	memcpy(copy->stateid.other, (void *)&s_stid->si_opaque,
>  		sizeof(stateid_opaque_t));
> -
> -	filp =  nfs42_ssc_open(ss_mnt, &fh, &stateid);
> -	if (IS_ERR(filp)) {
> -		nfsd4_interssc_disconnect(ss_mnt);
> -		return ERR_CAST(filp);
> -	}
> -	*src = filp;
>  out:
>  	return ss_mnt;
>  }
> @@ -1272,8 +1275,7 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
>  static struct vfsmount *
>  nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
>  			struct nfsd4_compound_state *cstate,
> -			struct nfsd4_copy *copy, struct file **src,
> -			struct file **dst)
> +			struct nfsd4_copy *copy)
>  {
>  	return ERR_PTR(-EINVAL);
>  }
> @@ -1289,13 +1291,13 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
>  static __be32
>  nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
>  		      struct nfsd4_compound_state *cstate,
> -		      struct nfsd4_copy *copy, struct file **src,
> -		      struct file **dst)
> +		      struct nfsd4_copy *copy)
>  {
>  	__be32 status;
>  
> -	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid, src,
> -				   &copy->cp_dst_stateid, dst);
> +	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid,
> +				   &copy->fh_src, &copy->cp_dst_stateid,
> +				   &copy->fh_dst);
>  	if (status)
>  		goto out;
>  
> @@ -1315,47 +1317,173 @@ extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
>  	fput(dst);
>  }
>  
> +static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
> +{
> +	struct nfsd4_copy *copy = container_of(cb, struct nfsd4_copy, cp_cb);
> +
> +	kfree(copy);
> +}
> +
> +static int nfsd4_cb_offload_done(struct nfsd4_callback *cb,
> +				 struct rpc_task *task)
> +{
> +	return 1;
> +}
> +
> +static const struct nfsd4_callback_ops nfsd4_cb_offload_ops = {
> +	.release = nfsd4_cb_offload_release,
> +	.done = nfsd4_cb_offload_done
> +};
> +
> +static int nfsd4_init_copy_res(struct nfsd4_copy *copy, bool sync)
> +{
> +	memcpy(&copy->cp_res.cb_stateid, &copy->cp_dst_stateid,
> +		sizeof(copy->cp_dst_stateid));
> +	copy->cp_res.wr_stable_how = NFS_UNSTABLE;
> +	copy->cp_consecutive = 1;
> +	copy->cp_synchronous = sync;
> +	gen_boot_verifier(&copy->cp_res.wr_verifier, copy->net);
> +
> +	return nfs_ok;
> +}
> +
> +static int _nfsd_copy_file_range(struct nfsd4_copy *copy)
> +{
> +	ssize_t bytes_copied = 0;
> +	size_t bytes_total = copy->cp_count;
> +	size_t bytes_to_copy;
> +	u64 src_pos = copy->cp_src_pos;
> +	u64 dst_pos = copy->cp_dst_pos;
> +
> +	do {
> +		bytes_to_copy = min_t(u64, bytes_total, MAX_RW_COUNT);
> +		bytes_copied = nfsd_copy_file_range(copy->fh_src, src_pos,
> +				copy->fh_dst, dst_pos, bytes_to_copy);
> +		if (bytes_copied <= 0)
> +			break;
> +		bytes_total -= bytes_copied;
> +		copy->cp_res.wr_bytes_written += bytes_copied;
> +		src_pos += bytes_copied;
> +		dst_pos += bytes_copied;
> +	} while (bytes_total > 0 && !copy->cp_synchronous);
> +	return bytes_copied;
> +}
> +
> +static int nfsd4_do_copy(struct nfsd4_copy *copy, bool sync)
> +{
> +	__be32 status;
> +	ssize_t bytes;
> +
> +	bytes = _nfsd_copy_file_range(copy);
> +	if (bytes < 0)
> +		status = nfserrno(bytes);
> +	else
> +		status = nfsd4_init_copy_res(copy, sync);
> +
> +	if (copy->cp_src.nl_nsvr > 0)   /* Inter server SSC */
> +		nfsd4_cleanup_inter_ssc(copy->ss_mnt, copy->fh_src,
> +				copy->fh_dst);
> +	else
> +		nfsd4_cleanup_intra_ssc(copy->fh_src, copy->fh_dst);
> +
> +	return status;
> +}
> +
> +static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
> +{
> +	memcpy(&dst->cp_src_stateid, &src->cp_src_stateid, sizeof(stateid_t));
> +	memcpy(&dst->cp_dst_stateid, &src->cp_dst_stateid, sizeof(stateid_t));
> +	dst->cp_src_pos = src->cp_src_pos;
> +	dst->cp_dst_pos = src->cp_dst_pos;
> +	dst->cp_count = src->cp_count;
> +	memcpy(&dst->cp_src, &src->cp_src, sizeof(struct nl4_servers));
> +	dst->cp_consecutive = src->cp_consecutive;
> +	dst->cp_synchronous = src->cp_synchronous;
> +	memcpy(&dst->cp_res, &src->cp_res, sizeof(src->cp_res));
> +	memcpy(&dst->stateid, &src->stateid, sizeof(src->stateid));
> +	memcpy(&dst->c_fh, &src->c_fh, sizeof(src->c_fh));
> +	/* skipping nfsd4_callback */
> +	memcpy(&dst->fh, &src->fh, sizeof(src->fh));
> +	dst->fh = src->fh;
> +	dst->cp_clp = src->cp_clp;
> +	dst->fh_src = src->fh_src;
> +	dst->fh_dst = src->fh_dst;
> +	dst->ss_mnt = src->ss_mnt;
> +	dst->net = src->net;
> +}
> +
> +static void nfsd4_do_async_copy(struct work_struct *work)
> +{
> +	struct nfsd4_copy *copy =
> +		container_of(work, struct nfsd4_copy, cp_work);
> +	struct nfsd4_copy *cb_copy;
> +
> +	if (copy->cp_src.nl_nsvr > 0) { /* Inter server SSC */
> +		copy->fh_src = nfs42_ssc_open(copy->ss_mnt, &copy->c_fh,
> +						&copy->stateid);
> +		if (IS_ERR(copy->fh_src)) {
> +			nfsd4_interssc_disconnect(copy->ss_mnt);
> +			return;
> +		}
> +	}
> +	copy->nfserr = nfsd4_do_copy(copy, 0);
> +	cb_copy = kzalloc(sizeof(struct nfsd4_copy), GFP_KERNEL);
> +	if (!cb_copy)
> +		goto out;
> +	memcpy(&cb_copy->cp_res, &copy->cp_res, sizeof(copy->cp_res));
> +	cb_copy->cp_clp = copy->cp_clp;
> +	cb_copy->nfserr = copy->nfserr;
> +	memcpy(&cb_copy->fh, &copy->fh, sizeof(copy->fh));
> +	nfsd4_init_cb(&cb_copy->cp_cb, cb_copy->cp_clp,
> +			&nfsd4_cb_offload_ops, NFSPROC4_CLNT_CB_OFFLOAD);
> +	nfsd4_run_cb(&cb_copy->cp_cb);
> +out:
> +	kfree(copy);
> +}
> +
>  static __be32
>  nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	struct nfsd4_copy *copy)
>  {
> -	struct vfsmount *ss_mnt = NULL;
> -	struct file *src, *dst;
>  	__be32 status;
> -	ssize_t bytes;
>  
>  	if (copy->cp_src.nl_nsvr > 0) { /* Inter server SSC */
> -		ss_mnt = nfsd4_setup_inter_ssc(rqstp, cstate, copy, &src, &dst);
> -		if (IS_ERR(ss_mnt)) {
> -			status = nfserrno(PTR_ERR(ss_mnt));
> +		copy->ss_mnt = nfsd4_setup_inter_ssc(rqstp, cstate, copy);
> +		if (IS_ERR(copy->ss_mnt)) {
> +			status = nfserrno(PTR_ERR(copy->ss_mnt));
>  			goto out;
>  		}
>  	} else {
> -		status = nfsd4_setup_intra_ssc(rqstp, cstate, copy, &src, &dst);
> +		status = nfsd4_setup_intra_ssc(rqstp, cstate, copy);
>  		if (status)
>  			goto out;
>  	}
>  
> -	bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
> -			dst, copy->cp_dst_pos, copy->cp_count);
> -
> -	if (bytes < 0)
> -		status = nfserrno(bytes);
> -	else {
> -		copy->cp_res.wr_bytes_written = bytes;
> -		copy->cp_res.wr_stable_how = NFS_UNSTABLE;
> -		copy->cp_consecutive = 1;
> -		copy->cp_synchronous = 1;
> -		gen_boot_verifier(&copy->cp_res.wr_verifier, SVC_NET(rqstp));
> -		status = nfs_ok;
> +	copy->cp_clp = cstate->clp;
> +	memcpy(&copy->fh, &cstate->current_fh.fh_handle,
> +		sizeof(struct knfsd_fh));
> +	copy->net = SVC_NET(rqstp);
> +	if (!copy->cp_synchronous) {
> +		struct nfsd4_copy *async_copy;
> +
> +		status = nfsd4_init_copy_res(copy, 0);
> +		async_copy = kzalloc(sizeof(struct nfsd4_copy), GFP_KERNEL);
> +		if (!async_copy)
> +			goto out_err;
> +		dup_copy_fields(copy, async_copy);
> +		memcpy(&copy->cp_res.cb_stateid, &copy->cp_dst_stateid,
> +			sizeof(copy->cp_dst_stateid));
> +		INIT_WORK(&async_copy->cp_work, nfsd4_do_async_copy);
> +		queue_work(copy_wq, &async_copy->cp_work);
> +	} else {
> +		status = nfsd4_do_copy(copy, 1);
>  	}
> -
> -	if (copy->cp_src.nl_nsvr > 0)   /* Inter server SSC */
> -		nfsd4_cleanup_inter_ssc(ss_mnt, src, dst);
> -	else
> -		nfsd4_cleanup_intra_ssc(src, dst);
>  out:
>  	return status;
> +out_err:
> +	nfsd4_interssc_disconnect(copy->ss_mnt);
> +	status = nfserrno(-ENOMEM);
> +	goto out;
>  }
>  
>  static __be32
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index b019fc8..d391ce5 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -7123,8 +7123,14 @@ static int nfs4_state_create_net(struct net *net)
>  		goto out_free_laundry;
>  
>  	set_max_delegations();
> -	return 0;
>  
> +	ret = nfsd4_create_copy_queue();
> +	if (ret)
> +		goto out_free_callback;
> +
> +	return 0;
> +out_free_callback:
> +	nfsd4_destroy_callback_queue();
>  out_free_laundry:
>  	destroy_workqueue(laundry_wq);
>  out_cleanup_cred:
> @@ -7187,6 +7193,7 @@ static int nfs4_state_create_net(struct net *net)
>  	destroy_workqueue(laundry_wq);
>  	nfsd4_destroy_callback_queue();
>  	cleanup_callback_cred();
> +	nfsd4_destroy_copy_queue();
>  }
>  
>  static void
> diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
> index 3b0da32..2acea23 100644
> --- a/fs/nfsd/state.h
> +++ b/fs/nfsd/state.h
> @@ -649,6 +649,8 @@ extern void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
>  extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(const char *name,
>  							struct nfsd_net *nn);
>  extern bool nfs4_has_reclaimed_state(const char *name, struct nfsd_net *nn);
> +extern int nfsd4_create_copy_queue(void);
> +extern void nfsd4_destroy_copy_queue(void);
>  
>  struct nfs4_file *find_file(struct knfsd_fh *fh);
>  void put_nfs4_file(struct nfs4_file *fi);
> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
> index c3e6907..d75e530 100644
> --- a/fs/nfsd/xdr4.h
> +++ b/fs/nfsd/xdr4.h
> @@ -532,10 +532,21 @@ struct nfsd4_copy {
>  	/* response */
>  	struct nfsd42_write_res	cp_res;
>  
> +	struct nfs_fh           c_fh;
> +	nfs4_stateid            stateid;
> +
>  	/* for cb_offload */
>  	struct nfsd4_callback	cp_cb;
>  	__be32			nfserr;
>  	struct knfsd_fh		fh;
> +
> +	struct work_struct	cp_work;
> +	struct nfs4_client	*cp_clp;
> +
> +	struct file		*fh_src;
> +	struct file		*fh_dst;
> +	struct vfsmount		*ss_mnt;
> +	struct net		*net;
>  };
>  
>  struct nfsd4_seek {
> 

Re: [RFC v3 16/42] NFS NFSD defining nl4_servers structure needed by both

[J. Bruce Fields]

On Tue, Jul 11, 2017 at 12:43:50PM -0400, Olga Kornievskaia wrote:
> These structures are needed by COPY_NOTIFY on the client and needed
> by the nfsd as well
> 
> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> ---
>  include/linux/nfs4.h | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
> index 7262908..4179c78 100644
> --- a/include/linux/nfs4.h
> +++ b/include/linux/nfs4.h
> @@ -15,6 +15,7 @@
>  #include <linux/list.h>
>  #include <linux/uidgid.h>
>  #include <uapi/linux/nfs4.h>
> +#include <linux/sunrpc/msg_prot.h>
>  
>  enum nfs4_acl_whotype {
>  	NFS4_ACL_WHO_NAMED = 0,
> @@ -659,4 +660,36 @@ struct nfs4_op_map {
>  	} u;
>  };
>  
> +struct nfs42_netaddr {
> +	unsigned int	na_netid_len;
> +	char		na_netid[RPCBIND_MAXNETIDLEN + 1];
> +	unsigned int	na_uaddr_len;
> +	char		na_uaddr[RPCBIND_MAXUADDRLEN + 1];
> +};
> +
> +enum netloc_type4 {
> +	NL4_NAME		= 1,
> +	NL4_URL			= 2,
> +	NL4_NETADDR		= 3,
> +};
> +
> +struct nl4_server {
> +	enum netloc_type4	nl4_type;
> +	union {
> +		struct { /* NL4_NAME, NL4_URL */
> +			int	nl4_str_sz;
> +			char	nl4_str[NFS4_OPAQUE_LIMIT + 1];
> +		};
> +		struct nfs42_netaddr	nl4_addr; /* NL4_NETADDR */
> +	} u;
> +};
> +
> +/*  support 1 nl4_server for now */
> +#define NFS42_MAX_SSC_SRC       1

In that case, let's just build that assumption into the data structures
and embed struct nl4_server in nl4_servers and drop the unnecessary
kmalloc()s.

--b.

> +
> +struct nl4_servers {
> +	int			nl_nsvr;
> +	struct nl4_server	*nl_svr;
> +};
> +
>  #endif
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 25/42] NFSD add COPY_NOTIFY operation

[J. Bruce Fields]

On Tue, Jul 11, 2017 at 12:43:59PM -0400, Olga Kornievskaia wrote:
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfsd/nfs4proc.c |  98 ++++++++++++++++++++++++++++++++++++++++++++++
>  fs/nfsd/nfs4xdr.c  | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  fs/nfsd/xdr4.h     |  13 +++++++
>  3 files changed, 221 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index dadb3bf..ed6b9f2 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -35,6 +35,7 @@
>  #include <linux/file.h>
>  #include <linux/falloc.h>
>  #include <linux/slab.h>
> +#include <linux/sunrpc/addr.h>
>  
>  #include "idmap.h"
>  #include "cache.h"
> @@ -1087,6 +1088,82 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>  }
>  
>  static __be32
> +nfsd4_set_src_nl4_netaddr(struct svc_rqst *rqstp, struct nl4_servers *svrs)
> +{
> +	const struct sockaddr *addr = (struct sockaddr *)&rqstp->rq_daddr;
> +	const struct sockaddr_in *sin = (const struct sockaddr_in *)addr;
> +	const struct sockaddr_in6 *sin6 = (const struct sockaddr_in6 *)addr;
> +	int uaddr_len = rqstp->rq_daddrlen + 4 + 1; /* port (4) and '\0' (1) */
> +	struct nfs42_netaddr *naddr;
> +	size_t ret;
> +	unsigned short port;
> +
> +	/* freed in nfsd4_encode_copy_notify */
> +	svrs->nl_svr = kmalloc_array(svrs->nl_nsvr, sizeof(struct nl4_server),
> +				GFP_KERNEL);
> +	if (svrs->nl_svr == NULL)
> +		return nfserrno(-ENOMEM);
> +
> +	svrs->nl_svr->nl4_type = NL4_NETADDR;
> +	naddr = &svrs->nl_svr->u.nl4_addr;
> +
> +	switch (addr->sa_family) {
> +	case AF_INET:
> +		port = ntohs(sin->sin_port);
> +		sprintf(naddr->na_netid, "tcp");
> +		naddr->na_netid_len = 3;
> +		break;
> +	case AF_INET6:
> +		port = ntohs(sin6->sin6_port);
> +		sprintf(naddr->na_netid, "tcp6");
> +		naddr->na_netid_len = 4;
> +		break;
> +	default:
> +		dprintk("NFSD  nfsd4_set_notify_src: unknown address type: %d",
> +			addr->sa_family);
> +		kfree(svrs->nl_svr);
> +		return nfserrno(-EINVAL);
> +	}
> +	ret = rpc_ntop(addr, naddr->na_uaddr, sizeof(naddr->na_uaddr));
> +	snprintf(naddr->na_uaddr + ret, uaddr_len, ".%u.%u",
> +		 port >> 8, port & 255);
> +	naddr->na_uaddr_len = strlen(naddr->na_uaddr);

I'm a little surprised there isn't already a standard helper for
this....  Actually, does net/sunrpc:addr.c:rpc_sockaddr2uaddr() do it?

(Looks like there's other potential users in
fs/nfsd/flexfilelayout.c:nfsd4_ff_proc_getdeviceinfo() and in flexfiles
code on client side and probably some other places too.)

--b.

> +	return 0;
> +}
> +
> +static __be32
> +nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> +		  struct nfsd4_copy_notify *cn)
> +{
> +	__be32 status;
> +	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
> +
> +	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
> +					&cn->cpn_src_stateid, RD_STATE, NULL,
> +					NULL);
> +	if (status)
> +		return status;
> +
> +	cn->cpn_sec = nn->nfsd4_lease;
> +	cn->cpn_nsec = 0;
> +
> +
> +	/** XXX Save cpn_src_statid, cpn_src, and any other returned source
> +	 * server addresses on which the source server is williing to accept
> +	 * connections from the destination e.g. what is returned in cpn_src,
> +	 * to verify READ from dest server.
> +	 */
> +
> +	/**
> +	 * For now, only return one server address in cpn_src, the
> +	 * address used by the client to connect to this server.
> +	 */
> +	cn->cpn_src.nl_nsvr = 1;
> +
> +	return nfsd4_set_src_nl4_netaddr(rqstp, &cn->cpn_src);
> +}
> +
> +static __be32
>  nfsd4_fallocate(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  		struct nfsd4_fallocate *fallocate, int flags)
>  {
> @@ -2042,6 +2119,21 @@ static inline u32 nfsd4_copy_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>  		1 /* cr_synchronous */) * sizeof(__be32);
>  }
>  
> +static inline u32 nfsd4_copy_notify_rsize(struct svc_rqst *rqstp,
> +					struct nfsd4_op *op)
> +{
> +	return (op_encode_hdr_size +
> +		3 /* cnr_lease_time */ +
> +		1 /* We support one cnr_source_server */ +
> +		1 /* cnr_stateid seq */ +
> +		op_encode_stateid_maxsz /* cnr_stateid */ +
> +		1 /* num cnr_source_server*/ +
> +		1 /* nl4_type */ +
> +		1 /* nl4 size */ +
> +		XDR_QUADLEN(NFS4_OPAQUE_LIMIT) /*nl4_loc + nl4_loc_sz */)
> +		* sizeof(__be32);
> +}
> +
>  #ifdef CONFIG_NFSD_PNFS
>  static inline u32 nfsd4_getdeviceinfo_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>  {
> @@ -2446,6 +2538,12 @@ static inline u32 nfsd4_seek_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>  		.op_name = "OP_SEEK",
>  		.op_rsize_bop = (nfsd4op_rsize)nfsd4_seek_rsize,
>  	},
> +	[OP_COPY_NOTIFY] = {
> +		.op_func = (nfsd4op_func)nfsd4_copy_notify,
> +		.op_flags = OP_MODIFIES_SOMETHING | OP_CACHEME,
> +		.op_name = "OP_COPY_NOTIFY",
> +		.op_rsize_bop = (nfsd4op_rsize)nfsd4_copy_notify_rsize,
> +	},
>  };
>  
>  /**
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 031d06d..3e08c15 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -1820,6 +1820,22 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
>  }
>  
>  static __be32
> +nfsd4_decode_copy_notify(struct nfsd4_compoundargs *argp,
> +			 struct nfsd4_copy_notify *cn)
> +{
> +	int status;
> +
> +	status = nfsd4_decode_stateid(argp, &cn->cpn_src_stateid);
> +	if (status)
> +		return status;
> +	status = nfsd4_decode_nl4_server(argp, &cn->cpn_dst);
> +	if (status)
> +		return status;
> +
> +	return status;
> +}
> +
> +static __be32
>  nfsd4_decode_seek(struct nfsd4_compoundargs *argp, struct nfsd4_seek *seek)
>  {
>  	DECODE_HEAD;
> @@ -1920,7 +1936,7 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
>  	/* new operations for NFSv4.2 */
>  	[OP_ALLOCATE]		= (nfsd4_dec)nfsd4_decode_fallocate,
>  	[OP_COPY]		= (nfsd4_dec)nfsd4_decode_copy,
> -	[OP_COPY_NOTIFY]	= (nfsd4_dec)nfsd4_decode_notsupp,
> +	[OP_COPY_NOTIFY]	= (nfsd4_dec)nfsd4_decode_copy_notify,
>  	[OP_DEALLOCATE]		= (nfsd4_dec)nfsd4_decode_fallocate,
>  	[OP_IO_ADVISE]		= (nfsd4_dec)nfsd4_decode_notsupp,
>  	[OP_LAYOUTERROR]	= (nfsd4_dec)nfsd4_decode_notsupp,
> @@ -4355,6 +4371,52 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>  }
>  
>  static __be32
> +nfsd42_encode_nl4_server(struct nfsd4_compoundres *resp, struct nl4_server *ns)
> +{
> +	struct xdr_stream *xdr = &resp->xdr;
> +	struct nfs42_netaddr *addr;
> +	__be32 *p;
> +
> +	p = xdr_reserve_space(xdr, 4);
> +	*p++ = cpu_to_be32(ns->nl4_type);
> +
> +	switch (ns->nl4_type) {
> +	case NL4_NAME:
> +	case NL4_URL:
> +		p = xdr_reserve_space(xdr, 4 /* url or name len */ +
> +				      (XDR_QUADLEN(ns->u.nl4_str_sz) * 4));
> +		if (!p)
> +			return nfserr_resource;
> +		*p++ = cpu_to_be32(ns->u.nl4_str_sz);
> +		p = xdr_encode_opaque_fixed(p, ns->u.nl4_str, ns->u.nl4_str_sz);
> +			break;
> +	case NL4_NETADDR:
> +		addr = &ns->u.nl4_addr;
> +
> +		/** netid_len, netid, uaddr_len, uaddr (port included
> +		 * in RPCBIND_MAXUADDRLEN)
> +		 */
> +		p = xdr_reserve_space(xdr,
> +			4 /* netid len */ +
> +			(XDR_QUADLEN(addr->na_netid_len) * 4) +
> +			4 /* uaddr len */ +
> +			(XDR_QUADLEN(addr->na_uaddr_len) * 4));
> +		if (!p)
> +			return nfserr_resource;
> +
> +		*p++ = cpu_to_be32(addr->na_netid_len);
> +		p = xdr_encode_opaque_fixed(p, addr->na_netid,
> +					    addr->na_netid_len);
> +		*p++ = cpu_to_be32(addr->na_uaddr_len);
> +		p = xdr_encode_opaque_fixed(p, addr->na_uaddr,
> +					addr->na_uaddr_len);
> +		break;
> +	}
> +
> +	return 0;
> +}
> +
> +static __be32
>  nfsd4_encode_copy(struct nfsd4_compoundres *resp, __be32 nfserr,
>  		  struct nfsd4_copy *copy)
>  {
> @@ -4375,6 +4437,52 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>  }
>  
>  static __be32
> +nfsd4_encode_copy_notify(struct nfsd4_compoundres *resp, __be32 nfserr,
> +			 struct nfsd4_copy_notify *cn)
> +{
> +	struct xdr_stream *xdr = &resp->xdr;
> +	struct nl4_server *ns;
> +	__be32 *p;
> +	int i;
> +
> +	if (nfserr)
> +		return nfserr;
> +
> +	/* 8 sec, 4 nsec */
> +	p = xdr_reserve_space(xdr, 12);
> +	if (!p)
> +		return nfserr_resource;
> +
> +	/* cnr_lease_time */
> +	p = xdr_encode_hyper(p, cn->cpn_sec);
> +	*p++ = cpu_to_be32(cn->cpn_nsec);
> +
> +	/* cnr_stateid */
> +	nfserr = nfsd4_encode_stateid(xdr, &cn->cpn_src_stateid);
> +	if (nfserr)
> +		return nfserr;
> +
> +	/* cnr_src.nl_nsvr */
> +	p = xdr_reserve_space(xdr, 4);
> +	if (!p)
> +		return nfserr_resource;
> +
> +	*p++ = cpu_to_be32(cn->cpn_src.nl_nsvr);
> +
> +	ns = cn->cpn_src.nl_svr;
> +	for (i = 0; i < cn->cpn_src.nl_nsvr; i++) {
> +		nfserr = nfsd42_encode_nl4_server(resp, ns);
> +		if (nfserr)
> +			return nfserr;
> +		ns++;
> +	}
> +
> +	/* allocated in nfsd4_copy_notify */
> +	kfree(cn->cpn_src.nl_svr);
> +	return nfserr;
> +}
> +
> +static __be32
>  nfsd4_encode_seek(struct nfsd4_compoundres *resp, __be32 nfserr,
>  		  struct nfsd4_seek *seek)
>  {
> @@ -4474,7 +4582,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>  	/* NFSv4.2 operations */
>  	[OP_ALLOCATE]		= (nfsd4_enc)nfsd4_encode_noop,
>  	[OP_COPY]		= (nfsd4_enc)nfsd4_encode_copy,
> -	[OP_COPY_NOTIFY]	= (nfsd4_enc)nfsd4_encode_noop,
> +	[OP_COPY_NOTIFY]	= (nfsd4_enc)nfsd4_encode_copy_notify,
>  	[OP_DEALLOCATE]		= (nfsd4_enc)nfsd4_encode_noop,
>  	[OP_IO_ADVISE]		= (nfsd4_enc)nfsd4_encode_noop,
>  	[OP_LAYOUTERROR]	= (nfsd4_enc)nfsd4_encode_noop,
> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
> index 6b1a61fc..5db7cd8 100644
> --- a/fs/nfsd/xdr4.h
> +++ b/fs/nfsd/xdr4.h
> @@ -540,6 +540,18 @@ struct nfsd4_seek {
>  	loff_t		seek_pos;
>  };
>  
> +struct nfsd4_copy_notify {
> +	/* request */
> +	stateid_t		cpn_src_stateid;
> +	struct nl4_server	cpn_dst;
> +
> +	/* response */
> +	/* Note: cpn_src_stateid is used for cnr_stateid */
> +	u64			cpn_sec;
> +	u32			cpn_nsec;
> +	struct nl4_servers	cpn_src;
> +};
> +
>  struct nfsd4_op {
>  	int					opnum;
>  	__be32					status;
> @@ -595,6 +607,7 @@ struct nfsd4_op {
>  		struct nfsd4_fallocate		deallocate;
>  		struct nfsd4_clone		clone;
>  		struct nfsd4_copy		copy;
> +		struct nfsd4_copy_notify	copy_notify;
>  		struct nfsd4_seek		seek;
>  	} u;
>  	struct nfs4_replay *			replay;
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 25/42] NFSD add COPY_NOTIFY operation

[J. Bruce Fields]

On Tue, Jul 11, 2017 at 12:43:59PM -0400, Olga Kornievskaia wrote:
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfsd/nfs4proc.c |  98 ++++++++++++++++++++++++++++++++++++++++++++++
>  fs/nfsd/nfs4xdr.c  | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  fs/nfsd/xdr4.h     |  13 +++++++
>  3 files changed, 221 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index dadb3bf..ed6b9f2 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -35,6 +35,7 @@
>  #include <linux/file.h>
>  #include <linux/falloc.h>
>  #include <linux/slab.h>
> +#include <linux/sunrpc/addr.h>
>  
>  #include "idmap.h"
>  #include "cache.h"
> @@ -1087,6 +1088,82 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>  }
>  
>  static __be32
> +nfsd4_set_src_nl4_netaddr(struct svc_rqst *rqstp, struct nl4_servers *svrs)
> +{
> +	const struct sockaddr *addr = (struct sockaddr *)&rqstp->rq_daddr;
> +	const struct sockaddr_in *sin = (const struct sockaddr_in *)addr;
> +	const struct sockaddr_in6 *sin6 = (const struct sockaddr_in6 *)addr;
> +	int uaddr_len = rqstp->rq_daddrlen + 4 + 1; /* port (4) and '\0' (1) */
> +	struct nfs42_netaddr *naddr;
> +	size_t ret;
> +	unsigned short port;
> +
> +	/* freed in nfsd4_encode_copy_notify */
> +	svrs->nl_svr = kmalloc_array(svrs->nl_nsvr, sizeof(struct nl4_server),
> +				GFP_KERNEL);
> +	if (svrs->nl_svr == NULL)
> +		return nfserrno(-ENOMEM);
> +
> +	svrs->nl_svr->nl4_type = NL4_NETADDR;
> +	naddr = &svrs->nl_svr->u.nl4_addr;
> +
> +	switch (addr->sa_family) {
> +	case AF_INET:
> +		port = ntohs(sin->sin_port);
> +		sprintf(naddr->na_netid, "tcp");
> +		naddr->na_netid_len = 3;
> +		break;
> +	case AF_INET6:
> +		port = ntohs(sin6->sin6_port);
> +		sprintf(naddr->na_netid, "tcp6");
> +		naddr->na_netid_len = 4;
> +		break;
> +	default:
> +		dprintk("NFSD  nfsd4_set_notify_src: unknown address type: %d",
> +			addr->sa_family);
> +		kfree(svrs->nl_svr);
> +		return nfserrno(-EINVAL);
> +	}
> +	ret = rpc_ntop(addr, naddr->na_uaddr, sizeof(naddr->na_uaddr));
> +	snprintf(naddr->na_uaddr + ret, uaddr_len, ".%u.%u",
> +		 port >> 8, port & 255);
> +	naddr->na_uaddr_len = strlen(naddr->na_uaddr);
> +	return 0;
> +}
> +
> +static __be32
> +nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> +		  struct nfsd4_copy_notify *cn)
> +{
> +	__be32 status;
> +	struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
> +
> +	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
> +					&cn->cpn_src_stateid, RD_STATE, NULL,
> +					NULL);
> +	if (status)
> +		return status;
> +
> +	cn->cpn_sec = nn->nfsd4_lease;
> +	cn->cpn_nsec = 0;
> +
> +
> +	/** XXX Save cpn_src_statid, cpn_src, and any other returned source
> +	 * server addresses on which the source server is williing to accept
> +	 * connections from the destination e.g. what is returned in cpn_src,
> +	 * to verify READ from dest server.
> +	 */

Either we should fix the stuff mentioned in the comment or figure out
why it's not necessary.

I guess I don't see what the value would be of verifying the IP address.
Our choice of IP address to return here is a little arbitrary anyway.
And I assume the eventualy READ calls are going to be verified with all
the usual export and file permission checks.

If we don't save the cna_src_stateid, then I guess that means the COPY
might still succeed after the original OPEN/LOCK stateid is no longer
valid?  I can't decide if that will cause problems in practice or not.

>  static __be32
> +nfsd42_encode_nl4_server(struct nfsd4_compoundres *resp, struct nl4_server *ns)
> +{
> +	struct xdr_stream *xdr = &resp->xdr;
> +	struct nfs42_netaddr *addr;
> +	__be32 *p;
> +
> +	p = xdr_reserve_space(xdr, 4);
> +	*p++ = cpu_to_be32(ns->nl4_type);
> +
> +	switch (ns->nl4_type) {
> +	case NL4_NAME:
> +	case NL4_URL:
> +		p = xdr_reserve_space(xdr, 4 /* url or name len */ +
> +				      (XDR_QUADLEN(ns->u.nl4_str_sz) * 4));
> +		if (!p)
> +			return nfserr_resource;
> +		*p++ = cpu_to_be32(ns->u.nl4_str_sz);
> +		p = xdr_encode_opaque_fixed(p, ns->u.nl4_str, ns->u.nl4_str_sz);
> +			break;

This code will never be used as far as I can tell.  We could
BUG_ON(nl4_type != NL4_NETADDR) (or even just remove nl4_type).

--b.

Re: [RFC v3 25/42] NFSD add COPY_NOTIFY operation

[J. Bruce Fields]

On Tue, Jul 11, 2017 at 12:43:59PM -0400, Olga Kornievskaia wrote:
> +struct nfsd4_copy_notify {
> +	/* request */
> +	stateid_t		cpn_src_stateid;
> +	struct nl4_server	cpn_dst;
> +
> +	/* response */
> +	/* Note: cpn_src_stateid is used for cnr_stateid */

That might be worth a little more explanation.

--b.

> +	u64			cpn_sec;
> +	u32			cpn_nsec;
> +	struct nl4_servers	cpn_src;
> +};
> +

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[J. Bruce Fields]

Some questions below, but really I think this approach can't work.  Keep
in mind this has to work not just for the one compound the current
client uses, but for *any* compound containing a COPY, including those
sent by future clients that may do something more complicated, or by
malcious clients that may intentionally send weird compounds to make the
server crash.

I think we want to flag the filehandle itself somehow, not the cstate.

And check very carefully to make sure that ops other than COPY can't get
tripped up by one of these foreign filehandles.

On Tue, Jul 11, 2017 at 12:44:01PM -0400, Olga Kornievskaia wrote:
> The inter server to server COPY source server filehandle
> is guaranteed to be stale as the COPY is sent to the destination
> server.
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
>  fs/nfsd/nfs4xdr.c  | 26 +++++++++++++++++++++++++-
>  fs/nfsd/nfsd.h     |  2 ++
>  fs/nfsd/xdr4.h     |  4 ++++
>  4 files changed, 77 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index b49ff31..ceee852 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -496,11 +496,19 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>  nfsd4_putfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	    struct nfsd4_putfh *putfh)
>  {
> +	__be32 ret;
> +
>  	fh_put(&cstate->current_fh);
>  	cstate->current_fh.fh_handle.fh_size = putfh->pf_fhlen;
>  	memcpy(&cstate->current_fh.fh_handle.fh_base, putfh->pf_fhval,
>  	       putfh->pf_fhlen);
> -	return fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
> +	ret = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
> +	if (ret == nfserr_stale && HAS_CSTATE_FLAG(cstate, NO_VERIFY_FH)) {
> +		CLEAR_CSTATE_FLAG(cstate, NO_VERIFY_FH);
> +		SET_CSTATE_FLAG(cstate, IS_STALE_FH);
> +		ret = 0;
> +	}
> +	return ret;
>  }
>  
>  static __be32
> @@ -533,6 +541,16 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>  nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>  	     void *arg)
>  {
> +	/**
> +	* This is either an inter COPY (most likely) or an intra COPY with a
> +	* stale file handle. If the latter, nfsd4_copy will reset the PUTFH to
> +	* return nfserr_stale. No fh_dentry, just copy the file handle
> +	* to use with the inter COPY READ.
> +	*/
> +	if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
> +		cstate->save_fh = cstate->current_fh;
> +		return nfs_ok;
> +	}
>  	if (!cstate->current_fh.fh_dentry)
>  		return nfserr_nofilehandle;
>  
> @@ -1067,6 +1085,13 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>  	if (status)
>  		goto out;
>  
> +	/* Intra copy source fh is stale. PUTFH will fail with ESTALE */
> +	if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
> +		CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
> +		cstate->status = nfserr_copy_stalefh;
> +		goto out_put;
> +	}
> +
>  	bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
>  			dst, copy->cp_dst_pos, copy->cp_count);
>  
> @@ -1081,6 +1106,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>  		status = nfs_ok;
>  	}
>  
> +out_put:
>  	fput(src);
>  	fput(dst);
>  out:
> @@ -1759,6 +1785,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>  	struct nfsd4_compound_state *cstate = &resp->cstate;
>  	struct svc_fh *current_fh = &cstate->current_fh;
>  	struct svc_fh *save_fh = &cstate->save_fh;
> +	int		i;
>  	__be32		status;
>  
>  	svcxdr_init_encode(rqstp, resp);
> @@ -1791,6 +1818,12 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>  		goto encode_op;
>  	}
>  
> +	/* NFSv4.2 COPY source file handle may be from a different server */
> +	for (i = 0; i < args->opcnt; i++) {
> +		op = &args->ops[i];
> +		if (op->opnum == OP_COPY)
> +			SET_CSTATE_FLAG(cstate, NO_VERIFY_FH);
> +	}

So, if a compound has a COPY anywhere in it, then a PUTFH with a stale
filehandle anywhere in the compound will temporarily succeed and result in
IS_STALE_FH being set.

>  	while (!status && resp->opcnt < args->opcnt) {
>  		op = &args->ops[resp->opcnt++];
>  
> @@ -1810,6 +1843,9 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>  
>  		opdesc = OPDESC(op);
>  
> +		if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
> +			goto call_op;
> +

That means that once IS_STALE_FH is set, we will skip:

	- nofilehandle checking
	- moved checking
	- fh_clear_wcc()
	- resp_size checking
	- current stateid capture

on every subsequent op.  Can that really be right?

>  		if (!current_fh->fh_dentry) {
>  			if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
>  				op->status = nfserr_nofilehandle;
> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>  
>  		if (opdesc->op_get_currentstateid)
>  			opdesc->op_get_currentstateid(cstate, &op->u);
> +call_op:
>  		op->status = opdesc->op_func(rqstp, cstate, &op->u);
>  
>  		/* Only from SEQUENCE */
> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>  			if (need_wrongsec_check(rqstp))
>  				op->status = check_nfsd_access(current_fh->fh_export, rqstp);
>  		}
> +		/* Only from intra COPY */
> +		if (cstate->status == nfserr_copy_stalefh) {
> +			dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
> +				__func__);
> +			status = nfserr_stale;
> +			nfsd4_adjust_encode(resp);

Are you sure it's safe just to throw away any operations since that
stale PUTFH?  What if some of those operations had side effects?

> +			goto out;
> +		}
>  encode_op:
>  		if (op->status == nfserr_replay_me) {
>  			op->replay = &cstate->replay_owner->so_replay;
> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> index 3e08c15..2896a11 100644
> --- a/fs/nfsd/nfs4xdr.c
> +++ b/fs/nfsd/nfs4xdr.c
> @@ -4624,15 +4624,28 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>  	return nfserr_rep_too_big;
>  }
>  
> +/** Rewind the encoding to return nfserr_stale on the PUTFH
> + * in this failed Intra COPY compound
> + */
> +void
> +nfsd4_adjust_encode(struct nfsd4_compoundres *resp)
> +{
> +	__be32 *p;
> +
> +	p = resp->cstate.putfh_errp;
> +	*p++ = nfserr_stale;
> +}
> +
>  void
>  nfsd4_encode_operation(struct nfsd4_compoundres *resp, struct nfsd4_op *op)
>  {
>  	struct xdr_stream *xdr = &resp->xdr;
>  	struct nfs4_stateowner *so = resp->cstate.replay_owner;
> +	struct nfsd4_compound_state *cstate = &resp->cstate;
>  	struct svc_rqst *rqstp = resp->rqstp;
>  	int post_err_offset;
>  	nfsd4_enc encoder;
> -	__be32 *p;
> +	__be32 *p, *statp;
>  
>  	p = xdr_reserve_space(xdr, 8);
>  	if (!p) {
> @@ -4641,9 +4654,20 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>  	}
>  	*p++ = cpu_to_be32(op->opnum);
>  	post_err_offset = xdr->buf->len;
> +	statp = p;
>  
>  	if (op->opnum == OP_ILLEGAL)
>  		goto status;
> +
> +	/** This is a COPY compound with a stale source server file handle.
> +	 * If OP_COPY processing determines that this is an intra server to
> +	 * server COPY, then this PUTFH should return nfserr_ stale so the
> +	 * putfh_errp will be set to nfserr_stale. If this is an inter server
> +	 * to server COPY, ignore the nfserr_stale.
> +	 */
> +	if (op->opnum == OP_PUTFH && HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
> +		cstate->putfh_errp = statp;
> +
>  	BUG_ON(op->opnum < 0 || op->opnum >= ARRAY_SIZE(nfsd4_enc_ops) ||
>  	       !nfsd4_enc_ops[op->opnum]);
>  	encoder = nfsd4_enc_ops[op->opnum];
> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
> index d966068..8d6fb0f 100644
> --- a/fs/nfsd/nfsd.h
> +++ b/fs/nfsd/nfsd.h
> @@ -272,6 +272,8 @@ static inline bool nfsd4_spo_must_allow(struct svc_rqst *rqstp)
>  #define	nfserr_replay_me	cpu_to_be32(11001)
>  /* nfs41 replay detected */
>  #define	nfserr_replay_cache	cpu_to_be32(11002)
> +/* nfs42 intra copy failed with nfserr_stale */
> +#define nfserr_copy_stalefh	cpu_to_be32(1103)
>  
>  /* Check for dir entries '.' and '..' */
>  #define isdotent(n, l)	(l < 3 && n[0] == '.' && (l == 1 || n[1] == '.'))
> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
> index 38fcb4f..aa94295 100644
> --- a/fs/nfsd/xdr4.h
> +++ b/fs/nfsd/xdr4.h
> @@ -45,6 +45,8 @@
>  
>  #define CURRENT_STATE_ID_FLAG (1<<0)
>  #define SAVED_STATE_ID_FLAG (1<<1)
> +#define NO_VERIFY_FH (1<<2)
> +#define IS_STALE_FH  (1<<3)
>  
>  #define SET_CSTATE_FLAG(c, f) ((c)->sid_flags |= (f))
>  #define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f))
> @@ -63,6 +65,7 @@ struct nfsd4_compound_state {
>  	size_t			iovlen;
>  	u32			minorversion;
>  	__be32			status;
> +	__be32			*putfh_errp;
>  	stateid_t	current_stateid;
>  	stateid_t	save_stateid;
>  	/* to indicate current and saved state id presents */
> @@ -705,6 +708,7 @@ int nfs4svc_decode_compoundargs(struct svc_rqst *, __be32 *,
>  int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
>  		struct nfsd4_compoundres *);
>  __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32);
> +void nfsd4_adjust_encode(struct nfsd4_compoundres *);
>  void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
>  void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op);
>  __be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words,
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[Olga Kornievskaia]

What if instead the client won't send the 2nd FH in case of the inter
server to server COPY and it would avoid all this stale handle
business. So for the "intra" COPY it'll still be PUTFH, SAVEFH, PUTFH,
COPY but for the "inter" COPY it'll be PUTFH, COPY.

On Fri, Sep 8, 2017 at 3:38 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> Some questions below, but really I think this approach can't work.  Keep
> in mind this has to work not just for the one compound the current
> client uses, but for *any* compound containing a COPY, including those
> sent by future clients that may do something more complicated, or by
> malcious clients that may intentionally send weird compounds to make the
> server crash.
>
> I think we want to flag the filehandle itself somehow, not the cstate.
>
> And check very carefully to make sure that ops other than COPY can't get
> tripped up by one of these foreign filehandles.
>
> On Tue, Jul 11, 2017 at 12:44:01PM -0400, Olga Kornievskaia wrote:
>> The inter server to server COPY source server filehandle
>> is guaranteed to be stale as the COPY is sent to the destination
>> server.
>>
>> Signed-off-by: Andy Adamson <andros@netapp.com>
>> ---
>>  fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
>>  fs/nfsd/nfs4xdr.c  | 26 +++++++++++++++++++++++++-
>>  fs/nfsd/nfsd.h     |  2 ++
>>  fs/nfsd/xdr4.h     |  4 ++++
>>  4 files changed, 77 insertions(+), 2 deletions(-)
>>
>> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
>> index b49ff31..ceee852 100644
>> --- a/fs/nfsd/nfs4proc.c
>> +++ b/fs/nfsd/nfs4proc.c
>> @@ -496,11 +496,19 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>>  nfsd4_putfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>>           struct nfsd4_putfh *putfh)
>>  {
>> +     __be32 ret;
>> +
>>       fh_put(&cstate->current_fh);
>>       cstate->current_fh.fh_handle.fh_size = putfh->pf_fhlen;
>>       memcpy(&cstate->current_fh.fh_handle.fh_base, putfh->pf_fhval,
>>              putfh->pf_fhlen);
>> -     return fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
>> +     ret = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
>> +     if (ret == nfserr_stale && HAS_CSTATE_FLAG(cstate, NO_VERIFY_FH)) {
>> +             CLEAR_CSTATE_FLAG(cstate, NO_VERIFY_FH);
>> +             SET_CSTATE_FLAG(cstate, IS_STALE_FH);
>> +             ret = 0;
>> +     }
>> +     return ret;
>>  }
>>
>>  static __be32
>> @@ -533,6 +541,16 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>>  nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>>            void *arg)
>>  {
>> +     /**
>> +     * This is either an inter COPY (most likely) or an intra COPY with a
>> +     * stale file handle. If the latter, nfsd4_copy will reset the PUTFH to
>> +     * return nfserr_stale. No fh_dentry, just copy the file handle
>> +     * to use with the inter COPY READ.
>> +     */
>> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
>> +             cstate->save_fh = cstate->current_fh;
>> +             return nfs_ok;
>> +     }
>>       if (!cstate->current_fh.fh_dentry)
>>               return nfserr_nofilehandle;
>>
>> @@ -1067,6 +1085,13 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>>       if (status)
>>               goto out;
>>
>> +     /* Intra copy source fh is stale. PUTFH will fail with ESTALE */
>> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
>> +             CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
>> +             cstate->status = nfserr_copy_stalefh;
>> +             goto out_put;
>> +     }
>> +
>>       bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
>>                       dst, copy->cp_dst_pos, copy->cp_count);
>>
>> @@ -1081,6 +1106,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>>               status = nfs_ok;
>>       }
>>
>> +out_put:
>>       fput(src);
>>       fput(dst);
>>  out:
>> @@ -1759,6 +1785,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>       struct nfsd4_compound_state *cstate = &resp->cstate;
>>       struct svc_fh *current_fh = &cstate->current_fh;
>>       struct svc_fh *save_fh = &cstate->save_fh;
>> +     int             i;
>>       __be32          status;
>>
>>       svcxdr_init_encode(rqstp, resp);
>> @@ -1791,6 +1818,12 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>               goto encode_op;
>>       }
>>
>> +     /* NFSv4.2 COPY source file handle may be from a different server */
>> +     for (i = 0; i < args->opcnt; i++) {
>> +             op = &args->ops[i];
>> +             if (op->opnum == OP_COPY)
>> +                     SET_CSTATE_FLAG(cstate, NO_VERIFY_FH);
>> +     }
>
> So, if a compound has a COPY anywhere in it, then a PUTFH with a stale
> filehandle anywhere in the compound will temporarily succeed and result in
> IS_STALE_FH being set.
>
>>       while (!status && resp->opcnt < args->opcnt) {
>>               op = &args->ops[resp->opcnt++];
>>
>> @@ -1810,6 +1843,9 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>
>>               opdesc = OPDESC(op);
>>
>> +             if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
>> +                     goto call_op;
>> +
>
> That means that once IS_STALE_FH is set, we will skip:
>
>         - nofilehandle checking
>         - moved checking
>         - fh_clear_wcc()
>         - resp_size checking
>         - current stateid capture
>
> on every subsequent op.  Can that really be right?
>
>>               if (!current_fh->fh_dentry) {
>>                       if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
>>                               op->status = nfserr_nofilehandle;
>> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>
>>               if (opdesc->op_get_currentstateid)
>>                       opdesc->op_get_currentstateid(cstate, &op->u);
>> +call_op:
>>               op->status = opdesc->op_func(rqstp, cstate, &op->u);
>>
>>               /* Only from SEQUENCE */
>> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>                       if (need_wrongsec_check(rqstp))
>>                               op->status = check_nfsd_access(current_fh->fh_export, rqstp);
>>               }
>> +             /* Only from intra COPY */
>> +             if (cstate->status == nfserr_copy_stalefh) {
>> +                     dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
>> +                             __func__);
>> +                     status = nfserr_stale;
>> +                     nfsd4_adjust_encode(resp);
>
> Are you sure it's safe just to throw away any operations since that
> stale PUTFH?  What if some of those operations had side effects?
>
>> +                     goto out;
>> +             }
>>  encode_op:
>>               if (op->status == nfserr_replay_me) {
>>                       op->replay = &cstate->replay_owner->so_replay;
>> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
>> index 3e08c15..2896a11 100644
>> --- a/fs/nfsd/nfs4xdr.c
>> +++ b/fs/nfsd/nfs4xdr.c
>> @@ -4624,15 +4624,28 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>>       return nfserr_rep_too_big;
>>  }
>>
>> +/** Rewind the encoding to return nfserr_stale on the PUTFH
>> + * in this failed Intra COPY compound
>> + */
>> +void
>> +nfsd4_adjust_encode(struct nfsd4_compoundres *resp)
>> +{
>> +     __be32 *p;
>> +
>> +     p = resp->cstate.putfh_errp;
>> +     *p++ = nfserr_stale;
>> +}
>> +
>>  void
>>  nfsd4_encode_operation(struct nfsd4_compoundres *resp, struct nfsd4_op *op)
>>  {
>>       struct xdr_stream *xdr = &resp->xdr;
>>       struct nfs4_stateowner *so = resp->cstate.replay_owner;
>> +     struct nfsd4_compound_state *cstate = &resp->cstate;
>>       struct svc_rqst *rqstp = resp->rqstp;
>>       int post_err_offset;
>>       nfsd4_enc encoder;
>> -     __be32 *p;
>> +     __be32 *p, *statp;
>>
>>       p = xdr_reserve_space(xdr, 8);
>>       if (!p) {
>> @@ -4641,9 +4654,20 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>>       }
>>       *p++ = cpu_to_be32(op->opnum);
>>       post_err_offset = xdr->buf->len;
>> +     statp = p;
>>
>>       if (op->opnum == OP_ILLEGAL)
>>               goto status;
>> +
>> +     /** This is a COPY compound with a stale source server file handle.
>> +      * If OP_COPY processing determines that this is an intra server to
>> +      * server COPY, then this PUTFH should return nfserr_ stale so the
>> +      * putfh_errp will be set to nfserr_stale. If this is an inter server
>> +      * to server COPY, ignore the nfserr_stale.
>> +      */
>> +     if (op->opnum == OP_PUTFH && HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
>> +             cstate->putfh_errp = statp;
>> +
>>       BUG_ON(op->opnum < 0 || op->opnum >= ARRAY_SIZE(nfsd4_enc_ops) ||
>>              !nfsd4_enc_ops[op->opnum]);
>>       encoder = nfsd4_enc_ops[op->opnum];
>> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
>> index d966068..8d6fb0f 100644
>> --- a/fs/nfsd/nfsd.h
>> +++ b/fs/nfsd/nfsd.h
>> @@ -272,6 +272,8 @@ static inline bool nfsd4_spo_must_allow(struct svc_rqst *rqstp)
>>  #define      nfserr_replay_me        cpu_to_be32(11001)
>>  /* nfs41 replay detected */
>>  #define      nfserr_replay_cache     cpu_to_be32(11002)
>> +/* nfs42 intra copy failed with nfserr_stale */
>> +#define nfserr_copy_stalefh  cpu_to_be32(1103)
>>
>>  /* Check for dir entries '.' and '..' */
>>  #define isdotent(n, l)       (l < 3 && n[0] == '.' && (l == 1 || n[1] == '.'))
>> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
>> index 38fcb4f..aa94295 100644
>> --- a/fs/nfsd/xdr4.h
>> +++ b/fs/nfsd/xdr4.h
>> @@ -45,6 +45,8 @@
>>
>>  #define CURRENT_STATE_ID_FLAG (1<<0)
>>  #define SAVED_STATE_ID_FLAG (1<<1)
>> +#define NO_VERIFY_FH (1<<2)
>> +#define IS_STALE_FH  (1<<3)
>>
>>  #define SET_CSTATE_FLAG(c, f) ((c)->sid_flags |= (f))
>>  #define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f))
>> @@ -63,6 +65,7 @@ struct nfsd4_compound_state {
>>       size_t                  iovlen;
>>       u32                     minorversion;
>>       __be32                  status;
>> +     __be32                  *putfh_errp;
>>       stateid_t       current_stateid;
>>       stateid_t       save_stateid;
>>       /* to indicate current and saved state id presents */
>> @@ -705,6 +708,7 @@ int nfs4svc_decode_compoundargs(struct svc_rqst *, __be32 *,
>>  int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
>>               struct nfsd4_compoundres *);
>>  __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32);
>> +void nfsd4_adjust_encode(struct nfsd4_compoundres *);
>>  void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
>>  void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op);
>>  __be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words,
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[J. Bruce Fields]

On Fri, Sep 08, 2017 at 03:52:53PM -0400, Olga Kornievskaia wrote:
> What if instead the client won't send the 2nd FH in case of the inter
> server to server COPY and it would avoid all this stale handle
> business. So for the "intra" COPY it'll still be PUTFH, SAVEFH, PUTFH,
> COPY but for the "inter" COPY it'll be PUTFH, COPY.

That might be easier to implement, but it'd require a change to the
protocol, right?  I may not understand what you're proposing.

--b.

> 
> On Fri, Sep 8, 2017 at 3:38 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > Some questions below, but really I think this approach can't work.  Keep
> > in mind this has to work not just for the one compound the current
> > client uses, but for *any* compound containing a COPY, including those
> > sent by future clients that may do something more complicated, or by
> > malcious clients that may intentionally send weird compounds to make the
> > server crash.
> >
> > I think we want to flag the filehandle itself somehow, not the cstate.
> >
> > And check very carefully to make sure that ops other than COPY can't get
> > tripped up by one of these foreign filehandles.
> >
> > On Tue, Jul 11, 2017 at 12:44:01PM -0400, Olga Kornievskaia wrote:
> >> The inter server to server COPY source server filehandle
> >> is guaranteed to be stale as the COPY is sent to the destination
> >> server.
> >>
> >> Signed-off-by: Andy Adamson <andros@netapp.com>
> >> ---
> >>  fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
> >>  fs/nfsd/nfs4xdr.c  | 26 +++++++++++++++++++++++++-
> >>  fs/nfsd/nfsd.h     |  2 ++
> >>  fs/nfsd/xdr4.h     |  4 ++++
> >>  4 files changed, 77 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> >> index b49ff31..ceee852 100644
> >> --- a/fs/nfsd/nfs4proc.c
> >> +++ b/fs/nfsd/nfs4proc.c
> >> @@ -496,11 +496,19 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
> >>  nfsd4_putfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> >>           struct nfsd4_putfh *putfh)
> >>  {
> >> +     __be32 ret;
> >> +
> >>       fh_put(&cstate->current_fh);
> >>       cstate->current_fh.fh_handle.fh_size = putfh->pf_fhlen;
> >>       memcpy(&cstate->current_fh.fh_handle.fh_base, putfh->pf_fhval,
> >>              putfh->pf_fhlen);
> >> -     return fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
> >> +     ret = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
> >> +     if (ret == nfserr_stale && HAS_CSTATE_FLAG(cstate, NO_VERIFY_FH)) {
> >> +             CLEAR_CSTATE_FLAG(cstate, NO_VERIFY_FH);
> >> +             SET_CSTATE_FLAG(cstate, IS_STALE_FH);
> >> +             ret = 0;
> >> +     }
> >> +     return ret;
> >>  }
> >>
> >>  static __be32
> >> @@ -533,6 +541,16 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
> >>  nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> >>            void *arg)
> >>  {
> >> +     /**
> >> +     * This is either an inter COPY (most likely) or an intra COPY with a
> >> +     * stale file handle. If the latter, nfsd4_copy will reset the PUTFH to
> >> +     * return nfserr_stale. No fh_dentry, just copy the file handle
> >> +     * to use with the inter COPY READ.
> >> +     */
> >> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
> >> +             cstate->save_fh = cstate->current_fh;
> >> +             return nfs_ok;
> >> +     }
> >>       if (!cstate->current_fh.fh_dentry)
> >>               return nfserr_nofilehandle;
> >>
> >> @@ -1067,6 +1085,13 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
> >>       if (status)
> >>               goto out;
> >>
> >> +     /* Intra copy source fh is stale. PUTFH will fail with ESTALE */
> >> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
> >> +             CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
> >> +             cstate->status = nfserr_copy_stalefh;
> >> +             goto out_put;
> >> +     }
> >> +
> >>       bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
> >>                       dst, copy->cp_dst_pos, copy->cp_count);
> >>
> >> @@ -1081,6 +1106,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
> >>               status = nfs_ok;
> >>       }
> >>
> >> +out_put:
> >>       fput(src);
> >>       fput(dst);
> >>  out:
> >> @@ -1759,6 +1785,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >>       struct nfsd4_compound_state *cstate = &resp->cstate;
> >>       struct svc_fh *current_fh = &cstate->current_fh;
> >>       struct svc_fh *save_fh = &cstate->save_fh;
> >> +     int             i;
> >>       __be32          status;
> >>
> >>       svcxdr_init_encode(rqstp, resp);
> >> @@ -1791,6 +1818,12 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >>               goto encode_op;
> >>       }
> >>
> >> +     /* NFSv4.2 COPY source file handle may be from a different server */
> >> +     for (i = 0; i < args->opcnt; i++) {
> >> +             op = &args->ops[i];
> >> +             if (op->opnum == OP_COPY)
> >> +                     SET_CSTATE_FLAG(cstate, NO_VERIFY_FH);
> >> +     }
> >
> > So, if a compound has a COPY anywhere in it, then a PUTFH with a stale
> > filehandle anywhere in the compound will temporarily succeed and result in
> > IS_STALE_FH being set.
> >
> >>       while (!status && resp->opcnt < args->opcnt) {
> >>               op = &args->ops[resp->opcnt++];
> >>
> >> @@ -1810,6 +1843,9 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >>
> >>               opdesc = OPDESC(op);
> >>
> >> +             if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
> >> +                     goto call_op;
> >> +
> >
> > That means that once IS_STALE_FH is set, we will skip:
> >
> >         - nofilehandle checking
> >         - moved checking
> >         - fh_clear_wcc()
> >         - resp_size checking
> >         - current stateid capture
> >
> > on every subsequent op.  Can that really be right?
> >
> >>               if (!current_fh->fh_dentry) {
> >>                       if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
> >>                               op->status = nfserr_nofilehandle;
> >> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >>
> >>               if (opdesc->op_get_currentstateid)
> >>                       opdesc->op_get_currentstateid(cstate, &op->u);
> >> +call_op:
> >>               op->status = opdesc->op_func(rqstp, cstate, &op->u);
> >>
> >>               /* Only from SEQUENCE */
> >> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >>                       if (need_wrongsec_check(rqstp))
> >>                               op->status = check_nfsd_access(current_fh->fh_export, rqstp);
> >>               }
> >> +             /* Only from intra COPY */
> >> +             if (cstate->status == nfserr_copy_stalefh) {
> >> +                     dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
> >> +                             __func__);
> >> +                     status = nfserr_stale;
> >> +                     nfsd4_adjust_encode(resp);
> >
> > Are you sure it's safe just to throw away any operations since that
> > stale PUTFH?  What if some of those operations had side effects?
> >
> >> +                     goto out;
> >> +             }
> >>  encode_op:
> >>               if (op->status == nfserr_replay_me) {
> >>                       op->replay = &cstate->replay_owner->so_replay;
> >> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
> >> index 3e08c15..2896a11 100644
> >> --- a/fs/nfsd/nfs4xdr.c
> >> +++ b/fs/nfsd/nfs4xdr.c
> >> @@ -4624,15 +4624,28 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
> >>       return nfserr_rep_too_big;
> >>  }
> >>
> >> +/** Rewind the encoding to return nfserr_stale on the PUTFH
> >> + * in this failed Intra COPY compound
> >> + */
> >> +void
> >> +nfsd4_adjust_encode(struct nfsd4_compoundres *resp)
> >> +{
> >> +     __be32 *p;
> >> +
> >> +     p = resp->cstate.putfh_errp;
> >> +     *p++ = nfserr_stale;
> >> +}
> >> +
> >>  void
> >>  nfsd4_encode_operation(struct nfsd4_compoundres *resp, struct nfsd4_op *op)
> >>  {
> >>       struct xdr_stream *xdr = &resp->xdr;
> >>       struct nfs4_stateowner *so = resp->cstate.replay_owner;
> >> +     struct nfsd4_compound_state *cstate = &resp->cstate;
> >>       struct svc_rqst *rqstp = resp->rqstp;
> >>       int post_err_offset;
> >>       nfsd4_enc encoder;
> >> -     __be32 *p;
> >> +     __be32 *p, *statp;
> >>
> >>       p = xdr_reserve_space(xdr, 8);
> >>       if (!p) {
> >> @@ -4641,9 +4654,20 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
> >>       }
> >>       *p++ = cpu_to_be32(op->opnum);
> >>       post_err_offset = xdr->buf->len;
> >> +     statp = p;
> >>
> >>       if (op->opnum == OP_ILLEGAL)
> >>               goto status;
> >> +
> >> +     /** This is a COPY compound with a stale source server file handle.
> >> +      * If OP_COPY processing determines that this is an intra server to
> >> +      * server COPY, then this PUTFH should return nfserr_ stale so the
> >> +      * putfh_errp will be set to nfserr_stale. If this is an inter server
> >> +      * to server COPY, ignore the nfserr_stale.
> >> +      */
> >> +     if (op->opnum == OP_PUTFH && HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
> >> +             cstate->putfh_errp = statp;
> >> +
> >>       BUG_ON(op->opnum < 0 || op->opnum >= ARRAY_SIZE(nfsd4_enc_ops) ||
> >>              !nfsd4_enc_ops[op->opnum]);
> >>       encoder = nfsd4_enc_ops[op->opnum];
> >> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
> >> index d966068..8d6fb0f 100644
> >> --- a/fs/nfsd/nfsd.h
> >> +++ b/fs/nfsd/nfsd.h
> >> @@ -272,6 +272,8 @@ static inline bool nfsd4_spo_must_allow(struct svc_rqst *rqstp)
> >>  #define      nfserr_replay_me        cpu_to_be32(11001)
> >>  /* nfs41 replay detected */
> >>  #define      nfserr_replay_cache     cpu_to_be32(11002)
> >> +/* nfs42 intra copy failed with nfserr_stale */
> >> +#define nfserr_copy_stalefh  cpu_to_be32(1103)
> >>
> >>  /* Check for dir entries '.' and '..' */
> >>  #define isdotent(n, l)       (l < 3 && n[0] == '.' && (l == 1 || n[1] == '.'))
> >> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
> >> index 38fcb4f..aa94295 100644
> >> --- a/fs/nfsd/xdr4.h
> >> +++ b/fs/nfsd/xdr4.h
> >> @@ -45,6 +45,8 @@
> >>
> >>  #define CURRENT_STATE_ID_FLAG (1<<0)
> >>  #define SAVED_STATE_ID_FLAG (1<<1)
> >> +#define NO_VERIFY_FH (1<<2)
> >> +#define IS_STALE_FH  (1<<3)
> >>
> >>  #define SET_CSTATE_FLAG(c, f) ((c)->sid_flags |= (f))
> >>  #define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f))
> >> @@ -63,6 +65,7 @@ struct nfsd4_compound_state {
> >>       size_t                  iovlen;
> >>       u32                     minorversion;
> >>       __be32                  status;
> >> +     __be32                  *putfh_errp;
> >>       stateid_t       current_stateid;
> >>       stateid_t       save_stateid;
> >>       /* to indicate current and saved state id presents */
> >> @@ -705,6 +708,7 @@ int nfs4svc_decode_compoundargs(struct svc_rqst *, __be32 *,
> >>  int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
> >>               struct nfsd4_compoundres *);
> >>  __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32);
> >> +void nfsd4_adjust_encode(struct nfsd4_compoundres *);
> >>  void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
> >>  void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op);
> >>  __be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words,
> >> --
> >> 1.8.3.1
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[Olga Kornievskaia]

On Fri, Sep 8, 2017 at 3:57 PM, J. Bruce Fields <bfields@redhat.com> wrote:
> On Fri, Sep 08, 2017 at 03:52:53PM -0400, Olga Kornievskaia wrote:
>> What if instead the client won't send the 2nd FH in case of the inter
>> server to server COPY and it would avoid all this stale handle
>> business. So for the "intra" COPY it'll still be PUTFH, SAVEFH, PUTFH,
>> COPY but for the "inter" COPY it'll be PUTFH, COPY.
>
> That might be easier to implement, but it'd require a change to the
> protocol, right?  I may not understand what you're proposing.

Sigh. You are right.

>
> --b.
>
>>
>> On Fri, Sep 8, 2017 at 3:38 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
>> > Some questions below, but really I think this approach can't work.  Keep
>> > in mind this has to work not just for the one compound the current
>> > client uses, but for *any* compound containing a COPY, including those
>> > sent by future clients that may do something more complicated, or by
>> > malcious clients that may intentionally send weird compounds to make the
>> > server crash.
>> >
>> > I think we want to flag the filehandle itself somehow, not the cstate.
>> >
>> > And check very carefully to make sure that ops other than COPY can't get
>> > tripped up by one of these foreign filehandles.
>> >
>> > On Tue, Jul 11, 2017 at 12:44:01PM -0400, Olga Kornievskaia wrote:
>> >> The inter server to server COPY source server filehandle
>> >> is guaranteed to be stale as the COPY is sent to the destination
>> >> server.
>> >>
>> >> Signed-off-by: Andy Adamson <andros@netapp.com>
>> >> ---
>> >>  fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
>> >>  fs/nfsd/nfs4xdr.c  | 26 +++++++++++++++++++++++++-
>> >>  fs/nfsd/nfsd.h     |  2 ++
>> >>  fs/nfsd/xdr4.h     |  4 ++++
>> >>  4 files changed, 77 insertions(+), 2 deletions(-)
>> >>
>> >> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
>> >> index b49ff31..ceee852 100644
>> >> --- a/fs/nfsd/nfs4proc.c
>> >> +++ b/fs/nfsd/nfs4proc.c
>> >> @@ -496,11 +496,19 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>> >>  nfsd4_putfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>> >>           struct nfsd4_putfh *putfh)
>> >>  {
>> >> +     __be32 ret;
>> >> +
>> >>       fh_put(&cstate->current_fh);
>> >>       cstate->current_fh.fh_handle.fh_size = putfh->pf_fhlen;
>> >>       memcpy(&cstate->current_fh.fh_handle.fh_base, putfh->pf_fhval,
>> >>              putfh->pf_fhlen);
>> >> -     return fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
>> >> +     ret = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
>> >> +     if (ret == nfserr_stale && HAS_CSTATE_FLAG(cstate, NO_VERIFY_FH)) {
>> >> +             CLEAR_CSTATE_FLAG(cstate, NO_VERIFY_FH);
>> >> +             SET_CSTATE_FLAG(cstate, IS_STALE_FH);
>> >> +             ret = 0;
>> >> +     }
>> >> +     return ret;
>> >>  }
>> >>
>> >>  static __be32
>> >> @@ -533,6 +541,16 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>> >>  nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>> >>            void *arg)
>> >>  {
>> >> +     /**
>> >> +     * This is either an inter COPY (most likely) or an intra COPY with a
>> >> +     * stale file handle. If the latter, nfsd4_copy will reset the PUTFH to
>> >> +     * return nfserr_stale. No fh_dentry, just copy the file handle
>> >> +     * to use with the inter COPY READ.
>> >> +     */
>> >> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
>> >> +             cstate->save_fh = cstate->current_fh;
>> >> +             return nfs_ok;
>> >> +     }
>> >>       if (!cstate->current_fh.fh_dentry)
>> >>               return nfserr_nofilehandle;
>> >>
>> >> @@ -1067,6 +1085,13 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>> >>       if (status)
>> >>               goto out;
>> >>
>> >> +     /* Intra copy source fh is stale. PUTFH will fail with ESTALE */
>> >> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
>> >> +             CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
>> >> +             cstate->status = nfserr_copy_stalefh;
>> >> +             goto out_put;
>> >> +     }
>> >> +
>> >>       bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
>> >>                       dst, copy->cp_dst_pos, copy->cp_count);
>> >>
>> >> @@ -1081,6 +1106,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>> >>               status = nfs_ok;
>> >>       }
>> >>
>> >> +out_put:
>> >>       fput(src);
>> >>       fput(dst);
>> >>  out:
>> >> @@ -1759,6 +1785,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>> >>       struct nfsd4_compound_state *cstate = &resp->cstate;
>> >>       struct svc_fh *current_fh = &cstate->current_fh;
>> >>       struct svc_fh *save_fh = &cstate->save_fh;
>> >> +     int             i;
>> >>       __be32          status;
>> >>
>> >>       svcxdr_init_encode(rqstp, resp);
>> >> @@ -1791,6 +1818,12 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>> >>               goto encode_op;
>> >>       }
>> >>
>> >> +     /* NFSv4.2 COPY source file handle may be from a different server */
>> >> +     for (i = 0; i < args->opcnt; i++) {
>> >> +             op = &args->ops[i];
>> >> +             if (op->opnum == OP_COPY)
>> >> +                     SET_CSTATE_FLAG(cstate, NO_VERIFY_FH);
>> >> +     }
>> >
>> > So, if a compound has a COPY anywhere in it, then a PUTFH with a stale
>> > filehandle anywhere in the compound will temporarily succeed and result in
>> > IS_STALE_FH being set.
>> >
>> >>       while (!status && resp->opcnt < args->opcnt) {
>> >>               op = &args->ops[resp->opcnt++];
>> >>
>> >> @@ -1810,6 +1843,9 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>> >>
>> >>               opdesc = OPDESC(op);
>> >>
>> >> +             if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
>> >> +                     goto call_op;
>> >> +
>> >
>> > That means that once IS_STALE_FH is set, we will skip:
>> >
>> >         - nofilehandle checking
>> >         - moved checking
>> >         - fh_clear_wcc()
>> >         - resp_size checking
>> >         - current stateid capture
>> >
>> > on every subsequent op.  Can that really be right?
>> >
>> >>               if (!current_fh->fh_dentry) {
>> >>                       if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
>> >>                               op->status = nfserr_nofilehandle;
>> >> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>> >>
>> >>               if (opdesc->op_get_currentstateid)
>> >>                       opdesc->op_get_currentstateid(cstate, &op->u);
>> >> +call_op:
>> >>               op->status = opdesc->op_func(rqstp, cstate, &op->u);
>> >>
>> >>               /* Only from SEQUENCE */
>> >> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>> >>                       if (need_wrongsec_check(rqstp))
>> >>                               op->status = check_nfsd_access(current_fh->fh_export, rqstp);
>> >>               }
>> >> +             /* Only from intra COPY */
>> >> +             if (cstate->status == nfserr_copy_stalefh) {
>> >> +                     dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
>> >> +                             __func__);
>> >> +                     status = nfserr_stale;
>> >> +                     nfsd4_adjust_encode(resp);
>> >
>> > Are you sure it's safe just to throw away any operations since that
>> > stale PUTFH?  What if some of those operations had side effects?
>> >
>> >> +                     goto out;
>> >> +             }
>> >>  encode_op:
>> >>               if (op->status == nfserr_replay_me) {
>> >>                       op->replay = &cstate->replay_owner->so_replay;
>> >> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
>> >> index 3e08c15..2896a11 100644
>> >> --- a/fs/nfsd/nfs4xdr.c
>> >> +++ b/fs/nfsd/nfs4xdr.c
>> >> @@ -4624,15 +4624,28 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>> >>       return nfserr_rep_too_big;
>> >>  }
>> >>
>> >> +/** Rewind the encoding to return nfserr_stale on the PUTFH
>> >> + * in this failed Intra COPY compound
>> >> + */
>> >> +void
>> >> +nfsd4_adjust_encode(struct nfsd4_compoundres *resp)
>> >> +{
>> >> +     __be32 *p;
>> >> +
>> >> +     p = resp->cstate.putfh_errp;
>> >> +     *p++ = nfserr_stale;
>> >> +}
>> >> +
>> >>  void
>> >>  nfsd4_encode_operation(struct nfsd4_compoundres *resp, struct nfsd4_op *op)
>> >>  {
>> >>       struct xdr_stream *xdr = &resp->xdr;
>> >>       struct nfs4_stateowner *so = resp->cstate.replay_owner;
>> >> +     struct nfsd4_compound_state *cstate = &resp->cstate;
>> >>       struct svc_rqst *rqstp = resp->rqstp;
>> >>       int post_err_offset;
>> >>       nfsd4_enc encoder;
>> >> -     __be32 *p;
>> >> +     __be32 *p, *statp;
>> >>
>> >>       p = xdr_reserve_space(xdr, 8);
>> >>       if (!p) {
>> >> @@ -4641,9 +4654,20 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>> >>       }
>> >>       *p++ = cpu_to_be32(op->opnum);
>> >>       post_err_offset = xdr->buf->len;
>> >> +     statp = p;
>> >>
>> >>       if (op->opnum == OP_ILLEGAL)
>> >>               goto status;
>> >> +
>> >> +     /** This is a COPY compound with a stale source server file handle.
>> >> +      * If OP_COPY processing determines that this is an intra server to
>> >> +      * server COPY, then this PUTFH should return nfserr_ stale so the
>> >> +      * putfh_errp will be set to nfserr_stale. If this is an inter server
>> >> +      * to server COPY, ignore the nfserr_stale.
>> >> +      */
>> >> +     if (op->opnum == OP_PUTFH && HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
>> >> +             cstate->putfh_errp = statp;
>> >> +
>> >>       BUG_ON(op->opnum < 0 || op->opnum >= ARRAY_SIZE(nfsd4_enc_ops) ||
>> >>              !nfsd4_enc_ops[op->opnum]);
>> >>       encoder = nfsd4_enc_ops[op->opnum];
>> >> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
>> >> index d966068..8d6fb0f 100644
>> >> --- a/fs/nfsd/nfsd.h
>> >> +++ b/fs/nfsd/nfsd.h
>> >> @@ -272,6 +272,8 @@ static inline bool nfsd4_spo_must_allow(struct svc_rqst *rqstp)
>> >>  #define      nfserr_replay_me        cpu_to_be32(11001)
>> >>  /* nfs41 replay detected */
>> >>  #define      nfserr_replay_cache     cpu_to_be32(11002)
>> >> +/* nfs42 intra copy failed with nfserr_stale */
>> >> +#define nfserr_copy_stalefh  cpu_to_be32(1103)
>> >>
>> >>  /* Check for dir entries '.' and '..' */
>> >>  #define isdotent(n, l)       (l < 3 && n[0] == '.' && (l == 1 || n[1] == '.'))
>> >> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
>> >> index 38fcb4f..aa94295 100644
>> >> --- a/fs/nfsd/xdr4.h
>> >> +++ b/fs/nfsd/xdr4.h
>> >> @@ -45,6 +45,8 @@
>> >>
>> >>  #define CURRENT_STATE_ID_FLAG (1<<0)
>> >>  #define SAVED_STATE_ID_FLAG (1<<1)
>> >> +#define NO_VERIFY_FH (1<<2)
>> >> +#define IS_STALE_FH  (1<<3)
>> >>
>> >>  #define SET_CSTATE_FLAG(c, f) ((c)->sid_flags |= (f))
>> >>  #define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f))
>> >> @@ -63,6 +65,7 @@ struct nfsd4_compound_state {
>> >>       size_t                  iovlen;
>> >>       u32                     minorversion;
>> >>       __be32                  status;
>> >> +     __be32                  *putfh_errp;
>> >>       stateid_t       current_stateid;
>> >>       stateid_t       save_stateid;
>> >>       /* to indicate current and saved state id presents */
>> >> @@ -705,6 +708,7 @@ int nfs4svc_decode_compoundargs(struct svc_rqst *, __be32 *,
>> >>  int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
>> >>               struct nfsd4_compoundres *);
>> >>  __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32);
>> >> +void nfsd4_adjust_encode(struct nfsd4_compoundres *);
>> >>  void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
>> >>  void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op);
>> >>  __be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words,
>> >> --
>> >> 1.8.3.1
>> >>
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 28/42] NFSD add nfs4 inter ssc to nfsd4_copy

[J. Bruce Fields]

On Tue, Jul 11, 2017 at 12:44:02PM -0400, Olga Kornievskaia wrote:
> Given a universal address, mount the source server from the destination
> server.  Use an internal mount. Call the NFS client nfs42_ssc_open to
> obtain the NFS struct file suitable for nfsd_copy_range.
> 
> Add Kconfig dependencies for inter server to server copy
> 
> Signed-off-by: Andy Adamson <andros@netapp.com>
> ---
>  fs/nfsd/Kconfig      |  10 ++
>  fs/nfsd/nfs4proc.c   | 262 +++++++++++++++++++++++++++++++++++++++++++++++++--
>  include/linux/nfs4.h |   1 +
>  3 files changed, 263 insertions(+), 10 deletions(-)
> 
> diff --git a/fs/nfsd/Kconfig b/fs/nfsd/Kconfig
> index 20b1c17..37ff3d5 100644
> --- a/fs/nfsd/Kconfig
> +++ b/fs/nfsd/Kconfig
> @@ -131,6 +131,16 @@ config NFSD_FLEXFILELAYOUT
>  
>  	  If unsure, say N.
>  
> +config NFSD_V4_2_INTER_SSC
> +	bool "NFSv4.2 inter server to server COPY"
> +	depends on NFSD_V4 && NFS_V4_1 && NFS_V4_2
> +	help
> +	  This option enables support for NFSv4.2 inter server to
> +	  server copy where the destination server calls the NFSv4.2
> +	  client to read the data to copy from the source server.
> +
> +	  If unsure, say N.
> +
>  config NFSD_V4_SECURITY_LABEL
>  	bool "Provide Security Label support for NFSv4 server"
>  	depends on NFSD_V4 && SECURITY
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index ceee852..b1095e9 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -1072,16 +1072,227 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>  	return status;
>  }
>  
> +#ifdef CONFIG_NFSD_V4_2_INTER_SSC
> +
> +extern struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
> +				   struct nfs_fh *src_fh,
> +				   nfs4_stateid *stateid);
> +extern struct file *nfs42_ssc_close(struct file *filep);
> +
> +extern void nfs_sb_deactive(struct super_block *sb);
> +
> +#define NFSD42_INTERSSC_RAWDATA "minorversion=2,vers=4,addr=%s,clientaddr=%s"

INTERSCC_RAWDATA isn't the most helpful name.  Maybe MOUNTOPTS could be
in there?

> +
> +/**
> + * Support one copy source server for now.
> + */
> +static struct vfsmount *
> +nfsd4_interssc_connect(struct nl4_servers *nss, struct svc_rqst *rqstp)
> +{
> +	struct file_system_type *type;
> +	struct vfsmount *ss_mnt;
> +	struct nfs42_netaddr *naddr;
> +	struct sockaddr_storage tmp_addr;
> +	size_t tmp_addrlen, match_netid_len = 3;
> +	char *startsep = "", *endsep = "", *match_netid = "tcp";
> +	char *ipaddr, *ipaddr2, *raw_data;
> +	int len, raw_len, status = -EINVAL;
> +
> +	/* Currently support for one NL4_NETADDR source server */
> +	if (nss->nl_svr->nl4_type != NL4_NETADDR) {
> +		WARN(nss->nl_svr->nl4_type != NL4_NETADDR,
> +			"nfsd4_copy src server not NL4_NETADDR\n");
> +		goto out_err;
> +	}

We already checked this at xdr decoding time, didn't we?  Well, maybe
i'ts still worth a WARN.  Better, as mentioned before, maybe the data
structures just should assume only 1 NL4_NETADDR source server.

> +
> +	naddr = &nss->nl_svr->u.nl4_addr;
> +
> +	tmp_addrlen = rpc_uaddr2sockaddr(SVC_NET(rqstp), naddr->na_uaddr,
> +					naddr->na_uaddr_len,
> +					(struct sockaddr *)&tmp_addr,
> +					sizeof(tmp_addr));
> +	if (tmp_addrlen == 0)
> +		goto out_err;
> +
> +	if (tmp_addr.ss_family == AF_INET6) {
> +		startsep = "[";
> +		endsep = "]";
> +		match_netid = "tcp6";
> +		match_netid_len = 4;
> +	}

What about the tcp/ipv4 case?  Oh, I see, it's handled by the
initialization above.  That's a little confusing.

> +
> +	if (naddr->na_netid_len != match_netid_len ||
> +	    strncmp(naddr->na_netid, match_netid, naddr->na_netid_len))
> +		goto out_err;
> +
> +	/* Construct the raw data for the vfs_kern_mount call */
> +	len = RPC_MAX_ADDRBUFLEN + 1;
> +	ipaddr = kzalloc(len, GFP_KERNEL);
> +	if (!ipaddr)
> +		goto out_err;
> +
> +	rpc_ntop((struct sockaddr *)&tmp_addr, ipaddr, len);
> +
> +	/* 2 for ipv6 endsep and startsep. 3 for ":/" and trailing '/0'*/
> +	ipaddr2 = kzalloc(len + 5, GFP_KERNEL);
> +	if (!ipaddr2)
> +		goto out_free_ipaddr;
> +
> +	rpc_ntop((struct sockaddr *)&rqstp->rq_daddr, ipaddr2, len + 5);
> +
> +	raw_len = strlen(NFSD42_INTERSSC_RAWDATA) + strlen(ipaddr) +
> +			strlen(ipaddr2);
> +	raw_data = kzalloc(raw_len, GFP_KERNEL);
> +	if (!raw_data)
> +		goto out_free_ipaddr2;
> +
> +	snprintf(raw_data, raw_len, NFSD42_INTERSSC_RAWDATA, ipaddr,
> +		 ipaddr2);

Could "raw_data" be named more helpfully?

> +
> +	status = -ENODEV;
> +	type = get_fs_type("nfs");
> +	if (!type)
> +		goto out_free_rawdata;
> +
> +	/* Set the server:<export> for the vfs_kerne_mount call */
> +	memset(ipaddr2, 0, len + 5);
> +	snprintf(ipaddr2, len + 5, "%s%s%s:/", startsep, ipaddr, endsep);
> +
> +	dprintk("%s  Raw mount data:  %s server:export %s\n", __func__,
> +		raw_data, ipaddr2);
> +
> +	/* Use an 'internal' mount: MS_KERNMOUNT -> MNT_INTERNAL */
> +	ss_mnt = vfs_kern_mount(type, MS_KERNMOUNT, ipaddr2, raw_data);

I wonder if creating a mount (even a kernel internal one) has any
unexpected side effects.  Is it a problem that it could share caches
with any already-existing mounts of that export?

> +	if (IS_ERR(ss_mnt)) {
> +		status = PTR_ERR(ss_mnt);
> +		goto out_free_rawdata;
> +	}
> +
> +	kfree(raw_data);
> +	kfree(ipaddr2);
> +	kfree(ipaddr);
> +
> +	return ss_mnt;
> +
> +out_free_rawdata:
> +	kfree(raw_data);
> +out_free_ipaddr2:
> +	kfree(ipaddr2);
> +out_free_ipaddr:
> +	kfree(ipaddr);
> +out_err:
> +	dprintk("--> %s ERROR %d\n", __func__, status);
> +	return ERR_PTR(status);
> +}
> +
> +static void
> +nfsd4_interssc_disconnect(struct vfsmount *ss_mnt)
> +{
> +	mntput(ss_mnt);
> +	nfs_sb_deactive(ss_mnt->mnt_sb);

OK, I don't claim to understand that.

Are you sure it's still safe to dereference ss_mnt after calling
mntput() on it?

> +}
> +
> +/**
> + * nfsd4_setup_inter_ssc
> + *
> + * Verify COPY destination stateid.
> + * Connect to the source server with NFSv4.1.
> + * Create the source struct file for nfsd_copy_range.
> + * Called with COPY cstate:
> + *    SAVED_FH: source filehandle
> + *    CURRENT_FH: destination filehandle
> + *
> + * Returns errno (not nfserrxxx)
> + */
> +static struct vfsmount *
> +nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
> +			struct nfsd4_compound_state *cstate,
> +			struct nfsd4_copy *copy, struct file **src,
> +			struct file **dst)
> +{
> +	struct svc_fh *s_fh = NULL;
> +	stateid_t *s_stid = &copy->cp_src_stateid;
> +	struct nfs_fh fh;
> +	nfs4_stateid stateid;
> +	struct file *filp;
> +	struct vfsmount *ss_mnt;
> +	__be32 status;
> +
> +	/* Verify the destination stateid and set dst struct file*/
> +	status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
> +					&copy->cp_dst_stateid,
> +					WR_STATE, dst, NULL);
> +	if (status) {
> +		ss_mnt = ERR_PTR(be32_to_cpu(status));
> +		goto out;
> +	}
> +
> +	/* Inter copy source fh is always stale */

Not necessarily.

> +	CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
> +
> +	ss_mnt = nfsd4_interssc_connect(&copy->cp_src, rqstp);
> +	if (IS_ERR(ss_mnt))
> +		goto out;
> +
> +	s_fh = &cstate->save_fh;
> +
> +	fh.size = s_fh->fh_handle.fh_size;
> +	memcpy(fh.data, &s_fh->fh_handle.fh_base, fh.size);
> +	stateid.seqid = s_stid->si_generation;
> +	memcpy(stateid.other, (void *)&s_stid->si_opaque,
> +		sizeof(stateid_opaque_t));
> +
> +	filp =  nfs42_ssc_open(ss_mnt, &fh, &stateid);

So that was defined in the client-side patches?

> +	if (IS_ERR(filp)) {
> +		nfsd4_interssc_disconnect(ss_mnt);
> +		return ERR_CAST(filp);
> +	}
> +	*src = filp;
> +out:
> +	return ss_mnt;
> +}
> +
> +static void
> +nfsd4_cleanup_inter_ssc(struct vfsmount *ss_mnt, struct file *src,
> +			struct file *dst)
> +{
> +	nfs42_ssc_close(src);
> +	fput(src);
> +	fput(dst);
> +
> +	nfsd4_interssc_disconnect(ss_mnt);
> +
> +}
> +
> +#else /* CONFIG_NFSD_V4_2_INTER_SSC */
> +
> +static struct vfsmount *
> +nfsd4_setup_inter_ssc(struct svc_rqst *rqstp,
> +			struct nfsd4_compound_state *cstate,
> +			struct nfsd4_copy *copy, struct file **src,
> +			struct file **dst)
> +{
> +	return ERR_PTR(-EINVAL);
> +}
> +
> +static void
> +nfsd4_cleanup_inter_ssc(struct vfsmount *ss_mnt, struct file *src,
> +			struct file *dst)
> +{
> +}
> +
> +#endif /* CONFIG_NFSD_V4_2_INTER_SSC */
> +
>  static __be32
> -nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> -		struct nfsd4_copy *copy)
> +nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
> +		      struct nfsd4_compound_state *cstate,
> +		      struct nfsd4_copy *copy, struct file **src,
> +		      struct file **dst)
>  {
> -	struct file *src, *dst;
>  	__be32 status;
> -	ssize_t bytes;
>  
> -	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid, &src,
> -				   &copy->cp_dst_stateid, &dst);
> +	status = nfsd4_verify_copy(rqstp, cstate, &copy->cp_src_stateid, src,
> +				   &copy->cp_dst_stateid, dst);
>  	if (status)
>  		goto out;
>  
> @@ -1089,7 +1300,37 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>  	if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
>  		CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
>  		cstate->status = nfserr_copy_stalefh;
> -		goto out_put;
> +	}
> +out:
> +	return status;
> +}
> +
> +static void
> +nfsd4_cleanup_intra_ssc(struct file *src, struct file *dst)
> +{
> +	fput(src);
> +	fput(dst);
> +}
> +
> +static __be32
> +nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> +	struct nfsd4_copy *copy)
> +{
> +	struct vfsmount *ss_mnt = NULL;
> +	struct file *src, *dst;
> +	__be32 status;
> +	ssize_t bytes;
> +
> +	if (copy->cp_src.nl_nsvr > 0) { /* Inter server SSC */
> +		ss_mnt = nfsd4_setup_inter_ssc(rqstp, cstate, copy, &src, &dst);
> +		if (IS_ERR(ss_mnt)) {
> +			status = nfserrno(PTR_ERR(ss_mnt));
> +			goto out;

Touched on before, I think: this could use some checking to make sure
that the error returned is useful to the client.  (I'm not sure exactly
what the most likely failures are, though.)

--b.

> +		}
> +	} else {
> +		status = nfsd4_setup_intra_ssc(rqstp, cstate, copy, &src, &dst);
> +		if (status)
> +			goto out;
>  	}
>  
>  	bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
> @@ -1106,9 +1347,10 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>  		status = nfs_ok;
>  	}
>  
> -out_put:
> -	fput(src);
> -	fput(dst);
> +	if (copy->cp_src.nl_nsvr > 0)   /* Inter server SSC */
> +		nfsd4_cleanup_inter_ssc(ss_mnt, src, dst);
> +	else
> +		nfsd4_cleanup_intra_ssc(src, dst);
>  out:
>  	return status;
>  }
> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
> index 5aa36ac..843443b 100644
> --- a/include/linux/nfs4.h
> +++ b/include/linux/nfs4.h
> @@ -16,6 +16,7 @@
>  #include <linux/uidgid.h>
>  #include <uapi/linux/nfs4.h>
>  #include <linux/sunrpc/msg_prot.h>
> +#include <linux/nfs.h>
>  
>  enum nfs4_acl_whotype {
>  	NFS4_ACL_WHO_NAMED = 0,
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 16/42] NFS NFSD defining nl4_servers structure needed by both

[Olga Kornievskaia]

On Wed, Sep 6, 2017 at 4:35 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Tue, Jul 11, 2017 at 12:43:50PM -0400, Olga Kornievskaia wrote:
>> These structures are needed by COPY_NOTIFY on the client and needed
>> by the nfsd as well
>>
>> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
>> ---
>>  include/linux/nfs4.h | 33 +++++++++++++++++++++++++++++++++
>>  1 file changed, 33 insertions(+)
>>
>> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
>> index 7262908..4179c78 100644
>> --- a/include/linux/nfs4.h
>> +++ b/include/linux/nfs4.h
>> @@ -15,6 +15,7 @@
>>  #include <linux/list.h>
>>  #include <linux/uidgid.h>
>>  #include <uapi/linux/nfs4.h>
>> +#include <linux/sunrpc/msg_prot.h>
>>
>>  enum nfs4_acl_whotype {
>>       NFS4_ACL_WHO_NAMED = 0,
>> @@ -659,4 +660,36 @@ struct nfs4_op_map {
>>       } u;
>>  };
>>
>> +struct nfs42_netaddr {
>> +     unsigned int    na_netid_len;
>> +     char            na_netid[RPCBIND_MAXNETIDLEN + 1];
>> +     unsigned int    na_uaddr_len;
>> +     char            na_uaddr[RPCBIND_MAXUADDRLEN + 1];
>> +};
>> +
>> +enum netloc_type4 {
>> +     NL4_NAME                = 1,
>> +     NL4_URL                 = 2,
>> +     NL4_NETADDR             = 3,
>> +};
>> +
>> +struct nl4_server {
>> +     enum netloc_type4       nl4_type;
>> +     union {
>> +             struct { /* NL4_NAME, NL4_URL */
>> +                     int     nl4_str_sz;
>> +                     char    nl4_str[NFS4_OPAQUE_LIMIT + 1];
>> +             };
>> +             struct nfs42_netaddr    nl4_addr; /* NL4_NETADDR */
>> +     } u;
>> +};
>> +
>> +/*  support 1 nl4_server for now */
>> +#define NFS42_MAX_SSC_SRC       1
>
> In that case, let's just build that assumption into the data structures
> and embed struct nl4_server in nl4_servers and drop the unnecessary
> kmalloc()s.

Wouldn't we still want to malloc as the structure is too big to be
allocated on the stack?

Anna has also suggested dropping nl4_servers since the code sets maximum to 1.
I have resisted making the change because I felt like it was easier to
increase the
MAX_SSC_SRC to say something like 10 and support a multi-home server. Then
the destination server can add smarts to chose between the available address to
connect to the source server with.

Is this not a good argument to keep the structures? Should I change
value to 2 to
indicate desire to support multi-path?

>
> --b.
>
>> +
>> +struct nl4_servers {
>> +     int                     nl_nsvr;
>> +     struct nl4_server       *nl_svr;
>> +};
>> +
>>  #endif
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 16/42] NFS NFSD defining nl4_servers structure needed by both

[J. Bruce Fields]

On Fri, Sep 08, 2017 at 04:51:59PM -0400, Olga Kornievskaia wrote:
> On Wed, Sep 6, 2017 at 4:35 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > On Tue, Jul 11, 2017 at 12:43:50PM -0400, Olga Kornievskaia wrote:
> >> These structures are needed by COPY_NOTIFY on the client and needed
> >> by the nfsd as well
> >>
> >> Signed-off-by: Olga Kornievskaia <kolga@netapp.com>
> >> ---
> >>  include/linux/nfs4.h | 33 +++++++++++++++++++++++++++++++++
> >>  1 file changed, 33 insertions(+)
> >>
> >> diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h
> >> index 7262908..4179c78 100644
> >> --- a/include/linux/nfs4.h
> >> +++ b/include/linux/nfs4.h
> >> @@ -15,6 +15,7 @@
> >>  #include <linux/list.h>
> >>  #include <linux/uidgid.h>
> >>  #include <uapi/linux/nfs4.h>
> >> +#include <linux/sunrpc/msg_prot.h>
> >>
> >>  enum nfs4_acl_whotype {
> >>       NFS4_ACL_WHO_NAMED = 0,
> >> @@ -659,4 +660,36 @@ struct nfs4_op_map {
> >>       } u;
> >>  };
> >>
> >> +struct nfs42_netaddr {
> >> +     unsigned int    na_netid_len;
> >> +     char            na_netid[RPCBIND_MAXNETIDLEN + 1];
> >> +     unsigned int    na_uaddr_len;
> >> +     char            na_uaddr[RPCBIND_MAXUADDRLEN + 1];
> >> +};
> >> +
> >> +enum netloc_type4 {
> >> +     NL4_NAME                = 1,
> >> +     NL4_URL                 = 2,
> >> +     NL4_NETADDR             = 3,
> >> +};
> >> +
> >> +struct nl4_server {
> >> +     enum netloc_type4       nl4_type;
> >> +     union {
> >> +             struct { /* NL4_NAME, NL4_URL */
> >> +                     int     nl4_str_sz;
> >> +                     char    nl4_str[NFS4_OPAQUE_LIMIT + 1];
> >> +             };
> >> +             struct nfs42_netaddr    nl4_addr; /* NL4_NETADDR */
> >> +     } u;
> >> +};
> >> +
> >> +/*  support 1 nl4_server for now */
> >> +#define NFS42_MAX_SSC_SRC       1
> >
> > In that case, let's just build that assumption into the data structures
> > and embed struct nl4_server in nl4_servers and drop the unnecessary
> > kmalloc()s.
> 
> Wouldn't we still want to malloc as the structure is too big to be
> allocated on the stack?

Is it allocated on the stack?  I thought it was embedded in
nfsd4_compoundargs.

Anyway, I'm not that fixated on removing kmalloc()'s, it's OK if you
need them.

Looking at that definition again, NFS4_OPAQUE_LIMIT is 1K, that's bigger
than I'd want embedded in nfsd4_compoundargs too, so yes, it might still
be worth kmalloc()'ing.  Or consider whether we really need the maximum
size for our current use (just ascii-encoded IP addresses?), or whether
we should do the parsing earlier and just store the resulting socket
address.

> Anna has also suggested dropping nl4_servers since the code sets maximum to 1.
> I have resisted making the change because I felt like it was easier to
> increase the
> MAX_SSC_SRC to say something like 10 and support a multi-home server. Then
> the destination server can add smarts to chose between the available address to
> connect to the source server with.
> 
> Is this not a good argument to keep the structures? Should I change
> value to 2 to
> indicate desire to support multi-path?

In the first version of this patchset I recommend dropping anything
that's not necessary.

It's easy enough to add this back in later.  The hard part is going to
be getting in any server-to-server patches at all, so we need to make
them as simple as possible.

The only reason to keep this in the initial version, I think, would be
if you think it's an important enough feature that server-to-server copy
isn't useful at all without it.

--b.

Re: [RFC v3 25/42] NFSD add COPY_NOTIFY operation

[Olga Kornievskaia]

On Wed, Sep 6, 2017 at 5:37 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Tue, Jul 11, 2017 at 12:43:59PM -0400, Olga Kornievskaia wrote:
>> +struct nfsd4_copy_notify {
>> +     /* request */
>> +     stateid_t               cpn_src_stateid;
>> +     struct nl4_server       cpn_dst;
>> +
>> +     /* response */
>> +     /* Note: cpn_src_stateid is used for cnr_stateid */
>
> That might be worth a little more explanation.

I'm opting for removing it from this patch? It's just valid for that
patch saying that for the
moment the same cpn_src_stateid is returned fort he cnr_stateid in the
later patch (29)
we will add the generation of the unique stateid used for the copy.

>
> --b.
>
>> +     u64                     cpn_sec;
>> +     u32                     cpn_nsec;
>> +     struct nl4_servers      cpn_src;
>> +};
>> +
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 25/42] NFSD add COPY_NOTIFY operation

[Olga Kornievskaia]

On Wed, Sep 6, 2017 at 5:34 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Tue, Jul 11, 2017 at 12:43:59PM -0400, Olga Kornievskaia wrote:
>> Signed-off-by: Andy Adamson <andros@netapp.com>
>> ---
>>  fs/nfsd/nfs4proc.c |  98 ++++++++++++++++++++++++++++++++++++++++++++++
>>  fs/nfsd/nfs4xdr.c  | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>  fs/nfsd/xdr4.h     |  13 +++++++
>>  3 files changed, 221 insertions(+), 2 deletions(-)
>>
>> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
>> index dadb3bf..ed6b9f2 100644
>> --- a/fs/nfsd/nfs4proc.c
>> +++ b/fs/nfsd/nfs4proc.c
>> @@ -35,6 +35,7 @@
>>  #include <linux/file.h>
>>  #include <linux/falloc.h>
>>  #include <linux/slab.h>
>> +#include <linux/sunrpc/addr.h>
>>
>>  #include "idmap.h"
>>  #include "cache.h"
>> @@ -1087,6 +1088,82 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>>  }
>>
>>  static __be32
>> +nfsd4_set_src_nl4_netaddr(struct svc_rqst *rqstp, struct nl4_servers *svrs)
>> +{
>> +     const struct sockaddr *addr = (struct sockaddr *)&rqstp->rq_daddr;
>> +     const struct sockaddr_in *sin = (const struct sockaddr_in *)addr;
>> +     const struct sockaddr_in6 *sin6 = (const struct sockaddr_in6 *)addr;
>> +     int uaddr_len = rqstp->rq_daddrlen + 4 + 1; /* port (4) and '\0' (1) */
>> +     struct nfs42_netaddr *naddr;
>> +     size_t ret;
>> +     unsigned short port;
>> +
>> +     /* freed in nfsd4_encode_copy_notify */
>> +     svrs->nl_svr = kmalloc_array(svrs->nl_nsvr, sizeof(struct nl4_server),
>> +                             GFP_KERNEL);
>> +     if (svrs->nl_svr == NULL)
>> +             return nfserrno(-ENOMEM);
>> +
>> +     svrs->nl_svr->nl4_type = NL4_NETADDR;
>> +     naddr = &svrs->nl_svr->u.nl4_addr;
>> +
>> +     switch (addr->sa_family) {
>> +     case AF_INET:
>> +             port = ntohs(sin->sin_port);
>> +             sprintf(naddr->na_netid, "tcp");
>> +             naddr->na_netid_len = 3;
>> +             break;
>> +     case AF_INET6:
>> +             port = ntohs(sin6->sin6_port);
>> +             sprintf(naddr->na_netid, "tcp6");
>> +             naddr->na_netid_len = 4;
>> +             break;
>> +     default:
>> +             dprintk("NFSD  nfsd4_set_notify_src: unknown address type: %d",
>> +                     addr->sa_family);
>> +             kfree(svrs->nl_svr);
>> +             return nfserrno(-EINVAL);
>> +     }
>> +     ret = rpc_ntop(addr, naddr->na_uaddr, sizeof(naddr->na_uaddr));
>> +     snprintf(naddr->na_uaddr + ret, uaddr_len, ".%u.%u",
>> +              port >> 8, port & 255);
>> +     naddr->na_uaddr_len = strlen(naddr->na_uaddr);
>> +     return 0;
>> +}
>> +
>> +static __be32
>> +nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>> +               struct nfsd4_copy_notify *cn)
>> +{
>> +     __be32 status;
>> +     struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
>> +
>> +     status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
>> +                                     &cn->cpn_src_stateid, RD_STATE, NULL,
>> +                                     NULL);
>> +     if (status)
>> +             return status;
>> +
>> +     cn->cpn_sec = nn->nfsd4_lease;
>> +     cn->cpn_nsec = 0;
>> +
>> +
>> +     /** XXX Save cpn_src_statid, cpn_src, and any other returned source
>> +      * server addresses on which the source server is williing to accept
>> +      * connections from the destination e.g. what is returned in cpn_src,
>> +      * to verify READ from dest server.
>> +      */
>
> Either we should fix the stuff mentioned in the comment or figure out
> why it's not necessary.

Sorry yes this comment should be removed from this patch. Another patch (29)
removes it.

> I guess I don't see what the value would be of verifying the IP address.

Correct no value is there. Also the READ request can come from a different IP
then what the client sends in COPY_NOTIFY. Destination server can be
multi-homed and share a high speed link with the source server and not with
the client.

> Our choice of IP address to return here is a little arbitrary anyway.

What the source server is returning shouldn't be arbitrary it should return
all the IPs (networks) that it might want to receive requests from the
destination
server.

> And I assume the eventualy READ calls are going to be verified with all
> the usual export and file permission checks.
>
> If we don't save the cna_src_stateid, then I guess that means the COPY
> might still succeed after the original OPEN/LOCK stateid is no longer
> valid?  I can't decide if that will cause problems in practice or not.

No we store the stateid (it's a part of the nfsd4_copy_notify structure and
gets saved from nfs4_preprocess_stateid_op. Then in patch 29 we generate
the unique stateid to be used for the copy itself and its validity is
tied to the
open/lock stateid and also it has to start to be used within the configured
lease time (nfsd_net->nfsd4_lease value).



>
>>  static __be32
>> +nfsd42_encode_nl4_server(struct nfsd4_compoundres *resp, struct nl4_server *ns)
>> +{
>> +     struct xdr_stream *xdr = &resp->xdr;
>> +     struct nfs42_netaddr *addr;
>> +     __be32 *p;
>> +
>> +     p = xdr_reserve_space(xdr, 4);
>> +     *p++ = cpu_to_be32(ns->nl4_type);
>> +
>> +     switch (ns->nl4_type) {
>> +     case NL4_NAME:
>> +     case NL4_URL:
>> +             p = xdr_reserve_space(xdr, 4 /* url or name len */ +
>> +                                   (XDR_QUADLEN(ns->u.nl4_str_sz) * 4));
>> +             if (!p)
>> +                     return nfserr_resource;
>> +             *p++ = cpu_to_be32(ns->u.nl4_str_sz);
>> +             p = xdr_encode_opaque_fixed(p, ns->u.nl4_str, ns->u.nl4_str_sz);
>> +                     break;
>
> This code will never be used as far as I can tell.  We could
> BUG_ON(nl4_type != NL4_NETADDR) (or even just remove nl4_type).

Sure.

>
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 25/42] NFSD add COPY_NOTIFY operation

[Olga Kornievskaia]

On Wed, Sep 6, 2017 at 4:45 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Tue, Jul 11, 2017 at 12:43:59PM -0400, Olga Kornievskaia wrote:
>> Signed-off-by: Andy Adamson <andros@netapp.com>
>> ---
>>  fs/nfsd/nfs4proc.c |  98 ++++++++++++++++++++++++++++++++++++++++++++++
>>  fs/nfsd/nfs4xdr.c  | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>  fs/nfsd/xdr4.h     |  13 +++++++
>>  3 files changed, 221 insertions(+), 2 deletions(-)
>>
>> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
>> index dadb3bf..ed6b9f2 100644
>> --- a/fs/nfsd/nfs4proc.c
>> +++ b/fs/nfsd/nfs4proc.c
>> @@ -35,6 +35,7 @@
>>  #include <linux/file.h>
>>  #include <linux/falloc.h>
>>  #include <linux/slab.h>
>> +#include <linux/sunrpc/addr.h>
>>
>>  #include "idmap.h"
>>  #include "cache.h"
>> @@ -1087,6 +1088,82 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>>  }
>>
>>  static __be32
>> +nfsd4_set_src_nl4_netaddr(struct svc_rqst *rqstp, struct nl4_servers *svrs)
>> +{
>> +     const struct sockaddr *addr = (struct sockaddr *)&rqstp->rq_daddr;
>> +     const struct sockaddr_in *sin = (const struct sockaddr_in *)addr;
>> +     const struct sockaddr_in6 *sin6 = (const struct sockaddr_in6 *)addr;
>> +     int uaddr_len = rqstp->rq_daddrlen + 4 + 1; /* port (4) and '\0' (1) */
>> +     struct nfs42_netaddr *naddr;
>> +     size_t ret;
>> +     unsigned short port;
>> +
>> +     /* freed in nfsd4_encode_copy_notify */
>> +     svrs->nl_svr = kmalloc_array(svrs->nl_nsvr, sizeof(struct nl4_server),
>> +                             GFP_KERNEL);
>> +     if (svrs->nl_svr == NULL)
>> +             return nfserrno(-ENOMEM);
>> +
>> +     svrs->nl_svr->nl4_type = NL4_NETADDR;
>> +     naddr = &svrs->nl_svr->u.nl4_addr;
>> +
>> +     switch (addr->sa_family) {
>> +     case AF_INET:
>> +             port = ntohs(sin->sin_port);
>> +             sprintf(naddr->na_netid, "tcp");
>> +             naddr->na_netid_len = 3;
>> +             break;
>> +     case AF_INET6:
>> +             port = ntohs(sin6->sin6_port);
>> +             sprintf(naddr->na_netid, "tcp6");
>> +             naddr->na_netid_len = 4;
>> +             break;
>> +     default:
>> +             dprintk("NFSD  nfsd4_set_notify_src: unknown address type: %d",
>> +                     addr->sa_family);
>> +             kfree(svrs->nl_svr);
>> +             return nfserrno(-EINVAL);
>> +     }
>> +     ret = rpc_ntop(addr, naddr->na_uaddr, sizeof(naddr->na_uaddr));
>> +     snprintf(naddr->na_uaddr + ret, uaddr_len, ".%u.%u",
>> +              port >> 8, port & 255);
>> +     naddr->na_uaddr_len = strlen(naddr->na_uaddr);
>
> I'm a little surprised there isn't already a standard helper for
> this....  Actually, does net/sunrpc:addr.c:rpc_sockaddr2uaddr() do it?

That only creates the string which is just a part of the data
structure. So I don't think it would be useful.

> (Looks like there's other potential users in
> fs/nfsd/flexfilelayout.c:nfsd4_ff_proc_getdeviceinfo() and in flexfiles
> code on client side and probably some other places too.)

Both flex files and copy can be re-written to share code though. They
are setting up similar .._netaddr structure.

> --b.
>
>> +     return 0;
>> +}
>> +
>> +static __be32
>> +nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>> +               struct nfsd4_copy_notify *cn)
>> +{
>> +     __be32 status;
>> +     struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
>> +
>> +     status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
>> +                                     &cn->cpn_src_stateid, RD_STATE, NULL,
>> +                                     NULL);
>> +     if (status)
>> +             return status;
>> +
>> +     cn->cpn_sec = nn->nfsd4_lease;
>> +     cn->cpn_nsec = 0;
>> +
>> +
>> +     /** XXX Save cpn_src_statid, cpn_src, and any other returned source
>> +      * server addresses on which the source server is williing to accept
>> +      * connections from the destination e.g. what is returned in cpn_src,
>> +      * to verify READ from dest server.
>> +      */
>> +
>> +     /**
>> +      * For now, only return one server address in cpn_src, the
>> +      * address used by the client to connect to this server.
>> +      */
>> +     cn->cpn_src.nl_nsvr = 1;
>> +
>> +     return nfsd4_set_src_nl4_netaddr(rqstp, &cn->cpn_src);
>> +}
>> +
>> +static __be32
>>  nfsd4_fallocate(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>>               struct nfsd4_fallocate *fallocate, int flags)
>>  {
>> @@ -2042,6 +2119,21 @@ static inline u32 nfsd4_copy_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>>               1 /* cr_synchronous */) * sizeof(__be32);
>>  }
>>
>> +static inline u32 nfsd4_copy_notify_rsize(struct svc_rqst *rqstp,
>> +                                     struct nfsd4_op *op)
>> +{
>> +     return (op_encode_hdr_size +
>> +             3 /* cnr_lease_time */ +
>> +             1 /* We support one cnr_source_server */ +
>> +             1 /* cnr_stateid seq */ +
>> +             op_encode_stateid_maxsz /* cnr_stateid */ +
>> +             1 /* num cnr_source_server*/ +
>> +             1 /* nl4_type */ +
>> +             1 /* nl4 size */ +
>> +             XDR_QUADLEN(NFS4_OPAQUE_LIMIT) /*nl4_loc + nl4_loc_sz */)
>> +             * sizeof(__be32);
>> +}
>> +
>>  #ifdef CONFIG_NFSD_PNFS
>>  static inline u32 nfsd4_getdeviceinfo_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>>  {
>> @@ -2446,6 +2538,12 @@ static inline u32 nfsd4_seek_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)
>>               .op_name = "OP_SEEK",
>>               .op_rsize_bop = (nfsd4op_rsize)nfsd4_seek_rsize,
>>       },
>> +     [OP_COPY_NOTIFY] = {
>> +             .op_func = (nfsd4op_func)nfsd4_copy_notify,
>> +             .op_flags = OP_MODIFIES_SOMETHING | OP_CACHEME,
>> +             .op_name = "OP_COPY_NOTIFY",
>> +             .op_rsize_bop = (nfsd4op_rsize)nfsd4_copy_notify_rsize,
>> +     },
>>  };
>>
>>  /**
>> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
>> index 031d06d..3e08c15 100644
>> --- a/fs/nfsd/nfs4xdr.c
>> +++ b/fs/nfsd/nfs4xdr.c
>> @@ -1820,6 +1820,22 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
>>  }
>>
>>  static __be32
>> +nfsd4_decode_copy_notify(struct nfsd4_compoundargs *argp,
>> +                      struct nfsd4_copy_notify *cn)
>> +{
>> +     int status;
>> +
>> +     status = nfsd4_decode_stateid(argp, &cn->cpn_src_stateid);
>> +     if (status)
>> +             return status;
>> +     status = nfsd4_decode_nl4_server(argp, &cn->cpn_dst);
>> +     if (status)
>> +             return status;
>> +
>> +     return status;
>> +}
>> +
>> +static __be32
>>  nfsd4_decode_seek(struct nfsd4_compoundargs *argp, struct nfsd4_seek *seek)
>>  {
>>       DECODE_HEAD;
>> @@ -1920,7 +1936,7 @@ static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
>>       /* new operations for NFSv4.2 */
>>       [OP_ALLOCATE]           = (nfsd4_dec)nfsd4_decode_fallocate,
>>       [OP_COPY]               = (nfsd4_dec)nfsd4_decode_copy,
>> -     [OP_COPY_NOTIFY]        = (nfsd4_dec)nfsd4_decode_notsupp,
>> +     [OP_COPY_NOTIFY]        = (nfsd4_dec)nfsd4_decode_copy_notify,
>>       [OP_DEALLOCATE]         = (nfsd4_dec)nfsd4_decode_fallocate,
>>       [OP_IO_ADVISE]          = (nfsd4_dec)nfsd4_decode_notsupp,
>>       [OP_LAYOUTERROR]        = (nfsd4_dec)nfsd4_decode_notsupp,
>> @@ -4355,6 +4371,52 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>>  }
>>
>>  static __be32
>> +nfsd42_encode_nl4_server(struct nfsd4_compoundres *resp, struct nl4_server *ns)
>> +{
>> +     struct xdr_stream *xdr = &resp->xdr;
>> +     struct nfs42_netaddr *addr;
>> +     __be32 *p;
>> +
>> +     p = xdr_reserve_space(xdr, 4);
>> +     *p++ = cpu_to_be32(ns->nl4_type);
>> +
>> +     switch (ns->nl4_type) {
>> +     case NL4_NAME:
>> +     case NL4_URL:
>> +             p = xdr_reserve_space(xdr, 4 /* url or name len */ +
>> +                                   (XDR_QUADLEN(ns->u.nl4_str_sz) * 4));
>> +             if (!p)
>> +                     return nfserr_resource;
>> +             *p++ = cpu_to_be32(ns->u.nl4_str_sz);
>> +             p = xdr_encode_opaque_fixed(p, ns->u.nl4_str, ns->u.nl4_str_sz);
>> +                     break;
>> +     case NL4_NETADDR:
>> +             addr = &ns->u.nl4_addr;
>> +
>> +             /** netid_len, netid, uaddr_len, uaddr (port included
>> +              * in RPCBIND_MAXUADDRLEN)
>> +              */
>> +             p = xdr_reserve_space(xdr,
>> +                     4 /* netid len */ +
>> +                     (XDR_QUADLEN(addr->na_netid_len) * 4) +
>> +                     4 /* uaddr len */ +
>> +                     (XDR_QUADLEN(addr->na_uaddr_len) * 4));
>> +             if (!p)
>> +                     return nfserr_resource;
>> +
>> +             *p++ = cpu_to_be32(addr->na_netid_len);
>> +             p = xdr_encode_opaque_fixed(p, addr->na_netid,
>> +                                         addr->na_netid_len);
>> +             *p++ = cpu_to_be32(addr->na_uaddr_len);
>> +             p = xdr_encode_opaque_fixed(p, addr->na_uaddr,
>> +                                     addr->na_uaddr_len);
>> +             break;
>> +     }
>> +
>> +     return 0;
>> +}
>> +
>> +static __be32
>>  nfsd4_encode_copy(struct nfsd4_compoundres *resp, __be32 nfserr,
>>                 struct nfsd4_copy *copy)
>>  {
>> @@ -4375,6 +4437,52 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>>  }
>>
>>  static __be32
>> +nfsd4_encode_copy_notify(struct nfsd4_compoundres *resp, __be32 nfserr,
>> +                      struct nfsd4_copy_notify *cn)
>> +{
>> +     struct xdr_stream *xdr = &resp->xdr;
>> +     struct nl4_server *ns;
>> +     __be32 *p;
>> +     int i;
>> +
>> +     if (nfserr)
>> +             return nfserr;
>> +
>> +     /* 8 sec, 4 nsec */
>> +     p = xdr_reserve_space(xdr, 12);
>> +     if (!p)
>> +             return nfserr_resource;
>> +
>> +     /* cnr_lease_time */
>> +     p = xdr_encode_hyper(p, cn->cpn_sec);
>> +     *p++ = cpu_to_be32(cn->cpn_nsec);
>> +
>> +     /* cnr_stateid */
>> +     nfserr = nfsd4_encode_stateid(xdr, &cn->cpn_src_stateid);
>> +     if (nfserr)
>> +             return nfserr;
>> +
>> +     /* cnr_src.nl_nsvr */
>> +     p = xdr_reserve_space(xdr, 4);
>> +     if (!p)
>> +             return nfserr_resource;
>> +
>> +     *p++ = cpu_to_be32(cn->cpn_src.nl_nsvr);
>> +
>> +     ns = cn->cpn_src.nl_svr;
>> +     for (i = 0; i < cn->cpn_src.nl_nsvr; i++) {
>> +             nfserr = nfsd42_encode_nl4_server(resp, ns);
>> +             if (nfserr)
>> +                     return nfserr;
>> +             ns++;
>> +     }
>> +
>> +     /* allocated in nfsd4_copy_notify */
>> +     kfree(cn->cpn_src.nl_svr);
>> +     return nfserr;
>> +}
>> +
>> +static __be32
>>  nfsd4_encode_seek(struct nfsd4_compoundres *resp, __be32 nfserr,
>>                 struct nfsd4_seek *seek)
>>  {
>> @@ -4474,7 +4582,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
>>       /* NFSv4.2 operations */
>>       [OP_ALLOCATE]           = (nfsd4_enc)nfsd4_encode_noop,
>>       [OP_COPY]               = (nfsd4_enc)nfsd4_encode_copy,
>> -     [OP_COPY_NOTIFY]        = (nfsd4_enc)nfsd4_encode_noop,
>> +     [OP_COPY_NOTIFY]        = (nfsd4_enc)nfsd4_encode_copy_notify,
>>       [OP_DEALLOCATE]         = (nfsd4_enc)nfsd4_encode_noop,
>>       [OP_IO_ADVISE]          = (nfsd4_enc)nfsd4_encode_noop,
>>       [OP_LAYOUTERROR]        = (nfsd4_enc)nfsd4_encode_noop,
>> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
>> index 6b1a61fc..5db7cd8 100644
>> --- a/fs/nfsd/xdr4.h
>> +++ b/fs/nfsd/xdr4.h
>> @@ -540,6 +540,18 @@ struct nfsd4_seek {
>>       loff_t          seek_pos;
>>  };
>>
>> +struct nfsd4_copy_notify {
>> +     /* request */
>> +     stateid_t               cpn_src_stateid;
>> +     struct nl4_server       cpn_dst;
>> +
>> +     /* response */
>> +     /* Note: cpn_src_stateid is used for cnr_stateid */
>> +     u64                     cpn_sec;
>> +     u32                     cpn_nsec;
>> +     struct nl4_servers      cpn_src;
>> +};
>> +
>>  struct nfsd4_op {
>>       int                                     opnum;
>>       __be32                                  status;
>> @@ -595,6 +607,7 @@ struct nfsd4_op {
>>               struct nfsd4_fallocate          deallocate;
>>               struct nfsd4_clone              clone;
>>               struct nfsd4_copy               copy;
>> +             struct nfsd4_copy_notify        copy_notify;
>>               struct nfsd4_seek               seek;
>>       } u;
>>       struct nfs4_replay *                    replay;
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 25/42] NFSD add COPY_NOTIFY operation

[J. Bruce Fields]

On Wed, Sep 13, 2017 at 10:38:46AM -0400, Olga Kornievskaia wrote:
> On Wed, Sep 6, 2017 at 5:37 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > On Tue, Jul 11, 2017 at 12:43:59PM -0400, Olga Kornievskaia wrote:
> >> +struct nfsd4_copy_notify {
> >> +     /* request */
> >> +     stateid_t               cpn_src_stateid;
> >> +     struct nl4_server       cpn_dst;
> >> +
> >> +     /* response */
> >> +     /* Note: cpn_src_stateid is used for cnr_stateid */
> >
> > That might be worth a little more explanation.
> 
> I'm opting for removing it from this patch? It's just valid for that
> patch saying that for the
> moment the same cpn_src_stateid is returned fort he cnr_stateid in the
> later patch (29)
> we will add the generation of the unique stateid used for the copy.

Got it, yes, that did confuse me, dropping the comment and noting the
coming change in the changelog would be great.--b.

> 
> >
> > --b.
> >
> >> +     u64                     cpn_sec;
> >> +     u32                     cpn_nsec;
> >> +     struct nl4_servers      cpn_src;
> >> +};
> >> +
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[Olga Kornievskaia]

On Fri, Sep 8, 2017 at 3:38 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> Some questions below, but really I think this approach can't work.  Keep
> in mind this has to work not just for the one compound the current
> client uses, but for *any* compound containing a COPY, including those
> sent by future clients that may do something more complicated, or by
> malcious clients that may intentionally send weird compounds to make the
> server crash.
> I think we want to flag the filehandle itself somehow, not the state.

> And check very carefully to make sure that ops other than COPY can't get
> tripped up by one of these foreign filehandles.
>
> On Tue, Jul 11, 2017 at 12:44:01PM -0400, Olga Kornievskaia wrote:
>> The inter server to server COPY source server filehandle
>> is guaranteed to be stale as the COPY is sent to the destination
>> server.
>>
>> Signed-off-by: Andy Adamson <andros@netapp.com>
>> ---
>>  fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
>>  fs/nfsd/nfs4xdr.c  | 26 +++++++++++++++++++++++++-
>>  fs/nfsd/nfsd.h     |  2 ++
>>  fs/nfsd/xdr4.h     |  4 ++++
>>  4 files changed, 77 insertions(+), 2 deletions(-)
>>
>> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
>> index b49ff31..ceee852 100644
>> --- a/fs/nfsd/nfs4proc.c
>> +++ b/fs/nfsd/nfs4proc.c
>> @@ -496,11 +496,19 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>>  nfsd4_putfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>>           struct nfsd4_putfh *putfh)
>>  {
>> +     __be32 ret;
>> +
>>       fh_put(&cstate->current_fh);
>>       cstate->current_fh.fh_handle.fh_size = putfh->pf_fhlen;
>>       memcpy(&cstate->current_fh.fh_handle.fh_base, putfh->pf_fhval,
>>              putfh->pf_fhlen);
>> -     return fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
>> +     ret = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
>> +     if (ret == nfserr_stale && HAS_CSTATE_FLAG(cstate, NO_VERIFY_FH)) {
>> +             CLEAR_CSTATE_FLAG(cstate, NO_VERIFY_FH);
>> +             SET_CSTATE_FLAG(cstate, IS_STALE_FH);
>> +             ret = 0;
>> +     }
>> +     return ret;
>>  }
>>
>>  static __be32
>> @@ -533,6 +541,16 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat
>>  nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>>            void *arg)
>>  {
>> +     /**
>> +     * This is either an inter COPY (most likely) or an intra COPY with a
>> +     * stale file handle. If the latter, nfsd4_copy will reset the PUTFH to
>> +     * return nfserr_stale. No fh_dentry, just copy the file handle
>> +     * to use with the inter COPY READ.
>> +     */
>> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
>> +             cstate->save_fh = cstate->current_fh;
>> +             return nfs_ok;
>> +     }
>>       if (!cstate->current_fh.fh_dentry)
>>               return nfserr_nofilehandle;
>>
>> @@ -1067,6 +1085,13 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>>       if (status)
>>               goto out;
>>
>> +     /* Intra copy source fh is stale. PUTFH will fail with ESTALE */
>> +     if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH)) {
>> +             CLEAR_CSTATE_FLAG(cstate, IS_STALE_FH);
>> +             cstate->status = nfserr_copy_stalefh;
>> +             goto out_put;
>> +     }
>> +
>>       bytes = nfsd_copy_file_range(src, copy->cp_src_pos,
>>                       dst, copy->cp_dst_pos, copy->cp_count);
>>
>> @@ -1081,6 +1106,7 @@ static int fill_in_write_vector(struct kvec *vec, struct nfsd4_write *write)
>>               status = nfs_ok;
>>       }
>>
>> +out_put:
>>       fput(src);
>>       fput(dst);
>>  out:
>> @@ -1759,6 +1785,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>       struct nfsd4_compound_state *cstate = &resp->cstate;
>>       struct svc_fh *current_fh = &cstate->current_fh;
>>       struct svc_fh *save_fh = &cstate->save_fh;
>> +     int             i;
>>       __be32          status;
>>
>>       svcxdr_init_encode(rqstp, resp);
>> @@ -1791,6 +1818,12 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>               goto encode_op;
>>       }
>>
>> +     /* NFSv4.2 COPY source file handle may be from a different server */
>> +     for (i = 0; i < args->opcnt; i++) {
>> +             op = &args->ops[i];
>> +             if (op->opnum == OP_COPY)
>> +                     SET_CSTATE_FLAG(cstate, NO_VERIFY_FH);
>> +     }
>
> So, if a compound has a COPY anywhere in it, then a PUTFH with a stale
> filehandle anywhere in the compound will temporarily succeed and result in
> IS_STALE_FH being set.

How about changing it to be more restrictive by checking within the
while loop that
if it's a PUTFH and it's followed by the SAVE_FH+PUTFH+COPY only
then set the NO_VERIFY_FH.

        while (!status && resp->opcnt < args->opcnt) {
                op = &args->ops[resp->opcnt++];

                if (op->opnum == OP_PUTFH &&
                        args->ops[resp->opcnt].opnum == OP_SAVEFH &&
                        args->ops[resp->opcnt+1].opnum == OP_PUTFH &&
                        args->ops[resp->opcnt+2].opnum == OP_COPY) {
                        SET_CSTATE_FLAG(cstate, NO_VERIFY_FH);
                }

Would that address your concern or is this too restrictive?

I guess we can allow some other operations between the 2nd PUTFH and COPY
because the 2nd filehandle will be validated and must be valid to continue
processing the compound. It doesn't make sense to allow for operation after the
1st PUTFH as it can be a foreign handle.

Somewhere else you were talking about how a "foreign" file handle can mean
something to the server. If that's the case and we do allow for operations
between putfh, savefh, putfh then we'll get into trouble that I can't
think we can
get out of.

If it's a inter copy and the source file handle means something to the
server I can
think of the following scenario: the state won't be flagged IS_STALE then the
filehandle would go thru the checks you listed below and can (unintentionally)
result in an error (eg., err_moved?).

To really handle this, i can see doing something like, for a putfh with a copy
elsewhere in the compound, skip ahead to the copy, find out if it's an
inter copy,
then we know to mark the filehandle (as you suggested in the beginning) and
check in all operations that they are not being done on a foreign handle. That
seems very messy... Isn't restricting to putfh, savefh, putfh, copy a better way
for the current situation?

>
>>       while (!status && resp->opcnt < args->opcnt) {
>>               op = &args->ops[resp->opcnt++];
>>
>> @@ -1810,6 +1843,9 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>
>>               opdesc = OPDESC(op);
>>
>> +             if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
>> +                     goto call_op;
>> +
>
> That means that once IS_STALE_FH is set, we will skip:
>
>         - nofilehandle checking
>         - moved checking
>         - fh_clear_wcc()
>         - resp_size checking
>         - current stateid capture
>
> on every subsequent op.  Can that really be right?

This should be changed to
if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH) && op->opnum == OP_SAVEFH)
then we need to skip the checks for savefh as it has no valid file handle.

Does that address your concern?


>
>>               if (!current_fh->fh_dentry) {
>>                       if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
>>                               op->status = nfserr_nofilehandle;
>> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>
>>               if (opdesc->op_get_currentstateid)
>>                       opdesc->op_get_currentstateid(cstate, &op->u);
>> +call_op:
>>               op->status = opdesc->op_func(rqstp, cstate, &op->u);
>>
>>               /* Only from SEQUENCE */
>> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>>                       if (need_wrongsec_check(rqstp))
>>                               op->status = check_nfsd_access(current_fh->fh_export, rqstp);
>>               }
>> +             /* Only from intra COPY */
>> +             if (cstate->status == nfserr_copy_stalefh) {
>> +                     dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
>> +                             __func__);
>> +                     status = nfserr_stale;
>> +                     nfsd4_adjust_encode(resp);
>
> Are you sure it's safe just to throw away any operations since that
> stale PUTFH?  What if some of those operations had side effects?

If COPY comes in PUTFH,SAVEFH, PUTFH,COPY compound then
I think it's safe?

>
>> +                     goto out;
>> +             }
>>  encode_op:
>>               if (op->status == nfserr_replay_me) {
>>                       op->replay = &cstate->replay_owner->so_replay;
>> diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
>> index 3e08c15..2896a11 100644
>> --- a/fs/nfsd/nfs4xdr.c
>> +++ b/fs/nfsd/nfs4xdr.c
>> @@ -4624,15 +4624,28 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>>       return nfserr_rep_too_big;
>>  }
>>
>> +/** Rewind the encoding to return nfserr_stale on the PUTFH
>> + * in this failed Intra COPY compound
>> + */
>> +void
>> +nfsd4_adjust_encode(struct nfsd4_compoundres *resp)
>> +{
>> +     __be32 *p;
>> +
>> +     p = resp->cstate.putfh_errp;
>> +     *p++ = nfserr_stale;
>> +}
>> +
>>  void
>>  nfsd4_encode_operation(struct nfsd4_compoundres *resp, struct nfsd4_op *op)
>>  {
>>       struct xdr_stream *xdr = &resp->xdr;
>>       struct nfs4_stateowner *so = resp->cstate.replay_owner;
>> +     struct nfsd4_compound_state *cstate = &resp->cstate;
>>       struct svc_rqst *rqstp = resp->rqstp;
>>       int post_err_offset;
>>       nfsd4_enc encoder;
>> -     __be32 *p;
>> +     __be32 *p, *statp;
>>
>>       p = xdr_reserve_space(xdr, 8);
>>       if (!p) {
>> @@ -4641,9 +4654,20 @@ __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize)
>>       }
>>       *p++ = cpu_to_be32(op->opnum);
>>       post_err_offset = xdr->buf->len;
>> +     statp = p;
>>
>>       if (op->opnum == OP_ILLEGAL)
>>               goto status;
>> +
>> +     /** This is a COPY compound with a stale source server file handle.
>> +      * If OP_COPY processing determines that this is an intra server to
>> +      * server COPY, then this PUTFH should return nfserr_ stale so the
>> +      * putfh_errp will be set to nfserr_stale. If this is an inter server
>> +      * to server COPY, ignore the nfserr_stale.
>> +      */
>> +     if (op->opnum == OP_PUTFH && HAS_CSTATE_FLAG(cstate, IS_STALE_FH))
>> +             cstate->putfh_errp = statp;
>> +
>>       BUG_ON(op->opnum < 0 || op->opnum >= ARRAY_SIZE(nfsd4_enc_ops) ||
>>              !nfsd4_enc_ops[op->opnum]);
>>       encoder = nfsd4_enc_ops[op->opnum];
>> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
>> index d966068..8d6fb0f 100644
>> --- a/fs/nfsd/nfsd.h
>> +++ b/fs/nfsd/nfsd.h
>> @@ -272,6 +272,8 @@ static inline bool nfsd4_spo_must_allow(struct svc_rqst *rqstp)
>>  #define      nfserr_replay_me        cpu_to_be32(11001)
>>  /* nfs41 replay detected */
>>  #define      nfserr_replay_cache     cpu_to_be32(11002)
>> +/* nfs42 intra copy failed with nfserr_stale */
>> +#define nfserr_copy_stalefh  cpu_to_be32(1103)
>>
>>  /* Check for dir entries '.' and '..' */
>>  #define isdotent(n, l)       (l < 3 && n[0] == '.' && (l == 1 || n[1] == '.'))
>> diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
>> index 38fcb4f..aa94295 100644
>> --- a/fs/nfsd/xdr4.h
>> +++ b/fs/nfsd/xdr4.h
>> @@ -45,6 +45,8 @@
>>
>>  #define CURRENT_STATE_ID_FLAG (1<<0)
>>  #define SAVED_STATE_ID_FLAG (1<<1)
>> +#define NO_VERIFY_FH (1<<2)
>> +#define IS_STALE_FH  (1<<3)
>>
>>  #define SET_CSTATE_FLAG(c, f) ((c)->sid_flags |= (f))
>>  #define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f))
>> @@ -63,6 +65,7 @@ struct nfsd4_compound_state {
>>       size_t                  iovlen;
>>       u32                     minorversion;
>>       __be32                  status;
>> +     __be32                  *putfh_errp;
>>       stateid_t       current_stateid;
>>       stateid_t       save_stateid;
>>       /* to indicate current and saved state id presents */
>> @@ -705,6 +708,7 @@ int nfs4svc_decode_compoundargs(struct svc_rqst *, __be32 *,
>>  int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *,
>>               struct nfsd4_compoundres *);
>>  __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32);
>> +void nfsd4_adjust_encode(struct nfsd4_compoundres *);
>>  void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
>>  void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op);
>>  __be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words,
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[J. Bruce Fields]

On Thu, Sep 14, 2017 at 02:44:32PM -0400, Olga Kornievskaia wrote:
> How about changing it to be more restrictive by checking within the
> while loop that
> if it's a PUTFH and it's followed by the SAVE_FH+PUTFH+COPY only
> then set the NO_VERIFY_FH.

I agree that the only op that could reasonably follow the first PUTFH of
a foreign filehandle is a SAVEFH.  But once we've saved the foreign
filehandle, the client could in theory do all sorts of stuff, as you
say:

> I guess we can allow some other operations between the 2nd PUTFH and
> COPY because the 2nd filehandle will be validated and must be valid to
> continue processing the compound.

PUTHF+SAVEFH+PUTFH+GETATTR+COPY might be useful to get pre-copy
attributes of the target file, for example?

I'd rather not restrict this if we don't need to.

We could do something like:

	- if this op is PUTFH
	- if the following is not SAVEFH, stop and verify the
	  filehandle.
	- otherwise, skip to the next operation that uses a saved
	  filehandle.  The possibilities are:
		- RENAME, LINK, RESTOREFH, CLONE: stop and verify the
		  filehandle.
		- COPY: if it's a local copy, stop and verify the
		  filehandle.  Otherwise, allow the PUTFH to succeed and
		  delay verification.

> Somewhere else you were talking about how a "foreign" file handle can
> mean something to the server. If that's the case and we do allow for
> operations between putfh, savefh, putfh then we'll get into trouble
> that I can't think we can get out of.
> 
> If it's a inter copy and the source file handle means something to the
> server I can think of the following scenario: the state won't be
> flagged IS_STALE then the filehandle would go thru the checks you
> listed below and can (unintentionally) result in an error (eg.,
> err_moved?).

I don't see any problem with just leaving those checks in.  If the
current filehandle is not validated, then there's already a
current_fh->fh_dentry check that skips the ABSENT_FH check, for example.

> This should be changed to
> if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH) && op->opnum == OP_SAVEFH)
> then we need to skip the checks for savefh as it has no valid file handle.
> 
> Does that address your concern?

I think you only need to skip the nofilehandle check in this case, not
the other stuff.

I don't think the IS_STALE_FH flag is needed at all.

> >>               if (!current_fh->fh_dentry) {
> >>                       if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
> >>                               op->status = nfserr_nofilehandle;
> >> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >>
> >>               if (opdesc->op_get_currentstateid)
> >>                       opdesc->op_get_currentstateid(cstate, &op->u);
> >> +call_op:
> >>               op->status = opdesc->op_func(rqstp, cstate, &op->u);
> >>
> >>               /* Only from SEQUENCE */
> >> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >>                       if (need_wrongsec_check(rqstp))
> >>                               op->status = check_nfsd_access(current_fh->fh_export, rqstp);
> >>               }
> >> +             /* Only from intra COPY */
> >> +             if (cstate->status == nfserr_copy_stalefh) {
> >> +                     dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
> >> +                             __func__);
> >> +                     status = nfserr_stale;
> >> +                     nfsd4_adjust_encode(resp);
> >
> > Are you sure it's safe just to throw away any operations since that
> > stale PUTFH?  What if some of those operations had side effects?
> 
> If COPY comes in PUTFH,SAVEFH, PUTFH,COPY compound then
> I think it's safe?

The spec says "If a server supports the inter-server copy feature, a
PUTFH followed by a SAVEFH MUST NOT return NFS4ERR_STALE for either
operation.  These restrictions do not pose substantial difficulties for
servers.  CURRENT_FH and SAVED_FH may be validated in the context of the
operation referencing them and an NFS4ERR_STALE error returned for an
invalid filehandle at that point."

So we're supposed to return NFS4ERR_STALE on the COPY, not the PUTFH.
So there's no need for this backtracking.

--b.

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[Olga Kornievskaia]

On Thu, Sep 14, 2017 at 9:47 PM, J. Bruce Fields <bfields@redhat.com> wrote:
> On Thu, Sep 14, 2017 at 02:44:32PM -0400, Olga Kornievskaia wrote:
>> How about changing it to be more restrictive by checking within the
>> while loop that
>> if it's a PUTFH and it's followed by the SAVE_FH+PUTFH+COPY only
>> then set the NO_VERIFY_FH.
>
> I agree that the only op that could reasonably follow the first PUTFH of
> a foreign filehandle is a SAVEFH.  But once we've saved the foreign
> filehandle, the client could in theory do all sorts of stuff, as you
> say:
>
>> I guess we can allow some other operations between the 2nd PUTFH and
>> COPY because the 2nd filehandle will be validated and must be valid to
>> continue processing the compound.
>
> PUTHF+SAVEFH+PUTFH+GETATTR+COPY might be useful to get pre-copy
> attributes of the target file, for example?
>
> I'd rather not restrict this if we don't need to.
>
> We could do something like:
>
>         - if this op is PUTFH
>         - if the following is not SAVEFH, stop and verify the
>           filehandle.
>         - otherwise, skip to the next operation that uses a saved
>           filehandle.  The possibilities are:
>                 - RENAME, LINK, RESTOREFH, CLONE: stop and verify the
>                   filehandle.
>                 - COPY: if it's a local copy, stop and verify the
>                   filehandle.  Otherwise, allow the PUTFH to succeed and
>                   delay verification.

OK so I can peep into the upcoming copy and see if it's inter or intra and
then based on that I can set NO_VERIFY_FH for processing the putfh.

>> Somewhere else you were talking about how a "foreign" file handle can
>> mean something to the server. If that's the case and we do allow for
>> operations between putfh, savefh, putfh then we'll get into trouble
>> that I can't think we can get out of.
>>
>> If it's a inter copy and the source file handle means something to the
>> server I can think of the following scenario: the state won't be
>> flagged IS_STALE then the filehandle would go thru the checks you
>> listed below and can (unintentionally) result in an error (eg.,
>> err_moved?).
>
> I don't see any problem with just leaving those checks in.  If the
> current filehandle is not validated, then there's already a
> current_fh->fh_dentry check that skips the ABSENT_FH check, for example.
>
>> This should be changed to
>> if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH) && op->opnum == OP_SAVEFH)
>> then we need to skip the checks for savefh as it has no valid file handle.
>>
>> Does that address your concern?
>
> I think you only need to skip the nofilehandle check in this case, not
> the other stuff.

As long as I can add that current_fh->fh_export is not null for the 2nd check
otherwise it leads to an oops.

> I don't think the IS_STALE_FH flag is needed at all.

We still need it so that the savefh process can skip the check for the
filehandle.
Or I can use something like checking that fh_dentry and fh_export are both null
and use that to mean I need to check the filehandle check in savefh.

I think explicitly marking it makes the code either to understand instead of
'knowning/remembering' that lack of fh_dentry+fh_export means inter copy?

>> >>               if (!current_fh->fh_dentry) {
>> >>                       if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
>> >>                               op->status = nfserr_nofilehandle;
>> >> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>> >>
>> >>               if (opdesc->op_get_currentstateid)
>> >>                       opdesc->op_get_currentstateid(cstate, &op->u);
>> >> +call_op:
>> >>               op->status = opdesc->op_func(rqstp, cstate, &op->u);
>> >>
>> >>               /* Only from SEQUENCE */
>> >> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
>> >>                       if (need_wrongsec_check(rqstp))
>> >>                               op->status = check_nfsd_access(current_fh->fh_export, rqstp);
>> >>               }
>> >> +             /* Only from intra COPY */
>> >> +             if (cstate->status == nfserr_copy_stalefh) {
>> >> +                     dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
>> >> +                             __func__);
>> >> +                     status = nfserr_stale;
>> >> +                     nfsd4_adjust_encode(resp);
>> >
>> > Are you sure it's safe just to throw away any operations since that
>> > stale PUTFH?  What if some of those operations had side effects?
>>
>> If COPY comes in PUTFH,SAVEFH, PUTFH,COPY compound then
>> I think it's safe?
>
> The spec says "If a server supports the inter-server copy feature, a
> PUTFH followed by a SAVEFH MUST NOT return NFS4ERR_STALE for either
> operation.  These restrictions do not pose substantial difficulties for
> servers.  CURRENT_FH and SAVED_FH may be validated in the context of the
> operation referencing them and an NFS4ERR_STALE error returned for an
> invalid filehandle at that point."
>
> So we're supposed to return NFS4ERR_STALE on the COPY, not the PUTFH.
> So there's no need for this backtracking.

You are right.

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[J. Bruce Fields]

On Fri, Sep 15, 2017 at 03:46:03PM -0400, Olga Kornievskaia wrote:
> On Thu, Sep 14, 2017 at 9:47 PM, J. Bruce Fields <bfields@redhat.com> wrote:
> > On Thu, Sep 14, 2017 at 02:44:32PM -0400, Olga Kornievskaia wrote:
> >> How about changing it to be more restrictive by checking within the
> >> while loop that
> >> if it's a PUTFH and it's followed by the SAVE_FH+PUTFH+COPY only
> >> then set the NO_VERIFY_FH.
> >
> > I agree that the only op that could reasonably follow the first PUTFH of
> > a foreign filehandle is a SAVEFH.  But once we've saved the foreign
> > filehandle, the client could in theory do all sorts of stuff, as you
> > say:
> >
> >> I guess we can allow some other operations between the 2nd PUTFH and
> >> COPY because the 2nd filehandle will be validated and must be valid to
> >> continue processing the compound.
> >
> > PUTHF+SAVEFH+PUTFH+GETATTR+COPY might be useful to get pre-copy
> > attributes of the target file, for example?
> >
> > I'd rather not restrict this if we don't need to.
> >
> > We could do something like:
> >
> >         - if this op is PUTFH
> >         - if the following is not SAVEFH, stop and verify the
> >           filehandle.
> >         - otherwise, skip to the next operation that uses a saved
> >           filehandle.  The possibilities are:
> >                 - RENAME, LINK, RESTOREFH, CLONE: stop and verify the
> >                   filehandle.
> >                 - COPY: if it's a local copy, stop and verify the
> >                   filehandle.  Otherwise, allow the PUTFH to succeed and
> >                   delay verification.
> 
> OK so I can peep into the upcoming copy and see if it's inter or intra and
> then based on that I can set NO_VERIFY_FH for processing the putfh.
> 
> >> Somewhere else you were talking about how a "foreign" file handle can
> >> mean something to the server. If that's the case and we do allow for
> >> operations between putfh, savefh, putfh then we'll get into trouble
> >> that I can't think we can get out of.
> >>
> >> If it's a inter copy and the source file handle means something to the
> >> server I can think of the following scenario: the state won't be
> >> flagged IS_STALE then the filehandle would go thru the checks you
> >> listed below and can (unintentionally) result in an error (eg.,
> >> err_moved?).
> >
> > I don't see any problem with just leaving those checks in.  If the
> > current filehandle is not validated, then there's already a
> > current_fh->fh_dentry check that skips the ABSENT_FH check, for example.
> >
> >> This should be changed to
> >> if (HAS_CSTATE_FLAG(cstate, IS_STALE_FH) && op->opnum == OP_SAVEFH)
> >> then we need to skip the checks for savefh as it has no valid file handle.
> >>
> >> Does that address your concern?
> >
> > I think you only need to skip the nofilehandle check in this case, not
> > the other stuff.
> 
> As long as I can add that current_fh->fh_export is not null for the 2nd check
> otherwise it leads to an oops.

fh_export should be set whenever fh_dentry is.

> > I don't think the IS_STALE_FH flag is needed at all.
> 
> We still need it so that the savefh process can skip the check for the
> filehandle.
>
> Or I can use something like checking that fh_dentry and fh_export are both null
> and use that to mean I need to check the filehandle check in savefh.

Yes, or you can just check the fh_dentry.

> I think explicitly marking it makes the code either to understand instead of
> 'knowning/remembering' that lack of fh_dentry+fh_export means inter copy?

If it turns out we still need a flag I think I'd rather it be set on the
filehandle somehow rather than in the cstate.  It's the filehandle
that's unverified.

--b.

> 
> >> >>               if (!current_fh->fh_dentry) {
> >> >>                       if (!(opdesc->op_flags & ALLOWED_WITHOUT_FH)) {
> >> >>                               op->status = nfserr_nofilehandle;
> >> >> @@ -1844,6 +1880,7 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >> >>
> >> >>               if (opdesc->op_get_currentstateid)
> >> >>                       opdesc->op_get_currentstateid(cstate, &op->u);
> >> >> +call_op:
> >> >>               op->status = opdesc->op_func(rqstp, cstate, &op->u);
> >> >>
> >> >>               /* Only from SEQUENCE */
> >> >> @@ -1862,6 +1899,14 @@ static void svcxdr_init_encode(struct svc_rqst *rqstp,
> >> >>                       if (need_wrongsec_check(rqstp))
> >> >>                               op->status = check_nfsd_access(current_fh->fh_export, rqstp);
> >> >>               }
> >> >> +             /* Only from intra COPY */
> >> >> +             if (cstate->status == nfserr_copy_stalefh) {
> >> >> +                     dprintk("%s NFS4.2 intra COPY stale src filehandle\n",
> >> >> +                             __func__);
> >> >> +                     status = nfserr_stale;
> >> >> +                     nfsd4_adjust_encode(resp);
> >> >
> >> > Are you sure it's safe just to throw away any operations since that
> >> > stale PUTFH?  What if some of those operations had side effects?
> >>
> >> If COPY comes in PUTFH,SAVEFH, PUTFH,COPY compound then
> >> I think it's safe?
> >
> > The spec says "If a server supports the inter-server copy feature, a
> > PUTFH followed by a SAVEFH MUST NOT return NFS4ERR_STALE for either
> > operation.  These restrictions do not pose substantial difficulties for
> > servers.  CURRENT_FH and SAVED_FH may be validated in the context of the
> > operation referencing them and an NFS4ERR_STALE error returned for an
> > invalid filehandle at that point."
> >
> > So we're supposed to return NFS4ERR_STALE on the COPY, not the PUTFH.
> > So there's no need for this backtracking.
> 
> You are right.

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[J. Bruce Fields]

On Fri, Sep 15, 2017 at 04:02:21PM -0400, J. Bruce Fields wrote:
> On Fri, Sep 15, 2017 at 03:46:03PM -0400, Olga Kornievskaia wrote:
> > We still need it so that the savefh process can skip the check for the
> > filehandle.

By the way, I think the current check in nfsd4_savefh is redundant; we
can probably apply something like the following.  (Untested.)

--b.

commit c1bab7c64408
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Fri Sep 15 16:02:52 2017 -0400

    nfsd: remove unnecessary nofilehandle checks
    
    These checks should have already be done centrally in
    nfsd4_proc_compound, the checks in each individual operation are
    unnecessary.
    
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 3c69db7d4905..07ebae912e0e 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -485,9 +485,6 @@ static __be32
 nfsd4_getfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	    union nfsd4_op_u *u)
 {
-	if (!cstate->current_fh.fh_dentry)
-		return nfserr_nofilehandle;
-
 	u->getfh = &cstate->current_fh;
 	return nfs_ok;
 }
@@ -535,9 +532,6 @@ static __be32
 nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	     union nfsd4_op_u *u)
 {
-	if (!cstate->current_fh.fh_dentry)
-		return nfserr_nofilehandle;
-
 	fh_dup2(&cstate->save_fh, &cstate->current_fh);
 	if (HAS_STATE_ID(cstate, CURRENT_STATE_ID_FLAG)) {
 		memcpy(&cstate->save_stateid, &cstate->current_stateid, sizeof(stateid_t));
@@ -703,10 +697,7 @@ nfsd4_link(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	   union nfsd4_op_u *u)
 {
 	struct nfsd4_link *link = &u->link;
-	__be32 status = nfserr_nofilehandle;
 
-	if (!cstate->save_fh.fh_dentry)
-		return status;
 	status = nfsd_link(rqstp, &cstate->current_fh,
 			   link->li_name, link->li_namelen, &cstate->save_fh);
 	if (!status)
@@ -850,10 +841,7 @@ nfsd4_rename(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	     union nfsd4_op_u *u)
 {
 	struct nfsd4_rename *rename = &u->rename;
-	__be32 status = nfserr_nofilehandle;
 
-	if (!cstate->save_fh.fh_dentry)
-		return status;
 	if (opens_in_grace(SVC_NET(rqstp)) &&
 		!(cstate->save_fh.fh_export->ex_flags & NFSEXP_NOSUBTREECHECK))
 		return nfserr_grace;

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[Olga Kornievskaia]

On Fri, Sep 15, 2017 at 4:06 PM, J. Bruce Fields <bfields@redhat.com> wrote:
> On Fri, Sep 15, 2017 at 04:02:21PM -0400, J. Bruce Fields wrote:
>> On Fri, Sep 15, 2017 at 03:46:03PM -0400, Olga Kornievskaia wrote:
>> > We still need it so that the savefh process can skip the check for the
>> > filehandle.
>
> By the way, I think the current check in nfsd4_savefh is redundant; we
> can probably apply something like the following.  (Untested.)

If those checks are not necessary then IS_STALE_FH is not necessary.

>
> --b.
>
> commit c1bab7c64408
> Author: J. Bruce Fields <bfields@redhat.com>
> Date:   Fri Sep 15 16:02:52 2017 -0400
>
>     nfsd: remove unnecessary nofilehandle checks
>
>     These checks should have already be done centrally in
>     nfsd4_proc_compound, the checks in each individual operation are
>     unnecessary.
>
>     Signed-off-by: J. Bruce Fields <bfields@redhat.com>
>
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 3c69db7d4905..07ebae912e0e 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -485,9 +485,6 @@ static __be32
>  nfsd4_getfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>             union nfsd4_op_u *u)
>  {
> -       if (!cstate->current_fh.fh_dentry)
> -               return nfserr_nofilehandle;
> -
>         u->getfh = &cstate->current_fh;
>         return nfs_ok;
>  }
> @@ -535,9 +532,6 @@ static __be32
>  nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>              union nfsd4_op_u *u)
>  {
> -       if (!cstate->current_fh.fh_dentry)
> -               return nfserr_nofilehandle;
> -
>         fh_dup2(&cstate->save_fh, &cstate->current_fh);
>         if (HAS_STATE_ID(cstate, CURRENT_STATE_ID_FLAG)) {
>                 memcpy(&cstate->save_stateid, &cstate->current_stateid, sizeof(stateid_t));
> @@ -703,10 +697,7 @@ nfsd4_link(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>            union nfsd4_op_u *u)
>  {
>         struct nfsd4_link *link = &u->link;
> -       __be32 status = nfserr_nofilehandle;
>
> -       if (!cstate->save_fh.fh_dentry)
> -               return status;
>         status = nfsd_link(rqstp, &cstate->current_fh,
>                            link->li_name, link->li_namelen, &cstate->save_fh);
>         if (!status)
> @@ -850,10 +841,7 @@ nfsd4_rename(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
>              union nfsd4_op_u *u)
>  {
>         struct nfsd4_rename *rename = &u->rename;
> -       __be32 status = nfserr_nofilehandle;
>
> -       if (!cstate->save_fh.fh_dentry)
> -               return status;
>         if (opens_in_grace(SVC_NET(rqstp)) &&
>                 !(cstate->save_fh.fh_export->ex_flags & NFSEXP_NOSUBTREECHECK))
>                 return nfserr_grace;

Re: [RFC v3 27/42] NFSD: allow inter server COPY to have a STALE source server fh

[J. Bruce Fields]

On Fri, Sep 15, 2017 at 04:06:29PM -0400, J. Bruce Fields wrote:
> By the way, I think the current check in nfsd4_savefh is redundant; we
> can probably apply something like the following.  (Untested.)

Fixed and tested version for 4.15.

--b.

commit 6eb90459308b
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Fri Sep 15 16:02:52 2017 -0400

    nfsd: remove unnecessary nofilehandle checks
    
    These checks should have already be done centrally in
    nfsd4_proc_compound, the checks in each individual operation are
    unnecessary.
    
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 3c69db7d4905..7aff6d383d34 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -485,9 +485,6 @@ static __be32
 nfsd4_getfh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	    union nfsd4_op_u *u)
 {
-	if (!cstate->current_fh.fh_dentry)
-		return nfserr_nofilehandle;
-
 	u->getfh = &cstate->current_fh;
 	return nfs_ok;
 }
@@ -535,9 +532,6 @@ static __be32
 nfsd4_savefh(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	     union nfsd4_op_u *u)
 {
-	if (!cstate->current_fh.fh_dentry)
-		return nfserr_nofilehandle;
-
 	fh_dup2(&cstate->save_fh, &cstate->current_fh);
 	if (HAS_STATE_ID(cstate, CURRENT_STATE_ID_FLAG)) {
 		memcpy(&cstate->save_stateid, &cstate->current_stateid, sizeof(stateid_t));
@@ -703,10 +697,8 @@ nfsd4_link(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	   union nfsd4_op_u *u)
 {
 	struct nfsd4_link *link = &u->link;
-	__be32 status = nfserr_nofilehandle;
+	__be32 status;
 
-	if (!cstate->save_fh.fh_dentry)
-		return status;
 	status = nfsd_link(rqstp, &cstate->current_fh,
 			   link->li_name, link->li_namelen, &cstate->save_fh);
 	if (!status)
@@ -850,10 +842,8 @@ nfsd4_rename(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	     union nfsd4_op_u *u)
 {
 	struct nfsd4_rename *rename = &u->rename;
-	__be32 status = nfserr_nofilehandle;
+	__be32 status;
 
-	if (!cstate->save_fh.fh_dentry)
-		return status;
 	if (opens_in_grace(SVC_NET(rqstp)) &&
 		!(cstate->save_fh.fh_export->ex_flags & NFSEXP_NOSUBTREECHECK))
 		return nfserr_grace;