[PATCH 0/1] Enable inter server to server copies on a export

[PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

This kernel patch and an upcoming nfs-utils patch
adds a new export option 's2sc' which will allow
inter server to server copies.

The option needs to be set on the destination server.

The idea behind the option is to allow admins the 
ability to control which export can do these 
types of copies.

Steve Dickson (1):
  nfsd4_copy: Adds the ability to do inter server to server on an export

 fs/nfsd/nfs4proc.c               | 3 ++-
 include/uapi/linux/nfsd/export.h | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

-- 
2.31.1

[PATCH 1/1] nfsd4_copy: Adds the ability to do inter server to server on an export

[Steve Dickson]

This adds the 's2sc' export option allowing inter server
to server copies on the destination server.

Signed-off-by: Steve Dickson <steved@redhat.com>
---
 fs/nfsd/nfs4proc.c               | 3 ++-
 include/uapi/linux/nfsd/export.h | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 486c5dba4b65..df3ca5f7f86f 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1654,9 +1654,10 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfsd4_copy *copy = &u->copy;
 	__be32 status;
 	struct nfsd4_copy *async_copy = NULL;
+	int s2sc = (cstate->current_fh.fh_export->ex_flags & NFSEXP_S2SC);
 
 	if (!copy->cp_intra) { /* Inter server SSC */
-		if (!inter_copy_offload_enable || copy->cp_synchronous) {
+		if (s2sc == 0 && (!inter_copy_offload_enable || copy->cp_synchronous)) {
 			status = nfserr_notsupp;
 			goto out;
 		}
diff --git a/include/uapi/linux/nfsd/export.h b/include/uapi/linux/nfsd/export.h
index 2124ba904779..53ba8d989689 100644
--- a/include/uapi/linux/nfsd/export.h
+++ b/include/uapi/linux/nfsd/export.h
@@ -53,6 +53,7 @@
  */
 #define	NFSEXP_V4ROOT		0x10000
 #define NFSEXP_PNFS		0x20000
+#define NFSEXP_S2SC		0x40000
 
 /* All flags that we claim to support.  (Note we don't support NOACL.) */
 #define NFSEXP_ALLFLAGS		0x3FEFF
-- 
2.31.1

Re: [PATCH 0/1] Enable inter server to server copies on a export

[J. Bruce Fields]

On Thu, Oct 28, 2021 at 10:48:50AM -0400, Steve Dickson wrote:
> This kernel patch and an upcoming nfs-utils patch
> adds a new export option 's2sc' which will allow
> inter server to server copies.

They're already allowed by a module option.  Why is an export option
better?  And why should it be set on the destination server and not the
source server?

Really, first I think we should try to identify what the problem is that
we're trying to solve.

--b.

> 
> The option needs to be set on the destination server.
> 
> The idea behind the option is to allow admins the 
> ability to control which export can do these 
> types of copies.
> 
> Steve Dickson (1):
>   nfsd4_copy: Adds the ability to do inter server to server on an export
> 
>  fs/nfsd/nfs4proc.c               | 3 ++-
>  include/uapi/linux/nfsd/export.h | 1 +
>  2 files changed, 3 insertions(+), 1 deletion(-)
> 
> -- 
> 2.31.1

Re: [PATCH 0/1] Enable inter server to server copies on a export

[J. Bruce Fields]

On Fri, Oct 29, 2021 at 09:45:34AM -0400, bfields wrote:
> On Thu, Oct 28, 2021 at 10:48:50AM -0400, Steve Dickson wrote:
> > This kernel patch and an upcoming nfs-utils patch
> > adds a new export option 's2sc' which will allow
> > inter server to server copies.
> 
> They're already allowed by a module option.  Why is an export option
> better?  And why should it be set on the destination server and not the
> source server?
> 
> Really, first I think we should try to identify what the problem is that
> we're trying to solve.

I guess we're thinking: we expect server-to-server copy to be a win in
some cases, but not others.

What would those cases look like?

Say you've got a client "C" and two servers, "S" and "T", and C is
copying a file from S to T.

I'd expect bandwidth of the traditional read-write-loop copy to be the
minimum of the network bandwidth between S and C, and the network
bandwidth between C and T.  Are there common cases were the S-to-T
bandwidth would be significantly less than both of those?

My guess would've been that that's relatively unusual.  So as a first
pass, just turning on COPY unconditionally doesn't seem so bad.

I know you're thinking about in cases where S and T are connected by a
high-performance network not necessarily available to C.

In that case, we know we want to use server-to-server copy for copies
between S and T.  But is it necessarily a problem to also use it for
copies between some other server and T?  Also, does knowing the export
containing the destination file on T really tell you whether or not the
copy will be coming from S and not from some other server?

I'd think the bigger issue in that case is figuring out how to configure
S so that it returns the right IP address in the cnr_source_server field
of the COPY_NOTIFY reply.  Currently we return address that the client
sent the COPY_NOTIFY, and I don't know if that's correct for that case.

--b.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

Hey!

On 10/29/21 09:45, J. Bruce Fields wrote:
> On Thu, Oct 28, 2021 at 10:48:50AM -0400, Steve Dickson wrote:
>> This kernel patch and an upcoming nfs-utils patch
>> adds a new export option 's2sc' which will allow
>> inter server to server copies.
> 
> They're already allowed by a module option.  Why is an export option
> better?  
I was talking with an old admin... that you might know :-)
and he was suggestion he would what to control which
export used these types of copies... And I agreed with him.

> And why should it be set on the destination server and not the
> source server?
Maybe I'm missing something... but I believe nfsd4_copy() is only
called on the destination server since it is the only
function that looks at the inter_copy_offload_enable param.

> 
> Really, first I think we should try to identify what the problem is that
> we're trying to solve.
I'm not trying to solve any problem... I'm just trying to enable a
feature in a sane and safe way. By only have one module param it
is on all the time on all copies... With an export opt, admins
can pick and choose which exports use it. I'm thinking that
is a bit less risky.

The option of setting the inter_copy_offload_enable is still
there... if admins want to go that route.

steved.
> 
> --b.
> 
>>
>> The option needs to be set on the destination server.
>>
>> The idea behind the option is to allow admins the
>> ability to control which export can do these
>> types of copies.
>>
>> Steve Dickson (1):
>>    nfsd4_copy: Adds the ability to do inter server to server on an export
>>
>>   fs/nfsd/nfs4proc.c               | 3 ++-
>>   include/uapi/linux/nfsd/export.h | 1 +
>>   2 files changed, 3 insertions(+), 1 deletion(-)
>>
>> -- 
>> 2.31.1
> 

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

On 10/29/21 10:26, J. Bruce Fields wrote:
> On Fri, Oct 29, 2021 at 09:45:34AM -0400, bfields wrote:
>> On Thu, Oct 28, 2021 at 10:48:50AM -0400, Steve Dickson wrote:
>>> This kernel patch and an upcoming nfs-utils patch
>>> adds a new export option 's2sc' which will allow
>>> inter server to server copies.
>>
>> They're already allowed by a module option.  Why is an export option
>> better?  And why should it be set on the destination server and not the
>> source server?
>>
>> Really, first I think we should try to identify what the problem is that
>> we're trying to solve.
> 
> I guess we're thinking: we expect server-to-server copy to be a win in
> some cases, but not others.
Right!

> 
> What would those cases look like?
> 
> Say you've got a client "C" and two servers, "S" and "T", and C is
> copying a file from S to T.
> 
> I'd expect bandwidth of the traditional read-write-loop copy to be the
> minimum of the network bandwidth between S and C, and the network
> bandwidth between C and T.  Are there common cases were the S-to-T
> bandwidth would be significantly less than both of those?
I was thinking the connection between S-to-T would be much
faster (come type of cluster interconnect) that the connection
between C-S or C-T.

> 
> My guess would've been that that's relatively unusual.  So as a first
> pass, just turning on COPY unconditionally doesn't seem so bad.
> 
> I know you're thinking about in cases where S and T are connected by a
> high-performance network not necessarily available to C.
Exactly... well said.

> 
> In that case, we know we want to use server-to-server copy for copies
> between S and T.  But is it necessarily a problem to also use it for
> copies between some other server and T?  Also, does knowing the export
> containing the destination file on T really tell you whether or not the
> copy will be coming from S and not from some other server?
No... All the export is doing is allowing the COPY request
to succeed. I don't think T has any idea where the
COPY is coming from... but you would know better that I :-)

> 
> I'd think the bigger issue in that case is figuring out how to configure
> S so that it returns the right IP address in the cnr_source_server field
> of the COPY_NOTIFY reply.  Currently we return address that the client
> sent the COPY_NOTIFY, and I don't know if that's correct for that case.
I didn't look at this part of the transaction since it was
not controlled by the module param... But How would S know its the
"right IP address" since it is C that did the mount to T and
is giving S the address?

Keeping in context of what I'm trying to do... All I'm
trying to do is enable the feature, and with your points,
I think you are making my case there should be an
alternate of enabling these copies.

steved.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[J. Bruce Fields]

On Fri, Oct 29, 2021 at 10:56:24AM -0400, Steve Dickson wrote:
> On 10/29/21 09:45, J. Bruce Fields wrote:
> >Really, first I think we should try to identify what the problem is that
> >we're trying to solve.
> I'm not trying to solve any problem... I'm just trying to enable a
> feature in a sane and safe way. By only have one module param it
> is on all the time on all copies... With an export opt, admins
> can pick and choose which exports use it. I'm thinking that
> is a bit less risky.

So, I'm not sure which risk you're thinking about.

If it's the risk of the client telling the destination server to copy
from a rogue server--I'm kind of regretting bringing that up.  If there
are bugs in that area, I'd rather they be fixed, than that we introduce
new configuration to work around bugs that may or may not exist.  They'd
need to be fixed anyway, for other reasons.  I think Dai has volunteered
to look through the code that handles replies to the small number of
operations concerned, and I think that's good enough due diligence.  And
I'm not getting the impression Trond is particularly worried about the
client code here either.

If the risk is performance--like I say, I'm not sure exactly what those
cases look like.

I've been thinking about it in terms of bandwidth, but maybe that's
wrong--the bigger problem may be when the source server is only
accessible to the client for some reason (like, you're copying from a
local server to a destination server that's outside a firewall).  Maybe
the destination server will end up waiting a long time trying to reach
the source.

But, again, I'm not sure an export option helps, because I don't see why
that problem would necessarily be per-destination-server-export.

Instead, we may just need to make sure the destination server is quick
to timeout, or has some mechanism to blacklist source servers so it's
not repeatedly timing out trying to copy from the same server.  I don't
know, I'm just thinking out loud.

> The option of setting the inter_copy_offload_enable is still
> there... if admins want to go that route.

Let's just stick with that for now, and leave it off by default until
we're sure it's mature enough.  Let's not introduce new configuration to
work around problems that we haven't really analyzed yet.

It's not as though turning on a module option is any more difficult than
setting an export option.

--b.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

On 10/29/21 12:40, J. Bruce Fields wrote:
> On Fri, Oct 29, 2021 at 10:56:24AM -0400, Steve Dickson wrote:
>> On 10/29/21 09:45, J. Bruce Fields wrote:
>>> Really, first I think we should try to identify what the problem is that
>>> we're trying to solve.
>> I'm not trying to solve any problem... I'm just trying to enable a
>> feature in a sane and safe way. By only have one module param it
>> is on all the time on all copies... With an export opt, admins
>> can pick and choose which exports use it. I'm thinking that
>> is a bit less risky.
> 
> So, I'm not sure which risk you're thinking about.
Now, I don't know how often a client copies files
from one server to another... but change the path on
all those copies... without making the admins aware
that is happening... is the risk I'm thinking about.

> 
> If it's the risk of the client telling the destination server to copy
> from a rogue server--I'm kind of regretting bringing that up.  If there
> are bugs in that area, I'd rather they be fixed, than that we introduce
> new configuration to work around bugs that may or may not exist.  They'd
> need to be fixed anyway, for other reasons.  I think Dai has volunteered
> to look through the code that handles replies to the small number of
> operations concerned, and I think that's good enough due diligence.  And
> I'm not getting the impression Trond is particularly worried about the
> client code here either.
I agree we should fix the bugs is better than working around them.
But having the bug effect all these types of copies verses
a few of them on a select number of exports... We still
would see the bugs.

> 
> If the risk is performance--like I say, I'm not sure exactly what those
> cases look like.
Well Jorge seem to show there was a was a big performance gain [1]

> 
> I've been thinking about it in terms of bandwidth, but maybe that's
> wrong--the bigger problem may be when the source server is only
> accessible to the client for some reason (like, you're copying from a
> local server to a destination server that's outside a firewall).  Maybe
> the destination server will end up waiting a long time trying to reach
> the source.
> 
> But, again, I'm not sure an export option helps, because I don't see why
> that problem would necessarily be per-destination-server-export.
It helps enabling the technology without out the big hammer approach.
Allowing these type of copies on a couple exports verses all
exports would still show any problem.

> 
> Instead, we may just need to make sure the destination server is quick
> to timeout, or has some mechanism to blacklist source servers so it's
> not repeatedly timing out trying to copy from the same server.  I don't
> know, I'm just thinking out loud.
This sounds like you not happy with the design... And it appears
where is there today is what the RPC specified...
Again... I'm just trying to enable it...

> 
>> The option of setting the inter_copy_offload_enable is still
>> there... if admins want to go that route.
> 
> Let's just stick with that for now, and leave it off by default until
> we're sure it's mature enough.  Let's not introduce new configuration to
> work around problems that we haven't really analyzed yet.
How is this going to find problems? At least with the export option
it is documented and it more if a stick you toe in the pool verses
jumping in...

steved.
> 
> It's not as though turning on a module option is any more difficult than
> setting an export option.
> 


[1] https://marc.info/?l=linux-nfs&m=149201305018467&w=2

Re: [PATCH 0/1] Enable inter server to server copies on a export

[J. Bruce Fields]

On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
> On 10/29/21 12:40, J. Bruce Fields wrote:
> >Let's just stick with that for now, and leave it off by default until
> >we're sure it's mature enough.  Let's not introduce new configuration to
> >work around problems that we haven't really analyzed yet.
> How is this going to find problems? At least with the export option
> it is documented

That sounds fixable.  We need documentation of module parameters anyway.

> and it more if a stick you toe in the pool verses
> jumping in...

If we want more fine-grained control, I'm not yet seeing the argument
that an export option on the destination server side is the way to do
it.

Let's document the module parameter and go with that for now.

--b.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

Hey!

On 10/29/21 15:14, J. Bruce Fields wrote:
> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
>> On 10/29/21 12:40, J. Bruce Fields wrote:
>>> Let's just stick with that for now, and leave it off by default until
>>> we're sure it's mature enough.  Let's not introduce new configuration to
>>> work around problems that we haven't really analyzed yet.
>> How is this going to find problems? At least with the export option
>> it is documented
> 
> That sounds fixable.  We need documentation of module parameters anyway.
Yeah I just took I don't see any documentation of module
parameters anywhere for any of the modules. But by documentation
I meant having the feature in the exports(5) manpage.

> 
>> and it more if a stick you toe in the pool verses
>> jumping in...
> 
> If we want more fine-grained control, I'm not yet seeing the argument
> that an export option on the destination server side is the way to do
> it.
> 
> Let's document the module parameter and go with that for now.
Now that cp will use copy_file_range() when available,
what are the steps needed to enable these fast copies?

steved.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[J. Bruce Fields]

On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
> Hey!
> 
> On 10/29/21 15:14, J. Bruce Fields wrote:
> >On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
> >>On 10/29/21 12:40, J. Bruce Fields wrote:
> >>>Let's just stick with that for now, and leave it off by default until
> >>>we're sure it's mature enough.  Let's not introduce new configuration to
> >>>work around problems that we haven't really analyzed yet.
> >>How is this going to find problems? At least with the export option
> >>it is documented
> >
> >That sounds fixable.  We need documentation of module parameters anyway.
> Yeah I just took I don't see any documentation of module
> parameters anywhere for any of the modules. But by documentation
> I meant having the feature in the exports(5) manpage.

I think I'd probably create a new page for sysctls (this isn't the only
one needing documentation), and make sure it's listed in the "SEE ALSO"
section of the other man pages.

> >>and it more if a stick you toe in the pool verses
> >>jumping in...
> >
> >If we want more fine-grained control, I'm not yet seeing the argument
> >that an export option on the destination server side is the way to do
> >it.
> >
> >Let's document the module parameter and go with that for now.
> Now that cp will use copy_file_range() when available,
> what are the steps needed to enable these fast copies?

1) Make sure client and both servers support NFSv4.2 and
server-to-server copy.

2) Make sure destination server can access (at least for read) any
exports on the source that you want to be able to copy from.

3) echo 1 >/sys/module/nfsd/parameters/inter_copy_offload_enable on the
destination server.

--b.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Chuck Lever III]

> On Nov 1, 2021, at 11:40 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> 
> On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
>> Hey!
>> 
>> On 10/29/21 15:14, J. Bruce Fields wrote:
>>> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
>>>> On 10/29/21 12:40, J. Bruce Fields wrote:
>>>>> Let's just stick with that for now, and leave it off by default until
>>>>> we're sure it's mature enough.  Let's not introduce new configuration to
>>>>> work around problems that we haven't really analyzed yet.
>>>> How is this going to find problems? At least with the export option
>>>> it is documented
>>> 
>>> That sounds fixable.  We need documentation of module parameters anyway.
>> Yeah I just took I don't see any documentation of module
>> parameters anywhere for any of the modules. But by documentation
>> I meant having the feature in the exports(5) manpage.
> 
> I think I'd probably create a new page for sysctls (this isn't the only
> one needing documentation), and make sure it's listed in the "SEE ALSO"
> section of the other man pages.

Aren't sysctls documented under Documentation/ ?


>>>> and it more if a stick you toe in the pool verses
>>>> jumping in...
>>> 
>>> If we want more fine-grained control, I'm not yet seeing the argument
>>> that an export option on the destination server side is the way to do
>>> it.
>>> 
>>> Let's document the module parameter and go with that for now.
>> Now that cp will use copy_file_range() when available,
>> what are the steps needed to enable these fast copies?
> 
> 1) Make sure client and both servers support NFSv4.2 and
> server-to-server copy.
> 
> 2) Make sure destination server can access (at least for read) any
> exports on the source that you want to be able to copy from.
> 
> 3) echo 1 >/sys/module/nfsd/parameters/inter_copy_offload_enable on the
> destination server.
> 
> --b.

--
Chuck Lever

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

On 11/1/21 12:55, Chuck Lever III wrote:
> 
>> On Nov 1, 2021, at 11:40 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
>>
>> On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
>>> Hey!
>>>
>>> On 10/29/21 15:14, J. Bruce Fields wrote:
>>>> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
>>>>> On 10/29/21 12:40, J. Bruce Fields wrote:
>>>>>> Let's just stick with that for now, and leave it off by default until
>>>>>> we're sure it's mature enough.  Let's not introduce new configuration to
>>>>>> work around problems that we haven't really analyzed yet.
>>>>> How is this going to find problems? At least with the export option
>>>>> it is documented
>>>>
>>>> That sounds fixable.  We need documentation of module parameters anyway.
>>> Yeah I just took I don't see any documentation of module
>>> parameters anywhere for any of the modules. But by documentation
>>> I meant having the feature in the exports(5) manpage.
>>
>> I think I'd probably create a new page for sysctls (this isn't the only
>> one needing documentation), and make sure it's listed in the "SEE ALSO"
>> section of the other man pages.
> 
> Aren't sysctls documented under Documentation/ ?No... there is a Documentation/admin-guide/sysctl/sunrpc.rst
but it talks about rpcdebug commands and there is nothing
that documents the module parameters under
Documentation/filesystems/nfs

steved.

> 
> 
>>>>> and it more if a stick you toe in the pool verses
>>>>> jumping in...
>>>>
>>>> If we want more fine-grained control, I'm not yet seeing the argument
>>>> that an export option on the destination server side is the way to do
>>>> it.
>>>>
>>>> Let's document the module parameter and go with that for now.
>>> Now that cp will use copy_file_range() when available,
>>> what are the steps needed to enable these fast copies?
>>
>> 1) Make sure client and both servers support NFSv4.2 and
>> server-to-server copy.
>>
>> 2) Make sure destination server can access (at least for read) any
>> exports on the source that you want to be able to copy from.
>>
>> 3) echo 1 >/sys/module/nfsd/parameters/inter_copy_offload_enable on the
>> destination server.
>>
>> --b.
> 
> --
> Chuck Lever
> 
> 
> 

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Bruce Fields]

On Mon, Nov 01, 2021 at 04:55:45PM +0000, Chuck Lever III wrote:
> 
> > On Nov 1, 2021, at 11:40 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > 
> > On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
> >> Hey!
> >> 
> >> On 10/29/21 15:14, J. Bruce Fields wrote:
> >>> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
> >>>> On 10/29/21 12:40, J. Bruce Fields wrote:
> >>>>> Let's just stick with that for now, and leave it off by default until
> >>>>> we're sure it's mature enough.  Let's not introduce new configuration to
> >>>>> work around problems that we haven't really analyzed yet.
> >>>> How is this going to find problems? At least with the export option
> >>>> it is documented
> >>> 
> >>> That sounds fixable.  We need documentation of module parameters anyway.
> >> Yeah I just took I don't see any documentation of module
> >> parameters anywhere for any of the modules. But by documentation
> >> I meant having the feature in the exports(5) manpage.
> > 
> > I think I'd probably create a new page for sysctls (this isn't the only
> > one needing documentation), and make sure it's listed in the "SEE ALSO"
> > section of the other man pages.
> 
> Aren't sysctls documented under Documentation/ ?

Sorry, I meant "module parameter".  Anyway, yes, looks like both are
listed in Documentation/admin-guide/kernel-parameters.txt.

Might be nice if these were in a man page too somewhere, but maybe
that's not the most important thing these days.

--b.

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 91ba391f9b32..14bc3f0b0149 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3243,6 +3243,19 @@
 			driver. A non-zero value sets the minimum interval
 			in seconds between layoutstats transmissions.
 
+	nfsd.inter_copy_offload_enable =
+			[NFSv4.2] When set to 1, the server will support
+			server-to-server copies for which this server is
+			the destination of the copy.
+
+	nfsd.nfsd4_ssc_umount_timeout =
+			[NFSv4.2] When used as the destination of a
+			server-to-server copy, knfsd temporarily mounts
+			the source server.  It caches the mount in case
+			it will be needed again, and discards it if not
+			used for the number of milliseconds specified by
+			this parameter.
+
 	nfsd.nfs4_disable_idmapping=
 			[NFSv4] When set to the default of '1', the NFSv4
 			server will return only numeric uids and gids to
@@ -3250,6 +3263,7 @@
 			and gids from such clients.  This is intended to ease
 			migration from NFSv2/v3.
 
+
 	nmi_backtrace.backtrace_idle [KNL]
 			Dump stacks even of idle CPUs in response to an
 			NMI stack-backtrace request.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Chuck Lever III]

> On Nov 1, 2021, at 2:39 PM, Bruce Fields <bfields@fieldses.org> wrote:
> 
> On Mon, Nov 01, 2021 at 04:55:45PM +0000, Chuck Lever III wrote:
>> 
>>> On Nov 1, 2021, at 11:40 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
>>> 
>>> On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
>>>> Hey!
>>>> 
>>>> On 10/29/21 15:14, J. Bruce Fields wrote:
>>>>> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
>>>>>> On 10/29/21 12:40, J. Bruce Fields wrote:
>>>>>>> Let's just stick with that for now, and leave it off by default until
>>>>>>> we're sure it's mature enough.  Let's not introduce new configuration to
>>>>>>> work around problems that we haven't really analyzed yet.
>>>>>> How is this going to find problems? At least with the export option
>>>>>> it is documented
>>>>> 
>>>>> That sounds fixable.  We need documentation of module parameters anyway.
>>>> Yeah I just took I don't see any documentation of module
>>>> parameters anywhere for any of the modules. But by documentation
>>>> I meant having the feature in the exports(5) manpage.
>>> 
>>> I think I'd probably create a new page for sysctls (this isn't the only
>>> one needing documentation), and make sure it's listed in the "SEE ALSO"
>>> section of the other man pages.
>> 
>> Aren't sysctls documented under Documentation/ ?
> 
> Sorry, I meant "module parameter".  Anyway, yes, looks like both are
> listed in Documentation/admin-guide/kernel-parameters.txt.
> 
> Might be nice if these were in a man page too somewhere, but maybe
> that's not the most important thing these days.
> 
> --b.

This looks like a step forward to me.


> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 91ba391f9b32..14bc3f0b0149 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -3243,6 +3243,19 @@
> 			driver. A non-zero value sets the minimum interval
> 			in seconds between layoutstats transmissions.
> 
> +	nfsd.inter_copy_offload_enable =
> +			[NFSv4.2] When set to 1, the server will support
> +			server-to-server copies for which this server is
> +			the destination of the copy.
> +
> +	nfsd.nfsd4_ssc_umount_timeout =
> +			[NFSv4.2] When used as the destination of a
> +			server-to-server copy, knfsd temporarily mounts
> +			the source server.  It caches the mount in case
> +			it will be needed again, and discards it if not
> +			used for the number of milliseconds specified by
> +			this parameter.
> +
> 	nfsd.nfs4_disable_idmapping=
> 			[NFSv4] When set to the default of '1', the NFSv4
> 			server will return only numeric uids and gids to
> @@ -3250,6 +3263,7 @@
> 			and gids from such clients.  This is intended to ease
> 			migration from NFSv2/v3.
> 
> +
> 	nmi_backtrace.backtrace_idle [KNL]
> 			Dump stacks even of idle CPUs in response to an
> 			NMI stack-backtrace request.

--
Chuck Lever

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

On 11/1/21 11:40, J. Bruce Fields wrote:
> On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
>> Hey!
>>
>> On 10/29/21 15:14, J. Bruce Fields wrote:
>>> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
>>>> On 10/29/21 12:40, J. Bruce Fields wrote:
>>>>> Let's just stick with that for now, and leave it off by default until
>>>>> we're sure it's mature enough.  Let's not introduce new configuration to
>>>>> work around problems that we haven't really analyzed yet.
>>>> How is this going to find problems? At least with the export option
>>>> it is documented
>>>
>>> That sounds fixable.  We need documentation of module parameters anyway.
>> Yeah I just took I don't see any documentation of module
>> parameters anywhere for any of the modules. But by documentation
>> I meant having the feature in the exports(5) manpage.
> 
> I think I'd probably create a new page for sysctls (this isn't the only
> one needing documentation), and make sure it's listed in the "SEE ALSO"
> section of the other man pages.
> 
>>>> and it more if a stick you toe in the pool verses
>>>> jumping in...
>>>
>>> If we want more fine-grained control, I'm not yet seeing the argument
>>> that an export option on the destination server side is the way to do
>>> it.
>>>
>>> Let's document the module parameter and go with that for now.
>> Now that cp will use copy_file_range() when available,
>> what are the steps needed to enable these fast copies?
> 
> 1) Make sure client and both servers support NFSv4.2 and
> server-to-server copy.
Something is already figuring this out... The only time
the client sends a COPY_NOTIFY and COPY is when both
mounts are 4.2. I have not looked into but that is what
the network traces are showing.

> 
> 2) Make sure destination server can access (at least for read) any
> exports on the source that you want to be able to copy from.
How can one server know what the other server has exported
or access to??

> 
> 3) echo 1 >/sys/module/nfsd/parameters/inter_copy_offload_enable on the
> destination server.
Who would be doing this? Plus this would not survive over a reboot.
An export would as well a /etc/modprobe.d/ file.

I can see the admin setting up the export but I really
don't see the admin doing the echo or creating the file
esp since the neither would is documented.

steved.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

On 11/1/21 14:39, Bruce Fields wrote:
> On Mon, Nov 01, 2021 at 04:55:45PM +0000, Chuck Lever III wrote:
>>
>>> On Nov 1, 2021, at 11:40 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
>>>
>>> On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
>>>> Hey!
>>>>
>>>> On 10/29/21 15:14, J. Bruce Fields wrote:
>>>>> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
>>>>>> On 10/29/21 12:40, J. Bruce Fields wrote:
>>>>>>> Let's just stick with that for now, and leave it off by default until
>>>>>>> we're sure it's mature enough.  Let's not introduce new configuration to
>>>>>>> work around problems that we haven't really analyzed yet.
>>>>>> How is this going to find problems? At least with the export option
>>>>>> it is documented
>>>>>
>>>>> That sounds fixable.  We need documentation of module parameters anyway.
>>>> Yeah I just took I don't see any documentation of module
>>>> parameters anywhere for any of the modules. But by documentation
>>>> I meant having the feature in the exports(5) manpage.
>>>
>>> I think I'd probably create a new page for sysctls (this isn't the only
>>> one needing documentation), and make sure it's listed in the "SEE ALSO"
>>> section of the other man pages.
>>
>> Aren't sysctls documented under Documentation/ ?
> 
> Sorry, I meant "module parameter".  Anyway, yes, looks like both are
> listed in Documentation/admin-guide/kernel-parameters.txt.
> 
> Might be nice if these were in a man page too somewhere, but maybe
> that's not the most important thing these days.
> 
> --b.
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 91ba391f9b32..14bc3f0b0149 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -3243,6 +3243,19 @@
>   			driver. A non-zero value sets the minimum interval
>   			in seconds between layoutstats transmissions.
>   
> +	nfsd.inter_copy_offload_enable =
> +			[NFSv4.2] When set to 1, the server will support
> +			server-to-server copies for which this server is
> +			the destination of the copy.
> +
> +	nfsd.nfsd4_ssc_umount_timeout =
> +			[NFSv4.2] When used as the destination of a
> +			server-to-server copy, knfsd temporarily mounts
> +			the source server.  It caches the mount in case
> +			it will be needed again, and discards it if not
> +			used for the number of milliseconds specified by
> +			this parameter.
> +
>   	nfsd.nfs4_disable_idmapping=
>   			[NFSv4] When set to the default of '1', the NFSv4
>   			server will return only numeric uids and gids to
> @@ -3250,6 +3263,7 @@
>   			and gids from such clients.  This is intended to ease
>   			migration from NFSv2/v3.
>   
> +
>   	nmi_backtrace.backtrace_idle [KNL]
>   			Dump stacks even of idle CPUs in response to an
>   			NMI stack-backtrace request.
> 
Interesting... I don't see these in the Linus tree I'm looking at [1]
find Documentation/ -type f  | xargs grep -i inter_copy_offload_enable

steved

[1] git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Re: [PATCH 0/1] Enable inter server to server copies on a export

[dai.ngo]

On 11/1/21 12:02 PM, Steve Dickson wrote:
>
>
> On 11/1/21 11:40, J. Bruce Fields wrote:
>> On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
>>> Hey!
>>>
>>> On 10/29/21 15:14, J. Bruce Fields wrote:
>>>> On Fri, Oct 29, 2021 at 01:30:36PM -0400, Steve Dickson wrote:
>>>>> On 10/29/21 12:40, J. Bruce Fields wrote:
>>>>>> Let's just stick with that for now, and leave it off by default 
>>>>>> until
>>>>>> we're sure it's mature enough.  Let's not introduce new 
>>>>>> configuration to
>>>>>> work around problems that we haven't really analyzed yet.
>>>>> How is this going to find problems? At least with the export option
>>>>> it is documented
>>>>
>>>> That sounds fixable.  We need documentation of module parameters 
>>>> anyway.
>>> Yeah I just took I don't see any documentation of module
>>> parameters anywhere for any of the modules. But by documentation
>>> I meant having the feature in the exports(5) manpage.
>>
>> I think I'd probably create a new page for sysctls (this isn't the only
>> one needing documentation), and make sure it's listed in the "SEE ALSO"
>> section of the other man pages.
>>
>>>>> and it more if a stick you toe in the pool verses
>>>>> jumping in...
>>>>
>>>> If we want more fine-grained control, I'm not yet seeing the argument
>>>> that an export option on the destination server side is the way to do
>>>> it.
>>>>
>>>> Let's document the module parameter and go with that for now.
>>> Now that cp will use copy_file_range() when available,
>>> what are the steps needed to enable these fast copies?
>>
>> 1) Make sure client and both servers support NFSv4.2 and
>> server-to-server copy.
> Something is already figuring this out... The only time
> the client sends a COPY_NOTIFY and COPY is when both
> mounts are 4.2. I have not looked into but that is what
> the network traces are showing.
>
>>
>> 2) Make sure destination server can access (at least for read) any
>> exports on the source that you want to be able to copy from.
> How can one server know what the other server has exported
> or access to??
>
>>
>> 3) echo 1 >/sys/module/nfsd/parameters/inter_copy_offload_enable on the
>> destination server.
> Who would be doing this? Plus this would not survive over a reboot.
> An export would as well a /etc/modprobe.d/ file.

You can add a line in /etc/modprobe.d/nfsd.conf:

options nfsd inter_copy_offload_enable=Y

to enable the option.

-Dai


>
> I can see the admin setting up the export but I really
> don't see the admin doing the echo or creating the file
> esp since the neither would is documented.
>
> steved.
>

Re: [PATCH 0/1] Enable inter server to server copies on a export

[J. Bruce Fields]

On Mon, Nov 01, 2021 at 12:22:15PM -0700, dai.ngo@oracle.com wrote:
> 
> On 11/1/21 12:02 PM, Steve Dickson wrote:
> >
> >
> >On 11/1/21 11:40, J. Bruce Fields wrote:
> >>On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
> >>>Now that cp will use copy_file_range() when available,
> >>>what are the steps needed to enable these fast copies?
> >>
> >>1) Make sure client and both servers support NFSv4.2 and
> >>server-to-server copy.
> >Something is already figuring this out... The only time
> >the client sends a COPY_NOTIFY and COPY is when both
> >mounts are 4.2. I have not looked into but that is what
> >the network traces are showing.

Right.  I was thinking what I'd tell an admin who wanted to set up
server-to-server copy.  The first thing they'd need to do was check that
their clients and servers were new enough.

> >>2) Make sure destination server can access (at least for read) any
> >>exports on the source that you want to be able to copy from.
> >How can one server know what the other server has exported
> >or access to??

And the second is to make sure that the destination server is able to
read from the source.

> >>3) echo 1 >/sys/module/nfsd/parameters/inter_copy_offload_enable on the
> >>destination server.
> >Who would be doing this? Plus this would not survive over a reboot.
> >An export would as well a /etc/modprobe.d/ file.
> 
> You can add a line in /etc/modprobe.d/nfsd.conf:
> 
> options nfsd inter_copy_offload_enable=Y
> 
> to enable the option.

Yep, it would be better to document it that way, thanks.

--b.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Bruce Fields]

On Mon, Nov 01, 2021 at 03:10:31PM -0400, Steve Dickson wrote:
> 
> 
> On 11/1/21 14:39, Bruce Fields wrote:
> >diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> >index 91ba391f9b32..14bc3f0b0149 100644
> >--- a/Documentation/admin-guide/kernel-parameters.txt
> >+++ b/Documentation/admin-guide/kernel-parameters.txt
> >@@ -3243,6 +3243,19 @@
> >  			driver. A non-zero value sets the minimum interval
> >  			in seconds between layoutstats transmissions.
> >+	nfsd.inter_copy_offload_enable =
> >+			[NFSv4.2] When set to 1, the server will support
> >+			server-to-server copies for which this server is
> >+			the destination of the copy.
> >+
> >+	nfsd.nfsd4_ssc_umount_timeout =
> >+			[NFSv4.2] When used as the destination of a
> >+			server-to-server copy, knfsd temporarily mounts
> >+			the source server.  It caches the mount in case
> >+			it will be needed again, and discards it if not
> >+			used for the number of milliseconds specified by
> >+			this parameter.
> >+
> >  	nfsd.nfs4_disable_idmapping=
> >  			[NFSv4] When set to the default of '1', the NFSv4
> >  			server will return only numeric uids and gids to
> >@@ -3250,6 +3263,7 @@
> >  			and gids from such clients.  This is intended to ease
> >  			migration from NFSv2/v3.
> >+
> >  	nmi_backtrace.backtrace_idle [KNL]
> >  			Dump stacks even of idle CPUs in response to an
> >  			NMI stack-backtrace request.
> >
> Interesting... I don't see these in the Linus tree I'm looking at [1]
> find Documentation/ -type f  | xargs grep -i inter_copy_offload_enable

I was suggesting that as a patch.  It's queued up for 5.16 now.

--b.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

On 11/1/21 15:25, J. Bruce Fields wrote:
> On Mon, Nov 01, 2021 at 12:22:15PM -0700, dai.ngo@oracle.com wrote:
>>
>> On 11/1/21 12:02 PM, Steve Dickson wrote:
>>>
>>>
>>> On 11/1/21 11:40, J. Bruce Fields wrote:
>>>> On Mon, Nov 01, 2021 at 11:30:48AM -0400, Steve Dickson wrote:
>>>>> Now that cp will use copy_file_range() when available,
>>>>> what are the steps needed to enable these fast copies?
>>>>
>>>> 1) Make sure client and both servers support NFSv4.2 and
>>>> server-to-server copy.
>>> Something is already figuring this out... The only time
>>> the client sends a COPY_NOTIFY and COPY is when both
>>> mounts are 4.2. I have not looked into but that is what
>>> the network traces are showing.
> 
> Right.  I was thinking what I'd tell an admin who wanted to set up
> server-to-server copy.  The first thing they'd need to do was check that
> their clients and servers were new enough.
Well the code went in over two years ago
ce0887ac96d35 (Olga Kornievskaia 2019-10-09 11:50:48 -0400 1663)
so most modern kernel will have the feature.

The real question is does the cp command have the
copy_file_range() support which didn't go in until
Mar of this year
* Wed Mar 24 2021 Kamil Dudka <kdudka@redhat.com> - 8.32-20
- cp: use copy_file_range if available

> 
>>>> 2) Make sure destination server can access (at least for read) any
>>>> exports on the source that you want to be able to copy from.
>>> How can one server know what the other server has exported
>>> or access to??
> 
> And the second is to make sure that the destination server is able to
> read from the source.
> 
>>>> 3) echo 1 >/sys/module/nfsd/parameters/inter_copy_offload_enable on the
>>>> destination server.
>>> Who would be doing this? Plus this would not survive over a reboot.
>>> An export would as well a /etc/modprobe.d/ file.
>>
>> You can add a line in /etc/modprobe.d/nfsd.conf:
>>
>> options nfsd inter_copy_offload_enable=Y
>>
>> to enable the option.
Yes... This is what I meant by "a /etc/modprobe.d/ file"

> 
> Yep, it would be better to document it that way, thanks.
But the question is who would create this file?
nfs-utils or the admin?

If it is nfs-utils, it is a global switch unbeknown by
the admin (but documented). If by the admin, the person
would know about but it still will not documented.

With a export option, it would be per export switch
and documented. Just saying... :-)

steved.

Re: [PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

On 11/1/21 15:26, Bruce Fields wrote:
> On Mon, Nov 01, 2021 at 03:10:31PM -0400, Steve Dickson wrote:
>>
>>
>> On 11/1/21 14:39, Bruce Fields wrote:
>>> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
>>> index 91ba391f9b32..14bc3f0b0149 100644
>>> --- a/Documentation/admin-guide/kernel-parameters.txt
>>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>>> @@ -3243,6 +3243,19 @@
>>>   			driver. A non-zero value sets the minimum interval
>>>   			in seconds between layoutstats transmissions.
>>> +	nfsd.inter_copy_offload_enable =
>>> +			[NFSv4.2] When set to 1, the server will support
>>> +			server-to-server copies for which this server is
>>> +			the destination of the copy.
>>> +
>>> +	nfsd.nfsd4_ssc_umount_timeout =
>>> +			[NFSv4.2] When used as the destination of a
>>> +			server-to-server copy, knfsd temporarily mounts
>>> +			the source server.  It caches the mount in case
>>> +			it will be needed again, and discards it if not
>>> +			used for the number of milliseconds specified by
>>> +			this parameter.
>>> +
>>>   	nfsd.nfs4_disable_idmapping=
>>>   			[NFSv4] When set to the default of '1', the NFSv4
>>>   			server will return only numeric uids and gids to
>>> @@ -3250,6 +3263,7 @@
>>>   			and gids from such clients.  This is intended to ease
>>>   			migration from NFSv2/v3.
>>> +
>>>   	nmi_backtrace.backtrace_idle [KNL]
>>>   			Dump stacks even of idle CPUs in response to an
>>>   			NMI stack-backtrace request.
>>>
>> Interesting... I don't see these in the Linus tree I'm looking at [1]
>> find Documentation/ -type f  | xargs grep -i inter_copy_offload_enable
> 
> I was suggesting that as a patch.  It's queued up for 5.16 now.
Sorry I did miss the patch part of the email...

Does it make sense to document both nfs and nfsd module
parameters in a couple man pages?

steved.

[PATCH 0/1] Enable inter server to server copies on a export

[Steve Dickson]

The patch introduces the 's2sc' export option to exportfs
and along with the recently sent kernel patch will
enable inter server to server copies on specified
exports

The idea being when servers have very interconnect
(as in cluster environments) admins can use this 
option to create very fast copy paths. 

Steve Dickson (1):
  exportfs: Add the 's2sc' option allowing inter server to server copies

 support/include/nfs/export.h |  1 +
 support/nfs/exports.c        |  5 +++++
 utils/exportfs/exportfs.c    |  2 ++
 utils/exportfs/exports.man   | 14 ++++++++++++++
 4 files changed, 22 insertions(+)

-- 
2.31.1