* server-to-server copy by default
@ 2021-10-20 15:54 J. Bruce Fields
2021-10-20 16:00 ` Chuck Lever III
` (2 more replies)
0 siblings, 3 replies; 27+ messages in thread
From: J. Bruce Fields @ 2021-10-20 15:54 UTC (permalink / raw)
To: linux-nfs; +Cc: dai.ngo, Olga Kornievskaia, Steve Dickson, Chuck Lever
knfsd has supported server-to-server copy for a couple years (since
5.5). You have set a module parameter to enable it. I'm getting asked
when we could turn that parameter on by default.
I've got a couple vague criteria: one just general maturity, the other a
security question:
1. General maturity: the only reports I recall seeing are from testers.
Is anyone using this? Does it work for them? Do they find a benefit?
Maybe we could turn it on by default in one distro (Fedora?) and promote
it a little and see what that turns up?
2. Security question: with server-to-server copy enabled, you can send
the server a COPY call with any random address, and the server will
mount that address, open a file, and read from it. Is that safe?
Normally we only mount servers that were chosen by root. Here we'll
mount any random server that some client told us to. What's the worst
that random server can do? Do we trust our xdr decoding? Can it DOS us
by throwing the client's state recovery code into some loop with weird
error returns? Etc.
Maybe it's fine. I'm OK with some level of risk. I just want to make
sure somebody's thought this through.
There's also interest in allowing unprivileged NFS mounts, but I don't
think we've turned that on yet, partly for similar reasons. This is a
subset of that problem.
--b.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: server-to-server copy by default 2021-10-20 15:54 server-to-server copy by default J. Bruce Fields @ 2021-10-20 16:00 ` Chuck Lever III 2021-10-20 16:33 ` Olga Kornievskaia 2021-10-20 17:24 ` Steve Dickson 2021-10-20 16:37 ` Olga Kornievskaia 2021-11-01 18:22 ` Charles Hedrick 2 siblings, 2 replies; 27+ messages in thread From: Chuck Lever III @ 2021-10-20 16:00 UTC (permalink / raw) To: Bruce Fields Cc: Linux NFS Mailing List, Dai Ngo, Olga Kornievskaia, Steve Dickson > On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: > > knfsd has supported server-to-server copy for a couple years (since > 5.5). You have set a module parameter to enable it. I'm getting asked > when we could turn that parameter on by default. > > I've got a couple vague criteria: one just general maturity, the other a > security question: > > 1. General maturity: the only reports I recall seeing are from testers. > Is anyone using this? Does it work for them? Do they find a benefit? > Maybe we could turn it on by default in one distro (Fedora?) and promote > it a little and see what that turns up? I like the idea of enabling it in one of the technology preview distributions. But wrt the maturity question, is the work finished? Or, perhaps a better question is do we have a minimum viable product here that can be enabled, or is more work needed to meet even that bar? One thing that I recall is missing is support for Kerberos in the server-to-server copy operation. Is that in plan, or deemed unimportant? > 2. Security question: with server-to-server copy enabled, you can send > the server a COPY call with any random address, and the server will > mount that address, open a file, and read from it. Is that safe? > > Normally we only mount servers that were chosen by root. Here we'll > mount any random server that some client told us to. What's the worst > that random server can do? Do we trust our xdr decoding? Can it DOS us > by throwing the client's state recovery code into some loop with weird > error returns? Etc. A basic question is what is in distribution QE test suites that could exercise this feature? Should upstream be tasked with providing any missing pieces (as part of, say, pynfs, or nfstests)? > Maybe it's fine. I'm OK with some level of risk. I just want to make > sure somebody's thought this through. > > There's also interest in allowing unprivileged NFS mounts, but I don't > think we've turned that on yet, partly for similar reasons. This is a > subset of that problem. > > --b. -- Chuck Lever ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 16:00 ` Chuck Lever III @ 2021-10-20 16:33 ` Olga Kornievskaia 2021-10-20 19:03 ` dai.ngo 2021-10-20 17:24 ` Steve Dickson 1 sibling, 1 reply; 27+ messages in thread From: Olga Kornievskaia @ 2021-10-20 16:33 UTC (permalink / raw) To: Chuck Lever III Cc: Bruce Fields, Linux NFS Mailing List, Dai Ngo, Steve Dickson On Wed, Oct 20, 2021 at 12:00 PM Chuck Lever III <chuck.lever@oracle.com> wrote: > > > > > On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: > > > > knfsd has supported server-to-server copy for a couple years (since > > 5.5). You have set a module parameter to enable it. I'm getting asked > > when we could turn that parameter on by default. > > > > I've got a couple vague criteria: one just general maturity, the other a > > security question: > > > > 1. General maturity: the only reports I recall seeing are from testers. > > Is anyone using this? Does it work for them? Do they find a benefit? > > Maybe we could turn it on by default in one distro (Fedora?) and promote > > it a little and see what that turns up? > > I like the idea of enabling it in one of the technology > preview distributions. > > But wrt the maturity question, is the work finished? Or, > perhaps a better question is do we have a minimum viable > product here that can be enabled, or is more work needed > to meet even that bar? > > One thing that I recall is missing is support for Kerberos > in the server-to-server copy operation. Is that in plan, > or deemed unimportant? Netapp has some code for gssv3 support which is required for server-to-server and possibly some copy offload pieces (Andy's work before his retirement). Anna was picking up the gssv3 work but hasn't had the cycles yet to complete it. We can make it more of a priority if that is a show stopper. > > 2. Security question: with server-to-server copy enabled, you can send > > the server a COPY call with any random address, and the server will > > mount that address, open a file, and read from it. Is that safe? > > > > Normally we only mount servers that were chosen by root. Here we'll > > mount any random server that some client told us to. What's the worst > > that random server can do? Do we trust our xdr decoding? Can it DOS us > > by throwing the client's state recovery code into some loop with weird > > error returns? Etc. > > A basic question is what is in distribution QE test suites > that could exercise this feature? Should upstream be tasked > with providing any missing pieces (as part of, say, pynfs, > or nfstests)? There are server-to-server tests in the nfstest testing suite. I'm not sure if any of the xfstest copy_ofload exercise server to server capability. Anna wrote the tests. > > Maybe it's fine. I'm OK with some level of risk. I just want to make > > sure somebody's thought this through. > > > > There's also interest in allowing unprivileged NFS mounts, but I don't > > think we've turned that on yet, partly for similar reasons. This is a > > subset of that problem. > > > > --b. > > -- > Chuck Lever > > > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 16:33 ` Olga Kornievskaia @ 2021-10-20 19:03 ` dai.ngo 2021-10-20 20:29 ` Bruce Fields 0 siblings, 1 reply; 27+ messages in thread From: dai.ngo @ 2021-10-20 19:03 UTC (permalink / raw) To: Bruce Fields Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On 10/20/21 9:33 AM, Olga Kornievskaia wrote: > On Wed, Oct 20, 2021 at 12:00 PM Chuck Lever III <chuck.lever@oracle.com> wrote: >> >> >>> On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: >>> >>> knfsd has supported server-to-server copy for a couple years (since >>> 5.5). You have set a module parameter to enable it. I'm getting asked >>> when we could turn that parameter on by default. >>> >>> I've got a couple vague criteria: one just general maturity, the other a >>> security question: >>> >>> 1. General maturity: the only reports I recall seeing are from testers. >>> Is anyone using this? Does it work for them? Do they find a benefit? >>> Maybe we could turn it on by default in one distro (Fedora?) and promote >>> it a little and see what that turns up? >> I like the idea of enabling it in one of the technology >> preview distributions. >> >> But wrt the maturity question, is the work finished? Or, >> perhaps a better question is do we have a minimum viable >> product here that can be enabled, or is more work needed >> to meet even that bar? >> >> One thing that I recall is missing is support for Kerberos >> in the server-to-server copy operation. Is that in plan, >> or deemed unimportant? > Netapp has some code for gssv3 support which is required for > server-to-server and possibly some copy offload pieces (Andy's work > before his retirement). Anna was picking up the gssv3 work but hasn't > had the cycles yet to complete it. We can make it more of a priority > if that is a show stopper. yes, afaik, the gssv3 support for server-to-server is the only missing functionality (besides the potential security related issues mentioned in this thread). It'd be good if we can implement this, or as Steve suggested we can wait a little while for the technology is stable before adding gssv3 support. Do you consider this missing functionality as a show stopper? > >>> 2. Security question: with server-to-server copy enabled, you can send >>> the server a COPY call with any random address, and the server will >>> mount that address, open a file, and read from it. Is that safe? The client already has write access to the share on the destination server, it can write any data to the destination file. If the client sends a COPY with random address of the source server, that source server has to export the share in such a way that allows the destination server to mount. If the share on the random source server is open for everyone, then isn't it the same as the client writes random data from its local file to the destination file without server-to-server copy? -Dai >>> >>> Normally we only mount servers that were chosen by root. Here we'll >>> mount any random server that some client told us to. What's the worst >>> that random server can do? Do we trust our xdr decoding? Can it DOS us >>> by throwing the client's state recovery code into some loop with weird >>> error returns? Etc. >> A basic question is what is in distribution QE test suites >> that could exercise this feature? Should upstream be tasked >> with providing any missing pieces (as part of, say, pynfs, >> or nfstests)? > There are server-to-server tests in the nfstest testing suite. I'm not > sure if any of the xfstest copy_ofload exercise server to server > capability. Anna wrote the tests. > >>> Maybe it's fine. I'm OK with some level of risk. I just want to make >>> sure somebody's thought this through. >>> >>> There's also interest in allowing unprivileged NFS mounts, but I don't >>> think we've turned that on yet, partly for similar reasons. This is a >>> subset of that problem. >>> >>> --b. >> -- >> Chuck Lever >> >> >> ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 19:03 ` dai.ngo @ 2021-10-20 20:29 ` Bruce Fields 2021-10-21 5:00 ` dai.ngo 0 siblings, 1 reply; 27+ messages in thread From: Bruce Fields @ 2021-10-20 20:29 UTC (permalink / raw) To: dai.ngo Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On Wed, Oct 20, 2021 at 12:03:46PM -0700, dai.ngo@oracle.com wrote: > > On 10/20/21 9:33 AM, Olga Kornievskaia wrote: > >On Wed, Oct 20, 2021 at 12:00 PM Chuck Lever III <chuck.lever@oracle.com> wrote: > >>>2. Security question: with server-to-server copy enabled, you can send > >>>the server a COPY call with any random address, and the server will > >>>mount that address, open a file, and read from it. Is that safe? > > The client already has write access to the share on the destination > server, it can write any data to the destination file. Agreed. Please look back at what I said; I'm not thinking about attacks on the source server, I'm thinking about attacks on the destination (the one that receives the COPY). --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 20:29 ` Bruce Fields @ 2021-10-21 5:00 ` dai.ngo 2021-10-21 14:02 ` Bruce Fields 0 siblings, 1 reply; 27+ messages in thread From: dai.ngo @ 2021-10-21 5:00 UTC (permalink / raw) To: Bruce Fields Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On 10/20/21 1:29 PM, Bruce Fields wrote: > On Wed, Oct 20, 2021 at 12:03:46PM -0700, dai.ngo@oracle.com wrote: >> On 10/20/21 9:33 AM, Olga Kornievskaia wrote: >>> On Wed, Oct 20, 2021 at 12:00 PM Chuck Lever III <chuck.lever@oracle.com> wrote: >>>>> 2. Security question: with server-to-server copy enabled, you can send >>>>> the server a COPY call with any random address, and the server will >>>>> mount that address, open a file, and read from it. Is that safe? >> The client already has write access to the share on the destination >> server, it can write any data to the destination file. > Agreed. Please look back at what I said; I'm not thinking about attacks > on the source server, I'm thinking about attacks on the destination (the > one that receives the COPY). Sorry for missing you point. If I understand correctly, your concern is that a malicious client can direct the destination server to mount a malicious source server that can generate DOS attack to the destination server. The attack can come from the replies of the source server or requests from the source server to the destination server via the back channel. One of possible attack in the reply is BAD_STATEID which was handled by the client code as mentioned by Olga. Here is the list of NFS requests made from the destination to the source server: EXCHANGE_ID CREATE_SESSION RECLAIM_COMLETE SEQUENCE PUTROOTFH PUTHF GETFH GETATTR READ/READ_PLUS DESTROY_SESSION DESTROY_CLIENTID Do you think we should review all replies from these requests to make sure error replies do not cause problems for the destination server? same for the back channel ops: OP_CB_GETATTR OP_CB_RECALL OP_CB_LAYOUTRECALL OP_CB_NOTIFY OP_CB_PUSH_DELEG OP_CB_RECALL_ANY OP_CB_RECALLABLE_OBJ_AVAIL OP_CB_RECALL_SLOT OP_CB_SEQUENCE OP_CB_WANTS_CANCELLED OP_CB_NOTIFY_LOCK OP_CB_NOTIFY_DEVICEID OP_CB_OFFLOAD -Dai > --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-21 5:00 ` dai.ngo @ 2021-10-21 14:02 ` Bruce Fields 2021-10-22 6:34 ` dai.ngo 0 siblings, 1 reply; 27+ messages in thread From: Bruce Fields @ 2021-10-21 14:02 UTC (permalink / raw) To: dai.ngo Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever Thanks for the persistence: On Wed, Oct 20, 2021 at 10:00:41PM -0700, dai.ngo@oracle.com wrote: > The attack can come from the replies of the source server or requests > from the source server to the destination server via the back channel. > One of possible attack in the reply is BAD_STATEID which was handled > by the client code as mentioned by Olga. > > Here is the list of NFS requests made from the destination to the > source server: > > EXCHANGE_ID > CREATE_SESSION > RECLAIM_COMLETE > SEQUENCE > PUTROOTFH > PUTHF > GETFH > GETATTR > READ/READ_PLUS > DESTROY_SESSION > DESTROY_CLIENTID > > Do you think we should review all replies from these requests to make > sure error replies do not cause problems for the destination server? That's the exactly the sort of analysis I was curious to see, yes. (I doubt the PUTROOTFH, PUTFH, GETFH, and GETATTR are really necessary, I wonder if there's any way we could just bypass them in our case. I don't know, maybe that's more trouble than it's worth.) > same for the back channel ops: > > OP_CB_GETATTR > OP_CB_RECALL > OP_CB_LAYOUTRECALL > OP_CB_NOTIFY > OP_CB_PUSH_DELEG > OP_CB_RECALL_ANY > OP_CB_RECALLABLE_OBJ_AVAIL > OP_CB_RECALL_SLOT > OP_CB_SEQUENCE > OP_CB_WANTS_CANCELLED > OP_CB_NOTIFY_LOCK > OP_CB_NOTIFY_DEVICEID > OP_CB_OFFLOAD There shouldn't be any need for callbacks at all. We might be able to get away without even setting up a backchannel. But, yes, if the server tries to send one anyway, it'd be good to know we do something reasonable. --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-21 14:02 ` Bruce Fields @ 2021-10-22 6:34 ` dai.ngo 2021-10-22 12:58 ` Bruce Fields 2021-11-01 17:37 ` dai.ngo 0 siblings, 2 replies; 27+ messages in thread From: dai.ngo @ 2021-10-22 6:34 UTC (permalink / raw) To: Bruce Fields Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On 10/21/21 7:02 AM, Bruce Fields wrote: > Thanks for the persistence: > > On Wed, Oct 20, 2021 at 10:00:41PM -0700, dai.ngo@oracle.com wrote: >> The attack can come from the replies of the source server or requests >> from the source server to the destination server via the back channel. >> One of possible attack in the reply is BAD_STATEID which was handled >> by the client code as mentioned by Olga. >> >> Here is the list of NFS requests made from the destination to the >> source server: >> >> EXCHANGE_ID >> CREATE_SESSION >> RECLAIM_COMLETE >> SEQUENCE >> PUTROOTFH >> PUTHF >> GETFH >> GETATTR >> READ/READ_PLUS >> DESTROY_SESSION >> DESTROY_CLIENTID >> >> Do you think we should review all replies from these requests to make >> sure error replies do not cause problems for the destination server? > That's the exactly the sort of analysis I was curious to see, yes. I will go through these requests to see if is there is anything that we need to do to ensure the destination does not react negatively on the replies. > > (I doubt the PUTROOTFH, PUTFH, GETFH, and GETATTR are really necessary, > I wonder if there's any way we could just bypass them in our case. I > don't know, maybe that's more trouble than it's worth.) I'll take a look but I think we should avoid modifying the client code if possible. > >> same for the back channel ops: >> >> OP_CB_GETATTR >> OP_CB_RECALL >> OP_CB_LAYOUTRECALL >> OP_CB_NOTIFY >> OP_CB_PUSH_DELEG >> OP_CB_RECALL_ANY >> OP_CB_RECALLABLE_OBJ_AVAIL >> OP_CB_RECALL_SLOT >> OP_CB_SEQUENCE >> OP_CB_WANTS_CANCELLED >> OP_CB_NOTIFY_LOCK >> OP_CB_NOTIFY_DEVICEID >> OP_CB_OFFLOAD > There shouldn't be any need for callbacks at all. We might be able to > get away without even setting up a backchannel. But, yes, if the server > tries to send one anyway, it'd be good to know we do something > reasonable. or do not specify the back channel when creating the session somehow. I will report back. -Dai > > --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-22 6:34 ` dai.ngo @ 2021-10-22 12:58 ` Bruce Fields 2021-11-01 17:37 ` dai.ngo 1 sibling, 0 replies; 27+ messages in thread From: Bruce Fields @ 2021-10-22 12:58 UTC (permalink / raw) To: dai.ngo Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On Thu, Oct 21, 2021 at 11:34:44PM -0700, dai.ngo@oracle.com wrote: > On 10/21/21 7:02 AM, Bruce Fields wrote: > >Thanks for the persistence: > > > >On Wed, Oct 20, 2021 at 10:00:41PM -0700, dai.ngo@oracle.com wrote: > >>The attack can come from the replies of the source server or requests > >>from the source server to the destination server via the back channel. > >>One of possible attack in the reply is BAD_STATEID which was handled > >>by the client code as mentioned by Olga. > >> > >>Here is the list of NFS requests made from the destination to the > >>source server: > >> > >> EXCHANGE_ID > >> CREATE_SESSION > >> RECLAIM_COMLETE > >> SEQUENCE > >> PUTROOTFH > >> PUTHF > >> GETFH > >> GETATTR > >> READ/READ_PLUS > >> DESTROY_SESSION > >> DESTROY_CLIENTID > >> > >>Do you think we should review all replies from these requests to make > >>sure error replies do not cause problems for the destination server? > >That's the exactly the sort of analysis I was curious to see, yes. > > I will go through these requests to see if is there is anything that > we need to do to ensure the destination does not react negatively > on the replies. > > > > >(I doubt the PUTROOTFH, PUTFH, GETFH, and GETATTR are really necessary, > >I wonder if there's any way we could just bypass them in our case. I > >don't know, maybe that's more trouble than it's worth.) > > I'll take a look but I think we should avoid modifying the client > code if possible. > > > > >>same for the back channel ops: > >> > >> OP_CB_GETATTR > >> OP_CB_RECALL > >> OP_CB_LAYOUTRECALL > >> OP_CB_NOTIFY > >> OP_CB_PUSH_DELEG > >> OP_CB_RECALL_ANY > >> OP_CB_RECALLABLE_OBJ_AVAIL > >> OP_CB_RECALL_SLOT > >> OP_CB_SEQUENCE > >> OP_CB_WANTS_CANCELLED > >> OP_CB_NOTIFY_LOCK > >> OP_CB_NOTIFY_DEVICEID > >> OP_CB_OFFLOAD > >There shouldn't be any need for callbacks at all. We might be able to > >get away without even setting up a backchannel. But, yes, if the server > >tries to send one anyway, it'd be good to know we do something > >reasonable. > > or do not specify the back channel when creating the session somehow. > I will report back. Thank you, Dai! --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-22 6:34 ` dai.ngo 2021-10-22 12:58 ` Bruce Fields @ 2021-11-01 17:37 ` dai.ngo 2021-11-01 19:33 ` Bruce Fields 1 sibling, 1 reply; 27+ messages in thread From: dai.ngo @ 2021-11-01 17:37 UTC (permalink / raw) To: Bruce Fields Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On 10/21/21 11:34 PM, dai.ngo@oracle.com wrote: > On 10/21/21 7:02 AM, Bruce Fields wrote: >> On Wed, Oct 20, 2021 at 10:00:41PM -0700, dai.ngo@oracle.com wrote: >>> The attack can come from the replies of the source server or requests >>> from the source server to the destination server via the back channel. >>> One of possible attack in the reply is BAD_STATEID which was handled >>> by the client code as mentioned by Olga. >>> >>> Here is the list of NFS requests made from the destination to the >>> source server: >>> >>> EXCHANGE_ID >>> CREATE_SESSION >>> RECLAIM_COMLETE >>> SEQUENCE >>> PUTROOTFH >>> PUTHF >>> GETFH >>> GETATTR >>> READ/READ_PLUS >>> DESTROY_SESSION >>> DESTROY_CLIENTID >>> >>> Do you think we should review all replies from these requests to make >>> sure error replies do not cause problems for the destination server? >> That's the exactly the sort of analysis I was curious to see, yes. > > I will go through these requests to see if is there is anything that > we need to do to ensure the destination does not react negatively > on the replies. still need to be done. > >> >> (I doubt the PUTROOTFH, PUTFH, GETFH, and GETATTR are really necessary, >> I wonder if there's any way we could just bypass them in our case. I >> don't know, maybe that's more trouble than it's worth.) > > I'll take a look but I think we should avoid modifying the client > code if possible. > >> >>> same for the back channel ops: >>> >>> OP_CB_GETATTR >>> OP_CB_RECALL >>> OP_CB_LAYOUTRECALL >>> OP_CB_NOTIFY >>> OP_CB_PUSH_DELEG >>> OP_CB_RECALL_ANY >>> OP_CB_RECALLABLE_OBJ_AVAIL >>> OP_CB_RECALL_SLOT >>> OP_CB_SEQUENCE >>> OP_CB_WANTS_CANCELLED >>> OP_CB_NOTIFY_LOCK >>> OP_CB_NOTIFY_DEVICEID >>> OP_CB_OFFLOAD >> There shouldn't be any need for callbacks at all. We might be able to >> get away without even setting up a backchannel. But, yes, if the server >> tries to send one anyway, it'd be good to know we do something >> reasonable. > > or do not specify the back channel when creating the session somehow. > I will report back. We can not disable the back channel of the SSC v4.2 mount since it might share the same connection with a regular NFSv4.2 mount from the destination to the source server. We need to be able to identify whether the back channel request is for the regular mount or the SSC mount and if it's for the SSC mount then drop the request. To differentiate back channel request for SSC vs regular mount I plan to do the following: Mark the nfs_server of the SSC mount with with a flag (NFS_MOUNT_SSC) When a back channel request comes in, we check all the nfs_server's that share the same nfs_client based on the clientid in the request. If there is one or more nfs_server's sharing the same nfs_client and none of them is marked as NFS_MOUNT_SSC then we allow the request to be processed as normal (non-SSC case). If there are multiple nfs_server's and one of then is marked as NFS_MOUNT_SSC then we allow the request to be processed. This is because if there is a regular mount from destination to source server that means the source server is already trusted by the destination's admin. If there is only one nfs_server and it's marked as NFS_MOUNT_SSC then we drop that request. Do see any problem with this approach or you have any suggestion on how to handle this? Thanks, -Dai ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-11-01 17:37 ` dai.ngo @ 2021-11-01 19:33 ` Bruce Fields 2021-11-01 19:55 ` dai.ngo 0 siblings, 1 reply; 27+ messages in thread From: Bruce Fields @ 2021-11-01 19:33 UTC (permalink / raw) To: dai.ngo Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On Mon, Nov 01, 2021 at 10:37:11AM -0700, dai.ngo@oracle.com wrote: > > On 10/21/21 11:34 PM, dai.ngo@oracle.com wrote: > >On 10/21/21 7:02 AM, Bruce Fields wrote: > >>On Wed, Oct 20, 2021 at 10:00:41PM -0700, dai.ngo@oracle.com wrote: > >>>The attack can come from the replies of the source server or requests > >>>from the source server to the destination server via the back channel. > >>>One of possible attack in the reply is BAD_STATEID which was handled > >>>by the client code as mentioned by Olga. > >>> > >>>Here is the list of NFS requests made from the destination to the > >>>source server: > >>> > >>> EXCHANGE_ID > >>> CREATE_SESSION > >>> RECLAIM_COMLETE > >>> SEQUENCE > >>> PUTROOTFH > >>> PUTHF > >>> GETFH > >>> GETATTR > >>> READ/READ_PLUS > >>> DESTROY_SESSION > >>> DESTROY_CLIENTID > >>> > >>>Do you think we should review all replies from these requests to make > >>>sure error replies do not cause problems for the destination server? > >>That's the exactly the sort of analysis I was curious to see, yes. > > > >I will go through these requests to see if is there is anything that > >we need to do to ensure the destination does not react negatively > >on the replies. > > still need to be done. > > > > >> > >>(I doubt the PUTROOTFH, PUTFH, GETFH, and GETATTR are really necessary, > >>I wonder if there's any way we could just bypass them in our case. I > >>don't know, maybe that's more trouble than it's worth.) > > > >I'll take a look but I think we should avoid modifying the client > >code if possible. > > > >> > >>>same for the back channel ops: > >>> > >>> OP_CB_GETATTR > >>> OP_CB_RECALL > >>> OP_CB_LAYOUTRECALL > >>> OP_CB_NOTIFY > >>> OP_CB_PUSH_DELEG > >>> OP_CB_RECALL_ANY > >>> OP_CB_RECALLABLE_OBJ_AVAIL > >>> OP_CB_RECALL_SLOT > >>> OP_CB_SEQUENCE > >>> OP_CB_WANTS_CANCELLED > >>> OP_CB_NOTIFY_LOCK > >>> OP_CB_NOTIFY_DEVICEID > >>> OP_CB_OFFLOAD > >>There shouldn't be any need for callbacks at all. We might be able to > >>get away without even setting up a backchannel. But, yes, if the server > >>tries to send one anyway, it'd be good to know we do something > >>reasonable. > > > >or do not specify the back channel when creating the session somehow. > >I will report back. > > We can not disable the back channel of the SSC v4.2 mount since it might > share the same connection with a regular NFSv4.2 mount from the destination > to the source server. Hm. Well, now that I think of it, a backchannel is probably required for the SSC case anyway. (I think CB_RECALL_SLOT is mandatory to support?) --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-11-01 19:33 ` Bruce Fields @ 2021-11-01 19:55 ` dai.ngo 0 siblings, 0 replies; 27+ messages in thread From: dai.ngo @ 2021-11-01 19:55 UTC (permalink / raw) To: Bruce Fields Cc: Linux NFS Mailing List, Steve Dickson, Olga Kornievskaia, Chuck Lever On 11/1/21 12:33 PM, Bruce Fields wrote: > On Mon, Nov 01, 2021 at 10:37:11AM -0700, dai.ngo@oracle.com wrote: >> On 10/21/21 11:34 PM, dai.ngo@oracle.com wrote: >>> On 10/21/21 7:02 AM, Bruce Fields wrote: >>>> On Wed, Oct 20, 2021 at 10:00:41PM -0700, dai.ngo@oracle.com wrote: >>>>> The attack can come from the replies of the source server or requests >>>> >from the source server to the destination server via the back channel. >>>>> One of possible attack in the reply is BAD_STATEID which was handled >>>>> by the client code as mentioned by Olga. >>>>> >>>>> Here is the list of NFS requests made from the destination to the >>>>> source server: >>>>> >>>>> EXCHANGE_ID >>>>> CREATE_SESSION >>>>> RECLAIM_COMLETE >>>>> SEQUENCE >>>>> PUTROOTFH >>>>> PUTHF >>>>> GETFH >>>>> GETATTR >>>>> READ/READ_PLUS >>>>> DESTROY_SESSION >>>>> DESTROY_CLIENTID >>>>> >>>>> Do you think we should review all replies from these requests to make >>>>> sure error replies do not cause problems for the destination server? >>>> That's the exactly the sort of analysis I was curious to see, yes. >>> I will go through these requests to see if is there is anything that >>> we need to do to ensure the destination does not react negatively >>> on the replies. >> still need to be done. >> >>>> (I doubt the PUTROOTFH, PUTFH, GETFH, and GETATTR are really necessary, >>>> I wonder if there's any way we could just bypass them in our case. I >>>> don't know, maybe that's more trouble than it's worth.) >>> I'll take a look but I think we should avoid modifying the client >>> code if possible. >>> >>>>> same for the back channel ops: >>>>> >>>>> OP_CB_GETATTR >>>>> OP_CB_RECALL >>>>> OP_CB_LAYOUTRECALL >>>>> OP_CB_NOTIFY >>>>> OP_CB_PUSH_DELEG >>>>> OP_CB_RECALL_ANY >>>>> OP_CB_RECALLABLE_OBJ_AVAIL >>>>> OP_CB_RECALL_SLOT >>>>> OP_CB_SEQUENCE >>>>> OP_CB_WANTS_CANCELLED >>>>> OP_CB_NOTIFY_LOCK >>>>> OP_CB_NOTIFY_DEVICEID >>>>> OP_CB_OFFLOAD >>>> There shouldn't be any need for callbacks at all. We might be able to >>>> get away without even setting up a backchannel. But, yes, if the server >>>> tries to send one anyway, it'd be good to know we do something >>>> reasonable. >>> or do not specify the back channel when creating the session somehow. >>> I will report back. >> We can not disable the back channel of the SSC v4.2 mount since it might >> share the same connection with a regular NFSv4.2 mount from the destination >> to the source server. > Hm. > > Well, now that I think of it, a backchannel is probably required for the > SSC case anyway. (I think CB_RECALL_SLOT is mandatory to support?) I think the back channel is not required. From Sec 13 of RFC 7862: The REQUIRED or OPTIONAL designation for callback operations sent by the server is for both the client and server. Generally, the client has the option of creating the backchannel and sending the operations on the forechannel that will be a catalyst for the server sending callback operations. A partial exception is CB_RECALL_SLOT; the only way the client can avoid supporting this operation is by not creating a backchannel. I have not found a clean/simple way to not creating the back channel for the SSC mount. The reasons is that the SSC mount can be using an existing connection of a regular mount to the source server, or a regular mount can happen after the SSC mount. It seems that we should support back channel for SSC mount but do a due diligent check to make sure the requests come from a trusted source and also limit the number of ops supported on the SSC back channel. -Dai > > --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 16:00 ` Chuck Lever III 2021-10-20 16:33 ` Olga Kornievskaia @ 2021-10-20 17:24 ` Steve Dickson 2021-10-20 17:51 ` Chuck Lever III 1 sibling, 1 reply; 27+ messages in thread From: Steve Dickson @ 2021-10-20 17:24 UTC (permalink / raw) To: Chuck Lever III, Bruce Fields Cc: Linux NFS Mailing List, Dai Ngo, Olga Kornievskaia Hey! On 10/20/21 12:00, Chuck Lever III wrote: > > >> On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: >> >> knfsd has supported server-to-server copy for a couple years (since >> 5.5). You have set a module parameter to enable it. I'm getting asked >> when we could turn that parameter on by default. >> >> I've got a couple vague criteria: one just general maturity, the other a >> security question: >> >> 1. General maturity: the only reports I recall seeing are from testers. >> Is anyone using this? Does it work for them? Do they find a benefit? >> Maybe we could turn it on by default in one distro (Fedora?) and promote >> it a little and see what that turns up? > > I like the idea of enabling it in one of the technology > preview distributions. My thoughts on this... we can do this one of two ways. either do a kernel patch to enable inter_copy_offload_enable by default or I could have nfs-utils drop a nfsd.conf file in /etc/modprobe.d/ enabling it. The kernel patch is more of a commitment but the nfs-utils change is easier to back out. > > But wrt the maturity question, is the work finished? Or, > perhaps a better question is do we have a minimum viable > product here that can be enabled, or is more work needed > to meet even that bar? I've been testing it and it seems to be pretty solid. Question, Olga mentioned Dia did a patch that eliminations the (rsize*14) file size limit... Meaning the file has to be greater that (rsize*14) for the SSC to happen. Was that patch committed? I have not looked that hard but I haven't found it... > > One thing that I recall is missing is support for Kerberos > in the server-to-server copy operation. Is that in plan, > or deemed unimportant? Personally I think we should make sure the technology is stable before adding things on to it.. IMHO. > > >> 2. Security question: with server-to-server copy enabled, you can send >> the server a COPY call with any random address, and the server will >> mount that address, open a file, and read from it. Is that safe? >> >> Normally we only mount servers that were chosen by root. Here we'll >> mount any random server that some client told us to. What's the worst >> that random server can do? Do we trust our xdr decoding? Can it DOS us >> by throwing the client's state recovery code into some loop with weird >> error returns? Etc. > > A basic question is what is in distribution QE test suites > that could exercise this feature? Should upstream be tasked > with providing any missing pieces (as part of, say, pynfs, > or nfstests)? As Olga pointed out... nfstests already has a test... my two cents, steved. > > >> Maybe it's fine. I'm OK with some level of risk. I just want to make >> sure somebody's thought this through. >> >> There's also interest in allowing unprivileged NFS mounts, but I don't >> think we've turned that on yet, partly for similar reasons. This is a >> subset of that problem. >> >> --b. > > -- > Chuck Lever > > > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 17:24 ` Steve Dickson @ 2021-10-20 17:51 ` Chuck Lever III 0 siblings, 0 replies; 27+ messages in thread From: Chuck Lever III @ 2021-10-20 17:51 UTC (permalink / raw) To: Steve Dickson Cc: Bruce Fields, Linux NFS Mailing List, Dai Ngo, Olga Kornievskaia > On Oct 20, 2021, at 1:24 PM, Steve Dickson <steved@redhat.com> wrote: > > Hey! > > On 10/20/21 12:00, Chuck Lever III wrote: >>> On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: >>> >>> knfsd has supported server-to-server copy for a couple years (since >>> 5.5). You have set a module parameter to enable it. I'm getting asked >>> when we could turn that parameter on by default. >>> >>> I've got a couple vague criteria: one just general maturity, the other a >>> security question: >>> >>> 1. General maturity: the only reports I recall seeing are from testers. >>> Is anyone using this? Does it work for them? Do they find a benefit? >>> Maybe we could turn it on by default in one distro (Fedora?) and promote >>> it a little and see what that turns up? > >> But wrt the maturity question, is the work finished? Or, >> perhaps a better question is do we have a minimum viable >> product here that can be enabled, or is more work needed >> to meet even that bar? > I've been testing it and it seems to be pretty solid. Well I was more interested in knowing if the capability is in a state where it is now useful to users rather than just being a prototype for a new protocol element. Along with having tests that QE teams can run, we probably also need to know whether there is adequate documentation. -- Chuck Lever ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 15:54 server-to-server copy by default J. Bruce Fields 2021-10-20 16:00 ` Chuck Lever III @ 2021-10-20 16:37 ` Olga Kornievskaia 2021-10-20 17:45 ` Chuck Lever III 2021-10-20 18:00 ` J. Bruce Fields 2021-11-01 18:22 ` Charles Hedrick 2 siblings, 2 replies; 27+ messages in thread From: Olga Kornievskaia @ 2021-10-20 16:37 UTC (permalink / raw) To: J. Bruce Fields; +Cc: linux-nfs, Dai Ngo, Steve Dickson, Chuck Lever On Wed, Oct 20, 2021 at 11:54 AM J. Bruce Fields <bfields@fieldses.org> wrote: > > knfsd has supported server-to-server copy for a couple years (since > 5.5). You have set a module parameter to enable it. I'm getting asked > when we could turn that parameter on by default. > > I've got a couple vague criteria: one just general maturity, the other a > security question: > > 1. General maturity: the only reports I recall seeing are from testers. > Is anyone using this? Does it work for them? Do they find a benefit? > Maybe we could turn it on by default in one distro (Fedora?) and promote > it a little and see what that turns up? > > 2. Security question: with server-to-server copy enabled, you can send > the server a COPY call with any random address, and the server will > mount that address, open a file, and read from it. Is that safe? How about adding a piece then on the server (a policy) that would only control that? The concept behind the server-to-server was that servers might have a private/fast network between them that they would want to utilize. A more restrictive policy could be to only allow predefined network space to do the COPY? I know that more work. But sound like perhaps it might be something that provides more control to the server. But as Chuck pointed out perhaps the kerberos piece would make this concern irrelevant. > Normally we only mount servers that were chosen by root. Here we'll > mount any random server that some client told us to. What's the worst > that random server can do? Do we trust our xdr decoding? Can it DOS us > by throwing the client's state recovery code into some loop with weird > error returns? Etc. Client code has been modified to know about special copy stateids that if the client gets BAD_STATEID it knows not to try to do recovery and instead it errors back to the "application", it being nfsd. > Maybe it's fine. I'm OK with some level of risk. I just want to make > sure somebody's thought this through. > > There's also interest in allowing unprivileged NFS mounts, but I don't > think we've turned that on yet, partly for similar reasons. This is a > subset of that problem. > > --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 16:37 ` Olga Kornievskaia @ 2021-10-20 17:45 ` Chuck Lever III 2021-10-20 18:15 ` Bruce Fields 2021-10-20 18:00 ` J. Bruce Fields 1 sibling, 1 reply; 27+ messages in thread From: Chuck Lever III @ 2021-10-20 17:45 UTC (permalink / raw) To: Olga Kornievskaia Cc: Bruce Fields, Linux NFS Mailing List, Dai Ngo, Steve Dickson > On Oct 20, 2021, at 12:37 PM, Olga Kornievskaia <olga.kornievskaia@gmail.com> wrote: > > On Wed, Oct 20, 2021 at 11:54 AM J. Bruce Fields <bfields@fieldses.org> wrote: >> >> knfsd has supported server-to-server copy for a couple years (since >> 5.5). You have set a module parameter to enable it. I'm getting asked >> when we could turn that parameter on by default. >> >> I've got a couple vague criteria: one just general maturity, the other a >> security question: >> >> 1. General maturity: the only reports I recall seeing are from testers. >> Is anyone using this? Does it work for them? Do they find a benefit? >> Maybe we could turn it on by default in one distro (Fedora?) and promote >> it a little and see what that turns up? >> >> 2. Security question: with server-to-server copy enabled, you can send >> the server a COPY call with any random address, and the server will >> mount that address, open a file, and read from it. Is that safe? > > How about adding a piece then on the server (a policy) that would only > control that? The concept behind the server-to-server was that servers > might have a private/fast network between them that they would want to > utilize. A more restrictive policy could be to only allow predefined > network space to do the COPY? I know that more work. But sound like > perhaps it might be something that provides more control to the > server. > > But as Chuck pointed out perhaps the kerberos piece would make this > concern irrelevant. I like the idea of having a server-side policy setting that controls whether s2sc is permitted, and maybe establishes a range of IP addresses allowed to be destination servers. -- Chuck Lever ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 17:45 ` Chuck Lever III @ 2021-10-20 18:15 ` Bruce Fields 2021-10-20 19:04 ` Chuck Lever III 0 siblings, 1 reply; 27+ messages in thread From: Bruce Fields @ 2021-10-20 18:15 UTC (permalink / raw) To: Chuck Lever III Cc: Olga Kornievskaia, Linux NFS Mailing List, Dai Ngo, Steve Dickson On Wed, Oct 20, 2021 at 05:45:58PM +0000, Chuck Lever III wrote: > > On Oct 20, 2021, at 12:37 PM, Olga Kornievskaia <olga.kornievskaia@gmail.com> wrote: > > > > On Wed, Oct 20, 2021 at 11:54 AM J. Bruce Fields <bfields@fieldses.org> wrote: > >> > >> knfsd has supported server-to-server copy for a couple years (since > >> 5.5). You have set a module parameter to enable it. I'm getting asked > >> when we could turn that parameter on by default. > >> > >> I've got a couple vague criteria: one just general maturity, the other a > >> security question: > >> > >> 1. General maturity: the only reports I recall seeing are from testers. > >> Is anyone using this? Does it work for them? Do they find a benefit? > >> Maybe we could turn it on by default in one distro (Fedora?) and promote > >> it a little and see what that turns up? > >> > >> 2. Security question: with server-to-server copy enabled, you can send > >> the server a COPY call with any random address, and the server will > >> mount that address, open a file, and read from it. Is that safe? > > > > How about adding a piece then on the server (a policy) that would only > > control that? The concept behind the server-to-server was that servers > > might have a private/fast network between them that they would want to > > utilize. A more restrictive policy could be to only allow predefined > > network space to do the COPY? I know that more work. But sound like > > perhaps it might be something that provides more control to the > > server. > > > > But as Chuck pointed out perhaps the kerberos piece would make this > > concern irrelevant. > > I like the idea of having a server-side policy setting that > controls whether s2sc is permitted, and maybe establishes a > range of IP addresses allowed to be destination servers. Maybe, but: 1) Couldn't you get something awfully close to that with firewall configuration? 2) I'm getting asked why server-side copy isn't on by default. So I guess the requirement to set inter_copy_offload_enable is too much. How does requiring more complicated configuration answer that concern? 3) There's interest in allowing unprivileged NFS mounts. That's more of a security risk than this. What's the client maintainers' judgement about unprivileged NFS mounts? Do they think that would be safe to allow by default in distros? If so, then we're certainly fine here. --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 18:15 ` Bruce Fields @ 2021-10-20 19:04 ` Chuck Lever III 2021-10-21 13:43 ` Steve Dickson ` (2 more replies) 0 siblings, 3 replies; 27+ messages in thread From: Chuck Lever III @ 2021-10-20 19:04 UTC (permalink / raw) To: Bruce Fields Cc: Olga Kornievskaia, Linux NFS Mailing List, Dai Ngo, Steve Dickson > On Oct 20, 2021, at 2:15 PM, Bruce Fields <bfields@fieldses.org> wrote: > > On Wed, Oct 20, 2021 at 05:45:58PM +0000, Chuck Lever III wrote: >>> On Oct 20, 2021, at 12:37 PM, Olga Kornievskaia <olga.kornievskaia@gmail.com> wrote: >>> >>> On Wed, Oct 20, 2021 at 11:54 AM J. Bruce Fields <bfields@fieldses.org> wrote: >>>> >>>> knfsd has supported server-to-server copy for a couple years (since >>>> 5.5). You have set a module parameter to enable it. I'm getting asked >>>> when we could turn that parameter on by default. >>>> >>>> I've got a couple vague criteria: one just general maturity, the other a >>>> security question: >>>> >>>> 1. General maturity: the only reports I recall seeing are from testers. >>>> Is anyone using this? Does it work for them? Do they find a benefit? >>>> Maybe we could turn it on by default in one distro (Fedora?) and promote >>>> it a little and see what that turns up? >>>> >>>> 2. Security question: with server-to-server copy enabled, you can send >>>> the server a COPY call with any random address, and the server will >>>> mount that address, open a file, and read from it. Is that safe? >>> >>> How about adding a piece then on the server (a policy) that would only >>> control that? The concept behind the server-to-server was that servers >>> might have a private/fast network between them that they would want to >>> utilize. A more restrictive policy could be to only allow predefined >>> network space to do the COPY? I know that more work. But sound like >>> perhaps it might be something that provides more control to the >>> server. >>> >>> But as Chuck pointed out perhaps the kerberos piece would make this >>> concern irrelevant. >> >> I like the idea of having a server-side policy setting that >> controls whether s2sc is permitted, and maybe establishes a >> range of IP addresses allowed to be destination servers. > > Maybe, but: > > 1) Couldn't you get something awfully close to that with > firewall configuration? Not if the s2sc policy setting is on each export. > 2) I'm getting asked why server-side copy isn't on by default. And your answer to that was "we haven't figured out how to guarantee security when it's enabled". > So I guess the requirement to set inter_copy_offload_enable is > too much. How does requiring more complicated configuration > answer that concern? It answers the concern by letting local administrators choose to enable or disable s2sc based on their own security needs. > 3) There's interest in allowing unprivileged NFS mounts. That's > more of a security risk than this. What's the client > maintainers' judgement about unprivileged NFS mounts? Do they > think that would be safe to allow by default in distros? If so, > then we're certainly fine here. Unprivileged mounting seems like a different question to me. Related, possibly, but not the same. I'd rather leave that discussion to another thread. -- Chuck Lever ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 19:04 ` Chuck Lever III @ 2021-10-21 13:43 ` Steve Dickson 2021-10-21 13:56 ` Bruce Fields 2021-10-21 14:13 ` Bruce Fields 2 siblings, 0 replies; 27+ messages in thread From: Steve Dickson @ 2021-10-21 13:43 UTC (permalink / raw) To: Chuck Lever III, Bruce Fields Cc: Olga Kornievskaia, Linux NFS Mailing List, Dai Ngo On 10/20/21 15:04, Chuck Lever III wrote: > > >> On Oct 20, 2021, at 2:15 PM, Bruce Fields <bfields@fieldses.org> wrote: >> >> On Wed, Oct 20, 2021 at 05:45:58PM +0000, Chuck Lever III wrote: >>>> On Oct 20, 2021, at 12:37 PM, Olga Kornievskaia <olga.kornievskaia@gmail.com> wrote: >>>> >>>> On Wed, Oct 20, 2021 at 11:54 AM J. Bruce Fields <bfields@fieldses.org> wrote: >>>>> >>>>> knfsd has supported server-to-server copy for a couple years (since >>>>> 5.5). You have set a module parameter to enable it. I'm getting asked >>>>> when we could turn that parameter on by default. >>>>> >>>>> I've got a couple vague criteria: one just general maturity, the other a >>>>> security question: >>>>> >>>>> 1. General maturity: the only reports I recall seeing are from testers. >>>>> Is anyone using this? Does it work for them? Do they find a benefit? >>>>> Maybe we could turn it on by default in one distro (Fedora?) and promote >>>>> it a little and see what that turns up? >>>>> >>>>> 2. Security question: with server-to-server copy enabled, you can send >>>>> the server a COPY call with any random address, and the server will >>>>> mount that address, open a file, and read from it. Is that safe? >>>> >>>> How about adding a piece then on the server (a policy) that would only >>>> control that? The concept behind the server-to-server was that servers >>>> might have a private/fast network between them that they would want to >>>> utilize. A more restrictive policy could be to only allow predefined >>>> network space to do the COPY? I know that more work. But sound like >>>> perhaps it might be something that provides more control to the >>>> server. >>>> >>>> But as Chuck pointed out perhaps the kerberos piece would make this >>>> concern irrelevant. >>> >>> I like the idea of having a server-side policy setting that >>> controls whether s2sc is permitted, and maybe establishes a >>> range of IP addresses allowed to be destination servers. >> >> Maybe, but: >> >> 1) Couldn't you get something awfully close to that with >> firewall configuration? > > Not if the s2sc policy setting is on each export. Is this level complication really necessary... I just don't see why people would not want to make copies on all exports faster. Is not having this option a showstopper to enabling it? > > >> 2) I'm getting asked why server-side copy isn't on by default. > > And your answer to that was "we haven't figured out how to > guarantee security when it's enabled". I'm thinking the servers will be behind a firewall which by definition makes them secure. Now if there is a malicious app throwing COPY calls with rouge address behind the firewall is that something we really need to protect from? The network has already been compromised. As Olga pointed out... clustered servers will have a will have a very fast connection between them which is something we should take advantage of... IMHO steved. > > >> So I guess the requirement to set inter_copy_offload_enable is >> too much. How does requiring more complicated configuration >> answer that concern? > > It answers the concern by letting local administrators choose > to enable or disable s2sc based on their own security needs. > > >> 3) There's interest in allowing unprivileged NFS mounts. That's >> more of a security risk than this. What's the client >> maintainers' judgement about unprivileged NFS mounts? Do they >> think that would be safe to allow by default in distros? If so, >> then we're certainly fine here. > > Unprivileged mounting seems like a different question to me. > Related, possibly, but not the same. I'd rather leave that > discussion to another thread. > > > -- > Chuck Lever > > > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 19:04 ` Chuck Lever III 2021-10-21 13:43 ` Steve Dickson @ 2021-10-21 13:56 ` Bruce Fields 2021-10-21 14:13 ` Bruce Fields 2 siblings, 0 replies; 27+ messages in thread From: Bruce Fields @ 2021-10-21 13:56 UTC (permalink / raw) To: Chuck Lever III Cc: Olga Kornievskaia, Linux NFS Mailing List, Dai Ngo, Steve Dickson On Wed, Oct 20, 2021 at 07:04:53PM +0000, Chuck Lever III wrote: > Not if the s2sc policy setting is on each export. I don't want anyone investing time in writing code to enable this configuration for the kernel and mountd and exportfs, documenting it, explaining it to users repeatedly, etc., all for a problem we're like to eventually declare not a problem. --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 19:04 ` Chuck Lever III 2021-10-21 13:43 ` Steve Dickson 2021-10-21 13:56 ` Bruce Fields @ 2021-10-21 14:13 ` Bruce Fields 2021-10-21 14:22 ` Trond Myklebust 2 siblings, 1 reply; 27+ messages in thread From: Bruce Fields @ 2021-10-21 14:13 UTC (permalink / raw) To: Trond Myklebust, Anna Schumaker Cc: Chuck Lever III, Olga Kornievskaia, Linux NFS Mailing List, Dai Ngo, Steve Dickson On Wed, Oct 20, 2021 at 07:04:53PM +0000, Chuck Lever III wrote: > Unprivileged mounting seems like a different question to me. > Related, possibly, but not the same. I'd rather leave that > discussion to another thread. Well, I'd be curious if client maintainers have any thoughts. The NFS client still disallows unprivileged mounts, right? Is it something you think could be supported, and if so, do you have an idea what's left to do? Trond, I remember asking you about unprivileged mounts at a bakeathon a few years ago, and at the time you seemed to think it'd be a reasonable thing to do eventually, and the one obstacle you mentioned was that the client wasn't capable of maintaining separate state in different namespaces. That's fixed, isn't it? --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-21 14:13 ` Bruce Fields @ 2021-10-21 14:22 ` Trond Myklebust 2021-10-21 14:38 ` bfields 0 siblings, 1 reply; 27+ messages in thread From: Trond Myklebust @ 2021-10-21 14:22 UTC (permalink / raw) To: bfields, schumakeranna Cc: linux-nfs, dai.ngo, steved, olga.kornievskaia, chuck.lever On Thu, 2021-10-21 at 10:13 -0400, Bruce Fields wrote: > On Wed, Oct 20, 2021 at 07:04:53PM +0000, Chuck Lever III wrote: > > Unprivileged mounting seems like a different question to me. > > Related, possibly, but not the same. I'd rather leave that > > discussion to another thread. > > Well, I'd be curious if client maintainers have any thoughts. > > The NFS client still disallows unprivileged mounts, right? Is it > something you think could be supported, and if so, do you have an > idea > what's left to do? > > Trond, I remember asking you about unprivileged mounts at a bakeathon > a > few years ago, and at the time you seemed to think it'd be a > reasonable > thing to do eventually, and the one obstacle you mentioned was that > the > client wasn't capable of maintaining separate state in different > namespaces. That's fixed, isn't it? > Yes, that's mostly fixed. As far as I'm concerned, there should be no major obstacles to allowing unprivileged mounts in their own private net namespace. The one thing to note, though, is that AUTH_SYS still required that the container be given a CAP_NET_BIND_SERVICE privilege to allow binding to a privileged port. -- Trond Myklebust Linux NFS client maintainer, Hammerspace trond.myklebust@hammerspace.com ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-21 14:22 ` Trond Myklebust @ 2021-10-21 14:38 ` bfields 0 siblings, 0 replies; 27+ messages in thread From: bfields @ 2021-10-21 14:38 UTC (permalink / raw) To: Trond Myklebust Cc: schumakeranna, linux-nfs, dai.ngo, steved, olga.kornievskaia, chuck.lever On Thu, Oct 21, 2021 at 02:22:13PM +0000, Trond Myklebust wrote: > Yes, that's mostly fixed. As far as I'm concerned, there should be no > major obstacles to allowing unprivileged mounts in their own private > net namespace. Do you think it'd be a reasonable thing to turn on now by default in distros or something the admin should have to opt-in to only on trusted networks? I'm wondering how much confidence we have in the client's robustness in the face of possibly compromised servers. > The one thing to note, though, is that AUTH_SYS still required that the > container be given a CAP_NET_BIND_SERVICE privilege to allow binding to > a privileged port. Got it, thanks. --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 16:37 ` Olga Kornievskaia 2021-10-20 17:45 ` Chuck Lever III @ 2021-10-20 18:00 ` J. Bruce Fields 1 sibling, 0 replies; 27+ messages in thread From: J. Bruce Fields @ 2021-10-20 18:00 UTC (permalink / raw) To: Olga Kornievskaia; +Cc: linux-nfs, Dai Ngo, Steve Dickson, Chuck Lever On Wed, Oct 20, 2021 at 12:37:08PM -0400, Olga Kornievskaia wrote: > On Wed, Oct 20, 2021 at 11:54 AM J. Bruce Fields <bfields@fieldses.org> wrote: > > > > knfsd has supported server-to-server copy for a couple years (since > > 5.5). You have set a module parameter to enable it. I'm getting asked > > when we could turn that parameter on by default. > > > > I've got a couple vague criteria: one just general maturity, the other a > > security question: > > > > 1. General maturity: the only reports I recall seeing are from testers. > > Is anyone using this? Does it work for them? Do they find a benefit? > > Maybe we could turn it on by default in one distro (Fedora?) and promote > > it a little and see what that turns up? > > > > 2. Security question: with server-to-server copy enabled, you can send > > the server a COPY call with any random address, and the server will > > mount that address, open a file, and read from it. Is that safe? (Whoops, I forgot, there's no open, just reads. And I don't know how much actual protocol there is involved in the mount.) > How about adding a piece then on the server (a policy) that would only > control that? The concept behind the server-to-server was that servers > might have a private/fast network between them that they would want to > utilize. A more restrictive policy could be to only allow predefined > network space to do the COPY? I know that more work. But sound like > perhaps it might be something that provides more control to the > server. That sounds like a step backwards if you're trying to enable it by default. But in the case there's a special server-to-server network, the way to handle that is by configuring the source server to return addresses on that network in the cnr_source_server field of the COPY_NOTIFY reply, right? > But as Chuck pointed out perhaps the kerberos piece would make this > concern irrelevant. I don't think kerberos addresses this. (It may make increase the attack surface, in fact.) > > Normally we only mount servers that were chosen by root. Here we'll > > mount any random server that some client told us to. What's the worst > > that random server can do? Do we trust our xdr decoding? Can it DOS us > > by throwing the client's state recovery code into some loop with weird > > error returns? Etc. > > Client code has been modified to know about special copy stateids that > if the client gets BAD_STATEID it knows not to try to do recovery and > instead it errors back to the "application", it being nfsd. Good to know, thanks. What are the list of rpc calls that are made to the source server--is it just READ, or does it at least need to create a client and a session? Are there any other error handling paths that we wouldn't want to go down? Etc. --b. > > Maybe it's fine. I'm OK with some level of risk. I just want to make > > sure somebody's thought this through. > > > > There's also interest in allowing unprivileged NFS mounts, but I don't > > think we've turned that on yet, partly for similar reasons. This is a > > subset of that problem. > > > > --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-10-20 15:54 server-to-server copy by default J. Bruce Fields 2021-10-20 16:00 ` Chuck Lever III 2021-10-20 16:37 ` Olga Kornievskaia @ 2021-11-01 18:22 ` Charles Hedrick 2021-11-01 19:25 ` Steve Dickson 2 siblings, 1 reply; 27+ messages in thread From: Charles Hedrick @ 2021-11-01 18:22 UTC (permalink / raw) To: J. Bruce Fields Cc: Linux NFS Mailing List, dai.ngo, Olga Kornievskaia, Steve Dickson, Chuck Lever I am in general concerned about turning on new features before basic ones work reliably. We’ve had enough different failures that we’ve backup up to NFS 3 for file systems with heavy use. We first tried turning off delegation. That helped a lot. But we just ran into a two different machine hung trying to lock Chome’s profile. (I sent a bit of information on that one previously.) We had to restart NFS on the server to fix it, and that caused us to lose a bunch of VMs. (That shouldn’t have happened. It looks like ESX misbehaved.) If I could turn off NFS4 locking I would. > On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: > > knfsd has supported server-to-server copy for a couple years (since > 5.5). You have set a module parameter to enable it. I'm getting asked > when we could turn that parameter on by default. > > I've got a couple vague criteria: one just general maturity, the other a > security question: > > 1. General maturity: the only reports I recall seeing are from testers. > Is anyone using this? Does it work for them? Do they find a benefit? > Maybe we could turn it on by default in one distro (Fedora?) and promote > it a little and see what that turns up? > > 2. Security question: with server-to-server copy enabled, you can send > the server a COPY call with any random address, and the server will > mount that address, open a file, and read from it. Is that safe? > > Normally we only mount servers that were chosen by root. Here we'll > mount any random server that some client told us to. What's the worst > that random server can do? Do we trust our xdr decoding? Can it DOS us > by throwing the client's state recovery code into some loop with weird > error returns? Etc. > > Maybe it's fine. I'm OK with some level of risk. I just want to make > sure somebody's thought this through. > > There's also interest in allowing unprivileged NFS mounts, but I don't > think we've turned that on yet, partly for similar reasons. This is a > subset of that problem. > > --b. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-11-01 18:22 ` Charles Hedrick @ 2021-11-01 19:25 ` Steve Dickson 2021-11-01 19:44 ` Charles Hedrick 0 siblings, 1 reply; 27+ messages in thread From: Steve Dickson @ 2021-11-01 19:25 UTC (permalink / raw) To: Charles Hedrick, J. Bruce Fields Cc: Linux NFS Mailing List, dai.ngo, Olga Kornievskaia, Chuck Lever Hello, On 11/1/21 14:22, Charles Hedrick wrote: > I am in general concerned about turning on new features before basic ones work reliably. We’ve had enough different failures that we’ve backup up to NFS 3 for file systems with heavy use. > > We first tried turning off delegation. That helped a lot. But we just ran into a two different machine hung trying to lock Chome’s profile. (I sent a bit of information on that one previously.) We had to restart NFS on the server to fix it, and that caused us to lose a bunch of VMs. (That shouldn’t have happened. It looks like ESX misbehaved.) If I could turn off NFS4 locking I would. This is the reason I was hopping not make this a global switch but a per export switch... Question... Do you do many server to server copies in your world? Meaning a client coping from one server to another? steved. > >> On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: >> >> knfsd has supported server-to-server copy for a couple years (since >> 5.5). You have set a module parameter to enable it. I'm getting asked >> when we could turn that parameter on by default. >> >> I've got a couple vague criteria: one just general maturity, the other a >> security question: >> >> 1. General maturity: the only reports I recall seeing are from testers. >> Is anyone using this? Does it work for them? Do they find a benefit? >> Maybe we could turn it on by default in one distro (Fedora?) and promote >> it a little and see what that turns up? >> >> 2. Security question: with server-to-server copy enabled, you can send >> the server a COPY call with any random address, and the server will >> mount that address, open a file, and read from it. Is that safe? >> >> Normally we only mount servers that were chosen by root. Here we'll >> mount any random server that some client told us to. What's the worst >> that random server can do? Do we trust our xdr decoding? Can it DOS us >> by throwing the client's state recovery code into some loop with weird >> error returns? Etc. >> >> Maybe it's fine. I'm OK with some level of risk. I just want to make >> sure somebody's thought this through. >> >> There's also interest in allowing unprivileged NFS mounts, but I don't >> think we've turned that on yet, partly for similar reasons. This is a >> subset of that problem. >> >> --b. > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: server-to-server copy by default 2021-11-01 19:25 ` Steve Dickson @ 2021-11-01 19:44 ` Charles Hedrick 0 siblings, 0 replies; 27+ messages in thread From: Charles Hedrick @ 2021-11-01 19:44 UTC (permalink / raw) To: Steve Dickson Cc: J. Bruce Fields, Linux NFS Mailing List, dai.ngo, Olga Kornievskaia, Chuck Lever I don’t think it’s a common operation. But there’s enough going on that it’s hard to be sure. > On Nov 1, 2021, at 3:25 PM, Steve Dickson <steved@redhat.com> wrote: > > Hello, > > On 11/1/21 14:22, Charles Hedrick wrote: >> I am in general concerned about turning on new features before basic ones work reliably. We’ve had enough different failures that we’ve backup up to NFS 3 for file systems with heavy use. >> We first tried turning off delegation. That helped a lot. But we just ran into a two different machine hung trying to lock Chome’s profile. (I sent a bit of information on that one previously.) We had to restart NFS on the server to fix it, and that caused us to lose a bunch of VMs. (That shouldn’t have happened. It looks like ESX misbehaved.) If I could turn off NFS4 locking I would. > This is the reason I was hopping not make this a global switch > but a per export switch... > > Question... Do you do many server to server copies in your world? > Meaning a client coping from one server to another? > > steved. > >>> On Oct 20, 2021, at 11:54 AM, J. Bruce Fields <bfields@fieldses.org> wrote: >>> >>> knfsd has supported server-to-server copy for a couple years (since >>> 5.5). You have set a module parameter to enable it. I'm getting asked >>> when we could turn that parameter on by default. >>> >>> I've got a couple vague criteria: one just general maturity, the other a >>> security question: >>> >>> 1. General maturity: the only reports I recall seeing are from testers. >>> Is anyone using this? Does it work for them? Do they find a benefit? >>> Maybe we could turn it on by default in one distro (Fedora?) and promote >>> it a little and see what that turns up? >>> >>> 2. Security question: with server-to-server copy enabled, you can send >>> the server a COPY call with any random address, and the server will >>> mount that address, open a file, and read from it. Is that safe? >>> >>> Normally we only mount servers that were chosen by root. Here we'll >>> mount any random server that some client told us to. What's the worst >>> that random server can do? Do we trust our xdr decoding? Can it DOS us >>> by throwing the client's state recovery code into some loop with weird >>> error returns? Etc. >>> >>> Maybe it's fine. I'm OK with some level of risk. I just want to make >>> sure somebody's thought this through. >>> >>> There's also interest in allowing unprivileged NFS mounts, but I don't >>> think we've turned that on yet, partly for similar reasons. This is a >>> subset of that problem. >>> >>> --b. > ^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2021-11-01 19:55 UTC | newest] Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-10-20 15:54 server-to-server copy by default J. Bruce Fields 2021-10-20 16:00 ` Chuck Lever III 2021-10-20 16:33 ` Olga Kornievskaia 2021-10-20 19:03 ` dai.ngo 2021-10-20 20:29 ` Bruce Fields 2021-10-21 5:00 ` dai.ngo 2021-10-21 14:02 ` Bruce Fields 2021-10-22 6:34 ` dai.ngo 2021-10-22 12:58 ` Bruce Fields 2021-11-01 17:37 ` dai.ngo 2021-11-01 19:33 ` Bruce Fields 2021-11-01 19:55 ` dai.ngo 2021-10-20 17:24 ` Steve Dickson 2021-10-20 17:51 ` Chuck Lever III 2021-10-20 16:37 ` Olga Kornievskaia 2021-10-20 17:45 ` Chuck Lever III 2021-10-20 18:15 ` Bruce Fields 2021-10-20 19:04 ` Chuck Lever III 2021-10-21 13:43 ` Steve Dickson 2021-10-21 13:56 ` Bruce Fields 2021-10-21 14:13 ` Bruce Fields 2021-10-21 14:22 ` Trond Myklebust 2021-10-21 14:38 ` bfields 2021-10-20 18:00 ` J. Bruce Fields 2021-11-01 18:22 ` Charles Hedrick 2021-11-01 19:25 ` Steve Dickson 2021-11-01 19:44 ` Charles Hedrick
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.