Specific IP per mounted share from same server

Specific IP per mounted share from same server

[Samy Ascha]

Hi!

I have an NFS server with 2 NICs, configured on different subnets. These subnets are on different network segments (2 dedicated switches).

I have clients that mount 2 distinct shares from this server. Expecting to spread the network load, I specified the mounts like so, in fstab:

10.110.1.235:/srv/nfs/www    /var/www    nfs defaults,rw,intr,nosuid,noatime,vers=4.1 0 0
10.110.0.235:/srv/nfs/backup    /backup   nfs defaults,rw,intr,nosuid,noatime,vers=4.1 0 0

However, as you may find totally expected, I see both mounts connected to the same IP on the server. Specificly, the one specified for the first mount:

10.110.1.235:/srv/nfs/www on /var/www type nfs4 (rw,...)
10.110.1.235:/srv/nfs/backup on /backup type nfs4 (rw,...)

Is this expected? Am I wrong to assume that this is a possible setup? If possible/forcible, what am I missing / doing wrong? :)

Thanks much,
Samy

Re: Specific IP per mounted share from same server

[Olga Kornievskaia]

Hi Samy,

Are you going against a linux server? I don't think linux server has
the functionality you are looking for. What you are really looking for
I believe is session trunking and neither the linux client nor server
fully support that (though we plan to add that functionality in the
near future).

Bruce, correct me if I'm wrong but linux server doesn't support
multi-home (multi-node?) therefore, it has no ability to distinguish
requests coming from different network interfaces and then present
different server major/minor/scope values and also return different
clientids in that case as well. So what happens now even though the
client is establishing a new TCP connection via the 2nd network, the
server returns back to the client same clientid and server identity as
the 1st mount thus client will use an existing connection it already
has.

On Fri, Nov 8, 2019 at 3:48 AM Samy Ascha <samy@ascha.org> wrote:
>
> Hi!
>
> I have an NFS server with 2 NICs, configured on different subnets. These subnets are on different network segments (2 dedicated switches).
>
> I have clients that mount 2 distinct shares from this server. Expecting to spread the network load, I specified the mounts like so, in fstab:
>
> 10.110.1.235:/srv/nfs/www    /var/www    nfs defaults,rw,intr,nosuid,noatime,vers=4.1 0 0
> 10.110.0.235:/srv/nfs/backup    /backup   nfs defaults,rw,intr,nosuid,noatime,vers=4.1 0 0
>
> However, as you may find totally expected, I see both mounts connected to the same IP on the server. Specificly, the one specified for the first mount:
>
> 10.110.1.235:/srv/nfs/www on /var/www type nfs4 (rw,...)
> 10.110.1.235:/srv/nfs/backup on /backup type nfs4 (rw,...)
>
> Is this expected? Am I wrong to assume that this is a possible setup? If possible/forcible, what am I missing / doing wrong? :)
>
> Thanks much,
> Samy

Re: Specific IP per mounted share from same server

[J. Bruce Fields]

On Fri, Nov 08, 2019 at 11:09:10AM -0500, Olga Kornievskaia wrote:
> Are you going against a linux server? I don't think linux server has
> the functionality you are looking for. What you are really looking for
> I believe is session trunking and neither the linux client nor server
> fully support that (though we plan to add that functionality in the
> near future).  Bruce, correct me if I'm wrong but linux server doesn't
> support multi-home (multi-node?)


The server should have complete support for session and clientid
trunking and multi-homing.  But I think we're just using those words in
slightly different ways:

> therefore, it has no ability to distinguish
> requests coming from different network interfaces and then present
> different server major/minor/scope values and also return different
> clientids in that case as well. So what happens now even though the
> client is establishing a new TCP connection via the 2nd network, the
> server returns back to the client same clientid and server identity as
> the 1st mount thus client will use an existing connection it already
> has.

So, this part is correct, it treats requests coming from different
addresses the same way.

Although you *can* actually make the server behave this way with
containers if you run nfsd in two different containers each using one of
the two network namespaces.

There are also things the client could do to distribute traffic across
the two IP addresses.  There's been some work on implementing that, but
I've lost track of where it stands at this point....

--b.

Re: Specific IP per mounted share from same server

[Olga Kornievskaia]

On Fri, Nov 8, 2019 at 11:29 AM J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Fri, Nov 08, 2019 at 11:09:10AM -0500, Olga Kornievskaia wrote:
> > Are you going against a linux server? I don't think linux server has
> > the functionality you are looking for. What you are really looking for
> > I believe is session trunking and neither the linux client nor server
> > fully support that (though we plan to add that functionality in the
> > near future).  Bruce, correct me if I'm wrong but linux server doesn't
> > support multi-home (multi-node?)
>
>
> The server should have complete support for session and clientid
> trunking and multi-homing.  But I think we're just using those words in
> slightly different ways:

By the full support, I mean neither client not server support trunking
discovery unless that sneaked in when I wasn't looking. That was the
piece that I knew needed to be done for sure.

When you say it fully support trunking do you mean it when each nfsd
node runs in its own container, right? Otherwise, I thought more code
would be needed to support the case presented here. nfsd would need to
have a notion of running something like a cluster node on a particular
interface and distinguish between requests coming from different
interface and act accordingly?

>
> > therefore, it has no ability to distinguish
> > requests coming from different network interfaces and then present
> > different server major/minor/scope values and also return different
> > clientids in that case as well. So what happens now even though the
> > client is establishing a new TCP connection via the 2nd network, the
> > server returns back to the client same clientid and server identity as
> > the 1st mount thus client will use an existing connection it already
> > has.
>
> So, this part is correct, it treats requests coming from different
> addresses the same way.
>
> Although you *can* actually make the server behave this way with
> containers if you run nfsd in two different containers each using one of
> the two network namespaces.
>
> There are also things the client could do to distribute traffic across
> the two IP addresses.  There's been some work on implementing that, but
> I've lost track of where it stands at this point....

Client can and will do trunking in case of pNFS to the data servers if
the server presents multiple IPs. Otherwise, we just have nconnect
feature but that doesn't split traffic between different network
paths.

>
> --b.

Re: Specific IP per mounted share from same server

[J. Bruce Fields]

On Fri, Nov 08, 2019 at 12:06:00PM -0500, Olga Kornievskaia wrote:
> On Fri, Nov 8, 2019 at 11:29 AM J. Bruce Fields <bfields@fieldses.org> wrote:
> >
> > On Fri, Nov 08, 2019 at 11:09:10AM -0500, Olga Kornievskaia wrote:
> > > Are you going against a linux server? I don't think linux server has
> > > the functionality you are looking for. What you are really looking for
> > > I believe is session trunking and neither the linux client nor server
> > > fully support that (though we plan to add that functionality in the
> > > near future).  Bruce, correct me if I'm wrong but linux server doesn't
> > > support multi-home (multi-node?)
> >
> >
> > The server should have complete support for session and clientid
> > trunking and multi-homing.  But I think we're just using those words in
> > slightly different ways:
> 
> By the full support, I mean neither client not server support trunking
> discovery unless that sneaked in when I wasn't looking. That was the
> piece that I knew needed to be done for sure.

I'd also say that the client supports trunking discovery--in a way,
that's the source of the problem here.  The client is probing and
discovering that the two IP addresses point to the same server, and
using that information to share the same connection.  That's trunking
discovery, to my understanding.

> When you say it fully support trunking do you mean it when each nfsd
> node runs in its own container, right?

No, I mean that we support a client using multiple sessions (clientid
trunking), or multiple connections per session (session trunking).

> Otherwise, I thought more code would be needed to support the case
> presented here. nfsd would need to have a notion of running something
> like a cluster node on a particular interface and distinguish between
> requests coming from different interface and act accordingly?

I guess it'd be possible to do something like that.  For now we don't
have any plans outside of the namespace/container work.  I wouldn't call
that trunking support.

But I think we agree on what's actually supported and not supported
here, we just disagree about the meanings of words like "trunking
support".

> Client can and will do trunking in case of pNFS to the data servers if
> the server presents multiple IPs. Otherwise, we just have nconnect
> feature but that doesn't split traffic between different network
> paths.

Got it, thanks.

--b.

Re: Specific IP per mounted share from same server

[Samy Ascha]

Hi,

Sorry for not answering back earlier.

Thank you for your answers. I understand now, why this is all happening as it is.

As I'm dealing with many clients, I also have plenty options to balance per client, instead of per share.

I, for now, with the current setups, won't be able to balance on NICs per client, but I have enough bandwidth and the NICs will still be useful in case of network failover scenario's.

Thanks again and have a good week!

Samy

> On 8 Nov 2019, at 18:06, Olga Kornievskaia <aglo@umich.edu> wrote:
> 
> On Fri, Nov 8, 2019 at 11:29 AM J. Bruce Fields <bfields@fieldses.org> wrote:
>> 
>> On Fri, Nov 08, 2019 at 11:09:10AM -0500, Olga Kornievskaia wrote:
>>> Are you going against a linux server? I don't think linux server has
>>> the functionality you are looking for. What you are really looking for
>>> I believe is session trunking and neither the linux client nor server
>>> fully support that (though we plan to add that functionality in the
>>> near future).  Bruce, correct me if I'm wrong but linux server doesn't
>>> support multi-home (multi-node?)
>> 
>> 
>> The server should have complete support for session and clientid
>> trunking and multi-homing.  But I think we're just using those words in
>> slightly different ways:
> 
> By the full support, I mean neither client not server support trunking
> discovery unless that sneaked in when I wasn't looking. That was the
> piece that I knew needed to be done for sure.
> 
> When you say it fully support trunking do you mean it when each nfsd
> node runs in its own container, right? Otherwise, I thought more code
> would be needed to support the case presented here. nfsd would need to
> have a notion of running something like a cluster node on a particular
> interface and distinguish between requests coming from different
> interface and act accordingly?
> 
>> 
>>> therefore, it has no ability to distinguish
>>> requests coming from different network interfaces and then present
>>> different server major/minor/scope values and also return different
>>> clientids in that case as well. So what happens now even though the
>>> client is establishing a new TCP connection via the 2nd network, the
>>> server returns back to the client same clientid and server identity as
>>> the 1st mount thus client will use an existing connection it already
>>> has.
>> 
>> So, this part is correct, it treats requests coming from different
>> addresses the same way.
>> 
>> Although you *can* actually make the server behave this way with
>> containers if you run nfsd in two different containers each using one of
>> the two network namespaces.
>> 
>> There are also things the client could do to distribute traffic across
>> the two IP addresses.  There's been some work on implementing that, but
>> I've lost track of where it stands at this point....
> 
> Client can and will do trunking in case of pNFS to the data servers if
> the server presents multiple IPs. Otherwise, we just have nconnect
> feature but that doesn't split traffic between different network
> paths.
> 
>> 
>> --b.