[PATCH 0/1] Add Host Transport Interface option (host-triface)

[PATCH 0/1] Add Host Transport Interface option (host-triface)

[Martin Belanger]

From: Martin Belanger <martin.belanger@dell.com>

In our application, we need a way to force TCP connections to go out a
specific interface instead of letting Linux select the interface based
on the routing tables. This patch adds the option 'host-triface' to allow
specifying the interface to use. Note that corresponding changes to the
nvme-cli utility will follow.

Martin Belanger (1):
  Add 'Transport Interface' (triface) option. This can be used to
    specify the IP interface to use for the connection. The driver uses
    that to set SO_BINDTODEVICE on the socket before connecting.

 drivers/nvme/host/core.c    |  5 +++++
 drivers/nvme/host/fabrics.c | 14 +++++++++++++
 drivers/nvme/host/fabrics.h |  6 +++++-
 drivers/nvme/host/tcp.c     | 41 ++++++++++++++++++++++++++++++++++---
 4 files changed, 62 insertions(+), 4 deletions(-)

--
2.25.1


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)

[James Smart]

On 4/15/2021 12:28 PM, Martin Belanger wrote:
> From: Martin Belanger <martin.belanger@dell.com>
> 
> In our application, we need a way to force TCP connections to go out a
> specific interface instead of letting Linux select the interface based
> on the routing tables. This patch adds the option 'host-triface' to allow
> specifying the interface to use. Note that corresponding changes to the
> nvme-cli utility will follow.
> 
> Martin Belanger (1):
>    Add 'Transport Interface' (triface) option. This can be used to
>      specify the IP interface to use for the connection. The driver uses
>      that to set SO_BINDTODEVICE on the socket before connecting.
> 
>   drivers/nvme/host/core.c    |  5 +++++
>   drivers/nvme/host/fabrics.c | 14 +++++++++++++
>   drivers/nvme/host/fabrics.h |  6 +++++-
>   drivers/nvme/host/tcp.c     | 41 ++++++++++++++++++++++++++++++++++---
>   4 files changed, 62 insertions(+), 4 deletions(-)
> 

This wasn't able to be done via HOST_TRADDR ?

-- james


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)

[Belanger, Martin]

Re-sending in Plain Text mode.

Hi James,

With HOST_TRADDR, we use bind() to bind to an address just before invoking connect(). I also used to think that this was the way to force packets to be routed onto the interface matching the address used in the bind(). But that is not the case (I tested it). bind() does not control the routing of transmitted packets. The only thing that bind() does is to set the source address on the socket so that when a remote process queries the peer address (getpeername), it will see that address as the source address.

It's only a setsockopt() with SO_BINDTODEVICE that can force packets to go out on a specific interface (I tested that as well).

References:
https://codingrelic.geekhold.com/2009/10/code-snippet-sobindtodevice.html
https://www.javaer101.com/en/article/1866348.html
https://wiki.treck.com/Appendix_C:_Strong_End_System_Model_/_Weak_End_System_Model

Regards,
Martin

________________________________________
From: James Smart <jsmart2021@gmail.com>
Sent: Thursday, April 15, 2021 19:05
To: Martin Belanger; linux-nvme@lists.infradead.org
Cc: kbusch@kernel.org; axboe@fb.com; hch@lst.de; sagi@grimberg.me; Belanger, Martin
Subject: Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)


[EXTERNAL EMAIL]

On 4/15/2021 12:28 PM, Martin Belanger wrote:
> From: Martin Belanger <martin.belanger@dell.com>
>
> In our application, we need a way to force TCP connections to go out a
> specific interface instead of letting Linux select the interface based
> on the routing tables. This patch adds the option 'host-triface' to allow
> specifying the interface to use. Note that corresponding changes to the
> nvme-cli utility will follow.
>
> Martin Belanger (1):
>    Add 'Transport Interface' (triface) option. This can be used to
>      specify the IP interface to use for the connection. The driver uses
>      that to set SO_BINDTODEVICE on the socket before connecting.
>
>   drivers/nvme/host/core.c    |  5 +++++
>   drivers/nvme/host/fabrics.c | 14 +++++++++++++
>   drivers/nvme/host/fabrics.h |  6 +++++-
>   drivers/nvme/host/tcp.c     | 41 ++++++++++++++++++++++++++++++++++---
>   4 files changed, 62 insertions(+), 4 deletions(-)
>

This wasn't able to be done via HOST_TRADDR ?

-- james


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)

[Hannes Reinecke]

On 4/16/21 2:41 AM, Belanger, Martin wrote:
> Re-sending in Plain Text mode.
> 
> Hi James,
> 
> With HOST_TRADDR, we use bind() to bind to an address just before invoking connect(). I also used
> to think that this was the way to force packets to be routed onto the interface matching the address
> used in the bind(). But that is not the case (I tested it). bind() does not control the routing
> of transmitted packets. The only thing that bind() does is to set the source address on the socket
> so that when a remote process queries the peer address (getpeername), it will see that address as
> the source address.
> 
> It's only a setsockopt() with SO_BINDTODEVICE that can force packets to go out on a specific
> interface (I tested that as well).
>
Hmm. But arguably this was what 'host_traddr' was supposed to achieve.
If you compare commit 8f4e8dace3d1 ("nvme-rdma: add support for
host_traddr") it clearly states that the request should be _send_ via
the interface specified with host_traddr.
So in the light of this I'd rather modify the current handling of
host_traddr for tcp.

Unless this was intentionally coded this way. Sagi?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare@suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)

[Belanger, Martin]

Hi Hannes,

For our use case, we need both features. We need to be able to specify the "interface" and we need to be able to specify the "source address". And we need both features to be controlled with separate "knobs".

Regards,
Martin

________________________________________
From: Hannes Reinecke <hare@suse.de>
Sent: Friday, April 16, 2021 10:26
To: Belanger, Martin; James Smart; Martin Belanger; linux-nvme@lists.infradead.org
Cc: kbusch@kernel.org; axboe@fb.com; hch@lst.de; sagi@grimberg.me
Subject: Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)


[EXTERNAL EMAIL]

On 4/16/21 2:41 AM, Belanger, Martin wrote:
> Re-sending in Plain Text mode.
>
> Hi James,
>
> With HOST_TRADDR, we use bind() to bind to an address just before invoking connect(). I also used
> to think that this was the way to force packets to be routed onto the interface matching the address
> used in the bind(). But that is not the case (I tested it). bind() does not control the routing
> of transmitted packets. The only thing that bind() does is to set the source address on the socket
> so that when a remote process queries the peer address (getpeername), it will see that address as
> the source address.
>
> It's only a setsockopt() with SO_BINDTODEVICE that can force packets to go out on a specific
> interface (I tested that as well).
>
Hmm. But arguably this was what 'host_traddr' was supposed to achieve.
If you compare commit 8f4e8dace3d1 ("nvme-rdma: add support for
host_traddr") it clearly states that the request should be _send_ via
the interface specified with host_traddr.
So in the light of this I'd rather modify the current handling of
host_traddr for tcp.

Unless this was intentionally coded this way. Sagi?

Cheers,

Hannes
--
Dr. Hannes Reinecke                     Kernel Storage Architect
hare@suse.de                                   +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)

[Belanger, Martin]

Hi Hannes,

Just checking if there were any update on this.

I understand that you'd prefer if we used the existing option --host-traddr to specify the interface. However, this option only sets the source address on the socket. But that's ok because we do need an option to set the source address without affecting which interface is used for the connection.

A separate option, --host-triface, specifically designed to configure the interface makes more sense. Here's why:
1) It's more intuitive to specify the "interface name" than the associated "IP address". Using an IP Address to specify an interface is confusing. One needs to perform a reverse lookup to figure out which IP address is associated with an interface.
2) There is no "hard" association between IP addresses and interfaces. The IP address associated with an interface can simply be changed with the command "ip addr [add|delete|change]". Worse, the same IP address can be assigned to more than one interface (e.g. ip addr add 20.20.20.20/24 dev enp2s0 && ip addr add 20.20.20.20/24 dev enp2s1). That may be a silly thing to do, but it's allowed.

Regards,
Martin

________________________________________
From: Belanger, Martin <Martin_Belanger@Dell.com>
Sent: Friday, April 16, 2021 12:15
To: Hannes Reinecke; James Smart; Martin Belanger; linux-nvme@lists.infradead.org
Cc: kbusch@kernel.org; axboe@fb.com; hch@lst.de; sagi@grimberg.me
Subject: Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)

Hi Hannes,

For our use case, we need both features. We need to be able to specify the "interface" and we need to be able to specify the "source address". And we need both features to be controlled with separate "knobs".

Regards,
Martin

________________________________________
From: Hannes Reinecke <hare@suse.de>
Sent: Friday, April 16, 2021 10:26
To: Belanger, Martin; James Smart; Martin Belanger; linux-nvme@lists.infradead.org
Cc: kbusch@kernel.org; axboe@fb.com; hch@lst.de; sagi@grimberg.me
Subject: Re: [PATCH 0/1] Add Host Transport Interface option (host-triface)


[EXTERNAL EMAIL]

On 4/16/21 2:41 AM, Belanger, Martin wrote:
> Re-sending in Plain Text mode.
>
> Hi James,
>
> With HOST_TRADDR, we use bind() to bind to an address just before invoking connect(). I also used
> to think that this was the way to force packets to be routed onto the interface matching the address
> used in the bind(). But that is not the case (I tested it). bind() does not control the routing
> of transmitted packets. The only thing that bind() does is to set the source address on the socket
> so that when a remote process queries the peer address (getpeername), it will see that address as
> the source address.
>
> It's only a setsockopt() with SO_BINDTODEVICE that can force packets to go out on a specific
> interface (I tested that as well).
>
Hmm. But arguably this was what 'host_traddr' was supposed to achieve.
If you compare commit 8f4e8dace3d1 ("nvme-rdma: add support for
host_traddr") it clearly states that the request should be _send_ via
the interface specified with host_traddr.
So in the light of this I'd rather modify the current handling of
host_traddr for tcp.

Unless this was intentionally coded this way. Sagi?

Cheers,

Hannes
--
Dr. Hannes Reinecke                     Kernel Storage Architect
hare@suse.de                                   +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme