Re: [PATCH 1/1] Add 'Transport Interface' (triface) option. This can be used to specify the IP interface to use for the connection. The driver uses that to set SO_BINDTODEVICE on the socket before connecting.

[Hannes Reinecke <hare@suse.de>]

On 5/6/21 8:05 AM, Hannes Reinecke wrote:
> On 5/5/21 4:31 PM, Belanger, Martin wrote:
[ .. ]
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
>> group default qlen 1000
>>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>      inet 100.0.0.100/24 scope global lo
>>         valid_lft forever preferred_lft forever
>> 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
>> state UP group default qlen 1000
>>      link/ether 08:00:27:21:65:ec brd ff:ff:ff:ff:ff:ff
>>      inet 100.0.0.100/24 scope global enp0s3
>>         valid_lft forever preferred_lft forever
>> 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
>> state UP group default qlen 1000
>>      link/ether 08:00:27:4f:95:5c brd ff:ff:ff:ff:ff:ff
>>      inet 100.0.0.100/24 scope global enp0s8
>>         valid_lft forever preferred_lft forever
>>
>> The above is a VM that I configured with the same IP address
>> (100.0.0.100) on all interfaces. Doing a reverse
>> lookup to identify the unique interface associated with 100.0.0.100
>> would simply not work here. And this is why
>> the option host_iface is required. I understand that the above config
>> does not represent a standard host system,
>> but I'm using this to prove a point: "we can never know how a user
>> will configure their system and the above
>> configuration is perfectly fine by Linux".
>>
> 
> ... and messing up any switch MAC address caching when doing so. I guess
> the network admin will come down hard on you if you try that on a
> production system.
> And I sincerely question whether this is a valid use-case; I'm already
> getting grief from our network admins if I dare to put two network
> interfaces from the same machine in the same network.
> 
>> The current TCP implementation for host_traddr uses
>> bind()-before-connect(). This is a common construct to set the
>> source IP address on the socket before connecting. This has no effect
>> on how Linux will select the interface for the
>> connection. That's because Linux uses the Weak End System model as
>> described in RFC1122 [2]. Setting the source address
>> on a connection is a common requirement that linux-nvme needs to
>> support. In fact, specifying the Source IP address
>> is a mandatory FedGov requirement (e.g. connection to a RADIUS/TACACS+
>> server). Consider the following configuration.
>>
>> $ ip addr list dev enp0s8
>> 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
>> state UP group default qlen 1000
>>      link/ether 08:00:27:4f:95:5c brd ff:ff:ff:ff:ff:ff
>>      inet 192.168.56.101/24 brd 192.168.56.255 scope global dynamic
>> noprefixroute enp0s8
>>         valid_lft 426sec preferred_lft 426sec
>>      inet 192.168.56.102/24 scope global secondary enp0s8
>>         valid_lft forever preferred_lft forever
>>      inet 192.168.56.103/24 scope global secondary enp0s8
>>         valid_lft forever preferred_lft forever
>>      inet 192.168.56.104/24 scope global secondary enp0s8
>>         valid_lft forever preferred_lft forever
>>
>> Here we can see that several addresses are associated with interface
>> enp0s8. By default, Linux will select the
>> default IP address, 192.168.56.101, as the source address when
>> connecting over interface enp0s8. Some users,
>> however, want the ability to specify a different address (e.g.,
>> 192.168.56.103) to be used as the source address.
>> The option host_traddr can be used as-is to perform this function (I
>> tested it).
>>
> 
> No disagreement here.
> 
>> In conclusion, I believe that for TCP we need 2 options. One that can
>> be used to specify an interface. And one
>> that can be used to set the source address. And users should be
>> allowed to use one or the other, or both, or none.
>> Of course, the documentation for host_traddr will need some
>> clarification. It should state that when used for TCP
>> connection, this option only sets the source address. And the
>> documentation for host_iface should say that this
>> option only applies to TCP connections.
>>
> 
> I'm with James Smart here. I do fail to see the need for 'host_iface'
> _without_ 'host_traddr'; especially for IPv6 where several addresses are
> standard just specifying 'host_iface' simply is not enough, and one has
> to specify 'host_traddr' additionally.
> 
> So 'host_iface' should be contingent on 'host_traddr', meaning we can
> just expand the syntax of 'host_traddr'.
> One easy possibility would be to add ',nobind' to the host_traddr syntax
> which would indicate that we should _not_ bind to the underlying
> interface; I do think that binding to the respective interface should be
> the default.
> 
A-ha. Just spoke to our network folks, and they clarified the usage of
binding to an IP address vs binding to a network interface.
Apparently, binding to a source IP address does just that, setting the
source IP address of the outgoing packet. That packet will _still_ be
subjected to the normal routing table, as the routing table is just
influenced by the _destination_ IP address.
So if we want to have it routed via a specific interface (and thereby
influencing the routing table) we need to bind it to that interface.

The only valid scenario our network folks could come up with where we do
_not_ want to bind to an interface is for asymmetric flows, ie in cases
where the outgoing flow is routed to one interface and the incoming flow
is arriving on another interface. But even they admitted that it's not a
common scenario, and probably will be killed by anti-spoofing software
running on the core switches ...

But if we want to support _that_ then clearly binding to a specific
interface doesn't work.

So I would vote for making binding to the network interface holding the
IP address the default, and add an option ',nobind' to host_traddr to
skip it.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare@suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme