updates needed to Documentation/librdmacm.md

updates needed to Documentation/librdmacm.md

[Rehm, Kevan Flint]

Greetings,
 
Getting multiple infiniband interfaces on a node to work required more changes than the hints provided in Documentation/librdmacm.md.  We have a few suggestions for additions to that page that might save others a lot of debugging time.
 
1. accept_local must be set to 1 as documented in in librdmacm.md.

2. The documentation says to set arp_ignore to the value 2.   That only works as long as all the interfaces on clients and servers are in the same logical subnet, e.g.   ib0 == 10.0.0.27, ib1 == 10.0.1.27, prefix=16.   If you happened to create separate logical subnets, e.g. prefix=24, then the value 2 does not work, you have to use the value 1.

3. The documentation does not mention the rp_filter parameter, but it must be modified even if you have created a single logical subnet.  The value of rp_filter cannot be 1, you must set it to either 0 or 2, with 2 being more secure.   The default value on CentOS 7 is 1.   We are using 2.
 
We were surprised to find that the value of rp_filter would also get reset after a “systemctl restart network”, not just on reboots, so the setting you want must be persisted in a /etc/sysctl.d file if you want the value to stay across network restarts.  The other parameters held their values across network restarts.
 
With the above changes persisted in a new /etc/sysctl.d/95-daos.conf file, we have been able to successfully use multiple infiniband interfaces per node.
 
Distribution is CentOS 7.
Kernel is 3.10.0-1062.18.1.el7.x86_64
 
CentOS 7 sysctl defaults:
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.rp_filter = 1
 
If you need more information, let me know,
 
Regards, Kevan

Re: updates needed to Documentation/librdmacm.md

[Jason Gunthorpe]

On Tue, Mar 31, 2020 at 10:57:05PM +0000, Rehm, Kevan Flint wrote:
> Greetings,
>  
> Getting multiple infiniband interfaces on a node to work required
> more changes than the hints provided in Documentation/librdmacm.md.
> We have a few suggestions for additions to that page that might save
> others a lot of debugging time.

You must have some very strange configuration to need these settings
changed - this really only impacts cases where there are multiple
interfaces with overlapping routing.

Generally the default should work for unambiguous cases

If there is overlapping routing then you need to follow the usual linux
guidance which usually also includes policy routing and the various
ARP adjustments

Jason