Re: [PATCH RFC 3/3] RDMA/siw: Require non-zero 6-byte MACs for soft iWARP

[Chuck Lever III <chuck.lever@oracle.com>]

> On May 31, 2023, at 4:09 PM, Jason Gunthorpe <jgg@ziepe.ca> wrote:
> 
> On Wed, May 31, 2023 at 07:11:52PM +0000, Chuck Lever III wrote:
>> 
>> 
>>> On May 31, 2023, at 3:04 PM, Jason Gunthorpe <jgg@ziepe.ca> wrote:
>>> 
>>> On Tue, May 23, 2023 at 07:18:18PM +0000, Chuck Lever III wrote:
>>> 
>>>> The core address resolution code wants to find an L2 address
>>>> for the egress device. The underlying ib_device, where a made-up
>>>> GID might be stored, is not involved with address resolution
>>>> AFAICT.
>>> 
>>> Where are you hitting this?
>> 
>>     kworker/2:0-26    [002]   551.962874: funcgraph_entry:                   |  addr_resolve() {
>>     kworker/2:0-26    [002]   551.962874: bprint:               addr_resolve: resolve_neigh=true resolve_by_gid_attr=false
>>     kworker/2:0-26    [002]   551.962874: funcgraph_entry:                   |    addr4_resolve.constprop.0() {
>>     kworker/2:0-26    [002]   551.962875: bprint:               addr4_resolve.constprop.0: src_in=0.0.0.0:35173 dst_in=100.72.1.2:20049
>>     kworker/2:0-26    [002]   551.962875: funcgraph_entry:                   |      ip_route_output_flow() {
>>     kworker/2:0-26    [002]   551.962875: funcgraph_entry:                   |        ip_route_output_key_hash() {
>>     kworker/2:0-26    [002]   551.962876: funcgraph_entry:                   |          ip_route_output_key_hash_rcu() {
>>     kworker/2:0-26    [002]   551.962876: funcgraph_entry:        4.526 us   |            __fib_lookup();
>>     kworker/2:0-26    [002]   551.962881: funcgraph_entry:        0.264 us   |            fib_select_path();
>>     kworker/2:0-26    [002]   551.962881: funcgraph_entry:        1.022 us   |            __mkroute_output();
>>     kworker/2:0-26    [002]   551.962882: funcgraph_exit:         6.705 us   |          }
>>     kworker/2:0-26    [002]   551.962882: funcgraph_exit:         7.283 us   |        }
>>     kworker/2:0-26    [002]   551.962883: funcgraph_exit:         7.624 us   |      }
>>     kworker/2:0-26    [002]   551.962883: funcgraph_exit:         8.395 us   |    }
>>     kworker/2:0-26    [002]   551.962883: funcgraph_entry:                   |    rdma_set_src_addr_rcu.constprop.0() {
>>     kworker/2:0-26    [002]   551.962883: bprint:               rdma_set_src_addr_rcu.constprop.0: ndev=0xffff91f5135a4000 name=tailscale0
>>     kworker/2:0-26    [002]   551.962884: funcgraph_entry:                   |      copy_src_l2_addr() {
>>     kworker/2:0-26    [002]   551.962884: funcgraph_entry:        0.984 us   |        iff_flags2string();
>>     kworker/2:0-26    [002]   551.962885: bprint:               copy_src_l2_addr: ndev=0xffff91f5135a4000 dst_in=100.72.1.2:20049 flags=UP|POINTOPOINT|NOARP|MULTICAST
>>     kworker/2:0-26    [002]   551.962885: funcgraph_entry:                   |        rdma_copy_src_l2_addr() {
>>     kworker/2:0-26    [002]   551.962886: funcgraph_entry:        0.148 us   |          devtype2string();
>>     kworker/2:0-26    [002]   551.962887: bprint:               rdma_copy_src_l2_addr: name=tailscale0 type=NONE src_dev_addr=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 broadcast=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ifindex=3
>>     kworker/2:0-26    [002]   551.962887: funcgraph_exit:         1.488 us   |        }
>>     kworker/2:0-26    [002]   551.962887: bprint:               copy_src_l2_addr: network type=IB
>>     kworker/2:0-26    [002]   551.962887: funcgraph_exit:         3.636 us   |      }
>>     kworker/2:0-26    [002]   551.962887: funcgraph_exit:         4.275 us   |    }
>> 
>> 
>> Address resolution finds the right device, but there's
>> a zero-value L2 address.
> 
> Sure, but why is that a problem?
> 
> This got to rdma_set_src_addr_rcu, so the resolution suceeded, where
> is the failure? From the above trace I think addr_resolve() succeeded?

Possibly it did succeed. But the ULP consumer sees CM_ADDR_ERROR_EVENT,
and does not proceed to route resolution.


>> Thus it cannot form a unique GID from that. Perhaps there needs to
>> be a call to query_gid in here?
> 
> So your issue is cma_iw_acquire_dev() which looks like it is encoding
> the MAC into the GID for some reason? We don't do that on rocee, the
> GID encodes the IP address

Well, I'm not getting there at all on the initiator side.
cma_iw_acquire_dev() is called only for listeners, I thought.


> 
> I have no idea how iWarp works, but this is surprising that it puts a
> MAC in the GID..
> 
> If the iwarp device has only one GID ever and it is always the "MAC"
> the cma_iw_acquire_dev()'s logic is simply wrong, it should check that
> the dev_addr's netdev matches the one and only GID and just use the
> GID. No reason to search for GIDs.
> 
> A small edit to cma_validate_port() might make sense, it is kind of
> wrong to force the gid_type to IB_GID_TYPE_IB for whatever ARPHRD type
> the tunnel is using.

I will have a look.

--
Chuck Lever