Re: Bug or mis configuration for mlx5e lag and multipath

[Roi Dayan <roid@mellanox.com>]

On 24/05/2019 04:02, wenxu wrote:
> Hi,
> 
> I can get the right log from demsg
> 
>  mlx5_core 0000:81:00.0: modify lag map port 1:1 port 2:2
> 
> 
> I debug with the driver, I find the rule be add on mlx_pf0vf0 and the peer one pf1,
> 
> So I think the esw0 and esw1 both have the rule.
> 
> The test case is based on the master branch of the net git tree.

ok thanks for reporting. we'll have to check it.

> 
> 在 2019/5/23 23:15, Roi Dayan 写道:
>>
>> On 20/05/2019 04:53, wenxu wrote:
>>> Hi Roi & Saeed,
>>>
>>> I just test the mlx5e lag and mutipath feature. There are some suituation the outgoing can't be offloaded.
>>>
>>> ovs configureation as following.
>>>
>>> # ovs-vsctl show
>>> dfd71dfb-6e22-423e-b088-d2022103af6b
>>>     Bridge "br0"
>>>         Port "mlx_pf0vf0"
>>>             Interface "mlx_pf0vf0"
>>>         Port gre
>>>             Interface gre
>>>                 type: gre
>>>                 options: {key="1000", local_ip="172.168.152.75", remote_ip="172.168.152.241"}
>>>         Port "br0"
>>>             Interface "br0"
>>>                 type: internal
>>>
>>> set the mlx5e driver:
>>>
>>>
>>> modprobe mlx5_core
>>> echo 0 > /sys/class/net/eth2/device/sriov_numvfs
>>> echo 0 > /sys/class/net/eth3/device/sriov_numvfs
>>> echo 2 > /sys/class/net/eth2/device/sriov_numvfs
>>> echo 2 > /sys/class/net/eth3/device/sriov_numvfs
>>> lspci -nn | grep Mellanox
>>> echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
>>> echo 0000:81:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind
>>> echo 0000:81:03.6 > /sys/bus/pci/drivers/mlx5_core/unbind
>>> echo 0000:81:03.7 > /sys/bus/pci/drivers/mlx5_core/unbind
>>>
>>> devlink dev eswitch set pci/0000:81:00.0  mode switchdev encap enable
>>> devlink dev eswitch set pci/0000:81:00.1  mode switchdev encap enable
>>>
>>> modprobe bonding mode=802.3ad miimon=100 lacp_rate=1
>>> ip l del dev bond0
>>> ifconfig mlx_p0 down
>>> ifconfig mlx_p1 down
>>> ip l add dev bond0 type bond mode 802.3ad
>>> ifconfig bond0 172.168.152.75/24 up
>>> echo 1 > /sys/class/net/bond0/bonding/xmit_hash_policy
>>> ip l set dev mlx_p0 master bond0
>>> ip l set dev mlx_p1 master bond0
>>> ifconfig mlx_p0 up
>>> ifconfig mlx_p1 up
>>>
>>> systemctl start openvswitch
>>> ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
>>> systemctl restart openvswitch
>>>
>>>
>>> mlx_pf0vf0 is assigned to vm. The tc rule show in_hw
>>>
>>> # tc filter ls dev mlx_pf0vf0 ingress
>>> filter protocol ip pref 2 flower
>>> filter protocol ip pref 2 flower handle 0x1
>>>   dst_mac 8e:c0:bd:bf:72:c3
>>>   src_mac 52:54:00:00:12:75
>>>   eth_type ipv4
>>>   ip_tos 0/3
>>>   ip_flags nofrag
>>>   in_hw
>>>     action order 1: tunnel_key set
>>>     src_ip 172.168.152.75
>>>     dst_ip 172.168.152.241
>>>     key_id 1000 pipe
>>>     index 2 ref 1 bind 1
>>>  
>>>     action order 2: mirred (Egress Redirect to device gre_sys) stolen
>>>      index 2 ref 1 bind 1
>>>
>>> In the vm:  the mlx5e driver enable xps default (by the way I think it is better not enable xps in default kernel can select queue by each flow),  in the lag mode different vf queue associate with hw PF.
>>>
>>> with command taskset -c 2 ping 10.0.0.241
>>>
>>> the packet can be offloaded , the outgoing pf is mlx_p0
>>>
>>> but with command taskset -c 1 ping 10.0.0.241
>>>
>>> the packet can't be offloaded, I can capture the packet on the mlx_pf0vf0, the outgoing pf is mlx_p1. Althrough the tc flower rule show in_hw
>>>
>>>
>>> I check with the driver  both mlx_pf0vf0 and peer(mlx_p1) install the tc rule success
>>>
>>> I think it's a problem of lag mode. Or I miss some configureation?
>>>
>>>
>>> BR
>>>
>>> wenxu
>>>
>>>
>>>
>>>
>>>
>> Hi,
>>
>> we need to verify the driver detected to be in lag mode and
>> duplicated the offload rule to both eswitches.
>> do you see lag map messages in dmesg?
>> something like "lag map port 1:1 port 2:2"
>> this is to make sure the driver actually in lag mode.
>> in this mode a rule added to mlx_pf0vf0 will be added to esw of pf0 and esw of pf1.
>> then when u send a packet it could be handled in esw0 or esw1
>> if the rule is not in esw1 then it wont be offloaded when using pf1.
>>
>> thanks,
>> Roi