Re: Question about flow table offload in mlx5e

[Paul Blakey <paulb@mellanox.com>]

On 12/9/2019 5:18 AM, wenxu wrote:
> Hi  Paul,
>
>
> Thanks for your fix, I will test it.
>
> On 12/8/2019 5:39 PM, Paul Blakey wrote:
>> Here's the temp fix:
>>
>>
>> The problem is TC + FT offload, and this revealed a bug in the driver.
>>
>> For the tunnel test, we changed tc block offload to ft callback, and
>> didn't change the indr block offload.
>>
>> So the tunnel unset rule is offloaded from indr tc callback (it's
>> indirect because it's on tun1 device):
>>
>> mlx5e_rep_indr_setup_block_cb
> Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup?
>
> Or all indr setup through TC offload?

Adding a "mlx5e_rep_indr_setup_ft_cb" with the correct flags (FT) and 
(EGRESS) should work as well, but this is just a test...

For upstream, I see you're talking with pablo about the pending tunnel 
offload support.


>> this offloaded the rule to hardware in the TC domain.
>>
>> Now the tunnel set (encap) rule was offloaded to FT domain.
>>
>>
>> Since TC comes before FT in software, we should have connected the miss
>> on TC domain to FT domain,
>>
>> but this didn't work.
>>
>> The below fix should fix that connection:
>>
>>
>> ------------------------------------------
>>
>>
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>> @@ -763,9 +763,6 @@ static struct mlx5_flow_table
>> *find_closest_ft_recursive(struct fs_node  *root,
>>           struct fs_node *iter = list_entry(start, struct fs_node, list);
>>           struct mlx5_flow_table *ft = NULL;
>>
>> -       if (!root || root->type == FS_TYPE_PRIO_CHAINS)
>> -               return NULL;
>> -
>>           list_for_each_advance_continue(iter, &root->children, reverse) {
>>                   if (iter->type == FS_TYPE_FLOW_TABLE) {
>>                           fs_get_obj(ft, iter);
>> @@ -792,7 +789,10 @@ static struct mlx5_flow_table
>> *find_closest_ft(struct fs_prio *prio, bool revers
>>           parent = prio->node.parent;
>>           curr_node = &prio->node;
>>           while (!ft && parent) {
>> -               ft = find_closest_ft_recursive(parent, &curr_node->list,
>> reverse);
>> +               if (parent->type != FS_TYPE_PRIO_CHAINS)
>> +                       ft = find_closest_ft_recursive(parent,
>> + &curr_node->list,
>> +                                                      reverse);
>>                   curr_node = parent;
>>                   parent = curr_node->parent;
>>           }
>>
>>
>> ------------------------------------------
>>
>>
>> I will do this patch for upstream if needed after our last patchset that
>> also touched this area.
>>
>> Thanks,
>>
>> Paul.
>>
>>
>>
>>
>> On 12/5/2019 5:17 PM, Paul Blakey wrote:
>>> On 12/2/2019 5:37 AM, wenxu wrote:
>>>
>>>> Hi Paul,
>>>>
>>>>
>>>> Sorry for trouble you again. I think it is a problem in ft callback.
>>>>
>>>> Can your help me fix it. Thx!
>>>>
>>>> I did the test like you with route tc rules to ft callback.
>>>>
>>>> # ifconfig mlx_p0 172.168.152.75/24 up
>>>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>>>>
>>>> # ip l add dev tun1 type gretap external
>>>> # tc qdisc add dev tun1 ingress
>>>> # tc qdisc add dev mlx_pf0vf0 ingress
>>>>
>>>> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower
>>>> skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id
>>>> 1000 nocsum pipe action mirred egress redirect dev tun1
>>>>
>>>>
>>>> In The vm:
>>>> # ifconfig eth0 10.0.0.75/24 up
>>>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>>>>
>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>
>>>> The syn packets can be offloaded successfully.
>>>>
>>>> # # tc -s filter ls dev mlx_pf0vf0 ingress
>>>> filter protocol ip pref 2 flower chain 0
>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>     eth_type ipv4
>>>>     skip_sw
>>>>     in_hw in_hw_count 1
>>>>      action order 1: tunnel_key  set
>>>>      src_ip 0.0.0.0
>>>>      dst_ip 172.168.152.241
>>>>      key_id 1000
>>>>      nocsum pipe
>>>>       index 1 ref 1 bind 1 installed 252 sec used 252 sec
>>>>      Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>      action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>        index 1 ref 1 bind 1 installed 252 sec used 110 sec
>>>>        Action statistics:
>>>>      Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      Sent software 0 bytes 0 pkt
>>>>      Sent hardware 3420 bytes 11 pkt
>>>>      backlog 0b 0p requeues 0
>>>>
>>>> But Then I add another decap filter on tun1:
>>>>
>>>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id
>>>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action
>>>> mirred egress redirect dev mlx_pf0vf0
>>>>
>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>
>>>> The syn packets can't be offloaded. The tc filter counter is also not
>>>> increase.
>>>>
>>>>
>>>> # tc -s filter ls dev mlx_pf0vf0 ingress
>>>> filter protocol ip pref 2 flower chain 0
>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>     eth_type ipv4
>>>>     skip_sw
>>>>     in_hw in_hw_count 1
>>>>      action order 1: tunnel_key  set
>>>>      src_ip 0.0.0.0
>>>>      dst_ip 172.168.152.241
>>>>      key_id 1000
>>>>      nocsum pipe
>>>>       index 1 ref 1 bind 1 installed 320 sec used 320 sec
>>>>      Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>      action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>        index 1 ref 1 bind 1 installed 320 sec used 178 sec
>>>>        Action statistics:
>>>>      Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      Sent software 0 bytes 0 pkt
>>>>      Sent hardware 3420 bytes 11 pkt
>>>>      backlog 0b 0p requeues 0
>>>>
>>>> # tc -s filter ls dev tun1 ingress
>>>> filter protocol ip pref 2 flower chain 0
>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>     eth_type ipv4
>>>>     enc_src_ip 172.168.152.241
>>>>     enc_key_id 1000
>>>>     in_hw in_hw_count 1
>>>>      action order 1: tunnel_key  unset pipe
>>>>       index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>      Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>      action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>>>        index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>        Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>
>>>> So there maybe some problem for ft callback setup. When there is
>>>> another reverse
>>>> decap rule add in tunnel device, The encap rule will not offloaded
>>>> the packets.
>>>>
>>>> Expect your help Thx!
>>>>
>>>>
>>>> BR
>>>> wenxu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> Hi I reproduced it.
>>>
>>> I'll find the reason and fix for it and get back to you soon.
>>>
>>> We are planing on expanding our chain and prio supported range, and in
>>> that we also move the FT offload code a bit.
>>>
>>> If what I think happens happened it would fix it anyway.
>>>
>>> Thanks.
>>>