Re: Question about flow table offload in mlx5e

[wenxu <wenxu@ucloud.cn>]

Hi  Paul,


Thanks for your fix, I will test it.

On 12/8/2019 5:39 PM, Paul Blakey wrote:
> Here's the temp fix:
>
>
> The problem is TC + FT offload, and this revealed a bug in the driver.
>
> For the tunnel test, we changed tc block offload to ft callback, and 
> didn't change the indr block offload.
>
> So the tunnel unset rule is offloaded from indr tc callback (it's 
> indirect because it's on tun1 device):
>
> mlx5e_rep_indr_setup_block_cb

Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup?

Or all indr setup through TC offload?

>
> this offloaded the rule to hardware in the TC domain.
>
> Now the tunnel set (encap) rule was offloaded to FT domain.
>
>
> Since TC comes before FT in software, we should have connected the miss 
> on TC domain to FT domain,
>
> but this didn't work.
>
> The below fix should fix that connection:
>
>
> ------------------------------------------
>
>
> --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
> @@ -763,9 +763,6 @@ static struct mlx5_flow_table 
> *find_closest_ft_recursive(struct fs_node  *root,
>          struct fs_node *iter = list_entry(start, struct fs_node, list);
>          struct mlx5_flow_table *ft = NULL;
>
> -       if (!root || root->type == FS_TYPE_PRIO_CHAINS)
> -               return NULL;
> -
>          list_for_each_advance_continue(iter, &root->children, reverse) {
>                  if (iter->type == FS_TYPE_FLOW_TABLE) {
>                          fs_get_obj(ft, iter);
> @@ -792,7 +789,10 @@ static struct mlx5_flow_table 
> *find_closest_ft(struct fs_prio *prio, bool revers
>          parent = prio->node.parent;
>          curr_node = &prio->node;
>          while (!ft && parent) {
> -               ft = find_closest_ft_recursive(parent, &curr_node->list, 
> reverse);
> +               if (parent->type != FS_TYPE_PRIO_CHAINS)
> +                       ft = find_closest_ft_recursive(parent,
> + &curr_node->list,
> +                                                      reverse);
>                  curr_node = parent;
>                  parent = curr_node->parent;
>          }
>
>
> ------------------------------------------
>
>
> I will do this patch for upstream if needed after our last patchset that 
> also touched this area.
>
> Thanks,
>
> Paul.
>
>
>
>
> On 12/5/2019 5:17 PM, Paul Blakey wrote:
>> On 12/2/2019 5:37 AM, wenxu wrote:
>>
>>> Hi Paul,
>>>
>>>
>>> Sorry for trouble you again. I think it is a problem in ft callback.
>>>
>>> Can your help me fix it. Thx!
>>>
>>> I did the test like you with route tc rules to ft callback.
>>>
>>> # ifconfig mlx_p0 172.168.152.75/24 up
>>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>>>
>>> # ip l add dev tun1 type gretap external
>>> # tc qdisc add dev tun1 ingress
>>> # tc qdisc add dev mlx_pf0vf0 ingress
>>>
>>> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower 
>>> skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 
>>> 1000 nocsum pipe action mirred egress redirect dev tun1
>>>
>>>
>>> In The vm:
>>> # ifconfig eth0 10.0.0.75/24 up
>>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>>>
>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>
>>> The syn packets can be offloaded successfully.
>>>
>>> # # tc -s filter ls dev mlx_pf0vf0 ingress
>>> filter protocol ip pref 2 flower chain 0
>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>    eth_type ipv4
>>>    skip_sw
>>>    in_hw in_hw_count 1
>>>     action order 1: tunnel_key  set
>>>     src_ip 0.0.0.0
>>>     dst_ip 172.168.152.241
>>>     key_id 1000
>>>     nocsum pipe
>>>      index 1 ref 1 bind 1 installed 252 sec used 252 sec
>>>     Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>     action order 2: mirred (Egress Redirect to device tun1) stolen
>>>       index 1 ref 1 bind 1 installed 252 sec used 110 sec
>>>       Action statistics:
>>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>     Sent software 0 bytes 0 pkt
>>>     Sent hardware 3420 bytes 11 pkt
>>>     backlog 0b 0p requeues 0
>>>
>>> But Then I add another decap filter on tun1:
>>>
>>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id 
>>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action 
>>> mirred egress redirect dev mlx_pf0vf0
>>>
>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>
>>> The syn packets can't be offloaded. The tc filter counter is also not 
>>> increase.
>>>
>>>
>>> # tc -s filter ls dev mlx_pf0vf0 ingress
>>> filter protocol ip pref 2 flower chain 0
>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>    eth_type ipv4
>>>    skip_sw
>>>    in_hw in_hw_count 1
>>>     action order 1: tunnel_key  set
>>>     src_ip 0.0.0.0
>>>     dst_ip 172.168.152.241
>>>     key_id 1000
>>>     nocsum pipe
>>>      index 1 ref 1 bind 1 installed 320 sec used 320 sec
>>>     Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>     action order 2: mirred (Egress Redirect to device tun1) stolen
>>>       index 1 ref 1 bind 1 installed 320 sec used 178 sec
>>>       Action statistics:
>>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>     Sent software 0 bytes 0 pkt
>>>     Sent hardware 3420 bytes 11 pkt
>>>     backlog 0b 0p requeues 0
>>>
>>> # tc -s filter ls dev tun1 ingress
>>> filter protocol ip pref 2 flower chain 0
>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>    eth_type ipv4
>>>    enc_src_ip 172.168.152.241
>>>    enc_key_id 1000
>>>    in_hw in_hw_count 1
>>>     action order 1: tunnel_key  unset pipe
>>>      index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>     Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>     action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>>       index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>       Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>
>>> So there maybe some problem for ft callback setup. When there is 
>>> another reverse
>>> decap rule add in tunnel device, The encap rule will not offloaded 
>>> the packets.
>>>
>>> Expect your help Thx!
>>>
>>>
>>> BR
>>> wenxu
>>>
>>>
>>>
>>>
>>>
>>>
>> Hi I reproduced it.
>>
>> I'll find the reason and fix for it and get back to you soon.
>>
>> We are planing on expanding our chain and prio supported range, and in 
>> that we also move the FT offload code a bit.
>>
>> If what I think happens happened it would fix it anyway.
>>
>> Thanks.
>>