From: wenxu <wenxu@ucloud.cn>
To: Paul Blakey <paulb@mellanox.com>
Cc: "netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: Question about flow table offload in mlx5e
Date: Mon, 9 Dec 2019 18:48:52 +0800 [thread overview]
Message-ID: <52204ad2-e700-2e45-d562-7ceaa0e08b1b@ucloud.cn> (raw)
In-Reply-To: <01602d82-b46c-07d2-dea7-baa3545db80f@mellanox.com>
On 12/9/2019 3:48 PM, Paul Blakey wrote:
> On 12/9/2019 5:18 AM, wenxu wrote:
>> Hi Paul,
>>
>>
>> Thanks for your fix, I will test it.
>>
>> On 12/8/2019 5:39 PM, Paul Blakey wrote:
>>> Here's the temp fix:
>>>
>>>
>>> The problem is TC + FT offload, and this revealed a bug in the driver.
>>>
>>> For the tunnel test, we changed tc block offload to ft callback, and
>>> didn't change the indr block offload.
>>>
>>> So the tunnel unset rule is offloaded from indr tc callback (it's
>>> indirect because it's on tun1 device):
>>>
>>> mlx5e_rep_indr_setup_block_cb
>> Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup?
>>
>> Or all indr setup through TC offload?
> Adding a "mlx5e_rep_indr_setup_ft_cb" with the correct flags (FT) and
> (EGRESS) should work as well, but this is just a test...
>
> For upstream, I see you're talking with pablo about the pending tunnel
> offload support.
I test your fix patch, it is work now.
But if there is no your fix patch and I only add a mlx5e_rep_indr_setup_ft_cb
with the FT flags, it also can't work (both rep_setup and indr_rep_setup with
FT flags. I can make it work with adding your fix patch. It seems not a problem of
TC + FT offload?
>
>
>>> this offloaded the rule to hardware in the TC domain.
>>>
>>> Now the tunnel set (encap) rule was offloaded to FT domain.
>>>
>>>
>>> Since TC comes before FT in software, we should have connected the miss
>>> on TC domain to FT domain,
>>>
>>> but this didn't work.
>>>
>>> The below fix should fix that connection:
>>>
>>>
>>> ------------------------------------------
>>>
>>>
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>>> @@ -763,9 +763,6 @@ static struct mlx5_flow_table
>>> *find_closest_ft_recursive(struct fs_node *root,
>>> struct fs_node *iter = list_entry(start, struct fs_node, list);
>>> struct mlx5_flow_table *ft = NULL;
>>>
>>> - if (!root || root->type == FS_TYPE_PRIO_CHAINS)
>>> - return NULL;
>>> -
>>> list_for_each_advance_continue(iter, &root->children, reverse) {
>>> if (iter->type == FS_TYPE_FLOW_TABLE) {
>>> fs_get_obj(ft, iter);
>>> @@ -792,7 +789,10 @@ static struct mlx5_flow_table
>>> *find_closest_ft(struct fs_prio *prio, bool revers
>>> parent = prio->node.parent;
>>> curr_node = &prio->node;
>>> while (!ft && parent) {
>>> - ft = find_closest_ft_recursive(parent, &curr_node->list,
>>> reverse);
>>> + if (parent->type != FS_TYPE_PRIO_CHAINS)
>>> + ft = find_closest_ft_recursive(parent,
>>> + &curr_node->list,
>>> + reverse);
>>> curr_node = parent;
>>> parent = curr_node->parent;
>>> }
>>>
>>>
>>> ------------------------------------------
>>>
>>>
>>> I will do this patch for upstream if needed after our last patchset that
>>> also touched this area.
>>>
>>> Thanks,
>>>
>>> Paul.
>>>
>>>
>>>
>>>
>>> On 12/5/2019 5:17 PM, Paul Blakey wrote:
>>>> On 12/2/2019 5:37 AM, wenxu wrote:
>>>>
>>>>> Hi Paul,
>>>>>
>>>>>
>>>>> Sorry for trouble you again. I think it is a problem in ft callback.
>>>>>
>>>>> Can your help me fix it. Thx!
>>>>>
>>>>> I did the test like you with route tc rules to ft callback.
>>>>>
>>>>> # ifconfig mlx_p0 172.168.152.75/24 up
>>>>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>>>>>
>>>>> # ip l add dev tun1 type gretap external
>>>>> # tc qdisc add dev tun1 ingress
>>>>> # tc qdisc add dev mlx_pf0vf0 ingress
>>>>>
>>>>> # tc filter add dev mlx_pf0vf0 pref 2 ingress protocol ip flower
>>>>> skip_sw action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id
>>>>> 1000 nocsum pipe action mirred egress redirect dev tun1
>>>>>
>>>>>
>>>>> In The vm:
>>>>> # ifconfig eth0 10.0.0.75/24 up
>>>>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>>>>>
>>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>>
>>>>> The syn packets can be offloaded successfully.
>>>>>
>>>>> # # tc -s filter ls dev mlx_pf0vf0 ingress
>>>>> filter protocol ip pref 2 flower chain 0
>>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>> eth_type ipv4
>>>>> skip_sw
>>>>> in_hw in_hw_count 1
>>>>> action order 1: tunnel_key set
>>>>> src_ip 0.0.0.0
>>>>> dst_ip 172.168.152.241
>>>>> key_id 1000
>>>>> nocsum pipe
>>>>> index 1 ref 1 bind 1 installed 252 sec used 252 sec
>>>>> Action statistics:
>>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>> backlog 0b 0p requeues 0
>>>>>
>>>>> action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>> index 1 ref 1 bind 1 installed 252 sec used 110 sec
>>>>> Action statistics:
>>>>> Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>> Sent software 0 bytes 0 pkt
>>>>> Sent hardware 3420 bytes 11 pkt
>>>>> backlog 0b 0p requeues 0
>>>>>
>>>>> But Then I add another decap filter on tun1:
>>>>>
>>>>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id
>>>>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action
>>>>> mirred egress redirect dev mlx_pf0vf0
>>>>>
>>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>>
>>>>> The syn packets can't be offloaded. The tc filter counter is also not
>>>>> increase.
>>>>>
>>>>>
>>>>> # tc -s filter ls dev mlx_pf0vf0 ingress
>>>>> filter protocol ip pref 2 flower chain 0
>>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>> eth_type ipv4
>>>>> skip_sw
>>>>> in_hw in_hw_count 1
>>>>> action order 1: tunnel_key set
>>>>> src_ip 0.0.0.0
>>>>> dst_ip 172.168.152.241
>>>>> key_id 1000
>>>>> nocsum pipe
>>>>> index 1 ref 1 bind 1 installed 320 sec used 320 sec
>>>>> Action statistics:
>>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>> backlog 0b 0p requeues 0
>>>>>
>>>>> action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>> index 1 ref 1 bind 1 installed 320 sec used 178 sec
>>>>> Action statistics:
>>>>> Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>> Sent software 0 bytes 0 pkt
>>>>> Sent hardware 3420 bytes 11 pkt
>>>>> backlog 0b 0p requeues 0
>>>>>
>>>>> # tc -s filter ls dev tun1 ingress
>>>>> filter protocol ip pref 2 flower chain 0
>>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>> eth_type ipv4
>>>>> enc_src_ip 172.168.152.241
>>>>> enc_key_id 1000
>>>>> in_hw in_hw_count 1
>>>>> action order 1: tunnel_key unset pipe
>>>>> index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>> Action statistics:
>>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>> backlog 0b 0p requeues 0
>>>>>
>>>>> action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>>>> index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>> Action statistics:
>>>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>> backlog 0b 0p requeues 0
>>>>>
>>>>>
>>>>> So there maybe some problem for ft callback setup. When there is
>>>>> another reverse
>>>>> decap rule add in tunnel device, The encap rule will not offloaded
>>>>> the packets.
>>>>>
>>>>> Expect your help Thx!
>>>>>
>>>>>
>>>>> BR
>>>>> wenxu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Hi I reproduced it.
>>>>
>>>> I'll find the reason and fix for it and get back to you soon.
>>>>
>>>> We are planing on expanding our chain and prio supported range, and in
>>>> that we also move the FT offload code a bit.
>>>>
>>>> If what I think happens happened it would fix it anyway.
>>>>
>>>> Thanks.
>>>>
next prev parent reply other threads:[~2019-12-09 10:49 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-19 7:08 [PATCH net-next] ip_gre: Make none-tun-dst gre tunnel keep tunnel info wenxu
2019-11-20 0:39 ` David Miller
2019-11-21 7:30 ` Question about flow table offload in mlx5e wenxu
2019-11-21 7:42 ` Paul Blakey
2019-11-21 8:28 ` wenxu
2019-11-21 11:39 ` Paul Blakey
2019-11-21 11:40 ` Paul Blakey
2019-11-21 12:35 ` wenxu
2019-11-21 13:05 ` Paul Blakey
2019-11-21 13:39 ` wenxu
2019-11-22 6:12 ` wenxu
[not found] ` <64285654-bc9a-c76e-5875-dc6e434dc4d4@ucloud.cn>
2019-11-24 8:46 ` Paul Blakey
2019-11-24 11:14 ` wenxu
2019-11-26 8:18 ` wenxu
[not found] ` <84874b42-c525-2149-539d-e7510d15f6a6@mellanox.com>
2019-11-27 12:16 ` wenxu
2019-11-27 12:16 ` wenxu
2019-11-27 13:11 ` Paul Blakey
2019-11-27 13:20 ` Paul Blakey
2019-11-27 13:45 ` wenxu
2019-12-02 3:37 ` wenxu
2019-12-05 15:17 ` Paul Blakey
2019-12-08 9:39 ` Paul Blakey
2019-12-09 3:18 ` wenxu
2019-12-09 7:48 ` Paul Blakey
2019-12-09 10:48 ` wenxu [this message]
2019-12-10 6:53 ` wenxu
2019-11-28 5:03 ` Bad performance for VF outgoing in offloaded mode wenxu
2019-12-04 13:50 ` Roi Dayan
2019-12-04 14:32 ` wenxu
2019-12-05 3:41 ` wenxu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=52204ad2-e700-2e45-d562-7ceaa0e08b1b@ucloud.cn \
--to=wenxu@ucloud.cn \
--cc=netdev@vger.kernel.org \
--cc=paulb@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).