From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8EB5C2D0BF for ; Mon, 9 Dec 2019 03:19:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9C34F2071A for ; Mon, 9 Dec 2019 03:19:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727047AbfLIDTH (ORCPT ); Sun, 8 Dec 2019 22:19:07 -0500 Received: from m9784.mail.qiye.163.com ([220.181.97.84]:30767 "EHLO m9784.mail.qiye.163.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726748AbfLIDTG (ORCPT ); Sun, 8 Dec 2019 22:19:06 -0500 Received: from [192.168.188.14] (unknown [120.132.1.226]) by m9784.mail.qiye.163.com (Hmail) with ESMTPA id 094654193C; Mon, 9 Dec 2019 11:19:03 +0800 (CST) Subject: Re: Question about flow table offload in mlx5e To: Paul Blakey Cc: "netdev@vger.kernel.org" References: <1574147331-31096-1-git-send-email-wenxu@ucloud.cn> <746ba973-3c58-31f8-42ce-db880fd1d8f4@ucloud.cn> <64285654-bc9a-c76e-5875-dc6e434dc4d4@ucloud.cn> <1b13e159-1030-2ea3-f69e-578041504ee6@ucloud.cn> <84874b42-c525-2149-539d-e7510d15f6a6@mellanox.com> <057b0ab1-5ce3-61f0-a59e-1c316e414c84@mellanox.com> <4ecddff0-5ba4-51f7-1544-3d76d43b6b39@mellanox.com> <5ce27064-97ee-a36d-8f20-10a0afe739cf@ucloud.cn> <052c1c18-89cb-53ed-344c-decd4d296db3@mellanox.com> From: wenxu Message-ID: Date: Mon, 9 Dec 2019 11:18:59 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <052c1c18-89cb-53ed-344c-decd4d296db3@mellanox.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-HM-Spam-Status: e1kfGhgUHx5ZQUtXWQgYFAkeWUFZVkpVSEJOS0tLSktLSkhOQ1lXWShZQU lCN1dZLVlBSVdZCQ4XHghZQVk1NCk2OjckKS43PlkG X-HM-Sender-Digest: e1kMHhlZQR0aFwgeV1kSHx4VD1lBWUc6PzY6ARw5ATgwLUocKyFMODBR E1YKC01VSlVKTkxOQ01KTk9ISkNOVTMWGhIXVQweFQMOOw4YFxQOH1UYFUVZV1kSC1lBWUpJS1VK SElVSlVJSU1ZV1kIAVlBSktITEw3Bg++ X-HM-Tid: 0a6ee8aae0b12086kuqy094654193c Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Hi  Paul, Thanks for your fix, I will test it. On 12/8/2019 5:39 PM, Paul Blakey wrote: > Here's the temp fix: > > > The problem is TC + FT offload, and this revealed a bug in the driver. > > For the tunnel test, we changed tc block offload to ft callback, and > didn't change the indr block offload. > > So the tunnel unset rule is offloaded from indr tc callback (it's > indirect because it's on tun1 device): > > mlx5e_rep_indr_setup_block_cb Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup? Or all indr setup through TC offload? > > this offloaded the rule to hardware in the TC domain. > > Now the tunnel set (encap) rule was offloaded to FT domain. > > > Since TC comes before FT in software, we should have connected the miss > on TC domain to FT domain, > > but this didn't work. > > The below fix should fix that connection: > > > ------------------------------------------ > > > --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c > +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c > @@ -763,9 +763,6 @@ static struct mlx5_flow_table > *find_closest_ft_recursive(struct fs_node  *root, >         struct fs_node *iter = list_entry(start, struct fs_node, list); >         struct mlx5_flow_table *ft = NULL; > > -       if (!root || root->type == FS_TYPE_PRIO_CHAINS) > -               return NULL; > - >         list_for_each_advance_continue(iter, &root->children, reverse) { >                 if (iter->type == FS_TYPE_FLOW_TABLE) { >                         fs_get_obj(ft, iter); > @@ -792,7 +789,10 @@ static struct mlx5_flow_table > *find_closest_ft(struct fs_prio *prio, bool revers >         parent = prio->node.parent; >         curr_node = &prio->node; >         while (!ft && parent) { > -               ft = find_closest_ft_recursive(parent, &curr_node->list, > reverse); > +               if (parent->type != FS_TYPE_PRIO_CHAINS) > +                       ft = find_closest_ft_recursive(parent, > + &curr_node->list, > +                                                      reverse); >                 curr_node = parent; >                 parent = curr_node->parent; >         } > > > ------------------------------------------ > > > I will do this patch for upstream if needed after our last patchset that > also touched this area. > > Thanks, > > Paul. > > > > > On 12/5/2019 5:17 PM, Paul Blakey wrote: >> On 12/2/2019 5:37 AM, wenxu wrote: >> >>> Hi Paul, >>> >>> >>> Sorry for trouble you again. I think it is a problem in ft callback. >>> >>> Can your help me fix it. Thx! >>> >>> I did the test like you with route tc rules to ft callback. >>> >>> # ifconfig mlx_p0 172.168.152.75/24 up >>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0 >>> >>> # ip l add dev tun1 type gretap external >>> # tc qdisc add dev tun1 ingress >>> # tc qdisc add dev mlx_pf0vf0 ingress >>> >>> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower >>> skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id >>> 1000 nocsum pipe action mirred egress redirect dev tun1 >>> >>> >>> In The vm: >>> # ifconfig eth0 10.0.0.75/24 up >>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0 >>> >>> # iperf -c 10.0.0.77 -t 100 -i 2 >>> >>> The syn packets can be offloaded successfully. >>> >>> # # tc -s filter ls dev mlx_pf0vf0 ingress >>> filter protocol ip pref 2 flower chain 0 >>> filter protocol ip pref 2 flower chain 0 handle 0x1 >>>    eth_type ipv4 >>>    skip_sw >>>    in_hw in_hw_count 1 >>>     action order 1: tunnel_key  set >>>     src_ip 0.0.0.0 >>>     dst_ip 172.168.152.241 >>>     key_id 1000 >>>     nocsum pipe >>>      index 1 ref 1 bind 1 installed 252 sec used 252 sec >>>     Action statistics: >>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) >>>     backlog 0b 0p requeues 0 >>> >>>     action order 2: mirred (Egress Redirect to device tun1) stolen >>>       index 1 ref 1 bind 1 installed 252 sec used 110 sec >>>       Action statistics: >>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0) >>>     Sent software 0 bytes 0 pkt >>>     Sent hardware 3420 bytes 11 pkt >>>     backlog 0b 0p requeues 0 >>> >>> But Then I add another decap filter on tun1: >>> >>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id >>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action >>> mirred egress redirect dev mlx_pf0vf0 >>> >>> # iperf -c 10.0.0.77 -t 100 -i 2 >>> >>> The syn packets can't be offloaded. The tc filter counter is also not >>> increase. >>> >>> >>> # tc -s filter ls dev mlx_pf0vf0 ingress >>> filter protocol ip pref 2 flower chain 0 >>> filter protocol ip pref 2 flower chain 0 handle 0x1 >>>    eth_type ipv4 >>>    skip_sw >>>    in_hw in_hw_count 1 >>>     action order 1: tunnel_key  set >>>     src_ip 0.0.0.0 >>>     dst_ip 172.168.152.241 >>>     key_id 1000 >>>     nocsum pipe >>>      index 1 ref 1 bind 1 installed 320 sec used 320 sec >>>     Action statistics: >>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) >>>     backlog 0b 0p requeues 0 >>> >>>     action order 2: mirred (Egress Redirect to device tun1) stolen >>>       index 1 ref 1 bind 1 installed 320 sec used 178 sec >>>       Action statistics: >>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0) >>>     Sent software 0 bytes 0 pkt >>>     Sent hardware 3420 bytes 11 pkt >>>     backlog 0b 0p requeues 0 >>> >>> # tc -s filter ls dev tun1 ingress >>> filter protocol ip pref 2 flower chain 0 >>> filter protocol ip pref 2 flower chain 0 handle 0x1 >>>    eth_type ipv4 >>>    enc_src_ip 172.168.152.241 >>>    enc_key_id 1000 >>>    in_hw in_hw_count 1 >>>     action order 1: tunnel_key  unset pipe >>>      index 2 ref 1 bind 1 installed 391 sec used 391 sec >>>     Action statistics: >>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) >>>     backlog 0b 0p requeues 0 >>> >>>     action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen >>>       index 2 ref 1 bind 1 installed 391 sec used 391 sec >>>       Action statistics: >>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) >>>     backlog 0b 0p requeues 0 >>> >>> >>> So there maybe some problem for ft callback setup. When there is >>> another reverse >>> decap rule add in tunnel device, The encap rule will not offloaded >>> the packets. >>> >>> Expect your help Thx! >>> >>> >>> BR >>> wenxu >>> >>> >>> >>> >>> >>> >> Hi I reproduced it. >> >> I'll find the reason and fix for it and get back to you soon. >> >> We are planing on expanding our chain and prio supported range, and in >> that we also move the FT offload code a bit. >> >> If what I think happens happened it would fix it anyway. >> >> Thanks. >>