[PATCH net-next] ip_gre: Make none-tun-dst gre tunnel keep tunnel info

[PATCH net-next] ip_gre: Make none-tun-dst gre tunnel keep tunnel info

[wenxu]

From: wenxu <wenxu@ucloud.cn>

Currently only collect_md gre tunnel keep tunnel info.
But the nono-tun-dst gre tunnel already can send packte through
lwtunnel.

For non-tun-dst gre tunnel should keep the tunnel info to make
the arp response can send success through the tunnel_info in
iptunnel_metadata_reply.

The following is the test script:

ip netns add cl
ip l add dev vethc type veth peer name eth0 netns cl

ifconfig vethc 172.168.0.7/24 up
ip l add dev tun1000 type gretap key 1000

ip link add user1000 type vrf table 1
ip l set user1000 up
ip l set dev tun1000 master user1000
ifconfig tun1000 10.0.1.1/24 up

ip netns exec cl ifconfig eth0 172.168.0.17/24 up
ip netns exec cl ip l add dev tun type gretap local 172.168.0.17 remote 172.168.0.7 key 1000
ip netns exec cl ifconfig tun 10.0.1.7/24 up
ip r r 10.0.1.7 encap ip id 1000 dst 172.168.0.17 key dev tun1000 table 1

With this patch
ip netns exec cl ping 10.0.1.1 can success

Signed-off-by: wenxu <wenxu@ucloud.cn>
---
 net/ipv4/ip_gre.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
index 10636fb..572b630 100644
--- a/net/ipv4/ip_gre.c
+++ b/net/ipv4/ip_gre.c
@@ -340,6 +340,8 @@ static int __ipgre_rcv(struct sk_buff *skb, const struct tnl_ptk_info *tpi,
 				  iph->saddr, iph->daddr, tpi->key);
 
 	if (tunnel) {
+		const struct iphdr *tnl_params;
+
 		if (__iptunnel_pull_header(skb, hdr_len, tpi->proto,
 					   raw_proto, false) < 0)
 			goto drop;
@@ -348,7 +350,9 @@ static int __ipgre_rcv(struct sk_buff *skb, const struct tnl_ptk_info *tpi,
 			skb_pop_mac_header(skb);
 		else
 			skb_reset_mac_header(skb);
-		if (tunnel->collect_md) {
+
+		tnl_params = &tunnel->parms.iph;
+		if (tunnel->collect_md || tnl_params->daddr == 0) {
 			__be16 flags;
 			__be64 tun_id;
 
-- 
1.8.3.1

Re: [PATCH net-next] ip_gre: Make none-tun-dst gre tunnel keep tunnel info

[David Miller]

From: wenxu@ucloud.cn
Date: Tue, 19 Nov 2019 15:08:51 +0800

> From: wenxu <wenxu@ucloud.cn>
> 
> Currently only collect_md gre tunnel keep tunnel info.
> But the nono-tun-dst gre tunnel already can send packte through
> lwtunnel.
> 
> For non-tun-dst gre tunnel should keep the tunnel info to make
> the arp response can send success through the tunnel_info in
> iptunnel_metadata_reply.

I know that English is not your native language, but there are many
typos in here and I do not understand from your description how all
of this works and what needs to be fixed.

Please try explain things more clearly, showing how collect_md works
for these tunnel types, exactly, compared to non-tun-dst tunnels.

Thank you.

Question about flow table offload in mlx5e

[wenxu]

Hi  paul,

The flow table offload in the mlx5e is based on TC_SETUP_FT.


It is almost the same as TC_SETUP_BLOCK.

It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change cls_flower.common.chain_index = FDB_FT_CHAIN;

In following codes line 1380 and 1392

1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void *type_data,
1369                                  void *cb_priv)
1370 {
1371         struct flow_cls_offload *f = type_data;
1372         struct flow_cls_offload cls_flower;
1373         struct mlx5e_priv *priv = cb_priv;
1374         struct mlx5_eswitch *esw;
1375         unsigned long flags;
1376         int err;
1377
1378         flags = MLX5_TC_FLAG(INGRESS) |
1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
1380                 MLX5_TC_FLAG(FT_OFFLOAD);
1381         esw = priv->mdev->priv.eswitch;
1382
1383         switch (type) {
1384         case TC_SETUP_CLSFLOWER:
1385                 if (!mlx5_eswitch_prios_supported(esw) || f->common.chain_index)
1386                         return -EOPNOTSUPP;
1387
1388                 /* Re-use tc offload path by moving the ft flow to the
1389                  * reserved ft chain.
1390                  */
1391                 memcpy(&cls_flower, f, sizeof(*f));
1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower, flags);
1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));


I want to add tunnel offload support in the flow table, I  add some patches in nf_flow_table_offload.

Also add the indr setup support in the mlx driver. And Now I can  flow table offload with decap.


But I meet a problem with the encap.  The encap rule can be added in hardware  successfully But it can't be offloaded.

But I think the rule I added is correct.  If I mask the line 1392. The rule also can be add success and can be offloaded.

So there are some limit for encap operation for FT_OFFLOAD in FDB_FT_CHAIN?


BR

wenxu

RE: Question about flow table offload in mlx5e

[Paul Blakey]

Hi,

The original design was the block setup to use TC_SETUP_FT type, and the tc event type to be case TC_SETUP_CLSFLOWER.
We will post a patch to change that. I would advise to wait till we fix that 😊
I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it the nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and not TC_SETUP_FT.

In our driver en_rep.c we have:
>-------switch (type) {
>-------case TC_SETUP_BLOCK:
>------->-------return flow_block_cb_setup_simple(type_data,
>------->------->------->------->------->-------  &mlx5e_rep_block_tc_cb_list,
>------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
>------->------->------->------->------->-------  priv, priv, true);
>-------case TC_SETUP_FT:
>------->-------return flow_block_cb_setup_simple(type_data,
>------->------->------->------->------->-------  &mlx5e_rep_block_ft_cb_list,
>------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
>------->------->------->------->------->-------  priv, priv, true);
>-------default:
>------->-------return -EOPNOTSUPP;
>-------}

In nf_flow_table_offload.c:
>-------bo.binder_type>-= FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>-------bo.extack>------= &extack;
>-------INIT_LIST_HEAD(&bo.cb_list);

>-------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK, &bo);
>-------if (err < 0)
>------->-------return err;

>-------return nf_flow_table_block_setup(flowtable, &bo, cmd);
}
EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);


So unless you changed that as well, you should have gotten to mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.

Regarding the encap action, there should be no difference on which chain the rule is on.


> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn>
> Sent: Thursday, November 21, 2019 9:30 AM
> To: Paul Blakey <paulb@mellanox.com>
> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> <markb@mellanox.com>
> Subject: Question about flow table offload in mlx5e
> 
> Hi  paul,
> 
> The flow table offload in the mlx5e is based on TC_SETUP_FT.
> 
> 
> It is almost the same as TC_SETUP_BLOCK.
> 
> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
> cls_flower.common.chain_index = FDB_FT_CHAIN;
> 
> In following codes line 1380 and 1392
> 
> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
> *type_data,
> 1369                                  void *cb_priv)
> 1370 {
> 1371         struct flow_cls_offload *f = type_data;
> 1372         struct flow_cls_offload cls_flower;
> 1373         struct mlx5e_priv *priv = cb_priv;
> 1374         struct mlx5_eswitch *esw;
> 1375         unsigned long flags;
> 1376         int err;
> 1377
> 1378         flags = MLX5_TC_FLAG(INGRESS) |
> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
> 1381         esw = priv->mdev->priv.eswitch;
> 1382
> 1383         switch (type) {
> 1384         case TC_SETUP_CLSFLOWER:
> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
> >common.chain_index)
> 1386                         return -EOPNOTSUPP;
> 1387
> 1388                 /* Re-use tc offload path by moving the ft flow to the
> 1389                  * reserved ft chain.
> 1390                  */
> 1391                 memcpy(&cls_flower, f, sizeof(*f));
> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower, flags);
> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
> 
> 
> I want to add tunnel offload support in the flow table, I  add some patches in
> nf_flow_table_offload.
> 
> Also add the indr setup support in the mlx driver. And Now I can  flow table
> offload with decap.
> 
> 
> But I meet a problem with the encap.  The encap rule can be added in
> hardware  successfully But it can't be offloaded.
> 
> But I think the rule I added is correct.  If I mask the line 1392. The rule also can
> be add success and can be offloaded.
> 
> So there are some limit for encap operation for FT_OFFLOAD in
> FDB_FT_CHAIN?
> 
> 
> BR
> 
> wenxu
> 

Re: Question about flow table offload in mlx5e

[wenxu]

On 11/21/2019 3:42 PM, Paul Blakey wrote:
> Hi,
>
> The original design was the block setup to use TC_SETUP_FT type, and the tc event type to be case TC_SETUP_CLSFLOWER.
> We will post a patch to change that. I would advise to wait till we fix that 😊
> I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it the nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and not TC_SETUP_FT.


Yes I change the TC_SETUP_BLOCK to TC_SETUP_FT in the nf_flow_table_offload_setup.

Two fixes patch provide:

http://patchwork.ozlabs.org/patch/1197818/

http://patchwork.ozlabs.org/patch/1197876/

So this change made by me is not correct currently?

>
> In our driver en_rep.c we have:
>> -------switch (type) {
>> -------case TC_SETUP_BLOCK:
>> ------->-------return flow_block_cb_setup_simple(type_data,
>> ------->------->------->------->------->-------  &mlx5e_rep_block_tc_cb_list,
>> ------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
>> ------->------->------->------->------->-------  priv, priv, true);
>> -------case TC_SETUP_FT:
>> ------->-------return flow_block_cb_setup_simple(type_data,
>> ------->------->------->------->------->-------  &mlx5e_rep_block_ft_cb_list,
>> ------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
>> ------->------->------->------->------->-------  priv, priv, true);
>> -------default:
>> ------->-------return -EOPNOTSUPP;
>> -------}
> In nf_flow_table_offload.c:
>> -------bo.binder_type>-= FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>> -------bo.extack>------= &extack;
>> -------INIT_LIST_HEAD(&bo.cb_list);
>> -------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK, &bo);
>> -------if (err < 0)
>> ------->-------return err;
>> -------return nf_flow_table_block_setup(flowtable, &bo, cmd);
> }
> EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>
>
> So unless you changed that as well, you should have gotten to mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.
>
> Regarding the encap action, there should be no difference on which chain the rule is on.


But for the same encap rule can be real offloaded when setup through through TC_SETUP_BLOCK. But TC_SETUP_FT can't.

So it is the problem of TC_SETUP_FT in mlx5e_rep_setup_ft_cb ?

>
>
>> -----Original Message-----
>> From: wenxu <wenxu@ucloud.cn>
>> Sent: Thursday, November 21, 2019 9:30 AM
>> To: Paul Blakey <paulb@mellanox.com>
>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>> <markb@mellanox.com>
>> Subject: Question about flow table offload in mlx5e
>>
>> Hi  paul,
>>
>> The flow table offload in the mlx5e is based on TC_SETUP_FT.
>>
>>
>> It is almost the same as TC_SETUP_BLOCK.
>>
>> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>> cls_flower.common.chain_index = FDB_FT_CHAIN;
>>
>> In following codes line 1380 and 1392
>>
>> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>> *type_data,
>> 1369                                  void *cb_priv)
>> 1370 {
>> 1371         struct flow_cls_offload *f = type_data;
>> 1372         struct flow_cls_offload cls_flower;
>> 1373         struct mlx5e_priv *priv = cb_priv;
>> 1374         struct mlx5_eswitch *esw;
>> 1375         unsigned long flags;
>> 1376         int err;
>> 1377
>> 1378         flags = MLX5_TC_FLAG(INGRESS) |
>> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
>> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
>> 1381         esw = priv->mdev->priv.eswitch;
>> 1382
>> 1383         switch (type) {
>> 1384         case TC_SETUP_CLSFLOWER:
>> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
>>> common.chain_index)
>> 1386                         return -EOPNOTSUPP;
>> 1387
>> 1388                 /* Re-use tc offload path by moving the ft flow to the
>> 1389                  * reserved ft chain.
>> 1390                  */
>> 1391                 memcpy(&cls_flower, f, sizeof(*f));
>> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
>> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower, flags);
>> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
>>
>>
>> I want to add tunnel offload support in the flow table, I  add some patches in
>> nf_flow_table_offload.
>>
>> Also add the indr setup support in the mlx driver. And Now I can  flow table
>> offload with decap.
>>
>>
>> But I meet a problem with the encap.  The encap rule can be added in
>> hardware  successfully But it can't be offloaded.
>>
>> But I think the rule I added is correct.  If I mask the line 1392. The rule also can
>> be add success and can be offloaded.
>>
>> So there are some limit for encap operation for FT_OFFLOAD in
>> FDB_FT_CHAIN?
>>
>>
>> BR
>>
>> wenxu
>>

RE: Question about flow table offload in mlx5e

[Paul Blakey]

They are good fixes, exactly what we had when we tested this, thanks.

Regarding encap, I don't know what changes you did, how does the encap rule look? Is it a FWD to vxlan device? If not it should be, as our driver expects that.

I tried it on my setup via tc, by changing the callback of tc (mlx5e_rep_setup_tc_cb) to that of ft (mlx5e_rep_setup_ft_cb),
and testing a vxlan encap rule:
sudo tc qdisc add dev ens1f0_0 ingress
sudo ifconfig ens1f0 7.7.7.7/24 up
sudo ip link add name vxlan0 type vxlan dev ens1f0 remote 7.7.7.8 dstport 4789 external
sudo ifconfig vxlan0 up
sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0 protocol ip flower dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action mirred egress redirect dev vxlan

then tc show:
filter protocol ip pref 1 flower chain 0 handle 0x1 dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw in_hw in_hw_count 1
        tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id 1234 dst_port 4789 csum pipe
        Stats: used 119 sec      0 pkt
        mirred (Egress Redirect to device vxlan0)
        Stats: used 119 sec      0 pkt



> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn>
> Sent: Thursday, November 21, 2019 10:29 AM
> To: Paul Blakey <paulb@mellanox.com>
> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> <markb@mellanox.com>
> Subject: Re: Question about flow table offload in mlx5e
> 
> 
> On 11/21/2019 3:42 PM, Paul Blakey wrote:
> > Hi,
> >
> > The original design was the block setup to use TC_SETUP_FT type, and the
> tc event type to be case TC_SETUP_CLSFLOWER.
> > We will post a patch to change that. I would advise to wait till we fix that
> 😊
> > I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it the
> nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and not
> TC_SETUP_FT.
> 
> 
> Yes I change the TC_SETUP_BLOCK to TC_SETUP_FT in the
> nf_flow_table_offload_setup.
> 
> Two fixes patch provide:
> 
> http://patchwork.ozlabs.org/patch/1197818/
> 
> http://patchwork.ozlabs.org/patch/1197876/
> 
> So this change made by me is not correct currently?
> 
> >
> > In our driver en_rep.c we have:
> >> -------switch (type) {
> >> -------case TC_SETUP_BLOCK:
> >> ------->-------return flow_block_cb_setup_simple(type_data,
> >> ------->------->------->------->------->-------  &mlx5e_rep_block_tc_cb_list,
> >> ------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
> >> ------->------->------->------->------->-------  priv, priv, true);
> >> -------case TC_SETUP_FT:
> >> ------->-------return flow_block_cb_setup_simple(type_data,
> >> ------->------->------->------->------->-------  &mlx5e_rep_block_ft_cb_list,
> >> ------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
> >> ------->------->------->------->------->-------  priv, priv, true);
> >> -------default:
> >> ------->-------return -EOPNOTSUPP;
> >> -------}
> > In nf_flow_table_offload.c:
> >> -------bo.binder_type>-= FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
> >> -------bo.extack>------= &extack;
> >> -------INIT_LIST_HEAD(&bo.cb_list);
> >> -------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK,
> &bo);
> >> -------if (err < 0)
> >> ------->-------return err;
> >> -------return nf_flow_table_block_setup(flowtable, &bo, cmd);
> > }
> > EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
> >
> >
> > So unless you changed that as well, you should have gotten to
> mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.
> >
> > Regarding the encap action, there should be no difference on which chain
> the rule is on.
> 
> 
> But for the same encap rule can be real offloaded when setup through
> through TC_SETUP_BLOCK. But TC_SETUP_FT can't.
> 
> So it is the problem of TC_SETUP_FT in mlx5e_rep_setup_ft_cb ?
> 
> >
> >
> >> -----Original Message-----
> >> From: wenxu <wenxu@ucloud.cn>
> >> Sent: Thursday, November 21, 2019 9:30 AM
> >> To: Paul Blakey <paulb@mellanox.com>
> >> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> >> <markb@mellanox.com>
> >> Subject: Question about flow table offload in mlx5e
> >>
> >> Hi  paul,
> >>
> >> The flow table offload in the mlx5e is based on TC_SETUP_FT.
> >>
> >>
> >> It is almost the same as TC_SETUP_BLOCK.
> >>
> >> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
> >> cls_flower.common.chain_index = FDB_FT_CHAIN;
> >>
> >> In following codes line 1380 and 1392
> >>
> >> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
> >> *type_data,
> >> 1369                                  void *cb_priv)
> >> 1370 {
> >> 1371         struct flow_cls_offload *f = type_data;
> >> 1372         struct flow_cls_offload cls_flower;
> >> 1373         struct mlx5e_priv *priv = cb_priv;
> >> 1374         struct mlx5_eswitch *esw;
> >> 1375         unsigned long flags;
> >> 1376         int err;
> >> 1377
> >> 1378         flags = MLX5_TC_FLAG(INGRESS) |
> >> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
> >> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
> >> 1381         esw = priv->mdev->priv.eswitch;
> >> 1382
> >> 1383         switch (type) {
> >> 1384         case TC_SETUP_CLSFLOWER:
> >> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
> >>> common.chain_index)
> >> 1386                         return -EOPNOTSUPP;
> >> 1387
> >> 1388                 /* Re-use tc offload path by moving the ft flow to the
> >> 1389                  * reserved ft chain.
> >> 1390                  */
> >> 1391                 memcpy(&cls_flower, f, sizeof(*f));
> >> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
> >> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
> flags);
> >> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
> >>
> >>
> >> I want to add tunnel offload support in the flow table, I  add some patches
> in
> >> nf_flow_table_offload.
> >>
> >> Also add the indr setup support in the mlx driver. And Now I can  flow
> table
> >> offload with decap.
> >>
> >>
> >> But I meet a problem with the encap.  The encap rule can be added in
> >> hardware  successfully But it can't be offloaded.
> >>
> >> But I think the rule I added is correct.  If I mask the line 1392. The rule also
> can
> >> be add success and can be offloaded.
> >>
> >> So there are some limit for encap operation for FT_OFFLOAD in
> >> FDB_FT_CHAIN?
> >>
> >>
> >> BR
> >>
> >> wenxu
> >>

RE: Question about flow table offload in mlx5e

[Paul Blakey]

Also, can u CC me on those patches? Thanks.

> -----Original Message-----
> From: Paul Blakey
> Sent: Thursday, November 21, 2019 1:40 PM
> To: wenxu <wenxu@ucloud.cn>
> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> <markb@mellanox.com>
> Subject: RE: Question about flow table offload in mlx5e
> 
> They are good fixes, exactly what we had when we tested this, thanks.
> 
> Regarding encap, I don't know what changes you did, how does the encap
> rule look? Is it a FWD to vxlan device? If not it should be, as our driver
> expects that.
> 
> I tried it on my setup via tc, by changing the callback of tc
> (mlx5e_rep_setup_tc_cb) to that of ft (mlx5e_rep_setup_ft_cb),
> and testing a vxlan encap rule:
> sudo tc qdisc add dev ens1f0_0 ingress
> sudo ifconfig ens1f0 7.7.7.7/24 up
> sudo ip link add name vxlan0 type vxlan dev ens1f0 remote 7.7.7.8 dstport
> 4789 external
> sudo ifconfig vxlan0 up
> sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0 protocol ip flower
> dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action tunnel_key set
> src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action mirred egress
> redirect dev vxlan
> 
> then tc show:
> filter protocol ip pref 1 flower chain 0 handle 0x1 dst_mac aa:bb:cc:dd:ee:ff
> ip_proto udp skip_sw in_hw in_hw_count 1
>         tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id 1234 dst_port 4789
> csum pipe
>         Stats: used 119 sec      0 pkt
>         mirred (Egress Redirect to device vxlan0)
>         Stats: used 119 sec      0 pkt
> 
> 
> 
> > -----Original Message-----
> > From: wenxu <wenxu@ucloud.cn>
> > Sent: Thursday, November 21, 2019 10:29 AM
> > To: Paul Blakey <paulb@mellanox.com>
> > Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> > <markb@mellanox.com>
> > Subject: Re: Question about flow table offload in mlx5e
> >
> >
> > On 11/21/2019 3:42 PM, Paul Blakey wrote:
> > > Hi,
> > >
> > > The original design was the block setup to use TC_SETUP_FT type, and
> the
> > tc event type to be case TC_SETUP_CLSFLOWER.
> > > We will post a patch to change that. I would advise to wait till we fix that
> > 😊
> > > I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it
> the
> > nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and
> not
> > TC_SETUP_FT.
> >
> >
> > Yes I change the TC_SETUP_BLOCK to TC_SETUP_FT in the
> > nf_flow_table_offload_setup.
> >
> > Two fixes patch provide:
> >
> > http://patchwork.ozlabs.org/patch/1197818/
> >
> > http://patchwork.ozlabs.org/patch/1197876/
> >
> > So this change made by me is not correct currently?
> >
> > >
> > > In our driver en_rep.c we have:
> > >> -------switch (type) {
> > >> -------case TC_SETUP_BLOCK:
> > >> ------->-------return flow_block_cb_setup_simple(type_data,
> > >> ------->------->------->------->------->-------
> &mlx5e_rep_block_tc_cb_list,
> > >> ------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
> > >> ------->------->------->------->------->-------  priv, priv, true);
> > >> -------case TC_SETUP_FT:
> > >> ------->-------return flow_block_cb_setup_simple(type_data,
> > >> ------->------->------->------->------->-------
> &mlx5e_rep_block_ft_cb_list,
> > >> ------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
> > >> ------->------->------->------->------->-------  priv, priv, true);
> > >> -------default:
> > >> ------->-------return -EOPNOTSUPP;
> > >> -------}
> > > In nf_flow_table_offload.c:
> > >> -------bo.binder_type>-=
> FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
> > >> -------bo.extack>------= &extack;
> > >> -------INIT_LIST_HEAD(&bo.cb_list);
> > >> -------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK,
> > &bo);
> > >> -------if (err < 0)
> > >> ------->-------return err;
> > >> -------return nf_flow_table_block_setup(flowtable, &bo, cmd);
> > > }
> > > EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
> > >
> > >
> > > So unless you changed that as well, you should have gotten to
> > mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.
> > >
> > > Regarding the encap action, there should be no difference on which chain
> > the rule is on.
> >
> >
> > But for the same encap rule can be real offloaded when setup through
> > through TC_SETUP_BLOCK. But TC_SETUP_FT can't.
> >
> > So it is the problem of TC_SETUP_FT in mlx5e_rep_setup_ft_cb ?
> >
> > >
> > >
> > >> -----Original Message-----
> > >> From: wenxu <wenxu@ucloud.cn>
> > >> Sent: Thursday, November 21, 2019 9:30 AM
> > >> To: Paul Blakey <paulb@mellanox.com>
> > >> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> > >> <markb@mellanox.com>
> > >> Subject: Question about flow table offload in mlx5e
> > >>
> > >> Hi  paul,
> > >>
> > >> The flow table offload in the mlx5e is based on TC_SETUP_FT.
> > >>
> > >>
> > >> It is almost the same as TC_SETUP_BLOCK.
> > >>
> > >> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
> > >> cls_flower.common.chain_index = FDB_FT_CHAIN;
> > >>
> > >> In following codes line 1380 and 1392
> > >>
> > >> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
> > >> *type_data,
> > >> 1369                                  void *cb_priv)
> > >> 1370 {
> > >> 1371         struct flow_cls_offload *f = type_data;
> > >> 1372         struct flow_cls_offload cls_flower;
> > >> 1373         struct mlx5e_priv *priv = cb_priv;
> > >> 1374         struct mlx5_eswitch *esw;
> > >> 1375         unsigned long flags;
> > >> 1376         int err;
> > >> 1377
> > >> 1378         flags = MLX5_TC_FLAG(INGRESS) |
> > >> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
> > >> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
> > >> 1381         esw = priv->mdev->priv.eswitch;
> > >> 1382
> > >> 1383         switch (type) {
> > >> 1384         case TC_SETUP_CLSFLOWER:
> > >> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
> > >>> common.chain_index)
> > >> 1386                         return -EOPNOTSUPP;
> > >> 1387
> > >> 1388                 /* Re-use tc offload path by moving the ft flow to the
> > >> 1389                  * reserved ft chain.
> > >> 1390                  */
> > >> 1391                 memcpy(&cls_flower, f, sizeof(*f));
> > >> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
> > >> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
> > flags);
> > >> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
> > >>
> > >>
> > >> I want to add tunnel offload support in the flow table, I  add some
> patches
> > in
> > >> nf_flow_table_offload.
> > >>
> > >> Also add the indr setup support in the mlx driver. And Now I can  flow
> > table
> > >> offload with decap.
> > >>
> > >>
> > >> But I meet a problem with the encap.  The encap rule can be added in
> > >> hardware  successfully But it can't be offloaded.
> > >>
> > >> But I think the rule I added is correct.  If I mask the line 1392. The rule
> also
> > can
> > >> be add success and can be offloaded.
> > >>
> > >> So there are some limit for encap operation for FT_OFFLOAD in
> > >> FDB_FT_CHAIN?
> > >>
> > >>
> > >> BR
> > >>
> > >> wenxu
> > >>

Re: Question about flow table offload in mlx5e

[wenxu]

在 2019/11/21 19:39, Paul Blakey 写道:
> They are good fixes, exactly what we had when we tested this, thanks.
>
> Regarding encap, I don't know what changes you did, how does the encap rule look? Is it a FWD to vxlan device? If not it should be, as our driver expects that.
It is fwd to a gretap devices
>
> I tried it on my setup via tc, by changing the callback of tc (mlx5e_rep_setup_tc_cb) to that of ft (mlx5e_rep_setup_ft_cb),
> and testing a vxlan encap rule:
> sudo tc qdisc add dev ens1f0_0 ingress
> sudo ifconfig ens1f0 7.7.7.7/24 up
> sudo ip link add name vxlan0 type vxlan dev ens1f0 remote 7.7.7.8 dstport 4789 external
> sudo ifconfig vxlan0 up
> sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0 protocol ip flower dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action mirred egress redirect dev vxlan
>
> then tc show:
> filter protocol ip pref 1 flower chain 0 handle 0x1 dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw in_hw in_hw_count 1
>         tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id 1234 dst_port 4789 csum pipe
>         Stats: used 119 sec      0 pkt
>         mirred (Egress Redirect to device vxlan0)
>         Stats: used 119 sec      0 pkt

Can you send packet that match this offloaded flow to check it is real offloaded?

In the flowtable offload with my patches both TC_SETUP_BLOCK and TC_SETUP_FT can offload the rule success

But in the TC_SETUP_FT case the packet is not real offloaded.


I  will test like u did.

>
>
>
>> -----Original Message-----
>> From: wenxu <wenxu@ucloud.cn>
>> Sent: Thursday, November 21, 2019 10:29 AM
>> To: Paul Blakey <paulb@mellanox.com>
>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>> <markb@mellanox.com>
>> Subject: Re: Question about flow table offload in mlx5e
>>
>>
>> On 11/21/2019 3:42 PM, Paul Blakey wrote:
>>> Hi,
>>>
>>> The original design was the block setup to use TC_SETUP_FT type, and the
>> tc event type to be case TC_SETUP_CLSFLOWER.
>>> We will post a patch to change that. I would advise to wait till we fix that
>> 😊
>>> I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it the
>> nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and not
>> TC_SETUP_FT.
>>
>>
>> Yes I change the TC_SETUP_BLOCK to TC_SETUP_FT in the
>> nf_flow_table_offload_setup.
>>
>> Two fixes patch provide:
>>
>> http://patchwork.ozlabs.org/patch/1197818/
>>
>> http://patchwork.ozlabs.org/patch/1197876/
>>
>> So this change made by me is not correct currently?
>>
>>> In our driver en_rep.c we have:
>>>> -------switch (type) {
>>>> -------case TC_SETUP_BLOCK:
>>>> ------->-------return flow_block_cb_setup_simple(type_data,
>>>> ------->------->------->------->------->-------  &mlx5e_rep_block_tc_cb_list,
>>>> ------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
>>>> ------->------->------->------->------->-------  priv, priv, true);
>>>> -------case TC_SETUP_FT:
>>>> ------->-------return flow_block_cb_setup_simple(type_data,
>>>> ------->------->------->------->------->-------  &mlx5e_rep_block_ft_cb_list,
>>>> ------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
>>>> ------->------->------->------->------->-------  priv, priv, true);
>>>> -------default:
>>>> ------->-------return -EOPNOTSUPP;
>>>> -------}
>>> In nf_flow_table_offload.c:
>>>> -------bo.binder_type>-= FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>>>> -------bo.extack>------= &extack;
>>>> -------INIT_LIST_HEAD(&bo.cb_list);
>>>> -------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK,
>> &bo);
>>>> -------if (err < 0)
>>>> ------->-------return err;
>>>> -------return nf_flow_table_block_setup(flowtable, &bo, cmd);
>>> }
>>> EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>>>
>>>
>>> So unless you changed that as well, you should have gotten to
>> mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.
>>> Regarding the encap action, there should be no difference on which chain
>> the rule is on.
>>
>>
>> But for the same encap rule can be real offloaded when setup through
>> through TC_SETUP_BLOCK. But TC_SETUP_FT can't.
>>
>> So it is the problem of TC_SETUP_FT in mlx5e_rep_setup_ft_cb ?
>>
>>>
>>>> -----Original Message-----
>>>> From: wenxu <wenxu@ucloud.cn>
>>>> Sent: Thursday, November 21, 2019 9:30 AM
>>>> To: Paul Blakey <paulb@mellanox.com>
>>>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>>>> <markb@mellanox.com>
>>>> Subject: Question about flow table offload in mlx5e
>>>>
>>>> Hi  paul,
>>>>
>>>> The flow table offload in the mlx5e is based on TC_SETUP_FT.
>>>>
>>>>
>>>> It is almost the same as TC_SETUP_BLOCK.
>>>>
>>>> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>>>> cls_flower.common.chain_index = FDB_FT_CHAIN;
>>>>
>>>> In following codes line 1380 and 1392
>>>>
>>>> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>>>> *type_data,
>>>> 1369                                  void *cb_priv)
>>>> 1370 {
>>>> 1371         struct flow_cls_offload *f = type_data;
>>>> 1372         struct flow_cls_offload cls_flower;
>>>> 1373         struct mlx5e_priv *priv = cb_priv;
>>>> 1374         struct mlx5_eswitch *esw;
>>>> 1375         unsigned long flags;
>>>> 1376         int err;
>>>> 1377
>>>> 1378         flags = MLX5_TC_FLAG(INGRESS) |
>>>> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
>>>> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
>>>> 1381         esw = priv->mdev->priv.eswitch;
>>>> 1382
>>>> 1383         switch (type) {
>>>> 1384         case TC_SETUP_CLSFLOWER:
>>>> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
>>>>> common.chain_index)
>>>> 1386                         return -EOPNOTSUPP;
>>>> 1387
>>>> 1388                 /* Re-use tc offload path by moving the ft flow to the
>>>> 1389                  * reserved ft chain.
>>>> 1390                  */
>>>> 1391                 memcpy(&cls_flower, f, sizeof(*f));
>>>> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
>>>> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>> flags);
>>>> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
>>>>
>>>>
>>>> I want to add tunnel offload support in the flow table, I  add some patches
>> in
>>>> nf_flow_table_offload.
>>>>
>>>> Also add the indr setup support in the mlx driver. And Now I can  flow
>> table
>>>> offload with decap.
>>>>
>>>>
>>>> But I meet a problem with the encap.  The encap rule can be added in
>>>> hardware  successfully But it can't be offloaded.
>>>>
>>>> But I think the rule I added is correct.  If I mask the line 1392. The rule also
>> can
>>>> be add success and can be offloaded.
>>>>
>>>> So there are some limit for encap operation for FT_OFFLOAD in
>>>> FDB_FT_CHAIN?
>>>>
>>>>
>>>> BR
>>>>
>>>> wenxu
>>>>

RE: Question about flow table offload in mlx5e

[Paul Blakey]

I see, I will test that, and how about normal FWD rules?

Paul.


> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn>
> Sent: Thursday, November 21, 2019 2:35 PM
> To: Paul Blakey <paulb@mellanox.com>
> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> <markb@mellanox.com>
> Subject: Re: Question about flow table offload in mlx5e
> 
> 
> 在 2019/11/21 19:39, Paul Blakey 写道:
> > They are good fixes, exactly what we had when we tested this, thanks.
> >
> > Regarding encap, I don't know what changes you did, how does the encap
> rule look? Is it a FWD to vxlan device? If not it should be, as our driver
> expects that.
> It is fwd to a gretap devices
> >
> > I tried it on my setup via tc, by changing the callback of tc
> (mlx5e_rep_setup_tc_cb) to that of ft (mlx5e_rep_setup_ft_cb),
> > and testing a vxlan encap rule:
> > sudo tc qdisc add dev ens1f0_0 ingress
> > sudo ifconfig ens1f0 7.7.7.7/24 up
> > sudo ip link add name vxlan0 type vxlan dev ens1f0 remote 7.7.7.8 dstport
> 4789 external
> > sudo ifconfig vxlan0 up
> > sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0 protocol ip flower
> dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action tunnel_key set
> src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action mirred egress
> redirect dev vxlan
> >
> > then tc show:
> > filter protocol ip pref 1 flower chain 0 handle 0x1 dst_mac aa:bb:cc:dd:ee:ff
> ip_proto udp skip_sw in_hw in_hw_count 1
> >         tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id 1234 dst_port 4789
> csum pipe
> >         Stats: used 119 sec      0 pkt
> >         mirred (Egress Redirect to device vxlan0)
> >         Stats: used 119 sec      0 pkt
> 
> Can you send packet that match this offloaded flow to check it is real
> offloaded?
> 
> In the flowtable offload with my patches both TC_SETUP_BLOCK and
> TC_SETUP_FT can offload the rule success
> 
> But in the TC_SETUP_FT case the packet is not real offloaded.
> 
> 
> I  will test like u did.
> 
> >
> >
> >
> >> -----Original Message-----
> >> From: wenxu <wenxu@ucloud.cn>
> >> Sent: Thursday, November 21, 2019 10:29 AM
> >> To: Paul Blakey <paulb@mellanox.com>
> >> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> >> <markb@mellanox.com>
> >> Subject: Re: Question about flow table offload in mlx5e
> >>
> >>
> >> On 11/21/2019 3:42 PM, Paul Blakey wrote:
> >>> Hi,
> >>>
> >>> The original design was the block setup to use TC_SETUP_FT type, and
> the
> >> tc event type to be case TC_SETUP_CLSFLOWER.
> >>> We will post a patch to change that. I would advise to wait till we fix that
> >> 😊
> >>> I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it
> the
> >> nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and
> not
> >> TC_SETUP_FT.
> >>
> >>
> >> Yes I change the TC_SETUP_BLOCK to TC_SETUP_FT in the
> >> nf_flow_table_offload_setup.
> >>
> >> Two fixes patch provide:
> >>
> >> http://patchwork.ozlabs.org/patch/1197818/
> >>
> >> http://patchwork.ozlabs.org/patch/1197876/
> >>
> >> So this change made by me is not correct currently?
> >>
> >>> In our driver en_rep.c we have:
> >>>> -------switch (type) {
> >>>> -------case TC_SETUP_BLOCK:
> >>>> ------->-------return flow_block_cb_setup_simple(type_data,
> >>>> ------->------->------->------->------->-------
> &mlx5e_rep_block_tc_cb_list,
> >>>> ------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
> >>>> ------->------->------->------->------->-------  priv, priv, true);
> >>>> -------case TC_SETUP_FT:
> >>>> ------->-------return flow_block_cb_setup_simple(type_data,
> >>>> ------->------->------->------->------->-------
> &mlx5e_rep_block_ft_cb_list,
> >>>> ------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
> >>>> ------->------->------->------->------->-------  priv, priv, true);
> >>>> -------default:
> >>>> ------->-------return -EOPNOTSUPP;
> >>>> -------}
> >>> In nf_flow_table_offload.c:
> >>>> -------bo.binder_type>-=
> FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
> >>>> -------bo.extack>------= &extack;
> >>>> -------INIT_LIST_HEAD(&bo.cb_list);
> >>>> -------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK,
> >> &bo);
> >>>> -------if (err < 0)
> >>>> ------->-------return err;
> >>>> -------return nf_flow_table_block_setup(flowtable, &bo, cmd);
> >>> }
> >>> EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
> >>>
> >>>
> >>> So unless you changed that as well, you should have gotten to
> >> mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.
> >>> Regarding the encap action, there should be no difference on which
> chain
> >> the rule is on.
> >>
> >>
> >> But for the same encap rule can be real offloaded when setup through
> >> through TC_SETUP_BLOCK. But TC_SETUP_FT can't.
> >>
> >> So it is the problem of TC_SETUP_FT in mlx5e_rep_setup_ft_cb ?
> >>
> >>>
> >>>> -----Original Message-----
> >>>> From: wenxu <wenxu@ucloud.cn>
> >>>> Sent: Thursday, November 21, 2019 9:30 AM
> >>>> To: Paul Blakey <paulb@mellanox.com>
> >>>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> >>>> <markb@mellanox.com>
> >>>> Subject: Question about flow table offload in mlx5e
> >>>>
> >>>> Hi  paul,
> >>>>
> >>>> The flow table offload in the mlx5e is based on TC_SETUP_FT.
> >>>>
> >>>>
> >>>> It is almost the same as TC_SETUP_BLOCK.
> >>>>
> >>>> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
> >>>> cls_flower.common.chain_index = FDB_FT_CHAIN;
> >>>>
> >>>> In following codes line 1380 and 1392
> >>>>
> >>>> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
> >>>> *type_data,
> >>>> 1369                                  void *cb_priv)
> >>>> 1370 {
> >>>> 1371         struct flow_cls_offload *f = type_data;
> >>>> 1372         struct flow_cls_offload cls_flower;
> >>>> 1373         struct mlx5e_priv *priv = cb_priv;
> >>>> 1374         struct mlx5_eswitch *esw;
> >>>> 1375         unsigned long flags;
> >>>> 1376         int err;
> >>>> 1377
> >>>> 1378         flags = MLX5_TC_FLAG(INGRESS) |
> >>>> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
> >>>> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
> >>>> 1381         esw = priv->mdev->priv.eswitch;
> >>>> 1382
> >>>> 1383         switch (type) {
> >>>> 1384         case TC_SETUP_CLSFLOWER:
> >>>> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
> >>>>> common.chain_index)
> >>>> 1386                         return -EOPNOTSUPP;
> >>>> 1387
> >>>> 1388                 /* Re-use tc offload path by moving the ft flow to the
> >>>> 1389                  * reserved ft chain.
> >>>> 1390                  */
> >>>> 1391                 memcpy(&cls_flower, f, sizeof(*f));
> >>>> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
> >>>> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
> >> flags);
> >>>> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
> >>>>
> >>>>
> >>>> I want to add tunnel offload support in the flow table, I  add some
> patches
> >> in
> >>>> nf_flow_table_offload.
> >>>>
> >>>> Also add the indr setup support in the mlx driver. And Now I can  flow
> >> table
> >>>> offload with decap.
> >>>>
> >>>>
> >>>> But I meet a problem with the encap.  The encap rule can be added in
> >>>> hardware  successfully But it can't be offloaded.
> >>>>
> >>>> But I think the rule I added is correct.  If I mask the line 1392. The rule
> also
> >> can
> >>>> be add success and can be offloaded.
> >>>>
> >>>> So there are some limit for encap operation for FT_OFFLOAD in
> >>>> FDB_FT_CHAIN?
> >>>>
> >>>>
> >>>> BR
> >>>>
> >>>> wenxu
> >>>>

Re: Question about flow table offload in mlx5e

[wenxu]

The normal FWD rules that fwd to pf-rep or vf-rep can real offloaded

在 2019/11/21 21:05, Paul Blakey 写道:
> I see, I will test that, and how about normal FWD rules?
>
> Paul.
>
>
>> -----Original Message-----
>> From: wenxu <wenxu@ucloud.cn>
>> Sent: Thursday, November 21, 2019 2:35 PM
>> To: Paul Blakey <paulb@mellanox.com>
>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>> <markb@mellanox.com>
>> Subject: Re: Question about flow table offload in mlx5e
>>
>>
>> 在 2019/11/21 19:39, Paul Blakey 写道:
>>> They are good fixes, exactly what we had when we tested this, thanks.
>>>
>>> Regarding encap, I don't know what changes you did, how does the encap
>> rule look? Is it a FWD to vxlan device? If not it should be, as our driver
>> expects that.
>> It is fwd to a gretap devices
>>> I tried it on my setup via tc, by changing the callback of tc
>> (mlx5e_rep_setup_tc_cb) to that of ft (mlx5e_rep_setup_ft_cb),
>>> and testing a vxlan encap rule:
>>> sudo tc qdisc add dev ens1f0_0 ingress
>>> sudo ifconfig ens1f0 7.7.7.7/24 up
>>> sudo ip link add name vxlan0 type vxlan dev ens1f0 remote 7.7.7.8 dstport
>> 4789 external
>>> sudo ifconfig vxlan0 up
>>> sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0 protocol ip flower
>> dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action tunnel_key set
>> src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action mirred egress
>> redirect dev vxlan
>>> then tc show:
>>> filter protocol ip pref 1 flower chain 0 handle 0x1 dst_mac aa:bb:cc:dd:ee:ff
>> ip_proto udp skip_sw in_hw in_hw_count 1
>>>         tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id 1234 dst_port 4789
>> csum pipe
>>>         Stats: used 119 sec      0 pkt
>>>         mirred (Egress Redirect to device vxlan0)
>>>         Stats: used 119 sec      0 pkt
>> Can you send packet that match this offloaded flow to check it is real
>> offloaded?
>>
>> In the flowtable offload with my patches both TC_SETUP_BLOCK and
>> TC_SETUP_FT can offload the rule success
>>
>> But in the TC_SETUP_FT case the packet is not real offloaded.
>>
>>
>> I  will test like u did.
>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: wenxu <wenxu@ucloud.cn>
>>>> Sent: Thursday, November 21, 2019 10:29 AM
>>>> To: Paul Blakey <paulb@mellanox.com>
>>>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>>>> <markb@mellanox.com>
>>>> Subject: Re: Question about flow table offload in mlx5e
>>>>
>>>>
>>>> On 11/21/2019 3:42 PM, Paul Blakey wrote:
>>>>> Hi,
>>>>>
>>>>> The original design was the block setup to use TC_SETUP_FT type, and
>> the
>>>> tc event type to be case TC_SETUP_CLSFLOWER.
>>>>> We will post a patch to change that. I would advise to wait till we fix that
>>>> 😊
>>>>> I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it
>> the
>>>> nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and
>> not
>>>> TC_SETUP_FT.
>>>>
>>>>
>>>> Yes I change the TC_SETUP_BLOCK to TC_SETUP_FT in the
>>>> nf_flow_table_offload_setup.
>>>>
>>>> Two fixes patch provide:
>>>>
>>>> http://patchwork.ozlabs.org/patch/1197818/
>>>>
>>>> http://patchwork.ozlabs.org/patch/1197876/
>>>>
>>>> So this change made by me is not correct currently?
>>>>
>>>>> In our driver en_rep.c we have:
>>>>>> -------switch (type) {
>>>>>> -------case TC_SETUP_BLOCK:
>>>>>> ------->-------return flow_block_cb_setup_simple(type_data,
>>>>>> ------->------->------->------->------->-------
>> &mlx5e_rep_block_tc_cb_list,
>>>>>> ------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
>>>>>> ------->------->------->------->------->-------  priv, priv, true);
>>>>>> -------case TC_SETUP_FT:
>>>>>> ------->-------return flow_block_cb_setup_simple(type_data,
>>>>>> ------->------->------->------->------->-------
>> &mlx5e_rep_block_ft_cb_list,
>>>>>> ------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
>>>>>> ------->------->------->------->------->-------  priv, priv, true);
>>>>>> -------default:
>>>>>> ------->-------return -EOPNOTSUPP;
>>>>>> -------}
>>>>> In nf_flow_table_offload.c:
>>>>>> -------bo.binder_type>-=
>> FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>>>>>> -------bo.extack>------= &extack;
>>>>>> -------INIT_LIST_HEAD(&bo.cb_list);
>>>>>> -------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK,
>>>> &bo);
>>>>>> -------if (err < 0)
>>>>>> ------->-------return err;
>>>>>> -------return nf_flow_table_block_setup(flowtable, &bo, cmd);
>>>>> }
>>>>> EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>>>>>
>>>>>
>>>>> So unless you changed that as well, you should have gotten to
>>>> mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.
>>>>> Regarding the encap action, there should be no difference on which
>> chain
>>>> the rule is on.
>>>>
>>>>
>>>> But for the same encap rule can be real offloaded when setup through
>>>> through TC_SETUP_BLOCK. But TC_SETUP_FT can't.
>>>>
>>>> So it is the problem of TC_SETUP_FT in mlx5e_rep_setup_ft_cb ?
>>>>
>>>>>> -----Original Message-----
>>>>>> From: wenxu <wenxu@ucloud.cn>
>>>>>> Sent: Thursday, November 21, 2019 9:30 AM
>>>>>> To: Paul Blakey <paulb@mellanox.com>
>>>>>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>>>>>> <markb@mellanox.com>
>>>>>> Subject: Question about flow table offload in mlx5e
>>>>>>
>>>>>> Hi  paul,
>>>>>>
>>>>>> The flow table offload in the mlx5e is based on TC_SETUP_FT.
>>>>>>
>>>>>>
>>>>>> It is almost the same as TC_SETUP_BLOCK.
>>>>>>
>>>>>> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>>>>>> cls_flower.common.chain_index = FDB_FT_CHAIN;
>>>>>>
>>>>>> In following codes line 1380 and 1392
>>>>>>
>>>>>> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>>>>>> *type_data,
>>>>>> 1369                                  void *cb_priv)
>>>>>> 1370 {
>>>>>> 1371         struct flow_cls_offload *f = type_data;
>>>>>> 1372         struct flow_cls_offload cls_flower;
>>>>>> 1373         struct mlx5e_priv *priv = cb_priv;
>>>>>> 1374         struct mlx5_eswitch *esw;
>>>>>> 1375         unsigned long flags;
>>>>>> 1376         int err;
>>>>>> 1377
>>>>>> 1378         flags = MLX5_TC_FLAG(INGRESS) |
>>>>>> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
>>>>>> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
>>>>>> 1381         esw = priv->mdev->priv.eswitch;
>>>>>> 1382
>>>>>> 1383         switch (type) {
>>>>>> 1384         case TC_SETUP_CLSFLOWER:
>>>>>> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
>>>>>>> common.chain_index)
>>>>>> 1386                         return -EOPNOTSUPP;
>>>>>> 1387
>>>>>> 1388                 /* Re-use tc offload path by moving the ft flow to the
>>>>>> 1389                  * reserved ft chain.
>>>>>> 1390                  */
>>>>>> 1391                 memcpy(&cls_flower, f, sizeof(*f));
>>>>>> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
>>>>>> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>>>> flags);
>>>>>> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
>>>>>>
>>>>>>
>>>>>> I want to add tunnel offload support in the flow table, I  add some
>> patches
>>>> in
>>>>>> nf_flow_table_offload.
>>>>>>
>>>>>> Also add the indr setup support in the mlx driver. And Now I can  flow
>>>> table
>>>>>> offload with decap.
>>>>>>
>>>>>>
>>>>>> But I meet a problem with the encap.  The encap rule can be added in
>>>>>> hardware  successfully But it can't be offloaded.
>>>>>>
>>>>>> But I think the rule I added is correct.  If I mask the line 1392. The rule
>> also
>>>> can
>>>>>> be add success and can be offloaded.
>>>>>>
>>>>>> So there are some limit for encap operation for FT_OFFLOAD in
>>>>>> FDB_FT_CHAIN?
>>>>>>
>>>>>>
>>>>>> BR
>>>>>>
>>>>>> wenxu
>>>>>>

Re: Question about flow table offload in mlx5e

[wenxu]

Hi Paul,


There are some update.


ifconfig mlx_p0 172.168.152.75/24 up

ip l add dev tun1 type gretap external
tc qdisc add dev tun1 ingress
tc qdisc add dev mlx_pf0vf0 ingress

tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001 tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000 nocsum pipe action mirred egress redirect dev tun1

tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip 10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5 enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action mirred egress redirect dev mlx_pf0vf0


If you run this script on the host，  and in the virtual machine  run "iperf -c 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 1000"

The tcp syn packet will not be offloaded


But if you  only run the script  without the last filter as following , The tcp syn packet will be offloaded.

ifconfig mlx_p0 172.168.152.75/24 up

ip l add dev tun1 type gretap external
tc qdisc add dev tun1 ingress
tc qdisc add dev mlx_pf0vf0 ingress

tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001 tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000 nocsum pipe action mirred egress redirect dev tun1





On 11/21/2019 9:05 PM, Paul Blakey wrote:
> I see, I will test that, and how about normal FWD rules?
>
> Paul.
>
>
>> -----Original Message-----
>> From: wenxu <wenxu@ucloud.cn>
>> Sent: Thursday, November 21, 2019 2:35 PM
>> To: Paul Blakey <paulb@mellanox.com>
>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>> <markb@mellanox.com>
>> Subject: Re: Question about flow table offload in mlx5e
>>
>>
>> 在 2019/11/21 19:39, Paul Blakey 写道:
>>> They are good fixes, exactly what we had when we tested this, thanks.
>>>
>>> Regarding encap, I don't know what changes you did, how does the encap
>> rule look? Is it a FWD to vxlan device? If not it should be, as our driver
>> expects that.
>> It is fwd to a gretap devices
>>> I tried it on my setup via tc, by changing the callback of tc
>> (mlx5e_rep_setup_tc_cb) to that of ft (mlx5e_rep_setup_ft_cb),
>>> and testing a vxlan encap rule:
>>> sudo tc qdisc add dev ens1f0_0 ingress
>>> sudo ifconfig ens1f0 7.7.7.7/24 up
>>> sudo ip link add name vxlan0 type vxlan dev ens1f0 remote 7.7.7.8 dstport
>> 4789 external
>>> sudo ifconfig vxlan0 up
>>> sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0 protocol ip flower
>> dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action tunnel_key set
>> src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action mirred egress
>> redirect dev vxlan
>>> then tc show:
>>> filter protocol ip pref 1 flower chain 0 handle 0x1 dst_mac aa:bb:cc:dd:ee:ff
>> ip_proto udp skip_sw in_hw in_hw_count 1
>>>         tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id 1234 dst_port 4789
>> csum pipe
>>>         Stats: used 119 sec      0 pkt
>>>         mirred (Egress Redirect to device vxlan0)
>>>         Stats: used 119 sec      0 pkt
>> Can you send packet that match this offloaded flow to check it is real
>> offloaded?
>>
>> In the flowtable offload with my patches both TC_SETUP_BLOCK and
>> TC_SETUP_FT can offload the rule success
>>
>> But in the TC_SETUP_FT case the packet is not real offloaded.
>>
>>
>> I  will test like u did.
>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: wenxu <wenxu@ucloud.cn>
>>>> Sent: Thursday, November 21, 2019 10:29 AM
>>>> To: Paul Blakey <paulb@mellanox.com>
>>>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>>>> <markb@mellanox.com>
>>>> Subject: Re: Question about flow table offload in mlx5e
>>>>
>>>>
>>>> On 11/21/2019 3:42 PM, Paul Blakey wrote:
>>>>> Hi,
>>>>>
>>>>> The original design was the block setup to use TC_SETUP_FT type, and
>> the
>>>> tc event type to be case TC_SETUP_CLSFLOWER.
>>>>> We will post a patch to change that. I would advise to wait till we fix that
>>>> 😊
>>>>> I'm not sure how you get to this function mlx5e_rep_setup_ft_cb() if it
>> the
>>>> nf_flow_table_offload ndo_setup_tc event was TC_SETUP_BLOCK, and
>> not
>>>> TC_SETUP_FT.
>>>>
>>>>
>>>> Yes I change the TC_SETUP_BLOCK to TC_SETUP_FT in the
>>>> nf_flow_table_offload_setup.
>>>>
>>>> Two fixes patch provide:
>>>>
>>>> http://patchwork.ozlabs.org/patch/1197818/
>>>>
>>>> http://patchwork.ozlabs.org/patch/1197876/
>>>>
>>>> So this change made by me is not correct currently?
>>>>
>>>>> In our driver en_rep.c we have:
>>>>>> -------switch (type) {
>>>>>> -------case TC_SETUP_BLOCK:
>>>>>> ------->-------return flow_block_cb_setup_simple(type_data,
>>>>>> ------->------->------->------->------->-------
>> &mlx5e_rep_block_tc_cb_list,
>>>>>> ------->------->------->------->------->-------  mlx5e_rep_setup_tc_cb,
>>>>>> ------->------->------->------->------->-------  priv, priv, true);
>>>>>> -------case TC_SETUP_FT:
>>>>>> ------->-------return flow_block_cb_setup_simple(type_data,
>>>>>> ------->------->------->------->------->-------
>> &mlx5e_rep_block_ft_cb_list,
>>>>>> ------->------->------->------->------->-------  mlx5e_rep_setup_ft_cb,
>>>>>> ------->------->------->------->------->-------  priv, priv, true);
>>>>>> -------default:
>>>>>> ------->-------return -EOPNOTSUPP;
>>>>>> -------}
>>>>> In nf_flow_table_offload.c:
>>>>>> -------bo.binder_type>-=
>> FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>>>>>> -------bo.extack>------= &extack;
>>>>>> -------INIT_LIST_HEAD(&bo.cb_list);
>>>>>> -------err = dev->netdev_ops->ndo_setup_tc(dev, TC_SETUP_BLOCK,
>>>> &bo);
>>>>>> -------if (err < 0)
>>>>>> ------->-------return err;
>>>>>> -------return nf_flow_table_block_setup(flowtable, &bo, cmd);
>>>>> }
>>>>> EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>>>>>
>>>>>
>>>>> So unless you changed that as well, you should have gotten to
>>>> mlx5e_rep_setup_tc_cb and not mlx5e_rep_setup_tc_ft.
>>>>> Regarding the encap action, there should be no difference on which
>> chain
>>>> the rule is on.
>>>>
>>>>
>>>> But for the same encap rule can be real offloaded when setup through
>>>> through TC_SETUP_BLOCK. But TC_SETUP_FT can't.
>>>>
>>>> So it is the problem of TC_SETUP_FT in mlx5e_rep_setup_ft_cb ?
>>>>
>>>>>> -----Original Message-----
>>>>>> From: wenxu <wenxu@ucloud.cn>
>>>>>> Sent: Thursday, November 21, 2019 9:30 AM
>>>>>> To: Paul Blakey <paulb@mellanox.com>
>>>>>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>>>>>> <markb@mellanox.com>
>>>>>> Subject: Question about flow table offload in mlx5e
>>>>>>
>>>>>> Hi  paul,
>>>>>>
>>>>>> The flow table offload in the mlx5e is based on TC_SETUP_FT.
>>>>>>
>>>>>>
>>>>>> It is almost the same as TC_SETUP_BLOCK.
>>>>>>
>>>>>> It just set MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>>>>>> cls_flower.common.chain_index = FDB_FT_CHAIN;
>>>>>>
>>>>>> In following codes line 1380 and 1392
>>>>>>
>>>>>> 1368 static int mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>>>>>> *type_data,
>>>>>> 1369                                  void *cb_priv)
>>>>>> 1370 {
>>>>>> 1371         struct flow_cls_offload *f = type_data;
>>>>>> 1372         struct flow_cls_offload cls_flower;
>>>>>> 1373         struct mlx5e_priv *priv = cb_priv;
>>>>>> 1374         struct mlx5_eswitch *esw;
>>>>>> 1375         unsigned long flags;
>>>>>> 1376         int err;
>>>>>> 1377
>>>>>> 1378         flags = MLX5_TC_FLAG(INGRESS) |
>>>>>> 1379                 MLX5_TC_FLAG(ESW_OFFLOAD) |
>>>>>> 1380                 MLX5_TC_FLAG(FT_OFFLOAD);
>>>>>> 1381         esw = priv->mdev->priv.eswitch;
>>>>>> 1382
>>>>>> 1383         switch (type) {
>>>>>> 1384         case TC_SETUP_CLSFLOWER:
>>>>>> 1385                 if (!mlx5_eswitch_prios_supported(esw) || f-
>>>>>>> common.chain_index)
>>>>>> 1386                         return -EOPNOTSUPP;
>>>>>> 1387
>>>>>> 1388                 /* Re-use tc offload path by moving the ft flow to the
>>>>>> 1389                  * reserved ft chain.
>>>>>> 1390                  */
>>>>>> 1391                 memcpy(&cls_flower, f, sizeof(*f));
>>>>>> 1392                cls_flower.common.chain_index = FDB_FT_CHAIN;
>>>>>> 1393                 err = mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>>>> flags);
>>>>>> 1394                 memcpy(&f->stats, &cls_flower.stats, sizeof(f->stats));
>>>>>>
>>>>>>
>>>>>> I want to add tunnel offload support in the flow table, I  add some
>> patches
>>>> in
>>>>>> nf_flow_table_offload.
>>>>>>
>>>>>> Also add the indr setup support in the mlx driver. And Now I can  flow
>>>> table
>>>>>> offload with decap.
>>>>>>
>>>>>>
>>>>>> But I meet a problem with the encap.  The encap rule can be added in
>>>>>> hardware  successfully But it can't be offloaded.
>>>>>>
>>>>>> But I think the rule I added is correct.  If I mask the line 1392. The rule
>> also
>>>> can
>>>>>> be add success and can be offloaded.
>>>>>>
>>>>>> So there are some limit for encap operation for FT_OFFLOAD in
>>>>>> FDB_FT_CHAIN?
>>>>>>
>>>>>>
>>>>>> BR
>>>>>>
>>>>>> wenxu
>>>>>>

RE: Question about flow table offload in mlx5e

[Paul Blakey]

Hi,

The syn packet might not be actually offloaded because there isn't a neighbor to resolve the destination mac for the tunnel destination ip (next hop mac).
Try setting the neighbor via "ip neigh replace dev mlx5_p0 172.168.152.241 lladdr <next hop mac>"
Or running ping to 172.168.152.241 before adding (or in background) the rule to resolve the mac and make available.
I'll test it on my end.


Thanks,
Paul.


> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn>
> Sent: Friday, November 22, 2019 8:26 AM
> To: Paul Blakey <paulb@mellanox.com>
> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
> <markb@mellanox.com>
> Subject: Re: Question about flow table offload in mlx5e
> 
> Hi Paul,
> 
> 
> There are some update. I also test it through replacing mlx5e_rep_setup_tc
> _cb with mlx5e_rep_setup_ft_cb
> 
> 
> ifconfig mlx_p0 172.168.152.75/24 up
> 
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
> 
> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1
> 
> tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip
> 10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
> enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
> action mirred egress redirect dev mlx_pf0vf0
> 
> 
> If you run this script on the host，  and in the virtual machine  run "iperf -c
> 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 1000"
> 
> The tcp syn packet will not be offloaded
> 
> 
> But if you  only run the script  without the last filter as following , The tcp syn
> packet will be offloaded.
> 
> ifconfig mlx_p0 172.168.152.75/24 up
> 
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
> 
> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1.
> 
> I think there are some problem in mlx5e_rep_setup_ft_cb.
> 
> On 11/21/2019 9:05 PM, Paul Blakey wrote:
> 
> 
> 	I see, I will test that, and how about normal FWD rules?
> 
> 	Paul.
> 
> 
> 
> 		-----Original Message-----
> 		From: wenxu <wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn>
> 		Sent: Thursday, November 21, 2019 2:35 PM
> 		To: Paul Blakey <paulb@mellanox.com>
> <mailto:paulb@mellanox.com>
> 		Cc: pablo@netfilter.org <mailto:pablo@netfilter.org> ;
> netdev@vger.kernel.org <mailto:netdev@vger.kernel.org> ; Mark Bloch
> 		<markb@mellanox.com> <mailto:markb@mellanox.com>
> 		Subject: Re: Question about flow table offload in mlx5e
> 
> 
> 		在 2019/11/21 19:39, Paul Blakey 写道:
> 
> 			They are good fixes, exactly what we had when we
> tested this, thanks.
> 
> 			Regarding encap, I don't know what changes you did,
> how does the encap
> 
> 		rule look? Is it a FWD to vxlan device? If not it should be, as
> our driver
> 		expects that.
> 		It is fwd to a gretap devices
> 
> 
> 			I tried it on my setup via tc, by changing the callback
> of tc
> 
> 		(mlx5e_rep_setup_tc_cb) to that of ft
> (mlx5e_rep_setup_ft_cb),
> 
> 			and testing a vxlan encap rule:
> 			sudo tc qdisc add dev ens1f0_0 ingress
> 			sudo ifconfig ens1f0 7.7.7.7/24 up
> 			sudo ip link add name vxlan0 type vxlan dev ens1f0
> remote 7.7.7.8 dstport
> 
> 		4789 external
> 
> 			sudo ifconfig vxlan0 up
> 			sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0
> protocol ip flower
> 
> 		dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action
> tunnel_key set
> 		src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action
> mirred egress
> 		redirect dev vxlan
> 
> 
> 			then tc show:
> 			filter protocol ip pref 1 flower chain 0 handle 0x1
> dst_mac aa:bb:cc:dd:ee:ff
> 
> 		ip_proto udp skip_sw in_hw in_hw_count 1
> 
> 			        tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id
> 1234 dst_port 4789
> 
> 		csum pipe
> 
> 			        Stats: used 119 sec      0 pkt
> 			        mirred (Egress Redirect to device vxlan0)
> 			        Stats: used 119 sec      0 pkt
> 
> 
> 		Can you send packet that match this offloaded flow to check
> it is real
> 		offloaded?
> 
> 		In the flowtable offload with my patches both
> TC_SETUP_BLOCK and
> 		TC_SETUP_FT can offload the rule success
> 
> 		But in the TC_SETUP_FT case the packet is not real offloaded.
> 
> 
> 		I  will test like u did.
> 
> 
> 
> 
> 
> 
> 				-----Original Message-----
> 				From: wenxu <wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn>
> 				Sent: Thursday, November 21, 2019 10:29 AM
> 				To: Paul Blakey <paulb@mellanox.com>
> <mailto:paulb@mellanox.com>
> 				Cc: pablo@netfilter.org
> <mailto:pablo@netfilter.org> ; netdev@vger.kernel.org
> <mailto:netdev@vger.kernel.org> ; Mark Bloch
> 				<markb@mellanox.com>
> <mailto:markb@mellanox.com>
> 				Subject: Re: Question about flow table
> offload in mlx5e
> 
> 
> 				On 11/21/2019 3:42 PM, Paul Blakey wrote:
> 
> 					Hi,
> 
> 					The original design was the block
> setup to use TC_SETUP_FT type, and
> 
> 		the
> 
> 				tc event type to be case
> TC_SETUP_CLSFLOWER.
> 
> 					We will post a patch to change that. I
> would advise to wait till we fix that
> 
> 				😊
> 
> 					I'm not sure how you get to this
> function mlx5e_rep_setup_ft_cb() if it
> 
> 		the
> 
> 				nf_flow_table_offload ndo_setup_tc event
> was TC_SETUP_BLOCK, and
> 
> 		not
> 
> 				TC_SETUP_FT.
> 
> 
> 				Yes I change the TC_SETUP_BLOCK to
> TC_SETUP_FT in the
> 				nf_flow_table_offload_setup.
> 
> 				Two fixes patch provide:
> 
> 				http://patchwork.ozlabs.org/patch/1197818/
> 
> 				http://patchwork.ozlabs.org/patch/1197876/
> 
> 				So this change made by me is not correct
> currently?
> 
> 
> 					In our driver en_rep.c we have:
> 
> 					-------switch (type) {
> 					-------case TC_SETUP_BLOCK:
> 					------->-------return
> flow_block_cb_setup_simple(type_data,
> 					------->------->------->------->------->---
> ----
> 
> 		&mlx5e_rep_block_tc_cb_list,
> 
> 					------->------->------->------->------->---
> ----  mlx5e_rep_setup_tc_cb,
> 					------->------->------->------->------->---
> ----  priv, priv, true);
> 					-------case TC_SETUP_FT:
> 					------->-------return
> flow_block_cb_setup_simple(type_data,
> 					------->------->------->------->------->---
> ----
> 
> 		&mlx5e_rep_block_ft_cb_list,
> 
> 					------->------->------->------->------->---
> ----  mlx5e_rep_setup_ft_cb,
> 					------->------->------->------->------->---
> ----  priv, priv, true);
> 					-------default:
> 					------->-------return -EOPNOTSUPP;
> 					-------}
> 
> 					In nf_flow_table_offload.c:
> 
> 					-------bo.binder_type>-=
> 
> 		FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
> 
> 					-------bo.extack>------= &extack;
> 					-------INIT_LIST_HEAD(&bo.cb_list);
> 					-------err = dev->netdev_ops-
> >ndo_setup_tc(dev, TC_SETUP_BLOCK,
> 
> 				&bo);
> 
> 					-------if (err < 0)
> 					------->-------return err;
> 					-------return
> nf_flow_table_block_setup(flowtable, &bo, cmd);
> 
> 					}
> 
> 	EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
> 
> 
> 					So unless you changed that as well,
> you should have gotten to
> 
> 				mlx5e_rep_setup_tc_cb and not
> mlx5e_rep_setup_tc_ft.
> 
> 					Regarding the encap action, there
> should be no difference on which
> 
> 		chain
> 
> 				the rule is on.
> 
> 
> 				But for the same encap rule can be real
> offloaded when setup through
> 				through TC_SETUP_BLOCK. But TC_SETUP_FT
> can't.
> 
> 				So it is the problem of TC_SETUP_FT in
> mlx5e_rep_setup_ft_cb ?
> 
> 
> 
> 
> 					-----Original Message-----
> 					From: wenxu <wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn>
> 					Sent: Thursday, November 21, 2019
> 9:30 AM
> 					To: Paul Blakey
> <paulb@mellanox.com> <mailto:paulb@mellanox.com>
> 					Cc: pablo@netfilter.org
> <mailto:pablo@netfilter.org> ; netdev@vger.kernel.org
> <mailto:netdev@vger.kernel.org> ; Mark Bloch
> 					<markb@mellanox.com>
> <mailto:markb@mellanox.com>
> 					Subject: Question about flow table
> offload in mlx5e
> 
> 					Hi  paul,
> 
> 					The flow table offload in the mlx5e is
> based on TC_SETUP_FT.
> 
> 
> 					It is almost the same as
> TC_SETUP_BLOCK.
> 
> 					It just set
> MLX5_TC_FLAG(FT_OFFLOAD) flags and change
> 					cls_flower.common.chain_index =
> FDB_FT_CHAIN;
> 
> 					In following codes line 1380 and 1392
> 
> 					1368 static int
> mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
> 					*type_data,
> 					1369                                  void *cb_priv)
> 					1370 {
> 					1371         struct flow_cls_offload *f =
> type_data;
> 					1372         struct flow_cls_offload
> cls_flower;
> 					1373         struct mlx5e_priv *priv =
> cb_priv;
> 					1374         struct mlx5_eswitch *esw;
> 					1375         unsigned long flags;
> 					1376         int err;
> 					1377
> 					1378         flags =
> MLX5_TC_FLAG(INGRESS) |
> 					1379
> MLX5_TC_FLAG(ESW_OFFLOAD) |
> 					1380
> MLX5_TC_FLAG(FT_OFFLOAD);
> 					1381         esw = priv->mdev-
> >priv.eswitch;
> 					1382
> 					1383         switch (type) {
> 					1384         case
> TC_SETUP_CLSFLOWER:
> 					1385                 if
> (!mlx5_eswitch_prios_supported(esw) || f-
> 
> 					common.chain_index)
> 
> 					1386                         return -
> EOPNOTSUPP;
> 					1387
> 					1388                 /* Re-use tc offload
> path by moving the ft flow to the
> 					1389                  * reserved ft chain.
> 					1390                  */
> 					1391                 memcpy(&cls_flower, f,
> sizeof(*f));
> 					1392
> cls_flower.common.chain_index = FDB_FT_CHAIN;
> 					1393                 err =
> mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
> 
> 				flags);
> 
> 					1394                 memcpy(&f->stats,
> &cls_flower.stats, sizeof(f->stats));
> 
> 
> 					I want to add tunnel offload support
> in the flow table, I  add some
> 
> 		patches
> 
> 				in
> 
> 					nf_flow_table_offload.
> 
> 					Also add the indr setup support in the
> mlx driver. And Now I can  flow
> 
> 				table
> 
> 					offload with decap.
> 
> 
> 					But I meet a problem with the encap.
> The encap rule can be added in
> 					hardware  successfully But it can't be
> offloaded.
> 
> 					But I think the rule I added is correct.
> If I mask the line 1392. The rule
> 
> 		also
> 
> 				can
> 
> 					be add success and can be offloaded.
> 
> 					So there are some limit for encap
> operation for FT_OFFLOAD in
> 					FDB_FT_CHAIN?
> 
> 
> 					BR
> 
> 					wenxu
> 

Re: Question about flow table offload in mlx5e

[wenxu]

sorry for miss some information, There is a real remote and the mac always available.  Not just one packets can't be offloaded. ALL the syn packets with repeated can't be offloaded.  But if there is no the last filter rule. All syn packets can be offload

在 2019/11/24 16:46, Paul Blakey 写道:
> Hi,
>
> The syn packet might not be actually offloaded because there isn't a neighbor to resolve the destination mac for the tunnel destination ip (next hop mac).
> Try setting the neighbor via "ip neigh replace dev mlx5_p0 172.168.152.241 lladdr <next hop mac>"
> Or running ping to 172.168.152.241 before adding (or in background) the rule to resolve the mac and make available.
> I'll test it on my end.
>
>
> Thanks,
> Paul.
>
>
>> -----Original Message-----
>> From: wenxu <wenxu@ucloud.cn>
>> Sent: Friday, November 22, 2019 8:26 AM
>> To: Paul Blakey <paulb@mellanox.com>
>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>> <markb@mellanox.com>
>> Subject: Re: Question about flow table offload in mlx5e
>>
>> Hi Paul,
>>
>>
>> There are some update. I also test it through replacing mlx5e_rep_setup_tc
>> _cb with mlx5e_rep_setup_ft_cb
>>
>>
>> ifconfig mlx_p0 172.168.152.75/24 up
>>
>> ip l add dev tun1 type gretap external
>> tc qdisc add dev tun1 ingress
>> tc qdisc add dev mlx_pf0vf0 ingress
>>
>> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
>> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
>> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
>> nocsum pipe action mirred egress redirect dev tun1
>>
>> tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip
>> 10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
>> enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
>> action mirred egress redirect dev mlx_pf0vf0
>>
>>
>> If you run this script on the host，  and in the virtual machine  run "iperf -c
>> 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 1000"
>>
>> The tcp syn packet will not be offloaded
>>
>>
>> But if you  only run the script  without the last filter as following , The tcp syn
>> packet will be offloaded.
>>
>> ifconfig mlx_p0 172.168.152.75/24 up
>>
>> ip l add dev tun1 type gretap external
>> tc qdisc add dev tun1 ingress
>> tc qdisc add dev mlx_pf0vf0 ingress
>>
>> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
>> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
>> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
>> nocsum pipe action mirred egress redirect dev tun1.
>>
>> I think there are some problem in mlx5e_rep_setup_ft_cb.
>>
>> On 11/21/2019 9:05 PM, Paul Blakey wrote:
>>
>>
>> 	I see, I will test that, and how about normal FWD rules?
>>
>> 	Paul.
>>
>>
>>
>> 		-----Original Message-----
>> 		From: wenxu <wenxu@ucloud.cn>
>> <mailto:wenxu@ucloud.cn>
>> 		Sent: Thursday, November 21, 2019 2:35 PM
>> 		To: Paul Blakey <paulb@mellanox.com>
>> <mailto:paulb@mellanox.com>
>> 		Cc: pablo@netfilter.org <mailto:pablo@netfilter.org> ;
>> netdev@vger.kernel.org <mailto:netdev@vger.kernel.org> ; Mark Bloch
>> 		<markb@mellanox.com> <mailto:markb@mellanox.com>
>> 		Subject: Re: Question about flow table offload in mlx5e
>>
>>
>> 		在 2019/11/21 19:39, Paul Blakey 写道:
>>
>> 			They are good fixes, exactly what we had when we
>> tested this, thanks.
>>
>> 			Regarding encap, I don't know what changes you did,
>> how does the encap
>>
>> 		rule look? Is it a FWD to vxlan device? If not it should be, as
>> our driver
>> 		expects that.
>> 		It is fwd to a gretap devices
>>
>>
>> 			I tried it on my setup via tc, by changing the callback
>> of tc
>>
>> 		(mlx5e_rep_setup_tc_cb) to that of ft
>> (mlx5e_rep_setup_ft_cb),
>>
>> 			and testing a vxlan encap rule:
>> 			sudo tc qdisc add dev ens1f0_0 ingress
>> 			sudo ifconfig ens1f0 7.7.7.7/24 up
>> 			sudo ip link add name vxlan0 type vxlan dev ens1f0
>> remote 7.7.7.8 dstport
>>
>> 		4789 external
>>
>> 			sudo ifconfig vxlan0 up
>> 			sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0
>> protocol ip flower
>>
>> 		dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action
>> tunnel_key set
>> 		src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action
>> mirred egress
>> 		redirect dev vxlan
>>
>>
>> 			then tc show:
>> 			filter protocol ip pref 1 flower chain 0 handle 0x1
>> dst_mac aa:bb:cc:dd:ee:ff
>>
>> 		ip_proto udp skip_sw in_hw in_hw_count 1
>>
>> 			        tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id
>> 1234 dst_port 4789
>>
>> 		csum pipe
>>
>> 			        Stats: used 119 sec      0 pkt
>> 			        mirred (Egress Redirect to device vxlan0)
>> 			        Stats: used 119 sec      0 pkt
>>
>>
>> 		Can you send packet that match this offloaded flow to check
>> it is real
>> 		offloaded?
>>
>> 		In the flowtable offload with my patches both
>> TC_SETUP_BLOCK and
>> 		TC_SETUP_FT can offload the rule success
>>
>> 		But in the TC_SETUP_FT case the packet is not real offloaded.
>>
>>
>> 		I  will test like u did.
>>
>>
>>
>>
>>
>>
>> 				-----Original Message-----
>> 				From: wenxu <wenxu@ucloud.cn>
>> <mailto:wenxu@ucloud.cn>
>> 				Sent: Thursday, November 21, 2019 10:29 AM
>> 				To: Paul Blakey <paulb@mellanox.com>
>> <mailto:paulb@mellanox.com>
>> 				Cc: pablo@netfilter.org
>> <mailto:pablo@netfilter.org> ; netdev@vger.kernel.org
>> <mailto:netdev@vger.kernel.org> ; Mark Bloch
>> 				<markb@mellanox.com>
>> <mailto:markb@mellanox.com>
>> 				Subject: Re: Question about flow table
>> offload in mlx5e
>>
>>
>> 				On 11/21/2019 3:42 PM, Paul Blakey wrote:
>>
>> 					Hi,
>>
>> 					The original design was the block
>> setup to use TC_SETUP_FT type, and
>>
>> 		the
>>
>> 				tc event type to be case
>> TC_SETUP_CLSFLOWER.
>>
>> 					We will post a patch to change that. I
>> would advise to wait till we fix that
>>
>> 				😊
>>
>> 					I'm not sure how you get to this
>> function mlx5e_rep_setup_ft_cb() if it
>>
>> 		the
>>
>> 				nf_flow_table_offload ndo_setup_tc event
>> was TC_SETUP_BLOCK, and
>>
>> 		not
>>
>> 				TC_SETUP_FT.
>>
>>
>> 				Yes I change the TC_SETUP_BLOCK to
>> TC_SETUP_FT in the
>> 				nf_flow_table_offload_setup.
>>
>> 				Two fixes patch provide:
>>
>> 				http://patchwork.ozlabs.org/patch/1197818/
>>
>> 				http://patchwork.ozlabs.org/patch/1197876/
>>
>> 				So this change made by me is not correct
>> currently?
>>
>>
>> 					In our driver en_rep.c we have:
>>
>> 					-------switch (type) {
>> 					-------case TC_SETUP_BLOCK:
>> 					------->-------return
>> flow_block_cb_setup_simple(type_data,
>> 					------->------->------->------->------->---
>> ----
>>
>> 		&mlx5e_rep_block_tc_cb_list,
>>
>> 					------->------->------->------->------->---
>> ----  mlx5e_rep_setup_tc_cb,
>> 					------->------->------->------->------->---
>> ----  priv, priv, true);
>> 					-------case TC_SETUP_FT:
>> 					------->-------return
>> flow_block_cb_setup_simple(type_data,
>> 					------->------->------->------->------->---
>> ----
>>
>> 		&mlx5e_rep_block_ft_cb_list,
>>
>> 					------->------->------->------->------->---
>> ----  mlx5e_rep_setup_ft_cb,
>> 					------->------->------->------->------->---
>> ----  priv, priv, true);
>> 					-------default:
>> 					------->-------return -EOPNOTSUPP;
>> 					-------}
>>
>> 					In nf_flow_table_offload.c:
>>
>> 					-------bo.binder_type>-=
>>
>> 		FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>>
>> 					-------bo.extack>------= &extack;
>> 					-------INIT_LIST_HEAD(&bo.cb_list);
>> 					-------err = dev->netdev_ops-
>>> ndo_setup_tc(dev, TC_SETUP_BLOCK,
>> 				&bo);
>>
>> 					-------if (err < 0)
>> 					------->-------return err;
>> 					-------return
>> nf_flow_table_block_setup(flowtable, &bo, cmd);
>>
>> 					}
>>
>> 	EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>>
>>
>> 					So unless you changed that as well,
>> you should have gotten to
>>
>> 				mlx5e_rep_setup_tc_cb and not
>> mlx5e_rep_setup_tc_ft.
>>
>> 					Regarding the encap action, there
>> should be no difference on which
>>
>> 		chain
>>
>> 				the rule is on.
>>
>>
>> 				But for the same encap rule can be real
>> offloaded when setup through
>> 				through TC_SETUP_BLOCK. But TC_SETUP_FT
>> can't.
>>
>> 				So it is the problem of TC_SETUP_FT in
>> mlx5e_rep_setup_ft_cb ?
>>
>>
>>
>>
>> 					-----Original Message-----
>> 					From: wenxu <wenxu@ucloud.cn>
>> <mailto:wenxu@ucloud.cn>
>> 					Sent: Thursday, November 21, 2019
>> 9:30 AM
>> 					To: Paul Blakey
>> <paulb@mellanox.com> <mailto:paulb@mellanox.com>
>> 					Cc: pablo@netfilter.org
>> <mailto:pablo@netfilter.org> ; netdev@vger.kernel.org
>> <mailto:netdev@vger.kernel.org> ; Mark Bloch
>> 					<markb@mellanox.com>
>> <mailto:markb@mellanox.com>
>> 					Subject: Question about flow table
>> offload in mlx5e
>>
>> 					Hi  paul,
>>
>> 					The flow table offload in the mlx5e is
>> based on TC_SETUP_FT.
>>
>>
>> 					It is almost the same as
>> TC_SETUP_BLOCK.
>>
>> 					It just set
>> MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>> 					cls_flower.common.chain_index =
>> FDB_FT_CHAIN;
>>
>> 					In following codes line 1380 and 1392
>>
>> 					1368 static int
>> mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>> 					*type_data,
>> 					1369                                  void *cb_priv)
>> 					1370 {
>> 					1371         struct flow_cls_offload *f =
>> type_data;
>> 					1372         struct flow_cls_offload
>> cls_flower;
>> 					1373         struct mlx5e_priv *priv =
>> cb_priv;
>> 					1374         struct mlx5_eswitch *esw;
>> 					1375         unsigned long flags;
>> 					1376         int err;
>> 					1377
>> 					1378         flags =
>> MLX5_TC_FLAG(INGRESS) |
>> 					1379
>> MLX5_TC_FLAG(ESW_OFFLOAD) |
>> 					1380
>> MLX5_TC_FLAG(FT_OFFLOAD);
>> 					1381         esw = priv->mdev-
>>> priv.eswitch;
>> 					1382
>> 					1383         switch (type) {
>> 					1384         case
>> TC_SETUP_CLSFLOWER:
>> 					1385                 if
>> (!mlx5_eswitch_prios_supported(esw) || f-
>>
>> 					common.chain_index)
>>
>> 					1386                         return -
>> EOPNOTSUPP;
>> 					1387
>> 					1388                 /* Re-use tc offload
>> path by moving the ft flow to the
>> 					1389                  * reserved ft chain.
>> 					1390                  */
>> 					1391                 memcpy(&cls_flower, f,
>> sizeof(*f));
>> 					1392
>> cls_flower.common.chain_index = FDB_FT_CHAIN;
>> 					1393                 err =
>> mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>>
>> 				flags);
>>
>> 					1394                 memcpy(&f->stats,
>> &cls_flower.stats, sizeof(f->stats));
>>
>>
>> 					I want to add tunnel offload support
>> in the flow table, I  add some
>>
>> 		patches
>>
>> 				in
>>
>> 					nf_flow_table_offload.
>>
>> 					Also add the indr setup support in the
>> mlx driver. And Now I can  flow
>>
>> 				table
>>
>> 					offload with decap.
>>
>>
>> 					But I meet a problem with the encap.
>> The encap rule can be added in
>> 					hardware  successfully But it can't be
>> offloaded.
>>
>> 					But I think the rule I added is correct.
>> If I mask the line 1392. The rule
>>
>> 		also
>>
>> 				can
>>
>> 					be add success and can be offloaded.
>>
>> 					So there are some limit for encap
>> operation for FT_OFFLOAD in
>> 					FDB_FT_CHAIN?
>>
>>
>> 					BR
>>
>> 					wenxu
>>

Re: Question about flow table offload in mlx5e

[wenxu]

Hi Paul,

Did your test for this case? There are some problem that I reported?


BR

wenxu 

On 11/24/2019 4:46 PM, Paul Blakey wrote:
> Hi,
>
> The syn packet might not be actually offloaded because there isn't a neighbor to resolve the destination mac for the tunnel destination ip (next hop mac).
> Try setting the neighbor via "ip neigh replace dev mlx5_p0 172.168.152.241 lladdr <next hop mac>"
> Or running ping to 172.168.152.241 before adding (or in background) the rule to resolve the mac and make available.
> I'll test it on my end.
>
>
> Thanks,
> Paul.
>
>
>> -----Original Message-----
>> From: wenxu <wenxu@ucloud.cn>
>> Sent: Friday, November 22, 2019 8:26 AM
>> To: Paul Blakey <paulb@mellanox.com>
>> Cc: pablo@netfilter.org; netdev@vger.kernel.org; Mark Bloch
>> <markb@mellanox.com>
>> Subject: Re: Question about flow table offload in mlx5e
>>
>> Hi Paul,
>>
>>
>> There are some update. I also test it through replacing mlx5e_rep_setup_tc
>> _cb with mlx5e_rep_setup_ft_cb
>>
>>
>> ifconfig mlx_p0 172.168.152.75/24 up
>>
>> ip l add dev tun1 type gretap external
>> tc qdisc add dev tun1 ingress
>> tc qdisc add dev mlx_pf0vf0 ingress
>>
>> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
>> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
>> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
>> nocsum pipe action mirred egress redirect dev tun1
>>
>> tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip
>> 10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
>> enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
>> action mirred egress redirect dev mlx_pf0vf0
>>
>>
>> If you run this script on the host，  and in the virtual machine  run "iperf -c
>> 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 1000"
>>
>> The tcp syn packet will not be offloaded
>>
>>
>> But if you  only run the script  without the last filter as following , The tcp syn
>> packet will be offloaded.
>>
>> ifconfig mlx_p0 172.168.152.75/24 up
>>
>> ip l add dev tun1 type gretap external
>> tc qdisc add dev tun1 ingress
>> tc qdisc add dev mlx_pf0vf0 ingress
>>
>> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
>> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
>> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
>> nocsum pipe action mirred egress redirect dev tun1.
>>
>> I think there are some problem in mlx5e_rep_setup_ft_cb.
>>
>> On 11/21/2019 9:05 PM, Paul Blakey wrote:
>>
>>
>> 	I see, I will test that, and how about normal FWD rules?
>>
>> 	Paul.
>>
>>
>>
>> 		-----Original Message-----
>> 		From: wenxu <wenxu@ucloud.cn>
>> <mailto:wenxu@ucloud.cn>
>> 		Sent: Thursday, November 21, 2019 2:35 PM
>> 		To: Paul Blakey <paulb@mellanox.com>
>> <mailto:paulb@mellanox.com>
>> 		Cc: pablo@netfilter.org <mailto:pablo@netfilter.org> ;
>> netdev@vger.kernel.org <mailto:netdev@vger.kernel.org> ; Mark Bloch
>> 		<markb@mellanox.com> <mailto:markb@mellanox.com>
>> 		Subject: Re: Question about flow table offload in mlx5e
>>
>>
>> 		在 2019/11/21 19:39, Paul Blakey 写道:
>>
>> 			They are good fixes, exactly what we had when we
>> tested this, thanks.
>>
>> 			Regarding encap, I don't know what changes you did,
>> how does the encap
>>
>> 		rule look? Is it a FWD to vxlan device? If not it should be, as
>> our driver
>> 		expects that.
>> 		It is fwd to a gretap devices
>>
>>
>> 			I tried it on my setup via tc, by changing the callback
>> of tc
>>
>> 		(mlx5e_rep_setup_tc_cb) to that of ft
>> (mlx5e_rep_setup_ft_cb),
>>
>> 			and testing a vxlan encap rule:
>> 			sudo tc qdisc add dev ens1f0_0 ingress
>> 			sudo ifconfig ens1f0 7.7.7.7/24 up
>> 			sudo ip link add name vxlan0 type vxlan dev ens1f0
>> remote 7.7.7.8 dstport
>>
>> 		4789 external
>>
>> 			sudo ifconfig vxlan0 up
>> 			sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0
>> protocol ip flower
>>
>> 		dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action
>> tunnel_key set
>> 		src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action
>> mirred egress
>> 		redirect dev vxlan
>>
>>
>> 			then tc show:
>> 			filter protocol ip pref 1 flower chain 0 handle 0x1
>> dst_mac aa:bb:cc:dd:ee:ff
>>
>> 		ip_proto udp skip_sw in_hw in_hw_count 1
>>
>> 			        tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id
>> 1234 dst_port 4789
>>
>> 		csum pipe
>>
>> 			        Stats: used 119 sec      0 pkt
>> 			        mirred (Egress Redirect to device vxlan0)
>> 			        Stats: used 119 sec      0 pkt
>>
>>
>> 		Can you send packet that match this offloaded flow to check
>> it is real
>> 		offloaded?
>>
>> 		In the flowtable offload with my patches both
>> TC_SETUP_BLOCK and
>> 		TC_SETUP_FT can offload the rule success
>>
>> 		But in the TC_SETUP_FT case the packet is not real offloaded.
>>
>>
>> 		I  will test like u did.
>>
>>
>>
>>
>>
>>
>> 				-----Original Message-----
>> 				From: wenxu <wenxu@ucloud.cn>
>> <mailto:wenxu@ucloud.cn>
>> 				Sent: Thursday, November 21, 2019 10:29 AM
>> 				To: Paul Blakey <paulb@mellanox.com>
>> <mailto:paulb@mellanox.com>
>> 				Cc: pablo@netfilter.org
>> <mailto:pablo@netfilter.org> ; netdev@vger.kernel.org
>> <mailto:netdev@vger.kernel.org> ; Mark Bloch
>> 				<markb@mellanox.com>
>> <mailto:markb@mellanox.com>
>> 				Subject: Re: Question about flow table
>> offload in mlx5e
>>
>>
>> 				On 11/21/2019 3:42 PM, Paul Blakey wrote:
>>
>> 					Hi,
>>
>> 					The original design was the block
>> setup to use TC_SETUP_FT type, and
>>
>> 		the
>>
>> 				tc event type to be case
>> TC_SETUP_CLSFLOWER.
>>
>> 					We will post a patch to change that. I
>> would advise to wait till we fix that
>>
>> 				😊
>>
>> 					I'm not sure how you get to this
>> function mlx5e_rep_setup_ft_cb() if it
>>
>> 		the
>>
>> 				nf_flow_table_offload ndo_setup_tc event
>> was TC_SETUP_BLOCK, and
>>
>> 		not
>>
>> 				TC_SETUP_FT.
>>
>>
>> 				Yes I change the TC_SETUP_BLOCK to
>> TC_SETUP_FT in the
>> 				nf_flow_table_offload_setup.
>>
>> 				Two fixes patch provide:
>>
>> 				http://patchwork.ozlabs.org/patch/1197818/
>>
>> 				http://patchwork.ozlabs.org/patch/1197876/
>>
>> 				So this change made by me is not correct
>> currently?
>>
>>
>> 					In our driver en_rep.c we have:
>>
>> 					-------switch (type) {
>> 					-------case TC_SETUP_BLOCK:
>> 					------->-------return
>> flow_block_cb_setup_simple(type_data,
>> 					------->------->------->------->------->---
>> ----
>>
>> 		&mlx5e_rep_block_tc_cb_list,
>>
>> 					------->------->------->------->------->---
>> ----  mlx5e_rep_setup_tc_cb,
>> 					------->------->------->------->------->---
>> ----  priv, priv, true);
>> 					-------case TC_SETUP_FT:
>> 					------->-------return
>> flow_block_cb_setup_simple(type_data,
>> 					------->------->------->------->------->---
>> ----
>>
>> 		&mlx5e_rep_block_ft_cb_list,
>>
>> 					------->------->------->------->------->---
>> ----  mlx5e_rep_setup_ft_cb,
>> 					------->------->------->------->------->---
>> ----  priv, priv, true);
>> 					-------default:
>> 					------->-------return -EOPNOTSUPP;
>> 					-------}
>>
>> 					In nf_flow_table_offload.c:
>>
>> 					-------bo.binder_type>-=
>>
>> 		FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>>
>> 					-------bo.extack>------= &extack;
>> 					-------INIT_LIST_HEAD(&bo.cb_list);
>> 					-------err = dev->netdev_ops-
>>> ndo_setup_tc(dev, TC_SETUP_BLOCK,
>> 				&bo);
>>
>> 					-------if (err < 0)
>> 					------->-------return err;
>> 					-------return
>> nf_flow_table_block_setup(flowtable, &bo, cmd);
>>
>> 					}
>>
>> 	EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>>
>>
>> 					So unless you changed that as well,
>> you should have gotten to
>>
>> 				mlx5e_rep_setup_tc_cb and not
>> mlx5e_rep_setup_tc_ft.
>>
>> 					Regarding the encap action, there
>> should be no difference on which
>>
>> 		chain
>>
>> 				the rule is on.
>>
>>
>> 				But for the same encap rule can be real
>> offloaded when setup through
>> 				through TC_SETUP_BLOCK. But TC_SETUP_FT
>> can't.
>>
>> 				So it is the problem of TC_SETUP_FT in
>> mlx5e_rep_setup_ft_cb ?
>>
>>
>>
>>
>> 					-----Original Message-----
>> 					From: wenxu <wenxu@ucloud.cn>
>> <mailto:wenxu@ucloud.cn>
>> 					Sent: Thursday, November 21, 2019
>> 9:30 AM
>> 					To: Paul Blakey
>> <paulb@mellanox.com> <mailto:paulb@mellanox.com>
>> 					Cc: pablo@netfilter.org
>> <mailto:pablo@netfilter.org> ; netdev@vger.kernel.org
>> <mailto:netdev@vger.kernel.org> ; Mark Bloch
>> 					<markb@mellanox.com>
>> <mailto:markb@mellanox.com>
>> 					Subject: Question about flow table
>> offload in mlx5e
>>
>> 					Hi  paul,
>>
>> 					The flow table offload in the mlx5e is
>> based on TC_SETUP_FT.
>>
>>
>> 					It is almost the same as
>> TC_SETUP_BLOCK.
>>
>> 					It just set
>> MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>> 					cls_flower.common.chain_index =
>> FDB_FT_CHAIN;
>>
>> 					In following codes line 1380 and 1392
>>
>> 					1368 static int
>> mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>> 					*type_data,
>> 					1369                                  void *cb_priv)
>> 					1370 {
>> 					1371         struct flow_cls_offload *f =
>> type_data;
>> 					1372         struct flow_cls_offload
>> cls_flower;
>> 					1373         struct mlx5e_priv *priv =
>> cb_priv;
>> 					1374         struct mlx5_eswitch *esw;
>> 					1375         unsigned long flags;
>> 					1376         int err;
>> 					1377
>> 					1378         flags =
>> MLX5_TC_FLAG(INGRESS) |
>> 					1379
>> MLX5_TC_FLAG(ESW_OFFLOAD) |
>> 					1380
>> MLX5_TC_FLAG(FT_OFFLOAD);
>> 					1381         esw = priv->mdev-
>>> priv.eswitch;
>> 					1382
>> 					1383         switch (type) {
>> 					1384         case
>> TC_SETUP_CLSFLOWER:
>> 					1385                 if
>> (!mlx5_eswitch_prios_supported(esw) || f-
>>
>> 					common.chain_index)
>>
>> 					1386                         return -
>> EOPNOTSUPP;
>> 					1387
>> 					1388                 /* Re-use tc offload
>> path by moving the ft flow to the
>> 					1389                  * reserved ft chain.
>> 					1390                  */
>> 					1391                 memcpy(&cls_flower, f,
>> sizeof(*f));
>> 					1392
>> cls_flower.common.chain_index = FDB_FT_CHAIN;
>> 					1393                 err =
>> mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>>
>> 				flags);
>>
>> 					1394                 memcpy(&f->stats,
>> &cls_flower.stats, sizeof(f->stats));
>>
>>
>> 					I want to add tunnel offload support
>> in the flow table, I  add some
>>
>> 		patches
>>
>> 				in
>>
>> 					nf_flow_table_offload.
>>
>> 					Also add the indr setup support in the
>> mlx driver. And Now I can  flow
>>
>> 				table
>>
>> 					offload with decap.
>>
>>
>> 					But I meet a problem with the encap.
>> The encap rule can be added in
>> 					hardware  successfully But it can't be
>> offloaded.
>>
>> 					But I think the rule I added is correct.
>> If I mask the line 1392. The rule
>>
>> 		also
>>
>> 				can
>>
>> 					be add success and can be offloaded.
>>
>> 					So there are some limit for encap
>> operation for FT_OFFLOAD in
>> 					FDB_FT_CHAIN?
>>
>>
>> 					BR
>>
>> 					wenxu
>>

Re: Question about flow table offload in mlx5e

[wenxu]

Sorry maybe something mess you,  Ignore with my patches.


I also did the test like you with route tc rules to ft callback.


please also did the following test:  mlx_p0 is the pf and mlx_pf0vf0 is the vf .

ifconfig mlx_p0 172.168.152.75/24 up
ip n replace 172.168.152.241 dev mlx_p0 lladdr aa:bb:cc:dd:ee:ff

ip l add dev tun1 type gretap external
tc qdisc add dev tun1 ingress
tc qdisc add dev mlx_pf0vf0 ingress

tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
nocsum pipe action mirred egress redirect dev tun1

In the virtual machine:
ifconfig eth0 10.0.0.75/24
ip r a default via 10.0.0.1
ip n replace 10.0.0.1 dev eth0 lladdr aa:bb:cc:dd:ee:01

iperf -c 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 10


The script above can offload the syn packets, The packet can't be captured on mlx_pf0vf0.

I think the rule is ok.  The problem is that if I add another rule in device tun1 as following.
It will lead the syn packet can't be offloaded

tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip
10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
action mirred egress redirect dev mlx_pf0vf0






在 2019/11/27 19:51, Paul Blakey 写道:
> Sorry I didn't have time apply your patches.
>
> I did test it again with route tc rules to ft callback, here's the diff:
>
> @@ -1291,7 +1304,7 @@ static int mlx5e_rep_setup_tc(struct net_device *dev, enum tc_setup_type type,
>         case TC_SETUP_BLOCK:
>                 return flow_block_cb_setup_simple(type_data,
>                                                   &mlx5e_rep_block_tc_cb_list,
> -                                                 mlx5e_rep_setup_tc_cb,
> +                                                 mlx5e_rep_setup_ft_cb,
>                                                   priv, priv, true);
>         case TC_SETUP_FT:
>                 return flow_block_cb_setup_simple(type_data,
>
>
> I ran this script after creating a VF (ens1f2) and entering switchdev mode (creating represntor ens1f0_0):
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> ifconfig tun1 up
>
> ifconfig ens1f0_0 0 up
> tc qdisc add dev ens1f0_0 ingress
>
> ifconfig ens1f0 172.168.152.75/24 up
> ip n replace 172.168.162.241 dev ens1f0 lladdr aa:bb:cc:dd:ee:01
>
> tc filter del dev ens1f0_0 ingress
>
> tc filter add dev ens1f0_0 pref 2 ingress proto ip flower \
>      skip_sw \
>      ip_proto tcp dst_ip 5.5.5.6 src_ip 5.5.5.5 tcp_flags 0/0x5 \
>      action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000 nocsum pipe \
>      action mirred egress redirect dev tun1
>
> tc filter add dev ens1f0_0 pref 1 ingress proto ip flower \
>      skip_hw \
>      action drop
>
> ifconfig ens1f2 5.5.5.5/24 up
> ip n replace 5.5.5.6 dev ens1f2 lladdr aa:bb:cc:dd:ee:ff
>
> timeout 3 iperf -c 5.5.5.6
>
> tc -s filter show dev ens1f0_0 ingress
>
>
>
> Notice I run iperf client without Iperf server, so I get only syn packets.
>
> Here is the tcpdump on the VF (ens1f2):
>
> Executing: sudo tcpdump -nnep  -i ens1f2
>
> Executing: sudo tcpdump -nnep  -i ens1f2
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on ens1f2, link-type EN10MB (Ethernet), capture size 262144 bytes
> 13:49:10.610376 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249857484 ecr 0,nop,wscale 7], length 0
> 13:49:11.616262 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249858489 ecr 0,nop,wscale 7], length 0
> 13:49:13.664261 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249860537 ecr 0,nop,wscale 7], length 0
> 13:49:17.696261 24:8a:07:a5:28:01 > aa:bb:cc:dd:e
>
> I get:
>
> filter protocol ip pref 1 flower chain 0
> filter protocol ip pref 1 flower chain 0 handle 0x1
>   eth_type ipv4
>   skip_hw
>   not_in_hw
>         action order 1: gact action drop
>          random type none pass val 0
>          index 1 ref 1 bind 1 installed 3 sec used 3 sec
>         Action statistics:
>         Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>         backlog 0b 0p requeues 0
>
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
>   eth_type ipv4
>   ip_proto tcp
>   dst_ip 5.5.5.6
>   src_ip 5.5.5.5
>   tcp_flags 0/5
>   skip_sw
>   in_hw
>         action order 1: tunnel_key set
>         src_ip 0.0.0.0
>         dst_ip 172.168.152.241
>         key_id 1000
>         nocsum pipe
>         index 1 ref 1 bind 1 installed 3 sec used 3 sec
>         Action statistics:
>         Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>         backlog 0b 0p requeues 0
>
>         action order 2: mirred (Egress Redirect to device tun1) stolen
>         index 1 ref 1 bind 1 installed 3 sec used 1 sec
>         Action statistics:
>         Sent 232 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
>         backlog 0b 0p requeues 0
>
>
> And it counts the 2 syn packets in hardware, the packets are leaving the VF (ens1f2) and not arriving at the mlx5 representor device (ens1f0_0),
>
> which means hardware got them. It's also couned in the above encap rule. And the software only (skip_hw, prio 1) rule didn't catch any packets.
>
>
> Thanks,
>
> Paul.
>
>
>
> On 11/26/2019 10:18 AM, wenxu wrote:
>
> Hi Paul,
>
> Did your test for this case? There are some problem that I reported?
>
>
> BR
>
> wenxu
>
> On 11/24/2019 4:46 PM, Paul Blakey wrote:
>
>
> Hi,
>
> The syn packet might not be actually offloaded because there isn't a neighbor to resolve the destination mac for the tunnel destination ip (next hop mac).
> Try setting the neighbor via "ip neigh replace dev mlx5_p0 172.168.152.241 lladdr <next hop mac>"
> Or running ping to 172.168.152.241 before adding (or in background) the rule to resolve the mac and make available.
> I'll test it on my end.
>
>
> Thanks,
> Paul.
>
>
>
>
> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> Sent: Friday, November 22, 2019 8:26 AM
> To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>; Mark Bloch
> <markb@mellanox.com><mailto:markb@mellanox.com>
> Subject: Re: Question about flow table offload in mlx5e
>
> Hi Paul,
>
>
> There are some update. I also test it through replacing mlx5e_rep_setup_tc
> _cb with mlx5e_rep_setup_ft_cb
>
>
> ifconfig mlx_p0 172.168.152.75/24 up
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
>
> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1
>
> tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip
> 10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
> enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
> action mirred egress redirect dev mlx_pf0vf0
>
>
> If you run this script on the host，  and in the virtual machine  run "iperf -c
> 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 1000"
>
> The tcp syn packet will not be offloaded
>
>
> But if you  only run the script  without the last filter as following , The tcp syn
> packet will be offloaded.
>
> ifconfig mlx_p0 172.168.152.75/24 up
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
>
> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1.
>
> I think there are some problem in mlx5e_rep_setup_ft_cb.
>
> On 11/21/2019 9:05 PM, Paul Blakey wrote:
>
>
>         I see, I will test that, and how about normal FWD rules?
>
>         Paul.
>
>
>
>                 -----Original Message-----
>                 From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
>                 Sent: Thursday, November 21, 2019 2:35 PM
>                 To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
>                 Cc: pablo@netfilter.org<mailto:pablo@netfilter.org> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ;
> netdev@vger.kernel.org<mailto:netdev@vger.kernel.org> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
>                 <markb@mellanox.com><mailto:markb@mellanox.com> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
>                 Subject: Re: Question about flow table offload in mlx5e
>
>
>                 在 2019/11/21 19:39, Paul Blakey 写道:
>
>                         They are good fixes, exactly what we had when we
> tested this, thanks.
>
>                         Regarding encap, I don't know what changes you did,
> how does the encap
>
>                 rule look? Is it a FWD to vxlan device? If not it should be, as
> our driver
>                 expects that.
>                 It is fwd to a gretap devices
>
>
>                         I tried it on my setup via tc, by changing the callback
> of tc
>
>                 (mlx5e_rep_setup_tc_cb) to that of ft
> (mlx5e_rep_setup_ft_cb),
>
>                         and testing a vxlan encap rule:
>                         sudo tc qdisc add dev ens1f0_0 ingress
>                         sudo ifconfig ens1f0 7.7.7.7/24 up
>                         sudo ip link add name vxlan0 type vxlan dev ens1f0
> remote 7.7.7.8 dstport
>
>                 4789 external
>
>                         sudo ifconfig vxlan0 up
>                         sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0
> protocol ip flower
>
>                 dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action
> tunnel_key set
>                 src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action
> mirred egress
>                 redirect dev vxlan
>
>
>                         then tc show:
>                         filter protocol ip pref 1 flower chain 0 handle 0x1
> dst_mac aa:bb:cc:dd:ee:ff
>
>                 ip_proto udp skip_sw in_hw in_hw_count 1
>
>                                 tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id
> 1234 dst_port 4789
>
>                 csum pipe
>
>                                 Stats: used 119 sec      0 pkt
>                                 mirred (Egress Redirect to device vxlan0)
>                                 Stats: used 119 sec      0 pkt
>
>
>                 Can you send packet that match this offloaded flow to check
> it is real
>                 offloaded?
>
>                 In the flowtable offload with my patches both
> TC_SETUP_BLOCK and
>                 TC_SETUP_FT can offload the rule success
>
>                 But in the TC_SETUP_FT case the packet is not real offloaded.
>
>
>                 I  will test like u did.
>
>
>
>
>
>
>                                 -----Original Message-----
>                                 From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
>                                 Sent: Thursday, November 21, 2019 10:29 AM
>                                 To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
>                                 Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>
> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>
> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
>                                 <markb@mellanox.com><mailto:markb@mellanox.com>
> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
>                                 Subject: Re: Question about flow table
> offload in mlx5e
>
>
>                                 On 11/21/2019 3:42 PM, Paul Blakey wrote:
>
>                                         Hi,
>
>                                         The original design was the block
> setup to use TC_SETUP_FT type, and
>
>                 the
>
>                                 tc event type to be case
> TC_SETUP_CLSFLOWER.
>
>                                         We will post a patch to change that. I
> would advise to wait till we fix that
>
>                                 😊
>
>                                         I'm not sure how you get to this
> function mlx5e_rep_setup_ft_cb() if it
>
>                 the
>
>                                 nf_flow_table_offload ndo_setup_tc event
> was TC_SETUP_BLOCK, and
>
>                 not
>
>                                 TC_SETUP_FT.
>
>
>                                 Yes I change the TC_SETUP_BLOCK to
> TC_SETUP_FT in the
>                                 nf_flow_table_offload_setup.
>
>                                 Two fixes patch provide:
>
>                                 http://patchwork.ozlabs.org/patch/1197818/
>
>                                 http://patchwork.ozlabs.org/patch/1197876/
>
>                                 So this change made by me is not correct
> currently?
>
>
>                                         In our driver en_rep.c we have:
>
>                                         -------switch (type) {
>                                         -------case TC_SETUP_BLOCK:
>                                         ------->-------return
> flow_block_cb_setup_simple(type_data,
>                                         ------->------->------->------->------->---
> ----
>
>                 &mlx5e_rep_block_tc_cb_list,
>
>                                         ------->------->------->------->------->---
> ----  mlx5e_rep_setup_tc_cb,
>                                         ------->------->------->------->------->---
> ----  priv, priv, true);
>                                         -------case TC_SETUP_FT:
>                                         ------->-------return
> flow_block_cb_setup_simple(type_data,
>                                         ------->------->------->------->------->---
> ----
>
>                 &mlx5e_rep_block_ft_cb_list,
>
>                                         ------->------->------->------->------->---
> ----  mlx5e_rep_setup_ft_cb,
>                                         ------->------->------->------->------->---
> ----  priv, priv, true);
>                                         -------default:
>                                         ------->-------return -EOPNOTSUPP;
>                                         -------}
>
>                                         In nf_flow_table_offload.c:
>
>                                         -------bo.binder_type>-=
>
>                 FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>
>                                         -------bo.extack>------= &extack;
>                                         -------INIT_LIST_HEAD(&bo.cb_list);
>                                         -------err = dev->netdev_ops-
>
>
> ndo_setup_tc(dev, TC_SETUP_BLOCK,
>
>
>                                 &bo);
>
>                                         -------if (err < 0)
>                                         ------->-------return err;
>                                         -------return
> nf_flow_table_block_setup(flowtable, &bo, cmd);
>
>                                         }
>
>         EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>
>
>                                         So unless you changed that as well,
> you should have gotten to
>
>                                 mlx5e_rep_setup_tc_cb and not
> mlx5e_rep_setup_tc_ft.
>
>                                         Regarding the encap action, there
> should be no difference on which
>
>                 chain
>
>                                 the rule is on.
>
>
>                                 But for the same encap rule can be real
> offloaded when setup through
>                                 through TC_SETUP_BLOCK. But TC_SETUP_FT
> can't.
>
>                                 So it is the problem of TC_SETUP_FT in
> mlx5e_rep_setup_ft_cb ?
>
>
>
>
>                                         -----Original Message-----
>                                         From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
>                                         Sent: Thursday, November 21, 2019
> 9:30 AM
>                                         To: Paul Blakey
> <paulb@mellanox.com><mailto:paulb@mellanox.com> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
>                                         Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>
> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>
> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
>                                         <markb@mellanox.com><mailto:markb@mellanox.com>
> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
>                                         Subject: Question about flow table
> offload in mlx5e
>
>                                         Hi  paul,
>
>                                         The flow table offload in the mlx5e is
> based on TC_SETUP_FT.
>
>
>                                         It is almost the same as
> TC_SETUP_BLOCK.
>
>                                         It just set
> MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>                                         cls_flower.common.chain_index =
> FDB_FT_CHAIN;
>
>                                         In following codes line 1380 and 1392
>
>                                         1368 static int
> mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>                                         *type_data,
>                                         1369                                  void *cb_priv)
>                                         1370 {
>                                         1371         struct flow_cls_offload *f =
> type_data;
>                                         1372         struct flow_cls_offload
> cls_flower;
>                                         1373         struct mlx5e_priv *priv =
> cb_priv;
>                                         1374         struct mlx5_eswitch *esw;
>                                         1375         unsigned long flags;
>                                         1376         int err;
>                                         1377
>                                         1378         flags =
> MLX5_TC_FLAG(INGRESS) |
>                                         1379
> MLX5_TC_FLAG(ESW_OFFLOAD) |
>                                         1380
> MLX5_TC_FLAG(FT_OFFLOAD);
>                                         1381         esw = priv->mdev-
>
>
> priv.eswitch;
>
>
>                                         1382
>                                         1383         switch (type) {
>                                         1384         case
> TC_SETUP_CLSFLOWER:
>                                         1385                 if
> (!mlx5_eswitch_prios_supported(esw) || f-
>
>                                         common.chain_index)
>
>                                         1386                         return -
> EOPNOTSUPP;
>                                         1387
>                                         1388                 /* Re-use tc offload
> path by moving the ft flow to the
>                                         1389                  * reserved ft chain.
>                                         1390                  */
>                                         1391                 memcpy(&cls_flower, f,
> sizeof(*f));
>                                         1392
> cls_flower.common.chain_index = FDB_FT_CHAIN;
>                                         1393                 err =
> mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>
>                                 flags);
>
>                                         1394                 memcpy(&f->stats,
> &cls_flower.stats, sizeof(f->stats));
>
>
>                                         I want to add tunnel offload support
> in the flow table, I  add some
>
>                 patches
>
>                                 in
>
>                                         nf_flow_table_offload.
>
>                                         Also add the indr setup support in the
> mlx driver. And Now I can  flow
>
>                                 table
>
>                                         offload with decap.
>
>
>                                         But I meet a problem with the encap.
> The encap rule can be added in
>                                         hardware  successfully But it can't be
> offloaded.
>
>                                         But I think the rule I added is correct.
> If I mask the line 1392. The rule
>
>                 also
>
>                                 can
>
>                                         be add success and can be offloaded.
>
>                                         So there are some limit for encap
> operation for FT_OFFLOAD in
>                                         FDB_FT_CHAIN?
>
>
>                                         BR
>
>                                         wenxu
>
>

Re: Question about flow table offload in mlx5e

[wenxu]

Sorry maybe something mess you,  Ignore with my patches.


I also did the test like you with route tc rules to ft callback.


please also did the following test:  mlx_p0 is the pf and mlx_pf0vf0 is the vf .

ifconfig mlx_p0 172.168.152.75/24 up
ip n replace 172.168.152.241 dev mlx_p0 lladdr aa:bb:cc:dd:ee:ff

ip l add dev tun1 type gretap external
tc qdisc add dev tun1 ingress
tc qdisc add dev mlx_pf0vf0 ingress

tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
nocsum pipe action mirred egress redirect dev tun1

In the virtual machine:
ifconfig eth0 10.0.0.75/24
ip r a default via 10.0.0.1
ip n replace 10.0.0.1 dev eth0 lladdr aa:bb:cc:dd:ee:01

iperf -c 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 10


The script above can offload the syn packets, The packet can't be captured on mlx_pf0vf0.

I think the rule is ok.  The problem is that if I add another rule in device tun1 as following
It will lead the syn packet can't be offloaded

tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip
10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
action mirred egress redirect dev mlx_pf0vf0






在 2019/11/27 19:51, Paul Blakey 写道:
> Sorry I didn't have time apply your patches.
>
> I did test it again with route tc rules to ft callback, here's the diff:
>
> @@ -1291,7 +1304,7 @@ static int mlx5e_rep_setup_tc(struct net_device *dev, enum tc_setup_type type,
>         case TC_SETUP_BLOCK:
>                 return flow_block_cb_setup_simple(type_data,
>                                                   &mlx5e_rep_block_tc_cb_list,
> -                                                 mlx5e_rep_setup_tc_cb,
> +                                                 mlx5e_rep_setup_ft_cb,
>                                                   priv, priv, true);
>         case TC_SETUP_FT:
>                 return flow_block_cb_setup_simple(type_data,
>
>
> I ran this script after creating a VF (ens1f2) and entering switchdev mode (creating represntor ens1f0_0):
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> ifconfig tun1 up
>
> ifconfig ens1f0_0 0 up
> tc qdisc add dev ens1f0_0 ingress
>
> ifconfig ens1f0 172.168.152.75/24 up
> ip n replace 172.168.162.241 dev ens1f0 lladdr aa:bb:cc:dd:ee:01
>
> tc filter del dev ens1f0_0 ingress
>
> tc filter add dev ens1f0_0 pref 2 ingress proto ip flower \
>      skip_sw \
>      ip_proto tcp dst_ip 5.5.5.6 src_ip 5.5.5.5 tcp_flags 0/0x5 \
>      action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000 nocsum pipe \
>      action mirred egress redirect dev tun1
>
> tc filter add dev ens1f0_0 pref 1 ingress proto ip flower \
>      skip_hw \
>      action drop
>
> ifconfig ens1f2 5.5.5.5/24 up
> ip n replace 5.5.5.6 dev ens1f2 lladdr aa:bb:cc:dd:ee:ff
>
> timeout 3 iperf -c 5.5.5.6
>
> tc -s filter show dev ens1f0_0 ingress
>
>
>
> Notice I run iperf client without Iperf server, so I get only syn packets.
>
> Here is the tcpdump on the VF (ens1f2):
>
> Executing: sudo tcpdump -nnep  -i ens1f2
>
> Executing: sudo tcpdump -nnep  -i ens1f2
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on ens1f2, link-type EN10MB (Ethernet), capture size 262144 bytes
> 13:49:10.610376 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249857484 ecr 0,nop,wscale 7], length 0
> 13:49:11.616262 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249858489 ecr 0,nop,wscale 7], length 0
> 13:49:13.664261 24:8a:07:a5:28:01 > aa:bb:cc:dd:ee:ff, ethertype IPv4 (0x0800), length 74: 5.5.5.5.49846 > 5.5.5.6.5001: Flags [S], seq 1395738427, win 64240, options [mss 1460,sackOK,TS val 2249860537 ecr 0,nop,wscale 7], length 0
> 13:49:17.696261 24:8a:07:a5:28:01 > aa:bb:cc:dd:e
>
> I get:
>
> filter protocol ip pref 1 flower chain 0
> filter protocol ip pref 1 flower chain 0 handle 0x1
>   eth_type ipv4
>   skip_hw
>   not_in_hw
>         action order 1: gact action drop
>          random type none pass val 0
>          index 1 ref 1 bind 1 installed 3 sec used 3 sec
>         Action statistics:
>         Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>         backlog 0b 0p requeues 0
>
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
>   eth_type ipv4
>   ip_proto tcp
>   dst_ip 5.5.5.6
>   src_ip 5.5.5.5
>   tcp_flags 0/5
>   skip_sw
>   in_hw
>         action order 1: tunnel_key set
>         src_ip 0.0.0.0
>         dst_ip 172.168.152.241
>         key_id 1000
>         nocsum pipe
>         index 1 ref 1 bind 1 installed 3 sec used 3 sec
>         Action statistics:
>         Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>         backlog 0b 0p requeues 0
>
>         action order 2: mirred (Egress Redirect to device tun1) stolen
>         index 1 ref 1 bind 1 installed 3 sec used 1 sec
>         Action statistics:
>         Sent 232 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
>         backlog 0b 0p requeues 0
>
>
> And it counts the 2 syn packets in hardware, the packets are leaving the VF (ens1f2) and not arriving at the mlx5 representor device (ens1f0_0),
>
> which means hardware got them. It's also couned in the above encap rule. And the software only (skip_hw, prio 1) rule didn't catch any packets.
>
>
> Thanks,
>
> Paul.
>
>
>
> On 11/26/2019 10:18 AM, wenxu wrote:
>
> Hi Paul,
>
> Did your test for this case? There are some problem that I reported?
>
>
> BR
>
> wenxu
>
> On 11/24/2019 4:46 PM, Paul Blakey wrote:
>
>
> Hi,
>
> The syn packet might not be actually offloaded because there isn't a neighbor to resolve the destination mac for the tunnel destination ip (next hop mac).
> Try setting the neighbor via "ip neigh replace dev mlx5_p0 172.168.152.241 lladdr <next hop mac>"
> Or running ping to 172.168.152.241 before adding (or in background) the rule to resolve the mac and make available.
> I'll test it on my end.
>
>
> Thanks,
> Paul.
>
>
>
>
> -----Original Message-----
> From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> Sent: Friday, November 22, 2019 8:26 AM
> To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>; Mark Bloch
> <markb@mellanox.com><mailto:markb@mellanox.com>
> Subject: Re: Question about flow table offload in mlx5e
>
> Hi Paul,
>
>
> There are some update. I also test it through replacing mlx5e_rep_setup_tc
> _cb with mlx5e_rep_setup_ft_cb
>
>
> ifconfig mlx_p0 172.168.152.75/24 up
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
>
> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1
>
> tc filter add dev tun1 pref 2 ingress  protocol ip flower ip_proto tcp src_ip
> 10.0.1.241 dst_ip 10.0.0.75 src_port 5001 dst_port 5002 tcp_flags 0/0x5
> enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe
> action mirred egress redirect dev mlx_pf0vf0
>
>
> If you run this script on the host，  and in the virtual machine  run "iperf -c
> 10.0.1.241  -i 2  -B 10.0.0.75:5002  -t 1000"
>
> The tcp syn packet will not be offloaded
>
>
> But if you  only run the script  without the last filter as following , The tcp syn
> packet will be offloaded.
>
> ifconfig mlx_p0 172.168.152.75/24 up
>
> ip l add dev tun1 type gretap external
> tc qdisc add dev tun1 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
>
> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw
> ip_proto tcp dst_ip 10.0.1.241 src_ip 10.0.0.75 src_port 5002 dst_port 5001
> tcp_flags 0/0x5  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000
> nocsum pipe action mirred egress redirect dev tun1.
>
> I think there are some problem in mlx5e_rep_setup_ft_cb.
>
> On 11/21/2019 9:05 PM, Paul Blakey wrote:
>
>
>         I see, I will test that, and how about normal FWD rules?
>
>         Paul.
>
>
>
>                 -----Original Message-----
>                 From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
>                 Sent: Thursday, November 21, 2019 2:35 PM
>                 To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
>                 Cc: pablo@netfilter.org<mailto:pablo@netfilter.org> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ;
> netdev@vger.kernel.org<mailto:netdev@vger.kernel.org> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
>                 <markb@mellanox.com><mailto:markb@mellanox.com> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
>                 Subject: Re: Question about flow table offload in mlx5e
>
>
>                 在 2019/11/21 19:39, Paul Blakey 写道:
>
>                         They are good fixes, exactly what we had when we
> tested this, thanks.
>
>                         Regarding encap, I don't know what changes you did,
> how does the encap
>
>                 rule look? Is it a FWD to vxlan device? If not it should be, as
> our driver
>                 expects that.
>                 It is fwd to a gretap devices
>
>
>                         I tried it on my setup via tc, by changing the callback
> of tc
>
>                 (mlx5e_rep_setup_tc_cb) to that of ft
> (mlx5e_rep_setup_ft_cb),
>
>                         and testing a vxlan encap rule:
>                         sudo tc qdisc add dev ens1f0_0 ingress
>                         sudo ifconfig ens1f0 7.7.7.7/24 up
>                         sudo ip link add name vxlan0 type vxlan dev ens1f0
> remote 7.7.7.8 dstport
>
>                 4789 external
>
>                         sudo ifconfig vxlan0 up
>                         sudo tc filter add dev ens1f0_0 ingress prio 1 chain 0
> protocol ip flower
>
>                 dst_mac aa:bb:cc:dd:ee:ff ip_proto udp skip_sw  action
> tunnel_key set
>                 src_ip 0.0.0.0 dst_ip 7.7.7.8 id 1234 dst_port 4789 pipe action
> mirred egress
>                 redirect dev vxlan
>
>
>                         then tc show:
>                         filter protocol ip pref 1 flower chain 0 handle 0x1
> dst_mac aa:bb:cc:dd:ee:ff
>
>                 ip_proto udp skip_sw in_hw in_hw_count 1
>
>                                 tunnel_key set src_ip 0.0.0.0 dst_ip 7.7.7.8 key_id
> 1234 dst_port 4789
>
>                 csum pipe
>
>                                 Stats: used 119 sec      0 pkt
>                                 mirred (Egress Redirect to device vxlan0)
>                                 Stats: used 119 sec      0 pkt
>
>
>                 Can you send packet that match this offloaded flow to check
> it is real
>                 offloaded?
>
>                 In the flowtable offload with my patches both
> TC_SETUP_BLOCK and
>                 TC_SETUP_FT can offload the rule success
>
>                 But in the TC_SETUP_FT case the packet is not real offloaded.
>
>
>                 I  will test like u did.
>
>
>
>
>
>
>                                 -----Original Message-----
>                                 From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
>                                 Sent: Thursday, November 21, 2019 10:29 AM
>                                 To: Paul Blakey <paulb@mellanox.com><mailto:paulb@mellanox.com>
> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
>                                 Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>
> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>
> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
>                                 <markb@mellanox.com><mailto:markb@mellanox.com>
> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
>                                 Subject: Re: Question about flow table
> offload in mlx5e
>
>
>                                 On 11/21/2019 3:42 PM, Paul Blakey wrote:
>
>                                         Hi,
>
>                                         The original design was the block
> setup to use TC_SETUP_FT type, and
>
>                 the
>
>                                 tc event type to be case
> TC_SETUP_CLSFLOWER.
>
>                                         We will post a patch to change that. I
> would advise to wait till we fix that
>
>                                 😊
>
>                                         I'm not sure how you get to this
> function mlx5e_rep_setup_ft_cb() if it
>
>                 the
>
>                                 nf_flow_table_offload ndo_setup_tc event
> was TC_SETUP_BLOCK, and
>
>                 not
>
>                                 TC_SETUP_FT.
>
>
>                                 Yes I change the TC_SETUP_BLOCK to
> TC_SETUP_FT in the
>                                 nf_flow_table_offload_setup.
>
>                                 Two fixes patch provide:
>
>                                 http://patchwork.ozlabs.org/patch/1197818/
>
>                                 http://patchwork.ozlabs.org/patch/1197876/
>
>                                 So this change made by me is not correct
> currently?
>
>
>                                         In our driver en_rep.c we have:
>
>                                         -------switch (type) {
>                                         -------case TC_SETUP_BLOCK:
>                                         ------->-------return
> flow_block_cb_setup_simple(type_data,
>                                         ------->------->------->------->------->---
> ----
>
>                 &mlx5e_rep_block_tc_cb_list,
>
>                                         ------->------->------->------->------->---
> ----  mlx5e_rep_setup_tc_cb,
>                                         ------->------->------->------->------->---
> ----  priv, priv, true);
>                                         -------case TC_SETUP_FT:
>                                         ------->-------return
> flow_block_cb_setup_simple(type_data,
>                                         ------->------->------->------->------->---
> ----
>
>                 &mlx5e_rep_block_ft_cb_list,
>
>                                         ------->------->------->------->------->---
> ----  mlx5e_rep_setup_ft_cb,
>                                         ------->------->------->------->------->---
> ----  priv, priv, true);
>                                         -------default:
>                                         ------->-------return -EOPNOTSUPP;
>                                         -------}
>
>                                         In nf_flow_table_offload.c:
>
>                                         -------bo.binder_type>-=
>
>                 FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
>
>                                         -------bo.extack>------= &extack;
>                                         -------INIT_LIST_HEAD(&bo.cb_list);
>                                         -------err = dev->netdev_ops-
>
>
> ndo_setup_tc(dev, TC_SETUP_BLOCK,
>
>
>                                 &bo);
>
>                                         -------if (err < 0)
>                                         ------->-------return err;
>                                         -------return
> nf_flow_table_block_setup(flowtable, &bo, cmd);
>
>                                         }
>
>         EXPORT_SYMBOL_GPL(nf_flow_table_offload_setup);
>
>
>                                         So unless you changed that as well,
> you should have gotten to
>
>                                 mlx5e_rep_setup_tc_cb and not
> mlx5e_rep_setup_tc_ft.
>
>                                         Regarding the encap action, there
> should be no difference on which
>
>                 chain
>
>                                 the rule is on.
>
>
>                                 But for the same encap rule can be real
> offloaded when setup through
>                                 through TC_SETUP_BLOCK. But TC_SETUP_FT
> can't.
>
>                                 So it is the problem of TC_SETUP_FT in
> mlx5e_rep_setup_ft_cb ?
>
>
>
>
>                                         -----Original Message-----
>                                         From: wenxu <wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
> <mailto:wenxu@ucloud.cn><mailto:wenxu@ucloud.cn>
>                                         Sent: Thursday, November 21, 2019
> 9:30 AM
>                                         To: Paul Blakey
> <paulb@mellanox.com><mailto:paulb@mellanox.com> <mailto:paulb@mellanox.com><mailto:paulb@mellanox.com>
>                                         Cc: pablo@netfilter.org<mailto:pablo@netfilter.org>
> <mailto:pablo@netfilter.org><mailto:pablo@netfilter.org> ; netdev@vger.kernel.org<mailto:netdev@vger.kernel.org>
> <mailto:netdev@vger.kernel.org><mailto:netdev@vger.kernel.org> ; Mark Bloch
>                                         <markb@mellanox.com><mailto:markb@mellanox.com>
> <mailto:markb@mellanox.com><mailto:markb@mellanox.com>
>                                         Subject: Question about flow table
> offload in mlx5e
>
>                                         Hi  paul,
>
>                                         The flow table offload in the mlx5e is
> based on TC_SETUP_FT.
>
>
>                                         It is almost the same as
> TC_SETUP_BLOCK.
>
>                                         It just set
> MLX5_TC_FLAG(FT_OFFLOAD) flags and change
>                                         cls_flower.common.chain_index =
> FDB_FT_CHAIN;
>
>                                         In following codes line 1380 and 1392
>
>                                         1368 static int
> mlx5e_rep_setup_ft_cb(enum tc_setup_type type, void
>                                         *type_data,
>                                         1369                                  void *cb_priv)
>                                         1370 {
>                                         1371         struct flow_cls_offload *f =
> type_data;
>                                         1372         struct flow_cls_offload
> cls_flower;
>                                         1373         struct mlx5e_priv *priv =
> cb_priv;
>                                         1374         struct mlx5_eswitch *esw;
>                                         1375         unsigned long flags;
>                                         1376         int err;
>                                         1377
>                                         1378         flags =
> MLX5_TC_FLAG(INGRESS) |
>                                         1379
> MLX5_TC_FLAG(ESW_OFFLOAD) |
>                                         1380
> MLX5_TC_FLAG(FT_OFFLOAD);
>                                         1381         esw = priv->mdev-
>
>
> priv.eswitch;
>
>
>                                         1382
>                                         1383         switch (type) {
>                                         1384         case
> TC_SETUP_CLSFLOWER:
>                                         1385                 if
> (!mlx5_eswitch_prios_supported(esw) || f-
>
>                                         common.chain_index)
>
>                                         1386                         return -
> EOPNOTSUPP;
>                                         1387
>                                         1388                 /* Re-use tc offload
> path by moving the ft flow to the
>                                         1389                  * reserved ft chain.
>                                         1390                  */
>                                         1391                 memcpy(&cls_flower, f,
> sizeof(*f));
>                                         1392
> cls_flower.common.chain_index = FDB_FT_CHAIN;
>                                         1393                 err =
> mlx5e_rep_setup_tc_cls_flower(priv, &cls_flower,
>
>                                 flags);
>
>                                         1394                 memcpy(&f->stats,
> &cls_flower.stats, sizeof(f->stats));
>
>
>                                         I want to add tunnel offload support
> in the flow table, I  add some
>
>                 patches
>
>                                 in
>
>                                         nf_flow_table_offload.
>
>                                         Also add the indr setup support in the
> mlx driver. And Now I can  flow
>
>                                 table
>
>                                         offload with decap.
>
>
>                                         But I meet a problem with the encap.
> The encap rule can be added in
>                                         hardware  successfully But it can't be
> offloaded.
>
>                                         But I think the rule I added is correct.
> If I mask the line 1392. The rule
>
>                 also
>
>                                 can
>
>                                         be add success and can be offloaded.
>
>                                         So there are some limit for encap
> operation for FT_OFFLOAD in
>                                         FDB_FT_CHAIN?
>
>
>                                         BR
>
>                                         wenxu
>
>

Re: Question about flow table offload in mlx5e

[Paul Blakey]

On 11/27/2019 2:16 PM, wenxu wrote:

> Sorry maybe something mess you,  Ignore with my patches.
>
>
> I also did the test like you with route tc rules to ft callback.
>
>
> please also did the following test:  mlx_p0 is the pf and mlx_pf0vf0 is the vf .
>
Are you  in switchdev mode (via devlink) or default legacy mode?

Re: Question about flow table offload in mlx5e

[Paul Blakey]

On 11/27/2019 3:11 PM, Paul Blakey wrote:
> On 11/27/2019 2:16 PM, wenxu wrote:
>
>> Sorry maybe something mess you,  Ignore with my patches.
>>
>>
>> I also did the test like you with route tc rules to ft callback.
>>
>>
>> please also did the following test:  mlx_p0 is the pf and mlx_pf0vf0 
>> is the vf .
>>
> Are you  in switchdev mode (via devlink) or default legacy mode?
>
>

mlx_pf0vf0  is representor device created after entring switchdev mode? and eth0 in vm is the binded mlx5 VF?

Can you run this command:

sudo grep -ri "" /sys/class/net/*/phys_* 2>/dev/null

example:
/sys/class/net/ens1f0_0/phys_port_name:pf0vf0
/sys/class/net/ens1f0_0/phys_switch_id:b828a50003078a24
/sys/class/net/ens1f0_1/phys_port_name:pf0vf1
/sys/class/net/ens1f0_1/phys_switch_id:b828a50003078a24
/sys/class/net/ens1f0/phys_port_name:p0
/sys/class/net/ens1f0/phys_switch_id:b828a50003078a24

and
sudo ls /sys/class/net/*/device/virtfn*/net

example:
/sys/class/net/ens1f0/device/virtfn0/net:
ens1f2

/sys/class/net/ens1f0/device/virtfn1/net:
ens1f3

and even

lspci | grep -i mellanox ; ls -l /sys/class/net






Thansk.

Re: Question about flow table offload in mlx5e

[wenxu]

在 2019/11/27 21:20, Paul Blakey 写道:
> On 11/27/2019 3:11 PM, Paul Blakey wrote:
>> On 11/27/2019 2:16 PM, wenxu wrote:
>>
>>> Sorry maybe something mess you,  Ignore with my patches.
>>>
>>>
>>> I also did the test like you with route tc rules to ft callback.
>>>
>>>
>>> please also did the following test:  mlx_p0 is the pf and mlx_pf0vf0 
>>> is the vf .
>>>
>> Are you  in switchdev mode (via devlink) or default legacy mode?
>>
>>
> mlx_pf0vf0  is representor device created after entring switchdev mode? and eth0 in vm is the binded mlx5 VF?

Yes, mlx_pf0vf0 is the representor and eth0 in vm is VF. It also in the switchdev mode.


sudo grep -ri "" /sys/class/net/*/phys_* 2>/dev/null
/sys/class/net/mlx_p0/phys_port_name:p0
/sys/class/net/mlx_p0/phys_switch_id:34ebc100034b6b50
/sys/class/net/mlx_pf0vf0/phys_port_name:pf0vf0
/sys/class/net/mlx_pf0vf0/phys_switch_id:34ebc100034b6b50
/sys/class/net/mlx_pf0vf1/phys_port_name:pf0vf1
/sys/class/net/mlx_pf0vf1/phys_switch_id:34ebc100034b6b50

The problem is when the last filter add in the tun1 will lead the outgoing syn packets can't be real offloaded

>
> Can you run this command:
>
> sudo grep -ri "" /sys/class/net/*/phys_* 2>/dev/null
>
> example:
> /sys/class/net/ens1f0_0/phys_port_name:pf0vf0
> /sys/class/net/ens1f0_0/phys_switch_id:b828a50003078a24
> /sys/class/net/ens1f0_1/phys_port_name:pf0vf1
> /sys/class/net/ens1f0_1/phys_switch_id:b828a50003078a24
> /sys/class/net/ens1f0/phys_port_name:p0
> /sys/class/net/ens1f0/phys_switch_id:b828a50003078a24
>
> and
> sudo ls /sys/class/net/*/device/virtfn*/net
>
> example:
> /sys/class/net/ens1f0/device/virtfn0/net:
> ens1f2
>
> /sys/class/net/ens1f0/device/virtfn1/net:
> ens1f3
>
> and even
>
> lspci | grep -i mellanox ; ls -l /sys/class/net
>
>
>
>
>
>
> Thansk.
>
>

Re: Question about flow table offload in mlx5e

[wenxu]

Hi Paul,


Sorry for trouble you again. I think it is a problem in ft callback.

Can your help me fix it. Thx!

I did the test like you with route tc rules to ft callback.

# ifconfig mlx_p0 172.168.152.75/24 up
# ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0

# ip l add dev tun1 type gretap external
# tc qdisc add dev tun1 ingress
# tc qdisc add dev mlx_pf0vf0 ingress

# tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000 nocsum pipe action mirred egress redirect dev tun1


In The vm:
# ifconfig eth0 10.0.0.75/24 up
# ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0

# iperf -c 10.0.0.77 -t 100 -i 2

The syn packets can be offloaded successfully.

# # tc -s filter ls dev mlx_pf0vf0 ingress
filter protocol ip pref 2 flower chain 0 
filter protocol ip pref 2 flower chain 0 handle 0x1 
  eth_type ipv4
  skip_sw
  in_hw in_hw_count 1
	action order 1: tunnel_key  set
	src_ip 0.0.0.0
	dst_ip 172.168.152.241
	key_id 1000
	nocsum pipe
	 index 1 ref 1 bind 1 installed 252 sec used 252 sec
	Action statistics:
	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
	backlog 0b 0p requeues 0

	action order 2: mirred (Egress Redirect to device tun1) stolen
 	index 1 ref 1 bind 1 installed 252 sec used 110 sec
 	Action statistics:
	Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0) 
	Sent software 0 bytes 0 pkt
	Sent hardware 3420 bytes 11 pkt
	backlog 0b 0p requeues 0

But Then I add another decap filter on tun1:

tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action mirred egress redirect dev mlx_pf0vf0

# iperf -c 10.0.0.77 -t 100 -i 2

The syn packets can't be offloaded. The tc filter counter is also not increase.


# tc -s filter ls dev mlx_pf0vf0 ingress
filter protocol ip pref 2 flower chain 0 
filter protocol ip pref 2 flower chain 0 handle 0x1 
  eth_type ipv4
  skip_sw
  in_hw in_hw_count 1
	action order 1: tunnel_key  set
	src_ip 0.0.0.0
	dst_ip 172.168.152.241
	key_id 1000
	nocsum pipe
	 index 1 ref 1 bind 1 installed 320 sec used 320 sec
	Action statistics:
	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
	backlog 0b 0p requeues 0

	action order 2: mirred (Egress Redirect to device tun1) stolen
 	index 1 ref 1 bind 1 installed 320 sec used 178 sec
 	Action statistics:
	Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0) 
	Sent software 0 bytes 0 pkt
	Sent hardware 3420 bytes 11 pkt
	backlog 0b 0p requeues 0

# tc -s filter ls dev tun1 ingress
filter protocol ip pref 2 flower chain 0 
filter protocol ip pref 2 flower chain 0 handle 0x1 
  eth_type ipv4
  enc_src_ip 172.168.152.241
  enc_key_id 1000
  in_hw in_hw_count 1
	action order 1: tunnel_key  unset pipe
	 index 2 ref 1 bind 1 installed 391 sec used 391 sec
	Action statistics:
	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
	backlog 0b 0p requeues 0

	action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
 	index 2 ref 1 bind 1 installed 391 sec used 391 sec
 	Action statistics:
	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
	backlog 0b 0p requeues 0


So there maybe some problem for ft callback setup. When there is another reverse
decap rule add in tunnel device, The encap rule will not offloaded the packets.

Expect your help Thx!


BR
wenxu

Re: Question about flow table offload in mlx5e

[Paul Blakey]

On 12/2/2019 5:37 AM, wenxu wrote:

> Hi Paul,
>
>
> Sorry for trouble you again. I think it is a problem in ft callback.
>
> Can your help me fix it. Thx!
>
> I did the test like you with route tc rules to ft callback.
>
> # ifconfig mlx_p0 172.168.152.75/24 up
> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>
> # ip l add dev tun1 type gretap external
> # tc qdisc add dev tun1 ingress
> # tc qdisc add dev mlx_pf0vf0 ingress
>
> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 1000 nocsum pipe action mirred egress redirect dev tun1
>
>
> In The vm:
> # ifconfig eth0 10.0.0.75/24 up
> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>
> # iperf -c 10.0.0.77 -t 100 -i 2
>
> The syn packets can be offloaded successfully.
>
> # # tc -s filter ls dev mlx_pf0vf0 ingress
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
>    eth_type ipv4
>    skip_sw
>    in_hw in_hw_count 1
> 	action order 1: tunnel_key  set
> 	src_ip 0.0.0.0
> 	dst_ip 172.168.152.241
> 	key_id 1000
> 	nocsum pipe
> 	 index 1 ref 1 bind 1 installed 252 sec used 252 sec
> 	Action statistics:
> 	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> 	backlog 0b 0p requeues 0
>
> 	action order 2: mirred (Egress Redirect to device tun1) stolen
>   	index 1 ref 1 bind 1 installed 252 sec used 110 sec
>   	Action statistics:
> 	Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
> 	Sent software 0 bytes 0 pkt
> 	Sent hardware 3420 bytes 11 pkt
> 	backlog 0b 0p requeues 0
>
> But Then I add another decap filter on tun1:
>
> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action mirred egress redirect dev mlx_pf0vf0
>
> # iperf -c 10.0.0.77 -t 100 -i 2
>
> The syn packets can't be offloaded. The tc filter counter is also not increase.
>
>
> # tc -s filter ls dev mlx_pf0vf0 ingress
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
>    eth_type ipv4
>    skip_sw
>    in_hw in_hw_count 1
> 	action order 1: tunnel_key  set
> 	src_ip 0.0.0.0
> 	dst_ip 172.168.152.241
> 	key_id 1000
> 	nocsum pipe
> 	 index 1 ref 1 bind 1 installed 320 sec used 320 sec
> 	Action statistics:
> 	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> 	backlog 0b 0p requeues 0
>
> 	action order 2: mirred (Egress Redirect to device tun1) stolen
>   	index 1 ref 1 bind 1 installed 320 sec used 178 sec
>   	Action statistics:
> 	Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
> 	Sent software 0 bytes 0 pkt
> 	Sent hardware 3420 bytes 11 pkt
> 	backlog 0b 0p requeues 0
>
> # tc -s filter ls dev tun1 ingress
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
>    eth_type ipv4
>    enc_src_ip 172.168.152.241
>    enc_key_id 1000
>    in_hw in_hw_count 1
> 	action order 1: tunnel_key  unset pipe
> 	 index 2 ref 1 bind 1 installed 391 sec used 391 sec
> 	Action statistics:
> 	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> 	backlog 0b 0p requeues 0
>
> 	action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>   	index 2 ref 1 bind 1 installed 391 sec used 391 sec
>   	Action statistics:
> 	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> 	backlog 0b 0p requeues 0
>
>
> So there maybe some problem for ft callback setup. When there is another reverse
> decap rule add in tunnel device, The encap rule will not offloaded the packets.
>
> Expect your help Thx!
>
>
> BR
> wenxu
>
>
>
>
>
>
Hi I reproduced it.

I'll find the reason and fix for it and get back to you soon.

We are planing on expanding our chain and prio supported range, and in 
that we also move the FT offload code a bit.

If what I think happens happened it would fix it anyway.

Thanks.

Re: Question about flow table offload in mlx5e

[Paul Blakey]

Here's the temp fix:


The problem is TC + FT offload, and this revealed a bug in the driver.

For the tunnel test, we changed tc block offload to ft callback, and 
didn't change the indr block offload.

So the tunnel unset rule is offloaded from indr tc callback (it's 
indirect because it's on tun1 device):

mlx5e_rep_indr_setup_block_cb

this offloaded the rule to hardware in the TC domain.

Now the tunnel set (encap) rule was offloaded to FT domain.


Since TC comes before FT in software, we should have connected the miss 
on TC domain to FT domain,

but this didn't work.

The below fix should fix that connection:


------------------------------------------


--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -763,9 +763,6 @@ static struct mlx5_flow_table 
*find_closest_ft_recursive(struct fs_node  *root,
         struct fs_node *iter = list_entry(start, struct fs_node, list);
         struct mlx5_flow_table *ft = NULL;

-       if (!root || root->type == FS_TYPE_PRIO_CHAINS)
-               return NULL;
-
         list_for_each_advance_continue(iter, &root->children, reverse) {
                 if (iter->type == FS_TYPE_FLOW_TABLE) {
                         fs_get_obj(ft, iter);
@@ -792,7 +789,10 @@ static struct mlx5_flow_table 
*find_closest_ft(struct fs_prio *prio, bool revers
         parent = prio->node.parent;
         curr_node = &prio->node;
         while (!ft && parent) {
-               ft = find_closest_ft_recursive(parent, &curr_node->list, 
reverse);
+               if (parent->type != FS_TYPE_PRIO_CHAINS)
+                       ft = find_closest_ft_recursive(parent,
+ &curr_node->list,
+                                                      reverse);
                 curr_node = parent;
                 parent = curr_node->parent;
         }


------------------------------------------


I will do this patch for upstream if needed after our last patchset that 
also touched this area.

Thanks,

Paul.




On 12/5/2019 5:17 PM, Paul Blakey wrote:
> On 12/2/2019 5:37 AM, wenxu wrote:
>
>> Hi Paul,
>>
>>
>> Sorry for trouble you again. I think it is a problem in ft callback.
>>
>> Can your help me fix it. Thx!
>>
>> I did the test like you with route tc rules to ft callback.
>>
>> # ifconfig mlx_p0 172.168.152.75/24 up
>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>>
>> # ip l add dev tun1 type gretap external
>> # tc qdisc add dev tun1 ingress
>> # tc qdisc add dev mlx_pf0vf0 ingress
>>
>> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower 
>> skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 
>> 1000 nocsum pipe action mirred egress redirect dev tun1
>>
>>
>> In The vm:
>> # ifconfig eth0 10.0.0.75/24 up
>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>>
>> # iperf -c 10.0.0.77 -t 100 -i 2
>>
>> The syn packets can be offloaded successfully.
>>
>> # # tc -s filter ls dev mlx_pf0vf0 ingress
>> filter protocol ip pref 2 flower chain 0
>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>    eth_type ipv4
>>    skip_sw
>>    in_hw in_hw_count 1
>>     action order 1: tunnel_key  set
>>     src_ip 0.0.0.0
>>     dst_ip 172.168.152.241
>>     key_id 1000
>>     nocsum pipe
>>      index 1 ref 1 bind 1 installed 252 sec used 252 sec
>>     Action statistics:
>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>     backlog 0b 0p requeues 0
>>
>>     action order 2: mirred (Egress Redirect to device tun1) stolen
>>       index 1 ref 1 bind 1 installed 252 sec used 110 sec
>>       Action statistics:
>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 3420 bytes 11 pkt
>>     backlog 0b 0p requeues 0
>>
>> But Then I add another decap filter on tun1:
>>
>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id 
>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action 
>> mirred egress redirect dev mlx_pf0vf0
>>
>> # iperf -c 10.0.0.77 -t 100 -i 2
>>
>> The syn packets can't be offloaded. The tc filter counter is also not 
>> increase.
>>
>>
>> # tc -s filter ls dev mlx_pf0vf0 ingress
>> filter protocol ip pref 2 flower chain 0
>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>    eth_type ipv4
>>    skip_sw
>>    in_hw in_hw_count 1
>>     action order 1: tunnel_key  set
>>     src_ip 0.0.0.0
>>     dst_ip 172.168.152.241
>>     key_id 1000
>>     nocsum pipe
>>      index 1 ref 1 bind 1 installed 320 sec used 320 sec
>>     Action statistics:
>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>     backlog 0b 0p requeues 0
>>
>>     action order 2: mirred (Egress Redirect to device tun1) stolen
>>       index 1 ref 1 bind 1 installed 320 sec used 178 sec
>>       Action statistics:
>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 3420 bytes 11 pkt
>>     backlog 0b 0p requeues 0
>>
>> # tc -s filter ls dev tun1 ingress
>> filter protocol ip pref 2 flower chain 0
>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>    eth_type ipv4
>>    enc_src_ip 172.168.152.241
>>    enc_key_id 1000
>>    in_hw in_hw_count 1
>>     action order 1: tunnel_key  unset pipe
>>      index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>     Action statistics:
>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>     backlog 0b 0p requeues 0
>>
>>     action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>       index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>       Action statistics:
>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>     backlog 0b 0p requeues 0
>>
>>
>> So there maybe some problem for ft callback setup. When there is 
>> another reverse
>> decap rule add in tunnel device, The encap rule will not offloaded 
>> the packets.
>>
>> Expect your help Thx!
>>
>>
>> BR
>> wenxu
>>
>>
>>
>>
>>
>>
> Hi I reproduced it.
>
> I'll find the reason and fix for it and get back to you soon.
>
> We are planing on expanding our chain and prio supported range, and in 
> that we also move the FT offload code a bit.
>
> If what I think happens happened it would fix it anyway.
>
> Thanks.
>

Re: Question about flow table offload in mlx5e

[wenxu]

Hi  Paul,


Thanks for your fix, I will test it.

On 12/8/2019 5:39 PM, Paul Blakey wrote:
> Here's the temp fix:
>
>
> The problem is TC + FT offload, and this revealed a bug in the driver.
>
> For the tunnel test, we changed tc block offload to ft callback, and 
> didn't change the indr block offload.
>
> So the tunnel unset rule is offloaded from indr tc callback (it's 
> indirect because it's on tun1 device):
>
> mlx5e_rep_indr_setup_block_cb

Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup?

Or all indr setup through TC offload?

>
> this offloaded the rule to hardware in the TC domain.
>
> Now the tunnel set (encap) rule was offloaded to FT domain.
>
>
> Since TC comes before FT in software, we should have connected the miss 
> on TC domain to FT domain,
>
> but this didn't work.
>
> The below fix should fix that connection:
>
>
> ------------------------------------------
>
>
> --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
> @@ -763,9 +763,6 @@ static struct mlx5_flow_table 
> *find_closest_ft_recursive(struct fs_node  *root,
>          struct fs_node *iter = list_entry(start, struct fs_node, list);
>          struct mlx5_flow_table *ft = NULL;
>
> -       if (!root || root->type == FS_TYPE_PRIO_CHAINS)
> -               return NULL;
> -
>          list_for_each_advance_continue(iter, &root->children, reverse) {
>                  if (iter->type == FS_TYPE_FLOW_TABLE) {
>                          fs_get_obj(ft, iter);
> @@ -792,7 +789,10 @@ static struct mlx5_flow_table 
> *find_closest_ft(struct fs_prio *prio, bool revers
>          parent = prio->node.parent;
>          curr_node = &prio->node;
>          while (!ft && parent) {
> -               ft = find_closest_ft_recursive(parent, &curr_node->list, 
> reverse);
> +               if (parent->type != FS_TYPE_PRIO_CHAINS)
> +                       ft = find_closest_ft_recursive(parent,
> + &curr_node->list,
> +                                                      reverse);
>                  curr_node = parent;
>                  parent = curr_node->parent;
>          }
>
>
> ------------------------------------------
>
>
> I will do this patch for upstream if needed after our last patchset that 
> also touched this area.
>
> Thanks,
>
> Paul.
>
>
>
>
> On 12/5/2019 5:17 PM, Paul Blakey wrote:
>> On 12/2/2019 5:37 AM, wenxu wrote:
>>
>>> Hi Paul,
>>>
>>>
>>> Sorry for trouble you again. I think it is a problem in ft callback.
>>>
>>> Can your help me fix it. Thx!
>>>
>>> I did the test like you with route tc rules to ft callback.
>>>
>>> # ifconfig mlx_p0 172.168.152.75/24 up
>>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>>>
>>> # ip l add dev tun1 type gretap external
>>> # tc qdisc add dev tun1 ingress
>>> # tc qdisc add dev mlx_pf0vf0 ingress
>>>
>>> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower 
>>> skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id 
>>> 1000 nocsum pipe action mirred egress redirect dev tun1
>>>
>>>
>>> In The vm:
>>> # ifconfig eth0 10.0.0.75/24 up
>>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>>>
>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>
>>> The syn packets can be offloaded successfully.
>>>
>>> # # tc -s filter ls dev mlx_pf0vf0 ingress
>>> filter protocol ip pref 2 flower chain 0
>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>    eth_type ipv4
>>>    skip_sw
>>>    in_hw in_hw_count 1
>>>     action order 1: tunnel_key  set
>>>     src_ip 0.0.0.0
>>>     dst_ip 172.168.152.241
>>>     key_id 1000
>>>     nocsum pipe
>>>      index 1 ref 1 bind 1 installed 252 sec used 252 sec
>>>     Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>     action order 2: mirred (Egress Redirect to device tun1) stolen
>>>       index 1 ref 1 bind 1 installed 252 sec used 110 sec
>>>       Action statistics:
>>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>     Sent software 0 bytes 0 pkt
>>>     Sent hardware 3420 bytes 11 pkt
>>>     backlog 0b 0p requeues 0
>>>
>>> But Then I add another decap filter on tun1:
>>>
>>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id 
>>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action 
>>> mirred egress redirect dev mlx_pf0vf0
>>>
>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>
>>> The syn packets can't be offloaded. The tc filter counter is also not 
>>> increase.
>>>
>>>
>>> # tc -s filter ls dev mlx_pf0vf0 ingress
>>> filter protocol ip pref 2 flower chain 0
>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>    eth_type ipv4
>>>    skip_sw
>>>    in_hw in_hw_count 1
>>>     action order 1: tunnel_key  set
>>>     src_ip 0.0.0.0
>>>     dst_ip 172.168.152.241
>>>     key_id 1000
>>>     nocsum pipe
>>>      index 1 ref 1 bind 1 installed 320 sec used 320 sec
>>>     Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>     action order 2: mirred (Egress Redirect to device tun1) stolen
>>>       index 1 ref 1 bind 1 installed 320 sec used 178 sec
>>>       Action statistics:
>>>     Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>     Sent software 0 bytes 0 pkt
>>>     Sent hardware 3420 bytes 11 pkt
>>>     backlog 0b 0p requeues 0
>>>
>>> # tc -s filter ls dev tun1 ingress
>>> filter protocol ip pref 2 flower chain 0
>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>    eth_type ipv4
>>>    enc_src_ip 172.168.152.241
>>>    enc_key_id 1000
>>>    in_hw in_hw_count 1
>>>     action order 1: tunnel_key  unset pipe
>>>      index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>     Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>     action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>>       index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>       Action statistics:
>>>     Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>     backlog 0b 0p requeues 0
>>>
>>>
>>> So there maybe some problem for ft callback setup. When there is 
>>> another reverse
>>> decap rule add in tunnel device, The encap rule will not offloaded 
>>> the packets.
>>>
>>> Expect your help Thx!
>>>
>>>
>>> BR
>>> wenxu
>>>
>>>
>>>
>>>
>>>
>>>
>> Hi I reproduced it.
>>
>> I'll find the reason and fix for it and get back to you soon.
>>
>> We are planing on expanding our chain and prio supported range, and in 
>> that we also move the FT offload code a bit.
>>
>> If what I think happens happened it would fix it anyway.
>>
>> Thanks.
>>

Re: Question about flow table offload in mlx5e

[Paul Blakey]

On 12/9/2019 5:18 AM, wenxu wrote:
> Hi  Paul,
>
>
> Thanks for your fix, I will test it.
>
> On 12/8/2019 5:39 PM, Paul Blakey wrote:
>> Here's the temp fix:
>>
>>
>> The problem is TC + FT offload, and this revealed a bug in the driver.
>>
>> For the tunnel test, we changed tc block offload to ft callback, and
>> didn't change the indr block offload.
>>
>> So the tunnel unset rule is offloaded from indr tc callback (it's
>> indirect because it's on tun1 device):
>>
>> mlx5e_rep_indr_setup_block_cb
> Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup?
>
> Or all indr setup through TC offload?

Adding a "mlx5e_rep_indr_setup_ft_cb" with the correct flags (FT) and 
(EGRESS) should work as well, but this is just a test...

For upstream, I see you're talking with pablo about the pending tunnel 
offload support.


>> this offloaded the rule to hardware in the TC domain.
>>
>> Now the tunnel set (encap) rule was offloaded to FT domain.
>>
>>
>> Since TC comes before FT in software, we should have connected the miss
>> on TC domain to FT domain,
>>
>> but this didn't work.
>>
>> The below fix should fix that connection:
>>
>>
>> ------------------------------------------
>>
>>
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>> @@ -763,9 +763,6 @@ static struct mlx5_flow_table
>> *find_closest_ft_recursive(struct fs_node  *root,
>>           struct fs_node *iter = list_entry(start, struct fs_node, list);
>>           struct mlx5_flow_table *ft = NULL;
>>
>> -       if (!root || root->type == FS_TYPE_PRIO_CHAINS)
>> -               return NULL;
>> -
>>           list_for_each_advance_continue(iter, &root->children, reverse) {
>>                   if (iter->type == FS_TYPE_FLOW_TABLE) {
>>                           fs_get_obj(ft, iter);
>> @@ -792,7 +789,10 @@ static struct mlx5_flow_table
>> *find_closest_ft(struct fs_prio *prio, bool revers
>>           parent = prio->node.parent;
>>           curr_node = &prio->node;
>>           while (!ft && parent) {
>> -               ft = find_closest_ft_recursive(parent, &curr_node->list,
>> reverse);
>> +               if (parent->type != FS_TYPE_PRIO_CHAINS)
>> +                       ft = find_closest_ft_recursive(parent,
>> + &curr_node->list,
>> +                                                      reverse);
>>                   curr_node = parent;
>>                   parent = curr_node->parent;
>>           }
>>
>>
>> ------------------------------------------
>>
>>
>> I will do this patch for upstream if needed after our last patchset that
>> also touched this area.
>>
>> Thanks,
>>
>> Paul.
>>
>>
>>
>>
>> On 12/5/2019 5:17 PM, Paul Blakey wrote:
>>> On 12/2/2019 5:37 AM, wenxu wrote:
>>>
>>>> Hi Paul,
>>>>
>>>>
>>>> Sorry for trouble you again. I think it is a problem in ft callback.
>>>>
>>>> Can your help me fix it. Thx!
>>>>
>>>> I did the test like you with route tc rules to ft callback.
>>>>
>>>> # ifconfig mlx_p0 172.168.152.75/24 up
>>>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>>>>
>>>> # ip l add dev tun1 type gretap external
>>>> # tc qdisc add dev tun1 ingress
>>>> # tc qdisc add dev mlx_pf0vf0 ingress
>>>>
>>>> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower
>>>> skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id
>>>> 1000 nocsum pipe action mirred egress redirect dev tun1
>>>>
>>>>
>>>> In The vm:
>>>> # ifconfig eth0 10.0.0.75/24 up
>>>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>>>>
>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>
>>>> The syn packets can be offloaded successfully.
>>>>
>>>> # # tc -s filter ls dev mlx_pf0vf0 ingress
>>>> filter protocol ip pref 2 flower chain 0
>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>     eth_type ipv4
>>>>     skip_sw
>>>>     in_hw in_hw_count 1
>>>>      action order 1: tunnel_key  set
>>>>      src_ip 0.0.0.0
>>>>      dst_ip 172.168.152.241
>>>>      key_id 1000
>>>>      nocsum pipe
>>>>       index 1 ref 1 bind 1 installed 252 sec used 252 sec
>>>>      Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>      action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>        index 1 ref 1 bind 1 installed 252 sec used 110 sec
>>>>        Action statistics:
>>>>      Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      Sent software 0 bytes 0 pkt
>>>>      Sent hardware 3420 bytes 11 pkt
>>>>      backlog 0b 0p requeues 0
>>>>
>>>> But Then I add another decap filter on tun1:
>>>>
>>>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id
>>>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action
>>>> mirred egress redirect dev mlx_pf0vf0
>>>>
>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>
>>>> The syn packets can't be offloaded. The tc filter counter is also not
>>>> increase.
>>>>
>>>>
>>>> # tc -s filter ls dev mlx_pf0vf0 ingress
>>>> filter protocol ip pref 2 flower chain 0
>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>     eth_type ipv4
>>>>     skip_sw
>>>>     in_hw in_hw_count 1
>>>>      action order 1: tunnel_key  set
>>>>      src_ip 0.0.0.0
>>>>      dst_ip 172.168.152.241
>>>>      key_id 1000
>>>>      nocsum pipe
>>>>       index 1 ref 1 bind 1 installed 320 sec used 320 sec
>>>>      Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>      action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>        index 1 ref 1 bind 1 installed 320 sec used 178 sec
>>>>        Action statistics:
>>>>      Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      Sent software 0 bytes 0 pkt
>>>>      Sent hardware 3420 bytes 11 pkt
>>>>      backlog 0b 0p requeues 0
>>>>
>>>> # tc -s filter ls dev tun1 ingress
>>>> filter protocol ip pref 2 flower chain 0
>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>     eth_type ipv4
>>>>     enc_src_ip 172.168.152.241
>>>>     enc_key_id 1000
>>>>     in_hw in_hw_count 1
>>>>      action order 1: tunnel_key  unset pipe
>>>>       index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>      Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>      action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>>>        index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>        Action statistics:
>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>      backlog 0b 0p requeues 0
>>>>
>>>>
>>>> So there maybe some problem for ft callback setup. When there is
>>>> another reverse
>>>> decap rule add in tunnel device, The encap rule will not offloaded
>>>> the packets.
>>>>
>>>> Expect your help Thx!
>>>>
>>>>
>>>> BR
>>>> wenxu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> Hi I reproduced it.
>>>
>>> I'll find the reason and fix for it and get back to you soon.
>>>
>>> We are planing on expanding our chain and prio supported range, and in
>>> that we also move the FT offload code a bit.
>>>
>>> If what I think happens happened it would fix it anyway.
>>>
>>> Thanks.
>>>

Re: Question about flow table offload in mlx5e

[wenxu]

On 12/9/2019 3:48 PM, Paul Blakey wrote:
> On 12/9/2019 5:18 AM, wenxu wrote:
>> Hi  Paul,
>>
>>
>> Thanks for your fix, I will test it.
>>
>> On 12/8/2019 5:39 PM, Paul Blakey wrote:
>>> Here's the temp fix:
>>>
>>>
>>> The problem is TC + FT offload, and this revealed a bug in the driver.
>>>
>>> For the tunnel test, we changed tc block offload to ft callback, and
>>> didn't change the indr block offload.
>>>
>>> So the tunnel unset rule is offloaded from indr tc callback (it's
>>> indirect because it's on tun1 device):
>>>
>>> mlx5e_rep_indr_setup_block_cb
>> Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup?
>>
>> Or all indr setup through TC offload?
> Adding a "mlx5e_rep_indr_setup_ft_cb" with the correct flags (FT) and 
> (EGRESS) should work as well, but this is just a test...
>
> For upstream, I see you're talking with pablo about the pending tunnel 
> offload support.

I test your fix patch, it is work now.

But if there is no your fix patch and  I only add a  mlx5e_rep_indr_setup_ft_cb

with the FT flags, it also can't work (both rep_setup and indr_rep_setup with

FT flags. I can make it work with adding your fix patch.  It seems not a problem of

TC + FT offload?


>
>
>>> this offloaded the rule to hardware in the TC domain.
>>>
>>> Now the tunnel set (encap) rule was offloaded to FT domain.
>>>
>>>
>>> Since TC comes before FT in software, we should have connected the miss
>>> on TC domain to FT domain,
>>>
>>> but this didn't work.
>>>
>>> The below fix should fix that connection:
>>>
>>>
>>> ------------------------------------------
>>>
>>>
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
>>> @@ -763,9 +763,6 @@ static struct mlx5_flow_table
>>> *find_closest_ft_recursive(struct fs_node  *root,
>>>           struct fs_node *iter = list_entry(start, struct fs_node, list);
>>>           struct mlx5_flow_table *ft = NULL;
>>>
>>> -       if (!root || root->type == FS_TYPE_PRIO_CHAINS)
>>> -               return NULL;
>>> -
>>>           list_for_each_advance_continue(iter, &root->children, reverse) {
>>>                   if (iter->type == FS_TYPE_FLOW_TABLE) {
>>>                           fs_get_obj(ft, iter);
>>> @@ -792,7 +789,10 @@ static struct mlx5_flow_table
>>> *find_closest_ft(struct fs_prio *prio, bool revers
>>>           parent = prio->node.parent;
>>>           curr_node = &prio->node;
>>>           while (!ft && parent) {
>>> -               ft = find_closest_ft_recursive(parent, &curr_node->list,
>>> reverse);
>>> +               if (parent->type != FS_TYPE_PRIO_CHAINS)
>>> +                       ft = find_closest_ft_recursive(parent,
>>> + &curr_node->list,
>>> +                                                      reverse);
>>>                   curr_node = parent;
>>>                   parent = curr_node->parent;
>>>           }
>>>
>>>
>>> ------------------------------------------
>>>
>>>
>>> I will do this patch for upstream if needed after our last patchset that
>>> also touched this area.
>>>
>>> Thanks,
>>>
>>> Paul.
>>>
>>>
>>>
>>>
>>> On 12/5/2019 5:17 PM, Paul Blakey wrote:
>>>> On 12/2/2019 5:37 AM, wenxu wrote:
>>>>
>>>>> Hi Paul,
>>>>>
>>>>>
>>>>> Sorry for trouble you again. I think it is a problem in ft callback.
>>>>>
>>>>> Can your help me fix it. Thx!
>>>>>
>>>>> I did the test like you with route tc rules to ft callback.
>>>>>
>>>>> # ifconfig mlx_p0 172.168.152.75/24 up
>>>>> # ip n r 172.16.152.241 lladdr fa:fa:ff:ff:ff:ff dev mlx_p0
>>>>>
>>>>> # ip l add dev tun1 type gretap external
>>>>> # tc qdisc add dev tun1 ingress
>>>>> # tc qdisc add dev mlx_pf0vf0 ingress
>>>>>
>>>>> # tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower
>>>>> skip_sw  action tunnel_key set dst_ip 172.168.152.241 src_ip 0 id
>>>>> 1000 nocsum pipe action mirred egress redirect dev tun1
>>>>>
>>>>>
>>>>> In The vm:
>>>>> # ifconfig eth0 10.0.0.75/24 up
>>>>> # ip n r 10.0.0.77 lladdr fa:ff:ff:ff:ff:ff dev eth0
>>>>>
>>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>>
>>>>> The syn packets can be offloaded successfully.
>>>>>
>>>>> # # tc -s filter ls dev mlx_pf0vf0 ingress
>>>>> filter protocol ip pref 2 flower chain 0
>>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>>     eth_type ipv4
>>>>>     skip_sw
>>>>>     in_hw in_hw_count 1
>>>>>      action order 1: tunnel_key  set
>>>>>      src_ip 0.0.0.0
>>>>>      dst_ip 172.168.152.241
>>>>>      key_id 1000
>>>>>      nocsum pipe
>>>>>       index 1 ref 1 bind 1 installed 252 sec used 252 sec
>>>>>      Action statistics:
>>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>>      backlog 0b 0p requeues 0
>>>>>
>>>>>      action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>>        index 1 ref 1 bind 1 installed 252 sec used 110 sec
>>>>>        Action statistics:
>>>>>      Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>>      Sent software 0 bytes 0 pkt
>>>>>      Sent hardware 3420 bytes 11 pkt
>>>>>      backlog 0b 0p requeues 0
>>>>>
>>>>> But Then I add another decap filter on tun1:
>>>>>
>>>>> tc filter add dev tun1 pref 2 ingress protocol ip flower enc_key_id
>>>>> 1000 enc_src_ip 172.168.152.241 action tunnel_key unset pipe action
>>>>> mirred egress redirect dev mlx_pf0vf0
>>>>>
>>>>> # iperf -c 10.0.0.77 -t 100 -i 2
>>>>>
>>>>> The syn packets can't be offloaded. The tc filter counter is also not
>>>>> increase.
>>>>>
>>>>>
>>>>> # tc -s filter ls dev mlx_pf0vf0 ingress
>>>>> filter protocol ip pref 2 flower chain 0
>>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>>     eth_type ipv4
>>>>>     skip_sw
>>>>>     in_hw in_hw_count 1
>>>>>      action order 1: tunnel_key  set
>>>>>      src_ip 0.0.0.0
>>>>>      dst_ip 172.168.152.241
>>>>>      key_id 1000
>>>>>      nocsum pipe
>>>>>       index 1 ref 1 bind 1 installed 320 sec used 320 sec
>>>>>      Action statistics:
>>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>>      backlog 0b 0p requeues 0
>>>>>
>>>>>      action order 2: mirred (Egress Redirect to device tun1) stolen
>>>>>        index 1 ref 1 bind 1 installed 320 sec used 178 sec
>>>>>        Action statistics:
>>>>>      Sent 3420 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
>>>>>      Sent software 0 bytes 0 pkt
>>>>>      Sent hardware 3420 bytes 11 pkt
>>>>>      backlog 0b 0p requeues 0
>>>>>
>>>>> # tc -s filter ls dev tun1 ingress
>>>>> filter protocol ip pref 2 flower chain 0
>>>>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>>>>     eth_type ipv4
>>>>>     enc_src_ip 172.168.152.241
>>>>>     enc_key_id 1000
>>>>>     in_hw in_hw_count 1
>>>>>      action order 1: tunnel_key  unset pipe
>>>>>       index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>>      Action statistics:
>>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>>      backlog 0b 0p requeues 0
>>>>>
>>>>>      action order 2: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>>>>        index 2 ref 1 bind 1 installed 391 sec used 391 sec
>>>>>        Action statistics:
>>>>>      Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>>>      backlog 0b 0p requeues 0
>>>>>
>>>>>
>>>>> So there maybe some problem for ft callback setup. When there is
>>>>> another reverse
>>>>> decap rule add in tunnel device, The encap rule will not offloaded
>>>>> the packets.
>>>>>
>>>>> Expect your help Thx!
>>>>>
>>>>>
>>>>> BR
>>>>> wenxu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Hi I reproduced it.
>>>>
>>>> I'll find the reason and fix for it and get back to you soon.
>>>>
>>>> We are planing on expanding our chain and prio supported range, and in
>>>> that we also move the FT offload code a bit.
>>>>
>>>> If what I think happens happened it would fix it anyway.
>>>>
>>>> Thanks.
>>>>

Re: Question about flow table offload in mlx5e

[wenxu]

On 12/9/2019 3:48 PM, Paul Blakey wrote:
> On 12/9/2019 5:18 AM, wenxu wrote:
>> Hi  Paul,
>>
>>
>> Thanks for your fix, I will test it.
>>
>> On 12/8/2019 5:39 PM, Paul Blakey wrote:
>>> Here's the temp fix:
>>>
>>>
>>> The problem is TC + FT offload, and this revealed a bug in the driver.
>>>
>>> For the tunnel test, we changed tc block offload to ft callback, and
>>> didn't change the indr block offload.
>>>
>>> So the tunnel unset rule is offloaded from indr tc callback (it's
>>> indirect because it's on tun1 device):
>>>
>>> mlx5e_rep_indr_setup_block_cb
>> Maybe It should add a "mlx5e_rep_indr_setup_ft_cb" makes the FT offload can support the indr setup?
>>
>> Or all indr setup through TC offload?
> Adding a "mlx5e_rep_indr_setup_ft_cb" with the correct flags (FT) and 
> (EGRESS) should work as well, but this is just a test...

Indeed, I test with a mlx5e_rep_indr_setup_ft_c, it is also work as well.

Thanks paul!

Bad performance for VF outgoing in offloaded mode

[wenxu]

Hi mellanox team,


I did a performance test for tc offload with upstream kernel:

I setup a vm with a VF as eth0

In the vm:

ifconfig eth0 10.0.0.75/24 up


On the host the mlx_p0 is the pf representor and mlx_pf0vf0 is the vf representor

The device in the switchdev mode

# grep -ri "" /sys/class/net/*/phys_* 2>/dev/null
/sys/class/net/mlx_p0/phys_port_name:p0
/sys/class/net/mlx_p0/phys_switch_id:34ebc100034b6b50
/sys/class/net/mlx_pf0vf0/phys_port_name:pf0vf0
/sys/class/net/mlx_pf0vf0/phys_switch_id:34ebc100034b6b50
/sys/class/net/mlx_pf0vf1/phys_port_name:pf0vf1
/sys/class/net/mlx_pf0vf1/phys_switch_id:34ebc100034b6b50


The tc filter as following: just forward ip/arp packets  in mlx_p0 and mlx_pf0vf0 each other

tc qdisc add dev mlx_p0 ingress
tc qdisc add dev mlx_pf0vf0 ingress

tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_p0
tc filter add dev mlx_p0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_pf0vf0

tc filter add dev mlx_pf0vf0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_p0
tc filter add dev mlx_p0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_pf0vf0


The remote server device eth0:

ifconfig eth0 10.0.0.241/24


test case 1:   tcp recieve from VF to PF

In the vm: iperf -s

On the remote server:

iperf -c 10.0.0.75 -t 10 -i 2
------------------------------------------------------------
Client connecting to 10.0.0.75, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.241 port 59708 connected with 10.0.0.75 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  5.40 GBytes  23.2 Gbits/sec
[  3]  2.0- 4.0 sec  5.35 GBytes  23.0 Gbits/sec
[  3]  4.0- 6.0 sec  5.46 GBytes  23.5 Gbits/sec
[  3]  6.0- 8.0 sec  5.10 GBytes  21.9 Gbits/sec
[  3]  8.0-10.0 sec  5.36 GBytes  23.0 Gbits/sec
[  3]  0.0-10.0 sec  26.7 GBytes  22.9 Gbits/sec


Good performance with offload.

# tc -s filter ls dev mlx_p0 ingress
filter protocol arp pref 1 flower chain 0
filter protocol arp pref 1 flower chain 0 handle 0x1
  eth_type arp
  skip_sw
  in_hw in_hw_count 1
    action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
     index 4 ref 1 bind 1 installed 971 sec used 82 sec
     Action statistics:
    Sent 420 bytes 7 pkt (dropped 0, overlimits 0 requeues 0)
    Sent software 0 bytes 0 pkt
    Sent hardware 420 bytes 7 pkt
    backlog 0b 0p requeues 0

filter protocol ip pref 2 flower chain 0
filter protocol ip pref 2 flower chain 0 handle 0x1
  eth_type ipv4
  skip_sw
  in_hw in_hw_count 1
    action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
     index 2 ref 1 bind 1 installed 972 sec used 67 sec
     Action statistics:
    Sent 79272204362 bytes 91511261 pkt (dropped 0, overlimits 0 requeues 0)
    Sent software 0 bytes 0 pkt
    Sent hardware 79272204362 bytes 91511261 pkt
    backlog 0b 0p requeues 0

#  tc -s filter ls dev mlx_pf0vf0 ingress
filter protocol arp pref 1 flower chain 0
filter protocol arp pref 1 flower chain 0 handle 0x1
  eth_type arp
  skip_sw
  in_hw in_hw_count 1
    action order 1: mirred (Egress Redirect to device mlx_p0) stolen
     index 3 ref 1 bind 1 installed 978 sec used 88 sec
     Action statistics:
    Sent 600 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
    Sent software 0 bytes 0 pkt
    Sent hardware 600 bytes 10 pkt
    backlog 0b 0p requeues 0

filter protocol ip pref 2 flower chain 0
filter protocol ip pref 2 flower chain 0 handle 0x1
  eth_type ipv4
  skip_sw
  in_hw in_hw_count 1
    action order 1: mirred (Egress Redirect to device mlx_p0) stolen
     index 1 ref 1 bind 1 installed 978 sec used 73 sec
     Action statistics:
    Sent 71556027574 bytes 47805525 pkt (dropped 0, overlimits 0 requeues 0)
    Sent software 0 bytes 0 pkt
    Sent hardware 71556027574 bytes 47805525 pkt
    backlog 0b 0p requeues 0



test case 2:  tcp send from VF to PF

On the reomte server: iperf -s

in the vm:

# iperf -c 10.0.0.241 -t 10 -i 2

------------------------------------------------------------
Client connecting to 10.0.0.241, TCP port 5001
TCP window size:  230 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.75 port 53166 connected with 10.0.0.241 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec   939 MBytes  3.94 Gbits/sec
[  3]  2.0- 4.0 sec   944 MBytes  3.96 Gbits/sec
[  3]  4.0- 6.0 sec  1.01 GBytes  4.34 Gbits/sec
[  3]  6.0- 8.0 sec  1.03 GBytes  4.44 Gbits/sec
[  3]  8.0-10.0 sec  1.02 GBytes  4.39 Gbits/sec
[  3]  0.0-10.0 sec  4.90 GBytes  4.21 Gbits/sec


Bad performance with offload.  All the packet are offloaded. 

It is the offload problem in the hardware?


BR

wenxu

Re: Bad performance for VF outgoing in offloaded mode

[Roi Dayan]

On 2019-11-28 7:03 AM, wenxu wrote:
> Hi mellanox team,
> 
> 
> I did a performance test for tc offload with upstream kernel:
> 
> I setup a vm with a VF as eth0
> 
> In the vm:
> 
> ifconfig eth0 10.0.0.75/24 up
> 
> 
> On the host the mlx_p0 is the pf representor and mlx_pf0vf0 is the vf representor
> 
> The device in the switchdev mode
> 
> # grep -ri "" /sys/class/net/*/phys_* 2>/dev/null
> /sys/class/net/mlx_p0/phys_port_name:p0
> /sys/class/net/mlx_p0/phys_switch_id:34ebc100034b6b50
> /sys/class/net/mlx_pf0vf0/phys_port_name:pf0vf0
> /sys/class/net/mlx_pf0vf0/phys_switch_id:34ebc100034b6b50
> /sys/class/net/mlx_pf0vf1/phys_port_name:pf0vf1
> /sys/class/net/mlx_pf0vf1/phys_switch_id:34ebc100034b6b50
> 
> 
> The tc filter as following: just forward ip/arp packets  in mlx_p0 and mlx_pf0vf0 each other
> 
> tc qdisc add dev mlx_p0 ingress
> tc qdisc add dev mlx_pf0vf0 ingress
> 
> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_p0
> tc filter add dev mlx_p0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_pf0vf0
> 
> tc filter add dev mlx_pf0vf0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_p0
> tc filter add dev mlx_p0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_pf0vf0
> 
> 
> The remote server device eth0:
> 
> ifconfig eth0 10.0.0.241/24
> 
> 
> test case 1:   tcp recieve from VF to PF
> 
> In the vm: iperf -s
> 
> On the remote server:
> 
> iperf -c 10.0.0.75 -t 10 -i 2
> ------------------------------------------------------------
> Client connecting to 10.0.0.75, TCP port 5001
> TCP window size: 85.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.241 port 59708 connected with 10.0.0.75 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 2.0 sec  5.40 GBytes  23.2 Gbits/sec
> [  3]  2.0- 4.0 sec  5.35 GBytes  23.0 Gbits/sec
> [  3]  4.0- 6.0 sec  5.46 GBytes  23.5 Gbits/sec
> [  3]  6.0- 8.0 sec  5.10 GBytes  21.9 Gbits/sec
> [  3]  8.0-10.0 sec  5.36 GBytes  23.0 Gbits/sec
> [  3]  0.0-10.0 sec  26.7 GBytes  22.9 Gbits/sec
> 
> 
> Good performance with offload.
> 
> # tc -s filter ls dev mlx_p0 ingress
> filter protocol arp pref 1 flower chain 0
> filter protocol arp pref 1 flower chain 0 handle 0x1
>   eth_type arp
>   skip_sw
>   in_hw in_hw_count 1
>     action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>      index 4 ref 1 bind 1 installed 971 sec used 82 sec
>      Action statistics:
>     Sent 420 bytes 7 pkt (dropped 0, overlimits 0 requeues 0)
>     Sent software 0 bytes 0 pkt
>     Sent hardware 420 bytes 7 pkt
>     backlog 0b 0p requeues 0
> 
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
>   eth_type ipv4
>   skip_sw
>   in_hw in_hw_count 1
>     action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>      index 2 ref 1 bind 1 installed 972 sec used 67 sec
>      Action statistics:
>     Sent 79272204362 bytes 91511261 pkt (dropped 0, overlimits 0 requeues 0)
>     Sent software 0 bytes 0 pkt
>     Sent hardware 79272204362 bytes 91511261 pkt
>     backlog 0b 0p requeues 0
> 
> #  tc -s filter ls dev mlx_pf0vf0 ingress
> filter protocol arp pref 1 flower chain 0
> filter protocol arp pref 1 flower chain 0 handle 0x1
>   eth_type arp
>   skip_sw
>   in_hw in_hw_count 1
>     action order 1: mirred (Egress Redirect to device mlx_p0) stolen
>      index 3 ref 1 bind 1 installed 978 sec used 88 sec
>      Action statistics:
>     Sent 600 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
>     Sent software 0 bytes 0 pkt
>     Sent hardware 600 bytes 10 pkt
>     backlog 0b 0p requeues 0
> 
> filter protocol ip pref 2 flower chain 0
> filter protocol ip pref 2 flower chain 0 handle 0x1
>   eth_type ipv4
>   skip_sw
>   in_hw in_hw_count 1
>     action order 1: mirred (Egress Redirect to device mlx_p0) stolen
>      index 1 ref 1 bind 1 installed 978 sec used 73 sec
>      Action statistics:
>     Sent 71556027574 bytes 47805525 pkt (dropped 0, overlimits 0 requeues 0)
>     Sent software 0 bytes 0 pkt
>     Sent hardware 71556027574 bytes 47805525 pkt
>     backlog 0b 0p requeues 0
> 
> 
> 
> test case 2:  tcp send from VF to PF
> 
> On the reomte server: iperf -s
> 
> in the vm:
> 
> # iperf -c 10.0.0.241 -t 10 -i 2
> 
> ------------------------------------------------------------
> Client connecting to 10.0.0.241, TCP port 5001
> TCP window size:  230 KByte (default)
> ------------------------------------------------------------
> [  3] local 10.0.0.75 port 53166 connected with 10.0.0.241 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0- 2.0 sec   939 MBytes  3.94 Gbits/sec
> [  3]  2.0- 4.0 sec   944 MBytes  3.96 Gbits/sec
> [  3]  4.0- 6.0 sec  1.01 GBytes  4.34 Gbits/sec
> [  3]  6.0- 8.0 sec  1.03 GBytes  4.44 Gbits/sec
> [  3]  8.0-10.0 sec  1.02 GBytes  4.39 Gbits/sec
> [  3]  0.0-10.0 sec  4.90 GBytes  4.21 Gbits/sec
> 
> 
> Bad performance with offload.  All the packet are offloaded. 
> 
> It is the offload problem in the hardware?
> 
> 
> BR
> 
> wenxu
> 
> 

Hi Wenxu,

We didn't notice this behavior.
Could it be your VM doesn't have enough resources to generate the traffic?
As a listener it's only sending the acks.

Thanks,
Roi

Re: Bad performance for VF outgoing in offloaded mode

[wenxu]

在 2019/12/4 21:50, Roi Dayan 写道:
>
> On 2019-11-28 7:03 AM, wenxu wrote:
>> Hi mellanox team,
>>
>>
>> I did a performance test for tc offload with upstream kernel:
>>
>> I setup a vm with a VF as eth0
>>
>> In the vm:
>>
>> ifconfig eth0 10.0.0.75/24 up
>>
>>
>> On the host the mlx_p0 is the pf representor and mlx_pf0vf0 is the vf representor
>>
>> The device in the switchdev mode
>>
>> # grep -ri "" /sys/class/net/*/phys_* 2>/dev/null
>> /sys/class/net/mlx_p0/phys_port_name:p0
>> /sys/class/net/mlx_p0/phys_switch_id:34ebc100034b6b50
>> /sys/class/net/mlx_pf0vf0/phys_port_name:pf0vf0
>> /sys/class/net/mlx_pf0vf0/phys_switch_id:34ebc100034b6b50
>> /sys/class/net/mlx_pf0vf1/phys_port_name:pf0vf1
>> /sys/class/net/mlx_pf0vf1/phys_switch_id:34ebc100034b6b50
>>
>>
>> The tc filter as following: just forward ip/arp packets  in mlx_p0 and mlx_pf0vf0 each other
>>
>> tc qdisc add dev mlx_p0 ingress
>> tc qdisc add dev mlx_pf0vf0 ingress
>>
>> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_p0
>> tc filter add dev mlx_p0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_pf0vf0
>>
>> tc filter add dev mlx_pf0vf0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_p0
>> tc filter add dev mlx_p0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_pf0vf0
>>
>>
>> The remote server device eth0:
>>
>> ifconfig eth0 10.0.0.241/24
>>
>>
>> test case 1:   tcp recieve from VF to PF
>>
>> In the vm: iperf -s
>>
>> On the remote server:
>>
>> iperf -c 10.0.0.75 -t 10 -i 2
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.75, TCP port 5001
>> TCP window size: 85.0 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.241 port 59708 connected with 10.0.0.75 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 2.0 sec  5.40 GBytes  23.2 Gbits/sec
>> [  3]  2.0- 4.0 sec  5.35 GBytes  23.0 Gbits/sec
>> [  3]  4.0- 6.0 sec  5.46 GBytes  23.5 Gbits/sec
>> [  3]  6.0- 8.0 sec  5.10 GBytes  21.9 Gbits/sec
>> [  3]  8.0-10.0 sec  5.36 GBytes  23.0 Gbits/sec
>> [  3]  0.0-10.0 sec  26.7 GBytes  22.9 Gbits/sec
>>
>>
>> Good performance with offload.
>>
>> # tc -s filter ls dev mlx_p0 ingress
>> filter protocol arp pref 1 flower chain 0
>> filter protocol arp pref 1 flower chain 0 handle 0x1
>>   eth_type arp
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>      index 4 ref 1 bind 1 installed 971 sec used 82 sec
>>      Action statistics:
>>     Sent 420 bytes 7 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 420 bytes 7 pkt
>>     backlog 0b 0p requeues 0
>>
>> filter protocol ip pref 2 flower chain 0
>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>   eth_type ipv4
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>      index 2 ref 1 bind 1 installed 972 sec used 67 sec
>>      Action statistics:
>>     Sent 79272204362 bytes 91511261 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 79272204362 bytes 91511261 pkt
>>     backlog 0b 0p requeues 0
>>
>> #  tc -s filter ls dev mlx_pf0vf0 ingress
>> filter protocol arp pref 1 flower chain 0
>> filter protocol arp pref 1 flower chain 0 handle 0x1
>>   eth_type arp
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_p0) stolen
>>      index 3 ref 1 bind 1 installed 978 sec used 88 sec
>>      Action statistics:
>>     Sent 600 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 600 bytes 10 pkt
>>     backlog 0b 0p requeues 0
>>
>> filter protocol ip pref 2 flower chain 0
>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>   eth_type ipv4
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_p0) stolen
>>      index 1 ref 1 bind 1 installed 978 sec used 73 sec
>>      Action statistics:
>>     Sent 71556027574 bytes 47805525 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 71556027574 bytes 47805525 pkt
>>     backlog 0b 0p requeues 0
>>
>>
>>
>> test case 2:  tcp send from VF to PF
>>
>> On the reomte server: iperf -s
>>
>> in the vm:
>>
>> # iperf -c 10.0.0.241 -t 10 -i 2
>>
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.241, TCP port 5001
>> TCP window size:  230 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.75 port 53166 connected with 10.0.0.241 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 2.0 sec   939 MBytes  3.94 Gbits/sec
>> [  3]  2.0- 4.0 sec   944 MBytes  3.96 Gbits/sec
>> [  3]  4.0- 6.0 sec  1.01 GBytes  4.34 Gbits/sec
>> [  3]  6.0- 8.0 sec  1.03 GBytes  4.44 Gbits/sec
>> [  3]  8.0-10.0 sec  1.02 GBytes  4.39 Gbits/sec
>> [  3]  0.0-10.0 sec  4.90 GBytes  4.21 Gbits/sec
>>
>>
>> Bad performance with offload.  All the packet are offloaded. 
>>
>> It is the offload problem in the hardware?
>>
>>
>> BR
>>
>> wenxu
>>
>>
> Hi Wenxu,
>
> We didn't notice this behavior.
> Could it be your VM doesn't have enough resources to generate the traffic?
> As a listener it's only sending the acks.

I  don't think so. If delete the ingress qdisc on mlx_pf0vf0 and set ifconfig mlx_pf0vf0 10.0.0.241/24"

iperf -s on the host.

In the vm.

# iperf -c 10.0.0.241 -t 10 -i 2
------------------------------------------------------------
Client connecting to 10.0.0.241, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[  3] local 10.0.0.75 port 50960 connected with 10.0.0.241 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  6.34 GBytes  27.3 Gbits/sec
[  3]  2.0- 4.0 sec  6.45 GBytes  27.7 Gbits/sec
[  3]  4.0- 6.0 sec  6.65 GBytes  28.6 Gbits/sec
[  3]  6.0- 8.0 sec  6.60 GBytes  28.3 Gbits/sec
[  3]  8.0-10.0 sec  6.29 GBytes  27.0 Gbits/sec
[  3]  0.0-10.0 sec  32.3 GBytes  27.8 Gbits/sec


The VM can generic enough traffic. Maybe you can test my case with upstream kernel

> Thanks,
> Roi

Re: Bad performance for VF outgoing in offloaded mode

[wenxu]

On 12/4/2019 9:50 PM, Roi Dayan wrote:
>
> On 2019-11-28 7:03 AM, wenxu wrote:
>> Hi mellanox team,
>>
>>
>> I did a performance test for tc offload with upstream kernel:
>>
>> I setup a vm with a VF as eth0
>>
>> In the vm:
>>
>> ifconfig eth0 10.0.0.75/24 up
>>
>>
>> On the host the mlx_p0 is the pf representor and mlx_pf0vf0 is the vf representor
>>
>> The device in the switchdev mode
>>
>> # grep -ri "" /sys/class/net/*/phys_* 2>/dev/null
>> /sys/class/net/mlx_p0/phys_port_name:p0
>> /sys/class/net/mlx_p0/phys_switch_id:34ebc100034b6b50
>> /sys/class/net/mlx_pf0vf0/phys_port_name:pf0vf0
>> /sys/class/net/mlx_pf0vf0/phys_switch_id:34ebc100034b6b50
>> /sys/class/net/mlx_pf0vf1/phys_port_name:pf0vf1
>> /sys/class/net/mlx_pf0vf1/phys_switch_id:34ebc100034b6b50
>>
>>
>> The tc filter as following: just forward ip/arp packets  in mlx_p0 and mlx_pf0vf0 each other
>>
>> tc qdisc add dev mlx_p0 ingress
>> tc qdisc add dev mlx_pf0vf0 ingress
>>
>> tc filter add dev mlx_pf0vf0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_p0
>> tc filter add dev mlx_p0 pref 2 ingress  protocol ip flower skip_sw action mirred egress redirect dev mlx_pf0vf0
>>
>> tc filter add dev mlx_pf0vf0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_p0
>> tc filter add dev mlx_p0 pref 1 ingress  protocol arp flower skip_sw action mirred egress redirect dev mlx_pf0vf0
>>
>>
>> The remote server device eth0:
>>
>> ifconfig eth0 10.0.0.241/24
>>
>>
>> test case 1:   tcp recieve from VF to PF
>>
>> In the vm: iperf -s
>>
>> On the remote server:
>>
>> iperf -c 10.0.0.75 -t 10 -i 2
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.75, TCP port 5001
>> TCP window size: 85.0 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.241 port 59708 connected with 10.0.0.75 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 2.0 sec  5.40 GBytes  23.2 Gbits/sec
>> [  3]  2.0- 4.0 sec  5.35 GBytes  23.0 Gbits/sec
>> [  3]  4.0- 6.0 sec  5.46 GBytes  23.5 Gbits/sec
>> [  3]  6.0- 8.0 sec  5.10 GBytes  21.9 Gbits/sec
>> [  3]  8.0-10.0 sec  5.36 GBytes  23.0 Gbits/sec
>> [  3]  0.0-10.0 sec  26.7 GBytes  22.9 Gbits/sec
>>
>>
>> Good performance with offload.
>>
>> # tc -s filter ls dev mlx_p0 ingress
>> filter protocol arp pref 1 flower chain 0
>> filter protocol arp pref 1 flower chain 0 handle 0x1
>>   eth_type arp
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>      index 4 ref 1 bind 1 installed 971 sec used 82 sec
>>      Action statistics:
>>     Sent 420 bytes 7 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 420 bytes 7 pkt
>>     backlog 0b 0p requeues 0
>>
>> filter protocol ip pref 2 flower chain 0
>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>   eth_type ipv4
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_pf0vf0) stolen
>>      index 2 ref 1 bind 1 installed 972 sec used 67 sec
>>      Action statistics:
>>     Sent 79272204362 bytes 91511261 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 79272204362 bytes 91511261 pkt
>>     backlog 0b 0p requeues 0
>>
>> #  tc -s filter ls dev mlx_pf0vf0 ingress
>> filter protocol arp pref 1 flower chain 0
>> filter protocol arp pref 1 flower chain 0 handle 0x1
>>   eth_type arp
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_p0) stolen
>>      index 3 ref 1 bind 1 installed 978 sec used 88 sec
>>      Action statistics:
>>     Sent 600 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 600 bytes 10 pkt
>>     backlog 0b 0p requeues 0
>>
>> filter protocol ip pref 2 flower chain 0
>> filter protocol ip pref 2 flower chain 0 handle 0x1
>>   eth_type ipv4
>>   skip_sw
>>   in_hw in_hw_count 1
>>     action order 1: mirred (Egress Redirect to device mlx_p0) stolen
>>      index 1 ref 1 bind 1 installed 978 sec used 73 sec
>>      Action statistics:
>>     Sent 71556027574 bytes 47805525 pkt (dropped 0, overlimits 0 requeues 0)
>>     Sent software 0 bytes 0 pkt
>>     Sent hardware 71556027574 bytes 47805525 pkt
>>     backlog 0b 0p requeues 0
>>
>>
>>
>> test case 2:  tcp send from VF to PF
>>
>> On the reomte server: iperf -s
>>
>> in the vm:
>>
>> # iperf -c 10.0.0.241 -t 10 -i 2
>>
>> ------------------------------------------------------------
>> Client connecting to 10.0.0.241, TCP port 5001
>> TCP window size:  230 KByte (default)
>> ------------------------------------------------------------
>> [  3] local 10.0.0.75 port 53166 connected with 10.0.0.241 port 5001
>> [ ID] Interval       Transfer     Bandwidth
>> [  3]  0.0- 2.0 sec   939 MBytes  3.94 Gbits/sec
>> [  3]  2.0- 4.0 sec   944 MBytes  3.96 Gbits/sec
>> [  3]  4.0- 6.0 sec  1.01 GBytes  4.34 Gbits/sec
>> [  3]  6.0- 8.0 sec  1.03 GBytes  4.44 Gbits/sec
>> [  3]  8.0-10.0 sec  1.02 GBytes  4.39 Gbits/sec
>> [  3]  0.0-10.0 sec  4.90 GBytes  4.21 Gbits/sec
>>
>>
>> Bad performance with offload.  All the packet are offloaded. 
>>
>> It is the offload problem in the hardware?
>>
>>
>> BR
>>
>> wenxu
>>
>>
> Hi Wenxu,
>
> We didn't notice this behavior.
> Could it be your VM doesn't have enough resources to generate the traffic?
> As a listener it's only sending the acks.

Sorry, I found it is the problem of remote 10.0.0.241 server.

>
> Thanks,
> Roi