Re: [PATCH net-next v2] net/mlx5e: Add mlx5e_flower_parse_meta support

[Saeed Mahameed <saeedm@mellanox.com>]

On Tue, 2020-01-07 at 17:16 +0800, wenxu@ucloud.cn wrote:
> From: wenxu <wenxu@ucloud.cn>
> 
> In the flowtables offload all the devices in the flowtables
> share the same flow_block. An offload rule will be installed on
> all the devices. This scenario is not correct.
> 
> It is no problem if there are only two devices in the flowtable,
> The rule with ingress and egress on the same device can be reject
> by driver.
> 
> But more than two devices in the flowtable will install the wrong
> rules on hardware.
> 
> For example:
> Three devices in a offload flowtables: dev_a, dev_b, dev_c
> 
> A rule ingress from dev_a and egress to dev_b:
> The rule will install on device dev_a.
> The rule will try to install on dev_b but failed for ingress
> and egress on the same device.
> The rule will install on dev_c. This is not correct.
> 
> The flowtables offload avoid this case through restricting the
> ingress dev
> with FLOW_DISSECTOR_KEY_META as following patch.
> http://patchwork.ozlabs.org/patch/1218109/
> 
> So the mlx5e driver also should support the FLOW_DISSECTOR_KEY_META
> parse.
> 
> Signed-off-by: wenxu <wenxu@ucloud.cn>
> ---
> v2: remap the patch description
> 
>  drivers/net/ethernet/mellanox/mlx5/core/en_tc.c | 39
> +++++++++++++++++++++++++

Hi Wenxu,

According to our regression Team, This commit is breaking vxlan offload
CC'ed Paul, Roi and Vlad, they will assist you if you need any help.

can you please investigate the following call trace ?


[  363.789805] #####################################
 [  363.791150] ## TEST test-vxlan-port-offload.sh ##
 [  363.792499] #####################################
 [  364.916862] :test: Test adding vxlan rule after creating vxlan
interface and reloading modules
 [  364.926877] :test: - create vxlan interface
 [  364.938976] :test: - reload modules
 [  365.689825] mlx5_core 0000:00:08.3 enp0s8f3: Link down
 [  365.790063] mlx5_core 0000:00:08.2 enp0s8f2: Link down
 [  366.040167] mlx5_core 0000:00:08.6 enp0s8f6: Link down
 [  370.399771] mlx5_core 0000:00:08.1: E-Switch: Disable:
mode(OFFLOADS), nvfs(1), active vports(2)
 [  371.006097] mlx5_core 0000:00:08.1: E-Switch: cleanup
 [  371.581179] mlx5_core 0000:00:08.1: driver left SR-IOV enabled
after remove
 [  371.585021] mlx5_core 0000:00:08.0: E-Switch: Disable:
mode(OFFLOADS), nvfs(3), active vports(4)
 [  371.743029]
=======================================================================
======
 [  371.746917] BUG mlx5_fs_ftes (Not tainted): Objects remaining in
mlx5_fs_ftes on __kmem_cache_shutdown()
 [  371.751265] -----------------------------------------------------
------------------------
 [  371.751265] 
 [  371.754662] Disabling lock debugging due to kernel taint
 [  371.755672] INFO: Slab 0x00000000c3fc424e objects=27 used=1
fp=0x00000000b711cbdf flags=0x8000000000010200
 [  371.757523] CPU: 0 PID: 13112 Comm: modprobe Tainted:
G    B             5.5.0-rc4-J8553-G8b261307a8b0 #1
 [  371.759458] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
 [  371.761692] Call Trace:
 [  371.762378]  dump_stack+0x50/0x70
 [  371.763168]  slab_err+0xb7/0xdc
 [  371.763936]  ? slub_cpu_dead+0x10/0xa0
 [  371.764873]  ? on_each_cpu_cond_mask+0x7a/0xa0
 [  371.765919]  ? __kmalloc+0x19f/0x1f0
 [  371.766749]  __kmem_cache_shutdown.cold+0x1b/0x12f
 [  371.767769]  kmem_cache_destroy+0x45/0xe0
 [  371.768669]  mlx5_cleanup_fs+0x12c/0x150 [mlx5_core]
 [  371.769753]  mlx5_unload+0x1e/0x70 [mlx5_core]
 [  371.770744]  mlx5_unload_one+0x49/0xb0 [mlx5_core]
 [  371.771772]  remove_one+0x38/0x80 [mlx5_core]
 [  371.772724]  pci_device_remove+0x36/0xa0
 [  371.773625]  device_release_driver_internal+0xe1/0x1b0
 [  371.774731]  driver_detach+0x3f/0x79
 [  371.775532]  bus_remove_driver+0x58/0xcc
 [  371.776407]  pci_unregister_driver+0x2d/0x90
 [  371.777436]  cleanup+0x11/0x254 [mlx5_core]
 [  371.778376]  __x64_sys_delete_module+0x11b/0x1f0
 [  371.779358]  ? exit_to_usermode_loop+0xac/0xb0
 [  371.780314]  do_syscall_64+0x42/0x130
 [  371.781129]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
 [  371.782305] RIP: 0033:0x7f0496589ae7
 [  371.783107] Code: 73 01 c3 48 8b 0d 99 a3 2b 00 f7 d8 64 89 01 48
83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 b0 00 00 00
0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 a3 2b 00 f7 d8 64 89 01
48
 [  371.786595] RSP: 002b:00007ffd14aff778 EFLAGS: 00000206 ORIG_RAX:
00000000000000b0
 [  371.788122] RAX: ffffffffffffffda RBX: 000055e5a261bf70 RCX:
00007f0496589ae7
 [  371.799948] RDX: 0000000000000000 RSI: 0000000000000800 RDI:
000055e5a261bfd8
 [  371.801336] RBP: 000055e5a261bf70 R08: 00007ffd14afe721 R09:
0000000000000000
 [  371.802773] R10: 00007f04965f81e0 R11: 0000000000000206 R12:
000055e5a261bfd8
 [  371.804182] R13: 000055e5a261c8e0 R14: 000055e5a261bc38 R15:
00007ffd14b00b98
 [  371.805633] INFO: Object 0x0000000057a96e69 @offset=9000
 [  371.806851] kmem_cache_destroy mlx5_fs_ftes: Slab cache still has
objects
 [  371.808330] CPU: 0 PID: 13112 Comm: modprobe Tainted:
G    B             5.5.0-rc4-J8553-G8b261307a8b0 #1
 [  371.810311] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014


>  1 file changed, 39 insertions(+)
> 
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
> b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
> index 9b32a9c..33d1ce5 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
> @@ -1805,6 +1805,40 @@ static void *get_match_headers_value(u32
> flags,
>  			     outer_headers);
>  }
>  
> +static int mlx5e_flower_parse_meta(struct net_device *filter_dev,
> +				   struct flow_cls_offload *f)
> +{
> +	struct flow_rule *rule = flow_cls_offload_flow_rule(f);
> +	struct netlink_ext_ack *extack = f->common.extack;
> +	struct net_device *ingress_dev;
> +	struct flow_match_meta match;
> +
> +	if (!flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_META))
> +		return 0;
> +
> +	flow_rule_match_meta(rule, &match);
> +	if (match.mask->ingress_ifindex != 0xFFFFFFFF) {
> +		NL_SET_ERR_MSG_MOD(extack, "Unsupported ingress ifindex
> mask");
> +		return -EINVAL;
> +	}
> +
> +	ingress_dev = __dev_get_by_index(dev_net(filter_dev),
> +					 match.key->ingress_ifindex);
> +	if (!ingress_dev) {
> +		NL_SET_ERR_MSG_MOD(extack,
> +				   "Can't find the ingress port to
> match on");
> +		return -EINVAL;
> +	}
> +
> +	if (ingress_dev != filter_dev) {
> +		NL_SET_ERR_MSG_MOD(extack,
> +				   "Can't match on the ingress filter
> port");
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
>  static int __parse_cls_flower(struct mlx5e_priv *priv,
>  			      struct mlx5_flow_spec *spec,
>  			      struct flow_cls_offload *f,
> @@ -1825,6 +1859,7 @@ static int __parse_cls_flower(struct mlx5e_priv
> *priv,
>  	u16 addr_type = 0;
>  	u8 ip_proto = 0;
>  	u8 *match_level;
> +	int err;
>  
>  	match_level = outer_match_level;
>  
> @@ -1868,6 +1903,10 @@ static int __parse_cls_flower(struct
> mlx5e_priv *priv,
>  						    spec);
>  	}
>  
> +	err = mlx5e_flower_parse_meta(filter_dev, f);
> +	if (err)
> +		return err;
> +
>  	if (flow_rule_match_key(rule, FLOW_DISSECTOR_KEY_BASIC)) {
>  		struct flow_match_basic match;
>