[BUG] kernel warning from br_nf_local_in+0x157/0x180

[BUG] kernel warning from br_nf_local_in+0x157/0x180

[Jianbo Liu]

Hi Florian and Pablo,

We hit the following warning from br_nf_local_in+0x157/0x180.

[   57.571874] WARNING: CPU: 1 PID: 0 at
net/bridge/br_netfilter_hooks.c:616 br_nf_local_in+0x157/0x180
[br_netfilter]
[   57.572749] Modules linked in: xt_MASQUERADE nf_conntrack_netlink
nfnetlink iptable_nat xt_addrtype xt_conntrack nf_nat br_netfilter
rpcsec_gss_krb5 auth_rpcgss oid_registry overlay rpcrdma rdma_ucm
ib_iser libiscsi scsi_transport_isc
si ib_umad rdma_cm ib_ipoib iw_cm ib_cm mlx5_ib ib_uverbs ib_core
mlx5ctl mlx5_core
[   57.575158] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.8.0+ #19
[   57.575700] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[   57.576662] RIP: 0010:br_nf_local_in+0x157/0x180 [br_netfilter]
[   57.577195] Code: fe ff ff 41 bd 04 00 00 00 be 04 00 00 00 e9 4a ff
ff ff be 04 00 00 00 48 89 ef e8 f3 a9 3c e1 66 83 ad b4 00 00 00 04 eb
91 <0f> 0b e9 f1 fe ff ff 0f 0b e9 df fe ff ff 48 89 df e8 b3 53 47 e1
[   57.578722] RSP: 0018:ffff88885f845a08 EFLAGS: 00010202
[   57.579207] RAX: 0000000000000002 RBX: ffff88812dfe8000 RCX:
0000000000000000
[   57.579830] RDX: ffff88885f845a60 RSI: ffff8881022dc300 RDI:
0000000000000000
[   57.580454] RBP: ffff88885f845a60 R08: 0000000000000001 R09:
0000000000000003
[   57.581076] R10: 00000000ffff1300 R11: 0000000000000002 R12:
0000000000000000
[   57.581695] R13: ffff8881047ffe00 R14: ffff888108dbee00 R15:
ffff88814519b800
[   57.582313] FS:  0000000000000000(0000) GS:ffff88885f840000(0000)
knlGS:0000000000000000
[   57.583040] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   57.583564] CR2: 000000c4206aa000 CR3: 0000000103847001 CR4:
0000000000370eb0
[   57.584194] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   57.584820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[   57.585440] Call Trace:
[   57.585721]  <IRQ>
[   57.585976]  ? __warn+0x7d/0x130
[   57.586323]  ? br_nf_local_in+0x157/0x180 [br_netfilter]
[   57.586811]  ? report_bug+0xf1/0x1c0
[   57.587177]  ? handle_bug+0x3f/0x70
[   57.587539]  ? exc_invalid_op+0x13/0x60
[   57.587929]  ? asm_exc_invalid_op+0x16/0x20
[   57.588336]  ? br_nf_local_in+0x157/0x180 [br_netfilter]
[   57.588825]  nf_hook_slow+0x3d/0xd0
[   57.589188]  ? br_handle_vlan+0x4b/0x110
[   57.589579]  br_pass_frame_up+0xfc/0x150
[   57.589970]  ? br_port_flags_change+0x40/0x40
[   57.590396]  br_handle_frame_finish+0x346/0x5e0
[   57.590837]  ? ipt_do_table+0x32e/0x430
[   57.591221]  ? br_handle_local_finish+0x20/0x20
[   57.591656]  br_nf_hook_thresh+0x4b/0xf0 [br_netfilter]
[   57.592286]  ? br_handle_local_finish+0x20/0x20
[   57.592802]  br_nf_pre_routing_finish+0x178/0x480 [br_netfilter]
[   57.593348]  ? br_handle_local_finish+0x20/0x20
[   57.593782]  ? nf_nat_ipv4_pre_routing+0x25/0x60 [nf_nat]
[   57.594279]  br_nf_pre_routing+0x24c/0x550 [br_netfilter]
[   57.594780]  ? br_nf_hook_thresh+0xf0/0xf0 [br_netfilter]
[   57.595280]  br_handle_frame+0x1f3/0x3d0
[   57.595676]  ? br_handle_local_finish+0x20/0x20
[   57.596118]  ? br_handle_frame_finish+0x5e0/0x5e0
[   57.596566]  __netif_receive_skb_core+0x25b/0xfc0
[   57.597017]  ? __napi_build_skb+0x37/0x40
[   57.597418]  __netif_receive_skb_list_core+0xfb/0x220
[   57.597887]  netif_receive_skb_list_internal+0x1c0/0x2d0
[   57.598376]  ? mlx5e_handle_rx_cqe_mpwrq+0x10f/0x1f0 [mlx5_core]
[   57.598969]  napi_complete_done+0x101/0x180
[   57.599383]  mlx5e_napi_poll+0x1a2/0x6c0 [mlx5_core]
[   57.599894]  __napi_poll+0x23/0x1b0
[   57.600257]  net_rx_action+0x256/0x2b0
[   57.600641]  __do_softirq+0xc1/0x297
[   57.601014]  irq_exit_rcu+0x6a/0x90
[   57.601377]  common_interrupt+0x5d/0xa0
[   57.601765]  </IRQ>
[   57.602028]  <TASK>
[   57.602287]  asm_common_interrupt+0x22/0x40
[   57.602701] RIP: 0010:default_idle+0x13/0x20
[   57.603125] Code: c0 08 00 00 00 4d 29 c8 4c 01 c7 4c 29 c2 e9 72 ff
ff ff cc cc cc cc 8b 05 7a bf f0 00 85 c0 7e 07 0f 00 2d cf 50 2c 00 fb
f4 <fa> c3 90 66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 04 25 80 b5 02 00
[   57.604718] RSP: 0018:ffff888101873ee0 EFLAGS: 00000242
[   57.605202] RAX: 0000000000000001 RBX: 0000000000000001 RCX:
4000000000000000
[   57.605826] RDX: 0000000000000000 RSI: 0000000000000083 RDI:
000000000002ea4c
[   57.606451] RBP: ffffffff82434ae0 R08: 0000000000000001 R09:
0000000000000000
[   57.607079] R10: 0000000000000075 R11: 0000000000000000 R12:
0000000000000000
[   57.607704] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[   57.608337]  default_idle_call+0x30/0xb0
[   57.608730]  do_idle+0x177/0x1e0
[   57.609072]  ? swake_up_locked.part.48+0x1a/0x40
[   57.609517]  cpu_startup_entry+0x26/0x30
[   57.609909]  start_secondary+0x107/0x130
[   57.610302]  secondary_startup_64_no_verify+0x15d/0x16b
[   57.610787]  </TASK>

It is from the commit: "netfilter: bridge: confirm multicast packets
before passing them up the stack"
https://lore.kernel.org/netdev/20240229000135.8780-3-pablo@netfilter.org/
And the issue can be easliy reproduced on two B2B machines with the
following configs.

On hostA:
BR=tst1
NIC1=enp8s0f0
NIC2=enp8s0f1
ip link add $BR type bridge
iptables -A FORWARD -i $BR -j ACCEPT
ip link set $NIC1 master $BR
ip link set $NIC2 master $BR
bridge vlan add dev $NIC1 vid 2 pvid untagged
bridge vlan add dev $NIC2 vid 2
ip link set $BR type bridge vlan_filtering 1
ip link set $NIC1 up
ip link set $NIC2 up
ip link set $BR up
 
On hostB:
ns1=ns1
ns2=ns2
NIC1=enp8s0f0
NIC2=enp8s0f1
ip netns add $ns1 2>/dev/null
ip netns add $ns2 2>/dev/null
ip link set dev $NIC1 netns $ns1
ip -net ${ns1} addr add 1.1.1.1/24 dev $NIC1
ip -net ${ns1} link set dev $NIC1 up
ip link set dev $NIC2 netns $ns2
ip -net ${ns2} link add link $NIC2 name $NIC2.2 type vlan id 2
ip -net ${ns2} addr add 1.1.1.2/24 dev $NIC2.2
ip -net ${ns2} link set dev $NIC2 up
ip -net ${ns2} link set dev $NIC2.2 up
 
Then run tcpdump on brigde interface (tst1) on hostA.
The warning appears immediately after ping on hostB: ip netns exec ns1
ping 1.1.1.2 -c 1

Thanks!
Jianbo

Re: [BUG] kernel warning from br_nf_local_in+0x157/0x180

[Florian Westphal]

Jianbo Liu <jianbol@nvidia.com> wrote:
> Hi Florian and Pablo,

	I am sorry for being such a failire.

I can't figure this out.

I'm done. I am resigning from maintainer role.

Re: [BUG] kernel warning from br_nf_local_in+0x157/0x180

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

Hi Jianbo,

On Mon, Mar 18, 2024 at 09:41:46AM +0000, Jianbo Liu wrote:
> Hi Florian and Pablo,
> 
> We hit the following warning from br_nf_local_in+0x157/0x180.

Can you give a try to this patch?

This fix is not yet complete but it should fix the splat for this
test.

Thanks.

[-- Attachment #2: x.patch --]
[-- Type: text/x-diff, Size: 568 bytes --]

diff --git a/net/bridge/br_netfilter_hooks.c b/net/bridge/br_netfilter_hooks.c
index 35e10c5a766d..085d3f751b3f 100644
--- a/net/bridge/br_netfilter_hooks.c
+++ b/net/bridge/br_netfilter_hooks.c
@@ -612,6 +612,13 @@ static unsigned int br_nf_local_in(void *priv,
 	if (likely(nf_ct_is_confirmed(ct)))
 		return NF_ACCEPT;
 
+	if (skb->pkt_type != PACKET_BROADCAST &&
+	    skb->pkt_type != PACKET_MULTICAST) {
+		skb->_nfct = 0ul;
+		nf_conntrack_put(nfct);
+		return NF_ACCEPT;
+	}
+
 	WARN_ON_ONCE(skb_shared(skb));
 	WARN_ON_ONCE(refcount_read(&nfct->use) != 1);
 

Re: [BUG] kernel warning from br_nf_local_in+0x157/0x180

[Jianbo Liu]

On Fri, 2024-03-22 at 13:27 +0100, Pablo Neira Ayuso wrote:
> Hi Jianbo,
> 
> On Mon, Mar 18, 2024 at 09:41:46AM +0000, Jianbo Liu wrote:
> > Hi Florian and Pablo,
> > 
> > We hit the following warning from br_nf_local_in+0x157/0x180.
> 
> Can you give a try to this patch?
> 

Sorry, it doesn't fix.
It looks fine when running the test manually. But the warning still
appeared in our regression tests.

Thanks!
Jianbo

> This fix is not yet complete but it should fix the splat for this
> test.
> 
> Thanks.

Re: [BUG] kernel warning from br_nf_local_in+0x157/0x180

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

On Sat, Mar 23, 2024 at 10:37:16AM +0000, Jianbo Liu wrote:
> On Fri, 2024-03-22 at 13:27 +0100, Pablo Neira Ayuso wrote:
> > Hi Jianbo,
> > 
> > On Mon, Mar 18, 2024 at 09:41:46AM +0000, Jianbo Liu wrote:
> > > Hi Florian and Pablo,
> > > 
> > > We hit the following warning from br_nf_local_in+0x157/0x180.
> > 
> > Can you give a try to this patch?
> > 
> 
> Sorry, it doesn't fix.
> It looks fine when running the test manually. But the warning still
> appeared in our regression tests.

You mean different test triggers now the warning splat?

Not sure yet if this is the bug that is triggering the issue in your
testbed yet, but I find it odd that packets hitting the local_in hook
because promisc flag is set on can confirm conntrack entries.

Would you please give a try to this patch? Thanks!

[-- Attachment #2: x.patch --]
[-- Type: text/x-diff, Size: 2877 bytes --]

diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index f21097e73482..0abdec77f0b0 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -137,7 +137,9 @@ int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb
 	if (p->flags & BR_LEARNING)
 		br_fdb_update(br, p, eth_hdr(skb)->h_source, vid, 0);
 
-	local_rcv = !!(br->dev->flags & IFF_PROMISC);
+	BR_INPUT_SKB_CB(skb)->promisc = !!(br->dev->flags & IFF_PROMISC);
+	local_rcv = BR_INPUT_SKB_CB(skb)->promisc;
+
 	if (is_multicast_ether_addr(eth_hdr(skb)->h_dest)) {
 		/* by definition the broadcast is also a multicast address */
 		if (is_broadcast_ether_addr(eth_hdr(skb)->h_dest)) {
diff --git a/net/bridge/br_netfilter_hooks.c b/net/bridge/br_netfilter_hooks.c
index 35e10c5a766d..22e35623c148 100644
--- a/net/bridge/br_netfilter_hooks.c
+++ b/net/bridge/br_netfilter_hooks.c
@@ -600,11 +600,17 @@ static unsigned int br_nf_local_in(void *priv,
 				   struct sk_buff *skb,
 				   const struct nf_hook_state *state)
 {
+	bool promisc = BR_INPUT_SKB_CB(skb)->promisc;
 	struct nf_conntrack *nfct = skb_nfct(skb);
 	const struct nf_ct_hook *ct_hook;
 	struct nf_conn *ct;
 	int ret;
 
+	if (promisc) {
+		nf_reset_ct(skb);
+		return NF_ACCEPT;
+	}
+
 	if (!nfct || skb->pkt_type == PACKET_HOST)
 		return NF_ACCEPT;
 
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index 86ea5e6689b5..d4bedc87b1d8 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -589,6 +589,7 @@ struct br_input_skb_cb {
 #endif
 	u8 proxyarp_replied:1;
 	u8 src_port_isolated:1;
+	u8 promisc:1;
 #ifdef CONFIG_BRIDGE_VLAN_FILTERING
 	u8 vlan_filtered:1;
 #endif
diff --git a/net/bridge/netfilter/nf_conntrack_bridge.c b/net/bridge/netfilter/nf_conntrack_bridge.c
index 6f877e31709b..c3c51b9a6826 100644
--- a/net/bridge/netfilter/nf_conntrack_bridge.c
+++ b/net/bridge/netfilter/nf_conntrack_bridge.c
@@ -294,18 +294,24 @@ static unsigned int nf_ct_bridge_pre(void *priv, struct sk_buff *skb,
 static unsigned int nf_ct_bridge_in(void *priv, struct sk_buff *skb,
 				    const struct nf_hook_state *state)
 {
-	enum ip_conntrack_info ctinfo;
+	bool promisc = BR_INPUT_SKB_CB(skb)->promisc;
+	struct nf_conntrack *nfct = skb_nfct(skb);
 	struct nf_conn *ct;
 
-	if (skb->pkt_type == PACKET_HOST)
+	if (promisc) {
+		nf_reset_ct(skb);
+		return NF_ACCEPT;
+	}
+
+	if (!nfct || skb->pkt_type == PACKET_HOST)
 		return NF_ACCEPT;
 
 	/* nf_conntrack_confirm() cannot handle concurrent clones,
 	 * this happens for broad/multicast frames with e.g. macvlan on top
 	 * of the bridge device.
 	 */
-	ct = nf_ct_get(skb, &ctinfo);
-	if (!ct || nf_ct_is_confirmed(ct) || nf_ct_is_template(ct))
+	ct = container_of(nfct, struct nf_conn, ct_general);
+	if (nf_ct_is_confirmed(ct) || nf_ct_is_template(ct))
 		return NF_ACCEPT;
 
 	/* let inet prerouting call conntrack again */

Re: [BUG] kernel warning from br_nf_local_in+0x157/0x180

[Jianbo Liu]

On Tue, 2024-03-26 at 02:28 +0100, Pablo Neira Ayuso wrote:
> On Sat, Mar 23, 2024 at 10:37:16AM +0000, Jianbo Liu wrote:
> > On Fri, 2024-03-22 at 13:27 +0100, Pablo Neira Ayuso wrote:
> > > Hi Jianbo,
> > > 
> > > On Mon, Mar 18, 2024 at 09:41:46AM +0000, Jianbo Liu wrote:
> > > > Hi Florian and Pablo,
> > > > 
> > > > We hit the following warning from br_nf_local_in+0x157/0x180.
> > > 
> > > Can you give a try to this patch?
> > > 
> > 
> > Sorry, it doesn't fix.
> > It looks fine when running the test manually. But the warning still
> > appeared in our regression tests.
> 
> You mean different test triggers now the warning splat?

It should be the same. The test I run manually was configured by the
scripts I posted in this thread.

> 
> Not sure yet if this is the bug that is triggering the issue in your
> testbed yet, but I find it odd that packets hitting the local_in hook
> because promisc flag is set on can confirm conntrack entries.
> 

This fix seems ok. I didn't see the warning or any other issue.

Thanks!
Jianbo

> Would you please give a try to this patch? Thanks!