WARNING: CPU: 9 PID: 0 at net/netfilter/nf_conntrack_core.c:977 __nf_conntrack_confirm+0x4e5/0x6f0 [nf_conntrack]

WARNING: CPU: 9 PID: 0 at net/netfilter/nf_conntrack_core.c:977 __nf_conntrack_confirm+0x4e5/0x6f0 [nf_conntrack]

[Harald Dunkel]

[-- Attachment #1: Type: text/plain, Size: 7829 bytes --]

Hi folks,

5.3.12 gave me

Nov 23 11:56:01 srvl064 kernel: [ 1372.693308] WARNING: CPU: 9 PID: 0 at net/netfilter/nf_conntrack_core.c:977 __nf_conntrack_confirm+0x4e5/0x6f0 [nf_conntrack]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693311] Modules linked in: xt_nat xt_tcpudp macvlan xt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_counter nft_chain_nat xt_addrtype nft_compat xt_conntrack br_netfilter overlay rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache nf_tables nfnetlink bridge stp llc binfmt_misc intel_powerclamp coretemp kvm_intel kvm irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_pcm aesni_intel snd_timer input_leds joydev snd sg i5500_temp aes_x86_64 i7core_edac soundcore ioatdma crypto_simd evdev cryptd pcspkr glue_helper ipmi_si intel_cstate ipmi_devintf ipmi_msghandler acpi_cpufreq button iptable_nat iptable_mangle iptable_filter ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 auth_rpcgss nf_defrag_ipv4 sunrpc ip6table_mangle ip6table_filter ip6_tables ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic 
raid1 raid0 multipath linear md_mod sr_mod
Nov 23 11:56:01 srvl064 kernel: [ 1372.693360]  sd_mod cdrom hid_cherry hid_generic usbhid hid ata_generic ata_piix ehci_pci uhci_hcd 3w_sas libata ehci_hcd crc32c_intel lpc_ich i2c_i801 igb mfd_core i2c_algo_bit usbcore scsi_mod dca
Nov 23 11:56:01 srvl064 kernel: [ 1372.693376] CPU: 9 PID: 0 Comm: swapper/9 Not tainted 5.3.12-raw #1
Nov 23 11:56:01 srvl064 kernel: [ 1372.693377] Hardware name: Intel Corporation S5520HC/S5520HC, BIOS S5500.86B.01.00.0060.090920111354 09/09/2011
Nov 23 11:56:01 srvl064 kernel: [ 1372.693382] RIP: 0010:__nf_conntrack_confirm+0x4e5/0x6f0 [nf_conntrack]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693385] Code: 41 39 46 30 0f 84 9a 00 00 00 4d 85 f6 0f 84 15 02 00 00 f0 41 ff 0e 0f 85 28 fd ff ff 4c 89 f7 e8 d0 49 20 e1 e9 1b fd ff ff <0f> 0b 44 89 e6 89 df 45 31 f6 e8 1c d8 ff ff be 00 02 00 00 48 c7
Nov 23 11:56:01 srvl064 kernel: [ 1372.693386] RSP: 0018:ffffc900065048a0 EFLAGS: 00010202
Nov 23 11:56:01 srvl064 kernel: [ 1372.693388] RAX: 0000000000000188 RBX: 0000000000003b0c RCX: 000000000000841c
Nov 23 11:56:01 srvl064 kernel: [ 1372.693390] RDX: 0000000000000001 RSI: 000000000000001c RDI: 0000000000000001
Nov 23 11:56:01 srvl064 kernel: [ 1372.693391] RBP: ffffffff820b2c80 R08: 000000001960e409 R09: ffff8898201f7e00
Nov 23 11:56:01 srvl064 kernel: [ 1372.693392] R10: 0000000000000001 R11: ffff88983aa0a000 R12: 000000000000841c
Nov 23 11:56:01 srvl064 kernel: [ 1372.693393] R13: 0000000000000000 R14: ffff8898201f7e00 R15: ffff8898201f7e58
Nov 23 11:56:01 srvl064 kernel: [ 1372.693395] FS:  0000000000000000(0000) GS:ffff88983fb00000(0000) knlGS:0000000000000000
Nov 23 11:56:01 srvl064 kernel: [ 1372.693397] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 23 11:56:01 srvl064 kernel: [ 1372.693398] CR2: 00007f4577f78380 CR3: 000000000200a002 CR4: 00000000000206e0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693399] Call Trace:
Nov 23 11:56:01 srvl064 kernel: [ 1372.693404]  <IRQ>
Nov 23 11:56:01 srvl064 kernel: [ 1372.693412]  nf_confirm+0xdb/0xf0 [nf_conntrack]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693418]  nf_hook_slow+0x3b/0xa0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693424]  ip_local_deliver+0xa6/0xf0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693428]  ? ip_protocol_deliver_rcu+0x1f0/0x1f0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693432]  ip_sabotage_in+0x52/0x60 [br_netfilter]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693434]  nf_hook_slow+0x3b/0xa0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693437]  ip_rcv+0x8d/0xe0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693440]  ? ip_rcv_finish_core.isra.0+0x400/0x400
Nov 23 11:56:01 srvl064 kernel: [ 1372.693446]  __netif_receive_skb_one_core+0x7b/0x90
Nov 23 11:56:01 srvl064 kernel: [ 1372.693449]  netif_receive_skb_internal+0x3b/0xc0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693452]  netif_receive_skb+0x18/0xe0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693462]  br_pass_frame_up+0x14e/0x170 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693469]  ? br_flood+0x62/0x120 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693475]  br_handle_frame_finish+0x16a/0x430 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693482]  ? br_pass_frame_up+0x170/0x170 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693485]  br_nf_hook_thresh+0xc7/0xd0 [br_netfilter]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693492]  ? br_pass_frame_up+0x170/0x170 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693495]  br_nf_pre_routing_finish+0x135/0x330 [br_netfilter]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693501]  ? br_pass_frame_up+0x170/0x170 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693506]  ? nf_nat_ipv4_in+0x20/0x60 [nf_nat]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693509]  br_nf_pre_routing+0x2d1/0x510 [br_netfilter]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693513]  ? br_nf_forward_ip+0x4e0/0x4e0 [br_netfilter]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693520]  br_handle_frame+0x1a4/0x370 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693527]  ? br_pass_frame_up+0x170/0x170 [bridge]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693530]  __netif_receive_skb_core+0x221/0xcb0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693534]  ? inet_gro_receive+0x25f/0x2e0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693537]  __netif_receive_skb_one_core+0x32/0x90
Nov 23 11:56:01 srvl064 kernel: [ 1372.693540]  netif_receive_skb_internal+0x3b/0xc0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693542]  napi_gro_receive+0x6d/0x180
Nov 23 11:56:01 srvl064 kernel: [ 1372.693550]  igb_poll+0x76c/0xf20 [igb]
Nov 23 11:56:01 srvl064 kernel: [ 1372.693553]  net_rx_action+0x12e/0x380
Nov 23 11:56:01 srvl064 kernel: [ 1372.693558]  __do_softirq+0x103/0x33a
Nov 23 11:56:01 srvl064 kernel: [ 1372.693562]  ? _raw_spin_lock+0xe/0x30
Nov 23 11:56:01 srvl064 kernel: [ 1372.693566]  irq_exit+0x93/0xb0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693568]  do_IRQ+0xb6/0x120
Nov 23 11:56:01 srvl064 kernel: [ 1372.693572]  common_interrupt+0xf/0xf
Nov 23 11:56:01 srvl064 kernel: [ 1372.693573]  </IRQ>
Nov 23 11:56:01 srvl064 kernel: [ 1372.693577] RIP: 0010:cpuidle_enter_state+0xac/0x460
Nov 23 11:56:01 srvl064 kernel: [ 1372.693579] Code: e8 19 a3 b7 ff 49 89 c5 8b 05 38 a3 ba 00 85 c0 0f 8f 30 02 00 00 31 ff e8 91 af b7 ff 45 84 f6 0f 85 b2 00 00 00 fb 45 85 e4 <0f> 89 cd 00 00 00 c7 45 10 00 00 00 00 48 83 c4 10 44 89 e0 5b 5d
Nov 23 11:56:01 srvl064 kernel: [ 1372.693580] RSP: 0018:ffffc90006337e70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffdd
Nov 23 11:56:01 srvl064 kernel: [ 1372.693582] RAX: ffff88983fb00000 RBX: ffffffff8208f080 RCX: 000000000000001f
Nov 23 11:56:01 srvl064 kernel: [ 1372.693583] RDX: 0000000000000000 RSI: 0000000035558106 RDI: 0000000000000000
Nov 23 11:56:01 srvl064 kernel: [ 1372.693585] RBP: ffffe8ffffd0e110 R08: 0000013f9ae4648e R09: 0000013f9e862951
Nov 23 11:56:01 srvl064 kernel: [ 1372.693586] R10: ffff88983fb28344 R11: ffff88983fb28324 R12: 0000000000000004
Nov 23 11:56:01 srvl064 kernel: [ 1372.693587] R13: 0000013f9ae4648e R14: 0000000000000000 R15: 0000013f98782725
Nov 23 11:56:01 srvl064 kernel: [ 1372.693592]  cpuidle_enter+0x24/0x40
Nov 23 11:56:01 srvl064 kernel: [ 1372.693597]  do_idle+0x1c9/0x240
Nov 23 11:56:01 srvl064 kernel: [ 1372.693600]  cpu_startup_entry+0x14/0x20
Nov 23 11:56:01 srvl064 kernel: [ 1372.693603]  start_secondary+0x14c/0x180
Nov 23 11:56:01 srvl064 kernel: [ 1372.693607]  secondary_startup_64+0xa4/0xb0
Nov 23 11:56:01 srvl064 kernel: [ 1372.693611] ---[ end trace 00d1dbfb4175aafa ]---

Full kern.log file is attached.

Please mail if I can help to track down this problem.


Regards
Harri

[-- Attachment #2: kern.log.txt.gz --]
[-- Type: application/gzip, Size: 20763 bytes --]

Re: WARNING: CPU: 9 PID: 0 at net/netfilter/nf_conntrack_core.c:977 __nf_conntrack_confirm+0x4e5/0x6f0 [nf_conntrack]

[Harald Dunkel]

PS: Platform is Debian 10.2. Docker is version 18.06.1~ce~3-0~debian.

I had setup a macvlan using

	docker network create -d macvlan -o parent=br0 --subnet=172.19.96.0/20 greylan

/etc/network/interfaces:

	auto eth0
	iface eth0 inet manual

	auto br0
	iface br0 inet static
	        address 172.19.96.191/20
	        gateway 172.19.96.1
	        bridge_ports eth0
	        bridge_stp off
	        bridge_waitport 0

There was one Docker container running. compose file:

	version: '3.7'
	services:
	  jenkins:
	    image: jenkins/jenkins:lts
	    container_name: jenkins
	    environment:
	      - JAVA_OPTS=-Xmx2G
	      - JENKINS_OPTS=--httpsPort=8443 --httpsPrivateKey=/var/jenkins_home/ssl/platjenkins.key.pem --httpsCertificate=/var/jenkins_home/ssl/platjenkins.cert.pem
	    ports:
	      - "8443:8443"
	      - "50000:50000"
	    volumes:
	      - /export/docker-persistent/jenkins_home:/var/jenkins_home
	    networks:
	      greylan:
	         ipv4_address: 172.19.96.46
	    restart: unless-stopped

	networks:
	  greylan:
	    external: true


Please note the port mapping lines that were kept by accident. Since I
omitted these lines there was no hiccup anymore


Regards
Harri

Re: WARNING: CPU: 9 PID: 0 at net/netfilter/nf_conntrack_core.c:977 __nf_conntrack_confirm+0x4e5/0x6f0 [nf_conntrack]

[Harald Dunkel]

Anybody interested?

Dropping the obsolete port mapping did *not* help. It crashed again.

Currently I would assume that the problem is triggered by the combination
of bridge and macvlan interfaces. Since I have bound the macvlan to eth0
(2 days ago) there was no incident anymore. Before it died within a
few hours.


Regards
Harri