BPF Archive on lore.kernel.org
 help / color / Atom feed
* Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb
@ 2020-06-30 14:28 Rudi Ratloser
  2020-06-30 14:56 ` Daniel Borkmann
                   ` (3 more replies)
  0 siblings, 4 replies; 14+ messages in thread
From: Rudi Ratloser @ 2020-06-30 14:28 UTC (permalink / raw)
  To: bpf

We have experienced a kernel BPF null pointer dereference issue on all
our machines since mid of June. It might be related to an upgrade of
libvirt/kvm/qemu at that point of time. But we’re not sure.

None of the servers can be used with this bug, as they crash latest
one hour after reboot. The time period until kernel panic can be
easily reduced down to 2 minutes, when starting one or more
applications of the following list:
- LXD daemon (4.2.1)
- libvirtd daemon (6.4.0) with qemu/kvm guests
- NFS server 2.5.1
- Mozilla Firefox
- Mozilla Thunderbird

If none of the applications run, the systems seem to be stable.

Intermediate solution:
Downgrade Linux kernel to 4.9.226 LTS or 4.4.226  LTS on all the machines

Why this solution works is not clear, yet. One of the major
differences we saw is, that both kernel packages have been configured
with user namespaces disabled.

We experienced the kernel freeze on following Arch Linux kernels:
- 5.7.0 (5.7.0-3-MANJARO x64)
- 5.6.16 (5.6.16-1-MANJARO x64)
- 5.4.44 (5.4.44-1-MANJARO x64)
- 4.19.126 (4.19.126-1-MANJARO x64)
- 4.14.183 (4.14.183-1-MANJARO x64)
Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.

Subsequent e-mails will contain the relevant extracts from journal or
netconsole logs.

Help and support on this issue is welcome.

^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re:  BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb
@ 2020-06-30 15:11 Rudi Ratloser
  0 siblings, 0 replies; 14+ messages in thread
From: Rudi Ratloser @ 2020-06-30 15:11 UTC (permalink / raw)
  To: bpf


> We have experienced a kernel BPF null pointer dereference issue on all
> our machines since mid of June. It might be related to an upgrade of
> libvirt/kvm/qemu at that point of time. But we’re not sure.
...
> We experienced the kernel freeze on following Arch Linux kernels:
> - 5.7.0 (5.7.0-3-MANJARO x64)
> - 5.6.16 (5.6.16-1-MANJARO x64)
> - 5.4.44 (5.4.44-1-MANJARO x64)
> - 4.19.126 (4.19.126-1-MANJARO x64)
> - 4.14.183 (4.14.183-1-MANJARO x64)
> Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.
> 
> Subsequent e-mails will contain the relevant extracts from journal or
> netconsole logs.

Kernel 5.7.0 (5.7.0-3-MANJARO x64)

BUG: kernel NULL pointer dereference, address: 0000000000000010
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 1 PID: 1132 Comm: nfsd Not tainted 5.7.0-3-MANJARO #1
Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
RIP: 0010:__cgroup_bpf_run_filter_skb+0x196/0x230
Code: 48 89 73 18 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 e8 d8 cb ec ff e8 93 12 f2 ff 48 8b 85 38 06 00 00 31 ed <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 34 49 8b 46 08 65 48 89 05 01
RSP: 0018:ffffaddac09eba20 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff93e20832d0e0 RCX: 0000000000000034
RDX: 0000000000000000 RSI: ffff93e1f0af0000 RDI: ffffffff9b7f6888
RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff93e20fe80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003d158e004 CR4: 00000000001626e0
Call Trace:
ip_finish_output+0x68/0xa0
ip_output+0x76/0x130
? __ip_finish_output+0x1e0/0x1e0
__ip_queue_xmit+0x186/0x440
? __switch_to_asm+0x34/0x70
? __switch_to_asm+0x40/0x70
__tcp_transmit_skb+0x53e/0xbf0
? __switch_to_asm+0x34/0x70
tcp_write_xmit+0x391/0x11b0
__tcp_push_pending_frames+0x32/0xf0
tcp_sendmsg_locked+0xa3c/0xb50
tcp_sendmsg+0x28/0x40
sock_sendmsg+0x57/0x60
xprt_sock_sendmsg+0xe8/0x2b0 [sunrpc]
? nfsd_destroy+0x60/0x60 [nfsd]
svc_tcp_sendto+0x77/0xd0 [sunrpc]
svc_send+0x80/0x1f0 [sunrpc]
nfsd+0xed/0x150 [nfsd]
kthread+0x13e/0x160
? __kthread_bind_mask+0x60/0x60
ret_from_fork+0x35/0x40
Modules linked in: rpcsec_gss_krb5 scsi_transport_iscsi veth xt_CHECKSUM vhost_net vhost tap vhost_iotlb tun ebtable_filter ebtables ip6table_filter ip6_tables xt_MASQUERADE xt_recent xt_comment ipt_REJECT nf_reject_ipv4 xt_addrtype br_netfilter xt_physdev iptable_nat xt_mark iptable_mangle xt_TCPMSS xt_hashlimit xt_tcpudp xt_CT iptable_raw xt_multiport xt_conntrack nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_nat nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp llc fuse nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ofpart cmdlinepart intel_spi_platform intel_spi mei_hdcp i915 eeepc_wmi spi_nor asus_wmi mtd iTCO_wdt iTCO_vendor_support battery snd_hda_codec_realtek sparse_keymap wmi_bmof rfkill snd_hda_codec_generic aesni_intel ledtrig_audio crypto_simd snd_hda_intel snd_intel_dspcfg cryptd glue_helper i2c_algo_bit snd_hda_codec intel_cstate intel_uncore snd_hda_core snd_hwdep drm_kms_helper r8169 intel_rapl_perf snd_pcm joydev realtek i2c_i801 libphy snd_timer mousedev cec snd rc_core mei_me input_leds intel_gtt syscopyarea sysfillrect e1000e lpc_ich sysimgblt mei soundcore fb_sys_fops wmi evdev mac_hid nfsd usbip_host drm usbip_core nfs_acl auth_rpcgss lockd grace uinput crypto_user sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio dm_mod
crc32c_intel sr_mod cdrom xhci_pci xhci_hcd ehci_pci ehci_hcd
CR2: 0000000000000010
---[ end trace 6fe9bf5a0db7a0b9 ]---
RIP: 0010:__cgroup_bpf_run_filter_skb+0x196/0x230
Code: 48 89 73 18 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 e8 d8 cb ec ff e8 93 12 f2 ff 48 8b 85 38 06 00 00 31 ed <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 34 49 8b 46 08 65 48 89 05 01
RSP: 0018:ffffaddac09eba20 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff93e20832d0e0 RCX: 0000000000000034
RDX: 0000000000000000 RSI: ffff93e1f0af0000 RDI: ffffffff9b7f6888
RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff93e20fe80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003d158e004 CR4: 00000000001626e0
note: nfsd[1132] exited with preempt_count 1
-- Reboot --

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, back to index

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20200530074608.GA60664@fnst.localdomain>
2020-06-02 21:46 ` BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb Brenden Blanco
2020-06-02 22:17   ` Alexei Starovoitov
2020-06-03  6:20     ` Lu Fengqi
2020-06-03  8:22       ` Lu Fengqi
2020-06-09 20:50     ` Daniel Borkmann
2020-06-10  1:37       ` Zefan Li
2020-06-03  6:16   ` Lu Fengqi
2020-06-30 14:28 Rudi Ratloser
2020-06-30 14:56 ` Daniel Borkmann
2020-07-01  7:08   ` Thomas Reim
2020-07-01  6:46 ` Thomas Reim
2020-07-01  6:51 ` Thomas Reim
2020-07-01  6:58 ` Thomas Reim
2020-06-30 15:11 Rudi Ratloser

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
		bpf@vger.kernel.org
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.bpf


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git