Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Rudi Ratloser]

We have experienced a kernel BPF null pointer dereference issue on all
our machines since mid of June. It might be related to an upgrade of
libvirt/kvm/qemu at that point of time. But we’re not sure.

None of the servers can be used with this bug, as they crash latest
one hour after reboot. The time period until kernel panic can be
easily reduced down to 2 minutes, when starting one or more
applications of the following list:
- LXD daemon (4.2.1)
- libvirtd daemon (6.4.0) with qemu/kvm guests
- NFS server 2.5.1
- Mozilla Firefox
- Mozilla Thunderbird

If none of the applications run, the systems seem to be stable.

Intermediate solution:
Downgrade Linux kernel to 4.9.226 LTS or 4.4.226  LTS on all the machines

Why this solution works is not clear, yet. One of the major
differences we saw is, that both kernel packages have been configured
with user namespaces disabled.

We experienced the kernel freeze on following Arch Linux kernels:
- 5.7.0 (5.7.0-3-MANJARO x64)
- 5.6.16 (5.6.16-1-MANJARO x64)
- 5.4.44 (5.4.44-1-MANJARO x64)
- 4.19.126 (4.19.126-1-MANJARO x64)
- 4.14.183 (4.14.183-1-MANJARO x64)
Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.

Subsequent e-mails will contain the relevant extracts from journal or
netconsole logs.

Help and support on this issue is welcome.

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Daniel Borkmann]

On 6/30/20 4:28 PM, Rudi Ratloser wrote:
> We have experienced a kernel BPF null pointer dereference issue on all
> our machines since mid of June. It might be related to an upgrade of
> libvirt/kvm/qemu at that point of time. But we’re not sure.
> 
> None of the servers can be used with this bug, as they crash latest
> one hour after reboot. The time period until kernel panic can be
> easily reduced down to 2 minutes, when starting one or more
> applications of the following list:
> - LXD daemon (4.2.1)
> - libvirtd daemon (6.4.0) with qemu/kvm guests
> - NFS server 2.5.1
> - Mozilla Firefox
> - Mozilla Thunderbird
> 
> If none of the applications run, the systems seem to be stable.
> 
> Intermediate solution:
> Downgrade Linux kernel to 4.9.226 LTS or 4.4.226  LTS on all the machines
> 
> Why this solution works is not clear, yet. One of the major
> differences we saw is, that both kernel packages have been configured
> with user namespaces disabled.
> 
> We experienced the kernel freeze on following Arch Linux kernels:
> - 5.7.0 (5.7.0-3-MANJARO x64)
> - 5.6.16 (5.6.16-1-MANJARO x64)
> - 5.4.44 (5.4.44-1-MANJARO x64)
> - 4.19.126 (4.19.126-1-MANJARO x64)
> - 4.14.183 (4.14.183-1-MANJARO x64)
> Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.
> 
> Subsequent e-mails will contain the relevant extracts from journal or
> netconsole logs.
> 
> Help and support on this issue is welcome.

Fix is under discussion here:

   https://lore.kernel.org/netdev/20200616180352.18602-1-xiyou.wangcong@gmail.com/

Thanks,
Daniel

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Thomas Reim]

>> We have experienced a kernel BPF null pointer dereference issue on all
>> our machines since mid of June. It might be related to an upgrade of
>> libvirt/kvm/qemu at that point of time. But we’re not sure.
>>
>> None of the servers can be used with this bug, as they crash latest
>> one hour after reboot. The time period until kernel panic can be
>> easily reduced down to 2 minutes, when starting one or more
>> applications of the following list:
>> - LXD daemon (4.2.1)
>> - libvirtd daemon (6.4.0) with qemu/kvm guests
>> - NFS server 2.5.1
>> - Mozilla Firefox
>> - Mozilla Thunderbird
>>
>> If none of the applications run, the systems seem to be stable.
>>
>> Intermediate solution:
>> Downgrade Linux kernel to 4.9.226 LTS or 4.4.226  LTS on all the machines
>>
>> Why this solution works is not clear, yet. One of the major
>> differences we saw is, that both kernel packages have been configured
>> with user namespaces disabled.
>>
>> We experienced the kernel freeze on following Arch Linux kernels:
>> - 5.7.0 (5.7.0-3-MANJARO x64)
>> - 5.6.16 (5.6.16-1-MANJARO x64)
>> - 5.4.44 (5.4.44-1-MANJARO x64)
>> - 4.19.126 (4.19.126-1-MANJARO x64)
>> - 4.14.183 (4.14.183-1-MANJARO x64)
>> Kernel configs can be taken from 
>> https://gitlab.manjaro.org/packages/core.
>>
>> Subsequent e-mails will contain the relevant extracts from journal or
>> netconsole logs.
>>
>> Help and support on this issue is welcome.
> 
> Fix is under discussion here:
> 
>    
> https://lore.kernel.org/netdev/20200616180352.18602-1-xiyou.wangcong@gmail.com/ 
> 
> 
> Thanks,
> Daniel

Dear Daniel,

thank you for the hint. I will try to follow-up the discussion. For your 
convenience I have added some of our many and various logs to this 
thread. Maybe it will be of some help for the team.

Below you will find one log from kernel 4.14, which maybe outlines a 
different issue. Do we need another thread or do you judge it to have 
the same root cause?

Kernel 4.14.183 (4.14.183-1-MANJARO x64)

BUG: unable to handle kernel paging request at 0000200000000002
IP: __cgroup_bpf_run_filter_skb+0xca/0x1b0
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun 
ebtable_filter ebtables devlink ip6table_filter ip6_tables 
iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid 
nls_iso8859_1 nls_cp437 vfat fat input_leds joydev mousedev 
snd_hda_codec_hdmi eeepc_wmi iTCO_wdt asus_wmi mei_wdt sparse_keymap 
rfkill intel_rapl iTCO_vendor_support led_class wmi_bmof 
x86_pkg_temp_thermal intel_powerclamp coretemp evdev mac_hid kvm_intel 
i915 snd_hda_codec_realtek snd_hda_codec_generic kvm irqbypass 
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_hda_intel 
aesni_intel aes_x86_64 crypto_simd glue_helper cryptd i2c_algo_bit 
snd_hda_codec intel_cstate drm_kms_helper pcspkr snd_hda_core 
intel_rapl_perf snd_hwdep e1000e snd_pcm r8169 i2c_i801 intel_gtt mii 
syscopyarea snd_timer sysfillrect
  sysimgblt snd ptp lpc_ich mei_me fb_sys_fops soundcore shpchp pps_core 
mei wmi thermal fan pcc_cpufreq video button sch_fq_codel nfsd 
auth_rpcgss oid_registry drm nfs_acl lockd grace agpgart sunrpc 
ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto dm_thin_pool 
dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio 
hid_generic hid_logitech_hidpp dm_mod hid_logitech_dj usbhid hid sr_mod 
sd_mod cdrom ahci libahci ehci_pci xhci_pci ehci_hcd libata xhci_hcd 
crc32c_intel scsi_mod usbcore usb_common
CPU: 0 PID: 1313 Comm: vhost-1306 Not tainted 4.14.183-1-MANJARO #1
Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
task: ffff90a042548000 task.stack: ffff9c4e82b4c000
RIP: 0010:__cgroup_bpf_run_filter_skb+0xca/0x1b0
RSP: 0018:ffff9c4e82b4f9a8 EFLAGS: 00010296
RAX: ffff909fa973804e RBX: ffff909efbb6d800 RCX: 0000000000000001
RDX: ffff909fa973804e RSI: ffff909efbb6d800 RDI: ffff90a06c0a2000
RBP: 0000000000000014 R08: 0000000000000001 R09: ffff90a06c0a2000
R10: 000000000000af02 R11: 000000000300a8c0 R12: 0000200000000000
R13: 0000000000000000 R14: 0000000000000014 R15: ffff909fa973804e
FS:  0000000000000000(0000) GS:ffff90a09fa00000(0000) knlGS:0000000000000000
BUG: unable to handle kernel paging request at 0000200000000002
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000000002 CR3: 00000003c2438001 CR4: 00000000001626f0
Call Trace:
  sk_filter_trim_cap+0xd1/0x1a0
  tcp_v4_rcv+0x921/0xbc0
  ? ip_local_deliver+0xbf/0x120
IP: __cgroup_bpf_run_filter_skb+0xca/0x1b0
  ip_local_deliver_finish+0x66/0x200
PGD 0 P4D 0
  __netif_receive_skb_core+0x35e/0xb40
  ? nf_hook_slow+0x3f/0xb0
  netif_receive_skb_internal+0x4b/0x130
Oops: 0000 [#1] PREEMPT SMP PTI
  br_handle_frame_finish+0x148/0x510 [bridge]
  ? try_to_wake_up+0x54/0x4a0
  ? br_handle_frame_finish+0x510/0x510 [bridge]
  br_handle_frame+0x146/0x330 [bridge]
  __netif_receive_skb_core+0x3e9/0xb40
  ? __skb_get_hash_symmetric+0x74/0xc0
  netif_receive_skb_internal+0x4b/0x130
  tun_get_user+0x956/0xf00 [tun]
  ? __switch_to_asm+0x35/0x70
  ? __switch_to_asm+0x41/0x70
  ? __switch_to_asm+0x35/0x70
  ? __switch_to_asm+0x41/0x70
  tun_sendmsg+0x60/0x90 [tun]
  handle_tx+0x360/0x5f0 [vhost_net]
  vhost_worker+0xa7/0x100 [vhost]
  kthread+0x102/0x140
  ? vhost_dev_reset_owner+0x50/0x50 [vhost]
  ? kthread_create_on_node+0x60/0x60
  ret_from_fork+0x35/0x40
Code: 00 00 48 03 93 d0 00 00 00 4c 8b 6b 18 48 89 6b 18 49 89 c6 49 29 
d6 44 01 b3 80 00 00 00 44 89 f5 48 29 e8 48 89 83 d8 00 00 00 <41> f6 
44 24 02 08 75 7c 49 8b 44 24 28 49 8d 74 24 30 48 89 df
Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun 
ebtable_filter ebtables devlink ip6table_filter ip6_tables 
iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid 
nls_iso8859_1 nls_cp437 vfat fat input_leds joydev mousedev 
snd_hda_codec_hdmi eeepc_wmi iTCO_wdt asus_wmi mei_wdt sparse_keymap 
rfkill intel_rapl iTCO_vendor_support led_class wmi_bmof 
x86_pkg_temp_thermal intel_powerclamp coretemp evdev mac_hid kvm_intel 
i915 snd_hda_codec_realtek snd_hda_codec_generic kvm irqbypass 
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_hda_intel 
aesni_intel aes_x86_64 crypto_simd glue_helper cryptd i2c_algo_bit 
snd_hda_codec intel_cstate drm_kms_helper pcspkr snd_hda_core 
intel_rapl_perf snd_hwdep e1000e snd_pcm r8169 i2c_i801 intel_gtt mii 
syscopyarea snd_timer sysfillrect
  sysimgblt snd ptp lpc_ich mei_me fb_sys_fops soundcore shpchp pps_core 
mei wmi thermal fan pcc_cpufreq video button sch_fq_codel nfsd 
auth_rpcgss oid_registry drm nfs_acl lockd grace agpgart sunrpc 
ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto dm_thin_pool 
dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio 
hid_generic hid_logitech_hidpp dm_mod hid_logitech_dj usbhid hid sr_mod 
sd_mod cdrom ahci libahci ehci_pci xhci_pci ehci_hcd libata xhci_hcd 
crc32c_intel scsi_mod usbcore usb_common
RIP: __cgroup_bpf_run_filter_skb+0xca/0x1b0 RSP: ffff9c4e82b4f9a8
CR2: 0000200000000002
---[ end trace cb04f0196a7eba73 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: 0x3a000000 from 0xffffffff81000000 (relocation range: 
0xffffffff80000000-0xffffffffbfffffff)
CPU: 0 PID: 1313 Comm: vhost-1306 Not tainted 4.14.183-1-MANJARO #1
---[ end Kernel panic - not syncing: Fatal exception in interrupt
Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
task: ffff90a042548000 task.stack: ffff9c4e82b4c000
RIP: 0010:__cgroup_bpf_run_filter_skb+0xca/0x1b0
RSP: 0018:ffff9c4e82b4f9a8 EFLAGS: 00010296
RAX: ffff909fa973804e RBX: ffff909efbb6d800 RCX: 0000000000000001
RDX: ffff909fa973804e RSI: ffff909efbb6d800 RDI: ffff90a06c0a2000
RBP: 0000000000000014 R08: 0000000000000001 R09: ffff90a06c0a2000
R10: 000000000000af02 R11: 000000000300a8c0 R12: 0000200000000000
R13: 0000000000000000 R14: 0000000000000014 R15: ffff909fa973804e
FS:  0000000000000000(0000) GS:ffff90a09fa00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000000002 CR3: 00000003c2438001 CR4: 00000000001626f0
Call Trace:
  sk_filter_trim_cap+0xd1/0x1a0
  tcp_v4_rcv+0x921/0xbc0
  ? ip_local_deliver+0xbf/0x120
  ip_local_deliver_finish+0x66/0x200
  __netif_receive_skb_core+0x35e/0xb40
  ? nf_hook_slow+0x3f/0xb0
  netif_receive_skb_internal+0x4b/0x130
  br_handle_frame_finish+0x148/0x510 [bridge]
  ? try_to_wake_up+0x54/0x4a0
  ? br_handle_frame_finish+0x510/0x510 [bridge]
  br_handle_frame+0x146/0x330 [bridge]
  __netif_receive_skb_core+0x3e9/0xb40
  ? __skb_get_hash_symmetric+0x74/0xc0
  netif_receive_skb_internal+0x4b/0x130
  tun_get_user+0x956/0xf00 [tun]
  ? __switch_to_asm+0x35/0x70
  ? __switch_to_asm+0x41/0x70
  ? __switch_to_asm+0x35/0x70
  ? __switch_to_asm+0x41/0x70
  tun_sendmsg+0x60/0x90 [tun]
  handle_tx+0x360/0x5f0 [vhost_net]
  vhost_worker+0xa7/0x100 [vhost]
  kthread+0x102/0x140
  ? vhost_dev_reset_owner+0x50/0x50 [vhost]
  ? kthread_create_on_node+0x60/0x60
  ret_from_fork+0x35/0x40
Code: 00 00 48 03 93 d0 00 00 00 4c 8b 6b 18 48 89 6b 18 49 89 c6 49 29 
d6 44 01 b3 80 00 00 00 44 89 f5 48 29 e8 48 89 83 d8 00 00 00 <41> f6 
44 24 02 08 75 7c 49 8b 44 24 28 49 8d 74 24 30 48 89 df
RIP: __cgroup_bpf_run_filter_skb+0xca/0x1b0 RSP: ffff9c4e82b4f9a8
CR2: 0000200000000002
---[ end trace cb04f0196a7eba73 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: 0x3a000000 from 0xffffffff81000000 (relocation range: 
0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: Fatal exception in interrupt

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Thomas Reim]

>>
>> Fix is under discussion here:
>>
>> https://lore.kernel.org/netdev/20200616180352.18602-1-xiyou.wangcong@gmail.com/ 
>>
>>
>> Thanks,
>> Daniel
> 
> Dear Daniel,
> 
> thank you for the hint. I will try to follow-up the discussion. For your 
> convenience I have added some of our many and various logs to this 
> thread. Maybe it will be of some help for the team.
> 

There seems to be not much progress in above mentioned thread. Don't 
know if there have been other discussions that have resulted in a patch.

But last week we successfully tested kernel 5.8.11 (5.8.11-1-MANJARO 
x64) without experiencing a kernel panic/freeze.. In the userspace 
systemd 246.6 was running. No idea which changes have solved our issue. 
But here kernel is back stable again.

We will switch the workstations from intermediate kernel 4.9 LTS, which 
was stable all the time, back to kernel 5.8.

Thank you.

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Thomas Reim]

:
> We have experienced a kernel BPF null pointer dereference issue on all
> our machines since mid of June. It might be related to an upgrade of
> libvirt/kvm/qemu at that point of time. But we’re not sure.
>
...
> We experienced the kernel freeze on following Arch Linux kernels:
> - 5.7.0 (5.7.0-3-MANJARO x64)
> - 5.6.16 (5.6.16-1-MANJARO x64)
> - 5.4.44 (5.4.44-1-MANJARO x64)
> - 4.19.126 (4.19.126-1-MANJARO x64)
> - 4.14.183 (4.14.183-1-MANJARO x64)
> Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.
>
> Subsequent e-mails will contain the relevant extracts from journal or
> netconsole logs.
>
> Help and support on this issue is welcome.
>
Linux Kernel 5.6.16 (5.6.16-1-MANJARO x64)

BUG: kernel NULL pointer dereference, address: 0000000000000010
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 2 PID: 988 Comm: nfsd Not tainted 5.6.16-1-MANJARO #1
Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
RIP: 0010:__cgroup_bpf_run_filter_skb+0x196/0x230
Code: 48 89 73 18 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 e8 38 ef ec ff e8 f3 2d f2 ff 48 8b 85 38 06 00 00 31 ed <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 34 49 8b 46 08 65 48 89 05 d1
RSP: 0018:ffffa3e54097f9f0 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff962908bb82e0 RCX: 0000000000000034
RDX: 0000000000000000 RSI: ffff962907408900 RDI: ffffffffa2df2178
RBP: 0000000000000000 R08: ffff96290981ed20 R09: 000000000000fa4c
R10: 0000000000007d26 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff96290ff00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003e185e005 CR4: 00000000001626e0
Call Trace:
  ? __local_bh_enable_ip+0x33/0x70
  ip_finish_output+0x68/0xa0
  ip_output+0x76/0x130
  ? __ip_local_out+0x4b/0x170
  __ip_queue_xmit+0x186/0x440
  ? __switch_to_asm+0x34/0x70
  ? __switch_to_asm+0x40/0x70
  __tcp_transmit_skb+0x53e/0xbf0
  ? __switch_to_asm+0x34/0x70
  tcp_write_xmit+0x391/0x11b0
  __tcp_push_pending_frames+0x32/0xf0
  do_tcp_sendpages+0x5f8/0x630
  tcp_sendpage+0x48/0x80
  inet_sendpage+0x52/0x90
  kernel_sendpage+0x1a/0x30
  svc_send_common+0x62/0x150 [sunrpc]
  svc_sendto+0xd7/0x240 [sunrpc]
  svc_tcp_sendto+0x36/0x50 [sunrpc]
  svc_send+0x7b/0x190 [sunrpc]
  nfsd+0xed/0x150 [nfsd]
  ? nfsd_destroy+0x60/0x60 [nfsd]
  kthread+0x117/0x130
  ? __kthread_bind_mask+0x60/0x60
  ret_from_fork+0x35/0x40
Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun fuse bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat joydev mousedev input_leds intel_rapl_msr ofpart cmdlinepart intel_spi_platform intel_spi mei_wdt mei_hdcp spi_nor mtd iTCO_wdt iTCO_vendor_support eeepc_wmi asus_wmi battery sparse_keymap rfkill wmi_bmof intel_rapl_common snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm i915 irqbypass crct10dif_pclmul snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio crc32_pclmul snd_hda_intel i2c_algo_bit ghash_clmulni_intel snd_intel_dspcfg aesni_intel crypto_simd snd_hda_codec drm_kms_helper cryptd glue_helper intel_cstate pcspkr i2c_i801 intel_uncore snd_hda_core intel_rapl_perf snd_hwdep cec snd_pcm r8169 rc_core realtek intel_gtt snd_timer syscopyarea mei_me libphy lpc_ich snd mei e1000e sysfillrect soundcore sysimgblt fb_sys_fops wmi evdev mac_hid nfsd nfsv4 dns_resolver nfs_acl nfs lockd auth_rpcgss grace drm sunrpc
  fscache agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp hid_logitech_dj dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_generic usbhid hid dm_mod crc32c_intel sr_mod xhci_pci cdrom xhci_hcd ehci_pci ehci_hcd
CR2: 0000000000000010
---[ end trace 50bcc1a93a161137 ]---
RIP: 0010:__cgroup_bpf_run_filter_skb+0x196/0x230
Code: 48 89 73 18 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 e8 38 ef ec ff e8 f3 2d f2 ff 48 8b 85 38 06 00 00 31 ed <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 34 49 8b 46 08 65 48 89 05 d1
RSP: 0018:ffffa3e54097f9f0 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff962908bb82e0 RCX: 0000000000000034
RDX: 0000000000000000 RSI: ffff962907408900 RDI: ffffffffa2df2178
RBP: 0000000000000000 R08: ffff96290981ed20 R09: 000000000000fa4c
R10: 0000000000007d26 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff96290ff00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003e185e005 CR4: 00000000001626e0
note: nfsd[988] exited with preempt_count 1
-- Reboot --

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Thomas Reim]

:
> We have experienced a kernel BPF null pointer dereference issue on all
> our machines since mid of June. It might be related to an upgrade of
> libvirt/kvm/qemu at that point of time. But we’re not sure.
...
> We experienced the kernel freeze on following Arch Linux kernels:
> - 5.7.0 (5.7.0-3-MANJARO x64)
> - 5.6.16 (5.6.16-1-MANJARO x64)
> - 5.4.44 (5.4.44-1-MANJARO x64)
> - 4.19.126 (4.19.126-1-MANJARO x64)
> - 4.14.183 (4.14.183-1-MANJARO x64)
> Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.
>
> Subsequent e-mails will contain the relevant extracts from journal or
> netconsole logs.
>
> Help and support on this issue is welcome.

Kernel 5.4.44 (5.4.44-1-MANJARO x64)

BUG: kernel NULL pointer dereference, address: 0000000000000010
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 3 PID: 1405 Comm: vhost-1399 Not tainted 5.4.44-1-MANJARO #1
Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
RIP: 0010:__cgroup_bpf_run_filter_skb+0xd9/0x230
Code: 00 48 01 c8 48 89 43 50 41 83 ff 01 0f 84 c2 00 00 00 e8 da a4 ed ff e8 c5 ce f2 ff 44 89 fa 48 8d 84 d5 30 06 00 00 48 8b 00 <48> 8b 78 10 4c 8d 78 10 48 85 ff 0f 84 29 01 00 00 bd 01 00 00 00
RSP: 0018:ffffbc1780b077f8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffffa32ce1bed600 RCX: 0000000000000034
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffffa32cf1e22000 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000dc02 R11: ffffa32cfa1100a0 R12: 0000000000000014
R13: 0000000000000014 R14: ffffa32be074f662 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa32d0fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003cab56004 CR4: 00000000001626e0
Call Trace:
  sk_filter_trim_cap+0x12f/0x270
  ? tcp_v4_inbound_md5_hash+0x56/0x170
  tcp_v4_rcv+0x9bc/0xc10
  ? arp_process+0x221/0x7e0
  ip_protocol_deliver_rcu+0x2b/0x1e0
  ip_local_deliver_finish+0x55/0x70
  ip_local_deliver+0x115/0x130
  ? ip_protocol_deliver_rcu+0x1e0/0x1e0
  ip_rcv+0x62/0x110
  __netif_receive_skb_one_core+0x87/0xa0
  netif_receive_skb_internal+0x93/0xe0
  netif_receive_skb+0x18/0xd0
  br_pass_frame_up+0xf0/0x1d0 [bridge]
  ? br_port_flags_change+0x70/0x70 [bridge]
  br_handle_frame_finish+0x18a/0x450 [bridge]
  br_handle_frame+0x238/0x380 [bridge]
  ? br_handle_local_finish+0xa0/0xa0 [bridge]
  __netif_receive_skb_core+0x3e7/0xc20
  ? kvm_irq_delivery_to_apic_fast+0x86/0x170 [kvm]
  __netif_receive_skb_one_core+0x3d/0xa0
  netif_receive_skb_internal+0x93/0xe0
  netif_receive_skb+0x18/0xd0
  tun_sendmsg+0x3a7/0x5d0 [tun]
  vhost_tx_batch.constprop.0+0x65/0xf0 [vhost_net]
  handle_tx_copy+0x187/0x5b0 [vhost_net]
  handle_tx+0xa5/0xe0 [vhost_net]
  vhost_worker+0xb9/0x130 [vhost]
  ? vhost_new_umem_range+0x1b0/0x1b0 [vhost]
  kthread+0x117/0x130
  ? __kthread_bind_mask+0x60/0x60
  ret_from_fork+0x35/0x40
Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat joydev mousedev input_leds intel_rapl_msr ofpart eeepc_wmi intel_rapl_common asus_wmi cmdlinepart x86_pkg_temp_thermal intel_powerclamp intel_spi_platform intel_spi mei_hdcp coretemp mei_wdt kvm_intel spi_nor kvm mtd iTCO_wdt iTCO_vendor_support snd_hda_codec_hdmi wmi_bmof irqbypass battery sparse_keymap rfkill crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_nhlt snd_hda_codec aesni_intel snd_hda_core crypto_simd cryptd glue_helper intel_cstate i2c_algo_bit intel_uncore intel_rapl_perf pcspkr i2c_i801 snd_hwdep drm_kms_helper snd_pcm r8169 mei_me snd_timer intel_gtt syscopyarea realtek e1000e mei libphy lpc_ich snd sysfillrect sysimgblt soundcore fb_sys_fops wmi evdev mac_hid nfsd
  nfs_acl lockd auth_rpcgss grace drm sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci libata crc32c_intel scsi_mod xhci_pci ehci_pci xhci_hcd ehci_hcd
CR2: 0000000000000010
---[ end trace ad97e7cc46d7ce69 ]---
RIP: 0010:__cgroup_bpf_run_filter_skb+0xd9/0x230
Code: 00 48 01 c8 48 89 43 50 41 83 ff 01 0f 84 c2 00 00 00 e8 da a4 ed ff e8 c5 ce f2 ff 44 89 fa 48 8d 84 d5 30 06 00 00 48 8b 00 <48> 8b 78 10 4c 8d 78 10 48 85 ff 0f 84 29 01 00 00 bd 01 00 00 00
RSP: 0018:ffffbc1780b077f8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffffa32ce1bed600 RCX: 0000000000000034
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffffa32cf1e22000 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000dc02 R11: ffffa32cfa1100a0 R12: 0000000000000014
R13: 0000000000000014 R14: ffffa32be074f662 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa32d0fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003cab56004 CR4: 00000000001626e0
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: 0x27400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Thomas Reim]

> We have experienced a kernel BPF null pointer dereference issue on all
> our machines since mid of June. It might be related to an upgrade of
> libvirt/kvm/qemu at that point of time. But we’re not sure.
>
> [...]
>
> We experienced the kernel freeze on following Arch Linux kernels:
> - 5.7.0 (5.7.0-3-MANJARO x64)
> - 5.6.16 (5.6.16-1-MANJARO x64)
> - 5.4.44 (5.4.44-1-MANJARO x64)
> - 4.19.126 (4.19.126-1-MANJARO x64)
> - 4.14.183 (4.14.183-1-MANJARO x64)
> Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.
>
> Subsequent e-mails will contain the relevant extracts from journal or
> netconsole logs.
>
> Help and support on this issue is welcome.

Kernel 4.19.126 (4.19.126-1-MANJARO x64)

We logged two different kind of kernel freezes:
- Gneral protection fault (1) (2)
- Unable to handle kernel paging request (3)

1. Kernel freeze preceeded by failing ethernet interface
[10979.771448] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10979.771448]   TDH                  <b>#012[10979.771448]   TDT                  <14>#012[10979.771448]   next_to_use          <14>#012[10979.771448]   next_to_clean        <a>#012[10979.771448] buffer_info[next_to_clean]:#012[10979.771448]   time_stamp           <100104b1a>#012[10979.771448]   next_to_watch        <b>#012[10979.771448]   jiffies              <100104b80>#012[10979.771448]   next_to_watch.status <0>#012[10979.771448] MAC Status             <80083>#012[10979.771448] PHY Status             <796d>#012[10979.771448] PHY 1000BASE-T Status  <3c00>#012[10979.771448] PHY Extended Status    <3000>#012[10979.771448] PCI Status             <10>
[10979.771448] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10979.771448]   TDH                  <b>#012[10979.771448]   TDT                  <14>#012[10979.771448]   next_to_use          <14>#012[10979.771448]   next_to_clean        <a>#012[10979.771448] buffer_info[next_to_clean]:#012[10979.771448]   time_stamp           <100104b1a>#012[10979.771448]   next_to_watch        <b>#012[10979.771448]   jiffies              <100104b80>#012[10979.771448]   next_to_watch.status <0>#012[10979.771448] MAC Status             <80083>#012[10979.771448] PHY Status             <796d>#012[10979.771448] PHY 1000BASE-T Status  <3c00>#012[10979.771448] PHY Extended Status    <3000>#012[10979.771448] PCI Status             <10>
[10981.771576] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10981.771576]   TDH                  <b>#012[10981.771576]   TDT                  <14>#012[10981.771576]   next_to_use          <14>#012[10981.771576]   next_to_clean        <a>#012[10981.771576] buffer_info[next_to_clean]:#012[10981.771576]   time_stamp           <100104b1a>#012[10981.771576]   next_to_watch        <b>#012[10981.771576]   jiffies              <100104c48>#012[10981.771576]   next_to_watch.status <0>#012[10981.771576] MAC Status             <80083>#012[10981.771576] PHY Status             <796d>#012[10981.771576] PHY 1000BASE-T Status  <3c00>#012[10981.771576] PHY Extended Status    <3000>#012[10981.771576] PCI Status             <10>
[10981.771576] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10981.771576]   TDH                  <b>#012[10981.771576]   TDT                  <14>#012[10981.771576]   next_to_use          <14>#012[10981.771576]   next_to_clean        <a>#012[10981.771576] buffer_info[next_to_clean]:#012[10981.771576]   time_stamp           <100104b1a>#012[10981.771576]   next_to_watch        <b>#012[10981.771576]   jiffies              <100104c48>#012[10981.771576]   next_to_watch.status <0>#012[10981.771576] MAC Status             <80083>#012[10981.771576] PHY Status             <796d>#012[10981.771576] PHY 1000BASE-T Status  <3c00>#012[10981.771576] PHY Extended Status    <3000>#012[10981.771576] PCI Status             <10>
[10983.771682] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10983.771682]   TDH                  <b>#012[10983.771682]   TDT                  <14>#012[10983.771682]   next_to_use          <14>#012[10983.771682]   next_to_clean        <a>#012[10983.771682] buffer_info[next_to_clean]:#012[10983.771682]   time_stamp           <100104b1a>#012[10983.771682]   next_to_watch        <b>#012[10983.771682]   jiffies              <100104d10>#012[10983.771682]   next_to_watch.status <0>#012[10983.771682] MAC Status             <80083>#012[10983.771682] PHY Status             <796d>#012[10983.771682] PHY 1000BASE-T Status  <3c00>#012[10983.771682] PHY Extended Status    <3000>#012[10983.771682] PCI Status             <10>
[10983.771682] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10983.771682]   TDH                  <b>#012[10983.771682]   TDT                  <14>#012[10983.771682]   next_to_use          <14>#012[10983.771682]   next_to_clean        <a>#012[10983.771682] buffer_info[next_to_clean]:#012[10983.771682]   time_stamp           <100104b1a>#012[10983.771682]   next_to_watch        <b>#012[10983.771682]   jiffies              <100104d10>#012[10983.771682]   next_to_watch.status <0>#012[10983.771682] MAC Status             <80083>#012[10983.771682] PHY Status             <796d>#012[10983.771682] PHY 1000BASE-T Status  <3c00>#012[10983.771682] PHY Extended Status    <3000>#012[10983.771682] PCI Status             <10>
[10985.771787] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10985.771787]   TDH                  <b>#012[10985.771787]   TDT                  <14>#012[10985.771787]   next_to_use          <14>#012[10985.771787]   next_to_clean        <a>#012[10985.771787] buffer_info[next_to_clean]:#012[10985.771787]   time_stamp           <100104b1a>#012[10985.771787]   next_to_watch        <b>#012[10985.771787]   jiffies              <100104dd8>#012[10985.771787]   next_to_watch.status <0>#012[10985.771787] MAC Status             <80083>#012[10985.771787] PHY Status             <796d>#012[10985.771787] PHY 1000BASE-T Status  <3c00>#012[10985.771787] PHY Extended Status    <3000>#012[10985.771787] PCI Status             <10>
[10985.771787] e1000e 0000:00:19.0 ethlocal: Detected Hardware Unit Hang:#012[10985.771787]   TDH                  <b>#012[10985.771787]   TDT                  <14>#012[10985.771787]   next_to_use          <14>#012[10985.771787]   next_to_clean        <a>#012[10985.771787] buffer_info[next_to_clean]:#012[10985.771787]   time_stamp           <100104b1a>#012[10985.771787]   next_to_watch        <b>#012[10985.771787]   jiffies              <100104dd8>#012[10985.771787]   next_to_watch.status <0>#012[10985.771787] MAC Status             <80083>#012[10985.771787] PHY Status             <796d>#012[10985.771787] PHY 1000BASE-T Status  <3c00>#012[10985.771787] PHY Extended Status    <3000>#012[10985.771787] PCI Status             <10>
[10986.731462] ------------[ cut here ]------------
[10986.731509] NETDEV WATCHDOG: ethlocal (e1000e): transmit queue 0 timed out
[10986.731559] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:465 dev_watchdog+0x212/0x220
[10986.731574] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat joydev mousedev input_leds intel_rapl snd_hda_codec_hdmi ofpart mei_wdt cmdlinepart intel_spi_platform intel_spi spi_nor mtd x86_pkg_temp_thermal iTCO_wdt intel_powerclamp iTCO_vendor_support eeepc_wmi asus_wmi coretemp sparse_keymap rfkill wmi_bmof kvm_intel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc i915 aesni_intel aes_x86_64 kvmgt crypto_simd cryptd vfio_mdev mdev glue_helper vfio_iommu_type1 vfio kvm intel_cstate intel_uncore snd_hda_codec_realtek irqbypass intel_rapl_perf snd_hda_codec_generic i2c_algo_bit drm_kms_helper pcspkr i2c_i801 snd_hda_intel drm snd_hda_codec
[10986.731462] ------------[ cut here ]------------
[10986.731647]  snd_hda_core snd_hwdep intel_gtt snd_pcm agpgart mei_me r8169 syscopyarea sysfillrect snd_timer sysimgblt pcc_cpufreq realtek snd lpc_ich libphy e1000e wmi soundcore mei fb_sys_fops evdev mac_hid nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci libata crc32c_intel scsi_mod xhci_pci xhci_hcd ehci_pci ehci_hcd
[10986.731712] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.126-1-MANJARO #1
[10986.731725] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[10986.731745] RIP: 0010:dev_watchdog+0x212/0x220
[10986.731763] Code: 63 74 24 e0 eb 8c 4c 89 f7 c6 05 e6 fd b8 00 01 e8 d3 b1 fc ff 44 89 e9 4c 89 f6 48 c7 c7 c8 ca d1 a9 48 89 c2 e8 5d 83 90 ff <0f> 0b eb bd 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 c7 47
[10986.731783] RSP: 0018:ffff8d258fd03e70 EFLAGS: 00010282
[10986.731802] RAX: 0000000000000000 RBX: ffff8d258bd00c00 RCX: 000000000000083f
[10986.731509] NETDEV WATCHDOG: ethlocal (e1000e): transmit queue 0 timed out
[10986.731820] RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
[10986.731837] RBP: ffff8d258a06045c R08: ffff8d258fd165b8 R09: 0000000000000000
[10986.731857] R10: ffffffffa8d04510 R11: 0000000000000000 R12: ffff8d258a060480
[10986.731875] R13: 0000000000000000 R14: ffff8d258a060000 R15: ffff8d258bd00c80
[10986.731895] FS:  0000000000000000(0000) GS:ffff8d258fd00000(0000) knlGS:0000000000000000
[10986.731913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10986.731931] CR2: 00007fcf213ea770 CR3: 00000001e640a006 CR4: 00000000001626e0
[10986.731948] Call Trace:
[10986.731968]  <IRQ>
[10986.731990]  ? qdisc_reset+0xd0/0xd0
[10986.732012]  call_timer_fn+0x2b/0x130
[10986.732036]  expire_timers+0x9c/0x100
[10986.732058]  run_timer_softirq+0x8f/0x180
[10986.731559] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:465 dev_watchdog+0x212/0x220
[10986.732158]  ? __hrtimer_run_queues+0x138/0x2a0
[10986.732175]  ? sched_clock+0x5/0x10
[10986.732198]  ? sched_clock_cpu+0xc/0xb0
[10986.732218]  __do_softirq+0xee/0x2e1
[10986.732238]  irq_exit+0xa4/0xe0
[10986.732256]  smp_apic_timer_interrupt+0x78/0x140
[10986.731574] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat joydev mousedev input_leds intel_rapl snd_hda_codec_hdmi ofpart mei_wdt cmdlinepart intel_spi_platform intel_spi spi_nor mtd x86_pkg_temp_thermal iTCO_wdt intel_powerclamp iTCO_vendor_support eeepc_wmi asus_wmi coretemp sparse_keymap rfkill wmi_bmof kvm_intel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc i915 aesni_intel aes_x86_64 kvmgt crypto_simd cryptd vfio_mdev mdev glue_helper vfio_iommu_type1 vfio kvm intel_cstate intel_uncore snd_hda_codec_realtek irqbypass intel_rapl_perf snd_hda_codec_generic i2c_algo_bit drm_kms_helper pcspkr i2c_i801 snd_hda_intel drm snd_hda_codec
[10986.732274]  apic_timer_interrupt+0xf/0x20
[10986.732289]  </IRQ>
[10986.732312] RIP: 0010:cpuidle_enter_state+0xfc/0x2c0
[10986.732328] Code: e8 e9 9f 9c ff 80 7c 24 03 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 9a 01 00 00 31 ff e8 7b ac a2 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 4c 29 f5 ba ff ff ff 7f 48 39 c5 7f
[10986.732344] RSP: 0018:ffffae8f81933e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[10986.732362] RAX: ffff8d258fd21cc0 RBX: ffff8d258fd2bf00 RCX: 000000000000001f
[10986.731647]  snd_hda_core snd_hwdep intel_gtt snd_pcm agpgart mei_me r8169 syscopyarea sysfillrect snd_timer sysimgblt pcc_cpufreq realtek snd lpc_ich libphy e1000e wmi soundcore mei fb_sys_fops evdev mac_hid nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci libata crc32c_intel scsi_mod xhci_pci xhci_hcd ehci_pci ehci_hcd
[10986.732378] RDX: 0000000000000000 RSI: 000000002c235370 RDI: 0000000000000000
[10986.732394] RBP: 000009fe0c399f97 R08: 000009fe0c399f97 R09: 00000000ffffffff
[10986.732409] R10: 0000000000002645 R11: ffff8d258fd20c48 R12: 0000000000000005
[10986.732429] R13: ffffffffa9eb88d8 R14: 000009fe0aeeae21 R15: 0000000000000000
[10986.732453]  ? cpuidle_enter_state+0xd7/0x2c0
[10986.731712] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.126-1-MANJARO #1
[10986.732473]  do_idle+0x1bf/0x240
[10986.732494]  cpu_startup_entry+0x6f/0x80
[10986.732513]  start_secondary+0x1a2/0x200
[10986.732533]  secondary_startup_64+0xa4/0xb0
[10986.732551] ---[ end trace 50a63959e2687b98 ]---
[10986.731725] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[10986.731745] RIP: 0010:dev_watchdog+0x212/0x220
[10986.731763] Code: 63 74 24 e0 eb 8c 4c 89 f7 c6 05 e6 fd b8 00 01 e8 d3 b1 fc ff 44 89 e9 4c 89 f6 48 c7 c7 c8 ca d1 a9 48 89 c2 e8 5d 83 90 ff <0f> 0b eb bd 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 c7 47
[10986.731783] RSP: 0018:ffff8d258fd03e70 EFLAGS: 00010282
[10986.731802] RAX: 0000000000000000 RBX: ffff8d258bd00c00 RCX: 000000000000083f
[10986.731820] RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
[10986.733084] systemd-journald[348]: Compressed data object 815 -> 682 using LZ4
[10986.731837] RBP: ffff8d258a06045c R08: ffff8d258fd165b8 R09: 0000000000000000
[10986.731857] R10: ffffffffa8d04510 R11: 0000000000000000 R12: ffff8d258a060480
[10986.731875] R13: 0000000000000000 R14: ffff8d258a060000 R15: ffff8d258bd00c80
[10986.733258] systemd-journald[348]: Compressed data object 534 -> 477 using LZ4
[10986.731895] FS:  0000000000000000(0000) GS:ffff8d258fd00000(0000) knlGS:0000000000000000
[10986.731913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10986.731931] CR2: 00007fcf213ea770 CR3: 00000001e640a006 CR4: 00000000001626e0
[10986.731948] Call Trace:
[10986.731968]  <IRQ>
[10986.731990]  ? qdisc_reset+0xd0/0xd0
[10986.732012]  call_timer_fn+0x2b/0x130
[10986.732036]  expire_timers+0x9c/0x100
[10986.732058]  run_timer_softirq+0x8f/0x180
[10986.732158]  ? __hrtimer_run_queues+0x138/0x2a0
[10986.732175]  ? sched_clock+0x5/0x10
[10986.732198]  ? sched_clock_cpu+0xc/0xb0
[10986.732218]  __do_softirq+0xee/0x2e1
[10986.732238]  irq_exit+0xa4/0xe0
[10986.732256]  smp_apic_timer_interrupt+0x78/0x140
[10986.732274]  apic_timer_interrupt+0xf/0x20
[10986.732289]  </IRQ>
[10986.732312] RIP: 0010:cpuidle_enter_state+0xfc/0x2c0
[10986.732328] Code: e8 e9 9f 9c ff 80 7c 24 03 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 9a 01 00 00 31 ff e8 7b ac a2 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 4c 29 f5 ba ff ff ff 7f 48 39 c5 7f
[10986.732344] RSP: 0018:ffffae8f81933e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[10986.736147] systemd-journald[348]: Sent WATCHDOG=1 notification.
[10986.732362] RAX: ffff8d258fd21cc0 RBX: ffff8d258fd2bf00 RCX: 000000000000001f
[10986.736204] e1000e 0000:00:19.0 ethlocal: Reset adapter unexpectedly
[10986.732378] RDX: 0000000000000000 RSI: 000000002c235370 RDI: 0000000000000000
[10986.732394] RBP: 000009fe0c399f97 R08: 000009fe0c399f97 R09: 00000000ffffffff
[10986.732409] R10: 0000000000002645 R11: ffff8d258fd20c48 R12: 0000000000000005
[10986.732429] R13: ffffffffa9eb88d8 R14: 000009fe0aeeae21 R15: 0000000000000000
[10986.736616] br0: port 1(ethlocal) entered disabled state
[10986.732453]  ? cpuidle_enter_state+0xd7/0x2c0
[10986.732473]  do_idle+0x1bf/0x240
[10986.732494]  cpu_startup_entry+0x6f/0x80
[10986.732513]  start_secondary+0x1a2/0x200
[10986.732533]  secondary_startup_64+0xa4/0xb0
[10986.732551] ---[ end trace 50a63959e2687b98 ]---
[10986.733084] systemd-journald[348]: Compressed data object 815 -> 682 using LZ4
[10986.733258] systemd-journald[348]: Compressed data object 534 -> 477 using LZ4
[10986.736147] systemd-journald[348]: Sent WATCHDOG=1 notification.
[10986.736204] e1000e 0000:00:19.0 ethlocal: Reset adapter unexpectedly
[10986.736616] br0: port 1(ethlocal) entered disabled state
[10990.503856] e1000e: ethlocal NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[10990.503914] br0: port 1(ethlocal) entered blocking state
[10990.503920] br0: port 1(ethlocal) entered forwarding state
[10990.503856] e1000e: ethlocal NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[10990.503914] br0: port 1(ethlocal) entered blocking state
[10990.503920] br0: port 1(ethlocal) entered forwarding state

[11011.263415] general protection fault: 0000 [#1] SMP PTI
[11011.263458] CPU: 0 PID: 1183 Comm: vhost-1178 Tainted: G        W         4.19.126-1-MANJARO #1
[11011.263468] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[11011.263492] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[11011.263502] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[11011.263512] RSP: 0018:ffffae8f837b3840 EFLAGS: 00010296
[11011.263522] RAX: 46e6c3fabfad8c89 RBX: ffff8d2589a47700 RCX: 0000000000000014
[11011.263531] RDX: 0000000000000000 RSI: ffff8d25725c8800 RDI: 0000000000000000
[11011.263571] RBP: ffff8d23f9639000 R08: ffff8d2589a47700 R09: 0000000000000001
[11011.263596] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000014
[11011.263610] R13: 0000000000000020 R14: ffffffffa9ef1d80 R15: ffff8d23f9639000
[11011.263627] FS:  0000000000000000(0000) GS:ffff8d258fc00000(0000) knlGS:0000000000000000
[11011.263642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11011.263654] CR2: 00007f6b95687fd8 CR3: 00000003df862004 CR4: 00000000001626f0
[11011.263666] Call Trace:
[11011.263694]  ? rt_cache_route+0xbe/0xd0
[11011.263415] general protection fault: 0000 [#1] SMP PTI
[11011.263720]  ? ipt_do_table+0x379/0x640 [ip_tables]
[11011.263747]  sk_filter_trim_cap+0xfe/0x1b0
[11011.263771]  tcp_v4_rcv+0xaf7/0xdf0
[11011.263791]  ip_local_deliver_finish+0x9c/0x1e0
[11011.263813]  ip_local_deliver+0x78/0x120
[11011.263458] CPU: 0 PID: 1183 Comm: vhost-1178 Tainted: G        W         4.19.126-1-MANJARO #1
[11011.263830]  ? ip_sublist_rcv_finish+0x60/0x60
[11011.263850]  ip_rcv+0x76/0x100
[11011.263873]  __netif_receive_skb_one_core+0x5b/0x80
[11011.263895]  netif_receive_skb_internal+0x4a/0xc0
[11011.263936]  br_pass_frame_up+0x108/0x1b0 [bridge]
[11011.263468] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[11011.263973]  ? br_port_flags_change+0x70/0x70 [bridge]
[11011.264007]  br_handle_frame_finish+0x181/0x450 [bridge]
[11011.264041]  br_handle_frame+0x175/0x360 [bridge]
[11011.263492] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[11011.264074]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[11011.264095]  __netif_receive_skb_core+0x4be/0xc60
[11011.264123]  ? tun_build_skb+0x2b1/0x520 [tun]
[11011.264148]  __netif_receive_skb_one_core+0x3d/0x80
[11011.263502] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[11011.264168]  netif_receive_skb_internal+0x4a/0xc0
[11011.264193]  tun_get_user+0x1021/0x12b0 [tun]
[11011.264221]  tun_sendmsg+0x55/0x70 [tun]
[11011.263512] RSP: 0018:ffffae8f837b3840 EFLAGS: 00010296
[11011.264247]  handle_tx_copy+0x142/0x280 [vhost_net]
[11011.264267]  handle_tx+0xa5/0xe0 [vhost_net]
[11011.264291]  vhost_worker+0xaa/0x100 [vhost]
[11011.264318]  kthread+0xfb/0x130
[11011.263522] RAX: 46e6c3fabfad8c89 RBX: ffff8d2589a47700 RCX: 0000000000000014
[11011.264342]  ? vhost_flush_work+0x10/0x10 [vhost]
[11011.264361]  ? kthread_park+0x80/0x80
[11011.264381]  ret_from_fork+0x35/0x40
[11011.263531] RDX: 0000000000000000 RSI: ffff8d25725c8800 RDI: 0000000000000000
[11011.264401] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat joydev mousedev input_leds intel_rapl snd_hda_codec_hdmi ofpart mei_wdt cmdlinepart intel_spi_platform intel_spi spi_nor mtd x86_pkg_temp_thermal iTCO_wdt intel_powerclamp iTCO_vendor_support eeepc_wmi asus_wmi coretemp sparse_keymap rfkill wmi_bmof kvm_intel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc i915 aesni_intel aes_x86_64 kvmgt crypto_simd cryptd vfio_mdev mdev glue_helper vfio_iommu_type1 vfio kvm intel_cstate intel_uncore snd_hda_codec_realtek irqbypass intel_rapl_perf snd_hda_codec_generic i2c_algo_bit drm_kms_helper pcspkr i2c_i801 snd_hda_intel drm snd_hda_codec
[11011.264502]  snd_hda_core snd_hwdep intel_gtt snd_pcm agpgart mei_me r8169 syscopyarea sysfillrect snd_timer sysimgblt pcc_cpufreq realtek snd lpc_ich libphy e1000e wmi soundcore mei fb_sys_fops evdev mac_hid nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci libata crc32c_intel scsi_mod xhci_pci xhci_hcd ehci_pci ehci_hcd
[11011.264593] ---[ end trace 50a63959e2687b99 ]---
[11011.264617] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[11011.264639] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[11011.264658] RSP: 0018:ffffae8f837b3840 EFLAGS: 00010296
[11011.264677] RAX: 46e6c3fabfad8c89 RBX: ffff8d2589a47700 RCX: 0000000000000014
[11011.264697] RDX: 0000000000000000 RSI: ffff8d25725c8800 RDI: 0000000000000000
[11011.264715] RBP: ffff8d23f9639000 R08: ffff8d2589a47700 R09: 0000000000000001
[11011.264734] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000014
[11011.264751] R13: 0000000000000020 R14: ffffffffa9ef1d80 R15: ffff8d23f9639000
[11011.264770] FS:  0000000000000000(0000) GS:ffff8d258fc00000(0000) knlGS:0000000000000000
[11011.264788] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11011.264807] CR2: 00007f6b95687fd8 CR3: 00000003df862004 CR4: 00000000001626f0
[11011.264831] Kernel panic - not syncing: Fatal exception in interrupt
[11011.264866] Kernel Offset: 0x27c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[11011.264879] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[11011.263571] RBP: ffff8d23f9639000 R08: ffff8d2589a47700 R09: 0000000000000001
[11011.263596] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000014
[11011.263610] R13: 0000000000000020 R14: ffffffffa9ef1d80 R15: ffff8d23f9639000
[11011.263627] FS:  0000000000000000(0000) GS:ffff8d258fc00000(0000) knlGS:0000000000000000
[11011.263642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11011.263654] CR2: 00007f6b95687fd8 CR3: 00000003df862004 CR4: 00000000001626f0
[11011.263666] Call Trace:
[11011.263694]  ? rt_cache_route+0xbe/0xd0
[11011.263720]  ? ipt_do_table+0x379/0x640 [ip_tables]
[11011.263747]  sk_filter_trim_cap+0xfe/0x1b0
[11011.263771]  tcp_v4_rcv+0xaf7/0xdf0
[11011.263791]  ip_local_deliver_finish+0x9c/0x1e0
[11011.263813]  ip_local_deliver+0x78/0x120
[11011.263830]  ? ip_sublist_rcv_finish+0x60/0x60
[11011.263850]  ip_rcv+0x76/0x100
[11011.263873]  __netif_receive_skb_one_core+0x5b/0x80
[11011.263895]  netif_receive_skb_internal+0x4a/0xc0
[11011.263936]  br_pass_frame_up+0x108/0x1b0 [bridge]
[11011.263973]  ? br_port_flags_change+0x70/0x70 [bridge]
[11011.264007]  br_handle_frame_finish+0x181/0x450 [bridge]
[11011.264041]  br_handle_frame+0x175/0x360 [bridge]
[11011.264074]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[11011.264095]  __netif_receive_skb_core+0x4be/0xc60
[11011.264123]  ? tun_build_skb+0x2b1/0x520 [tun]
[11011.264148]  __netif_receive_skb_one_core+0x3d/0x80
[11011.264168]  netif_receive_skb_internal+0x4a/0xc0
[11011.264193]  tun_get_user+0x1021/0x12b0 [tun]
[11011.264221]  tun_sendmsg+0x55/0x70 [tun]
[11011.264247]  handle_tx_copy+0x142/0x280 [vhost_net]
[11011.264267]  handle_tx+0xa5/0xe0 [vhost_net]
[11011.264291]  vhost_worker+0xaa/0x100 [vhost]
[11011.264318]  kthread+0xfb/0x130
[11011.264342]  ? vhost_flush_work+0x10/0x10 [vhost]
[11011.264361]  ? kthread_park+0x80/0x80
[11011.264381]  ret_from_fork+0x35/0x40
[11011.264401] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat joydev mousedev input_leds intel_rapl snd_hda_codec_hdmi ofpart mei_wdt cmdlinepart intel_spi_platform intel_spi spi_nor mtd x86_pkg_temp_thermal iTCO_wdt intel_powerclamp iTCO_vendor_support eeepc_wmi asus_wmi coretemp sparse_keymap rfkill wmi_bmof kvm_intel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc i915 aesni_intel aes_x86_64 kvmgt crypto_simd cryptd vfio_mdev mdev glue_helper vfio_iommu_type1 vfio kvm intel_cstate intel_uncore snd_hda_codec_realtek irqbypass intel_rapl_perf snd_hda_codec_generic i2c_algo_bit drm_kms_helper pcspkr i2c_i801 snd_hda_intel drm snd_hda_codec
[11011.264502]  snd_hda_core snd_hwdep intel_gtt snd_pcm agpgart mei_me r8169 syscopyarea sysfillrect snd_timer sysimgblt pcc_cpufreq realtek snd lpc_ich libphy e1000e wmi soundcore mei fb_sys_fops evdev mac_hid nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci libata crc32c_intel scsi_mod xhci_pci xhci_hcd ehci_pci ehci_hcd
[11011.264593] ---[ end trace 50a63959e2687b99 ]---
[11011.264617] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[11011.264639] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[11011.264658] RSP: 0018:ffffae8f837b3840 EFLAGS: 00010296
[11011.264677] RAX: 46e6c3fabfad8c89 RBX: ffff8d2589a47700 RCX: 0000000000000014
[11011.264697] RDX: 0000000000000000 RSI: ffff8d25725c8800 RDI: 0000000000000000
[11011.264715] RBP: ffff8d23f9639000 R08: ffff8d2589a47700 R09: 0000000000000001
[11011.264734] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000014
[11011.264751] R13: 0000000000000020 R14: ffffffffa9ef1d80 R15: ffff8d23f9639000
[11011.264770] FS:  0000000000000000(0000) GS:ffff8d258fc00000(0000) knlGS:0000000000000000
[11011.264788] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11011.264807] CR2: 00007f6b95687fd8 CR3: 00000003df862004 CR4: 00000000001626f0
[11011.264831] Kernel panic - not syncing: Fatal exception in interrupt
[11011.264866] Kernel Offset: 0x27c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[11011.264879] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

2. Kernel freeze (general protection fault)
[10185.416945] general protection fault: 0000 [#1] SMP PTI
[10185.416972] CPU: 0 PID: 38176 Comm: vhost-38171 Not tainted 4.19.126-1-MANJARO #1
[10185.416977] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[10185.416988] RIP: 0010:__cgroup_bpf_run_filter_skb+0xe3/0x1e0
[10185.416994] Code: c8 03 00 00 48 8b 00 48 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd 01 00 00 00 49 8b 46 08 48 85 c0 74 0f <48> 8b 00 48 83 c0 10 65 48 89 05 86 1c 06 45 f6 43 02 10 0f 85 9c
[10185.417000] RSP: 0018:ffffab4103fe7680 EFLAGS: 00010286
[10185.417006] RAX: f083ff937272e800 RBX: 00000cbe407f8b48 RCX: 0000000000000014
[10185.417010] RDX: 0000000000000000 RSI: ffff9583755c3800 RDI: 0000000000000000
[10185.417014] RBP: 0000000000000001 R08: ffff958386cf8880 R09: 0000000000000000
[10185.417019] R10: ffff95838f803800 R11: 0000000000000801 R12: 0000000000000014
[10185.417024] R13: ffff95832ec93530 R14: ffffffffbb5598e0 R15: ffff95832ec93500
[10185.417028] FS:  0000000000000000(0000) GS:ffff95838fc00000(0000) knlGS:0000000000000000
[10185.417033] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10185.417037] CR2: 00007f6a6000d2c0 CR3: 0000000354366004 CR4: 00000000001626f0
[10185.417041] Call Trace:
[10185.417054]  sk_filter_trim_cap+0xfe/0x1b0
[10185.417062]  tcp_v4_rcv+0xaf7/0xdf0
[10185.417068]  ip_local_deliver_finish+0x9c/0x1e0
[10185.417074]  ip_local_deliver+0x78/0x120
[10185.417078]  ? ip_sublist_rcv_finish+0x60/0x60
[10185.417085]  ip_sabotage_in+0x42/0x50 [br_netfilter]
[10185.417091]  nf_hook_slow+0x3f/0xb0
[10185.417096]  ip_rcv+0xdd/0x100
[10185.417100]  ? ip_sublist_rcv+0x2a0/0x2a0
[10185.417176]  __netif_receive_skb_one_core+0x5b/0x80
[10185.417182]  netif_receive_skb_internal+0x4a/0xc0
[10185.417193]  br_pass_frame_up+0x108/0x1b0 [bridge]
[10185.417203]  ? br_port_flags_change+0x70/0x70 [bridge]
[10185.417211]  br_handle_frame_finish+0x181/0x450 [bridge]
[10185.417219]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417225]  br_nf_hook_thresh+0xdf/0xf0 [br_netfilter]
[10185.417232]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417237]  br_nf_pre_routing_finish+0x148/0x380 [br_netfilter]
[10185.417245]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417250]  br_nf_pre_routing+0x383/0x470 [br_netfilter]
[10185.417254]  ? br_nf_forward_ip+0x490/0x490 [br_netfilter]
[10185.417260]  nf_hook_slow+0x3f/0xb0
[10185.417268]  br_handle_frame+0x217/0x360 [bridge]
[10185.417275]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417282]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[10185.417286]  __netif_receive_skb_core+0x4be/0xc60
[10185.417293]  ? tun_build_skb+0x2b1/0x520 [tun]
[10185.417299]  __netif_receive_skb_one_core+0x3d/0x80
[10185.417303]  netif_receive_skb_internal+0x4a/0xc0
[10185.417308]  tun_get_user+0x1021/0x12b0 [tun]
[10185.417315]  tun_sendmsg+0x55/0x70 [tun]
[10185.417320]  handle_tx_copy+0x142/0x280 [vhost_net]
[10185.417326]  handle_tx+0xa5/0xe0 [vhost_net]
[10185.417387]  vhost_worker+0xaa/0x100 [vhost]
[10185.417395]  kthread+0xfb/0x130
[10185.417406]  ? vhost_flush_work+0x10/0x10 [vhost]
[10185.417411]  ? kthread_park+0x80/0x80
[10185.417416]  ret_from_fork+0x35/0x40
[10185.417420] Modules linked in: ipt_MASQUERADE xt_recent xt_comment ipt_REJECT nf_reject_ipv4 xt_addrtype br_netfilter xt_physdev iptable_nat nf_nat_ipv4 xt_mark iptable_mangle xt_TCPMSS xt_hashlimit xt_tcpudp xt_CT iptable_raw xt_multiport xt_conntrack nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_nat nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter
[10185.416945] general protection fault: 0000 [#1] SMP PTI
[10185.417447]  ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev input_leds joydev snd_hda_codec_hdmi ofpart cmdlinepart intel_spi_platform intel_spi spi_nor intel_rapl iTCO_wdt mtd iTCO_vendor_support mei_wdt wmi_bmof eeepc_wmi asus_wmi sparse_keymap rfkill i915 x86_pkg_temp_thermal intel_powerclamp coretemp kvmgt kvm_intel vfio_mdev mdev vfio_iommu_type1 crct10dif_pclmul vfio crc32_pclmul ghash_clmulni_intel pcbc kvm aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate irqbypass snd_hda_codec_realtek i2c_algo_bit intel_uncore drm_kms_helper intel_rapl_perf snd_hda_codec_generic pcspkr drm snd_hda_intel snd_hda_codec intel_gtt r8169 snd_hda_core agpgart syscopyarea snd_hwdep snd_pcm lpc_ich realtek
[10185.417480]  sysfillrect sysimgblt snd_timer snd i2c_i801 mei_me libphy e1000e pcc_cpufreq fb_sys_fops mei soundcore wmi evdev mac_hid nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci libata crc32c_intel scsi_mod xhci_pci ehci_pci xhci_hcd ehci_hcd
[10185.417517] ---[ end trace 7f69e8167d4880de ]---
[10185.417526] RIP: 0010:__cgroup_bpf_run_filter_skb+0xe3/0x1e0
[10185.417532] Code: c8 03 00 00 48 8b 00 48 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd 01 00 00 00 49 8b 46 08 48 85 c0 74 0f <48> 8b 00 48 83 c0 10 65 48 89 05 86 1c 06 45 f6 43 02 10 0f 85 9c
[10185.417537] RSP: 0018:ffffab4103fe7680 EFLAGS: 00010286
[10185.417541] RAX: f083ff937272e800 RBX: 00000cbe407f8b48 RCX: 0000000000000014
[10185.417544] RDX: 0000000000000000 RSI: ffff9583755c3800 RDI: 0000000000000000
[10185.417548] RBP: 0000000000000001 R08: ffff958386cf8880 R09: 0000000000000000
[10185.417551] R10: ffff95838f803800 R11: 0000000000000801 R12: 0000000000000014
[10185.417555] R13: ffff95832ec93530 R14: ffffffffbb5598e0 R15: ffff95832ec93500
[10185.417558] FS:  0000000000000000(0000) GS:ffff95838fc00000(0000) knlGS:0000000000000000
[10185.417562] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10185.417565] CR2: 00007f6a6000d2c0 CR3: 0000000354366004 CR4: 00000000001626f0
[10185.417569] Kernel panic - not syncing: Fatal exception in interrupt
[10185.417580] Kernel Offset: 0x39e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[10185.417585] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[10185.416972] CPU: 0 PID: 38176 Comm: vhost-38171 Not tainted 4.19.126-1-MANJARO #1
[10185.416977] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[10185.416988] RIP: 0010:__cgroup_bpf_run_filter_skb+0xe3/0x1e0
[10185.416994] Code: c8 03 00 00 48 8b 00 48 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd 01 00 00 00 49 8b 46 08 48 85 c0 74 0f <48> 8b 00 48 83 c0 10 65 48 89 05 86 1c 06 45 f6 43 02 10 0f 85 9c
[10185.417000] RSP: 0018:ffffab4103fe7680 EFLAGS: 00010286
[10185.417006] RAX: f083ff937272e800 RBX: 00000cbe407f8b48 RCX: 0000000000000014
[10185.417010] RDX: 0000000000000000 RSI: ffff9583755c3800 RDI: 0000000000000000
[10185.417014] RBP: 0000000000000001 R08: ffff958386cf8880 R09: 0000000000000000
[10185.417019] R10: ffff95838f803800 R11: 0000000000000801 R12: 0000000000000014
[10185.417024] R13: ffff95832ec93530 R14: ffffffffbb5598e0 R15: ffff95832ec93500
[10185.417028] FS:  0000000000000000(0000) GS:ffff95838fc00000(0000) knlGS:0000000000000000
[10185.417033] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10185.417037] CR2: 00007f6a6000d2c0 CR3: 0000000354366004 CR4: 00000000001626f0
[10185.417041] Call Trace:
[10185.417054]  sk_filter_trim_cap+0xfe/0x1b0
[10185.417062]  tcp_v4_rcv+0xaf7/0xdf0
[10185.417068]  ip_local_deliver_finish+0x9c/0x1e0
[10185.417074]  ip_local_deliver+0x78/0x120
[10185.417078]  ? ip_sublist_rcv_finish+0x60/0x60
[10185.417085]  ip_sabotage_in+0x42/0x50 [br_netfilter]
[10185.417091]  nf_hook_slow+0x3f/0xb0
[10185.417096]  ip_rcv+0xdd/0x100
[10185.417100]  ? ip_sublist_rcv+0x2a0/0x2a0
[10185.417176]  __netif_receive_skb_one_core+0x5b/0x80
[10185.417182]  netif_receive_skb_internal+0x4a/0xc0
[10185.417193]  br_pass_frame_up+0x108/0x1b0 [bridge]
[10185.417203]  ? br_port_flags_change+0x70/0x70 [bridge]
[10185.417211]  br_handle_frame_finish+0x181/0x450 [bridge]
[10185.417219]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417225]  br_nf_hook_thresh+0xdf/0xf0 [br_netfilter]
[10185.417232]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417237]  br_nf_pre_routing_finish+0x148/0x380 [br_netfilter]
[10185.417245]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417250]  br_nf_pre_routing+0x383/0x470 [br_netfilter]
[10185.417254]  ? br_nf_forward_ip+0x490/0x490 [br_netfilter]
[10185.417260]  nf_hook_slow+0x3f/0xb0
[10185.417268]  br_handle_frame+0x217/0x360 [bridge]
[10185.417275]  ? br_pass_frame_up+0x1b0/0x1b0 [bridge]
[10185.417282]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[10185.417286]  __netif_receive_skb_core+0x4be/0xc60
[10185.417293]  ? tun_build_skb+0x2b1/0x520 [tun]
[10185.417299]  __netif_receive_skb_one_core+0x3d/0x80
[10185.417303]  netif_receive_skb_internal+0x4a/0xc0
[10185.417308]  tun_get_user+0x1021/0x12b0 [tun]
[10185.417315]  tun_sendmsg+0x55/0x70 [tun]
[10185.417320]  handle_tx_copy+0x142/0x280 [vhost_net]
[10185.417326]  handle_tx+0xa5/0xe0 [vhost_net]
[10185.417387]  vhost_worker+0xaa/0x100 [vhost]
[10185.417395]  kthread+0xfb/0x130
[10185.417406]  ? vhost_flush_work+0x10/0x10 [vhost]
[10185.417411]  ? kthread_park+0x80/0x80
[10185.417416]  ret_from_fork+0x35/0x40
[10185.417420] Modules linked in: ipt_MASQUERADE xt_recent xt_comment ipt_REJECT nf_reject_ipv4 xt_addrtype br_netfilter xt_physdev iptable_nat nf_nat_ipv4 xt_mark iptable_mangle xt_TCPMSS xt_hashlimit xt_tcpudp xt_CT iptable_raw xt_multiport xt_conntrack nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_nat nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter
[10185.417447]  ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev input_leds joydev snd_hda_codec_hdmi ofpart cmdlinepart intel_spi_platform intel_spi spi_nor intel_rapl iTCO_wdt mtd iTCO_vendor_support mei_wdt wmi_bmof eeepc_wmi asus_wmi sparse_keymap rfkill i915 x86_pkg_temp_thermal intel_powerclamp coretemp kvmgt kvm_intel vfio_mdev mdev vfio_iommu_type1 crct10dif_pclmul vfio crc32_pclmul ghash_clmulni_intel pcbc kvm aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate irqbypass snd_hda_codec_realtek i2c_algo_bit intel_uncore drm_kms_helper intel_rapl_perf snd_hda_codec_generic pcspkr drm snd_hda_intel snd_hda_codec intel_gtt r8169 snd_hda_core agpgart syscopyarea snd_hwdep snd_pcm lpc_ich realtek
[10185.417480]  sysfillrect sysimgblt snd_timer snd i2c_i801 mei_me libphy e1000e pcc_cpufreq fb_sys_fops mei soundcore wmi evdev mac_hid nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci libata crc32c_intel scsi_mod xhci_pci ehci_pci xhci_hcd ehci_hcd
[10185.417517] ---[ end trace 7f69e8167d4880de ]---
[10185.417526] RIP: 0010:__cgroup_bpf_run_filter_skb+0xe3/0x1e0
[10185.417532] Code: c8 03 00 00 48 8b 00 48 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd 01 00 00 00 49 8b 46 08 48 85 c0 74 0f <48> 8b 00 48 83 c0 10 65 48 89 05 86 1c 06 45 f6 43 02 10 0f 85 9c
[10185.417537] RSP: 0018:ffffab4103fe7680 EFLAGS: 00010286
[10185.417541] RAX: f083ff937272e800 RBX: 00000cbe407f8b48 RCX: 0000000000000014
[10185.417544] RDX: 0000000000000000 RSI: ffff9583755c3800 RDI: 0000000000000000
[10185.417548] RBP: 0000000000000001 R08: ffff958386cf8880 R09: 0000000000000000
[10185.417551] R10: ffff95838f803800 R11: 0000000000000801 R12: 0000000000000014
[10185.417555] R13: ffff95832ec93530 R14: ffffffffbb5598e0 R15: ffff95832ec93500
[10185.417558] FS:  0000000000000000(0000) GS:ffff95838fc00000(0000) knlGS:0000000000000000
[10185.417562] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10185.417565] CR2: 00007f6a6000d2c0 CR3: 0000000354366004 CR4: 00000000001626f0
[10185.417569] Kernel panic - not syncing: Fatal exception in interrupt
[10185.417580] Kernel Offset: 0x39e00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[10185.417585] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

3. Kernel Freeze (Paging Request)
[ 3719.735568] BUG: unable to handle kernel paging request at 0000740100706d71
[ 3719.735605] PGD 0 P4D 0
[ 3719.735621] Oops: 0000 [#1] SMP PTI
[ 3719.735635] CPU: 1 PID: 1203 Comm: vhost-1198 Not tainted 4.19.126-1-MANJARO #1
[ 3719.735643] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[ 3719.735663] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[ 3719.735679] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[ 3719.735690] RSP: 0018:ffffb49f03bc7840 EFLAGS: 00010296
[ 3719.735705] RAX: 0000740100706d61 RBX: ffff9d103572d500 RCX: 0000000000000014
[ 3719.735715] RDX: 0000000000000000 RSI: ffff9d103ccdb000 RDI: 0000000000000000
[ 3719.735728] RBP: ffff9d10398d0300 R08: ffff9d103572d500 R09: 0000000000000001
[ 3719.735738] R10: 000000000000f203 R11: 0000000000000000 R12: 0000000000000014
[ 3719.735747] R13: 0000000000000020 R14: ffffffff942f1d80 R15: ffff9d10398d0300
[ 3719.735758] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.735767] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.735775] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.735785] Call Trace:
[ 3719.735808]  ? ipt_do_table+0x379/0x640 [ip_tables]
[ 3719.735568] BUG: unable to handle kernel paging request at 0000740100706d71
[ 3719.735830]  sk_filter_trim_cap+0xfe/0x1b0
[ 3719.735854]  tcp_v4_rcv+0xaf7/0xdf0
[ 3719.735872]  ip_local_deliver_finish+0x9c/0x1e0
[ 3719.735890]  ip_local_deliver+0x78/0x120
[ 3719.735903]  ? ip_sublist_rcv_finish+0x60/0x60
[ 3719.735914]  ip_rcv+0x76/0x100
[ 3719.735928]  __netif_receive_skb_one_core+0x5b/0x80
[ 3719.735605] PGD 0 P4D 0
[ 3719.735943]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.735970]  br_pass_frame_up+0x108/0x1b0 [bridge]
[ 3719.735996]  ? br_port_flags_change+0x70/0x70 [bridge]
[ 3719.736019]  br_handle_frame_finish+0x181/0x450 [bridge]
[ 3719.735621] Oops: 0000 [#1] SMP PTI
[ 3719.736045]  br_handle_frame+0x175/0x360 [bridge]
[ 3719.736072]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[ 3719.736086]  __netif_receive_skb_core+0x4be/0xc60
[ 3719.736101]  ? tun_build_skb+0x2b1/0x520 [tun]
[ 3719.736116]  __netif_receive_skb_one_core+0x3d/0x80
[ 3719.736134]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.735635] CPU: 1 PID: 1203 Comm: vhost-1198 Not tainted 4.19.126-1-MANJARO #1
[ 3719.736148]  tun_get_user+0x1021/0x12b0 [tun]
[ 3719.736167]  tun_sendmsg+0x55/0x70 [tun]
[ 3719.736188]  handle_tx_copy+0x142/0x280 [vhost_net]
[ 3719.736208]  handle_tx+0xa5/0xe0 [vhost_net]
[ 3719.736229]  vhost_worker+0xaa/0x100 [vhost]
[ 3719.735643] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[ 3719.736245]  kthread+0xfb/0x130
[ 3719.736260]  ? vhost_flush_work+0x10/0x10 [vhost]
[ 3719.736274]  ? kthread_park+0x80/0x80
[ 3719.736294]  ret_from_fork+0x35/0x40
[ 3719.735663] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[ 3719.736313] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev joydev input_leds intel_rapl ofpart cmdlinepart intel_spi_platform snd_hda_codec_hdmi intel_spi iTCO_wdt spi_nor mtd iTCO_vendor_support mei_wdt x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul eeepc_wmi crc32_pclmul asus_wmi ghash_clmulni_intel wmi_bmof sparse_keymap rfkill pcbc aesni_intel i915 aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore kvmgt vfio_mdev mdev vfio_iommu_type1 vfio snd_hda_codec_realtek intel_rapl_perf kvm pcspkr snd_hda_codec_generic irqbypass i2c_algo_bit r8169 snd_hda_intel snd_hda_codec realtek snd_hda_core
[ 3719.736376]  drm_kms_helper libphy i2c_i801 e1000e intel_gtt lpc_ich snd_hwdep snd_pcm syscopyarea mei_me sysfillrect pcc_cpufreq snd_timer evdev sysimgblt mei fb_sys_fops mac_hid wmi snd soundcore nfsd nfs_acl lockd drm auth_rpcgss grace sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci crc32c_intel xhci_pci libata ehci_pci scsi_mod xhci_hcd ehci_hcd
[ 3719.736432] CR2: 0000740100706d71
[ 3719.736453] ---[ end trace e5ae8b80d90db0da ]---
[ 3719.736471] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[ 3719.736485] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[ 3719.736500] RSP: 0018:ffffb49f03bc7840 EFLAGS: 00010296
[ 3719.736518] RAX: 0000740100706d61 RBX: ffff9d103572d500 RCX: 0000000000000014
[ 3719.736528] RDX: 0000000000000000 RSI: ffff9d103ccdb000 RDI: 0000000000000000
[ 3719.736540] RBP: ffff9d10398d0300 R08: ffff9d103572d500 R09: 0000000000000001
[ 3719.736556] R10: 000000000000f203 R11: 0000000000000000 R12: 0000000000000014
[ 3719.736571] R13: 0000000000000020 R14: ffffffff942f1d80 R15: ffff9d10398d0300
[ 3719.736582] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.736592] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.736600] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.736611] Kernel panic - not syncing: Fatal exception in interrupt
[ 3719.736644] Kernel Offset: 0x12000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 3719.736657] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[ 3719.736686] WARNING: CPU: 1 PID: 1203 at kernel/sched/core.c:1164 set_task_cpu+0x173/0x180
[ 3719.736695] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev joydev input_leds intel_rapl ofpart cmdlinepart intel_spi_platform snd_hda_codec_hdmi intel_spi iTCO_wdt spi_nor mtd iTCO_vendor_support mei_wdt x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul eeepc_wmi crc32_pclmul asus_wmi ghash_clmulni_intel wmi_bmof sparse_keymap rfkill pcbc aesni_intel i915 aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore kvmgt vfio_mdev mdev vfio_iommu_type1 vfio snd_hda_codec_realtek intel_rapl_perf kvm pcspkr snd_hda_codec_generic irqbypass i2c_algo_bit r8169 snd_hda_intel snd_hda_codec realtek snd_hda_core
[ 3719.736743]  drm_kms_helper libphy i2c_i801 e1000e intel_gtt lpc_ich snd_hwdep snd_pcm syscopyarea mei_me sysfillrect pcc_cpufreq snd_timer evdev sysimgblt mei fb_sys_fops mac_hid wmi snd soundcore nfsd nfs_acl lockd drm auth_rpcgss grace sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci crc32c_intel xhci_pci libata ehci_pci scsi_mod xhci_hcd ehci_hcd
[ 3719.736784] CPU: 1 PID: 1203 Comm: vhost-1198 Tainted: G      D           4.19.126-1-MANJARO #1
[ 3719.736791] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[ 3719.736803] RIP: 0010:set_task_cpu+0x173/0x180
[ 3719.736813] Code: e9 54 ff ff ff 0f 0b e9 dc fe ff ff 8b 43 60 85 c0 0f 84 e8 fe ff ff 8b 43 60 83 f8 02 0f 84 dc fe ff ff 0f 0b e9 d5 fe ff ff <0f> 0b e9 df fe ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 41 55 49 89
[ 3719.736823] RSP: 0018:ffff9d104fc83cb8 EFLAGS: 00010006
[ 3719.736832] RAX: 0000000000000200 RBX: ffff9d10486bd940 RCX: ffff9d104fd80000
[ 3719.736840] RDX: ffff9d10486bd940 RSI: 0000000000000003 RDI: ffff9d10486bd940
[ 3719.736848] RBP: ffff9d10486bd940 R08: 0000000000000003 R09: 0000000000000001
[ 3719.736857] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000003
[ 3719.735679] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[ 3719.736865] R13: 0000000000000003 R14: 0000000000000046 R15: ffff9d10486be084
[ 3719.736875] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.736885] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.736893] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.736900] Call Trace:
[ 3719.736915]  <IRQ>
[ 3719.737005]  try_to_wake_up+0x185/0x4c0
[ 3719.737019]  __wake_up_common+0x7a/0x140
[ 3719.737033]  ep_poll_callback+0x152/0x270
[ 3719.737046]  __wake_up_common+0x7a/0x140
[ 3719.737058]  __wake_up_common_lock+0x7f/0xc0
[ 3719.737072]  irq_work_run_list+0x4f/0x70
[ 3719.737086]  update_process_times+0x84/0x90
[ 3719.737097]  tick_sched_handle+0x22/0x60
[ 3719.737108]  tick_sched_timer+0x51/0xb0
[ 3719.735690] RSP: 0018:ffffb49f03bc7840 EFLAGS: 00010296
[ 3719.737119]  ? tick_do_update_jiffies64.part.0+0xd0/0xd0
[ 3719.737132]  __hrtimer_run_queues+0x128/0x2a0
[ 3719.737145]  hrtimer_interrupt+0x10e/0x280
[ 3719.737158]  smp_apic_timer_interrupt+0x6e/0x140
[ 3719.737169]  apic_timer_interrupt+0xf/0x20
[ 3719.737179]  </IRQ>
[ 3719.737190] RIP: 0010:panic+0x204/0x24a
[ 3719.737199] Code: eb a6 83 3d 15 4f 80 01 00 74 05 e8 0e 54 02 00 48 c7 c6 00 b1 88 94 48 c7 c7 18 f4 07 94 e8 89 77 06 00 fb 66 0f 1f 44 00 00 <31> db e8 1d b2 0c 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 bd 4e 80
[ 3719.737208] RSP: 0018:ffffb49f03bc76f8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3719.737217] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 3719.737228] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff9d104fc965b0
[ 3719.737237] RBP: ffffb49f03bc7768 R08: 0000000000000610 R09: 0000000000000000
[ 3719.737245] R10: ffffffff93104510 R11: ffffe42f4c8dd600 R12: 0000000000000000
[ 3719.737253] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 3719.735705] RAX: 0000740100706d61 RBX: ffff9d103572d500 RCX: 0000000000000014
[ 3719.737264]  ? swiotlb_tbl_unmap_single+0x110/0x110
[ 3719.737349]  ? panic+0x1fd/0x24a
[ 3719.737363]  oops_end.cold+0xc/0x18
[ 3719.737373]  page_fault+0x1e/0x30
[ 3719.737386] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[ 3719.737394] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[ 3719.737402] RSP: 0018:ffffb49f03bc7840 EFLAGS: 00010296
[ 3719.737411] RAX: 0000740100706d61 RBX: ffff9d103572d500 RCX: 0000000000000014
[ 3719.737418] RDX: 0000000000000000 RSI: ffff9d103ccdb000 RDI: 0000000000000000
[ 3719.737426] RBP: ffff9d10398d0300 R08: ffff9d103572d500 R09: 0000000000000001
[ 3719.735715] RDX: 0000000000000000 RSI: ffff9d103ccdb000 RDI: 0000000000000000
[ 3719.737434] R10: 000000000000f203 R11: 0000000000000000 R12: 0000000000000014
[ 3719.737442] R13: 0000000000000020 R14: ffffffff942f1d80 R15: ffff9d10398d0300
[ 3719.737458]  ? ipt_do_table+0x379/0x640 [ip_tables]
[ 3719.737473]  sk_filter_trim_cap+0xfe/0x1b0
[ 3719.737487]  tcp_v4_rcv+0xaf7/0xdf0
[ 3719.737499]  ip_local_deliver_finish+0x9c/0x1e0
[ 3719.737512]  ip_local_deliver+0x78/0x120
[ 3719.735728] RBP: ffff9d10398d0300 R08: ffff9d103572d500 R09: 0000000000000001
[ 3719.737522]  ? ip_sublist_rcv_finish+0x60/0x60
[ 3719.737532]  ip_rcv+0x76/0x100
[ 3719.737543]  __netif_receive_skb_one_core+0x5b/0x80
[ 3719.737553]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.737574]  br_pass_frame_up+0x108/0x1b0 [bridge]
[ 3719.737592]  ? br_port_flags_change+0x70/0x70 [bridge]
[ 3719.737610]  br_handle_frame_finish+0x181/0x450 [bridge]
[ 3719.735738] R10: 000000000000f203 R11: 0000000000000000 R12: 0000000000000014
[ 3719.737703]  br_handle_frame+0x175/0x360 [bridge]
[ 3719.735747] R13: 0000000000000020 R14: ffffffff942f1d80 R15: ffff9d10398d0300
[ 3719.737721]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[ 3719.737732]  __netif_receive_skb_core+0x4be/0xc60
[ 3719.737745]  ? tun_build_skb+0x2b1/0x520 [tun]
[ 3719.737755]  __netif_receive_skb_one_core+0x3d/0x80
[ 3719.737765]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.737777]  tun_get_user+0x1021/0x12b0 [tun]
[ 3719.737791]  tun_sendmsg+0x55/0x70 [tun]
[ 3719.735758] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.737802]  handle_tx_copy+0x142/0x280 [vhost_net]
[ 3719.737815]  handle_tx+0xa5/0xe0 [vhost_net]
[ 3719.737828]  vhost_worker+0xaa/0x100 [vhost]
[ 3719.737841]  kthread+0xfb/0x130
[ 3719.737853]  ? vhost_flush_work+0x10/0x10 [vhost]
[ 3719.737864]  ? kthread_park+0x80/0x80
[ 3719.737874]  ret_from_fork+0x35/0x40
[ 3719.737884] ---[ end trace e5ae8b80d90db0db ]---
[ 3719.737897] ------------[ cut here ]------------
[ 3719.737905] sched: Unexpected reschedule of offline CPU#3!
[ 3719.735767] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.737919] WARNING: CPU: 1 PID: 1203 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x34/0x40
[ 3719.737929] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev joydev input_leds intel_rapl ofpart cmdlinepart intel_spi_platform snd_hda_codec_hdmi intel_spi iTCO_wdt spi_nor mtd iTCO_vendor_support mei_wdt x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul eeepc_wmi crc32_pclmul asus_wmi ghash_clmulni_intel wmi_bmof sparse_keymap rfkill pcbc aesni_intel i915 aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore kvmgt vfio_mdev mdev vfio_iommu_type1 vfio snd_hda_codec_realtek intel_rapl_perf kvm pcspkr snd_hda_codec_generic irqbypass i2c_algo_bit r8169 snd_hda_intel snd_hda_codec realtek snd_hda_core
[ 3719.735775] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.737974]  drm_kms_helper libphy i2c_i801 e1000e intel_gtt lpc_ich snd_hwdep snd_pcm syscopyarea mei_me sysfillrect pcc_cpufreq snd_timer evdev sysimgblt mei fb_sys_fops mac_hid wmi snd soundcore nfsd nfs_acl lockd drm auth_rpcgss grace sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci crc32c_intel xhci_pci libata ehci_pci scsi_mod xhci_hcd ehci_hcd
[ 3719.738014] CPU: 1 PID: 1203 Comm: vhost-1198 Tainted: G      D W         4.19.126-1-MANJARO #1
[ 3719.738024] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[ 3719.735785] Call Trace:
[ 3719.738034] RIP: 0010:native_smp_send_reschedule+0x34/0x40
[ 3719.738115] Code: 05 c1 89 2f 01 73 15 48 8b 05 78 71 0d 01 be fd 00 00 00 48 8b 40 30 e9 ca eb ba 00 89 fe 48 c7 c7 30 58 07 94 e8 fb 1a 03 00 <0f> 0b c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 05 6d 7a 83
[ 3719.738124] RSP: 0018:ffff9d104fc83ca0 EFLAGS: 00010086
[ 3719.738132] RAX: 0000000000000000 RBX: ffff9d104fda1cc0 RCX: 0000000000000006
[ 3719.738140] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff9d104fc965b0
[ 3719.738147] RBP: ffff9d10486bd940 R08: 0000000000000663 R09: 0000000000000000
[ 3719.738156] R10: ffffffff93104510 R11: 0000000000000000 R12: ffff9d10486bd940
[ 3719.738163] R13: ffff9d104fc83cf0 R14: 0000000000000046 R15: ffff9d10486be084
[ 3719.738172] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.738180] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.738188] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.738195] Call Trace:
[ 3719.738203]  <IRQ>
[ 3719.735808]  ? ipt_do_table+0x379/0x640 [ip_tables]
[ 3719.738214]  check_preempt_curr+0x4d/0x90
[ 3719.738225]  ttwu_do_wakeup+0x19/0x150
[ 3719.738236]  try_to_wake_up+0x210/0x4c0
[ 3719.738250]  __wake_up_common+0x7a/0x140
[ 3719.738265]  ep_poll_callback+0x152/0x270
[ 3719.738280]  __wake_up_common+0x7a/0x140
[ 3719.738292]  __wake_up_common_lock+0x7f/0xc0
[ 3719.735830]  sk_filter_trim_cap+0xfe/0x1b0
[ 3719.738304]  irq_work_run_list+0x4f/0x70
[ 3719.738316]  update_process_times+0x84/0x90
[ 3719.738329]  tick_sched_handle+0x22/0x60
[ 3719.738340]  tick_sched_timer+0x51/0xb0
[ 3719.735854]  tcp_v4_rcv+0xaf7/0xdf0
[ 3719.738424]  ? tick_do_update_jiffies64.part.0+0xd0/0xd0
[ 3719.738435]  __hrtimer_run_queues+0x128/0x2a0
[ 3719.738448]  hrtimer_interrupt+0x10e/0x280
[ 3719.738460]  smp_apic_timer_interrupt+0x6e/0x140
[ 3719.738470]  apic_timer_interrupt+0xf/0x20
[ 3719.738478]  </IRQ>
[ 3719.738488] RIP: 0010:panic+0x204/0x24a
[ 3719.738496] Code: eb a6 83 3d 15 4f 80 01 00 74 05 e8 0e 54 02 00 48 c7 c6 00 b1 88 94 48 c7 c7 18 f4 07 94 e8 89 77 06 00 fb 66 0f 1f 44 00 00 <31> db e8 1d b2 0c 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 bd 4e 80
[ 3719.735872]  ip_local_deliver_finish+0x9c/0x1e0
[ 3719.738505] RSP: 0018:ffffb49f03bc76f8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3719.738514] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 3719.738522] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff9d104fc965b0
[ 3719.738531] RBP: ffffb49f03bc7768 R08: 0000000000000610 R09: 0000000000000000
[ 3719.738539] R10: ffffffff93104510 R11: ffffe42f4c8dd600 R12: 0000000000000000
[ 3719.738547] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 3719.738558]  ? swiotlb_tbl_unmap_single+0x110/0x110
[ 3719.738571]  ? panic+0x1fd/0x24a
[ 3719.738584]  oops_end.cold+0xc/0x18
[ 3719.738593]  page_fault+0x1e/0x30
[ 3719.735890]  ip_local_deliver+0x78/0x120
[ 3719.738605] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[ 3719.738616] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[ 3719.738625] RSP: 0018:ffffb49f03bc7840 EFLAGS: 00010296
[ 3719.738633] RAX: 0000740100706d61 RBX: ffff9d103572d500 RCX: 0000000000000014
[ 3719.738640] RDX: 0000000000000000 RSI: ffff9d103ccdb000 RDI: 0000000000000000
[ 3719.735903]  ? ip_sublist_rcv_finish+0x60/0x60
[ 3719.738648] RBP: ffff9d10398d0300 R08: ffff9d103572d500 R09: 0000000000000001
[ 3719.738730] R10: 000000000000f203 R11: 0000000000000000 R12: 0000000000000014
[ 3719.738738] R13: 0000000000000020 R14: ffffffff942f1d80 R15: ffff9d10398d0300
[ 3719.738751]  ? ipt_do_table+0x379/0x640 [ip_tables]
[ 3719.738765]  sk_filter_trim_cap+0xfe/0x1b0
[ 3719.738778]  tcp_v4_rcv+0xaf7/0xdf0
[ 3719.738789]  ip_local_deliver_finish+0x9c/0x1e0
[ 3719.738799]  ip_local_deliver+0x78/0x120
[ 3719.735914]  ip_rcv+0x76/0x100
[ 3719.738809]  ? ip_sublist_rcv_finish+0x60/0x60
[ 3719.738819]  ip_rcv+0x76/0x100
[ 3719.738830]  __netif_receive_skb_one_core+0x5b/0x80
[ 3719.738841]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.738861]  br_pass_frame_up+0x108/0x1b0 [bridge]
[ 3719.738880]  ? br_port_flags_change+0x70/0x70 [bridge]
[ 3719.738898]  br_handle_frame_finish+0x181/0x450 [bridge]
[ 3719.735928]  __netif_receive_skb_one_core+0x5b/0x80
[ 3719.738916]  br_handle_frame+0x175/0x360 [bridge]
[ 3719.738934]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[ 3719.738946]  __netif_receive_skb_core+0x4be/0xc60
[ 3719.738959]  ? tun_build_skb+0x2b1/0x520 [tun]
[ 3719.738972]  __netif_receive_skb_one_core+0x3d/0x80
[ 3719.738981]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.738993]  tun_get_user+0x1021/0x12b0 [tun]
[ 3719.739006]  tun_sendmsg+0x55/0x70 [tun]
[ 3719.735943]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.739020]  handle_tx_copy+0x142/0x280 [vhost_net]
[ 3719.739104]  handle_tx+0xa5/0xe0 [vhost_net]
[ 3719.739116]  vhost_worker+0xaa/0x100 [vhost]
[ 3719.735970]  br_pass_frame_up+0x108/0x1b0 [bridge]
[ 3719.739128]  kthread+0xfb/0x130
[ 3719.739140]  ? vhost_flush_work+0x10/0x10 [vhost]
[ 3719.739149]  ? kthread_park+0x80/0x80
[ 3719.739158]  ret_from_fork+0x35/0x40
[ 3719.739168] ---[ end trace e5ae8b80d90db0dc ]---
[ 3719.735996]  ? br_port_flags_change+0x70/0x70 [bridge]
[ 3719.736019]  br_handle_frame_finish+0x181/0x450 [bridge]
[ 3719.736045]  br_handle_frame+0x175/0x360 [bridge]
[ 3719.736072]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[ 3719.736086]  __netif_receive_skb_core+0x4be/0xc60
[ 3719.736101]  ? tun_build_skb+0x2b1/0x520 [tun]
[ 3719.736116]  __netif_receive_skb_one_core+0x3d/0x80
[ 3719.736134]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.736148]  tun_get_user+0x1021/0x12b0 [tun]
[ 3719.736167]  tun_sendmsg+0x55/0x70 [tun]
[ 3719.736188]  handle_tx_copy+0x142/0x280 [vhost_net]
[ 3719.736208]  handle_tx+0xa5/0xe0 [vhost_net]
[ 3719.736229]  vhost_worker+0xaa/0x100 [vhost]
[ 3719.736245]  kthread+0xfb/0x130
[ 3719.736260]  ? vhost_flush_work+0x10/0x10 [vhost]
[ 3719.736274]  ? kthread_park+0x80/0x80
[ 3719.736294]  ret_from_fork+0x35/0x40
[ 3719.736313] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev joydev input_leds intel_rapl ofpart cmdlinepart intel_spi_platform snd_hda_codec_hdmi intel_spi iTCO_wdt spi_nor mtd iTCO_vendor_support mei_wdt x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul eeepc_wmi crc32_pclmul asus_wmi ghash_clmulni_intel wmi_bmof sparse_keymap rfkill pcbc aesni_intel i915 aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore kvmgt vfio_mdev mdev vfio_iommu_type1 vfio snd_hda_codec_realtek intel_rapl_perf kvm pcspkr snd_hda_codec_generic irqbypass i2c_algo_bit r8169 snd_hda_intel snd_hda_codec realtek snd_hda_core
[ 3719.736376]  drm_kms_helper libphy i2c_i801 e1000e intel_gtt lpc_ich snd_hwdep snd_pcm syscopyarea mei_me sysfillrect pcc_cpufreq snd_timer evdev sysimgblt mei fb_sys_fops mac_hid wmi snd soundcore nfsd nfs_acl lockd drm auth_rpcgss grace sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci crc32c_intel xhci_pci libata ehci_pci scsi_mod xhci_hcd ehci_hcd
[ 3719.736432] CR2: 0000740100706d71
[ 3719.736453] ---[ end trace e5ae8b80d90db0da ]---
[ 3719.736471] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[ 3719.736485] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[ 3719.736500] RSP: 0018:ffffb49f03bc7840 EFLAGS: 00010296
[ 3719.736518] RAX: 0000740100706d61 RBX: ffff9d103572d500 RCX: 0000000000000014
[ 3719.736528] RDX: 0000000000000000 RSI: ffff9d103ccdb000 RDI: 0000000000000000
[ 3719.736540] RBP: ffff9d10398d0300 R08: ffff9d103572d500 R09: 0000000000000001
[ 3719.736556] R10: 000000000000f203 R11: 0000000000000000 R12: 0000000000000014
[ 3719.736571] R13: 0000000000000020 R14: ffffffff942f1d80 R15: ffff9d10398d0300
[ 3719.736582] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.736592] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.736600] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.736611] Kernel panic - not syncing: Fatal exception in interrupt
[ 3719.736644] Kernel Offset: 0x12000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 3719.736657] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[ 3719.736686] WARNING: CPU: 1 PID: 1203 at kernel/sched/core.c:1164 set_task_cpu+0x173/0x180
[ 3719.736695] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev joydev input_leds intel_rapl ofpart cmdlinepart intel_spi_platform snd_hda_codec_hdmi intel_spi iTCO_wdt spi_nor mtd iTCO_vendor_support mei_wdt x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul eeepc_wmi crc32_pclmul asus_wmi ghash_clmulni_intel wmi_bmof sparse_keymap rfkill pcbc aesni_intel i915 aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore kvmgt vfio_mdev mdev vfio_iommu_type1 vfio snd_hda_codec_realtek intel_rapl_perf kvm pcspkr snd_hda_codec_generic irqbypass i2c_algo_bit r8169 snd_hda_intel snd_hda_codec realtek snd_hda_core
[ 3719.736743]  drm_kms_helper libphy i2c_i801 e1000e intel_gtt lpc_ich snd_hwdep snd_pcm syscopyarea mei_me sysfillrect pcc_cpufreq snd_timer evdev sysimgblt mei fb_sys_fops mac_hid wmi snd soundcore nfsd nfs_acl lockd drm auth_rpcgss grace sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci crc32c_intel xhci_pci libata ehci_pci scsi_mod xhci_hcd ehci_hcd
[ 3719.736784] CPU: 1 PID: 1203 Comm: vhost-1198 Tainted: G      D           4.19.126-1-MANJARO #1
[ 3719.736791] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[ 3719.736803] RIP: 0010:set_task_cpu+0x173/0x180
[ 3719.736813] Code: e9 54 ff ff ff 0f 0b e9 dc fe ff ff 8b 43 60 85 c0 0f 84 e8 fe ff ff 8b 43 60 83 f8 02 0f 84 dc fe ff ff 0f 0b e9 d5 fe ff ff <0f> 0b e9 df fe ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 41 55 49 89
[ 3719.736823] RSP: 0018:ffff9d104fc83cb8 EFLAGS: 00010006
[ 3719.736832] RAX: 0000000000000200 RBX: ffff9d10486bd940 RCX: ffff9d104fd80000
[ 3719.736840] RDX: ffff9d10486bd940 RSI: 0000000000000003 RDI: ffff9d10486bd940
[ 3719.736848] RBP: ffff9d10486bd940 R08: 0000000000000003 R09: 0000000000000001
[ 3719.736857] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000003
[ 3719.736865] R13: 0000000000000003 R14: 0000000000000046 R15: ffff9d10486be084
[ 3719.736875] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.736885] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.736893] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.736900] Call Trace:
[ 3719.736915]  <IRQ>
[ 3719.737005]  try_to_wake_up+0x185/0x4c0
[ 3719.737019]  __wake_up_common+0x7a/0x140
[ 3719.737033]  ep_poll_callback+0x152/0x270
[ 3719.737046]  __wake_up_common+0x7a/0x140
[ 3719.737058]  __wake_up_common_lock+0x7f/0xc0
[ 3719.737072]  irq_work_run_list+0x4f/0x70
[ 3719.737086]  update_process_times+0x84/0x90
[ 3719.737097]  tick_sched_handle+0x22/0x60
[ 3719.737108]  tick_sched_timer+0x51/0xb0
[ 3719.737119]  ? tick_do_update_jiffies64.part.0+0xd0/0xd0
[ 3719.737132]  __hrtimer_run_queues+0x128/0x2a0
[ 3719.737145]  hrtimer_interrupt+0x10e/0x280
[ 3719.737158]  smp_apic_timer_interrupt+0x6e/0x140
[ 3719.737169]  apic_timer_interrupt+0xf/0x20
[ 3719.737179]  </IRQ>
[ 3719.737190] RIP: 0010:panic+0x204/0x24a
[ 3719.737199] Code: eb a6 83 3d 15 4f 80 01 00 74 05 e8 0e 54 02 00 48 c7 c6 00 b1 88 94 48 c7 c7 18 f4 07 94 e8 89 77 06 00 fb 66 0f 1f 44 00 00 <31> db e8 1d b2 0c 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 bd 4e 80
[ 3719.737208] RSP: 0018:ffffb49f03bc76f8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[ 3719.737217] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 3719.737228] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff9d104fc965b0
[ 3719.737237] RBP: ffffb49f03bc7768 R08: 0000000000000610 R09: 0000000000000000
[ 3719.737245] R10: ffffffff93104510 R11: ffffe42f4c8dd600 R12: 0000000000000000
[ 3719.737253] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 3719.737264]  ? swiotlb_tbl_unmap_single+0x110/0x110
[ 3719.737349]  ? panic+0x1fd/0x24a
[ 3719.737363]  oops_end.cold+0xc/0x18
[ 3719.737373]  page_fault+0x1e/0x30
[ 3719.737386] RIP: 0010:__cgroup_bpf_run_filter_skb+0xc0/0x1e0
[ 3719.737394] Code: 48 89 3c 24 44 89 e1 45 01 a7 80 00 00 00 48 29 c8 48 89 4c 24 08 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 c8 03 00 00 48 8b 00 <48> 8b 58 10 4c 8d 70 10 48 85 db 0f 84 fe 00 00 00 4d 8d 6f 30 bd
[ 3719.737402] RSP: 0018:ffffb49f03bc7840 EFLAGS: 00010296
[ 3719.737411] RAX: 0000740100706d61 RBX: ffff9d103572d500 RCX: 0000000000000014
[ 3719.737418] RDX: 0000000000000000 RSI: ffff9d103ccdb000 RDI: 0000000000000000
[ 3719.737426] RBP: ffff9d10398d0300 R08: ffff9d103572d500 R09: 0000000000000001
[ 3719.737434] R10: 000000000000f203 R11: 0000000000000000 R12: 0000000000000014
[ 3719.737442] R13: 0000000000000020 R14: ffffffff942f1d80 R15: ffff9d10398d0300
[ 3719.737458]  ? ipt_do_table+0x379/0x640 [ip_tables]
[ 3719.737473]  sk_filter_trim_cap+0xfe/0x1b0
[ 3719.737487]  tcp_v4_rcv+0xaf7/0xdf0
[ 3719.737499]  ip_local_deliver_finish+0x9c/0x1e0
[ 3719.737512]  ip_local_deliver+0x78/0x120
[ 3719.737522]  ? ip_sublist_rcv_finish+0x60/0x60
[ 3719.737532]  ip_rcv+0x76/0x100
[ 3719.737543]  __netif_receive_skb_one_core+0x5b/0x80
[ 3719.737553]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.737574]  br_pass_frame_up+0x108/0x1b0 [bridge]
[ 3719.737592]  ? br_port_flags_change+0x70/0x70 [bridge]
[ 3719.737610]  br_handle_frame_finish+0x181/0x450 [bridge]
[ 3719.737703]  br_handle_frame+0x175/0x360 [bridge]
[ 3719.737721]  ? br_handle_local_finish+0xa0/0xa0 [bridge]
[ 3719.737732]  __netif_receive_skb_core+0x4be/0xc60
[ 3719.737745]  ? tun_build_skb+0x2b1/0x520 [tun]
[ 3719.737755]  __netif_receive_skb_one_core+0x3d/0x80
[ 3719.737765]  netif_receive_skb_internal+0x4a/0xc0
[ 3719.737777]  tun_get_user+0x1021/0x12b0 [tun]
[ 3719.737791]  tun_sendmsg+0x55/0x70 [tun]
[ 3719.737802]  handle_tx_copy+0x142/0x280 [vhost_net]
[ 3719.737815]  handle_tx+0xa5/0xe0 [vhost_net]
[ 3719.737828]  vhost_worker+0xaa/0x100 [vhost]
[ 3719.737841]  kthread+0xfb/0x130
[ 3719.737853]  ? vhost_flush_work+0x10/0x10 [vhost]
[ 3719.737864]  ? kthread_park+0x80/0x80
[ 3719.737874]  ret_from_fork+0x35/0x40
[ 3719.737884] ---[ end trace e5ae8b80d90db0db ]---
[ 3719.737897] ------------[ cut here ]------------
[ 3719.737905] sched: Unexpected reschedule of offline CPU#3!
[ 3719.737919] WARNING: CPU: 1 PID: 1203 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x34/0x40
[ 3719.737929] Modules linked in: rpcsec_gss_krb5 vhost_net vhost tap tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse netconsole bridge stp llc nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat mousedev joydev input_leds intel_rapl ofpart cmdlinepart intel_spi_platform snd_hda_codec_hdmi intel_spi iTCO_wdt spi_nor mtd iTCO_vendor_support mei_wdt x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul eeepc_wmi crc32_pclmul asus_wmi ghash_clmulni_intel wmi_bmof sparse_keymap rfkill pcbc aesni_intel i915 aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore kvmgt vfio_mdev mdev vfio_iommu_type1 vfio snd_hda_codec_realtek intel_rapl_perf kvm pcspkr snd_hda_codec_generic irqbypass i2c_algo_bit r8169 snd_hda_intel snd_hda_codec realtek snd_hda_core
[ 3719.737974]  drm_kms_helper libphy i2c_i801 e1000e intel_gtt lpc_ich snd_hwdep snd_pcm syscopyarea mei_me sysfillrect pcc_cpufreq snd_timer evdev sysimgblt mei fb_sys_fops mac_hid wmi snd soundcore nfsd nfs_acl lockd drm auth_rpcgss grace sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio hid_logitech_dj hid_generic usbhid hid dm_mod sr_mod cdrom sd_mod ahci libahci crc32c_intel xhci_pci libata ehci_pci scsi_mod xhci_hcd ehci_hcd
[ 3719.738014] CPU: 1 PID: 1203 Comm: vhost-1198 Tainted: G      D W         4.19.126-1-MANJARO #1
[ 3719.738024] Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
[ 3719.738034] RIP: 0010:native_smp_send_reschedule+0x34/0x40
[ 3719.738115] Code: 05 c1 89 2f 01 73 15 48 8b 05 78 71 0d 01 be fd 00 00 00 48 8b 40 30 e9 ca eb ba 00 89 fe 48 c7 c7 30 58 07 94 e8 fb 1a 03 00 <0f> 0b c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 05 6d 7a 83
[ 3719.738124] RSP: 0018:ffff9d104fc83ca0 EFLAGS: 00010086
[ 3719.738132] RAX: 0000000000000000 RBX: ffff9d104fda1cc0 RCX: 0000000000000006
[ 3719.738140] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff9d104fc965b0
[ 3719.738147] RBP: ffff9d10486bd940 R08: 0000000000000663 R09: 0000000000000000
[ 3719.738156] R10: ffffffff93104510 R11: 0000000000000000 R12: ffff9d10486bd940
[ 3719.738163] R13: ffff9d104fc83cf0 R14: 0000000000000046 R15: ffff9d10486be084
[ 3719.738172] FS:  0000000000000000(0000) GS:ffff9d104fc80000(0000) knlGS:0000000000000000
[ 3719.738180] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3719.738188] CR2: 0000740100706d71 CR3: 00000003dc4a0002 CR4: 00000000001626e0
[ 3719.738195] Call Trace:
[ 3719.738203]  <IRQ>
[ 3719.738214]  check_preempt_curr+0x4d/0x90
[ 3719.738225]  ttwu_do_wakeup+0x19/0x150
[ 3719.738236]  try_to_wake_up+0x210/0x4c0
[ 3719.738250]  __wake_up_common+0x7a/0x140
[ 3719.738265]  ep_poll_callback+0x152/0x270
[ 3719.738280]  __wake_up_common+0x7a/0x140
[ 3719.738292]  __wake_up_common_lock+0x7f/0xc0
[ 3719.738304]  irq_work_run_list+0x4f/0x70
[ 3719.738316]  update_process_times+0x84/0x90
[ 3719.738329]  tick_sched_handle+0x22/0x60
[ 3719.738340]  tick_sched_timer+0x51/0xb0
[ 3719.738424]  ? tick_do_update_jiffies64.part.0+0xd0/0xd0
[ 3719.738435]  __hrtimer_run_queues+0x128/0x2a0

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Rudi Ratloser]


> We have experienced a kernel BPF null pointer dereference issue on all
> our machines since mid of June. It might be related to an upgrade of
> libvirt/kvm/qemu at that point of time. But we’re not sure.
...
> We experienced the kernel freeze on following Arch Linux kernels:
> - 5.7.0 (5.7.0-3-MANJARO x64)
> - 5.6.16 (5.6.16-1-MANJARO x64)
> - 5.4.44 (5.4.44-1-MANJARO x64)
> - 4.19.126 (4.19.126-1-MANJARO x64)
> - 4.14.183 (4.14.183-1-MANJARO x64)
> Kernel configs can be taken from https://gitlab.manjaro.org/packages/core.
> 
> Subsequent e-mails will contain the relevant extracts from journal or
> netconsole logs.

Kernel 5.7.0 (5.7.0-3-MANJARO x64)

BUG: kernel NULL pointer dereference, address: 0000000000000010
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 1 PID: 1132 Comm: nfsd Not tainted 5.7.0-3-MANJARO #1
Hardware name: ASUS All Series/CS-B, BIOS 3602 03/26/2018
RIP: 0010:__cgroup_bpf_run_filter_skb+0x196/0x230
Code: 48 89 73 18 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 e8 d8 cb ec ff e8 93 12 f2 ff 48 8b 85 38 06 00 00 31 ed <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 34 49 8b 46 08 65 48 89 05 01
RSP: 0018:ffffaddac09eba20 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff93e20832d0e0 RCX: 0000000000000034
RDX: 0000000000000000 RSI: ffff93e1f0af0000 RDI: ffffffff9b7f6888
RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff93e20fe80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003d158e004 CR4: 00000000001626e0
Call Trace:
ip_finish_output+0x68/0xa0
ip_output+0x76/0x130
? __ip_finish_output+0x1e0/0x1e0
__ip_queue_xmit+0x186/0x440
? __switch_to_asm+0x34/0x70
? __switch_to_asm+0x40/0x70
__tcp_transmit_skb+0x53e/0xbf0
? __switch_to_asm+0x34/0x70
tcp_write_xmit+0x391/0x11b0
__tcp_push_pending_frames+0x32/0xf0
tcp_sendmsg_locked+0xa3c/0xb50
tcp_sendmsg+0x28/0x40
sock_sendmsg+0x57/0x60
xprt_sock_sendmsg+0xe8/0x2b0 [sunrpc]
? nfsd_destroy+0x60/0x60 [nfsd]
svc_tcp_sendto+0x77/0xd0 [sunrpc]
svc_send+0x80/0x1f0 [sunrpc]
nfsd+0xed/0x150 [nfsd]
kthread+0x13e/0x160
? __kthread_bind_mask+0x60/0x60
ret_from_fork+0x35/0x40
Modules linked in: rpcsec_gss_krb5 scsi_transport_iscsi veth xt_CHECKSUM vhost_net vhost tap vhost_iotlb tun ebtable_filter ebtables ip6table_filter ip6_tables xt_MASQUERADE xt_recent xt_comment ipt_REJECT nf_reject_ipv4 xt_addrtype br_netfilter xt_physdev iptable_nat xt_mark iptable_mangle xt_TCPMSS xt_hashlimit xt_tcpudp xt_CT iptable_raw xt_multiport xt_conntrack nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_nat nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp llc fuse nct6775 hwmon_vid nls_iso8859_1 nls_cp437 vfat fat intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ofpart cmdlinepart intel_spi_platform intel_spi mei_hdcp i915 eeepc_wmi spi_nor asus_wmi mtd iTCO_wdt iTCO_vendor_support battery snd_hda_codec_realtek sparse_keymap wmi_bmof rfkill snd_hda_codec_generic aesni_intel ledtrig_audio crypto_simd snd_hda_intel snd_intel_dspcfg cryptd glue_helper i2c_algo_bit snd_hda_codec intel_cstate intel_uncore snd_hda_core snd_hwdep drm_kms_helper r8169 intel_rapl_perf snd_pcm joydev realtek i2c_i801 libphy snd_timer mousedev cec snd rc_core mei_me input_leds intel_gtt syscopyarea sysfillrect e1000e lpc_ich sysimgblt mei soundcore fb_sys_fops wmi evdev mac_hid nfsd usbip_host drm usbip_core nfs_acl auth_rpcgss lockd grace uinput crypto_user sunrpc agpgart ip_tables x_tables ext4 crc16 mbcache jbd2 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid dm_thin_pool dm_persistent_data libcrc32c crc32c_generic dm_bio_prison dm_bufio dm_mod
crc32c_intel sr_mod cdrom xhci_pci xhci_hcd ehci_pci ehci_hcd
CR2: 0000000000000010
---[ end trace 6fe9bf5a0db7a0b9 ]---
RIP: 0010:__cgroup_bpf_run_filter_skb+0x196/0x230
Code: 48 89 73 18 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 e8 d8 cb ec ff e8 93 12 f2 ff 48 8b 85 38 06 00 00 31 ed <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 34 49 8b 46 08 65 48 89 05 01
RSP: 0018:ffffaddac09eba20 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff93e20832d0e0 RCX: 0000000000000034
RDX: 0000000000000000 RSI: ffff93e1f0af0000 RDI: ffffffff9b7f6888
RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff93e20fe80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000010 CR3: 00000003d158e004 CR4: 00000000001626e0
note: nfsd[1132] exited with preempt_count 1
-- Reboot --

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Brenden Blanco]

On Sat, May 30, 2020 at 12:51 AM Lu Fengqi <lufq.fnst@cn.fujitsu.com> wrote:
>
> Hello,
>
> I encountered a reproducible NULL pointer dereference using the mainline
> kernel v5.7-rc7-44-g75caf310d16c(which also happened multiple times on
> 5.6.14). The machine is installed with archlinux, used as a kubernetes
> v1.18.3 node, and uses calico v3.13.2 as a cni plugin. I use kdump/crash
> to see the value of the bpf_prog pointer in cgroup.bpf is 0x0 or 0x800.
>
> I am not sure whether this is caused by kernel bpf or calico? If you need
> me to provide more information, please let me know. Any suggestions are
> very helpful.

I encountered a similar set of crashes. I was able to workaround it by
disabling the systemd IPAddressDeny feature until the number of
bpf-progs in use by systemd reached 0 (via lsof inspection). I hit the
crash in kernels 5.4.43 through 5.7.

[40188.268677] BUG: kernel NULL pointer dereference, address: 0000000000000010
[40188.268736] #PF: supervisor read access in kernel mode
[40188.268773] #PF: error_code(0x0000) - not-present page
[40188.268819] PGD 0 P4D 0
[40188.268842] Oops: 0000 [#1] SMP PTI
[40188.268871] CPU: 7 PID: 2834 Comm: nfsd Tainted: P           OE
5.4.43-1-lts #1
[40188.268915] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
3.2 11/22/2019
[40188.268970] RIP: 0010:__cgroup_bpf_run_filter_skb+0x155/0x1d0
[40188.269013] Code: 48 8b 4c 24 08 4c 01 ab c8 00 00 00 48 89 4b 18
48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 48 8b 86 38
06 00 00 <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 5f 31 ed 49 8b 46 08 65
48 89
[40188.269115] RSP: 0018:ffffb0c581cf3918 EFLAGS: 00010246
[40188.269153] RAX: 0000000000000000 RBX: ffff8e32156bfae0 RCX: 0000000000000048
[40188.269202] RDX: 0000000000000000 RSI: ffff8e31f9d1e000 RDI: ffff8e31f9bc8940
[40188.269250] RBP: ffff8e31f9bc8940 R08: ffff8e3215c74a40 R09: 0000000000000001
[40188.269299] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
[40188.269348] R13: 0000000000000000 R14: 000000000000e400 R15: 0000000000000001
[40188.269391] FS:  0000000000000000(0000) GS:ffff8e321fbc0000(0000)
knlGS:0000000000000000
[40188.269446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[40188.269481] CR2: 0000000000000010 CR3: 0000000271c0a001 CR4: 00000000001606e0
[40188.269530] Call Trace:
[40188.269559]  ip6_finish_output+0x68/0xa0
[40188.269587]  ip6_output+0x6e/0x130
[40188.269615]  ? __ip6_finish_output+0x110/0x110
[40188.269648]  ip6_xmit+0x2cf/0x5e0
[40188.269675]  ? ipv6_anycast_cleanup+0x50/0x50
[40188.269711]  inet6_csk_xmit+0xb6/0x100
[40188.269742]  __tcp_transmit_skb+0x4ff/0xb10
[40188.269776]  tcp_write_xmit+0x517/0x1030
[40188.269807]  __tcp_push_pending_frames+0x32/0xf0
[40188.269843]  do_tcp_sendpages+0x5fa/0x630
[40188.269875]  tcp_sendpage+0x48/0x80
[40188.269904]  inet_sendpage+0x52/0x90
[40188.269931]  kernel_sendpage+0x1a/0x30
[40188.269989]  svc_send_common+0x136/0x150 [sunrpc]
[40188.270044]  svc_sendto+0xd7/0x240 [sunrpc]
[40188.270096]  svc_tcp_sendto+0x36/0x50 [sunrpc]
[40188.271531]  svc_send+0x7b/0x150 [sunrpc]
[40188.272961]  nfsd+0xe3/0x140 [nfsd]
[40188.274381]  ? nfsd_destroy+0x50/0x50 [nfsd]
[40188.275785]  kthread+0x117/0x130
[40188.277166]  ? __kthread_bind_mask+0x60/0x60
[40188.278520]  ret_from_fork+0x35/0x40
[40188.279820] Modules linked in: netconsole veth macvlan xt_nat
xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
iptable_nat rpcsec_gss_krb5 nls_iso8859_1 nls_cp437 vfat fat 8021q
garp mrp twofish_generic twofish_avx_x86_64 twofish_x86_64_3way
twofish_x86_64 twofish_common intel_rapl_msr intel_rapl_common
algif_skcipher af_alg zfs(POE) sb_edac x86_pkg_temp_thermal
intel_powerclamp iTCO_wdt zunicode(POE) ipmi_ssif zavl(POE)
iTCO_vendor_support icp(POE) coretemp kvm_intel kvm irqbypass
zcommon(POE) znvpair(POE) intel_cstate spl(OE) intel_uncore zlua(POE)
intel_rapl_perf ast drm_vram_helper pcspkr ttm ixgbe drm_kms_helper
i2c_i801 joydev mei_me syscopyarea sysfillrect sysimgblt igb libphy
fb_sys_fops mousedev ioatdma i2c_algo_bit mdio input_leds lpc_ich mei
dca ipmi_si ipmi_devintf ipmi_msghandler evdev mac_hid
acpi_power_meter ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT
nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc xt_limit
xt_addrtype xt_tcpudp xt_physdev
[40188.279868]  xt_conntrack ip6table_filter ip6_tables
nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
iptable_filter nfsd sha256_ssse3 drm sha1_ssse3 auth_rpcgss nfs_acl
lockd grace agpgart sunrpc ip_tables x_tables ext4 crc32c_generic
crc16 mbcache jbd2 raid1 md_mod hid_generic usbhid hid sd_mod dm_crypt
dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
ahci aesni_intel libahci xhci_pci crypto_simd libata xhci_hcd cryptd
glue_helper scsi_mod ehci_pci ehci_hcd wmi
[40188.300047] CR2: 0000000000000010
[40188.301739] ---[ end trace 44ac77af42fe7195 ]---

I also hit an interesting (related?) warning on one of the iterations
during boot.

[ 8891.070041] ------------[ cut here ]------------
[ 8891.070093] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after
switching to atomic
[ 8891.070117] WARNING: CPU: 7 PID: 54 at lib/percpu-refcount.c:160
percpu_ref_switch_to_atomic_rcu+0x12f/0x140
[ 8891.070178] Modules linked in: netconsole veth macvlan xt_nat
xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
iptable_nat nls_iso8859_1 nls_cp437 vfat fat 8021q garp mrp
twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64
twofish_common algif_skcipher af_alg intel_rapl_msr intel_rapl_common
zfs(POE) zunicode(POE) zavl(POE) iTCO_wdt icp(POE) iTCO_vendor_support
ipmi_ssif sb_edac x86_pkg_temp_thermal zcommon(POE) intel_powerclamp
znvpair(POE) kvm_intel spl(OE) zlua(POE) kvm irqbypass ast
intel_cstate intel_uncore drm_vram_helper ttm intel_rapl_perf pcspkr
drm_kms_helper i2c_i801 syscopyarea joydev lpc_ich sysfillrect
mousedev input_leds ixgbe sysimgblt fb_sys_fops mei_me igb mei libphy
ioatdma i2c_algo_bit mdio dca ipmi_si acpi_power_meter ipmi_devintf
ipmi_msghandler evdev mac_hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt
ipt_REJECT nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc
xt_limit xt_addrtype xt_tcpudp xt_physdev xt_conntrack ip6table_filter
[ 8891.070220]  ip6_tables nf_conntrack_netbios_ns
nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter nfsd
sha256_ssse3 drm sha1_ssse3 nfs_acl lockd auth_rpcgss grace sunrpc
agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
raid1 hid_generic usbhid hid md_mod sd_mod dm_crypt dm_mod
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
libahci aesni_intel libata crypto_simd cryptd xhci_pci glue_helper
ehci_pci xhci_hcd scsi_mod ehci_hcd wmi [last unloaded: coretemp]
[ 8891.070664] CPU: 7 PID: 54 Comm: ksoftirqd/7 Tainted: P
OE     5.4.43-1-lts #1
[ 8891.070691] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
3.2 11/22/2019
[ 8891.070721] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x12f/0x140
[ 8891.070745] Code: eb 99 80 3d 33 da eb 00 00 0f 85 4d ff ff ff 48
8b 55 d8 48 8b 75 e8 48 c7 c7 20 1c 50 8a c6 05 17 da eb 00 01 e8 0f
4c c3 ff <0f> 0b e9 2b ff ff ff 0f 0b eb a2 90 90 90 90 90 90 8d 8c 16
ef be
[ 8891.070812] RSP: 0018:ffffbf72c027fe00 EFLAGS: 00010286
[ 8891.070833] RAX: 0000000000000000 RBX: 8000000000000002 RCX: 0000000000000000
[ 8891.070857] RDX: 0000000000000046 RSI: ffffffff8acd7b46 RDI: 0000000000000246
[ 8891.070885] RBP: ffffa0b458b1f8e8 R08: 000008161d131aa4 R09: 0000000000000046
[ 8891.070913] R10: 0000000080000007 R11: ffffffff8acd7b2b R12: 00003ebe60014fc8
[ 8891.070938] R13: ffffa0b45fbeb350 R14: ffffa0b45b953c00 R15: ffffa0b45b953c00
[ 8891.070968] FS:  0000000000000000(0000) GS:ffffa0b45fbc0000(0000)
knlGS:0000000000000000
[ 8891.071001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8891.071025] CR2: 00007fae6b351e10 CR3: 000000082b21c005 CR4: 00000000001606e0
[ 8891.071058] Call Trace:
[ 8891.071075]  rcu_core+0x1ba/0x4e0
[ 8891.071093]  __do_softirq+0xe9/0x2dc
[ 8891.071110]  run_ksoftirqd+0x26/0x40
[ 8891.072222]  smpboot_thread_fn+0xc5/0x160
[ 8891.073320]  ? smpboot_unregister_percpu_thread+0x60/0x60
[ 8891.074423]  kthread+0x117/0x130
[ 8891.075542]  ? __kthread_bind_mask+0x60/0x60
[ 8891.076672]  ret_from_fork+0x35/0x40
[ 8891.077815] ---[ end trace 727b9af96c86f011 ]---

Hope this is useful.

>
> Attachments:
> kernel_config is the config used to compile the kernel
> __cgroup_bpf_run_filter_skb is the result of "dis -l __cgroup_bpf_run_filter_skb"
> log. * is dmesg
> bt_FF. * is the stack frames when NULL Pointer dereference occurs
> cgroup.bpf. * is the bpf member of the cgroup structure in the __cgroup_bpf_run_filter_skb function
> bpf. * is the currently loaded bpf programs
>
> --
> Thanks,
> Lu
>
>

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Alexei Starovoitov]

On Tue, Jun 2, 2020 at 2:46 PM Brenden Blanco <bblanco@gmail.com> wrote:
>
> On Sat, May 30, 2020 at 12:51 AM Lu Fengqi <lufq.fnst@cn.fujitsu.com> wrote:
> >
> > Hello,
> >
> > I encountered a reproducible NULL pointer dereference using the mainline
> > kernel v5.7-rc7-44-g75caf310d16c(which also happened multiple times on
> > 5.6.14). The machine is installed with archlinux, used as a kubernetes
> > v1.18.3 node, and uses calico v3.13.2 as a cni plugin. I use kdump/crash
> > to see the value of the bpf_prog pointer in cgroup.bpf is 0x0 or 0x800.
> >
> > I am not sure whether this is caused by kernel bpf or calico? If you need
> > me to provide more information, please let me know. Any suggestions are
> > very helpful.
>
> I encountered a similar set of crashes. I was able to workaround it by
> disabling the systemd IPAddressDeny feature until the number of
> bpf-progs in use by systemd reached 0 (via lsof inspection). I hit the
> crash in kernels 5.4.43 through 5.7.
>
> [40188.268677] BUG: kernel NULL pointer dereference, address: 0000000000000010
> [40188.268736] #PF: supervisor read access in kernel mode
> [40188.268773] #PF: error_code(0x0000) - not-present page
> [40188.268819] PGD 0 P4D 0
> [40188.268842] Oops: 0000 [#1] SMP PTI
> [40188.268871] CPU: 7 PID: 2834 Comm: nfsd Tainted: P           OE
> 5.4.43-1-lts #1
> [40188.268915] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
> 3.2 11/22/2019
> [40188.268970] RIP: 0010:__cgroup_bpf_run_filter_skb+0x155/0x1d0
> [40188.269013] Code: 48 8b 4c 24 08 4c 01 ab c8 00 00 00 48 89 4b 18
> 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 48 8b 86 38
> 06 00 00 <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 5f 31 ed 49 8b 46 08 65
> 48 89
> [40188.269115] RSP: 0018:ffffb0c581cf3918 EFLAGS: 00010246
> [40188.269153] RAX: 0000000000000000 RBX: ffff8e32156bfae0 RCX: 0000000000000048
> [40188.269202] RDX: 0000000000000000 RSI: ffff8e31f9d1e000 RDI: ffff8e31f9bc8940
> [40188.269250] RBP: ffff8e31f9bc8940 R08: ffff8e3215c74a40 R09: 0000000000000001
> [40188.269299] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
> [40188.269348] R13: 0000000000000000 R14: 000000000000e400 R15: 0000000000000001
> [40188.269391] FS:  0000000000000000(0000) GS:ffff8e321fbc0000(0000)
> knlGS:0000000000000000
> [40188.269446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [40188.269481] CR2: 0000000000000010 CR3: 0000000271c0a001 CR4: 00000000001606e0
> [40188.269530] Call Trace:
> [40188.269559]  ip6_finish_output+0x68/0xa0
> [40188.269587]  ip6_output+0x6e/0x130
> [40188.269615]  ? __ip6_finish_output+0x110/0x110
> [40188.269648]  ip6_xmit+0x2cf/0x5e0
> [40188.269675]  ? ipv6_anycast_cleanup+0x50/0x50
> [40188.269711]  inet6_csk_xmit+0xb6/0x100
> [40188.269742]  __tcp_transmit_skb+0x4ff/0xb10
> [40188.269776]  tcp_write_xmit+0x517/0x1030
> [40188.269807]  __tcp_push_pending_frames+0x32/0xf0
> [40188.269843]  do_tcp_sendpages+0x5fa/0x630
> [40188.269875]  tcp_sendpage+0x48/0x80
> [40188.269904]  inet_sendpage+0x52/0x90
> [40188.269931]  kernel_sendpage+0x1a/0x30
> [40188.269989]  svc_send_common+0x136/0x150 [sunrpc]
> [40188.270044]  svc_sendto+0xd7/0x240 [sunrpc]
> [40188.270096]  svc_tcp_sendto+0x36/0x50 [sunrpc]
> [40188.271531]  svc_send+0x7b/0x150 [sunrpc]
> [40188.272961]  nfsd+0xe3/0x140 [nfsd]
> [40188.274381]  ? nfsd_destroy+0x50/0x50 [nfsd]
> [40188.275785]  kthread+0x117/0x130
> [40188.277166]  ? __kthread_bind_mask+0x60/0x60
> [40188.278520]  ret_from_fork+0x35/0x40
> [40188.279820] Modules linked in: netconsole veth macvlan xt_nat
> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
> iptable_nat rpcsec_gss_krb5 nls_iso8859_1 nls_cp437 vfat fat 8021q
> garp mrp twofish_generic twofish_avx_x86_64 twofish_x86_64_3way
> twofish_x86_64 twofish_common intel_rapl_msr intel_rapl_common
> algif_skcipher af_alg zfs(POE) sb_edac x86_pkg_temp_thermal
> intel_powerclamp iTCO_wdt zunicode(POE) ipmi_ssif zavl(POE)
> iTCO_vendor_support icp(POE) coretemp kvm_intel kvm irqbypass
> zcommon(POE) znvpair(POE) intel_cstate spl(OE) intel_uncore zlua(POE)
> intel_rapl_perf ast drm_vram_helper pcspkr ttm ixgbe drm_kms_helper
> i2c_i801 joydev mei_me syscopyarea sysfillrect sysimgblt igb libphy
> fb_sys_fops mousedev ioatdma i2c_algo_bit mdio input_leds lpc_ich mei
> dca ipmi_si ipmi_devintf ipmi_msghandler evdev mac_hid
> acpi_power_meter ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT
> nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc xt_limit
> xt_addrtype xt_tcpudp xt_physdev
> [40188.279868]  xt_conntrack ip6table_filter ip6_tables
> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
> nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
> iptable_filter nfsd sha256_ssse3 drm sha1_ssse3 auth_rpcgss nfs_acl
> lockd grace agpgart sunrpc ip_tables x_tables ext4 crc32c_generic
> crc16 mbcache jbd2 raid1 md_mod hid_generic usbhid hid sd_mod dm_crypt
> dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
> ahci aesni_intel libahci xhci_pci crypto_simd libata xhci_hcd cryptd
> glue_helper scsi_mod ehci_pci ehci_hcd wmi
> [40188.300047] CR2: 0000000000000010
> [40188.301739] ---[ end trace 44ac77af42fe7195 ]---
>
> I also hit an interesting (related?) warning on one of the iterations
> during boot.
>
> [ 8891.070041] ------------[ cut here ]------------
> [ 8891.070093] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after
> switching to atomic
> [ 8891.070117] WARNING: CPU: 7 PID: 54 at lib/percpu-refcount.c:160
> percpu_ref_switch_to_atomic_rcu+0x12f/0x140
> [ 8891.070178] Modules linked in: netconsole veth macvlan xt_nat
> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
> iptable_nat nls_iso8859_1 nls_cp437 vfat fat 8021q garp mrp
> twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64
> twofish_common algif_skcipher af_alg intel_rapl_msr intel_rapl_common
> zfs(POE) zunicode(POE) zavl(POE) iTCO_wdt icp(POE) iTCO_vendor_support
> ipmi_ssif sb_edac x86_pkg_temp_thermal zcommon(POE) intel_powerclamp
> znvpair(POE) kvm_intel spl(OE) zlua(POE) kvm irqbypass ast
> intel_cstate intel_uncore drm_vram_helper ttm intel_rapl_perf pcspkr
> drm_kms_helper i2c_i801 syscopyarea joydev lpc_ich sysfillrect
> mousedev input_leds ixgbe sysimgblt fb_sys_fops mei_me igb mei libphy
> ioatdma i2c_algo_bit mdio dca ipmi_si acpi_power_meter ipmi_devintf
> ipmi_msghandler evdev mac_hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt
> ipt_REJECT nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc
> xt_limit xt_addrtype xt_tcpudp xt_physdev xt_conntrack ip6table_filter
> [ 8891.070220]  ip6_tables nf_conntrack_netbios_ns
> nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
> nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter nfsd
> sha256_ssse3 drm sha1_ssse3 nfs_acl lockd auth_rpcgss grace sunrpc
> agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
> raid1 hid_generic usbhid hid md_mod sd_mod dm_crypt dm_mod
> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
> libahci aesni_intel libata crypto_simd cryptd xhci_pci glue_helper
> ehci_pci xhci_hcd scsi_mod ehci_hcd wmi [last unloaded: coretemp]
> [ 8891.070664] CPU: 7 PID: 54 Comm: ksoftirqd/7 Tainted: P
> OE     5.4.43-1-lts #1
> [ 8891.070691] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
> 3.2 11/22/2019
> [ 8891.070721] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x12f/0x140
> [ 8891.070745] Code: eb 99 80 3d 33 da eb 00 00 0f 85 4d ff ff ff 48
> 8b 55 d8 48 8b 75 e8 48 c7 c7 20 1c 50 8a c6 05 17 da eb 00 01 e8 0f
> 4c c3 ff <0f> 0b e9 2b ff ff ff 0f 0b eb a2 90 90 90 90 90 90 8d 8c 16
> ef be
> [ 8891.070812] RSP: 0018:ffffbf72c027fe00 EFLAGS: 00010286
> [ 8891.070833] RAX: 0000000000000000 RBX: 8000000000000002 RCX: 0000000000000000
> [ 8891.070857] RDX: 0000000000000046 RSI: ffffffff8acd7b46 RDI: 0000000000000246
> [ 8891.070885] RBP: ffffa0b458b1f8e8 R08: 000008161d131aa4 R09: 0000000000000046
> [ 8891.070913] R10: 0000000080000007 R11: ffffffff8acd7b2b R12: 00003ebe60014fc8
> [ 8891.070938] R13: ffffa0b45fbeb350 R14: ffffa0b45b953c00 R15: ffffa0b45b953c00
> [ 8891.070968] FS:  0000000000000000(0000) GS:ffffa0b45fbc0000(0000)
> knlGS:0000000000000000
> [ 8891.071001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 8891.071025] CR2: 00007fae6b351e10 CR3: 000000082b21c005 CR4: 00000000001606e0
> [ 8891.071058] Call Trace:
> [ 8891.071075]  rcu_core+0x1ba/0x4e0
> [ 8891.071093]  __do_softirq+0xe9/0x2dc
> [ 8891.071110]  run_ksoftirqd+0x26/0x40
> [ 8891.072222]  smpboot_thread_fn+0xc5/0x160
> [ 8891.073320]  ? smpboot_unregister_percpu_thread+0x60/0x60
> [ 8891.074423]  kthread+0x117/0x130
> [ 8891.075542]  ? __kthread_bind_mask+0x60/0x60
> [ 8891.076672]  ret_from_fork+0x35/0x40
> [ 8891.077815] ---[ end trace 727b9af96c86f011 ]---

Thanks for the reporting.
We've seen very similar stack trace due to out of order cgroup destroy.
But it was fixed in
commit e10360f815ca ("bpf: cgroup: prevent out-of-order release of cgroup bpf")
Sounds like it wasn't fixed completely?
If somebody can reproduce could you please
revert both the fix e10360f815ca and offending
commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from
cgroup itself")
and see whether it reproduces ?
It will help us narrow down the problem.

Thanks

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Lu Fengqi]

On Tue, Jun 02, 2020 at 03:17:18PM -0700, Alexei Starovoitov wrote:
>On Tue, Jun 2, 2020 at 2:46 PM Brenden Blanco <bblanco@gmail.com> wrote:
>>
>> On Sat, May 30, 2020 at 12:51 AM Lu Fengqi <lufq.fnst@cn.fujitsu.com> wrote:
>> >
>> > Hello,
>> >
>> > I encountered a reproducible NULL pointer dereference using the mainline
>> > kernel v5.7-rc7-44-g75caf310d16c(which also happened multiple times on
>> > 5.6.14). The machine is installed with archlinux, used as a kubernetes
>> > v1.18.3 node, and uses calico v3.13.2 as a cni plugin. I use kdump/crash
>> > to see the value of the bpf_prog pointer in cgroup.bpf is 0x0 or 0x800.
>> >
>> > I am not sure whether this is caused by kernel bpf or calico? If you need
>> > me to provide more information, please let me know. Any suggestions are
>> > very helpful.
>>
>> I encountered a similar set of crashes. I was able to workaround it by
>> disabling the systemd IPAddressDeny feature until the number of
>> bpf-progs in use by systemd reached 0 (via lsof inspection). I hit the
>> crash in kernels 5.4.43 through 5.7.
>>
>> [40188.268677] BUG: kernel NULL pointer dereference, address: 0000000000000010
>> [40188.268736] #PF: supervisor read access in kernel mode
>> [40188.268773] #PF: error_code(0x0000) - not-present page
>> [40188.268819] PGD 0 P4D 0
>> [40188.268842] Oops: 0000 [#1] SMP PTI
>> [40188.268871] CPU: 7 PID: 2834 Comm: nfsd Tainted: P           OE
>> 5.4.43-1-lts #1
>> [40188.268915] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>> 3.2 11/22/2019
>> [40188.268970] RIP: 0010:__cgroup_bpf_run_filter_skb+0x155/0x1d0
>> [40188.269013] Code: 48 8b 4c 24 08 4c 01 ab c8 00 00 00 48 89 4b 18
>> 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 48 8b 86 38
>> 06 00 00 <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 5f 31 ed 49 8b 46 08 65
>> 48 89
>> [40188.269115] RSP: 0018:ffffb0c581cf3918 EFLAGS: 00010246
>> [40188.269153] RAX: 0000000000000000 RBX: ffff8e32156bfae0 RCX: 0000000000000048
>> [40188.269202] RDX: 0000000000000000 RSI: ffff8e31f9d1e000 RDI: ffff8e31f9bc8940
>> [40188.269250] RBP: ffff8e31f9bc8940 R08: ffff8e3215c74a40 R09: 0000000000000001
>> [40188.269299] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
>> [40188.269348] R13: 0000000000000000 R14: 000000000000e400 R15: 0000000000000001
>> [40188.269391] FS:  0000000000000000(0000) GS:ffff8e321fbc0000(0000)
>> knlGS:0000000000000000
>> [40188.269446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [40188.269481] CR2: 0000000000000010 CR3: 0000000271c0a001 CR4: 00000000001606e0
>> [40188.269530] Call Trace:
>> [40188.269559]  ip6_finish_output+0x68/0xa0
>> [40188.269587]  ip6_output+0x6e/0x130
>> [40188.269615]  ? __ip6_finish_output+0x110/0x110
>> [40188.269648]  ip6_xmit+0x2cf/0x5e0
>> [40188.269675]  ? ipv6_anycast_cleanup+0x50/0x50
>> [40188.269711]  inet6_csk_xmit+0xb6/0x100
>> [40188.269742]  __tcp_transmit_skb+0x4ff/0xb10
>> [40188.269776]  tcp_write_xmit+0x517/0x1030
>> [40188.269807]  __tcp_push_pending_frames+0x32/0xf0
>> [40188.269843]  do_tcp_sendpages+0x5fa/0x630
>> [40188.269875]  tcp_sendpage+0x48/0x80
>> [40188.269904]  inet_sendpage+0x52/0x90
>> [40188.269931]  kernel_sendpage+0x1a/0x30
>> [40188.269989]  svc_send_common+0x136/0x150 [sunrpc]
>> [40188.270044]  svc_sendto+0xd7/0x240 [sunrpc]
>> [40188.270096]  svc_tcp_sendto+0x36/0x50 [sunrpc]
>> [40188.271531]  svc_send+0x7b/0x150 [sunrpc]
>> [40188.272961]  nfsd+0xe3/0x140 [nfsd]
>> [40188.274381]  ? nfsd_destroy+0x50/0x50 [nfsd]
>> [40188.275785]  kthread+0x117/0x130
>> [40188.277166]  ? __kthread_bind_mask+0x60/0x60
>> [40188.278520]  ret_from_fork+0x35/0x40
>> [40188.279820] Modules linked in: netconsole veth macvlan xt_nat
>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>> iptable_nat rpcsec_gss_krb5 nls_iso8859_1 nls_cp437 vfat fat 8021q
>> garp mrp twofish_generic twofish_avx_x86_64 twofish_x86_64_3way
>> twofish_x86_64 twofish_common intel_rapl_msr intel_rapl_common
>> algif_skcipher af_alg zfs(POE) sb_edac x86_pkg_temp_thermal
>> intel_powerclamp iTCO_wdt zunicode(POE) ipmi_ssif zavl(POE)
>> iTCO_vendor_support icp(POE) coretemp kvm_intel kvm irqbypass
>> zcommon(POE) znvpair(POE) intel_cstate spl(OE) intel_uncore zlua(POE)
>> intel_rapl_perf ast drm_vram_helper pcspkr ttm ixgbe drm_kms_helper
>> i2c_i801 joydev mei_me syscopyarea sysfillrect sysimgblt igb libphy
>> fb_sys_fops mousedev ioatdma i2c_algo_bit mdio input_leds lpc_ich mei
>> dca ipmi_si ipmi_devintf ipmi_msghandler evdev mac_hid
>> acpi_power_meter ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT
>> nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc xt_limit
>> xt_addrtype xt_tcpudp xt_physdev
>> [40188.279868]  xt_conntrack ip6table_filter ip6_tables
>> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
>> nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
>> iptable_filter nfsd sha256_ssse3 drm sha1_ssse3 auth_rpcgss nfs_acl
>> lockd grace agpgart sunrpc ip_tables x_tables ext4 crc32c_generic
>> crc16 mbcache jbd2 raid1 md_mod hid_generic usbhid hid sd_mod dm_crypt
>> dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
>> ahci aesni_intel libahci xhci_pci crypto_simd libata xhci_hcd cryptd
>> glue_helper scsi_mod ehci_pci ehci_hcd wmi
>> [40188.300047] CR2: 0000000000000010
>> [40188.301739] ---[ end trace 44ac77af42fe7195 ]---
>>
>> I also hit an interesting (related?) warning on one of the iterations
>> during boot.
>>
>> [ 8891.070041] ------------[ cut here ]------------
>> [ 8891.070093] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after
>> switching to atomic
>> [ 8891.070117] WARNING: CPU: 7 PID: 54 at lib/percpu-refcount.c:160
>> percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>> [ 8891.070178] Modules linked in: netconsole veth macvlan xt_nat
>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>> iptable_nat nls_iso8859_1 nls_cp437 vfat fat 8021q garp mrp
>> twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64
>> twofish_common algif_skcipher af_alg intel_rapl_msr intel_rapl_common
>> zfs(POE) zunicode(POE) zavl(POE) iTCO_wdt icp(POE) iTCO_vendor_support
>> ipmi_ssif sb_edac x86_pkg_temp_thermal zcommon(POE) intel_powerclamp
>> znvpair(POE) kvm_intel spl(OE) zlua(POE) kvm irqbypass ast
>> intel_cstate intel_uncore drm_vram_helper ttm intel_rapl_perf pcspkr
>> drm_kms_helper i2c_i801 syscopyarea joydev lpc_ich sysfillrect
>> mousedev input_leds ixgbe sysimgblt fb_sys_fops mei_me igb mei libphy
>> ioatdma i2c_algo_bit mdio dca ipmi_si acpi_power_meter ipmi_devintf
>> ipmi_msghandler evdev mac_hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt
>> ipt_REJECT nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc
>> xt_limit xt_addrtype xt_tcpudp xt_physdev xt_conntrack ip6table_filter
>> [ 8891.070220]  ip6_tables nf_conntrack_netbios_ns
>> nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
>> nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter nfsd
>> sha256_ssse3 drm sha1_ssse3 nfs_acl lockd auth_rpcgss grace sunrpc
>> agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
>> raid1 hid_generic usbhid hid md_mod sd_mod dm_crypt dm_mod
>> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
>> libahci aesni_intel libata crypto_simd cryptd xhci_pci glue_helper
>> ehci_pci xhci_hcd scsi_mod ehci_hcd wmi [last unloaded: coretemp]
>> [ 8891.070664] CPU: 7 PID: 54 Comm: ksoftirqd/7 Tainted: P
>> OE     5.4.43-1-lts #1
>> [ 8891.070691] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>> 3.2 11/22/2019
>> [ 8891.070721] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>> [ 8891.070745] Code: eb 99 80 3d 33 da eb 00 00 0f 85 4d ff ff ff 48
>> 8b 55 d8 48 8b 75 e8 48 c7 c7 20 1c 50 8a c6 05 17 da eb 00 01 e8 0f
>> 4c c3 ff <0f> 0b e9 2b ff ff ff 0f 0b eb a2 90 90 90 90 90 90 8d 8c 16
>> ef be
>> [ 8891.070812] RSP: 0018:ffffbf72c027fe00 EFLAGS: 00010286
>> [ 8891.070833] RAX: 0000000000000000 RBX: 8000000000000002 RCX: 0000000000000000
>> [ 8891.070857] RDX: 0000000000000046 RSI: ffffffff8acd7b46 RDI: 0000000000000246
>> [ 8891.070885] RBP: ffffa0b458b1f8e8 R08: 000008161d131aa4 R09: 0000000000000046
>> [ 8891.070913] R10: 0000000080000007 R11: ffffffff8acd7b2b R12: 00003ebe60014fc8
>> [ 8891.070938] R13: ffffa0b45fbeb350 R14: ffffa0b45b953c00 R15: ffffa0b45b953c00
>> [ 8891.070968] FS:  0000000000000000(0000) GS:ffffa0b45fbc0000(0000)
>> knlGS:0000000000000000
>> [ 8891.071001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 8891.071025] CR2: 00007fae6b351e10 CR3: 000000082b21c005 CR4: 00000000001606e0
>> [ 8891.071058] Call Trace:
>> [ 8891.071075]  rcu_core+0x1ba/0x4e0
>> [ 8891.071093]  __do_softirq+0xe9/0x2dc
>> [ 8891.071110]  run_ksoftirqd+0x26/0x40
>> [ 8891.072222]  smpboot_thread_fn+0xc5/0x160
>> [ 8891.073320]  ? smpboot_unregister_percpu_thread+0x60/0x60
>> [ 8891.074423]  kthread+0x117/0x130
>> [ 8891.075542]  ? __kthread_bind_mask+0x60/0x60
>> [ 8891.076672]  ret_from_fork+0x35/0x40
>> [ 8891.077815] ---[ end trace 727b9af96c86f011 ]---
>
>Thanks for the reporting.
>We've seen very similar stack trace due to out of order cgroup destroy.
>But it was fixed in
>commit e10360f815ca ("bpf: cgroup: prevent out-of-order release of cgroup bpf")
>Sounds like it wasn't fixed completely?

Yes, I also found this patch and confirmed that it has indeed been applied.

>If somebody can reproduce could you please
>revert both the fix e10360f815ca and offending
>commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from
>cgroup itself")
>and see whether it reproduces ?

I will revert these commits on the upstream kernel, once I capture the issue
again I will send it here.

>It will help us narrow down the problem.

-- 
Thanks,
Lu

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Lu Fengqi]

On Wed, Jun 03, 2020 at 02:20:52PM +0800, Lu Fengqi wrote:
>On Tue, Jun 02, 2020 at 03:17:18PM -0700, Alexei Starovoitov wrote:
>>On Tue, Jun 2, 2020 at 2:46 PM Brenden Blanco <bblanco@gmail.com> wrote:
>>>
>>> On Sat, May 30, 2020 at 12:51 AM Lu Fengqi <lufq.fnst@cn.fujitsu.com> wrote:
>>> >
>>> > Hello,
>>> >
>>> > I encountered a reproducible NULL pointer dereference using the mainline
>>> > kernel v5.7-rc7-44-g75caf310d16c(which also happened multiple times on
>>> > 5.6.14). The machine is installed with archlinux, used as a kubernetes
>>> > v1.18.3 node, and uses calico v3.13.2 as a cni plugin. I use kdump/crash
>>> > to see the value of the bpf_prog pointer in cgroup.bpf is 0x0 or 0x800.
>>> >
>>> > I am not sure whether this is caused by kernel bpf or calico? If you need
>>> > me to provide more information, please let me know. Any suggestions are
>>> > very helpful.
>>>
>>> I encountered a similar set of crashes. I was able to workaround it by
>>> disabling the systemd IPAddressDeny feature until the number of
>>> bpf-progs in use by systemd reached 0 (via lsof inspection). I hit the
>>> crash in kernels 5.4.43 through 5.7.
>>>
>>> [40188.268677] BUG: kernel NULL pointer dereference, address: 0000000000000010
>>> [40188.268736] #PF: supervisor read access in kernel mode
>>> [40188.268773] #PF: error_code(0x0000) - not-present page
>>> [40188.268819] PGD 0 P4D 0
>>> [40188.268842] Oops: 0000 [#1] SMP PTI
>>> [40188.268871] CPU: 7 PID: 2834 Comm: nfsd Tainted: P           OE
>>> 5.4.43-1-lts #1
>>> [40188.268915] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>>> 3.2 11/22/2019
>>> [40188.268970] RIP: 0010:__cgroup_bpf_run_filter_skb+0x155/0x1d0
>>> [40188.269013] Code: 48 8b 4c 24 08 4c 01 ab c8 00 00 00 48 89 4b 18
>>> 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 48 8b 86 38
>>> 06 00 00 <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 5f 31 ed 49 8b 46 08 65
>>> 48 89
>>> [40188.269115] RSP: 0018:ffffb0c581cf3918 EFLAGS: 00010246
>>> [40188.269153] RAX: 0000000000000000 RBX: ffff8e32156bfae0 RCX: 0000000000000048
>>> [40188.269202] RDX: 0000000000000000 RSI: ffff8e31f9d1e000 RDI: ffff8e31f9bc8940
>>> [40188.269250] RBP: ffff8e31f9bc8940 R08: ffff8e3215c74a40 R09: 0000000000000001
>>> [40188.269299] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
>>> [40188.269348] R13: 0000000000000000 R14: 000000000000e400 R15: 0000000000000001
>>> [40188.269391] FS:  0000000000000000(0000) GS:ffff8e321fbc0000(0000)
>>> knlGS:0000000000000000
>>> [40188.269446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [40188.269481] CR2: 0000000000000010 CR3: 0000000271c0a001 CR4: 00000000001606e0
>>> [40188.269530] Call Trace:
>>> [40188.269559]  ip6_finish_output+0x68/0xa0
>>> [40188.269587]  ip6_output+0x6e/0x130
>>> [40188.269615]  ? __ip6_finish_output+0x110/0x110
>>> [40188.269648]  ip6_xmit+0x2cf/0x5e0
>>> [40188.269675]  ? ipv6_anycast_cleanup+0x50/0x50
>>> [40188.269711]  inet6_csk_xmit+0xb6/0x100
>>> [40188.269742]  __tcp_transmit_skb+0x4ff/0xb10
>>> [40188.269776]  tcp_write_xmit+0x517/0x1030
>>> [40188.269807]  __tcp_push_pending_frames+0x32/0xf0
>>> [40188.269843]  do_tcp_sendpages+0x5fa/0x630
>>> [40188.269875]  tcp_sendpage+0x48/0x80
>>> [40188.269904]  inet_sendpage+0x52/0x90
>>> [40188.269931]  kernel_sendpage+0x1a/0x30
>>> [40188.269989]  svc_send_common+0x136/0x150 [sunrpc]
>>> [40188.270044]  svc_sendto+0xd7/0x240 [sunrpc]
>>> [40188.270096]  svc_tcp_sendto+0x36/0x50 [sunrpc]
>>> [40188.271531]  svc_send+0x7b/0x150 [sunrpc]
>>> [40188.272961]  nfsd+0xe3/0x140 [nfsd]
>>> [40188.274381]  ? nfsd_destroy+0x50/0x50 [nfsd]
>>> [40188.275785]  kthread+0x117/0x130
>>> [40188.277166]  ? __kthread_bind_mask+0x60/0x60
>>> [40188.278520]  ret_from_fork+0x35/0x40
>>> [40188.279820] Modules linked in: netconsole veth macvlan xt_nat
>>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>>> iptable_nat rpcsec_gss_krb5 nls_iso8859_1 nls_cp437 vfat fat 8021q
>>> garp mrp twofish_generic twofish_avx_x86_64 twofish_x86_64_3way
>>> twofish_x86_64 twofish_common intel_rapl_msr intel_rapl_common
>>> algif_skcipher af_alg zfs(POE) sb_edac x86_pkg_temp_thermal
>>> intel_powerclamp iTCO_wdt zunicode(POE) ipmi_ssif zavl(POE)
>>> iTCO_vendor_support icp(POE) coretemp kvm_intel kvm irqbypass
>>> zcommon(POE) znvpair(POE) intel_cstate spl(OE) intel_uncore zlua(POE)
>>> intel_rapl_perf ast drm_vram_helper pcspkr ttm ixgbe drm_kms_helper
>>> i2c_i801 joydev mei_me syscopyarea sysfillrect sysimgblt igb libphy
>>> fb_sys_fops mousedev ioatdma i2c_algo_bit mdio input_leds lpc_ich mei
>>> dca ipmi_si ipmi_devintf ipmi_msghandler evdev mac_hid
>>> acpi_power_meter ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT
>>> nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc xt_limit
>>> xt_addrtype xt_tcpudp xt_physdev
>>> [40188.279868]  xt_conntrack ip6table_filter ip6_tables
>>> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
>>> nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
>>> iptable_filter nfsd sha256_ssse3 drm sha1_ssse3 auth_rpcgss nfs_acl
>>> lockd grace agpgart sunrpc ip_tables x_tables ext4 crc32c_generic
>>> crc16 mbcache jbd2 raid1 md_mod hid_generic usbhid hid sd_mod dm_crypt
>>> dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
>>> ahci aesni_intel libahci xhci_pci crypto_simd libata xhci_hcd cryptd
>>> glue_helper scsi_mod ehci_pci ehci_hcd wmi
>>> [40188.300047] CR2: 0000000000000010
>>> [40188.301739] ---[ end trace 44ac77af42fe7195 ]---
>>>
>>> I also hit an interesting (related?) warning on one of the iterations
>>> during boot.
>>>
>>> [ 8891.070041] ------------[ cut here ]------------
>>> [ 8891.070093] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after
>>> switching to atomic
>>> [ 8891.070117] WARNING: CPU: 7 PID: 54 at lib/percpu-refcount.c:160
>>> percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>>> [ 8891.070178] Modules linked in: netconsole veth macvlan xt_nat
>>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>>> iptable_nat nls_iso8859_1 nls_cp437 vfat fat 8021q garp mrp
>>> twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64
>>> twofish_common algif_skcipher af_alg intel_rapl_msr intel_rapl_common
>>> zfs(POE) zunicode(POE) zavl(POE) iTCO_wdt icp(POE) iTCO_vendor_support
>>> ipmi_ssif sb_edac x86_pkg_temp_thermal zcommon(POE) intel_powerclamp
>>> znvpair(POE) kvm_intel spl(OE) zlua(POE) kvm irqbypass ast
>>> intel_cstate intel_uncore drm_vram_helper ttm intel_rapl_perf pcspkr
>>> drm_kms_helper i2c_i801 syscopyarea joydev lpc_ich sysfillrect
>>> mousedev input_leds ixgbe sysimgblt fb_sys_fops mei_me igb mei libphy
>>> ioatdma i2c_algo_bit mdio dca ipmi_si acpi_power_meter ipmi_devintf
>>> ipmi_msghandler evdev mac_hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt
>>> ipt_REJECT nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc
>>> xt_limit xt_addrtype xt_tcpudp xt_physdev xt_conntrack ip6table_filter
>>> [ 8891.070220]  ip6_tables nf_conntrack_netbios_ns
>>> nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
>>> nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter nfsd
>>> sha256_ssse3 drm sha1_ssse3 nfs_acl lockd auth_rpcgss grace sunrpc
>>> agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
>>> raid1 hid_generic usbhid hid md_mod sd_mod dm_crypt dm_mod
>>> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
>>> libahci aesni_intel libata crypto_simd cryptd xhci_pci glue_helper
>>> ehci_pci xhci_hcd scsi_mod ehci_hcd wmi [last unloaded: coretemp]
>>> [ 8891.070664] CPU: 7 PID: 54 Comm: ksoftirqd/7 Tainted: P
>>> OE     5.4.43-1-lts #1
>>> [ 8891.070691] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>>> 3.2 11/22/2019
>>> [ 8891.070721] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>>> [ 8891.070745] Code: eb 99 80 3d 33 da eb 00 00 0f 85 4d ff ff ff 48
>>> 8b 55 d8 48 8b 75 e8 48 c7 c7 20 1c 50 8a c6 05 17 da eb 00 01 e8 0f
>>> 4c c3 ff <0f> 0b e9 2b ff ff ff 0f 0b eb a2 90 90 90 90 90 90 8d 8c 16
>>> ef be
>>> [ 8891.070812] RSP: 0018:ffffbf72c027fe00 EFLAGS: 00010286
>>> [ 8891.070833] RAX: 0000000000000000 RBX: 8000000000000002 RCX: 0000000000000000
>>> [ 8891.070857] RDX: 0000000000000046 RSI: ffffffff8acd7b46 RDI: 0000000000000246
>>> [ 8891.070885] RBP: ffffa0b458b1f8e8 R08: 000008161d131aa4 R09: 0000000000000046
>>> [ 8891.070913] R10: 0000000080000007 R11: ffffffff8acd7b2b R12: 00003ebe60014fc8
>>> [ 8891.070938] R13: ffffa0b45fbeb350 R14: ffffa0b45b953c00 R15: ffffa0b45b953c00
>>> [ 8891.070968] FS:  0000000000000000(0000) GS:ffffa0b45fbc0000(0000)
>>> knlGS:0000000000000000
>>> [ 8891.071001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 8891.071025] CR2: 00007fae6b351e10 CR3: 000000082b21c005 CR4: 00000000001606e0
>>> [ 8891.071058] Call Trace:
>>> [ 8891.071075]  rcu_core+0x1ba/0x4e0
>>> [ 8891.071093]  __do_softirq+0xe9/0x2dc
>>> [ 8891.071110]  run_ksoftirqd+0x26/0x40
>>> [ 8891.072222]  smpboot_thread_fn+0xc5/0x160
>>> [ 8891.073320]  ? smpboot_unregister_percpu_thread+0x60/0x60
>>> [ 8891.074423]  kthread+0x117/0x130
>>> [ 8891.075542]  ? __kthread_bind_mask+0x60/0x60
>>> [ 8891.076672]  ret_from_fork+0x35/0x40
>>> [ 8891.077815] ---[ end trace 727b9af96c86f011 ]---
>>
>>Thanks for the reporting.
>>We've seen very similar stack trace due to out of order cgroup destroy.
>>But it was fixed in
>>commit e10360f815ca ("bpf: cgroup: prevent out-of-order release of cgroup bpf")
>>Sounds like it wasn't fixed completely?
>
>Yes, I also found this patch and confirmed that it has indeed been applied.
>
>>If somebody can reproduce could you please
>>revert both the fix e10360f815ca and offending
>>commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from
>>cgroup itself")
>>and see whether it reproduces ?
>
>I will revert these commits on the upstream kernel, once I capture the issue
>again I will send it here.

I try to revert commit 4bfc0bb2c60e ("bpf: decouple the lifetime of
cgroup_bpf from cgroup itself") on the upstream kernel, but it seems
unlikely reverted without conflict. How about I checkout the commit
37b54aed123f ("samples/bpf: fix a couple of style issues in bpf_load") and
wait the issue be reproduced?

-- 
Thanks,
Lu

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Daniel Borkmann]

[ +Zefan, + TJ ]

On 6/3/20 12:17 AM, Alexei Starovoitov wrote:
> On Tue, Jun 2, 2020 at 2:46 PM Brenden Blanco <bblanco@gmail.com> wrote:
>> On Sat, May 30, 2020 at 12:51 AM Lu Fengqi <lufq.fnst@cn.fujitsu.com> wrote:
>>>
>>> I encountered a reproducible NULL pointer dereference using the mainline
>>> kernel v5.7-rc7-44-g75caf310d16c(which also happened multiple times on
>>> 5.6.14). The machine is installed with archlinux, used as a kubernetes
>>> v1.18.3 node, and uses calico v3.13.2 as a cni plugin. I use kdump/crash
>>> to see the value of the bpf_prog pointer in cgroup.bpf is 0x0 or 0x800.
>>>
>>> I am not sure whether this is caused by kernel bpf or calico? If you need
>>> me to provide more information, please let me know. Any suggestions are
>>> very helpful.
>>
>> I encountered a similar set of crashes. I was able to workaround it by
>> disabling the systemd IPAddressDeny feature until the number of
>> bpf-progs in use by systemd reached 0 (via lsof inspection). I hit the
>> crash in kernels 5.4.43 through 5.7.
>>
>> [40188.268677] BUG: kernel NULL pointer dereference, address: 0000000000000010
>> [40188.268736] #PF: supervisor read access in kernel mode
>> [40188.268773] #PF: error_code(0x0000) - not-present page
>> [40188.268819] PGD 0 P4D 0
>> [40188.268842] Oops: 0000 [#1] SMP PTI
>> [40188.268871] CPU: 7 PID: 2834 Comm: nfsd Tainted: P           OE
>> 5.4.43-1-lts #1
>> [40188.268915] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>> 3.2 11/22/2019
>> [40188.268970] RIP: 0010:__cgroup_bpf_run_filter_skb+0x155/0x1d0
>> [40188.269013] Code: 48 8b 4c 24 08 4c 01 ab c8 00 00 00 48 89 4b 18
>> 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 48 8b 86 38
>> 06 00 00 <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 5f 31 ed 49 8b 46 08 65
>> 48 89
>> [40188.269115] RSP: 0018:ffffb0c581cf3918 EFLAGS: 00010246
>> [40188.269153] RAX: 0000000000000000 RBX: ffff8e32156bfae0 RCX: 0000000000000048
>> [40188.269202] RDX: 0000000000000000 RSI: ffff8e31f9d1e000 RDI: ffff8e31f9bc8940
>> [40188.269250] RBP: ffff8e31f9bc8940 R08: ffff8e3215c74a40 R09: 0000000000000001
>> [40188.269299] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
>> [40188.269348] R13: 0000000000000000 R14: 000000000000e400 R15: 0000000000000001
>> [40188.269391] FS:  0000000000000000(0000) GS:ffff8e321fbc0000(0000)
>> knlGS:0000000000000000
>> [40188.269446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [40188.269481] CR2: 0000000000000010 CR3: 0000000271c0a001 CR4: 00000000001606e0
>> [40188.269530] Call Trace:
>> [40188.269559]  ip6_finish_output+0x68/0xa0
>> [40188.269587]  ip6_output+0x6e/0x130
>> [40188.269615]  ? __ip6_finish_output+0x110/0x110
>> [40188.269648]  ip6_xmit+0x2cf/0x5e0
>> [40188.269675]  ? ipv6_anycast_cleanup+0x50/0x50
>> [40188.269711]  inet6_csk_xmit+0xb6/0x100
>> [40188.269742]  __tcp_transmit_skb+0x4ff/0xb10
>> [40188.269776]  tcp_write_xmit+0x517/0x1030
>> [40188.269807]  __tcp_push_pending_frames+0x32/0xf0
>> [40188.269843]  do_tcp_sendpages+0x5fa/0x630
>> [40188.269875]  tcp_sendpage+0x48/0x80
>> [40188.269904]  inet_sendpage+0x52/0x90
>> [40188.269931]  kernel_sendpage+0x1a/0x30
>> [40188.269989]  svc_send_common+0x136/0x150 [sunrpc]
>> [40188.270044]  svc_sendto+0xd7/0x240 [sunrpc]
>> [40188.270096]  svc_tcp_sendto+0x36/0x50 [sunrpc]
>> [40188.271531]  svc_send+0x7b/0x150 [sunrpc]
>> [40188.272961]  nfsd+0xe3/0x140 [nfsd]
>> [40188.274381]  ? nfsd_destroy+0x50/0x50 [nfsd]
>> [40188.275785]  kthread+0x117/0x130
>> [40188.277166]  ? __kthread_bind_mask+0x60/0x60
>> [40188.278520]  ret_from_fork+0x35/0x40
>> [40188.279820] Modules linked in: netconsole veth macvlan xt_nat
>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>> iptable_nat rpcsec_gss_krb5 nls_iso8859_1 nls_cp437 vfat fat 8021q
>> garp mrp twofish_generic twofish_avx_x86_64 twofish_x86_64_3way
>> twofish_x86_64 twofish_common intel_rapl_msr intel_rapl_common
>> algif_skcipher af_alg zfs(POE) sb_edac x86_pkg_temp_thermal
>> intel_powerclamp iTCO_wdt zunicode(POE) ipmi_ssif zavl(POE)
>> iTCO_vendor_support icp(POE) coretemp kvm_intel kvm irqbypass
>> zcommon(POE) znvpair(POE) intel_cstate spl(OE) intel_uncore zlua(POE)
>> intel_rapl_perf ast drm_vram_helper pcspkr ttm ixgbe drm_kms_helper
>> i2c_i801 joydev mei_me syscopyarea sysfillrect sysimgblt igb libphy
>> fb_sys_fops mousedev ioatdma i2c_algo_bit mdio input_leds lpc_ich mei
>> dca ipmi_si ipmi_devintf ipmi_msghandler evdev mac_hid
>> acpi_power_meter ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT
>> nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc xt_limit
>> xt_addrtype xt_tcpudp xt_physdev
>> [40188.279868]  xt_conntrack ip6table_filter ip6_tables
>> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
>> nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
>> iptable_filter nfsd sha256_ssse3 drm sha1_ssse3 auth_rpcgss nfs_acl
>> lockd grace agpgart sunrpc ip_tables x_tables ext4 crc32c_generic
>> crc16 mbcache jbd2 raid1 md_mod hid_generic usbhid hid sd_mod dm_crypt
>> dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
>> ahci aesni_intel libahci xhci_pci crypto_simd libata xhci_hcd cryptd
>> glue_helper scsi_mod ehci_pci ehci_hcd wmi
>> [40188.300047] CR2: 0000000000000010
>> [40188.301739] ---[ end trace 44ac77af42fe7195 ]---
>>
>> I also hit an interesting (related?) warning on one of the iterations
>> during boot.
>>
>> [ 8891.070041] ------------[ cut here ]------------
>> [ 8891.070093] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after
>> switching to atomic
>> [ 8891.070117] WARNING: CPU: 7 PID: 54 at lib/percpu-refcount.c:160
>> percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>> [ 8891.070178] Modules linked in: netconsole veth macvlan xt_nat
>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>> iptable_nat nls_iso8859_1 nls_cp437 vfat fat 8021q garp mrp
>> twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64
>> twofish_common algif_skcipher af_alg intel_rapl_msr intel_rapl_common
>> zfs(POE) zunicode(POE) zavl(POE) iTCO_wdt icp(POE) iTCO_vendor_support
>> ipmi_ssif sb_edac x86_pkg_temp_thermal zcommon(POE) intel_powerclamp
>> znvpair(POE) kvm_intel spl(OE) zlua(POE) kvm irqbypass ast
>> intel_cstate intel_uncore drm_vram_helper ttm intel_rapl_perf pcspkr
>> drm_kms_helper i2c_i801 syscopyarea joydev lpc_ich sysfillrect
>> mousedev input_leds ixgbe sysimgblt fb_sys_fops mei_me igb mei libphy
>> ioatdma i2c_algo_bit mdio dca ipmi_si acpi_power_meter ipmi_devintf
>> ipmi_msghandler evdev mac_hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt
>> ipt_REJECT nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc
>> xt_limit xt_addrtype xt_tcpudp xt_physdev xt_conntrack ip6table_filter
>> [ 8891.070220]  ip6_tables nf_conntrack_netbios_ns
>> nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
>> nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter nfsd
>> sha256_ssse3 drm sha1_ssse3 nfs_acl lockd auth_rpcgss grace sunrpc
>> agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
>> raid1 hid_generic usbhid hid md_mod sd_mod dm_crypt dm_mod
>> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
>> libahci aesni_intel libata crypto_simd cryptd xhci_pci glue_helper
>> ehci_pci xhci_hcd scsi_mod ehci_hcd wmi [last unloaded: coretemp]
>> [ 8891.070664] CPU: 7 PID: 54 Comm: ksoftirqd/7 Tainted: P
>> OE     5.4.43-1-lts #1
>> [ 8891.070691] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>> 3.2 11/22/2019
>> [ 8891.070721] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>> [ 8891.070745] Code: eb 99 80 3d 33 da eb 00 00 0f 85 4d ff ff ff 48
>> 8b 55 d8 48 8b 75 e8 48 c7 c7 20 1c 50 8a c6 05 17 da eb 00 01 e8 0f
>> 4c c3 ff <0f> 0b e9 2b ff ff ff 0f 0b eb a2 90 90 90 90 90 90 8d 8c 16
>> ef be
>> [ 8891.070812] RSP: 0018:ffffbf72c027fe00 EFLAGS: 00010286
>> [ 8891.070833] RAX: 0000000000000000 RBX: 8000000000000002 RCX: 0000000000000000
>> [ 8891.070857] RDX: 0000000000000046 RSI: ffffffff8acd7b46 RDI: 0000000000000246
>> [ 8891.070885] RBP: ffffa0b458b1f8e8 R08: 000008161d131aa4 R09: 0000000000000046
>> [ 8891.070913] R10: 0000000080000007 R11: ffffffff8acd7b2b R12: 00003ebe60014fc8
>> [ 8891.070938] R13: ffffa0b45fbeb350 R14: ffffa0b45b953c00 R15: ffffa0b45b953c00
>> [ 8891.070968] FS:  0000000000000000(0000) GS:ffffa0b45fbc0000(0000)
>> knlGS:0000000000000000
>> [ 8891.071001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 8891.071025] CR2: 00007fae6b351e10 CR3: 000000082b21c005 CR4: 00000000001606e0
>> [ 8891.071058] Call Trace:
>> [ 8891.071075]  rcu_core+0x1ba/0x4e0
>> [ 8891.071093]  __do_softirq+0xe9/0x2dc
>> [ 8891.071110]  run_ksoftirqd+0x26/0x40
>> [ 8891.072222]  smpboot_thread_fn+0xc5/0x160
>> [ 8891.073320]  ? smpboot_unregister_percpu_thread+0x60/0x60
>> [ 8891.074423]  kthread+0x117/0x130
>> [ 8891.075542]  ? __kthread_bind_mask+0x60/0x60
>> [ 8891.076672]  ret_from_fork+0x35/0x40
>> [ 8891.077815] ---[ end trace 727b9af96c86f011 ]---
> 
> Thanks for the reporting.
> We've seen very similar stack trace due to out of order cgroup destroy.
> But it was fixed in
> commit e10360f815ca ("bpf: cgroup: prevent out-of-order release of cgroup bpf")
> Sounds like it wasn't fixed completely?
> If somebody can reproduce could you please
> revert both the fix e10360f815ca and offending
> commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from
> cgroup itself")
> and see whether it reproduces ?
> It will help us narrow down the problem.

Recent updates in [0] suggest that reverting [1] would fix the panic.

Zefan, please take a look.

   [0] https://bugzilla.kernel.org/show_bug.cgi?id=208003
   [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=090e28b229af92dc5b40786ca673999d59e73056

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Zefan Li]

On 2020/6/10 4:50, Daniel Borkmann wrote:
> [ +Zefan, + TJ ]
> 
> On 6/3/20 12:17 AM, Alexei Starovoitov wrote:
>> On Tue, Jun 2, 2020 at 2:46 PM Brenden Blanco <bblanco@gmail.com> wrote:
>>> On Sat, May 30, 2020 at 12:51 AM Lu Fengqi <lufq.fnst@cn.fujitsu.com> wrote:
>>>>
>>>> I encountered a reproducible NULL pointer dereference using the mainline
>>>> kernel v5.7-rc7-44-g75caf310d16c(which also happened multiple times on
>>>> 5.6.14). The machine is installed with archlinux, used as a kubernetes
>>>> v1.18.3 node, and uses calico v3.13.2 as a cni plugin. I use kdump/crash
>>>> to see the value of the bpf_prog pointer in cgroup.bpf is 0x0 or 0x800.
>>>>
>>>> I am not sure whether this is caused by kernel bpf or calico? If you need
>>>> me to provide more information, please let me know. Any suggestions are
>>>> very helpful.
>>>
>>> I encountered a similar set of crashes. I was able to workaround it by
>>> disabling the systemd IPAddressDeny feature until the number of
>>> bpf-progs in use by systemd reached 0 (via lsof inspection). I hit the
>>> crash in kernels 5.4.43 through 5.7.
>>>
>>> [40188.268677] BUG: kernel NULL pointer dereference, address: 0000000000000010
>>> [40188.268736] #PF: supervisor read access in kernel mode
>>> [40188.268773] #PF: error_code(0x0000) - not-present page
>>> [40188.268819] PGD 0 P4D 0
>>> [40188.268842] Oops: 0000 [#1] SMP PTI
>>> [40188.268871] CPU: 7 PID: 2834 Comm: nfsd Tainted: P           OE
>>> 5.4.43-1-lts #1
>>> [40188.268915] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>>> 3.2 11/22/2019
>>> [40188.268970] RIP: 0010:__cgroup_bpf_run_filter_skb+0x155/0x1d0
>>> [40188.269013] Code: 48 8b 4c 24 08 4c 01 ab c8 00 00 00 48 89 4b 18
>>> 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 48 8b 86 38
>>> 06 00 00 <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 5f 31 ed 49 8b 46 08 65
>>> 48 89
>>> [40188.269115] RSP: 0018:ffffb0c581cf3918 EFLAGS: 00010246
>>> [40188.269153] RAX: 0000000000000000 RBX: ffff8e32156bfae0 RCX: 0000000000000048
>>> [40188.269202] RDX: 0000000000000000 RSI: ffff8e31f9d1e000 RDI: ffff8e31f9bc8940
>>> [40188.269250] RBP: ffff8e31f9bc8940 R08: ffff8e3215c74a40 R09: 0000000000000001
>>> [40188.269299] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
>>> [40188.269348] R13: 0000000000000000 R14: 000000000000e400 R15: 0000000000000001
>>> [40188.269391] FS:  0000000000000000(0000) GS:ffff8e321fbc0000(0000)
>>> knlGS:0000000000000000
>>> [40188.269446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [40188.269481] CR2: 0000000000000010 CR3: 0000000271c0a001 CR4: 00000000001606e0
>>> [40188.269530] Call Trace:
>>> [40188.269559]  ip6_finish_output+0x68/0xa0
>>> [40188.269587]  ip6_output+0x6e/0x130
>>> [40188.269615]  ? __ip6_finish_output+0x110/0x110
>>> [40188.269648]  ip6_xmit+0x2cf/0x5e0
>>> [40188.269675]  ? ipv6_anycast_cleanup+0x50/0x50
>>> [40188.269711]  inet6_csk_xmit+0xb6/0x100
>>> [40188.269742]  __tcp_transmit_skb+0x4ff/0xb10
>>> [40188.269776]  tcp_write_xmit+0x517/0x1030
>>> [40188.269807]  __tcp_push_pending_frames+0x32/0xf0
>>> [40188.269843]  do_tcp_sendpages+0x5fa/0x630
>>> [40188.269875]  tcp_sendpage+0x48/0x80
>>> [40188.269904]  inet_sendpage+0x52/0x90
>>> [40188.269931]  kernel_sendpage+0x1a/0x30
>>> [40188.269989]  svc_send_common+0x136/0x150 [sunrpc]
>>> [40188.270044]  svc_sendto+0xd7/0x240 [sunrpc]
>>> [40188.270096]  svc_tcp_sendto+0x36/0x50 [sunrpc]
>>> [40188.271531]  svc_send+0x7b/0x150 [sunrpc]
>>> [40188.272961]  nfsd+0xe3/0x140 [nfsd]
>>> [40188.274381]  ? nfsd_destroy+0x50/0x50 [nfsd]
>>> [40188.275785]  kthread+0x117/0x130
>>> [40188.277166]  ? __kthread_bind_mask+0x60/0x60
>>> [40188.278520]  ret_from_fork+0x35/0x40
>>> [40188.279820] Modules linked in: netconsole veth macvlan xt_nat
>>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>>> iptable_nat rpcsec_gss_krb5 nls_iso8859_1 nls_cp437 vfat fat 8021q
>>> garp mrp twofish_generic twofish_avx_x86_64 twofish_x86_64_3way
>>> twofish_x86_64 twofish_common intel_rapl_msr intel_rapl_common
>>> algif_skcipher af_alg zfs(POE) sb_edac x86_pkg_temp_thermal
>>> intel_powerclamp iTCO_wdt zunicode(POE) ipmi_ssif zavl(POE)
>>> iTCO_vendor_support icp(POE) coretemp kvm_intel kvm irqbypass
>>> zcommon(POE) znvpair(POE) intel_cstate spl(OE) intel_uncore zlua(POE)
>>> intel_rapl_perf ast drm_vram_helper pcspkr ttm ixgbe drm_kms_helper
>>> i2c_i801 joydev mei_me syscopyarea sysfillrect sysimgblt igb libphy
>>> fb_sys_fops mousedev ioatdma i2c_algo_bit mdio input_leds lpc_ich mei
>>> dca ipmi_si ipmi_devintf ipmi_msghandler evdev mac_hid
>>> acpi_power_meter ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT
>>> nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc xt_limit
>>> xt_addrtype xt_tcpudp xt_physdev
>>> [40188.279868]  xt_conntrack ip6table_filter ip6_tables
>>> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
>>> nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
>>> iptable_filter nfsd sha256_ssse3 drm sha1_ssse3 auth_rpcgss nfs_acl
>>> lockd grace agpgart sunrpc ip_tables x_tables ext4 crc32c_generic
>>> crc16 mbcache jbd2 raid1 md_mod hid_generic usbhid hid sd_mod dm_crypt
>>> dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
>>> ahci aesni_intel libahci xhci_pci crypto_simd libata xhci_hcd cryptd
>>> glue_helper scsi_mod ehci_pci ehci_hcd wmi
>>> [40188.300047] CR2: 0000000000000010
>>> [40188.301739] ---[ end trace 44ac77af42fe7195 ]---
>>>
>>> I also hit an interesting (related?) warning on one of the iterations
>>> during boot.
>>>
>>> [ 8891.070041] ------------[ cut here ]------------
>>> [ 8891.070093] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after
>>> switching to atomic
>>> [ 8891.070117] WARNING: CPU: 7 PID: 54 at lib/percpu-refcount.c:160
>>> percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>>> [ 8891.070178] Modules linked in: netconsole veth macvlan xt_nat
>>> xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>>> iptable_nat nls_iso8859_1 nls_cp437 vfat fat 8021q garp mrp
>>> twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64
>>> twofish_common algif_skcipher af_alg intel_rapl_msr intel_rapl_common
>>> zfs(POE) zunicode(POE) zavl(POE) iTCO_wdt icp(POE) iTCO_vendor_support
>>> ipmi_ssif sb_edac x86_pkg_temp_thermal zcommon(POE) intel_powerclamp
>>> znvpair(POE) kvm_intel spl(OE) zlua(POE) kvm irqbypass ast
>>> intel_cstate intel_uncore drm_vram_helper ttm intel_rapl_perf pcspkr
>>> drm_kms_helper i2c_i801 syscopyarea joydev lpc_ich sysfillrect
>>> mousedev input_leds ixgbe sysimgblt fb_sys_fops mei_me igb mei libphy
>>> ioatdma i2c_algo_bit mdio dca ipmi_si acpi_power_meter ipmi_devintf
>>> ipmi_msghandler evdev mac_hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt
>>> ipt_REJECT nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc
>>> xt_limit xt_addrtype xt_tcpudp xt_physdev xt_conntrack ip6table_filter
>>> [ 8891.070220]  ip6_tables nf_conntrack_netbios_ns
>>> nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
>>> nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter nfsd
>>> sha256_ssse3 drm sha1_ssse3 nfs_acl lockd auth_rpcgss grace sunrpc
>>> agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
>>> raid1 hid_generic usbhid hid md_mod sd_mod dm_crypt dm_mod
>>> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
>>> libahci aesni_intel libata crypto_simd cryptd xhci_pci glue_helper
>>> ehci_pci xhci_hcd scsi_mod ehci_hcd wmi [last unloaded: coretemp]
>>> [ 8891.070664] CPU: 7 PID: 54 Comm: ksoftirqd/7 Tainted: P
>>> OE     5.4.43-1-lts #1
>>> [ 8891.070691] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>>> 3.2 11/22/2019
>>> [ 8891.070721] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>>> [ 8891.070745] Code: eb 99 80 3d 33 da eb 00 00 0f 85 4d ff ff ff 48
>>> 8b 55 d8 48 8b 75 e8 48 c7 c7 20 1c 50 8a c6 05 17 da eb 00 01 e8 0f
>>> 4c c3 ff <0f> 0b e9 2b ff ff ff 0f 0b eb a2 90 90 90 90 90 90 8d 8c 16
>>> ef be
>>> [ 8891.070812] RSP: 0018:ffffbf72c027fe00 EFLAGS: 00010286
>>> [ 8891.070833] RAX: 0000000000000000 RBX: 8000000000000002 RCX: 0000000000000000
>>> [ 8891.070857] RDX: 0000000000000046 RSI: ffffffff8acd7b46 RDI: 0000000000000246
>>> [ 8891.070885] RBP: ffffa0b458b1f8e8 R08: 000008161d131aa4 R09: 0000000000000046
>>> [ 8891.070913] R10: 0000000080000007 R11: ffffffff8acd7b2b R12: 00003ebe60014fc8
>>> [ 8891.070938] R13: ffffa0b45fbeb350 R14: ffffa0b45b953c00 R15: ffffa0b45b953c00
>>> [ 8891.070968] FS:  0000000000000000(0000) GS:ffffa0b45fbc0000(0000)
>>> knlGS:0000000000000000
>>> [ 8891.071001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 8891.071025] CR2: 00007fae6b351e10 CR3: 000000082b21c005 CR4: 00000000001606e0
>>> [ 8891.071058] Call Trace:
>>> [ 8891.071075]  rcu_core+0x1ba/0x4e0
>>> [ 8891.071093]  __do_softirq+0xe9/0x2dc
>>> [ 8891.071110]  run_ksoftirqd+0x26/0x40
>>> [ 8891.072222]  smpboot_thread_fn+0xc5/0x160
>>> [ 8891.073320]  ? smpboot_unregister_percpu_thread+0x60/0x60
>>> [ 8891.074423]  kthread+0x117/0x130
>>> [ 8891.075542]  ? __kthread_bind_mask+0x60/0x60
>>> [ 8891.076672]  ret_from_fork+0x35/0x40
>>> [ 8891.077815] ---[ end trace 727b9af96c86f011 ]---
>>
>> Thanks for the reporting.
>> We've seen very similar stack trace due to out of order cgroup destroy.
>> But it was fixed in
>> commit e10360f815ca ("bpf: cgroup: prevent out-of-order release of cgroup bpf")
>> Sounds like it wasn't fixed completely?
>> If somebody can reproduce could you please
>> revert both the fix e10360f815ca and offending
>> commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from
>> cgroup itself")
>> and see whether it reproduces ?
>> It will help us narrow down the problem.
> 
> Recent updates in [0] suggest that reverting [1] would fix the panic.
> 
> Zefan, please take a look.
> 
>   [0] https://bugzilla.kernel.org/show_bug.cgi?id=208003
>   [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=090e28b229af92dc5b40786ca673999d59e73056
> 

Haven't looked into it, but my guess is the bug is in somewhere else,
and the above commit just unveiled it.

Re: BUG: kernel NULL pointer dereference in __cgroup_bpf_run_filter_skb

[Lu Fengqi]

On Tue, Jun 02, 2020 at 02:46:41PM -0700, Brenden Blanco wrote:
>On Sat, May 30, 2020 at 12:51 AM Lu Fengqi <lufq.fnst@cn.fujitsu.com> wrote:
>>
>> Hello,
>>
>> I encountered a reproducible NULL pointer dereference using the mainline
>> kernel v5.7-rc7-44-g75caf310d16c(which also happened multiple times on
>> 5.6.14). The machine is installed with archlinux, used as a kubernetes
>> v1.18.3 node, and uses calico v3.13.2 as a cni plugin. I use kdump/crash
>> to see the value of the bpf_prog pointer in cgroup.bpf is 0x0 or 0x800.
>>
>> I am not sure whether this is caused by kernel bpf or calico? If you need
>> me to provide more information, please let me know. Any suggestions are
>> very helpful.
>
>I encountered a similar set of crashes. I was able to workaround it by
>disabling the systemd IPAddressDeny feature until the number of
>bpf-progs in use by systemd reached 0 (via lsof inspection). I hit the
>crash in kernels 5.4.43 through 5.7.
>
>[40188.268677] BUG: kernel NULL pointer dereference, address: 0000000000000010
>[40188.268736] #PF: supervisor read access in kernel mode
>[40188.268773] #PF: error_code(0x0000) - not-present page
>[40188.268819] PGD 0 P4D 0
>[40188.268842] Oops: 0000 [#1] SMP PTI
>[40188.268871] CPU: 7 PID: 2834 Comm: nfsd Tainted: P           OE
>5.4.43-1-lts #1
>[40188.268915] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>3.2 11/22/2019
>[40188.268970] RIP: 0010:__cgroup_bpf_run_filter_skb+0x155/0x1d0
>[40188.269013] Code: 48 8b 4c 24 08 4c 01 ab c8 00 00 00 48 89 4b 18
>48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c0 c3 c3 48 8b 86 38
>06 00 00 <48> 8b 78 10 4c 8d 70 10 48 85 ff 74 5f 31 ed 49 8b 46 08 65
>48 89
>[40188.269115] RSP: 0018:ffffb0c581cf3918 EFLAGS: 00010246
>[40188.269153] RAX: 0000000000000000 RBX: ffff8e32156bfae0 RCX: 0000000000000048
>[40188.269202] RDX: 0000000000000000 RSI: ffff8e31f9d1e000 RDI: ffff8e31f9bc8940
>[40188.269250] RBP: ffff8e31f9bc8940 R08: ffff8e3215c74a40 R09: 0000000000000001
>[40188.269299] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
>[40188.269348] R13: 0000000000000000 R14: 000000000000e400 R15: 0000000000000001
>[40188.269391] FS:  0000000000000000(0000) GS:ffff8e321fbc0000(0000)
>knlGS:0000000000000000
>[40188.269446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>[40188.269481] CR2: 0000000000000010 CR3: 0000000271c0a001 CR4: 00000000001606e0
>[40188.269530] Call Trace:
>[40188.269559]  ip6_finish_output+0x68/0xa0
>[40188.269587]  ip6_output+0x6e/0x130
>[40188.269615]  ? __ip6_finish_output+0x110/0x110
>[40188.269648]  ip6_xmit+0x2cf/0x5e0
>[40188.269675]  ? ipv6_anycast_cleanup+0x50/0x50
>[40188.269711]  inet6_csk_xmit+0xb6/0x100
>[40188.269742]  __tcp_transmit_skb+0x4ff/0xb10
>[40188.269776]  tcp_write_xmit+0x517/0x1030
>[40188.269807]  __tcp_push_pending_frames+0x32/0xf0
>[40188.269843]  do_tcp_sendpages+0x5fa/0x630
>[40188.269875]  tcp_sendpage+0x48/0x80
>[40188.269904]  inet_sendpage+0x52/0x90
>[40188.269931]  kernel_sendpage+0x1a/0x30
>[40188.269989]  svc_send_common+0x136/0x150 [sunrpc]
>[40188.270044]  svc_sendto+0xd7/0x240 [sunrpc]
>[40188.270096]  svc_tcp_sendto+0x36/0x50 [sunrpc]
>[40188.271531]  svc_send+0x7b/0x150 [sunrpc]
>[40188.272961]  nfsd+0xe3/0x140 [nfsd]
>[40188.274381]  ? nfsd_destroy+0x50/0x50 [nfsd]
>[40188.275785]  kthread+0x117/0x130
>[40188.277166]  ? __kthread_bind_mask+0x60/0x60
>[40188.278520]  ret_from_fork+0x35/0x40
>[40188.279820] Modules linked in: netconsole veth macvlan xt_nat
>xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>iptable_nat rpcsec_gss_krb5 nls_iso8859_1 nls_cp437 vfat fat 8021q
>garp mrp twofish_generic twofish_avx_x86_64 twofish_x86_64_3way
>twofish_x86_64 twofish_common intel_rapl_msr intel_rapl_common
>algif_skcipher af_alg zfs(POE) sb_edac x86_pkg_temp_thermal
>intel_powerclamp iTCO_wdt zunicode(POE) ipmi_ssif zavl(POE)
>iTCO_vendor_support icp(POE) coretemp kvm_intel kvm irqbypass
>zcommon(POE) znvpair(POE) intel_cstate spl(OE) intel_uncore zlua(POE)
>intel_rapl_perf ast drm_vram_helper pcspkr ttm ixgbe drm_kms_helper
>i2c_i801 joydev mei_me syscopyarea sysfillrect sysimgblt igb libphy
>fb_sys_fops mousedev ioatdma i2c_algo_bit mdio input_leds lpc_ich mei
>dca ipmi_si ipmi_devintf ipmi_msghandler evdev mac_hid
>acpi_power_meter ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ipt_REJECT
>nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc xt_limit
>xt_addrtype xt_tcpudp xt_physdev
>[40188.279868]  xt_conntrack ip6table_filter ip6_tables
>nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat
>nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
>iptable_filter nfsd sha256_ssse3 drm sha1_ssse3 auth_rpcgss nfs_acl
>lockd grace agpgart sunrpc ip_tables x_tables ext4 crc32c_generic
>crc16 mbcache jbd2 raid1 md_mod hid_generic usbhid hid sd_mod dm_crypt
>dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
>ahci aesni_intel libahci xhci_pci crypto_simd libata xhci_hcd cryptd
>glue_helper scsi_mod ehci_pci ehci_hcd wmi
>[40188.300047] CR2: 0000000000000010
>[40188.301739] ---[ end trace 44ac77af42fe7195 ]---
>
>I also hit an interesting (related?) warning on one of the iterations
>during boot.
>
>[ 8891.070041] ------------[ cut here ]------------
>[ 8891.070093] percpu ref (cgroup_bpf_release_fn) <= 0 (-1) after
>switching to atomic
>[ 8891.070117] WARNING: CPU: 7 PID: 54 at lib/percpu-refcount.c:160
>percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>[ 8891.070178] Modules linked in: netconsole veth macvlan xt_nat
>xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo
>iptable_nat nls_iso8859_1 nls_cp437 vfat fat 8021q garp mrp
>twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64
>twofish_common algif_skcipher af_alg intel_rapl_msr intel_rapl_common
>zfs(POE) zunicode(POE) zavl(POE) iTCO_wdt icp(POE) iTCO_vendor_support
>ipmi_ssif sb_edac x86_pkg_temp_thermal zcommon(POE) intel_powerclamp
>znvpair(POE) kvm_intel spl(OE) zlua(POE) kvm irqbypass ast
>intel_cstate intel_uncore drm_vram_helper ttm intel_rapl_perf pcspkr
>drm_kms_helper i2c_i801 syscopyarea joydev lpc_ich sysfillrect
>mousedev input_leds ixgbe sysimgblt fb_sys_fops mei_me igb mei libphy
>ioatdma i2c_algo_bit mdio dca ipmi_si acpi_power_meter ipmi_devintf
>ipmi_msghandler evdev mac_hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt
>ipt_REJECT nf_reject_ipv4 xt_multiport br_netfilter bridge stp llc
>xt_limit xt_addrtype xt_tcpudp xt_physdev xt_conntrack ip6table_filter
>[ 8891.070220]  ip6_tables nf_conntrack_netbios_ns
>nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
>nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_filter nfsd
>sha256_ssse3 drm sha1_ssse3 nfs_acl lockd auth_rpcgss grace sunrpc
>agpgart ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
>raid1 hid_generic usbhid hid md_mod sd_mod dm_crypt dm_mod
>crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ahci
>libahci aesni_intel libata crypto_simd cryptd xhci_pci glue_helper
>ehci_pci xhci_hcd scsi_mod ehci_hcd wmi [last unloaded: coretemp]
>[ 8891.070664] CPU: 7 PID: 54 Comm: ksoftirqd/7 Tainted: P
>OE     5.4.43-1-lts #1
>[ 8891.070691] Hardware name: Supermicro Super Server/X10SRi-F, BIOS
>3.2 11/22/2019
>[ 8891.070721] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x12f/0x140
>[ 8891.070745] Code: eb 99 80 3d 33 da eb 00 00 0f 85 4d ff ff ff 48
>8b 55 d8 48 8b 75 e8 48 c7 c7 20 1c 50 8a c6 05 17 da eb 00 01 e8 0f
>4c c3 ff <0f> 0b e9 2b ff ff ff 0f 0b eb a2 90 90 90 90 90 90 8d 8c 16
>ef be
>[ 8891.070812] RSP: 0018:ffffbf72c027fe00 EFLAGS: 00010286
>[ 8891.070833] RAX: 0000000000000000 RBX: 8000000000000002 RCX: 0000000000000000
>[ 8891.070857] RDX: 0000000000000046 RSI: ffffffff8acd7b46 RDI: 0000000000000246
>[ 8891.070885] RBP: ffffa0b458b1f8e8 R08: 000008161d131aa4 R09: 0000000000000046
>[ 8891.070913] R10: 0000000080000007 R11: ffffffff8acd7b2b R12: 00003ebe60014fc8
>[ 8891.070938] R13: ffffa0b45fbeb350 R14: ffffa0b45b953c00 R15: ffffa0b45b953c00
>[ 8891.070968] FS:  0000000000000000(0000) GS:ffffa0b45fbc0000(0000)
>knlGS:0000000000000000
>[ 8891.071001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>[ 8891.071025] CR2: 00007fae6b351e10 CR3: 000000082b21c005 CR4: 00000000001606e0
>[ 8891.071058] Call Trace:
>[ 8891.071075]  rcu_core+0x1ba/0x4e0
>[ 8891.071093]  __do_softirq+0xe9/0x2dc
>[ 8891.071110]  run_ksoftirqd+0x26/0x40
>[ 8891.072222]  smpboot_thread_fn+0xc5/0x160
>[ 8891.073320]  ? smpboot_unregister_percpu_thread+0x60/0x60
>[ 8891.074423]  kthread+0x117/0x130
>[ 8891.075542]  ? __kthread_bind_mask+0x60/0x60
>[ 8891.076672]  ret_from_fork+0x35/0x40
>[ 8891.077815] ---[ end trace 727b9af96c86f011 ]---
>
>Hope this is useful.

Thank you so much for your reply, it at least proves calico's innocence.

-- 
Thanks,
Lu