Seemingly random crashes with CONFIG_HARDENED_USERCOPY=y on ppc64be

Seemingly random crashes with CONFIG_HARDENED_USERCOPY=y on ppc64be

[Stijn Tintel]

Hi,

When CONFIG_HARDENED_USERCOPY=y is enabled, I seem to be having random
crashes on a ppc64be system. The bug always triggers by OpenWrt's
firewall3, but in different parts of the netfilter code. See the traces
below. With the default firewall3 config, the system boots fine. Then
copying the config that triggers the crash, and calling firewall3, does
not trigger the crash. It only seems to trigger the crash when that
specific config is applied during boot. The randomness leads me to
believe the problem is in the CONFIG_HARDENED_USERCOPY* checks, but as
this is really over my head, I could use some guidance.

The device in question is a WatchGuard Firebox M300, based on NXP's
QorIQ T2081 SoC. The OpenWrt image is built from my OpenWrt staging tree
at [1]. I realize this is very OpenWrt specific, so if any additional
info is needed, I'll gladly supply it.


[   26.333649] usercopy: Kernel memory overwrite attempt detected to SLUB object not in SLUB page?! (offset 0, size 8)!
[   26.344273] ------------[ cut here ]------------
[   26.348908] Kernel BUG at .usercopy_abort+0x94/0x9c [verbose debug info unavailable]
[   26.356667] Oops: Exception in kernel mode, sig: 5 [#1]
[   26.361899] BE PAGE_SIZE=4K SMP NR_CPUS=24 CoreNet Generic
[   26.367387] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount l2tp_ppp iptable_nat cdc_mbim xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT wireguard pppox ppp
_generic nft_redir nft_ct nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_flow_table nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netl
ink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast nf_conntrack_amanda nf_conntrack libchacha20poly1305 libblake2s ipt_REJECT cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic xt_recent xt_quota xt_policy xt_pkttype xt_physdev xt_owner xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_esp xt_ecn xt_dscp xt_comment xt_cgroup xt_bpf xt_addrtype xt_TRACE xt_TCPMSS xt_NFQUEUE xt_LOG xt_HL xt_DSCP xt_CLASSIFY xfrm_interface w83793 usbnet ts_kmp ts_fsm ts_bm slhc sch_cake ptp_qoriq
[   26.367603]  nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_quota nft_objref nft_numgen nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev nft_counter nfnetlink_queue nf_tables nf_reject_ipv4 nf_log_ipv4 nf_dup_netdev nf_defrag_ipv6 nf_defrag_ipv4 macvlan libpoly1305 libcurve25519_generic libcrc32c libchacha libblake2s_generic iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ECN ip6table_raw ip_tables crc_ccitt cdc_wdm br_netfilter sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact i2c_dev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle
[   26.455354]  ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 hwmon_vid ifb sit l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 netlink_diag tunnel6 tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha512_generic sha256_generic libsha256 sha1_generic seqiv jitterentropy_rng drbg md5 hmac echainiv deflate zlib_inflate cbc crypto_acompress leds_gpio rtc_rs5c372 ehci_platform tpm_i2c_atmel i2c_core ptp mii tpm
[   26.591596] CPU: 5 PID: 5633 Comm: fw3 Not tainted 5.10.60 #0
[   26.597357] NIP:  c000000000203f38 LR: c000000000203f34 CTR: c000000000038560
[   26.604503] REGS: c00000008d9c3550 TRAP: 0700   Not tainted  (5.10.60)
[   26.611035] MSR:  0000000080029002 <CE,EE,ME>  CR: 28004248  XER: 20000000
[   26.617924] IRQMASK: 0
[   26.617924] GPR00: c000000000203f34 c00000008d9c37e0 c000000000c6d400 0000000000000068
[   26.617924] GPR04: c0000000ffea0678 c0000000ffea5f40 0000000000000027 c0000000ffea0680
[   26.617924] GPR08: 0000000000000023 0000000000000000 0000000000000000 0000000000000001
[   26.617924] GPR12: 0000000024004448 c00000003fffdc40 0000000000000000 0000000000000000
[   26.617924] GPR16: 0000000000000000 0000000000000000 00003fff9d113880 00003fff9d20bb80
[   26.617924] GPR20: 0000000017c8a640 00003fff9d1137e0 0000000000000009 0000000000000001
[   26.617924] GPR24: 0000000000000000 00003ffff411d430 c0000000831b64e0 0000000000000008
[   26.617924] GPR28: 8000000001000008 0000000000000000 0000000000000008 8000000001000000
[   26.684450] NIP [c000000000203f38] .usercopy_abort+0x94/0x9c
[   26.690109] LR [c000000000203f34] .usercopy_abort+0x90/0x9c
[   26.695679] Call Trace:
[   26.698122] [c00000008d9c37e0] [c000000000203f34] .usercopy_abort+0x90/0x9c (unreliable)
[   26.706225] [c00000008d9c3860] [c0000000001f1820] .__check_heap_object+0x170/0x190
[   26.713800] [c00000008d9c38d0] [c0000000002040c0] .__check_object_size+0x180/0x1f0
[   26.721384] [c00000008d9c3960] [80000000004244f4] .ip_set_sockfn_get+0xb4/0x380 [ip_set]
[   26.729482] [c00000008d9c3a10] [c000000000774b18] .nf_getsockopt+0x78/0xf0
[   26.736368] [c00000008d9c3ab0] [c000000000788a3c] .ip_getsockopt+0xcc/0x120
[   26.743340] [c00000008d9c3b50] [c0000000007c358c] .raw_getsockopt+0x10c/0x1a0
[   26.750490] [c00000008d9c3be0] [c0000000006a0d1c] .sock_common_getsockopt+0x2c/0x40
[   26.758152] [c00000008d9c3c50] [c00000000069f240] .__sys_getsockopt+0xa0/0x220
[   26.765380] [c00000008d9c3d00] [c00000000069f3dc] .__se_sys_getsockopt+0x1c/0x30
[   26.772784] [c00000008d9c3d70] [c00000000000fb5c] .system_call_exception+0x11c/0x220
[   26.780534] [c00000008d9c3e20] [c000000000000678] system_call_common+0xf8/0x200
[   26.787848] --- interrupt: c00 at 0x3fff9d25915c
[   26.787848]     LR = 0x100077e4
[   26.795592] Instruction dump:
[   26.798558] 392929d0 48000014 3d02ffd5 3908ee40 7d074378 7d094378 f8c10070 7c661b78
[   26.806319] 3c62ffd5 38639ae0 4be9abf5 60000000 <0fe00000> 60000000 3d22ffde 81298ab8
[   26.814258] ---[ end trace 70b7c82100ca71f1 ]---


[   29.817617] usercopy: Kernel memory exposure attempt detected from
SLUB object not in SLUB page?! (offset 0, size 8912)!
[   29.834303] ------------[ cut here ]------------
[   29.838922] kernel BUG at mm/usercopy.c:99!
[   29.843104] Oops: Exception in kernel mode, sig: 5 [#1]
[   29.848330] BE PAGE_SIZE=4K SMP NR_CPUS=24 CoreNet Generic
[   29.853813] Modules linked in: xt_connlimit pppoe ppp_async
nf_conncount l2tp_ppp iptable_nat cdc_mbim xt_state xt_nat xt_helper
xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE
xt_FLOWOFFLOAD
 xt_CT wireguard pppox ppp_generic nft_redir nft_ct nf_nat_tftp
nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323
nf_nat_ftp nf_nat_amanda nf_nat nf_flow_table nf_conntrack_tftp
nf_conntrack_snmp
nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast
nf_conntrack_amanda nf_conntrack libchacha20poly1305 libblake2s
ipt_REJECT cdc_
ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic
xt_recent xt_quota xt_policy xt_pkttype xt_physdev xt_owner xt_multiport
xt_mark xt_mac xt_limit xt_length xt_hl xt_esp xt_ecn xt_dscp xt_comment
 xt_cgroup xt_bpf xt_addrtype xt_TRACE xt_TCPMSS xt_NFQUEUE xt_LOG xt_HL
xt_DSCP xt_CLASSIFY xfrm_interface w83793 usbnet ts_kmp ts_fsm ts_bm
slhc sch_cake ptp_qoriq
[   29.854028]  nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet
nft_reject_bridge nft_reject nft_quota nft_objref nft_numgen
nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev
nft_counter
nfnetlink_queue nf_tables nf_reject_ipv4 nf_log_ipv4 nf_dup_netdev
nf_defrag_ipv6 nf_defrag_ipv4 macvlan libpoly1305 libcurve25519_generic
libcrc32c libchacha libblake2s_generic iptable_raw iptable_mangle iptab
le_filter ipt_ah ipt_ECN ip6table_raw ip_tables crc_ccitt cdc_wdm
br_netfilter sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32
cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit
act_mi
rred act_gact i2c_dev xt_set ip_set_list_set ip_set_hash_netportnet
ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface
ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet
ip_set_hash_ipportip ip_set_
hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port
ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6
nf_log_common ip6table_mangle
[   29.941760]  ip6table_filter ip6_tables ip6t_REJECT x_tables
nf_reject_ipv6 hwmon_vid ifb sit l2tp_netlink l2tp_core udp_tunnel
ip6_udp_tunnel ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4
ah4 netli
nk_diag tunnel6 tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo
crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg
sha512_generic sha256_generic libsha256 sha1_generic seqiv jitterentrop
y_rng drbg md5 hmac echainiv deflate zlib_inflate cbc crypto_acompress
leds_gpio rtc_rs5c372 ehci_platform tpm_i2c_atmel i2c_core ptp mii tpm
[   30.077962] CPU: 5 PID: 5314 Comm: iptables Not tainted 5.10.60 #0
[   30.084144] NIP:  c000000000203f68 LR: c000000000203f64 CTR:
c0000000000087e0
[   30.091280] REGS: c00000008733b3f0 TRAP: 0700   Not tainted  (5.10.60)
[   30.097807] MSR:  0000000080029002 <CE,EE,ME>  CR: 28002448  XER:
00000000
[   30.104695] IRQMASK: 0
[   30.104695] GPR00: c000000000203f64 c00000008733b680 c000000000c70400
000000000000006c
[   30.104695] GPR04: c0000000ffea0678 c0000000ffea5f40 0000000000000027
c0000000ffea0680
[   30.104695] GPR08: 0000000000000023 0000000000000000 0000000000000000
0000000000000001
[   30.104695] GPR12: 0000000024002842 c00000003fffdc40 0000000000000028
0000000000700098
[   30.104695] GPR16: 0000000000000000 0000000000000040 0000000017e64b92
0000000010031d60
[   30.104695] GPR20: 0000000017e64a00 0000000017e64700 c000000000bb8700
c000000000c79d38
[   30.104695] GPR24: 0000000017e8ff80 00000000000022d0 c000000000bb8700
0000000000000018
[   30.104695] GPR28: 80000000010042d0 0000000000000001 00000000000022d0
8000000001002000
[   30.171224] NIP [c000000000203f68] .usercopy_abort+0x94/0x9c
[   30.176883] LR [c000000000203f64] .usercopy_abort+0x90/0x9c
[   30.182455] Call Trace:
[   30.184900] [c00000008733b680] [c000000000203f64]
.usercopy_abort+0x90/0x9c (unreliable)
[   30.193002] [c00000008733b700] [c0000000001f1850]
.__check_heap_object+0x170/0x190
[   30.200578] [c00000008733b770] [c0000000002040f0]
.__check_object_size+0x180/0x1f0
[   30.208160] [c00000008733b800] [8000000000403ee4]
.__do_replace+0x2d4/0x3b0 [ip_tables]
[   30.216170] [c00000008733b8d0] [8000000000405d7c]
.do_ipt_set_ctl+0x48c/0x530 [ip_tables]
[   30.224355] [c00000008733b9d0] [c000000000774bf4]
.nf_setsockopt+0x84/0x100
[   30.231329] [c00000008733ba70] [c00000000078a0b4]
.ip_setsockopt+0x524/0x1920
[   30.238475] [c00000008733bb60] [c0000000007c36dc]
.raw_setsockopt+0xdc/0x110
[   30.245537] [c00000008733bbf0] [c0000000006a0dfc]
.sock_common_setsockopt+0x2c/0x40
[   30.253200] [c00000008733bc60] [c00000000069f06c]
.__sys_setsockopt+0xbc/0x1e0
[   30.260428] [c00000008733bd00] [c00000000069f1b0]
.__se_sys_setsockopt+0x20/0x30
[   30.267831] [c00000008733bd70] [c00000000000fb5c]
.system_call_exception+0x11c/0x220
[   30.275581] [c00000008733be20] [c000000000000678]
system_call_common+0xf8/0x200
[   30.282895] --- interrupt: c00 at 0x3fffaa864714
[   30.282895]     LR = 0x3fffaa7815a0
[   30.290987] Instruction dump:
[   30.293953] 39290e90 48000014 3d02ffd5 3908d1c0 7d074378 7d094378
f8c10070 7c661b78
[   30.301715] 3c62ffd4 38637a80 4be9abf5 60000000 <0fe00000> 60000000
3d22ffde 81298838
[   30.309654] ---[ end trace 924262dfe54d8433 ]---


[   36.560294] usercopy: Kernel memory exposure attempt detected from
SLUB object not in SLUB page?! (offset 0, size 8880)!
[   36.575184] ------------[ cut here ]------------
[   36.579801] kernel BUG at mm/usercopy.c:99!
[   36.583984] Oops: Exception in kernel mode, sig: 5 [#1]
[   36.589209] BE PAGE_SIZE=4K SMP NR_CPUS=24 CoreNet Generic
[   36.594692] Modules linked in: xt_connlimit pppoe ppp_async
nf_conncount l2tp_ppp iptable_nat cdc_mbim xt_state xt_nat xt_helper
xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE
xt_FLOWOFFLOAD
 xt_CT wireguard pppox ppp_generic nft_redir nft_ct nf_nat_tftp
nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323
nf_nat_ftp nf_nat_amanda nf_nat nf_flow_table nf_conntrack_tftp
nf_conntrack_snmp
nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast
nf_conntrack_amanda nf_conntrack libchacha20poly1305 libblake2s
ipt_REJECT cdc_
ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic
xt_recent xt_quota xt_policy xt_pkttype xt_physdev xt_owner xt_multiport
xt_mark xt_mac xt_limit xt_length xt_hl xt_esp xt_ecn xt_dscp xt_comment
 xt_cgroup xt_bpf xt_addrtype xt_TRACE xt_TCPMSS xt_NFQUEUE xt_LOG xt_HL
xt_DSCP xt_CLASSIFY xfrm_interface w83793 usbnet ts_kmp ts_fsm ts_bm
slhc sch_cake ptp_qoriq
[   36.594911]  nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet
nft_reject_bridge nft_reject nft_quota nft_objref nft_numgen
nft_meta_bridge nft_log nft_limit nft_hash nft_fwd_netdev nft_dup_netdev
nft_counter
nfnetlink_queue nf_tables nf_reject_ipv4 nf_log_ipv4 nf_dup_netdev
nf_defrag_ipv6 nf_defrag_ipv4 macvlan libpoly1305 libcurve25519_generic
libcrc32c libchacha libblake2s_generic iptable_raw iptable_mangle iptab
le_filter ipt_ah ipt_ECN ip6table_raw ip_tables crc_ccitt cdc_wdm
br_netfilter sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32
cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit
act_mi
rred act_gact i2c_dev xt_set ip_set_list_set ip_set_hash_netportnet
ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface
ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet
ip_set_hash_ipportip ip_set_
hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port
ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6
nf_log_common ip6table_mangle
[   36.682648]  ip6table_filter ip6_tables ip6t_REJECT x_tables
nf_reject_ipv6 hwmon_vid ifb sit l2tp_netlink l2tp_core udp_tunnel
ip6_udp_tunnel ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4
ah4 netli
nk_diag tunnel6 tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo
crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg
sha512_generic sha256_generic libsha256 sha1_generic seqiv jitterentrop
y_rng drbg md5 hmac echainiv deflate zlib_inflate cbc crypto_acompress
leds_gpio rtc_rs5c372 ehci_platform tpm_i2c_atmel i2c_core ptp mii tpm
[   36.818853] CPU: 5 PID: 5145 Comm: fw3 Not tainted 5.10.60 #0
[   36.824599] NIP:  c000000000203f68 LR: c000000000203f64 CTR:
c0000000000087e0
[   36.831736] REGS: c000000083337430 TRAP: 0700   Not tainted  (5.10.60)
[   36.838263] MSR:  0000000080029002 <CE,EE,ME>  CR: 28002448  XER:
00000000
[   36.845153] IRQMASK: 0
[   36.845153] GPR00: c000000000203f64 c0000000833376c0 c000000000c70400
000000000000006c
[   36.845153] GPR04: c0000000ffea0678 c0000000ffea5f40 0000000000000027
c0000000ffea0680
[   36.845153] GPR08: 0000000000000023 0000000000000000 0000000000000000
fffffffffffea968
[   36.845153] GPR12: 0000000024002842 c00000003fffdc40 0000000000000028
0000000000a800d0
[   36.845153] GPR16: 0000000000000000 0000000000000040 00003fff847a2860
0000000046a8c240
[   36.845153] GPR20: 00000000469ed0c0 0000000046affc60 0000000046adbf70
c000000000c79d38
[   36.845153] GPR24: 0000000046adc220 00000000000022b0 c000000000bb8700
0000000000000018
[   36.845153] GPR28: 80000000010042b0 0000000000000001 00000000000022b0
8000000001002000
[   36.911686] NIP [c000000000203f68] .usercopy_abort+0x94/0x9c
[   36.917345] LR [c000000000203f64] .usercopy_abort+0x90/0x9c
[   36.922917] Call Trace:
[   36.925361] [c0000000833376c0] [c000000000203f64]
.usercopy_abort+0x90/0x9c (unreliable)
[   36.933463] [c000000083337740] [c0000000001f1850]
.__check_heap_object+0x170/0x190
[   36.941038] [c0000000833377b0] [c0000000002040f0]
.__check_object_size+0x180/0x1f0
[   36.948618] [c000000083337840] [8000000000677504]
.__do_replace+0x2d4/0x3b0 [ip6_tables]
[   36.956715] [c000000083337910] [8000000000679abc]
.do_ip6t_set_ctl+0x48c/0x530 [ip6_tables]
[   36.965074] [c000000083337a10] [c000000000774bf4]
.nf_setsockopt+0x84/0x100
[   36.972048] [c000000083337ab0] [c00000000085e498]
.ipv6_setsockopt+0x128/0x130
[   36.979280] [c000000083337b50] [c000000000868f18]
.rawv6_setsockopt+0x58/0x290
[   36.986518] [c000000083337bf0] [c0000000006a0dfc]
.sock_common_setsockopt+0x2c/0x40
[   36.994179] [c000000083337c60] [c00000000069f06c]
.__sys_setsockopt+0xbc/0x1e0
[   37.001407] [c000000083337d00] [c00000000069f1b0]
.__se_sys_setsockopt+0x20/0x30
[   37.008812] [c000000083337d70] [c00000000000fb5c]
.system_call_exception+0x11c/0x220
[   37.016561] [c000000083337e20] [c000000000000678]
system_call_common+0xf8/0x200
[   37.023876] --- interrupt: c00 at 0x3fff8490e714
[   37.023876]     LR = 0x3fff847fa738
[   37.031969] Instruction dump:
[   37.034934] 39290e90 48000014 3d02ffd5 3908d1c0 7d074378 7d094378
f8c10070 7c661b78
[   37.042696] 3c62ffd4 38637a80 4be9abf5 60000000 <0fe00000> 60000000
3d22ffde 81298838
[   37.050636] ---[ end trace 1a74d40a19fa7b96 ]---

Thanks,
Stijn

[1]
https://git.openwrt.org/?p=openwrt/staging/stintel.git;a=shortlog;h=refs/heads/qoriq