[bug report] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000058

[bug report] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000058

[Changhui Zhong]

Hello,
Below issue was triggered with ( v6.0.15-996-g988abd970566), pls help check it

[ 7845.648246] Unable to handle kernel NULL pointer dereference at
virtual address 0000000000000058
[ 7845.648776] Mem abort info:
[ 7845.648938]   ESR = 0x0000000096000004
[ 7845.649155]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 7845.649462]   SET = 0, FnV = 0
[ 7845.649639]   EA = 0, S1PTW = 0
[ 7845.649821]   FSC = 0x04: level 0 translation fault
[ 7845.650105] Data abort info:
[ 7845.650274]   ISV = 0, ISS = 0x00000004
[ 7845.650496]   CM = 0, WnR = 0
[ 7845.650670] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000103cba000
[ 7845.651043] [0000000000000058] pgd=0000000000000000, p4d=0000000000000000
[ 7845.651446] Internal error: Oops: 96000004 [#1] SMP
[ 7845.651764] Modules linked in: snd_aloop snd_dummy snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore ansi_cprng crypto_user
veth vrf ipvlan echainiv esp4 des_generic libdes tun geneve ip6_tables
ip_vs ip_set xt_sctp nf_conntrack_netlink nft_chain_nat xt_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables
nfnetlink tcp_dctcp ah6 ah4 binfmt_misc can_j1939 l2tp_core bnep hidp
can_bcm pptp gre can_raw rfcomm bluetooth ieee802154_socket ieee802154
af_key qrtr pppoe pppox ppp_generic slhc mpls_router ip_tunnel
vsock_loopback vmw_vsock_virtio_transport_common
vmw_vsock_vmci_transport vmw_vmci vsock fcrypt pcbc rxrpc smc ib_core
kcm can macsec llc sctp ip6_udp_tunnel udp_tunnel mlx4_en mlx4_core
nfp tls loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver fscache
netfs rfkill sunrpc vfat fat virtio_net net_failover failover fuse
zram xfs crct10dif_ce polyval_ce virtio_console polyval_generic
ghash_ce virtio_blk virtio_mmio qemu_fw_cfg [last unloaded: vxlan]
[ 7845.656785] CPU: 5 PID: 789199 Comm: bash Not tainted 6.0.15 #1
[ 7845.657126] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 7845.657523] pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 7845.657932] pc : blk_mq_quiesce_queue+0x50/0xa0
[ 7845.658208] lr : blk_mq_quiesce_queue+0x4c/0xa0
[ 7845.658489] sp : ffff80000dcdbb30
[ 7845.658696] x29: ffff80000dcdbb30 x28: ffff0007fe9cc480 x27: 0000000000000000
[ 7845.659137] x26: 0000000000000000 x25: 0000000000001000 x24: ffff0000c2e4d140
[ 7845.659577] x23: ffffb77a8eef4db0 x22: ffffb77a8eeeee50 x21: ffff0000ca8042f8
[ 7845.660019] x20: 0000000000000000 x19: ffff0000ca804280 x18: 0000000000000000
[ 7845.660457] x17: 0000000000000000 x16: 0000000000000000 x15: 0000aaaacb2fb7c0
[ 7845.660899] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 7845.661348] x11: ffffb77a8eeef6d8 x10: 0000000000001db0 x9 : ffffb77a8d727ab8
[ 7845.661788] x8 : ffff80000dcdbac8 x7 : 0000000000000000 x6 : ffff80000dcdbb20
[ 7845.662230] x5 : ffff80000dcd8000 x4 : ffff80000dcdbb10 x3 : ffff0000ca8042e8
[ 7845.662671] x2 : 0000000000000001 x1 : ffff0007fe9cc480 x0 : 0000000000000000
[ 7845.663110] Call trace:
[ 7845.663264]  blk_mq_quiesce_queue+0x50/0xa0
[ 7845.663524]  del_gendisk+0x1e4/0x32c
[ 7845.663748]  zram_remove+0xd0/0xec [zram]
[ 7845.664003]  hot_remove_store+0x6c/0x100 [zram]
[ 7845.664286]  class_attr_store+0x24/0x40
[ 7845.664527]  sysfs_kf_write+0x4c/0x5c
[ 7845.664758]  kernfs_fop_write_iter+0x120/0x1f0
[ 7845.665032]  vfs_write+0x1cc/0x380
[ 7845.665245]  ksys_write+0x68/0xf0
[ 7845.665452]  __arm64_sys_write+0x24/0x30
[ 7845.665696]  invoke_syscall+0x78/0x100
[ 7845.665931]  el0_svc_common.constprop.0+0x4c/0xf4
[ 7845.666225]  do_el0_svc+0x34/0x4c
[ 7845.666432]  el0_svc+0x34/0x10c
[ 7845.666631]  el0t_64_sync_handler+0xf4/0x120
[ 7845.666898]  el0t_64_sync+0x190/0x194
[ 7845.667129] Code: aa1403e1 aa1503e0 94257716 f9419660 (b9405801)
[ 7845.667507] ---[ end trace 0000000000000000 ]---
[ 7845.667794] Kernel panic - not syncing: Oops: Fatal exception
[ 7845.668150] SMP: stopping secondary CPUs
[ 7845.668403] Kernel Offset: 0x377a846d0000 from 0xffff800008000000
[ 7845.668770] PHYS_OFFSET: 0x40000000
[ 7845.668971] CPU features: 0x0000,0085c021,19805c82
[ 7845.669251] Memory Limit: none
[ 7845.669426] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---


--
Best Regards,
Changhui Zhong

Re: [bug report] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000058

[Ming Lei]

On Mon, Dec 26, 2022 at 11:11:44AM +0800, Changhui Zhong wrote:
> Hello,
> Below issue was triggered with ( v6.0.15-996-g988abd970566), pls help check it
> 
> [ 7845.648246] Unable to handle kernel NULL pointer dereference at
> virtual address 0000000000000058
> [ 7845.648776] Mem abort info:
> [ 7845.648938]   ESR = 0x0000000096000004
> [ 7845.649155]   EC = 0x25: DABT (current EL), IL = 32 bits
> [ 7845.649462]   SET = 0, FnV = 0
> [ 7845.649639]   EA = 0, S1PTW = 0
> [ 7845.649821]   FSC = 0x04: level 0 translation fault
> [ 7845.650105] Data abort info:
> [ 7845.650274]   ISV = 0, ISS = 0x00000004
> [ 7845.650496]   CM = 0, WnR = 0
> [ 7845.650670] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000103cba000
> [ 7845.651043] [0000000000000058] pgd=0000000000000000, p4d=0000000000000000
> [ 7845.651446] Internal error: Oops: 96000004 [#1] SMP
> [ 7845.651764] Modules linked in: snd_aloop snd_dummy snd_seq
> snd_seq_device snd_pcm snd_timer snd soundcore ansi_cprng crypto_user
> veth vrf ipvlan echainiv esp4 des_generic libdes tun geneve ip6_tables
> ip_vs ip_set xt_sctp nf_conntrack_netlink nft_chain_nat xt_nat nf_nat
> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables
> nfnetlink tcp_dctcp ah6 ah4 binfmt_misc can_j1939 l2tp_core bnep hidp
> can_bcm pptp gre can_raw rfcomm bluetooth ieee802154_socket ieee802154
> af_key qrtr pppoe pppox ppp_generic slhc mpls_router ip_tunnel
> vsock_loopback vmw_vsock_virtio_transport_common
> vmw_vsock_vmci_transport vmw_vmci vsock fcrypt pcbc rxrpc smc ib_core
> kcm can macsec llc sctp ip6_udp_tunnel udp_tunnel mlx4_en mlx4_core
> nfp tls loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver fscache
> netfs rfkill sunrpc vfat fat virtio_net net_failover failover fuse
> zram xfs crct10dif_ce polyval_ce virtio_console polyval_generic
> ghash_ce virtio_blk virtio_mmio qemu_fw_cfg [last unloaded: vxlan]
> [ 7845.656785] CPU: 5 PID: 789199 Comm: bash Not tainted 6.0.15 #1
> [ 7845.657126] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
> [ 7845.657523] pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 7845.657932] pc : blk_mq_quiesce_queue+0x50/0xa0

Hi Changhui,

Can you figure out the fault source code by gdb?

gdb vmlinux
gdb> l *(blk_mq_quiesce_queue+0x50)


thanks, 
Ming

Re: [bug report] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000058

[Ming Lei]

Hi Changhui,

On Mon, Dec 26, 2022 at 11:11:44AM +0800, Changhui Zhong wrote:
> Hello,
> Below issue was triggered with ( v6.0.15-996-g988abd970566), pls help check it

There isn't commit 988abd970566 in linux-6.0.y, so I guess the above
build must integrate other patches not in 6.0.y

From the source code in cki build[1], looks commit 80bd4a7aab4c ("blk-mq: move
the srcu_struct used for quiescing to the tagset") is included, but
commit 8537380bb988 ("blk-mq: skip non-mq queues in blk_mq_quiesce_queue")
is missed, that is why this panic is triggered.

BTW, if possible, I'd suggest to share kernel tree being tested in cki test
if non official kernel is tested.


[1] https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/731863135/build%20aarch64/3522994262/artifacts/kernel-stable-queue-redhat_731863135_aarch64.tar.gz


thanks,
Ming

Re: [bug report] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000058

[Ming Lei]

Hi Greg,

On Tue, Dec 27, 2022 at 04:00:37PM +0800, Ming Lei wrote:
> Hi Changhui,
> 
> On Mon, Dec 26, 2022 at 11:11:44AM +0800, Changhui Zhong wrote:
> > Hello,
> > Below issue was triggered with ( v6.0.15-996-g988abd970566), pls help check it
> 
> There isn't commit 988abd970566 in linux-6.0.y, so I guess the above
> build must integrate other patches not in 6.0.y
> 
> From the source code in cki build[1], looks commit 80bd4a7aab4c ("blk-mq: move
> the srcu_struct used for quiescing to the tagset") is included, but
> commit 8537380bb988 ("blk-mq: skip non-mq queues in blk_mq_quiesce_queue")
> is missed, that is why this panic is triggered.

I just found that patch of blk-mq-move-the-srcu_struct-used-for-quiescing-to-th.patch
is queued in stable-queue/queue-6.0, but that patch depends on
commit 8537380bb988 ("blk-mq: skip non-mq queues in blk_mq_quiesce_queue").
which needs to be added to queue-6.0 too.


Thanks,
Ming

Re: [bug report] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000058

[Changhui Zhong]

On Tue, Dec 27, 2022 at 4:00 PM Ming Lei <ming.lei@redhat.com> wrote:
>
> Hi Changhui,
>
> On Mon, Dec 26, 2022 at 11:11:44AM +0800, Changhui Zhong wrote:
> > Hello,
> > Below issue was triggered with ( v6.0.15-996-g988abd970566), pls help check it
>
> There isn't commit 988abd970566 in linux-6.0.y, so I guess the above
> build must integrate other patches not in 6.0.y
>
> From the source code in cki build[1], looks commit 80bd4a7aab4c ("blk-mq: move
> the srcu_struct used for quiescing to the tagset") is included, but
> commit 8537380bb988 ("blk-mq: skip non-mq queues in blk_mq_quiesce_queue")
> is missed, that is why this panic is triggered.
>
> BTW, if possible, I'd suggest to share kernel tree being tested in cki test
> if non official kernel is tested.
>
>
> [1] https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/731863135/build%20aarch64/3522994262/artifacts/kernel-stable-queue-redhat_731863135_aarch64.tar.gz
>
>
> thanks,
> Ming
>


Hi,Ming

thanks for your investigation,
I first saw this issue at
https://datawarehouse.cki-project.org/kcidb/checkouts/64206#issue-1783,
I'm not familiar with which repo CKI use to compile, but I see
repository is https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git@queue/6.0

Thanks，
Changhui