5.15.0-rc1 armv7hl Workqueue: btrfs-delalloc btrfs_work_helper PC is at mmiocpy LR is at ZSTD_compressStream

5.15.0-rc1 armv7hl Workqueue: btrfs-delalloc btrfs_work_helper PC is at mmiocpy LR is at ZSTD_compressStream

[Chris Murphy]

https://kojipkgs.fedoraproject.org//work/tasks/8820/75928820/oz-armhfp.log

The oops happens about 80% of the way down. This is a snippet.


00:09:50,669 ALERT kernel:8<--- cut here ---
00:09:50,669 ALERT kernel:Unable to handle kernel NULL pointer
dereference at virtual address 00000000
00:09:50,669 ALERT kernel:pgd = b0c5a5c4
00:09:50,670 ALERT kernel:[00000000] *pgd=00000000
00:09:50,670 EMERG kernel:Internal error: Oops: 5 [#1] SMP ARM
00:09:50,670 WARNING kernel:Modules linked in: vfat fat libfc
scsi_transport_fc uinput scsi_dh_rdac scsi_dh_emc scsi_dh_alua joydev
fuse zram overlay loop virtio_gpu virtio_dma_buf drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops cec gpio_keys drm
virtio_mmio virtio_net virtio_blk virtio_console net_failover failover
uhci_hcd rfkill sunrpc lrw dm_crypt trusted asn1_encoder tee
dm_round_robin dm_multipath linear raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx raid1 raid0 squashfs cramfs
be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi
libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi
00:09:50,670 WARNING kernel:CPU: 1 PID: 134 Comm: kworker/u8:2 Not
tainted 5.15.0-0.rc1.20210917gitbdb575f87217.15.fc36.armv7hl #1
00:09:50,671 WARNING kernel:Hardware name: QEMU KVM Virtual Machine,
BIOS 0.0.0 02/06/2015
00:09:50,671 WARNING kernel:Workqueue: btrfs-delalloc btrfs_work_helper
00:09:50,671 WARNING kernel:PC is at mmiocpy+0x48/0x330
00:09:50,671 WARNING kernel:LR is at ZSTD_compressStream+0x184/0x294
00:09:50,671 WARNING kernel:pc : [<c08bd768>]    lr : [<c089c314>]
psr: 200e0013
00:09:50,672 WARNING kernel:sp : c2a67d14  ip : 00000000  fp : 00000000
00:09:50,672 WARNING kernel:r10: 00000200  r9 : dcb27000  r8 : 00000400
00:09:50,672 WARNING kernel:r7 : 00000200  r6 : c503eb28  r5 :
c503eb34  r4 : df800008
00:09:50,672 WARNING kernel:r3 : 00000200  r2 : 00000180  r1 :
00000000  r0 : df8027c0
00:09:50,672 WARNING kernel:Flags: nzCv  IRQs on  FIQs on  Mode SVC_32
 ISA ARM  Segment none
00:09:50,672 WARNING kernel:Control: 10c5383d  Table: 4497406a  DAC: 00000051
00:09:50,673 ALERT kernel:Register r0 information: non-slab/vmalloc memory
00:09:50,673 ALERT kernel:Register r1 information: NULL pointer
00:09:50,673 ALERT kernel:Register r2 information: non-paged memory
00:09:50,673 ALERT kernel:Register r3 information: non-paged memory
00:09:50,673 ALERT kernel:Register r4 information: non-slab/vmalloc memory
00:09:50,673 ALERT kernel:Register r5 information: slab kmalloc-64
start c503eb00 pointer offset 52 size 64
00:09:50,674 ALERT kernel:Register r6 information: slab kmalloc-64
start c503eb00 pointer offset 40 size 64
00:09:50,674 ALERT kernel:Register r7 information: non-paged memory
00:09:50,674 ALERT kernel:Register r8 information: non-paged memory
00:09:50,674 ALERT kernel:Register r9 information: non-slab/vmalloc memory
00:09:50,674 ALERT kernel:Register r10 information: non-paged memory
00:09:50,674 ALERT kernel:Register r11 information: NULL pointer
00:09:50,675 ALERT kernel:Register r12 information: NULL pointer
00:09:50,675 EMERG kernel:Process kworker/u8:2 (pid: 134, stack limit
= 0xcad4b4eb)
00:09:50,675 EMERG kernel:Stack: (0xc2a67d14 to 0xc2a68000)
00:09:50,675 EMERG kernel:7d00:
      ???????? ???????? ????????
00:09:50,675 EMERG kernel:7d20: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,675 EMERG kernel:7d40: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,676 EMERG kernel:7d60: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,676 EMERG kernel:7d80: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,676 EMERG kernel:7da0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,676 EMERG kernel:7dc0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,676 EMERG kernel:7de0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,677 EMERG kernel:7e00: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,677 EMERG kernel:7e20: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,677 EMERG kernel:7e40: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,677 EMERG kernel:7e60: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,677 EMERG kernel:7e80: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,677 EMERG kernel:7ea0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,678 EMERG kernel:7ec0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,678 EMERG kernel:7ee0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,678 EMERG kernel:7f00: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,678 EMERG kernel:7f20: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,678 EMERG kernel:7f40: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,678 EMERG kernel:7f60: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,679 EMERG kernel:7f80: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,679 EMERG kernel:7fa0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,679 EMERG kernel:7fc0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,679 EMERG kernel:7fe0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,679 EMERG kernel:[<c08bd768>] (mmiocpy) from [<c089c314>]
(ZSTD_compressStream+0x184/0x294)
00:09:50,679 EMERG kernel:[<c089c314>] (ZSTD_compressStream) from
[<c0737980>] (zstd_compress_pages+0x174/0x390)
00:09:50,680 EMERG kernel:[<c0737980>] (zstd_compress_pages) from
[<c0739ba4>] (btrfs_compress_pages+0x104/0x10c)
00:09:50,680 EMERG kernel:[<c0739ba4>] (btrfs_compress_pages) from
[<c06f1df8>] (compress_file_range+0x25c/0x5dc)
00:09:50,680 EMERG kernel:[<c06f1df8>] (compress_file_range) from
[<c06f2190>] (async_cow_start+0x18/0x30)
00:09:50,680 EMERG kernel:[<c06f2190>] (async_cow_start) from
[<c071c704>] (btrfs_work_helper+0x1ac/0x39c)
00:09:50,680 EMERG kernel:[<c071c704>] (btrfs_work_helper) from
[<c036d7c4>] (process_one_work+0x2f8/0x4f4)
00:09:50,680 EMERG kernel:[<c036d7c4>] (process_one_work) from
[<c036e4f0>] (worker_thread+0x234/0x2f4)
00:09:50,681 EMERG kernel:[<c036e4f0>] (worker_thread) from
[<c037508c>] (kthread+0x134/0x140)
00:09:50,681 EMERG kernel:[<c037508c>] (kthread) from [<c0300160>]
(ret_from_fork+0x14/0x34)
00:09:50,681 EMERG kernel:Exception stack(0xc2a67fb0 to 0xc2a67ff8)
00:09:50,681 EMERG kernel:7fa0:
???????? ???????? ???????? ????????
00:09:50,681 EMERG kernel:7fc0: ???????? ???????? ???????? ????????
???????? ???????? ???????? ????????
00:09:50,681 EMERG kernel:7fe0: ???????? ???????? ???????? ????????
???????? ????????
00:09:50,682 EMERG kernel:Code: ba000002 f5d1f03c f5d1f05c f5d1f07c (e8b151f8)
00:09:50,682 WARNING kernel:---[ end trace 9051d8c5bdb7aaf3 ]---



-- 
Chris Murphy

Re: 5.15.0-rc1 armv7hl Workqueue: btrfs-delalloc btrfs_work_helper PC is at mmiocpy LR is at ZSTD_compressStream

[Chris Murphy]

Downstream bug is in Fedora Rawhide

https://bugzilla.redhat.com/show_bug.cgi?id=2006295

Re: 5.15.0-rc1 armv7hl Workqueue: btrfs-delalloc btrfs_work_helper PC is at mmiocpy LR is at ZSTD_compressStream

[David Sterba]

On Mon, Sep 20, 2021 at 06:13:07PM -0600, Chris Murphy wrote:
> https://kojipkgs.fedoraproject.org//work/tasks/8820/75928820/oz-armhfp.log

> 00:09:50,679 EMERG kernel:[<c08bd768>] (mmiocpy) from [<c089c314>]
> (ZSTD_compressStream+0x184/0x294)

The last function to fail is inside ZSTD implementation in kernel.
If btrfs passes wrong data to zstd we'd see that also on non-arm builds,
so guessing by mmiocpy as a copy-something function it could be some
sort of alignment problem that works on x86_64 but throws an exception
on arm as it's stricter about alignment.

Re: 5.15.0-rc1 armv7hl Workqueue: btrfs-delalloc btrfs_work_helper PC is at mmiocpy LR is at ZSTD_compressStream

[Su Yue]

On Tue 21 Sep 2021 at 21:56, David Sterba <dsterba@suse.cz> wrote:

> On Mon, Sep 20, 2021 at 06:13:07PM -0600, Chris Murphy wrote:
>> https://kojipkgs.fedoraproject.org//work/tasks/8820/75928820/oz-armhfp.log
>
>> 00:09:50,679 EMERG kernel:[<c08bd768>] (mmiocpy) from 
>> [<c089c314>]
>> (ZSTD_compressStream+0x184/0x294)
>
> The last function to fail is inside ZSTD implementation in 
> kernel.
> If btrfs passes wrong data to zstd we'd see that also on non-arm 
> builds,
> so guessing by mmiocpy as a copy-something function it could be 
> some
> sort of alignment problem that works on x86_64 but throws an 
> exception
> on arm as it's stricter about alignment.

It's about 32bits platforms. There is another report about x86 
cpu.
And it's reproduciable by running btrfs/004 (lzo and zstd)in x86 
VM.
In my arm64, all things are fine.
And the bisection result is the highmem related patchset
'[PATCH 0/6] Remove highmem allocations, kmap/kunmap'.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=214565

--
Su

Re: 5.15.0-rc1 armv7hl Workqueue: btrfs-delalloc btrfs_work_helper PC is at mmiocpy LR is at ZSTD_compressStream

[David Sterba]

On Mon, Oct 04, 2021 at 07:57:25PM +0800, Su Yue wrote:
> 
> On Tue 21 Sep 2021 at 21:56, David Sterba <dsterba@suse.cz> wrote:
> 
> > On Mon, Sep 20, 2021 at 06:13:07PM -0600, Chris Murphy wrote:
> >> https://kojipkgs.fedoraproject.org//work/tasks/8820/75928820/oz-armhfp.log
> >
> >> 00:09:50,679 EMERG kernel:[<c08bd768>] (mmiocpy) from 
> >> [<c089c314>]
> >> (ZSTD_compressStream+0x184/0x294)
> >
> > The last function to fail is inside ZSTD implementation in 
> > kernel.
> > If btrfs passes wrong data to zstd we'd see that also on non-arm 
> > builds,
> > so guessing by mmiocpy as a copy-something function it could be 
> > some
> > sort of alignment problem that works on x86_64 but throws an 
> > exception
> > on arm as it's stricter about alignment.
> 
> It's about 32bits platforms. There is another report about x86 
> cpu.
> And it's reproduciable by running btrfs/004 (lzo and zstd)in x86 
> VM.
> In my arm64, all things are fine.
> And the bisection result is the highmem related patchset
> '[PATCH 0/6] Remove highmem allocations, kmap/kunmap'.

I must be missing something then, if the pages are not allocated from
HIGHMEM we don't need kmap. If the pages come from outside and are
potentially allocated from HIGHMEM kmap is used, that's for all the
file/inode pages.