* Re: [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-25 11:55 [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0 bugzilla-daemon
@ 2022-09-26 5:02 ` Theodore Ts'o
2022-09-27 18:10 ` Ritesh Harjani (IBM)
2022-09-26 5:02 ` [Bug 216529] " bugzilla-daemon
` (4 subsequent siblings)
5 siblings, 1 reply; 10+ messages in thread
From: Theodore Ts'o @ 2022-09-26 5:02 UTC (permalink / raw)
To: bugzilla-daemon; +Cc: linux-ext4, Ritesh Harjani (IBM)
On Sun, Sep 25, 2022 at 11:55:29AM +0000, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=216529
>
>
> Hit a panic on ppc64le, by running generic/048 with 1k block size:
Hmm, does this reproduce reliably for you? I test with a 1k block
size on x86_64 as a proxy 4k block sizes on PPC64, where the blocksize
< pagesize... and this isn't reproducing for me on x86, and I don't
have access to a PPC64LE system.
Ritesh, is this something you can take a look at it? Thanks!
- Ted
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-26 5:02 ` Theodore Ts'o
@ 2022-09-27 18:10 ` Ritesh Harjani (IBM)
2022-10-10 17:01 ` Ritesh Harjani (IBM)
0 siblings, 1 reply; 10+ messages in thread
From: Ritesh Harjani (IBM) @ 2022-09-27 18:10 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: bugzilla-daemon, linux-ext4
On 22/09/26 01:02AM, Theodore Ts'o wrote:
> On Sun, Sep 25, 2022 at 11:55:29AM +0000, bugzilla-daemon@kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=216529
> >
> >
> > Hit a panic on ppc64le, by running generic/048 with 1k block size:
>
> Hmm, does this reproduce reliably for you? I test with a 1k block
> size on x86_64 as a proxy 4k block sizes on PPC64, where the blocksize
> < pagesize... and this isn't reproducing for me on x86, and I don't
> have access to a PPC64LE system.
>
> Ritesh, is this something you can take a look at it? Thanks!
I was away for some personal work for last few days, but I am back to work from
today. Sure, I will take a look at this and will get back.
I did give this test a couple of runs though, but wasn't able to reproduce it.
But let me try few more things along with more iterations. Will update
accordingly.
-ritesh
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-27 18:10 ` Ritesh Harjani (IBM)
@ 2022-10-10 17:01 ` Ritesh Harjani (IBM)
0 siblings, 0 replies; 10+ messages in thread
From: Ritesh Harjani (IBM) @ 2022-10-10 17:01 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: bugzilla-daemon, linux-ext4
On 22/09/27 11:40PM, Ritesh Harjani (IBM) wrote:
> On 22/09/26 01:02AM, Theodore Ts'o wrote:
> > On Sun, Sep 25, 2022 at 11:55:29AM +0000, bugzilla-daemon@kernel.org wrote:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=216529
> > >
> > >
> > > Hit a panic on ppc64le, by running generic/048 with 1k block size:
> >
> > Hmm, does this reproduce reliably for you? I test with a 1k block
> > size on x86_64 as a proxy 4k block sizes on PPC64, where the blocksize
> > < pagesize... and this isn't reproducing for me on x86, and I don't
> > have access to a PPC64LE system.
> >
> > Ritesh, is this something you can take a look at it? Thanks!
>
> I was away for some personal work for last few days, but I am back to work from
> today. Sure, I will take a look at this and will get back.
>
> I did give this test a couple of runs though, but wasn't able to reproduce it.
> But let me try few more things along with more iterations. Will update
> accordingly.
I thought I had updated this. But I guess I forgot to update on this mail
thread...
I tested this for quite some time in a loop and also gave it a overnight run,
but I couldn't hit this issue. I had kept low memory size guest, so that we
could see more reclaim activity (which I also ensured by doing perf trace to see
if we are going over that path or not while test was running).
I am not sure whether this could be a timing issue or what. Maybe if you could
share your defconfig, I could give a try with that on my setup once.
-ritesh
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Bug 216529] [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-25 11:55 [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0 bugzilla-daemon
2022-09-26 5:02 ` Theodore Ts'o
@ 2022-09-26 5:02 ` bugzilla-daemon
2022-09-27 0:47 ` bugzilla-daemon
` (3 subsequent siblings)
5 siblings, 0 replies; 10+ messages in thread
From: bugzilla-daemon @ 2022-09-26 5:02 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=216529
--- Comment #1 from Theodore Tso (tytso@mit.edu) ---
On Sun, Sep 25, 2022 at 11:55:29AM +0000, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=216529
>
>
> Hit a panic on ppc64le, by running generic/048 with 1k block size:
Hmm, does this reproduce reliably for you? I test with a 1k block
size on x86_64 as a proxy 4k block sizes on PPC64, where the blocksize
< pagesize... and this isn't reproducing for me on x86, and I don't
have access to a PPC64LE system.
Ritesh, is this something you can take a look at it? Thanks!
- Ted
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Bug 216529] [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-25 11:55 [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0 bugzilla-daemon
2022-09-26 5:02 ` Theodore Ts'o
2022-09-26 5:02 ` [Bug 216529] " bugzilla-daemon
@ 2022-09-27 0:47 ` bugzilla-daemon
2022-09-27 18:06 ` Theodore Ts'o
2022-09-27 18:06 ` bugzilla-daemon
` (2 subsequent siblings)
5 siblings, 1 reply; 10+ messages in thread
From: bugzilla-daemon @ 2022-09-27 0:47 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=216529
--- Comment #2 from Zorro Lang (zlang@redhat.com) ---
(In reply to Theodore Tso from comment #1)
> On Sun, Sep 25, 2022 at 11:55:29AM +0000, bugzilla-daemon@kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=216529
> >
> >
> > Hit a panic on ppc64le, by running generic/048 with 1k block size:
>
> Hmm, does this reproduce reliably for you? I test with a 1k block
> size on x86_64 as a proxy 4k block sizes on PPC64, where the blocksize
> < pagesize... and this isn't reproducing for me on x86, and I don't
> have access to a PPC64LE system.
Hi Ted,
Yes, it's reproducible for me, I just reproduced it again on another ppc64le
(P8) machine [1]. But it's not easy to reproduce by running generic/048 (maybe
there's a better way to reproduce it).
And this time the call trace is a little different, it might be a folio [mm]
related bug? Maybe I should cc linux-mm list to get more checking?
Thanks,
Zorro
[ 1254.857035] run fstests generic/048 at 2022-09-26 12:12:26
[ 1257.651002] EXT4-fs (sda3): mounted filesystem with ordered data mode. Quota
mode: none.
[ 1257.666754] EXT4-fs (sda3): shut down requested (1)
[ 1257.666773] Aborting journal on device sda3-8.
[ 1257.696046] EXT4-fs (sda3): unmounting filesystem.
[ 1259.216580] EXT4-fs (sda3): mounted filesystem with ordered data mode. Quota
mode: none.
[ 1273.042962] restraintd[2251]: *** Current Time: Mon Sep 26 12:12:45 2022
Localwatchdog at: Wed Sep 28 11:54:44 2022
[ 1333.319238] restraintd[2251]: *** Current Time: Mon Sep 26 12:13:45 2022
Localwatchdog at: Wed Sep 28 11:54:44 2022
[ 1394.828503] restraintd[2251]: *** Current Time: Mon Sep 26 12:14:47 2022
Localwatchdog at: Wed Sep 28 11:54:44 2022
[ 1403.799008] BUG: Kernel NULL pointer dereference at 0x00000062
[ 1403.799218] Faulting instruction address: 0xc00000000068edfc
[ 1403.799228] Oops: Kernel access of bad area, sig: 11 [#1]
[ 1403.799233] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
[ 1403.799241] Modules linked in: ext4 mbcache jbd2 bonding tls rfkill sunrpc
pseries_rng drm fuse drm_panel_orientation_quirks xfs libcrc32c sd_mod t10_pi
sg ibmvscsi ibmveth scsi_transport_srp vmx_crypto
[ 1403.799280] CPU: 4 PID: 82 Comm: kswapd0 Kdump: loaded Not tainted 6.0.0-rc7
#1
[ 1403.799293] NIP: c00000000068edfc LR: c00000000068f2a8 CTR:
0000000000000000
[ 1403.799300] REGS: c00000000a44b560 TRAP: 0380 Not tainted (6.0.0-rc7)
[ 1403.799308] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR:
28028244 XER: 00000001
[ 1403.799327] CFAR: c00000000068ede4 IRQMASK: 0
[ 1403.799327] GPR00: c00000000068f2a8 c00000000a44b800 c000000002cf1700
c00c0000001c0bc0
[ 1403.799327] GPR04: c00000000a44b860 0000000000000002 00000003fb290000
c000000002de7dc8
[ 1403.799327] GPR08: 0000000ae4f08f42 0000000000000000 c00c0000001c0bc0
0000000000008000
[ 1403.799327] GPR12: 00000003fb290000 c00000000ffcc080 c000000000194288
c0000003fff9c480
[ 1403.799327] GPR16: c000000069d30050 0000000000000007 0000000000000000
0000000000000000
[ 1403.799327] GPR20: 0000000000000001 c00000000a44b8f8 c00000000146bad8
5deadbeef0000100
[ 1403.799327] GPR24: 5deadbeef0000122 c000000069d30000 c00000000a44bc00
c00000000a44b8e8
[ 1403.799327] GPR28: c00000000a44b860 c00c0000001c0bc0 0000000000000002
0000000000000002
[ 1403.799413] NIP [c00000000068edfc] drop_buffers.constprop.0+0x4c/0x1c0
[ 1403.799423] LR [c00000000068f2a8] try_to_free_buffers+0x128/0x150
[ 1403.799431] Call Trace:
[ 1403.799434] [c00000000a44b840] [c00000000a44bc00] 0xc00000000a44bc00
[ 1403.799443] [c00000000a44b890] [c0000000004986f8]
filemap_release_folio+0x88/0xb0
[ 1403.799452] [c00000000a44b8b0] [c0000000004c51b0]
shrink_active_list+0x490/0x750
[ 1403.799462] [c00000000a44b9b0] [c0000000004c9f78] shrink_lruvec+0x3f8/0x430
[ 1403.799470] [c00000000a44baa0] [c0000000004ca1e4]
shrink_node_memcgs+0x234/0x290
[ 1403.799478] [c00000000a44bb10] [c0000000004ca3b4] shrink_node+0x174/0x6b0
[ 1403.799486] [c00000000a44bbc0] [c0000000004cace0] balance_pgdat+0x3f0/0x970
[ 1403.799494] [c00000000a44bd20] [c0000000004cb430] kswapd+0x1d0/0x450
[ 1403.799501] [c00000000a44bdc0] [c0000000001943c8] kthread+0x148/0x150
[ 1403.799510] [c00000000a44be10] [c00000000000cbe4]
ret_from_kernel_thread+0x5c/0x64
[ 1403.799520] Instruction dump:
[ 1403.799525] fbc1fff0 f821ffc1 7c7d1b78 7c9c2378 ebc30028 7fdff378 48000018
60000000
[ 1403.799540] 60000000 ebff0008 7c3ef840 41820048 <815f0060> e93f0000 5529077c
7d295378
[ 1403.799554] ---[ end trace 0000000000000000 ]---
[ 1403.806330]
[-- MARK -- Mon Sep 26 16:15:00 2022]
[ 1415.093395] EXT4-fs (sda3): shut down requested (2)
[ 1415.093410] Aborting journal on device sda3-8.
[ 1429.107188] EXT4-fs (sda3): unmounting filesystem.
[ 1429.926262] EXT4-fs (sda3): recovery complete
[ 1429.983938] EXT4-fs (sda3): mounted filesystem with ordered data mode. Quota
mode: none.
[ 1429.988189] EXT4-fs (sda3): unmounting filesystem.
[ 1430.166549] EXT4-fs (sda3): mounted filesystem with ordered data mode. Quota
mode: none.
[ 1453.015796] restraintd[2251]: *** Current Time: Mon Sep 26 12:15:45 2022
Localwatchdog at: Wed Sep 28 11:54:44 2022
[ 1454.708150] EXT4-fs (sda5): unmounting filesystem.
[ 1455.225112] EXT4-fs (sda3): unmounting filesystem.
[ 1456.128026] EXT4-fs (sda3): mounted filesystem with ordered data mode. Quota
mode: none.
[ 1456.139102] EXT4-fs (sda3): unmounting filesystem.
[ 1456.396367] EXT4-fs (sda5): mounted filesystem with ordered data mode. Quota
mode: none.
[ 1462.317449] EXT4-fs (sda3): mounted filesystem with ordered data mode. Quota
mode: none.
[ 1462.326680] EXT4-fs (sda3): unmounting filesystem.
[ 1462.427320] EXT4-fs (sda5): unmounting filesystem.
[ 1463.259690] EXT4-fs (sda5): mounted filesystem with ordered data mode. Quota
mode: none.
>
> Ritesh, is this something you can take a look at it? Thanks!
>
> - Ted
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bug 216529] [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-27 0:47 ` bugzilla-daemon
@ 2022-09-27 18:06 ` Theodore Ts'o
0 siblings, 0 replies; 10+ messages in thread
From: Theodore Ts'o @ 2022-09-27 18:06 UTC (permalink / raw)
To: bugzilla-daemon; +Cc: linux-ext4
On Tue, Sep 27, 2022 at 12:47:02AM +0000, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=216529
>
> Yes, it's reproducible for me, I just reproduced it again on another ppc64le
> (P8) machine [1]. But it's not easy to reproduce by running generic/048 (maybe
> there's a better way to reproduce it).
Can you give a rough percentage of how often it reproduces? e.g.,
does it reproduces 10% of the time? 50% of the time? 2-3 times after
100 tries, so 2-3%? etc. If it reproduces but rarely, it'll be a lot
harder to try to bisect.
Something perhaps to try is to enable KASAN, since both stack traces
seem to involve a null pointer derference while trying to free
buffers. Maybe that will give us some hints towards the cause....
Thanks,
- Ted
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Bug 216529] [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-25 11:55 [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0 bugzilla-daemon
` (2 preceding siblings ...)
2022-09-27 0:47 ` bugzilla-daemon
@ 2022-09-27 18:06 ` bugzilla-daemon
2022-09-27 18:10 ` bugzilla-daemon
2022-10-10 17:01 ` bugzilla-daemon
5 siblings, 0 replies; 10+ messages in thread
From: bugzilla-daemon @ 2022-09-27 18:06 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=216529
--- Comment #3 from Theodore Tso (tytso@mit.edu) ---
On Tue, Sep 27, 2022 at 12:47:02AM +0000, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=216529
>
> Yes, it's reproducible for me, I just reproduced it again on another ppc64le
> (P8) machine [1]. But it's not easy to reproduce by running generic/048
> (maybe
> there's a better way to reproduce it).
Can you give a rough percentage of how often it reproduces? e.g.,
does it reproduces 10% of the time? 50% of the time? 2-3 times after
100 tries, so 2-3%? etc. If it reproduces but rarely, it'll be a lot
harder to try to bisect.
Something perhaps to try is to enable KASAN, since both stack traces
seem to involve a null pointer derference while trying to free
buffers. Maybe that will give us some hints towards the cause....
Thanks,
- Ted
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Bug 216529] [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-25 11:55 [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0 bugzilla-daemon
` (3 preceding siblings ...)
2022-09-27 18:06 ` bugzilla-daemon
@ 2022-09-27 18:10 ` bugzilla-daemon
2022-10-10 17:01 ` bugzilla-daemon
5 siblings, 0 replies; 10+ messages in thread
From: bugzilla-daemon @ 2022-09-27 18:10 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=216529
--- Comment #4 from ritesh.list@gmail.com ---
On 22/09/26 01:02AM, Theodore Ts'o wrote:
> On Sun, Sep 25, 2022 at 11:55:29AM +0000, bugzilla-daemon@kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=216529
> >
> >
> > Hit a panic on ppc64le, by running generic/048 with 1k block size:
>
> Hmm, does this reproduce reliably for you? I test with a 1k block
> size on x86_64 as a proxy 4k block sizes on PPC64, where the blocksize
> < pagesize... and this isn't reproducing for me on x86, and I don't
> have access to a PPC64LE system.
>
> Ritesh, is this something you can take a look at it? Thanks!
I was away for some personal work for last few days, but I am back to work from
today. Sure, I will take a look at this and will get back.
I did give this test a couple of runs though, but wasn't able to reproduce it.
But let me try few more things along with more iterations. Will update
accordingly.
-ritesh
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Bug 216529] [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0
2022-09-25 11:55 [Bug 216529] New: [fstests generic/048] BUG: Kernel NULL pointer dereference at 0x00000069, filemap_release_folio+0x88/0xb0 bugzilla-daemon
` (4 preceding siblings ...)
2022-09-27 18:10 ` bugzilla-daemon
@ 2022-10-10 17:01 ` bugzilla-daemon
5 siblings, 0 replies; 10+ messages in thread
From: bugzilla-daemon @ 2022-10-10 17:01 UTC (permalink / raw)
To: linux-ext4
https://bugzilla.kernel.org/show_bug.cgi?id=216529
--- Comment #5 from ritesh.list@gmail.com ---
On 22/09/27 11:40PM, Ritesh Harjani (IBM) wrote:
> On 22/09/26 01:02AM, Theodore Ts'o wrote:
> > On Sun, Sep 25, 2022 at 11:55:29AM +0000, bugzilla-daemon@kernel.org wrote:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=216529
> > >
> > >
> > > Hit a panic on ppc64le, by running generic/048 with 1k block size:
> >
> > Hmm, does this reproduce reliably for you? I test with a 1k block
> > size on x86_64 as a proxy 4k block sizes on PPC64, where the blocksize
> > < pagesize... and this isn't reproducing for me on x86, and I don't
> > have access to a PPC64LE system.
> >
> > Ritesh, is this something you can take a look at it? Thanks!
>
> I was away for some personal work for last few days, but I am back to work
> from
> today. Sure, I will take a look at this and will get back.
>
> I did give this test a couple of runs though, but wasn't able to reproduce
> it.
> But let me try few more things along with more iterations. Will update
> accordingly.
I thought I had updated this. But I guess I forgot to update on this mail
thread...
I tested this for quite some time in a loop and also gave it a overnight run,
but I couldn't hit this issue. I had kept low memory size guest, so that we
could see more reclaim activity (which I also ensured by doing perf trace to
see
if we are going over that path or not while test was running).
I am not sure whether this could be a timing issue or what. Maybe if you could
share your defconfig, I could give a try with that on my setup once.
-ritesh
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
^ permalink raw reply [flat|nested] 10+ messages in thread