* kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
@ 2019-08-21 1:39 jane.chu
2019-08-21 3:48 ` Dan Williams
0 siblings, 1 reply; 5+ messages in thread
From: jane.chu @ 2019-08-21 1:39 UTC (permalink / raw)
To: CHUCK_LEVER, bfields, linux-nfs
Cc: Dan Williams, JANE.CHU, Linux Kernel Mailing List
Hi,
Apology if there is a better channel reporting the issue, if so, please
let me know.
I just saw below regression in 5.3-rc5 kernel, but not in 5.2-rc7 or
earlier kernels.
[ 3533.659787] mce: Uncorrected hardware memory error in user-access at
383e202000
[ 3533.659903] Memory failure: 0x383e202: Sending SIGBUS to
read_poison:14493 due to hardware memory corruption
[ 3533.679041] Memory failure: 0x383e202: recovery action for dax page:
Recovered
[ 3564.624934] BUG: kernel NULL pointer dereference, address:
00000000000001f9
[ 3564.632707] #PF: supervisor read access in kernel mode
[ 3564.638440] #PF: error_code(0x0000) - not-present page
[ 3564.644174] PGD acd7b47067 P4D acd7b47067 PUD acd7aba067 PMD 0
[ 3564.650784] Oops: 0000 [#1] SMP NOPTI
[ 3564.654869] CPU: 58 PID: 15026 Comm: sosreport Tainted: G M
5.3.0-rc5.master.20190820.ol7.x86_64 #1
[ 3564.666420] Hardware name: Oracle Corporation ORACLE SERVER
X8-2L/ASM,MTHRBD,2U, BIOS 52020101 05/07/2019
[ 3564.677112] RIP: 0010:nfsd_reply_cache_stats_show+0x11/0x110 [nfsd]
[ 3564.684106] Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 63
47 ec 48 89 e5 5d c3 90 0f 1f 44 00 00 55 31 c0 48 89 e5 41 54 49 89 f4
53 <8b> 96 f8 01 00 00 48 c7 c6 d9 b8 74 c0 48 89 fb e8 9a ae bc f2 41
[ 3564.705062] RSP: 0018:ffffaa140f87fe18 EFLAGS: 00010246
[ 3564.710894] RAX: 0000000000000000 RBX: ffff9f7c9b562ca8 RCX:
0000000000005c19
[ 3564.718858] RDX: 0000000000001000 RSI: 0000000000000001 RDI:
ffff9f7c9b562c80
[ 3564.726822] RBP: ffffaa140f87fe28 R08: ffff9f801fab01a0 R09:
ffff9ed347c06600
[ 3564.734785] R10: ffff9f801e287000 R11: ffff9f8012f8d638 R12:
0000000000000001
[ 3564.742749] R13: ffff9f8012f8d600 R14: ffff9f7c9b562c80 R15:
0000000000000001
[ 3564.750712] FS: 00007f3cfaa92700(0000) GS:ffff9f801fa80000(0000)
knlGS:0000000000000000
[ 3564.759743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3564.766156] CR2: 00000000000001f9 CR3: 000000add1894004 CR4:
00000000007606e0
[ 3564.774120] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 3564.782084] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 3564.790050] PKRU: 55555554
[ 3564.793068] Call Trace:
[ 3564.795800] seq_read+0x13b/0x390
[ 3564.799502] __vfs_read+0x1b/0x40
[ 3564.803202] vfs_read+0x8e/0x140
[ 3564.806794] ksys_read+0x61/0xd0
[ 3564.810394] __x64_sys_read+0x1a/0x20
[ 3564.814484] do_syscall_64+0x60/0x1e0
[ 3564.818572] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3564.824199] RIP: 0033:0x7f3d163a304d
The panic is reproducible every time in 5.3-rc5, here is the steps.
On system with Intel DC PMEM, configure at AppDirect NonInterleave mode,
# ndctl create-namespace -m devdax
{
"dev":"namespace1.0",
"mode":"devdax",
"map":"dev",
"size":"124.03 GiB (133.18 GB)",
..
"align":2097152,
"devices":[
{
"chardev":"dax1.0",
# ndctl inject-error namespace1.0 -B 16 --count=1
# ./read_poison -x dax1.0 -o 8192 -m 1
Read poison location at (16 * 512 = 8192)
About a little under 30sec later, kernel panics.
thanks,
-jane
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
2019-08-21 1:39 kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11 jane.chu
@ 2019-08-21 3:48 ` Dan Williams
2019-08-21 14:12 ` J. Bruce Fields
2019-08-21 18:11 ` jane.chu
0 siblings, 2 replies; 5+ messages in thread
From: Dan Williams @ 2019-08-21 3:48 UTC (permalink / raw)
To: Jane Chu
Cc: CHUCK_LEVER, J. Bruce Fields, linux-nfs, Linux Kernel Mailing List
On Tue, Aug 20, 2019 at 6:39 PM <jane.chu@oracle.com> wrote:
>
> Hi,
>
> Apology if there is a better channel reporting the issue, if so, please
> let me know.
>
> I just saw below regression in 5.3-rc5 kernel, but not in 5.2-rc7 or
> earlier kernels.
Is the error stable enough to bisect?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
2019-08-21 3:48 ` Dan Williams
@ 2019-08-21 14:12 ` J. Bruce Fields
2019-08-21 21:28 ` jane.chu
2019-08-21 18:11 ` jane.chu
1 sibling, 1 reply; 5+ messages in thread
From: J. Bruce Fields @ 2019-08-21 14:12 UTC (permalink / raw)
To: Dan Williams; +Cc: Jane Chu, CHUCK_LEVER, linux-nfs, Linux Kernel Mailing List
Probably just needs the following.
I've been slow to get some bugfixes upstream, sorry--I'll go send a pull
request now....
--b.
commit 78e70e780b28
Author: He Zhe <zhe.he@windriver.com>
Date: Tue Aug 6 17:41:04 2019 +0800
nfsd4: Fix kernel crash when reading proc file reply_cache_stats
reply_cache_stats uses wrong parameter as seq file private structure and
thus causes the following kernel crash when users read
/proc/fs/nfsd/reply_cache_stats
BUG: kernel NULL pointer dereference, address: 00000000000001f9
PGD 0 P4D 0
Oops: 0000 [#3] SMP PTI
CPU: 6 PID: 1502 Comm: cat Tainted: G D 5.3.0-rc3+ #1
Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015
RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0
Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00
RSP: 0018:ffffaa520106fe08 EFLAGS: 00010246
RAX: 000000cfe1a77123 RBX: 0000000000000000 RCX: 0000000000291b46
RDX: 000000cf00000000 RSI: 0000000000000006 RDI: 0000000000291b28
RBP: ffffaa520106fe20 R08: 0000000000000006 R09: 000000cfe17e55dd
R10: ffffa424e47c0000 R11: 000000000000030b R12: 0000000000000001
R13: ffffa424e5697000 R14: 0000000000000001 R15: ffffa424e5697000
FS: 00007f805735f580(0000) GS:ffffa424f8f80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001f9 CR3: 00000000655ce005 CR4: 00000000003606e0
Call Trace:
seq_read+0x194/0x3e0
__vfs_read+0x1b/0x40
vfs_read+0x95/0x140
ksys_read+0x61/0xe0
__x64_sys_read+0x1a/0x20
do_syscall_64+0x4d/0x120
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f805728b861
Code: fe ff ff 50 48 8d 3d 86 b4 09 00 e8 79 e0 01 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 d9 19 0d 00 8b 00 85 c0 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 48 83 ec 28 48 89 54
RSP: 002b:00007ffea1ce3c38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f805728b861
RDX: 0000000000020000 RSI: 00007f8057183000 RDI: 0000000000000003
RBP: 00007f8057183000 R08: 00007f8057182010 R09: 0000000000000000
R10: 0000000000000022 R11: 0000000000000246 R12: 0000559a60e8ff10
R13: 0000000000000003 R14: 0000000000020000 R15: 0000000000020000
Modules linked in:
CR2: 00000000000001f9
---[ end trace 01613595153f0cba ]---
RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0
Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00
RSP: 0018:ffffaa52004b3e08 EFLAGS: 00010246
RAX: 0000002bab45a7c6 RBX: 0000000000000000 RCX: 0000000000291b4c
RDX: 0000002b00000000 RSI: 0000000000000004 RDI: 0000000000291b28
RBP: ffffaa52004b3e20 R08: 0000000000000004 R09: 0000002bab1c8c7a
R10: ffffa424e5500000 R11: 00000000000002a9 R12: 0000000000000001
R13: ffffa424e4475000 R14: 0000000000000001 R15: ffffa424e4475000
FS: 00007f805735f580(0000) GS:ffffa424f8f80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001f9 CR3: 00000000655ce005 CR4: 00000000003606e0
Killed
Fixes: 3ba75830ce17 ("nfsd4: drc containerization")
Signed-off-by: He Zhe <zhe.he@windriver.com>
Signed-off-by: J. Bruce Fields <bfields@redhat.com>
diff --git a/fs/nfsd/nfscache.c b/fs/nfsd/nfscache.c
index 26ad75ae2be0..96352ab7bd81 100644
--- a/fs/nfsd/nfscache.c
+++ b/fs/nfsd/nfscache.c
@@ -571,7 +571,7 @@ nfsd_cache_append(struct svc_rqst *rqstp, struct kvec *data)
*/
static int nfsd_reply_cache_stats_show(struct seq_file *m, void *v)
{
- struct nfsd_net *nn = v;
+ struct nfsd_net *nn = m->private;
seq_printf(m, "max entries: %u\n", nn->max_drc_entries);
seq_printf(m, "num entries: %u\n",
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
2019-08-21 3:48 ` Dan Williams
2019-08-21 14:12 ` J. Bruce Fields
@ 2019-08-21 18:11 ` jane.chu
1 sibling, 0 replies; 5+ messages in thread
From: jane.chu @ 2019-08-21 18:11 UTC (permalink / raw)
To: Dan Williams
Cc: CHUCK_LEVER, J. Bruce Fields, linux-nfs, Linux Kernel Mailing List
Hi, Dan,
On 8/20/19 8:48 PM, Dan Williams wrote:
> On Tue, Aug 20, 2019 at 6:39 PM <jane.chu@oracle.com> wrote:
>>
>> Hi,
>>
>> Apology if there is a better channel reporting the issue, if so, please
>> let me know.
>>
>> I just saw below regression in 5.3-rc5 kernel, but not in 5.2-rc7 or
>> earlier kernels.
>
> Is the error stable enough to bisect?
>
The error is stable, I haven't tried bisect, thought to report the issue
first since it's late in the 5.3 process.
Then saw Bruce' email, I'll report soon whether the patch takes care the
issue.
thanks!
-jane
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
2019-08-21 14:12 ` J. Bruce Fields
@ 2019-08-21 21:28 ` jane.chu
0 siblings, 0 replies; 5+ messages in thread
From: jane.chu @ 2019-08-21 21:28 UTC (permalink / raw)
To: J. Bruce Fields, Dan Williams
Cc: CHUCK_LEVER, linux-nfs, Linux Kernel Mailing List
Hi, Bruce, Dan,
This patch took care the panic issue.
thanks,
-jane
On 8/21/19 7:12 AM, J. Bruce Fields wrote:
> Probably just needs the following.
>
> I've been slow to get some bugfixes upstream, sorry--I'll go send a pull
> request now....
>
> --b.
>
> commit 78e70e780b28
> Author: He Zhe <zhe.he@windriver.com>
> Date: Tue Aug 6 17:41:04 2019 +0800
>
> nfsd4: Fix kernel crash when reading proc file reply_cache_stats
>
> reply_cache_stats uses wrong parameter as seq file private structure and
> thus causes the following kernel crash when users read
> /proc/fs/nfsd/reply_cache_stats
>
> BUG: kernel NULL pointer dereference, address: 00000000000001f9
> PGD 0 P4D 0
> Oops: 0000 [#3] SMP PTI
> CPU: 6 PID: 1502 Comm: cat Tainted: G D 5.3.0-rc3+ #1
> Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015
> RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0
> Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00
> RSP: 0018:ffffaa520106fe08 EFLAGS: 00010246
> RAX: 000000cfe1a77123 RBX: 0000000000000000 RCX: 0000000000291b46
> RDX: 000000cf00000000 RSI: 0000000000000006 RDI: 0000000000291b28
> RBP: ffffaa520106fe20 R08: 0000000000000006 R09: 000000cfe17e55dd
> R10: ffffa424e47c0000 R11: 000000000000030b R12: 0000000000000001
> R13: ffffa424e5697000 R14: 0000000000000001 R15: ffffa424e5697000
> FS: 00007f805735f580(0000) GS:ffffa424f8f80000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000001f9 CR3: 00000000655ce005 CR4: 00000000003606e0
> Call Trace:
> seq_read+0x194/0x3e0
> __vfs_read+0x1b/0x40
> vfs_read+0x95/0x140
> ksys_read+0x61/0xe0
> __x64_sys_read+0x1a/0x20
> do_syscall_64+0x4d/0x120
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7f805728b861
> Code: fe ff ff 50 48 8d 3d 86 b4 09 00 e8 79 e0 01 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 d9 19 0d 00 8b 00 85 c0 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 48 83 ec 28 48 89 54
> RSP: 002b:00007ffea1ce3c38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f805728b861
> RDX: 0000000000020000 RSI: 00007f8057183000 RDI: 0000000000000003
> RBP: 00007f8057183000 R08: 00007f8057182010 R09: 0000000000000000
> R10: 0000000000000022 R11: 0000000000000246 R12: 0000559a60e8ff10
> R13: 0000000000000003 R14: 0000000000020000 R15: 0000000000020000
> Modules linked in:
> CR2: 00000000000001f9
> ---[ end trace 01613595153f0cba ]---
> RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0
> Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00
> RSP: 0018:ffffaa52004b3e08 EFLAGS: 00010246
> RAX: 0000002bab45a7c6 RBX: 0000000000000000 RCX: 0000000000291b4c
> RDX: 0000002b00000000 RSI: 0000000000000004 RDI: 0000000000291b28
> RBP: ffffaa52004b3e20 R08: 0000000000000004 R09: 0000002bab1c8c7a
> R10: ffffa424e5500000 R11: 00000000000002a9 R12: 0000000000000001
> R13: ffffa424e4475000 R14: 0000000000000001 R15: ffffa424e4475000
> FS: 00007f805735f580(0000) GS:ffffa424f8f80000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000001f9 CR3: 00000000655ce005 CR4: 00000000003606e0
> Killed
>
> Fixes: 3ba75830ce17 ("nfsd4: drc containerization")
> Signed-off-by: He Zhe <zhe.he@windriver.com>
> Signed-off-by: J. Bruce Fields <bfields@redhat.com>
>
> diff --git a/fs/nfsd/nfscache.c b/fs/nfsd/nfscache.c
> index 26ad75ae2be0..96352ab7bd81 100644
> --- a/fs/nfsd/nfscache.c
> +++ b/fs/nfsd/nfscache.c
> @@ -571,7 +571,7 @@ nfsd_cache_append(struct svc_rqst *rqstp, struct kvec *data)
> */
> static int nfsd_reply_cache_stats_show(struct seq_file *m, void *v)
> {
> - struct nfsd_net *nn = v;
> + struct nfsd_net *nn = m->private;
>
> seq_printf(m, "max entries: %u\n", nn->max_drc_entries);
> seq_printf(m, "num entries: %u\n",
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-08-21 21:36 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-21 1:39 kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11 jane.chu
2019-08-21 3:48 ` Dan Williams
2019-08-21 14:12 ` J. Bruce Fields
2019-08-21 21:28 ` jane.chu
2019-08-21 18:11 ` jane.chu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).