* kasan: paging percpu + kasan causes a double fault
@ 2019-07-08 15:05 Dennis Zhou
2019-07-18 15:51 ` Dmitry Vyukov
0 siblings, 1 reply; 3+ messages in thread
From: Dennis Zhou @ 2019-07-08 15:05 UTC (permalink / raw)
To: Andrey Ryabinin, Alexander Potapenko, Dmitry Vyukov
Cc: Tejun Heo, Kefeng Wang, kasan-dev, linux-mm, linux-kernel
Hi Andrey, Alexander, and Dmitry,
It was reported to me that when percpu is ran with param
percpu_alloc=page or the embed allocation scheme fails and falls back to
page that a double fault occurs.
I don't know much about how kasan works, but a difference between the
two is that we manually reserve vm area via vm_area_register_early().
I guessed it had something to do with the stack canary or the irq_stack,
and manually mapped the shadow vm area with kasan_add_zero_shadow(), but
that didn't seem to do the trick.
RIP resolves to the fixed_percpu_data declaration.
Double fault below:
[ 0.000000] PANIC: double fault, error_code: 0x0
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc7-00007-ge0afe6d4d12c-dirty #299
[ 0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
[ 0.000000] RIP: 0010:no_context+0x38/0x4b0
[ 0.000000] Code: df 41 57 41 56 4c 8d bf 88 00 00 00 41 55 49 89 d5 41 54 49 89 f4 55 48 89 fd 4c8
[ 0.000000] RSP: 0000:ffffc8ffffffff28 EFLAGS: 00010096
[ 0.000000] RAX: dffffc0000000000 RBX: ffffc8ffffffff50 RCX: 000000000000000b
[ 0.000000] RDX: fffff52000000030 RSI: 0000000000000003 RDI: ffffc90000000130
[ 0.000000] RBP: ffffc900000000a8 R08: 0000000000000001 R09: 0000000000000000
[ 0.000000] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[ 0.000000] R13: fffff52000000030 R14: 0000000000000000 R15: ffffc90000000130
[ 0.000000] FS: 0000000000000000(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000
[ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.000000] CR2: ffffc8ffffffff18 CR3: 0000000002e0d001 CR4: 00000000000606b0
[ 0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.000000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 0.000000] Call Trace:
[ 0.000000] Kernel panic - not syncing: Machine halted.
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc7-00007-ge0afe6d4d12c-dirty #299
[ 0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
[ 0.000000] Call Trace:
[ 0.000000] <#DF>
[ 0.000000] dump_stack+0x5b/0x90
[ 0.000000] panic+0x17e/0x36e
[ 0.000000] ? __warn_printk+0xdb/0xdb
[ 0.000000] ? spurious_kernel_fault_check+0x1a/0x60
[ 0.000000] df_debug+0x2e/0x39
[ 0.000000] do_double_fault+0x89/0xb0
[ 0.000000] double_fault+0x1e/0x30
[ 0.000000] RIP: 0010:no_context+0x38/0x4b0
[ 0.000000] Code: df 41 57 41 56 4c 8d bf 88 00 00 00 41 55 49 89 d5 41 54 49 89 f4 55 48 89 fd 4c8
[ 0.000000] RSP: 0000:ffffc8ffffffff28 EFLAGS: 00010096
[ 0.000000] RAX: dffffc0000000000 RBX: ffffc8ffffffff50 RCX: 000000000000000b
[ 0.000000] RDX: fffff52000000030 RSI: 0000000000000003 RDI: ffffc90000000130
[ 0.000000] RBP: ffffc900000000a8 R08: 0000000000000001 R09: 0000000000000000
[ 0.000000] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[ 0.000000] R13: fffff52000000030 R14: 0000000000000000 R15: ffffc90000000130
Thanks,
Dennis
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: kasan: paging percpu + kasan causes a double fault
2019-07-08 15:05 kasan: paging percpu + kasan causes a double fault Dennis Zhou
@ 2019-07-18 15:51 ` Dmitry Vyukov
2019-07-18 16:20 ` Andrey Ryabinin
0 siblings, 1 reply; 3+ messages in thread
From: Dmitry Vyukov @ 2019-07-18 15:51 UTC (permalink / raw)
To: Dennis Zhou
Cc: Andrey Ryabinin, Alexander Potapenko, Tejun Heo, Kefeng Wang,
kasan-dev, Linux-MM, LKML
On Mon, Jul 8, 2019 at 5:05 PM Dennis Zhou <dennis@kernel.org> wrote:
>
> Hi Andrey, Alexander, and Dmitry,
>
> It was reported to me that when percpu is ran with param
> percpu_alloc=page or the embed allocation scheme fails and falls back to
> page that a double fault occurs.
>
> I don't know much about how kasan works, but a difference between the
> two is that we manually reserve vm area via vm_area_register_early().
> I guessed it had something to do with the stack canary or the irq_stack,
> and manually mapped the shadow vm area with kasan_add_zero_shadow(), but
> that didn't seem to do the trick.
>
> RIP resolves to the fixed_percpu_data declaration.
>
> Double fault below:
> [ 0.000000] PANIC: double fault, error_code: 0x0
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc7-00007-ge0afe6d4d12c-dirty #299
> [ 0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
> [ 0.000000] RIP: 0010:no_context+0x38/0x4b0
> [ 0.000000] Code: df 41 57 41 56 4c 8d bf 88 00 00 00 41 55 49 89 d5 41 54 49 89 f4 55 48 89 fd 4c8
> [ 0.000000] RSP: 0000:ffffc8ffffffff28 EFLAGS: 00010096
> [ 0.000000] RAX: dffffc0000000000 RBX: ffffc8ffffffff50 RCX: 000000000000000b
> [ 0.000000] RDX: fffff52000000030 RSI: 0000000000000003 RDI: ffffc90000000130
> [ 0.000000] RBP: ffffc900000000a8 R08: 0000000000000001 R09: 0000000000000000
> [ 0.000000] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
> [ 0.000000] R13: fffff52000000030 R14: 0000000000000000 R15: ffffc90000000130
> [ 0.000000] FS: 0000000000000000(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000
> [ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 0.000000] CR2: ffffc8ffffffff18 CR3: 0000000002e0d001 CR4: 00000000000606b0
> [ 0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 0.000000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 0.000000] Call Trace:
> [ 0.000000] Kernel panic - not syncing: Machine halted.
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc7-00007-ge0afe6d4d12c-dirty #299
> [ 0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
> [ 0.000000] Call Trace:
> [ 0.000000] <#DF>
> [ 0.000000] dump_stack+0x5b/0x90
> [ 0.000000] panic+0x17e/0x36e
> [ 0.000000] ? __warn_printk+0xdb/0xdb
> [ 0.000000] ? spurious_kernel_fault_check+0x1a/0x60
> [ 0.000000] df_debug+0x2e/0x39
> [ 0.000000] do_double_fault+0x89/0xb0
> [ 0.000000] double_fault+0x1e/0x30
> [ 0.000000] RIP: 0010:no_context+0x38/0x4b0
> [ 0.000000] Code: df 41 57 41 56 4c 8d bf 88 00 00 00 41 55 49 89 d5 41 54 49 89 f4 55 48 89 fd 4c8
> [ 0.000000] RSP: 0000:ffffc8ffffffff28 EFLAGS: 00010096
> [ 0.000000] RAX: dffffc0000000000 RBX: ffffc8ffffffff50 RCX: 000000000000000b
> [ 0.000000] RDX: fffff52000000030 RSI: 0000000000000003 RDI: ffffc90000000130
> [ 0.000000] RBP: ffffc900000000a8 R08: 0000000000000001 R09: 0000000000000000
> [ 0.000000] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
> [ 0.000000] R13: fffff52000000030 R14: 0000000000000000 R15: ffffc90000000130
Hi Dennis,
I don't have lots of useful info, but a naive question: could you stop
using percpu_alloc=page with KASAN? That should resolve the problem :)
We could even add a runtime check that will clearly say that this
combintation does not work.
I see that setup_per_cpu_areas is called after kasan_init which is
called from setup_arch. So KASAN should already map final shadow at
that point.
The only potential reason that I see is that setup_per_cpu_areas maps
the percpu region at address that is not covered/expected by
kasan_init. Where is page-based percpu is mapped? Is that covered by
kasan_init?
Otherwise, seeing the full stack trace of the fault may shed some light.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: kasan: paging percpu + kasan causes a double fault
2019-07-18 15:51 ` Dmitry Vyukov
@ 2019-07-18 16:20 ` Andrey Ryabinin
0 siblings, 0 replies; 3+ messages in thread
From: Andrey Ryabinin @ 2019-07-18 16:20 UTC (permalink / raw)
To: Dmitry Vyukov, Dennis Zhou
Cc: Alexander Potapenko, Tejun Heo, Kefeng Wang, kasan-dev, Linux-MM, LKML
On 7/18/19 6:51 PM, Dmitry Vyukov wrote:
> On Mon, Jul 8, 2019 at 5:05 PM Dennis Zhou <dennis@kernel.org> wrote:
>>
>> Hi Andrey, Alexander, and Dmitry,
>>
>> It was reported to me that when percpu is ran with param
>> percpu_alloc=page or the embed allocation scheme fails and falls back to
>> page that a double fault occurs.
>>
>> I don't know much about how kasan works, but a difference between the
>> two is that we manually reserve vm area via vm_area_register_early().
>> I guessed it had something to do with the stack canary or the irq_stack,
>> and manually mapped the shadow vm area with kasan_add_zero_shadow(), but
>> that didn't seem to do the trick.
>>
>> RIP resolves to the fixed_percpu_data declaration.
>>
>> Double fault below:
>> [ 0.000000] PANIC: double fault, error_code: 0x0
>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc7-00007-ge0afe6d4d12c-dirty #299
>> [ 0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
>> [ 0.000000] RIP: 0010:no_context+0x38/0x4b0
>> [ 0.000000] Code: df 41 57 41 56 4c 8d bf 88 00 00 00 41 55 49 89 d5 41 54 49 89 f4 55 48 89 fd 4c8
>> [ 0.000000] RSP: 0000:ffffc8ffffffff28 EFLAGS: 00010096
>> [ 0.000000] RAX: dffffc0000000000 RBX: ffffc8ffffffff50 RCX: 000000000000000b
>> [ 0.000000] RDX: fffff52000000030 RSI: 0000000000000003 RDI: ffffc90000000130
>> [ 0.000000] RBP: ffffc900000000a8 R08: 0000000000000001 R09: 0000000000000000
>> [ 0.000000] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
>> [ 0.000000] R13: fffff52000000030 R14: 0000000000000000 R15: ffffc90000000130
>> [ 0.000000] FS: 0000000000000000(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000
>> [ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 0.000000] CR2: ffffc8ffffffff18 CR3: 0000000002e0d001 CR4: 00000000000606b0
>> [ 0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 0.000000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [ 0.000000] Call Trace:
>> [ 0.000000] Kernel panic - not syncing: Machine halted.
>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc7-00007-ge0afe6d4d12c-dirty #299
>> [ 0.000000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-2.el7 04/01/2014
>> [ 0.000000] Call Trace:
>> [ 0.000000] <#DF>
>> [ 0.000000] dump_stack+0x5b/0x90
>> [ 0.000000] panic+0x17e/0x36e
>> [ 0.000000] ? __warn_printk+0xdb/0xdb
>> [ 0.000000] ? spurious_kernel_fault_check+0x1a/0x60
>> [ 0.000000] df_debug+0x2e/0x39
>> [ 0.000000] do_double_fault+0x89/0xb0
>> [ 0.000000] double_fault+0x1e/0x30
>> [ 0.000000] RIP: 0010:no_context+0x38/0x4b0
>> [ 0.000000] Code: df 41 57 41 56 4c 8d bf 88 00 00 00 41 55 49 89 d5 41 54 49 89 f4 55 48 89 fd 4c8
>> [ 0.000000] RSP: 0000:ffffc8ffffffff28 EFLAGS: 00010096
>> [ 0.000000] RAX: dffffc0000000000 RBX: ffffc8ffffffff50 RCX: 000000000000000b
>> [ 0.000000] RDX: fffff52000000030 RSI: 0000000000000003 RDI: ffffc90000000130
>> [ 0.000000] RBP: ffffc900000000a8 R08: 0000000000000001 R09: 0000000000000000
>> [ 0.000000] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
>> [ 0.000000] R13: fffff52000000030 R14: 0000000000000000 R15: ffffc90000000130
>
>
> Hi Dennis,
>
> I don't have lots of useful info, but a naive question: could you stop
> using percpu_alloc=page with KASAN? That should resolve the problem :)
> We could even add a runtime check that will clearly say that this
> combintation does not work.
>
> I see that setup_per_cpu_areas is called after kasan_init which is
> called from setup_arch. So KASAN should already map final shadow at
> that point.
> The only potential reason that I see is that setup_per_cpu_areas maps
> the percpu region at address that is not covered/expected by
> kasan_init. Where is page-based percpu is mapped? Is that covered by
> kasan_init?
> Otherwise, seeing the full stack trace of the fault may shed some light.
>
percpu_alloc=page maps percpu areas into vmalloc, which don't have RW KASAN shadow mem.
irq stack are percpu thus we have GPF on attempt to poison stack redzones in irq.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-07-18 16:20 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-08 15:05 kasan: paging percpu + kasan causes a double fault Dennis Zhou
2019-07-18 15:51 ` Dmitry Vyukov
2019-07-18 16:20 ` Andrey Ryabinin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).