* kvm: WARNING in mmu_spte_clear_track_bits
@ 2016-12-13 19:50 Dmitry Vyukov
2017-01-13 11:15 ` Dmitry Vyukov
0 siblings, 1 reply; 8+ messages in thread
From: Dmitry Vyukov @ 2016-12-13 19:50 UTC (permalink / raw)
To: Paolo Bonzini, Radim Krčmář,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86, KVM list,
LKML, Alan Stern, Steve Rutherford
Cc: syzkaller
Hello,
The following program:
https://gist.githubusercontent.com/dvyukov/23d8bd622fd526d7701ac2057bbbc9c2/raw/aacd20451e6f460232f5e1da262b653fb3155613/gistfile1.txt
leads to WARNING in mmu_spte_clear_track_bits and later to splash of
BUG: Bad page state in process a.out pfn:619b5
On commit e7aa8c2eb11ba69b1b69099c3c7bd6be3087b0ba (Dec 12).
------------[ cut here ]------------
WARNING: CPU: 0 PID: 6907 at mmu_spte_clear_track_bits+0x326/0x3a0
arch/x86/kvm/mmu.c:614
Modules linked in:
CPU: 0 PID: 6907 Comm: a.out Not tainted 4.9.0+ #85
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
[< inline >] __dump_stack lib/dump_stack.c:15
[< none >] dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
[< none >] __warn+0x1a4/0x1e0 kernel/panic.c:550
[< none >] warn_slowpath_null+0x31/0x40 kernel/panic.c:585
[< none >] mmu_spte_clear_track_bits+0x326/0x3a0
arch/x86/kvm/mmu.c:614
[< none >] drop_spte+0x29/0x220 arch/x86/kvm/mmu.c:1182
[< none >] mmu_page_zap_pte+0x209/0x300 arch/x86/kvm/mmu.c:2306
[< inline >] kvm_mmu_page_unlink_children arch/x86/kvm/mmu.c:2328
[< none >] kvm_mmu_prepare_zap_page+0x1cd/0x1240
arch/x86/kvm/mmu.c:2372
[< inline >] kvm_zap_obsolete_pages arch/x86/kvm/mmu.c:4915
[< none >] kvm_mmu_invalidate_zap_all_pages+0x4af/0x6f0
arch/x86/kvm/mmu.c:4956
[< none >] kvm_arch_flush_shadow_all+0x1a/0x20
arch/x86/kvm/x86.c:8177
[< none >] kvm_mmu_notifier_release+0x76/0xb0
arch/x86/kvm/../../../virt/kvm/kvm_main.c:467
[< none >] __mmu_notifier_release+0x1fe/0x6c0 mm/mmu_notifier.c:74
[< inline >] mmu_notifier_release ./include/linux/mmu_notifier.h:235
[< none >] exit_mmap+0x3d1/0x4a0 mm/mmap.c:2918
[< inline >] __mmput kernel/fork.c:868
[< none >] mmput+0x1fd/0x690 kernel/fork.c:890
[< inline >] exit_mm kernel/exit.c:521
[< none >] do_exit+0x9e7/0x2930 kernel/exit.c:826
[< none >] do_group_exit+0x14e/0x420 kernel/exit.c:943
[< inline >] SYSC_exit_group kernel/exit.c:954
[< none >] SyS_exit_group+0x22/0x30 kernel/exit.c:952
[< none >] entry_SYSCALL_64_fastpath+0x23/0xc6
arch/x86/entry/entry_64.S:203
RIP: 0033:0x43f4d9
RSP: 002b:00007ffc7e83f548 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00000000006d6660 RCX: 000000000043f4d9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000001 R08: 000000000000003c R09: 00000000000000e7
R10: ffffffffffffffd0 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fe58e3869c0 R15: 00007fe58e386700
---[ end trace 37ef4e3d7e4c81a9 ]---
BUG: Bad page state in process a.out pfn:61fb5
page:ffffea000187ed40 count:0 mapcount:0 mapping: (null) index:0x0
flags: 0x5fffc0000000014(referenced|dirty)
raw: 05fffc0000000014 0000000000000000 0000000000000000 00000000ffffffff
raw: 0000000000000000 ffffea000187ed60 0000000000000000 0000000000000000
page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag set
bad because of flags: 0x14(referenced|dirty)
Modules linked in:
CPU: 2 PID: 7169 Comm: a.out Tainted: G W 4.9.0+ #85
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
[< inline >] __dump_stack lib/dump_stack.c:15
[< none >] dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
[< none >] bad_page+0x29c/0x320 mm/page_alloc.c:550
[< none >] check_new_page_bad+0x203/0x2f0 mm/page_alloc.c:1682
[< inline >] check_new_page mm/page_alloc.c:1694
[< inline >] check_new_pages mm/page_alloc.c:1731
[< none >] buffered_rmqueue+0x1770/0x2900 mm/page_alloc.c:2668
[< none >] get_page_from_freelist+0x213/0x1180
mm/page_alloc.c:2985
[< none >] __alloc_pages_nodemask+0x3b2/0xc90 mm/page_alloc.c:3801
[< inline >] __alloc_pages ./include/linux/gfp.h:433
[< inline >] __alloc_pages_node ./include/linux/gfp.h:446
[< none >] alloc_pages_vma+0x723/0xa30 mm/mempolicy.c:2012
[< none >] do_huge_pmd_anonymous_page+0x35f/0x1b10
mm/huge_memory.c:704
[< inline >] create_huge_pmd mm/memory.c:3476
[< inline >] __handle_mm_fault mm/memory.c:3626
[< none >] handle_mm_fault+0x1975/0x2b90 mm/memory.c:3687
[< none >] __do_page_fault+0x4fb/0xb60 arch/x86/mm/fault.c:1396
[< none >] trace_do_page_fault+0x159/0x810
arch/x86/mm/fault.c:1489
[< none >] do_async_page_fault+0x77/0xd0 arch/x86/kernel/kvm.c:264
[< none >] async_page_fault+0x28/0x30
arch/x86/entry/entry_64.S:1011
RIP: 0033:0x401f5f
RSP: 002b:00007fe592b8ece0 EFLAGS: 00010246
RAX: 0000000020017fe0 RBX: 0000000000000000 RCX: 0000000000403894
RDX: b93bc4d4f06f7d0e RSI: 0000000000000000 RDI: 00007fe592b8f608
RBP: 00007fe592b8ed10 R08: 00007fe592b8f700 R09: 00007fe592b8f700
R10: 00007fe592b8f9d0 R11: 0000000000000202 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fe592b8f9c0 R15: 00007fe592b8f700
BUG: Bad page state in process a.out pfn:619b5
page:ffffea0001866d40 count:0 mapcount:0 mapping: (null) index:0x0
flags: 0x5fffc0000000014(referenced|dirty)
raw: 05fffc0000000014 0000000000000000 0000000000000000 00000000ffffffff
raw: 0000000000000000 ffffea0001866d60 0000000000000000 0000000000000000
page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag set
bad because of flags: 0x14(referenced|dirty)
Modules linked in:
CPU: 2 PID: 7169 Comm: a.out Tainted: G B W 4.9.0+ #85
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
[< inline >] __dump_stack lib/dump_stack.c:15
[< none >] dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
[< none >] bad_page+0x29c/0x320 mm/page_alloc.c:550
[< none >] check_new_page_bad+0x203/0x2f0 mm/page_alloc.c:1682
[< inline >] check_new_page mm/page_alloc.c:1694
[< inline >] check_new_pages mm/page_alloc.c:1731
[< none >] buffered_rmqueue+0x1770/0x2900 mm/page_alloc.c:2668
[< none >] get_page_from_freelist+0x213/0x1180
mm/page_alloc.c:2985
[< none >] __alloc_pages_nodemask+0x3b2/0xc90 mm/page_alloc.c:3801
[< inline >] __alloc_pages ./include/linux/gfp.h:433
[< inline >] __alloc_pages_node ./include/linux/gfp.h:446
[< none >] alloc_pages_vma+0x723/0xa30 mm/mempolicy.c:2012
[< none >] do_huge_pmd_anonymous_page+0x35f/0x1b10
mm/huge_memory.c:704
[< inline >] create_huge_pmd mm/memory.c:3476
[< inline >] __handle_mm_fault mm/memory.c:3626
[< none >] handle_mm_fault+0x1975/0x2b90 mm/memory.c:3687
[< none >] __do_page_fault+0x4fb/0xb60 arch/x86/mm/fault.c:1396
[< none >] trace_do_page_fault+0x159/0x810
arch/x86/mm/fault.c:1489
[< none >] do_async_page_fault+0x77/0xd0 arch/x86/kernel/kvm.c:264
[< none >] async_page_fault+0x28/0x30
arch/x86/entry/entry_64.S:1011
RIP: 0033:0x401f5f
RSP: 002b:00007fe592b8ece0 EFLAGS: 00010246
RAX: 0000000020017fe0 RBX: 0000000000000000 RCX: 0000000000403894
RDX: b93bc4d4f06f7d0e RSI: 0000000000000000 RDI: 00007fe592b8f608
RBP: 00007fe592b8ed10 R08: 00007fe592b8f700 R09: 00007fe592b8f700
R10: 00007fe592b8f9d0 R11: 0000000000000202 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fe592b8f9c0 R15: 00007fe592b8f700
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kvm: WARNING in mmu_spte_clear_track_bits
2016-12-13 19:50 kvm: WARNING in mmu_spte_clear_track_bits Dmitry Vyukov
@ 2017-01-13 11:15 ` Dmitry Vyukov
2017-01-17 15:20 ` Paolo Bonzini
0 siblings, 1 reply; 8+ messages in thread
From: Dmitry Vyukov @ 2017-01-13 11:15 UTC (permalink / raw)
To: Paolo Bonzini, Radim Krčmář,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86, KVM list,
LKML, Alan Stern, Steve Rutherford, Xiao Guangrong,
haozhong.zhang
Cc: syzkaller
On Tue, Dec 13, 2016 at 8:50 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> Hello,
>
> The following program:
> https://gist.githubusercontent.com/dvyukov/23d8bd622fd526d7701ac2057bbbc9c2/raw/aacd20451e6f460232f5e1da262b653fb3155613/gistfile1.txt
>
> leads to WARNING in mmu_spte_clear_track_bits and later to splash of
> BUG: Bad page state in process a.out pfn:619b5
>
> On commit e7aa8c2eb11ba69b1b69099c3c7bd6be3087b0ba (Dec 12).
>
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 6907 at mmu_spte_clear_track_bits+0x326/0x3a0
> arch/x86/kvm/mmu.c:614
> Modules linked in:
> CPU: 0 PID: 6907 Comm: a.out Not tainted 4.9.0+ #85
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> [< inline >] __dump_stack lib/dump_stack.c:15
> [< none >] dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
> [< none >] __warn+0x1a4/0x1e0 kernel/panic.c:550
> [< none >] warn_slowpath_null+0x31/0x40 kernel/panic.c:585
> [< none >] mmu_spte_clear_track_bits+0x326/0x3a0
> arch/x86/kvm/mmu.c:614
> [< none >] drop_spte+0x29/0x220 arch/x86/kvm/mmu.c:1182
> [< none >] mmu_page_zap_pte+0x209/0x300 arch/x86/kvm/mmu.c:2306
> [< inline >] kvm_mmu_page_unlink_children arch/x86/kvm/mmu.c:2328
> [< none >] kvm_mmu_prepare_zap_page+0x1cd/0x1240
> arch/x86/kvm/mmu.c:2372
> [< inline >] kvm_zap_obsolete_pages arch/x86/kvm/mmu.c:4915
> [< none >] kvm_mmu_invalidate_zap_all_pages+0x4af/0x6f0
> arch/x86/kvm/mmu.c:4956
> [< none >] kvm_arch_flush_shadow_all+0x1a/0x20
> arch/x86/kvm/x86.c:8177
> [< none >] kvm_mmu_notifier_release+0x76/0xb0
> arch/x86/kvm/../../../virt/kvm/kvm_main.c:467
> [< none >] __mmu_notifier_release+0x1fe/0x6c0 mm/mmu_notifier.c:74
> [< inline >] mmu_notifier_release ./include/linux/mmu_notifier.h:235
> [< none >] exit_mmap+0x3d1/0x4a0 mm/mmap.c:2918
> [< inline >] __mmput kernel/fork.c:868
> [< none >] mmput+0x1fd/0x690 kernel/fork.c:890
> [< inline >] exit_mm kernel/exit.c:521
> [< none >] do_exit+0x9e7/0x2930 kernel/exit.c:826
> [< none >] do_group_exit+0x14e/0x420 kernel/exit.c:943
> [< inline >] SYSC_exit_group kernel/exit.c:954
> [< none >] SyS_exit_group+0x22/0x30 kernel/exit.c:952
> [< none >] entry_SYSCALL_64_fastpath+0x23/0xc6
> arch/x86/entry/entry_64.S:203
> RIP: 0033:0x43f4d9
> RSP: 002b:00007ffc7e83f548 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> RAX: ffffffffffffffda RBX: 00000000006d6660 RCX: 000000000043f4d9
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: 0000000000000001 R08: 000000000000003c R09: 00000000000000e7
> R10: ffffffffffffffd0 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000000 R14: 00007fe58e3869c0 R15: 00007fe58e386700
> ---[ end trace 37ef4e3d7e4c81a9 ]---
>
>
> BUG: Bad page state in process a.out pfn:61fb5
> page:ffffea000187ed40 count:0 mapcount:0 mapping: (null) index:0x0
> flags: 0x5fffc0000000014(referenced|dirty)
> raw: 05fffc0000000014 0000000000000000 0000000000000000 00000000ffffffff
> raw: 0000000000000000 ffffea000187ed60 0000000000000000 0000000000000000
> page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag set
> bad because of flags: 0x14(referenced|dirty)
> Modules linked in:
> CPU: 2 PID: 7169 Comm: a.out Tainted: G W 4.9.0+ #85
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> [< inline >] __dump_stack lib/dump_stack.c:15
> [< none >] dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
> [< none >] bad_page+0x29c/0x320 mm/page_alloc.c:550
> [< none >] check_new_page_bad+0x203/0x2f0 mm/page_alloc.c:1682
> [< inline >] check_new_page mm/page_alloc.c:1694
> [< inline >] check_new_pages mm/page_alloc.c:1731
> [< none >] buffered_rmqueue+0x1770/0x2900 mm/page_alloc.c:2668
> [< none >] get_page_from_freelist+0x213/0x1180
> mm/page_alloc.c:2985
> [< none >] __alloc_pages_nodemask+0x3b2/0xc90 mm/page_alloc.c:3801
> [< inline >] __alloc_pages ./include/linux/gfp.h:433
> [< inline >] __alloc_pages_node ./include/linux/gfp.h:446
> [< none >] alloc_pages_vma+0x723/0xa30 mm/mempolicy.c:2012
> [< none >] do_huge_pmd_anonymous_page+0x35f/0x1b10
> mm/huge_memory.c:704
> [< inline >] create_huge_pmd mm/memory.c:3476
> [< inline >] __handle_mm_fault mm/memory.c:3626
> [< none >] handle_mm_fault+0x1975/0x2b90 mm/memory.c:3687
> [< none >] __do_page_fault+0x4fb/0xb60 arch/x86/mm/fault.c:1396
> [< none >] trace_do_page_fault+0x159/0x810
> arch/x86/mm/fault.c:1489
> [< none >] do_async_page_fault+0x77/0xd0 arch/x86/kernel/kvm.c:264
> [< none >] async_page_fault+0x28/0x30
> arch/x86/entry/entry_64.S:1011
> RIP: 0033:0x401f5f
> RSP: 002b:00007fe592b8ece0 EFLAGS: 00010246
> RAX: 0000000020017fe0 RBX: 0000000000000000 RCX: 0000000000403894
> RDX: b93bc4d4f06f7d0e RSI: 0000000000000000 RDI: 00007fe592b8f608
> RBP: 00007fe592b8ed10 R08: 00007fe592b8f700 R09: 00007fe592b8f700
> R10: 00007fe592b8f9d0 R11: 0000000000000202 R12: 0000000000000000
> R13: 0000000000000000 R14: 00007fe592b8f9c0 R15: 00007fe592b8f700
>
> BUG: Bad page state in process a.out pfn:619b5
> page:ffffea0001866d40 count:0 mapcount:0 mapping: (null) index:0x0
> flags: 0x5fffc0000000014(referenced|dirty)
> raw: 05fffc0000000014 0000000000000000 0000000000000000 00000000ffffffff
> raw: 0000000000000000 ffffea0001866d60 0000000000000000 0000000000000000
> page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag set
> bad because of flags: 0x14(referenced|dirty)
> Modules linked in:
> CPU: 2 PID: 7169 Comm: a.out Tainted: G B W 4.9.0+ #85
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> [< inline >] __dump_stack lib/dump_stack.c:15
> [< none >] dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
> [< none >] bad_page+0x29c/0x320 mm/page_alloc.c:550
> [< none >] check_new_page_bad+0x203/0x2f0 mm/page_alloc.c:1682
> [< inline >] check_new_page mm/page_alloc.c:1694
> [< inline >] check_new_pages mm/page_alloc.c:1731
> [< none >] buffered_rmqueue+0x1770/0x2900 mm/page_alloc.c:2668
> [< none >] get_page_from_freelist+0x213/0x1180
> mm/page_alloc.c:2985
> [< none >] __alloc_pages_nodemask+0x3b2/0xc90 mm/page_alloc.c:3801
> [< inline >] __alloc_pages ./include/linux/gfp.h:433
> [< inline >] __alloc_pages_node ./include/linux/gfp.h:446
> [< none >] alloc_pages_vma+0x723/0xa30 mm/mempolicy.c:2012
> [< none >] do_huge_pmd_anonymous_page+0x35f/0x1b10
> mm/huge_memory.c:704
> [< inline >] create_huge_pmd mm/memory.c:3476
> [< inline >] __handle_mm_fault mm/memory.c:3626
> [< none >] handle_mm_fault+0x1975/0x2b90 mm/memory.c:3687
> [< none >] __do_page_fault+0x4fb/0xb60 arch/x86/mm/fault.c:1396
> [< none >] trace_do_page_fault+0x159/0x810
> arch/x86/mm/fault.c:1489
> [< none >] do_async_page_fault+0x77/0xd0 arch/x86/kernel/kvm.c:264
> [< none >] async_page_fault+0x28/0x30
> arch/x86/entry/entry_64.S:1011
> RIP: 0033:0x401f5f
> RSP: 002b:00007fe592b8ece0 EFLAGS: 00010246
> RAX: 0000000020017fe0 RBX: 0000000000000000 RCX: 0000000000403894
> RDX: b93bc4d4f06f7d0e RSI: 0000000000000000 RDI: 00007fe592b8f608
> RBP: 00007fe592b8ed10 R08: 00007fe592b8f700 R09: 00007fe592b8f700
> R10: 00007fe592b8f9d0 R11: 0000000000000202 R12: 0000000000000000
> R13: 0000000000000000 R14: 00007fe592b8f9c0 R15: 00007fe592b8f700
I've commented out the WARNING for now, but I am seeing lots of
use-after-free's and rcu stalls involving mmu_spte_clear_track_bits:
BUG: KASAN: use-after-free in mmu_spte_clear_track_bits+0x186/0x190
arch/x86/kvm/mmu.c:597 at addr ffff880068ae2008
Read of size 8 by task syz-executor2/16715
page:ffffea00016e6170 count:0 mapcount:0 mapping: (null) index:0x0
flags: 0x500000000000000()
raw: 0500000000000000 0000000000000000 0000000000000000 00000000ffffffff
raw: ffffea00017ec5a0 ffffea0001783d48 ffff88006aec5d98
page dumped because: kasan: bad access detected
CPU: 2 PID: 16715 Comm: syz-executor2 Not tainted 4.10.0-rc3+ #163
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x292/0x3a2 lib/dump_stack.c:51
kasan_report_error mm/kasan/report.c:213 [inline]
kasan_report+0x42d/0x460 mm/kasan/report.c:307
__asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:333
mmu_spte_clear_track_bits+0x186/0x190 arch/x86/kvm/mmu.c:597
drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401
kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412
kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565
kvm_unmap_hva_range+0x2e/0x40 arch/x86/kvm/mmu.c:1591
kvm_mmu_notifier_invalidate_range_start+0xae/0x140
arch/x86/kvm/../../../virt/kvm/kvm_main.c:360
__mmu_notifier_invalidate_range_start+0x1f8/0x300 mm/mmu_notifier.c:199
mmu_notifier_invalidate_range_start include/linux/mmu_notifier.h:282 [inline]
unmap_vmas+0x14b/0x1b0 mm/memory.c:1368
unmap_region+0x2f8/0x560 mm/mmap.c:2460
do_munmap+0x7b8/0xfa0 mm/mmap.c:2657
mmap_region+0x68f/0x18e0 mm/mmap.c:1612
do_mmap+0x6a2/0xd40 mm/mmap.c:1450
do_mmap_pgoff include/linux/mm.h:2031 [inline]
vm_mmap_pgoff+0x1a9/0x200 mm/util.c:305
SYSC_mmap_pgoff mm/mmap.c:1500 [inline]
SyS_mmap_pgoff+0x22c/0x5d0 mm/mmap.c:1458
SYSC_mmap arch/x86/kernel/sys_x86_64.c:95 [inline]
SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:86
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x445329
RSP: 002b:00007fb33933cb58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
RAX: ffffffffffffffda RBX: 0000000020000000 RCX: 0000000000445329
RDX: 0000000000000003 RSI: 0000000000af1000 RDI: 0000000020000000
RBP: 00000000006dfe90 R08: ffffffffffffffff R09: 0000000000000000
R10: 0000000000000032 R11: 0000000000000282 R12: 0000000000700000
R13: 0000000000000006 R14: ffffffffffffffff R15: 0000000020001000
Memory state around the buggy address:
ffff880068ae1f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff880068ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff880068ae2000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
^
ffff880068ae2080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ffff880068ae2100: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
==================================================================
INFO: rcu_sched self-detected stall on CPU
3-...: (62755 ticks this GP) idle=83f/140000000000001/0
softirq=314032/314032 fqs=14158
(t=65000 jiffies g=108568 c=108567 q=491)
Task dump for CPU 3:
syz-executor2 R running task 11768 19364 2894 0x0000000c
Call Trace:
<IRQ>
sched_show_task+0x3e4/0x540 kernel/sched/core.c:5217
dump_cpu_task+0x71/0x90 kernel/sched/core.c:8822
rcu_dump_cpu_stacks+0x318/0x35e kernel/rcu/tree.c:1290
print_cpu_stall+0x3ad/0x6f0 kernel/rcu/tree.c:1434
check_cpu_stall.isra.64+0x702/0xe80 kernel/rcu/tree.c:1502
__rcu_pending kernel/rcu/tree.c:3469 [inline]
rcu_pending kernel/rcu/tree.c:3533 [inline]
rcu_check_callbacks+0x27f/0xda0 kernel/rcu/tree.c:2867
update_process_times+0x30/0x60 kernel/time/timer.c:1612
tick_sched_handle.isra.18+0xb3/0xe0 kernel/time/tick-sched.c:151
tick_sched_timer+0x72/0x120 kernel/time/tick-sched.c:1158
__run_hrtimer kernel/time/hrtimer.c:1238 [inline]
__hrtimer_run_queues+0x385/0xf80 kernel/time/hrtimer.c:1302
hrtimer_interrupt+0x1c2/0x5e0 kernel/time/hrtimer.c:1336
local_apic_timer_interrupt+0x6f/0xe0 arch/x86/kernel/apic/apic.c:936
smp_apic_timer_interrupt+0x71/0xa0 arch/x86/kernel/apic/apic.c:960
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:484
RIP: 0010:__write_once_size include/linux/compiler.h:272 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x52/0x60 kernel/kcov.c:96
RSP: 0018:ffff88006a8db870 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: 0000000000010000 RBX: 0000000000000000 RCX: ffffc9000186f000
RDX: 0000000000010000 RSI: ffffffff810fb9ba RDI: 0000000000000000
RBP: ffff88006a8db870 R08: 0000000000000001 R09: 0000000000000000
R10: a48dc350e42c518e R11: dffffc0000000000 R12: 1ffff1000d51b710
R13: ffff88006885a008 R14: ffff88006a8db8e0 R15: 0000000000000001
</IRQ>
is_shadow_present_pte arch/x86/kvm/mmu.c:310 [inline]
mmu_spte_clear_track_bits+0xba/0x190 arch/x86/kvm/mmu.c:604
drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401
kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412
kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565
kvm_handle_hva arch/x86/kvm/mmu.c:1581 [inline]
kvm_unmap_hva+0x2a/0x30 arch/x86/kvm/mmu.c:1586
kvm_mmu_notifier_invalidate_page+0xa6/0x140
arch/x86/kvm/../../../virt/kvm/kvm_main.c:316
__mmu_notifier_invalidate_page+0x1ea/0x2f0 mm/mmu_notifier.c:185
mmu_notifier_invalidate_page include/linux/mmu_notifier.h:275 [inline]
try_to_unmap_one+0xd94/0x17c0 mm/rmap.c:1585
rmap_walk_file+0x446/0xb20 mm/rmap.c:1837
rmap_walk+0x1d8/0x300 mm/rmap.c:1857
try_to_unmap+0x57a/0x760 mm/rmap.c:1657
__unmap_and_move+0x464/0xf80 mm/migrate.c:1024
unmap_and_move mm/migrate.c:1113 [inline]
migrate_pages+0x958/0x2350 mm/migrate.c:1338
compact_zone+0xf49/0x24c0 mm/compaction.c:1556
compact_zone_order+0x3f7/0x700 mm/compaction.c:1663
try_to_compact_pages+0x347/0xc20 mm/compaction.c:1712
__alloc_pages_direct_compact+0x77/0x260 mm/page_alloc.c:3152
__alloc_pages_slowpath+0x9a7/0x1910 mm/page_alloc.c:3671
__alloc_pages_nodemask+0x8f4/0xbb0 mm/page_alloc.c:3810
alloc_pages_current+0x1c7/0x6b0 mm/mempolicy.c:2066
alloc_pages include/linux/gfp.h:462 [inline]
get_huge_zero_page mm/huge_memory.c:69 [inline]
mm_get_huge_zero_page+0x293/0x3e0 mm/huge_memory.c:103
do_huge_pmd_anonymous_page+0xac6/0x16b0 mm/huge_memory.c:677
create_huge_pmd mm/memory.c:3476 [inline]
__handle_mm_fault mm/memory.c:3628 [inline]
handle_mm_fault+0x1bc7/0x2630 mm/memory.c:3689
faultin_page mm/gup.c:391 [inline]
__get_user_pages+0x554/0x15f0 mm/gup.c:590
__get_user_pages_locked mm/gup.c:763 [inline]
get_user_pages+0xa3/0xd0 mm/gup.c:985
get_user_page_nowait arch/x86/kvm/../../../virt/kvm/kvm_main.c:1359 [inline]
hva_to_pfn_slow arch/x86/kvm/../../../virt/kvm/kvm_main.c:1420 [inline]
hva_to_pfn arch/x86/kvm/../../../virt/kvm/kvm_main.c:1537 [inline]
__gfn_to_pfn_memslot+0x3c6/0xff0 arch/x86/kvm/../../../virt/kvm/kvm_main.c:1593
try_async_pf+0x14b/0xd50 arch/x86/kvm/mmu.c:3512
tdp_page_fault+0x52c/0xc00 arch/x86/kvm/mmu.c:3577
kvm_mmu_page_fault+0xe7/0x270 arch/x86/kvm/mmu.c:4534
handle_ept_violation+0x18e/0x610 arch/x86/kvm/vmx.c:6412
vmx_handle_exit+0x2b7/0x38b0 arch/x86/kvm/vmx.c:8625
vcpu_enter_guest arch/x86/kvm/x86.c:6888 [inline]
vcpu_run arch/x86/kvm/x86.c:6947 [inline]
kvm_arch_vcpu_ioctl_run+0xf3d/0x4660 arch/x86/kvm/x86.c:7105
kvm_vcpu_ioctl+0x673/0x1120 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2569
vfs_ioctl fs/ioctl.c:43 [inline]
do_vfs_ioctl+0x1bf/0x1780 fs/ioctl.c:683
SYSC_ioctl fs/ioctl.c:698 [inline]
SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x445329
RSP: 002b:00007f6b3ce26b58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 0000000000445329
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015
RBP: 00000000006deb50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000700000
R13: 00007f6b3d5a9768 R14: 00007f6b3d5b4d80 R15: 0000000020007000
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kvm: WARNING in mmu_spte_clear_track_bits
2017-01-13 11:15 ` Dmitry Vyukov
@ 2017-01-17 15:20 ` Paolo Bonzini
2017-01-17 16:00 ` Dmitry Vyukov
0 siblings, 1 reply; 8+ messages in thread
From: Paolo Bonzini @ 2017-01-17 15:20 UTC (permalink / raw)
To: Dmitry Vyukov, Radim Krčmář,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86, KVM list,
LKML, Alan Stern, Steve Rutherford, Xiao Guangrong,
haozhong.zhang
Cc: syzkaller
On 13/01/2017 12:15, Dmitry Vyukov wrote:
>
> I've commented out the WARNING for now, but I am seeing lots of
> use-after-free's and rcu stalls involving mmu_spte_clear_track_bits:
>
>
> BUG: KASAN: use-after-free in mmu_spte_clear_track_bits+0x186/0x190
> arch/x86/kvm/mmu.c:597 at addr ffff880068ae2008
> Read of size 8 by task syz-executor2/16715
> page:ffffea00016e6170 count:0 mapcount:0 mapping: (null) index:0x0
> flags: 0x500000000000000()
> raw: 0500000000000000 0000000000000000 0000000000000000 00000000ffffffff
> raw: ffffea00017ec5a0 ffffea0001783d48 ffff88006aec5d98
> page dumped because: kasan: bad access detected
> CPU: 2 PID: 16715 Comm: syz-executor2 Not tainted 4.10.0-rc3+ #163
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:15 [inline]
> dump_stack+0x292/0x3a2 lib/dump_stack.c:51
> kasan_report_error mm/kasan/report.c:213 [inline]
> kasan_report+0x42d/0x460 mm/kasan/report.c:307
> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:333
> mmu_spte_clear_track_bits+0x186/0x190 arch/x86/kvm/mmu.c:597
> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
> kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401
> kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412
> kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565
> kvm_unmap_hva_range+0x2e/0x40 arch/x86/kvm/mmu.c:1591
> kvm_mmu_notifier_invalidate_range_start+0xae/0x140
> arch/x86/kvm/../../../virt/kvm/kvm_main.c:360
> __mmu_notifier_invalidate_range_start+0x1f8/0x300 mm/mmu_notifier.c:199
> mmu_notifier_invalidate_range_start include/linux/mmu_notifier.h:282 [inline]
> unmap_vmas+0x14b/0x1b0 mm/memory.c:1368
> unmap_region+0x2f8/0x560 mm/mmap.c:2460
> do_munmap+0x7b8/0xfa0 mm/mmap.c:2657
> mmap_region+0x68f/0x18e0 mm/mmap.c:1612
> do_mmap+0x6a2/0xd40 mm/mmap.c:1450
> do_mmap_pgoff include/linux/mm.h:2031 [inline]
> vm_mmap_pgoff+0x1a9/0x200 mm/util.c:305
> SYSC_mmap_pgoff mm/mmap.c:1500 [inline]
> SyS_mmap_pgoff+0x22c/0x5d0 mm/mmap.c:1458
> SYSC_mmap arch/x86/kernel/sys_x86_64.c:95 [inline]
> SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:86
> entry_SYSCALL_64_fastpath+0x1f/0xc2
> RIP: 0033:0x445329
> RSP: 002b:00007fb33933cb58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
> RAX: ffffffffffffffda RBX: 0000000020000000 RCX: 0000000000445329
> RDX: 0000000000000003 RSI: 0000000000af1000 RDI: 0000000020000000
> RBP: 00000000006dfe90 R08: ffffffffffffffff R09: 0000000000000000
> R10: 0000000000000032 R11: 0000000000000282 R12: 0000000000700000
> R13: 0000000000000006 R14: ffffffffffffffff R15: 0000000020001000
> Memory state around the buggy address:
> ffff880068ae1f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ffff880068ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ffff880068ae2000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> ^
> ffff880068ae2080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> ffff880068ae2100: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> ==================================================================
This could be related to the gfn_to_rmap issues.
Paolo
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kvm: WARNING in mmu_spte_clear_track_bits
2017-01-17 15:20 ` Paolo Bonzini
@ 2017-01-17 16:00 ` Dmitry Vyukov
2017-03-12 11:20 ` Dmitry Vyukov
0 siblings, 1 reply; 8+ messages in thread
From: Dmitry Vyukov @ 2017-01-17 16:00 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Radim Krčmář,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86, KVM list,
LKML, Alan Stern, Steve Rutherford, Xiao Guangrong,
haozhong.zhang, syzkaller
On Tue, Jan 17, 2017 at 4:20 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>
> On 13/01/2017 12:15, Dmitry Vyukov wrote:
>>
>> I've commented out the WARNING for now, but I am seeing lots of
>> use-after-free's and rcu stalls involving mmu_spte_clear_track_bits:
>>
>>
>> BUG: KASAN: use-after-free in mmu_spte_clear_track_bits+0x186/0x190
>> arch/x86/kvm/mmu.c:597 at addr ffff880068ae2008
>> Read of size 8 by task syz-executor2/16715
>> page:ffffea00016e6170 count:0 mapcount:0 mapping: (null) index:0x0
>> flags: 0x500000000000000()
>> raw: 0500000000000000 0000000000000000 0000000000000000 00000000ffffffff
>> raw: ffffea00017ec5a0 ffffea0001783d48 ffff88006aec5d98
>> page dumped because: kasan: bad access detected
>> CPU: 2 PID: 16715 Comm: syz-executor2 Not tainted 4.10.0-rc3+ #163
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> Call Trace:
>> __dump_stack lib/dump_stack.c:15 [inline]
>> dump_stack+0x292/0x3a2 lib/dump_stack.c:51
>> kasan_report_error mm/kasan/report.c:213 [inline]
>> kasan_report+0x42d/0x460 mm/kasan/report.c:307
>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:333
>> mmu_spte_clear_track_bits+0x186/0x190 arch/x86/kvm/mmu.c:597
>> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
>> kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401
>> kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412
>> kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565
>> kvm_unmap_hva_range+0x2e/0x40 arch/x86/kvm/mmu.c:1591
>> kvm_mmu_notifier_invalidate_range_start+0xae/0x140
>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:360
>> __mmu_notifier_invalidate_range_start+0x1f8/0x300 mm/mmu_notifier.c:199
>> mmu_notifier_invalidate_range_start include/linux/mmu_notifier.h:282 [inline]
>> unmap_vmas+0x14b/0x1b0 mm/memory.c:1368
>> unmap_region+0x2f8/0x560 mm/mmap.c:2460
>> do_munmap+0x7b8/0xfa0 mm/mmap.c:2657
>> mmap_region+0x68f/0x18e0 mm/mmap.c:1612
>> do_mmap+0x6a2/0xd40 mm/mmap.c:1450
>> do_mmap_pgoff include/linux/mm.h:2031 [inline]
>> vm_mmap_pgoff+0x1a9/0x200 mm/util.c:305
>> SYSC_mmap_pgoff mm/mmap.c:1500 [inline]
>> SyS_mmap_pgoff+0x22c/0x5d0 mm/mmap.c:1458
>> SYSC_mmap arch/x86/kernel/sys_x86_64.c:95 [inline]
>> SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:86
>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>> RIP: 0033:0x445329
>> RSP: 002b:00007fb33933cb58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
>> RAX: ffffffffffffffda RBX: 0000000020000000 RCX: 0000000000445329
>> RDX: 0000000000000003 RSI: 0000000000af1000 RDI: 0000000020000000
>> RBP: 00000000006dfe90 R08: ffffffffffffffff R09: 0000000000000000
>> R10: 0000000000000032 R11: 0000000000000282 R12: 0000000000700000
>> R13: 0000000000000006 R14: ffffffffffffffff R15: 0000000020001000
>> Memory state around the buggy address:
>> ffff880068ae1f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ffff880068ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> ffff880068ae2000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>> ^
>> ffff880068ae2080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>> ffff880068ae2100: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>> ==================================================================
>
> This could be related to the gfn_to_rmap issues.
Humm... That's possible. Potentially I am not seeing any more of
spte-related crashes after I applied the following patch:
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -968,8 +968,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
/* Check for overlaps */
r = -EEXIST;
kvm_for_each_memslot(slot, __kvm_memslots(kvm, as_id)) {
- if ((slot->id >= KVM_USER_MEM_SLOTS) ||
- (slot->id == id))
+ if (slot->id == id)
continue;
if (!((base_gfn + npages <= slot->base_gfn) ||
(base_gfn >= slot->base_gfn + slot->npages)))
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kvm: WARNING in mmu_spte_clear_track_bits
2017-01-17 16:00 ` Dmitry Vyukov
@ 2017-03-12 11:20 ` Dmitry Vyukov
2017-03-14 15:17 ` Radim Krčmář
0 siblings, 1 reply; 8+ messages in thread
From: Dmitry Vyukov @ 2017-03-12 11:20 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Radim Krčmář,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86, KVM list,
LKML, Alan Stern, Steve Rutherford, Xiao Guangrong,
Haozhong Zhang, syzkaller
On Tue, Jan 17, 2017 at 5:00 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Tue, Jan 17, 2017 at 4:20 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>>
>> On 13/01/2017 12:15, Dmitry Vyukov wrote:
>>>
>>> I've commented out the WARNING for now, but I am seeing lots of
>>> use-after-free's and rcu stalls involving mmu_spte_clear_track_bits:
>>>
>>>
>>> BUG: KASAN: use-after-free in mmu_spte_clear_track_bits+0x186/0x190
>>> arch/x86/kvm/mmu.c:597 at addr ffff880068ae2008
>>> Read of size 8 by task syz-executor2/16715
>>> page:ffffea00016e6170 count:0 mapcount:0 mapping: (null) index:0x0
>>> flags: 0x500000000000000()
>>> raw: 0500000000000000 0000000000000000 0000000000000000 00000000ffffffff
>>> raw: ffffea00017ec5a0 ffffea0001783d48 ffff88006aec5d98
>>> page dumped because: kasan: bad access detected
>>> CPU: 2 PID: 16715 Comm: syz-executor2 Not tainted 4.10.0-rc3+ #163
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>> Call Trace:
>>> __dump_stack lib/dump_stack.c:15 [inline]
>>> dump_stack+0x292/0x3a2 lib/dump_stack.c:51
>>> kasan_report_error mm/kasan/report.c:213 [inline]
>>> kasan_report+0x42d/0x460 mm/kasan/report.c:307
>>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:333
>>> mmu_spte_clear_track_bits+0x186/0x190 arch/x86/kvm/mmu.c:597
>>> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
>>> kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401
>>> kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412
>>> kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565
>>> kvm_unmap_hva_range+0x2e/0x40 arch/x86/kvm/mmu.c:1591
>>> kvm_mmu_notifier_invalidate_range_start+0xae/0x140
>>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:360
>>> __mmu_notifier_invalidate_range_start+0x1f8/0x300 mm/mmu_notifier.c:199
>>> mmu_notifier_invalidate_range_start include/linux/mmu_notifier.h:282 [inline]
>>> unmap_vmas+0x14b/0x1b0 mm/memory.c:1368
>>> unmap_region+0x2f8/0x560 mm/mmap.c:2460
>>> do_munmap+0x7b8/0xfa0 mm/mmap.c:2657
>>> mmap_region+0x68f/0x18e0 mm/mmap.c:1612
>>> do_mmap+0x6a2/0xd40 mm/mmap.c:1450
>>> do_mmap_pgoff include/linux/mm.h:2031 [inline]
>>> vm_mmap_pgoff+0x1a9/0x200 mm/util.c:305
>>> SYSC_mmap_pgoff mm/mmap.c:1500 [inline]
>>> SyS_mmap_pgoff+0x22c/0x5d0 mm/mmap.c:1458
>>> SYSC_mmap arch/x86/kernel/sys_x86_64.c:95 [inline]
>>> SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:86
>>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>>> RIP: 0033:0x445329
>>> RSP: 002b:00007fb33933cb58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
>>> RAX: ffffffffffffffda RBX: 0000000020000000 RCX: 0000000000445329
>>> RDX: 0000000000000003 RSI: 0000000000af1000 RDI: 0000000020000000
>>> RBP: 00000000006dfe90 R08: ffffffffffffffff R09: 0000000000000000
>>> R10: 0000000000000032 R11: 0000000000000282 R12: 0000000000700000
>>> R13: 0000000000000006 R14: ffffffffffffffff R15: 0000000020001000
>>> Memory state around the buggy address:
>>> ffff880068ae1f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> ffff880068ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>>> ffff880068ae2000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>> ^
>>> ffff880068ae2080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>> ffff880068ae2100: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>> ==================================================================
>>
>> This could be related to the gfn_to_rmap issues.
>
>
> Humm... That's possible. Potentially I am not seeing any more of
> spte-related crashes after I applied the following patch:
>
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -968,8 +968,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
> /* Check for overlaps */
> r = -EEXIST;
> kvm_for_each_memslot(slot, __kvm_memslots(kvm, as_id)) {
> - if ((slot->id >= KVM_USER_MEM_SLOTS) ||
> - (slot->id == id))
> + if (slot->id == id)
> continue;
> if (!((base_gfn + npages <= slot->base_gfn) ||
> (base_gfn >= slot->base_gfn + slot->npages)))
Friendly ping. Just hit it on
mmotm/86292b33d4b79ee03e2f43ea0381ef85f077c760 (without the above
change):
------------[ cut here ]------------
WARNING: CPU: 1 PID: 31060 at arch/x86/kvm/mmu.c:682
mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682
CPU: 1 PID: 31060 Comm: syz-executor0 Not tainted 4.11.0-rc1+ #328
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x1a7/0x26a lib/dump_stack.c:52
panic+0x1f8/0x40f kernel/panic.c:180
__warn+0x1c4/0x1e0 kernel/panic.c:541
warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682
drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1323
mmu_page_zap_pte+0x223/0x350 arch/x86/kvm/mmu.c:2438
kvm_mmu_page_unlink_children arch/x86/kvm/mmu.c:2460 [inline]
kvm_mmu_prepare_zap_page+0x1ce/0x13d0 arch/x86/kvm/mmu.c:2504
kvm_zap_obsolete_pages arch/x86/kvm/mmu.c:5134 [inline]
kvm_mmu_invalidate_zap_all_pages+0x4d4/0x6b0 arch/x86/kvm/mmu.c:5175
kvm_arch_flush_shadow_all+0x15/0x20 arch/x86/kvm/x86.c:8364
kvm_mmu_notifier_release+0x71/0xb0
arch/x86/kvm/../../../virt/kvm/kvm_main.c:472
__mmu_notifier_release+0x1e5/0x6b0 mm/mmu_notifier.c:75
mmu_notifier_release include/linux/mmu_notifier.h:235 [inline]
exit_mmap+0x3a3/0x470 mm/mmap.c:2941
__mmput kernel/fork.c:890 [inline]
mmput+0x228/0x700 kernel/fork.c:912
exit_mm kernel/exit.c:558 [inline]
do_exit+0x9e8/0x1c20 kernel/exit.c:866
do_group_exit+0x149/0x400 kernel/exit.c:983
get_signal+0x6d9/0x1840 kernel/signal.c:2318
do_signal+0x94/0x1f30 arch/x86/kernel/signal.c:808
exit_to_usermode_loop+0x1e5/0x2d0 arch/x86/entry/common.c:157
prepare_exit_to_usermode arch/x86/entry/common.c:191 [inline]
syscall_return_slowpath+0x3bd/0x460 arch/x86/entry/common.c:260
entry_SYSCALL_64_fastpath+0xc0/0xc2
RIP: 0033:0x4458d9
RSP: 002b:00007ffa472c3b58 EFLAGS: 00000286 ORIG_RAX: 00000000000000ce
RAX: fffffffffffffff4 RBX: 0000000000708000 RCX: 00000000004458d9
RDX: 0000000000000000 RSI: 000000002006bff8 RDI: 000000000000a05b
RBP: 0000000000000fe0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 00000000006df0a0
R13: 000000000000a05b R14: 000000002006bff8 R15: 0000000000000000
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kvm: WARNING in mmu_spte_clear_track_bits
2017-03-12 11:20 ` Dmitry Vyukov
@ 2017-03-14 15:17 ` Radim Krčmář
2017-03-23 16:39 ` Dmitry Vyukov
0 siblings, 1 reply; 8+ messages in thread
From: Radim Krčmář @ 2017-03-14 15:17 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Paolo Bonzini, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86,
KVM list, LKML, Alan Stern, Steve Rutherford, Xiao Guangrong,
Haozhong Zhang, syzkaller
2017-03-12 12:20+0100, Dmitry Vyukov:
> On Tue, Jan 17, 2017 at 5:00 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> On Tue, Jan 17, 2017 at 4:20 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>
>>>
>>> On 13/01/2017 12:15, Dmitry Vyukov wrote:
>>>>
>>>> I've commented out the WARNING for now, but I am seeing lots of
>>>> use-after-free's and rcu stalls involving mmu_spte_clear_track_bits:
>>>>
>>>>
>>>> BUG: KASAN: use-after-free in mmu_spte_clear_track_bits+0x186/0x190
>>>> arch/x86/kvm/mmu.c:597 at addr ffff880068ae2008
>>>> Read of size 8 by task syz-executor2/16715
>>>> page:ffffea00016e6170 count:0 mapcount:0 mapping: (null) index:0x0
>>>> flags: 0x500000000000000()
>>>> raw: 0500000000000000 0000000000000000 0000000000000000 00000000ffffffff
>>>> raw: ffffea00017ec5a0 ffffea0001783d48 ffff88006aec5d98
>>>> page dumped because: kasan: bad access detected
>>>> CPU: 2 PID: 16715 Comm: syz-executor2 Not tainted 4.10.0-rc3+ #163
>>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>>> Call Trace:
>>>> __dump_stack lib/dump_stack.c:15 [inline]
>>>> dump_stack+0x292/0x3a2 lib/dump_stack.c:51
>>>> kasan_report_error mm/kasan/report.c:213 [inline]
>>>> kasan_report+0x42d/0x460 mm/kasan/report.c:307
>>>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:333
>>>> mmu_spte_clear_track_bits+0x186/0x190 arch/x86/kvm/mmu.c:597
>>>> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
>>>> kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401
>>>> kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412
>>>> kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565
>>>> kvm_unmap_hva_range+0x2e/0x40 arch/x86/kvm/mmu.c:1591
>>>> kvm_mmu_notifier_invalidate_range_start+0xae/0x140
>>>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:360
>>>> __mmu_notifier_invalidate_range_start+0x1f8/0x300 mm/mmu_notifier.c:199
>>>> mmu_notifier_invalidate_range_start include/linux/mmu_notifier.h:282 [inline]
>>>> unmap_vmas+0x14b/0x1b0 mm/memory.c:1368
>>>> unmap_region+0x2f8/0x560 mm/mmap.c:2460
>>>> do_munmap+0x7b8/0xfa0 mm/mmap.c:2657
>>>> mmap_region+0x68f/0x18e0 mm/mmap.c:1612
>>>> do_mmap+0x6a2/0xd40 mm/mmap.c:1450
>>>> do_mmap_pgoff include/linux/mm.h:2031 [inline]
>>>> vm_mmap_pgoff+0x1a9/0x200 mm/util.c:305
>>>> SYSC_mmap_pgoff mm/mmap.c:1500 [inline]
>>>> SyS_mmap_pgoff+0x22c/0x5d0 mm/mmap.c:1458
>>>> SYSC_mmap arch/x86/kernel/sys_x86_64.c:95 [inline]
>>>> SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:86
>>>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>>>> RIP: 0033:0x445329
>>>> RSP: 002b:00007fb33933cb58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
>>>> RAX: ffffffffffffffda RBX: 0000000020000000 RCX: 0000000000445329
>>>> RDX: 0000000000000003 RSI: 0000000000af1000 RDI: 0000000020000000
>>>> RBP: 00000000006dfe90 R08: ffffffffffffffff R09: 0000000000000000
>>>> R10: 0000000000000032 R11: 0000000000000282 R12: 0000000000700000
>>>> R13: 0000000000000006 R14: ffffffffffffffff R15: 0000000020001000
>>>> Memory state around the buggy address:
>>>> ffff880068ae1f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>>> ffff880068ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>>>> ffff880068ae2000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>> ^
>>>> ffff880068ae2080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>> ffff880068ae2100: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>> ==================================================================
>>>
>>> This could be related to the gfn_to_rmap issues.
>>
>>
>> Humm... That's possible. Potentially I am not seeing any more of
>> spte-related crashes after I applied the following patch:
>>
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -968,8 +968,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
>> /* Check for overlaps */
>> r = -EEXIST;
>> kvm_for_each_memslot(slot, __kvm_memslots(kvm, as_id)) {
>> - if ((slot->id >= KVM_USER_MEM_SLOTS) ||
>> - (slot->id == id))
>> + if (slot->id == id)
>> continue;
>> if (!((base_gfn + npages <= slot->base_gfn) ||
>> (base_gfn >= slot->base_gfn + slot->npages)))
I don't understand how this fixes the test: the only memslot that the
test creates is at memory range 0x0-0x1000, which should not overlap
with any private memslots.
There should be just the IDENTITY_PAGETABLE_PRIVATE_MEMSLOT @
0xfffbc000ul.
Do you get any ouput with this hunk?
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index a17d78759727..7e1929432232 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -888,6 +888,14 @@ static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
return old_memslots;
}
+void kvm_dump_slot(struct kvm_memory_slot *slot)
+{
+ printk("kvm_memory_slot %p { .id = %u, .base_gfn = %#llx, .npages = %lu, "
+ ".userspace_addr = %#lx, .flags = %u, .dirty_bitmap = %p, .arch = ? }\n",
+ slot, slot->id, slot->base_gfn, slot->npages,
+ slot->userspace_addr, slot->flags, slot->dirty_bitmap);
+}
+
/*
* Allocate some memory and give it an address in the guest physical address
* space.
@@ -978,12 +986,14 @@ int __kvm_set_memory_region(struct kvm *kvm,
/* Check for overlaps */
r = -EEXIST;
kvm_for_each_memslot(slot, __kvm_memslots(kvm, as_id)) {
- if ((slot->id >= KVM_USER_MEM_SLOTS) ||
- (slot->id == id))
+ if (slot->id == id)
continue;
if (!((base_gfn + npages <= slot->base_gfn) ||
- (base_gfn >= slot->base_gfn + slot->npages)))
+ (base_gfn >= slot->base_gfn + slot->npages))) {
+ kvm_dump_slot(&new);
+ kvm_dump_slot(slot);
goto out;
+ }
}
}
> Friendly ping. Just hit it on
And the warning happens at mmap ... I can't reproduce, but does the bug
happen on the second mmap()? (Test line 210 when i = 0.)
The change above makes sense as memslots currently cannot overlap
anywhere. There are three private memslots that can cause this problem:
TSS, IDENTITY_MAP and APIC.
TSS and IDENTITY_MAP can be configured by userspace and must not
conflict by design, so we can safely enforce that.
APIC memslot doesn't provide such guarantees and should be overlaid over
any memory, but assuming that userspace doesn't configure memslots there
seems bearable.
Still, I'd like to understand why that patch would fix this bug.
Thanks.
> mmotm/86292b33d4b79ee03e2f43ea0381ef85f077c760 (without the above
> change):
>
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 31060 at arch/x86/kvm/mmu.c:682
> mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682
> CPU: 1 PID: 31060 Comm: syz-executor0 Not tainted 4.11.0-rc1+ #328
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:16 [inline]
> dump_stack+0x1a7/0x26a lib/dump_stack.c:52
> panic+0x1f8/0x40f kernel/panic.c:180
> __warn+0x1c4/0x1e0 kernel/panic.c:541
> warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
> mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682
> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1323
> mmu_page_zap_pte+0x223/0x350 arch/x86/kvm/mmu.c:2438
> kvm_mmu_page_unlink_children arch/x86/kvm/mmu.c:2460 [inline]
> kvm_mmu_prepare_zap_page+0x1ce/0x13d0 arch/x86/kvm/mmu.c:2504
> kvm_zap_obsolete_pages arch/x86/kvm/mmu.c:5134 [inline]
> kvm_mmu_invalidate_zap_all_pages+0x4d4/0x6b0 arch/x86/kvm/mmu.c:5175
> kvm_arch_flush_shadow_all+0x15/0x20 arch/x86/kvm/x86.c:8364
> kvm_mmu_notifier_release+0x71/0xb0
> arch/x86/kvm/../../../virt/kvm/kvm_main.c:472
> __mmu_notifier_release+0x1e5/0x6b0 mm/mmu_notifier.c:75
> mmu_notifier_release include/linux/mmu_notifier.h:235 [inline]
> exit_mmap+0x3a3/0x470 mm/mmap.c:2941
> __mmput kernel/fork.c:890 [inline]
> mmput+0x228/0x700 kernel/fork.c:912
> exit_mm kernel/exit.c:558 [inline]
> do_exit+0x9e8/0x1c20 kernel/exit.c:866
> do_group_exit+0x149/0x400 kernel/exit.c:983
> get_signal+0x6d9/0x1840 kernel/signal.c:2318
> do_signal+0x94/0x1f30 arch/x86/kernel/signal.c:808
> exit_to_usermode_loop+0x1e5/0x2d0 arch/x86/entry/common.c:157
> prepare_exit_to_usermode arch/x86/entry/common.c:191 [inline]
> syscall_return_slowpath+0x3bd/0x460 arch/x86/entry/common.c:260
> entry_SYSCALL_64_fastpath+0xc0/0xc2
> RIP: 0033:0x4458d9
> RSP: 002b:00007ffa472c3b58 EFLAGS: 00000286 ORIG_RAX: 00000000000000ce
> RAX: fffffffffffffff4 RBX: 0000000000708000 RCX: 00000000004458d9
> RDX: 0000000000000000 RSI: 000000002006bff8 RDI: 000000000000a05b
> RBP: 0000000000000fe0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000286 R12: 00000000006df0a0
> R13: 000000000000a05b R14: 000000002006bff8 R15: 0000000000000000
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: kvm: WARNING in mmu_spte_clear_track_bits
2017-03-14 15:17 ` Radim Krčmář
@ 2017-03-23 16:39 ` Dmitry Vyukov
0 siblings, 0 replies; 8+ messages in thread
From: Dmitry Vyukov @ 2017-03-23 16:39 UTC (permalink / raw)
To: Radim Krčmář
Cc: Paolo Bonzini, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86,
KVM list, LKML, Alan Stern, Steve Rutherford, Xiao Guangrong,
Haozhong Zhang, syzkaller
On Tue, Mar 14, 2017 at 4:17 PM, Radim Krčmář <rkrcmar@redhat.com> wrote:
> 2017-03-12 12:20+0100, Dmitry Vyukov:
>> On Tue, Jan 17, 2017 at 5:00 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>>> On Tue, Jan 17, 2017 at 4:20 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>>
>>>>
>>>> On 13/01/2017 12:15, Dmitry Vyukov wrote:
>>>>>
>>>>> I've commented out the WARNING for now, but I am seeing lots of
>>>>> use-after-free's and rcu stalls involving mmu_spte_clear_track_bits:
>>>>>
>>>>>
>>>>> BUG: KASAN: use-after-free in mmu_spte_clear_track_bits+0x186/0x190
>>>>> arch/x86/kvm/mmu.c:597 at addr ffff880068ae2008
>>>>> Read of size 8 by task syz-executor2/16715
>>>>> page:ffffea00016e6170 count:0 mapcount:0 mapping: (null) index:0x0
>>>>> flags: 0x500000000000000()
>>>>> raw: 0500000000000000 0000000000000000 0000000000000000 00000000ffffffff
>>>>> raw: ffffea00017ec5a0 ffffea0001783d48 ffff88006aec5d98
>>>>> page dumped because: kasan: bad access detected
>>>>> CPU: 2 PID: 16715 Comm: syz-executor2 Not tainted 4.10.0-rc3+ #163
>>>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>>>> Call Trace:
>>>>> __dump_stack lib/dump_stack.c:15 [inline]
>>>>> dump_stack+0x292/0x3a2 lib/dump_stack.c:51
>>>>> kasan_report_error mm/kasan/report.c:213 [inline]
>>>>> kasan_report+0x42d/0x460 mm/kasan/report.c:307
>>>>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:333
>>>>> mmu_spte_clear_track_bits+0x186/0x190 arch/x86/kvm/mmu.c:597
>>>>> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
>>>>> kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401
>>>>> kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412
>>>>> kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565
>>>>> kvm_unmap_hva_range+0x2e/0x40 arch/x86/kvm/mmu.c:1591
>>>>> kvm_mmu_notifier_invalidate_range_start+0xae/0x140
>>>>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:360
>>>>> __mmu_notifier_invalidate_range_start+0x1f8/0x300 mm/mmu_notifier.c:199
>>>>> mmu_notifier_invalidate_range_start include/linux/mmu_notifier.h:282 [inline]
>>>>> unmap_vmas+0x14b/0x1b0 mm/memory.c:1368
>>>>> unmap_region+0x2f8/0x560 mm/mmap.c:2460
>>>>> do_munmap+0x7b8/0xfa0 mm/mmap.c:2657
>>>>> mmap_region+0x68f/0x18e0 mm/mmap.c:1612
>>>>> do_mmap+0x6a2/0xd40 mm/mmap.c:1450
>>>>> do_mmap_pgoff include/linux/mm.h:2031 [inline]
>>>>> vm_mmap_pgoff+0x1a9/0x200 mm/util.c:305
>>>>> SYSC_mmap_pgoff mm/mmap.c:1500 [inline]
>>>>> SyS_mmap_pgoff+0x22c/0x5d0 mm/mmap.c:1458
>>>>> SYSC_mmap arch/x86/kernel/sys_x86_64.c:95 [inline]
>>>>> SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:86
>>>>> entry_SYSCALL_64_fastpath+0x1f/0xc2
>>>>> RIP: 0033:0x445329
>>>>> RSP: 002b:00007fb33933cb58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009
>>>>> RAX: ffffffffffffffda RBX: 0000000020000000 RCX: 0000000000445329
>>>>> RDX: 0000000000000003 RSI: 0000000000af1000 RDI: 0000000020000000
>>>>> RBP: 00000000006dfe90 R08: ffffffffffffffff R09: 0000000000000000
>>>>> R10: 0000000000000032 R11: 0000000000000282 R12: 0000000000700000
>>>>> R13: 0000000000000006 R14: ffffffffffffffff R15: 0000000020001000
>>>>> Memory state around the buggy address:
>>>>> ffff880068ae1f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>>>> ffff880068ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>>>>> ffff880068ae2000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>> ^
>>>>> ffff880068ae2080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>> ffff880068ae2100: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>>>>> ==================================================================
>>>>
>>>> This could be related to the gfn_to_rmap issues.
>>>
>>>
>>> Humm... That's possible. Potentially I am not seeing any more of
>>> spte-related crashes after I applied the following patch:
>>>
>>> --- a/virt/kvm/kvm_main.c
>>> +++ b/virt/kvm/kvm_main.c
>>> @@ -968,8 +968,7 @@ int __kvm_set_memory_region(struct kvm *kvm,
>>> /* Check for overlaps */
>>> r = -EEXIST;
>>> kvm_for_each_memslot(slot, __kvm_memslots(kvm, as_id)) {
>>> - if ((slot->id >= KVM_USER_MEM_SLOTS) ||
>>> - (slot->id == id))
>>> + if (slot->id == id)
>>> continue;
>>> if (!((base_gfn + npages <= slot->base_gfn) ||
>>> (base_gfn >= slot->base_gfn + slot->npages)))
>
> I don't understand how this fixes the test: the only memslot that the
> test creates is at memory range 0x0-0x1000, which should not overlap
> with any private memslots.
> There should be just the IDENTITY_PAGETABLE_PRIVATE_MEMSLOT @
> 0xfffbc000ul.
>
> Do you get any ouput with this hunk?
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index a17d78759727..7e1929432232 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -888,6 +888,14 @@ static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
> return old_memslots;
> }
>
> +void kvm_dump_slot(struct kvm_memory_slot *slot)
> +{
> + printk("kvm_memory_slot %p { .id = %u, .base_gfn = %#llx, .npages = %lu, "
> + ".userspace_addr = %#lx, .flags = %u, .dirty_bitmap = %p, .arch = ? }\n",
> + slot, slot->id, slot->base_gfn, slot->npages,
> + slot->userspace_addr, slot->flags, slot->dirty_bitmap);
> +}
> +
> /*
> * Allocate some memory and give it an address in the guest physical address
> * space.
> @@ -978,12 +986,14 @@ int __kvm_set_memory_region(struct kvm *kvm,
> /* Check for overlaps */
> r = -EEXIST;
> kvm_for_each_memslot(slot, __kvm_memslots(kvm, as_id)) {
> - if ((slot->id >= KVM_USER_MEM_SLOTS) ||
> - (slot->id == id))
> + if (slot->id == id)
> continue;
> if (!((base_gfn + npages <= slot->base_gfn) ||
> - (base_gfn >= slot->base_gfn + slot->npages)))
> + (base_gfn >= slot->base_gfn + slot->npages))) {
> + kvm_dump_slot(&new);
> + kvm_dump_slot(slot);
> goto out;
> + }
> }
> }
>
>
>> Friendly ping. Just hit it on
>
> And the warning happens at mmap ... I can't reproduce, but does the bug
> happen on the second mmap()? (Test line 210 when i = 0.)
>
> The change above makes sense as memslots currently cannot overlap
> anywhere. There are three private memslots that can cause this problem:
> TSS, IDENTITY_MAP and APIC.
>
> TSS and IDENTITY_MAP can be configured by userspace and must not
> conflict by design, so we can safely enforce that.
> APIC memslot doesn't provide such guarantees and should be overlaid over
> any memory, but assuming that userspace doesn't configure memslots there
> seems bearable.
>
> Still, I'd like to understand why that patch would fix this bug.
>
> Thanks.
Humm... I cannot reproduce it anymore. Maybe it was fixed by something else...
However this looks very close and is still not fixed:
https://groups.google.com/d/msg/syzkaller/IqkesiRS-t0/aLcJuMXqBgAJ
Maybe it's another reincarnation of the same problem...
>> mmotm/86292b33d4b79ee03e2f43ea0381ef85f077c760 (without the above
>> change):
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 1 PID: 31060 at arch/x86/kvm/mmu.c:682
>> mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682
>> CPU: 1 PID: 31060 Comm: syz-executor0 Not tainted 4.11.0-rc1+ #328
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> Call Trace:
>> __dump_stack lib/dump_stack.c:16 [inline]
>> dump_stack+0x1a7/0x26a lib/dump_stack.c:52
>> panic+0x1f8/0x40f kernel/panic.c:180
>> __warn+0x1c4/0x1e0 kernel/panic.c:541
>> warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
>> mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682
>> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1323
>> mmu_page_zap_pte+0x223/0x350 arch/x86/kvm/mmu.c:2438
>> kvm_mmu_page_unlink_children arch/x86/kvm/mmu.c:2460 [inline]
>> kvm_mmu_prepare_zap_page+0x1ce/0x13d0 arch/x86/kvm/mmu.c:2504
>> kvm_zap_obsolete_pages arch/x86/kvm/mmu.c:5134 [inline]
>> kvm_mmu_invalidate_zap_all_pages+0x4d4/0x6b0 arch/x86/kvm/mmu.c:5175
>> kvm_arch_flush_shadow_all+0x15/0x20 arch/x86/kvm/x86.c:8364
>> kvm_mmu_notifier_release+0x71/0xb0
>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:472
>> __mmu_notifier_release+0x1e5/0x6b0 mm/mmu_notifier.c:75
>> mmu_notifier_release include/linux/mmu_notifier.h:235 [inline]
>> exit_mmap+0x3a3/0x470 mm/mmap.c:2941
>> __mmput kernel/fork.c:890 [inline]
>> mmput+0x228/0x700 kernel/fork.c:912
>> exit_mm kernel/exit.c:558 [inline]
>> do_exit+0x9e8/0x1c20 kernel/exit.c:866
>> do_group_exit+0x149/0x400 kernel/exit.c:983
>> get_signal+0x6d9/0x1840 kernel/signal.c:2318
>> do_signal+0x94/0x1f30 arch/x86/kernel/signal.c:808
>> exit_to_usermode_loop+0x1e5/0x2d0 arch/x86/entry/common.c:157
>> prepare_exit_to_usermode arch/x86/entry/common.c:191 [inline]
>> syscall_return_slowpath+0x3bd/0x460 arch/x86/entry/common.c:260
>> entry_SYSCALL_64_fastpath+0xc0/0xc2
>> RIP: 0033:0x4458d9
>> RSP: 002b:00007ffa472c3b58 EFLAGS: 00000286 ORIG_RAX: 00000000000000ce
>> RAX: fffffffffffffff4 RBX: 0000000000708000 RCX: 00000000004458d9
>> RDX: 0000000000000000 RSI: 000000002006bff8 RDI: 000000000000a05b
>> RBP: 0000000000000fe0 R08: 0000000000000000 R09: 0000000000000000
>> R10: 0000000000000000 R11: 0000000000000286 R12: 00000000006df0a0
>> R13: 000000000000a05b R14: 000000002006bff8 R15: 0000000000000000
^ permalink raw reply [flat|nested] 8+ messages in thread
* kvm: WARNING in mmu_spte_clear_track_bits
@ 2017-01-23 14:19 Dmitry Vyukov
0 siblings, 0 replies; 8+ messages in thread
From: Dmitry Vyukov @ 2017-01-23 14:19 UTC (permalink / raw)
To: Paolo Bonzini, Radim Krčmář,
KVM list, LKML, Steve Rutherford
Cc: syzkaller
Hello,
I've started seeing the following WARNING while running syzkaller fuzzer:
Out of memory: Kill process 30627 (syz-execprog) score 57 or sacrifice child
Killed process 30962 (syz-executor) total-vm:20996kB, anon-rss:64kB,
file-rss:0kB, shmem-rss:0kB
oom_reaper: reaped process 3916 (syz-executor), now anon-rss:0kB,
file-rss:0kB, shmem-rss:4kB
------------[ cut here ]------------
WARNING: CPU: 8 PID: 3916 at arch/x86/kvm/mmu.c:614
mmu_spte_clear_track_bits+0x32d/0x3a0 arch/x86/kvm/mmu.c:614
Kernel panic - not syncing: panic_on_warn set ...
CPU: 8 PID: 3916 Comm: syz-executor Not tainted 4.10.0-rc5+ #186
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
panic+0x1fb/0x412 kernel/panic.c:179
__warn+0x1c4/0x1e0 kernel/panic.c:539
warn_slowpath_null+0x2c/0x40 kernel/panic.c:582
mmu_spte_clear_track_bits+0x32d/0x3a0 arch/x86/kvm/mmu.c:614
drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182
mmu_page_zap_pte+0x204/0x300 arch/x86/kvm/mmu.c:2298
kvm_mmu_page_unlink_children arch/x86/kvm/mmu.c:2320 [inline]
kvm_mmu_prepare_zap_page+0x1b6/0x1320 arch/x86/kvm/mmu.c:2364
kvm_zap_obsolete_pages arch/x86/kvm/mmu.c:4932 [inline]
kvm_mmu_invalidate_zap_all_pages+0x488/0x660 arch/x86/kvm/mmu.c:4973
kvm_arch_flush_shadow_all+0x15/0x20 arch/x86/kvm/x86.c:8264
kvm_mmu_notifier_release+0x71/0xb0
arch/x86/kvm/../../../virt/kvm/kvm_main.c:470
__mmu_notifier_release+0x1f9/0x6c0 mm/mmu_notifier.c:74
mmu_notifier_release include/linux/mmu_notifier.h:235 [inline]
exit_mmap+0x3cc/0x490 mm/mmap.c:2918
__mmput kernel/fork.c:873 [inline]
mmput+0x22b/0x6e0 kernel/fork.c:895
exit_mm kernel/exit.c:521 [inline]
do_exit+0x9cf/0x28a0 kernel/exit.c:826
do_group_exit+0x149/0x420 kernel/exit.c:943
get_signal+0x7e0/0x1820 kernel/signal.c:2313
do_signal+0xd2/0x2190 arch/x86/kernel/signal.c:807
exit_to_usermode_loop+0x200/0x2a0 arch/x86/entry/common.c:156
syscall_return_slowpath arch/x86/entry/common.c:190 [inline]
do_syscall_64+0x6fc/0x930 arch/x86/entry/common.c:285
entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x447859
RSP: 002b:0000000001a2fbd0 EFLAGS: 00000202 ORIG_RAX: 0000000000000038
RAX: fffffffffffffdff RBX: 00007fbd700c9700 RCX: 0000000000447859
RDX: 00007fbd700c99d0 RSI: 00007fbd700c8db0 RDI: 00000000003d0f00
RBP: 0000000001a2fcd8 R08: 00007fbd700c9700 R09: 00007fbd700c9700
R10: 00007fbd700c99d0 R11: 0000000000000202 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fbd700c99c0 R15: 00007fbd700c9700
These warnings are always preceded by oom reaping the process. So
that's probably related.
On commit 7a308bb3016f57e5be11a677d15b821536419d36 (Jan 22).
I also have some local kvm-related changes on top to fix/silence known
kvm bugs, but hey should be unrealted:
https://gist.githubusercontent.com/dvyukov/fab99e9750bd93eefdab90418f04ea3d/raw/86a4e1475d73e8447782da6dc1824b2b7b31fa9e/gistfile1.txt
FTR, this was reproduced by running 150 of the following programs in
parallel in a VM with 512MB of memory:
https://gist.githubusercontent.com/dvyukov/cdf6dd9f7dbf9b207f836eac088dc11d/raw/686d73ec312492428bedd9f6438f28a7f559a916/gistfile1.txt
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-03-23 16:39 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-12-13 19:50 kvm: WARNING in mmu_spte_clear_track_bits Dmitry Vyukov
2017-01-13 11:15 ` Dmitry Vyukov
2017-01-17 15:20 ` Paolo Bonzini
2017-01-17 16:00 ` Dmitry Vyukov
2017-03-12 11:20 ` Dmitry Vyukov
2017-03-14 15:17 ` Radim Krčmář
2017-03-23 16:39 ` Dmitry Vyukov
2017-01-23 14:19 Dmitry Vyukov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).