From: Zdenek Kaspar <zkaspar82@gmail.com>
To: "kvm@vger.kernel.org" <kvm@vger.kernel.org>
Subject: Re: Bad performance since 5.9-rc1
Date: Fri, 18 Dec 2020 20:33:10 +0100 [thread overview]
Message-ID: <20201218203310.5025c17e.zkaspar82@gmail.com> (raw)
In-Reply-To: <20201201073537.6749e2d7.zkaspar82@gmail.com>
On Tue, 1 Dec 2020 07:35:37 +0100
Zdenek Kaspar <zkaspar82@gmail.com> wrote:
> On Thu, 19 Nov 2020 04:05:26 +0100
> Zdenek Kaspar <zkaspar82@gmail.com> wrote:
>
> > Hi,
> >
> > in my initial report
> > (https://marc.info/?l=kvm&m=160502183220080&w=2 - now fixed by
> > c887c9b9ca62c051d339b1c7b796edf2724029ed) I saw degraded
> > performance going back somewhere between v5.8 - v5.9-rc1.
> >
> > OpenBSD 6.8 (GENERIC.MP) guest performance (time ./test-build.sh)
> > good: 0m13.54s real 0m10.51s user 0m10.96s system
> > bad : 6m20.07s real 11m42.93s user 0m13.57s system
> >
> > bisected to first bad commit:
> > 6b82ef2c9cf18a48726e4bb359aa9014632f6466
> >
> > git bisect log:
> > # bad: [e47c4aee5bde03e7018f4fde45ba21028a8f8438] KVM: x86/mmu:
> > Rename page_header() to to_shadow_page() # good:
> > [01c3b2b5cdae39af8dfcf6e40fdf484ae0e812c5] KVM: SVM: Rename
> > svm_nested_virtualize_tpr() to nested_svm_virtualize_tpr() git
> > bisect start 'e47c4aee5bde' '01c3b2b5cdae' # bad:
> > [ebdb292dac7993425c8e31e2c21c9978e914a676] KVM: x86/mmu: Batch zap
> > MMU pages when shrinking the slab git bisect bad
> > ebdb292dac7993425c8e31e2c21c9978e914a676 # good:
> > [fb58a9c345f645f1774dcf6a36fda169253008ae] KVM: x86/mmu: Optimize
> > MMU page cache lookup for fully direct MMUs git bisect good
> > fb58a9c345f645f1774dcf6a36fda169253008ae # bad:
> > [6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap
> > MMU pages when recycling oldest pages git bisect bad
> > 6b82ef2c9cf18a48726e4bb359aa9014632f6466 # good:
> > [f95eec9bed76d42194c23153cb1cc8f186bf91cb] KVM: x86/mmu: Don't put
> > invalid SPs back on the list of active pages git bisect good
> > f95eec9bed76d42194c23153cb1cc8f186bf91cb # first bad commit:
> > [6b82ef2c9cf18a48726e4bb359aa9014632f6466] KVM: x86/mmu: Batch zap
> > MMU pages when recycling oldest pages
> >
> > Host machine is old Intel Core2 without EPT (TDP).
> >
> > TIA, Z.
>
> Hi, with v5.10-rc6:
> get_mmio_spte: detect reserved bits on spte, addr 0xfe00d000, dump
> hierarchy: ------ spte 0x8000030e level 3.
> ------ spte 0xaf82027 level 2.
> ------ spte 0x2038001ffe00d407 level 1.
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 355 at kvm_mmu_page_fault.cold+0x42/0x4f [kvm]
> ...
> CPU: 1 PID: 355 Comm: qemu-build Not tainted 5.10.0-rc6-amd64 #1
> Hardware name: /DG35EC, BIOS ECG3510M.86A.0118.2010.0113.1426
> 01/13/2010 RIP: 0010:kvm_mmu_page_fault.cold+0x42/0x4f [kvm]
> Code: e2 ec 44 8b 04 24 8b 5c 24 0c 44 89 c5 89 da 83 eb 01 48 c7 c7
> 20 b2 65 c0 48 63 c3 48 8b 74 c4 30 e8 dd 74 e2 ec 39 dd 7e e3 <0f>
> 0b 41 b8 ea ff ff ff e9 27 99 ff ff 0f 0b 48 8b 54 24 10 48 83 RSP:
> 0018:ffffb67400653d30 EFLAGS: 00010202 RAX: 0000000000000027 RBX:
> 0000000000000000 RCX: ffffa271ff2976f8 RDX: 00000000ffffffd8 RSI:
> 0000000000000027 RDI: ffffa271ff2976f0 RBP: 0000000000000001 R08:
> ffffffffadd02ae8 R09: 0000000000000003 R10: 00000000ffffe000 R11:
> 3fffffffffffffff R12: 00000000fe00d000 R13: 0000000000000000 R14:
> 0000000000000000 R15: 0000000000000001 FS: 00007fc10ae3d640(0000)
> GS:ffffa271ff280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000
> ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3:
> 0000000002dc2000 CR4: 00000000000026e0 Call Trace:
> kvm_arch_vcpu_ioctl_run+0xbaf/0x18f0 [kvm] ? do_futex+0x7c4/0xb80
> kvm_vcpu_ioctl+0x203/0x520 [kvm]
> ? set_next_entity+0x5b/0x80
> ? __switch_to_asm+0x32/0x60
> ? finish_task_switch+0x70/0x260
> __x64_sys_ioctl+0x338/0x720
> ? __x64_sys_futex+0x120/0x190
> do_syscall_64+0x33/0x40
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x7fc10c389f6b
> Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0
> 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48>
> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 ae 0c 00 f7 d8 64 89 01 48 RSP:
> 002b:00007fc10ae3c628 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fc10c389f6b
> RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012
> RBP: 000055ad3767baf0 R08: 000055ad36be4850 R09: 00000000000000ff
> R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> R13: 000055ad371d9800 R14: 0000000000000001 R15: 0000000000000002
> ---[ end trace c5f7ae690f5abcc4 ]---
>
> without: kvm: x86/mmu: Fix get_mmio_spte() on CPUs supporting 5-level
> PT I can run guest again, but with degraded performance as before.
>
> Z.
With: KVM: x86/mmu: Bug fixes and cleanups in get_mmio_spte() series
I can run guest again and performance is slightly better:
v5.8: 0m13.54s real 0m10.51s user 0m10.96s system
v5.9: 6m20.07s real 11m42.93s user 0m13.57s system
v5.10+fixes: 5m50.77s real 10m38.29s user 0m15.96s system
perf top from host when guest (openbsd) is compiling:
26.85% [kernel] [k] queued_spin_lock_slowpath
8.49% [kvm] [k] mmu_page_zap_pte
7.47% [kvm] [k] __kvm_mmu_prepare_zap_page
3.61% [kernel] [k] clear_page_rep
2.43% [kernel] [k] page_counter_uncharge
2.30% [kvm] [k] paging64_page_fault
2.03% [kvm_intel] [k] vmx_vcpu_run
2.02% [kvm] [k] kvm_vcpu_gfn_to_memslot
1.95% [kernel] [k] internal_get_user_pages_fast
1.64% [kvm] [k] kvm_mmu_get_page
1.55% [kernel] [k] page_counter_try_charge
1.33% [kernel] [k] propagate_protected_usage
1.29% [kvm] [k] kvm_arch_vcpu_ioctl_run
1.13% [kernel] [k] get_page_from_freelist
1.01% [kvm] [k] paging64_walk_addr_generic
0.83% [kernel] [k] ___slab_alloc.constprop.0
0.83% [kernel] [k] kmem_cache_free
0.82% [kvm] [k] __pte_list_remove
0.77% [kernel] [k] try_grab_compound_head
0.76% [kvm_intel] [k] 0x000000000001cfa0
0.74% [kvm] [k] pte_list_add
HTH, Z.
next prev parent reply other threads:[~2020-12-18 19:34 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-19 3:05 Bad performance since 5.9-rc1 Zdenek Kaspar
2020-12-01 6:35 ` Zdenek Kaspar
2020-12-18 19:33 ` Zdenek Kaspar [this message]
2020-12-21 19:41 ` Sean Christopherson
2020-12-21 21:13 ` Zdenek Kaspar
2020-12-22 17:07 ` Sean Christopherson
2020-12-22 21:26 ` Zdenek Kaspar
2021-01-12 11:18 ` Zdenek Kaspar
2021-01-13 20:17 ` Sean Christopherson
2021-01-13 22:17 ` Zdenek Kaspar
2020-12-02 0:31 ` Sean Christopherson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201218203310.5025c17e.zkaspar82@gmail.com \
--to=zkaspar82@gmail.com \
--cc=kvm@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).