From: Keqian Zhu <zhukeqian1@huawei.com>
To: <linux-kernel@vger.kernel.org>,
<linux-arm-kernel@lists.infradead.org>, <kvm@vger.kernel.org>,
<kvmarm@lists.cs.columbia.edu>, Marc Zyngier <maz@kernel.org>,
Will Deacon <will@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
Kirti Wankhede <kwankhede@nvidia.com>,
Cornelia Huck <cohuck@redhat.com>,
Mark Rutland <mark.rutland@arm.com>,
James Morse <james.morse@arm.com>,
Robin Murphy <robin.murphy@arm.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
<wanghaibin.wang@huawei.com>, <jiangkunkun@huawei.com>,
<xiexiangyou@huawei.com>, <zhengchuan@huawei.com>,
<yubihong@huawei.com>
Subject: Re: [RFC PATCH 0/7] kvm: arm64: Implement SW/HW combined dirty log
Date: Tue, 2 Mar 2021 19:23:05 +0800 [thread overview]
Message-ID: <4716a83b-5dad-4dbc-6661-e1f05abbd29c@huawei.com> (raw)
In-Reply-To: <20210126124444.27136-1-zhukeqian1@huawei.com>
Hi everyone,
Any comments are welcome :).
Thanks,
Keqian
On 2021/1/26 20:44, Keqian Zhu wrote:
> The intention:
>
> On arm64 platform, we tracking dirty log of vCPU through guest memory abort.
> KVM occupys some vCPU time of guest to change stage2 mapping and mark dirty.
> This leads to heavy side effect on VM, especially when multi vCPU race and
> some of them block on kvm mmu_lock.
>
> DBM is a HW auxiliary approach to log dirty. MMU chages PTE to be writable if
> its DBM bit is set. Then KVM doesn't occupy vCPU time to log dirty.
>
> About this patch series:
>
> The biggest problem of apply DBM for stage2 is that software must scan PTs to
> collect dirty state, which may cost much time and affect downtime of migration.
>
> This series realize a SW/HW combined dirty log that can effectively solve this
> problem (The smmu side can also use this approach to solve dma dirty log tracking).
>
> The core idea is that we do not enable hardware dirty at start (do not add DBM bit).
> When a arbitrary PT occurs fault, we execute soft tracking for this PT and enable
> hardware tracking for its *nearby* PTs (e.g. Add DBM bit for nearby 16PTs). Then when
> sync dirty log, we have known all PTs with hardware dirty enabled, so we do not need
> to scan all PTs.
>
> mem abort point mem abort point
> ↓ ↓
> ---------------------------------------------------------------
> |********| | |********| | |
> ---------------------------------------------------------------
> ↑ ↑
> set DBM bit of set DBM bit of
> this PT section (64PTEs) this PT section (64PTEs)
>
> We may worry that when dirty rate is over-high we still need to scan too much PTs.
> We mainly concern the VM stop time. With Qemu dirty rate throttling, the dirty memory
> is closing to the VM stop threshold, so there is a little PTs to scan after VM stop.
>
> It has the advantages of hardware tracking that minimizes side effect on vCPU,
> and also has the advantages of software tracking that controls vCPU dirty rate.
> Moreover, software tracking helps us to scan PTs at some fixed points, which
> greatly reduces scanning time. And the biggest benefit is that we can apply this
> solution for dma dirty tracking.
>
> Test:
>
> Host: Kunpeng 920 with 128 CPU 512G RAM. Disable Transparent Hugepage (Ensure test result
> is not effected by dissolve of block page table at the early stage of migration).
> VM: 16 CPU 16GB RAM. Run 4 pair of (redis_benchmark+redis_server).
>
> Each run 5 times for software dirty log and SW/HW conbined dirty log.
>
> Test result:
>
> Gain 5%~7% improvement of redis QPS during VM migration.
> VM downtime is not affected fundamentally.
> About 56.7% of DBM is effectively used.
>
> Keqian Zhu (7):
> arm64: cpufeature: Add API to report system support of HWDBM
> kvm: arm64: Use atomic operation when update PTE
> kvm: arm64: Add level_apply parameter for stage2_attr_walker
> kvm: arm64: Add some HW_DBM related pgtable interfaces
> kvm: arm64: Add some HW_DBM related mmu interfaces
> kvm: arm64: Only write protect selected PTE
> kvm: arm64: Start up SW/HW combined dirty log
>
> arch/arm64/include/asm/cpufeature.h | 12 +++
> arch/arm64/include/asm/kvm_host.h | 6 ++
> arch/arm64/include/asm/kvm_mmu.h | 7 ++
> arch/arm64/include/asm/kvm_pgtable.h | 45 ++++++++++
> arch/arm64/kvm/arm.c | 125 ++++++++++++++++++++++++++
> arch/arm64/kvm/hyp/pgtable.c | 130 ++++++++++++++++++++++-----
> arch/arm64/kvm/mmu.c | 47 +++++++++-
> arch/arm64/kvm/reset.c | 8 +-
> 8 files changed, 351 insertions(+), 29 deletions(-)
>
prev parent reply other threads:[~2021-03-02 16:07 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-26 12:44 [RFC PATCH 0/7] kvm: arm64: Implement SW/HW combined dirty log Keqian Zhu
2021-01-26 12:44 ` [RFC PATCH 1/7] arm64: cpufeature: Add API to report system support of HWDBM Keqian Zhu
2021-01-26 12:44 ` [RFC PATCH 2/7] kvm: arm64: Use atomic operation when update PTE Keqian Zhu
2021-01-26 12:44 ` [RFC PATCH 3/7] kvm: arm64: Add level_apply parameter for stage2_attr_walker Keqian Zhu
2021-01-26 12:44 ` [RFC PATCH 4/7] kvm: arm64: Add some HW_DBM related pgtable interfaces Keqian Zhu
2021-01-26 12:44 ` [RFC PATCH 5/7] kvm: arm64: Add some HW_DBM related mmu interfaces Keqian Zhu
2021-01-26 12:44 ` [RFC PATCH 6/7] kvm: arm64: Only write protect selected PTE Keqian Zhu
2021-01-26 12:44 ` [RFC PATCH 7/7] kvm: arm64: Start up SW/HW combined dirty log Keqian Zhu
2021-02-01 13:12 ` [RFC PATCH 0/7] kvm: arm64: Implement " Keqian Zhu
2021-02-01 13:17 ` Marc Zyngier
2021-02-01 13:25 ` Keqian Zhu
2021-03-02 11:23 ` Keqian Zhu [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4716a83b-5dad-4dbc-6661-e1f05abbd29c@huawei.com \
--to=zhukeqian1@huawei.com \
--cc=alex.williamson@redhat.com \
--cc=catalin.marinas@arm.com \
--cc=cohuck@redhat.com \
--cc=james.morse@arm.com \
--cc=jiangkunkun@huawei.com \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=kwankhede@nvidia.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=maz@kernel.org \
--cc=robin.murphy@arm.com \
--cc=suzuki.poulose@arm.com \
--cc=wanghaibin.wang@huawei.com \
--cc=will@kernel.org \
--cc=xiexiangyou@huawei.com \
--cc=yubihong@huawei.com \
--cc=zhengchuan@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).