All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC PATCH 0/3] ARM64: Guest performance improvement during dirty
@ 2022-01-10 21:04 ` Jing Zhang
  0 siblings, 0 replies; 40+ messages in thread
From: Jing Zhang @ 2022-01-10 21:04 UTC (permalink / raw)
  To: KVM, KVMARM, Marc Zyngier, Will Deacon, Paolo Bonzini,
	David Matlack, Oliver Upton, Reiji Watanabe
  Cc: Jing Zhang

This patch is to reduce the performance degradation of guest workload during
dirty logging on ARM64. A fast path is added to handle permission relaxation
during dirty logging. The MMU lock is replaced with rwlock, by which all
permision relaxations on leaf pte can be performed under the read lock. This
greatly reduces the MMU lock contention during dirty logging. With this
solution, the source guest workload performance degradation can be improved
by more than 60%.

Problem:
  * A Google internal live migration test shows that the source guest workload
  performance has >99% degradation for about 105 seconds, >50% degradation
  for about 112 seconds, >10% degradation for about 112 seconds on ARM64.
  This shows that most of the time, the guest workload degradtion is above
  99%, which obviously needs some improvement compared to the test result
  on x86 (>99% for 6s, >50% for 9s, >10% for 27s).
  * Tested H/W: Ampere Altra 3GHz, #CPU: 64, #Mem: 256GB
  * VM spec: #vCPU: 48, #Mem/vCPU: 4GB

Analysis:
  * We enabled CONFIG_LOCK_STAT in kernel and used dirty_log_perf_test to get
    the number of contentions of MMU lock and the "dirty memory time" on
    various VM spec.
    By using test command
    ./dirty_log_perf_test -b 2G -m 2 -i 2 -s anonymous_hugetlb_2mb -v [#vCPU]
    Below are the results:
    +-------+------------------------+-----------------------+
    | #vCPU | dirty memory time (ms) | number of contentions |
    +-------+------------------------+-----------------------+
    | 1     | 926                    | 0                     |
    +-------+------------------------+-----------------------+
    | 2     | 1189                   | 4732558               |
    +-------+------------------------+-----------------------+
    | 4     | 2503                   | 11527185              |
    +-------+------------------------+-----------------------+
    | 8     | 5069                   | 24881677              |
    +-------+------------------------+-----------------------+
    | 16    | 10340                  | 50347956              |
    +-------+------------------------+-----------------------+
    | 32    | 20351                  | 100605720             |
    +-------+------------------------+-----------------------+
    | 64    | 40994                  | 201442478             |
    +-------+------------------------+-----------------------+

  * From the test results above, the "dirty memory time" and the number of
    MMU lock contention scale with the number of vCPUs. That means all the
    dirty memory operations from all vCPU threads have been serialized by
    the MMU lock. Further analysis also shows that the permission relaxation
    during dirty logging is where vCPU threads get serialized.

Solution:
  * On ARM64, there is no mechanism as PML (Page Modification Logging) and
    the dirty-bit solution for dirty logging is much complicated compared to
    the write-protection solution. The straight way to reduce the guest
    performance degradation is to enhance the concurrency for the permission
    fault path during dirty logging.
  * In this patch, we only put leaf PTE permission relaxation for dirty
    logging under read lock, all others would go under write lock.
    Below are the results based on the solution:
    +-------+------------------------+
    | #vCPU | dirty memory time (ms) |
    +-------+------------------------+
    | 1     | 803                    |
    +-------+------------------------+
    | 2     | 843                    |
    +-------+------------------------+
    | 4     | 942                    |
    +-------+------------------------+
    | 8     | 1458                   |
    +-------+------------------------+
    | 16    | 2853                   |
    +-------+------------------------+
    | 32    | 5886                   |
    +-------+------------------------+
    | 64    | 12190                  |
    +-------+------------------------+
    All "dirty memory time" have been reduced by more than 60% when the
    number of vCPU grows.
    
---

Jing Zhang (3):
  KVM: arm64: Use read/write spin lock for MMU protection
  KVM: arm64: Add fast path to handle permission relaxation during dirty
    logging
  KVM: selftests: Add vgic initialization for dirty log perf test for
    ARM

 arch/arm64/include/asm/kvm_host.h             |  2 +
 arch/arm64/kvm/mmu.c                          | 86 +++++++++++++++----
 .../selftests/kvm/dirty_log_perf_test.c       | 10 +++
 3 files changed, 80 insertions(+), 18 deletions(-)


base-commit: fea31d1690945e6dd6c3e89ec5591490857bc3d4
-- 
2.34.1.575.g55b058a8bb-goog


^ permalink raw reply	[flat|nested] 40+ messages in thread

end of thread, other threads:[~2022-01-13  6:12 UTC | newest]

Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-10 21:04 [RFC PATCH 0/3] ARM64: Guest performance improvement during dirty Jing Zhang
2022-01-10 21:04 ` Jing Zhang
2022-01-10 21:04 ` [RFC PATCH 1/3] KVM: arm64: Use read/write spin lock for MMU protection Jing Zhang
2022-01-10 21:04   ` Jing Zhang
2022-01-11 10:23   ` Marc Zyngier
2022-01-11 10:23     ` Marc Zyngier
2022-01-11 22:12     ` Jing Zhang
2022-01-11 22:12       ` Jing Zhang
2022-01-10 21:04 ` [RFC PATCH 2/3] KVM: arm64: Add fast path to handle permission relaxation during dirty logging Jing Zhang
2022-01-10 21:04   ` Jing Zhang
2022-01-11 10:22   ` Marc Zyngier
2022-01-11 10:22     ` Marc Zyngier
2022-01-11 10:50   ` Marc Zyngier
2022-01-11 10:50     ` Marc Zyngier
2022-01-11 22:12     ` Jing Zhang
2022-01-11 22:12       ` Jing Zhang
2022-01-10 21:04 ` [RFC PATCH 3/3] KVM: selftests: Add vgic initialization for dirty log perf test for ARM Jing Zhang
2022-01-10 21:04   ` Jing Zhang
2022-01-11  9:55   ` Andrew Jones
2022-01-11  9:55     ` Andrew Jones
2022-01-11 22:12     ` Jing Zhang
2022-01-11 22:12       ` Jing Zhang
2022-01-11 10:30   ` Marc Zyngier
2022-01-11 10:30     ` Marc Zyngier
2022-01-11 22:16     ` Jing Zhang
2022-01-11 22:16       ` Jing Zhang
2022-01-12 11:37       ` Marc Zyngier
2022-01-12 11:37         ` Marc Zyngier
2022-01-12 17:40         ` Jing Zhang
2022-01-12 17:40           ` Jing Zhang
2022-01-11 11:54 ` [RFC PATCH 0/3] ARM64: Guest performance improvement during dirty Marc Zyngier
2022-01-11 11:54   ` Marc Zyngier
2022-01-11 22:12   ` Jing Zhang
2022-01-11 22:12     ` Jing Zhang
2022-01-13  2:49 ` Ricardo Koller
2022-01-13  2:49   ` Ricardo Koller
2022-01-13  3:50   ` Jing Zhang
2022-01-13  3:50     ` Jing Zhang
2022-01-13  6:12     ` Ricardo Koller
2022-01-13  6:12       ` Ricardo Koller

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.