[RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
performance benefit in waiting for a period of time for an event to
arrive before taking the trap as it is common that event will arrive
“quite soon” after executing the WFE instruction.

This series adds support for TWED feature and implements TWE delay
value dynamic adjustment.

Thanks for Shameer's advice on this series. The function of this patch
has been tested on TWED supported hardware and the performance of it is
still on test, any advice will be welcomed.

Jingyi Wang (2):
  KVM: arm64: Make use of TWED feature
  KVM: arm64: Use dynamic TWE Delay value

Zengruan Ye (2):
  arm64: cpufeature: TWED support detection
  KVM: arm64: Add trace for TWED update

 arch/arm64/Kconfig                   | 10 +++++
 arch/arm64/include/asm/cpucaps.h     |  3 +-
 arch/arm64/include/asm/kvm_arm.h     |  5 +++
 arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
 arch/arm64/include/asm/virt.h        |  8 ++++
 arch/arm64/kernel/cpufeature.c       | 12 ++++++
 arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 +
 arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
 10 files changed, 174 insertions(+), 2 deletions(-)

-- 
2.19.1

[RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
performance benefit in waiting for a period of time for an event to
arrive before taking the trap as it is common that event will arrive
“quite soon” after executing the WFE instruction.

This series adds support for TWED feature and implements TWE delay
value dynamic adjustment.

Thanks for Shameer's advice on this series. The function of this patch
has been tested on TWED supported hardware and the performance of it is
still on test, any advice will be welcomed.

Jingyi Wang (2):
  KVM: arm64: Make use of TWED feature
  KVM: arm64: Use dynamic TWE Delay value

Zengruan Ye (2):
  arm64: cpufeature: TWED support detection
  KVM: arm64: Add trace for TWED update

 arch/arm64/Kconfig                   | 10 +++++
 arch/arm64/include/asm/cpucaps.h     |  3 +-
 arch/arm64/include/asm/kvm_arm.h     |  5 +++
 arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
 arch/arm64/include/asm/virt.h        |  8 ++++
 arch/arm64/kernel/cpufeature.c       | 12 ++++++
 arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 +
 arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
 10 files changed, 174 insertions(+), 2 deletions(-)

-- 
2.19.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
performance benefit in waiting for a period of time for an event to
arrive before taking the trap as it is common that event will arrive
“quite soon” after executing the WFE instruction.

This series adds support for TWED feature and implements TWE delay
value dynamic adjustment.

Thanks for Shameer's advice on this series. The function of this patch
has been tested on TWED supported hardware and the performance of it is
still on test, any advice will be welcomed.

Jingyi Wang (2):
  KVM: arm64: Make use of TWED feature
  KVM: arm64: Use dynamic TWE Delay value

Zengruan Ye (2):
  arm64: cpufeature: TWED support detection
  KVM: arm64: Add trace for TWED update

 arch/arm64/Kconfig                   | 10 +++++
 arch/arm64/include/asm/cpucaps.h     |  3 +-
 arch/arm64/include/asm/kvm_arm.h     |  5 +++
 arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
 arch/arm64/include/asm/virt.h        |  8 ++++
 arch/arm64/kernel/cpufeature.c       | 12 ++++++
 arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 +
 arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
 10 files changed, 174 insertions(+), 2 deletions(-)

-- 
2.19.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 1/4] arm64: cpufeature: TWED support detection

[Jingyi Wang]

From: Zengruan Ye <yezengruan@huawei.com>

TWE Delay is an optional feature in ARMv8.6 Extentions. This patch
detect this feature.

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/Kconfig               | 10 ++++++++++
 arch/arm64/include/asm/cpucaps.h |  3 ++-
 arch/arm64/kernel/cpufeature.c   | 12 ++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6d232837cbee..fe66f4fc5f49 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1666,6 +1666,16 @@ config ARCH_RANDOM
 
 endmenu
 
+menu "ARMv8.6 architectural features"
+
+config ARM64_TWED
+	bool "Enable suppot for delayed trapping of WFE"
+	default y
+	help
+	  Delayed Trapping of WFE (part of the ARMv8.6 Extensions)
+
+endmenu
+
 config ARM64_SVE
 	bool "ARM Scalable Vector Extension support"
 	default y
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 07b643a70710..7b8f018d142b 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -64,7 +64,8 @@
 #define ARM64_BTI				54
 #define ARM64_HAS_ARMv8_4_TTL			55
 #define ARM64_HAS_TLB_RANGE			56
+#define ARM64_HAS_TWED				57
 
-#define ARM64_NCAPS				57
+#define ARM64_NCAPS				58
 
 #endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 6424584be01e..d042d32a1144 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2120,6 +2120,18 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.min_field_value = ID_AA64PFR1_BT_BTI,
 		.sign = FTR_UNSIGNED,
 	},
+#endif
+#ifdef CONFIG_ARM64_TWED
+	{
+		.desc = "Delayed Trapping of WFE",
+		.capability = ARM64_HAS_TWED,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_cpuid_feature,
+		.sys_reg = SYS_ID_AA64MMFR1_EL1,
+		.field_pos = ID_AA64MMFR1_TWED_SHIFT,
+		.sign = FTR_UNSIGNED,
+		.min_field_value = 1,
+	},
 #endif
 	{},
 };
-- 
2.19.1

[RFC PATCH 1/4] arm64: cpufeature: TWED support detection

[Jingyi Wang]

From: Zengruan Ye <yezengruan@huawei.com>

TWE Delay is an optional feature in ARMv8.6 Extentions. This patch
detect this feature.

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/Kconfig               | 10 ++++++++++
 arch/arm64/include/asm/cpucaps.h |  3 ++-
 arch/arm64/kernel/cpufeature.c   | 12 ++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6d232837cbee..fe66f4fc5f49 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1666,6 +1666,16 @@ config ARCH_RANDOM
 
 endmenu
 
+menu "ARMv8.6 architectural features"
+
+config ARM64_TWED
+	bool "Enable suppot for delayed trapping of WFE"
+	default y
+	help
+	  Delayed Trapping of WFE (part of the ARMv8.6 Extensions)
+
+endmenu
+
 config ARM64_SVE
 	bool "ARM Scalable Vector Extension support"
 	default y
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 07b643a70710..7b8f018d142b 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -64,7 +64,8 @@
 #define ARM64_BTI				54
 #define ARM64_HAS_ARMv8_4_TTL			55
 #define ARM64_HAS_TLB_RANGE			56
+#define ARM64_HAS_TWED				57
 
-#define ARM64_NCAPS				57
+#define ARM64_NCAPS				58
 
 #endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 6424584be01e..d042d32a1144 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2120,6 +2120,18 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.min_field_value = ID_AA64PFR1_BT_BTI,
 		.sign = FTR_UNSIGNED,
 	},
+#endif
+#ifdef CONFIG_ARM64_TWED
+	{
+		.desc = "Delayed Trapping of WFE",
+		.capability = ARM64_HAS_TWED,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_cpuid_feature,
+		.sys_reg = SYS_ID_AA64MMFR1_EL1,
+		.field_pos = ID_AA64MMFR1_TWED_SHIFT,
+		.sign = FTR_UNSIGNED,
+		.min_field_value = 1,
+	},
 #endif
 	{},
 };
-- 
2.19.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[RFC PATCH 1/4] arm64: cpufeature: TWED support detection

[Jingyi Wang]

From: Zengruan Ye <yezengruan@huawei.com>

TWE Delay is an optional feature in ARMv8.6 Extentions. This patch
detect this feature.

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/Kconfig               | 10 ++++++++++
 arch/arm64/include/asm/cpucaps.h |  3 ++-
 arch/arm64/kernel/cpufeature.c   | 12 ++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6d232837cbee..fe66f4fc5f49 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1666,6 +1666,16 @@ config ARCH_RANDOM
 
 endmenu
 
+menu "ARMv8.6 architectural features"
+
+config ARM64_TWED
+	bool "Enable suppot for delayed trapping of WFE"
+	default y
+	help
+	  Delayed Trapping of WFE (part of the ARMv8.6 Extensions)
+
+endmenu
+
 config ARM64_SVE
 	bool "ARM Scalable Vector Extension support"
 	default y
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 07b643a70710..7b8f018d142b 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -64,7 +64,8 @@
 #define ARM64_BTI				54
 #define ARM64_HAS_ARMv8_4_TTL			55
 #define ARM64_HAS_TLB_RANGE			56
+#define ARM64_HAS_TWED				57
 
-#define ARM64_NCAPS				57
+#define ARM64_NCAPS				58
 
 #endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 6424584be01e..d042d32a1144 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2120,6 +2120,18 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.min_field_value = ID_AA64PFR1_BT_BTI,
 		.sign = FTR_UNSIGNED,
 	},
+#endif
+#ifdef CONFIG_ARM64_TWED
+	{
+		.desc = "Delayed Trapping of WFE",
+		.capability = ARM64_HAS_TWED,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_cpuid_feature,
+		.sys_reg = SYS_ID_AA64MMFR1_EL1,
+		.field_pos = ID_AA64MMFR1_TWED_SHIFT,
+		.sign = FTR_UNSIGNED,
+		.min_field_value = 1,
+	},
 #endif
 	{},
 };
-- 
2.19.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 2/4] KVM: arm64: Make use of TWED feature

[Jingyi Wang]

For HCR_EL2, TWEDEn(bit[59]) decides whether TWED is enabled, and
when the configurable delay is enabled, TWEDEL (bits[63:60]) encodes
the minimum delay in taking a trap of WFE caused by the TWE bit in
this register as 2^(TWEDEL + 8) cycles.

We use two kernel parameters "twed_enable" and "twed" to configure the
register.

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/include/asm/kvm_arm.h     |  5 +++++
 arch/arm64/include/asm/kvm_emulate.h | 32 ++++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 13 +++++++++++
 arch/arm64/include/asm/virt.h        |  8 +++++++
 arch/arm64/kvm/arm.c                 | 22 +++++++++++++++++++
 5 files changed, 80 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 1da8e3dc4455..03afb2921f1c 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -12,6 +12,11 @@
 #include <asm/types.h>
 
 /* Hyp Configuration Register (HCR) bits */
+#define HCR_TWEDEL_SHIFT	60
+#define HCR_TWEDEL_MAX		(UL(0xf))
+#define HCR_TWEDEL_MASK		(HCR_TWEDEL_MAX << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEL	(UL(1) << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEN	(UL(1) << 59)
 #define HCR_FWB		(UL(1) << 46)
 #define HCR_API		(UL(1) << 41)
 #define HCR_APK		(UL(1) << 40)
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 1cc5f5f72d0b..1a9cce836170 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -102,6 +102,38 @@ static inline void vcpu_set_wfx_traps(struct kvm_vcpu *vcpu)
 	vcpu->arch.hcr_el2 |= HCR_TWI;
 }
 
+#ifdef CONFIG_ARM64_TWED
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 |= HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.twed = (u64)twed;
+}
+
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
+{
+	u64 delay = vcpu->arch.twed;
+	if (delay > HCR_TWEDEL_MAX)
+		delay = HCR_TWEDEL_MAX;
+
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
+	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+}
+#else
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu) {};
+#endif
+
 static inline void vcpu_ptrauth_enable(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK);
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 905c2b87e05a..380cd9c8ad0f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -371,6 +371,11 @@ struct kvm_vcpu_arch {
 		u64 last_steal;
 		gpa_t base;
 	} steal;
+
+#ifdef CONFIG_ARM64_TWED
+	/* WFE trap delay */
+	u64 twed;
+#endif
 };
 
 /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -688,4 +693,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define kvm_arm_vcpu_sve_finalized(vcpu) \
 	((vcpu)->arch.flags & KVM_ARM64_VCPU_SVE_FINALIZED)
 
+#ifdef CONFIG_ARM64_TWED
+#define use_twed() (has_twed() && twed_enable)
+extern bool twed_enable;
+extern unsigned int twed;
+#else
+#define use_twed() false
+#endif
+
 #endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h
index 09977acc007d..1a9e437e5bf6 100644
--- a/arch/arm64/include/asm/virt.h
+++ b/arch/arm64/include/asm/virt.h
@@ -98,6 +98,14 @@ static __always_inline bool has_vhe(void)
 		return cpus_have_final_cap(ARM64_HAS_VIRT_HOST_EXTN);
 }
 
+static __always_inline bool has_twed(void)
+{
+	if (cpus_have_const_cap(ARM64_HAS_TWED))
+		return true;
+
+	return false;
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* ! __ASM__VIRT_H */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b588c3b5c2f0..5b5e8b14dcd5 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -59,6 +59,14 @@ static bool vgic_present;
 static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled);
 DEFINE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
 
+#ifdef CONFIG_ARM64_TWED
+bool twed_enable = false;
+module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
+
+unsigned int twed = 0;
+module_param(twed, uint, S_IRUGO | S_IWUSR);
+#endif
+
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
 {
 	return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
@@ -270,6 +278,13 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
 
+	if (use_twed()) {
+		vcpu_twed_enable(vcpu);
+		vcpu_twed_init(vcpu);
+	} else {
+		vcpu_twed_disable(vcpu);
+	}
+
 	err = kvm_vgic_vcpu_init(vcpu);
 	if (err)
 		return err;
@@ -736,6 +751,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		kvm_arm_setup_debug(vcpu);
 
+		if (use_twed()) {
+			vcpu_twed_enable(vcpu);
+			vcpu_set_twed(vcpu);
+		} else {
+			vcpu_twed_disable(vcpu);
+		}
+
 		/**************************************************************
 		 * Enter the guest
 		 */
-- 
2.19.1

[RFC PATCH 2/4] KVM: arm64: Make use of TWED feature

[Jingyi Wang]

For HCR_EL2, TWEDEn(bit[59]) decides whether TWED is enabled, and
when the configurable delay is enabled, TWEDEL (bits[63:60]) encodes
the minimum delay in taking a trap of WFE caused by the TWE bit in
this register as 2^(TWEDEL + 8) cycles.

We use two kernel parameters "twed_enable" and "twed" to configure the
register.

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/include/asm/kvm_arm.h     |  5 +++++
 arch/arm64/include/asm/kvm_emulate.h | 32 ++++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 13 +++++++++++
 arch/arm64/include/asm/virt.h        |  8 +++++++
 arch/arm64/kvm/arm.c                 | 22 +++++++++++++++++++
 5 files changed, 80 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 1da8e3dc4455..03afb2921f1c 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -12,6 +12,11 @@
 #include <asm/types.h>
 
 /* Hyp Configuration Register (HCR) bits */
+#define HCR_TWEDEL_SHIFT	60
+#define HCR_TWEDEL_MAX		(UL(0xf))
+#define HCR_TWEDEL_MASK		(HCR_TWEDEL_MAX << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEL	(UL(1) << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEN	(UL(1) << 59)
 #define HCR_FWB		(UL(1) << 46)
 #define HCR_API		(UL(1) << 41)
 #define HCR_APK		(UL(1) << 40)
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 1cc5f5f72d0b..1a9cce836170 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -102,6 +102,38 @@ static inline void vcpu_set_wfx_traps(struct kvm_vcpu *vcpu)
 	vcpu->arch.hcr_el2 |= HCR_TWI;
 }
 
+#ifdef CONFIG_ARM64_TWED
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 |= HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.twed = (u64)twed;
+}
+
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
+{
+	u64 delay = vcpu->arch.twed;
+	if (delay > HCR_TWEDEL_MAX)
+		delay = HCR_TWEDEL_MAX;
+
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
+	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+}
+#else
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu) {};
+#endif
+
 static inline void vcpu_ptrauth_enable(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK);
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 905c2b87e05a..380cd9c8ad0f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -371,6 +371,11 @@ struct kvm_vcpu_arch {
 		u64 last_steal;
 		gpa_t base;
 	} steal;
+
+#ifdef CONFIG_ARM64_TWED
+	/* WFE trap delay */
+	u64 twed;
+#endif
 };
 
 /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -688,4 +693,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define kvm_arm_vcpu_sve_finalized(vcpu) \
 	((vcpu)->arch.flags & KVM_ARM64_VCPU_SVE_FINALIZED)
 
+#ifdef CONFIG_ARM64_TWED
+#define use_twed() (has_twed() && twed_enable)
+extern bool twed_enable;
+extern unsigned int twed;
+#else
+#define use_twed() false
+#endif
+
 #endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h
index 09977acc007d..1a9e437e5bf6 100644
--- a/arch/arm64/include/asm/virt.h
+++ b/arch/arm64/include/asm/virt.h
@@ -98,6 +98,14 @@ static __always_inline bool has_vhe(void)
 		return cpus_have_final_cap(ARM64_HAS_VIRT_HOST_EXTN);
 }
 
+static __always_inline bool has_twed(void)
+{
+	if (cpus_have_const_cap(ARM64_HAS_TWED))
+		return true;
+
+	return false;
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* ! __ASM__VIRT_H */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b588c3b5c2f0..5b5e8b14dcd5 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -59,6 +59,14 @@ static bool vgic_present;
 static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled);
 DEFINE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
 
+#ifdef CONFIG_ARM64_TWED
+bool twed_enable = false;
+module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
+
+unsigned int twed = 0;
+module_param(twed, uint, S_IRUGO | S_IWUSR);
+#endif
+
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
 {
 	return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
@@ -270,6 +278,13 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
 
+	if (use_twed()) {
+		vcpu_twed_enable(vcpu);
+		vcpu_twed_init(vcpu);
+	} else {
+		vcpu_twed_disable(vcpu);
+	}
+
 	err = kvm_vgic_vcpu_init(vcpu);
 	if (err)
 		return err;
@@ -736,6 +751,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		kvm_arm_setup_debug(vcpu);
 
+		if (use_twed()) {
+			vcpu_twed_enable(vcpu);
+			vcpu_set_twed(vcpu);
+		} else {
+			vcpu_twed_disable(vcpu);
+		}
+
 		/**************************************************************
 		 * Enter the guest
 		 */
-- 
2.19.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[RFC PATCH 2/4] KVM: arm64: Make use of TWED feature

[Jingyi Wang]

For HCR_EL2, TWEDEn(bit[59]) decides whether TWED is enabled, and
when the configurable delay is enabled, TWEDEL (bits[63:60]) encodes
the minimum delay in taking a trap of WFE caused by the TWE bit in
this register as 2^(TWEDEL + 8) cycles.

We use two kernel parameters "twed_enable" and "twed" to configure the
register.

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/include/asm/kvm_arm.h     |  5 +++++
 arch/arm64/include/asm/kvm_emulate.h | 32 ++++++++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h    | 13 +++++++++++
 arch/arm64/include/asm/virt.h        |  8 +++++++
 arch/arm64/kvm/arm.c                 | 22 +++++++++++++++++++
 5 files changed, 80 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 1da8e3dc4455..03afb2921f1c 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -12,6 +12,11 @@
 #include <asm/types.h>
 
 /* Hyp Configuration Register (HCR) bits */
+#define HCR_TWEDEL_SHIFT	60
+#define HCR_TWEDEL_MAX		(UL(0xf))
+#define HCR_TWEDEL_MASK		(HCR_TWEDEL_MAX << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEL	(UL(1) << HCR_TWEDEL_SHIFT)
+#define HCR_TWEDEN	(UL(1) << 59)
 #define HCR_FWB		(UL(1) << 46)
 #define HCR_API		(UL(1) << 41)
 #define HCR_APK		(UL(1) << 40)
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 1cc5f5f72d0b..1a9cce836170 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -102,6 +102,38 @@ static inline void vcpu_set_wfx_traps(struct kvm_vcpu *vcpu)
 	vcpu->arch.hcr_el2 |= HCR_TWI;
 }
 
+#ifdef CONFIG_ARM64_TWED
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 |= HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEN;
+}
+
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.twed = (u64)twed;
+}
+
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
+{
+	u64 delay = vcpu->arch.twed;
+	if (delay > HCR_TWEDEL_MAX)
+		delay = HCR_TWEDEL_MAX;
+
+	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
+	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+}
+#else
+static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_twed_init(struct kvm_vcpu *vcpu) {};
+static inline void vcpu_set_twed(struct kvm_vcpu *vcpu) {};
+#endif
+
 static inline void vcpu_ptrauth_enable(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK);
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 905c2b87e05a..380cd9c8ad0f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -371,6 +371,11 @@ struct kvm_vcpu_arch {
 		u64 last_steal;
 		gpa_t base;
 	} steal;
+
+#ifdef CONFIG_ARM64_TWED
+	/* WFE trap delay */
+	u64 twed;
+#endif
 };
 
 /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -688,4 +693,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define kvm_arm_vcpu_sve_finalized(vcpu) \
 	((vcpu)->arch.flags & KVM_ARM64_VCPU_SVE_FINALIZED)
 
+#ifdef CONFIG_ARM64_TWED
+#define use_twed() (has_twed() && twed_enable)
+extern bool twed_enable;
+extern unsigned int twed;
+#else
+#define use_twed() false
+#endif
+
 #endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h
index 09977acc007d..1a9e437e5bf6 100644
--- a/arch/arm64/include/asm/virt.h
+++ b/arch/arm64/include/asm/virt.h
@@ -98,6 +98,14 @@ static __always_inline bool has_vhe(void)
 		return cpus_have_final_cap(ARM64_HAS_VIRT_HOST_EXTN);
 }
 
+static __always_inline bool has_twed(void)
+{
+	if (cpus_have_const_cap(ARM64_HAS_TWED))
+		return true;
+
+	return false;
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* ! __ASM__VIRT_H */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b588c3b5c2f0..5b5e8b14dcd5 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -59,6 +59,14 @@ static bool vgic_present;
 static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled);
 DEFINE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
 
+#ifdef CONFIG_ARM64_TWED
+bool twed_enable = false;
+module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
+
+unsigned int twed = 0;
+module_param(twed, uint, S_IRUGO | S_IWUSR);
+#endif
+
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
 {
 	return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
@@ -270,6 +278,13 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
 
+	if (use_twed()) {
+		vcpu_twed_enable(vcpu);
+		vcpu_twed_init(vcpu);
+	} else {
+		vcpu_twed_disable(vcpu);
+	}
+
 	err = kvm_vgic_vcpu_init(vcpu);
 	if (err)
 		return err;
@@ -736,6 +751,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 
 		kvm_arm_setup_debug(vcpu);
 
+		if (use_twed()) {
+			vcpu_twed_enable(vcpu);
+			vcpu_set_twed(vcpu);
+		} else {
+			vcpu_twed_disable(vcpu);
+		}
+
 		/**************************************************************
 		 * Enter the guest
 		 */
-- 
2.19.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 3/4] KVM: arm64: Use dynamic TWE Delay value

[Jingyi Wang]

We implement two new kernel parameters for changing the delay adaptively:
twed_grow and twed_shrink
twed_grow affects the delay on WFE trap and twed_shrink does it on
sched_in; depending on their value, the delay is modifier like this:

     twed_shrink/ |
     twed_grow    | WFE trap exit | sched_in
    --------------+---------------+------------
     < 1          | = twed        | = twed
     otherwise    | = twed + 1    | = twed - 1

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/include/asm/kvm_emulate.h |  6 ++++++
 arch/arm64/include/asm/kvm_host.h    |  6 +++++-
 arch/arm64/kvm/arm.c                 | 32 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 ++
 4 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 1a9cce836170..546d10b3b534 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -116,16 +116,22 @@ static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
 static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.twed = (u64)twed;
+	vcpu->arch.twed_dirty = true;
 }
 
 static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
 {
 	u64 delay = vcpu->arch.twed;
+
+	if (!vcpu->arch.twed_dirty)
+		return;
+
 	if (delay > HCR_TWEDEL_MAX)
 		delay = HCR_TWEDEL_MAX;
 
 	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
 	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+	vcpu->arch.twed_dirty = false;
 }
 #else
 static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 380cd9c8ad0f..35d1953d9d35 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -375,6 +375,7 @@ struct kvm_vcpu_arch {
 #ifdef CONFIG_ARM64_TWED
 	/* WFE trap delay */
 	u64 twed;
+	bool twed_dirty;
 #endif
 };
 
@@ -595,7 +596,6 @@ void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu);
 
 static inline void kvm_arch_hardware_unsetup(void) {}
 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
-static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
 static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {}
 
 void kvm_arm_init_debug(void);
@@ -697,8 +697,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define use_twed() (has_twed() && twed_enable)
 extern bool twed_enable;
 extern unsigned int twed;
+void grow_twed(struct kvm_vcpu *vcpu);
+void shrink_twed(struct kvm_vcpu *vcpu);
 #else
 #define use_twed() false
+static inline void grow_twed(struct kvm_vcpu *vcpu) {};
+static inline void shrink_twed(struct kvm_vcpu *vcpu) {};
 #endif
 
 #endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 5b5e8b14dcd5..989bffdcb3e9 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -65,6 +65,32 @@ module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
 
 unsigned int twed = 0;
 module_param(twed, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_grow = 0;
+module_param(twed_grow, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_shrink = 0;
+module_param(twed_shrink, uint, S_IRUGO | S_IWUSR);
+
+void grow_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old < HCR_TWEDEL_MAX && twed_grow) {
+		vcpu->arch.twed += 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
+
+void shrink_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old > 0 && twed_shrink) {
+		vcpu->arch.twed -= 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
 #endif
 
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
@@ -108,6 +134,12 @@ static int kvm_arm_default_max_vcpus(void)
 	return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS;
 }
 
+void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu)
+{
+	if (use_twed())
+		shrink_twed(vcpu);
+}
+
 /**
  * kvm_arch_init_vm - initializes a VM data structure
  * @kvm:	pointer to the KVM struct
diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c
index 5d690d60ccad..2ad72defa3c9 100644
--- a/arch/arm64/kvm/handle_exit.c
+++ b/arch/arm64/kvm/handle_exit.c
@@ -91,6 +91,8 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu)
 {
 	if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE) {
 		trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true);
+		if (use_twed())
+			grow_twed(vcpu);
 		vcpu->stat.wfe_exit_stat++;
 		kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu));
 	} else {
-- 
2.19.1

[RFC PATCH 3/4] KVM: arm64: Use dynamic TWE Delay value

[Jingyi Wang]

We implement two new kernel parameters for changing the delay adaptively:
twed_grow and twed_shrink
twed_grow affects the delay on WFE trap and twed_shrink does it on
sched_in; depending on their value, the delay is modifier like this:

     twed_shrink/ |
     twed_grow    | WFE trap exit | sched_in
    --------------+---------------+------------
     < 1          | = twed        | = twed
     otherwise    | = twed + 1    | = twed - 1

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/include/asm/kvm_emulate.h |  6 ++++++
 arch/arm64/include/asm/kvm_host.h    |  6 +++++-
 arch/arm64/kvm/arm.c                 | 32 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 ++
 4 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 1a9cce836170..546d10b3b534 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -116,16 +116,22 @@ static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
 static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.twed = (u64)twed;
+	vcpu->arch.twed_dirty = true;
 }
 
 static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
 {
 	u64 delay = vcpu->arch.twed;
+
+	if (!vcpu->arch.twed_dirty)
+		return;
+
 	if (delay > HCR_TWEDEL_MAX)
 		delay = HCR_TWEDEL_MAX;
 
 	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
 	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+	vcpu->arch.twed_dirty = false;
 }
 #else
 static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 380cd9c8ad0f..35d1953d9d35 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -375,6 +375,7 @@ struct kvm_vcpu_arch {
 #ifdef CONFIG_ARM64_TWED
 	/* WFE trap delay */
 	u64 twed;
+	bool twed_dirty;
 #endif
 };
 
@@ -595,7 +596,6 @@ void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu);
 
 static inline void kvm_arch_hardware_unsetup(void) {}
 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
-static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
 static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {}
 
 void kvm_arm_init_debug(void);
@@ -697,8 +697,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define use_twed() (has_twed() && twed_enable)
 extern bool twed_enable;
 extern unsigned int twed;
+void grow_twed(struct kvm_vcpu *vcpu);
+void shrink_twed(struct kvm_vcpu *vcpu);
 #else
 #define use_twed() false
+static inline void grow_twed(struct kvm_vcpu *vcpu) {};
+static inline void shrink_twed(struct kvm_vcpu *vcpu) {};
 #endif
 
 #endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 5b5e8b14dcd5..989bffdcb3e9 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -65,6 +65,32 @@ module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
 
 unsigned int twed = 0;
 module_param(twed, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_grow = 0;
+module_param(twed_grow, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_shrink = 0;
+module_param(twed_shrink, uint, S_IRUGO | S_IWUSR);
+
+void grow_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old < HCR_TWEDEL_MAX && twed_grow) {
+		vcpu->arch.twed += 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
+
+void shrink_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old > 0 && twed_shrink) {
+		vcpu->arch.twed -= 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
 #endif
 
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
@@ -108,6 +134,12 @@ static int kvm_arm_default_max_vcpus(void)
 	return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS;
 }
 
+void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu)
+{
+	if (use_twed())
+		shrink_twed(vcpu);
+}
+
 /**
  * kvm_arch_init_vm - initializes a VM data structure
  * @kvm:	pointer to the KVM struct
diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c
index 5d690d60ccad..2ad72defa3c9 100644
--- a/arch/arm64/kvm/handle_exit.c
+++ b/arch/arm64/kvm/handle_exit.c
@@ -91,6 +91,8 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu)
 {
 	if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE) {
 		trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true);
+		if (use_twed())
+			grow_twed(vcpu);
 		vcpu->stat.wfe_exit_stat++;
 		kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu));
 	} else {
-- 
2.19.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[RFC PATCH 3/4] KVM: arm64: Use dynamic TWE Delay value

[Jingyi Wang]

We implement two new kernel parameters for changing the delay adaptively:
twed_grow and twed_shrink
twed_grow affects the delay on WFE trap and twed_shrink does it on
sched_in; depending on their value, the delay is modifier like this:

     twed_shrink/ |
     twed_grow    | WFE trap exit | sched_in
    --------------+---------------+------------
     < 1          | = twed        | = twed
     otherwise    | = twed + 1    | = twed - 1

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/include/asm/kvm_emulate.h |  6 ++++++
 arch/arm64/include/asm/kvm_host.h    |  6 +++++-
 arch/arm64/kvm/arm.c                 | 32 ++++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c         |  2 ++
 4 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 1a9cce836170..546d10b3b534 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -116,16 +116,22 @@ static inline void vcpu_twed_disable(struct kvm_vcpu *vcpu)
 static inline void vcpu_twed_init(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.twed = (u64)twed;
+	vcpu->arch.twed_dirty = true;
 }
 
 static inline void vcpu_set_twed(struct kvm_vcpu *vcpu)
 {
 	u64 delay = vcpu->arch.twed;
+
+	if (!vcpu->arch.twed_dirty)
+		return;
+
 	if (delay > HCR_TWEDEL_MAX)
 		delay = HCR_TWEDEL_MAX;
 
 	vcpu->arch.hcr_el2 &= ~HCR_TWEDEL_MASK;
 	vcpu->arch.hcr_el2 |= (delay << HCR_TWEDEL_SHIFT);
+	vcpu->arch.twed_dirty = false;
 }
 #else
 static inline void vcpu_twed_enable(struct kvm_vcpu *vcpu) {};
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 380cd9c8ad0f..35d1953d9d35 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -375,6 +375,7 @@ struct kvm_vcpu_arch {
 #ifdef CONFIG_ARM64_TWED
 	/* WFE trap delay */
 	u64 twed;
+	bool twed_dirty;
 #endif
 };
 
@@ -595,7 +596,6 @@ void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu);
 
 static inline void kvm_arch_hardware_unsetup(void) {}
 static inline void kvm_arch_sync_events(struct kvm *kvm) {}
-static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
 static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {}
 
 void kvm_arm_init_debug(void);
@@ -697,8 +697,12 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
 #define use_twed() (has_twed() && twed_enable)
 extern bool twed_enable;
 extern unsigned int twed;
+void grow_twed(struct kvm_vcpu *vcpu);
+void shrink_twed(struct kvm_vcpu *vcpu);
 #else
 #define use_twed() false
+static inline void grow_twed(struct kvm_vcpu *vcpu) {};
+static inline void shrink_twed(struct kvm_vcpu *vcpu) {};
 #endif
 
 #endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 5b5e8b14dcd5..989bffdcb3e9 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -65,6 +65,32 @@ module_param(twed_enable, bool, S_IRUGO | S_IWUSR);
 
 unsigned int twed = 0;
 module_param(twed, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_grow = 0;
+module_param(twed_grow, uint, S_IRUGO | S_IWUSR);
+
+static unsigned int twed_shrink = 0;
+module_param(twed_shrink, uint, S_IRUGO | S_IWUSR);
+
+void grow_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old < HCR_TWEDEL_MAX && twed_grow) {
+		vcpu->arch.twed += 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
+
+void shrink_twed(struct kvm_vcpu *vcpu)
+{
+	u64 old = vcpu->arch.twed;
+
+	if (old > 0 && twed_shrink) {
+		vcpu->arch.twed -= 1;
+		vcpu->arch.twed_dirty = true;
+	}
+}
 #endif
 
 int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
@@ -108,6 +134,12 @@ static int kvm_arm_default_max_vcpus(void)
 	return vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS;
 }
 
+void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu)
+{
+	if (use_twed())
+		shrink_twed(vcpu);
+}
+
 /**
  * kvm_arch_init_vm - initializes a VM data structure
  * @kvm:	pointer to the KVM struct
diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c
index 5d690d60ccad..2ad72defa3c9 100644
--- a/arch/arm64/kvm/handle_exit.c
+++ b/arch/arm64/kvm/handle_exit.c
@@ -91,6 +91,8 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu)
 {
 	if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE) {
 		trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true);
+		if (use_twed())
+			grow_twed(vcpu);
 		vcpu->stat.wfe_exit_stat++;
 		kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu));
 	} else {
-- 
2.19.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[RFC PATCH 4/4] KVM: arm64: Add trace for TWED update

[Jingyi Wang]

From: Zengruan Ye <yezengruan@huawei.com>

Add tracepoints for TWE delay value update

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/kvm/arm.c       |  4 ++++
 arch/arm64/kvm/trace_arm.h | 21 +++++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 989bffdcb3e9..c3d7a326bf1b 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -80,6 +80,8 @@ void grow_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed += 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 
 void shrink_twed(struct kvm_vcpu *vcpu)
@@ -90,6 +92,8 @@ void shrink_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed -= 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 #endif
 
diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h
index ff0444352bba..5081266399e8 100644
--- a/arch/arm64/kvm/trace_arm.h
+++ b/arch/arm64/kvm/trace_arm.h
@@ -367,6 +367,27 @@ TRACE_EVENT(kvm_timer_emulate,
 		  __entry->timer_idx, __entry->should_fire)
 );
 
+TRACE_EVENT(kvm_twed_update,
+	TP_PROTO(unsigned int vcpu_id, unsigned int new, unsigned int old),
+	TP_ARGS(vcpu_id, new, old),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	vcpu_id		)
+		__field(	unsigned int,	new		)
+		__field(	unsigned int,	old		)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_id	= vcpu_id;
+		__entry->new		= new;
+		__entry->old		= old;
+	),
+
+	TP_printk("vcpu %u old %u new %u (%s)",
+		  __entry->vcpu_id, __entry->old, __entry->new,
+		  __entry->old < __entry->new ? "growed" : "shrinked")
+);
+
 #endif /* _TRACE_ARM_ARM64_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
-- 
2.19.1

[RFC PATCH 4/4] KVM: arm64: Add trace for TWED update

[Jingyi Wang]

From: Zengruan Ye <yezengruan@huawei.com>

Add tracepoints for TWE delay value update

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/kvm/arm.c       |  4 ++++
 arch/arm64/kvm/trace_arm.h | 21 +++++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 989bffdcb3e9..c3d7a326bf1b 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -80,6 +80,8 @@ void grow_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed += 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 
 void shrink_twed(struct kvm_vcpu *vcpu)
@@ -90,6 +92,8 @@ void shrink_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed -= 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 #endif
 
diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h
index ff0444352bba..5081266399e8 100644
--- a/arch/arm64/kvm/trace_arm.h
+++ b/arch/arm64/kvm/trace_arm.h
@@ -367,6 +367,27 @@ TRACE_EVENT(kvm_timer_emulate,
 		  __entry->timer_idx, __entry->should_fire)
 );
 
+TRACE_EVENT(kvm_twed_update,
+	TP_PROTO(unsigned int vcpu_id, unsigned int new, unsigned int old),
+	TP_ARGS(vcpu_id, new, old),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	vcpu_id		)
+		__field(	unsigned int,	new		)
+		__field(	unsigned int,	old		)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_id	= vcpu_id;
+		__entry->new		= new;
+		__entry->old		= old;
+	),
+
+	TP_printk("vcpu %u old %u new %u (%s)",
+		  __entry->vcpu_id, __entry->old, __entry->new,
+		  __entry->old < __entry->new ? "growed" : "shrinked")
+);
+
 #endif /* _TRACE_ARM_ARM64_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
-- 
2.19.1

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[RFC PATCH 4/4] KVM: arm64: Add trace for TWED update

[Jingyi Wang]

From: Zengruan Ye <yezengruan@huawei.com>

Add tracepoints for TWE delay value update

Signed-off-by: Zengruan Ye <yezengruan@huawei.com>
Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arch/arm64/kvm/arm.c       |  4 ++++
 arch/arm64/kvm/trace_arm.h | 21 +++++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 989bffdcb3e9..c3d7a326bf1b 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -80,6 +80,8 @@ void grow_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed += 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 
 void shrink_twed(struct kvm_vcpu *vcpu)
@@ -90,6 +92,8 @@ void shrink_twed(struct kvm_vcpu *vcpu)
 		vcpu->arch.twed -= 1;
 		vcpu->arch.twed_dirty = true;
 	}
+
+	trace_kvm_twed_update(vcpu->vcpu_id, vcpu->arch.twed, old);
 }
 #endif
 
diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h
index ff0444352bba..5081266399e8 100644
--- a/arch/arm64/kvm/trace_arm.h
+++ b/arch/arm64/kvm/trace_arm.h
@@ -367,6 +367,27 @@ TRACE_EVENT(kvm_timer_emulate,
 		  __entry->timer_idx, __entry->should_fire)
 );
 
+TRACE_EVENT(kvm_twed_update,
+	TP_PROTO(unsigned int vcpu_id, unsigned int new, unsigned int old),
+	TP_ARGS(vcpu_id, new, old),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	vcpu_id		)
+		__field(	unsigned int,	new		)
+		__field(	unsigned int,	old		)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_id	= vcpu_id;
+		__entry->new		= new;
+		__entry->old		= old;
+	),
+
+	TP_printk("vcpu %u old %u new %u (%s)",
+		  __entry->vcpu_id, __entry->old, __entry->new,
+		  __entry->old < __entry->new ? "growed" : "shrinked")
+);
+
 #endif /* _TRACE_ARM_ARM64_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
-- 
2.19.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Marc Zyngier]

On 2020-09-29 10:17, Jingyi Wang wrote:
> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
> performance benefit in waiting for a period of time for an event to
> arrive before taking the trap as it is common that event will arrive
> “quite soon” after executing the WFE instruction.

Define "quite soon". Quantify "performance benefits". Which are the
workloads that actually benefit from this imitation of the x86 PLE?

I was opposed to this when the spec was drafted, and I still am given
that there is zero supporting evidence that it bring any gain over
immediate trapping in an oversubscribed environment (which is the only
case where it matters).

Thanks,

         M.

> 
> This series adds support for TWED feature and implements TWE delay
> value dynamic adjustment.
> 
> Thanks for Shameer's advice on this series. The function of this patch
> has been tested on TWED supported hardware and the performance of it is
> still on test, any advice will be welcomed.
> 
> Jingyi Wang (2):
>   KVM: arm64: Make use of TWED feature
>   KVM: arm64: Use dynamic TWE Delay value
> 
> Zengruan Ye (2):
>   arm64: cpufeature: TWED support detection
>   KVM: arm64: Add trace for TWED update
> 
>  arch/arm64/Kconfig                   | 10 +++++
>  arch/arm64/include/asm/cpucaps.h     |  3 +-
>  arch/arm64/include/asm/kvm_arm.h     |  5 +++
>  arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>  arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>  arch/arm64/include/asm/virt.h        |  8 ++++
>  arch/arm64/kernel/cpufeature.c       | 12 ++++++
>  arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>  arch/arm64/kvm/handle_exit.c         |  2 +
>  arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>  10 files changed, 174 insertions(+), 2 deletions(-)

-- 
Jazz is not dead. It just smells funny...

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Marc Zyngier]

On 2020-09-29 10:17, Jingyi Wang wrote:
> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
> performance benefit in waiting for a period of time for an event to
> arrive before taking the trap as it is common that event will arrive
> “quite soon” after executing the WFE instruction.

Define "quite soon". Quantify "performance benefits". Which are the
workloads that actually benefit from this imitation of the x86 PLE?

I was opposed to this when the spec was drafted, and I still am given
that there is zero supporting evidence that it bring any gain over
immediate trapping in an oversubscribed environment (which is the only
case where it matters).

Thanks,

         M.

> 
> This series adds support for TWED feature and implements TWE delay
> value dynamic adjustment.
> 
> Thanks for Shameer's advice on this series. The function of this patch
> has been tested on TWED supported hardware and the performance of it is
> still on test, any advice will be welcomed.
> 
> Jingyi Wang (2):
>   KVM: arm64: Make use of TWED feature
>   KVM: arm64: Use dynamic TWE Delay value
> 
> Zengruan Ye (2):
>   arm64: cpufeature: TWED support detection
>   KVM: arm64: Add trace for TWED update
> 
>  arch/arm64/Kconfig                   | 10 +++++
>  arch/arm64/include/asm/cpucaps.h     |  3 +-
>  arch/arm64/include/asm/kvm_arm.h     |  5 +++
>  arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>  arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>  arch/arm64/include/asm/virt.h        |  8 ++++
>  arch/arm64/kernel/cpufeature.c       | 12 ++++++
>  arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>  arch/arm64/kvm/handle_exit.c         |  2 +
>  arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>  10 files changed, 174 insertions(+), 2 deletions(-)

-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Marc Zyngier]

On 2020-09-29 10:17, Jingyi Wang wrote:
> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
> performance benefit in waiting for a period of time for an event to
> arrive before taking the trap as it is common that event will arrive
> “quite soon” after executing the WFE instruction.

Define "quite soon". Quantify "performance benefits". Which are the
workloads that actually benefit from this imitation of the x86 PLE?

I was opposed to this when the spec was drafted, and I still am given
that there is zero supporting evidence that it bring any gain over
immediate trapping in an oversubscribed environment (which is the only
case where it matters).

Thanks,

         M.

> 
> This series adds support for TWED feature and implements TWE delay
> value dynamic adjustment.
> 
> Thanks for Shameer's advice on this series. The function of this patch
> has been tested on TWED supported hardware and the performance of it is
> still on test, any advice will be welcomed.
> 
> Jingyi Wang (2):
>   KVM: arm64: Make use of TWED feature
>   KVM: arm64: Use dynamic TWE Delay value
> 
> Zengruan Ye (2):
>   arm64: cpufeature: TWED support detection
>   KVM: arm64: Add trace for TWED update
> 
>  arch/arm64/Kconfig                   | 10 +++++
>  arch/arm64/include/asm/cpucaps.h     |  3 +-
>  arch/arm64/include/asm/kvm_arm.h     |  5 +++
>  arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>  arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>  arch/arm64/include/asm/virt.h        |  8 ++++
>  arch/arm64/kernel/cpufeature.c       | 12 ++++++
>  arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>  arch/arm64/kvm/handle_exit.c         |  2 +
>  arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>  10 files changed, 174 insertions(+), 2 deletions(-)

-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc,

On 9/29/2020 6:50 PM, Marc Zyngier wrote:
> On 2020-09-29 10:17, Jingyi Wang wrote:
>> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
>> performance benefit in waiting for a period of time for an event to
>> arrive before taking the trap as it is common that event will arrive
>> “quite soon” after executing the WFE instruction.
> 
> Define "quite soon". Quantify "performance benefits". Which are the
> workloads that actually benefit from this imitation of the x86 PLE?
> 
> I was opposed to this when the spec was drafted, and I still am given
> that there is zero supporting evidence that it bring any gain over
> immediate trapping in an oversubscribed environment (which is the only
> case where it matters).
> 
> Thanks,
> 
>          M.

Sure, I will do more performance tests and post the results as soon as
possible.

Thanks,
Jingyi

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc,

On 9/29/2020 6:50 PM, Marc Zyngier wrote:
> On 2020-09-29 10:17, Jingyi Wang wrote:
>> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
>> performance benefit in waiting for a period of time for an event to
>> arrive before taking the trap as it is common that event will arrive
>> “quite soon” after executing the WFE instruction.
> 
> Define "quite soon". Quantify "performance benefits". Which are the
> workloads that actually benefit from this imitation of the x86 PLE?
> 
> I was opposed to this when the spec was drafted, and I still am given
> that there is zero supporting evidence that it bring any gain over
> immediate trapping in an oversubscribed environment (which is the only
> case where it matters).
> 
> Thanks,
> 
>          M.

Sure, I will do more performance tests and post the results as soon as
possible.

Thanks,
Jingyi
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc,

On 9/29/2020 6:50 PM, Marc Zyngier wrote:
> On 2020-09-29 10:17, Jingyi Wang wrote:
>> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
>> performance benefit in waiting for a period of time for an event to
>> arrive before taking the trap as it is common that event will arrive
>> “quite soon” after executing the WFE instruction.
> 
> Define "quite soon". Quantify "performance benefits". Which are the
> workloads that actually benefit from this imitation of the x86 PLE?
> 
> I was opposed to this when the spec was drafted, and I still am given
> that there is zero supporting evidence that it bring any gain over
> immediate trapping in an oversubscribed environment (which is the only
> case where it matters).
> 
> Thanks,
> 
>          M.

Sure, I will do more performance tests and post the results as soon as
possible.

Thanks,
Jingyi

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi all，

Sorry for the delay. I have been testing the TWED feature performance
lately. We select unixbench as the benchmark for some items of it is 
lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and 
here is the result.

      twed_value   | fstime        | fsbuffer   | fsdisk
     --------------+---------------+------------+------------
      disable      | 16.0          | 14.1       | 18.0
      0            | 16.3          | 13.5       | 17.2
      1            | 17.5          | 14.7       | 17.4
      2            | 17.3          | 15.3       | 18.0
      3            | 17.7          | 15.2       | 18.9
      4            | 17.9          | 14.3       | 18.2
      5            | 17.2          | 14.1       | 19.0
      6            | 5.8           | 4.2        | 5.7
      7            | 6.2           | 5.6        | 12.8

Note:
fstime: File Copy 1024 bufsize 2000 maxblocks
fsbuffer: File Copy 256 bufsize 500 maxblocks
fsdisk: File Copy 4096 bufsize 8000 maxblocks
The index of unixbench, higher is better.

It is shown that, compared to the circumstance that TWED is disabled,
lock-intensive testing items have better performance if an appropriate
TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench
test is run to prove that other testing items are not sensitive to this
parameter.

Thanks
Jingyi

On 9/29/2020 5:17 PM, Jingyi Wang wrote:
> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
> performance benefit in waiting for a period of time for an event to
> arrive before taking the trap as it is common that event will arrive
> “quite soon” after executing the WFE instruction.
> 
> This series adds support for TWED feature and implements TWE delay
> value dynamic adjustment.
> 
> Thanks for Shameer's advice on this series. The function of this patch
> has been tested on TWED supported hardware and the performance of it is
> still on test, any advice will be welcomed.
> 
> Jingyi Wang (2):
>    KVM: arm64: Make use of TWED feature
>    KVM: arm64: Use dynamic TWE Delay value
> 
> Zengruan Ye (2):
>    arm64: cpufeature: TWED support detection
>    KVM: arm64: Add trace for TWED update
> 
>   arch/arm64/Kconfig                   | 10 +++++
>   arch/arm64/include/asm/cpucaps.h     |  3 +-
>   arch/arm64/include/asm/kvm_arm.h     |  5 +++
>   arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>   arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>   arch/arm64/include/asm/virt.h        |  8 ++++
>   arch/arm64/kernel/cpufeature.c       | 12 ++++++
>   arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>   arch/arm64/kvm/handle_exit.c         |  2 +
>   arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>   10 files changed, 174 insertions(+), 2 deletions(-)
> 

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi all，

Sorry for the delay. I have been testing the TWED feature performance
lately. We select unixbench as the benchmark for some items of it is 
lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and 
here is the result.

      twed_value   | fstime        | fsbuffer   | fsdisk
     --------------+---------------+------------+------------
      disable      | 16.0          | 14.1       | 18.0
      0            | 16.3          | 13.5       | 17.2
      1            | 17.5          | 14.7       | 17.4
      2            | 17.3          | 15.3       | 18.0
      3            | 17.7          | 15.2       | 18.9
      4            | 17.9          | 14.3       | 18.2
      5            | 17.2          | 14.1       | 19.0
      6            | 5.8           | 4.2        | 5.7
      7            | 6.2           | 5.6        | 12.8

Note:
fstime: File Copy 1024 bufsize 2000 maxblocks
fsbuffer: File Copy 256 bufsize 500 maxblocks
fsdisk: File Copy 4096 bufsize 8000 maxblocks
The index of unixbench, higher is better.

It is shown that, compared to the circumstance that TWED is disabled,
lock-intensive testing items have better performance if an appropriate
TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench
test is run to prove that other testing items are not sensitive to this
parameter.

Thanks
Jingyi

On 9/29/2020 5:17 PM, Jingyi Wang wrote:
> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
> performance benefit in waiting for a period of time for an event to
> arrive before taking the trap as it is common that event will arrive
> “quite soon” after executing the WFE instruction.
> 
> This series adds support for TWED feature and implements TWE delay
> value dynamic adjustment.
> 
> Thanks for Shameer's advice on this series. The function of this patch
> has been tested on TWED supported hardware and the performance of it is
> still on test, any advice will be welcomed.
> 
> Jingyi Wang (2):
>    KVM: arm64: Make use of TWED feature
>    KVM: arm64: Use dynamic TWE Delay value
> 
> Zengruan Ye (2):
>    arm64: cpufeature: TWED support detection
>    KVM: arm64: Add trace for TWED update
> 
>   arch/arm64/Kconfig                   | 10 +++++
>   arch/arm64/include/asm/cpucaps.h     |  3 +-
>   arch/arm64/include/asm/kvm_arm.h     |  5 +++
>   arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>   arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>   arch/arm64/include/asm/virt.h        |  8 ++++
>   arch/arm64/kernel/cpufeature.c       | 12 ++++++
>   arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>   arch/arm64/kvm/handle_exit.c         |  2 +
>   arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>   10 files changed, 174 insertions(+), 2 deletions(-)
> 
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi all，

Sorry for the delay. I have been testing the TWED feature performance
lately. We select unixbench as the benchmark for some items of it is 
lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and 
here is the result.

      twed_value   | fstime        | fsbuffer   | fsdisk
     --------------+---------------+------------+------------
      disable      | 16.0          | 14.1       | 18.0
      0            | 16.3          | 13.5       | 17.2
      1            | 17.5          | 14.7       | 17.4
      2            | 17.3          | 15.3       | 18.0
      3            | 17.7          | 15.2       | 18.9
      4            | 17.9          | 14.3       | 18.2
      5            | 17.2          | 14.1       | 19.0
      6            | 5.8           | 4.2        | 5.7
      7            | 6.2           | 5.6        | 12.8

Note:
fstime: File Copy 1024 bufsize 2000 maxblocks
fsbuffer: File Copy 256 bufsize 500 maxblocks
fsdisk: File Copy 4096 bufsize 8000 maxblocks
The index of unixbench, higher is better.

It is shown that, compared to the circumstance that TWED is disabled,
lock-intensive testing items have better performance if an appropriate
TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench
test is run to prove that other testing items are not sensitive to this
parameter.

Thanks
Jingyi

On 9/29/2020 5:17 PM, Jingyi Wang wrote:
> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
> performance benefit in waiting for a period of time for an event to
> arrive before taking the trap as it is common that event will arrive
> “quite soon” after executing the WFE instruction.
> 
> This series adds support for TWED feature and implements TWE delay
> value dynamic adjustment.
> 
> Thanks for Shameer's advice on this series. The function of this patch
> has been tested on TWED supported hardware and the performance of it is
> still on test, any advice will be welcomed.
> 
> Jingyi Wang (2):
>    KVM: arm64: Make use of TWED feature
>    KVM: arm64: Use dynamic TWE Delay value
> 
> Zengruan Ye (2):
>    arm64: cpufeature: TWED support detection
>    KVM: arm64: Add trace for TWED update
> 
>   arch/arm64/Kconfig                   | 10 +++++
>   arch/arm64/include/asm/cpucaps.h     |  3 +-
>   arch/arm64/include/asm/kvm_arm.h     |  5 +++
>   arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>   arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>   arch/arm64/include/asm/virt.h        |  8 ++++
>   arch/arm64/kernel/cpufeature.c       | 12 ++++++
>   arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>   arch/arm64/kvm/handle_exit.c         |  2 +
>   arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>   10 files changed, 174 insertions(+), 2 deletions(-)
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc,

Gentle ping, could you please give some comments on this patch or the
current test results? Thanks in advance.

Thanks,
Jingyi

On 11/13/2020 3:54 PM, Jingyi Wang wrote:
> Hi all，
> 
> Sorry for the delay. I have been testing the TWED feature performance
> lately. We select unixbench as the benchmark for some items of it is 
> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and 
> here is the result.
> 
>       twed_value  | fstime     | fsbuffer  | fsdisk
>      --------------+---------------+------------+------------
>       disable   | 16.0      | 14.1    | 18.0
>       0      | 16.3      | 13.5    | 17.2
>       1      | 17.5      | 14.7    | 17.4
>       2      | 17.3      | 15.3    | 18.0
>       3      | 17.7      | 15.2    | 18.9
>       4      | 17.9      | 14.3    | 18.2
>       5      | 17.2      | 14.1    | 19.0
>       6      | 5.8       | 4.2     | 5.7
>       7      | 6.2      | 5.6     | 12.8
> 
> Note:
> fstime: File Copy 1024 bufsize 2000 maxblocks
> fsbuffer: File Copy 256 bufsize 500 maxblocks
> fsdisk: File Copy 4096 bufsize 8000 maxblocks
> The index of unixbench, higher is better.
> 
> It is shown that, compared to the circumstance that TWED is disabled,
> lock-intensive testing items have better performance if an appropriate
> TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench
> test is run to prove that other testing items are not sensitive to this
> parameter.
> 
> Thanks
> Jingyi
> 
> On 9/29/2020 5:17 PM, Jingyi Wang wrote:
>> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
>> performance benefit in waiting for a period of time for an event to
>> arrive before taking the trap as it is common that event will arrive
>> “quite soon” after executing the WFE instruction.
>>
>> This series adds support for TWED feature and implements TWE delay
>> value dynamic adjustment.
>>
>> Thanks for Shameer's advice on this series. The function of this patch
>> has been tested on TWED supported hardware and the performance of it is
>> still on test, any advice will be welcomed.
>>
>> Jingyi Wang (2):
>>    KVM: arm64: Make use of TWED feature
>>    KVM: arm64: Use dynamic TWE Delay value
>>
>> Zengruan Ye (2):
>>    arm64: cpufeature: TWED support detection
>>    KVM: arm64: Add trace for TWED update
>>
>>   arch/arm64/Kconfig                   | 10 +++++
>>   arch/arm64/include/asm/cpucaps.h     |  3 +-
>>   arch/arm64/include/asm/kvm_arm.h     |  5 +++
>>   arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>>   arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>>   arch/arm64/include/asm/virt.h        |  8 ++++
>>   arch/arm64/kernel/cpufeature.c       | 12 ++++++
>>   arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>>   arch/arm64/kvm/handle_exit.c         |  2 +
>>   arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>>   10 files changed, 174 insertions(+), 2 deletions(-)
>>

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc,

Gentle ping, could you please give some comments on this patch or the
current test results? Thanks in advance.

Thanks,
Jingyi

On 11/13/2020 3:54 PM, Jingyi Wang wrote:
> Hi all，
> 
> Sorry for the delay. I have been testing the TWED feature performance
> lately. We select unixbench as the benchmark for some items of it is 
> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and 
> here is the result.
> 
>       twed_value  | fstime     | fsbuffer  | fsdisk
>      --------------+---------------+------------+------------
>       disable   | 16.0      | 14.1    | 18.0
>       0      | 16.3      | 13.5    | 17.2
>       1      | 17.5      | 14.7    | 17.4
>       2      | 17.3      | 15.3    | 18.0
>       3      | 17.7      | 15.2    | 18.9
>       4      | 17.9      | 14.3    | 18.2
>       5      | 17.2      | 14.1    | 19.0
>       6      | 5.8       | 4.2     | 5.7
>       7      | 6.2      | 5.6     | 12.8
> 
> Note:
> fstime: File Copy 1024 bufsize 2000 maxblocks
> fsbuffer: File Copy 256 bufsize 500 maxblocks
> fsdisk: File Copy 4096 bufsize 8000 maxblocks
> The index of unixbench, higher is better.
> 
> It is shown that, compared to the circumstance that TWED is disabled,
> lock-intensive testing items have better performance if an appropriate
> TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench
> test is run to prove that other testing items are not sensitive to this
> parameter.
> 
> Thanks
> Jingyi
> 
> On 9/29/2020 5:17 PM, Jingyi Wang wrote:
>> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
>> performance benefit in waiting for a period of time for an event to
>> arrive before taking the trap as it is common that event will arrive
>> “quite soon” after executing the WFE instruction.
>>
>> This series adds support for TWED feature and implements TWE delay
>> value dynamic adjustment.
>>
>> Thanks for Shameer's advice on this series. The function of this patch
>> has been tested on TWED supported hardware and the performance of it is
>> still on test, any advice will be welcomed.
>>
>> Jingyi Wang (2):
>>    KVM: arm64: Make use of TWED feature
>>    KVM: arm64: Use dynamic TWE Delay value
>>
>> Zengruan Ye (2):
>>    arm64: cpufeature: TWED support detection
>>    KVM: arm64: Add trace for TWED update
>>
>>   arch/arm64/Kconfig                   | 10 +++++
>>   arch/arm64/include/asm/cpucaps.h     |  3 +-
>>   arch/arm64/include/asm/kvm_arm.h     |  5 +++
>>   arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>>   arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>>   arch/arm64/include/asm/virt.h        |  8 ++++
>>   arch/arm64/kernel/cpufeature.c       | 12 ++++++
>>   arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>>   arch/arm64/kvm/handle_exit.c         |  2 +
>>   arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>>   10 files changed, 174 insertions(+), 2 deletions(-)
>>
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc,

Gentle ping, could you please give some comments on this patch or the
current test results? Thanks in advance.

Thanks,
Jingyi

On 11/13/2020 3:54 PM, Jingyi Wang wrote:
> Hi all，
> 
> Sorry for the delay. I have been testing the TWED feature performance
> lately. We select unixbench as the benchmark for some items of it is 
> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and 
> here is the result.
> 
>       twed_value  | fstime     | fsbuffer  | fsdisk
>      --------------+---------------+------------+------------
>       disable   | 16.0      | 14.1    | 18.0
>       0      | 16.3      | 13.5    | 17.2
>       1      | 17.5      | 14.7    | 17.4
>       2      | 17.3      | 15.3    | 18.0
>       3      | 17.7      | 15.2    | 18.9
>       4      | 17.9      | 14.3    | 18.2
>       5      | 17.2      | 14.1    | 19.0
>       6      | 5.8       | 4.2     | 5.7
>       7      | 6.2      | 5.6     | 12.8
> 
> Note:
> fstime: File Copy 1024 bufsize 2000 maxblocks
> fsbuffer: File Copy 256 bufsize 500 maxblocks
> fsdisk: File Copy 4096 bufsize 8000 maxblocks
> The index of unixbench, higher is better.
> 
> It is shown that, compared to the circumstance that TWED is disabled,
> lock-intensive testing items have better performance if an appropriate
> TWED value is set(up to 5.6%~11.9%). Meanwhile, the complete unixbench
> test is run to prove that other testing items are not sensitive to this
> parameter.
> 
> Thanks
> Jingyi
> 
> On 9/29/2020 5:17 PM, Jingyi Wang wrote:
>> TWE Delay is an optional feature in ARMv8.6 Extentions. There is a
>> performance benefit in waiting for a period of time for an event to
>> arrive before taking the trap as it is common that event will arrive
>> “quite soon” after executing the WFE instruction.
>>
>> This series adds support for TWED feature and implements TWE delay
>> value dynamic adjustment.
>>
>> Thanks for Shameer's advice on this series. The function of this patch
>> has been tested on TWED supported hardware and the performance of it is
>> still on test, any advice will be welcomed.
>>
>> Jingyi Wang (2):
>>    KVM: arm64: Make use of TWED feature
>>    KVM: arm64: Use dynamic TWE Delay value
>>
>> Zengruan Ye (2):
>>    arm64: cpufeature: TWED support detection
>>    KVM: arm64: Add trace for TWED update
>>
>>   arch/arm64/Kconfig                   | 10 +++++
>>   arch/arm64/include/asm/cpucaps.h     |  3 +-
>>   arch/arm64/include/asm/kvm_arm.h     |  5 +++
>>   arch/arm64/include/asm/kvm_emulate.h | 38 ++++++++++++++++++
>>   arch/arm64/include/asm/kvm_host.h    | 19 ++++++++-
>>   arch/arm64/include/asm/virt.h        |  8 ++++
>>   arch/arm64/kernel/cpufeature.c       | 12 ++++++
>>   arch/arm64/kvm/arm.c                 | 58 ++++++++++++++++++++++++++++
>>   arch/arm64/kvm/handle_exit.c         |  2 +
>>   arch/arm64/kvm/trace_arm.h           | 21 ++++++++++
>>   10 files changed, 174 insertions(+), 2 deletions(-)
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Marc Zyngier]

On 2020-11-13 07:54, Jingyi Wang wrote:
> Hi all，
> 
> Sorry for the delay. I have been testing the TWED feature performance
> lately. We select unixbench as the benchmark for some items of it is
> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and
> here is the result.

How representative is this?

TBH, I only know of two real world configurations: one where
the vCPUs are pinned to different physical CPUs (and in this
case your patch has absolutely no effect as long as there is
no concurrent tasks), and one where there is oversubscription,
and the scheduler moves things around as it sees fit, depending
on the load.

Having two vCPUs pinned per CPU feels like a test that has been
picked to give the result you wanted. I'd like to see the full
picture, including the case that matters for current use cases.
I'm specially interested in the cases where the system is
oversubscribed, because TWED is definitely going to screw with
the scheduler latency.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Marc Zyngier]

On 2020-11-13 07:54, Jingyi Wang wrote:
> Hi all，
> 
> Sorry for the delay. I have been testing the TWED feature performance
> lately. We select unixbench as the benchmark for some items of it is
> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and
> here is the result.

How representative is this?

TBH, I only know of two real world configurations: one where
the vCPUs are pinned to different physical CPUs (and in this
case your patch has absolutely no effect as long as there is
no concurrent tasks), and one where there is oversubscription,
and the scheduler moves things around as it sees fit, depending
on the load.

Having two vCPUs pinned per CPU feels like a test that has been
picked to give the result you wanted. I'd like to see the full
picture, including the case that matters for current use cases.
I'm specially interested in the cases where the system is
oversubscribed, because TWED is definitely going to screw with
the scheduler latency.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Marc Zyngier]

On 2020-11-13 07:54, Jingyi Wang wrote:
> Hi all，
> 
> Sorry for the delay. I have been testing the TWED feature performance
> lately. We select unixbench as the benchmark for some items of it is
> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and
> here is the result.

How representative is this?

TBH, I only know of two real world configurations: one where
the vCPUs are pinned to different physical CPUs (and in this
case your patch has absolutely no effect as long as there is
no concurrent tasks), and one where there is oversubscription,
and the scheduler moves things around as it sees fit, depending
on the load.

Having two vCPUs pinned per CPU feels like a test that has been
picked to give the result you wanted. I'd like to see the full
picture, including the case that matters for current use cases.
I'm specially interested in the cases where the system is
oversubscribed, because TWED is definitely going to screw with
the scheduler latency.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc，

I will consider more circumstances in the later test. Thanks for the
advice.

Thanks,
Jingyi


On 11/24/2020 7:02 PM, Marc Zyngier wrote:
> On 2020-11-13 07:54, Jingyi Wang wrote:
>> Hi all，
>>
>> Sorry for the delay. I have been testing the TWED feature performance
>> lately. We select unixbench as the benchmark for some items of it is
>> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
>> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and
>> here is the result.
> 
> How representative is this?
> 
> TBH, I only know of two real world configurations: one where
> the vCPUs are pinned to different physical CPUs (and in this
> case your patch has absolutely no effect as long as there is
> no concurrent tasks), and one where there is oversubscription,
> and the scheduler moves things around as it sees fit, depending
> on the load.
> 
> Having two vCPUs pinned per CPU feels like a test that has been
> picked to give the result you wanted. I'd like to see the full
> picture, including the case that matters for current use cases.
> I'm specially interested in the cases where the system is
> oversubscribed, because TWED is definitely going to screw with
> the scheduler latency.
> 
> Thanks,
> 
>          M.

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc，

I will consider more circumstances in the later test. Thanks for the
advice.

Thanks,
Jingyi


On 11/24/2020 7:02 PM, Marc Zyngier wrote:
> On 2020-11-13 07:54, Jingyi Wang wrote:
>> Hi all，
>>
>> Sorry for the delay. I have been testing the TWED feature performance
>> lately. We select unixbench as the benchmark for some items of it is
>> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
>> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and
>> here is the result.
> 
> How representative is this?
> 
> TBH, I only know of two real world configurations: one where
> the vCPUs are pinned to different physical CPUs (and in this
> case your patch has absolutely no effect as long as there is
> no concurrent tasks), and one where there is oversubscription,
> and the scheduler moves things around as it sees fit, depending
> on the load.
> 
> Having two vCPUs pinned per CPU feels like a test that has been
> picked to give the result you wanted. I'd like to see the full
> picture, including the case that matters for current use cases.
> I'm specially interested in the cases where the system is
> oversubscribed, because TWED is definitely going to screw with
> the scheduler latency.
> 
> Thanks,
> 
>          M.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC PATCH 0/4] Add support for ARMv8.6 TWED feature

[Jingyi Wang]

Hi Marc，

I will consider more circumstances in the later test. Thanks for the
advice.

Thanks,
Jingyi


On 11/24/2020 7:02 PM, Marc Zyngier wrote:
> On 2020-11-13 07:54, Jingyi Wang wrote:
>> Hi all，
>>
>> Sorry for the delay. I have been testing the TWED feature performance
>> lately. We select unixbench as the benchmark for some items of it is
>> lock-intensive(fstime/fsbuffer/fsdisk). We run unixbench on a 4-VCPU
>> VM, and bind every two VCPUs on one PCPU. Fixed TWED value is used and
>> here is the result.
> 
> How representative is this?
> 
> TBH, I only know of two real world configurations: one where
> the vCPUs are pinned to different physical CPUs (and in this
> case your patch has absolutely no effect as long as there is
> no concurrent tasks), and one where there is oversubscription,
> and the scheduler moves things around as it sees fit, depending
> on the load.
> 
> Having two vCPUs pinned per CPU feels like a test that has been
> picked to give the result you wanted. I'd like to see the full
> picture, including the case that matters for current use cases.
> I'm specially interested in the cases where the system is
> oversubscribed, because TWED is definitely going to screw with
> the scheduler latency.
> 
> Thanks,
> 
>          M.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel