[PATCH 0/2] x86/svm: add pause filtering threshold for SVM

[PATCH 0/2] x86/svm: add pause filtering threshold for SVM

[Brian Woods]

This patch series adds support and enablement of the pause filtering
threshold.  Once there's pause filtering threshold amount of cycles
between pauses, the pause filtering counter resets to what was in the
VMCB.  This allows the pause filtering count to "reset" between pauses
and keeps the guset from getting intercepted by the hypervisor.  See AMD
APM vol 2 section 15.14.4 for more details.

In reply to this will be an email with graphs showing some benchmark
results of why the values of the counter and threshold were picked.

Brian Woods (2):
  x86/svm: add support for pause filtering threshold
  x86/svm: enable pause filtering threshold

 xen/arch/x86/hvm/svm/svm.c         | 1 +
 xen/arch/x86/hvm/svm/vmcb.c        | 3 +++
 xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
 xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
 4 files changed, 10 insertions(+), 2 deletions(-)

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 1/2] x86/svm: add support for pause filtering threshold

[Brian Woods]

Add support for enabling the pause filtering threshold feature.  This
causes the pause filtering count to reset if there's pause filtering
threshold cycles or greater between pauses.  See AMD APM Vol 2 Section
15.14.4 for more details.

The values of the pause filtering count and threshold were found by
iterating over different values of the count and threshold while running
kernbench and a pi spigot algorithm with yields placed in it.  A
balanced setting for both variable provides:

(Using averaged elapsed time with kernbench)
old = 852.0
new = 848.8
improvement = .4%

For system without pause filtering threshold, the change, from 3000 to
4000 for the count, should not negatively effect system performance.

Signed-off-by: Brian Woods <brian.woods@amd.com>
---
 xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
 xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/xen/include/asm-x86/hvm/svm/svm.h b/xen/include/asm-x86/hvm/svm/svm.h
index 462cb89b7c..593546fb56 100644
--- a/xen/include/asm-x86/hvm/svm/svm.h
+++ b/xen/include/asm-x86/hvm/svm/svm.h
@@ -64,6 +64,7 @@ extern u32 svm_feature_flags;
 #define SVM_FEATURE_FLUSHBYASID    6 /* TLB flush by ASID support */
 #define SVM_FEATURE_DECODEASSISTS  7 /* Decode assists support */
 #define SVM_FEATURE_PAUSEFILTER   10 /* Pause intercept filter support */
+#define SVM_FEATURE_PAUSETHRESH   12 /* Pause intercept filter support */
 #define SVM_FEATURE_VLOADSAVE     15 /* virtual vmload/vmsave */
 #define SVM_FEATURE_VGIF          16 /* Virtual GIF */
 
@@ -76,10 +77,12 @@ extern u32 svm_feature_flags;
 #define cpu_has_svm_decode    cpu_has_svm_feature(SVM_FEATURE_DECODEASSISTS)
 #define cpu_has_svm_vgif      cpu_has_svm_feature(SVM_FEATURE_VGIF)
 #define cpu_has_pause_filter  cpu_has_svm_feature(SVM_FEATURE_PAUSEFILTER)
+#define cpu_has_pause_thresh  cpu_has_svm_feature(SVM_FEATURE_PAUSETHRESH)
 #define cpu_has_tsc_ratio     cpu_has_svm_feature(SVM_FEATURE_TSCRATEMSR)
 #define cpu_has_svm_vloadsave cpu_has_svm_feature(SVM_FEATURE_VLOADSAVE)
 
-#define SVM_PAUSEFILTER_INIT    3000
+#define SVM_PAUSEFILTER_INIT    4000
+#define SVM_PAUSETHRESH_INIT    1000
 
 /* TSC rate */
 #define DEFAULT_TSC_RATIO       0x0000000100000000ULL
diff --git a/xen/include/asm-x86/hvm/svm/vmcb.h b/xen/include/asm-x86/hvm/svm/vmcb.h
index 9d5dfc58f2..de07429dff 100644
--- a/xen/include/asm-x86/hvm/svm/vmcb.h
+++ b/xen/include/asm-x86/hvm/svm/vmcb.h
@@ -412,7 +412,7 @@ struct vmcb_struct {
     u64 res04;                  /* offset 0x28 */
     u64 res05;                  /* offset 0x30 */
     u32 res06;                  /* offset 0x38 */
-    u16 res06a;                 /* offset 0x3C */
+    u16 _pause_filter_thresh;   /* offset 0x3C - cleanbit 0 */
     u16 _pause_filter_count;    /* offset 0x3E - cleanbit 0 */
     u64 _iopm_base_pa;          /* offset 0x40 - cleanbit 1 */
     u64 _msrpm_base_pa;         /* offset 0x48 - cleanbit 1 */
@@ -568,6 +568,7 @@ VMCB_ACCESSORS(exception_intercepts, intercepts)
 VMCB_ACCESSORS(general1_intercepts, intercepts)
 VMCB_ACCESSORS(general2_intercepts, intercepts)
 VMCB_ACCESSORS(pause_filter_count, intercepts)
+VMCB_ACCESSORS(pause_filter_thresh, intercepts)
 VMCB_ACCESSORS(tsc_offset, intercepts)
 VMCB_ACCESSORS(iopm_base_pa, iopm)
 VMCB_ACCESSORS(msrpm_base_pa, iopm)
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 2/2] x86/svm: enable pause filtering threshold

[Brian Woods]

If available, enable the pause filtering threshold feature.  See the
previous commit for more information.

Signed-off-by: Brian Woods <brian.woods@amd.com>
---
 xen/arch/x86/hvm/svm/svm.c  | 1 +
 xen/arch/x86/hvm/svm/vmcb.c | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 9f58afc2d8..b6b92365bf 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -1694,6 +1694,7 @@ const struct hvm_function_table * __init start_svm(void)
     P(cpu_has_svm_vloadsave, "Virtual VMLOAD/VMSAVE");
     P(cpu_has_svm_vgif, "Virtual GIF");
     P(cpu_has_pause_filter, "Pause-Intercept Filter");
+    P(cpu_has_pause_thresh, "Pause-Intercept Filter Threshold");
     P(cpu_has_tsc_ratio, "TSC Rate MSR");
 #undef P
 
diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c
index 0e6cba5b7b..ebe6f0c751 100644
--- a/xen/arch/x86/hvm/svm/vmcb.c
+++ b/xen/arch/x86/hvm/svm/vmcb.c
@@ -227,6 +227,9 @@ static int construct_vmcb(struct vcpu *v)
     {
         vmcb->_pause_filter_count = SVM_PAUSEFILTER_INIT;
         vmcb->_general1_intercepts |= GENERAL1_INTERCEPT_PAUSE;
+
+        if ( cpu_has_pause_thresh )
+            vmcb->_pause_filter_thresh = SVM_PAUSETHRESH_INIT;
     }
 
     vmcb->cleanbits.bytes = 0;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/2] x86/svm: add pause filtering threshold for SVM

[Brian Woods]

[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]

HW Setup:
	CPU: AMD EPYC 7401 24c/48t x 2 
	RAM: 128GB RAM
	Storage: HDD for dom0, SSDs for VMs
SW Setup:
	Xen base 4c7e478d597b0346eef3a256cfd6794ac778b608
	32 VMs
		6 VPCUs per VM
		3.5GB RAM per VM
		7.5GB disk per VM

Results:
	kernbench:
		half -> -j vpcu/2 = one job per host thread
		quad -> -j vpcu*4 = two VMs per host thread,
                                    four jobs per VM VCPU
	pi:
		03 ->  3 in parallel = one pi per host thread
		24 -> 24 in parallel = two VMs per host thread,
                                       four pis per VM VCPU

Kernbench: https://github.com/linux-test-project/ltp/blob/master/utils/benchmark/kernbench-0.42/kernbench
pi: based off of https://stackoverflow.com/questions/4084571/implementing-the-spigot-algorithm-for-%CF%80-pi
    but with yields and 10ns sleeps added in. 

Between the current usage of pause filtering counter, there's about an
~.5% improve in performance.  While this isn't huge, all that's
required is to write a value in the VMCB.

Attached: 
	elapsed_time_kernbenchhalf.pdf
	elapsed_time_kernbenchquad.pdf
	elapsed_time_pi03.pdf
	elapsed_time_pi24.pdf

-- 
Brian Woods

[-- Attachment #2: elapsed_time_kernbenchquad.pdf --]
[-- Type: application/pdf, Size: 9561 bytes --]

[-- Attachment #3: elapsed_time_kernbenchhalf.pdf --]
[-- Type: application/pdf, Size: 10552 bytes --]

[-- Attachment #4: elapsed_time_pi03.pdf --]
[-- Type: application/pdf, Size: 8972 bytes --]

[-- Attachment #5: elapsed_time_pi24.pdf --]
[-- Type: application/pdf, Size: 11710 bytes --]

[-- Attachment #6: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 1/2] x86/svm: add support for pause filtering threshold

[Moger, Babu]

Looks good. 
Reviewed-by: Babu Moger <babu.moger@amd.com>

> -----Original Message-----
> From: Woods, Brian
> Sent: Tuesday, February 20, 2018 6:13 PM
> To: xen-devel@lists.xen.org
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Jan Beulich <jbeulich@suse.com>;
> Andrew Cooper <andrew.cooper3@citrix.com>; Woods, Brian
> <Brian.Woods@amd.com>
> Subject: [PATCH 1/2] x86/svm: add support for pause filtering threshold
> 
> Add support for enabling the pause filtering threshold feature.  This
> causes the pause filtering count to reset if there's pause filtering
> threshold cycles or greater between pauses.  See AMD APM Vol 2 Section
> 15.14.4 for more details.
> 
> The values of the pause filtering count and threshold were found by
> iterating over different values of the count and threshold while running
> kernbench and a pi spigot algorithm with yields placed in it.  A
> balanced setting for both variable provides:
> 
> (Using averaged elapsed time with kernbench)
> old = 852.0
> new = 848.8
> improvement = .4%
> 
> For system without pause filtering threshold, the change, from 3000 to
> 4000 for the count, should not negatively effect system performance.
> 
> Signed-off-by: Brian Woods <brian.woods@amd.com>
> ---
>  xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
>  xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
>  2 files changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/include/asm-x86/hvm/svm/svm.h b/xen/include/asm-
> x86/hvm/svm/svm.h
> index 462cb89b7c..593546fb56 100644
> --- a/xen/include/asm-x86/hvm/svm/svm.h
> +++ b/xen/include/asm-x86/hvm/svm/svm.h
> @@ -64,6 +64,7 @@ extern u32 svm_feature_flags;
>  #define SVM_FEATURE_FLUSHBYASID    6 /* TLB flush by ASID support */
>  #define SVM_FEATURE_DECODEASSISTS  7 /* Decode assists support */
>  #define SVM_FEATURE_PAUSEFILTER   10 /* Pause intercept filter support
> */
> +#define SVM_FEATURE_PAUSETHRESH   12 /* Pause intercept filter support
> */
>  #define SVM_FEATURE_VLOADSAVE     15 /* virtual vmload/vmsave */
>  #define SVM_FEATURE_VGIF          16 /* Virtual GIF */
> 
> @@ -76,10 +77,12 @@ extern u32 svm_feature_flags;
>  #define cpu_has_svm_decode
> cpu_has_svm_feature(SVM_FEATURE_DECODEASSISTS)
>  #define cpu_has_svm_vgif      cpu_has_svm_feature(SVM_FEATURE_VGIF)
>  #define cpu_has_pause_filter
> cpu_has_svm_feature(SVM_FEATURE_PAUSEFILTER)
> +#define cpu_has_pause_thresh
> cpu_has_svm_feature(SVM_FEATURE_PAUSETHRESH)
>  #define cpu_has_tsc_ratio
> cpu_has_svm_feature(SVM_FEATURE_TSCRATEMSR)
>  #define cpu_has_svm_vloadsave
> cpu_has_svm_feature(SVM_FEATURE_VLOADSAVE)
> 
> -#define SVM_PAUSEFILTER_INIT    3000
> +#define SVM_PAUSEFILTER_INIT    4000
> +#define SVM_PAUSETHRESH_INIT    1000
> 
>  /* TSC rate */
>  #define DEFAULT_TSC_RATIO       0x0000000100000000ULL
> diff --git a/xen/include/asm-x86/hvm/svm/vmcb.h b/xen/include/asm-
> x86/hvm/svm/vmcb.h
> index 9d5dfc58f2..de07429dff 100644
> --- a/xen/include/asm-x86/hvm/svm/vmcb.h
> +++ b/xen/include/asm-x86/hvm/svm/vmcb.h
> @@ -412,7 +412,7 @@ struct vmcb_struct {
>      u64 res04;                  /* offset 0x28 */
>      u64 res05;                  /* offset 0x30 */
>      u32 res06;                  /* offset 0x38 */
> -    u16 res06a;                 /* offset 0x3C */
> +    u16 _pause_filter_thresh;   /* offset 0x3C - cleanbit 0 */
>      u16 _pause_filter_count;    /* offset 0x3E - cleanbit 0 */
>      u64 _iopm_base_pa;          /* offset 0x40 - cleanbit 1 */
>      u64 _msrpm_base_pa;         /* offset 0x48 - cleanbit 1 */
> @@ -568,6 +568,7 @@ VMCB_ACCESSORS(exception_intercepts, intercepts)
>  VMCB_ACCESSORS(general1_intercepts, intercepts)
>  VMCB_ACCESSORS(general2_intercepts, intercepts)
>  VMCB_ACCESSORS(pause_filter_count, intercepts)
> +VMCB_ACCESSORS(pause_filter_thresh, intercepts)
>  VMCB_ACCESSORS(tsc_offset, intercepts)
>  VMCB_ACCESSORS(iopm_base_pa, iopm)
>  VMCB_ACCESSORS(msrpm_base_pa, iopm)
> --
> 2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 2/2] x86/svm: enable pause filtering threshold

[Moger, Babu]

Looks good. 
Reviewed-by: Babu Moger <babu.moger@amd.com>

> -----Original Message-----
> From: Woods, Brian
> Sent: Tuesday, February 20, 2018 6:13 PM
> To: xen-devel@lists.xen.org
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Jan Beulich <jbeulich@suse.com>;
> Andrew Cooper <andrew.cooper3@citrix.com>; Woods, Brian
> <Brian.Woods@amd.com>
> Subject: [PATCH 2/2] x86/svm: enable pause filtering threshold
> 
> If available, enable the pause filtering threshold feature.  See the
> previous commit for more information.
> 
> Signed-off-by: Brian Woods <brian.woods@amd.com>
> ---
>  xen/arch/x86/hvm/svm/svm.c  | 1 +
>  xen/arch/x86/hvm/svm/vmcb.c | 3 +++
>  2 files changed, 4 insertions(+)
> 
> diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
> index 9f58afc2d8..b6b92365bf 100644
> --- a/xen/arch/x86/hvm/svm/svm.c
> +++ b/xen/arch/x86/hvm/svm/svm.c
> @@ -1694,6 +1694,7 @@ const struct hvm_function_table * __init
> start_svm(void)
>      P(cpu_has_svm_vloadsave, "Virtual VMLOAD/VMSAVE");
>      P(cpu_has_svm_vgif, "Virtual GIF");
>      P(cpu_has_pause_filter, "Pause-Intercept Filter");
> +    P(cpu_has_pause_thresh, "Pause-Intercept Filter Threshold");
>      P(cpu_has_tsc_ratio, "TSC Rate MSR");
>  #undef P
> 
> diff --git a/xen/arch/x86/hvm/svm/vmcb.c
> b/xen/arch/x86/hvm/svm/vmcb.c
> index 0e6cba5b7b..ebe6f0c751 100644
> --- a/xen/arch/x86/hvm/svm/vmcb.c
> +++ b/xen/arch/x86/hvm/svm/vmcb.c
> @@ -227,6 +227,9 @@ static int construct_vmcb(struct vcpu *v)
>      {
>          vmcb->_pause_filter_count = SVM_PAUSEFILTER_INIT;
>          vmcb->_general1_intercepts |= GENERAL1_INTERCEPT_PAUSE;
> +
> +        if ( cpu_has_pause_thresh )
> +            vmcb->_pause_filter_thresh = SVM_PAUSETHRESH_INIT;
>      }
> 
>      vmcb->cleanbits.bytes = 0;
> --
> 2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/2] x86/svm: add pause filtering threshold for SVM

[Boris Ostrovsky]

On 02/20/2018 07:12 PM, Brian Woods wrote:
> This patch series adds support and enablement of the pause filtering
> threshold.  Once there's pause filtering threshold amount of cycles
> between pauses, the pause filtering counter resets to what was in the
> VMCB.  This allows the pause filtering count to "reset" between pauses
> and keeps the guset from getting intercepted by the hypervisor.  See AMD
> APM vol 2 section 15.14.4 for more details.
>
> In reply to this will be an email with graphs showing some benchmark
> results of why the values of the counter and threshold were picked.
>
> Brian Woods (2):
>   x86/svm: add support for pause filtering threshold
>   x86/svm: enable pause filtering threshold
>
>  xen/arch/x86/hvm/svm/svm.c         | 1 +
>  xen/arch/x86/hvm/svm/vmcb.c        | 3 +++
>  xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
>  xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
>  4 files changed, 10 insertions(+), 2 deletions(-)
>

IIRC previous count value  (3000) was somewhat arbitrary so

Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 1/2] x86/svm: add support for pause filtering threshold

[Jan Beulich]

>>> On 21.02.18 at 01:12, <brian.woods@amd.com> wrote:
> Add support for enabling the pause filtering threshold feature.  This
> causes the pause filtering count to reset if there's pause filtering
> threshold cycles or greater between pauses.  See AMD APM Vol 2 Section
> 15.14.4 for more details.

FTR I'm going to commit this as is (since it has been acked), but in
the future please don't refer to documentation sections by number,
but instead by title (as the numbers tend to change over time).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel