* [Xen-devel] [PATCH v2 1/3] x86/hvm: allow ASID flush when v != current
2020-01-10 16:04 [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode Roger Pau Monne
@ 2020-01-10 16:04 ` Roger Pau Monne
2020-01-10 16:04 ` [Xen-devel] [PATCH v2 2/3] x86/hvm: rework HVMOP_flush_tlbs Roger Pau Monne
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Roger Pau Monne @ 2020-01-10 16:04 UTC (permalink / raw)
To: xen-devel; +Cc: Andrew Cooper, Wei Liu, Jan Beulich, Roger Pau Monne
Current implementation of hvm_asid_flush_vcpu is not safe to use
unless the target vCPU is either paused or the currently running one,
as it modifies the generation without any locking.
Fix this by using atomic operations when accessing the generation
field, both in hvm_asid_flush_vcpu_asid and other ASID functions. This
allows to safely flush the current ASID generation. Note that for the
flush to take effect if the vCPU is currently running a vmexit is
required.
Note the same could be achieved by introducing an extra field to
hvm_vcpu_asid that signals hvm_asid_handle_vmenter the need to call
hvm_asid_flush_vcpu on the given vCPU before vmentry, this however
seems unnecessary as hvm_asid_flush_vcpu itself only sets two vCPU
fields to 0, so there's no need to delay this to the vmentry ASID
helper.
This is not a bugfix as no callers that would violate the assumptions
listed in the first paragraph have been found, but a preparatory
change in order to allow remote flushing of HVM vCPUs.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
xen/arch/x86/hvm/asid.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/hvm/asid.c b/xen/arch/x86/hvm/asid.c
index 9d3c671a5f..80b73da89b 100644
--- a/xen/arch/x86/hvm/asid.c
+++ b/xen/arch/x86/hvm/asid.c
@@ -82,7 +82,7 @@ void hvm_asid_init(int nasids)
void hvm_asid_flush_vcpu_asid(struct hvm_vcpu_asid *asid)
{
- asid->generation = 0;
+ write_atomic(&asid->generation, 0);
}
void hvm_asid_flush_vcpu(struct vcpu *v)
@@ -120,7 +120,7 @@ bool_t hvm_asid_handle_vmenter(struct hvm_vcpu_asid *asid)
goto disabled;
/* Test if VCPU has valid ASID. */
- if ( asid->generation == data->core_asid_generation )
+ if ( read_atomic(&asid->generation) == data->core_asid_generation )
return 0;
/* If there are no free ASIDs, need to go to a new generation */
@@ -134,7 +134,7 @@ bool_t hvm_asid_handle_vmenter(struct hvm_vcpu_asid *asid)
/* Now guaranteed to be a free ASID. */
asid->asid = data->next_asid++;
- asid->generation = data->core_asid_generation;
+ write_atomic(&asid->generation, data->core_asid_generation);
/*
* When we assign ASID 1, flush all TLB entries as we are starting a new
--
2.24.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 6+ messages in thread* [Xen-devel] [PATCH v2 2/3] x86/hvm: rework HVMOP_flush_tlbs
2020-01-10 16:04 [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode Roger Pau Monne
2020-01-10 16:04 ` [Xen-devel] [PATCH v2 1/3] x86/hvm: allow ASID flush when v != current Roger Pau Monne
@ 2020-01-10 16:04 ` Roger Pau Monne
2020-01-10 16:04 ` [Xen-devel] [PATCH v2 3/3] x86/tlb: use Xen L0 assisted TLB flush when available Roger Pau Monne
2020-01-10 16:08 ` [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode Jan Beulich
3 siblings, 0 replies; 6+ messages in thread
From: Roger Pau Monne @ 2020-01-10 16:04 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Paul Durrant, Wei Liu, Jan Beulich, Roger Pau Monne
Current implementation of hvm_flush_vcpu_tlb is highly inefficient.
First of all the call to flush_tlb_mask is completely useless when
trying to flush the TLB of HVM guests, as this TLB flush is executed in
root mode, and hence doesn't flush any guest state cache.
Secondly, calling paging_update_cr3 albeit correct, is much more
expensive than strictly required. Instead a TLB flush can be achieved by
calling hvm_asid_flush_vcpu on each pCPU that has a domain vCPU state
currently loaded. This call will invalidate the current non-root
context, thus forcing a clean cache state on vmentry. If the guest is
not using ASIDs, the vmexit caused by the on_selected_cpus IPI will
already force a TLB flush.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
xen/arch/x86/hvm/hvm.c | 54 ++++++++++++----------------
xen/arch/x86/hvm/viridian/viridian.c | 7 +---
xen/include/asm-x86/hvm/hvm.h | 2 +-
3 files changed, 25 insertions(+), 38 deletions(-)
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 4723f5d09c..e4fef0afcd 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -3973,7 +3973,21 @@ static void hvm_s3_resume(struct domain *d)
}
}
-bool hvm_flush_vcpu_tlb(bool (*flush_vcpu)(void *ctxt, struct vcpu *v),
+static void do_flush(void *data)
+{
+ cpumask_t *mask = data;
+ unsigned int cpu = smp_processor_id();
+
+ ASSERT(cpumask_test_cpu(cpu, mask));
+ /*
+ * A vmexit/vmenter (caused by the IPI issued to execute this function) is
+ * enough to force a TLB flush since we have already ticked the vCPU ASID
+ * prior to issuing the IPI.
+ */
+ cpumask_clear_cpu(cpu, mask);
+}
+
+void hvm_flush_vcpu_tlb(bool (*flush_vcpu)(void *ctxt, struct vcpu *v),
void *ctxt)
{
static DEFINE_PER_CPU(cpumask_t, flush_cpumask);
@@ -3981,27 +3995,8 @@ bool hvm_flush_vcpu_tlb(bool (*flush_vcpu)(void *ctxt, struct vcpu *v),
struct domain *d = current->domain;
struct vcpu *v;
- /* Avoid deadlock if more than one vcpu tries this at the same time. */
- if ( !spin_trylock(&d->hypercall_deadlock_mutex) )
- return false;
-
- /* Pause all other vcpus. */
- for_each_vcpu ( d, v )
- if ( v != current && flush_vcpu(ctxt, v) )
- vcpu_pause_nosync(v);
-
- /* Now that all VCPUs are signalled to deschedule, we wait... */
- for_each_vcpu ( d, v )
- if ( v != current && flush_vcpu(ctxt, v) )
- while ( !vcpu_runnable(v) && v->is_running )
- cpu_relax();
-
- /* All other vcpus are paused, safe to unlock now. */
- spin_unlock(&d->hypercall_deadlock_mutex);
-
cpumask_clear(mask);
- /* Flush paging-mode soft state (e.g., va->gfn cache; PAE PDPE cache). */
for_each_vcpu ( d, v )
{
unsigned int cpu;
@@ -4009,22 +4004,17 @@ bool hvm_flush_vcpu_tlb(bool (*flush_vcpu)(void *ctxt, struct vcpu *v),
if ( !flush_vcpu(ctxt, v) )
continue;
- paging_update_cr3(v, false);
+ hvm_asid_flush_vcpu(v);
cpu = read_atomic(&v->dirty_cpu);
- if ( is_vcpu_dirty_cpu(cpu) )
+ if ( cpu != smp_processor_id() && is_vcpu_dirty_cpu(cpu) )
__cpumask_set_cpu(cpu, mask);
}
- /* Flush TLBs on all CPUs with dirty vcpu state. */
- flush_tlb_mask(mask);
+ on_selected_cpus(mask, do_flush, mask, 0);
- /* Done. */
- for_each_vcpu ( d, v )
- if ( v != current && flush_vcpu(ctxt, v) )
- vcpu_unpause(v);
-
- return true;
+ while ( !cpumask_empty(mask) )
+ cpu_relax();
}
static bool always_flush(void *ctxt, struct vcpu *v)
@@ -4037,7 +4027,9 @@ static int hvmop_flush_tlb_all(void)
if ( !is_hvm_domain(current->domain) )
return -EINVAL;
- return hvm_flush_vcpu_tlb(always_flush, NULL) ? 0 : -ERESTART;
+ hvm_flush_vcpu_tlb(always_flush, NULL);
+
+ return 0;
}
static int hvmop_set_evtchn_upcall_vector(
diff --git a/xen/arch/x86/hvm/viridian/viridian.c b/xen/arch/x86/hvm/viridian/viridian.c
index 44c8e6cac6..ec73361597 100644
--- a/xen/arch/x86/hvm/viridian/viridian.c
+++ b/xen/arch/x86/hvm/viridian/viridian.c
@@ -604,12 +604,7 @@ int viridian_hypercall(struct cpu_user_regs *regs)
if ( input_params.flags & HV_FLUSH_ALL_PROCESSORS )
input_params.vcpu_mask = ~0ul;
- /*
- * A false return means that another vcpu is currently trying
- * a similar operation, so back off.
- */
- if ( !hvm_flush_vcpu_tlb(need_flush, &input_params.vcpu_mask) )
- return HVM_HCALL_preempted;
+ hvm_flush_vcpu_tlb(need_flush, &input_params.vcpu_mask);
output.rep_complete = input.rep_count;
diff --git a/xen/include/asm-x86/hvm/hvm.h b/xen/include/asm-x86/hvm/hvm.h
index 09793c12e9..1f70ee0823 100644
--- a/xen/include/asm-x86/hvm/hvm.h
+++ b/xen/include/asm-x86/hvm/hvm.h
@@ -333,7 +333,7 @@ const char *hvm_efer_valid(const struct vcpu *v, uint64_t value,
signed int cr0_pg);
unsigned long hvm_cr4_guest_valid_bits(const struct domain *d, bool restore);
-bool hvm_flush_vcpu_tlb(bool (*flush_vcpu)(void *ctxt, struct vcpu *v),
+void hvm_flush_vcpu_tlb(bool (*flush_vcpu)(void *ctxt, struct vcpu *v),
void *ctxt);
#ifdef CONFIG_HVM
--
2.24.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 6+ messages in thread* [Xen-devel] [PATCH v2 3/3] x86/tlb: use Xen L0 assisted TLB flush when available
2020-01-10 16:04 [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode Roger Pau Monne
2020-01-10 16:04 ` [Xen-devel] [PATCH v2 1/3] x86/hvm: allow ASID flush when v != current Roger Pau Monne
2020-01-10 16:04 ` [Xen-devel] [PATCH v2 2/3] x86/hvm: rework HVMOP_flush_tlbs Roger Pau Monne
@ 2020-01-10 16:04 ` Roger Pau Monne
2020-01-10 16:08 ` [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode Jan Beulich
3 siblings, 0 replies; 6+ messages in thread
From: Roger Pau Monne @ 2020-01-10 16:04 UTC (permalink / raw)
To: xen-devel; +Cc: Andrew Cooper, Wei Liu, Jan Beulich, Roger Pau Monne
Use Xen's L0 HVMOP_flush_tlbs hypercall in order to perform flushes.
This greatly increases the performance of TLB flushes when running
with a high amount of vCPUs as a Xen guest, and is specially important
when running in shim mode.
The following figures are from a PV guest running `make -j32 xen` in
shim mode with 32 vCPUs.
Using x2APIC and ALLBUT shorthand:
real 4m35.973s
user 4m35.110s
sys 36m24.117s
Using L0 assisted flush:
real 1m14.206s
user 4m31.835s
sys 5m45.013s
The implementation adds a new hook to hypervisor_ops so other
enlightenments can also implement such assisted flush just by filling
the hook. Note that the Xen implementation completely ignores the
dirty CPU mask and the linear address passed in, and always performs a
global TLB flush on all vCPUs.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v1:
- Add a L0 assisted hook to hypervisor ops.
---
xen/arch/x86/guest/hypervisor.c | 9 +++++++++
xen/arch/x86/guest/xen/xen.c | 6 ++++++
xen/arch/x86/smp.c | 6 ++++++
xen/include/asm-x86/guest/hypervisor.h | 13 +++++++++++++
4 files changed, 34 insertions(+)
diff --git a/xen/arch/x86/guest/hypervisor.c b/xen/arch/x86/guest/hypervisor.c
index 4f27b98740..c793ba51c2 100644
--- a/xen/arch/x86/guest/hypervisor.c
+++ b/xen/arch/x86/guest/hypervisor.c
@@ -18,6 +18,7 @@
*
* Copyright (c) 2019 Microsoft.
*/
+#include <xen/cpumask.h>
#include <xen/init.h>
#include <xen/types.h>
@@ -64,6 +65,14 @@ void hypervisor_resume(void)
ops->resume();
}
+int hypervisor_flush_tlb(const cpumask_t *mask, const void *va)
+{
+ if ( ops && ops->flush_tlb )
+ return ops->flush_tlb(mask, va);
+
+ return -ENOSYS;
+}
+
/*
* Local variables:
* mode: C
diff --git a/xen/arch/x86/guest/xen/xen.c b/xen/arch/x86/guest/xen/xen.c
index 6dbc5f953f..47af773aca 100644
--- a/xen/arch/x86/guest/xen/xen.c
+++ b/xen/arch/x86/guest/xen/xen.c
@@ -310,11 +310,17 @@ static void resume(void)
pv_console_init();
}
+static int flush_tlb(const cpumask_t *mask, const void *va)
+{
+ return xen_hypercall_hvm_op(HVMOP_flush_tlbs, NULL);
+}
+
static const struct hypervisor_ops ops = {
.name = "Xen",
.setup = setup,
.ap_setup = ap_setup,
.resume = resume,
+ .flush_tlb = flush_tlb,
};
const struct hypervisor_ops *__init xg_probe(void)
diff --git a/xen/arch/x86/smp.c b/xen/arch/x86/smp.c
index 6510dd84ab..b3cb1ba90f 100644
--- a/xen/arch/x86/smp.c
+++ b/xen/arch/x86/smp.c
@@ -15,6 +15,7 @@
#include <xen/perfc.h>
#include <xen/spinlock.h>
#include <asm/current.h>
+#include <asm/guest.h>
#include <asm/smp.h>
#include <asm/mc146818rtc.h>
#include <asm/flushtlb.h>
@@ -262,6 +263,11 @@ void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags)
if ( (flags & ~FLUSH_ORDER_MASK) &&
!cpumask_subset(mask, cpumask_of(cpu)) )
{
+ if ( cpu_has_hypervisor &&
+ !(flags & ~(FLUSH_TLB | FLUSH_TLB_GLOBAL | FLUSH_VA_VALID)) &&
+ !hypervisor_flush_tlb(mask, va) )
+ return;
+
spin_lock(&flush_lock);
cpumask_and(&flush_cpumask, mask, &cpu_online_map);
cpumask_clear_cpu(cpu, &flush_cpumask);
diff --git a/xen/include/asm-x86/guest/hypervisor.h b/xen/include/asm-x86/guest/hypervisor.h
index 392f4b90ae..b7f1969796 100644
--- a/xen/include/asm-x86/guest/hypervisor.h
+++ b/xen/include/asm-x86/guest/hypervisor.h
@@ -28,6 +28,8 @@ struct hypervisor_ops {
void (*ap_setup)(void);
/* Resume from suspension */
void (*resume)(void);
+ /* L0 assisted TLB flush */
+ int (*flush_tlb)(const cpumask_t *mask, const void *va);
};
#ifdef CONFIG_GUEST
@@ -36,9 +38,16 @@ const char *hypervisor_probe(void);
void hypervisor_setup(void);
void hypervisor_ap_setup(void);
void hypervisor_resume(void);
+/*
+ * L0 assisted TLB flush.
+ * mask: cpumask of the dirty vCPUs that should be flushed.
+ * va: linear address to flush, or NULL for global flushes.
+ */
+int hypervisor_flush_tlb(const cpumask_t *mask, const void *va);
#else
+#include <xen/cpumask.h>
#include <xen/lib.h>
#include <xen/types.h>
@@ -46,6 +55,10 @@ static inline const char *hypervisor_probe(void) { return NULL; }
static inline void hypervisor_setup(void) { ASSERT_UNREACHABLE(); }
static inline void hypervisor_ap_setup(void) { ASSERT_UNREACHABLE(); }
static inline void hypervisor_resume(void) { ASSERT_UNREACHABLE(); }
+static inline int hypervisor_flush_tlb(const cpumask_t *mask, const void *va)
+{
+ return -ENOSYS;
+}
#endif /* CONFIG_GUEST */
--
2.24.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode
2020-01-10 16:04 [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode Roger Pau Monne
` (2 preceding siblings ...)
2020-01-10 16:04 ` [Xen-devel] [PATCH v2 3/3] x86/tlb: use Xen L0 assisted TLB flush when available Roger Pau Monne
@ 2020-01-10 16:08 ` Jan Beulich
2020-01-10 16:14 ` Roger Pau Monné
3 siblings, 1 reply; 6+ messages in thread
From: Jan Beulich @ 2020-01-10 16:08 UTC (permalink / raw)
To: Roger Pau Monne; +Cc: xen-devel, Paul Durrant, Wei Liu, Andrew Cooper
On 10.01.2020 17:04, Roger Pau Monne wrote:
> Patch #2 is likely the most controversial one, as it changes the
> implementation of assisted TLB flushes. I have to admit I haven't been
> able to figure out why HVM guest context flushes issued a
> flush_tlb_mask, and the commit introducing such behavior doesn't contain
> a helpful commit message.
A shadow mode thing, maybe?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode
2020-01-10 16:08 ` [Xen-devel] [PATCH v2 0/3] x86: improve assisted tlb flush and use it in guest mode Jan Beulich
@ 2020-01-10 16:14 ` Roger Pau Monné
0 siblings, 0 replies; 6+ messages in thread
From: Roger Pau Monné @ 2020-01-10 16:14 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel, Paul Durrant, Wei Liu, Andrew Cooper
On Fri, Jan 10, 2020 at 05:08:16PM +0100, Jan Beulich wrote:
> On 10.01.2020 17:04, Roger Pau Monne wrote:
> > Patch #2 is likely the most controversial one, as it changes the
> > implementation of assisted TLB flushes. I have to admit I haven't been
> > able to figure out why HVM guest context flushes issued a
> > flush_tlb_mask, and the commit introducing such behavior doesn't contain
> > a helpful commit message.
>
> A shadow mode thing, maybe?
Hm, I could be wrong, but that flush doesn't seem to make sense for
shadow mode either.
If VPID/ASID is used, ticking it will drop all the guest caches, and
if VPID/ASID not used a vmexit/vmentry will clear the cache.
According to my reading of the Intel SDM this applies regardless of
whether HAP (EPT) is used or not.
The flush done by flush_tlb_mask is in root mode, and hence doesn't
affect the guest (non-root) caches when SVM/VTx is used.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread