[PATCH 00/18] Provide a command line option to choose how to handle SErrors

[PATCH 00/18] Provide a command line option to choose how to handle SErrors

[Wei Chen]

From XSA-201 (see [1]), we know that, a guest could trigger SErrors when
accessing memory mapped HW in a non-conventional way. In the patches for
XSA-201, we crash the guest when we captured such asynchronous aborts to
avoid data corruption.

In order to distinguish guest-generated SErrors from hypervisor-generated
SErrors. We have to place SError checking code in every EL1 -> EL2 paths.
That will be an overhead on entries caused by dsb/isb.

But not all platforms want to categorize the SErrors. For example, a host
that is running with trusted guests. The administrator can confirm that
all guests that are running on the host will not trigger such SErrors. In
this user scene, we should provide some options to administrator to avoid
categorizing the SErrors. And then reduce the overhead of dsb/isb.

We provided following 3 options to administrator to determine how to handle
the SErrors:

* `diverse`:
  The hypervisor will distinguish guest SErrors from hypervisor SErrors.
  The guest generated SErrors will be forwarded to guests, the hypervisor
  generated SErrors will cause the whole system crash.
  It requires:
  1. Place dsb/isb on all EL1 -> EL2 trap entries to categorize SErrors
     correctly.
  2. Place dsb/isb on EL2 -> EL1 return paths to prevent slipping hypervisor
     SErrors to guests.
  3. Place dsb/isb in context switch to isolate the SErrors between 2 vCPUs.

* `forward`:
  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
  All SErrors will be forwarded to guests, except the SErrors generated when
  idle vCPU is running. The idle domain doesn't have the ability to hanle the
  SErrors, so we have to crash the whole system when we get SErros with idle
  vCPU. This option will avoid most overhead of the dsb/isb, except the dsb/isb
  in context switch which is used to isolate the SErrors between 2 vCPUs.

* `panic`:
  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
  All SErrors will crash the whole system. This option will avoid all overhead
  of the dsb/isb.


Wei Chen (18):
  xen/arm: Introduce a helper to get default HCR_EL2 flags
  xen/arm: Restore HCR_EL2 register
  xen/arm: Avoid setting/clearing HCR_RW at every context switch
  xen/arm: Save HCR_EL2 when a guest took the SError
  xen/arm: Save ESR_EL2 to avoid using mismatched value in syndrome
    check
  xen/arm: Introduce a virtual abort injection helper
  xen/arm: Introduce a command line parameter for SErrors/Aborts
  xen/arm: Introduce a initcall to update cpu_hwcaps by serror_op
  xen/arm64: Use alternative to skip the check of pending serrors
  xen/arm32: Use cpu_hwcaps to skip the check of pending serrors
  xen/arm: Move macro VABORT_GEN_BY_GUEST to common header
  xen/arm: Introduce new helpers to handle guest/hyp SErrors
  xen/arm: Replace do_trap_guest_serror with new helpers
  xen/arm: Unmask the Abort/SError bit in the exception entries
  xen/arm: Introduce a helper to synchronize SError
  xen/arm: Isolate the SError between the context switch of 2 vCPUs
  xen/arm: Prevent slipping hypervisor SError to guest
  xen/arm: Handle guest external abort as guest SError

 docs/misc/xen-command-line.markdown   |  44 ++++++++
 xen/arch/arm/arm32/asm-offsets.c      |   1 +
 xen/arch/arm/arm32/entry.S            |  37 ++++++-
 xen/arch/arm/arm32/traps.c            |   5 +-
 xen/arch/arm/arm64/asm-offsets.c      |   1 +
 xen/arch/arm/arm64/domctl.c           |   6 +
 xen/arch/arm/arm64/entry.S            | 105 ++++++++----------
 xen/arch/arm/domain.c                 |   9 ++
 xen/arch/arm/domain_build.c           |   7 ++
 xen/arch/arm/p2m.c                    |  16 ++-
 xen/arch/arm/traps.c                  | 200 ++++++++++++++++++++++++++++++----
 xen/include/asm-arm/arm32/processor.h |  12 +-
 xen/include/asm-arm/arm64/processor.h |  10 +-
 xen/include/asm-arm/cpufeature.h      |   3 +-
 xen/include/asm-arm/domain.h          |   4 +
 xen/include/asm-arm/processor.h       |  19 +++-
 16 files changed, 370 insertions(+), 109 deletions(-)

-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 01/18] xen/arm: Introduce a helper to get default HCR_EL2 flags

[Wei Chen]

We want to add HCR_EL2 register to Xen context switch. And each copy
of HCR_EL2 in vcpu structure will be initialized with the same set
of trap flags as the HCR_EL2 register. We introduce a helper here to
represent these flags to be reused easily.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c            | 11 ++++++++---
 xen/include/asm-arm/processor.h |  2 ++
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 614501f..d343c66 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -115,6 +115,13 @@ static void __init parse_vwfi(const char *s)
 }
 custom_param("vwfi", parse_vwfi);
 
+register_t get_default_hcr_flags(void)
+{
+    return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
+             (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
+             HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB);
+}
+
 void init_traps(void)
 {
     /* Setup Hyp vector base */
@@ -139,9 +146,7 @@ void init_traps(void)
                  CPTR_EL2);
 
     /* Setup hypervisor traps */
-    WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
-                 (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
-                 HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB,HCR_EL2);
+    WRITE_SYSREG(get_default_hcr_flags(), HCR_EL2);
     isb();
 }
 
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index afc0e9a..4b6338b 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -708,6 +708,8 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
 
 void do_trap_guest_error(struct cpu_user_regs *regs);
 
+register_t get_default_hcr_flags(void);
+
 #endif /* __ASSEMBLY__ */
 #endif /* __ASM_ARM_PROCESSOR_H */
 /*
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Wei Chen]

Different domains may have different HCR_EL2 flags. For example, the
64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
we give each domain a default HCR_EL2 value and save it in the VCPU's
context.

HCR_EL2 register has only one bit can be updated automatically without
explicit write (HCR_VSE). But we haven't used this bit currently, so
we can consider that the HCR_EL2 register will not be modified while
the guest is running. So save the HCR_EL2 while guest exiting to
hypervisor is not neccessary. We just have to restore this register for
each VCPU while leaving hypervisor.

We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
ctxt_switch_to. Because the leave_hypervisor_tail is the closest place
to the exception return. In this case, we don't need to warrant the
HCR_EL2 will not be changed between ctxt_switch_to and exception return.

Even though we have restored HCR_EL2 in leave_hypervisor_tail, we still
have to keep the write to HCR_EL2 in p2m_restore_state. That because
p2m_restore_state could be used to switch between two p2m and possibly
to do address translation using hardware. For instance when building
the hardware domain, we are using the instruction to before copying
data. During the translation, some bits of base register (such as SCTLR
and HCR) could be cached in TLB and used for the translation.

We had some issues in the past (see commit b3cbe129d "xen: arm: Ensure
HCR_EL2.RW is set correctly when building dom0"), so we should probably
keep the write to HCR_EL2 in p2m_restore_state.

Signed-off-by: wei chen <Wei.Chen@arm.com>
---
 xen/arch/arm/domain.c        |  2 ++
 xen/arch/arm/p2m.c           | 15 +++++++++------
 xen/arch/arm/traps.c         |  1 +
 xen/include/asm-arm/domain.h |  3 +++
 4 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index bb327da..5d18bb0 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -513,6 +513,8 @@ int vcpu_initialise(struct vcpu *v)
 
     v->arch.actlr = READ_SYSREG32(ACTLR_EL1);
 
+    v->arch.hcr_el2 = get_default_hcr_flags();
+
     processor_vcpu_initialise(v);
 
     if ( (rc = vcpu_vgic_init(v)) != 0 )
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 1fc6ca3..c49bfa6 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -128,26 +128,29 @@ void p2m_save_state(struct vcpu *p)
 
 void p2m_restore_state(struct vcpu *n)
 {
-    register_t hcr;
     struct p2m_domain *p2m = &n->domain->arch.p2m;
 
     if ( is_idle_vcpu(n) )
         return;
 
-    hcr = READ_SYSREG(HCR_EL2);
-
     WRITE_SYSREG64(p2m->vttbr, VTTBR_EL2);
     isb();
 
     if ( is_32bit_domain(n->domain) )
-        hcr &= ~HCR_RW;
+        n->arch.hcr_el2 &= ~HCR_RW;
     else
-        hcr |= HCR_RW;
+        n->arch.hcr_el2 |= HCR_RW;
 
     WRITE_SYSREG(n->arch.sctlr, SCTLR_EL1);
     isb();
 
-    WRITE_SYSREG(hcr, HCR_EL2);
+    /*
+     * p2m_restore_state could be used to switch between two p2m and possibly
+     * to do address translation using hardware. And these operations may
+     * happen during the interval between enter/leave hypervior, so we should
+     * probably keep the write to HCR_EL2 here.
+     */
+    WRITE_SYSREG(n->arch.hcr_el2, HCR_EL2);
     isb();
 }
 
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index d343c66..9792d02 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2811,6 +2811,7 @@ asmlinkage void leave_hypervisor_tail(void)
         local_irq_disable();
         if (!softirq_pending(smp_processor_id())) {
             gic_inject();
+            WRITE_SYSREG(current->arch.hcr_el2, HCR_EL2);
             return;
         }
         local_irq_enable();
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 09fe502..7b1dacc 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -204,6 +204,9 @@ struct arch_vcpu
     register_t tpidr_el1;
     register_t tpidrro_el0;
 
+    /* HYP configuration */
+    register_t hcr_el2;
+
     uint32_t teecr, teehbr; /* ThumbEE, 32-bit guests only */
 #ifdef CONFIG_ARM_32
     /*
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Wei Chen]

The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
DomU in vcpu_initialise, we didn't know the domain's address size
information. We had to use compatible flags to initialize HCR_EL2,
and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
at every context switch.

But, after we added the HCR_EL2 to vcpu's context, this behaviour
seems a little fussy. We can update the HCR_RW bit in vcpu's context
as soon as we get the domain's address size to avoid setting/clearing
HCR_RW at every context switch.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/arm64/domctl.c  | 6 ++++++
 xen/arch/arm/domain.c        | 5 +++++
 xen/arch/arm/domain_build.c  | 7 +++++++
 xen/arch/arm/p2m.c           | 5 -----
 xen/include/asm-arm/domain.h | 1 +
 5 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
index 44e1e7b..ab8781f 100644
--- a/xen/arch/arm/arm64/domctl.c
+++ b/xen/arch/arm/arm64/domctl.c
@@ -14,6 +14,8 @@
 
 static long switch_mode(struct domain *d, enum domain_type type)
 {
+    struct vcpu *v;
+
     if ( d == NULL )
         return -EINVAL;
     if ( d->tot_pages != 0 )
@@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum domain_type type)
 
     d->arch.type = type;
 
+    if ( is_64bit_domain(d) )
+        for_each_vcpu(d, v)
+            vcpu_switch_to_aarch64_mode(v);
+
     return 0;
 }
 
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 5d18bb0..69c2854 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
     free_xenheap_pages(v->arch.stack, STACK_ORDER);
 }
 
+void vcpu_switch_to_aarch64_mode(struct vcpu *v)
+{
+    v->arch.hcr_el2 |= HCR_RW;
+}
+
 int arch_domain_create(struct domain *d, unsigned int domcr_flags,
                        struct xen_arch_domainconfig *config)
 {
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index de59e5f..3abacc0 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2148,6 +2148,10 @@ int construct_dom0(struct domain *d)
         return -EINVAL;
     }
     d->arch.type = kinfo.type;
+
+    if ( is_64bit_domain(d) )
+        vcpu_switch_to_aarch64_mode(v);
+
 #endif
 
     allocate_memory(d, &kinfo);
@@ -2240,6 +2244,9 @@ int construct_dom0(struct domain *d)
             printk("Failed to allocate dom0 vcpu %d on pcpu %d\n", i, cpu);
             break;
         }
+
+        if ( is_64bit_domain(d) )
+            vcpu_switch_to_aarch64_mode(d->vcpu[i]);
     }
 
     return 0;
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index c49bfa6..1cba0d0 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -136,11 +136,6 @@ void p2m_restore_state(struct vcpu *n)
     WRITE_SYSREG64(p2m->vttbr, VTTBR_EL2);
     isb();
 
-    if ( is_32bit_domain(n->domain) )
-        n->arch.hcr_el2 &= ~HCR_RW;
-    else
-        n->arch.hcr_el2 |= HCR_RW;
-
     WRITE_SYSREG(n->arch.sctlr, SCTLR_EL1);
     isb();
 
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 7b1dacc..68185e2 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -268,6 +268,7 @@ struct arch_vcpu
 
 void vcpu_show_execution_state(struct vcpu *);
 void vcpu_show_registers(const struct vcpu *);
+void vcpu_switch_to_aarch64_mode(struct vcpu *);
 
 unsigned int domain_max_vcpus(const struct domain *);
 
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 04/18] xen/arm: Save HCR_EL2 when a guest took the SError

[Wei Chen]

The HCR_EL2.VSE (HCR.VA for aarch32) bit can be used to generate a
virtual abort to guest. The HCR_EL2.VSE bit has a peculiar feature
of getting cleared when the guest has taken the abort (this is the
only bit that behaves as such in HCR_EL2 register).

This means that if we set the HCR_EL2.VSE bit to signal such an abort,
we must preserve it in the guest context until it disappears from
HCR_EL2, and at which point it must be cleared from the context. This
is achieved by reading back from HCR_EL2 until the guest takes the
fault.

If we preserved a pending VSE in guest context, we have to restore
it to HCR_EL2 when context switch to this guest. This is achieved
by writing saved HCR_EL2 value in guest context back to HCR_EL2
register before return to guest. This had been done by the patch
of "Restore HCR_EL2 register".

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 9792d02..476e2be 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2641,7 +2641,18 @@ static void do_trap_smc(struct cpu_user_regs *regs, const union hsr hsr)
 static void enter_hypervisor_head(struct cpu_user_regs *regs)
 {
     if ( guest_mode(regs) )
+    {
+        /*
+         * If we pended a virtual abort, preserve it until it gets cleared.
+         * See ARM ARM DDI 0487A.j D1.14.3 (Virtual Interrupts) for details,
+         * but the crucial bit is "On taking a vSError interrupt, HCR_EL2.VSE
+         * (alias of HCR.VA) is cleared to 0."
+         */
+        if ( current->arch.hcr_el2 & HCR_VA )
+            current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
+
         gic_clear_lrs(current);
+    }
 }
 
 asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 05/18] xen/arm: Save ESR_EL2 to avoid using mismatched value in syndrome check

[Wei Chen]

Xen will do exception syndrome check while some types of exception
take place in EL2. The syndrome check code read the ESR_EL2 register
directly, but in some situation this register maybe overridden by
nested exception.

For example, if we re-enable IRQ before reading ESR_EL2 which means
Xen will enter in IRQ exception mode and return the processor with
clobbered ESR_EL2 (See ARM ARM DDI 0487A.j D7.2.25)

In this case the guest exception syndrome has been overridden, we will
check the syndrome for guest sync exception with a mismatched ESR_EL2
value. So we want to save ESR_EL2 to cpu_user_regs as soon as the
exception takes place in EL2 to avoid using a mismatched syndrome value.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/arm32/asm-offsets.c      |  1 +
 xen/arch/arm/arm32/entry.S            |  3 +++
 xen/arch/arm/arm64/asm-offsets.c      |  1 +
 xen/arch/arm/arm64/entry.S            | 13 +++++++++----
 xen/arch/arm/traps.c                  |  2 +-
 xen/include/asm-arm/arm32/processor.h |  2 +-
 xen/include/asm-arm/arm64/processor.h | 10 ++++++++--
 7 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/xen/arch/arm/arm32/asm-offsets.c b/xen/arch/arm/arm32/asm-offsets.c
index f8e6b53..5b543ab 100644
--- a/xen/arch/arm/arm32/asm-offsets.c
+++ b/xen/arch/arm/arm32/asm-offsets.c
@@ -26,6 +26,7 @@ void __dummy__(void)
    OFFSET(UREGS_lr, struct cpu_user_regs, lr);
    OFFSET(UREGS_pc, struct cpu_user_regs, pc);
    OFFSET(UREGS_cpsr, struct cpu_user_regs, cpsr);
+   OFFSET(UREGS_hsr, struct cpu_user_regs, hsr);
 
    OFFSET(UREGS_LR_usr, struct cpu_user_regs, lr_usr);
    OFFSET(UREGS_SP_usr, struct cpu_user_regs, sp_usr);
diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
index 2a6f4f0..2187226 100644
--- a/xen/arch/arm/arm32/entry.S
+++ b/xen/arch/arm/arm32/entry.S
@@ -23,6 +23,9 @@
         add r11, sp, #UREGS_kernel_sizeof+4;                            \
         str r11, [sp, #UREGS_sp];                                       \
                                                                         \
+        mrc CP32(r11, HSR);             /* Save exception syndrome */   \
+        str r11, [sp, #UREGS_hsr];                                      \
+                                                                        \
         mrs r11, SPSR_hyp;                                              \
         str r11, [sp, #UREGS_cpsr];                                     \
         and r11, #PSR_MODE_MASK;                                        \
diff --git a/xen/arch/arm/arm64/asm-offsets.c b/xen/arch/arm/arm64/asm-offsets.c
index 69ea92a..ce24e44 100644
--- a/xen/arch/arm/arm64/asm-offsets.c
+++ b/xen/arch/arm/arm64/asm-offsets.c
@@ -27,6 +27,7 @@ void __dummy__(void)
    OFFSET(UREGS_SP, struct cpu_user_regs, sp);
    OFFSET(UREGS_PC, struct cpu_user_regs, pc);
    OFFSET(UREGS_CPSR, struct cpu_user_regs, cpsr);
+   OFFSET(UREGS_ESR_el2, struct cpu_user_regs, hsr);
 
    OFFSET(UREGS_SPSR_el1, struct cpu_user_regs, spsr_el1);
 
diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index c181b5e..02802c0 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -121,9 +121,13 @@ lr      .req    x30             // link register
 
         stp     lr, x21, [sp, #UREGS_LR]
 
-        mrs     x22, elr_el2
-        mrs     x23, spsr_el2
-        stp     x22, x23, [sp, #UREGS_PC]
+        mrs     x21, elr_el2
+        str     x21, [sp, #UREGS_PC]
+
+        add     x21, sp, #UREGS_CPSR
+        mrs     x22, spsr_el2
+        mrs     x23, esr_el2
+        stp     w22, w23, [x21]
 
         .endm
 
@@ -307,7 +311,8 @@ ENTRY(return_to_new_vcpu64)
 return_from_trap:
         msr     daifset, #2 /* Mask interrupts */
 
-        ldp     x21, x22, [sp, #UREGS_PC]       // load ELR, SPSR
+        ldr     x21, [sp, #UREGS_PC]            // load ELR
+        ldr     w22, [sp, #UREGS_CPSR]          // load SPSR
 
         pop     x0, x1
         pop     x2, x3
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 476e2be..c11359d 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2657,7 +2657,7 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
 
 asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
 {
-    const union hsr hsr = { .bits = READ_SYSREG32(ESR_EL2) };
+    const union hsr hsr = { .bits = regs->hsr };
 
     enter_hypervisor_head(regs);
 
diff --git a/xen/include/asm-arm/arm32/processor.h b/xen/include/asm-arm/arm32/processor.h
index db3b17b..f6d5df3 100644
--- a/xen/include/asm-arm/arm32/processor.h
+++ b/xen/include/asm-arm/arm32/processor.h
@@ -37,7 +37,7 @@ struct cpu_user_regs
         uint32_t pc, pc32;
     };
     uint32_t cpsr; /* Return mode */
-    uint32_t pad0; /* Doubleword-align the kernel half of the frame */
+    uint32_t hsr;  /* Exception Syndrome */
 
     /* Outer guest frame only from here on... */
 
diff --git a/xen/include/asm-arm/arm64/processor.h b/xen/include/asm-arm/arm64/processor.h
index b0726ff..d381428 100644
--- a/xen/include/asm-arm/arm64/processor.h
+++ b/xen/include/asm-arm/arm64/processor.h
@@ -65,9 +65,15 @@ struct cpu_user_regs
 
     /* Return address and mode */
     __DECL_REG(pc,           pc32);             /* ELR_EL2 */
+    /*
+     * Be careful for 32-bit registers, if we use xN to save 32-bit register
+     * to stack, its next field on stack will be overridden.
+     * For example, if we use xN to save SPSR_EL2 to stack will override the
+     * hsr field on stack.
+     * So, it's better to use wN to save 32-bit registers to stack.
+     */
     uint32_t cpsr;                              /* SPSR_EL2 */
-
-    uint32_t pad0; /* Align end of kernel frame. */
+    uint32_t hsr;                               /* ESR_EL2 */
 
     /* Outer guest frame only from here on... */
 
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 06/18] xen/arm: Introduce a virtual abort injection helper

[Wei Chen]

When guest triggers async aborts, in most platform, such aborts
will be routed to hypervisor. But we don't want the hypervisor
to handle such aborts, so we have to route such aborts back to
the guest.

This helper is using the HCR_EL2.VSE (HCR.VA for aarch32) bit to
route such aborts back to the guest. If the guest PC had been
advanced by SVC/HVC/SMC instructions before we caught the SError
in hypervisor, we have to adjust the guest PC to exact address
while the SError generated.

About HSR_EC_SVC32/64, even thought we don't trap SVC32/64 today,
we would like them to be handled here. This would be useful when
VM introspection will gain support of SVC32/64 trapping.

This helper will be used by the later patches in this series, we
use #if 0 to disable it in this patch temporarily to remove the
warning message of unused function from compiler.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c            | 32 ++++++++++++++++++++++++++++++++
 xen/include/asm-arm/processor.h |  1 +
 2 files changed, 33 insertions(+)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index c11359d..e425832 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -618,6 +618,38 @@ static void inject_dabt_exception(struct cpu_user_regs *regs,
 #endif
 }
 
+#if 0
+/* Inject a virtual Abort/SError into the guest. */
+static void inject_vabt_exception(struct cpu_user_regs *regs)
+{
+    const union hsr hsr = { .bits = regs->hsr };
+
+    /*
+     * SVC/HVC/SMC already have an adjusted PC (See ARM ARM DDI 0487A.j
+     * D1.10.1 for more details), which we need to correct in order to
+     * return to after having injected the SError.
+     */
+    switch ( hsr.ec )
+    {
+    case HSR_EC_SVC32:
+    case HSR_EC_HVC32:
+    case HSR_EC_SMC32:
+#ifdef CONFIG_ARM_64
+    case HSR_EC_SVC64:
+    case HSR_EC_HVC64:
+    case HSR_EC_SMC64:
+#endif
+        regs->pc -= hsr.len ? 4 : 2;
+        break;
+
+    default:
+        break;
+    }
+
+    current->arch.hcr_el2 |= HCR_VA;
+}
+#endif
+
 struct reg_ctxt {
     /* Guest-side state */
     uint32_t sctlr_el1;
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index 4b6338b..d7b0711 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -252,6 +252,7 @@
 #define HSR_EC_HVC32                0x12
 #define HSR_EC_SMC32                0x13
 #ifdef CONFIG_ARM_64
+#define HSR_EC_SVC64                0x15
 #define HSR_EC_HVC64                0x16
 #define HSR_EC_SMC64                0x17
 #define HSR_EC_SYSREG               0x18
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 07/18] xen/arm: Introduce a command line parameter for SErrors/Aborts

[Wei Chen]

In order to distinguish guest-generated SErrors from hypervisor-generated
SErrors. We have to place SError checking code in every EL1 -> EL2 paths.
That will be an overhead on entries caused by dsb/isb.

But not all platforms want to categorize the SErrors. For example, a host
that is running with trusted guests. The administrator can confirm that
all guests that are running on the host will not trigger such SErrors. In
this user scene, we should provide some options to administrator to avoid
categorizing the SErrors. And then reduce the overhead of dsb/isb.

We provided following 3 options to administrator to determine how to handle
the SErrors:

* `diverse`:
  The hypervisor will distinguish guest SErrors from hypervisor SErrors.
  The guest generated SErrors will be forwarded to guests, the hypervisor
  generated SErrors will cause the whole system crash.
  It requires:
  1. Place dsb/isb on all EL1 -> EL2 trap entries to categorize SErrors
     correctly.
  2. Place dsb/isb on EL2 -> EL1 return paths to prevent slipping hypervisor
     SErrors to guests.
  3. Place dsb/isb in context switch to isolate the SErrors between 2 vCPUs.

* `forward`:
  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
  All SErrors will be forwarded to guests, except the SErrors generated when
  idle vCPU is running. The idle domain doesn't have the ability to hanle the
  SErrors, so we have to crash the whole system when we get SErros with idle
  vCPU. This option will avoid most overhead of the dsb/isb, except the dsb/isb
  in context switch which is used to isolate the SErrors between 2 vCPUs.

* `panic`:
  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
  All SErrors will crash the whole system. This option will avoid all overhead
  of the dsb/isb.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>

---
About adding dsb/isb to prevent slipping Hypervisor SErrors to Guests if the
selected option is "diverse". Some Hypervisor SErrors could not be avoid by
software, for example ECC Error. But I don't know whether it's worth adding
the overhead by default.
---
 docs/misc/xen-command-line.markdown | 44 +++++++++++++++++++++++++++++++++++++
 xen/arch/arm/traps.c                | 19 ++++++++++++++++
 2 files changed, 63 insertions(+)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 4daf5b5..79554ce 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1481,6 +1481,50 @@ enabling more sockets and cores to go into deeper sleep states.
 
 Set the serial transmit buffer size.
 
+### serrors (ARM)
+> `= diverse | forward | panic`
+
+> Default: `diverse`
+
+This parameter is provided to administrator to determine how to handle the
+SErrors.
+
+In order to distinguish guest-generated SErrors from hypervisor-generated
+SErrors. We have to place SError checking code in every EL1 -> EL2 paths.
+That will be an overhead on entries caused by dsb/isb. But not all platforms
+need to categorize the SErrors. For example, a host that is running with
+trusted guests. The administrator can confirm that all guests that are
+running on the host will not trigger such SErrors. In this case, the
+administrator can use this parameter to skip categorizing the SErrors. And
+reduce the overhead of dsb/isb.
+
+We provided following 3 options to administrator to determine how to handle
+the SErrors:
+
+* `diverse`:
+  The hypervisor will distinguish guest SErrors from hypervisor SErrors.
+  The guest generated SErrors will be forwarded to guests, the hypervisor
+  generated SErrors will cause the whole system crash.
+  It requires:
+  1. Place dsb/isb on all EL1 -> EL2 trap entries to categorize SErrors
+     correctly.
+  2. Place dsb/isb on EL2 -> EL1 return paths to prevent slipping hypervisor
+     SErrors to guests.
+  3. Place dsb/isb in context switch to isolate the SErrors between 2 vCPUs.
+
+* `forward`:
+  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
+  All SErrors will be forwarded to guests, except the SErrors generated when
+  idle vCPU is running. The idle domain doesn't have the ability to hanle the
+  SErrors, so we have to crash the whole system when we get SErros with idle
+  vCPU. This option will avoid most overhead of the dsb/isb, except the dsb/isb
+  in context switch which is used to isolate the SErrors between 2 vCPUs.
+
+* `panic`:
+  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
+  All SErrors will crash the whole system. This option will avoid all overhead
+  of the dsb/isb.
+
 ### smap
 > `= <boolean> | hvm`
 
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index e425832..5e31699 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -115,6 +115,25 @@ static void __init parse_vwfi(const char *s)
 }
 custom_param("vwfi", parse_vwfi);
 
+static enum {
+    SERRORS_DIVERSE,
+    SERRORS_FORWARD,
+    SERRORS_PANIC,
+} serrors_op;
+
+static void __init parse_serrors_behavior(const char *str)
+{
+    if ( !strcmp(str, "forward") )
+        serrors_op = SERRORS_FORWARD;
+    else if ( !strcmp(str, "panic") )
+        serrors_op = SERRORS_PANIC;
+    else
+        serrors_op = SERRORS_DIVERSE;
+
+    return;
+}
+custom_param("serrors", parse_serrors_behavior);
+
 register_t get_default_hcr_flags(void)
 {
     return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 08/18] xen/arm: Introduce a initcall to update cpu_hwcaps by serror_op

[Wei Chen]

In the later patches of this series, we want to use the alternative
patching framework to avoid checking serror_op in every entries.
So we define a new cpu feature "SKIP_CHECK_PENDING_VSERROR" for
serror_op. When serror_op is not equal to SERROR_DIVERSE, this
feature will be set to cpu_hwcaps.

But we could not update cpu_hwcaps directly in the serror parameter
parsing function. Because if the serror parameter is not placed in
the command line, the parsing function would not be invoked.

So, we introduce this initcall to guarantee the cpu_hwcaps can be
updated no matter the serror parameter is placed in the command line
or not.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c             | 9 +++++++++
 xen/include/asm-arm/cpufeature.h | 3 ++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 5e31699..053b7fc 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -134,6 +134,15 @@ static void __init parse_serrors_behavior(const char *str)
 }
 custom_param("serrors", parse_serrors_behavior);
 
+static __init int update_serrors_cpu_caps(void)
+{
+    if ( serrors_op != SERRORS_DIVERSE )
+        cpus_set_cap(SKIP_CHECK_PENDING_VSERROR);
+
+    return 0;
+}
+__initcall(update_serrors_cpu_caps);
+
 register_t get_default_hcr_flags(void)
 {
     return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
diff --git a/xen/include/asm-arm/cpufeature.h b/xen/include/asm-arm/cpufeature.h
index c0a25ae..ec3f9e5 100644
--- a/xen/include/asm-arm/cpufeature.h
+++ b/xen/include/asm-arm/cpufeature.h
@@ -40,8 +40,9 @@
 #define ARM32_WORKAROUND_766422 2
 #define ARM64_WORKAROUND_834220 3
 #define LIVEPATCH_FEATURE   4
+#define SKIP_CHECK_PENDING_VSERROR 5
 
-#define ARM_NCAPS           5
+#define ARM_NCAPS           6
 
 #ifndef __ASSEMBLY__
 
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 09/18] xen/arm64: Use alternative to skip the check of pending serrors

[Wei Chen]

We have provided an option to administrator to determine how to
handle the SErrors. In order to skip the check of pending SError,
in conventional way, we have to read the option every time before
we try to check the pending SError. This will add overhead to check
the option at every trap.

The ARM64 supports the alternative patching feature. We can use an
ALTERNATIVE to avoid checking option at every trap. We added a new
cpufeature named "SKIP_CHECK_PENDING_VSERROR". This feature will be
enabled when the option is not diverse.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/arm64/entry.S | 41 +++++++++++++++++++++++++----------------
 1 file changed, 25 insertions(+), 16 deletions(-)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 02802c0..4baa3cb 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -1,5 +1,6 @@
 #include <asm/asm_defns.h>
 #include <asm/regs.h>
+#include <asm/alternative.h>
 #include <public/xen.h>
 
 /*
@@ -229,12 +230,14 @@ hyp_irq:
 
 guest_sync:
         entry   hyp=0, compat=0
-        bl      check_pending_vserror
         /*
-         * If x0 is Non-zero, a vSError took place, the initial exception
-         * doesn't have any significance to be handled. Exit ASAP
+         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
+         * not set. If a vSError took place, the initial exception will be
+         * skipped. Exit ASAP
          */
-        cbnz    x0, 1f
+        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
+                    "nop; nop",
+                    SKIP_CHECK_PENDING_VSERROR)
         msr     daifclr, #2
         mov     x0, sp
         bl      do_trap_hypervisor
@@ -243,12 +246,14 @@ guest_sync:
 
 guest_irq:
         entry   hyp=0, compat=0
-        bl      check_pending_vserror
         /*
-         * If x0 is Non-zero, a vSError took place, the initial exception
-         * doesn't have any significance to be handled. Exit ASAP
+         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
+         * not set. If a vSError took place, the initial exception will be
+         * skipped. Exit ASAP
          */
-        cbnz    x0, 1f
+        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
+                    "nop; nop",
+                    SKIP_CHECK_PENDING_VSERROR)
         mov     x0, sp
         bl      do_trap_irq
 1:
@@ -267,12 +272,14 @@ guest_error:
 
 guest_sync_compat:
         entry   hyp=0, compat=1
-        bl      check_pending_vserror
         /*
-         * If x0 is Non-zero, a vSError took place, the initial exception
-         * doesn't have any significance to be handled. Exit ASAP
+         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
+         * not set. If a vSError took place, the initial exception will be
+         * skipped. Exit ASAP
          */
-        cbnz    x0, 1f
+        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
+                    "nop; nop",
+                    SKIP_CHECK_PENDING_VSERROR)
         msr     daifclr, #2
         mov     x0, sp
         bl      do_trap_hypervisor
@@ -281,12 +288,14 @@ guest_sync_compat:
 
 guest_irq_compat:
         entry   hyp=0, compat=1
-        bl      check_pending_vserror
         /*
-         * If x0 is Non-zero, a vSError took place, the initial exception
-         * doesn't have any significance to be handled. Exit ASAP
+         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
+         * not set. If a vSError took place, the initial exception will be
+         * skipped. Exit ASAP
          */
-        cbnz    x0, 1f
+        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
+                    "nop; nop",
+                    SKIP_CHECK_PENDING_VSERROR)
         mov     x0, sp
         bl      do_trap_irq
 1:
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 10/18] xen/arm32: Use cpu_hwcaps to skip the check of pending serrors

[Wei Chen]

We have provided an option to administrator to determine how to
handle the SErrors. In order to skip the check of pending SError,
in conventional way, we have to read the option every time before
we try to check the pending SError.

Currently, we haven't export the option to other source file. But,
in the previous patch, we had set "SKIP_CHECK_PENDING_VSERROR" to
cpu_hwcaps when the option doesn't need to check the SErrors. So we
can use checking cpu_hwcaps to replace checking the option directly.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>

---
This is a temporary solution, this would have to be dropped as soon
as ARM32 gain support of alternative patching to avoid potential misusage.
The alternative patching support patches for ARM32 are still in review
stage.
---
 xen/arch/arm/arm32/entry.S | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
index 2187226..79929ca 100644
--- a/xen/arch/arm/arm32/entry.S
+++ b/xen/arch/arm/arm32/entry.S
@@ -1,5 +1,6 @@
 #include <asm/asm_defns.h>
 #include <asm/regs.h>
+#include <asm/cpufeature.h>
 #include <public/xen.h>
 
 #define SAVE_ONE_BANKED(reg)    mrs r11, reg; str r11, [sp, #UREGS_##reg]
@@ -11,6 +12,21 @@
 #define RESTORE_BANKED(mode) \
         RESTORE_ONE_BANKED(SP_##mode) ; RESTORE_ONE_BANKED(LR_##mode) ; RESTORE_ONE_BANKED(SPSR_##mode)
 
+/*
+ * If the SKIP_CHECK_PENDING_VSERROR has been set in the cpu feature,
+ * the checking of pending SErrors will be skipped.
+ *
+ * As it is a temporary solution, we are assuming that
+ * SKIP_CHECK_PENDING_VSERROR will always be in the first word for
+ * cpu_hwcaps. This would have to be dropped as soon as ARM32 gain
+ * support of alternative.
+ */
+#define SKIP_VSERROR_CHECK                      \
+        ldr r1, =cpu_hwcaps;                    \
+        ldr r1, [r1];                           \
+        tst r1, #SKIP_CHECK_PENDING_VSERROR;    \
+        moveq pc, lr
+
 #define SAVE_ALL                                                        \
         sub sp, #(UREGS_SP_usr - UREGS_sp); /* SP, LR, SPSR, PC */      \
         push {r0-r12}; /* Save R0-R12 */                                \
@@ -44,6 +60,9 @@ save_guest_regs:
         SAVE_BANKED(fiq)
         SAVE_ONE_BANKED(R8_fiq); SAVE_ONE_BANKED(R9_fiq); SAVE_ONE_BANKED(R10_fiq)
         SAVE_ONE_BANKED(R11_fiq); SAVE_ONE_BANKED(R12_fiq);
+
+        SKIP_VSERROR_CHECK
+
         /*
          * Start to check pending virtual abort in the gap of Guest -> HYP
          * world switch.
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 11/18] xen/arm: Move macro VABORT_GEN_BY_GUEST to common header

[Wei Chen]

We want to move part of SErrors checking code from hyp_error assembly code
to a function. This new function will use this macro to distinguish the
guest SErrors from hypervisor SErrors. So we have to move this macro to
common header.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/arm64/entry.S            |  2 ++
 xen/include/asm-arm/arm32/processor.h | 10 ----------
 xen/include/asm-arm/processor.h       | 10 ++++++++++
 3 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 4baa3cb..113e1c3 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -380,10 +380,12 @@ check_pending_vserror:
          * exception handler, and the elr_el2 will be set to
          * abort_guest_exit_start or abort_guest_exit_end.
          */
+        .global abort_guest_exit_start
 abort_guest_exit_start:
 
         isb
 
+        .global abort_guest_exit_end
 abort_guest_exit_end:
         /* Mask PSTATE asynchronous abort bit, close the checking window. */
         msr     daifset, #4
diff --git a/xen/include/asm-arm/arm32/processor.h b/xen/include/asm-arm/arm32/processor.h
index f6d5df3..68cc821 100644
--- a/xen/include/asm-arm/arm32/processor.h
+++ b/xen/include/asm-arm/arm32/processor.h
@@ -56,16 +56,6 @@ struct cpu_user_regs
     uint32_t pad1; /* Doubleword-align the user half of the frame */
 };
 
-/* Functions for pending virtual abort checking window. */
-void abort_guest_exit_start(void);
-void abort_guest_exit_end(void);
-
-#define VABORT_GEN_BY_GUEST(r)  \
-( \
-    ( (unsigned long)abort_guest_exit_start == (r)->pc ) || \
-    ( (unsigned long)abort_guest_exit_end == (r)->pc ) \
-)
-
 #endif
 
 /* Layout as used in assembly, with src/dest registers mixed in */
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index d7b0711..148cc6f 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -709,6 +709,16 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
 
 void do_trap_guest_error(struct cpu_user_regs *regs);
 
+/* Functions for pending virtual abort checking window. */
+void abort_guest_exit_start(void);
+void abort_guest_exit_end(void);
+
+#define VABORT_GEN_BY_GUEST(r)  \
+( \
+    ( (unsigned long)abort_guest_exit_start == (r)->pc ) || \
+    ( (unsigned long)abort_guest_exit_end == (r)->pc ) \
+)
+
 register_t get_default_hcr_flags(void);
 
 #endif /* __ASSEMBLY__ */
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 12/18] xen/arm: Introduce new helpers to handle guest/hyp SErrors

[Wei Chen]

Currently, ARM32 and ARM64 has different SError exception handlers.
These handlers include lots of code to check SError handle options
and code to distinguish guest-generated SErrors from hypervisor
SErrors.

The new helpers: do_trap_guest_serror and do_trap_hyp_serror are
wrappers of __do_trap_serror with constant guest/hyp parameters.
__do_trap_serror moves the option checking code and SError checking
code from assembly to C source. This will make the code become more
readable and avoid placing check code in too many places.

These two helpers only handle the following 3 types of SErrors:
1) Guest-generated SError and had been delivered in EL1 and then
   been forwarded to EL2.
2) Guest-generated SError but hadn't been delivered in EL1 before
   trapping to EL2. This SError would be caught in EL2 as soon as
   we just unmasked the PSTATE.A bit.
3) Hypervisor generated native SError, that would be a bug.

In the new helpers, we have used the function "inject_vabt_exception"
which was disabled by "#if 0" before. Now, we can remove the "#if 0"
to make this function to be available.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c            | 69 +++++++++++++++++++++++++++++++++++++++--
 xen/include/asm-arm/processor.h |  4 +++
 2 files changed, 71 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 053b7fc..48cfc8e 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -646,7 +646,6 @@ static void inject_dabt_exception(struct cpu_user_regs *regs,
 #endif
 }
 
-#if 0
 /* Inject a virtual Abort/SError into the guest. */
 static void inject_vabt_exception(struct cpu_user_regs *regs)
 {
@@ -676,7 +675,59 @@ static void inject_vabt_exception(struct cpu_user_regs *regs)
 
     current->arch.hcr_el2 |= HCR_VA;
 }
-#endif
+
+/*
+ * SError exception handler. We only handle the following 3 types of SErrors:
+ * 1) Guest-generated SError and had been delivered in EL1 and then
+ *    been forwarded to EL2.
+ * 2) Guest-generated SError but hadn't been delivered in EL1 before
+ *    trapping to EL2. This SError would be caught in EL2 as soon as
+ *    we just unmasked the PSTATE.A bit.
+ * 3) Hypervisor generated native SError, that would be a bug.
+ *
+ * A true parameter "guest" means that the SError is type#1 or type#2.
+ */
+static void __do_trap_serror(struct cpu_user_regs *regs, bool guest)
+{
+    /*
+     * Only "DIVERSE" option needs to distinguish the guest-generated SErrors
+     * from hypervisor SErrors.
+     */
+    if ( serrors_op == SERRORS_DIVERSE )
+    {
+        /* Forward the type#1 and type#2 SErrors to guests. */
+        if ( guest )
+            return inject_vabt_exception(regs);
+
+        /* Type#3 SErrors will panic the whole system */
+        goto crash_system;
+    }
+
+    /*
+     * The "FORWARD" option will forward all SErrors to the guests, except
+     * idle domain generated SErrors.
+     */
+    if ( serrors_op == SERRORS_FORWARD )
+    {
+        /*
+         * Because the idle domain doesn't have the ability to handle the
+         * SErrors, we have to crash the whole system while we get a SError
+         * generated by idle domain.
+         */
+        if ( is_idle_vcpu(current) )
+            goto crash_system;
+
+        return inject_vabt_exception(regs);
+    }
+
+crash_system:
+    /* Three possibilities to crash the whole system:
+     * 1) "DIVERSE" option with Hypervisor generated SErrors.
+     * 2) "FORWARD" option with Idle Domain generated SErrors.
+     * 3) "PANIC" option with all SErrors.
+     */
+    do_unexpected_trap("SError", regs);
+}
 
 struct reg_ctxt {
     /* Guest-side state */
@@ -2863,6 +2914,20 @@ asmlinkage void do_trap_guest_error(struct cpu_user_regs *regs)
     domain_crash_synchronous();
 }
 
+asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
+{
+    enter_hypervisor_head(regs);
+
+    __do_trap_serror(regs, VABORT_GEN_BY_GUEST(regs));
+}
+
+asmlinkage void do_trap_guest_serror(struct cpu_user_regs *regs)
+{
+    enter_hypervisor_head(regs);
+
+    __do_trap_serror(regs, true);
+}
+
 asmlinkage void do_trap_irq(struct cpu_user_regs *regs)
 {
     enter_hypervisor_head(regs);
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index 148cc6f..885dbca 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -709,6 +709,10 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
 
 void do_trap_guest_error(struct cpu_user_regs *regs);
 
+void do_trap_hyp_serror(struct cpu_user_regs *regs);
+
+void do_trap_guest_serror(struct cpu_user_regs *regs);
+
 /* Functions for pending virtual abort checking window. */
 void abort_guest_exit_start(void);
 void abort_guest_exit_end(void);
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 13/18] xen/arm: Replace do_trap_guest_serror with new helpers

[Wei Chen]

We have introduced two helpers to handle the guest/hyp SErrors:
do_trap_guest_serror and do_trap_guest_hyp_serror. These handlers
can take the role of do_trap_guest_serror and reduce the assembly
code in the same time. So we use these two helpers to replace it
and drop it now.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/arm32/traps.c      |  5 +----
 xen/arch/arm/arm64/entry.S      | 36 +++---------------------------------
 xen/arch/arm/traps.c            | 15 ---------------
 xen/include/asm-arm/processor.h |  2 --
 4 files changed, 4 insertions(+), 54 deletions(-)

diff --git a/xen/arch/arm/arm32/traps.c b/xen/arch/arm/arm32/traps.c
index 4176f0e..5bc5f64 100644
--- a/xen/arch/arm/arm32/traps.c
+++ b/xen/arch/arm/arm32/traps.c
@@ -62,10 +62,7 @@ asmlinkage void do_trap_prefetch_abort(struct cpu_user_regs *regs)
 
 asmlinkage void do_trap_data_abort(struct cpu_user_regs *regs)
 {
-    if ( VABORT_GEN_BY_GUEST(regs) )
-        do_trap_guest_error(regs);
-    else
-        do_unexpected_trap("Data Abort", regs);
+    do_trap_hyp_serror(regs);
 }
 
 /*
diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 113e1c3..8d5a890 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -178,40 +178,10 @@ hyp_error_invalid:
         invalid BAD_ERROR
 
 hyp_error:
-        /*
-         * Only two possibilities:
-         * 1) Either we come from the exit path, having just unmasked
-         *    PSTATE.A: change the return code to an EL2 fault, and
-         *    carry on, as we're already in a sane state to handle it.
-         * 2) Or we come from anywhere else, and that's a bug: we panic.
-         */
         entry   hyp=1
         msr     daifclr, #2
-
-        /*
-         * The ELR_EL2 may be modified by an interrupt, so we have to use the
-         * saved value in cpu_user_regs to check whether we come from 1) or
-         * not.
-         */
-        ldr     x0, [sp, #UREGS_PC]
-        adr     x1, abort_guest_exit_start
-        cmp     x0, x1
-        adr     x1, abort_guest_exit_end
-        ccmp    x0, x1, #4, ne
         mov     x0, sp
-        mov     x1, #BAD_ERROR
-
-        /*
-         * Not equal, the exception come from 2). It's a bug, we have to
-         * panic the hypervisor.
-         */
-        b.ne    do_bad_mode
-
-        /*
-         * Otherwise, the exception come from 1). It happened because of
-         * the guest. Crash this guest.
-         */
-        bl      do_trap_guest_error
+        bl      do_trap_hyp_serror
         exit    hyp=1
 
 /* Traps taken in Current EL with SP_ELx */
@@ -267,7 +237,7 @@ guest_error:
         entry   hyp=0, compat=0
         msr     daifclr, #2
         mov     x0, sp
-        bl      do_trap_guest_error
+        bl      do_trap_guest_serror
         exit    hyp=0, compat=0
 
 guest_sync_compat:
@@ -309,7 +279,7 @@ guest_error_compat:
         entry   hyp=0, compat=1
         msr     daifclr, #2
         mov     x0, sp
-        bl      do_trap_guest_error
+        bl      do_trap_guest_serror
         exit    hyp=0, compat=1
 
 ENTRY(return_to_new_vcpu32)
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 48cfc8e..44a0281 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2899,21 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
     }
 }
 
-asmlinkage void do_trap_guest_error(struct cpu_user_regs *regs)
-{
-    enter_hypervisor_head(regs);
-
-    /*
-     * Currently, to ensure hypervisor safety, when we received a
-     * guest-generated vSerror/vAbort, we just crash the guest to protect
-     * the hypervisor. In future we can better handle this by injecting
-     * a vSerror/vAbort to the guest.
-     */
-    gdprintk(XENLOG_WARNING, "Guest(Dom-%u) will be crashed by vSError\n",
-             current->domain->domain_id);
-    domain_crash_synchronous();
-}
-
 asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
 {
     enter_hypervisor_head(regs);
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index 885dbca..afad78c 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -707,8 +707,6 @@ void vcpu_regs_user_to_hyp(struct vcpu *vcpu,
 int call_smc(register_t function_id, register_t arg0, register_t arg1,
              register_t arg2);
 
-void do_trap_guest_error(struct cpu_user_regs *regs);
-
 void do_trap_hyp_serror(struct cpu_user_regs *regs);
 
 void do_trap_guest_serror(struct cpu_user_regs *regs);
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Wei Chen]

Currently, we masked the Abort/SError bit in Xen exception entries.
So Xen could not capture any Abort/SError while it's running.
Now, Xen has the ability to handle the Abort/SError, we should unmask
the Abort/SError bit by default to let Xen capture Abort/SError while
it's running.

But in order to avoid receiving nested asynchronous abort, we don't
unmask Abort/SError bit in hyp_error and trap_data_abort.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
We haven't done this before, so I don't know how can this change
will affect the Xen. If the IRQ and Abort take place at the same
time, how can we handle them? If an abort is taking place while we're
handling the IRQ, the program jump to abort exception, and then enable
the IRQ. In this case, what will happen? So I think I need more
discussions from community.
---
 xen/arch/arm/arm32/entry.S | 15 ++++++++++++++-
 xen/arch/arm/arm64/entry.S | 13 ++++++++-----
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
index 79929ca..4d46239 100644
--- a/xen/arch/arm/arm32/entry.S
+++ b/xen/arch/arm/arm32/entry.S
@@ -125,6 +125,7 @@ abort_guest_exit_end:
 trap_##trap:                                                            \
         SAVE_ALL;                                                       \
         cpsie i;        /* local_irq_enable */                          \
+        cpsie a;        /* asynchronous abort enable */                 \
         adr lr, return_from_trap;                                       \
         mov r0, sp;                                                     \
         mov r11, sp;                                                    \
@@ -135,6 +136,18 @@ trap_##trap:                                                            \
         ALIGN;                                                          \
 trap_##trap:                                                            \
         SAVE_ALL;                                                       \
+        cpsie a;        /* asynchronous abort enable */                 \
+        adr lr, return_from_trap;                                       \
+        mov r0, sp;                                                     \
+        mov r11, sp;                                                    \
+        bic sp, #7; /* Align the stack pointer (noop on guest trap) */  \
+        b do_trap_##trap
+
+#define DEFINE_TRAP_ENTRY_NOABORT(trap)                                 \
+        ALIGN;                                                          \
+trap_##trap:                                                            \
+        SAVE_ALL;                                                       \
+        cpsie i;        /* local_irq_enable */                          \
         adr lr, return_from_trap;                                       \
         mov r0, sp;                                                     \
         mov r11, sp;                                                    \
@@ -155,10 +168,10 @@ GLOBAL(hyp_traps_vector)
 DEFINE_TRAP_ENTRY(undefined_instruction)
 DEFINE_TRAP_ENTRY(supervisor_call)
 DEFINE_TRAP_ENTRY(prefetch_abort)
-DEFINE_TRAP_ENTRY(data_abort)
 DEFINE_TRAP_ENTRY(hypervisor)
 DEFINE_TRAP_ENTRY_NOIRQ(irq)
 DEFINE_TRAP_ENTRY_NOIRQ(fiq)
+DEFINE_TRAP_ENTRY_NOABORT(data_abort)
 
 return_from_trap:
         mov sp, r11
diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
index 8d5a890..0401a41 100644
--- a/xen/arch/arm/arm64/entry.S
+++ b/xen/arch/arm/arm64/entry.S
@@ -187,13 +187,14 @@ hyp_error:
 /* Traps taken in Current EL with SP_ELx */
 hyp_sync:
         entry   hyp=1
-        msr     daifclr, #2
+        msr     daifclr, #6
         mov     x0, sp
         bl      do_trap_hypervisor
         exit    hyp=1
 
 hyp_irq:
         entry   hyp=1
+        msr     daifclr, #4
         mov     x0, sp
         bl      do_trap_irq
         exit    hyp=1
@@ -208,7 +209,7 @@ guest_sync:
         ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
                     "nop; nop",
                     SKIP_CHECK_PENDING_VSERROR)
-        msr     daifclr, #2
+        msr     daifclr, #6
         mov     x0, sp
         bl      do_trap_hypervisor
 1:
@@ -224,6 +225,7 @@ guest_irq:
         ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
                     "nop; nop",
                     SKIP_CHECK_PENDING_VSERROR)
+        msr     daifclr, #4
         mov     x0, sp
         bl      do_trap_irq
 1:
@@ -235,7 +237,7 @@ guest_fiq_invalid:
 
 guest_error:
         entry   hyp=0, compat=0
-        msr     daifclr, #2
+        msr     daifclr, #6
         mov     x0, sp
         bl      do_trap_guest_serror
         exit    hyp=0, compat=0
@@ -250,7 +252,7 @@ guest_sync_compat:
         ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
                     "nop; nop",
                     SKIP_CHECK_PENDING_VSERROR)
-        msr     daifclr, #2
+        msr     daifclr, #6
         mov     x0, sp
         bl      do_trap_hypervisor
 1:
@@ -266,6 +268,7 @@ guest_irq_compat:
         ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
                     "nop; nop",
                     SKIP_CHECK_PENDING_VSERROR)
+        msr     daifclr, #4
         mov     x0, sp
         bl      do_trap_irq
 1:
@@ -277,7 +280,7 @@ guest_fiq_invalid_compat:
 
 guest_error_compat:
         entry   hyp=0, compat=1
-        msr     daifclr, #2
+        msr     daifclr, #6
         mov     x0, sp
         bl      do_trap_guest_serror
         exit    hyp=0, compat=1
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 15/18] xen/arm: Introduce a helper to synchronize SError

[Wei Chen]

We may have to isolate the SError between the context switch of
2 vCPUs or may have to prevent slipping hypervisor SError to guest.
So we need a helper to synchronize SError before context switching
or returning to guest.

This function will be used by the later patches in this series, we
use "#if 0" to disable it temporarily to remove compiler warnning.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 44a0281..ee7865b 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2899,6 +2899,17 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
     }
 }
 
+#if 0
+static void synchronize_serror(void)
+{
+    /* Synchronize against in-flight ld/st. */
+    dsb(sy);
+
+    /* A single instruction exception window */
+    isb();
+}
+#endif
+
 asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
 {
     enter_hypervisor_head(regs);
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 16/18] xen/arm: Isolate the SError between the context switch of 2 vCPUs

[Wei Chen]

If there is a pending SError while we are doing context switch, if the
SError handle option is "FORWARD", We have to guranatee this serror to
be caught by current vCPU, otherwise it will be caught by next vCPU and
be forwarded to this wrong vCPU.

We don't want to export serror_op accessing to other source files and
use serror_op every place, so we add a helper to synchronize SError for
context switching. The synchronize_serror has been used in this helper,
so the "#if 0" can be removed.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/domain.c           |  2 ++
 xen/arch/arm/traps.c            | 14 ++++++++++++--
 xen/include/asm-arm/processor.h |  2 ++
 3 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 69c2854..a547fcd 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -312,6 +312,8 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
 
     local_irq_disable();
 
+    prevent_forward_serror_to_next_vcpu();
+
     set_current(next);
 
     prev = __context_switch(prev, next);
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index ee7865b..b8c8389 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2899,7 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
     }
 }
 
-#if 0
 static void synchronize_serror(void)
 {
     /* Synchronize against in-flight ld/st. */
@@ -2908,7 +2907,18 @@ static void synchronize_serror(void)
     /* A single instruction exception window */
     isb();
 }
-#endif
+
+/*
+ * If the SErrors option is "FORWARD", we have to prevent forwarding
+ * serror to wrong vCPU. So before context switch, we have to use the
+ * synchronize_serror to guarantee that the pending serror would be
+ * caught by current vCPU.
+ */
+void prevent_forward_serror_to_next_vcpu(void)
+{
+    if ( serrors_op == SERRORS_FORWARD )
+        synchronize_serror();
+}
 
 asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
 {
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index afad78c..3b43234 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -711,6 +711,8 @@ void do_trap_hyp_serror(struct cpu_user_regs *regs);
 
 void do_trap_guest_serror(struct cpu_user_regs *regs);
 
+void prevent_forward_serror_to_next_vcpu(void);
+
 /* Functions for pending virtual abort checking window. */
 void abort_guest_exit_start(void);
 void abort_guest_exit_end(void);
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 17/18] xen/arm: Prevent slipping hypervisor SError to guest

[Wei Chen]

If there is a pending SError while we're returning from trap. If the
SError handle option is "DIVERSE", we have to prevent slipping this
hypervisor SError to guest. So we have to use the dsb/isb to guarantee
that the pending hypervisor SError would be caught in hypervisor before
return to guest.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index b8c8389..3b84e80 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2953,6 +2953,16 @@ asmlinkage void leave_hypervisor_tail(void)
         local_irq_disable();
         if (!softirq_pending(smp_processor_id())) {
             gic_inject();
+
+            /*
+             * If the SErrors handle option is "DIVERSE", we have to prevent
+             * slipping the hypervisor SError to guest. So before returning
+             * from trap, we use the synchronize_serror to guarantee that the
+             * pending SError would be caught in hypervisor.
+             */
+            if ( serrors_op == SERRORS_DIVERSE )
+                synchronize_serror();
+
             WRITE_SYSREG(current->arch.hcr_el2, HCR_EL2);
             return;
         }
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 18/18] xen/arm: Handle guest external abort as guest SError

[Wei Chen]

The guest generated external data/instruction aborts can be treated
as guest SErrors. We already have a handler to handle the SErrors,
so we can reuse this handler to handle guest external aborts.

Signed-off-by: Wei Chen <Wei.Chen@arm.com>
---
 xen/arch/arm/traps.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 3b84e80..24511e5 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2558,12 +2558,12 @@ static void do_trap_instr_abort_guest(struct cpu_user_regs *regs,
 
     /*
      * If this bit has been set, it means that this instruction abort is caused
-     * by a guest external abort. Currently we crash the guest to protect the
-     * hypervisor. In future one can better handle this by injecting a virtual
-     * abort to the guest.
+     * by a guest external abort. We can handle this instruction abort as guest
+     * SError.
      */
     if ( hsr.iabt.eat )
-        domain_crash_synchronous();
+        return __do_trap_serror(regs, true);
+
 
     if ( hpfar_is_valid(hsr.iabt.s1ptw, fsc) )
         gpa = get_faulting_ipa(gva);
@@ -2661,12 +2661,10 @@ static void do_trap_data_abort_guest(struct cpu_user_regs *regs,
 
     /*
      * If this bit has been set, it means that this data abort is caused
-     * by a guest external abort. Currently we crash the guest to protect the
-     * hypervisor. In future one can better handle this by injecting a virtual
-     * abort to the guest.
+     * by a guest external abort. We treat this data abort as guest SError.
      */
     if ( dabt.eat )
-        domain_crash_synchronous();
+        return __do_trap_serror(regs, true);
 
     info.dabt = dabt;
 #ifdef CONFIG_ARM_32
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/18] xen/arm: Introduce a helper to get default HCR_EL2 flags

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> We want to add HCR_EL2 register to Xen context switch. And each copy
> of HCR_EL2 in vcpu structure will be initialized with the same set
> of trap flags as the HCR_EL2 register. We introduce a helper here to
> represent these flags to be reused easily.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/traps.c            | 11 ++++++++---
>  xen/include/asm-arm/processor.h |  2 ++
>  2 files changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 614501f..d343c66 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -115,6 +115,13 @@ static void __init parse_vwfi(const char *s)
>  }
>  custom_param("vwfi", parse_vwfi);
>  
> +register_t get_default_hcr_flags(void)
> +{
> +    return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
> +             (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
> +             HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB);
> +}

I haven't finished reading this series yet, but I would make this a
static inline function if possible


>  void init_traps(void)
>  {
>      /* Setup Hyp vector base */
> @@ -139,9 +146,7 @@ void init_traps(void)
>                   CPTR_EL2);
>  
>      /* Setup hypervisor traps */
> -    WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
> -                 (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
> -                 HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB,HCR_EL2);
> +    WRITE_SYSREG(get_default_hcr_flags(), HCR_EL2);
>      isb();
>  }
>  
> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
> index afc0e9a..4b6338b 100644
> --- a/xen/include/asm-arm/processor.h
> +++ b/xen/include/asm-arm/processor.h
> @@ -708,6 +708,8 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
>  
>  void do_trap_guest_error(struct cpu_user_regs *regs);
>  
> +register_t get_default_hcr_flags(void);
> +
>  #endif /* __ASSEMBLY__ */
>  #endif /* __ASM_ARM_PROCESSOR_H */
>  /*
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> Different domains may have different HCR_EL2 flags. For example, the
> 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
> we give each domain a default HCR_EL2 value and save it in the VCPU's
> context.
> 
> HCR_EL2 register has only one bit can be updated automatically without
> explicit write (HCR_VSE). But we haven't used this bit currently, so
> we can consider that the HCR_EL2 register will not be modified while
> the guest is running. So save the HCR_EL2 while guest exiting to
> hypervisor is not neccessary. We just have to restore this register for
> each VCPU while leaving hypervisor.
> 
> We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
> ctxt_switch_to. Because the leave_hypervisor_tail is the closest place
> to the exception return. In this case, we don't need to warrant the
> HCR_EL2 will not be changed between ctxt_switch_to and exception return.

Please explain a bit better why it is good to restore HCR_EL2 in
leave_hypervisor_tail. Why is it a good thing that is closer to the
exception return? What can be the cause of a change in HCR_EL2?



> Even though we have restored HCR_EL2 in leave_hypervisor_tail, we still
> have to keep the write to HCR_EL2 in p2m_restore_state. That because
> p2m_restore_state could be used to switch between two p2m and possibly
> to do address translation using hardware. For instance when building
> the hardware domain, we are using the instruction to before copying
> data. During the translation, some bits of base register (such as SCTLR
> and HCR) could be cached in TLB and used for the translation.
> 
> We had some issues in the past (see commit b3cbe129d "xen: arm: Ensure
> HCR_EL2.RW is set correctly when building dom0"), so we should probably
> keep the write to HCR_EL2 in p2m_restore_state.
> 
> Signed-off-by: wei chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/domain.c        |  2 ++
>  xen/arch/arm/p2m.c           | 15 +++++++++------
>  xen/arch/arm/traps.c         |  1 +
>  xen/include/asm-arm/domain.h |  3 +++
>  4 files changed, 15 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index bb327da..5d18bb0 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -513,6 +513,8 @@ int vcpu_initialise(struct vcpu *v)
>  
>      v->arch.actlr = READ_SYSREG32(ACTLR_EL1);
>  
> +    v->arch.hcr_el2 = get_default_hcr_flags();
> +
>      processor_vcpu_initialise(v);
>  
>      if ( (rc = vcpu_vgic_init(v)) != 0 )
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index 1fc6ca3..c49bfa6 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -128,26 +128,29 @@ void p2m_save_state(struct vcpu *p)
>  
>  void p2m_restore_state(struct vcpu *n)
>  {
> -    register_t hcr;
>      struct p2m_domain *p2m = &n->domain->arch.p2m;
>  
>      if ( is_idle_vcpu(n) )
>          return;
>  
> -    hcr = READ_SYSREG(HCR_EL2);
> -
>      WRITE_SYSREG64(p2m->vttbr, VTTBR_EL2);
>      isb();
>  
>      if ( is_32bit_domain(n->domain) )
> -        hcr &= ~HCR_RW;
> +        n->arch.hcr_el2 &= ~HCR_RW;
>      else
> -        hcr |= HCR_RW;
> +        n->arch.hcr_el2 |= HCR_RW;

It makes sense to move this if/else statement to one of the vcpu
initialization functions, but I can see that you are going to do that in
a later patch, so that's OK.


>      WRITE_SYSREG(n->arch.sctlr, SCTLR_EL1);
>      isb();
>  
> -    WRITE_SYSREG(hcr, HCR_EL2);
> +    /*
> +     * p2m_restore_state could be used to switch between two p2m and possibly
> +     * to do address translation using hardware. And these operations may
> +     * happen during the interval between enter/leave hypervior, so we should
> +     * probably keep the write to HCR_EL2 here.
> +     */

Please rewrite this to:

  p2m_restore_state could be used to switch between two p2m and possibly
  to do address translation using hardware. These operations may
  happen during the interval between enter/leave hypervior, so we need
  to restore the right HCR_EL2 here.


> +    WRITE_SYSREG(n->arch.hcr_el2, HCR_EL2);
>      isb();
>  }
>  
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index d343c66..9792d02 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2811,6 +2811,7 @@ asmlinkage void leave_hypervisor_tail(void)
>          local_irq_disable();
>          if (!softirq_pending(smp_processor_id())) {
>              gic_inject();

Please add another in-code comment:

  Set HCR_EL2 in leave_hypervisor_tail, because it is the closest code
  site to the exception return and this is important because....


> +            WRITE_SYSREG(current->arch.hcr_el2, HCR_EL2);
>              return;
>          }
>          local_irq_enable();
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 09fe502..7b1dacc 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -204,6 +204,9 @@ struct arch_vcpu
>      register_t tpidr_el1;
>      register_t tpidrro_el0;
>  
> +    /* HYP configuration */
> +    register_t hcr_el2;
> +
>      uint32_t teecr, teehbr; /* ThumbEE, 32-bit guests only */
>  #ifdef CONFIG_ARM_32
>      /*
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
> when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
> DomU in vcpu_initialise, we didn't know the domain's address size
> information. We had to use compatible flags to initialize HCR_EL2,
> and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
> at every context switch.
> 
> But, after we added the HCR_EL2 to vcpu's context, this behaviour
> seems a little fussy. We can update the HCR_RW bit in vcpu's context
> as soon as we get the domain's address size to avoid setting/clearing
> HCR_RW at every context switch.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/arm64/domctl.c  | 6 ++++++
>  xen/arch/arm/domain.c        | 5 +++++
>  xen/arch/arm/domain_build.c  | 7 +++++++
>  xen/arch/arm/p2m.c           | 5 -----
>  xen/include/asm-arm/domain.h | 1 +
>  5 files changed, 19 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
> index 44e1e7b..ab8781f 100644
> --- a/xen/arch/arm/arm64/domctl.c
> +++ b/xen/arch/arm/arm64/domctl.c
> @@ -14,6 +14,8 @@
>  
>  static long switch_mode(struct domain *d, enum domain_type type)
>  {
> +    struct vcpu *v;
> +
>      if ( d == NULL )
>          return -EINVAL;
>      if ( d->tot_pages != 0 )
> @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum domain_type type)
>  
>      d->arch.type = type;
>  
> +    if ( is_64bit_domain(d) )
> +        for_each_vcpu(d, v)
> +            vcpu_switch_to_aarch64_mode(v);
> +
>      return 0;
>  }
>  
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 5d18bb0..69c2854 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
>      free_xenheap_pages(v->arch.stack, STACK_ORDER);
>  }
>  
> +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
> +{
> +    v->arch.hcr_el2 |= HCR_RW;
> +}

if possible, I would write it as a static inline function


>  int arch_domain_create(struct domain *d, unsigned int domcr_flags,
>                         struct xen_arch_domainconfig *config)
>  {
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index de59e5f..3abacc0 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -2148,6 +2148,10 @@ int construct_dom0(struct domain *d)
>          return -EINVAL;
>      }
>      d->arch.type = kinfo.type;
> +
> +    if ( is_64bit_domain(d) )
> +        vcpu_switch_to_aarch64_mode(v);
> +
>  #endif
>  
>      allocate_memory(d, &kinfo);
> @@ -2240,6 +2244,9 @@ int construct_dom0(struct domain *d)
>              printk("Failed to allocate dom0 vcpu %d on pcpu %d\n", i, cpu);
>              break;
>          }
> +
> +        if ( is_64bit_domain(d) )
> +            vcpu_switch_to_aarch64_mode(d->vcpu[i]);
>      }
>  
>      return 0;
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index c49bfa6..1cba0d0 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -136,11 +136,6 @@ void p2m_restore_state(struct vcpu *n)
>      WRITE_SYSREG64(p2m->vttbr, VTTBR_EL2);
>      isb();
>  
> -    if ( is_32bit_domain(n->domain) )
> -        n->arch.hcr_el2 &= ~HCR_RW;
> -    else
> -        n->arch.hcr_el2 |= HCR_RW;
> -
>      WRITE_SYSREG(n->arch.sctlr, SCTLR_EL1);
>      isb();
>  
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 7b1dacc..68185e2 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -268,6 +268,7 @@ struct arch_vcpu
>  
>  void vcpu_show_execution_state(struct vcpu *);
>  void vcpu_show_registers(const struct vcpu *);
> +void vcpu_switch_to_aarch64_mode(struct vcpu *);
>  
>  unsigned int domain_max_vcpus(const struct domain *);
>  
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 04/18] xen/arm: Save HCR_EL2 when a guest took the SError

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> The HCR_EL2.VSE (HCR.VA for aarch32) bit can be used to generate a
> virtual abort to guest. The HCR_EL2.VSE bit has a peculiar feature
> of getting cleared when the guest has taken the abort (this is the
> only bit that behaves as such in HCR_EL2 register).
> 
> This means that if we set the HCR_EL2.VSE bit to signal such an abort,
> we must preserve it in the guest context until it disappears from
> HCR_EL2, and at which point it must be cleared from the context. This
> is achieved by reading back from HCR_EL2 until the guest takes the
> fault.
> 
> If we preserved a pending VSE in guest context, we have to restore
> it to HCR_EL2 when context switch to this guest. This is achieved
> by writing saved HCR_EL2 value in guest context back to HCR_EL2
> register before return to guest. This had been done by the patch
> of "Restore HCR_EL2 register".
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  xen/arch/arm/traps.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 9792d02..476e2be 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2641,7 +2641,18 @@ static void do_trap_smc(struct cpu_user_regs *regs, const union hsr hsr)
>  static void enter_hypervisor_head(struct cpu_user_regs *regs)
>  {
>      if ( guest_mode(regs) )
> +    {
> +        /*
> +         * If we pended a virtual abort, preserve it until it gets cleared.
> +         * See ARM ARM DDI 0487A.j D1.14.3 (Virtual Interrupts) for details,
> +         * but the crucial bit is "On taking a vSError interrupt, HCR_EL2.VSE
> +         * (alias of HCR.VA) is cleared to 0."
> +         */
> +        if ( current->arch.hcr_el2 & HCR_VA )
> +            current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
> +
>          gic_clear_lrs(current);
> +    }
>  }
>  
>  asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 06/18] xen/arm: Introduce a virtual abort injection helper

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> When guest triggers async aborts, in most platform, such aborts
> will be routed to hypervisor. But we don't want the hypervisor
> to handle such aborts, so we have to route such aborts back to
> the guest.
> 
> This helper is using the HCR_EL2.VSE (HCR.VA for aarch32) bit to
> route such aborts back to the guest. If the guest PC had been
> advanced by SVC/HVC/SMC instructions before we caught the SError
> in hypervisor, we have to adjust the guest PC to exact address
> while the SError generated.
> 
> About HSR_EC_SVC32/64, even thought we don't trap SVC32/64 today,
> we would like them to be handled here. This would be useful when
> VM introspection will gain support of SVC32/64 trapping.
> 
> This helper will be used by the later patches in this series, we
> use #if 0 to disable it in this patch temporarily to remove the
> warning message of unused function from compiler.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  xen/arch/arm/traps.c            | 32 ++++++++++++++++++++++++++++++++
>  xen/include/asm-arm/processor.h |  1 +
>  2 files changed, 33 insertions(+)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index c11359d..e425832 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -618,6 +618,38 @@ static void inject_dabt_exception(struct cpu_user_regs *regs,
>  #endif
>  }
>  
> +#if 0
> +/* Inject a virtual Abort/SError into the guest. */
> +static void inject_vabt_exception(struct cpu_user_regs *regs)
> +{
> +    const union hsr hsr = { .bits = regs->hsr };
> +
> +    /*
> +     * SVC/HVC/SMC already have an adjusted PC (See ARM ARM DDI 0487A.j
> +     * D1.10.1 for more details), which we need to correct in order to
> +     * return to after having injected the SError.
> +     */
> +    switch ( hsr.ec )
> +    {
> +    case HSR_EC_SVC32:
> +    case HSR_EC_HVC32:
> +    case HSR_EC_SMC32:
> +#ifdef CONFIG_ARM_64
> +    case HSR_EC_SVC64:
> +    case HSR_EC_HVC64:
> +    case HSR_EC_SMC64:
> +#endif
> +        regs->pc -= hsr.len ? 4 : 2;
> +        break;
> +
> +    default:
> +        break;
> +    }
> +
> +    current->arch.hcr_el2 |= HCR_VA;
> +}
> +#endif
> +
>  struct reg_ctxt {
>      /* Guest-side state */
>      uint32_t sctlr_el1;
> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
> index 4b6338b..d7b0711 100644
> --- a/xen/include/asm-arm/processor.h
> +++ b/xen/include/asm-arm/processor.h
> @@ -252,6 +252,7 @@
>  #define HSR_EC_HVC32                0x12
>  #define HSR_EC_SMC32                0x13
>  #ifdef CONFIG_ARM_64
> +#define HSR_EC_SVC64                0x15
>  #define HSR_EC_HVC64                0x16
>  #define HSR_EC_SMC64                0x17
>  #define HSR_EC_SYSREG               0x18
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 07/18] xen/arm: Introduce a command line parameter for SErrors/Aborts

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> In order to distinguish guest-generated SErrors from hypervisor-generated
> SErrors. We have to place SError checking code in every EL1 -> EL2 paths.
         ^ remove .

> That will be an overhead on entries caused by dsb/isb.
> 
> But not all platforms want to categorize the SErrors. For example, a host
> that is running with trusted guests. The administrator can confirm that
> all guests that are running on the host will not trigger such SErrors. In
> this user scene, we should provide some options to administrator to avoid
       ^use-case


> categorizing the SErrors. And then reduce the overhead of dsb/isb.
               ^ remove   ^ remove 


> We provided following 3 options to administrator to determine how to handle
> the SErrors:
> 
> * `diverse`:
>   The hypervisor will distinguish guest SErrors from hypervisor SErrors.
>   The guest generated SErrors will be forwarded to guests, the hypervisor
>   generated SErrors will cause the whole system crash.
>   It requires:
>   1. Place dsb/isb on all EL1 -> EL2 trap entries to categorize SErrors
>      correctly.
>   2. Place dsb/isb on EL2 -> EL1 return paths to prevent slipping hypervisor
>      SErrors to guests.
>   3. Place dsb/isb in context switch to isolate the SErrors between 2 vCPUs.
> 
> * `forward`:
>   The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
>   All SErrors will be forwarded to guests, except the SErrors generated when
>   idle vCPU is running. The idle domain doesn't have the ability to hanle the
>   SErrors, so we have to crash the whole system when we get SErros with idle
>   vCPU. This option will avoid most overhead of the dsb/isb, except the dsb/isb
>   in context switch which is used to isolate the SErrors between 2 vCPUs.
> 
> * `panic`:
>   The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
>   All SErrors will crash the whole system. This option will avoid all overhead
>   of the dsb/isb.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> 
> ---
> About adding dsb/isb to prevent slipping Hypervisor SErrors to Guests if the
> selected option is "diverse". Some Hypervisor SErrors could not be avoid by
> software, for example ECC Error. But I don't know whether it's worth adding
> the overhead by default.
> ---
>  docs/misc/xen-command-line.markdown | 44 +++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/traps.c                | 19 ++++++++++++++++
>  2 files changed, 63 insertions(+)
> 
> diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
> index 4daf5b5..79554ce 100644
> --- a/docs/misc/xen-command-line.markdown
> +++ b/docs/misc/xen-command-line.markdown
> @@ -1481,6 +1481,50 @@ enabling more sockets and cores to go into deeper sleep states.
>  
>  Set the serial transmit buffer size.
>  
> +### serrors (ARM)
> +> `= diverse | forward | panic`
> +
> +> Default: `diverse`
> +
> +This parameter is provided to administrator to determine how to handle the
> +SErrors.

  This parameter is provided to administrators to determine how the
  hypervisors handle SErrors.


> +In order to distinguish guest-generated SErrors from hypervisor-generated
> +SErrors. We have to place SError checking code in every EL1 -> EL2 paths.
          ^remove .

> +That will be an overhead on entries caused by dsb/isb. But not all platforms
   
   That will cause overhead on entries due to dsb/isb. However, not all platforms

> +need to categorize the SErrors. For example, a host that is running with
                       ^ remove the

> +trusted guests. The administrator can confirm that all guests that are
> +running on the host will not trigger such SErrors. In this case, the
> +administrator can use this parameter to skip categorizing the SErrors. And
                                                             ^ remove   ^ remove

> +reduce the overhead of dsb/isb.
> +
> +We provided following 3 options to administrator to determine how to handle
              ^ the following         ^ administrators

> +the SErrors:
   ^ remove the


> +
> +* `diverse`:
> +  The hypervisor will distinguish guest SErrors from hypervisor SErrors.
> +  The guest generated SErrors will be forwarded to guests, the hypervisor
> +  generated SErrors will cause the whole system crash.
                                                  ^ to crash

> +  It requires:
> +  1. Place dsb/isb on all EL1 -> EL2 trap entries to categorize SErrors
        ^ remove Place
> +     correctly.
> +  2. Place dsb/isb on EL2 -> EL1 return paths to prevent slipping hypervisor
        ^ remove Place
> +     SErrors to guests.
> +  3. Place dsb/isb in context switch to isolate the SErrors between 2 vCPUs.
        ^ remove Place                             ^ remove the
> +
> +* `forward`:
> +  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
> +  All SErrors will be forwarded to guests, except the SErrors generated when
> +  idle vCPU is running. The idle domain doesn't have the ability to hanle the
     ^ the idle                                                        ^ handle ^ remove the 

> +  SErrors, so we have to crash the whole system when we get SErros with idle
                                                                          ^ the idle

> +  vCPU. This option will avoid most overhead of the dsb/isb, except the dsb/isb
> +  in context switch which is used to isolate the SErrors between 2 vCPUs.
> +
> +* `panic`:
> +  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
> +  All SErrors will crash the whole system. This option will avoid all overhead
> +  of the dsb/isb.
    
    of the dsb/isb pairs.

Please make these changes to the commit message too, when applicable.
With these changes:

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


>  ### smap
>  > `= <boolean> | hvm`
>  
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index e425832..5e31699 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -115,6 +115,25 @@ static void __init parse_vwfi(const char *s)
>  }
>  custom_param("vwfi", parse_vwfi);
>  
> +static enum {
> +    SERRORS_DIVERSE,
> +    SERRORS_FORWARD,
> +    SERRORS_PANIC,
> +} serrors_op;
> +
> +static void __init parse_serrors_behavior(const char *str)
> +{
> +    if ( !strcmp(str, "forward") )
> +        serrors_op = SERRORS_FORWARD;
> +    else if ( !strcmp(str, "panic") )
> +        serrors_op = SERRORS_PANIC;
> +    else
> +        serrors_op = SERRORS_DIVERSE;
> +
> +    return;
> +}
> +custom_param("serrors", parse_serrors_behavior);
> +
>  register_t get_default_hcr_flags(void)
>  {
>      return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/18] xen/arm: Introduce a helper to get default HCR_EL2 flags

[Wei Chen]

Hi Stefano,

On 2017/3/15 8:24, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> We want to add HCR_EL2 register to Xen context switch. And each copy
>> of HCR_EL2 in vcpu structure will be initialized with the same set
>> of trap flags as the HCR_EL2 register. We introduce a helper here to
>> represent these flags to be reused easily.
>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>> ---
>>  xen/arch/arm/traps.c            | 11 ++++++++---
>>  xen/include/asm-arm/processor.h |  2 ++
>>  2 files changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index 614501f..d343c66 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -115,6 +115,13 @@ static void __init parse_vwfi(const char *s)
>>  }
>>  custom_param("vwfi", parse_vwfi);
>>
>> +register_t get_default_hcr_flags(void)
>> +{
>> +    return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
>> +             (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
>> +             HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB);
>> +}
>
> I haven't finished reading this series yet, but I would make this a
> static inline function if possible
>

I had considered to use static inline before. But it must move the

static enum {
	TRAP,
	NATIVE,
} vwfi;

to the header file at the same time. But get_default_hcr_flags would
not be used frequently. So I thought it didn't have enough value to
change a less relevant code to make this function become static inline.

>
>>  void init_traps(void)
>>  {
>>      /* Setup Hyp vector base */
>> @@ -139,9 +146,7 @@ void init_traps(void)
>>                   CPTR_EL2);
>>
>>      /* Setup hypervisor traps */
>> -    WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
>> -                 (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
>> -                 HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB,HCR_EL2);
>> +    WRITE_SYSREG(get_default_hcr_flags(), HCR_EL2);
>>      isb();
>>  }
>>
>> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
>> index afc0e9a..4b6338b 100644
>> --- a/xen/include/asm-arm/processor.h
>> +++ b/xen/include/asm-arm/processor.h
>> @@ -708,6 +708,8 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
>>
>>  void do_trap_guest_error(struct cpu_user_regs *regs);
>>
>> +register_t get_default_hcr_flags(void);
>> +
>>  #endif /* __ASSEMBLY__ */
>>  #endif /* __ASM_ARM_PROCESSOR_H */
>>  /*
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Wei Chen]

On 2017/3/15 8:25, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> Different domains may have different HCR_EL2 flags. For example, the
>> 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
>> we give each domain a default HCR_EL2 value and save it in the VCPU's
>> context.
>>
>> HCR_EL2 register has only one bit can be updated automatically without
>> explicit write (HCR_VSE). But we haven't used this bit currently, so
>> we can consider that the HCR_EL2 register will not be modified while
>> the guest is running. So save the HCR_EL2 while guest exiting to
>> hypervisor is not neccessary. We just have to restore this register for
>> each VCPU while leaving hypervisor.
>>
>> We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
>> ctxt_switch_to. Because the leave_hypervisor_tail is the closest place
>> to the exception return. In this case, we don't need to warrant the
>> HCR_EL2 will not be changed between ctxt_switch_to and exception return.
>
> Please explain a bit better why it is good to restore HCR_EL2 in
> leave_hypervisor_tail. Why is it a good thing that is closer to the
> exception return? What can be the cause of a change in HCR_EL2?
>

Ok, I will try to improve it in next version. In current Xen code, I
can't see any code would change the HCR_EL2 between ctxt_switch_to and
return_from_trap. But my concern is that, in the future, if someone
want to insert some HCR_EL2 change code between them, we can't guarantee
he will restore correct HCR_EL2 value for current vcpu.

>
>
>> Even though we have restored HCR_EL2 in leave_hypervisor_tail, we still
>> have to keep the write to HCR_EL2 in p2m_restore_state. That because
>> p2m_restore_state could be used to switch between two p2m and possibly
>> to do address translation using hardware. For instance when building
>> the hardware domain, we are using the instruction to before copying
>> data. During the translation, some bits of base register (such as SCTLR
>> and HCR) could be cached in TLB and used for the translation.
>>
>> We had some issues in the past (see commit b3cbe129d "xen: arm: Ensure
>> HCR_EL2.RW is set correctly when building dom0"), so we should probably
>> keep the write to HCR_EL2 in p2m_restore_state.
>>
>> Signed-off-by: wei chen <Wei.Chen@arm.com>
>> ---
>>  xen/arch/arm/domain.c        |  2 ++
>>  xen/arch/arm/p2m.c           | 15 +++++++++------
>>  xen/arch/arm/traps.c         |  1 +
>>  xen/include/asm-arm/domain.h |  3 +++
>>  4 files changed, 15 insertions(+), 6 deletions(-)
>>
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index bb327da..5d18bb0 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -513,6 +513,8 @@ int vcpu_initialise(struct vcpu *v)
>>
>>      v->arch.actlr = READ_SYSREG32(ACTLR_EL1);
>>
>> +    v->arch.hcr_el2 = get_default_hcr_flags();
>> +
>>      processor_vcpu_initialise(v);
>>
>>      if ( (rc = vcpu_vgic_init(v)) != 0 )
>> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
>> index 1fc6ca3..c49bfa6 100644
>> --- a/xen/arch/arm/p2m.c
>> +++ b/xen/arch/arm/p2m.c
>> @@ -128,26 +128,29 @@ void p2m_save_state(struct vcpu *p)
>>
>>  void p2m_restore_state(struct vcpu *n)
>>  {
>> -    register_t hcr;
>>      struct p2m_domain *p2m = &n->domain->arch.p2m;
>>
>>      if ( is_idle_vcpu(n) )
>>          return;
>>
>> -    hcr = READ_SYSREG(HCR_EL2);
>> -
>>      WRITE_SYSREG64(p2m->vttbr, VTTBR_EL2);
>>      isb();
>>
>>      if ( is_32bit_domain(n->domain) )
>> -        hcr &= ~HCR_RW;
>> +        n->arch.hcr_el2 &= ~HCR_RW;
>>      else
>> -        hcr |= HCR_RW;
>> +        n->arch.hcr_el2 |= HCR_RW;
>
> It makes sense to move this if/else statement to one of the vcpu
> initialization functions, but I can see that you are going to do that in
> a later patch, so that's OK.
>

Thanks.

>
>>      WRITE_SYSREG(n->arch.sctlr, SCTLR_EL1);
>>      isb();
>>
>> -    WRITE_SYSREG(hcr, HCR_EL2);
>> +    /*
>> +     * p2m_restore_state could be used to switch between two p2m and possibly
>> +     * to do address translation using hardware. And these operations may
>> +     * happen during the interval between enter/leave hypervior, so we should
>> +     * probably keep the write to HCR_EL2 here.
>> +     */
>
> Please rewrite this to:
>
>   p2m_restore_state could be used to switch between two p2m and possibly
>   to do address translation using hardware. These operations may
>   happen during the interval between enter/leave hypervior, so we need
>   to restore the right HCR_EL2 here.
>

Thanks, I will update the code comment in next version.

>
>> +    WRITE_SYSREG(n->arch.hcr_el2, HCR_EL2);
>>      isb();
>>  }
>>
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index d343c66..9792d02 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -2811,6 +2811,7 @@ asmlinkage void leave_hypervisor_tail(void)
>>          local_irq_disable();
>>          if (!softirq_pending(smp_processor_id())) {
>>              gic_inject();
>
> Please add another in-code comment:
>
>   Set HCR_EL2 in leave_hypervisor_tail, because it is the closest code
>   site to the exception return and this is important because....
>

Ok, I will add code comment in next version.

>
>> +            WRITE_SYSREG(current->arch.hcr_el2, HCR_EL2);
>>              return;
>>          }
>>          local_irq_enable();
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 09fe502..7b1dacc 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -204,6 +204,9 @@ struct arch_vcpu
>>      register_t tpidr_el1;
>>      register_t tpidrro_el0;
>>
>> +    /* HYP configuration */
>> +    register_t hcr_el2;
>> +
>>      uint32_t teecr, teehbr; /* ThumbEE, 32-bit guests only */
>>  #ifdef CONFIG_ARM_32
>>      /*
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Wei Chen]

Hi Stefano,

On 2017/3/15 8:25, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
>> when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
>> DomU in vcpu_initialise, we didn't know the domain's address size
>> information. We had to use compatible flags to initialize HCR_EL2,
>> and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
>> at every context switch.
>>
>> But, after we added the HCR_EL2 to vcpu's context, this behaviour
>> seems a little fussy. We can update the HCR_RW bit in vcpu's context
>> as soon as we get the domain's address size to avoid setting/clearing
>> HCR_RW at every context switch.
>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>> ---
>>  xen/arch/arm/arm64/domctl.c  | 6 ++++++
>>  xen/arch/arm/domain.c        | 5 +++++
>>  xen/arch/arm/domain_build.c  | 7 +++++++
>>  xen/arch/arm/p2m.c           | 5 -----
>>  xen/include/asm-arm/domain.h | 1 +
>>  5 files changed, 19 insertions(+), 5 deletions(-)
>>
>> diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
>> index 44e1e7b..ab8781f 100644
>> --- a/xen/arch/arm/arm64/domctl.c
>> +++ b/xen/arch/arm/arm64/domctl.c
>> @@ -14,6 +14,8 @@
>>
>>  static long switch_mode(struct domain *d, enum domain_type type)
>>  {
>> +    struct vcpu *v;
>> +
>>      if ( d == NULL )
>>          return -EINVAL;
>>      if ( d->tot_pages != 0 )
>> @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum domain_type type)
>>
>>      d->arch.type = type;
>>
>> +    if ( is_64bit_domain(d) )
>> +        for_each_vcpu(d, v)
>> +            vcpu_switch_to_aarch64_mode(v);
>> +
>>      return 0;
>>  }
>>
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index 5d18bb0..69c2854 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
>>      free_xenheap_pages(v->arch.stack, STACK_ORDER);
>>  }
>>
>> +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
>> +{
>> +    v->arch.hcr_el2 |= HCR_RW;
>> +}
>
> if possible, I would write it as a static inline function
>

I had tried to write it as a static inline function in asm/domain.h
But while the first source file (arm64/asm-offsets.c) include
xen/sched.h wanted to compile this inline function it could not find
'struct vcpu'. Because the xen/sched.h included the asm/domain.h
but defined the vcpu type later. Even though we had included the
xen/sched.h in asm/domain.h already.


>
>>  int arch_domain_create(struct domain *d, unsigned int domcr_flags,
>>                         struct xen_arch_domainconfig *config)
>>  {
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index de59e5f..3abacc0 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -2148,6 +2148,10 @@ int construct_dom0(struct domain *d)
>>          return -EINVAL;
>>      }
>>      d->arch.type = kinfo.type;
>> +
>> +    if ( is_64bit_domain(d) )
>> +        vcpu_switch_to_aarch64_mode(v);
>> +
>>  #endif
>>
>>      allocate_memory(d, &kinfo);
>> @@ -2240,6 +2244,9 @@ int construct_dom0(struct domain *d)
>>              printk("Failed to allocate dom0 vcpu %d on pcpu %d\n", i, cpu);
>>              break;
>>          }
>> +
>> +        if ( is_64bit_domain(d) )
>> +            vcpu_switch_to_aarch64_mode(d->vcpu[i]);
>>      }
>>
>>      return 0;
>> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
>> index c49bfa6..1cba0d0 100644
>> --- a/xen/arch/arm/p2m.c
>> +++ b/xen/arch/arm/p2m.c
>> @@ -136,11 +136,6 @@ void p2m_restore_state(struct vcpu *n)
>>      WRITE_SYSREG64(p2m->vttbr, VTTBR_EL2);
>>      isb();
>>
>> -    if ( is_32bit_domain(n->domain) )
>> -        n->arch.hcr_el2 &= ~HCR_RW;
>> -    else
>> -        n->arch.hcr_el2 |= HCR_RW;
>> -
>>      WRITE_SYSREG(n->arch.sctlr, SCTLR_EL1);
>>      isb();
>>
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 7b1dacc..68185e2 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -268,6 +268,7 @@ struct arch_vcpu
>>
>>  void vcpu_show_execution_state(struct vcpu *);
>>  void vcpu_show_registers(const struct vcpu *);
>> +void vcpu_switch_to_aarch64_mode(struct vcpu *);
>>
>>  unsigned int domain_max_vcpus(const struct domain *);
>>
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 07/18] xen/arm: Introduce a command line parameter for SErrors/Aborts

[Wei Chen]

Hi Stefano,

On 2017/3/15 8:45, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> In order to distinguish guest-generated SErrors from hypervisor-generated
>> SErrors. We have to place SError checking code in every EL1 -> EL2 paths.
>          ^ remove .
>

Ok.

>> That will be an overhead on entries caused by dsb/isb.
>>
>> But not all platforms want to categorize the SErrors. For example, a host
>> that is running with trusted guests. The administrator can confirm that
>> all guests that are running on the host will not trigger such SErrors. In
>> this user scene, we should provide some options to administrator to avoid
>        ^use-case
>

Ok.

>
>> categorizing the SErrors. And then reduce the overhead of dsb/isb.
>                ^ remove   ^ remove
>

Ok.

>
>> We provided following 3 options to administrator to determine how to handle
>> the SErrors:
>>
>> * `diverse`:
>>   The hypervisor will distinguish guest SErrors from hypervisor SErrors.
>>   The guest generated SErrors will be forwarded to guests, the hypervisor
>>   generated SErrors will cause the whole system crash.
>>   It requires:
>>   1. Place dsb/isb on all EL1 -> EL2 trap entries to categorize SErrors
>>      correctly.
>>   2. Place dsb/isb on EL2 -> EL1 return paths to prevent slipping hypervisor
>>      SErrors to guests.
>>   3. Place dsb/isb in context switch to isolate the SErrors between 2 vCPUs.
>>
>> * `forward`:
>>   The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
>>   All SErrors will be forwarded to guests, except the SErrors generated when
>>   idle vCPU is running. The idle domain doesn't have the ability to hanle the
>>   SErrors, so we have to crash the whole system when we get SErros with idle
>>   vCPU. This option will avoid most overhead of the dsb/isb, except the dsb/isb
>>   in context switch which is used to isolate the SErrors between 2 vCPUs.
>>
>> * `panic`:
>>   The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
>>   All SErrors will crash the whole system. This option will avoid all overhead
>>   of the dsb/isb.
>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>
>> ---
>> About adding dsb/isb to prevent slipping Hypervisor SErrors to Guests if the
>> selected option is "diverse". Some Hypervisor SErrors could not be avoid by
>> software, for example ECC Error. But I don't know whether it's worth adding
>> the overhead by default.
>> ---
>>  docs/misc/xen-command-line.markdown | 44 +++++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/traps.c                | 19 ++++++++++++++++
>>  2 files changed, 63 insertions(+)
>>
>> diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
>> index 4daf5b5..79554ce 100644
>> --- a/docs/misc/xen-command-line.markdown
>> +++ b/docs/misc/xen-command-line.markdown
>> @@ -1481,6 +1481,50 @@ enabling more sockets and cores to go into deeper sleep states.
>>
>>  Set the serial transmit buffer size.
>>
>> +### serrors (ARM)
>> +> `= diverse | forward | panic`
>> +
>> +> Default: `diverse`
>> +
>> +This parameter is provided to administrator to determine how to handle the
>> +SErrors.
>
>   This parameter is provided to administrators to determine how the
>   hypervisors handle SErrors.
>

Thanks for reorganization.

>
>> +In order to distinguish guest-generated SErrors from hypervisor-generated
>> +SErrors. We have to place SError checking code in every EL1 -> EL2 paths.
>           ^remove .
>

Ok.

>> +That will be an overhead on entries caused by dsb/isb. But not all platforms
>
>    That will cause overhead on entries due to dsb/isb. However, not all platforms
>

Thanks.

>> +need to categorize the SErrors. For example, a host that is running with
>                        ^ remove the
>

Ok.

>> +trusted guests. The administrator can confirm that all guests that are
>> +running on the host will not trigger such SErrors. In this case, the
>> +administrator can use this parameter to skip categorizing the SErrors. And
>                                                              ^ remove   ^ remove
>

Ok.

>> +reduce the overhead of dsb/isb.
>> +
>> +We provided following 3 options to administrator to determine how to handle
>               ^ the following         ^ administrators
>

Ok.

>> +the SErrors:
>    ^ remove the

Ok.

>
>
>> +
>> +* `diverse`:
>> +  The hypervisor will distinguish guest SErrors from hypervisor SErrors.
>> +  The guest generated SErrors will be forwarded to guests, the hypervisor
>> +  generated SErrors will cause the whole system crash.
>                                                   ^ to crash
>

Ok.

>> +  It requires:
>> +  1. Place dsb/isb on all EL1 -> EL2 trap entries to categorize SErrors
>         ^ remove Place

Ok.

>> +     correctly.
>> +  2. Place dsb/isb on EL2 -> EL1 return paths to prevent slipping hypervisor
>         ^ remove Place

Ok.

>> +     SErrors to guests.
>> +  3. Place dsb/isb in context switch to isolate the SErrors between 2 vCPUs.
>         ^ remove Place                             ^ remove the

Ok.

>> +
>> +* `forward`:
>> +  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
>> +  All SErrors will be forwarded to guests, except the SErrors generated when
>> +  idle vCPU is running. The idle domain doesn't have the ability to hanle the
>      ^ the idle                                                        ^ handle ^ remove the
>

Ok.

>> +  SErrors, so we have to crash the whole system when we get SErros with idle
>                                                                           ^ the idle
>

Ok.

>> +  vCPU. This option will avoid most overhead of the dsb/isb, except the dsb/isb
>> +  in context switch which is used to isolate the SErrors between 2 vCPUs.
>> +
>> +* `panic`:
>> +  The hypervisor will not distinguish guest SErrors from hypervisor SErrors.
>> +  All SErrors will crash the whole system. This option will avoid all overhead
>> +  of the dsb/isb.
>
>     of the dsb/isb pairs.
>

Ok.

> Please make these changes to the commit message too, when applicable.
> With these changes:
>

I will do it in next version.

> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>
>
>>  ### smap
>>  > `= <boolean> | hvm`
>>
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index e425832..5e31699 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -115,6 +115,25 @@ static void __init parse_vwfi(const char *s)
>>  }
>>  custom_param("vwfi", parse_vwfi);
>>
>> +static enum {
>> +    SERRORS_DIVERSE,
>> +    SERRORS_FORWARD,
>> +    SERRORS_PANIC,
>> +} serrors_op;
>> +
>> +static void __init parse_serrors_behavior(const char *str)
>> +{
>> +    if ( !strcmp(str, "forward") )
>> +        serrors_op = SERRORS_FORWARD;
>> +    else if ( !strcmp(str, "panic") )
>> +        serrors_op = SERRORS_PANIC;
>> +    else
>> +        serrors_op = SERRORS_DIVERSE;
>> +
>> +    return;
>> +}
>> +custom_param("serrors", parse_serrors_behavior);
>> +
>>  register_t get_default_hcr_flags(void)
>>  {
>>      return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/18] xen/arm: Introduce a helper to get default HCR_EL2 flags

[Julien Grall]

On 15/03/17 07:19, Wei Chen wrote:
> Hi Stefano,

Hello,

> On 2017/3/15 8:24, Stefano Stabellini wrote:
>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>> We want to add HCR_EL2 register to Xen context switch. And each copy
>>> of HCR_EL2 in vcpu structure will be initialized with the same set
>>> of trap flags as the HCR_EL2 register. We introduce a helper here to
>>> represent these flags to be reused easily.
>>>
>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>> ---
>>>  xen/arch/arm/traps.c            | 11 ++++++++---
>>>  xen/include/asm-arm/processor.h |  2 ++
>>>  2 files changed, 10 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>>> index 614501f..d343c66 100644
>>> --- a/xen/arch/arm/traps.c
>>> +++ b/xen/arch/arm/traps.c
>>> @@ -115,6 +115,13 @@ static void __init parse_vwfi(const char *s)
>>>  }
>>>  custom_param("vwfi", parse_vwfi);
>>>
>>> +register_t get_default_hcr_flags(void)
>>> +{
>>> +    return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
>>> +             (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
>>> +             HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB);
>>> +}
>>
>> I haven't finished reading this series yet, but I would make this a
>> static inline function if possible
>>
>
> I had considered to use static inline before. But it must move the
>
> static enum {
>         TRAP,
>         NATIVE,
> } vwfi;
>
> to the header file at the same time. But get_default_hcr_flags would
> not be used frequently. So I thought it didn't have enough value to
> change a less relevant code to make this function become static inline.

Looking at the spec, HCR_EL2 is controlling the behavior of the VM. We 
only need to ensure this to be set before going to EL1/EL0. Note that 
the reset value of some register are architecturally UNKNOWN, but I 
don't think we care here.

So I would suggest to drop the setting in init_traps and only rely on 
the one in when returning to the guest. And therefore this function 
could be moved in domain.c

Any opinions?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Julien Grall]

Hi Wei,

On 15/03/17 08:34, Wei Chen wrote:
> On 2017/3/15 8:25, Stefano Stabellini wrote:
>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>> Different domains may have different HCR_EL2 flags. For example, the
>>> 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
>>> we give each domain a default HCR_EL2 value and save it in the VCPU's
>>> context.
>>>
>>> HCR_EL2 register has only one bit can be updated automatically without
>>> explicit write (HCR_VSE). But we haven't used this bit currently, so
>>> we can consider that the HCR_EL2 register will not be modified while
>>> the guest is running. So save the HCR_EL2 while guest exiting to
>>> hypervisor is not neccessary. We just have to restore this register for
>>> each VCPU while leaving hypervisor.
>>>
>>> We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
>>> ctxt_switch_to. Because the leave_hypervisor_tail is the closest place
>>> to the exception return. In this case, we don't need to warrant the
>>> HCR_EL2 will not be changed between ctxt_switch_to and exception return.
>>
>> Please explain a bit better why it is good to restore HCR_EL2 in
>> leave_hypervisor_tail. Why is it a good thing that is closer to the
>> exception return? What can be the cause of a change in HCR_EL2?
>>
>
> Ok, I will try to improve it in next version. In current Xen code, I
> can't see any code would change the HCR_EL2 between ctxt_switch_to and
> return_from_trap. But my concern is that, in the future, if someone
> want to insert some HCR_EL2 change code between them, we can't guarantee
> he will restore correct HCR_EL2 value for current vcpu.

Well, the purpose of this series is to inject a virtual SError to the 
guest when a host SError is happening. The host SError will be received 
in the hypervisor whilst the vCPU is running and no context switch (e.g 
call to ctxt_switch) may happen if the scheduler decides to keep the 
vCPU running.

Also, one could argue that HCR_EL2 could be modified on-fly in the 
function but we may have other places in the future which will modify 
HCR_EL2. For instance, this would be the case when the monitor app will 
gain support of inspecting some registers.

So we want a single place to restore HCR_EL2. And leave_hypervisor_tail 
is the best place for that.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/18] xen/arm: Introduce a helper to get default HCR_EL2 flags

[Stefano Stabellini]

On Wed, 15 Mar 2017, Julien Grall wrote:
> On 15/03/17 07:19, Wei Chen wrote:
> > Hi Stefano,
> 
> Hello,
> 
> > On 2017/3/15 8:24, Stefano Stabellini wrote:
> > > On Mon, 13 Mar 2017, Wei Chen wrote:
> > > > We want to add HCR_EL2 register to Xen context switch. And each copy
> > > > of HCR_EL2 in vcpu structure will be initialized with the same set
> > > > of trap flags as the HCR_EL2 register. We introduce a helper here to
> > > > represent these flags to be reused easily.
> > > > 
> > > > Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> > > > ---
> > > >  xen/arch/arm/traps.c            | 11 ++++++++---
> > > >  xen/include/asm-arm/processor.h |  2 ++
> > > >  2 files changed, 10 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> > > > index 614501f..d343c66 100644
> > > > --- a/xen/arch/arm/traps.c
> > > > +++ b/xen/arch/arm/traps.c
> > > > @@ -115,6 +115,13 @@ static void __init parse_vwfi(const char *s)
> > > >  }
> > > >  custom_param("vwfi", parse_vwfi);
> > > > 
> > > > +register_t get_default_hcr_flags(void)
> > > > +{
> > > > +    return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
> > > > +             (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
> > > > +             HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB);
> > > > +}
> > > 
> > > I haven't finished reading this series yet, but I would make this a
> > > static inline function if possible
> > > 
> > 
> > I had considered to use static inline before. But it must move the
> > 
> > static enum {
> >         TRAP,
> >         NATIVE,
> > } vwfi;
> > 
> > to the header file at the same time. But get_default_hcr_flags would
> > not be used frequently. So I thought it didn't have enough value to
> > change a less relevant code to make this function become static inline.
> 
> Looking at the spec, HCR_EL2 is controlling the behavior of the VM. We only
> need to ensure this to be set before going to EL1/EL0. Note that the reset
> value of some register are architecturally UNKNOWN, but I don't think we care
> here.
> 
> So I would suggest to drop the setting in init_traps and only rely on the one
> in when returning to the guest. And therefore this function could be moved in
> domain.c
> 
> Any opinions?

Fine by me

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/18] xen/arm: Introduce a helper to get default HCR_EL2 flags

[Wei Chen]

Hi Julien,

On 2017/3/15 19:01, Julien Grall wrote:
> On 15/03/17 07:19, Wei Chen wrote:
>> Hi Stefano,
>
> Hello,
>
>> On 2017/3/15 8:24, Stefano Stabellini wrote:
>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>> We want to add HCR_EL2 register to Xen context switch. And each copy
>>>> of HCR_EL2 in vcpu structure will be initialized with the same set
>>>> of trap flags as the HCR_EL2 register. We introduce a helper here to
>>>> represent these flags to be reused easily.
>>>>
>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>> ---
>>>>  xen/arch/arm/traps.c            | 11 ++++++++---
>>>>  xen/include/asm-arm/processor.h |  2 ++
>>>>  2 files changed, 10 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>>>> index 614501f..d343c66 100644
>>>> --- a/xen/arch/arm/traps.c
>>>> +++ b/xen/arch/arm/traps.c
>>>> @@ -115,6 +115,13 @@ static void __init parse_vwfi(const char *s)
>>>>  }
>>>>  custom_param("vwfi", parse_vwfi);
>>>>
>>>> +register_t get_default_hcr_flags(void)
>>>> +{
>>>> +    return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
>>>> +             (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
>>>> +             HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB);
>>>> +}
>>>
>>> I haven't finished reading this series yet, but I would make this a
>>> static inline function if possible
>>>
>>
>> I had considered to use static inline before. But it must move the
>>
>> static enum {
>>         TRAP,
>>         NATIVE,
>> } vwfi;
>>
>> to the header file at the same time. But get_default_hcr_flags would
>> not be used frequently. So I thought it didn't have enough value to
>> change a less relevant code to make this function become static inline.
>
> Looking at the spec, HCR_EL2 is controlling the behavior of the VM. We
> only need to ensure this to be set before going to EL1/EL0. Note that
> the reset value of some register are architecturally UNKNOWN, but I
> don't think we care here.
>
> So I would suggest to drop the setting in init_traps and only rely on
> the one in when returning to the guest. And therefore this function
> could be moved in domain.c
>
> Any opinions?
>

It seems good. I will do some test, if everything pass, I will do this
change in next version.

> Cheers,
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Wei Chen]

Hi Julien,

On 2017/3/15 19:12, Julien Grall wrote:
> Hi Wei,
>
> On 15/03/17 08:34, Wei Chen wrote:
>> On 2017/3/15 8:25, Stefano Stabellini wrote:
>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>> Different domains may have different HCR_EL2 flags. For example, the
>>>> 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
>>>> we give each domain a default HCR_EL2 value and save it in the VCPU's
>>>> context.
>>>>
>>>> HCR_EL2 register has only one bit can be updated automatically without
>>>> explicit write (HCR_VSE). But we haven't used this bit currently, so
>>>> we can consider that the HCR_EL2 register will not be modified while
>>>> the guest is running. So save the HCR_EL2 while guest exiting to
>>>> hypervisor is not neccessary. We just have to restore this register for
>>>> each VCPU while leaving hypervisor.
>>>>
>>>> We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
>>>> ctxt_switch_to. Because the leave_hypervisor_tail is the closest place
>>>> to the exception return. In this case, we don't need to warrant the
>>>> HCR_EL2 will not be changed between ctxt_switch_to and exception return.
>>>
>>> Please explain a bit better why it is good to restore HCR_EL2 in
>>> leave_hypervisor_tail. Why is it a good thing that is closer to the
>>> exception return? What can be the cause of a change in HCR_EL2?
>>>
>>
>> Ok, I will try to improve it in next version. In current Xen code, I
>> can't see any code would change the HCR_EL2 between ctxt_switch_to and
>> return_from_trap. But my concern is that, in the future, if someone
>> want to insert some HCR_EL2 change code between them, we can't guarantee
>> he will restore correct HCR_EL2 value for current vcpu.
>
> Well, the purpose of this series is to inject a virtual SError to the
> guest when a host SError is happening. The host SError will be received
> in the hypervisor whilst the vCPU is running and no context switch (e.g
> call to ctxt_switch) may happen if the scheduler decides to keep the
> vCPU running.
>
> Also, one could argue that HCR_EL2 could be modified on-fly in the
> function but we may have other places in the future which will modify
> HCR_EL2. For instance, this would be the case when the monitor app will
> gain support of inspecting some registers.
>
> So we want a single place to restore HCR_EL2. And leave_hypervisor_tail
> is the best place for that.
>

Thanks! I will reference this comment to the commit message in next
version.

> Cheers,
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 05/18] xen/arm: Save ESR_EL2 to avoid using mismatched value in syndrome check

[Julien Grall]

Hi Wei,

On 03/13/2017 10:55 AM, Wei Chen wrote:
> Xen will do exception syndrome check while some types of exception
> take place in EL2. The syndrome check code read the ESR_EL2 register
> directly, but in some situation this register maybe overridden by
> nested exception.
>
> For example, if we re-enable IRQ before reading ESR_EL2 which means
> Xen will enter in IRQ exception mode and return the processor with

s/will/may/

> clobbered ESR_EL2 (See ARM ARM DDI 0487A.j D7.2.25)
>
> In this case the guest exception syndrome has been overridden, we will
> check the syndrome for guest sync exception with a mismatched ESR_EL2

s/mismatched/incorrect/ I think

> value. So we want to save ESR_EL2 to cpu_user_regs as soon as the
> exception takes place in EL2 to avoid using a mismatched syndrome value.

Ditto.

>
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/arm32/asm-offsets.c      |  1 +
>  xen/arch/arm/arm32/entry.S            |  3 +++
>  xen/arch/arm/arm64/asm-offsets.c      |  1 +
>  xen/arch/arm/arm64/entry.S            | 13 +++++++++----
>  xen/arch/arm/traps.c                  |  2 +-
>  xen/include/asm-arm/arm32/processor.h |  2 +-
>  xen/include/asm-arm/arm64/processor.h | 10 ++++++++--
>  7 files changed, 24 insertions(+), 8 deletions(-)
>
> diff --git a/xen/arch/arm/arm32/asm-offsets.c b/xen/arch/arm/arm32/asm-offsets.c
> index f8e6b53..5b543ab 100644
> --- a/xen/arch/arm/arm32/asm-offsets.c
> +++ b/xen/arch/arm/arm32/asm-offsets.c
> @@ -26,6 +26,7 @@ void __dummy__(void)
>     OFFSET(UREGS_lr, struct cpu_user_regs, lr);
>     OFFSET(UREGS_pc, struct cpu_user_regs, pc);
>     OFFSET(UREGS_cpsr, struct cpu_user_regs, cpsr);
> +   OFFSET(UREGS_hsr, struct cpu_user_regs, hsr);
>
>     OFFSET(UREGS_LR_usr, struct cpu_user_regs, lr_usr);
>     OFFSET(UREGS_SP_usr, struct cpu_user_regs, sp_usr);
> diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
> index 2a6f4f0..2187226 100644
> --- a/xen/arch/arm/arm32/entry.S
> +++ b/xen/arch/arm/arm32/entry.S
> @@ -23,6 +23,9 @@
>          add r11, sp, #UREGS_kernel_sizeof+4;                            \
>          str r11, [sp, #UREGS_sp];                                       \
>                                                                          \
> +        mrc CP32(r11, HSR);             /* Save exception syndrome */   \
> +        str r11, [sp, #UREGS_hsr];                                      \
> +                                                                        \
>          mrs r11, SPSR_hyp;                                              \
>          str r11, [sp, #UREGS_cpsr];                                     \
>          and r11, #PSR_MODE_MASK;                                        \
> diff --git a/xen/arch/arm/arm64/asm-offsets.c b/xen/arch/arm/arm64/asm-offsets.c
> index 69ea92a..ce24e44 100644
> --- a/xen/arch/arm/arm64/asm-offsets.c
> +++ b/xen/arch/arm/arm64/asm-offsets.c
> @@ -27,6 +27,7 @@ void __dummy__(void)
>     OFFSET(UREGS_SP, struct cpu_user_regs, sp);
>     OFFSET(UREGS_PC, struct cpu_user_regs, pc);
>     OFFSET(UREGS_CPSR, struct cpu_user_regs, cpsr);
> +   OFFSET(UREGS_ESR_el2, struct cpu_user_regs, hsr);
>
>     OFFSET(UREGS_SPSR_el1, struct cpu_user_regs, spsr_el1);
>
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index c181b5e..02802c0 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -121,9 +121,13 @@ lr      .req    x30             // link register
>
>          stp     lr, x21, [sp, #UREGS_LR]
>
> -        mrs     x22, elr_el2
> -        mrs     x23, spsr_el2
> -        stp     x22, x23, [sp, #UREGS_PC]
> +        mrs     x21, elr_el2
> +        str     x21, [sp, #UREGS_PC]

Please explain the commit message you modify the lines above ...

> +
> +        add     x21, sp, #UREGS_CPSR
> +        mrs     x22, spsr_el2
> +        mrs     x23, esr_el2
> +        stp     w22, w23, [x21]
>
>          .endm
>
> @@ -307,7 +311,8 @@ ENTRY(return_to_new_vcpu64)
>  return_from_trap:
>          msr     daifset, #2 /* Mask interrupts */
>
> -        ldp     x21, x22, [sp, #UREGS_PC]       // load ELR, SPSR
> +        ldr     x21, [sp, #UREGS_PC]            // load ELR
> +        ldr     w22, [sp, #UREGS_CPSR]          // load SPSR

as long as those one.

>
>          pop     x0, x1
>          pop     x2, x3

[...]

> diff --git a/xen/include/asm-arm/arm64/processor.h b/xen/include/asm-arm/arm64/processor.h
> index b0726ff..d381428 100644
> --- a/xen/include/asm-arm/arm64/processor.h
> +++ b/xen/include/asm-arm/arm64/processor.h
> @@ -65,9 +65,15 @@ struct cpu_user_regs
>
>      /* Return address and mode */
>      __DECL_REG(pc,           pc32);             /* ELR_EL2 */
> +    /*
> +     * Be careful for 32-bit registers, if we use xN to save 32-bit register
> +     * to stack, its next field on stack will be overridden.
> +     * For example, if we use xN to save SPSR_EL2 to stack will override the
> +     * hsr field on stack.
> +     * So, it's better to use wN to save 32-bit registers to stack.
> +     */

This comment is pointless. This is true for any 32-bit register, you 
should use wN unless you now that you have a padding after.

>      uint32_t cpsr;                              /* SPSR_EL2 */
> -
> -    uint32_t pad0; /* Align end of kernel frame. */
> +    uint32_t hsr;                               /* ESR_EL2 */
>
>      /* Outer guest frame only from here on... */
>
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 05/18] xen/arm: Save ESR_EL2 to avoid using mismatched value in syndrome check

[Stefano Stabellini]

On Thu, 16 Mar 2017, Julien Grall wrote:
> Hi Wei,
> 
> On 03/13/2017 10:55 AM, Wei Chen wrote:
> > Xen will do exception syndrome check while some types of exception
> > take place in EL2. The syndrome check code read the ESR_EL2 register
> > directly, but in some situation this register maybe overridden by
> > nested exception.
> > 
> > For example, if we re-enable IRQ before reading ESR_EL2 which means
> > Xen will enter in IRQ exception mode and return the processor with
> 
> s/will/may/
> 
> > clobbered ESR_EL2 (See ARM ARM DDI 0487A.j D7.2.25)
> > 
> > In this case the guest exception syndrome has been overridden, we will
> > check the syndrome for guest sync exception with a mismatched ESR_EL2
> 
> s/mismatched/incorrect/ I think
> 
> > value. So we want to save ESR_EL2 to cpu_user_regs as soon as the
> > exception takes place in EL2 to avoid using a mismatched syndrome value.
> 
> Ditto.

Please address Julien's comments. The code looks good.


> > 
> > Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> > ---
> >  xen/arch/arm/arm32/asm-offsets.c      |  1 +
> >  xen/arch/arm/arm32/entry.S            |  3 +++
> >  xen/arch/arm/arm64/asm-offsets.c      |  1 +
> >  xen/arch/arm/arm64/entry.S            | 13 +++++++++----
> >  xen/arch/arm/traps.c                  |  2 +-
> >  xen/include/asm-arm/arm32/processor.h |  2 +-
> >  xen/include/asm-arm/arm64/processor.h | 10 ++++++++--
> >  7 files changed, 24 insertions(+), 8 deletions(-)
> > 
> > diff --git a/xen/arch/arm/arm32/asm-offsets.c
> > b/xen/arch/arm/arm32/asm-offsets.c
> > index f8e6b53..5b543ab 100644
> > --- a/xen/arch/arm/arm32/asm-offsets.c
> > +++ b/xen/arch/arm/arm32/asm-offsets.c
> > @@ -26,6 +26,7 @@ void __dummy__(void)
> >     OFFSET(UREGS_lr, struct cpu_user_regs, lr);
> >     OFFSET(UREGS_pc, struct cpu_user_regs, pc);
> >     OFFSET(UREGS_cpsr, struct cpu_user_regs, cpsr);
> > +   OFFSET(UREGS_hsr, struct cpu_user_regs, hsr);
> > 
> >     OFFSET(UREGS_LR_usr, struct cpu_user_regs, lr_usr);
> >     OFFSET(UREGS_SP_usr, struct cpu_user_regs, sp_usr);
> > diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
> > index 2a6f4f0..2187226 100644
> > --- a/xen/arch/arm/arm32/entry.S
> > +++ b/xen/arch/arm/arm32/entry.S
> > @@ -23,6 +23,9 @@
> >          add r11, sp, #UREGS_kernel_sizeof+4;                            \
> >          str r11, [sp, #UREGS_sp];                                       \
> >                                                                          \
> > +        mrc CP32(r11, HSR);             /* Save exception syndrome */   \
> > +        str r11, [sp, #UREGS_hsr];                                      \
> > +                                                                        \
> >          mrs r11, SPSR_hyp;                                              \
> >          str r11, [sp, #UREGS_cpsr];                                     \
> >          and r11, #PSR_MODE_MASK;                                        \
> > diff --git a/xen/arch/arm/arm64/asm-offsets.c
> > b/xen/arch/arm/arm64/asm-offsets.c
> > index 69ea92a..ce24e44 100644
> > --- a/xen/arch/arm/arm64/asm-offsets.c
> > +++ b/xen/arch/arm/arm64/asm-offsets.c
> > @@ -27,6 +27,7 @@ void __dummy__(void)
> >     OFFSET(UREGS_SP, struct cpu_user_regs, sp);
> >     OFFSET(UREGS_PC, struct cpu_user_regs, pc);
> >     OFFSET(UREGS_CPSR, struct cpu_user_regs, cpsr);
> > +   OFFSET(UREGS_ESR_el2, struct cpu_user_regs, hsr);
> > 
> >     OFFSET(UREGS_SPSR_el1, struct cpu_user_regs, spsr_el1);
> > 
> > diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> > index c181b5e..02802c0 100644
> > --- a/xen/arch/arm/arm64/entry.S
> > +++ b/xen/arch/arm/arm64/entry.S
> > @@ -121,9 +121,13 @@ lr      .req    x30             // link register
> > 
> >          stp     lr, x21, [sp, #UREGS_LR]
> > 
> > -        mrs     x22, elr_el2
> > -        mrs     x23, spsr_el2
> > -        stp     x22, x23, [sp, #UREGS_PC]
> > +        mrs     x21, elr_el2
> > +        str     x21, [sp, #UREGS_PC]
> 
> Please explain the commit message you modify the lines above ...
> 
> > +
> > +        add     x21, sp, #UREGS_CPSR
> > +        mrs     x22, spsr_el2
> > +        mrs     x23, esr_el2
> > +        stp     w22, w23, [x21]
> > 
> >          .endm
> > 
> > @@ -307,7 +311,8 @@ ENTRY(return_to_new_vcpu64)
> >  return_from_trap:
> >          msr     daifset, #2 /* Mask interrupts */
> > 
> > -        ldp     x21, x22, [sp, #UREGS_PC]       // load ELR, SPSR
> > +        ldr     x21, [sp, #UREGS_PC]            // load ELR
> > +        ldr     w22, [sp, #UREGS_CPSR]          // load SPSR
> 
> as long as those one.
> 
> > 
> >          pop     x0, x1
> >          pop     x2, x3
> 
> [...]
> 
> > diff --git a/xen/include/asm-arm/arm64/processor.h
> > b/xen/include/asm-arm/arm64/processor.h
> > index b0726ff..d381428 100644
> > --- a/xen/include/asm-arm/arm64/processor.h
> > +++ b/xen/include/asm-arm/arm64/processor.h
> > @@ -65,9 +65,15 @@ struct cpu_user_regs
> > 
> >      /* Return address and mode */
> >      __DECL_REG(pc,           pc32);             /* ELR_EL2 */
> > +    /*
> > +     * Be careful for 32-bit registers, if we use xN to save 32-bit
> > register
> > +     * to stack, its next field on stack will be overridden.
> > +     * For example, if we use xN to save SPSR_EL2 to stack will override
> > the
> > +     * hsr field on stack.
> > +     * So, it's better to use wN to save 32-bit registers to stack.
> > +     */
> 
> This comment is pointless. This is true for any 32-bit register, you should
> use wN unless you now that you have a padding after.
> 
> >      uint32_t cpsr;                              /* SPSR_EL2 */
> > -
> > -    uint32_t pad0; /* Align end of kernel frame. */
> > +    uint32_t hsr;                               /* ESR_EL2 */
> > 
> >      /* Outer guest frame only from here on... */
> > 
> > 
> 
> Cheers,
> 
> -- 
> Julien Grall
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Stefano Stabellini]

On Wed, 15 Mar 2017, Julien Grall wrote:
> Hi Wei,
> 
> On 15/03/17 08:34, Wei Chen wrote:
> > On 2017/3/15 8:25, Stefano Stabellini wrote:
> > > On Mon, 13 Mar 2017, Wei Chen wrote:
> > > > Different domains may have different HCR_EL2 flags. For example, the
> > > > 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
> > > > we give each domain a default HCR_EL2 value and save it in the VCPU's
> > > > context.
> > > > 
> > > > HCR_EL2 register has only one bit can be updated automatically without
> > > > explicit write (HCR_VSE). But we haven't used this bit currently, so
> > > > we can consider that the HCR_EL2 register will not be modified while
> > > > the guest is running. So save the HCR_EL2 while guest exiting to
> > > > hypervisor is not neccessary. We just have to restore this register for
> > > > each VCPU while leaving hypervisor.
> > > > 
> > > > We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
> > > > ctxt_switch_to. Because the leave_hypervisor_tail is the closest place
> > > > to the exception return. In this case, we don't need to warrant the
> > > > HCR_EL2 will not be changed between ctxt_switch_to and exception return.
> > > 
> > > Please explain a bit better why it is good to restore HCR_EL2 in
> > > leave_hypervisor_tail. Why is it a good thing that is closer to the
> > > exception return? What can be the cause of a change in HCR_EL2?
> > > 
> > 
> > Ok, I will try to improve it in next version. In current Xen code, I
> > can't see any code would change the HCR_EL2 between ctxt_switch_to and
> > return_from_trap. But my concern is that, in the future, if someone
> > want to insert some HCR_EL2 change code between them, we can't guarantee
> > he will restore correct HCR_EL2 value for current vcpu.
> 
> Well, the purpose of this series is to inject a virtual SError to the guest
> when a host SError is happening. The host SError will be received in the
> hypervisor whilst the vCPU is running and no context switch (e.g call to
> ctxt_switch) may happen if the scheduler decides to keep the vCPU running.
> 
> Also, one could argue that HCR_EL2 could be modified on-fly in the function
> but we may have other places in the future which will modify HCR_EL2. For
> instance, this would be the case when the monitor app will gain support of
> inspecting some registers.
> 
> So we want a single place to restore HCR_EL2. And leave_hypervisor_tail is the
> best place for that.

You say that we want a single place to restore HCR_EL2, but this patch
introduces two places, one is p2m_restore_state, the other is
leave_hypervisor_tail. Are you saying Wei should remove the HCR_EL2
restore in p2m_restore_state and just keep the one in
leave_hypervisor_tail? 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Stefano Stabellini]

On Wed, 15 Mar 2017, Wei Chen wrote:
> Hi Stefano,
> 
> On 2017/3/15 8:25, Stefano Stabellini wrote:
> > On Mon, 13 Mar 2017, Wei Chen wrote:
> >> The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
> >> when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
> >> DomU in vcpu_initialise, we didn't know the domain's address size
> >> information. We had to use compatible flags to initialize HCR_EL2,
> >> and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
> >> at every context switch.
> >>
> >> But, after we added the HCR_EL2 to vcpu's context, this behaviour
> >> seems a little fussy. We can update the HCR_RW bit in vcpu's context
> >> as soon as we get the domain's address size to avoid setting/clearing
> >> HCR_RW at every context switch.
> >>
> >> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> >> ---
> >>  xen/arch/arm/arm64/domctl.c  | 6 ++++++
> >>  xen/arch/arm/domain.c        | 5 +++++
> >>  xen/arch/arm/domain_build.c  | 7 +++++++
> >>  xen/arch/arm/p2m.c           | 5 -----
> >>  xen/include/asm-arm/domain.h | 1 +
> >>  5 files changed, 19 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
> >> index 44e1e7b..ab8781f 100644
> >> --- a/xen/arch/arm/arm64/domctl.c
> >> +++ b/xen/arch/arm/arm64/domctl.c
> >> @@ -14,6 +14,8 @@
> >>
> >>  static long switch_mode(struct domain *d, enum domain_type type)
> >>  {
> >> +    struct vcpu *v;
> >> +
> >>      if ( d == NULL )
> >>          return -EINVAL;
> >>      if ( d->tot_pages != 0 )
> >> @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum domain_type type)
> >>
> >>      d->arch.type = type;
> >>
> >> +    if ( is_64bit_domain(d) )
> >> +        for_each_vcpu(d, v)
> >> +            vcpu_switch_to_aarch64_mode(v);
> >> +
> >>      return 0;
> >>  }
> >>
> >> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >> index 5d18bb0..69c2854 100644
> >> --- a/xen/arch/arm/domain.c
> >> +++ b/xen/arch/arm/domain.c
> >> @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
> >>      free_xenheap_pages(v->arch.stack, STACK_ORDER);
> >>  }
> >>
> >> +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
> >> +{
> >> +    v->arch.hcr_el2 |= HCR_RW;
> >> +}
> >
> > if possible, I would write it as a static inline function
> >
> 
> I had tried to write it as a static inline function in asm/domain.h
> But while the first source file (arm64/asm-offsets.c) include
> xen/sched.h wanted to compile this inline function it could not find
> 'struct vcpu'. Because the xen/sched.h included the asm/domain.h
> but defined the vcpu type later. Even though we had included the
> xen/sched.h in asm/domain.h already.

It might be too complex to disentangle. In this case, there isn't much
type safety to be gained by using a static inline over a macro, so it
would be OK to use a macro for this.

 
> >
> >>  int arch_domain_create(struct domain *d, unsigned int domcr_flags,
> >>                         struct xen_arch_domainconfig *config)
> >>  {
> >> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> >> index de59e5f..3abacc0 100644
> >> --- a/xen/arch/arm/domain_build.c
> >> +++ b/xen/arch/arm/domain_build.c
> >> @@ -2148,6 +2148,10 @@ int construct_dom0(struct domain *d)
> >>          return -EINVAL;
> >>      }
> >>      d->arch.type = kinfo.type;
> >> +
> >> +    if ( is_64bit_domain(d) )
> >> +        vcpu_switch_to_aarch64_mode(v);
> >> +
> >>  #endif
> >>
> >>      allocate_memory(d, &kinfo);
> >> @@ -2240,6 +2244,9 @@ int construct_dom0(struct domain *d)
> >>              printk("Failed to allocate dom0 vcpu %d on pcpu %d\n", i, cpu);
> >>              break;
> >>          }
> >> +
> >> +        if ( is_64bit_domain(d) )
> >> +            vcpu_switch_to_aarch64_mode(d->vcpu[i]);
> >>      }
> >>
> >>      return 0;
> >> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> >> index c49bfa6..1cba0d0 100644
> >> --- a/xen/arch/arm/p2m.c
> >> +++ b/xen/arch/arm/p2m.c
> >> @@ -136,11 +136,6 @@ void p2m_restore_state(struct vcpu *n)
> >>      WRITE_SYSREG64(p2m->vttbr, VTTBR_EL2);
> >>      isb();
> >>
> >> -    if ( is_32bit_domain(n->domain) )
> >> -        n->arch.hcr_el2 &= ~HCR_RW;
> >> -    else
> >> -        n->arch.hcr_el2 |= HCR_RW;
> >> -
> >>      WRITE_SYSREG(n->arch.sctlr, SCTLR_EL1);
> >>      isb();
> >>
> >> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> >> index 7b1dacc..68185e2 100644
> >> --- a/xen/include/asm-arm/domain.h
> >> +++ b/xen/include/asm-arm/domain.h
> >> @@ -268,6 +268,7 @@ struct arch_vcpu
> >>
> >>  void vcpu_show_execution_state(struct vcpu *);
> >>  void vcpu_show_registers(const struct vcpu *);
> >> +void vcpu_switch_to_aarch64_mode(struct vcpu *);
> >>
> >>  unsigned int domain_max_vcpus(const struct domain *);


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Julien Grall]

Hi Stefano

On 03/16/2017 10:33 PM, Stefano Stabellini wrote:
> On Wed, 15 Mar 2017, Julien Grall wrote:
>> Hi Wei,
>>
>> On 15/03/17 08:34, Wei Chen wrote:
>>> On 2017/3/15 8:25, Stefano Stabellini wrote:
>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>>> Different domains may have different HCR_EL2 flags. For example, the
>>>>> 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
>>>>> we give each domain a default HCR_EL2 value and save it in the VCPU's
>>>>> context.
>>>>>
>>>>> HCR_EL2 register has only one bit can be updated automatically without
>>>>> explicit write (HCR_VSE). But we haven't used this bit currently, so
>>>>> we can consider that the HCR_EL2 register will not be modified while
>>>>> the guest is running. So save the HCR_EL2 while guest exiting to
>>>>> hypervisor is not neccessary. We just have to restore this register for
>>>>> each VCPU while leaving hypervisor.
>>>>>
>>>>> We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
>>>>> ctxt_switch_to. Because the leave_hypervisor_tail is the closest place
>>>>> to the exception return. In this case, we don't need to warrant the
>>>>> HCR_EL2 will not be changed between ctxt_switch_to and exception return.
>>>>
>>>> Please explain a bit better why it is good to restore HCR_EL2 in
>>>> leave_hypervisor_tail. Why is it a good thing that is closer to the
>>>> exception return? What can be the cause of a change in HCR_EL2?
>>>>
>>>
>>> Ok, I will try to improve it in next version. In current Xen code, I
>>> can't see any code would change the HCR_EL2 between ctxt_switch_to and
>>> return_from_trap. But my concern is that, in the future, if someone
>>> want to insert some HCR_EL2 change code between them, we can't guarantee
>>> he will restore correct HCR_EL2 value for current vcpu.
>>
>> Well, the purpose of this series is to inject a virtual SError to the guest
>> when a host SError is happening. The host SError will be received in the
>> hypervisor whilst the vCPU is running and no context switch (e.g call to
>> ctxt_switch) may happen if the scheduler decides to keep the vCPU running.
>>
>> Also, one could argue that HCR_EL2 could be modified on-fly in the function
>> but we may have other places in the future which will modify HCR_EL2. For
>> instance, this would be the case when the monitor app will gain support of
>> inspecting some registers.
>>
>> So we want a single place to restore HCR_EL2. And leave_hypervisor_tail is the
>> best place for that.
>
> You say that we want a single place to restore HCR_EL2, but this patch
> introduces two places, one is p2m_restore_state, the other is
> leave_hypervisor_tail. Are you saying Wei should remove the HCR_EL2
> restore in p2m_restore_state and just keep the one in
> leave_hypervisor_tail?

p2m_restore_state may be used to switch temporarily to a p2m state. For 
instance for TLB flushing or even doing VA -> PA translation. Though the 
later does not yet work quite well when not using the current vCPU.

Some bits in HCR_EL2 (specifically HCR_EL2.RW) will be used to know how 
to interpret the stage-1 page table as the encoding differ between 
AArch64 and AArch32.

So I think we have to keep the one in p2m_restore_state. But I would 
like to keep the number very limited.

Cheers.

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Julien Grall]

On 03/16/2017 10:40 PM, Stefano Stabellini wrote:
> On Wed, 15 Mar 2017, Wei Chen wrote:
>> Hi Stefano,
>>
>> On 2017/3/15 8:25, Stefano Stabellini wrote:
>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>> The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
>>>> when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
>>>> DomU in vcpu_initialise, we didn't know the domain's address size
>>>> information. We had to use compatible flags to initialize HCR_EL2,
>>>> and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
>>>> at every context switch.
>>>>
>>>> But, after we added the HCR_EL2 to vcpu's context, this behaviour
>>>> seems a little fussy. We can update the HCR_RW bit in vcpu's context
>>>> as soon as we get the domain's address size to avoid setting/clearing
>>>> HCR_RW at every context switch.
>>>>
>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>> ---
>>>>  xen/arch/arm/arm64/domctl.c  | 6 ++++++
>>>>  xen/arch/arm/domain.c        | 5 +++++
>>>>  xen/arch/arm/domain_build.c  | 7 +++++++
>>>>  xen/arch/arm/p2m.c           | 5 -----
>>>>  xen/include/asm-arm/domain.h | 1 +
>>>>  5 files changed, 19 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
>>>> index 44e1e7b..ab8781f 100644
>>>> --- a/xen/arch/arm/arm64/domctl.c
>>>> +++ b/xen/arch/arm/arm64/domctl.c
>>>> @@ -14,6 +14,8 @@
>>>>
>>>>  static long switch_mode(struct domain *d, enum domain_type type)
>>>>  {
>>>> +    struct vcpu *v;
>>>> +
>>>>      if ( d == NULL )
>>>>          return -EINVAL;
>>>>      if ( d->tot_pages != 0 )
>>>> @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum domain_type type)
>>>>
>>>>      d->arch.type = type;
>>>>
>>>> +    if ( is_64bit_domain(d) )
>>>> +        for_each_vcpu(d, v)
>>>> +            vcpu_switch_to_aarch64_mode(v);
>>>> +
>>>>      return 0;
>>>>  }
>>>>
>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>> index 5d18bb0..69c2854 100644
>>>> --- a/xen/arch/arm/domain.c
>>>> +++ b/xen/arch/arm/domain.c
>>>> @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
>>>>      free_xenheap_pages(v->arch.stack, STACK_ORDER);
>>>>  }
>>>>
>>>> +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
>>>> +{
>>>> +    v->arch.hcr_el2 |= HCR_RW;
>>>> +}
>>>
>>> if possible, I would write it as a static inline function
>>>
>>
>> I had tried to write it as a static inline function in asm/domain.h
>> But while the first source file (arm64/asm-offsets.c) include
>> xen/sched.h wanted to compile this inline function it could not find
>> 'struct vcpu'. Because the xen/sched.h included the asm/domain.h
>> but defined the vcpu type later. Even though we had included the
>> xen/sched.h in asm/domain.h already.
>
> It might be too complex to disentangle. In this case, there isn't much
> type safety to be gained by using a static inline over a macro, so it
> would be OK to use a macro for this.

It is not like vCPU will be switch to AArch64 mode often? Only once at 
vCPU creation time.

So what do we gain to switch to static inline or even macro?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Stefano Stabellini]

On Thu, 16 Mar 2017, Julien Grall wrote:
> On 03/16/2017 10:40 PM, Stefano Stabellini wrote:
> > On Wed, 15 Mar 2017, Wei Chen wrote:
> > > Hi Stefano,
> > > 
> > > On 2017/3/15 8:25, Stefano Stabellini wrote:
> > > > On Mon, 13 Mar 2017, Wei Chen wrote:
> > > > > The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
> > > > > when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
> > > > > DomU in vcpu_initialise, we didn't know the domain's address size
> > > > > information. We had to use compatible flags to initialize HCR_EL2,
> > > > > and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
> > > > > at every context switch.
> > > > > 
> > > > > But, after we added the HCR_EL2 to vcpu's context, this behaviour
> > > > > seems a little fussy. We can update the HCR_RW bit in vcpu's context
> > > > > as soon as we get the domain's address size to avoid setting/clearing
> > > > > HCR_RW at every context switch.
> > > > > 
> > > > > Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> > > > > ---
> > > > >  xen/arch/arm/arm64/domctl.c  | 6 ++++++
> > > > >  xen/arch/arm/domain.c        | 5 +++++
> > > > >  xen/arch/arm/domain_build.c  | 7 +++++++
> > > > >  xen/arch/arm/p2m.c           | 5 -----
> > > > >  xen/include/asm-arm/domain.h | 1 +
> > > > >  5 files changed, 19 insertions(+), 5 deletions(-)
> > > > > 
> > > > > diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
> > > > > index 44e1e7b..ab8781f 100644
> > > > > --- a/xen/arch/arm/arm64/domctl.c
> > > > > +++ b/xen/arch/arm/arm64/domctl.c
> > > > > @@ -14,6 +14,8 @@
> > > > > 
> > > > >  static long switch_mode(struct domain *d, enum domain_type type)
> > > > >  {
> > > > > +    struct vcpu *v;
> > > > > +
> > > > >      if ( d == NULL )
> > > > >          return -EINVAL;
> > > > >      if ( d->tot_pages != 0 )
> > > > > @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum
> > > > > domain_type type)
> > > > > 
> > > > >      d->arch.type = type;
> > > > > 
> > > > > +    if ( is_64bit_domain(d) )
> > > > > +        for_each_vcpu(d, v)
> > > > > +            vcpu_switch_to_aarch64_mode(v);
> > > > > +
> > > > >      return 0;
> > > > >  }
> > > > > 
> > > > > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > > > > index 5d18bb0..69c2854 100644
> > > > > --- a/xen/arch/arm/domain.c
> > > > > +++ b/xen/arch/arm/domain.c
> > > > > @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
> > > > >      free_xenheap_pages(v->arch.stack, STACK_ORDER);
> > > > >  }
> > > > > 
> > > > > +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
> > > > > +{
> > > > > +    v->arch.hcr_el2 |= HCR_RW;
> > > > > +}
> > > > 
> > > > if possible, I would write it as a static inline function
> > > > 
> > > 
> > > I had tried to write it as a static inline function in asm/domain.h
> > > But while the first source file (arm64/asm-offsets.c) include
> > > xen/sched.h wanted to compile this inline function it could not find
> > > 'struct vcpu'. Because the xen/sched.h included the asm/domain.h
> > > but defined the vcpu type later. Even though we had included the
> > > xen/sched.h in asm/domain.h already.
> > 
> > It might be too complex to disentangle. In this case, there isn't much
> > type safety to be gained by using a static inline over a macro, so it
> > would be OK to use a macro for this.
> 
> It is not like vCPU will be switch to AArch64 mode often? Only once at vCPU
> creation time.
> 
> So what do we gain to switch to static inline or even macro?

To turn 5 lines of code into 1. Obviously it's no big deal.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 08/18] xen/arm: Introduce a initcall to update cpu_hwcaps by serror_op

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> In the later patches of this series, we want to use the alternative
> patching framework to avoid checking serror_op in every entries.
> So we define a new cpu feature "SKIP_CHECK_PENDING_VSERROR" for
> serror_op. When serror_op is not equal to SERROR_DIVERSE, this
> feature will be set to cpu_hwcaps.
> 
> But we could not update cpu_hwcaps directly in the serror parameter
> parsing function. Because if the serror parameter is not placed in
> the command line, the parsing function would not be invoked.

Wait, the only way to set serrors_op != SERRORS_DIVERSE is to pass the
"serror" command line parameter. The parsing function is always invoked
if serrors_op != SERRORS_DIVERSE.


> So, we introduce this initcall to guarantee the cpu_hwcaps can be
> updated no matter the serror parameter is placed in the command line
> or not.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/traps.c             | 9 +++++++++
>  xen/include/asm-arm/cpufeature.h | 3 ++-
>  2 files changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 5e31699..053b7fc 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -134,6 +134,15 @@ static void __init parse_serrors_behavior(const char *str)
>  }
>  custom_param("serrors", parse_serrors_behavior);
>  
> +static __init int update_serrors_cpu_caps(void)

I think the coding style is "static int __init" ...


> +{
> +    if ( serrors_op != SERRORS_DIVERSE )
> +        cpus_set_cap(SKIP_CHECK_PENDING_VSERROR);
> +
> +    return 0;
> +}
> +__initcall(update_serrors_cpu_caps);
> +
>  register_t get_default_hcr_flags(void)
>  {
>      return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
> diff --git a/xen/include/asm-arm/cpufeature.h b/xen/include/asm-arm/cpufeature.h
> index c0a25ae..ec3f9e5 100644
> --- a/xen/include/asm-arm/cpufeature.h
> +++ b/xen/include/asm-arm/cpufeature.h
> @@ -40,8 +40,9 @@
>  #define ARM32_WORKAROUND_766422 2
>  #define ARM64_WORKAROUND_834220 3
>  #define LIVEPATCH_FEATURE   4
> +#define SKIP_CHECK_PENDING_VSERROR 5
>  
> -#define ARM_NCAPS           5
> +#define ARM_NCAPS           6
>  
>  #ifndef __ASSEMBLY__
>  
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 09/18] xen/arm64: Use alternative to skip the check of pending serrors

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> We have provided an option to administrator to determine how to
> handle the SErrors. In order to skip the check of pending SError,
> in conventional way, we have to read the option every time before
> we try to check the pending SError. This will add overhead to check
> the option at every trap.
> 
> The ARM64 supports the alternative patching feature. We can use an
> ALTERNATIVE to avoid checking option at every trap. We added a new
> cpufeature named "SKIP_CHECK_PENDING_VSERROR". This feature will be
> enabled when the option is not diverse.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  xen/arch/arm/arm64/entry.S | 41 +++++++++++++++++++++++++----------------
>  1 file changed, 25 insertions(+), 16 deletions(-)
> 
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index 02802c0..4baa3cb 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -1,5 +1,6 @@
>  #include <asm/asm_defns.h>
>  #include <asm/regs.h>
> +#include <asm/alternative.h>
>  #include <public/xen.h>
>  
>  /*
> @@ -229,12 +230,14 @@ hyp_irq:
>  
>  guest_sync:
>          entry   hyp=0, compat=0
> -        bl      check_pending_vserror
>          /*
> -         * If x0 is Non-zero, a vSError took place, the initial exception
> -         * doesn't have any significance to be handled. Exit ASAP
> +         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
> +         * not set. If a vSError took place, the initial exception will be
> +         * skipped. Exit ASAP
>           */
> -        cbnz    x0, 1f
> +        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
> +                    "nop; nop",
> +                    SKIP_CHECK_PENDING_VSERROR)
>          msr     daifclr, #2
>          mov     x0, sp
>          bl      do_trap_hypervisor
> @@ -243,12 +246,14 @@ guest_sync:
>  
>  guest_irq:
>          entry   hyp=0, compat=0
> -        bl      check_pending_vserror
>          /*
> -         * If x0 is Non-zero, a vSError took place, the initial exception
> -         * doesn't have any significance to be handled. Exit ASAP
> +         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
> +         * not set. If a vSError took place, the initial exception will be
> +         * skipped. Exit ASAP
>           */
> -        cbnz    x0, 1f
> +        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
> +                    "nop; nop",
> +                    SKIP_CHECK_PENDING_VSERROR)
>          mov     x0, sp
>          bl      do_trap_irq
>  1:
> @@ -267,12 +272,14 @@ guest_error:
>  
>  guest_sync_compat:
>          entry   hyp=0, compat=1
> -        bl      check_pending_vserror
>          /*
> -         * If x0 is Non-zero, a vSError took place, the initial exception
> -         * doesn't have any significance to be handled. Exit ASAP
> +         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
> +         * not set. If a vSError took place, the initial exception will be
> +         * skipped. Exit ASAP
>           */
> -        cbnz    x0, 1f
> +        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
> +                    "nop; nop",
> +                    SKIP_CHECK_PENDING_VSERROR)
>          msr     daifclr, #2
>          mov     x0, sp
>          bl      do_trap_hypervisor
> @@ -281,12 +288,14 @@ guest_sync_compat:
>  
>  guest_irq_compat:
>          entry   hyp=0, compat=1
> -        bl      check_pending_vserror
>          /*
> -         * If x0 is Non-zero, a vSError took place, the initial exception
> -         * doesn't have any significance to be handled. Exit ASAP
> +         * The vSError will be checked while SKIP_CHECK_PENDING_VSERROR is
> +         * not set. If a vSError took place, the initial exception will be
> +         * skipped. Exit ASAP
>           */
> -        cbnz    x0, 1f
> +        ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
> +                    "nop; nop",
> +                    SKIP_CHECK_PENDING_VSERROR)
>          mov     x0, sp
>          bl      do_trap_irq
>  1:
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 10/18] xen/arm32: Use cpu_hwcaps to skip the check of pending serrors

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> We have provided an option to administrator to determine how to
> handle the SErrors. In order to skip the check of pending SError,
> in conventional way, we have to read the option every time before
> we try to check the pending SError.
> 
> Currently, we haven't export the option to other source file. But,
> in the previous patch, we had set "SKIP_CHECK_PENDING_VSERROR" to
> cpu_hwcaps when the option doesn't need to check the SErrors. So we
> can use checking cpu_hwcaps to replace checking the option directly.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> This is a temporary solution, this would have to be dropped as soon
> as ARM32 gain support of alternative patching to avoid potential misusage.
> The alternative patching support patches for ARM32 are still in review
> stage.
> ---
>  xen/arch/arm/arm32/entry.S | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
> index 2187226..79929ca 100644
> --- a/xen/arch/arm/arm32/entry.S
> +++ b/xen/arch/arm/arm32/entry.S
> @@ -1,5 +1,6 @@
>  #include <asm/asm_defns.h>
>  #include <asm/regs.h>
> +#include <asm/cpufeature.h>
>  #include <public/xen.h>
>  
>  #define SAVE_ONE_BANKED(reg)    mrs r11, reg; str r11, [sp, #UREGS_##reg]
> @@ -11,6 +12,21 @@
>  #define RESTORE_BANKED(mode) \
>          RESTORE_ONE_BANKED(SP_##mode) ; RESTORE_ONE_BANKED(LR_##mode) ; RESTORE_ONE_BANKED(SPSR_##mode)
>  
> +/*
> + * If the SKIP_CHECK_PENDING_VSERROR has been set in the cpu feature,
> + * the checking of pending SErrors will be skipped.
> + *
> + * As it is a temporary solution, we are assuming that
> + * SKIP_CHECK_PENDING_VSERROR will always be in the first word for
> + * cpu_hwcaps. This would have to be dropped as soon as ARM32 gain
> + * support of alternative.
> + */
> +#define SKIP_VSERROR_CHECK                      \
> +        ldr r1, =cpu_hwcaps;                    \
> +        ldr r1, [r1];                           \
> +        tst r1, #SKIP_CHECK_PENDING_VSERROR;    \
> +        moveq pc, lr
> +
>  #define SAVE_ALL                                                        \
>          sub sp, #(UREGS_SP_usr - UREGS_sp); /* SP, LR, SPSR, PC */      \
>          push {r0-r12}; /* Save R0-R12 */                                \
> @@ -44,6 +60,9 @@ save_guest_regs:
>          SAVE_BANKED(fiq)
>          SAVE_ONE_BANKED(R8_fiq); SAVE_ONE_BANKED(R9_fiq); SAVE_ONE_BANKED(R10_fiq)
>          SAVE_ONE_BANKED(R11_fiq); SAVE_ONE_BANKED(R12_fiq);
> +
> +        SKIP_VSERROR_CHECK
> +
>          /*
>           * Start to check pending virtual abort in the gap of Guest -> HYP
>           * world switch.
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 11/18] xen/arm: Move macro VABORT_GEN_BY_GUEST to common header

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> We want to move part of SErrors checking code from hyp_error assembly code
> to a function. This new function will use this macro to distinguish the
> guest SErrors from hypervisor SErrors. So we have to move this macro to
> common header.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/arm64/entry.S            |  2 ++
>  xen/include/asm-arm/arm32/processor.h | 10 ----------
>  xen/include/asm-arm/processor.h       | 10 ++++++++++
>  3 files changed, 12 insertions(+), 10 deletions(-)
> 
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index 4baa3cb..113e1c3 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -380,10 +380,12 @@ check_pending_vserror:
>           * exception handler, and the elr_el2 will be set to
>           * abort_guest_exit_start or abort_guest_exit_end.
>           */
> +        .global abort_guest_exit_start
>  abort_guest_exit_start:
>  
>          isb
>  
> +        .global abort_guest_exit_end
>  abort_guest_exit_end:
>          /* Mask PSTATE asynchronous abort bit, close the checking window. */
>          msr     daifset, #4

These changes are unexplained in the commit message.


> diff --git a/xen/include/asm-arm/arm32/processor.h b/xen/include/asm-arm/arm32/processor.h
> index f6d5df3..68cc821 100644
> --- a/xen/include/asm-arm/arm32/processor.h
> +++ b/xen/include/asm-arm/arm32/processor.h
> @@ -56,16 +56,6 @@ struct cpu_user_regs
>      uint32_t pad1; /* Doubleword-align the user half of the frame */
>  };
>  
> -/* Functions for pending virtual abort checking window. */
> -void abort_guest_exit_start(void);
> -void abort_guest_exit_end(void);
> -
> -#define VABORT_GEN_BY_GUEST(r)  \
> -( \
> -    ( (unsigned long)abort_guest_exit_start == (r)->pc ) || \
> -    ( (unsigned long)abort_guest_exit_end == (r)->pc ) \
> -)
> -
>  #endif
>  
>  /* Layout as used in assembly, with src/dest registers mixed in */
> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
> index d7b0711..148cc6f 100644
> --- a/xen/include/asm-arm/processor.h
> +++ b/xen/include/asm-arm/processor.h
> @@ -709,6 +709,16 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
>  
>  void do_trap_guest_error(struct cpu_user_regs *regs);
>  
> +/* Functions for pending virtual abort checking window. */
> +void abort_guest_exit_start(void);
> +void abort_guest_exit_end(void);
> +
> +#define VABORT_GEN_BY_GUEST(r)  \
> +( \
> +    ( (unsigned long)abort_guest_exit_start == (r)->pc ) || \
> +    ( (unsigned long)abort_guest_exit_end == (r)->pc ) \
> +)
> +
>  register_t get_default_hcr_flags(void);
>  
>  #endif /* __ASSEMBLY__ */
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 13/18] xen/arm: Replace do_trap_guest_serror with new helpers

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> We have introduced two helpers to handle the guest/hyp SErrors:
> do_trap_guest_serror and do_trap_guest_hyp_serror. These handlers
> can take the role of do_trap_guest_serror and reduce the assembly
> code in the same time. So we use these two helpers to replace it
> and drop it now.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  xen/arch/arm/arm32/traps.c      |  5 +----
>  xen/arch/arm/arm64/entry.S      | 36 +++---------------------------------
>  xen/arch/arm/traps.c            | 15 ---------------
>  xen/include/asm-arm/processor.h |  2 --
>  4 files changed, 4 insertions(+), 54 deletions(-)
> 
> diff --git a/xen/arch/arm/arm32/traps.c b/xen/arch/arm/arm32/traps.c
> index 4176f0e..5bc5f64 100644
> --- a/xen/arch/arm/arm32/traps.c
> +++ b/xen/arch/arm/arm32/traps.c
> @@ -62,10 +62,7 @@ asmlinkage void do_trap_prefetch_abort(struct cpu_user_regs *regs)
>  
>  asmlinkage void do_trap_data_abort(struct cpu_user_regs *regs)
>  {
> -    if ( VABORT_GEN_BY_GUEST(regs) )
> -        do_trap_guest_error(regs);
> -    else
> -        do_unexpected_trap("Data Abort", regs);
> +    do_trap_hyp_serror(regs);
>  }
>  
>  /*
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index 113e1c3..8d5a890 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -178,40 +178,10 @@ hyp_error_invalid:
>          invalid BAD_ERROR
>  
>  hyp_error:
> -        /*
> -         * Only two possibilities:
> -         * 1) Either we come from the exit path, having just unmasked
> -         *    PSTATE.A: change the return code to an EL2 fault, and
> -         *    carry on, as we're already in a sane state to handle it.
> -         * 2) Or we come from anywhere else, and that's a bug: we panic.
> -         */
>          entry   hyp=1
>          msr     daifclr, #2
> -
> -        /*
> -         * The ELR_EL2 may be modified by an interrupt, so we have to use the
> -         * saved value in cpu_user_regs to check whether we come from 1) or
> -         * not.
> -         */
> -        ldr     x0, [sp, #UREGS_PC]
> -        adr     x1, abort_guest_exit_start
> -        cmp     x0, x1
> -        adr     x1, abort_guest_exit_end
> -        ccmp    x0, x1, #4, ne
>          mov     x0, sp
> -        mov     x1, #BAD_ERROR
> -
> -        /*
> -         * Not equal, the exception come from 2). It's a bug, we have to
> -         * panic the hypervisor.
> -         */
> -        b.ne    do_bad_mode
> -
> -        /*
> -         * Otherwise, the exception come from 1). It happened because of
> -         * the guest. Crash this guest.
> -         */
> -        bl      do_trap_guest_error
> +        bl      do_trap_hyp_serror
>          exit    hyp=1
>  
>  /* Traps taken in Current EL with SP_ELx */
> @@ -267,7 +237,7 @@ guest_error:
>          entry   hyp=0, compat=0
>          msr     daifclr, #2
>          mov     x0, sp
> -        bl      do_trap_guest_error
> +        bl      do_trap_guest_serror
>          exit    hyp=0, compat=0
>  
>  guest_sync_compat:
> @@ -309,7 +279,7 @@ guest_error_compat:
>          entry   hyp=0, compat=1
>          msr     daifclr, #2
>          mov     x0, sp
> -        bl      do_trap_guest_error
> +        bl      do_trap_guest_serror
>          exit    hyp=0, compat=1
>  
>  ENTRY(return_to_new_vcpu32)
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 48cfc8e..44a0281 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2899,21 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
>      }
>  }
>  
> -asmlinkage void do_trap_guest_error(struct cpu_user_regs *regs)
> -{
> -    enter_hypervisor_head(regs);
> -
> -    /*
> -     * Currently, to ensure hypervisor safety, when we received a
> -     * guest-generated vSerror/vAbort, we just crash the guest to protect
> -     * the hypervisor. In future we can better handle this by injecting
> -     * a vSerror/vAbort to the guest.
> -     */
> -    gdprintk(XENLOG_WARNING, "Guest(Dom-%u) will be crashed by vSError\n",
> -             current->domain->domain_id);
> -    domain_crash_synchronous();
> -}
> -
>  asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
>  {
>      enter_hypervisor_head(regs);
> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
> index 885dbca..afad78c 100644
> --- a/xen/include/asm-arm/processor.h
> +++ b/xen/include/asm-arm/processor.h
> @@ -707,8 +707,6 @@ void vcpu_regs_user_to_hyp(struct vcpu *vcpu,
>  int call_smc(register_t function_id, register_t arg0, register_t arg1,
>               register_t arg2);
>  
> -void do_trap_guest_error(struct cpu_user_regs *regs);
> -
>  void do_trap_hyp_serror(struct cpu_user_regs *regs);
>  
>  void do_trap_guest_serror(struct cpu_user_regs *regs);
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 12/18] xen/arm: Introduce new helpers to handle guest/hyp SErrors

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> Currently, ARM32 and ARM64 has different SError exception handlers.
> These handlers include lots of code to check SError handle options
> and code to distinguish guest-generated SErrors from hypervisor
> SErrors.
> 
> The new helpers: do_trap_guest_serror and do_trap_hyp_serror are
> wrappers of __do_trap_serror with constant guest/hyp parameters.
> __do_trap_serror moves the option checking code and SError checking
> code from assembly to C source. This will make the code become more
> readable and avoid placing check code in too many places.
> 
> These two helpers only handle the following 3 types of SErrors:
> 1) Guest-generated SError and had been delivered in EL1 and then
>    been forwarded to EL2.
> 2) Guest-generated SError but hadn't been delivered in EL1 before
>    trapping to EL2. This SError would be caught in EL2 as soon as
>    we just unmasked the PSTATE.A bit.
> 3) Hypervisor generated native SError, that would be a bug.
> 
> In the new helpers, we have used the function "inject_vabt_exception"
> which was disabled by "#if 0" before. Now, we can remove the "#if 0"
> to make this function to be available.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  xen/arch/arm/traps.c            | 69 +++++++++++++++++++++++++++++++++++++++--
>  xen/include/asm-arm/processor.h |  4 +++
>  2 files changed, 71 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 053b7fc..48cfc8e 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -646,7 +646,6 @@ static void inject_dabt_exception(struct cpu_user_regs *regs,
>  #endif
>  }
>  
> -#if 0
>  /* Inject a virtual Abort/SError into the guest. */
>  static void inject_vabt_exception(struct cpu_user_regs *regs)
>  {
> @@ -676,7 +675,59 @@ static void inject_vabt_exception(struct cpu_user_regs *regs)
>  
>      current->arch.hcr_el2 |= HCR_VA;
>  }
> -#endif
> +
> +/*
> + * SError exception handler. We only handle the following 3 types of SErrors:
> + * 1) Guest-generated SError and had been delivered in EL1 and then
> + *    been forwarded to EL2.
> + * 2) Guest-generated SError but hadn't been delivered in EL1 before
> + *    trapping to EL2. This SError would be caught in EL2 as soon as
> + *    we just unmasked the PSTATE.A bit.
> + * 3) Hypervisor generated native SError, that would be a bug.
> + *
> + * A true parameter "guest" means that the SError is type#1 or type#2.
> + */
> +static void __do_trap_serror(struct cpu_user_regs *regs, bool guest)
> +{
> +    /*
> +     * Only "DIVERSE" option needs to distinguish the guest-generated SErrors
> +     * from hypervisor SErrors.
> +     */
> +    if ( serrors_op == SERRORS_DIVERSE )
> +    {
> +        /* Forward the type#1 and type#2 SErrors to guests. */
> +        if ( guest )
> +            return inject_vabt_exception(regs);
> +
> +        /* Type#3 SErrors will panic the whole system */
> +        goto crash_system;
> +    }
> +
> +    /*
> +     * The "FORWARD" option will forward all SErrors to the guests, except
> +     * idle domain generated SErrors.
> +     */
> +    if ( serrors_op == SERRORS_FORWARD )
> +    {
> +        /*
> +         * Because the idle domain doesn't have the ability to handle the
> +         * SErrors, we have to crash the whole system while we get a SError
> +         * generated by idle domain.
> +         */
> +        if ( is_idle_vcpu(current) )
> +            goto crash_system;
> +
> +        return inject_vabt_exception(regs);
> +    }
> +
> +crash_system:
> +    /* Three possibilities to crash the whole system:
> +     * 1) "DIVERSE" option with Hypervisor generated SErrors.
> +     * 2) "FORWARD" option with Idle Domain generated SErrors.
> +     * 3) "PANIC" option with all SErrors.
> +     */
> +    do_unexpected_trap("SError", regs);
> +}
>  
>  struct reg_ctxt {
>      /* Guest-side state */
> @@ -2863,6 +2914,20 @@ asmlinkage void do_trap_guest_error(struct cpu_user_regs *regs)
>      domain_crash_synchronous();
>  }
>  
> +asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
> +{
> +    enter_hypervisor_head(regs);
> +
> +    __do_trap_serror(regs, VABORT_GEN_BY_GUEST(regs));
> +}
> +
> +asmlinkage void do_trap_guest_serror(struct cpu_user_regs *regs)
> +{
> +    enter_hypervisor_head(regs);
> +
> +    __do_trap_serror(regs, true);
> +}
> +
>  asmlinkage void do_trap_irq(struct cpu_user_regs *regs)
>  {
>      enter_hypervisor_head(regs);
> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
> index 148cc6f..885dbca 100644
> --- a/xen/include/asm-arm/processor.h
> +++ b/xen/include/asm-arm/processor.h
> @@ -709,6 +709,10 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
>  
>  void do_trap_guest_error(struct cpu_user_regs *regs);
>  
> +void do_trap_hyp_serror(struct cpu_user_regs *regs);
> +
> +void do_trap_guest_serror(struct cpu_user_regs *regs);
> +
>  /* Functions for pending virtual abort checking window. */
>  void abort_guest_exit_start(void);
>  void abort_guest_exit_end(void);
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 05/18] xen/arm: Save ESR_EL2 to avoid using mismatched value in syndrome check

[Wei Chen]

Hi Julien,

On 2017/3/17 1:34, Julien Grall wrote:
> Hi Wei,
>
> On 03/13/2017 10:55 AM, Wei Chen wrote:
>> Xen will do exception syndrome check while some types of exception
>> take place in EL2. The syndrome check code read the ESR_EL2 register
>> directly, but in some situation this register maybe overridden by
>> nested exception.
>>
>> For example, if we re-enable IRQ before reading ESR_EL2 which means
>> Xen will enter in IRQ exception mode and return the processor with
>
> s/will/may/
>

Ok.

>> clobbered ESR_EL2 (See ARM ARM DDI 0487A.j D7.2.25)
>>
>> In this case the guest exception syndrome has been overridden, we will
>> check the syndrome for guest sync exception with a mismatched ESR_EL2
>
> s/mismatched/incorrect/ I think
>
>> value. So we want to save ESR_EL2 to cpu_user_regs as soon as the
>> exception takes place in EL2 to avoid using a mismatched syndrome value.
>
> Ditto.
>

Ok.

>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>> ---
>>  xen/arch/arm/arm32/asm-offsets.c      |  1 +
>>  xen/arch/arm/arm32/entry.S            |  3 +++
>>  xen/arch/arm/arm64/asm-offsets.c      |  1 +
>>  xen/arch/arm/arm64/entry.S            | 13 +++++++++----
>>  xen/arch/arm/traps.c                  |  2 +-
>>  xen/include/asm-arm/arm32/processor.h |  2 +-
>>  xen/include/asm-arm/arm64/processor.h | 10 ++++++++--
>>  7 files changed, 24 insertions(+), 8 deletions(-)
>>
>> diff --git a/xen/arch/arm/arm32/asm-offsets.c b/xen/arch/arm/arm32/asm-offsets.c
>> index f8e6b53..5b543ab 100644
>> --- a/xen/arch/arm/arm32/asm-offsets.c
>> +++ b/xen/arch/arm/arm32/asm-offsets.c
>> @@ -26,6 +26,7 @@ void __dummy__(void)
>>     OFFSET(UREGS_lr, struct cpu_user_regs, lr);
>>     OFFSET(UREGS_pc, struct cpu_user_regs, pc);
>>     OFFSET(UREGS_cpsr, struct cpu_user_regs, cpsr);
>> +   OFFSET(UREGS_hsr, struct cpu_user_regs, hsr);
>>
>>     OFFSET(UREGS_LR_usr, struct cpu_user_regs, lr_usr);
>>     OFFSET(UREGS_SP_usr, struct cpu_user_regs, sp_usr);
>> diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
>> index 2a6f4f0..2187226 100644
>> --- a/xen/arch/arm/arm32/entry.S
>> +++ b/xen/arch/arm/arm32/entry.S
>> @@ -23,6 +23,9 @@
>>          add r11, sp, #UREGS_kernel_sizeof+4;                            \
>>          str r11, [sp, #UREGS_sp];                                       \
>>                                                                          \
>> +        mrc CP32(r11, HSR);             /* Save exception syndrome */   \
>> +        str r11, [sp, #UREGS_hsr];                                      \
>> +                                                                        \
>>          mrs r11, SPSR_hyp;                                              \
>>          str r11, [sp, #UREGS_cpsr];                                     \
>>          and r11, #PSR_MODE_MASK;                                        \
>> diff --git a/xen/arch/arm/arm64/asm-offsets.c b/xen/arch/arm/arm64/asm-offsets.c
>> index 69ea92a..ce24e44 100644
>> --- a/xen/arch/arm/arm64/asm-offsets.c
>> +++ b/xen/arch/arm/arm64/asm-offsets.c
>> @@ -27,6 +27,7 @@ void __dummy__(void)
>>     OFFSET(UREGS_SP, struct cpu_user_regs, sp);
>>     OFFSET(UREGS_PC, struct cpu_user_regs, pc);
>>     OFFSET(UREGS_CPSR, struct cpu_user_regs, cpsr);
>> +   OFFSET(UREGS_ESR_el2, struct cpu_user_regs, hsr);
>>
>>     OFFSET(UREGS_SPSR_el1, struct cpu_user_regs, spsr_el1);
>>
>> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
>> index c181b5e..02802c0 100644
>> --- a/xen/arch/arm/arm64/entry.S
>> +++ b/xen/arch/arm/arm64/entry.S
>> @@ -121,9 +121,13 @@ lr      .req    x30             // link register
>>
>>          stp     lr, x21, [sp, #UREGS_LR]
>>
>> -        mrs     x22, elr_el2
>> -        mrs     x23, spsr_el2
>> -        stp     x22, x23, [sp, #UREGS_PC]
>> +        mrs     x21, elr_el2
>> +        str     x21, [sp, #UREGS_PC]
>
> Please explain the commit message you modify the lines above ...
>

I will do it in next version.

>> +
>> +        add     x21, sp, #UREGS_CPSR
>> +        mrs     x22, spsr_el2
>> +        mrs     x23, esr_el2
>> +        stp     w22, w23, [x21]
>>
>>          .endm
>>
>> @@ -307,7 +311,8 @@ ENTRY(return_to_new_vcpu64)
>>  return_from_trap:
>>          msr     daifset, #2 /* Mask interrupts */
>>
>> -        ldp     x21, x22, [sp, #UREGS_PC]       // load ELR, SPSR
>> +        ldr     x21, [sp, #UREGS_PC]            // load ELR
>> +        ldr     w22, [sp, #UREGS_CPSR]          // load SPSR
>
> as long as those one.
>

Ok.

>>
>>          pop     x0, x1
>>          pop     x2, x3
>
> [...]
>
>> diff --git a/xen/include/asm-arm/arm64/processor.h b/xen/include/asm-arm/arm64/processor.h
>> index b0726ff..d381428 100644
>> --- a/xen/include/asm-arm/arm64/processor.h
>> +++ b/xen/include/asm-arm/arm64/processor.h
>> @@ -65,9 +65,15 @@ struct cpu_user_regs
>>
>>      /* Return address and mode */
>>      __DECL_REG(pc,           pc32);             /* ELR_EL2 */
>> +    /*
>> +     * Be careful for 32-bit registers, if we use xN to save 32-bit register
>> +     * to stack, its next field on stack will be overridden.
>> +     * For example, if we use xN to save SPSR_EL2 to stack will override the
>> +     * hsr field on stack.
>> +     * So, it's better to use wN to save 32-bit registers to stack.
>> +     */
>
> This comment is pointless. This is true for any 32-bit register, you
> should use wN unless you now that you have a padding after.
>

Ok, I will remove it.

>>      uint32_t cpsr;                              /* SPSR_EL2 */
>> -
>> -    uint32_t pad0; /* Align end of kernel frame. */
>> +    uint32_t hsr;                               /* ESR_EL2 */
>>
>>      /* Outer guest frame only from here on... */
>>
>>
>
> Cheers,
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 05/18] xen/arm: Save ESR_EL2 to avoid using mismatched value in syndrome check

[Wei Chen]

Hi Stefano,

On 2017/3/17 6:27, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Julien Grall wrote:
>> Hi Wei,
>>
>> On 03/13/2017 10:55 AM, Wei Chen wrote:
>>> Xen will do exception syndrome check while some types of exception
>>> take place in EL2. The syndrome check code read the ESR_EL2 register
>>> directly, but in some situation this register maybe overridden by
>>> nested exception.
>>>
>>> For example, if we re-enable IRQ before reading ESR_EL2 which means
>>> Xen will enter in IRQ exception mode and return the processor with
>>
>> s/will/may/
>>
>>> clobbered ESR_EL2 (See ARM ARM DDI 0487A.j D7.2.25)
>>>
>>> In this case the guest exception syndrome has been overridden, we will
>>> check the syndrome for guest sync exception with a mismatched ESR_EL2
>>
>> s/mismatched/incorrect/ I think
>>
>>> value. So we want to save ESR_EL2 to cpu_user_regs as soon as the
>>> exception takes place in EL2 to avoid using a mismatched syndrome value.
>>
>> Ditto.
>
> Please address Julien's comments. The code looks good.
>

Thanks, I will address them in next version.

>
>>>
>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>> ---
>>>  xen/arch/arm/arm32/asm-offsets.c      |  1 +
>>>  xen/arch/arm/arm32/entry.S            |  3 +++
>>>  xen/arch/arm/arm64/asm-offsets.c      |  1 +
>>>  xen/arch/arm/arm64/entry.S            | 13 +++++++++----
>>>  xen/arch/arm/traps.c                  |  2 +-
>>>  xen/include/asm-arm/arm32/processor.h |  2 +-
>>>  xen/include/asm-arm/arm64/processor.h | 10 ++++++++--
>>>  7 files changed, 24 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/arm32/asm-offsets.c
>>> b/xen/arch/arm/arm32/asm-offsets.c
>>> index f8e6b53..5b543ab 100644
>>> --- a/xen/arch/arm/arm32/asm-offsets.c
>>> +++ b/xen/arch/arm/arm32/asm-offsets.c
>>> @@ -26,6 +26,7 @@ void __dummy__(void)
>>>     OFFSET(UREGS_lr, struct cpu_user_regs, lr);
>>>     OFFSET(UREGS_pc, struct cpu_user_regs, pc);
>>>     OFFSET(UREGS_cpsr, struct cpu_user_regs, cpsr);
>>> +   OFFSET(UREGS_hsr, struct cpu_user_regs, hsr);
>>>
>>>     OFFSET(UREGS_LR_usr, struct cpu_user_regs, lr_usr);
>>>     OFFSET(UREGS_SP_usr, struct cpu_user_regs, sp_usr);
>>> diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
>>> index 2a6f4f0..2187226 100644
>>> --- a/xen/arch/arm/arm32/entry.S
>>> +++ b/xen/arch/arm/arm32/entry.S
>>> @@ -23,6 +23,9 @@
>>>          add r11, sp, #UREGS_kernel_sizeof+4;                            \
>>>          str r11, [sp, #UREGS_sp];                                       \
>>>                                                                          \
>>> +        mrc CP32(r11, HSR);             /* Save exception syndrome */   \
>>> +        str r11, [sp, #UREGS_hsr];                                      \
>>> +                                                                        \
>>>          mrs r11, SPSR_hyp;                                              \
>>>          str r11, [sp, #UREGS_cpsr];                                     \
>>>          and r11, #PSR_MODE_MASK;                                        \
>>> diff --git a/xen/arch/arm/arm64/asm-offsets.c
>>> b/xen/arch/arm/arm64/asm-offsets.c
>>> index 69ea92a..ce24e44 100644
>>> --- a/xen/arch/arm/arm64/asm-offsets.c
>>> +++ b/xen/arch/arm/arm64/asm-offsets.c
>>> @@ -27,6 +27,7 @@ void __dummy__(void)
>>>     OFFSET(UREGS_SP, struct cpu_user_regs, sp);
>>>     OFFSET(UREGS_PC, struct cpu_user_regs, pc);
>>>     OFFSET(UREGS_CPSR, struct cpu_user_regs, cpsr);
>>> +   OFFSET(UREGS_ESR_el2, struct cpu_user_regs, hsr);
>>>
>>>     OFFSET(UREGS_SPSR_el1, struct cpu_user_regs, spsr_el1);
>>>
>>> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
>>> index c181b5e..02802c0 100644
>>> --- a/xen/arch/arm/arm64/entry.S
>>> +++ b/xen/arch/arm/arm64/entry.S
>>> @@ -121,9 +121,13 @@ lr      .req    x30             // link register
>>>
>>>          stp     lr, x21, [sp, #UREGS_LR]
>>>
>>> -        mrs     x22, elr_el2
>>> -        mrs     x23, spsr_el2
>>> -        stp     x22, x23, [sp, #UREGS_PC]
>>> +        mrs     x21, elr_el2
>>> +        str     x21, [sp, #UREGS_PC]
>>
>> Please explain the commit message you modify the lines above ...
>>
>>> +
>>> +        add     x21, sp, #UREGS_CPSR
>>> +        mrs     x22, spsr_el2
>>> +        mrs     x23, esr_el2
>>> +        stp     w22, w23, [x21]
>>>
>>>          .endm
>>>
>>> @@ -307,7 +311,8 @@ ENTRY(return_to_new_vcpu64)
>>>  return_from_trap:
>>>          msr     daifset, #2 /* Mask interrupts */
>>>
>>> -        ldp     x21, x22, [sp, #UREGS_PC]       // load ELR, SPSR
>>> +        ldr     x21, [sp, #UREGS_PC]            // load ELR
>>> +        ldr     w22, [sp, #UREGS_CPSR]          // load SPSR
>>
>> as long as those one.
>>
>>>
>>>          pop     x0, x1
>>>          pop     x2, x3
>>
>> [...]
>>
>>> diff --git a/xen/include/asm-arm/arm64/processor.h
>>> b/xen/include/asm-arm/arm64/processor.h
>>> index b0726ff..d381428 100644
>>> --- a/xen/include/asm-arm/arm64/processor.h
>>> +++ b/xen/include/asm-arm/arm64/processor.h
>>> @@ -65,9 +65,15 @@ struct cpu_user_regs
>>>
>>>      /* Return address and mode */
>>>      __DECL_REG(pc,           pc32);             /* ELR_EL2 */
>>> +    /*
>>> +     * Be careful for 32-bit registers, if we use xN to save 32-bit
>>> register
>>> +     * to stack, its next field on stack will be overridden.
>>> +     * For example, if we use xN to save SPSR_EL2 to stack will override
>>> the
>>> +     * hsr field on stack.
>>> +     * So, it's better to use wN to save 32-bit registers to stack.
>>> +     */
>>
>> This comment is pointless. This is true for any 32-bit register, you should
>> use wN unless you now that you have a padding after.
>>
>>>      uint32_t cpsr;                              /* SPSR_EL2 */
>>> -
>>> -    uint32_t pad0; /* Align end of kernel frame. */
>>> +    uint32_t hsr;                               /* ESR_EL2 */
>>>
>>>      /* Outer guest frame only from here on... */
>>>
>>>
>>
>> Cheers,
>>
>> --
>> Julien Grall
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Wei Chen]

Hi Stefano,

On 2017/3/17 7:17, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Julien Grall wrote:
>> On 03/16/2017 10:40 PM, Stefano Stabellini wrote:
>>> On Wed, 15 Mar 2017, Wei Chen wrote:
>>>> Hi Stefano,
>>>>
>>>> On 2017/3/15 8:25, Stefano Stabellini wrote:
>>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>>>> The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
>>>>>> when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
>>>>>> DomU in vcpu_initialise, we didn't know the domain's address size
>>>>>> information. We had to use compatible flags to initialize HCR_EL2,
>>>>>> and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
>>>>>> at every context switch.
>>>>>>
>>>>>> But, after we added the HCR_EL2 to vcpu's context, this behaviour
>>>>>> seems a little fussy. We can update the HCR_RW bit in vcpu's context
>>>>>> as soon as we get the domain's address size to avoid setting/clearing
>>>>>> HCR_RW at every context switch.
>>>>>>
>>>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>>>> ---
>>>>>>  xen/arch/arm/arm64/domctl.c  | 6 ++++++
>>>>>>  xen/arch/arm/domain.c        | 5 +++++
>>>>>>  xen/arch/arm/domain_build.c  | 7 +++++++
>>>>>>  xen/arch/arm/p2m.c           | 5 -----
>>>>>>  xen/include/asm-arm/domain.h | 1 +
>>>>>>  5 files changed, 19 insertions(+), 5 deletions(-)
>>>>>>
>>>>>> diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
>>>>>> index 44e1e7b..ab8781f 100644
>>>>>> --- a/xen/arch/arm/arm64/domctl.c
>>>>>> +++ b/xen/arch/arm/arm64/domctl.c
>>>>>> @@ -14,6 +14,8 @@
>>>>>>
>>>>>>  static long switch_mode(struct domain *d, enum domain_type type)
>>>>>>  {
>>>>>> +    struct vcpu *v;
>>>>>> +
>>>>>>      if ( d == NULL )
>>>>>>          return -EINVAL;
>>>>>>      if ( d->tot_pages != 0 )
>>>>>> @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum
>>>>>> domain_type type)
>>>>>>
>>>>>>      d->arch.type = type;
>>>>>>
>>>>>> +    if ( is_64bit_domain(d) )
>>>>>> +        for_each_vcpu(d, v)
>>>>>> +            vcpu_switch_to_aarch64_mode(v);
>>>>>> +
>>>>>>      return 0;
>>>>>>  }
>>>>>>
>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>> index 5d18bb0..69c2854 100644
>>>>>> --- a/xen/arch/arm/domain.c
>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>> @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
>>>>>>      free_xenheap_pages(v->arch.stack, STACK_ORDER);
>>>>>>  }
>>>>>>
>>>>>> +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
>>>>>> +{
>>>>>> +    v->arch.hcr_el2 |= HCR_RW;
>>>>>> +}
>>>>>
>>>>> if possible, I would write it as a static inline function
>>>>>
>>>>
>>>> I had tried to write it as a static inline function in asm/domain.h
>>>> But while the first source file (arm64/asm-offsets.c) include
>>>> xen/sched.h wanted to compile this inline function it could not find
>>>> 'struct vcpu'. Because the xen/sched.h included the asm/domain.h
>>>> but defined the vcpu type later. Even though we had included the
>>>> xen/sched.h in asm/domain.h already.
>>>
>>> It might be too complex to disentangle. In this case, there isn't much
>>> type safety to be gained by using a static inline over a macro, so it
>>> would be OK to use a macro for this.
>>
>> It is not like vCPU will be switch to AArch64 mode often? Only once at vCPU
>> creation time.
>>
>> So what do we gain to switch to static inline or even macro?
>
> To turn 5 lines of code into 1. Obviously it's no big deal.
>

Ok, I think switch to macro is easier then static inline. I will switch
it to a macro in next version.


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 08/18] xen/arm: Introduce a initcall to update cpu_hwcaps by serror_op

[Wei Chen]

Hi Stefano,

On 2017/3/17 7:30, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> In the later patches of this series, we want to use the alternative
>> patching framework to avoid checking serror_op in every entries.
>> So we define a new cpu feature "SKIP_CHECK_PENDING_VSERROR" for
>> serror_op. When serror_op is not equal to SERROR_DIVERSE, this
>> feature will be set to cpu_hwcaps.
>>
>> But we could not update cpu_hwcaps directly in the serror parameter
>> parsing function. Because if the serror parameter is not placed in
>> the command line, the parsing function would not be invoked.
>
> Wait, the only way to set serrors_op != SERRORS_DIVERSE is to pass the
> "serror" command line parameter. The parsing function is always invoked
> if serrors_op != SERRORS_DIVERSE.
>

Yes, but that because we set serrors_op to SERRORS_DIVERSE by default.
If we change the policy and set serrors_op to SERRORS_PANIC by default, 
then the parsing function would never be invoked.

>
>> So, we introduce this initcall to guarantee the cpu_hwcaps can be
>> updated no matter the serror parameter is placed in the command line
>> or not.
>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>> ---
>>  xen/arch/arm/traps.c             | 9 +++++++++
>>  xen/include/asm-arm/cpufeature.h | 3 ++-
>>  2 files changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index 5e31699..053b7fc 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -134,6 +134,15 @@ static void __init parse_serrors_behavior(const char *str)
>>  }
>>  custom_param("serrors", parse_serrors_behavior);
>>
>> +static __init int update_serrors_cpu_caps(void)
>
> I think the coding style is "static int __init" ...
>

Oh, thanks, I will fix it.

>
>> +{
>> +    if ( serrors_op != SERRORS_DIVERSE )
>> +        cpus_set_cap(SKIP_CHECK_PENDING_VSERROR);
>> +
>> +    return 0;
>> +}
>> +__initcall(update_serrors_cpu_caps);
>> +
>>  register_t get_default_hcr_flags(void)
>>  {
>>      return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
>> diff --git a/xen/include/asm-arm/cpufeature.h b/xen/include/asm-arm/cpufeature.h
>> index c0a25ae..ec3f9e5 100644
>> --- a/xen/include/asm-arm/cpufeature.h
>> +++ b/xen/include/asm-arm/cpufeature.h
>> @@ -40,8 +40,9 @@
>>  #define ARM32_WORKAROUND_766422 2
>>  #define ARM64_WORKAROUND_834220 3
>>  #define LIVEPATCH_FEATURE   4
>> +#define SKIP_CHECK_PENDING_VSERROR 5
>>
>> -#define ARM_NCAPS           5
>> +#define ARM_NCAPS           6
>>
>>  #ifndef __ASSEMBLY__
>>
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 11/18] xen/arm: Move macro VABORT_GEN_BY_GUEST to common header

[Wei Chen]

Hi Stefano,

On 2017/3/17 7:53, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> We want to move part of SErrors checking code from hyp_error assembly code
>> to a function. This new function will use this macro to distinguish the
>> guest SErrors from hypervisor SErrors. So we have to move this macro to
>> common header.
>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>> ---
>>  xen/arch/arm/arm64/entry.S            |  2 ++
>>  xen/include/asm-arm/arm32/processor.h | 10 ----------
>>  xen/include/asm-arm/processor.h       | 10 ++++++++++
>>  3 files changed, 12 insertions(+), 10 deletions(-)
>>
>> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
>> index 4baa3cb..113e1c3 100644
>> --- a/xen/arch/arm/arm64/entry.S
>> +++ b/xen/arch/arm/arm64/entry.S
>> @@ -380,10 +380,12 @@ check_pending_vserror:
>>           * exception handler, and the elr_el2 will be set to
>>           * abort_guest_exit_start or abort_guest_exit_end.
>>           */
>> +        .global abort_guest_exit_start
>>  abort_guest_exit_start:
>>
>>          isb
>>
>> +        .global abort_guest_exit_end
>>  abort_guest_exit_end:
>>          /* Mask PSTATE asynchronous abort bit, close the checking window. */
>>          msr     daifset, #4
>
> These changes are unexplained in the commit message.
>

I will add the explanation into the commit message in next version.

>
>> diff --git a/xen/include/asm-arm/arm32/processor.h b/xen/include/asm-arm/arm32/processor.h
>> index f6d5df3..68cc821 100644
>> --- a/xen/include/asm-arm/arm32/processor.h
>> +++ b/xen/include/asm-arm/arm32/processor.h
>> @@ -56,16 +56,6 @@ struct cpu_user_regs
>>      uint32_t pad1; /* Doubleword-align the user half of the frame */
>>  };
>>
>> -/* Functions for pending virtual abort checking window. */
>> -void abort_guest_exit_start(void);
>> -void abort_guest_exit_end(void);
>> -
>> -#define VABORT_GEN_BY_GUEST(r)  \
>> -( \
>> -    ( (unsigned long)abort_guest_exit_start == (r)->pc ) || \
>> -    ( (unsigned long)abort_guest_exit_end == (r)->pc ) \
>> -)
>> -
>>  #endif
>>
>>  /* Layout as used in assembly, with src/dest registers mixed in */
>> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
>> index d7b0711..148cc6f 100644
>> --- a/xen/include/asm-arm/processor.h
>> +++ b/xen/include/asm-arm/processor.h
>> @@ -709,6 +709,16 @@ int call_smc(register_t function_id, register_t arg0, register_t arg1,
>>
>>  void do_trap_guest_error(struct cpu_user_regs *regs);
>>
>> +/* Functions for pending virtual abort checking window. */
>> +void abort_guest_exit_start(void);
>> +void abort_guest_exit_end(void);
>> +
>> +#define VABORT_GEN_BY_GUEST(r)  \
>> +( \
>> +    ( (unsigned long)abort_guest_exit_start == (r)->pc ) || \
>> +    ( (unsigned long)abort_guest_exit_end == (r)->pc ) \
>> +)
>> +
>>  register_t get_default_hcr_flags(void);
>>
>>  #endif /* __ASSEMBLY__ */
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Julien Grall]

Hi Wei,

On 03/17/2017 06:51 AM, Wei Chen wrote:
> Hi Stefano,
>
> On 2017/3/17 7:17, Stefano Stabellini wrote:
>> On Thu, 16 Mar 2017, Julien Grall wrote:
>>> On 03/16/2017 10:40 PM, Stefano Stabellini wrote:
>>>> On Wed, 15 Mar 2017, Wei Chen wrote:
>>>>> Hi Stefano,
>>>>>
>>>>> On 2017/3/15 8:25, Stefano Stabellini wrote:
>>>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>>>>> The HCR_EL2 flags for 64-bit and 32-bit domains are different. But
>>>>>>> when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus of
>>>>>>> DomU in vcpu_initialise, we didn't know the domain's address size
>>>>>>> information. We had to use compatible flags to initialize HCR_EL2,
>>>>>>> and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit domain
>>>>>>> at every context switch.
>>>>>>>
>>>>>>> But, after we added the HCR_EL2 to vcpu's context, this behaviour
>>>>>>> seems a little fussy. We can update the HCR_RW bit in vcpu's context
>>>>>>> as soon as we get the domain's address size to avoid setting/clearing
>>>>>>> HCR_RW at every context switch.
>>>>>>>
>>>>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>>>>> ---
>>>>>>>  xen/arch/arm/arm64/domctl.c  | 6 ++++++
>>>>>>>  xen/arch/arm/domain.c        | 5 +++++
>>>>>>>  xen/arch/arm/domain_build.c  | 7 +++++++
>>>>>>>  xen/arch/arm/p2m.c           | 5 -----
>>>>>>>  xen/include/asm-arm/domain.h | 1 +
>>>>>>>  5 files changed, 19 insertions(+), 5 deletions(-)
>>>>>>>
>>>>>>> diff --git a/xen/arch/arm/arm64/domctl.c b/xen/arch/arm/arm64/domctl.c
>>>>>>> index 44e1e7b..ab8781f 100644
>>>>>>> --- a/xen/arch/arm/arm64/domctl.c
>>>>>>> +++ b/xen/arch/arm/arm64/domctl.c
>>>>>>> @@ -14,6 +14,8 @@
>>>>>>>
>>>>>>>  static long switch_mode(struct domain *d, enum domain_type type)
>>>>>>>  {
>>>>>>> +    struct vcpu *v;
>>>>>>> +
>>>>>>>      if ( d == NULL )
>>>>>>>          return -EINVAL;
>>>>>>>      if ( d->tot_pages != 0 )
>>>>>>> @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d, enum
>>>>>>> domain_type type)
>>>>>>>
>>>>>>>      d->arch.type = type;
>>>>>>>
>>>>>>> +    if ( is_64bit_domain(d) )
>>>>>>> +        for_each_vcpu(d, v)
>>>>>>> +            vcpu_switch_to_aarch64_mode(v);
>>>>>>> +
>>>>>>>      return 0;
>>>>>>>  }
>>>>>>>
>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>> index 5d18bb0..69c2854 100644
>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>> @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
>>>>>>>      free_xenheap_pages(v->arch.stack, STACK_ORDER);
>>>>>>>  }
>>>>>>>
>>>>>>> +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
>>>>>>> +{
>>>>>>> +    v->arch.hcr_el2 |= HCR_RW;
>>>>>>> +}
>>>>>>
>>>>>> if possible, I would write it as a static inline function
>>>>>>
>>>>>
>>>>> I had tried to write it as a static inline function in asm/domain.h
>>>>> But while the first source file (arm64/asm-offsets.c) include
>>>>> xen/sched.h wanted to compile this inline function it could not find
>>>>> 'struct vcpu'. Because the xen/sched.h included the asm/domain.h
>>>>> but defined the vcpu type later. Even though we had included the
>>>>> xen/sched.h in asm/domain.h already.
>>>>
>>>> It might be too complex to disentangle. In this case, there isn't much
>>>> type safety to be gained by using a static inline over a macro, so it
>>>> would be OK to use a macro for this.
>>>
>>> It is not like vCPU will be switch to AArch64 mode often? Only once at vCPU
>>> creation time.
>>>
>>> So what do we gain to switch to static inline or even macro?
>>
>> To turn 5 lines of code into 1. Obviously it's no big deal.
>>
>
> Ok, I think switch to macro is easier then static inline. I will switch
> it to a macro in next version.

Nack from my side. We are getting ride of macro in Xen in favor of 
static inline in the whole. I see no point here to do the macro, except 
saving 4 lines....

I don't want to see us in this game of trying to get the fewer number of 
lines just to claim we are small. Clarity and typesafety first.

In this case, a macro is not clarity nor safe.


Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 08/18] xen/arm: Introduce a initcall to update cpu_hwcaps by serror_op

[Stefano Stabellini]

On Fri, 17 Mar 2017, Wei Chen wrote:
> Hi Stefano,
> On 2017/3/17 7:30, Stefano Stabellini wrote:
> > On Mon, 13 Mar 2017, Wei Chen wrote:
> >> In the later patches of this series, we want to use the alternative
> >> patching framework to avoid checking serror_op in every entries.
> >> So we define a new cpu feature "SKIP_CHECK_PENDING_VSERROR" for
> >> serror_op. When serror_op is not equal to SERROR_DIVERSE, this
> >> feature will be set to cpu_hwcaps.
> >>
> >> But we could not update cpu_hwcaps directly in the serror parameter
> >> parsing function. Because if the serror parameter is not placed in
> >> the command line, the parsing function would not be invoked.
> >
> > Wait, the only way to set serrors_op != SERRORS_DIVERSE is to pass the
> > "serror" command line parameter. The parsing function is always invoked
> > if serrors_op != SERRORS_DIVERSE.
> >
> 
> Yes, but that because we set serrors_op to SERRORS_DIVERSE by default.
> If we change the policy and set serrors_op to SERRORS_PANIC by default, 
> then the parsing function would never be invoked.

Ah, I see. It's not actually "to guarantee the cpu_hwcaps can be updated
no matter the serror parameter is placed in the command line or not",
because today it would always be the case.

It's to future-proof the code, in case one day we'll change the default
SERRORS behavior.


> >
> >> So, we introduce this initcall to guarantee the cpu_hwcaps can be
> >> updated no matter the serror parameter is placed in the command line
> >> or not.
> >>
> >> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> >> ---
> >>  xen/arch/arm/traps.c             | 9 +++++++++
> >>  xen/include/asm-arm/cpufeature.h | 3 ++-
> >>  2 files changed, 11 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> >> index 5e31699..053b7fc 100644
> >> --- a/xen/arch/arm/traps.c
> >> +++ b/xen/arch/arm/traps.c
> >> @@ -134,6 +134,15 @@ static void __init parse_serrors_behavior(const char *str)
> >>  }
> >>  custom_param("serrors", parse_serrors_behavior);
> >>
> >> +static __init int update_serrors_cpu_caps(void)
> >
> > I think the coding style is "static int __init" ...
> >
> 
> Oh, thanks, I will fix it.
> 
> >
> >> +{
> >> +    if ( serrors_op != SERRORS_DIVERSE )
> >> +        cpus_set_cap(SKIP_CHECK_PENDING_VSERROR);
> >> +
> >> +    return 0;
> >> +}
> >> +__initcall(update_serrors_cpu_caps);
> >> +
> >>  register_t get_default_hcr_flags(void)
> >>  {
> >>      return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
> >> diff --git a/xen/include/asm-arm/cpufeature.h b/xen/include/asm-arm/cpufeature.h
> >> index c0a25ae..ec3f9e5 100644
> >> --- a/xen/include/asm-arm/cpufeature.h
> >> +++ b/xen/include/asm-arm/cpufeature.h
> >> @@ -40,8 +40,9 @@
> >>  #define ARM32_WORKAROUND_766422 2
> >>  #define ARM64_WORKAROUND_834220 3
> >>  #define LIVEPATCH_FEATURE   4
> >> +#define SKIP_CHECK_PENDING_VSERROR 5
> >>
> >> -#define ARM_NCAPS           5
> >> +#define ARM_NCAPS           6
> >>
> >>  #ifndef __ASSEMBLY__
> >>
> >> --
> >> 2.7.4
> >>
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> https://lists.xen.org/xen-devel
> >>
> >
> 
> 
> -- 
> Regards,
> Wei Chen
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/18] xen/arm: Avoid setting/clearing HCR_RW at every context switch

[Stefano Stabellini]

On Fri, 17 Mar 2017, Julien Grall wrote:
> Hi Wei,
> 
> On 03/17/2017 06:51 AM, Wei Chen wrote:
> > Hi Stefano,
> > 
> > On 2017/3/17 7:17, Stefano Stabellini wrote:
> > > On Thu, 16 Mar 2017, Julien Grall wrote:
> > > > On 03/16/2017 10:40 PM, Stefano Stabellini wrote:
> > > > > On Wed, 15 Mar 2017, Wei Chen wrote:
> > > > > > Hi Stefano,
> > > > > > 
> > > > > > On 2017/3/15 8:25, Stefano Stabellini wrote:
> > > > > > > On Mon, 13 Mar 2017, Wei Chen wrote:
> > > > > > > > The HCR_EL2 flags for 64-bit and 32-bit domains are different.
> > > > > > > > But
> > > > > > > > when we initialized the HCR_EL2 for vcpu0 of Dom0 and all vcpus
> > > > > > > > of
> > > > > > > > DomU in vcpu_initialise, we didn't know the domain's address
> > > > > > > > size
> > > > > > > > information. We had to use compatible flags to initialize
> > > > > > > > HCR_EL2,
> > > > > > > > and set HCR_RW for 64-bit domain or clear HCR_RW for 32-bit
> > > > > > > > domain
> > > > > > > > at every context switch.
> > > > > > > > 
> > > > > > > > But, after we added the HCR_EL2 to vcpu's context, this
> > > > > > > > behaviour
> > > > > > > > seems a little fussy. We can update the HCR_RW bit in vcpu's
> > > > > > > > context
> > > > > > > > as soon as we get the domain's address size to avoid
> > > > > > > > setting/clearing
> > > > > > > > HCR_RW at every context switch.
> > > > > > > > 
> > > > > > > > Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> > > > > > > > ---
> > > > > > > >  xen/arch/arm/arm64/domctl.c  | 6 ++++++
> > > > > > > >  xen/arch/arm/domain.c        | 5 +++++
> > > > > > > >  xen/arch/arm/domain_build.c  | 7 +++++++
> > > > > > > >  xen/arch/arm/p2m.c           | 5 -----
> > > > > > > >  xen/include/asm-arm/domain.h | 1 +
> > > > > > > >  5 files changed, 19 insertions(+), 5 deletions(-)
> > > > > > > > 
> > > > > > > > diff --git a/xen/arch/arm/arm64/domctl.c
> > > > > > > > b/xen/arch/arm/arm64/domctl.c
> > > > > > > > index 44e1e7b..ab8781f 100644
> > > > > > > > --- a/xen/arch/arm/arm64/domctl.c
> > > > > > > > +++ b/xen/arch/arm/arm64/domctl.c
> > > > > > > > @@ -14,6 +14,8 @@
> > > > > > > > 
> > > > > > > >  static long switch_mode(struct domain *d, enum domain_type
> > > > > > > > type)
> > > > > > > >  {
> > > > > > > > +    struct vcpu *v;
> > > > > > > > +
> > > > > > > >      if ( d == NULL )
> > > > > > > >          return -EINVAL;
> > > > > > > >      if ( d->tot_pages != 0 )
> > > > > > > > @@ -23,6 +25,10 @@ static long switch_mode(struct domain *d,
> > > > > > > > enum
> > > > > > > > domain_type type)
> > > > > > > > 
> > > > > > > >      d->arch.type = type;
> > > > > > > > 
> > > > > > > > +    if ( is_64bit_domain(d) )
> > > > > > > > +        for_each_vcpu(d, v)
> > > > > > > > +            vcpu_switch_to_aarch64_mode(v);
> > > > > > > > +
> > > > > > > >      return 0;
> > > > > > > >  }
> > > > > > > > 
> > > > > > > > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > > > > > > > index 5d18bb0..69c2854 100644
> > > > > > > > --- a/xen/arch/arm/domain.c
> > > > > > > > +++ b/xen/arch/arm/domain.c
> > > > > > > > @@ -537,6 +537,11 @@ void vcpu_destroy(struct vcpu *v)
> > > > > > > >      free_xenheap_pages(v->arch.stack, STACK_ORDER);
> > > > > > > >  }
> > > > > > > > 
> > > > > > > > +void vcpu_switch_to_aarch64_mode(struct vcpu *v)
> > > > > > > > +{
> > > > > > > > +    v->arch.hcr_el2 |= HCR_RW;
> > > > > > > > +}
> > > > > > > 
> > > > > > > if possible, I would write it as a static inline function
> > > > > > > 
> > > > > > 
> > > > > > I had tried to write it as a static inline function in asm/domain.h
> > > > > > But while the first source file (arm64/asm-offsets.c) include
> > > > > > xen/sched.h wanted to compile this inline function it could not find
> > > > > > 'struct vcpu'. Because the xen/sched.h included the asm/domain.h
> > > > > > but defined the vcpu type later. Even though we had included the
> > > > > > xen/sched.h in asm/domain.h already.
> > > > > 
> > > > > It might be too complex to disentangle. In this case, there isn't much
> > > > > type safety to be gained by using a static inline over a macro, so it
> > > > > would be OK to use a macro for this.
> > > > 
> > > > It is not like vCPU will be switch to AArch64 mode often? Only once at
> > > > vCPU
> > > > creation time.
> > > > 
> > > > So what do we gain to switch to static inline or even macro?
> > > 
> > > To turn 5 lines of code into 1. Obviously it's no big deal.
> > > 
> > 
> > Ok, I think switch to macro is easier then static inline. I will switch
> > it to a macro in next version.
> 
> Nack from my side. We are getting ride of macro in Xen in favor of static
> inline in the whole. I see no point here to do the macro, except saving 4
> lines....
> 
> I don't want to see us in this game of trying to get the fewer number of lines
> just to claim we are small. Clarity and typesafety first.
> 
> In this case, a macro is not clarity nor safe.

/me shakes my head

As everybody knows, I am no fan of macros, but they have a place. In
fact, we have a few of them already in the asm-arm headers. But arguing
about this is a waste of my keystrokes.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 08/18] xen/arm: Introduce a initcall to update cpu_hwcaps by serror_op

[Wei Chen]

Hi Stefano,

On 2017/3/18 1:21, Stefano Stabellini wrote:
> On Fri, 17 Mar 2017, Wei Chen wrote:
>> Hi Stefano,
>> On 2017/3/17 7:30, Stefano Stabellini wrote:
>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>> In the later patches of this series, we want to use the alternative
>>>> patching framework to avoid checking serror_op in every entries.
>>>> So we define a new cpu feature "SKIP_CHECK_PENDING_VSERROR" for
>>>> serror_op. When serror_op is not equal to SERROR_DIVERSE, this
>>>> feature will be set to cpu_hwcaps.
>>>>
>>>> But we could not update cpu_hwcaps directly in the serror parameter
>>>> parsing function. Because if the serror parameter is not placed in
>>>> the command line, the parsing function would not be invoked.
>>>
>>> Wait, the only way to set serrors_op != SERRORS_DIVERSE is to pass the
>>> "serror" command line parameter. The parsing function is always invoked
>>> if serrors_op != SERRORS_DIVERSE.
>>>
>>
>> Yes, but that because we set serrors_op to SERRORS_DIVERSE by default.
>> If we change the policy and set serrors_op to SERRORS_PANIC by default,
>> then the parsing function would never be invoked.
>
> Ah, I see. It's not actually "to guarantee the cpu_hwcaps can be updated
> no matter the serror parameter is placed in the command line or not",
> because today it would always be the case.
>
> It's to future-proof the code, in case one day we'll change the default
> SERRORS behavior.
>

Yes, it's future-proof. I will update the commit message to be more
accurate.

>
>>>
>>>> So, we introduce this initcall to guarantee the cpu_hwcaps can be
>>>> updated no matter the serror parameter is placed in the command line
>>>> or not.
>>>>
>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>> ---
>>>>  xen/arch/arm/traps.c             | 9 +++++++++
>>>>  xen/include/asm-arm/cpufeature.h | 3 ++-
>>>>  2 files changed, 11 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>>>> index 5e31699..053b7fc 100644
>>>> --- a/xen/arch/arm/traps.c
>>>> +++ b/xen/arch/arm/traps.c
>>>> @@ -134,6 +134,15 @@ static void __init parse_serrors_behavior(const char *str)
>>>>  }
>>>>  custom_param("serrors", parse_serrors_behavior);
>>>>
>>>> +static __init int update_serrors_cpu_caps(void)
>>>
>>> I think the coding style is "static int __init" ...
>>>
>>
>> Oh, thanks, I will fix it.
>>
>>>
>>>> +{
>>>> +    if ( serrors_op != SERRORS_DIVERSE )
>>>> +        cpus_set_cap(SKIP_CHECK_PENDING_VSERROR);
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +__initcall(update_serrors_cpu_caps);
>>>> +
>>>>  register_t get_default_hcr_flags(void)
>>>>  {
>>>>      return  (HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
>>>> diff --git a/xen/include/asm-arm/cpufeature.h b/xen/include/asm-arm/cpufeature.h
>>>> index c0a25ae..ec3f9e5 100644
>>>> --- a/xen/include/asm-arm/cpufeature.h
>>>> +++ b/xen/include/asm-arm/cpufeature.h
>>>> @@ -40,8 +40,9 @@
>>>>  #define ARM32_WORKAROUND_766422 2
>>>>  #define ARM64_WORKAROUND_834220 3
>>>>  #define LIVEPATCH_FEATURE   4
>>>> +#define SKIP_CHECK_PENDING_VSERROR 5
>>>>
>>>> -#define ARM_NCAPS           5
>>>> +#define ARM_NCAPS           6
>>>>
>>>>  #ifndef __ASSEMBLY__
>>>>
>>>> --
>>>> 2.7.4
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xen.org
>>>> https://lists.xen.org/xen-devel
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Wei Chen
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> Currently, we masked the Abort/SError bit in Xen exception entries.
> So Xen could not capture any Abort/SError while it's running.
> Now, Xen has the ability to handle the Abort/SError, we should unmask
> the Abort/SError bit by default to let Xen capture Abort/SError while
> it's running.
> 
> But in order to avoid receiving nested asynchronous abort, we don't
> unmask Abort/SError bit in hyp_error and trap_data_abort.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
> We haven't done this before, so I don't know how can this change
> will affect the Xen. If the IRQ and Abort take place at the same
> time, how can we handle them?

If the abort is for Xen, the hypervisor will crash and that's the end of
it. If the abort is for the guest, Xen will inject it into the VM, then
it will return from handling the abort, going back to handling the IRQ
as before. Isn't that right?


> If an abort is taking place while we're handling the IRQ, the program
> jump to abort exception, and then enable the IRQ. In this case, what
> will happen? So I think I need more discussions from community.

Do you know of an example scenario where Xen could have a problem with
this?


> ---
>  xen/arch/arm/arm32/entry.S | 15 ++++++++++++++-
>  xen/arch/arm/arm64/entry.S | 13 ++++++++-----
>  2 files changed, 22 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
> index 79929ca..4d46239 100644
> --- a/xen/arch/arm/arm32/entry.S
> +++ b/xen/arch/arm/arm32/entry.S
> @@ -125,6 +125,7 @@ abort_guest_exit_end:
>  trap_##trap:                                                            \
>          SAVE_ALL;                                                       \
>          cpsie i;        /* local_irq_enable */                          \
> +        cpsie a;        /* asynchronous abort enable */                 \
>          adr lr, return_from_trap;                                       \
>          mov r0, sp;                                                     \
>          mov r11, sp;                                                    \
> @@ -135,6 +136,18 @@ trap_##trap:                                                            \
>          ALIGN;                                                          \
>  trap_##trap:                                                            \
>          SAVE_ALL;                                                       \
> +        cpsie a;        /* asynchronous abort enable */                 \
> +        adr lr, return_from_trap;                                       \
> +        mov r0, sp;                                                     \
> +        mov r11, sp;                                                    \
> +        bic sp, #7; /* Align the stack pointer (noop on guest trap) */  \
> +        b do_trap_##trap
> +
> +#define DEFINE_TRAP_ENTRY_NOABORT(trap)                                 \
> +        ALIGN;                                                          \
> +trap_##trap:                                                            \
> +        SAVE_ALL;                                                       \
> +        cpsie i;        /* local_irq_enable */                          \
>          adr lr, return_from_trap;                                       \
>          mov r0, sp;                                                     \
>          mov r11, sp;                                                    \
> @@ -155,10 +168,10 @@ GLOBAL(hyp_traps_vector)
>  DEFINE_TRAP_ENTRY(undefined_instruction)
>  DEFINE_TRAP_ENTRY(supervisor_call)
>  DEFINE_TRAP_ENTRY(prefetch_abort)
> -DEFINE_TRAP_ENTRY(data_abort)
>  DEFINE_TRAP_ENTRY(hypervisor)
>  DEFINE_TRAP_ENTRY_NOIRQ(irq)
>  DEFINE_TRAP_ENTRY_NOIRQ(fiq)
> +DEFINE_TRAP_ENTRY_NOABORT(data_abort)
>  
>  return_from_trap:
>          mov sp, r11
> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
> index 8d5a890..0401a41 100644
> --- a/xen/arch/arm/arm64/entry.S
> +++ b/xen/arch/arm/arm64/entry.S
> @@ -187,13 +187,14 @@ hyp_error:
>  /* Traps taken in Current EL with SP_ELx */
>  hyp_sync:
>          entry   hyp=1
> -        msr     daifclr, #2
> +        msr     daifclr, #6
>          mov     x0, sp
>          bl      do_trap_hypervisor
>          exit    hyp=1
>  
>  hyp_irq:
>          entry   hyp=1
> +        msr     daifclr, #4
>          mov     x0, sp
>          bl      do_trap_irq
>          exit    hyp=1
> @@ -208,7 +209,7 @@ guest_sync:
>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>                      "nop; nop",
>                      SKIP_CHECK_PENDING_VSERROR)
> -        msr     daifclr, #2
> +        msr     daifclr, #6
>          mov     x0, sp
>          bl      do_trap_hypervisor
>  1:
> @@ -224,6 +225,7 @@ guest_irq:
>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>                      "nop; nop",
>                      SKIP_CHECK_PENDING_VSERROR)
> +        msr     daifclr, #4
>          mov     x0, sp
>          bl      do_trap_irq
>  1:
> @@ -235,7 +237,7 @@ guest_fiq_invalid:
>  
>  guest_error:
>          entry   hyp=0, compat=0
> -        msr     daifclr, #2
> +        msr     daifclr, #6
>          mov     x0, sp
>          bl      do_trap_guest_serror
>          exit    hyp=0, compat=0
> @@ -250,7 +252,7 @@ guest_sync_compat:
>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>                      "nop; nop",
>                      SKIP_CHECK_PENDING_VSERROR)
> -        msr     daifclr, #2
> +        msr     daifclr, #6
>          mov     x0, sp
>          bl      do_trap_hypervisor
>  1:
> @@ -266,6 +268,7 @@ guest_irq_compat:
>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>                      "nop; nop",
>                      SKIP_CHECK_PENDING_VSERROR)
> +        msr     daifclr, #4
>          mov     x0, sp
>          bl      do_trap_irq
>  1:
> @@ -277,7 +280,7 @@ guest_fiq_invalid_compat:
>  
>  guest_error_compat:
>          entry   hyp=0, compat=1
> -        msr     daifclr, #2
> +        msr     daifclr, #6
>          mov     x0, sp
>          bl      do_trap_guest_serror
>          exit    hyp=0, compat=1
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 15/18] xen/arm: Introduce a helper to synchronize SError

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> We may have to isolate the SError between the context switch of
> 2 vCPUs or may have to prevent slipping hypervisor SError to guest.

I thought the problem is not the risk of slipping hypervisor SErrors to
guest, but the risk of slipping previous guest SErrors to the next
guest. Is that right?


> So we need a helper to synchronize SError before context switching
> or returning to guest.
> 
> This function will be used by the later patches in this series, we
> use "#if 0" to disable it temporarily to remove compiler warnning.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/traps.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 44a0281..ee7865b 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2899,6 +2899,17 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
>      }
>  }
>  
> +#if 0
> +static void synchronize_serror(void)
> +{
> +    /* Synchronize against in-flight ld/st. */
> +    dsb(sy);
> +
> +    /* A single instruction exception window */
> +    isb();
> +}
> +#endif
> +
>  asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
>  {
>      enter_hypervisor_head(regs);
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 15/18] xen/arm: Introduce a helper to synchronize SError

[Stefano Stabellini]

On Mon, 20 Mar 2017, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
> > We may have to isolate the SError between the context switch of
> > 2 vCPUs or may have to prevent slipping hypervisor SError to guest.
> 
> I thought the problem is not the risk of slipping hypervisor SErrors to
> guest, but the risk of slipping previous guest SErrors to the next
> guest. Is that right?

I can see from the following patches that both situations are a risk.
Although the patch could benefit from a better commit description:

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>



> > So we need a helper to synchronize SError before context switching
> > or returning to guest.
> > 
> > This function will be used by the later patches in this series, we
> > use "#if 0" to disable it temporarily to remove compiler warnning.
> > 
> > Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> > ---
> >  xen/arch/arm/traps.c | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> > 
> > diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> > index 44a0281..ee7865b 100644
> > --- a/xen/arch/arm/traps.c
> > +++ b/xen/arch/arm/traps.c
> > @@ -2899,6 +2899,17 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
> >      }
> >  }
> >  
> > +#if 0
> > +static void synchronize_serror(void)
> > +{
> > +    /* Synchronize against in-flight ld/st. */
> > +    dsb(sy);
> > +
> > +    /* A single instruction exception window */
> > +    isb();
> > +}
> > +#endif
> > +
> >  asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
> >  {
> >      enter_hypervisor_head(regs);
> > -- 
> > 2.7.4
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > https://lists.xen.org/xen-devel
> > 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 16/18] xen/arm: Isolate the SError between the context switch of 2 vCPUs

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> If there is a pending SError while we are doing context switch, if the
> SError handle option is "FORWARD", We have to guranatee this serror to
> be caught by current vCPU, otherwise it will be caught by next vCPU and
> be forwarded to this wrong vCPU.
> 
> We don't want to export serror_op accessing to other source files and
> use serror_op every place, so we add a helper to synchronize SError for
> context switching. The synchronize_serror has been used in this helper,
> so the "#if 0" can be removed.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> ---
>  xen/arch/arm/domain.c           |  2 ++
>  xen/arch/arm/traps.c            | 14 ++++++++++++--
>  xen/include/asm-arm/processor.h |  2 ++
>  3 files changed, 16 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 69c2854..a547fcd 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -312,6 +312,8 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
>  
>      local_irq_disable();
>  
> +    prevent_forward_serror_to_next_vcpu();
> +
>      set_current(next);
>  
>      prev = __context_switch(prev, next);
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index ee7865b..b8c8389 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2899,7 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
>      }
>  }
>  
> -#if 0
>  static void synchronize_serror(void)
>  {
>      /* Synchronize against in-flight ld/st. */
> @@ -2908,7 +2907,18 @@ static void synchronize_serror(void)
>      /* A single instruction exception window */
>      isb();
>  }
> -#endif
> +
> +/*
> + * If the SErrors option is "FORWARD", we have to prevent forwarding
> + * serror to wrong vCPU. So before context switch, we have to use the
> + * synchronize_serror to guarantee that the pending serror would be
> + * caught by current vCPU.
> + */
> +void prevent_forward_serror_to_next_vcpu(void)
> +{
> +    if ( serrors_op == SERRORS_FORWARD )
> +        synchronize_serror();
> +}

I dislike introducing so many 2 lines functions. I would get rid of
prevent_forward_serror_to_next_vcpu and just add

   if ( serrors_op == SERRORS_FORWARD )
       synchronize_serror();

to context_switch, and I would make synchronize_serror a static inline.


>  asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
>  {
> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
> index afad78c..3b43234 100644
> --- a/xen/include/asm-arm/processor.h
> +++ b/xen/include/asm-arm/processor.h
> @@ -711,6 +711,8 @@ void do_trap_hyp_serror(struct cpu_user_regs *regs);
>  
>  void do_trap_guest_serror(struct cpu_user_regs *regs);
>  
> +void prevent_forward_serror_to_next_vcpu(void);
> +
>  /* Functions for pending virtual abort checking window. */
>  void abort_guest_exit_start(void);
>  void abort_guest_exit_end(void);
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 17/18] xen/arm: Prevent slipping hypervisor SError to guest

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> If there is a pending SError while we're returning from trap. If the
> SError handle option is "DIVERSE", we have to prevent slipping this
> hypervisor SError to guest. So we have to use the dsb/isb to guarantee
> that the pending hypervisor SError would be caught in hypervisor before
> return to guest.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  xen/arch/arm/traps.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index b8c8389..3b84e80 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2953,6 +2953,16 @@ asmlinkage void leave_hypervisor_tail(void)
>          local_irq_disable();
>          if (!softirq_pending(smp_processor_id())) {
>              gic_inject();
> +
> +            /*
> +             * If the SErrors handle option is "DIVERSE", we have to prevent
> +             * slipping the hypervisor SError to guest. So before returning
> +             * from trap, we use the synchronize_serror to guarantee that the
> +             * pending SError would be caught in hypervisor.
> +             */
> +            if ( serrors_op == SERRORS_DIVERSE )
> +                synchronize_serror();
> +
>              WRITE_SYSREG(current->arch.hcr_el2, HCR_EL2);
>              return;
>          }
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 18/18] xen/arm: Handle guest external abort as guest SError

[Stefano Stabellini]

On Mon, 13 Mar 2017, Wei Chen wrote:
> The guest generated external data/instruction aborts can be treated
> as guest SErrors. We already have a handler to handle the SErrors,
> so we can reuse this handler to handle guest external aborts.
> 
> Signed-off-by: Wei Chen <Wei.Chen@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  xen/arch/arm/traps.c | 14 ++++++--------
>  1 file changed, 6 insertions(+), 8 deletions(-)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 3b84e80..24511e5 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2558,12 +2558,12 @@ static void do_trap_instr_abort_guest(struct cpu_user_regs *regs,
>  
>      /*
>       * If this bit has been set, it means that this instruction abort is caused
> -     * by a guest external abort. Currently we crash the guest to protect the
> -     * hypervisor. In future one can better handle this by injecting a virtual
> -     * abort to the guest.
> +     * by a guest external abort. We can handle this instruction abort as guest
> +     * SError.
>       */
>      if ( hsr.iabt.eat )
> -        domain_crash_synchronous();
> +        return __do_trap_serror(regs, true);
> +
>  
>      if ( hpfar_is_valid(hsr.iabt.s1ptw, fsc) )
>          gpa = get_faulting_ipa(gva);
> @@ -2661,12 +2661,10 @@ static void do_trap_data_abort_guest(struct cpu_user_regs *regs,
>  
>      /*
>       * If this bit has been set, it means that this data abort is caused
> -     * by a guest external abort. Currently we crash the guest to protect the
> -     * hypervisor. In future one can better handle this by injecting a virtual
> -     * abort to the guest.
> +     * by a guest external abort. We treat this data abort as guest SError.
>       */
>      if ( dabt.eat )
> -        domain_crash_synchronous();
> +        return __do_trap_serror(regs, true);
>  
>      info.dabt = dabt;
>  #ifdef CONFIG_ARM_32
> -- 
> 2.7.4
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Stefano Stabellini]

On Thu, 16 Mar 2017, Julien Grall wrote:
> Hi Stefano
> 
> On 03/16/2017 10:33 PM, Stefano Stabellini wrote:
> > On Wed, 15 Mar 2017, Julien Grall wrote:
> > > Hi Wei,
> > > 
> > > On 15/03/17 08:34, Wei Chen wrote:
> > > > On 2017/3/15 8:25, Stefano Stabellini wrote:
> > > > > On Mon, 13 Mar 2017, Wei Chen wrote:
> > > > > > Different domains may have different HCR_EL2 flags. For example, the
> > > > > > 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
> > > > > > we give each domain a default HCR_EL2 value and save it in the
> > > > > > VCPU's
> > > > > > context.
> > > > > > 
> > > > > > HCR_EL2 register has only one bit can be updated automatically
> > > > > > without
> > > > > > explicit write (HCR_VSE). But we haven't used this bit currently, so
> > > > > > we can consider that the HCR_EL2 register will not be modified while
> > > > > > the guest is running. So save the HCR_EL2 while guest exiting to
> > > > > > hypervisor is not neccessary. We just have to restore this register
> > > > > > for
> > > > > > each VCPU while leaving hypervisor.
> > > > > > 
> > > > > > We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
> > > > > > ctxt_switch_to. Because the leave_hypervisor_tail is the closest
> > > > > > place
> > > > > > to the exception return. In this case, we don't need to warrant the
> > > > > > HCR_EL2 will not be changed between ctxt_switch_to and exception
> > > > > > return.
> > > > > 
> > > > > Please explain a bit better why it is good to restore HCR_EL2 in
> > > > > leave_hypervisor_tail. Why is it a good thing that is closer to the
> > > > > exception return? What can be the cause of a change in HCR_EL2?
> > > > > 
> > > > 
> > > > Ok, I will try to improve it in next version. In current Xen code, I
> > > > can't see any code would change the HCR_EL2 between ctxt_switch_to and
> > > > return_from_trap. But my concern is that, in the future, if someone
> > > > want to insert some HCR_EL2 change code between them, we can't guarantee
> > > > he will restore correct HCR_EL2 value for current vcpu.
> > > 
> > > Well, the purpose of this series is to inject a virtual SError to the
> > > guest
> > > when a host SError is happening. The host SError will be received in the
> > > hypervisor whilst the vCPU is running and no context switch (e.g call to
> > > ctxt_switch) may happen if the scheduler decides to keep the vCPU running.
> > > 
> > > Also, one could argue that HCR_EL2 could be modified on-fly in the
> > > function
> > > but we may have other places in the future which will modify HCR_EL2. For
> > > instance, this would be the case when the monitor app will gain support of
> > > inspecting some registers.
> > > 
> > > So we want a single place to restore HCR_EL2. And leave_hypervisor_tail is
> > > the
> > > best place for that.
> > 
> > You say that we want a single place to restore HCR_EL2, but this patch
> > introduces two places, one is p2m_restore_state, the other is
> > leave_hypervisor_tail. Are you saying Wei should remove the HCR_EL2
> > restore in p2m_restore_state and just keep the one in
> > leave_hypervisor_tail?
> 
> p2m_restore_state may be used to switch temporarily to a p2m state. For
> instance for TLB flushing or even doing VA -> PA translation. Though the later
> does not yet work quite well when not using the current vCPU.
> 
> Some bits in HCR_EL2 (specifically HCR_EL2.RW) will be used to know how to
> interpret the stage-1 page table as the encoding differ between AArch64 and
> AArch32.
>
> So I think we have to keep the one in p2m_restore_state. But I would like to
> keep the number very limited.

I understand, but restoring HCR_EL2 twice is very counter-intuitive. At
the very least we need a good comment above each of the two
write(HCR_EL2) calls.

Another option would be to only "fix" HCR_RW in p2m_restore_state, and
do the full restore in leave_hypervisor_tail. The purpose of doing that
is separation of responsibilities: it would be clearer what is the
responsibility of each of the two.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 15/18] xen/arm: Introduce a helper to synchronize SError

[Wei Chen]

Hi Stefano,

On 2017/3/21 5:44, Stefano Stabellini wrote:
> On Mon, 20 Mar 2017, Stefano Stabellini wrote:
>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>> We may have to isolate the SError between the context switch of
>>> 2 vCPUs or may have to prevent slipping hypervisor SError to guest.
>>
>> I thought the problem is not the risk of slipping hypervisor SErrors to
>> guest, but the risk of slipping previous guest SErrors to the next
>> guest. Is that right?
>
> I can see from the following patches that both situations are a risk.
> Although the patch could benefit from a better commit description:
>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>
>

Thanks, I will update the commit message in next version.

>
>>> So we need a helper to synchronize SError before context switching
>>> or returning to guest.
>>>
>>> This function will be used by the later patches in this series, we
>>> use "#if 0" to disable it temporarily to remove compiler warnning.
>>>
>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>> ---
>>>  xen/arch/arm/traps.c | 11 +++++++++++
>>>  1 file changed, 11 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>>> index 44a0281..ee7865b 100644
>>> --- a/xen/arch/arm/traps.c
>>> +++ b/xen/arch/arm/traps.c
>>> @@ -2899,6 +2899,17 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
>>>      }
>>>  }
>>>
>>> +#if 0
>>> +static void synchronize_serror(void)
>>> +{
>>> +    /* Synchronize against in-flight ld/st. */
>>> +    dsb(sy);
>>> +
>>> +    /* A single instruction exception window */
>>> +    isb();
>>> +}
>>> +#endif
>>> +
>>>  asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
>>>  {
>>>      enter_hypervisor_head(regs);
>>> --
>>> 2.7.4
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> https://lists.xen.org/xen-devel
>>>
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Wei Chen]

Hi Stefano,

On 2017/3/21 5:38, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> Currently, we masked the Abort/SError bit in Xen exception entries.
>> So Xen could not capture any Abort/SError while it's running.
>> Now, Xen has the ability to handle the Abort/SError, we should unmask
>> the Abort/SError bit by default to let Xen capture Abort/SError while
>> it's running.
>>
>> But in order to avoid receiving nested asynchronous abort, we don't
>> unmask Abort/SError bit in hyp_error and trap_data_abort.
>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>> ---
>> We haven't done this before, so I don't know how can this change
>> will affect the Xen. If the IRQ and Abort take place at the same
>> time, how can we handle them?
>
> If the abort is for Xen, the hypervisor will crash and that's the end of

Before the system crash, we have enable the IRQ, so that would not be
the end if an IRQ happens at the same time.

> it. If the abort is for the guest, Xen will inject it into the VM, then

Before we have inject the abort to VM, we have enable the IRQ.

> it will return from handling the abort, going back to handling the IRQ
> as before. Isn't that right?

If the abort has higher priority then IRQ, it's right.

>
>
>> If an abort is taking place while we're handling the IRQ, the program
>> jump to abort exception, and then enable the IRQ. In this case, what
>> will happen? So I think I need more discussions from community.
>
> Do you know of an example scenario where Xen could have a problem with
> this?
>

For example,
1. Trigger a SError in hypervisor.
2. Jump to hyp_error to handle SError.
3. Enable IRQ in hyp_error before PANIC
4. A timer IRQ happens.
5. Jump to hyp_irq and unmask abort again.
6. Jump hyp_error again?

>
>> ---
>>  xen/arch/arm/arm32/entry.S | 15 ++++++++++++++-
>>  xen/arch/arm/arm64/entry.S | 13 ++++++++-----
>>  2 files changed, 22 insertions(+), 6 deletions(-)
>>
>> diff --git a/xen/arch/arm/arm32/entry.S b/xen/arch/arm/arm32/entry.S
>> index 79929ca..4d46239 100644
>> --- a/xen/arch/arm/arm32/entry.S
>> +++ b/xen/arch/arm/arm32/entry.S
>> @@ -125,6 +125,7 @@ abort_guest_exit_end:
>>  trap_##trap:                                                            \
>>          SAVE_ALL;                                                       \
>>          cpsie i;        /* local_irq_enable */                          \
>> +        cpsie a;        /* asynchronous abort enable */                 \
>>          adr lr, return_from_trap;                                       \
>>          mov r0, sp;                                                     \
>>          mov r11, sp;                                                    \
>> @@ -135,6 +136,18 @@ trap_##trap:                                                            \
>>          ALIGN;                                                          \
>>  trap_##trap:                                                            \
>>          SAVE_ALL;                                                       \
>> +        cpsie a;        /* asynchronous abort enable */                 \
>> +        adr lr, return_from_trap;                                       \
>> +        mov r0, sp;                                                     \
>> +        mov r11, sp;                                                    \
>> +        bic sp, #7; /* Align the stack pointer (noop on guest trap) */  \
>> +        b do_trap_##trap
>> +
>> +#define DEFINE_TRAP_ENTRY_NOABORT(trap)                                 \
>> +        ALIGN;                                                          \
>> +trap_##trap:                                                            \
>> +        SAVE_ALL;                                                       \
>> +        cpsie i;        /* local_irq_enable */                          \
>>          adr lr, return_from_trap;                                       \
>>          mov r0, sp;                                                     \
>>          mov r11, sp;                                                    \
>> @@ -155,10 +168,10 @@ GLOBAL(hyp_traps_vector)
>>  DEFINE_TRAP_ENTRY(undefined_instruction)
>>  DEFINE_TRAP_ENTRY(supervisor_call)
>>  DEFINE_TRAP_ENTRY(prefetch_abort)
>> -DEFINE_TRAP_ENTRY(data_abort)
>>  DEFINE_TRAP_ENTRY(hypervisor)
>>  DEFINE_TRAP_ENTRY_NOIRQ(irq)
>>  DEFINE_TRAP_ENTRY_NOIRQ(fiq)
>> +DEFINE_TRAP_ENTRY_NOABORT(data_abort)
>>
>>  return_from_trap:
>>          mov sp, r11
>> diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S
>> index 8d5a890..0401a41 100644
>> --- a/xen/arch/arm/arm64/entry.S
>> +++ b/xen/arch/arm/arm64/entry.S
>> @@ -187,13 +187,14 @@ hyp_error:
>>  /* Traps taken in Current EL with SP_ELx */
>>  hyp_sync:
>>          entry   hyp=1
>> -        msr     daifclr, #2
>> +        msr     daifclr, #6
>>          mov     x0, sp
>>          bl      do_trap_hypervisor
>>          exit    hyp=1
>>
>>  hyp_irq:
>>          entry   hyp=1
>> +        msr     daifclr, #4
>>          mov     x0, sp
>>          bl      do_trap_irq
>>          exit    hyp=1
>> @@ -208,7 +209,7 @@ guest_sync:
>>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>>                      "nop; nop",
>>                      SKIP_CHECK_PENDING_VSERROR)
>> -        msr     daifclr, #2
>> +        msr     daifclr, #6
>>          mov     x0, sp
>>          bl      do_trap_hypervisor
>>  1:
>> @@ -224,6 +225,7 @@ guest_irq:
>>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>>                      "nop; nop",
>>                      SKIP_CHECK_PENDING_VSERROR)
>> +        msr     daifclr, #4
>>          mov     x0, sp
>>          bl      do_trap_irq
>>  1:
>> @@ -235,7 +237,7 @@ guest_fiq_invalid:
>>
>>  guest_error:
>>          entry   hyp=0, compat=0
>> -        msr     daifclr, #2
>> +        msr     daifclr, #6
>>          mov     x0, sp
>>          bl      do_trap_guest_serror
>>          exit    hyp=0, compat=0
>> @@ -250,7 +252,7 @@ guest_sync_compat:
>>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>>                      "nop; nop",
>>                      SKIP_CHECK_PENDING_VSERROR)
>> -        msr     daifclr, #2
>> +        msr     daifclr, #6
>>          mov     x0, sp
>>          bl      do_trap_hypervisor
>>  1:
>> @@ -266,6 +268,7 @@ guest_irq_compat:
>>          ALTERNATIVE("bl check_pending_vserror; cbnz x0, 1f",
>>                      "nop; nop",
>>                      SKIP_CHECK_PENDING_VSERROR)
>> +        msr     daifclr, #4
>>          mov     x0, sp
>>          bl      do_trap_irq
>>  1:
>> @@ -277,7 +280,7 @@ guest_fiq_invalid_compat:
>>
>>  guest_error_compat:
>>          entry   hyp=0, compat=1
>> -        msr     daifclr, #2
>> +        msr     daifclr, #6
>>          mov     x0, sp
>>          bl      do_trap_guest_serror
>>          exit    hyp=0, compat=1
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 16/18] xen/arm: Isolate the SError between the context switch of 2 vCPUs

[Wei Chen]

Hi Stefano,

On 2017/3/21 5:46, Stefano Stabellini wrote:
> On Mon, 13 Mar 2017, Wei Chen wrote:
>> If there is a pending SError while we are doing context switch, if the
>> SError handle option is "FORWARD", We have to guranatee this serror to
>> be caught by current vCPU, otherwise it will be caught by next vCPU and
>> be forwarded to this wrong vCPU.
>>
>> We don't want to export serror_op accessing to other source files and
>> use serror_op every place, so we add a helper to synchronize SError for
>> context switching. The synchronize_serror has been used in this helper,
>> so the "#if 0" can be removed.
>>
>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>> ---
>>  xen/arch/arm/domain.c           |  2 ++
>>  xen/arch/arm/traps.c            | 14 ++++++++++++--
>>  xen/include/asm-arm/processor.h |  2 ++
>>  3 files changed, 16 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index 69c2854..a547fcd 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -312,6 +312,8 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
>>
>>      local_irq_disable();
>>
>> +    prevent_forward_serror_to_next_vcpu();
>> +
>>      set_current(next);
>>
>>      prev = __context_switch(prev, next);
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index ee7865b..b8c8389 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -2899,7 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
>>      }
>>  }
>>
>> -#if 0
>>  static void synchronize_serror(void)
>>  {
>>      /* Synchronize against in-flight ld/st. */
>> @@ -2908,7 +2907,18 @@ static void synchronize_serror(void)
>>      /* A single instruction exception window */
>>      isb();
>>  }
>> -#endif
>> +
>> +/*
>> + * If the SErrors option is "FORWARD", we have to prevent forwarding
>> + * serror to wrong vCPU. So before context switch, we have to use the
>> + * synchronize_serror to guarantee that the pending serror would be
>> + * caught by current vCPU.
>> + */
>> +void prevent_forward_serror_to_next_vcpu(void)
>> +{
>> +    if ( serrors_op == SERRORS_FORWARD )
>> +        synchronize_serror();
>> +}
>
> I dislike introducing so many 2 lines functions. I would get rid of
> prevent_forward_serror_to_next_vcpu and just add
>
>    if ( serrors_op == SERRORS_FORWARD )
>        synchronize_serror();
>
> to context_switch, and I would make synchronize_serror a static inline.
>

I had done it before, but that made the serrors_op appear everywhere.
If export serrors_op to other files is acceptable, I would re-do it.

>
>>  asmlinkage void do_trap_hyp_serror(struct cpu_user_regs *regs)
>>  {
>> diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
>> index afad78c..3b43234 100644
>> --- a/xen/include/asm-arm/processor.h
>> +++ b/xen/include/asm-arm/processor.h
>> @@ -711,6 +711,8 @@ void do_trap_hyp_serror(struct cpu_user_regs *regs);
>>
>>  void do_trap_guest_serror(struct cpu_user_regs *regs);
>>
>> +void prevent_forward_serror_to_next_vcpu(void);
>> +
>>  /* Functions for pending virtual abort checking window. */
>>  void abort_guest_exit_start(void);
>>  void abort_guest_exit_end(void);
>> --
>> 2.7.4
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>>
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Julien Grall]

(CC Mark for the TLB question)

Hi Stefano,

On 21/03/17 00:31, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Julien Grall wrote:
>> Hi Stefano
>>
>> On 03/16/2017 10:33 PM, Stefano Stabellini wrote:
>>> On Wed, 15 Mar 2017, Julien Grall wrote:
>>>> Hi Wei,
>>>>
>>>> On 15/03/17 08:34, Wei Chen wrote:
>>>>> On 2017/3/15 8:25, Stefano Stabellini wrote:
>>>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>>>>> Different domains may have different HCR_EL2 flags. For example, the
>>>>>>> 64-bit domain needs HCR_RW flag but the 32-bit does not need it. So
>>>>>>> we give each domain a default HCR_EL2 value and save it in the
>>>>>>> VCPU's
>>>>>>> context.
>>>>>>>
>>>>>>> HCR_EL2 register has only one bit can be updated automatically
>>>>>>> without
>>>>>>> explicit write (HCR_VSE). But we haven't used this bit currently, so
>>>>>>> we can consider that the HCR_EL2 register will not be modified while
>>>>>>> the guest is running. So save the HCR_EL2 while guest exiting to
>>>>>>> hypervisor is not neccessary. We just have to restore this register
>>>>>>> for
>>>>>>> each VCPU while leaving hypervisor.
>>>>>>>
>>>>>>> We prefer to restore HCR_EL2 in leave_hypervisor_tail rather than in
>>>>>>> ctxt_switch_to. Because the leave_hypervisor_tail is the closest
>>>>>>> place
>>>>>>> to the exception return. In this case, we don't need to warrant the
>>>>>>> HCR_EL2 will not be changed between ctxt_switch_to and exception
>>>>>>> return.
>>>>>>
>>>>>> Please explain a bit better why it is good to restore HCR_EL2 in
>>>>>> leave_hypervisor_tail. Why is it a good thing that is closer to the
>>>>>> exception return? What can be the cause of a change in HCR_EL2?
>>>>>>
>>>>>
>>>>> Ok, I will try to improve it in next version. In current Xen code, I
>>>>> can't see any code would change the HCR_EL2 between ctxt_switch_to and
>>>>> return_from_trap. But my concern is that, in the future, if someone
>>>>> want to insert some HCR_EL2 change code between them, we can't guarantee
>>>>> he will restore correct HCR_EL2 value for current vcpu.
>>>>
>>>> Well, the purpose of this series is to inject a virtual SError to the
>>>> guest
>>>> when a host SError is happening. The host SError will be received in the
>>>> hypervisor whilst the vCPU is running and no context switch (e.g call to
>>>> ctxt_switch) may happen if the scheduler decides to keep the vCPU running.
>>>>
>>>> Also, one could argue that HCR_EL2 could be modified on-fly in the
>>>> function
>>>> but we may have other places in the future which will modify HCR_EL2. For
>>>> instance, this would be the case when the monitor app will gain support of
>>>> inspecting some registers.
>>>>
>>>> So we want a single place to restore HCR_EL2. And leave_hypervisor_tail is
>>>> the
>>>> best place for that.
>>>
>>> You say that we want a single place to restore HCR_EL2, but this patch
>>> introduces two places, one is p2m_restore_state, the other is
>>> leave_hypervisor_tail. Are you saying Wei should remove the HCR_EL2
>>> restore in p2m_restore_state and just keep the one in
>>> leave_hypervisor_tail?
>>
>> p2m_restore_state may be used to switch temporarily to a p2m state. For
>> instance for TLB flushing or even doing VA -> PA translation. Though the later
>> does not yet work quite well when not using the current vCPU.
>>
>> Some bits in HCR_EL2 (specifically HCR_EL2.RW) will be used to know how to
>> interpret the stage-1 page table as the encoding differ between AArch64 and
>> AArch32.
>>
>> So I think we have to keep the one in p2m_restore_state. But I would like to
>> keep the number very limited.
>
> I understand, but restoring HCR_EL2 twice is very counter-intuitive. At
> the very least we need a good comment above each of the two
> write(HCR_EL2) calls.
>
> Another option would be to only "fix" HCR_RW in p2m_restore_state, and
> do the full restore in leave_hypervisor_tail. The purpose of doing that
> is separation of responsibilities: it would be clearer what is the
> responsibility of each of the two.

I am not entirely sure if we can only restore HCR_RW. My concern is some 
of the bits are cached in the TLBs. Looking at the ARM ARM (both v7 and 
v8 see D4.7 in DDI0487A.k_iss10775): "In these descriptions, TLB entries 
for a translation regime for a particular Exception level are out of 
context when executing at a higher Exception level.". Which I interpret 
as TLB result cannot be cached when translating a guest VA to guest PA 
at EL2. So I guess restoring only HCR_RW might be fine.

But to be fair, the function p2m_restore_state is working by chance as 
technically more registers (SCTLR, TTBR,...) should be restored for some 
translation. I think it would benefit some rework but this is post Xen 
4.9 task.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Julien Grall]

Hi Wei,

On 22/03/17 08:49, Wei Chen wrote:
> Hi Stefano,
>
> On 2017/3/21 5:38, Stefano Stabellini wrote:
>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>> Currently, we masked the Abort/SError bit in Xen exception entries.
>>> So Xen could not capture any Abort/SError while it's running.
>>> Now, Xen has the ability to handle the Abort/SError, we should unmask
>>> the Abort/SError bit by default to let Xen capture Abort/SError while
>>> it's running.
>>>
>>> But in order to avoid receiving nested asynchronous abort, we don't
>>> unmask Abort/SError bit in hyp_error and trap_data_abort.
>>>
>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>> ---
>>> We haven't done this before, so I don't know how can this change
>>> will affect the Xen. If the IRQ and Abort take place at the same
>>> time, how can we handle them?
>>
>> If the abort is for Xen, the hypervisor will crash and that's the end of
>
> Before the system crash, we have enable the IRQ, so that would not be
> the end if an IRQ happens at the same time.
>
>> it. If the abort is for the guest, Xen will inject it into the VM, then
>
> Before we have inject the abort to VM, we have enable the IRQ.
>
>> it will return from handling the abort, going back to handling the IRQ
>> as before. Isn't that right?
>
> If the abort has higher priority then IRQ, it's right.
>
>>
>>
>>> If an abort is taking place while we're handling the IRQ, the program
>>> jump to abort exception, and then enable the IRQ. In this case, what
>>> will happen? So I think I need more discussions from community.
>>
>> Do you know of an example scenario where Xen could have a problem with
>> this?
>>
>
> For example,
> 1. Trigger a SError in hypervisor.
> 2. Jump to hyp_error to handle SError.
> 3. Enable IRQ in hyp_error before PANIC
> 4. A timer IRQ happens.
> 5. Jump to hyp_irq and unmask abort again.
> 6. Jump hyp_error again?

Technically you could end up in an infinite loop if hyp_error code 
generates a SError. It will stay pending until the end and then trigger 
again when SError is unmasked...

That's unfortunate but I don't think this is a big issue as if this is 
happening your platform is already doomed.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 16/18] xen/arm: Isolate the SError between the context switch of 2 vCPUs

[Julien Grall]

Hi Wei,

On 22/03/17 08:53, Wei Chen wrote:
> Hi Stefano,
>
> On 2017/3/21 5:46, Stefano Stabellini wrote:
>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>> If there is a pending SError while we are doing context switch, if the
>>> SError handle option is "FORWARD", We have to guranatee this serror to
>>> be caught by current vCPU, otherwise it will be caught by next vCPU and
>>> be forwarded to this wrong vCPU.
>>>
>>> We don't want to export serror_op accessing to other source files and
>>> use serror_op every place, so we add a helper to synchronize SError for
>>> context switching. The synchronize_serror has been used in this helper,
>>> so the "#if 0" can be removed.
>>>
>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>> ---
>>>  xen/arch/arm/domain.c           |  2 ++
>>>  xen/arch/arm/traps.c            | 14 ++++++++++++--
>>>  xen/include/asm-arm/processor.h |  2 ++
>>>  3 files changed, 16 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index 69c2854..a547fcd 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -312,6 +312,8 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
>>>
>>>      local_irq_disable();
>>>
>>> +    prevent_forward_serror_to_next_vcpu();
>>> +
>>>      set_current(next);
>>>
>>>      prev = __context_switch(prev, next);
>>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>>> index ee7865b..b8c8389 100644
>>> --- a/xen/arch/arm/traps.c
>>> +++ b/xen/arch/arm/traps.c
>>> @@ -2899,7 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
>>>      }
>>>  }
>>>
>>> -#if 0
>>>  static void synchronize_serror(void)
>>>  {
>>>      /* Synchronize against in-flight ld/st. */
>>> @@ -2908,7 +2907,18 @@ static void synchronize_serror(void)
>>>      /* A single instruction exception window */
>>>      isb();
>>>  }
>>> -#endif
>>> +
>>> +/*
>>> + * If the SErrors option is "FORWARD", we have to prevent forwarding
>>> + * serror to wrong vCPU. So before context switch, we have to use the
>>> + * synchronize_serror to guarantee that the pending serror would be
>>> + * caught by current vCPU.
>>> + */
>>> +void prevent_forward_serror_to_next_vcpu(void)
>>> +{
>>> +    if ( serrors_op == SERRORS_FORWARD )
>>> +        synchronize_serror();
>>> +}
>>
>> I dislike introducing so many 2 lines functions. I would get rid of
>> prevent_forward_serror_to_next_vcpu and just add
>>
>>    if ( serrors_op == SERRORS_FORWARD )
>>        synchronize_serror();
>>
>> to context_switch, and I would make synchronize_serror a static inline.
>>
>
> I had done it before, but that made the serrors_op appear everywhere.
> If export serrors_op to other files is acceptable, I would re-do it.

Why don't you introduce a new flag in cpu_hwcaps? This would allow us to 
take advantage of alternative framework in the future and avoid a branch.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Mark Rutland]

On Wed, Mar 22, 2017 at 12:16:20PM +0000, Julien Grall wrote:
> (CC Mark for the TLB question)

[Adding Marc since he should understand this better than I do]

I've trimmed a lot of context here, since it wasn't clear if it was
relevant to the question. If there's something I've missed, please point
that out explicitly.

> I am not entirely sure if we can only restore HCR_RW. My concern is
> some of the bits are cached in the TLBs. Looking at the ARM ARM
> (both v7 and v8 see D4.7 in DDI0487A.k_iss10775): "In these
> descriptions, TLB entries for a translation regime for a particular
> Exception level are out of context when executing at a higher
> Exception level.". Which I interpret as TLB result cannot be cached
> when translating a guest VA to guest PA at EL2. So I guess restoring
> only HCR_RW might be fine.

My understanding is that when a translation regime is out-of-context,
the only requirement is that the core does not begin speculative walks
for that translation regime. See ARM DDI 0487A.k_iss10775, page D4-1735,
"Use of out-of-context translation regimes".

I believe that an explicit address translation involving a translation
regime can validly make use of (any part of) that regime's state and/or
allocate new TLB entries for that regime.

It looks like you're asking if you can avoid installing all of the
registers related to the EL1&0 regime when performing an EL1&0
translation at EL2, right?

I do not believe that is valid; my understanding is that all of the
relevant registers need to be installed first.

Thanks,
Mark.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Marc Zyngier]

On 22/03/17 12:45, Mark Rutland wrote:
> On Wed, Mar 22, 2017 at 12:16:20PM +0000, Julien Grall wrote:
>> (CC Mark for the TLB question)
> 
> [Adding Marc since he should understand this better than I do]
> 
> I've trimmed a lot of context here, since it wasn't clear if it was
> relevant to the question. If there's something I've missed, please point
> that out explicitly.
> 
>> I am not entirely sure if we can only restore HCR_RW. My concern is
>> some of the bits are cached in the TLBs. Looking at the ARM ARM
>> (both v7 and v8 see D4.7 in DDI0487A.k_iss10775): "In these
>> descriptions, TLB entries for a translation regime for a particular
>> Exception level are out of context when executing at a higher
>> Exception level.". Which I interpret as TLB result cannot be cached
>> when translating a guest VA to guest PA at EL2. So I guess restoring
>> only HCR_RW might be fine.
> 
> My understanding is that when a translation regime is out-of-context,
> the only requirement is that the core does not begin speculative walks
> for that translation regime. See ARM DDI 0487A.k_iss10775, page D4-1735,
> "Use of out-of-context translation regimes".

+1. An AT instruction is certainly allowed to hit in the TLB, and the
only thing the core is not allowed to do is to speculate (because you
cannot restore a guest context atomically).

> I believe that an explicit address translation involving a translation
> regime can validly make use of (any part of) that regime's state and/or
> allocate new TLB entries for that regime.
> 
> It looks like you're asking if you can avoid installing all of the
> registers related to the EL1&0 regime when performing an EL1&0
> translation at EL2, right?
> 
> I do not believe that is valid; my understanding is that all of the
> relevant registers need to be installed first.

Indeed. I can't see how the you can perform an AT without first
restoring all of the context that define how the page tables are parsed
and including all the element that may be used to hit in the TLBs. Even
worse, you could create TLB entries (as you said, AT doesn't preclude
TLBs from being populated) that would conflict with an existing one on
the next lookup, generating a Conflict Abort.

In short, not restoring the full VM context is doomed.

Now, what is the actual question? ;-)

Cheers,

	M.
-- 
Jazz is not dead. It just smells funny...

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Stefano Stabellini]

On Wed, 22 Mar 2017, Marc Zyngier wrote:
> On 22/03/17 12:45, Mark Rutland wrote:
> > On Wed, Mar 22, 2017 at 12:16:20PM +0000, Julien Grall wrote:
> >> (CC Mark for the TLB question)
> > 
> > [Adding Marc since he should understand this better than I do]
> > 
> > I've trimmed a lot of context here, since it wasn't clear if it was
> > relevant to the question. If there's something I've missed, please point
> > that out explicitly.
> > 
> >> I am not entirely sure if we can only restore HCR_RW. My concern is
> >> some of the bits are cached in the TLBs. Looking at the ARM ARM
> >> (both v7 and v8 see D4.7 in DDI0487A.k_iss10775): "In these
> >> descriptions, TLB entries for a translation regime for a particular
> >> Exception level are out of context when executing at a higher
> >> Exception level.". Which I interpret as TLB result cannot be cached
> >> when translating a guest VA to guest PA at EL2. So I guess restoring
> >> only HCR_RW might be fine.
> > 
> > My understanding is that when a translation regime is out-of-context,
> > the only requirement is that the core does not begin speculative walks
> > for that translation regime. See ARM DDI 0487A.k_iss10775, page D4-1735,
> > "Use of out-of-context translation regimes".
> 
> +1. An AT instruction is certainly allowed to hit in the TLB, and the
> only thing the core is not allowed to do is to speculate (because you
> cannot restore a guest context atomically).
> 
> > I believe that an explicit address translation involving a translation
> > regime can validly make use of (any part of) that regime's state and/or
> > allocate new TLB entries for that regime.
> > 
> > It looks like you're asking if you can avoid installing all of the
> > registers related to the EL1&0 regime when performing an EL1&0
> > translation at EL2, right?
> > 
> > I do not believe that is valid; my understanding is that all of the
> > relevant registers need to be installed first.
> 
> Indeed. I can't see how the you can perform an AT without first
> restoring all of the context that define how the page tables are parsed
> and including all the element that may be used to hit in the TLBs. Even
> worse, you could create TLB entries (as you said, AT doesn't preclude
> TLBs from being populated) that would conflict with an existing one on
> the next lookup, generating a Conflict Abort.
> 
> In short, not restoring the full VM context is doomed.
> 
> Now, what is the actual question? ;-)

Hi Marc, thanks for jumping in.

When we receive an SError in Xen, we determine if it should be injected
into the guest or "handled" in Xen (by "handle" I mean crash the
system). In case it should be injected into the guest, we set the
relevant bit in vcpu->arch.hcr_el2 (the saved version of HCR_EL2). This
patch moves the WRITE(vcpu->arch.hcr_el2, HCR_EL2) from context switch
related functions (p2m_restore_state) to leave_hypervisor_tail, which is
the last point we can move it to, before exiting Xen. That way, we are
sure to inject an abort into the guest, no matter exactly when we
receive the SError. So far so good, right?

Now that leaves the question: what are we going to do at context switch
time? From the SErrors and HCR_VA point of view, there is no need to
restore the vcpu->arch.hcr_el2 of the next vcpu immediately, we can
still do it later in leave_hypervisor_tail. But we do need the right
HCR_RW immediately, because otherwise AT could behave erratically after
the context switch and before leave_hypervisor_tail.

Given that restoring vcpu->arch.hcr_el2 twice (once in
leave_hypervisor_tail and once at context switch) looks weird, I
suggested to only restore the HCR_RW bit at context switch time and do a
full restore of vcpu->arch.hcr_el2 in leave_hypervisor_tail: the benefit
would be a clear separation of responsibility of the two operations.
Hence the question above. But it looks like it's not safe?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Julien Grall]

Hi Stefano,

On 22/03/17 17:54, Stefano Stabellini wrote:
> On Wed, 22 Mar 2017, Marc Zyngier wrote:
>> On 22/03/17 12:45, Mark Rutland wrote:
>>> On Wed, Mar 22, 2017 at 12:16:20PM +0000, Julien Grall wrote:
>>>> (CC Mark for the TLB question)
>>>
>>> [Adding Marc since he should understand this better than I do]
>>>
>>> I've trimmed a lot of context here, since it wasn't clear if it was
>>> relevant to the question. If there's something I've missed, please point
>>> that out explicitly.
>>>
>>>> I am not entirely sure if we can only restore HCR_RW. My concern is
>>>> some of the bits are cached in the TLBs. Looking at the ARM ARM
>>>> (both v7 and v8 see D4.7 in DDI0487A.k_iss10775): "In these
>>>> descriptions, TLB entries for a translation regime for a particular
>>>> Exception level are out of context when executing at a higher
>>>> Exception level.". Which I interpret as TLB result cannot be cached
>>>> when translating a guest VA to guest PA at EL2. So I guess restoring
>>>> only HCR_RW might be fine.
>>>
>>> My understanding is that when a translation regime is out-of-context,
>>> the only requirement is that the core does not begin speculative walks
>>> for that translation regime. See ARM DDI 0487A.k_iss10775, page D4-1735,
>>> "Use of out-of-context translation regimes".
>>
>> +1. An AT instruction is certainly allowed to hit in the TLB, and the
>> only thing the core is not allowed to do is to speculate (because you
>> cannot restore a guest context atomically).
>>
>>> I believe that an explicit address translation involving a translation
>>> regime can validly make use of (any part of) that regime's state and/or
>>> allocate new TLB entries for that regime.
>>>
>>> It looks like you're asking if you can avoid installing all of the
>>> registers related to the EL1&0 regime when performing an EL1&0
>>> translation at EL2, right?
>>>
>>> I do not believe that is valid; my understanding is that all of the
>>> relevant registers need to be installed first.
>>
>> Indeed. I can't see how the you can perform an AT without first
>> restoring all of the context that define how the page tables are parsed
>> and including all the element that may be used to hit in the TLBs. Even
>> worse, you could create TLB entries (as you said, AT doesn't preclude
>> TLBs from being populated) that would conflict with an existing one on
>> the next lookup, generating a Conflict Abort.
>>
>> In short, not restoring the full VM context is doomed.
>>
>> Now, what is the actual question? ;-)
>
> Hi Marc, thanks for jumping in.
>
> When we receive an SError in Xen, we determine if it should be injected
> into the guest or "handled" in Xen (by "handle" I mean crash the
> system). In case it should be injected into the guest, we set the
> relevant bit in vcpu->arch.hcr_el2 (the saved version of HCR_EL2). This
> patch moves the WRITE(vcpu->arch.hcr_el2, HCR_EL2) from context switch
> related functions (p2m_restore_state) to leave_hypervisor_tail, which is
> the last point we can move it to, before exiting Xen. That way, we are
> sure to inject an abort into the guest, no matter exactly when we
> receive the SError. So far so good, right?
>
> Now that leaves the question: what are we going to do at context switch
> time? From the SErrors and HCR_VA point of view, there is no need to
> restore the vcpu->arch.hcr_el2 of the next vcpu immediately, we can
> still do it later in leave_hypervisor_tail. But we do need the right
> HCR_RW immediately, because otherwise AT could behave erratically after
> the context switch and before leave_hypervisor_tail.

Well, this is slightly untrue. The function p2m_restore_state is used 
for 2 purposes:
	- Context switch vCPU
	- Temporarily switching to a P2M to perform AT instruction on behalf of 
a vCPU.

For the former, AT instruction should never happen because as soon as we 
context switch we will return to the guest. So, where it matter is when 
p2m_restore_state is used to perform some AT translation (such as when 
DOM0 is booting or as it should be done in memaccess).

>
> Given that restoring vcpu->arch.hcr_el2 twice (once in
> leave_hypervisor_tail and once at context switch) looks weird, I
> suggested to only restore the HCR_RW bit at context switch time and do a
> full restore of vcpu->arch.hcr_el2 in leave_hypervisor_tail: the benefit
> would be a clear separation of responsibility of the two operations.
> Hence the question above. But it looks like it's not safe?

 From the discussion I had with Mar{c,k}, we should restore all the 
registers that could be cached in TLB when performing an AT instruction. 
Otherwise the TLB may generate conflict abort.

But as I said previously, the function p2m_restore_state is a mess. It 
should be reworked to differentiate the case where we context switch the 
vCPU and the one we perform an AT instruction. This would need some 
rework that I think is too late for Xen 4.9.

So for the time being, I would keep both and add a comment explaining why.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Mark Rutland]

On Wed, Mar 22, 2017 at 10:54:11AM -0700, Stefano Stabellini wrote:
> When we receive an SError in Xen, we determine if it should be injected
> into the guest or "handled" in Xen (by "handle" I mean crash the
> system). In case it should be injected into the guest, we set the
> relevant bit in vcpu->arch.hcr_el2 (the saved version of HCR_EL2). This
> patch moves the WRITE(vcpu->arch.hcr_el2, HCR_EL2) from context switch
> related functions (p2m_restore_state) to leave_hypervisor_tail, which is
> the last point we can move it to, before exiting Xen. That way, we are
> sure to inject an abort into the guest, no matter exactly when we
> receive the SError. So far so good, right?

I assume you're in the context of the guest at this point, so what's the
problem with doing:

	vcpu->arch.hcr_el2 |= HCR_VA;
	WRITE(vcpu->arch.hcr_el2, HCR_EL2);

That way you don't need to move the restoration of HCR_EL2, and it workd
regardless of whether you get preempted, at the cost of a potentially
redundant system register write on what should be an incredibly rare
path...

Within Linux we do similar when we manipulate system registers which are
context switched with the thread, e.g. when setting tpidrro_el0 in
tls_thread_flush() [1].

Surely similar applies for the manipulation of other system registers
while in the guest context?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kernel/process.c?h=v4.11-rc3#n217

Thanks,
Mark.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/18] xen/arm: Restore HCR_EL2 register

[Stefano Stabellini]

On Wed, 22 Mar 2017, Mark Rutland wrote:
> On Wed, Mar 22, 2017 at 10:54:11AM -0700, Stefano Stabellini wrote:
> > When we receive an SError in Xen, we determine if it should be injected
> > into the guest or "handled" in Xen (by "handle" I mean crash the
> > system). In case it should be injected into the guest, we set the
> > relevant bit in vcpu->arch.hcr_el2 (the saved version of HCR_EL2). This
> > patch moves the WRITE(vcpu->arch.hcr_el2, HCR_EL2) from context switch
> > related functions (p2m_restore_state) to leave_hypervisor_tail, which is
> > the last point we can move it to, before exiting Xen. That way, we are
> > sure to inject an abort into the guest, no matter exactly when we
> > receive the SError. So far so good, right?
> 
> I assume you're in the context of the guest at this point, so what's the
> problem with doing:
> 
> 	vcpu->arch.hcr_el2 |= HCR_VA;
> 	WRITE(vcpu->arch.hcr_el2, HCR_EL2);
> 
> That way you don't need to move the restoration of HCR_EL2, and it workd
> regardless of whether you get preempted, at the cost of a potentially
> redundant system register write on what should be an incredibly rare
> path...
> 
> Within Linux we do similar when we manipulate system registers which are
> context switched with the thread, e.g. when setting tpidrro_el0 in
> tls_thread_flush() [1].
> 
> Surely similar applies for the manipulation of other system registers
> while in the guest context?

Yes, this looks like the better option.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Stefano Stabellini]

On Wed, 22 Mar 2017, Julien Grall wrote:
> Hi Wei,
> 
> On 22/03/17 08:49, Wei Chen wrote:
> > Hi Stefano,
> > 
> > On 2017/3/21 5:38, Stefano Stabellini wrote:
> > > On Mon, 13 Mar 2017, Wei Chen wrote:
> > > > Currently, we masked the Abort/SError bit in Xen exception entries.
> > > > So Xen could not capture any Abort/SError while it's running.
> > > > Now, Xen has the ability to handle the Abort/SError, we should unmask
> > > > the Abort/SError bit by default to let Xen capture Abort/SError while
> > > > it's running.
> > > > 
> > > > But in order to avoid receiving nested asynchronous abort, we don't
> > > > unmask Abort/SError bit in hyp_error and trap_data_abort.
> > > > 
> > > > Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> > > > ---
> > > > We haven't done this before, so I don't know how can this change
> > > > will affect the Xen. If the IRQ and Abort take place at the same
> > > > time, how can we handle them?
> > > 
> > > If the abort is for Xen, the hypervisor will crash and that's the end of
> > 
> > Before the system crash, we have enable the IRQ, so that would not be
> > the end if an IRQ happens at the same time.
> > 
> > > it. If the abort is for the guest, Xen will inject it into the VM, then
> > 
> > Before we have inject the abort to VM, we have enable the IRQ.
> > 
> > > it will return from handling the abort, going back to handling the IRQ
> > > as before. Isn't that right?
> > 
> > If the abort has higher priority then IRQ, it's right.
> > 
> > > 
> > > 
> > > > If an abort is taking place while we're handling the IRQ, the program
> > > > jump to abort exception, and then enable the IRQ. In this case, what
> > > > will happen? So I think I need more discussions from community.
> > > 
> > > Do you know of an example scenario where Xen could have a problem with
> > > this?
> > > 
> > 
> > For example,
> > 1. Trigger a SError in hypervisor.
> > 2. Jump to hyp_error to handle SError.
> > 3. Enable IRQ in hyp_error before PANIC
> > 4. A timer IRQ happens.
> > 5. Jump to hyp_irq and unmask abort again.
> > 6. Jump hyp_error again?
> 
> Technically you could end up in an infinite loop if hyp_error code generates a
> SError. It will stay pending until the end and then trigger again when SError
> is unmasked...
> 
> That's unfortunate but I don't think this is a big issue as if this is
> happening your platform is already doomed.

I agree, but the scenario suggested by Wei is not like that: hyp_error
does not generate an serror, it only unmask irqs.

I think that it would be safer not to unmask IRQs in hyp_error (remove
msr daifclr, #2 at the beginning of hyp_error). IRQs can be enabled at
the end of do_trap_hyp_serror (if the hypervisor does not panic).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Wei Chen]

Hi Stefano,

On 2017/3/23 6:22, Stefano Stabellini wrote:
> On Wed, 22 Mar 2017, Julien Grall wrote:
>> Hi Wei,
>>
>> On 22/03/17 08:49, Wei Chen wrote:
>>> Hi Stefano,
>>>
>>> On 2017/3/21 5:38, Stefano Stabellini wrote:
>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>>> Currently, we masked the Abort/SError bit in Xen exception entries.
>>>>> So Xen could not capture any Abort/SError while it's running.
>>>>> Now, Xen has the ability to handle the Abort/SError, we should unmask
>>>>> the Abort/SError bit by default to let Xen capture Abort/SError while
>>>>> it's running.
>>>>>
>>>>> But in order to avoid receiving nested asynchronous abort, we don't
>>>>> unmask Abort/SError bit in hyp_error and trap_data_abort.
>>>>>
>>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>>> ---
>>>>> We haven't done this before, so I don't know how can this change
>>>>> will affect the Xen. If the IRQ and Abort take place at the same
>>>>> time, how can we handle them?
>>>>
>>>> If the abort is for Xen, the hypervisor will crash and that's the end of
>>>
>>> Before the system crash, we have enable the IRQ, so that would not be
>>> the end if an IRQ happens at the same time.
>>>
>>>> it. If the abort is for the guest, Xen will inject it into the VM, then
>>>
>>> Before we have inject the abort to VM, we have enable the IRQ.
>>>
>>>> it will return from handling the abort, going back to handling the IRQ
>>>> as before. Isn't that right?
>>>
>>> If the abort has higher priority then IRQ, it's right.
>>>
>>>>
>>>>
>>>>> If an abort is taking place while we're handling the IRQ, the program
>>>>> jump to abort exception, and then enable the IRQ. In this case, what
>>>>> will happen? So I think I need more discussions from community.
>>>>
>>>> Do you know of an example scenario where Xen could have a problem with
>>>> this?
>>>>
>>>
>>> For example,
>>> 1. Trigger a SError in hypervisor.
>>> 2. Jump to hyp_error to handle SError.
>>> 3. Enable IRQ in hyp_error before PANIC
>>> 4. A timer IRQ happens.
>>> 5. Jump to hyp_irq and unmask abort again.
>>> 6. Jump hyp_error again?
>>
>> Technically you could end up in an infinite loop if hyp_error code generates a
>> SError. It will stay pending until the end and then trigger again when SError
>> is unmasked...
>>
>> That's unfortunate but I don't think this is a big issue as if this is
>> happening your platform is already doomed.
>
> I agree, but the scenario suggested by Wei is not like that: hyp_error
> does not generate an serror, it only unmask irqs.
>
> I think that it would be safer not to unmask IRQs in hyp_error (remove
> msr daifclr, #2 at the beginning of hyp_error). IRQs can be enabled at
> the end of do_trap_hyp_serror (if the hypervisor does not panic).
>

Yes, it would be safer if we defined an end for exception loop. And
hyp_error is a good place to end up the exception loop.
1. If hyp_error will PANIC the system, enable IRQ in hyp_error to handle
    interrupts is non-significant.
2. If hyp_error will forward the SErrors, enable IRQ before forwarding
    SError to guest will make Xen enter exception loop. The SError would
    not have chance to be forwarded to guest.

So, I think enable IRQ at the end of do_trap_hyp_serror would be better.


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 16/18] xen/arm: Isolate the SError between the context switch of 2 vCPUs

[Wei Chen]

Hi Julien,

On 2017/3/22 20:29, Julien Grall wrote:
> Hi Wei,
>
> On 22/03/17 08:53, Wei Chen wrote:
>> Hi Stefano,
>>
>> On 2017/3/21 5:46, Stefano Stabellini wrote:
>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>> If there is a pending SError while we are doing context switch, if the
>>>> SError handle option is "FORWARD", We have to guranatee this serror to
>>>> be caught by current vCPU, otherwise it will be caught by next vCPU and
>>>> be forwarded to this wrong vCPU.
>>>>
>>>> We don't want to export serror_op accessing to other source files and
>>>> use serror_op every place, so we add a helper to synchronize SError for
>>>> context switching. The synchronize_serror has been used in this helper,
>>>> so the "#if 0" can be removed.
>>>>
>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>> ---
>>>>  xen/arch/arm/domain.c           |  2 ++
>>>>  xen/arch/arm/traps.c            | 14 ++++++++++++--
>>>>  xen/include/asm-arm/processor.h |  2 ++
>>>>  3 files changed, 16 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>> index 69c2854..a547fcd 100644
>>>> --- a/xen/arch/arm/domain.c
>>>> +++ b/xen/arch/arm/domain.c
>>>> @@ -312,6 +312,8 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
>>>>
>>>>      local_irq_disable();
>>>>
>>>> +    prevent_forward_serror_to_next_vcpu();
>>>> +
>>>>      set_current(next);
>>>>
>>>>      prev = __context_switch(prev, next);
>>>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>>>> index ee7865b..b8c8389 100644
>>>> --- a/xen/arch/arm/traps.c
>>>> +++ b/xen/arch/arm/traps.c
>>>> @@ -2899,7 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
>>>>      }
>>>>  }
>>>>
>>>> -#if 0
>>>>  static void synchronize_serror(void)
>>>>  {
>>>>      /* Synchronize against in-flight ld/st. */
>>>> @@ -2908,7 +2907,18 @@ static void synchronize_serror(void)
>>>>      /* A single instruction exception window */
>>>>      isb();
>>>>  }
>>>> -#endif
>>>> +
>>>> +/*
>>>> + * If the SErrors option is "FORWARD", we have to prevent forwarding
>>>> + * serror to wrong vCPU. So before context switch, we have to use the
>>>> + * synchronize_serror to guarantee that the pending serror would be
>>>> + * caught by current vCPU.
>>>> + */
>>>> +void prevent_forward_serror_to_next_vcpu(void)
>>>> +{
>>>> +    if ( serrors_op == SERRORS_FORWARD )
>>>> +        synchronize_serror();
>>>> +}
>>>
>>> I dislike introducing so many 2 lines functions. I would get rid of
>>> prevent_forward_serror_to_next_vcpu and just add
>>>
>>>    if ( serrors_op == SERRORS_FORWARD )
>>>        synchronize_serror();
>>>
>>> to context_switch, and I would make synchronize_serror a static inline.
>>>
>>
>> I had done it before, but that made the serrors_op appear everywhere.
>> If export serrors_op to other files is acceptable, I would re-do it.
>
> Why don't you introduce a new flag in cpu_hwcaps? This would allow us to
> take advantage of alternative framework in the future and avoid a branch.
>

Ah, that would be a good option, but should we take too much resources
of cpu_hwcaps for SErrors?

> Cheers,
>


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 16/18] xen/arm: Isolate the SError between the context switch of 2 vCPUs

[Stefano Stabellini]

On Thu, 23 Mar 2017, Wei Chen wrote:
> Hi Julien,
> 
> On 2017/3/22 20:29, Julien Grall wrote:
> > Hi Wei,
> >
> > On 22/03/17 08:53, Wei Chen wrote:
> >> Hi Stefano,
> >>
> >> On 2017/3/21 5:46, Stefano Stabellini wrote:
> >>> On Mon, 13 Mar 2017, Wei Chen wrote:
> >>>> If there is a pending SError while we are doing context switch, if the
> >>>> SError handle option is "FORWARD", We have to guranatee this serror to
> >>>> be caught by current vCPU, otherwise it will be caught by next vCPU and
> >>>> be forwarded to this wrong vCPU.
> >>>>
> >>>> We don't want to export serror_op accessing to other source files and
> >>>> use serror_op every place, so we add a helper to synchronize SError for
> >>>> context switching. The synchronize_serror has been used in this helper,
> >>>> so the "#if 0" can be removed.
> >>>>
> >>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> >>>> ---
> >>>>  xen/arch/arm/domain.c           |  2 ++
> >>>>  xen/arch/arm/traps.c            | 14 ++++++++++++--
> >>>>  xen/include/asm-arm/processor.h |  2 ++
> >>>>  3 files changed, 16 insertions(+), 2 deletions(-)
> >>>>
> >>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>> index 69c2854..a547fcd 100644
> >>>> --- a/xen/arch/arm/domain.c
> >>>> +++ b/xen/arch/arm/domain.c
> >>>> @@ -312,6 +312,8 @@ void context_switch(struct vcpu *prev, struct vcpu *next)
> >>>>
> >>>>      local_irq_disable();
> >>>>
> >>>> +    prevent_forward_serror_to_next_vcpu();
> >>>> +
> >>>>      set_current(next);
> >>>>
> >>>>      prev = __context_switch(prev, next);
> >>>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> >>>> index ee7865b..b8c8389 100644
> >>>> --- a/xen/arch/arm/traps.c
> >>>> +++ b/xen/arch/arm/traps.c
> >>>> @@ -2899,7 +2899,6 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
> >>>>      }
> >>>>  }
> >>>>
> >>>> -#if 0
> >>>>  static void synchronize_serror(void)
> >>>>  {
> >>>>      /* Synchronize against in-flight ld/st. */
> >>>> @@ -2908,7 +2907,18 @@ static void synchronize_serror(void)
> >>>>      /* A single instruction exception window */
> >>>>      isb();
> >>>>  }
> >>>> -#endif
> >>>> +
> >>>> +/*
> >>>> + * If the SErrors option is "FORWARD", we have to prevent forwarding
> >>>> + * serror to wrong vCPU. So before context switch, we have to use the
> >>>> + * synchronize_serror to guarantee that the pending serror would be
> >>>> + * caught by current vCPU.
> >>>> + */
> >>>> +void prevent_forward_serror_to_next_vcpu(void)
> >>>> +{
> >>>> +    if ( serrors_op == SERRORS_FORWARD )
> >>>> +        synchronize_serror();
> >>>> +}
> >>>
> >>> I dislike introducing so many 2 lines functions. I would get rid of
> >>> prevent_forward_serror_to_next_vcpu and just add
> >>>
> >>>    if ( serrors_op == SERRORS_FORWARD )
> >>>        synchronize_serror();
> >>>
> >>> to context_switch, and I would make synchronize_serror a static inline.
> >>>
> >>
> >> I had done it before, but that made the serrors_op appear everywhere.
> >> If export serrors_op to other files is acceptable, I would re-do it.
> >
> > Why don't you introduce a new flag in cpu_hwcaps? This would allow us to
> > take advantage of alternative framework in the future and avoid a branch.
> >
> 
> Ah, that would be a good option, but should we take too much resources
> of cpu_hwcaps for SErrors?

No worries, we have plenty.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Julien Grall]

Hi Wei,

On 23/03/17 03:13, Wei Chen wrote:
> On 2017/3/23 6:22, Stefano Stabellini wrote:
>> On Wed, 22 Mar 2017, Julien Grall wrote:
>>> Hi Wei,
>>>
>>> On 22/03/17 08:49, Wei Chen wrote:
>>>> Hi Stefano,
>>>>
>>>> On 2017/3/21 5:38, Stefano Stabellini wrote:
>>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>>>> Currently, we masked the Abort/SError bit in Xen exception entries.
>>>>>> So Xen could not capture any Abort/SError while it's running.
>>>>>> Now, Xen has the ability to handle the Abort/SError, we should unmask
>>>>>> the Abort/SError bit by default to let Xen capture Abort/SError while
>>>>>> it's running.
>>>>>>
>>>>>> But in order to avoid receiving nested asynchronous abort, we don't
>>>>>> unmask Abort/SError bit in hyp_error and trap_data_abort.
>>>>>>
>>>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>>>> ---
>>>>>> We haven't done this before, so I don't know how can this change
>>>>>> will affect the Xen. If the IRQ and Abort take place at the same
>>>>>> time, how can we handle them?
>>>>>
>>>>> If the abort is for Xen, the hypervisor will crash and that's the end of
>>>>
>>>> Before the system crash, we have enable the IRQ, so that would not be
>>>> the end if an IRQ happens at the same time.
>>>>
>>>>> it. If the abort is for the guest, Xen will inject it into the VM, then
>>>>
>>>> Before we have inject the abort to VM, we have enable the IRQ.
>>>>
>>>>> it will return from handling the abort, going back to handling the IRQ
>>>>> as before. Isn't that right?
>>>>
>>>> If the abort has higher priority then IRQ, it's right.
>>>>
>>>>>
>>>>>
>>>>>> If an abort is taking place while we're handling the IRQ, the program
>>>>>> jump to abort exception, and then enable the IRQ. In this case, what
>>>>>> will happen? So I think I need more discussions from community.
>>>>>
>>>>> Do you know of an example scenario where Xen could have a problem with
>>>>> this?
>>>>>
>>>>
>>>> For example,
>>>> 1. Trigger a SError in hypervisor.
>>>> 2. Jump to hyp_error to handle SError.
>>>> 3. Enable IRQ in hyp_error before PANIC
>>>> 4. A timer IRQ happens.
>>>> 5. Jump to hyp_irq and unmask abort again.
>>>> 6. Jump hyp_error again?
>>>
>>> Technically you could end up in an infinite loop if hyp_error code generates a
>>> SError. It will stay pending until the end and then trigger again when SError
>>> is unmasked...
>>>
>>> That's unfortunate but I don't think this is a big issue as if this is
>>> happening your platform is already doomed.
>>
>> I agree, but the scenario suggested by Wei is not like that: hyp_error
>> does not generate an serror, it only unmask irqs.
>>
>> I think that it would be safer not to unmask IRQs in hyp_error (remove
>> msr daifclr, #2 at the beginning of hyp_error). IRQs can be enabled at
>> the end of do_trap_hyp_serror (if the hypervisor does not panic).
>>
>
> Yes, it would be safer if we defined an end for exception loop. And
> hyp_error is a good place to end up the exception loop.
> 1. If hyp_error will PANIC the system, enable IRQ in hyp_error to handle
>     interrupts is non-significant.
> 2. If hyp_error will forward the SErrors, enable IRQ before forwarding
>     SError to guest will make Xen enter exception loop. The SError would
>     not have chance to be forwarded to guest.
>
> So, I think enable IRQ at the end of do_trap_hyp_serror would be better.

That's not going to help. If the IRQ path is triggering an SError again 
and again, there is no way to go out even if you re-enable IRQ later.

The HCR_VA will be set for the guest, but you will never go return to 
the guest. So you will still ending up in an infinite loop.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Stefano Stabellini]

On Thu, 23 Mar 2017, Julien Grall wrote:
> Hi Wei,
> 
> On 23/03/17 03:13, Wei Chen wrote:
> > On 2017/3/23 6:22, Stefano Stabellini wrote:
> > > On Wed, 22 Mar 2017, Julien Grall wrote:
> > > > Hi Wei,
> > > > 
> > > > On 22/03/17 08:49, Wei Chen wrote:
> > > > > Hi Stefano,
> > > > > 
> > > > > On 2017/3/21 5:38, Stefano Stabellini wrote:
> > > > > > On Mon, 13 Mar 2017, Wei Chen wrote:
> > > > > > > Currently, we masked the Abort/SError bit in Xen exception
> > > > > > > entries.
> > > > > > > So Xen could not capture any Abort/SError while it's running.
> > > > > > > Now, Xen has the ability to handle the Abort/SError, we should
> > > > > > > unmask
> > > > > > > the Abort/SError bit by default to let Xen capture Abort/SError
> > > > > > > while
> > > > > > > it's running.
> > > > > > > 
> > > > > > > But in order to avoid receiving nested asynchronous abort, we
> > > > > > > don't
> > > > > > > unmask Abort/SError bit in hyp_error and trap_data_abort.
> > > > > > > 
> > > > > > > Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> > > > > > > ---
> > > > > > > We haven't done this before, so I don't know how can this change
> > > > > > > will affect the Xen. If the IRQ and Abort take place at the same
> > > > > > > time, how can we handle them?
> > > > > > 
> > > > > > If the abort is for Xen, the hypervisor will crash and that's the
> > > > > > end of
> > > > > 
> > > > > Before the system crash, we have enable the IRQ, so that would not be
> > > > > the end if an IRQ happens at the same time.
> > > > > 
> > > > > > it. If the abort is for the guest, Xen will inject it into the VM,
> > > > > > then
> > > > > 
> > > > > Before we have inject the abort to VM, we have enable the IRQ.
> > > > > 
> > > > > > it will return from handling the abort, going back to handling the
> > > > > > IRQ
> > > > > > as before. Isn't that right?
> > > > > 
> > > > > If the abort has higher priority then IRQ, it's right.
> > > > > 
> > > > > > 
> > > > > > 
> > > > > > > If an abort is taking place while we're handling the IRQ, the
> > > > > > > program
> > > > > > > jump to abort exception, and then enable the IRQ. In this case,
> > > > > > > what
> > > > > > > will happen? So I think I need more discussions from community.
> > > > > > 
> > > > > > Do you know of an example scenario where Xen could have a problem
> > > > > > with
> > > > > > this?
> > > > > > 
> > > > > 
> > > > > For example,
> > > > > 1. Trigger a SError in hypervisor.
> > > > > 2. Jump to hyp_error to handle SError.
> > > > > 3. Enable IRQ in hyp_error before PANIC
> > > > > 4. A timer IRQ happens.
> > > > > 5. Jump to hyp_irq and unmask abort again.
> > > > > 6. Jump hyp_error again?
> > > > 
> > > > Technically you could end up in an infinite loop if hyp_error code
> > > > generates a
> > > > SError. It will stay pending until the end and then trigger again when
> > > > SError
> > > > is unmasked...
> > > > 
> > > > That's unfortunate but I don't think this is a big issue as if this is
> > > > happening your platform is already doomed.
> > > 
> > > I agree, but the scenario suggested by Wei is not like that: hyp_error
> > > does not generate an serror, it only unmask irqs.
> > > 
> > > I think that it would be safer not to unmask IRQs in hyp_error (remove
> > > msr daifclr, #2 at the beginning of hyp_error). IRQs can be enabled at
> > > the end of do_trap_hyp_serror (if the hypervisor does not panic).
> > > 
> > 
> > Yes, it would be safer if we defined an end for exception loop. And
> > hyp_error is a good place to end up the exception loop.
> > 1. If hyp_error will PANIC the system, enable IRQ in hyp_error to handle
> >     interrupts is non-significant.
> > 2. If hyp_error will forward the SErrors, enable IRQ before forwarding
> >     SError to guest will make Xen enter exception loop. The SError would
> >     not have chance to be forwarded to guest.
> > 
> > So, I think enable IRQ at the end of do_trap_hyp_serror would be better.
> 
> That's not going to help. If the IRQ path is triggering an SError again and
> again, there is no way to go out even if you re-enable IRQ later.

I understand where the misunderstanding comes from, I think it is due to
a different interpretation of how the hardware behaves. Only one is
correct, and I think it's yours.

In the following case, assuming only one SError is ever generated:

1. A guest causes an SError
2. Xen jumps to hyp_error to handle it
3. Xen enables IRQ in hyp_error
4. Xen receives a timer event
5. Xen jumps to hyp_irq and unmask abort
6. ???

In 6., Xen doesn't jump to hyp_error again because of the first SError,
right? The SError has already been received, Xen wouldn't trap again on
the same abort because something hasn't been cleared, right? In that
case, there is no need to delay unmasking IRQs, as you have been saying:

6. Xen handles the IRQ, injects a virtual irq to guest
7. Xen returns from IRQ, goes back to abort handler
8. Xen injects abort to guest

in that case this patch is good as is.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Wei Chen]

Hi Stefano,

On 2017/3/24 8:10, Stefano Stabellini wrote:
> On Thu, 23 Mar 2017, Julien Grall wrote:
>> Hi Wei,
>>
>> On 23/03/17 03:13, Wei Chen wrote:
>>> On 2017/3/23 6:22, Stefano Stabellini wrote:
>>>> On Wed, 22 Mar 2017, Julien Grall wrote:
>>>>> Hi Wei,
>>>>>
>>>>> On 22/03/17 08:49, Wei Chen wrote:
>>>>>> Hi Stefano,
>>>>>>
>>>>>> On 2017/3/21 5:38, Stefano Stabellini wrote:
>>>>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
>>>>>>>> Currently, we masked the Abort/SError bit in Xen exception
>>>>>>>> entries.
>>>>>>>> So Xen could not capture any Abort/SError while it's running.
>>>>>>>> Now, Xen has the ability to handle the Abort/SError, we should
>>>>>>>> unmask
>>>>>>>> the Abort/SError bit by default to let Xen capture Abort/SError
>>>>>>>> while
>>>>>>>> it's running.
>>>>>>>>
>>>>>>>> But in order to avoid receiving nested asynchronous abort, we
>>>>>>>> don't
>>>>>>>> unmask Abort/SError bit in hyp_error and trap_data_abort.
>>>>>>>>
>>>>>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
>>>>>>>> ---
>>>>>>>> We haven't done this before, so I don't know how can this change
>>>>>>>> will affect the Xen. If the IRQ and Abort take place at the same
>>>>>>>> time, how can we handle them?
>>>>>>>
>>>>>>> If the abort is for Xen, the hypervisor will crash and that's the
>>>>>>> end of
>>>>>>
>>>>>> Before the system crash, we have enable the IRQ, so that would not be
>>>>>> the end if an IRQ happens at the same time.
>>>>>>
>>>>>>> it. If the abort is for the guest, Xen will inject it into the VM,
>>>>>>> then
>>>>>>
>>>>>> Before we have inject the abort to VM, we have enable the IRQ.
>>>>>>
>>>>>>> it will return from handling the abort, going back to handling the
>>>>>>> IRQ
>>>>>>> as before. Isn't that right?
>>>>>>
>>>>>> If the abort has higher priority then IRQ, it's right.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> If an abort is taking place while we're handling the IRQ, the
>>>>>>>> program
>>>>>>>> jump to abort exception, and then enable the IRQ. In this case,
>>>>>>>> what
>>>>>>>> will happen? So I think I need more discussions from community.
>>>>>>>
>>>>>>> Do you know of an example scenario where Xen could have a problem
>>>>>>> with
>>>>>>> this?
>>>>>>>
>>>>>>
>>>>>> For example,
>>>>>> 1. Trigger a SError in hypervisor.
>>>>>> 2. Jump to hyp_error to handle SError.
>>>>>> 3. Enable IRQ in hyp_error before PANIC
>>>>>> 4. A timer IRQ happens.
>>>>>> 5. Jump to hyp_irq and unmask abort again.
>>>>>> 6. Jump hyp_error again?
>>>>>
>>>>> Technically you could end up in an infinite loop if hyp_error code
>>>>> generates a
>>>>> SError. It will stay pending until the end and then trigger again when
>>>>> SError
>>>>> is unmasked...
>>>>>
>>>>> That's unfortunate but I don't think this is a big issue as if this is
>>>>> happening your platform is already doomed.
>>>>
>>>> I agree, but the scenario suggested by Wei is not like that: hyp_error
>>>> does not generate an serror, it only unmask irqs.
>>>>
>>>> I think that it would be safer not to unmask IRQs in hyp_error (remove
>>>> msr daifclr, #2 at the beginning of hyp_error). IRQs can be enabled at
>>>> the end of do_trap_hyp_serror (if the hypervisor does not panic).
>>>>
>>>
>>> Yes, it would be safer if we defined an end for exception loop. And
>>> hyp_error is a good place to end up the exception loop.
>>> 1. If hyp_error will PANIC the system, enable IRQ in hyp_error to handle
>>>     interrupts is non-significant.
>>> 2. If hyp_error will forward the SErrors, enable IRQ before forwarding
>>>     SError to guest will make Xen enter exception loop. The SError would
>>>     not have chance to be forwarded to guest.
>>>
>>> So, I think enable IRQ at the end of do_trap_hyp_serror would be better.
>>
>> That's not going to help. If the IRQ path is triggering an SError again and
>> again, there is no way to go out even if you re-enable IRQ later.
>
> I understand where the misunderstanding comes from, I think it is due to
> a different interpretation of how the hardware behaves. Only one is
> correct, and I think it's yours.
>
> In the following case, assuming only one SError is ever generated:
>
> 1. A guest causes an SError
> 2. Xen jumps to hyp_error to handle it
> 3. Xen enables IRQ in hyp_error
> 4. Xen receives a timer event
> 5. Xen jumps to hyp_irq and unmask abort
> 6. ???
>
> In 6., Xen doesn't jump to hyp_error again because of the first SError,
> right? The SError has already been received, Xen wouldn't trap again on
> the same abort because something hasn't been cleared, right? In that

Yes, I think you are right. I have done a test, the hardware works as
you said. Xen would not jump to hyp_error again by the first SError.

> case, there is no need to delay unmasking IRQs, as you have been saying:
>
> 6. Xen handles the IRQ, injects a virtual irq to guest
> 7. Xen returns from IRQ, goes back to abort handler
> 8. Xen injects abort to guest
>
> in that case this patch is good as is.
>

Thanks, that reassures me :)


-- 
Regards,
Wei Chen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 14/18] xen/arm: Unmask the Abort/SError bit in the exception entries

[Stefano Stabellini]

On Fri, 24 Mar 2017, Wei Chen wrote:
> Hi Stefano,
> 
> On 2017/3/24 8:10, Stefano Stabellini wrote:
> > On Thu, 23 Mar 2017, Julien Grall wrote:
> >> Hi Wei,
> >>
> >> On 23/03/17 03:13, Wei Chen wrote:
> >>> On 2017/3/23 6:22, Stefano Stabellini wrote:
> >>>> On Wed, 22 Mar 2017, Julien Grall wrote:
> >>>>> Hi Wei,
> >>>>>
> >>>>> On 22/03/17 08:49, Wei Chen wrote:
> >>>>>> Hi Stefano,
> >>>>>>
> >>>>>> On 2017/3/21 5:38, Stefano Stabellini wrote:
> >>>>>>> On Mon, 13 Mar 2017, Wei Chen wrote:
> >>>>>>>> Currently, we masked the Abort/SError bit in Xen exception
> >>>>>>>> entries.
> >>>>>>>> So Xen could not capture any Abort/SError while it's running.
> >>>>>>>> Now, Xen has the ability to handle the Abort/SError, we should
> >>>>>>>> unmask
> >>>>>>>> the Abort/SError bit by default to let Xen capture Abort/SError
> >>>>>>>> while
> >>>>>>>> it's running.
> >>>>>>>>
> >>>>>>>> But in order to avoid receiving nested asynchronous abort, we
> >>>>>>>> don't
> >>>>>>>> unmask Abort/SError bit in hyp_error and trap_data_abort.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Wei Chen <Wei.Chen@arm.com>
> >>>>>>>> ---
> >>>>>>>> We haven't done this before, so I don't know how can this change
> >>>>>>>> will affect the Xen. If the IRQ and Abort take place at the same
> >>>>>>>> time, how can we handle them?
> >>>>>>>
> >>>>>>> If the abort is for Xen, the hypervisor will crash and that's the
> >>>>>>> end of
> >>>>>>
> >>>>>> Before the system crash, we have enable the IRQ, so that would not be
> >>>>>> the end if an IRQ happens at the same time.
> >>>>>>
> >>>>>>> it. If the abort is for the guest, Xen will inject it into the VM,
> >>>>>>> then
> >>>>>>
> >>>>>> Before we have inject the abort to VM, we have enable the IRQ.
> >>>>>>
> >>>>>>> it will return from handling the abort, going back to handling the
> >>>>>>> IRQ
> >>>>>>> as before. Isn't that right?
> >>>>>>
> >>>>>> If the abort has higher priority then IRQ, it's right.
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> If an abort is taking place while we're handling the IRQ, the
> >>>>>>>> program
> >>>>>>>> jump to abort exception, and then enable the IRQ. In this case,
> >>>>>>>> what
> >>>>>>>> will happen? So I think I need more discussions from community.
> >>>>>>>
> >>>>>>> Do you know of an example scenario where Xen could have a problem
> >>>>>>> with
> >>>>>>> this?
> >>>>>>>
> >>>>>>
> >>>>>> For example,
> >>>>>> 1. Trigger a SError in hypervisor.
> >>>>>> 2. Jump to hyp_error to handle SError.
> >>>>>> 3. Enable IRQ in hyp_error before PANIC
> >>>>>> 4. A timer IRQ happens.
> >>>>>> 5. Jump to hyp_irq and unmask abort again.
> >>>>>> 6. Jump hyp_error again?
> >>>>>
> >>>>> Technically you could end up in an infinite loop if hyp_error code
> >>>>> generates a
> >>>>> SError. It will stay pending until the end and then trigger again when
> >>>>> SError
> >>>>> is unmasked...
> >>>>>
> >>>>> That's unfortunate but I don't think this is a big issue as if this is
> >>>>> happening your platform is already doomed.
> >>>>
> >>>> I agree, but the scenario suggested by Wei is not like that: hyp_error
> >>>> does not generate an serror, it only unmask irqs.
> >>>>
> >>>> I think that it would be safer not to unmask IRQs in hyp_error (remove
> >>>> msr daifclr, #2 at the beginning of hyp_error). IRQs can be enabled at
> >>>> the end of do_trap_hyp_serror (if the hypervisor does not panic).
> >>>>
> >>>
> >>> Yes, it would be safer if we defined an end for exception loop. And
> >>> hyp_error is a good place to end up the exception loop.
> >>> 1. If hyp_error will PANIC the system, enable IRQ in hyp_error to handle
> >>>     interrupts is non-significant.
> >>> 2. If hyp_error will forward the SErrors, enable IRQ before forwarding
> >>>     SError to guest will make Xen enter exception loop. The SError would
> >>>     not have chance to be forwarded to guest.
> >>>
> >>> So, I think enable IRQ at the end of do_trap_hyp_serror would be better.
> >>
> >> That's not going to help. If the IRQ path is triggering an SError again and
> >> again, there is no way to go out even if you re-enable IRQ later.
> >
> > I understand where the misunderstanding comes from, I think it is due to
> > a different interpretation of how the hardware behaves. Only one is
> > correct, and I think it's yours.
> >
> > In the following case, assuming only one SError is ever generated:
> >
> > 1. A guest causes an SError
> > 2. Xen jumps to hyp_error to handle it
> > 3. Xen enables IRQ in hyp_error
> > 4. Xen receives a timer event
> > 5. Xen jumps to hyp_irq and unmask abort
> > 6. ???
> >
> > In 6., Xen doesn't jump to hyp_error again because of the first SError,
> > right? The SError has already been received, Xen wouldn't trap again on
> > the same abort because something hasn't been cleared, right? In that
> 
> Yes, I think you are right. I have done a test, the hardware works as
> you said. Xen would not jump to hyp_error again by the first SError.
> 
> > case, there is no need to delay unmasking IRQs, as you have been saying:
> >
> > 6. Xen handles the IRQ, injects a virtual irq to guest
> > 7. Xen returns from IRQ, goes back to abort handler
> > 8. Xen injects abort to guest
> >
> > in that case this patch is good as is.
> >
> 
> Thanks, that reassures me :)

Great, you can add my

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel