[PATCH for-4.5 v2 0/8] xen: arm: map normal memory as inner shareable, reduce scope of various barriers

[PATCH for-4.5 v2 0/8] xen: arm: map normal memory as inner shareable, reduce scope of various barriers

[Ian Campbell]

Currently Xen maps all RAM as Outer-Shareable, since that seemed like
the most conservative option early on when we didn't really know what
Inner- vs. Outer-Shareable meant. However we have long suspected that
actually Inner-Shareable would be the correct type to use.

After reading the docs many times, getting more confused each time, I
finally got a reasonable explanation from a man (and a dog) down the
pub: Inner-Shareable == the processors in an SMP system, while
Outer-Shareable == devices. (NB: Not a random man, he knew what he was
talking about...). With that in mind switch all of Xen's memory mapping,
page table walks and an appropriate subset of the barriers to be inner
shareable.

In addition I have switched barriers to use the correct read/write/any
variants for their types. Note that I have only tackled the generic
mb/rmb/wmb and smp_* barriers (mainly used by common code) here. There
are also quite a few open-coded full-system dsb's in the arch code which
I will look at another time.

v1 of this was back in June[0], and I deferred it due to the 4.3 freeze,
so given that we are now frozen for 4.4 this is clearly 4.5 material.

I've slightly forgotten what was changed since then, but apart from
rebasing the highlights are:

      * dropped all the bogus tlb flush stuff, which was wrong, or at
        best confused.
      * mfn_to_p2m_entry sets shareability based on mattr, dev mappings
        remain outer
      * some clarifications to the comments.

[0] http://lists.xen.org/archives/html/xen-devel/2013-06/msg02969.html

[PATCH for-4.5 v2 1/8] xen: arm: map memory as inner shareable.

[Ian Campbell]

The inner shareable domain contains all SMP processors, including different
clusters (e.g. big.LITTLE). Therefore this is the correct thing to use for Xen
memory mappings. The outer shareable domain is for devices on busses which are
coherent and barrier-aware (e.g. AMBA4 AXI with ACE). While the system domain
is for things behind bridges which are not.

One wrinkle is that Normal memory with attributes Inner Non-cacheable, Outer
Non-cacheable (which we call BUFFERABLE) must be mapped Outer Shareable on ARM
v7. Therefore change the prototype of mfn_to_xen_entry to take the attribute
index so we can DTRT. On ARMv8 the sharability is ignored and considered to
always be Outer Shareable.

Don't adjust the barriers, flushes etc, those remain as they were (which is
more than is now required).  I'll change those in a later patch.

Many thanks to Leif for explaining the difference between Inner- and
Outer-Shareable in words of two or less syllables, I hope I've replicated that
explanation properly above!

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2:
     split dsb sy changes into a separate patch

     comment clarifications from Leif.

     mfn_to_p2m_entry sets shareability based on mattr, dev mappings remain
     outer.
---
 xen/arch/arm/arm32/head.S  |    8 ++++----
 xen/arch/arm/arm64/head.S  |    8 ++++----
 xen/arch/arm/mm.c          |   34 +++++++++++++++++++---------------
 xen/arch/arm/p2m.c         |   18 ++++++++++++++++--
 xen/include/asm-arm/page.h |   37 ++++++++++++++++++++++++++++++++++---
 5 files changed, 77 insertions(+), 28 deletions(-)

diff --git a/xen/arch/arm/arm32/head.S b/xen/arch/arm/arm32/head.S
index 96230ac..60d5cd6 100644
--- a/xen/arch/arm/arm32/head.S
+++ b/xen/arch/arm/arm32/head.S
@@ -26,8 +26,8 @@
 
 #define ZIMAGE_MAGIC_NUMBER 0x016f2818
 
-#define PT_PT     0xe7f /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=111 T=1 P=1 */
-#define PT_MEM    0xe7d /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=111 T=0 P=1 */
+#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
+#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1 */
 #define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1 */
 #define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1 */
 
@@ -236,10 +236,10 @@ cpu_init_done:
         mcr   CP32(r1, HMAIR1)
 
         /* Set up the HTCR:
-         * PT walks use Outer-Shareable accesses,
+         * PT walks use Inner-Shareable accesses,
          * PT walks are write-back, write-allocate in both cache levels,
          * Full 32-bit address space goes through this table. */
-        ldr   r0, =0x80002500
+        ldr   r0, =0x80003500
         mcr   CP32(r0, HTCR)
 
         /* Set up the HSCTLR:
diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 31afdd0..bf9bb58 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -25,8 +25,8 @@
 #include <asm/asm_defns.h>
 #include <asm/early_printk.h>
 
-#define PT_PT     0xe7f /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=111 T=1 P=1 */
-#define PT_MEM    0xe7d /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=111 T=0 P=1 */
+#define PT_PT     0xf7f /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=1 P=1 */
+#define PT_MEM    0xf7d /* nG=1 AF=1 SH=11 AP=01 NS=1 ATTR=111 T=0 P=1 */
 #define PT_DEV    0xe71 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=0 P=1 */
 #define PT_DEV_L3 0xe73 /* nG=1 AF=1 SH=10 AP=01 NS=1 ATTR=100 T=1 P=1 */
 
@@ -227,10 +227,10 @@ skip_bss:
         /* Set up the HTCR:
          * PASize -- 40 bits / 1TB
          * Top byte is used
-         * PT walks use Outer-Shareable accesses,
+         * PT walks use Inner-Shareable accesses,
          * PT walks are write-back, write-allocate in both cache levels,
          * Full 64-bit address space goes through this table. */
-        ldr   x0, =0x80822500
+        ldr   x0, =0x80823500
         msr   tcr_el2, x0
 
         /* Set up the SCTLR_EL2:
diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 308a798..f608020 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -212,9 +212,8 @@ void dump_hyp_walk(vaddr_t addr)
 /* Map a 4k page in a fixmap entry */
 void set_fixmap(unsigned map, unsigned long mfn, unsigned attributes)
 {
-    lpae_t pte = mfn_to_xen_entry(mfn);
+    lpae_t pte = mfn_to_xen_entry(mfn, attributes);
     pte.pt.table = 1; /* 4k mappings always have this bit set */
-    pte.pt.ai = attributes;
     pte.pt.xn = 1;
     write_pte(xen_fixmap + third_table_offset(FIXMAP_ADDR(map)), pte);
     flush_xen_data_tlb_range_va(FIXMAP_ADDR(map), PAGE_SIZE);
@@ -270,7 +269,7 @@ void *map_domain_page(unsigned long mfn)
         else if ( map[slot].pt.avail == 0 )
         {
             /* Commandeer this 2MB slot */
-            pte = mfn_to_xen_entry(slot_mfn);
+            pte = mfn_to_xen_entry(slot_mfn, WRITEALLOC);
             pte.pt.avail = 1;
             write_pte(map + slot, pte);
             break;
@@ -401,7 +400,7 @@ static inline lpae_t pte_of_xenaddr(vaddr_t va)
 {
     paddr_t ma = va + phys_offset;
     unsigned long mfn = ma >> PAGE_SHIFT;
-    return mfn_to_xen_entry(mfn);
+    return mfn_to_xen_entry(mfn, WRITEALLOC);
 }
 
 void __init remove_early_mappings(void)
@@ -422,6 +421,12 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
     lpae_t pte, *p;
     int i;
 
+    /* Map the destination in the boot misc area. */
+    dest_va = BOOT_RELOC_VIRT_START;
+    pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
+    write_pte(xen_second + second_table_offset(dest_va), pte);
+    flush_xen_data_tlb_range_va(dest_va, SECOND_SIZE);
+
     /* Calculate virt-to-phys offset for the new location */
     phys_offset = xen_paddr - (unsigned long) _start;
 
@@ -455,7 +460,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
     /* Initialise xen second level entries ... */
     /* ... Xen's text etc */
 
-    pte = mfn_to_xen_entry(xen_paddr>>PAGE_SHIFT);
+    pte = mfn_to_xen_entry(xen_paddr>>PAGE_SHIFT, WRITEALLOC);
     pte.pt.xn = 0;/* Contains our text mapping! */
     xen_second[second_table_offset(XEN_VIRT_START)] = pte;
 
@@ -470,7 +475,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
 
     /* Map the destination in the boot misc area. */
     dest_va = BOOT_RELOC_VIRT_START;
-    pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT);
+    pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
     write_pte(boot_second + second_table_offset(dest_va), pte);
     flush_xen_data_tlb_range_va(dest_va, SECOND_SIZE);
 #ifdef CONFIG_ARM_64
@@ -499,7 +504,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr)
         unsigned long va = XEN_VIRT_START + (i << PAGE_SHIFT);
         if ( !is_kernel(va) )
             break;
-        pte = mfn_to_xen_entry(mfn);
+        pte = mfn_to_xen_entry(mfn, WRITEALLOC);
         pte.pt.table = 1; /* 4k mappings always have this bit set */
         if ( is_kernel_text(va) || is_kernel_inittext(va) )
         {
@@ -569,7 +574,7 @@ int init_secondary_pagetables(int cpu)
      * domheap mapping pages. */
     for ( i = 0; i < DOMHEAP_SECOND_PAGES; i++ )
     {
-        pte = mfn_to_xen_entry(virt_to_mfn(domheap+i*LPAE_ENTRIES));
+        pte = mfn_to_xen_entry(virt_to_mfn(domheap+i*LPAE_ENTRIES), WRITEALLOC);
         pte.pt.table = 1;
         write_pte(&first[first_table_offset(DOMHEAP_VIRT_START+i*FIRST_SIZE)], pte);
     }
@@ -614,7 +619,7 @@ static void __init create_32mb_mappings(lpae_t *second,
 
     count = nr_mfns / LPAE_ENTRIES;
     p = second + second_linear_offset(virt_offset);
-    pte = mfn_to_xen_entry(base_mfn);
+    pte = mfn_to_xen_entry(base_mfn, WRITEALLOC);
     pte.pt.contig = 1;  /* These maps are in 16-entry contiguous chunks. */
     for ( i = 0; i < count; i++ )
     {
@@ -686,13 +691,13 @@ void __init setup_xenheap_mappings(unsigned long base_mfn,
         else
         {
             unsigned long first_mfn = alloc_boot_pages(1, 1);
-            pte = mfn_to_xen_entry(first_mfn);
+            pte = mfn_to_xen_entry(first_mfn, WRITEALLOC);
             pte.pt.table = 1;
             write_pte(p, pte);
             first = mfn_to_virt(first_mfn);
         }
 
-        pte = mfn_to_xen_entry(base_mfn);
+        pte = mfn_to_xen_entry(base_mfn, WRITEALLOC);
         /* TODO: Set pte.pt.contig when appropriate. */
         write_pte(&first[first_table_offset(vaddr)], pte);
 
@@ -728,7 +733,7 @@ void __init setup_frametable_mappings(paddr_t ps, paddr_t pe)
     second = mfn_to_virt(second_base);
     for ( i = 0; i < nr_second; i++ )
     {
-        pte = mfn_to_xen_entry(second_base + i);
+        pte = mfn_to_xen_entry(second_base + i, WRITEALLOC);
         pte.pt.table = 1;
         write_pte(&xen_first[first_table_offset(FRAMETABLE_VIRT_START)+i], pte);
     }
@@ -780,7 +785,7 @@ static int create_xen_table(lpae_t *entry)
     if ( p == NULL )
         return -ENOMEM;
     clear_page(p);
-    pte = mfn_to_xen_entry(virt_to_mfn(p));
+    pte = mfn_to_xen_entry(virt_to_mfn(p), WRITEALLOC);
     pte.pt.table = 1;
     write_pte(entry, pte);
     return 0;
@@ -826,9 +831,8 @@ static int create_xen_entries(enum xenmap_operation op,
                            addr, mfn);
                     return -EINVAL;
                 }
-                pte = mfn_to_xen_entry(mfn);
+                pte = mfn_to_xen_entry(mfn, ai);
                 pte.pt.table = 1;
-                pte.pt.ai = ai;
                 write_pte(&third[third_table_offset(addr)], pte);
                 break;
             case REMOVE:
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index d00c882..b9d8ca6 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -145,10 +145,10 @@ static lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr,
                                p2m_type_t t)
 {
     paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
-    /* xn and write bit will be defined in the switch */
+    /* sh, xn and write bit will be defined in the following switches
+     * based on mattr and t. */
     lpae_t e = (lpae_t) {
         .p2m.af = 1,
-        .p2m.sh = LPAE_SH_OUTER,
         .p2m.read = 1,
         .p2m.mattr = mattr,
         .p2m.table = 1,
@@ -158,6 +158,20 @@ static lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr,
 
     BUILD_BUG_ON(p2m_max_real_type > (1 << 4));
 
+    switch (mattr)
+    {
+    case MATTR_MEM:
+        e.p2m.sh = LPAE_SH_INNER;
+        break;
+
+    case MATTR_DEV:
+        e.p2m.sh = LPAE_SH_OUTER;
+        break;
+    default:
+        BUG();
+        break;
+    }
+
     switch (t)
     {
     case p2m_ram_rw:
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index e00be9e..6dc7fa6 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -185,7 +185,7 @@ typedef union {
 /* Standard entry type that we'll use to build Xen's own pagetables.
  * We put the same permissions at every level, because they're ignored
  * by the walker in non-leaf entries. */
-static inline lpae_t mfn_to_xen_entry(unsigned long mfn)
+static inline lpae_t mfn_to_xen_entry(unsigned long mfn, unsigned attr)
 {
     paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
     lpae_t e = (lpae_t) {
@@ -193,10 +193,9 @@ static inline lpae_t mfn_to_xen_entry(unsigned long mfn)
             .xn = 1,              /* No need to execute outside .text */
             .ng = 1,              /* Makes TLB flushes easier */
             .af = 1,              /* No need for access tracking */
-            .sh = LPAE_SH_OUTER,  /* Xen mappings are globally coherent */
             .ns = 1,              /* Hyp mode is in the non-secure world */
             .user = 1,            /* See below */
-            .ai = WRITEALLOC,
+            .ai = attr,
             .table = 0,           /* Set to 1 for links and 4k maps */
             .valid = 1,           /* Mappings are present */
         }};;
@@ -205,6 +204,38 @@ static inline lpae_t mfn_to_xen_entry(unsigned long mfn)
      * pagetables un User mode it's OK.  If this changes, remember
      * to update the hard-coded values in head.S too */
 
+    switch ( attr )
+    {
+    case BUFFERABLE:
+        /*
+         * ARM ARM: Overlaying the shareability attribute (DDI
+         * 0406C.b B3-1376 to 1377)
+         *
+         * A memory region with a resultant memory type attribute of Normal,
+         * and a resultant cacheability attribute of Inner Non-cacheable,
+         * Outer Non-cacheable, must have a resultant shareability attribute
+         * of Outer Shareable, otherwise shareability is UNPREDICTABLE.
+         *
+         * On ARMv8 sharability is ignored and explicitly treated as Outer
+         * Shareable for Normal Inner Non_cacheable, Outer Non-cacheable.
+         */
+        e.pt.sh = LPAE_SH_OUTER;
+        break;
+    case UNCACHED:
+    case DEV_SHARED:
+        /* Shareability is ignored for non-Normal memory, Outer is as
+         * good as anything.
+         *
+         * On ARMv8 sharability is ignored and explicitly treated as Outer
+         * Shareable for any device memory type.
+         */
+        e.pt.sh = LPAE_SH_OUTER;
+        break;
+    default:
+        e.pt.sh = LPAE_SH_INNER;  /* Xen mappings are SMP coherent */
+        break;
+    }
+
     ASSERT(!(pa & ~PAGE_MASK));
     ASSERT(!(pa & ~PADDR_MASK));
 
-- 
1.7.10.4

[PATCH for-4.5 v2 2/8] xen: arm: Only upgrade guest barriers to inner shareable.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 xen/arch/arm/traps.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 21c7b26..72fd620 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -75,7 +75,7 @@ void __cpuinit init_traps(void)
     WRITE_SYSREG((vaddr_t)hyp_traps_vector, VBAR_EL2);
 
     /* Setup hypervisor traps */
-    WRITE_SYSREG(HCR_PTW|HCR_BSU_OUTER|HCR_AMO|HCR_IMO|HCR_VM|HCR_TWI|HCR_TSC|
+    WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_VM|HCR_TWI|HCR_TSC|
                  HCR_TAC, HCR_EL2);
     isb();
 }
-- 
1.7.10.4

[PATCH for-4.5 v2 3/8] xen: arm: consolidate barrier definitions

[Ian Campbell]

These are effectively identical on both 32- and 64-bit.

The only difference is that they implicit "sy" on 32-bit becomes explicit.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
---
 xen/include/asm-arm/arm32/system.h |   16 ----------------
 xen/include/asm-arm/arm64/system.h |   17 -----------------
 xen/include/asm-arm/system.h       |   16 ++++++++++++++++
 3 files changed, 16 insertions(+), 33 deletions(-)

diff --git a/xen/include/asm-arm/arm32/system.h b/xen/include/asm-arm/arm32/system.h
index 60148cb..9f233fe 100644
--- a/xen/include/asm-arm/arm32/system.h
+++ b/xen/include/asm-arm/arm32/system.h
@@ -2,22 +2,6 @@
 #ifndef __ASM_ARM32_SYSTEM_H
 #define __ASM_ARM32_SYSTEM_H
 
-#define sev() __asm__ __volatile__ ("sev" : : : "memory")
-#define wfe() __asm__ __volatile__ ("wfe" : : : "memory")
-#define wfi() __asm__ __volatile__ ("wfi" : : : "memory")
-
-#define isb() __asm__ __volatile__ ("isb" : : : "memory")
-#define dsb() __asm__ __volatile__ ("dsb" : : : "memory")
-#define dmb() __asm__ __volatile__ ("dmb" : : : "memory")
-
-#define mb()            dsb()
-#define rmb()           dsb()
-#define wmb()           mb()
-
-#define smp_mb()        mb()
-#define smp_rmb()       rmb()
-#define smp_wmb()       wmb()
-
 extern void __bad_xchg(volatile void *, int);
 
 static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int size)
diff --git a/xen/include/asm-arm/arm64/system.h b/xen/include/asm-arm/arm64/system.h
index d7e912f..570af5c 100644
--- a/xen/include/asm-arm/arm64/system.h
+++ b/xen/include/asm-arm/arm64/system.h
@@ -2,23 +2,6 @@
 #ifndef __ASM_ARM64_SYSTEM_H
 #define __ASM_ARM64_SYSTEM_H
 
-#define sev()           asm volatile("sev" : : : "memory")
-#define wfe()           asm volatile("wfe" : : : "memory")
-#define wfi()           asm volatile("wfi" : : : "memory")
-
-#define isb()           asm volatile("isb" : : : "memory")
-#define dsb()           asm volatile("dsb sy" : : : "memory")
-#define dmb()           asm volatile("dmb sy" : : : "memory")
-
-#define mb()            dsb()
-#define rmb()           dsb()
-#define wmb()           mb()
-
-#define smp_mb()        mb()
-#define smp_rmb()       rmb()
-#define smp_wmb()       wmb()
-
-
 extern void __bad_xchg(volatile void *, int);
 
 static inline unsigned long __xchg(unsigned long x, volatile void *ptr, int size)
diff --git a/xen/include/asm-arm/system.h b/xen/include/asm-arm/system.h
index 290d38d..e003624 100644
--- a/xen/include/asm-arm/system.h
+++ b/xen/include/asm-arm/system.h
@@ -8,6 +8,22 @@
 #define nop() \
     asm volatile ( "nop" )
 
+#define sev()           asm volatile("sev" : : : "memory")
+#define wfe()           asm volatile("wfe" : : : "memory")
+#define wfi()           asm volatile("wfi" : : : "memory")
+
+#define isb()           asm volatile("isb" : : : "memory")
+#define dsb()           asm volatile("dsb sy" : : : "memory")
+#define dmb()           asm volatile("dmb sy" : : : "memory")
+
+#define mb()            dsb()
+#define rmb()           dsb()
+#define wmb()           mb()
+
+#define smp_mb()        mb()
+#define smp_rmb()       rmb()
+#define smp_wmb()       wmb()
+
 #define xchg(ptr,x) \
         ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
 
-- 
1.7.10.4

[PATCH for-4.5 v2 4/8] xen: arm: Use SMP barriers when that is all which is required.

[Ian Campbell]

SMP barriers can be used when all we care about is synchronising against other
processors.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
---
v2: Turn cpu_die mb()s into smp_mb()s.
---
 xen/arch/arm/mm.c      |    2 +-
 xen/arch/arm/smpboot.c |   10 +++++-----
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index f608020..ff19e39 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -967,7 +967,7 @@ void share_xen_page_with_guest(struct page_info *page,
     page->u.inuse.type_info |= PGT_validated | 1;
 
     page_set_owner(page, d);
-    wmb(); /* install valid domain ptr before updating refcnt. */
+    smp_wmb(); /* install valid domain ptr before updating refcnt. */
     ASSERT((page->count_info & ~PGC_xen_heap) == 0);
 
     /* Only add to the allocation list if the domain isn't dying. */
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index a829957..ce68d34 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -298,12 +298,12 @@ void __cpuinit start_secondary(unsigned long boot_phys_offset,
 
     /* Run local notifiers */
     notify_cpu_starting(cpuid);
-    wmb();
+    smp_wmb();
 
     /* Now report this CPU is up */
     smp_up_cpu = MPIDR_INVALID;
     cpumask_set_cpu(cpuid, &cpu_online_map);
-    wmb();
+    smp_wmb();
 
     local_irq_enable();
     local_abort_enable();
@@ -330,7 +330,7 @@ void __cpu_disable(void)
 
     if ( cpu_disable_scheduler(cpu) )
         BUG();
-    mb();
+    smp_mb();
 
     /* Return to caller; eventually the IPI mechanism will unwind and the 
      * scheduler will drop to the idle loop, which will call stop_cpu(). */
@@ -411,10 +411,10 @@ void __cpu_die(unsigned int cpu)
         process_pending_softirqs();
         if ( (++i % 10) == 0 )
             printk(KERN_ERR "CPU %u still not dead...\n", cpu);
-        mb();
+        smp_mb();
     }
     cpu_is_dead = 0;
-    mb();
+    smp_mb();
 }
 
 /*
-- 
1.7.10.4

[PATCH for-4.5 v2 5/8] xen: arm: Use dmb for smp barriers

[Ian Campbell]

The full power of dsb is not required in this context.

Also change wmb() to be dsb() directly instead of indirectly via mb(), for
clarity.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
---
 xen/include/asm-arm/system.h |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/xen/include/asm-arm/system.h b/xen/include/asm-arm/system.h
index e003624..89c61ef 100644
--- a/xen/include/asm-arm/system.h
+++ b/xen/include/asm-arm/system.h
@@ -18,11 +18,11 @@
 
 #define mb()            dsb()
 #define rmb()           dsb()
-#define wmb()           mb()
+#define wmb()           dsb()
 
-#define smp_mb()        mb()
-#define smp_rmb()       rmb()
-#define smp_wmb()       wmb()
+#define smp_mb()        dmb()
+#define smp_rmb()       dmb()
+#define smp_wmb()       dmb()
 
 #define xchg(ptr,x) \
         ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
-- 
1.7.10.4

[PATCH for-4.5 v2 6/8] xen: arm: add scope to dsb and dmb macros

[Ian Campbell]

Everywhere currently passes "sy"stem, so no actual change.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
---
 xen/arch/arm/domain.c                |    2 +-
 xen/arch/arm/gic.c                   |   10 +++++-----
 xen/arch/arm/mm.c                    |    4 ++--
 xen/arch/arm/platforms/vexpress.c    |    6 +++---
 xen/arch/arm/smpboot.c               |    2 +-
 xen/arch/arm/time.c                  |    2 +-
 xen/drivers/video/arm_hdlcd.c        |    2 +-
 xen/include/asm-arm/arm32/flushtlb.h |   16 ++++++++--------
 xen/include/asm-arm/arm32/page.h     |    4 ++--
 xen/include/asm-arm/arm64/page.h     |    4 ++--
 xen/include/asm-arm/page.h           |    4 ++--
 xen/include/asm-arm/system.h         |   16 ++++++++--------
 12 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 8f20fdf..b27f32f 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -47,7 +47,7 @@ void idle_loop(void)
         local_irq_disable();
         if ( cpu_is_haltable(smp_processor_id()) )
         {
-            dsb();
+            dsb(sy);
             wfi();
         }
         local_irq_enable();
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 13bbf48..1467b69 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -137,7 +137,7 @@ static void gic_irq_enable(struct irq_desc *desc)
     spin_lock_irqsave(&desc->lock, flags);
     spin_lock(&gic.lock);
     desc->status &= ~IRQ_DISABLED;
-    dsb();
+    dsb(sy);
     /* Enable routing */
     GICD[GICD_ISENABLER + irq / 32] = (1u << (irq % 32));
     spin_unlock(&gic.lock);
@@ -478,7 +478,7 @@ void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
     cpumask_and(&online_mask, cpumask, &cpu_online_map);
     mask = gic_cpu_mask(&online_mask);
 
-    dsb();
+    dsb(sy);
 
     GICD[GICD_SGIR] = GICD_SGI_TARGET_LIST
         | (mask<<GICD_SGI_TARGET_SHIFT)
@@ -495,7 +495,7 @@ void send_SGI_self(enum gic_sgi sgi)
 {
     ASSERT(sgi < 16); /* There are only 16 SGIs */
 
-    dsb();
+    dsb(sy);
 
     GICD[GICD_SGIR] = GICD_SGI_TARGET_SELF
         | sgi;
@@ -505,7 +505,7 @@ void send_SGI_allbutself(enum gic_sgi sgi)
 {
    ASSERT(sgi < 16); /* There are only 16 SGIs */
 
-   dsb();
+   dsb(sy);
 
    GICD[GICD_SGIR] = GICD_SGI_TARGET_OTHERS
        | sgi;
@@ -589,7 +589,7 @@ static int __setup_irq(struct irq_desc *desc, unsigned int irq,
         return -EBUSY;
 
     desc->action  = new;
-    dsb();
+    dsb(sy);
 
     return 0;
 }
diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index ff19e39..20dbb90 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -345,10 +345,10 @@ void flush_page_to_ram(unsigned long mfn)
 {
     void *p, *v = map_domain_page(mfn);
 
-    dsb();           /* So the CPU issues all writes to the range */
+    dsb(sy);         /* So the CPU issues all writes to the range */
     for ( p = v; p < v + PAGE_SIZE ; p += cacheline_bytes )
         asm volatile (__clean_and_invalidate_xen_dcache_one(0) : : "r" (p));
-    dsb();           /* So we know the flushes happen before continuing */
+    dsb(sy);         /* So we know the flushes happen before continuing */
 
     unmap_domain_page(v);
 }
diff --git a/xen/arch/arm/platforms/vexpress.c b/xen/arch/arm/platforms/vexpress.c
index 6132056..8e6a4ea 100644
--- a/xen/arch/arm/platforms/vexpress.c
+++ b/xen/arch/arm/platforms/vexpress.c
@@ -48,7 +48,7 @@ static inline int vexpress_ctrl_start(uint32_t *syscfg, int write,
     /* wait for complete flag to be set */
     do {
         stat = syscfg[V2M_SYS_CFGSTAT/4];
-        dsb();
+        dsb(sy);
     } while ( !(stat & V2M_SYS_CFG_COMPLETE) );
 
     /* check error status and return error flag if set */
@@ -113,10 +113,10 @@ static void vexpress_reset(void)
 
     /* switch to slow mode */
     writel(0x3, sp810);
-    dsb(); isb();
+    dsb(sy); isb();
     /* writing any value to SCSYSSTAT reg will reset the system */
     writel(0x1, sp810 + 4);
-    dsb(); isb();
+    dsb(sy); isb();
 
     iounmap(sp810);
 }
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index ce68d34..7f28b68 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -341,7 +341,7 @@ void stop_cpu(void)
     local_irq_disable();
     cpu_is_dead = 1;
     /* Make sure the write happens before we sleep forever */
-    dsb();
+    dsb(sy);
     isb();
     while ( 1 )
         wfi();
diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index 81e3e28..93d957a 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -260,7 +260,7 @@ void udelay(unsigned long usecs)
     s_time_t deadline = get_s_time() + 1000 * (s_time_t) usecs;
     while ( get_s_time() - deadline < 0 )
         ;
-    dsb();
+    dsb(sy);
     isb();
 }
 
diff --git a/xen/drivers/video/arm_hdlcd.c b/xen/drivers/video/arm_hdlcd.c
index 647f22c..e5ad18d 100644
--- a/xen/drivers/video/arm_hdlcd.c
+++ b/xen/drivers/video/arm_hdlcd.c
@@ -78,7 +78,7 @@ void (*video_puts)(const char *) = vga_noop_puts;
 
 static void hdlcd_flush(void)
 {
-    dsb();
+    dsb(sy);
 }
 
 static int __init get_color_masks(const char* bpp, struct color_masks **masks)
diff --git a/xen/include/asm-arm/arm32/flushtlb.h b/xen/include/asm-arm/arm32/flushtlb.h
index 7183a07..bbcc82f 100644
--- a/xen/include/asm-arm/arm32/flushtlb.h
+++ b/xen/include/asm-arm/arm32/flushtlb.h
@@ -4,44 +4,44 @@
 /* Flush local TLBs, current VMID only */
 static inline void flush_tlb_local(void)
 {
-    dsb();
+    dsb(sy);
 
     WRITE_CP32((uint32_t) 0, TLBIALL);
 
-    dsb();
+    dsb(sy);
     isb();
 }
 
 /* Flush inner shareable TLBs, current VMID only */
 static inline void flush_tlb(void)
 {
-    dsb();
+    dsb(sy);
 
     WRITE_CP32((uint32_t) 0, TLBIALLIS);
 
-    dsb();
+    dsb(sy);
     isb();
 }
 
 /* Flush local TLBs, all VMIDs, non-hypervisor mode */
 static inline void flush_tlb_all_local(void)
 {
-    dsb();
+    dsb(sy);
 
     WRITE_CP32((uint32_t) 0, TLBIALLNSNH);
 
-    dsb();
+    dsb(sy);
     isb();
 }
 
 /* Flush innershareable TLBs, all VMIDs, non-hypervisor mode */
 static inline void flush_tlb_all(void)
 {
-    dsb();
+    dsb(sy);
 
     WRITE_CP32((uint32_t) 0, TLBIALLNSNHIS);
 
-    dsb();
+    dsb(sy);
     isb();
 }
 
diff --git a/xen/include/asm-arm/arm32/page.h b/xen/include/asm-arm/arm32/page.h
index b8221ca..191a108 100644
--- a/xen/include/asm-arm/arm32/page.h
+++ b/xen/include/asm-arm/arm32/page.h
@@ -67,13 +67,13 @@ static inline void flush_xen_data_tlb(void)
 static inline void flush_xen_data_tlb_range_va(unsigned long va, unsigned long size)
 {
     unsigned long end = va + size;
-    dsb(); /* Ensure preceding are visible */
+    dsb(sy); /* Ensure preceding are visible */
     while ( va < end ) {
         asm volatile(STORE_CP32(0, TLBIMVAH)
                      : : "r" (va) : "memory");
         va += PAGE_SIZE;
     }
-    dsb(); /* Ensure completion of the TLB flush */
+    dsb(sy); /* Ensure completion of the TLB flush */
     isb();
 }
 
diff --git a/xen/include/asm-arm/arm64/page.h b/xen/include/asm-arm/arm64/page.h
index 3352821..20b4c5a 100644
--- a/xen/include/asm-arm/arm64/page.h
+++ b/xen/include/asm-arm/arm64/page.h
@@ -60,13 +60,13 @@ static inline void flush_xen_data_tlb(void)
 static inline void flush_xen_data_tlb_range_va(unsigned long va, unsigned long size)
 {
     unsigned long end = va + size;
-    dsb(); /* Ensure preceding are visible */
+    dsb(sy); /* Ensure preceding are visible */
     while ( va < end ) {
         asm volatile("tlbi vae2, %0;"
                      : : "r" (va>>PAGE_SHIFT) : "memory");
         va += PAGE_SIZE;
     }
-    dsb(); /* Ensure completion of the TLB flush */
+    dsb(sy); /* Ensure completion of the TLB flush */
     isb();
 }
 
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index 6dc7fa6..5e4678e 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -263,10 +263,10 @@ extern size_t cacheline_bytes;
 static inline void clean_xen_dcache_va_range(void *p, unsigned long size)
 {
     void *end;
-    dsb();           /* So the CPU issues all writes to the range */
+    dsb(sy);           /* So the CPU issues all writes to the range */
     for ( end = p + size; p < end; p += cacheline_bytes )
         asm volatile (__clean_xen_dcache_one(0) : : "r" (p));
-    dsb();           /* So we know the flushes happen before continuing */
+    dsb(sy);           /* So we know the flushes happen before continuing */
 }
 
 /* Macro for flushing a single small item.  The predicate is always
diff --git a/xen/include/asm-arm/system.h b/xen/include/asm-arm/system.h
index 89c61ef..e1f126a 100644
--- a/xen/include/asm-arm/system.h
+++ b/xen/include/asm-arm/system.h
@@ -13,16 +13,16 @@
 #define wfi()           asm volatile("wfi" : : : "memory")
 
 #define isb()           asm volatile("isb" : : : "memory")
-#define dsb()           asm volatile("dsb sy" : : : "memory")
-#define dmb()           asm volatile("dmb sy" : : : "memory")
+#define dsb(scope)      asm volatile("dsb " #scope : : : "memory")
+#define dmb(scope)      asm volatile("dmb " #scope : : : "memory")
 
-#define mb()            dsb()
-#define rmb()           dsb()
-#define wmb()           dsb()
+#define mb()            dsb(sy)
+#define rmb()           dsb(sy)
+#define wmb()           dsb(sy)
 
-#define smp_mb()        dmb()
-#define smp_rmb()       dmb()
-#define smp_wmb()       dmb()
+#define smp_mb()        dmb(sy)
+#define smp_rmb()       dmb(sy)
+#define smp_wmb()       dmb(sy)
 
 #define xchg(ptr,x) \
         ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
-- 
1.7.10.4

[PATCH for-4.5 v2 7/8] xen: arm: weaken SMP barriers to inner shareable.

[Ian Campbell]

Since all processors are in the inner-shareable domain and we map everything
that way this is sufficient.

The non-SMP barriers remain full system. Although in principle they could
become outer shareable barriers for some hardware this would require us to
know which class a given device is. Given the small number of device drivers
in Xen itself its probably not worth worrying over, although maybe someone
will benchmark at some point.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
---
 xen/include/asm-arm/system.h |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/xen/include/asm-arm/system.h b/xen/include/asm-arm/system.h
index e1f126a..32ed277 100644
--- a/xen/include/asm-arm/system.h
+++ b/xen/include/asm-arm/system.h
@@ -20,9 +20,9 @@
 #define rmb()           dsb(sy)
 #define wmb()           dsb(sy)
 
-#define smp_mb()        dmb(sy)
-#define smp_rmb()       dmb(sy)
-#define smp_wmb()       dmb(sy)
+#define smp_mb()        dmb(ish)
+#define smp_rmb()       dmb(ish)
+#define smp_wmb()       dmb(ish)
 
 #define xchg(ptr,x) \
         ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
-- 
1.7.10.4

[PATCH for-4.5 v2 8/8] xen: arm: use more specific barriers for read and write barriers.

[Ian Campbell]

Note that 32-bit does not provide a load variant of the inner shareable
barrier, so that remains a full any-any barrier.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
---
 xen/include/asm-arm/system.h |   17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/xen/include/asm-arm/system.h b/xen/include/asm-arm/system.h
index 32ed277..7aaaf50 100644
--- a/xen/include/asm-arm/system.h
+++ b/xen/include/asm-arm/system.h
@@ -17,12 +17,21 @@
 #define dmb(scope)      asm volatile("dmb " #scope : : : "memory")
 
 #define mb()            dsb(sy)
-#define rmb()           dsb(sy)
-#define wmb()           dsb(sy)
+#ifdef CONFIG_ARM_64
+#define rmb()           dsb(ld)
+#else
+#define rmb()           dsb(sy) /* 32-bit has no ld variant. */
+#endif
+#define wmb()           dsb(st)
 
 #define smp_mb()        dmb(ish)
-#define smp_rmb()       dmb(ish)
-#define smp_wmb()       dmb(ish)
+#ifdef CONFIG_ARM_64
+#define smp_rmb()       dmb(ishld)
+#else
+#define smp_rmb()       dmb(ish) /* 32-bit has no ishld variant. */
+#endif
+
+#define smp_wmb()       dmb(ishst)
 
 #define xchg(ptr,x) \
         ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
-- 
1.7.10.4