[PATCH 0/38] arm: boot a dom1 to "Calibrating delay loop" then hang

[PATCH 0/38] arm: boot a dom1 to "Calibrating delay loop" then hang

[Ian Campbell]

Sorry for the enormous dump, I seem to have accumulated a boat load of
stuff in my tree...

There's actually a bunch of random guff in here, but the main bit is
xc_dom code in libxc to build a dom1 from a ARM zImage and enough
hypervisor support to run it to:
        [    0.000000] Linux version 3.2.0-rc5-arm-native+ (ianc@drall) (gcc version 4.6.0 (GCC) ) #174 Thu May 24 15:44:30 BST 2012
        [    0.000000] CPU: ARMv7 Processor [410fc0f0] revision 0 (ARMv7), cr=10c53c7d
        [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
        [    0.000000] Machine: ARM Versatile Express, model: V2P-AEMv7A
        [    0.000000] bootconsole [xenboot0] enabled
        [    0.000000] Memory policy: ECC disabled, Data cache writeback
        [    0.000000] Architected local timer running at 100.00MHz.
        [    0.000000] sched_clock: 32 bits at 100MHz, resolution 10ns, wraps every 42949ms
        [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
        [    0.000000] Kernel command line: earlyprintk=xen console=hvc0
        [    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
        [    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
        [    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
        [    0.000000] Memory: 128MB = 128MB total
        [    0.000000] Memory: 126256k/126256k available, 4816k reserved, 0K highmem
        [    0.000000] Virtual kernel memory layout:
        [    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
        [    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
        [    0.000000]     vmalloc : 0xc8800000 - 0xf8000000   ( 760 MB)
        [    0.000000]     lowmem  : 0xc0000000 - 0xc8000000   ( 128 MB)
        [    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
        [    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
        [    0.000000]       .text : 0xc0008000 - 0xc032cce0   (3220 kB)
        [    0.000000]       .init : 0xc032d000 - 0xc034e000   ( 132 kB)
        [    0.000000]       .data : 0xc034e000 - 0xc036f400   ( 133 kB)
        [    0.000000]        .bss : 0xc036f424 - 0xc038e0ec   ( 124 kB)
        [    0.000000] NR_IRQS:256
        [    0.000000] Console: colour dummy device 80x30
        [    0.191514] Calibrating delay loop... 

(Stefano is looking at virtual timers and interrupt injection as we
speak...)

There is some non-ARM impact, specifically teaching xc_dom_* about the
idea that RAM might not start at 0x000 (on the ARM platform we are
similar too RAM starts at 0x80000000), I think that makes at least the
libxc parts non-4.2 material at this stage.

There are also three (more than usually) hacky patches needed to get to
this point:
        HACK: arm: initial XENMAPSPACE_gmfn_foreign
        HACK: add simple xcbuild
        HACK: arm: disable hypercall continuations.
which I'm not proposing but I'm including because it doesn't work
without them ;-).

The rest I think is fair game for acking and cherry-picking the
acceptable bits for commit.

I still need to cleanup the Linux side of this, that probably won't
happen today and these patches probably don't work without those
changes... FWIW the XENMAPSPACE_gmfn_foreign thing is an even bigger
hack there. Ultimately we should do the same thing as hybrid x86

Ian.

The following changes since commit 5fcec8e92ea02240c2737c4fa982027cef053401:

  libxl: fix typos in libxl_cpuid_parse_config (2012-06-01 12:06:22 +0100)

are available in the git repository at:

  git://xenbits.xen.org/people/ianc/xen-unstable.git devel/arm

for you to fetch changes up to 5177d39f17feb3d5ccdedb976074cb0eee62ee69:

  HACK: arm: disable hypercall continuations. (2012-06-01 15:14:28 +0000)

----------------------------------------------------------------
Ian Campbell (38):
      arm: allocate top level p2m page for all non-idle domains
      arm: handy function to print a walk of the hypervisor page tables
      arm: handy function to print a walk of a domain's p2m.
      arm: correct and expand TLB flush CP15 registers
      arm: restore stack on return from trap.
      arm: enable interrupts while handling traps
      arm: hook up domctl and memory_op
      arm: allocate and setup a guest vcpu.
      arm: print domid as part of debug trap
      arm: remove unnecessarily verbose print from p2m_load_VTTBR
      arm: implement p2m lookup
      arm: remove hard tabs from init_idle_domain
      arm: stub out sync_vcpu_execstate
      arm: do not set max_vcpus = 8 in arch_domain_create.
      arm: implement stub version of flush_tlb_mask.
      arm: Add simple cpu_{sibling,core}_mask
      arm: allow p2m to be created with specific MATTR.
      arm: implement vpl011 (UART) emulator.
      arm: context switch a bunch of guest state.
      arm: dump a page table walk when va_to_par fails.
      arm: dump guest s1 walk on data abort which is not a stage 2 issue.
      arm: implement vcpu_show_execution_state
      arm: use correct attributes for mappings in copy_from_paddr()
      arm: map fixmaps non-executable.
      arm: remove old identity map of boot paddr when we are done with it.
      arm: fix locking in create_p2m_entries
      arm: split pending SPIs (global) out from pending PPIs and SGIs (per CPU)
      arm: map GICV in all domains, not just dom0.
      arm: delay enabling data-cache until paging enabled.
      arm: Upgrade guest barriers to Outer-Shareable. Enable Protected Table Walk.
      arm: gic.lock can be taken in interrupt context, so lock appropriately.
      arm: context switch virtual timer registers
      arm: the hyp timer seems to work now,  default to using it.
      HACK: arm: initial XENMAPSPACE_gmfn_foreign
      arm: move PSR flag definitions into interface, for tools use.
      libxc: add ARM support to xc_dom (PV domain building)
      HACK: add simple xcbuild
      HACK: arm: disable hypercall continuations.

 tools/libxc/Makefile                 |    1 +
 tools/libxc/xc_dom.h                 |    5 +-
 tools/libxc/xc_dom_arm.c             |  135 +++++++++++++++++++-
 tools/libxc/xc_dom_armzimageloader.c |  167 ++++++++++++++++++++++++
 tools/libxc/xc_dom_core.c            |   12 ++-
 tools/libxc/xg_private.h             |    4 +
 tools/xcutils/Makefile               |    6 +-
 tools/xcutils/xcbuild.c              |  100 +++++++++++++++
 xen/arch/arm/Makefile                |    1 +
 xen/arch/arm/domain.c                |  232 +++++++++++++++++++++++++++++++---
 xen/arch/arm/domain_build.c          |    7 +-
 xen/arch/arm/dummy.S                 |    8 --
 xen/arch/arm/entry.S                 |   23 +++-
 xen/arch/arm/gic.c                   |   56 ++++++---
 xen/arch/arm/gic.h                   |   11 ++-
 xen/arch/arm/head.S                  |    9 +-
 xen/arch/arm/io.c                    |    1 +
 xen/arch/arm/io.h                    |    1 +
 xen/arch/arm/kernel.c                |    8 +-
 xen/arch/arm/mm.c                    |   94 +++++++++++++--
 xen/arch/arm/p2m.c                   |  123 ++++++++++++++++--
 xen/arch/arm/setup.c                 |   15 ++-
 xen/arch/arm/smp.c                   |    9 ++
 xen/arch/arm/smpboot.c               |    5 +
 xen/arch/arm/time.c                  |    8 +-
 xen/arch/arm/traps.c                 |  145 +++++++++++++++++++---
 xen/arch/arm/vgic.c                  |   17 ++-
 xen/arch/arm/vpl011.c                |  155 +++++++++++++++++++++++
 xen/arch/arm/vpl011.h                |   34 +++++
 xen/arch/x86/mm.c                    |    2 +
 xen/include/asm-arm/cpregs.h         |   43 ++++++-
 xen/include/asm-arm/domain.h         |   56 ++++++++-
 xen/include/asm-arm/p2m.h            |    3 +
 xen/include/asm-arm/page.h           |   17 ++-
 xen/include/asm-arm/processor.h      |   26 +---
 xen/include/asm-arm/setup.h          |    2 +-
 xen/include/asm-arm/system.h         |    2 +-
 xen/include/public/arch-arm.h        |   36 ++++--
 xen/include/public/memory.h          |   12 +-
 xen/include/xen/sched.h              |    4 +
 40 files changed, 1426 insertions(+), 169 deletions(-)
 create mode 100644 tools/libxc/xc_dom_armzimageloader.c
 create mode 100644 tools/xcutils/xcbuild.c
 create mode 100644 xen/arch/arm/vpl011.c
 create mode 100644 xen/arch/arm/vpl011.h

[PATCH 01/38] arm: allocate top level p2m page for all non-idle domains

[Ian Campbell]

Not just dom0.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
---
 xen/arch/arm/domain.c |    3 +++
 xen/arch/arm/p2m.c    |    2 +-
 2 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 5702399..4b38790 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -201,6 +201,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
         clear_page(d->shared_info);
         share_xen_page_with_guest(
                 virt_to_page(d->shared_info), d, XENSHARE_writable);
+
+        if ( (rc = p2m_alloc_table(d)) != 0 )
+            goto fail;
     }
 
     d->max_vcpus = 8;
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 051a0e8..4f624d8 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -203,7 +203,7 @@ int p2m_alloc_table(struct domain *d)
     void *p;
 
     /* First level P2M is 2 consecutive pages */
-    page = alloc_domheap_pages(d, 1, 0);
+    page = alloc_domheap_pages(NULL, 1, 0);
     if ( page == NULL )
         return -ENOMEM;
 
-- 
1.7.9.1

[PATCH 02/38] arm: handy function to print a walk of the hypervisor page tables

[Ian Campbell]

Useful for debug but not actually used in this patch.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/mm.c          |   53 ++++++++++++++++++++++++++++++++++++++++++-
 xen/include/asm-arm/page.h |    2 +
 2 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 10ff883..c332e4c 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -42,6 +42,8 @@ static lpae_t xen_xenmap[LPAE_ENTRIES] __attribute__((__aligned__(4096)));
 /* Non-boot CPUs use this to find the correct pagetables. */
 uint64_t boot_httbr;
 
+static paddr_t xen_phys_offset;
+
 /* Limits of the Xen heap */
 unsigned long xenheap_mfn_start, xenheap_mfn_end;
 unsigned long xenheap_virt_end;
@@ -53,6 +55,53 @@ unsigned long max_page;
 
 extern char __init_begin[], __init_end[];
 
+void dump_pt_walk(uint32_t addr)
+{
+    paddr_t second_ma, third_ma;
+    lpae_t *first = NULL, *second = NULL, *third = NULL;
+    uint64_t httbr = READ_CP64(HTTBR);
+
+    printk("Walking Hypervisor VA %#"PRIx32" via HTTBR %#"PRIx64"\n",
+           addr, httbr);
+
+    BUG_ON( (lpae_t *)(unsigned long)(httbr - xen_phys_offset) != xen_pgtable );
+    first = xen_pgtable;
+    printk("1ST[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
+           first_table_offset(addr),
+           first, first_table_offset(addr),
+           virt_to_maddr(&first[first_table_offset(addr)]),
+           first[first_table_offset(addr)].bits);
+
+    if ( !first[first_table_offset(addr)].pt.valid ||
+         !first[first_table_offset(addr)].pt.table )
+        goto done;
+
+    second_ma = (paddr_t)first[first_table_offset(addr)].pt.base << PAGE_SHIFT;
+    second = (lpae_t *)(unsigned long)(second_ma - xen_phys_offset);
+    printk("2ND[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
+           second_table_offset(addr),
+           second, second_table_offset(addr),
+           virt_to_maddr(&second[second_table_offset(addr)]),
+           second[second_table_offset(addr)].bits);
+    if ( !second[second_table_offset(addr)].pt.valid ||
+         !second[second_table_offset(addr)].pt.table )
+        goto done;
+
+    third_ma = (paddr_t)second[second_table_offset(addr)].pt.base << PAGE_SHIFT;
+    third = (lpae_t *)(unsigned long)(third_ma - xen_phys_offset);
+    printk("3RD[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
+           third_table_offset(addr),
+           third, third_table_offset(addr),
+           virt_to_maddr(&third[third_table_offset(addr)]),
+           third[third_table_offset(addr)].bits);
+    if ( !third[third_table_offset(addr)].pt.valid ||
+         !third[third_table_offset(addr)].pt.table )
+        goto done;
+
+done:
+    return;
+}
+
 /* Map a 4k page in a fixmap entry */
 void set_fixmap(unsigned map, unsigned long mfn, unsigned attributes)
 {
@@ -173,8 +222,8 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
     flush_xen_data_tlb_va(dest_va);
 
     /* Calculate virt-to-phys offset for the new location */
-    phys_offset = xen_paddr - (unsigned long) _start;
-
+    xen_phys_offset = phys_offset = xen_paddr - (unsigned long) _start;
+    
     /* Copy */
     memcpy((void *) dest_va, _start, _end - _start);
 
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index b6df64e..22c56b5 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -312,6 +312,8 @@ static inline uint64_t gva_to_ipa(uint32_t va)
 /* Bits in the PAR returned by va_to_par */
 #define PAR_FAULT 0x1
 
+extern void dump_pt_walk(uint32_t addr);
+
 #endif /* __ASSEMBLY__ */
 
 /* These numbers add up to a 39-bit input address space.  The  ARMv7-A
-- 
1.7.9.1

[PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Ian Campbell]

Useful for debug but not actually used in this patch.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c         |   34 ++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/page.h |    1 +
 2 files changed, 35 insertions(+), 0 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 4f624d8..fdbecbc 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -5,6 +5,40 @@
 #include <xen/domain_page.h>
 #include <asm/flushtlb.h>
 
+void dump_p2m_lookup(struct domain *d, paddr_t addr)
+{
+    struct p2m_domain *p2m = &d->arch.p2m;
+    lpae_t *first = NULL, *second = NULL, *third = NULL;
+
+    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
+
+    first = __map_domain_page(p2m->first_level);
+    printk("1ST[%#03llx] = %#016llx\n",
+           first_table_offset(addr),
+           first[first_table_offset(addr)].bits);
+    if ( !first[first_table_offset(addr)].p2m.valid ||
+         !first[first_table_offset(addr)].p2m.table )
+        goto done;
+
+    second = map_domain_page(first[first_table_offset(addr)].p2m.base);
+    printk("2ND[%#03llx] = %#016llx\n",
+           second_table_offset(addr),
+           second[second_table_offset(addr)].bits);
+    if ( !second[second_table_offset(addr)].p2m.valid ||
+         !second[second_table_offset(addr)].p2m.table )
+        goto done;
+
+    third = map_domain_page(second[second_table_offset(addr)].p2m.base);
+    printk("3RD[%#03llx] = %#016llx\n",
+           third_table_offset(addr),
+           third[third_table_offset(addr)].bits);
+
+done:
+    if (third) unmap_domain_page(third);
+    if (second) unmap_domain_page(second);
+    if (first) unmap_domain_page(first);
+}
+
 void p2m_load_VTTBR(struct domain *d)
 {
     struct p2m_domain *p2m = &d->arch.p2m;
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index 22c56b5..c7b6530 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -313,6 +313,7 @@ static inline uint64_t gva_to_ipa(uint32_t va)
 #define PAR_FAULT 0x1
 
 extern void dump_pt_walk(uint32_t addr);
+extern void dump_p2m_lookup(struct domain *d, paddr_t addr);
 
 #endif /* __ASSEMBLY__ */
 
-- 
1.7.9.1

[PATCH 04/38] arm: correct and expand TLB flush CP15 registers

[Ian Campbell]

Correct spelling of TLBIALLHIS and correct definition of TLBIALLNSNHIS.

Add a few more.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/include/asm-arm/cpregs.h |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/xen/include/asm-arm/cpregs.h b/xen/include/asm-arm/cpregs.h
index ee8a287..7a0b49a 100644
--- a/xen/include/asm-arm/cpregs.h
+++ b/xen/include/asm-arm/cpregs.h
@@ -172,12 +172,19 @@
 #define TLBIMVAIS       p15,0,c8,c3,1   /* Invalidate unified TLB entry by MVA inner shareable */
 #define TLBIASIDIS      p15,0,c8,c3,2   /* Invalidate unified TLB by ASID match inner shareable */
 #define TLBIMVAAIS      p15,0,c8,c3,3   /* Invalidate unified TLB entry by MVA all ASID inner shareable */
+#define ITLBIALL        p15,0,c8,c5,0   /* Invalidate instruction TLB */
+#define ITLBIMVA        p15,0,c8,c5,1   /* Invalidate instruction TLB entry by MVA */
+#define ITLBIASID       p15,0,c8,c5,2   /* Invalidate instruction TLB by ASID match */
 #define DTLBIALL        p15,0,c8,c6,0   /* Invalidate data TLB */
 #define DTLBIMVA        p15,0,c8,c6,1   /* Invalidate data TLB entry by MVA */
 #define DTLBIASID       p15,0,c8,c6,2   /* Invalidate data TLB by ASID match */
-#define TLBILLHIS       p15,4,c8,c3,0   /* Invalidate Entire Hyp. Unified TLB inner shareable */
+#define TLBIALL         p15,0,c8,c7,0   /* invalidate unified TLB */
+#define TLBIMVA         p15,0,c8,c7,1   /* invalidate unified TLB entry by MVA */
+#define TLBIASID        p15,0,c8,c7,2   /* invalid unified TLB by ASID match */
+#define TLBIMVAA        p15,0,c8,c7,3   /* invalidate unified TLB entries by MVA all ASID */
+#define TLBIALLHIS      p15,4,c8,c3,0   /* Invalidate Entire Hyp. Unified TLB inner shareable */
 #define TLBIMVAHIS      p15,4,c8,c3,1   /* Invalidate Unified Hyp. TLB by MVA inner shareable */
-#define TLBIALLNSNHIS   p15,4,c8,c7,4   /* Invalidate Entire Non-Secure Non-Hyp. Unified TLB inner shareable */
+#define TLBIALLNSNHIS   p15,4,c8,c3,4   /* Invalidate Entire Non-Secure Non-Hyp. Unified TLB inner shareable */
 #define TLBIALLH        p15,4,c8,c7,0   /* Invalidate Entire Hyp. Unified TLB */
 #define TLBIMVAH        p15,4,c8,c7,1   /* Invalidate Unified Hyp. TLB by MVA */
 #define TLBIALLNSNH     p15,4,c8,c7,4   /* Invalidate Entire Non-Secure Non-Hyp. Unified TLB */
-- 
1.7.9.1

[PATCH 05/38] arm: restore stack on return from trap.

[Ian Campbell]

We align the stack before calling into C code but we weren't undoing this on
return.

Collapse continue_(non)idle_domain into continue_new_vcpu.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c |   16 +++-------------
 xen/arch/arm/entry.S  |    5 ++++-
 2 files changed, 7 insertions(+), 14 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 4b38790..9339a11 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -16,17 +16,6 @@
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
-static void continue_idle_domain(struct vcpu *v)
-{
-    reset_stack_and_jump(idle_loop);
-}
-
-static void continue_nonidle_domain(struct vcpu *v)
-{
-    /* check_wakeup_from_wait(); */
-    reset_stack_and_jump(return_from_trap);
-}
-
 void idle_loop(void)
 {
     for ( ; ; )
@@ -72,9 +61,10 @@ static void continue_new_vcpu(struct vcpu *prev)
     schedule_tail(prev);
 
     if ( is_idle_vcpu(current) )
-        continue_idle_domain(current);
+        reset_stack_and_jump(idle_loop);
     else
-        continue_nonidle_domain(current);
+        /* check_wakeup_from_wait(); */
+        reset_stack_and_jump(return_to_new_vcpu);
 }
 
 void context_switch(struct vcpu *prev, struct vcpu *next)
diff --git a/xen/arch/arm/entry.S b/xen/arch/arm/entry.S
index f261a9f..7a22e2d 100644
--- a/xen/arch/arm/entry.S
+++ b/xen/arch/arm/entry.S
@@ -72,7 +72,9 @@ DEFINE_TRAP_ENTRY(hypervisor)
 DEFINE_TRAP_ENTRY(irq)
 DEFINE_TRAP_ENTRY(fiq)
 
-ENTRY(return_from_trap)
+return_from_trap:
+	mov sp, r11
+ENTRY(return_to_new_vcpu)
 	ldr r11, [sp, #UREGS_cpsr]
 	and r11, #PSR_MODE_MASK
 	cmp r11, #PSR_MODE_HYP
@@ -82,6 +84,7 @@ ENTRY(return_to_guest)
 	mov r11, sp
 	bic sp, #7 /* Align the stack pointer */
 	bl leave_hypervisor_tail
+	mov sp, r11
 	RESTORE_ONE_BANKED(SP_usr)
 	/* LR_usr is the same physical register as lr and is restored below */
 	RESTORE_BANKED(svc)
-- 
1.7.9.1

[PATCH 06/38] arm: enable interrupts while handling traps

[Ian Campbell]

For most traps we can do this as soon as we have saved the necessary state.
For IRQs and FIQs we must wait until we have acked the interrupt with the GIC.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/entry.S |   17 ++++++++++++++---
 xen/arch/arm/gic.c   |    2 ++
 xen/arch/arm/traps.c |    1 -
 3 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/entry.S b/xen/arch/arm/entry.S
index 7a22e2d..5bc3906 100644
--- a/xen/arch/arm/entry.S
+++ b/xen/arch/arm/entry.S
@@ -46,6 +46,17 @@ save_guest_regs:
 	ALIGN;												\
 trap_##trap:												\
 	SAVE_ALL;											\
+	cpsie i; 	/* local_irq_enable */								\
+	adr lr, return_from_trap;									\
+	mov r0, sp;											\
+	mov r11, sp;											\
+	bic sp, #7; /* Align the stack pointer (noop on guest trap) */					\
+	b do_trap_##trap
+
+#define DEFINE_TRAP_ENTRY_NOIRQ(trap)									\
+	ALIGN;												\
+trap_##trap:												\
+	SAVE_ALL;											\
 	adr lr, return_from_trap;									\
 	mov r0, sp;											\
 	mov r11, sp;											\
@@ -69,8 +80,8 @@ DEFINE_TRAP_ENTRY(supervisor_call)
 DEFINE_TRAP_ENTRY(prefetch_abort)
 DEFINE_TRAP_ENTRY(data_abort)
 DEFINE_TRAP_ENTRY(hypervisor)
-DEFINE_TRAP_ENTRY(irq)
-DEFINE_TRAP_ENTRY(fiq)
+DEFINE_TRAP_ENTRY_NOIRQ(irq)
+DEFINE_TRAP_ENTRY_NOIRQ(fiq)
 
 return_from_trap:
 	mov sp, r11
@@ -83,7 +94,7 @@ ENTRY(return_to_new_vcpu)
 ENTRY(return_to_guest)
 	mov r11, sp
 	bic sp, #7 /* Align the stack pointer */
-	bl leave_hypervisor_tail
+	bl leave_hypervisor_tail /* Disables interrupts on return */
 	mov sp, r11
 	RESTORE_ONE_BANKED(SP_usr)
 	/* LR_usr is the same physical register as lr and is restored below */
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index cc9d37b..1a2b95f 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -509,6 +509,8 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
     uint32_t intack = GICC[GICC_IAR];
     unsigned int irq = intack & GICC_IA_IRQ;
 
+    local_irq_enable();
+
     if ( irq == 1023 )
         /* Spurious interrupt */
         return;
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index abc26a3..5ed754f 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -412,7 +412,6 @@ static void do_debug_trap(struct cpu_user_regs *regs, unsigned int code)
 static void do_trap_hypercall(struct cpu_user_regs *regs, unsigned long iss)
 {
     arm_hypercall_t *call = NULL;
-    local_irq_enable();
 
     if ( iss != XEN_HYPERCALL_TAG )
     {
-- 
1.7.9.1

[PATCH 07/38] arm: hook up domctl and memory_op

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/traps.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 5ed754f..5d8b7f9 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -373,6 +373,8 @@ typedef unsigned long arm_hypercall_t(
     [ __HYPERVISOR_ ## x ] = (arm_hypercall_t *) do_ ## x
 
 static arm_hypercall_t *arm_hypercall_table[] = {
+    HYPERCALL(memory_op),
+    HYPERCALL(domctl),
     HYPERCALL(arch_0),
     HYPERCALL(sched_op),
     HYPERCALL(console_io),
-- 
1.7.9.1

[PATCH 08/38] arm: allocate and setup a guest vcpu.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c         |   68 +++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/dummy.S          |    3 --
 xen/include/public/arch-arm.h |    9 -----
 3 files changed, 68 insertions(+), 12 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 9339a11..62a2f3a 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -144,6 +144,17 @@ void free_vcpu_struct(struct vcpu *v)
     free_xenheap_page(v);
 }
 
+struct vcpu_guest_context *alloc_vcpu_guest_context(void)
+{
+    return xmalloc(struct vcpu_guest_context);
+
+}
+
+void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
+{
+    xfree(vgc);
+}
+
 int vcpu_initialise(struct vcpu *v)
 {
     int rc = 0;
@@ -182,6 +193,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
     if ( (rc = p2m_init(d)) != 0 )
         goto fail;
 
+    if ( (rc = domain_vgic_init(d)) != 0 )
+        goto fail;
+
     if ( !is_idle_domain(d) )
     {
         rc = -ENOMEM;
@@ -212,6 +226,60 @@ void arch_domain_destroy(struct domain *d)
     /* domain_vgic_destroy */
 }
 
+static int is_guest_psr(uint32_t psr)
+{
+    switch (psr & PSR_MODE_MASK)
+    {
+    case PSR_MODE_USR:
+    case PSR_MODE_FIQ:
+    case PSR_MODE_IRQ:
+    case PSR_MODE_SVC:
+    case PSR_MODE_ABT:
+    case PSR_MODE_UND:
+    case PSR_MODE_SYS:
+        return 1;
+    case PSR_MODE_MON:
+    case PSR_MODE_HYP:
+    default:
+        return 0;
+    }
+}
+
+int arch_set_info_guest(
+    struct vcpu *v, vcpu_guest_context_u c)
+{
+    struct cpu_user_regs *regs = &c.nat->user_regs;
+
+    if ( !is_guest_psr(regs->cpsr) )
+        return -EINVAL;
+
+    if ( regs->spsr_svc && !is_guest_psr(regs->spsr_svc) )
+        return -EINVAL;
+    if ( regs->spsr_abt && !is_guest_psr(regs->spsr_abt) )
+        return -EINVAL;
+    if ( regs->spsr_und && !is_guest_psr(regs->spsr_und) )
+        return -EINVAL;
+    if ( regs->spsr_irq && !is_guest_psr(regs->spsr_irq) )
+        return -EINVAL;
+    if ( regs->spsr_fiq && !is_guest_psr(regs->spsr_fiq) )
+        return -EINVAL;
+
+    v->arch.cpu_info->guest_cpu_user_regs = *regs;
+
+    /* XXX other state:
+     * - SCTLR
+     * - TTBR0/1
+     * - TTBCR
+     */
+
+    //if ( flags & VGCF_online )
+        clear_bit(_VPF_down, &v->pause_flags);
+    //else
+    //    set_bit(_VPF_down, &v->pause_flags);
+
+    return 0;
+}
+
 void arch_dump_domain_info(struct domain *d)
 {
 }
diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
index 016340c..3b48917 100644
--- a/xen/arch/arm/dummy.S
+++ b/xen/arch/arm/dummy.S
@@ -20,11 +20,8 @@ DUMMY(pirq_guest_unbind);
 DUMMY(pirq_set_affinity);
 
 /* VCPU */
-DUMMY(alloc_vcpu_guest_context);
 DUMMY(arch_get_info_guest);
-DUMMY(arch_set_info_guest);
 DUMMY(arch_vcpu_reset);
-DUMMY(free_vcpu_guest_context);
 DUMMY(sync_vcpu_execstate);
 NOP(update_vcpu_system_time);
 DUMMY(vcpu_show_execution_state);
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 1b1bcf3..e439727 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -124,15 +124,6 @@ typedef uint32_t xen_ulong_t;
 
 struct vcpu_guest_context {
     struct cpu_user_regs user_regs;         /* User-level CPU registers     */
-    union {
-        uint32_t reg[16];
-        struct {
-            uint32_t __pad[12];
-            uint32_t sp; /* r13 */
-            uint32_t lr; /* r14 */
-            uint32_t pc; /* r15 */
-        };
-    };
 };
 typedef struct vcpu_guest_context vcpu_guest_context_t;
 DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
-- 
1.7.9.1

[PATCH 09/38] arm: print domid as part of debug trap

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/traps.c |   11 ++++++-----
 1 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 5d8b7f9..40bb375 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -388,25 +388,26 @@ static arm_hypercall_t *arm_hypercall_table[] = {
 static void do_debug_trap(struct cpu_user_regs *regs, unsigned int code)
 {
     uint32_t reg, *r;
-
+    uint32_t domid = current->domain->domain_id;
     switch ( code ) {
     case 0xe0 ... 0xef:
         reg = code - 0xe0;
         r = &regs->r0 + reg;
-        printk("R%d = %#010"PRIx32" at %#010"PRIx32"\n", reg, *r, regs->pc);
+        printk("DOM%d: R%d = %#010"PRIx32" at %#010"PRIx32"\n",
+               domid, reg, *r, regs->pc);
         break;
     case 0xfd:
-        printk("Reached %08"PRIx32"\n", regs->pc);
+        printk("DOM%d: Reached %#010"PRIx32"\n", domid, regs->pc);
         break;
     case 0xfe:
         printk("%c", (char)(regs->r0 & 0xff));
         break;
     case 0xff:
-        printk("DEBUG\n");
+        printk("DOM%d: DEBUG\n", domid);
         show_execution_state(regs);
         break;
     default:
-        panic("Unhandled debug trap %#x\n", code);
+        panic("DOM%d: Unhandled debug trap %#x\n", domid, code);
         break;
     }
 }
-- 
1.7.9.1

[PATCH 10/38] arm: remove unnecessarily verbose print from p2m_load_VTTBR

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index fdbecbc..095e608 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -47,8 +47,6 @@ void p2m_load_VTTBR(struct domain *d)
 
     vttbr |= ((uint64_t)p2m->vmid&0xff)<<48;
 
-    printk("VTTBR dom%d = %"PRIx64"\n", d->domain_id, vttbr);
-
     WRITE_CP64(vttbr, VTTBR);
     isb(); /* Ensure update is visible */
 }
-- 
1.7.9.1

[PATCH 11/38] arm: implement p2m lookup

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c        |   71 ++++++++++++++++++++++++++++++++++++++++++--
 xen/include/asm-arm/p2m.h |    3 ++
 2 files changed, 70 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 095e608..9b40e93 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -10,10 +10,20 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
     struct p2m_domain *p2m = &d->arch.p2m;
     lpae_t *first = NULL, *second = NULL, *third = NULL;
 
-    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
+    printk("dom%d IPA %#"PRIpaddr"\n", d->domain_id, addr);
+
+    printk("P2M @ %p mfn:%#lx (%#03llx,%#03llx,%#03llx)\n",
+           p2m->first_level,
+           page_to_mfn(p2m->first_level),
+           first_table_offset(addr),
+           second_table_offset(addr),
+           third_table_offset(addr));
+
+    if ( first_table_offset(addr) >= LPAE_ENTRIES )
+        goto done;
 
     first = __map_domain_page(p2m->first_level);
-    printk("1ST[%#03llx] = %#016llx\n",
+    printk("1ST[%#03llx] = %#"PRIpaddr"\n",
            first_table_offset(addr),
            first[first_table_offset(addr)].bits);
     if ( !first[first_table_offset(addr)].p2m.valid ||
@@ -21,7 +31,7 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
         goto done;
 
     second = map_domain_page(first[first_table_offset(addr)].p2m.base);
-    printk("2ND[%#03llx] = %#016llx\n",
+    printk("2ND[%#03llx] = %#"PRIpaddr"\n",
            second_table_offset(addr),
            second[second_table_offset(addr)].bits);
     if ( !second[second_table_offset(addr)].p2m.valid ||
@@ -29,7 +39,7 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
         goto done;
 
     third = map_domain_page(second[second_table_offset(addr)].p2m.base);
-    printk("3RD[%#03llx] = %#016llx\n",
+    printk("3RD[%#03llx] = %#"PRIpaddr"\n",
            third_table_offset(addr),
            third[third_table_offset(addr)].bits);
 
@@ -51,6 +61,59 @@ void p2m_load_VTTBR(struct domain *d)
     isb(); /* Ensure update is visible */
 }
 
+/*
+ * Lookup the MFN corresponding to a domain's PFN.
+ *
+ * There are no processor functions to do a stage 2 only lookup therefore we
+ * do a a software walk.
+ */
+paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
+{
+    struct p2m_domain *p2m = &d->arch.p2m;
+    lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
+    paddr_t maddr = INVALID_PADDR;
+
+    spin_lock(&p2m->lock);
+
+    first = __map_domain_page(p2m->first_level);
+    if ( !first[first_table_offset(paddr)].p2m.valid )
+        goto done_err;
+    if ( !first[first_table_offset(paddr)].p2m.table )
+    {
+        pte = first[first_table_offset(paddr)];
+        goto done;
+    }
+
+    second = map_domain_page(first[first_table_offset(paddr)].p2m.base);
+    if ( !second[second_table_offset(paddr)].p2m.valid )
+        goto done_err;
+    if ( !second[second_table_offset(paddr)].p2m.table )
+    {
+        pte = second[second_table_offset(paddr)];
+        goto done;
+    }
+
+    third = map_domain_page(second[second_table_offset(paddr)].p2m.base);
+    if ( !third[third_table_offset(paddr)].p2m.valid )
+        goto done_err;
+    if ( !third[third_table_offset(paddr)].p2m.table )
+        goto done_err;
+
+    pte = third[third_table_offset(paddr)];
+
+done:
+
+    maddr = (pte.bits & PADDR_MASK & PAGE_MASK) | (paddr & ~PAGE_MASK);
+done_err:
+    if (third) unmap_domain_page(third);
+    if (second) unmap_domain_page(second);
+    if (first) unmap_domain_page(first);
+
+    spin_unlock(&p2m->lock);
+
+    return maddr;
+}
+
 int guest_physmap_mark_populate_on_demand(struct domain *d,
                                           unsigned long gfn,
                                           unsigned int order)
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 349923a..1afd5cb 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -32,6 +32,9 @@ int p2m_alloc_table(struct domain *d);
 /* */
 void p2m_load_VTTBR(struct domain *d);
 
+/* */
+paddr_t p2m_lookup(struct domain *d, paddr_t gpfn);
+
 /* Setup p2m RAM mapping for domain d from start-end. */
 int p2m_populate_ram(struct domain *d, paddr_t start, paddr_t end);
 /* Map MMIO regions in the p2m: start_gaddr and end_gaddr is the range
-- 
1.7.9.1

[PATCH 12/38] arm: remove hard tabs from init_idle_domain

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/setup.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index 0df3c1a..81ababb 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -47,9 +47,9 @@ static __attribute_used__ void init_done(void)
 
 static void __init init_idle_domain(void)
 {
-        scheduler_init();
-        set_current(idle_vcpu[0]);
-        /* TODO: setup_idle_pagetable(); */
+    scheduler_init();
+    set_current(idle_vcpu[0]);
+    /* TODO: setup_idle_pagetable(); */
 }
 
 static void __init processor_id(void)
-- 
1.7.9.1

[PATCH 13/38] arm: stub out sync_vcpu_execstate

[Ian Campbell]

We don't do lazy exec state switching so there isn't actually anything to do.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c |    5 +++++
 xen/arch/arm/dummy.S  |    1 -
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 62a2f3a..bd900f9 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -96,6 +96,11 @@ void sync_local_execstate(void)
     /* Nothing to do -- no lazy switching */
 }
 
+void sync_vcpu_execstate(struct vcpu *v)
+{
+    /* Nothing to do -- no lazy switching */
+}
+
 void startup_cpu_idle_loop(void)
 {
     struct vcpu *v = current;
diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
index 3b48917..8eddd15 100644
--- a/xen/arch/arm/dummy.S
+++ b/xen/arch/arm/dummy.S
@@ -22,7 +22,6 @@ DUMMY(pirq_set_affinity);
 /* VCPU */
 DUMMY(arch_get_info_guest);
 DUMMY(arch_vcpu_reset);
-DUMMY(sync_vcpu_execstate);
 NOP(update_vcpu_system_time);
 DUMMY(vcpu_show_execution_state);
 
-- 
1.7.9.1

[PATCH 14/38] arm: do not set max_vcpus = 8 in arch_domain_create.

[Ian Campbell]

XEN_DOMCTL_max_vcpus cannot reduce max_vcpus and therefore we can't create a
smaller guest.

The limit of 8 (due to GIC limits) should be expressed elsewhere, likely in
MAX_VIRT_CPUS -- but making that change caused:
    (XEN) Unexpected Trap: Data Abort
    (XEN) ----[ Xen-4.2-unstable  x86_64  debug=y  Not tainted ]----
    (XEN) CPU:    0
    (XEN) PC:     00222e48 _spin_lock+0x28/0x6c
    (XEN) CPSR:   600001da MODE:HYP
    (XEN)      R0: 002c4389 R1: 800001da R2: 00000001 R3: 0000ffff
    (XEN)      R4: 002c4381 R5: 00000080 R6: 002c4380 R7: 002c4000
    (XEN)      R8: 002c4380 R9: 4000015a R10:00000080 R11:40017d6c R12:00000000
    (XEN)      SP: 40017d5c LR: 00222e34
    (XEN)
    [...]
    (XEN) Xen call trace:
    (XEN)    [<00222e48>] _spin_lock+0x28/0x6c
    (XEN)    [<0022623c>] init_timer+0xbc/0x160
    (XEN)    [<0021fbdc>] sched_init_vcpu+0x94/0x200
    (XEN)    [<002061a4>] alloc_vcpu+0x124/0x210
    (XEN)    [<00204890>] do_domctl+0xaa4/0x14e4
    (XEN)    [<00241ab8>] do_trap_hypervisor+0x588/0x8cc
    (XEN)    [<0023bbb0>] return_from_trap+0x0/0x4

so punt on that for now.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index bd900f9..e867cb2 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -215,8 +215,6 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
             goto fail;
     }
 
-    d->max_vcpus = 8;
-
     if ( (rc = domain_vgic_init(d)) != 0 )
         goto fail;
 
-- 
1.7.9.1

[PATCH 15/38] arm: implement stub version of flush_tlb_mask.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/dummy.S |    1 -
 xen/arch/arm/smp.c   |    9 +++++++++
 2 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
index 8eddd15..c001e8d 100644
--- a/xen/arch/arm/dummy.S
+++ b/xen/arch/arm/dummy.S
@@ -48,7 +48,6 @@ DUMMY(domain_get_maximum_gpfn);
 DUMMY(domain_relinquish_resources);
 DUMMY(domain_set_time_offset);
 DUMMY(dom_cow);
-DUMMY(flush_tlb_mask);
 DUMMY(gmfn_to_mfn);
 DUMMY(hypercall_create_continuation);
 DUMMY(send_timer_event);
diff --git a/xen/arch/arm/smp.c b/xen/arch/arm/smp.c
index cad84f5..824c8c8 100644
--- a/xen/arch/arm/smp.c
+++ b/xen/arch/arm/smp.c
@@ -1,5 +1,14 @@
 #include <xen/config.h>
+#include <asm/system.h>
 #include <asm/smp.h>
+#include <asm/cpregs.h>
+#include <asm/page.h>
+
+void flush_tlb_mask(const cpumask_t *mask)
+{
+    /* XXX IPI other processors */
+    flush_xen_data_tlb();
+}
 
 void smp_call_function(
     void (*func) (void *info),
-- 
1.7.9.1

[PATCH 16/38] arm: Add simple cpu_{sibling,core}_mask

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/dummy.S   |    2 --
 xen/arch/arm/setup.c   |    7 +++++++
 xen/arch/arm/smpboot.c |    5 +++++
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
index c001e8d..03f7489 100644
--- a/xen/arch/arm/dummy.S
+++ b/xen/arch/arm/dummy.S
@@ -7,8 +7,6 @@ x:	.word 0xe7f000f0 /* Undefined instruction */
 x:	mov pc, lr
 	
 /* SMP support */
-DUMMY(per_cpu__cpu_core_mask);
-DUMMY(per_cpu__cpu_sibling_mask);
 DUMMY(node_online_map);
 DUMMY(smp_send_state_dump);
 
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index 81ababb..b0cfacc 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -230,6 +230,13 @@ void __init start_xen(unsigned long boot_phys_offset,
         }
     }
 
+    if ( !zalloc_cpumask_var(&per_cpu(cpu_sibling_mask, 0)) ||
+         !zalloc_cpumask_var(&per_cpu(cpu_core_mask, 0)) )
+        BUG();
+
+    cpumask_clear(per_cpu(cpu_sibling_mask, 0));
+    cpumask_clear(per_cpu(cpu_core_mask, 0));
+
     printk("Brought up %ld CPUs\n", (long)num_online_cpus());
     /* TODO: smp_cpus_done(); */
 
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index ea05afc..8517d86 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -52,6 +52,11 @@ unsigned long __initdata ready_cpus = 0;
 
 /* ID of the PCPU we're running on */
 DEFINE_PER_CPU(unsigned int, cpu_id);
+/* XXX these seem awefully x86ish... */
+/* representing HT siblings of each logical CPU */
+DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask);
+/* representing HT and core siblings of each logical CPU */
+DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
 
 void __init
 smp_prepare_cpus (unsigned int max_cpus)
-- 
1.7.9.1

[PATCH 17/38] arm: allow p2m to be created with specific MATTR.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c         |   15 ++++++++-------
 xen/include/asm-arm/page.h |    6 ++++--
 2 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 9b40e93..46c6f17 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -148,7 +148,7 @@ static int p2m_create_entry(struct domain *d,
     clear_page(p);
     unmap_domain_page(p);
 
-    pte = mfn_to_p2m_entry(page_to_mfn(page));
+    pte = mfn_to_p2m_entry(page_to_mfn(page), MATTR_MEM);
 
     write_pte(entry, pte);
 
@@ -159,7 +159,8 @@ static int create_p2m_entries(struct domain *d,
                      int alloc,
                      paddr_t start_gpaddr,
                      paddr_t end_gpaddr,
-                     paddr_t maddr)
+                     paddr_t maddr,
+                     int mattr)
 {
     int rc;
     struct p2m_domain *p2m = &d->arch.p2m;
@@ -235,11 +236,11 @@ static int create_p2m_entries(struct domain *d,
                 goto out;
             }
 
-            pte = mfn_to_p2m_entry(page_to_mfn(page));
+            pte = mfn_to_p2m_entry(page_to_mfn(page), mattr);
 
             write_pte(&third[third_table_offset(addr)], pte);
         } else {
-            lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT);
+            lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT, mattr);
             write_pte(&third[third_table_offset(addr)], pte);
             maddr += PAGE_SIZE;
         }
@@ -263,7 +264,7 @@ int p2m_populate_ram(struct domain *d,
                      paddr_t start,
                      paddr_t end)
 {
-    return create_p2m_entries(d, 1, start, end, 0);
+    return create_p2m_entries(d, 1, start, end, 0, MATTR_MEM);
 }
 
 int map_mmio_regions(struct domain *d,
@@ -271,7 +272,7 @@ int map_mmio_regions(struct domain *d,
                      paddr_t end_gaddr,
                      paddr_t maddr)
 {
-    return create_p2m_entries(d, 0, start_gaddr, end_gaddr, maddr);
+    return create_p2m_entries(d, 0, start_gaddr, end_gaddr, maddr, MATTR_DEV);
 }
 
 int guest_physmap_add_page(struct domain *d,
@@ -281,7 +282,7 @@ int guest_physmap_add_page(struct domain *d,
 {
     return create_p2m_entries(d, 0, gpfn << PAGE_SHIFT,
                               (gpfn + (1<<page_order)) << PAGE_SHIFT,
-                              mfn << PAGE_SHIFT);
+                              mfn << PAGE_SHIFT, MATTR_MEM);
 }
 
 void guest_physmap_remove_page(struct domain *d,
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index c7b6530..bb1729a 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -46,6 +46,8 @@
 #define DEV_WC        BUFFERABLE
 #define DEV_CACHED    WRITEBACK
 
+#define MATTR_DEV     0x1
+#define MATTR_MEM     0xf
 
 #ifndef __ASSEMBLY__
 
@@ -169,7 +171,7 @@ static inline lpae_t mfn_to_xen_entry(unsigned long mfn)
     return e;
 }
 
-static inline lpae_t mfn_to_p2m_entry(unsigned long mfn)
+static inline lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
 {
     paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
     lpae_t e = (lpae_t) {
@@ -178,7 +180,7 @@ static inline lpae_t mfn_to_p2m_entry(unsigned long mfn)
         .p2m.sh = LPAE_SH_OUTER,
         .p2m.write = 1,
         .p2m.read = 1,
-        .p2m.mattr = 0xf,
+        .p2m.mattr = mattr,
         .p2m.table = 1,
         .p2m.valid = 1,
     };
-- 
1.7.9.1

[PATCH 18/38] arm: implement vpl011 (UART) emulator.

[Ian Campbell]

This is not interended to provide a full emulation, but rather just enough to
satisfy the use made by Linux' boot time decompressor code (which is too early
for DT etc)

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/Makefile        |    1 +
 xen/arch/arm/domain.c        |    4 +
 xen/arch/arm/io.c            |    1 +
 xen/arch/arm/io.h            |    1 +
 xen/arch/arm/vpl011.c        |  155 ++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vpl011.h        |   34 +++++++++
 xen/include/asm-arm/domain.h |    8 ++
 7 files changed, 204 insertions(+), 0 deletions(-)
 create mode 100644 xen/arch/arm/vpl011.c
 create mode 100644 xen/arch/arm/vpl011.h

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 9440a21..5a87ba6 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -25,6 +25,7 @@ obj-y += shutdown.o
 obj-y += traps.o
 obj-y += vgic.o
 obj-y += vtimer.o
+obj-y += vpl011.o
 
 #obj-bin-y += ....o
 
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index e867cb2..d830980 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -13,6 +13,7 @@
 
 #include "gic.h"
 #include "vtimer.h"
+#include "vpl011.h"
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
@@ -201,6 +202,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
     if ( (rc = domain_vgic_init(d)) != 0 )
         goto fail;
 
+    if ( (rc = domain_uart0_init(d)) != 0 )
+        goto fail;
+
     if ( !is_idle_domain(d) )
     {
         rc = -ENOMEM;
diff --git a/xen/arch/arm/io.c b/xen/arch/arm/io.c
index 4461225..18f6164 100644
--- a/xen/arch/arm/io.c
+++ b/xen/arch/arm/io.c
@@ -25,6 +25,7 @@
 static const struct mmio_handler *const mmio_handlers[] =
 {
     &vgic_distr_mmio_handler,
+    &uart0_mmio_handler,
 };
 #define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
 
diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
index 8cc5ca7..9a507f5 100644
--- a/xen/arch/arm/io.h
+++ b/xen/arch/arm/io.h
@@ -40,6 +40,7 @@ struct mmio_handler {
 };
 
 extern const struct mmio_handler vgic_distr_mmio_handler;
+extern const struct mmio_handler uart0_mmio_handler;
 
 extern int handle_mmio(mmio_info_t *info);
 
diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
new file mode 100644
index 0000000..1491dcc
--- /dev/null
+++ b/xen/arch/arm/vpl011.c
@@ -0,0 +1,155 @@
+/*
+ * xen/arch/arm/vpl011.c
+ *
+ * ARM PL011 UART Emulator (DEBUG)
+ *
+ * Ian Campbell <ian.campbell@citrix.com>
+ * Copyright (c) 2012 Citrix Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/*
+ * This is not intended to be a full emulation of a PL011
+ * device. Rather it is intended to provide a sufficient veneer of one
+ * that early code (such as Linux's boot time decompressor) which
+ * hardcodes output directly to such a device are able to make progress.
+ *
+ * This device is not intended to be enumerable or exposed to the OS
+ * (e.g. via Device Tree).
+ */
+
+#include <xen/config.h>
+#include <xen/lib.h>
+#include <xen/sched.h>
+#include <xen/errno.h>
+#include <xen/ctype.h>
+
+#include "io.h"
+
+#define UART0_BASE_ADDRESS 0x1c090000
+
+#define UARTDR 0x000
+#define UARTFR 0x018
+
+int domain_uart0_init(struct domain *d)
+{
+    int rc;
+    if ( d->domain_id == 0 )
+        return 0;
+
+    spin_lock_init(&d->arch.uart0.lock);
+    d->arch.uart0.idx = 0;
+
+    rc = -ENOMEM;
+    d->arch.uart0.buf = xzalloc_array(char, VPL011_BUF_SIZE);
+    if ( !d->arch.uart0.buf )
+        goto out;
+
+    rc = 0;
+out:
+    return rc;
+}
+
+static void uart0_print_char(char c)
+{
+    struct vpl011 *uart = &current->domain->arch.uart0;
+
+    /* Accept only printable characters, newline, and horizontal tab. */
+    if ( !isprint(c) && (c != '\n') && (c != '\t') )
+        return ;
+
+    spin_lock(&uart->lock);
+    uart->buf[uart->idx++] = c;
+    if ( (uart->idx == (VPL011_BUF_SIZE - 2)) || (c == '\n') )
+    {
+        if ( c != '\n' )
+            uart->buf[uart->idx++] = '\n';
+        uart->buf[uart->idx] = '\0';
+        printk(XENLOG_G_DEBUG "DOM%u: %s",
+               current->domain->domain_id, uart->buf);
+        uart->idx = 0;
+    }
+    spin_unlock(&uart->lock);
+}
+
+static int uart0_mmio_check(struct vcpu *v, paddr_t addr)
+{
+    return v->domain->domain_id && addr >= UART0_BASE_ADDRESS && addr < (UART0_BASE_ADDRESS+65536);
+}
+
+static int uart0_mmio_read(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    uint32_t *r = &regs->r0 + dabt.reg;
+    int offset = (int)(info->gpa - UART0_BASE_ADDRESS);
+
+    switch ( offset )
+    {
+    case UARTDR:
+        *r = 0;
+        return 1;
+    case UARTFR:
+        *r = 0x87; /* All holding registers empty, ready to send etc */
+        return 1;
+    default:
+        printk("VPL011: unhandled read r%d offset %#08x\n",
+               dabt.reg, offset);
+        domain_crash_synchronous();
+    }
+}
+
+static int uart0_mmio_write(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    uint32_t *r = &regs->r0 + dabt.reg;
+    int offset = (int)(info->gpa - UART0_BASE_ADDRESS);
+    int val = (int)((*r) & 0xFF);
+
+    if ( dabt.size != 0 )
+    {
+        printk("VPL011: Invalid %d-byte write to offset %#x\n",
+               1<<dabt.size, offset);
+        domain_crash_synchronous();
+    }
+
+    switch ( offset )
+    {
+    case UARTDR:
+        uart0_print_char(val);
+        return 1;
+    case UARTFR:
+        /* Silently ignore */
+        return 1;
+    default:
+        printk("VPL011: unhandled write r%d=%"PRIx32" offset %#08x\n",
+               dabt.reg, *r, offset);
+        domain_crash_synchronous();
+    }
+}
+
+const struct mmio_handler uart0_mmio_handler = {
+    .check_handler = uart0_mmio_check,
+    .read_handler  = uart0_mmio_read,
+    .write_handler = uart0_mmio_write,
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
diff --git a/xen/arch/arm/vpl011.h b/xen/arch/arm/vpl011.h
new file mode 100644
index 0000000..952d812
--- /dev/null
+++ b/xen/arch/arm/vpl011.h
@@ -0,0 +1,34 @@
+/*
+ * xen/arch/arm/vpl011.h
+ *
+ * ARM PL011 Emulation Support
+ *
+ * Ian Campbell <ian.campbell@citrix.com>
+ * Copyright (c) 2012 Citrix Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPL011_H__
+#define __ARCH_ARM_VPL011_H__
+
+extern int domain_uart0_init(struct domain *d);
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 10ed540..f295a82 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -48,6 +48,14 @@ struct arch_domain
         struct vgic_irq_rank *shared_irqs;
         struct pending_irq *pending_irqs;
     } vgic;
+
+    struct vpl011 {
+#define VPL011_BUF_SIZE 128
+        char                  *buf;
+        int                    idx;
+        spinlock_t             lock;
+    } uart0;
+
 }  __cacheline_aligned;
 
 struct arch_vcpu
-- 
1.7.9.1

[PATCH 19/38] arm: context switch a bunch of guest state.

[Ian Campbell]

I haven't investigated what if any of this could be done lazily.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c         |  122 ++++++++++++++++++++++++++++++++++++++++-
 xen/arch/arm/gic.c            |   25 ++++++++-
 xen/arch/arm/gic.h            |    9 ++-
 xen/include/asm-arm/cpregs.h  |   29 +++++++++-
 xen/include/asm-arm/domain.h  |   33 ++++++++++-
 xen/include/public/arch-arm.h |    3 +
 6 files changed, 210 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index d830980..a7fb227 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -36,12 +36,124 @@ void idle_loop(void)
 
 static void ctxt_switch_from(struct vcpu *p)
 {
+    /* CP 15 */
+    p->arch.csselr = READ_CP32(CSSELR);
+
+    /* Control Registers */
+    p->arch.actlr = READ_CP32(ACTLR);
+    p->arch.sctlr = READ_CP32(SCTLR);
+    p->arch.cpacr = READ_CP32(CPACR);
+
+    p->arch.contextidr = READ_CP32(CONTEXTIDR);
+    p->arch.tpidrurw = READ_CP32(TPIDRURW);
+    p->arch.tpidruro = READ_CP32(TPIDRURO);
+    p->arch.tpidrprw = READ_CP32(TPIDRPRW);
+
+    /* XXX only save these if ThumbEE e.g. ID_PFR0.THUMB_EE_SUPPORT */
+    p->arch.teecr = READ_CP32(TEECR);
+    p->arch.teehbr = READ_CP32(TEEHBR);
+
+    p->arch.joscr = READ_CP32(JOSCR);
+    p->arch.jmcr = READ_CP32(JMCR);
+
+    isb();
+
+    /* MMU */
+    p->arch.vbar = READ_CP32(VBAR);
+    p->arch.ttbcr = READ_CP32(TTBCR);
+    /* XXX save 64 bit TTBR if guest is LPAE */
+    p->arch.ttbr0 = READ_CP32(TTBR0);
+    p->arch.ttbr1 = READ_CP32(TTBR1);
+
+    p->arch.dacr = READ_CP32(DACR);
+    p->arch.par = READ_CP64(PAR);
+    p->arch.mair0 = READ_CP32(MAIR0);
+    p->arch.mair1 = READ_CP32(MAIR1);
+
+    /* Fault Status */
+    p->arch.dfar = READ_CP32(DFAR);
+    p->arch.ifar = READ_CP32(IFAR);
+    p->arch.dfsr = READ_CP32(DFSR);
+    p->arch.ifsr = READ_CP32(IFSR);
+    p->arch.adfsr = READ_CP32(ADFSR);
+    p->arch.aifsr = READ_CP32(AIFSR);
+
+    /* XXX MPU */
+
+    /* XXX VFP */
+
+    /* XXX VGIC */
+    gic_save_state(p);
+
+    isb();
     context_saved(p);
 }
 
 static void ctxt_switch_to(struct vcpu *n)
 {
+    uint32_t hcr;
+
+    hcr = READ_CP32(HCR);
+    WRITE_CP32(hcr & ~HCR_VM, HCR);
+    isb();
+
     p2m_load_VTTBR(n->domain);
+    isb();
+
+    /* XXX VGIC */
+    gic_restore_state(n);
+
+    /* XXX VFP */
+
+    /* XXX MPU */
+
+    /* Fault Status */
+    WRITE_CP32(n->arch.dfar, DFAR);
+    WRITE_CP32(n->arch.ifar, IFAR);
+    WRITE_CP32(n->arch.dfsr, DFSR);
+    WRITE_CP32(n->arch.ifsr, IFSR);
+    WRITE_CP32(n->arch.adfsr, ADFSR);
+    WRITE_CP32(n->arch.aifsr, AIFSR);
+
+    /* MMU */
+    WRITE_CP32(n->arch.vbar, VBAR);
+    WRITE_CP32(n->arch.ttbcr, TTBCR);
+    /* XXX restore 64 bit TTBR if guest is LPAE */
+    WRITE_CP32(n->arch.ttbr0, TTBR0);
+    WRITE_CP32(n->arch.ttbr1, TTBR1);
+
+    WRITE_CP32(n->arch.dacr, DACR);
+    WRITE_CP64(n->arch.par, PAR);
+    WRITE_CP32(n->arch.mair0, MAIR0);
+    WRITE_CP32(n->arch.mair1, MAIR1);
+    isb();
+
+    /* Control Registers */
+    WRITE_CP32(n->arch.actlr, ACTLR);
+    WRITE_CP32(n->arch.sctlr, SCTLR);
+    WRITE_CP32(n->arch.cpacr, CPACR);
+
+    WRITE_CP32(n->arch.contextidr, CONTEXTIDR);
+    WRITE_CP32(n->arch.tpidrurw, TPIDRURW);
+    WRITE_CP32(n->arch.tpidruro, TPIDRURO);
+    WRITE_CP32(n->arch.tpidrprw, TPIDRPRW);
+
+    /* XXX only restore these if ThumbEE e.g. ID_PFR0.THUMB_EE_SUPPORT */
+    WRITE_CP32(n->arch.teecr, TEECR);
+    WRITE_CP32(n->arch.teehbr, TEEHBR);
+
+    WRITE_CP32(n->arch.joscr, JOSCR);
+    WRITE_CP32(n->arch.jmcr, JMCR);
+
+    isb();
+
+    /* CP 15 */
+    WRITE_CP32(n->arch.csselr, CSSELR);
+
+    isb();
+
+    WRITE_CP32(hcr, HCR);
+    isb();
 }
 
 static void schedule_tail(struct vcpu *prev)
@@ -255,6 +367,7 @@ static int is_guest_psr(uint32_t psr)
 int arch_set_info_guest(
     struct vcpu *v, vcpu_guest_context_u c)
 {
+    struct vcpu_guest_context *ctxt = c.nat;
     struct cpu_user_regs *regs = &c.nat->user_regs;
 
     if ( !is_guest_psr(regs->cpsr) )
@@ -273,10 +386,13 @@ int arch_set_info_guest(
 
     v->arch.cpu_info->guest_cpu_user_regs = *regs;
 
+    v->arch.sctlr = ctxt->sctlr;
+    v->arch.ttbr0 = ctxt->ttbr0;
+    v->arch.ttbr1 = ctxt->ttbr1;
+    v->arch.ttbcr = ctxt->ttbcr;
+
     /* XXX other state:
-     * - SCTLR
-     * - TTBR0/1
-     * - TTBCR
+     * -
      */
 
     //if ( flags & VGCF_online )
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 1a2b95f..339c327 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -61,6 +61,30 @@ static struct {
 irq_desc_t irq_desc[NR_IRQS];
 unsigned nr_lrs;
 
+void gic_save_state(struct vcpu *v)
+{
+    int i;
+
+    for ( i=0; i<nr_lrs; i++)
+        v->arch.gic_lr[i] = GICH[GICH_LR + i];
+    /* Disable until next VCPU scheduled */
+    GICH[GICH_HCR] = 0;
+    isb();
+}
+
+void gic_restore_state(struct vcpu *v)
+{
+    int i;
+
+    if ( is_idle_vcpu(v) )
+        return;
+
+    for ( i=0; i<nr_lrs; i++)
+        GICH[GICH_LR + i] = v->arch.gic_lr[i];
+    GICH[GICH_HCR] = GICH_HCR_EN;
+    isb();
+}
+
 static unsigned int gic_irq_startup(struct irq_desc *desc)
 {
     uint32_t enabler;
@@ -263,7 +287,6 @@ static void __cpuinit gic_hyp_init(void)
     vtr = GICH[GICH_VTR];
     nr_lrs  = (vtr & GICH_VTR_NRLRGS) + 1;
 
-    GICH[GICH_HCR] = GICH_HCR_EN;
     GICH[GICH_MISR] = GICH_MISR_EOI;
 }
 
diff --git a/xen/arch/arm/gic.h b/xen/arch/arm/gic.h
index ff8d0a2..ac9cf3a 100644
--- a/xen/arch/arm/gic.h
+++ b/xen/arch/arm/gic.h
@@ -70,8 +70,8 @@
 #define GICH_MISR       (0x10/4)
 #define GICH_EISR0      (0x20/4)
 #define GICH_EISR1      (0x24/4)
-#define GICH_ELRSR0     (0x30/4)
-#define GICH_ELRSR1     (0x34/4)
+#define GICH_ELSR0      (0x30/4)
+#define GICH_ELSR1      (0x34/4)
 #define GICH_APR        (0xF0/4)
 #define GICH_LR         (0x100/4)
 
@@ -149,6 +149,11 @@ extern void gic_init_secondary_cpu(void);
 extern void gic_disable_cpu(void);
 /* setup the gic virtual interface for a guest */
 extern void gicv_setup(struct domain *d);
+
+/* Context switch */
+extern void gic_save_state(struct vcpu *v);
+extern void gic_restore_state(struct vcpu *v);
+
 #endif
 
 /*
diff --git a/xen/include/asm-arm/cpregs.h b/xen/include/asm-arm/cpregs.h
index 7a0b49a..bd46942 100644
--- a/xen/include/asm-arm/cpregs.h
+++ b/xen/include/asm-arm/cpregs.h
@@ -88,6 +88,19 @@
  * arguments, which are cp,opc1,crn,crm,opc2.
  */
 
+/* Coprocessor 14 */
+
+/* CP14 CR0: */
+#define TEECR           p14,6,c0,c0,0   /* ThumbEE Configuration Register */
+
+/* CP14 CR1: */
+#define TEEHBR          p14,6,c1,c0,0   /* ThumbEE Handler Base Register */
+#define JOSCR           p14,7,c1,c0,0   /* Jazelle OS Control Register */
+
+/* CP14 CR2: */
+#define JMCR            p14,7,c2,c0,0   /* Jazelle Main Configuration Register */
+
+
 /* Coprocessor 15 */
 
 /* CP15 CR0: CPUID and Cache Type Registers */
@@ -112,6 +125,8 @@
 
 /* CP15 CR1: System Control Registers */
 #define SCTLR           p15,0,c1,c0,0   /* System Control Register */
+#define ACTLR           p15,0,c1,c0,1   /* Auxiliary Control Register */
+#define CPACR           p15,0,c1,c0,2   /* Coprocessor Access Control Register */
 #define SCR             p15,0,c1,c1,0   /* Secure Configuration Register */
 #define NSACR           p15,0,c1,c1,2   /* Non-Secure Access Control Register */
 #define HSCTLR          p15,4,c1,c0,0   /* Hyp. System Control Register */
@@ -127,12 +142,15 @@
 #define VTTBR           p15,6,c2        /* Virtualization Translation Table Base Register */
 
 /* CP15 CR3: Domain Access Control Register */
+#define DACR            p15,0,c3,c0,0   /* Domain Access Control Register */
 
 /* CP15 CR4: */
 
 /* CP15 CR5: Fault Status Registers */
 #define DFSR            p15,0,c5,c0,0   /* Data Fault Status Register */
 #define IFSR            p15,0,c5,c0,1   /* Instruction Fault Status Register */
+#define ADFSR           p15,0,c5,c1,0   /* Auxiliary Data Fault Status Register */
+#define AIFSR           p15,0,c5,c1,1   /* Auxiliary Instruction Fault Status Register */
 #define HSR             p15,4,c5,c2,0   /* Hyp. Syndrome Register */
 
 /* CP15 CR6: Fault Address Registers */
@@ -144,6 +162,7 @@
 
 /* CP15 CR7: Cache and address translation operations */
 #define PAR             p15,0,c7        /* Physical Address Register */
+
 #define ICIALLUIS       p15,0,c7,c1,0   /* Invalidate all instruction caches to PoU inner shareable */
 #define BPIALLIS        p15,0,c7,c1,6   /* Invalidate entire branch predictor array inner shareable */
 #define ICIALLU         p15,0,c7,c5,0   /* Invalidate all instruction caches to PoU */
@@ -192,20 +211,24 @@
 /* CP15 CR9: */
 
 /* CP15 CR10: */
-#define MAIR0           p15,0,c10,c2,0  /* Memory Attribute Indirection Register 0 */
-#define MAIR1           p15,0,c10,c2,1  /* Memory Attribute Indirection Register 1 */
+#define MAIR0           p15,0,c10,c2,0  /* Memory Attribute Indirection Register 0 AKA PRRR */
+#define MAIR1           p15,0,c10,c2,1  /* Memory Attribute Indirection Register 1 AKA NMRR */
 #define HMAIR0          p15,4,c10,c2,0  /* Hyp. Memory Attribute Indirection Register 0 */
 #define HMAIR1          p15,4,c10,c2,1  /* Hyp. Memory Attribute Indirection Register 1 */
 
 /* CP15 CR11: DMA Operations for TCM Access */
 
 /* CP15 CR12:  */
+#define VBAR            p15,0,c12,c0,0  /* Vector Base Address Register */
 #define HVBAR           p15,4,c12,c0,0  /* Hyp. Vector Base Address Register */
 
 /* CP15 CR13:  */
 #define FCSEIDR         p15,0,c13,c0,0  /* FCSE Process ID Register */
 #define CONTEXTIDR      p15,0,c13,c0,1  /* Context ID Register */
-#define HTPIDR          p15,4,c13,c0,2  /* Hyp. Software Thread ID Register */
+#define TPIDRURW        p15,0,c13,c0,2  /* Software Thread ID, User, R/W */
+#define TPIDRURO        p15,0,c13,c0,3  /* Software Thread ID, User, R/O */
+#define TPIDRPRW        p15,0,c13,c0,4  /* Software Thread ID, Priveleged */
+#define HTPIDR          p15,4,c13,c0,2  /* HYp Software Thread Id Register */
 
 /* CP15 CR14:  */
 #define CNTPCT          p15,0,c14       /* Time counter value */
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index f295a82..620b26e 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -81,8 +81,37 @@ struct arch_vcpu
      */
     struct cpu_info *cpu_info;
 
-    uint32_t sctlr;
-    uint32_t ttbr0, ttbr1, ttbcr;
+    /* Fault Status */
+    uint32_t dfar, ifar;
+    uint32_t dfsr, ifsr;
+    uint32_t adfsr, aifsr;
+
+    /* MMU */
+    uint32_t vbar;
+    uint32_t ttbcr;
+    uint32_t ttbr0, ttbr1;
+
+    uint32_t dacr;
+    uint64_t par;
+    uint32_t mair0, mair1;
+
+    /* Control Registers */
+    uint32_t actlr, sctlr;
+    uint32_t cpacr;
+
+    uint32_t contextidr;
+    uint32_t tpidrurw;
+    uint32_t tpidruro;
+    uint32_t tpidrprw;
+
+    uint32_t teecr, teehbr;
+    uint32_t joscr, jmcr;
+
+    /* CP 15 */
+    uint32_t csselr;
+
+    uint32_t gic_hcr, gic_vmcr, gic_apr;
+    uint32_t gic_lr[64];
 
     struct {
         struct vgic_irq_rank private_irqs;
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index e439727..e915cbf 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -124,6 +124,9 @@ typedef uint32_t xen_ulong_t;
 
 struct vcpu_guest_context {
     struct cpu_user_regs user_regs;         /* User-level CPU registers     */
+
+    uint32_t sctlr;
+    uint32_t ttbr0, ttbr1, ttbcr;
 };
 typedef struct vcpu_guest_context vcpu_guest_context_t;
 DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
-- 
1.7.9.1

[PATCH 20/38] arm: dump a page table walk when va_to_par fails.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/include/asm-arm/page.h |   12 ++++++++----
 1 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index bb1729a..f36bf6f 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -254,6 +254,9 @@ static inline void flush_guest_tlb(void)
     WRITE_CP32(r0 /* dummy */, TLBIALLNSNH);
 }
 
+extern void dump_pt_walk(uint32_t addr);
+extern void dump_p2m_lookup(struct domain *d, paddr_t addr);
+
 /* Ask the MMU to translate a VA for us */
 static inline uint64_t __va_to_par(uint32_t va)
 {
@@ -270,7 +273,11 @@ static inline uint64_t va_to_par(uint32_t va)
 {
     uint64_t par = __va_to_par(va);
     /* It is not OK to call this with an invalid VA */
-    if ( par & PAR_F ) panic_PAR(par, "Hypervisor");
+    if ( par & PAR_F )
+    {
+        dump_pt_walk(va);
+        panic_PAR(par, "Hypervisor");
+    }
     return par;
 }
 
@@ -314,9 +321,6 @@ static inline uint64_t gva_to_ipa(uint32_t va)
 /* Bits in the PAR returned by va_to_par */
 #define PAR_FAULT 0x1
 
-extern void dump_pt_walk(uint32_t addr);
-extern void dump_p2m_lookup(struct domain *d, paddr_t addr);
-
 #endif /* __ASSEMBLY__ */
 
 /* These numbers add up to a 39-bit input address space.  The  ARMv7-A
-- 
1.7.9.1

[PATCH 21/38] arm: dump guest s1 walk on data abort which is not a stage 2 issue.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/traps.c            |   75 +++++++++++++++++++++++++++++++++++---
 xen/include/asm-arm/processor.h |    1 +
 2 files changed, 70 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 40bb375..35907ee 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -28,6 +28,7 @@
 #include <xen/errno.h>
 #include <xen/hypercall.h>
 #include <xen/softirq.h>
+#include <xen/domain_page.h>
 #include <public/xen.h>
 #include <asm/regs.h>
 #include <asm/cpregs.h>
@@ -528,6 +529,62 @@ static void do_cp15_64(struct cpu_user_regs *regs,
 
 }
 
+void dump_guest_s1_walk(struct domain *d, uint32_t addr)
+{
+    uint32_t ttbcr = READ_CP32(TTBCR);
+    uint32_t ttbr0 = READ_CP32(TTBR0);
+    paddr_t paddr;
+    uint32_t offset;
+    uint32_t *first = NULL, *second = NULL;
+
+    printk("dom%d VA %#010"PRIx32"\n", d->domain_id, addr);
+    printk("    TTBCR: %#010"PRIx32"\n", ttbcr);
+    printk("    TTBR0: %#010"PRIx32" = %#"PRIpaddr"\n",
+           ttbr0, p2m_lookup(d, ttbr0 & PAGE_MASK));
+
+    if ( ttbcr & TTBCR_EAE )
+    {
+        printk("Cannot handle LPAE guest PT walk\n");
+        return;
+    }
+    if ( (ttbcr & TTBCR_N_MASK) != 0 )
+    {
+        printk("Cannot handle TTBR1 guest walks\n");
+        return;
+    }
+
+    paddr = p2m_lookup(d, ttbr0 & PAGE_MASK);
+    if ( paddr == INVALID_PADDR )
+    {
+        printk("Failed TTBR0 maddr lookup\n");
+        goto done;
+    }
+    first = map_domain_page(paddr>>PAGE_SHIFT);
+
+    offset = addr >> (12+10);
+    printk("1ST[%#03"PRIx32"] (%#"PRIpaddr") = %#010"PRIx32"\n",
+           offset, paddr, first[offset]);
+    if ( !(first[offset] & 0x1) ||
+         !(first[offset] & 0x2) )
+        goto done;
+
+    paddr = p2m_lookup(d, first[offset] & PAGE_MASK);
+
+    if ( paddr == INVALID_PADDR )
+    {
+        printk("Failed L1 entry maddr lookup\n");
+        goto done;
+    }
+    second = map_domain_page(paddr>>PAGE_SHIFT);
+    offset = (addr >> 12) & 0x3FF;
+    printk("2ND[%#03"PRIx32"] (%#"PRIpaddr") = %#010"PRIx32"\n",
+           offset, paddr, second[offset]);
+
+done:
+    if (second) unmap_domain_page(second);
+    if (first) unmap_domain_page(first);
+}
+
 static void do_trap_data_abort_guest(struct cpu_user_regs *regs,
                                      struct hsr_dabt dabt)
 {
@@ -535,11 +592,12 @@ static void do_trap_data_abort_guest(struct cpu_user_regs *regs,
     int level = -1;
     mmio_info_t info;
 
+    info.dabt = dabt;
+    info.gva = READ_CP32(HDFAR);
+
     if (dabt.s1ptw)
         goto bad_data_abort;
 
-    info.dabt = dabt;
-    info.gva = READ_CP32(HDFAR);
     info.gpa = gva_to_ipa(info.gva);
 
     if (handle_mmio(&info))
@@ -553,18 +611,23 @@ bad_data_abort:
     msg = decode_fsc( dabt.dfsc, &level);
 
     printk("Guest data abort: %s%s%s\n"
-           "    gva=%"PRIx32" gpa=%"PRIpaddr"\n",
+           "    gva=%"PRIx32"\n",
            msg, dabt.s1ptw ? " S2 during S1" : "",
            fsc_level_str(level),
-           info.gva, info.gpa);
-    if (dabt.valid)
+           info.gva);
+    if ( !dabt.s1ptw )
+        printk("    gpa=%"PRIpaddr"\n", info.gpa);
+    if ( dabt.valid )
         printk("    size=%d sign=%d write=%d reg=%d\n",
                dabt.size, dabt.sign, dabt.write, dabt.reg);
     else
         printk("    instruction syndrome invalid\n");
     printk("    eat=%d cm=%d s1ptw=%d dfsc=%d\n",
            dabt.eat, dabt.cache, dabt.s1ptw, dabt.dfsc);
-
+    if ( !dabt.s1ptw )
+        dump_p2m_lookup(current->domain, info.gpa);
+    else
+        dump_guest_s1_walk(current->domain, info.gva);
     show_execution_state(regs);
     panic("Unhandled guest data abort\n");
 }
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index ec6fb48..81924a4 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -25,6 +25,7 @@
 #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
 
 /* TTBCR Translation Table Base Control Register */
+#define TTBCR_EAE    0x80000000
 #define TTBCR_N_MASK 0x07
 #define TTBCR_N_16KB 0x00
 #define TTBCR_N_8KB  0x01
-- 
1.7.9.1

[PATCH 22/38] arm: implement vcpu_show_execution_state

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/dummy.S |    1 -
 xen/arch/arm/traps.c |   56 +++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 51 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
index 03f7489..cab9522 100644
--- a/xen/arch/arm/dummy.S
+++ b/xen/arch/arm/dummy.S
@@ -21,7 +21,6 @@ DUMMY(pirq_set_affinity);
 DUMMY(arch_get_info_guest);
 DUMMY(arch_vcpu_reset);
 NOP(update_vcpu_system_time);
-DUMMY(vcpu_show_execution_state);
 
 /* Page Reference & Type Maintenance */
 DUMMY(get_page);
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 35907ee..ec74298 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -170,7 +170,13 @@ void panic_PAR(uint64_t par, const char *when)
     panic("Error during %s-to-physical address translation\n", when);
 }
 
-void show_registers(struct cpu_user_regs *regs)
+struct reg_ctxt {
+    uint32_t sctlr;
+    uint32_t ttbr0, ttbr1, ttbcr;
+};
+static void _show_registers(struct cpu_user_regs *regs,
+                            struct reg_ctxt *ctxt,
+                            int guest_mode)
 {
     static const char *mode_strings[] = {
        [PSR_MODE_USR] = "USR",
@@ -187,7 +193,7 @@ void show_registers(struct cpu_user_regs *regs)
     print_xen_info();
     printk("CPU:    %d\n", smp_processor_id());
     printk("PC:     %08"PRIx32, regs->pc);
-    if ( !guest_mode(regs) )
+    if ( !guest_mode )
             print_symbol(" %s", regs->pc);
     printk("\n");
     printk("CPSR:   %08"PRIx32" MODE:%s\n", regs->cpsr,
@@ -199,7 +205,7 @@ void show_registers(struct cpu_user_regs *regs)
     printk("     R8: %08"PRIx32" R9: %08"PRIx32" R10:%08"PRIx32" R11:%08"PRIx32" R12:%08"PRIx32"\n",
            regs->r8, regs->r9, regs->r10, regs->r11, regs->r12);
 
-    if ( guest_mode(regs) )
+    if ( guest_mode )
     {
         printk("USR: SP: %08"PRIx32" LR: %08"PRIx32" CPSR:%08"PRIx32"\n",
                regs->sp_usr, regs->lr_usr, regs->cpsr);
@@ -217,8 +223,8 @@ void show_registers(struct cpu_user_regs *regs)
                regs->r8_fiq, regs->r9_fiq, regs->r10_fiq, regs->r11_fiq, regs->r11_fiq);
         printk("\n");
         printk("TTBR0 %08"PRIx32" TTBR1 %08"PRIx32" TTBCR %08"PRIx32"\n",
-               READ_CP32(TTBR0), READ_CP32(TTBR1), READ_CP32(TTBCR));
-        printk("SCTLR %08"PRIx32"\n", READ_CP32(SCTLR));
+               ctxt->ttbr0, ctxt->ttbr1, ctxt->ttbcr);
+        printk("SCTLR %08"PRIx32"\n", ctxt->sctlr);
         printk("VTTBR %010"PRIx64"\n", READ_CP64(VTTBR));
         printk("\n");
     }
@@ -241,6 +247,26 @@ void show_registers(struct cpu_user_regs *regs)
     printk("\n");
 }
 
+void show_registers(struct cpu_user_regs *regs)
+{
+    struct reg_ctxt ctxt;
+    ctxt.sctlr = READ_CP32(SCTLR);
+    ctxt.ttbcr = READ_CP32(TTBCR);
+    ctxt.ttbr0 = READ_CP32(TTBR0);
+    ctxt.ttbr1 = READ_CP32(TTBR1);
+    _show_registers(regs, &ctxt, guest_mode(regs));
+}
+
+void vcpu_show_registers(const struct vcpu *v)
+{
+    struct reg_ctxt ctxt;
+    ctxt.sctlr = v->arch.sctlr;
+    ctxt.ttbcr = v->arch.ttbcr;
+    ctxt.ttbr0 = v->arch.ttbr0;
+    ctxt.ttbr1 = v->arch.ttbr1;
+    _show_registers(&v->arch.cpu_info->guest_cpu_user_regs, &ctxt, 1);
+}
+
 static void show_guest_stack(struct cpu_user_regs *regs)
 {
     printk("GUEST STACK GOES HERE\n");
@@ -334,6 +360,26 @@ void show_execution_state(struct cpu_user_regs *regs)
     show_stack(regs);
 }
 
+void vcpu_show_execution_state(struct vcpu *v)
+{
+    printk("*** Dumping Dom%d vcpu#%d state: ***\n",
+           v->domain->domain_id, v->vcpu_id);
+
+    if ( v == current )
+    {
+        show_execution_state(guest_cpu_user_regs());
+        return;
+    }
+
+    vcpu_pause(v); /* acceptably dangerous */
+
+    vcpu_show_registers(v);
+    if ( !usr_mode(&v->arch.cpu_info->guest_cpu_user_regs) )
+        show_guest_stack(&v->arch.cpu_info->guest_cpu_user_regs);
+
+    vcpu_unpause(v);
+}
+
 static void do_unexpected_trap(const char *msg, struct cpu_user_regs *regs)
 {
     printk("Unexpected Trap: %s\n", msg);
-- 
1.7.9.1

[PATCH 23/38] arm: use correct attributes for mappings in copy_from_paddr()

[Ian Campbell]

The DTB is in RAM (hence bufferable), kernel is in flash and therefor requires
a device type mapping (hence dev shared).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/kernel.c       |    8 ++++----
 xen/arch/arm/setup.c        |    2 +-
 xen/include/asm-arm/setup.h |    2 +-
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/kernel.c b/xen/arch/arm/kernel.c
index 130d488..1a705c9 100644
--- a/xen/arch/arm/kernel.c
+++ b/xen/arch/arm/kernel.c
@@ -39,7 +39,7 @@ struct minimal_dtb_header {
  * @paddr: source physical address
  * @len: length to copy
  */
-void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
+void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int mattr)
 {
     void *src = (void *)FIXMAP_ADDR(FIXMAP_MISC);
 
@@ -51,7 +51,7 @@ void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
         s = paddr & (PAGE_SIZE-1);
         l = min(PAGE_SIZE - s, len);
 
-        set_fixmap(FIXMAP_MISC, p, DEV_SHARED);
+        set_fixmap(FIXMAP_MISC, p, mattr);
         memcpy(dst, src + s, l);
 
         paddr += l;
@@ -111,7 +111,7 @@ static int kernel_try_zimage_prepare(struct kernel_info *info)
     /*
      * Check for an appended DTB.
      */
-    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr));
+    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr), DEV_SHARED);
     if (be32_to_cpu(dtb_hdr.magic) == DTB_MAGIC) {
         end += be32_to_cpu(dtb_hdr.total_size);
     }
@@ -151,7 +151,7 @@ static int kernel_try_elf_prepare(struct kernel_info *info)
     if ( info->kernel_img == NULL )
         panic("Cannot allocate temporary buffer for kernel.\n");
 
-    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE);
+    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE, DEV_SHARED);
 
     if ( (rc = elf_init(&info->elf.elf, info->kernel_img, KERNEL_FLASH_SIZE )) != 0 )
         return rc;
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index b0cfacc..f5473cd 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -122,7 +122,7 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
      * TODO: handle other payloads too.
      */
     device_tree_flattened = mfn_to_virt(alloc_boot_pages(dtb_pages, 1));
-    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size);
+    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size, BUFFERABLE);
 
     /* Add non-xenheap memory */
     init_boot_pages(pfn_to_paddr(xenheap_mfn_start + xenheap_pages),
diff --git a/xen/include/asm-arm/setup.h b/xen/include/asm-arm/setup.h
index 05ff89e..faadccc 100644
--- a/xen/include/asm-arm/setup.h
+++ b/xen/include/asm-arm/setup.h
@@ -3,7 +3,7 @@
 
 #include <public/version.h>
 
-void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len);
+void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int mattr);
 
 void arch_get_xen_caps(xen_capabilities_info_t *info);
 
-- 
1.7.9.1

[PATCH 24/38] arm: map fixmaps non-executable.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/mm.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index c332e4c..160a4e9 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -108,6 +108,7 @@ void set_fixmap(unsigned map, unsigned long mfn, unsigned attributes)
     lpae_t pte = mfn_to_xen_entry(mfn);
     pte.pt.table = 1; /* 4k mappings always have this bit set */
     pte.pt.ai = attributes;
+    pte.pt.xn = 1;
     write_pte(xen_fixmap + third_table_offset(FIXMAP_ADDR(map)), pte);
     flush_xen_data_tlb_va(FIXMAP_ADDR(map));
 }
-- 
1.7.9.1

[PATCH 25/38] arm: remove old identity map of boot paddr when we are done with it.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/mm.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 160a4e9..ab52171 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -214,6 +214,14 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
     lpae_t pte, *p;
     int i;
 
+    if ( boot_phys_offset != 0 )
+    {
+        /* Remove the old identity mapping of the boot paddr */
+        pte.bits = 0;
+        dest_va = (unsigned long)_start + boot_phys_offset;
+        write_pte(xen_second + second_linear_offset(dest_va), pte);
+    }
+
     xen_paddr = device_tree_get_xen_paddr();
 
     /* Map the destination in the boot misc area. */
-- 
1.7.9.1

[PATCH 26/38] arm: fix locking in create_p2m_entries

[Ian Campbell]

For some reason we were holding the lock over only the unmaps at the end of
the function, rather than for the whole walk.

We might want to be more clever in the future, but for now lets just lock for
the whole walk+create process.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 46c6f17..c4daf83 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -168,6 +168,8 @@ static int create_p2m_entries(struct domain *d,
     paddr_t addr;
     unsigned long cur_first_offset = ~0, cur_second_offset = ~0;
 
+    spin_lock(&p2m->lock);
+
     /* XXX Don't actually handle 40 bit guest physical addresses */
     BUG_ON(start_gpaddr & 0x8000000000ULL);
     BUG_ON(end_gpaddr   & 0x8000000000ULL);
@@ -249,8 +251,6 @@ static int create_p2m_entries(struct domain *d,
     rc = 0;
 
 out:
-    spin_lock(&p2m->lock);
-
     if (third) unmap_domain_page(third);
     if (second) unmap_domain_page(second);
     if (first) unmap_domain_page(first);
-- 
1.7.9.1

[PATCH 27/38] arm: split pending SPIs (global) out from pending PPIs and SGIs (per CPU)

[Ian Campbell]

This tracks SPIs in struct arch_domain and PPIs+SGIs in struct arch_vcpu which
seems more logical.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/vgic.c          |   17 ++++++++++-------
 xen/include/asm-arm/domain.h |   10 ++++++++++
 2 files changed, 20 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 629a0da..91d6166 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -82,9 +82,8 @@ int domain_vgic_init(struct domain *d)
     d->arch.vgic.shared_irqs =
         xmalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
     d->arch.vgic.pending_irqs =
-        xmalloc_array(struct pending_irq,
-                d->arch.vgic.nr_lines + (32 * d->max_vcpus));
-    for (i=0; i<d->arch.vgic.nr_lines + (32 * d->max_vcpus); i++)
+        xzalloc_array(struct pending_irq, d->arch.vgic.nr_lines);
+    for (i=0; i<d->arch.vgic.nr_lines; i++)
         INIT_LIST_HEAD(&d->arch.vgic.pending_irqs[i].inflight);
     for (i=0; i<DOMAIN_NR_RANKS(d); i++)
         spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
@@ -98,6 +97,10 @@ int vcpu_vgic_init(struct vcpu *v)
 
     spin_lock_init(&v->arch.vgic.private_irqs.lock);
 
+    memset(&v->arch.vgic.pending_irqs, 0, sizeof(v->arch.vgic.pending_irqs));
+    for (i = 0; i < 32; i++)
+        INIT_LIST_HEAD(&v->arch.vgic.pending_irqs[i].inflight);
+
     /* For SGI and PPI the target is always this CPU */
     for ( i = 0 ; i < 8 ; i++ )
         v->arch.vgic.private_irqs.itargets[i] =
@@ -535,8 +538,7 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
     /* Pending irqs allocation strategy: the first vgic.nr_lines irqs
      * are used for SPIs; the rests are used for per cpu irqs */
     if ( irq < 32 )
-        n = &v->domain->arch.vgic.pending_irqs[irq + (v->vcpu_id * 32)
-            + v->domain->arch.vgic.nr_lines];
+        n = &v->arch.vgic.pending_irqs[irq];
     else
         n = &v->domain->arch.vgic.pending_irqs[irq - 32];
     return n;
@@ -548,6 +550,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
     uint8_t priority;
     struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
     struct pending_irq *iter, *n = irq_to_pending(v, irq);
+    unsigned long flags;
 
     /* irq still pending */
     if (!list_empty(&n->inflight))
@@ -564,7 +567,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
 
     gic_set_guest_irq(irq, GICH_LR_PENDING, priority);
 
-    spin_lock(&v->arch.vgic.lock);
+    spin_lock_irqsave(&v->arch.vgic.lock, flags);
     list_for_each_entry ( iter, &v->arch.vgic.inflight_irqs, inflight )
     {
         if ( iter->priority > priority )
@@ -575,7 +578,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
         }
     }
     list_add_tail(&n->inflight, &v->arch.vgic.inflight_irqs);
-    spin_unlock(&v->arch.vgic.lock);
+    spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
     /* we have a new higher priority irq, inject it into the guest */
 }
 
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 620b26e..32deb52 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -46,6 +46,10 @@ struct arch_domain
         int ctlr;
         int nr_lines;
         struct vgic_irq_rank *shared_irqs;
+        /*
+         * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
+         * struct arch_vcpu.
+         */
         struct pending_irq *pending_irqs;
     } vgic;
 
@@ -114,7 +118,13 @@ struct arch_vcpu
     uint32_t gic_lr[64];
 
     struct {
+        /*
+         * SGIs and PPIs are per-VCPU, SPIs are domain global and in
+         * struct arch_domain.
+         */
+        struct pending_irq pending_irqs[32];
         struct vgic_irq_rank private_irqs;
+
         /* This list is ordered by IRQ priority and it is used to keep
          * track of the IRQs that the VGIC injected into the guest.
          * Depending on the availability of LR registers, the IRQs might
-- 
1.7.9.1

[PATCH 28/38] arm: map GICV in all domains, not just dom0.

[Ian Campbell]

This requires that we allocate all p2m pages from domheap without a particular
dom because max pages is not setup yet so there is no allocation available to
us.

At some point we should create a separate p2m allocation (similar to x86's shadow allocation) and use that.

Also we seem to have been calling p2m_alloc_table twice for dom0.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c       |   10 +++++++---
 xen/arch/arm/domain_build.c |    5 -----
 xen/arch/arm/gic.c          |    9 ++++-----
 xen/arch/arm/gic.h          |    2 +-
 xen/arch/arm/p2m.c          |    3 ++-
 5 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index a7fb227..e15c1e8 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -329,13 +329,17 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
 
         if ( (rc = p2m_alloc_table(d)) != 0 )
             goto fail;
-    }
 
-    if ( (rc = domain_vgic_init(d)) != 0 )
-        goto fail;
+        if ( (rc = gicv_setup(d)) != 0 )
+            goto fail;
+
+        if ( (rc = domain_vgic_init(d)) != 0 )
+            goto fail;
+    }
 
     rc = 0;
 fail:
+    /*XXX unwind allocations etc */
     return rc;
 }
 
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 72e775c..1b19e54 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -270,9 +270,6 @@ int construct_dom0(struct domain *d)
 
     d->max_pages = ~0U;
 
-    if ( (rc = p2m_alloc_table(d)) != 0 )
-        return rc;
-
     rc = prepare_dtb(d, &kinfo);
     if ( rc < 0 )
         return rc;
@@ -288,8 +285,6 @@ int construct_dom0(struct domain *d)
     printk("Map VGIC MMIO regions 1:1 in the P2M %#llx->%#llx\n", 0x2C008000ULL, 0x2DFFFFFFULL);
     map_mmio_regions(d, 0x2C008000, 0x2DFFFFFF, 0x2C008000);
 
-    gicv_setup(d);
-
     printk("Routing peripheral interrupts to guest\n");
     /* TODO Get from device tree */
     gic_route_irq_to_guest(d, 34, "timer0");
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 339c327..a398f92 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -541,14 +541,13 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
     do_IRQ(regs, irq, is_fiq);
 }
 
-void gicv_setup(struct domain *d)
+int gicv_setup(struct domain *d)
 {
+    printk("GICV setup for DOM%d\n", d->domain_id);
+
     /* map the gic virtual cpu interface in the gic cpu interface region of
      * the guest */
-    printk("mapping GICC at %#"PRIx32" to %#"PRIx32"\n",
-           GIC_BASE_ADDRESS + GIC_CR_OFFSET,
-           GIC_BASE_ADDRESS + GIC_VR_OFFSET);
-    map_mmio_regions(d, GIC_BASE_ADDRESS + GIC_CR_OFFSET,
+    return map_mmio_regions(d, GIC_BASE_ADDRESS + GIC_CR_OFFSET,
                         GIC_BASE_ADDRESS + GIC_CR_OFFSET + (2 * PAGE_SIZE) - 1,
                         GIC_BASE_ADDRESS + GIC_VR_OFFSET);
 }
diff --git a/xen/arch/arm/gic.h b/xen/arch/arm/gic.h
index ac9cf3a..018d820 100644
--- a/xen/arch/arm/gic.h
+++ b/xen/arch/arm/gic.h
@@ -148,7 +148,7 @@ extern void gic_init_secondary_cpu(void);
 /* Take down a CPU's per-CPU GIC interface */
 extern void gic_disable_cpu(void);
 /* setup the gic virtual interface for a guest */
-extern void gicv_setup(struct domain *d);
+extern int gicv_setup(struct domain *d);
 
 /* Context switch */
 extern void gic_save_state(struct vcpu *v);
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index c4daf83..0665445 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -4,6 +4,7 @@
 #include <xen/errno.h>
 #include <xen/domain_page.h>
 #include <asm/flushtlb.h>
+#include "gic.h"
 
 void dump_p2m_lookup(struct domain *d, paddr_t addr)
 {
@@ -138,7 +139,7 @@ static int p2m_create_entry(struct domain *d,
 
     BUG_ON(entry->p2m.valid);
 
-    page = alloc_domheap_page(d, 0);
+    page = alloc_domheap_page(NULL, 0);
     if ( page == NULL )
         return -ENOMEM;
 
-- 
1.7.9.1

[PATCH 29/38] arm: delay enabling data-cache until paging enabled.

[Ian Campbell]

With enough warnings enabled the model seemed to be complaining that pages
cached before paging was enabled had been mapped with to inconsistent sets of
attributes. I'm not convinced that isn't a model issue, nor am I convinced
this has really fixed anything, but it seems sensible enough.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/head.S |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/head.S b/xen/arch/arm/head.S
index 9a7714a..71197af 100644
--- a/xen/arch/arm/head.S
+++ b/xen/arch/arm/head.S
@@ -148,10 +148,11 @@ hyp:
 	 * Exceptions in LE ARM,
 	 * Low-latency IRQs disabled,
 	 * Write-implies-XN disabled (for now),
-	 * I-cache and d-cache enabled,
+	 * D-cache diabled (for now),
+	 * I-cache enabled,
 	 * Alignment checking enabled,
 	 * MMU translation disabled (for now). */
-	ldr   r0, =(HSCTLR_BASE|SCTLR_A|SCTLR_C)
+	ldr   r0, =(HSCTLR_BASE|SCTLR_A)
 	mcr   CP32(r0, HSCTLR)
 
 	/* Write Xen's PT's paddr into the HTTBR */
@@ -217,6 +218,10 @@ pt_ready:
 	mov   pc, r1                 /* Get a proper vaddr into PC */
 paging:
 
+	mrc   CP32(r0, HSCTLR)       /* Now enable data cache */
+	orr   r0, r0, #(SCTLR_C)
+	mcr   CP32(r0, HSCTLR)
+
 #ifdef EARLY_UART_ADDRESS
 	/* Recover the UART address in the new address space. */
 	lsl   r11, #11
-- 
1.7.9.1

[PATCH 30/38] arm: Upgrade guest barriers to Outer-Shareable. Enable Protected Table Walk.

[Ian Campbell]

Upgrading barriers is conservative and may not be necessary.

Protected Table Walk traps stage 1 page tables which refer to device memory
(per stage 2) using a non-device mapping. This generally indicates a guest
error but trapping it as a fault for now helps us know if something odd is
going on.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain_build.c     |    2 +-
 xen/include/asm-arm/processor.h |    4 ++++
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 1b19e54..a9e7f43 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -333,7 +333,7 @@ int construct_dom0(struct domain *d)
 
     WRITE_CP32(SCTLR_BASE, SCTLR);
 
-    WRITE_CP32(HCR_AMO|HCR_IMO|HCR_VM, HCR);
+    WRITE_CP32(HCR_PTW|HCR_BSU_OUTER|HCR_AMO|HCR_IMO|HCR_VM, HCR);
     isb();
 
     local_abort_enable();
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index 81924a4..9b3c9dd 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -76,6 +76,10 @@
 #define HCR_TWI         (1<<13)
 #define HCR_DC          (1<<12)
 #define HCR_BSU_MASK    (3<<10)
+#define HCR_BSU_NONE     (0<<10)
+#define HCR_BSU_INNER    (1<<10)
+#define HCR_BSU_OUTER    (2<<10)
+#define HCR_BSU_FULL     (3<<10)
 #define HCR_FB          (1<<9)
 #define HCR_VA          (1<<8)
 #define HCR_VI          (1<<7)
-- 
1.7.9.1

[PATCH 31/38] arm: gic.lock can be taken in interrupt context, so lock appropriately.

[Ian Campbell]

In particular it is taken by gic_set_guest_irq which is called by
vgic_vcpu_inject_irq

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/gic.c |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index a398f92..ededa99 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -329,19 +329,19 @@ int __init gic_init(void)
 /* Set up the per-CPU parts of the GIC for a secondary CPU */
 void __cpuinit gic_init_secondary_cpu(void)
 {
-    spin_lock(&gic.lock);
+    spin_lock_irq(&gic.lock);
     gic_cpu_init();
     gic_hyp_init();
-    spin_unlock(&gic.lock);
+    spin_unlock_irq(&gic.lock);
 }
 
 /* Shut down the per-CPU GIC interface */
 void gic_disable_cpu(void)
 {
-    spin_lock(&gic.lock);
+    spin_lock_irq(&gic.lock);
     gic_cpu_disable();
     gic_hyp_disable();
-    spin_unlock(&gic.lock);
+    spin_unlock_irq(&gic.lock);
 }
 
 void gic_route_irqs(void)
@@ -439,7 +439,7 @@ void gic_set_guest_irq(unsigned int virtual_irq,
 
     events_maintenance(current);
 
-    spin_lock(&gic.lock);
+    spin_lock_irq(&gic.lock);
 
     if ( list_empty(&gic.lr_pending) )
     {
@@ -465,7 +465,7 @@ void gic_set_guest_irq(unsigned int virtual_irq,
     list_add_tail(&n->lr_queue, &gic.lr_pending);
 
 out:
-    spin_unlock(&gic.lock);
+    spin_unlock_irq(&gic.lock);
     return;
 }
 
@@ -559,7 +559,7 @@ static void events_maintenance(struct vcpu *v)
             (unsigned long *)&vcpu_info(v, evtchn_upcall_pending));
 
     if (!already_pending && gic.event_mask != 0) {
-        spin_lock(&gic.lock);
+        spin_lock_irq(&gic.lock);
         while ((i = find_next_bit((const long unsigned int *) &gic.event_mask,
                         sizeof(uint64_t), i)) < sizeof(uint64_t)) {
 
@@ -569,7 +569,7 @@ static void events_maintenance(struct vcpu *v)
 
             i++;
         }
-        spin_unlock(&gic.lock);
+        spin_unlock_irq(&gic.lock);
     }
 }
 
@@ -585,7 +585,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
                               sizeof(eisr), i)) < sizeof(eisr)) {
         struct pending_irq *p;
 
-        spin_lock(&gic.lock);
+        spin_lock_irq(&gic.lock);
         lr = GICH[GICH_LR + i];
         virq = lr & GICH_LR_VIRTUAL_MASK;
         GICH[GICH_LR + i] = 0;
@@ -601,7 +601,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
         } else {
             gic_inject_irq_stop();
         }
-        spin_unlock(&gic.lock);
+        spin_unlock_irq(&gic.lock);
 
         spin_lock(&current->arch.vgic.lock);
         p = irq_to_pending(current, virq);
-- 
1.7.9.1

[PATCH 32/38] arm: context switch virtual timer registers

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c        |   10 ++++++++++
 xen/include/asm-arm/cpregs.h |    3 +++
 xen/include/asm-arm/domain.h |    5 +++++
 3 files changed, 18 insertions(+), 0 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index e15c1e8..893a169 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -49,6 +49,11 @@ static void ctxt_switch_from(struct vcpu *p)
     p->arch.tpidruro = READ_CP32(TPIDRURO);
     p->arch.tpidrprw = READ_CP32(TPIDRPRW);
 
+    /* Arch timer */
+    p->arch.cntvoff = READ_CP64(CNTVOFF);
+    p->arch.cntv_cval = READ_CP64(CNTV_CVAL);
+    p->arch.cntv_ctl = READ_CP32(CNTV_CTL);
+
     /* XXX only save these if ThumbEE e.g. ID_PFR0.THUMB_EE_SUPPORT */
     p->arch.teecr = READ_CP32(TEECR);
     p->arch.teehbr = READ_CP32(TEEHBR);
@@ -128,6 +133,11 @@ static void ctxt_switch_to(struct vcpu *n)
     WRITE_CP32(n->arch.mair1, MAIR1);
     isb();
 
+    /* Arch timer */
+    WRITE_CP64(n->arch.cntvoff, CNTVOFF);
+    WRITE_CP64(n->arch.cntv_cval, CNTV_CVAL);
+    WRITE_CP32(n->arch.cntv_ctl, CNTV_CTL);
+
     /* Control Registers */
     WRITE_CP32(n->arch.actlr, ACTLR);
     WRITE_CP32(n->arch.sctlr, SCTLR);
diff --git a/xen/include/asm-arm/cpregs.h b/xen/include/asm-arm/cpregs.h
index bd46942..34a9e93 100644
--- a/xen/include/asm-arm/cpregs.h
+++ b/xen/include/asm-arm/cpregs.h
@@ -238,10 +238,13 @@
 #define CNTP_CTL        p15,0,c14,c2,1  /* Physical Timer control register */
 #define CNTVCT          p15,1,c14       /* Time counter value + offset */
 #define CNTP_CVAL       p15,2,c14       /* Physical Timer comparator */
+#define CNTV_CVAL       p15,3,c14       /* Virt. Timer comparator */
 #define CNTVOFF         p15,4,c14       /* Time counter offset */
 #define CNTHCTL         p15,4,c14,c1,0  /* Time counter hyp. control */
 #define CNTHP_TVAL      p15,4,c14,c2,0  /* Hyp. Timer value */
 #define CNTHP_CTL       p15,4,c14,c2,1  /* Hyp. Timer control register */
+#define CNTV_TVAL       p15,0,c14,c3,0  /* Virt. Timer value */
+#define CNTV_CTL        p15,0,c14,c3,1  /* Virt. TImer control register */
 #define CNTHP_CVAL      p15,6,c14       /* Hyp. Timer comparator */
 
 /* CP15 CR15: Implementation Defined Registers */
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 32deb52..230ea8c 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -111,6 +111,11 @@ struct arch_vcpu
     uint32_t teecr, teehbr;
     uint32_t joscr, jmcr;
 
+    /* Arch timers */
+    uint64_t cntvoff;
+    uint64_t cntv_cval;
+    uint32_t cntv_ctl;
+
     /* CP 15 */
     uint32_t csselr;
 
-- 
1.7.9.1

[PATCH 33/38] arm: the hyp timer seems to work now, default to using it.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/time.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index 437dc71..1587fa2 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -27,8 +27,12 @@
 #include <xen/time.h>
 #include <asm/system.h>
 
-/* Unfortunately the hypervisor timer interrupt appears to be buggy */
-#define USE_HYP_TIMER 0
+/*
+ * Unfortunately the hypervisor timer interrupt appears to be buggy in
+ * some versions of the model. Disable this to use the physical timer
+ * instead.
+ */
+#define USE_HYP_TIMER 1
 
 /* For fine-grained timekeeping, we use the ARM "Generic Timer", a
  * register-mapped time source in the SoC. */
-- 
1.7.9.1

[PATCH 34/38] HACK: arm: initial XENMAPSPACE_gmfn_foreign

[Ian Campbell]

Should use same interface as hybrid x86.
---
 xen/arch/arm/mm.c             |   32 ++++++++++++++++++++++++++------
 xen/arch/x86/mm.c             |    2 ++
 xen/include/public/arch-arm.h |    1 +
 xen/include/public/memory.h   |   12 +++++++-----
 4 files changed, 36 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index ab52171..1832e7f 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -480,12 +480,32 @@ static int xenmem_add_to_physmap_once(
 
     switch ( xatp->space )
     {
-        case XENMAPSPACE_shared_info:
-            if ( xatp->idx == 0 )
-                mfn = virt_to_mfn(d->shared_info);
-            break;
-        default:
-            return -ENOSYS;
+    case XENMAPSPACE_shared_info:
+        if ( xatp->idx == 0 )
+            mfn = virt_to_mfn(d->shared_info);
+        break;
+    case XENMAPSPACE_gmfn_foreign:
+    {
+        paddr_t maddr;
+        struct domain *od;
+
+        rc = rcu_lock_target_domain_by_id(xatp->foreign_domid, &od);
+        if ( rc < 0 )
+            return rc;
+        maddr = p2m_lookup(od, xatp->idx << PAGE_SHIFT);
+        if ( maddr == INVALID_PADDR )
+        {
+            printk("bad p2m lookup\n");
+            dump_p2m_lookup(od, xatp->idx << PAGE_SHIFT);
+            rcu_unlock_domain(od);
+            return -EINVAL;
+        }
+        mfn = maddr >> PAGE_SHIFT;
+        rcu_unlock_domain(od);
+        break;
+    }
+    default:
+        return -ENOSYS;
     }
 
     domain_lock(d);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 876e1ef..d6c90f9 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -4572,6 +4572,8 @@ static int xenmem_add_to_physmap_once(
             mfn = idx;
             page = mfn_to_page(mfn);
             break;
+        case XENMAPSPACE_gmfn_foreign:
+            return -ENOSYS;
         }
         default:
             break;
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index e915cbf..b52bfc7 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -121,6 +121,7 @@ typedef uint64_t xen_pfn_t;
 #define XEN_LEGACY_MAX_VCPUS 1
 
 typedef uint32_t xen_ulong_t;
+#define PRI_xen_ulong PRIx32
 
 struct vcpu_guest_context {
     struct cpu_user_regs user_regs;         /* User-level CPU registers     */
diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
index 86d02c8..b2adfbe 100644
--- a/xen/include/public/memory.h
+++ b/xen/include/public/memory.h
@@ -212,11 +212,13 @@ struct xen_add_to_physmap {
     uint16_t    size;
 
     /* Source mapping space. */
-#define XENMAPSPACE_shared_info 0 /* shared info page */
-#define XENMAPSPACE_grant_table 1 /* grant table page */
-#define XENMAPSPACE_gmfn        2 /* GMFN */
-#define XENMAPSPACE_gmfn_range  3 /* GMFN range */
-    unsigned int space;
+#define XENMAPSPACE_shared_info  0 /* shared info page */
+#define XENMAPSPACE_grant_table  1 /* grant table page */
+#define XENMAPSPACE_gmfn         2 /* GMFN */
+#define XENMAPSPACE_gmfn_range   3 /* GMFN range */
+#define XENMAPSPACE_gmfn_foreign 4 /* GMFN from another guest */
+    uint16_t space;
+    domid_t foreign_domid; /* IFF gmfn_foreign */
 
 #define XENMAPIDX_grant_table_status 0x80000000
 
-- 
1.7.9.1

[PATCH 35/38] arm: move PSR flag definitions into interface, for tools use.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/entry.S            |    1 +
 xen/include/asm-arm/page.h      |    2 ++
 xen/include/asm-arm/processor.h |   21 ---------------------
 xen/include/asm-arm/system.h    |    2 +-
 xen/include/public/arch-arm.h   |   23 ++++++++++++++++++++++-
 5 files changed, 26 insertions(+), 23 deletions(-)

diff --git a/xen/arch/arm/entry.S b/xen/arch/arm/entry.S
index 5bc3906..2ff32a1 100644
--- a/xen/arch/arm/entry.S
+++ b/xen/arch/arm/entry.S
@@ -1,5 +1,6 @@
 #include <xen/config.h>
 #include <asm/asm_defns.h>
+#include <public/xen.h>
 
 #define SAVE_ONE_BANKED(reg)	mrs r11, reg; str r11, [sp, #UREGS_##reg]
 #define RESTORE_ONE_BANKED(reg)	ldr r11, [sp, #UREGS_##reg]; msr reg, r11
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index f36bf6f..12ab2e8 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -2,6 +2,8 @@
 #define __ARM_PAGE_H__
 
 #include <xen/config.h>
+#include <public/xen.h>
+#include <asm/processor.h>
 
 #define PADDR_BITS              40
 #define PADDR_MASK              ((1ULL << PADDR_BITS)-1)
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index 9b3c9dd..3849b23 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -3,27 +3,6 @@
 
 #include <asm/cpregs.h>
 
-/* PSR bits (CPSR, SPSR)*/
-
-/* 0-4: Mode */
-#define PSR_MODE_MASK 0x1f
-#define PSR_MODE_USR 0x10
-#define PSR_MODE_FIQ 0x11
-#define PSR_MODE_IRQ 0x12
-#define PSR_MODE_SVC 0x13
-#define PSR_MODE_MON 0x16
-#define PSR_MODE_ABT 0x17
-#define PSR_MODE_HYP 0x1a
-#define PSR_MODE_UND 0x1b
-#define PSR_MODE_SYS 0x1f
-
-#define PSR_THUMB       (1<<5)        /* Thumb Mode enable */
-#define PSR_FIQ_MASK    (1<<6)        /* Fast Interrupt mask */
-#define PSR_IRQ_MASK    (1<<7)        /* Interrupt mask */
-#define PSR_ABT_MASK    (1<<8)        /* Asynchronous Abort mask */
-#define PSR_BIG_ENDIAN  (1<<9)        /* Big Endian Mode */
-#define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
-
 /* TTBCR Translation Table Base Control Register */
 #define TTBCR_EAE    0x80000000
 #define TTBCR_N_MASK 0x07
diff --git a/xen/include/asm-arm/system.h b/xen/include/asm-arm/system.h
index 7963ea5..216ef1f 100644
--- a/xen/include/asm-arm/system.h
+++ b/xen/include/asm-arm/system.h
@@ -3,7 +3,7 @@
 #define __ASM_SYSTEM_H
 
 #include <xen/lib.h>
-#include <asm/processor.h>
+#include <public/arch-arm.h>
 
 #define nop() \
     asm volatile ( "nop" )
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index b52bfc7..7ebe966 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -139,7 +139,28 @@ struct arch_shared_info { };
 typedef struct arch_shared_info arch_shared_info_t;
 typedef uint64_t xen_callback_t;
 
-#endif
+#endif /* ifndef __ASSEMBLY __ */
+
+/* PSR bits (CPSR, SPSR)*/
+
+/* 0-4: Mode */
+#define PSR_MODE_MASK 0x1f
+#define PSR_MODE_USR 0x10
+#define PSR_MODE_FIQ 0x11
+#define PSR_MODE_IRQ 0x12
+#define PSR_MODE_SVC 0x13
+#define PSR_MODE_MON 0x16
+#define PSR_MODE_ABT 0x17
+#define PSR_MODE_HYP 0x1a
+#define PSR_MODE_UND 0x1b
+#define PSR_MODE_SYS 0x1f
+
+#define PSR_THUMB       (1<<5)        /* Thumb Mode enable */
+#define PSR_FIQ_MASK    (1<<6)        /* Fast Interrupt mask */
+#define PSR_IRQ_MASK    (1<<7)        /* Interrupt mask */
+#define PSR_ABT_MASK    (1<<8)        /* Asynchronous Abort mask */
+#define PSR_BIG_ENDIAN  (1<<9)        /* Big Endian Mode */
+#define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
 
 #endif /*  __XEN_PUBLIC_ARCH_ARM_H__ */
 
-- 
1.7.9.1

[PATCH 36/38] libxc: add ARM support to xc_dom (PV domain building)

[Ian Campbell]

Includes ARM zImage support.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxc/Makefile                 |    1 +
 tools/libxc/xc_dom.h                 |    5 +-
 tools/libxc/xc_dom_arm.c             |  135 +++++++++++++++++++++++++++-
 tools/libxc/xc_dom_armzimageloader.c |  167 ++++++++++++++++++++++++++++++++++
 tools/libxc/xc_dom_core.c            |   12 ++-
 tools/libxc/xg_private.h             |    4 +
 6 files changed, 315 insertions(+), 9 deletions(-)
 create mode 100644 tools/libxc/xc_dom_armzimageloader.c

diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index ca38cbd..a01d457 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -59,6 +59,7 @@ GUEST_SRCS-y += libelf-dominfo.c libelf-relocate.c
 GUEST_SRCS-y                 += xc_dom_core.c xc_dom_boot.c
 GUEST_SRCS-y                 += xc_dom_elfloader.c
 GUEST_SRCS-$(CONFIG_X86)     += xc_dom_bzimageloader.c
+GUEST_SRCS-$(CONFIG_ARM)     += xc_dom_armzimageloader.c
 GUEST_SRCS-y                 += xc_dom_binloader.c
 GUEST_SRCS-y                 += xc_dom_compat_linux.c
 
diff --git a/tools/libxc/xc_dom.h b/tools/libxc/xc_dom.h
index 2aef64a..4db8fad 100644
--- a/tools/libxc/xc_dom.h
+++ b/tools/libxc/xc_dom.h
@@ -93,6 +93,7 @@ struct xc_dom_image {
     void *p2m_guest;
 
     /* physical memory */
+    xen_pfn_t rambase_pfn;
     xen_pfn_t total_pages;
     struct xc_dom_phys *phys_pages;
     int realmodearea_log;
@@ -286,7 +287,7 @@ static inline xen_pfn_t xc_dom_p2m_host(struct xc_dom_image *dom, xen_pfn_t pfn)
 {
     if (dom->shadow_enabled)
         return pfn;
-    return dom->p2m_host[pfn];
+    return dom->p2m_host[pfn - dom->rambase_pfn];
 }
 
 static inline xen_pfn_t xc_dom_p2m_guest(struct xc_dom_image *dom,
@@ -294,7 +295,7 @@ static inline xen_pfn_t xc_dom_p2m_guest(struct xc_dom_image *dom,
 {
     if (xc_dom_feature_translated(dom))
         return pfn;
-    return dom->p2m_host[pfn];
+    return dom->p2m_host[pfn - dom->rambase_pfn];
 }
 
 /* --- arch bits --------------------------------------------------- */
diff --git a/tools/libxc/xc_dom_arm.c b/tools/libxc/xc_dom_arm.c
index 122d0e8..9099cad 100644
--- a/tools/libxc/xc_dom_arm.c
+++ b/tools/libxc/xc_dom_arm.c
@@ -18,14 +18,138 @@
  * Copyright (c) 2011, Citrix Systems
  */
 #include <inttypes.h>
+
 #include <xen/xen.h>
+#include <xen/io/protocols.h>
+
 #include "xg_private.h"
 #include "xc_dom.h"
 
+/* ------------------------------------------------------------------------ */
+/*
+ * arm guests are hybrid and start off with paging disabled, therefore no
+ * pagetables and nothing to do here.
+ */
+static int count_pgtables_arm(struct xc_dom_image *dom)
+{
+    DOMPRINTF_CALLED(dom->xch);
+    return 0;
+}
+
+static int setup_pgtables_arm(struct xc_dom_image *dom)
+{
+    DOMPRINTF_CALLED(dom->xch);
+    return 0;
+}
+
+/* ------------------------------------------------------------------------ */
+
+static int alloc_magic_pages(struct xc_dom_image *dom)
+{
+    DOMPRINTF_CALLED(dom->xch);
+    /* XXX
+     *   dom->p2m_guest
+     *   dom->start_info_pfn
+     *   dom->xenstore_pfn
+     *   dom->console_pfn
+     */
+    return 0;
+}
+
+/* ------------------------------------------------------------------------ */
+
+static int start_info_arm(struct xc_dom_image *dom)
+{
+    DOMPRINTF_CALLED(dom->xch);
+    /* XXX */
+    return 0;
+}
+
+static int shared_info_arm(struct xc_dom_image *dom, void *ptr)
+{
+    DOMPRINTF_CALLED(dom->xch);
+    /* XXX */
+    return 0;
+}
+
+/* ------------------------------------------------------------------------ */
+
+static int vcpu_arm(struct xc_dom_image *dom, void *ptr)
+{
+    vcpu_guest_context_t *ctxt = ptr;
+
+    DOMPRINTF_CALLED(dom->xch);
+
+    /* clear everything */
+    memset(ctxt, 0, sizeof(*ctxt));
+
+    ctxt->user_regs.pc = dom->parms.virt_entry;
+    ctxt->user_regs.r0 = 0; /* SBZ */
+    ctxt->user_regs.r1 = 2272; /* Machine NR: Versatile Express */
+
+    ctxt->user_regs.r2 = 0xffffffff; //devicetree_seg //dtb_paddr; //atags or dtb /* XXX using APPEND right now */
+    ctxt->user_regs.r3 = 0xdeadbeef;
+    ctxt->sctlr = /* #define SCTLR_BASE */0x00c50078;
+    ctxt->ttbr0 = 0;
+    ctxt->ttbr1 = 0;
+    ctxt->ttbcr = 0; /* Defined Reset Value */
+
+    ctxt->user_regs.cpsr = PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC;
+
+    DOMPRINTF("Initial state CPSR %#"PRIx32" PC %#"PRIx32,
+           ctxt->user_regs.cpsr, ctxt->user_regs.pc);
+
+    return 0;
+}
+
+/* ------------------------------------------------------------------------ */
+
+static struct xc_dom_arch xc_dom_32 = {
+    .guest_type = "xen-3.0-armv7l",
+    .native_protocol = XEN_IO_PROTO_ABI_ARM,
+    .page_shift = PAGE_SHIFT_ARM,
+    .sizeof_pfn = 8,
+    .alloc_magic_pages = alloc_magic_pages,
+    .count_pgtables = count_pgtables_arm,
+    .setup_pgtables = setup_pgtables_arm,
+    .start_info = start_info_arm,
+    .shared_info = shared_info_arm,
+    .vcpu = vcpu_arm,
+};
+
+static void __init register_arch_hooks(void)
+{
+    xc_dom_register_arch_hooks(&xc_dom_32);
+}
+
 int arch_setup_meminit(struct xc_dom_image *dom)
 {
-    errno = ENOSYS;
-    return -1;
+    int rc;
+    xen_pfn_t pfn, allocsz, i;
+
+    dom->shadow_enabled = 1;
+
+    dom->p2m_host = xc_dom_malloc(dom, sizeof(xen_pfn_t) * dom->total_pages);
+
+    /* setup initial p2m */
+    for ( pfn = 0; pfn < dom->total_pages; pfn++ )
+        dom->p2m_host[pfn] = pfn + dom->rambase_pfn;
+
+    /* allocate guest memory */
+    for ( i = rc = allocsz = 0;
+          (i < dom->total_pages) && !rc;
+          i += allocsz )
+    {
+        allocsz = dom->total_pages - i;
+        if ( allocsz > 1024*1024 )
+            allocsz = 1024*1024;
+
+        rc = xc_domain_populate_physmap_exact(
+            dom->xch, dom->guest_domid, allocsz,
+            0, 0, &dom->p2m_host[i]);
+    }
+
+    return 0;
 }
 
 int arch_setup_bootearly(struct xc_dom_image *dom)
@@ -36,9 +160,14 @@ int arch_setup_bootearly(struct xc_dom_image *dom)
 
 int arch_setup_bootlate(struct xc_dom_image *dom)
 {
-    DOMPRINTF("%s: doing nothing", __FUNCTION__);
+    /* XXX
+     *   map shared info
+     *   map grant tables
+     *   setup shared info
+     */
     return 0;
 }
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxc/xc_dom_armzimageloader.c b/tools/libxc/xc_dom_armzimageloader.c
new file mode 100644
index 0000000..220176d
--- /dev/null
+++ b/tools/libxc/xc_dom_armzimageloader.c
@@ -0,0 +1,167 @@
+/*
+ * Xen domain builder -- ARM zImage bits
+ *
+ * Parse and load ARM zImage kernel images.
+ *
+ * Copyright (C) 2012, Citrix Systems.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation;
+ * version 2.1 of the License.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <inttypes.h>
+
+#include "xg_private.h"
+#include "xc_dom.h"
+
+#include <arpa/inet.h> /* XXX ntohl is not the right function... */
+
+#define ZIMAGE_MAGIC_OFFSET 0x24
+#define ZIMAGE_START_OFFSET 0x28
+#define ZIMAGE_END_OFFSET   0x2c
+
+#define ZIMAGE_MAGIC 0x016f2818
+
+struct minimal_dtb_header {
+    uint32_t magic;
+    uint32_t total_size;
+    /* There are other fields but we don't use them yet. */
+};
+
+#define DTB_MAGIC 0xd00dfeed
+
+static int xc_dom_probe_zimage_kernel(struct xc_dom_image *dom)
+{
+    uint32_t *zimage;
+    uint32_t end;
+
+    if ( dom->kernel_blob == NULL )
+    {
+        xc_dom_panic(dom->xch, XC_INTERNAL_ERROR,
+                     "%s: no kernel image loaded", __FUNCTION__);
+        return -EINVAL;
+    }
+
+    if ( dom->kernel_size < 0x30 /*sizeof(struct setup_header)*/ )
+    {
+        xc_dom_printf(dom->xch, "%s: kernel image too small", __FUNCTION__);
+        return -EINVAL;
+    }
+
+    zimage = (uint32_t *)dom->kernel_blob;
+    if ( zimage[ZIMAGE_MAGIC_OFFSET/4] != ZIMAGE_MAGIC )
+    {
+        xc_dom_printf(dom->xch, "%s: kernel is not a bzImage", __FUNCTION__);
+        return -EINVAL;
+    }
+
+    end = zimage[ZIMAGE_END_OFFSET/4];
+
+    /*
+     * Check for an appended DTB.
+     */
+    if ( end + sizeof(struct minimal_dtb_header) < dom->kernel_size ) {
+        struct minimal_dtb_header *dtb_hdr;
+        dtb_hdr = (struct minimal_dtb_header *)(dom->kernel_blob + end);
+        if (ntohl/*be32_to_cpu*/(dtb_hdr->magic) == DTB_MAGIC) {
+            xc_dom_printf(dom->xch, "%s: found an appended DTB", __FUNCTION__);
+            end += ntohl/*be32_to_cpu*/(dtb_hdr->total_size);
+        }
+    }
+
+    dom->kernel_size = end;
+
+    return 0;
+}
+
+static int xc_dom_parse_zimage_kernel(struct xc_dom_image *dom)
+{
+    uint32_t *zimage;
+    uint32_t start, entry_addr;
+    uint64_t v_start, v_end;
+    uint64_t rambase = 0x80000000; /* XXX */
+
+    DOMPRINTF_CALLED(dom->xch);
+
+    zimage = (uint32_t *)dom->kernel_blob;
+
+    dom->rambase_pfn = rambase >> XC_PAGE_SHIFT;
+
+    v_start = rambase + 0x8000; /* XXX */
+    v_end = v_start + dom->kernel_size;
+
+    start = zimage[ZIMAGE_START_OFFSET/4];
+
+    if (start == 0)
+        entry_addr = v_start;
+    else
+        entry_addr = start;
+
+    /* find kernel segment */
+    dom->kernel_seg.vstart = v_start;
+    dom->kernel_seg.vend   = v_end;
+
+    dom->parms.virt_entry = entry_addr;
+
+    dom->guest_type = "xen-3.0-armv7l";
+    DOMPRINTF("%s: %s: RAM starts at %"PRI_xen_pfn,
+              __FUNCTION__, dom->guest_type, dom->rambase_pfn);
+    DOMPRINTF("%s: %s: 0x%" PRIx64 " -> 0x%" PRIx64 "",
+              __FUNCTION__, dom->guest_type,
+              dom->kernel_seg.vstart, dom->kernel_seg.vend);
+    return 0;
+}
+
+static int xc_dom_load_zimage_kernel(struct xc_dom_image *dom)
+{
+    void *dst;
+
+    DOMPRINTF_CALLED(dom->xch);
+
+    dst = xc_dom_seg_to_ptr(dom, &dom->kernel_seg);
+
+    DOMPRINTF("%s: kernel sed %#"PRIx64"-%#"PRIx64,
+              __func__, dom->kernel_seg.vstart, dom->kernel_seg.vend);
+    DOMPRINTF("%s: copy %zd bytes from blob %p to dst %p",
+              __func__, dom->kernel_size, dom->kernel_blob, dst);
+
+    memcpy(dst, dom->kernel_blob, dom->kernel_size);
+
+    return 0;
+}
+
+static struct xc_dom_loader zimage_loader = {
+    .name = "Linux zImage (ARM)",
+    .probe = xc_dom_probe_zimage_kernel,
+    .parser = xc_dom_parse_zimage_kernel,
+    .loader = xc_dom_load_zimage_kernel,
+};
+
+static void __init register_loader(void)
+{
+    xc_dom_register_loader(&zimage_loader);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxc/xc_dom_core.c b/tools/libxc/xc_dom_core.c
index fea9de5..b0d48d5 100644
--- a/tools/libxc/xc_dom_core.c
+++ b/tools/libxc/xc_dom_core.c
@@ -307,15 +307,17 @@ void *xc_dom_pfn_to_ptr(struct xc_dom_image *dom, xen_pfn_t pfn,
                         xen_pfn_t count)
 {
     struct xc_dom_phys *phys;
+    xen_pfn_t offset;
     unsigned int page_shift = XC_DOM_PAGE_SHIFT(dom);
     char *mode = "unset";
 
-    if ( pfn > dom->total_pages ||    /* multiple checks to avoid overflows */
+    offset = pfn-dom->rambase_pfn;
+    if ( offset > dom->total_pages ||    /* multiple checks to avoid overflows */
          count > dom->total_pages ||
-         pfn > dom->total_pages - count )
+         offset > dom->total_pages - count )
     {
-        DOMPRINTF("%s: pfn out of range (0x%" PRIpfn " > 0x%" PRIpfn ")",
-                  __FUNCTION__, pfn, dom->total_pages);
+        DOMPRINTF("%s: pfn %"PRI_xen_pfn" out of range (0x%" PRIpfn " > 0x%" PRIpfn ")",
+                  __FUNCTION__, pfn, offset, dom->total_pages);
         return NULL;
     }
 
@@ -599,6 +601,8 @@ struct xc_dom_image *xc_dom_allocate(xc_interface *xch,
     dom->parms.virt_hv_start_low = UNSET_ADDR;
     dom->parms.elf_paddr_offset = UNSET_ADDR;
 
+    dom->rambase_pfn = 0;
+
     dom->alloc_malloc += sizeof(*dom);
     return dom;
 
diff --git a/tools/libxc/xg_private.h b/tools/libxc/xg_private.h
index a29fa26..a271942 100644
--- a/tools/libxc/xg_private.h
+++ b/tools/libxc/xg_private.h
@@ -148,6 +148,10 @@ typedef l4_pgentry_64_t l4_pgentry_t;
 #define l4_table_offset(_a) l4_table_offset_x86_64(_a)
 #endif
 
+#define PAGE_SHIFT_ARM          12
+#define PAGE_SIZE_ARM           (1UL << PAGE_SHIFT_ARM)
+#define PAGE_MASK_ARM           (~(PAGE_SIZE_ARM-1))
+
 #define PAGE_SHIFT_X86          12
 #define PAGE_SIZE_X86           (1UL << PAGE_SHIFT_X86)
 #define PAGE_MASK_X86           (~(PAGE_SIZE_X86-1))
-- 
1.7.9.1

[PATCH 37/38] HACK: add simple xcbuild

[Ian Campbell]

Based on init-xenstore-domain.c.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/xcutils/Makefile  |    6 ++-
 tools/xcutils/xcbuild.c |  100 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 105 insertions(+), 1 deletions(-)
 create mode 100644 tools/xcutils/xcbuild.c

diff --git a/tools/xcutils/Makefile b/tools/xcutils/Makefile
index 6c502f1..dcd2c84 100644
--- a/tools/xcutils/Makefile
+++ b/tools/xcutils/Makefile
@@ -11,7 +11,7 @@
 XEN_ROOT	= $(CURDIR)/../..
 include $(XEN_ROOT)/tools/Rules.mk
 
-PROGRAMS = xc_restore xc_save readnotes lsevtchn
+PROGRAMS = xc_restore xc_save readnotes lsevtchn xcbuild
 
 CFLAGS += -Werror
 
@@ -19,6 +19,7 @@ CFLAGS_xc_restore.o := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
 CFLAGS_xc_save.o    := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore)
 CFLAGS_readnotes.o  := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
 CFLAGS_lsevtchn.o   := $(CFLAGS_libxenctrl)
+CFLAGS_xcbuild.o    := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
 
 .PHONY: all
 all: build
@@ -32,6 +33,9 @@ xc_restore: xc_restore.o
 xc_save: xc_save.o
 	$(CC) $(LDFLAGS) $^ -o $@ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
 
+xcbuild: xcbuild.o
+	$(CC) $(LDFLAGS) $^ -o $@ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
+
 readnotes: readnotes.o
 	$(CC) $(LDFLAGS) $^ -o $@ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
 
diff --git a/tools/xcutils/xcbuild.c b/tools/xcutils/xcbuild.c
new file mode 100644
index 0000000..8f8660e
--- /dev/null
+++ b/tools/xcutils/xcbuild.c
@@ -0,0 +1,100 @@
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+#include <errno.h>
+
+#include <xenctrl.h>
+#include <xentoollog.h>
+#include <xc_dom.h>
+
+int main(int argc, char **argv)
+{
+	xentoollog_logger *logger;
+	xc_interface *xch;
+	int rv;
+	const char *image;
+	uint32_t domid;
+	xen_domain_handle_t handle;
+	int maxmem = 128; /* MB */ //atoi(argv[2]);
+	int memory_kb = 2*(maxmem + 1)*1024; /* bit of slack... */
+	struct xc_dom_image *dom;
+
+	image = (argc < 2) ? "guest.img" : argv[1];
+	printf("Image: %s\n", image);
+	printf("Memory: %dKB\n", memory_kb);
+
+	logger = (xentoollog_logger*)
+		xtl_createlogger_stdiostream(stderr, XTL_DEBUG, 0);
+	if ( logger == NULL )
+	{
+		perror("xtl_createlogger_stdiostream");
+		exit(1);
+	}
+
+	xch = xc_interface_open(logger, logger, 0);
+	if ( xch == NULL )
+	{
+		perror("xc_interface_open");
+		exit(1);
+	}
+
+	rv = xc_dom_loginit(xch);
+	if (rv) return rv;
+
+	//rv = xc_flask_context_to_sid(xch, argv[3], strlen(argv[3]), &ssid);
+	//if (rv) return rv;
+
+	rv = xc_domain_create(xch, 0 /* ssid */, handle, 0 /* flags */, &domid);
+	printf("xc_domain_create: %d (%d)\n", rv, errno);
+	if ( rv < 0 )
+	{
+		perror("xc_domain_create");
+		exit(1);
+	}
+
+	printf("building dom%d\n", domid);
+
+	rv = xc_domain_max_vcpus(xch, domid, 1);
+	if ( rv < 0)
+	{
+		perror("xc_domain_max_vcpus");
+		exit(1);
+	}
+
+	rv = xc_domain_setmaxmem(xch, domid, memory_kb);
+	if ( rv < 0)
+	{
+		perror("xc_domain_setmaxmem");
+		exit(1);
+	}
+
+	dom = xc_dom_allocate(xch, "", NULL);
+	rv = xc_dom_kernel_file(dom, image);
+	if (rv) return rv;
+	rv = xc_dom_boot_xen_init(dom, xch, domid);
+	if (rv) return rv;
+	rv = xc_dom_parse_image(dom);
+	if (rv) return rv;
+	rv = xc_dom_mem_init(dom, 2*maxmem);/* XXX */
+	if (rv) return rv;
+	rv = xc_dom_boot_mem_init(dom);
+	if (rv) return rv;
+	rv = xc_dom_build_image(dom);
+	if (rv) return rv;
+	rv = xc_dom_boot_image(dom);
+	if (rv) return rv;
+
+	xc_dom_release(dom);
+
+	rv = xc_domain_unpause(xch, domid);
+	if ( rv )
+	{
+		perror("xc_domain_unpause");
+		exit(1);
+	}
+
+	xc_interface_close(xch);
+
+	return 0;
+}
-- 
1.7.9.1

[PATCH 38/38] HACK: arm: disable hypercall continuations.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/include/xen/sched.h |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 53804c8..15fa6b4 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -577,10 +577,14 @@ unsigned long hypercall_create_continuation(
     unsigned int op, const char *format, ...);
 void hypercall_cancel_continuation(void);
 
+#ifdef CONFIG_ARM
+#define hypercall_preempt_check() (0)
+#else
 #define hypercall_preempt_check() (unlikely(    \
         softirq_pending(smp_processor_id()) |   \
         local_events_need_delivery()            \
     ))
+#endif
 
 extern struct domain *domain_list;
 
-- 
1.7.9.1

Re: [PATCH 23/38] arm: use correct attributes for mappings in copy_from_paddr()

[David Vrabel]

On 01/06/12 16:39, Ian Campbell wrote:
> The DTB is in RAM (hence bufferable), kernel is in flash and therefor requires
> a device type mapping (hence dev shared).
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Potentially the bootloader could provide the DTB in flash but this seems
unlikely.

Acked-by: David Vrabel <david.vrabel@citrix.com>

David

Re: [PATCH 0/38] arm: boot a dom1 to "Calibrating delay loop" then hang

[Ian Campbell]

On Fri, 2012-06-01 at 16:38 +0100, Ian Campbell wrote:
> I still need to cleanup the Linux side of this, that probably won't
> happen today and these patches probably don't work without those
> changes... 

I've pushed my dev tree, which contains several hacks and lots of stuff
to cleanup, to:
        git://xenbits.xen.org/people/ianc/linux-2.6.git devel/arm-hacks

It's based of David's[0] vexpress-dt branch + Stefanos's[1]
vexpress-dt-privcmd branch.

It contains a backport of some old version of Mark Zynger's arch_timers
stuff, a very hacky XENMAPSPACE_gmfn_foreign, loads of mess around UARTs
and device tree stuff which make no sense etc etc.

At the moment you need CONFIG_ARM_ARCH_TIMER=y for domU and =n for
dom0... 

Nobody look too closely ;-)

Ian.

[0] git://xenbits.xen.org/people/dvrabel/linux.git
[1] git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git

Re: [PATCH 29/38] arm: delay enabling data-cache until paging enabled.

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565198), Ian Campbell wrote:
> With enough warnings enabled the model seemed to be complaining that pages
> cached before paging was enabled had been mapped with to inconsistent sets of
> attributes. I'm not convinced that isn't a model issue, nor am I convinced
> this has really fixed anything, but it seems sensible enough.

This might be what breaks secondary CPU bringup: pagetables built by CPU
0 may not have been flushed all the way to RAM when CPU 1 comes up, and
CPU 1 isn't participating in cache coherence protocols when it
starts to need them.

OTOH, if that's the case, it's surprising that CPU 1 passes the boot
gate.  I'll look into it next week, x86/mm workload permitting. 

Cheers,

Tim.

> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/head.S |    9 +++++++--
>  1 files changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/head.S b/xen/arch/arm/head.S
> index 9a7714a..71197af 100644
> --- a/xen/arch/arm/head.S
> +++ b/xen/arch/arm/head.S
> @@ -148,10 +148,11 @@ hyp:
>  	 * Exceptions in LE ARM,
>  	 * Low-latency IRQs disabled,
>  	 * Write-implies-XN disabled (for now),
> -	 * I-cache and d-cache enabled,
> +	 * D-cache diabled (for now),
> +	 * I-cache enabled,
>  	 * Alignment checking enabled,
>  	 * MMU translation disabled (for now). */
> -	ldr   r0, =(HSCTLR_BASE|SCTLR_A|SCTLR_C)
> +	ldr   r0, =(HSCTLR_BASE|SCTLR_A)
>  	mcr   CP32(r0, HSCTLR)
>  
>  	/* Write Xen's PT's paddr into the HTTBR */
> @@ -217,6 +218,10 @@ pt_ready:
>  	mov   pc, r1                 /* Get a proper vaddr into PC */
>  paging:
>  
> +	mrc   CP32(r0, HSCTLR)       /* Now enable data cache */
> +	orr   r0, r0, #(SCTLR_C)
> +	mcr   CP32(r0, HSCTLR)
> +
>  #ifdef EARLY_UART_ADDRESS
>  	/* Recover the UART address in the new address space. */
>  	lsl   r11, #11
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH 29/38] arm: delay enabling data-cache until paging enabled.

[Ian Campbell]

On Fri, 2012-06-01 at 18:05 +0100, Tim Deegan wrote:
> At 15:39 +0000 on 01 Jun (1338565198), Ian Campbell wrote:
> > With enough warnings enabled the model seemed to be complaining that pages
> > cached before paging was enabled had been mapped with to inconsistent sets of
> > attributes. I'm not convinced that isn't a model issue, nor am I convinced
> > this has really fixed anything, but it seems sensible enough.
> 
> This might be what breaks secondary CPU bringup: pagetables built by CPU
> 0 may not have been flushed all the way to RAM when CPU 1 comes up, and
> CPU 1 isn't participating in cache coherence protocols when it
> starts to need them.

The issue here is the lack of the necessary flush, rather than this
change particularly, right?

> OTOH, if that's the case, it's surprising that CPU 1 passes the boot
> gate.  I'll look into it next week, x86/mm workload permitting. 

Sounds good!

I tried to test SMP yesterday but ut turned out all the models I thought
were SMP enabled were actual UP Only, not sure how/when that happened!

Cheers,
Ian.

> 
> Cheers,
> 
> Tim.
> 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/head.S |    9 +++++++--
> >  1 files changed, 7 insertions(+), 2 deletions(-)
> > 
> > diff --git a/xen/arch/arm/head.S b/xen/arch/arm/head.S
> > index 9a7714a..71197af 100644
> > --- a/xen/arch/arm/head.S
> > +++ b/xen/arch/arm/head.S
> > @@ -148,10 +148,11 @@ hyp:
> >  	 * Exceptions in LE ARM,
> >  	 * Low-latency IRQs disabled,
> >  	 * Write-implies-XN disabled (for now),
> > -	 * I-cache and d-cache enabled,
> > +	 * D-cache diabled (for now),
> > +	 * I-cache enabled,
> >  	 * Alignment checking enabled,
> >  	 * MMU translation disabled (for now). */
> > -	ldr   r0, =(HSCTLR_BASE|SCTLR_A|SCTLR_C)
> > +	ldr   r0, =(HSCTLR_BASE|SCTLR_A)
> >  	mcr   CP32(r0, HSCTLR)
> >  
> >  	/* Write Xen's PT's paddr into the HTTBR */
> > @@ -217,6 +218,10 @@ pt_ready:
> >  	mov   pc, r1                 /* Get a proper vaddr into PC */
> >  paging:
> >  
> > +	mrc   CP32(r0, HSCTLR)       /* Now enable data cache */
> > +	orr   r0, r0, #(SCTLR_C)
> > +	mcr   CP32(r0, HSCTLR)
> > +
> >  #ifdef EARLY_UART_ADDRESS
> >  	/* Recover the UART address in the new address space. */
> >  	lsl   r11, #11
> > -- 
> > 1.7.9.1
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel

Re: [PATCH 19/38] arm: context switch a bunch of guest state.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 1a2b95f..339c327 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -61,6 +61,30 @@ static struct {
>  irq_desc_t irq_desc[NR_IRQS];
>  unsigned nr_lrs;
> 
> +void gic_save_state(struct vcpu *v)
> +{
> +    int i;
> +
> +    for ( i=0; i<nr_lrs; i++)
> +        v->arch.gic_lr[i] = GICH[GICH_LR + i];
> +    /* Disable until next VCPU scheduled */
> +    GICH[GICH_HCR] = 0;
> +    isb();
> +}
> +
> +void gic_restore_state(struct vcpu *v)
> +{
> +    int i;
> +
> +    if ( is_idle_vcpu(v) )
> +        return;
> +
> +    for ( i=0; i<nr_lrs; i++)
> +        GICH[GICH_LR + i] = v->arch.gic_lr[i];
> +    GICH[GICH_HCR] = GICH_HCR_EN;
> +    isb();
> +}
> +

it is still missing a bunch of stuff from the gic state but it is a step
in the right direction, so I'll send out patches to complete the gic
context switch separately, based on this one.

Re: [PATCH 04/38] arm: correct and expand TLB flush CP15 registers

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Correct spelling of TLBIALLHIS and correct definition of TLBIALLNSNHIS.
> 
> Add a few more.
>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---

Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/include/asm-arm/cpregs.h |   11 +++++++++--
>  1 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/include/asm-arm/cpregs.h b/xen/include/asm-arm/cpregs.h
> index ee8a287..7a0b49a 100644
> --- a/xen/include/asm-arm/cpregs.h
> +++ b/xen/include/asm-arm/cpregs.h
> @@ -172,12 +172,19 @@
>  #define TLBIMVAIS       p15,0,c8,c3,1   /* Invalidate unified TLB entry by MVA inner shareable */
>  #define TLBIASIDIS      p15,0,c8,c3,2   /* Invalidate unified TLB by ASID match inner shareable */
>  #define TLBIMVAAIS      p15,0,c8,c3,3   /* Invalidate unified TLB entry by MVA all ASID inner shareable */
> +#define ITLBIALL        p15,0,c8,c5,0   /* Invalidate instruction TLB */
> +#define ITLBIMVA        p15,0,c8,c5,1   /* Invalidate instruction TLB entry by MVA */
> +#define ITLBIASID       p15,0,c8,c5,2   /* Invalidate instruction TLB by ASID match */
>  #define DTLBIALL        p15,0,c8,c6,0   /* Invalidate data TLB */
>  #define DTLBIMVA        p15,0,c8,c6,1   /* Invalidate data TLB entry by MVA */
>  #define DTLBIASID       p15,0,c8,c6,2   /* Invalidate data TLB by ASID match */
> -#define TLBILLHIS       p15,4,c8,c3,0   /* Invalidate Entire Hyp. Unified TLB inner shareable */
> +#define TLBIALL         p15,0,c8,c7,0   /* invalidate unified TLB */
> +#define TLBIMVA         p15,0,c8,c7,1   /* invalidate unified TLB entry by MVA */
> +#define TLBIASID        p15,0,c8,c7,2   /* invalid unified TLB by ASID match */
> +#define TLBIMVAA        p15,0,c8,c7,3   /* invalidate unified TLB entries by MVA all ASID */
> +#define TLBIALLHIS      p15,4,c8,c3,0   /* Invalidate Entire Hyp. Unified TLB inner shareable */
>  #define TLBIMVAHIS      p15,4,c8,c3,1   /* Invalidate Unified Hyp. TLB by MVA inner shareable */
> -#define TLBIALLNSNHIS   p15,4,c8,c7,4   /* Invalidate Entire Non-Secure Non-Hyp. Unified TLB inner shareable */
> +#define TLBIALLNSNHIS   p15,4,c8,c3,4   /* Invalidate Entire Non-Secure Non-Hyp. Unified TLB inner shareable */
>  #define TLBIALLH        p15,4,c8,c7,0   /* Invalidate Entire Hyp. Unified TLB */
>  #define TLBIMVAH        p15,4,c8,c7,1   /* Invalidate Unified Hyp. TLB by MVA */
>  #define TLBIALLNSNH     p15,4,c8,c7,4   /* Invalidate Entire Non-Secure Non-Hyp. Unified TLB */
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 05/38] arm: restore stack on return from trap.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> We align the stack before calling into C code but we weren't undoing this on
> return.
> 
> Collapse continue_(non)idle_domain into continue_new_vcpu.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/domain.c |   16 +++-------------
>  xen/arch/arm/entry.S  |    5 ++++-
>  2 files changed, 7 insertions(+), 14 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 4b38790..9339a11 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -16,17 +16,6 @@
>  
>  DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>  
> -static void continue_idle_domain(struct vcpu *v)
> -{
> -    reset_stack_and_jump(idle_loop);
> -}
> -
> -static void continue_nonidle_domain(struct vcpu *v)
> -{
> -    /* check_wakeup_from_wait(); */
> -    reset_stack_and_jump(return_from_trap);
> -}
> -
>  void idle_loop(void)
>  {
>      for ( ; ; )
> @@ -72,9 +61,10 @@ static void continue_new_vcpu(struct vcpu *prev)
>      schedule_tail(prev);
>  
>      if ( is_idle_vcpu(current) )
> -        continue_idle_domain(current);
> +        reset_stack_and_jump(idle_loop);
>      else
> -        continue_nonidle_domain(current);
> +        /* check_wakeup_from_wait(); */
> +        reset_stack_and_jump(return_to_new_vcpu);
>  }
>  
>  void context_switch(struct vcpu *prev, struct vcpu *next)
> diff --git a/xen/arch/arm/entry.S b/xen/arch/arm/entry.S
> index f261a9f..7a22e2d 100644
> --- a/xen/arch/arm/entry.S
> +++ b/xen/arch/arm/entry.S
> @@ -72,7 +72,9 @@ DEFINE_TRAP_ENTRY(hypervisor)
>  DEFINE_TRAP_ENTRY(irq)
>  DEFINE_TRAP_ENTRY(fiq)
>  
> -ENTRY(return_from_trap)
> +return_from_trap:
> +	mov sp, r11
> +ENTRY(return_to_new_vcpu)
>  	ldr r11, [sp, #UREGS_cpsr]
>  	and r11, #PSR_MODE_MASK
>  	cmp r11, #PSR_MODE_HYP
> @@ -82,6 +84,7 @@ ENTRY(return_to_guest)
>  	mov r11, sp
>  	bic sp, #7 /* Align the stack pointer */
>  	bl leave_hypervisor_tail
> +	mov sp, r11
>  	RESTORE_ONE_BANKED(SP_usr)
>  	/* LR_usr is the same physical register as lr and is restored below */
>  	RESTORE_BANKED(svc)
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 06/38] arm: enable interrupts while handling traps

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> For most traps we can do this as soon as we have saved the necessary state.
> For IRQs and FIQs we must wait until we have acked the interrupt with the GIC.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/entry.S |   17 ++++++++++++++---
>  xen/arch/arm/gic.c   |    2 ++
>  xen/arch/arm/traps.c |    1 -
>  3 files changed, 16 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/entry.S b/xen/arch/arm/entry.S
> index 7a22e2d..5bc3906 100644
> --- a/xen/arch/arm/entry.S
> +++ b/xen/arch/arm/entry.S
> @@ -46,6 +46,17 @@ save_guest_regs:
>  	ALIGN;												\
>  trap_##trap:												\
>  	SAVE_ALL;											\
> +	cpsie i; 	/* local_irq_enable */								\
> +	adr lr, return_from_trap;									\
> +	mov r0, sp;											\
> +	mov r11, sp;											\
> +	bic sp, #7; /* Align the stack pointer (noop on guest trap) */					\
> +	b do_trap_##trap
> +
> +#define DEFINE_TRAP_ENTRY_NOIRQ(trap)									\
> +	ALIGN;												\
> +trap_##trap:												\
> +	SAVE_ALL;											\
>  	adr lr, return_from_trap;									\
>  	mov r0, sp;											\
>  	mov r11, sp;											\
> @@ -69,8 +80,8 @@ DEFINE_TRAP_ENTRY(supervisor_call)
>  DEFINE_TRAP_ENTRY(prefetch_abort)
>  DEFINE_TRAP_ENTRY(data_abort)
>  DEFINE_TRAP_ENTRY(hypervisor)
> -DEFINE_TRAP_ENTRY(irq)
> -DEFINE_TRAP_ENTRY(fiq)
> +DEFINE_TRAP_ENTRY_NOIRQ(irq)
> +DEFINE_TRAP_ENTRY_NOIRQ(fiq)
>  
>  return_from_trap:
>  	mov sp, r11
> @@ -83,7 +94,7 @@ ENTRY(return_to_new_vcpu)
>  ENTRY(return_to_guest)
>  	mov r11, sp
>  	bic sp, #7 /* Align the stack pointer */
> -	bl leave_hypervisor_tail
> +	bl leave_hypervisor_tail /* Disables interrupts on return */
>  	mov sp, r11
>  	RESTORE_ONE_BANKED(SP_usr)
>  	/* LR_usr is the same physical register as lr and is restored below */
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index cc9d37b..1a2b95f 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -509,6 +509,8 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
>      uint32_t intack = GICC[GICC_IAR];
>      unsigned int irq = intack & GICC_IA_IRQ;
>  
> +    local_irq_enable();
> +
>      if ( irq == 1023 )
>          /* Spurious interrupt */
>          return;
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index abc26a3..5ed754f 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -412,7 +412,6 @@ static void do_debug_trap(struct cpu_user_regs *regs, unsigned int code)
>  static void do_trap_hypercall(struct cpu_user_regs *regs, unsigned long iss)
>  {
>      arm_hypercall_t *call = NULL;
> -    local_irq_enable();
>  
>      if ( iss != XEN_HYPERCALL_TAG )
>      {
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 07/38] arm: hook up domctl and memory_op

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/traps.c |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 5ed754f..5d8b7f9 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -373,6 +373,8 @@ typedef unsigned long arm_hypercall_t(
>      [ __HYPERVISOR_ ## x ] = (arm_hypercall_t *) do_ ## x
>  
>  static arm_hypercall_t *arm_hypercall_table[] = {
> +    HYPERCALL(memory_op),
> +    HYPERCALL(domctl),
>      HYPERCALL(arch_0),
>      HYPERCALL(sched_op),
>      HYPERCALL(console_io),
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 08/38] arm: allocate and setup a guest vcpu.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/domain.c         |   68 +++++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/dummy.S          |    3 --
>  xen/include/public/arch-arm.h |    9 -----
>  3 files changed, 68 insertions(+), 12 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 9339a11..62a2f3a 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -144,6 +144,17 @@ void free_vcpu_struct(struct vcpu *v)
>      free_xenheap_page(v);
>  }
>  
> +struct vcpu_guest_context *alloc_vcpu_guest_context(void)
> +{
> +    return xmalloc(struct vcpu_guest_context);
> +
> +}
> +
> +void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
> +{
> +    xfree(vgc);
> +}
> +
>  int vcpu_initialise(struct vcpu *v)
>  {
>      int rc = 0;
> @@ -182,6 +193,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
>      if ( (rc = p2m_init(d)) != 0 )
>          goto fail;
>  
> +    if ( (rc = domain_vgic_init(d)) != 0 )
> +        goto fail;
> +

there is a call to domain_vgic_init already in arch_domain_create


>      if ( !is_idle_domain(d) )
>      {
>          rc = -ENOMEM;
> @@ -212,6 +226,60 @@ void arch_domain_destroy(struct domain *d)
>      /* domain_vgic_destroy */
>  }
>  
> +static int is_guest_psr(uint32_t psr)
> +{
> +    switch (psr & PSR_MODE_MASK)
> +    {
> +    case PSR_MODE_USR:
> +    case PSR_MODE_FIQ:
> +    case PSR_MODE_IRQ:
> +    case PSR_MODE_SVC:
> +    case PSR_MODE_ABT:
> +    case PSR_MODE_UND:
> +    case PSR_MODE_SYS:
> +        return 1;
> +    case PSR_MODE_MON:
> +    case PSR_MODE_HYP:
> +    default:
> +        return 0;
> +    }
> +}
> +
> +int arch_set_info_guest(
> +    struct vcpu *v, vcpu_guest_context_u c)
> +{
> +    struct cpu_user_regs *regs = &c.nat->user_regs;
> +
> +    if ( !is_guest_psr(regs->cpsr) )
> +        return -EINVAL;
> +
> +    if ( regs->spsr_svc && !is_guest_psr(regs->spsr_svc) )
> +        return -EINVAL;
> +    if ( regs->spsr_abt && !is_guest_psr(regs->spsr_abt) )
> +        return -EINVAL;
> +    if ( regs->spsr_und && !is_guest_psr(regs->spsr_und) )
> +        return -EINVAL;
> +    if ( regs->spsr_irq && !is_guest_psr(regs->spsr_irq) )
> +        return -EINVAL;
> +    if ( regs->spsr_fiq && !is_guest_psr(regs->spsr_fiq) )
> +        return -EINVAL;
> +
> +    v->arch.cpu_info->guest_cpu_user_regs = *regs;
> +
> +    /* XXX other state:
> +     * - SCTLR
> +     * - TTBR0/1
> +     * - TTBCR
> +     */
> +
> +    //if ( flags & VGCF_online )
> +        clear_bit(_VPF_down, &v->pause_flags);
> +    //else
> +    //    set_bit(_VPF_down, &v->pause_flags);
> +
> +    return 0;
> +}

Do we really need to add commented out code like this?
Also arch_set_info_guest could benefit by a couple of lines of comments
to explain what it is supposed to do.


>  void arch_dump_domain_info(struct domain *d)
>  {
>  }
> diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
> index 016340c..3b48917 100644
> --- a/xen/arch/arm/dummy.S
> +++ b/xen/arch/arm/dummy.S
> @@ -20,11 +20,8 @@ DUMMY(pirq_guest_unbind);
>  DUMMY(pirq_set_affinity);
>  
>  /* VCPU */
> -DUMMY(alloc_vcpu_guest_context);
>  DUMMY(arch_get_info_guest);
> -DUMMY(arch_set_info_guest);
>  DUMMY(arch_vcpu_reset);
> -DUMMY(free_vcpu_guest_context);
>  DUMMY(sync_vcpu_execstate);
>  NOP(update_vcpu_system_time);
>  DUMMY(vcpu_show_execution_state);
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 1b1bcf3..e439727 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -124,15 +124,6 @@ typedef uint32_t xen_ulong_t;
>  
>  struct vcpu_guest_context {
>      struct cpu_user_regs user_regs;         /* User-level CPU registers     */
> -    union {
> -        uint32_t reg[16];
> -        struct {
> -            uint32_t __pad[12];
> -            uint32_t sp; /* r13 */
> -            uint32_t lr; /* r14 */
> -            uint32_t pc; /* r15 */
> -        };
> -    };
>  };
>  typedef struct vcpu_guest_context vcpu_guest_context_t;
>  DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 09/38] arm: print domid as part of debug trap

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>


>  xen/arch/arm/traps.c |   11 ++++++-----
>  1 files changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 5d8b7f9..40bb375 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -388,25 +388,26 @@ static arm_hypercall_t *arm_hypercall_table[] = {
>  static void do_debug_trap(struct cpu_user_regs *regs, unsigned int code)
>  {
>      uint32_t reg, *r;
> -
> +    uint32_t domid = current->domain->domain_id;
>      switch ( code ) {
>      case 0xe0 ... 0xef:
>          reg = code - 0xe0;
>          r = &regs->r0 + reg;
> -        printk("R%d = %#010"PRIx32" at %#010"PRIx32"\n", reg, *r, regs->pc);
> +        printk("DOM%d: R%d = %#010"PRIx32" at %#010"PRIx32"\n",
> +               domid, reg, *r, regs->pc);
>          break;
>      case 0xfd:
> -        printk("Reached %08"PRIx32"\n", regs->pc);
> +        printk("DOM%d: Reached %#010"PRIx32"\n", domid, regs->pc);
>          break;
>      case 0xfe:
>          printk("%c", (char)(regs->r0 & 0xff));
>          break;
>      case 0xff:
> -        printk("DEBUG\n");
> +        printk("DOM%d: DEBUG\n", domid);
>          show_execution_state(regs);
>          break;
>      default:
> -        panic("Unhandled debug trap %#x\n", code);
> +        panic("DOM%d: Unhandled debug trap %#x\n", domid, code);
>          break;
>      }
>  }
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 10/38] arm: remove unnecessarily verbose print from p2m_load_VTTBR

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/p2m.c |    2 --
>  1 files changed, 0 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index fdbecbc..095e608 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -47,8 +47,6 @@ void p2m_load_VTTBR(struct domain *d)
>  
>      vttbr |= ((uint64_t)p2m->vmid&0xff)<<48;
>  
> -    printk("VTTBR dom%d = %"PRIx64"\n", d->domain_id, vttbr);
> -
>      WRITE_CP64(vttbr, VTTBR);
>      isb(); /* Ensure update is visible */
>  }
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 08/38] arm: allocate and setup a guest vcpu.

[Ian Campbell]

On Wed, 2012-06-06 at 14:46 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/domain.c         |   68 +++++++++++++++++++++++++++++++++++++++++
> >  xen/arch/arm/dummy.S          |    3 --
> >  xen/include/public/arch-arm.h |    9 -----
> >  3 files changed, 68 insertions(+), 12 deletions(-)
> > 
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > index 9339a11..62a2f3a 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -144,6 +144,17 @@ void free_vcpu_struct(struct vcpu *v)
> >      free_xenheap_page(v);
> >  }
> >  
> > +struct vcpu_guest_context *alloc_vcpu_guest_context(void)
> > +{
> > +    return xmalloc(struct vcpu_guest_context);
> > +
> > +}
> > +
> > +void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
> > +{
> > +    xfree(vgc);
> > +}
> > +
> >  int vcpu_initialise(struct vcpu *v)
> >  {
> >      int rc = 0;
> > @@ -182,6 +193,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
> >      if ( (rc = p2m_init(d)) != 0 )
> >          goto fail;
> >  
> > +    if ( (rc = domain_vgic_init(d)) != 0 )
> > +        goto fail;
> > +
> 
> there is a call to domain_vgic_init already in arch_domain_create

So there is!

I notice while checking that a bunch of stuff can/should be pushed under
the !idle_domain conditional, or better the idle domain case should bail
early.

> > +int arch_set_info_guest(
> > +    struct vcpu *v, vcpu_guest_context_u c)
> > +{
> > +    struct cpu_user_regs *regs = &c.nat->user_regs;
> > +
> > +    if ( !is_guest_psr(regs->cpsr) )
> > +        return -EINVAL;
> > +
> > +    if ( regs->spsr_svc && !is_guest_psr(regs->spsr_svc) )
> > +        return -EINVAL;
> > +    if ( regs->spsr_abt && !is_guest_psr(regs->spsr_abt) )
> > +        return -EINVAL;
> > +    if ( regs->spsr_und && !is_guest_psr(regs->spsr_und) )
> > +        return -EINVAL;
> > +    if ( regs->spsr_irq && !is_guest_psr(regs->spsr_irq) )
> > +        return -EINVAL;
> > +    if ( regs->spsr_fiq && !is_guest_psr(regs->spsr_fiq) )
> > +        return -EINVAL;
> > +
> > +    v->arch.cpu_info->guest_cpu_user_regs = *regs;
> > +
> > +    /* XXX other state:
> > +     * - SCTLR
> > +     * - TTBR0/1
> > +     * - TTBCR
> > +     */
> > +
> > +    //if ( flags & VGCF_online )
> > +        clear_bit(_VPF_down, &v->pause_flags);
> > +    //else
> > +    //    set_bit(_VPF_down, &v->pause_flags);
> > +
> > +    return 0;
> > +}
> 
> Do we really need to add commented out code like this?

Yeah, you're right, I copied from x86 which has this but we haven't
implemented it for ARM yet. I suppose an XXX would be better. Or maybe I
should just implement the flags...

> Also arch_set_info_guest could benefit by a couple of lines of comments
> to explain what it is supposed to do.

I'll add something.

Re: [PATCH 11/38] arm: implement p2m lookup

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/p2m.c        |   71 ++++++++++++++++++++++++++++++++++++++++++--
>  xen/include/asm-arm/p2m.h |    3 ++
>  2 files changed, 70 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index 095e608..9b40e93 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -10,10 +10,20 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
>      struct p2m_domain *p2m = &d->arch.p2m;
>      lpae_t *first = NULL, *second = NULL, *third = NULL;
>  
> -    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
> +    printk("dom%d IPA %#"PRIpaddr"\n", d->domain_id, addr);
> +
> +    printk("P2M @ %p mfn:%#lx (%#03llx,%#03llx,%#03llx)\n",
> +           p2m->first_level,
> +           page_to_mfn(p2m->first_level),
> +           first_table_offset(addr),
> +           second_table_offset(addr),
> +           third_table_offset(addr));
> +
> +    if ( first_table_offset(addr) >= LPAE_ENTRIES )
> +        goto done;
>  
>      first = __map_domain_page(p2m->first_level);
> -    printk("1ST[%#03llx] = %#016llx\n",
> +    printk("1ST[%#03llx] = %#"PRIpaddr"\n",
>             first_table_offset(addr),
>             first[first_table_offset(addr)].bits);
>      if ( !first[first_table_offset(addr)].p2m.valid ||
> @@ -21,7 +31,7 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
>          goto done;
>  
>      second = map_domain_page(first[first_table_offset(addr)].p2m.base);
> -    printk("2ND[%#03llx] = %#016llx\n",
> +    printk("2ND[%#03llx] = %#"PRIpaddr"\n",
>             second_table_offset(addr),
>             second[second_table_offset(addr)].bits);
>      if ( !second[second_table_offset(addr)].p2m.valid ||
> @@ -29,7 +39,7 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
>          goto done;
>  
>      third = map_domain_page(second[second_table_offset(addr)].p2m.base);
> -    printk("3RD[%#03llx] = %#016llx\n",
> +    printk("3RD[%#03llx] = %#"PRIpaddr"\n",
>             third_table_offset(addr),
>             third[third_table_offset(addr)].bits);
>  
> @@ -51,6 +61,59 @@ void p2m_load_VTTBR(struct domain *d)
>      isb(); /* Ensure update is visible */
>  }

there is no need to introduce p2m_lookup in the same patch you do these
unrelated printk adjustments, correct?


> +/*
> + * Lookup the MFN corresponding to a domain's PFN.
> + *
> + * There are no processor functions to do a stage 2 only lookup therefore we
> + * do a a software walk.
> + */
> +paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
> +{
> +    struct p2m_domain *p2m = &d->arch.p2m;
> +    lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
> +    paddr_t maddr = INVALID_PADDR;
> +
> +    spin_lock(&p2m->lock);
> +
> +    first = __map_domain_page(p2m->first_level);
> +    if ( !first[first_table_offset(paddr)].p2m.valid )
> +        goto done_err;
> +    if ( !first[first_table_offset(paddr)].p2m.table )
> +    {
> +        pte = first[first_table_offset(paddr)];
> +        goto done;
> +    }
> +
> +    second = map_domain_page(first[first_table_offset(paddr)].p2m.base);
> +    if ( !second[second_table_offset(paddr)].p2m.valid )
> +        goto done_err;
> +    if ( !second[second_table_offset(paddr)].p2m.table )
> +    {
> +        pte = second[second_table_offset(paddr)];
> +        goto done;
> +    }
> +
> +    third = map_domain_page(second[second_table_offset(paddr)].p2m.base);
> +    if ( !third[third_table_offset(paddr)].p2m.valid )
> +        goto done_err;
> +    if ( !third[third_table_offset(paddr)].p2m.table )
> +        goto done_err;
> +
> +    pte = third[third_table_offset(paddr)];
> +
> +done:
> +
> +    maddr = (pte.bits & PADDR_MASK & PAGE_MASK) | (paddr & ~PAGE_MASK);
> +done_err:
> +    if (third) unmap_domain_page(third);
> +    if (second) unmap_domain_page(second);
> +    if (first) unmap_domain_page(first);
> +
> +    spin_unlock(&p2m->lock);
> +
> +    return maddr;
> +}
> +

this function looks correct though

>  int guest_physmap_mark_populate_on_demand(struct domain *d,
>                                            unsigned long gfn,
>                                            unsigned int order)
> diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
> index 349923a..1afd5cb 100644
> --- a/xen/include/asm-arm/p2m.h
> +++ b/xen/include/asm-arm/p2m.h
> @@ -32,6 +32,9 @@ int p2m_alloc_table(struct domain *d);
>  /* */
>  void p2m_load_VTTBR(struct domain *d);
>  
> +/* */
> +paddr_t p2m_lookup(struct domain *d, paddr_t gpfn);
> +
>  /* Setup p2m RAM mapping for domain d from start-end. */
>  int p2m_populate_ram(struct domain *d, paddr_t start, paddr_t end);
>  /* Map MMIO regions in the p2m: start_gaddr and end_gaddr is the range
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 11/38] arm: implement p2m lookup

[Ian Campbell]

On Wed, 2012-06-06 at 15:01 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/p2m.c        |   71 ++++++++++++++++++++++++++++++++++++++++++--
> >  xen/include/asm-arm/p2m.h |    3 ++
> >  2 files changed, 70 insertions(+), 4 deletions(-)
> > 
> > diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> > index 095e608..9b40e93 100644
> > --- a/xen/arch/arm/p2m.c
> > +++ b/xen/arch/arm/p2m.c
> > @@ -10,10 +10,20 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
> >      struct p2m_domain *p2m = &d->arch.p2m;
> >      lpae_t *first = NULL, *second = NULL, *third = NULL;
> >  
> > -    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
> > +    printk("dom%d IPA %#"PRIpaddr"\n", d->domain_id, addr);
> > +
> > +    printk("P2M @ %p mfn:%#lx (%#03llx,%#03llx,%#03llx)\n",
> > +           p2m->first_level,
> > +           page_to_mfn(p2m->first_level),
> > +           first_table_offset(addr),
> > +           second_table_offset(addr),
> > +           third_table_offset(addr));
> > +
> > +    if ( first_table_offset(addr) >= LPAE_ENTRIES )
> > +        goto done;
> >  
> >      first = __map_domain_page(p2m->first_level);
> > -    printk("1ST[%#03llx] = %#016llx\n",
> > +    printk("1ST[%#03llx] = %#"PRIpaddr"\n",
> >             first_table_offset(addr),
> >             first[first_table_offset(addr)].bits);
> >      if ( !first[first_table_offset(addr)].p2m.valid ||
> > @@ -21,7 +31,7 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
> >          goto done;
> >  
> >      second = map_domain_page(first[first_table_offset(addr)].p2m.base);
> > -    printk("2ND[%#03llx] = %#016llx\n",
> > +    printk("2ND[%#03llx] = %#"PRIpaddr"\n",
> >             second_table_offset(addr),
> >             second[second_table_offset(addr)].bits);
> >      if ( !second[second_table_offset(addr)].p2m.valid ||
> > @@ -29,7 +39,7 @@ void dump_p2m_lookup(struct domain *d, paddr_t addr)
> >          goto done;
> >  
> >      third = map_domain_page(second[second_table_offset(addr)].p2m.base);
> > -    printk("3RD[%#03llx] = %#016llx\n",
> > +    printk("3RD[%#03llx] = %#"PRIpaddr"\n",
> >             third_table_offset(addr),
> >             third[third_table_offset(addr)].bits);
> >  
> > @@ -51,6 +61,59 @@ void p2m_load_VTTBR(struct domain *d)
> >      isb(); /* Ensure update is visible */
> >  }
> 
> there is no need to introduce p2m_lookup in the same patch you do these
> unrelated printk adjustments, correct?
> 

Right, I think I've just put them in the wrong patch by mistake, I'll
figure out what I meant to do ...
> 
> > +/*
> > + * Lookup the MFN corresponding to a domain's PFN.
> > + *
> > + * There are no processor functions to do a stage 2 only lookup therefore we
> > + * do a a software walk.
> > + */
> > +paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
> > +{
> > +    struct p2m_domain *p2m = &d->arch.p2m;
> > +    lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
> > +    paddr_t maddr = INVALID_PADDR;
> > +
> > +    spin_lock(&p2m->lock);
> > +
> > +    first = __map_domain_page(p2m->first_level);
> > +    if ( !first[first_table_offset(paddr)].p2m.valid )
> > +        goto done_err;
> > +    if ( !first[first_table_offset(paddr)].p2m.table )
> > +    {
> > +        pte = first[first_table_offset(paddr)];
> > +        goto done;
> > +    }
> > +
> > +    second = map_domain_page(first[first_table_offset(paddr)].p2m.base);
> > +    if ( !second[second_table_offset(paddr)].p2m.valid )
> > +        goto done_err;
> > +    if ( !second[second_table_offset(paddr)].p2m.table )
> > +    {
> > +        pte = second[second_table_offset(paddr)];
> > +        goto done;
> > +    }
> > +
> > +    third = map_domain_page(second[second_table_offset(paddr)].p2m.base);
> > +    if ( !third[third_table_offset(paddr)].p2m.valid )
> > +        goto done_err;
> > +    if ( !third[third_table_offset(paddr)].p2m.table )
> > +        goto done_err;
> > +
> > +    pte = third[third_table_offset(paddr)];
> > +
> > +done:
> > +
> > +    maddr = (pte.bits & PADDR_MASK & PAGE_MASK) | (paddr & ~PAGE_MASK);
> > +done_err:
> > +    if (third) unmap_domain_page(third);
> > +    if (second) unmap_domain_page(second);
> > +    if (first) unmap_domain_page(first);
> > +
> > +    spin_unlock(&p2m->lock);
> > +
> > +    return maddr;
> > +}
> > +
> 
> this function looks correct though
> 
> >  int guest_physmap_mark_populate_on_demand(struct domain *d,
> >                                            unsigned long gfn,
> >                                            unsigned int order)
> > diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
> > index 349923a..1afd5cb 100644
> > --- a/xen/include/asm-arm/p2m.h
> > +++ b/xen/include/asm-arm/p2m.h
> > @@ -32,6 +32,9 @@ int p2m_alloc_table(struct domain *d);
> >  /* */
> >  void p2m_load_VTTBR(struct domain *d);
> >  
> > +/* */
> > +paddr_t p2m_lookup(struct domain *d, paddr_t gpfn);
> > +
> >  /* Setup p2m RAM mapping for domain d from start-end. */
> >  int p2m_populate_ram(struct domain *d, paddr_t start, paddr_t end);
> >  /* Map MMIO regions in the p2m: start_gaddr and end_gaddr is the range
> > -- 
> > 1.7.9.1
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> > 

Re: [PATCH 12/38] arm: remove hard tabs from init_idle_domain

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/setup.c |    6 +++---
>  1 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index 0df3c1a..81ababb 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -47,9 +47,9 @@ static __attribute_used__ void init_done(void)
>  
>  static void __init init_idle_domain(void)
>  {
> -        scheduler_init();
> -        set_current(idle_vcpu[0]);
> -        /* TODO: setup_idle_pagetable(); */
> +    scheduler_init();
> +    set_current(idle_vcpu[0]);
> +    /* TODO: setup_idle_pagetable(); */
>  }
>  
>  static void __init processor_id(void)
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 13/38] arm: stub out sync_vcpu_execstate

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> We don't do lazy exec state switching so there isn't actually anything to do.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/domain.c |    5 +++++
>  xen/arch/arm/dummy.S  |    1 -
>  2 files changed, 5 insertions(+), 1 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 62a2f3a..bd900f9 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -96,6 +96,11 @@ void sync_local_execstate(void)
>      /* Nothing to do -- no lazy switching */
>  }
>  
> +void sync_vcpu_execstate(struct vcpu *v)
> +{
> +    /* Nothing to do -- no lazy switching */
> +}
> +
>  void startup_cpu_idle_loop(void)
>  {
>      struct vcpu *v = current;
> diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
> index 3b48917..8eddd15 100644
> --- a/xen/arch/arm/dummy.S
> +++ b/xen/arch/arm/dummy.S
> @@ -22,7 +22,6 @@ DUMMY(pirq_set_affinity);
>  /* VCPU */
>  DUMMY(arch_get_info_guest);
>  DUMMY(arch_vcpu_reset);
> -DUMMY(sync_vcpu_execstate);
>  NOP(update_vcpu_system_time);
>  DUMMY(vcpu_show_execution_state);
>  
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 19/38] arm: context switch a bunch of guest state.

[Ian Campbell]

On Tue, 2012-06-05 at 18:11 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> > index 1a2b95f..339c327 100644
> > --- a/xen/arch/arm/gic.c
> > +++ b/xen/arch/arm/gic.c
> > @@ -61,6 +61,30 @@ static struct {
> >  irq_desc_t irq_desc[NR_IRQS];
> >  unsigned nr_lrs;
> > 
> > +void gic_save_state(struct vcpu *v)
> > +{
> > +    int i;
> > +
> > +    for ( i=0; i<nr_lrs; i++)
> > +        v->arch.gic_lr[i] = GICH[GICH_LR + i];
> > +    /* Disable until next VCPU scheduled */
> > +    GICH[GICH_HCR] = 0;
> > +    isb();
> > +}
> > +
> > +void gic_restore_state(struct vcpu *v)
> > +{
> > +    int i;
> > +
> > +    if ( is_idle_vcpu(v) )
> > +        return;
> > +
> > +    for ( i=0; i<nr_lrs; i++)
> > +        GICH[GICH_LR + i] = v->arch.gic_lr[i];
> > +    GICH[GICH_HCR] = GICH_HCR_EN;
> > +    isb();
> > +}
> > +
> 
> it is still missing a bunch of stuff from the gic state but it is a step
> in the right direction, so I'll send out patches to complete the gic
> context switch separately, based on this one.

Can I take this as an Ack for this patch for what it does?

Ian.

Re: [PATCH 19/38] arm: context switch a bunch of guest state.

[Stefano Stabellini]

On Wed, 6 Jun 2012, Ian Campbell wrote:
> On Tue, 2012-06-05 at 18:11 +0100, Stefano Stabellini wrote:
> > On Fri, 1 Jun 2012, Ian Campbell wrote:
> > > diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> > > index 1a2b95f..339c327 100644
> > > --- a/xen/arch/arm/gic.c
> > > +++ b/xen/arch/arm/gic.c
> > > @@ -61,6 +61,30 @@ static struct {
> > >  irq_desc_t irq_desc[NR_IRQS];
> > >  unsigned nr_lrs;
> > > 
> > > +void gic_save_state(struct vcpu *v)
> > > +{
> > > +    int i;
> > > +
> > > +    for ( i=0; i<nr_lrs; i++)
> > > +        v->arch.gic_lr[i] = GICH[GICH_LR + i];
> > > +    /* Disable until next VCPU scheduled */
> > > +    GICH[GICH_HCR] = 0;
> > > +    isb();
> > > +}
> > > +
> > > +void gic_restore_state(struct vcpu *v)
> > > +{
> > > +    int i;
> > > +
> > > +    if ( is_idle_vcpu(v) )
> > > +        return;
> > > +
> > > +    for ( i=0; i<nr_lrs; i++)
> > > +        GICH[GICH_LR + i] = v->arch.gic_lr[i];
> > > +    GICH[GICH_HCR] = GICH_HCR_EN;
> > > +    isb();
> > > +}
> > > +
> > 
> > it is still missing a bunch of stuff from the gic state but it is a step
> > in the right direction, so I'll send out patches to complete the gic
> > context switch separately, based on this one.
> 
> Can I take this as an Ack for this patch for what it does?

yes

Re: [PATCH 14/38] arm: do not set max_vcpus = 8 in arch_domain_create.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> XEN_DOMCTL_max_vcpus cannot reduce max_vcpus and therefore we can't create a
> smaller guest.
> 
> The limit of 8 (due to GIC limits) should be expressed elsewhere, likely in
> MAX_VIRT_CPUS -- but making that change caused:

Are you sure? I made that change and I didn't see the error.
I think this patch should set MAX_VIRT_CPUS to 8 as well as removing
max_vcpus = 8.


>     (XEN) Unexpected Trap: Data Abort
>     (XEN) ----[ Xen-4.2-unstable  x86_64  debug=y  Not tainted ]----
>     (XEN) CPU:    0
>     (XEN) PC:     00222e48 _spin_lock+0x28/0x6c
>     (XEN) CPSR:   600001da MODE:HYP
>     (XEN)      R0: 002c4389 R1: 800001da R2: 00000001 R3: 0000ffff
>     (XEN)      R4: 002c4381 R5: 00000080 R6: 002c4380 R7: 002c4000
>     (XEN)      R8: 002c4380 R9: 4000015a R10:00000080 R11:40017d6c R12:00000000
>     (XEN)      SP: 40017d5c LR: 00222e34
>     (XEN)
>     [...]
>     (XEN) Xen call trace:
>     (XEN)    [<00222e48>] _spin_lock+0x28/0x6c
>     (XEN)    [<0022623c>] init_timer+0xbc/0x160
>     (XEN)    [<0021fbdc>] sched_init_vcpu+0x94/0x200
>     (XEN)    [<002061a4>] alloc_vcpu+0x124/0x210
>     (XEN)    [<00204890>] do_domctl+0xaa4/0x14e4
>     (XEN)    [<00241ab8>] do_trap_hypervisor+0x588/0x8cc
>     (XEN)    [<0023bbb0>] return_from_trap+0x0/0x4
> 
> so punt on that for now.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/domain.c |    2 --
>  1 files changed, 0 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index bd900f9..e867cb2 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -215,8 +215,6 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
>              goto fail;
>      }
>  
> -    d->max_vcpus = 8;
> -
>      if ( (rc = domain_vgic_init(d)) != 0 )
>          goto fail;
>  
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 15/38] arm: implement stub version of flush_tlb_mask.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/dummy.S |    1 -
>  xen/arch/arm/smp.c   |    9 +++++++++
>  2 files changed, 9 insertions(+), 1 deletions(-)
> 
> diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
> index 8eddd15..c001e8d 100644
> --- a/xen/arch/arm/dummy.S
> +++ b/xen/arch/arm/dummy.S
> @@ -48,7 +48,6 @@ DUMMY(domain_get_maximum_gpfn);
>  DUMMY(domain_relinquish_resources);
>  DUMMY(domain_set_time_offset);
>  DUMMY(dom_cow);
> -DUMMY(flush_tlb_mask);
>  DUMMY(gmfn_to_mfn);
>  DUMMY(hypercall_create_continuation);
>  DUMMY(send_timer_event);
> diff --git a/xen/arch/arm/smp.c b/xen/arch/arm/smp.c
> index cad84f5..824c8c8 100644
> --- a/xen/arch/arm/smp.c
> +++ b/xen/arch/arm/smp.c
> @@ -1,5 +1,14 @@
>  #include <xen/config.h>
> +#include <asm/system.h>
>  #include <asm/smp.h>
> +#include <asm/cpregs.h>
> +#include <asm/page.h>
> +
> +void flush_tlb_mask(const cpumask_t *mask)
> +{
> +    /* XXX IPI other processors */
> +    flush_xen_data_tlb();
> +}
>  
>  void smp_call_function(
>      void (*func) (void *info),
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 14/38] arm: do not set max_vcpus = 8 in arch_domain_create.

[Ian Campbell]

On Wed, 2012-06-06 at 16:26 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > XEN_DOMCTL_max_vcpus cannot reduce max_vcpus and therefore we can't create a
> > smaller guest.
> > 
> > The limit of 8 (due to GIC limits) should be expressed elsewhere, likely in
> > MAX_VIRT_CPUS -- but making that change caused:
> 
> Are you sure?

Reasonably.

>  I made that change and I didn't see the error.

Let me try it again.

> I think this patch should set MAX_VIRT_CPUS to 8 as well as removing
> max_vcpus = 8.

Yes, that would be ideal, but in the interim just removing the max_vcpus
= 8 is an improvement in its own right if changing MAX_VIRT_CPUS causes
grief.

Ian.
> 
> 
> >     (XEN) Unexpected Trap: Data Abort
> >     (XEN) ----[ Xen-4.2-unstable  x86_64  debug=y  Not tainted ]----
> >     (XEN) CPU:    0
> >     (XEN) PC:     00222e48 _spin_lock+0x28/0x6c
> >     (XEN) CPSR:   600001da MODE:HYP
> >     (XEN)      R0: 002c4389 R1: 800001da R2: 00000001 R3: 0000ffff
> >     (XEN)      R4: 002c4381 R5: 00000080 R6: 002c4380 R7: 002c4000
> >     (XEN)      R8: 002c4380 R9: 4000015a R10:00000080 R11:40017d6c R12:00000000
> >     (XEN)      SP: 40017d5c LR: 00222e34
> >     (XEN)
> >     [...]
> >     (XEN) Xen call trace:
> >     (XEN)    [<00222e48>] _spin_lock+0x28/0x6c
> >     (XEN)    [<0022623c>] init_timer+0xbc/0x160
> >     (XEN)    [<0021fbdc>] sched_init_vcpu+0x94/0x200
> >     (XEN)    [<002061a4>] alloc_vcpu+0x124/0x210
> >     (XEN)    [<00204890>] do_domctl+0xaa4/0x14e4
> >     (XEN)    [<00241ab8>] do_trap_hypervisor+0x588/0x8cc
> >     (XEN)    [<0023bbb0>] return_from_trap+0x0/0x4
> > 
> > so punt on that for now.
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/domain.c |    2 --
> >  1 files changed, 0 insertions(+), 2 deletions(-)
> > 
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > index bd900f9..e867cb2 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -215,8 +215,6 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
> >              goto fail;
> >      }
> >  
> > -    d->max_vcpus = 8;
> > -
> >      if ( (rc = domain_vgic_init(d)) != 0 )
> >          goto fail;
> >  
> > -- 
> > 1.7.9.1
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> > 

Re: [PATCH 16/38] arm: Add simple cpu_{sibling, core}_mask

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/dummy.S   |    2 --
>  xen/arch/arm/setup.c   |    7 +++++++
>  xen/arch/arm/smpboot.c |    5 +++++
>  3 files changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
> index c001e8d..03f7489 100644
> --- a/xen/arch/arm/dummy.S
> +++ b/xen/arch/arm/dummy.S
> @@ -7,8 +7,6 @@ x:	.word 0xe7f000f0 /* Undefined instruction */
>  x:	mov pc, lr
>  	
>  /* SMP support */
> -DUMMY(per_cpu__cpu_core_mask);
> -DUMMY(per_cpu__cpu_sibling_mask);
>  DUMMY(node_online_map);
>  DUMMY(smp_send_state_dump);
>  
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index 81ababb..b0cfacc 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -230,6 +230,13 @@ void __init start_xen(unsigned long boot_phys_offset,
>          }
>      }
>  
> +    if ( !zalloc_cpumask_var(&per_cpu(cpu_sibling_mask, 0)) ||
> +         !zalloc_cpumask_var(&per_cpu(cpu_core_mask, 0)) )
> +        BUG();
> +
> +    cpumask_clear(per_cpu(cpu_sibling_mask, 0));
> +    cpumask_clear(per_cpu(cpu_core_mask, 0));
> +
>      printk("Brought up %ld CPUs\n", (long)num_online_cpus());
>      /* TODO: smp_cpus_done(); */
>  
> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
> index ea05afc..8517d86 100644
> --- a/xen/arch/arm/smpboot.c
> +++ b/xen/arch/arm/smpboot.c
> @@ -52,6 +52,11 @@ unsigned long __initdata ready_cpus = 0;
>  
>  /* ID of the PCPU we're running on */
>  DEFINE_PER_CPU(unsigned int, cpu_id);
> +/* XXX these seem awefully x86ish... */
> +/* representing HT siblings of each logical CPU */
> +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask);
> +/* representing HT and core siblings of each logical CPU */
> +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
>  
>  void __init
>  smp_prepare_cpus (unsigned int max_cpus)
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 17/38] arm: allow p2m to be created with specific MATTR.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/p2m.c         |   15 ++++++++-------
>  xen/include/asm-arm/page.h |    6 ++++--
>  2 files changed, 12 insertions(+), 9 deletions(-)
> 
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index 9b40e93..46c6f17 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -148,7 +148,7 @@ static int p2m_create_entry(struct domain *d,
>      clear_page(p);
>      unmap_domain_page(p);
>  
> -    pte = mfn_to_p2m_entry(page_to_mfn(page));
> +    pte = mfn_to_p2m_entry(page_to_mfn(page), MATTR_MEM);
>  
>      write_pte(entry, pte);
>  

This works because p2m_create_entry is always called to create first or
second level entries only.
Maybe we should rename p2m_create_entry to p2m_create_table_entry for
clarity? Or add a comment?


> @@ -159,7 +159,8 @@ static int create_p2m_entries(struct domain *d,
>                       int alloc,
>                       paddr_t start_gpaddr,
>                       paddr_t end_gpaddr,
> -                     paddr_t maddr)
> +                     paddr_t maddr,
> +                     int mattr)
>  {
>      int rc;
>      struct p2m_domain *p2m = &d->arch.p2m;
> @@ -235,11 +236,11 @@ static int create_p2m_entries(struct domain *d,
>                  goto out;
>              }
>  
> -            pte = mfn_to_p2m_entry(page_to_mfn(page));
> +            pte = mfn_to_p2m_entry(page_to_mfn(page), mattr);
>  
>              write_pte(&third[third_table_offset(addr)], pte);
>          } else {
> -            lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT);
> +            lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT, mattr);
>              write_pte(&third[third_table_offset(addr)], pte);
>              maddr += PAGE_SIZE;
>          }
> @@ -263,7 +264,7 @@ int p2m_populate_ram(struct domain *d,
>                       paddr_t start,
>                       paddr_t end)
>  {
> -    return create_p2m_entries(d, 1, start, end, 0);
> +    return create_p2m_entries(d, 1, start, end, 0, MATTR_MEM);
>  }
>  
>  int map_mmio_regions(struct domain *d,
> @@ -271,7 +272,7 @@ int map_mmio_regions(struct domain *d,
>                       paddr_t end_gaddr,
>                       paddr_t maddr)
>  {
> -    return create_p2m_entries(d, 0, start_gaddr, end_gaddr, maddr);
> +    return create_p2m_entries(d, 0, start_gaddr, end_gaddr, maddr, MATTR_DEV);
>  }
>  
>  int guest_physmap_add_page(struct domain *d,
> @@ -281,7 +282,7 @@ int guest_physmap_add_page(struct domain *d,
>  {
>      return create_p2m_entries(d, 0, gpfn << PAGE_SHIFT,
>                                (gpfn + (1<<page_order)) << PAGE_SHIFT,
> -                              mfn << PAGE_SHIFT);
> +                              mfn << PAGE_SHIFT, MATTR_MEM);
>  }
>  
>  void guest_physmap_remove_page(struct domain *d,
> diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
> index c7b6530..bb1729a 100644
> --- a/xen/include/asm-arm/page.h
> +++ b/xen/include/asm-arm/page.h
> @@ -46,6 +46,8 @@
>  #define DEV_WC        BUFFERABLE
>  #define DEV_CACHED    WRITEBACK
>  
> +#define MATTR_DEV     0x1
> +#define MATTR_MEM     0xf
>  
>  #ifndef __ASSEMBLY__
>  
> @@ -169,7 +171,7 @@ static inline lpae_t mfn_to_xen_entry(unsigned long mfn)
>      return e;
>  }
>  
> -static inline lpae_t mfn_to_p2m_entry(unsigned long mfn)
> +static inline lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
>  {
>      paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
>      lpae_t e = (lpae_t) {
> @@ -178,7 +180,7 @@ static inline lpae_t mfn_to_p2m_entry(unsigned long mfn)
>          .p2m.sh = LPAE_SH_OUTER,
>          .p2m.write = 1,
>          .p2m.read = 1,
> -        .p2m.mattr = 0xf,
> +        .p2m.mattr = mattr,
>          .p2m.table = 1,
>          .p2m.valid = 1,
>      };

the rest of the patch is fine

Re: [PATCH 18/38] arm: implement vpl011 (UART) emulator.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> This is not interended to provide a full emulation, but rather just enough to
> satisfy the use made by Linux' boot time decompressor code (which is too early
> for DT etc)
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/Makefile        |    1 +
>  xen/arch/arm/domain.c        |    4 +
>  xen/arch/arm/io.c            |    1 +
>  xen/arch/arm/io.h            |    1 +
>  xen/arch/arm/vpl011.c        |  155 ++++++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vpl011.h        |   34 +++++++++
>  xen/include/asm-arm/domain.h |    8 ++
>  7 files changed, 204 insertions(+), 0 deletions(-)
>  create mode 100644 xen/arch/arm/vpl011.c
>  create mode 100644 xen/arch/arm/vpl011.h
> 
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 9440a21..5a87ba6 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -25,6 +25,7 @@ obj-y += shutdown.o
>  obj-y += traps.o
>  obj-y += vgic.o
>  obj-y += vtimer.o
> +obj-y += vpl011.o
>  
>  #obj-bin-y += ....o
>  
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index e867cb2..d830980 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -13,6 +13,7 @@
>  
>  #include "gic.h"
>  #include "vtimer.h"
> +#include "vpl011.h"
>  
>  DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>  
> @@ -201,6 +202,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
>      if ( (rc = domain_vgic_init(d)) != 0 )
>          goto fail;
>  
> +    if ( (rc = domain_uart0_init(d)) != 0 )
> +        goto fail;
> +
>      if ( !is_idle_domain(d) )
>      {
>          rc = -ENOMEM;
> diff --git a/xen/arch/arm/io.c b/xen/arch/arm/io.c
> index 4461225..18f6164 100644
> --- a/xen/arch/arm/io.c
> +++ b/xen/arch/arm/io.c
> @@ -25,6 +25,7 @@
>  static const struct mmio_handler *const mmio_handlers[] =
>  {
>      &vgic_distr_mmio_handler,
> +    &uart0_mmio_handler,
>  };
>  #define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
>  
> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
> index 8cc5ca7..9a507f5 100644
> --- a/xen/arch/arm/io.h
> +++ b/xen/arch/arm/io.h
> @@ -40,6 +40,7 @@ struct mmio_handler {
>  };
>  
>  extern const struct mmio_handler vgic_distr_mmio_handler;
> +extern const struct mmio_handler uart0_mmio_handler;
>  
>  extern int handle_mmio(mmio_info_t *info);
>  
> diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
> new file mode 100644
> index 0000000..1491dcc
> --- /dev/null
> +++ b/xen/arch/arm/vpl011.c
> @@ -0,0 +1,155 @@
> +/*
> + * xen/arch/arm/vpl011.c
> + *
> + * ARM PL011 UART Emulator (DEBUG)
> + *
> + * Ian Campbell <ian.campbell@citrix.com>
> + * Copyright (c) 2012 Citrix Systems.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +/*
> + * This is not intended to be a full emulation of a PL011
> + * device. Rather it is intended to provide a sufficient veneer of one
> + * that early code (such as Linux's boot time decompressor) which
> + * hardcodes output directly to such a device are able to make progress.
> + *
> + * This device is not intended to be enumerable or exposed to the OS
> + * (e.g. via Device Tree).
> + */
> +
> +#include <xen/config.h>
> +#include <xen/lib.h>
> +#include <xen/sched.h>
> +#include <xen/errno.h>
> +#include <xen/ctype.h>
> +
> +#include "io.h"
> +
> +#define UART0_BASE_ADDRESS 0x1c090000
> +
> +#define UARTDR 0x000
> +#define UARTFR 0x018
> +
> +int domain_uart0_init(struct domain *d)
> +{
> +    int rc;
> +    if ( d->domain_id == 0 )
> +        return 0;
> +
> +    spin_lock_init(&d->arch.uart0.lock);
> +    d->arch.uart0.idx = 0;
> +
> +    rc = -ENOMEM;
> +    d->arch.uart0.buf = xzalloc_array(char, VPL011_BUF_SIZE);
> +    if ( !d->arch.uart0.buf )
> +        goto out;
> +
> +    rc = 0;
> +out:
> +    return rc;
> +}
> +
> +static void uart0_print_char(char c)
> +{
> +    struct vpl011 *uart = &current->domain->arch.uart0;
> +
> +    /* Accept only printable characters, newline, and horizontal tab. */
> +    if ( !isprint(c) && (c != '\n') && (c != '\t') )
> +        return ;
> +
> +    spin_lock(&uart->lock);
> +    uart->buf[uart->idx++] = c;
> +    if ( (uart->idx == (VPL011_BUF_SIZE - 2)) || (c == '\n') )
> +    {
> +        if ( c != '\n' )
> +            uart->buf[uart->idx++] = '\n';
> +        uart->buf[uart->idx] = '\0';
> +        printk(XENLOG_G_DEBUG "DOM%u: %s",
> +               current->domain->domain_id, uart->buf);
> +        uart->idx = 0;
> +    }
> +    spin_unlock(&uart->lock);
> +}
> +
> +static int uart0_mmio_check(struct vcpu *v, paddr_t addr)
> +{
> +    return v->domain->domain_id && addr >= UART0_BASE_ADDRESS && addr < (UART0_BASE_ADDRESS+65536);
> +}

maybe we need UART0_BASE_ADDRESS_START and UART0_BASE_ADDRESS_END
instead of having an arbitrary +65536

Re: [PATCH 22/38] arm: implement vcpu_show_execution_state

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/dummy.S |    1 -
>  xen/arch/arm/traps.c |   56 +++++++++++++++++++++++++++++++++++++++++++++----
>  2 files changed, 51 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
> index 03f7489..cab9522 100644
> --- a/xen/arch/arm/dummy.S
> +++ b/xen/arch/arm/dummy.S
> @@ -21,7 +21,6 @@ DUMMY(pirq_set_affinity);
>  DUMMY(arch_get_info_guest);
>  DUMMY(arch_vcpu_reset);
>  NOP(update_vcpu_system_time);
> -DUMMY(vcpu_show_execution_state);
>  
>  /* Page Reference & Type Maintenance */
>  DUMMY(get_page);
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 35907ee..ec74298 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -170,7 +170,13 @@ void panic_PAR(uint64_t par, const char *when)
>      panic("Error during %s-to-physical address translation\n", when);
>  }
>  
> -void show_registers(struct cpu_user_regs *regs)
> +struct reg_ctxt {
> +    uint32_t sctlr;
> +    uint32_t ttbr0, ttbr1, ttbcr;
> +};
> +static void _show_registers(struct cpu_user_regs *regs,
> +                            struct reg_ctxt *ctxt,
> +                            int guest_mode)
>  {
>      static const char *mode_strings[] = {
>         [PSR_MODE_USR] = "USR",
> @@ -187,7 +193,7 @@ void show_registers(struct cpu_user_regs *regs)
>      print_xen_info();
>      printk("CPU:    %d\n", smp_processor_id());
>      printk("PC:     %08"PRIx32, regs->pc);
> -    if ( !guest_mode(regs) )
> +    if ( !guest_mode )
>              print_symbol(" %s", regs->pc);
>      printk("\n");
>      printk("CPSR:   %08"PRIx32" MODE:%s\n", regs->cpsr,
> @@ -199,7 +205,7 @@ void show_registers(struct cpu_user_regs *regs)
>      printk("     R8: %08"PRIx32" R9: %08"PRIx32" R10:%08"PRIx32" R11:%08"PRIx32" R12:%08"PRIx32"\n",
>             regs->r8, regs->r9, regs->r10, regs->r11, regs->r12);
>  
> -    if ( guest_mode(regs) )
> +    if ( guest_mode )
>      {
>          printk("USR: SP: %08"PRIx32" LR: %08"PRIx32" CPSR:%08"PRIx32"\n",
>                 regs->sp_usr, regs->lr_usr, regs->cpsr);
> @@ -217,8 +223,8 @@ void show_registers(struct cpu_user_regs *regs)
>                 regs->r8_fiq, regs->r9_fiq, regs->r10_fiq, regs->r11_fiq, regs->r11_fiq);
>          printk("\n");
>          printk("TTBR0 %08"PRIx32" TTBR1 %08"PRIx32" TTBCR %08"PRIx32"\n",
> -               READ_CP32(TTBR0), READ_CP32(TTBR1), READ_CP32(TTBCR));
> -        printk("SCTLR %08"PRIx32"\n", READ_CP32(SCTLR));
> +               ctxt->ttbr0, ctxt->ttbr1, ctxt->ttbcr);
> +        printk("SCTLR %08"PRIx32"\n", ctxt->sctlr);
>          printk("VTTBR %010"PRIx64"\n", READ_CP64(VTTBR));
>          printk("\n");
>      }
> @@ -241,6 +247,26 @@ void show_registers(struct cpu_user_regs *regs)
>      printk("\n");
>  }
>  
> +void show_registers(struct cpu_user_regs *regs)
> +{
> +    struct reg_ctxt ctxt;
> +    ctxt.sctlr = READ_CP32(SCTLR);
> +    ctxt.ttbcr = READ_CP32(TTBCR);
> +    ctxt.ttbr0 = READ_CP32(TTBR0);
> +    ctxt.ttbr1 = READ_CP32(TTBR1);
> +    _show_registers(regs, &ctxt, guest_mode(regs));
> +}
> +
> +void vcpu_show_registers(const struct vcpu *v)
> +{
> +    struct reg_ctxt ctxt;
> +    ctxt.sctlr = v->arch.sctlr;
> +    ctxt.ttbcr = v->arch.ttbcr;
> +    ctxt.ttbr0 = v->arch.ttbr0;
> +    ctxt.ttbr1 = v->arch.ttbr1;
> +    _show_registers(&v->arch.cpu_info->guest_cpu_user_regs, &ctxt, 1);
> +}
> +
>  static void show_guest_stack(struct cpu_user_regs *regs)
>  {
>      printk("GUEST STACK GOES HERE\n");
> @@ -334,6 +360,26 @@ void show_execution_state(struct cpu_user_regs *regs)
>      show_stack(regs);
>  }
>  
> +void vcpu_show_execution_state(struct vcpu *v)
> +{
> +    printk("*** Dumping Dom%d vcpu#%d state: ***\n",
> +           v->domain->domain_id, v->vcpu_id);
> +
> +    if ( v == current )
> +    {
> +        show_execution_state(guest_cpu_user_regs());
> +        return;
> +    }
> +
> +    vcpu_pause(v); /* acceptably dangerous */
> +
> +    vcpu_show_registers(v);
> +    if ( !usr_mode(&v->arch.cpu_info->guest_cpu_user_regs) )
> +        show_guest_stack(&v->arch.cpu_info->guest_cpu_user_regs);

isn't the if condition inverted?

> +    vcpu_unpause(v);
> +}
> +
>  static void do_unexpected_trap(const char *msg, struct cpu_user_regs *regs)
>  {
>      printk("Unexpected Trap: %s\n", msg);

Re: [PATCH 23/38] arm: use correct attributes for mappings in copy_from_paddr()

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> The DTB is in RAM (hence bufferable), kernel is in flash and therefor requires
> a device type mapping (hence dev shared).
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/kernel.c       |    8 ++++----
>  xen/arch/arm/setup.c        |    2 +-
>  xen/include/asm-arm/setup.h |    2 +-
>  3 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/arm/kernel.c b/xen/arch/arm/kernel.c
> index 130d488..1a705c9 100644
> --- a/xen/arch/arm/kernel.c
> +++ b/xen/arch/arm/kernel.c
> @@ -39,7 +39,7 @@ struct minimal_dtb_header {
>   * @paddr: source physical address
>   * @len: length to copy
>   */
> -void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
> +void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int mattr)
>  {
>      void *src = (void *)FIXMAP_ADDR(FIXMAP_MISC);
>  
> @@ -51,7 +51,7 @@ void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
>          s = paddr & (PAGE_SIZE-1);
>          l = min(PAGE_SIZE - s, len);
>  
> -        set_fixmap(FIXMAP_MISC, p, DEV_SHARED);
> +        set_fixmap(FIXMAP_MISC, p, mattr);
>          memcpy(dst, src + s, l);
>  
>          paddr += l;
> @@ -111,7 +111,7 @@ static int kernel_try_zimage_prepare(struct kernel_info *info)
>      /*
>       * Check for an appended DTB.
>       */
> -    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr));
> +    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr), DEV_SHARED);
>      if (be32_to_cpu(dtb_hdr.magic) == DTB_MAGIC) {
>          end += be32_to_cpu(dtb_hdr.total_size);
>      }
> @@ -151,7 +151,7 @@ static int kernel_try_elf_prepare(struct kernel_info *info)
>      if ( info->kernel_img == NULL )
>          panic("Cannot allocate temporary buffer for kernel.\n");
>  
> -    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE);
> +    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE, DEV_SHARED);
>  
>      if ( (rc = elf_init(&info->elf.elf, info->kernel_img, KERNEL_FLASH_SIZE )) != 0 )
>          return rc;
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index b0cfacc..f5473cd 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -122,7 +122,7 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
>       * TODO: handle other payloads too.
>       */
>      device_tree_flattened = mfn_to_virt(alloc_boot_pages(dtb_pages, 1));
> -    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size);
> +    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size, BUFFERABLE);
>  
>      /* Add non-xenheap memory */
>      init_boot_pages(pfn_to_paddr(xenheap_mfn_start + xenheap_pages),
> diff --git a/xen/include/asm-arm/setup.h b/xen/include/asm-arm/setup.h
> index 05ff89e..faadccc 100644
> --- a/xen/include/asm-arm/setup.h
> +++ b/xen/include/asm-arm/setup.h
> @@ -3,7 +3,7 @@
>  
>  #include <public/version.h>
>  
> -void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len);
> +void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int mattr);
>  
>  void arch_get_xen_caps(xen_capabilities_info_t *info);
>  

The patch is correct, but it shouldn't call the new parameter mattr
because it can easily be confused with the memattr bits (see
http://marc.info/?l=xen-devel&m=133856578918985).
I would call it "int attrindx" instead.

Re: [PATCH 24/38] arm: map fixmaps non-executable.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/mm.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> index c332e4c..160a4e9 100644
> --- a/xen/arch/arm/mm.c
> +++ b/xen/arch/arm/mm.c
> @@ -108,6 +108,7 @@ void set_fixmap(unsigned map, unsigned long mfn, unsigned attributes)
>      lpae_t pte = mfn_to_xen_entry(mfn);
>      pte.pt.table = 1; /* 4k mappings always have this bit set */
>      pte.pt.ai = attributes;
> +    pte.pt.xn = 1;
>      write_pte(xen_fixmap + third_table_offset(FIXMAP_ADDR(map)), pte);
>      flush_xen_data_tlb_va(FIXMAP_ADDR(map));
>  }
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 25/38] arm: remove old identity map of boot paddr when we are done with it.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/mm.c |    8 ++++++++
>  1 files changed, 8 insertions(+), 0 deletions(-)
> 
> diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> index 160a4e9..ab52171 100644
> --- a/xen/arch/arm/mm.c
> +++ b/xen/arch/arm/mm.c
> @@ -214,6 +214,14 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
>      lpae_t pte, *p;
>      int i;
>  
> +    if ( boot_phys_offset != 0 )
> +    {
> +        /* Remove the old identity mapping of the boot paddr */
> +        pte.bits = 0;
> +        dest_va = (unsigned long)_start + boot_phys_offset;
> +        write_pte(xen_second + second_linear_offset(dest_va), pte);
> +    }

It looks like we are already doing this few lines below.
Also now that I am thinking about it, considering that bits is a 64 bit
field, shouldn't this be:

pte.bits = 0ULL;

Re: [PATCH 25/38] arm: remove old identity map of boot paddr when we are done with it.

[Tim Deegan]

At 19:04 +0100 on 06 Jun (1339009473), Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/mm.c |    8 ++++++++
> >  1 files changed, 8 insertions(+), 0 deletions(-)
> > 
> > diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> > index 160a4e9..ab52171 100644
> > --- a/xen/arch/arm/mm.c
> > +++ b/xen/arch/arm/mm.c
> > @@ -214,6 +214,14 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
> >      lpae_t pte, *p;
> >      int i;
> >  
> > +    if ( boot_phys_offset != 0 )
> > +    {
> > +        /* Remove the old identity mapping of the boot paddr */
> > +        pte.bits = 0;
> > +        dest_va = (unsigned long)_start + boot_phys_offset;
> > +        write_pte(xen_second + second_linear_offset(dest_va), pte);
> > +    }
> 
> It looks like we are already doing this few lines below.

We used to do this here and now we do it futher down, after the copy.
That way the secondary CPUs can come up on the pre-copy tables where
the identity map still exists.

> Also now that I am thinking about it, considering that bits is a 64 bit
> field, shouldn't this be:
> 
> pte.bits = 0ULL;

It's OK; the 0 will be sign-extended by the compiler to 64 bits. 

Cheers,

Tim.

Re: [PATCH 02/38] arm: handy function to print a walk of the hypervisor page tables

[Tim Deegan]

Hi,

At 15:39 +0000 on 01 Jun (1338565171), Ian Campbell wrote:
> +void dump_pt_walk(uint32_t addr)
> +{
> +    paddr_t second_ma, third_ma;
> +    lpae_t *first = NULL, *second = NULL, *third = NULL;
> +    uint64_t httbr = READ_CP64(HTTBR);
> +
> +    printk("Walking Hypervisor VA %#"PRIx32" via HTTBR %#"PRIx64"\n",
> +           addr, httbr);
> +
> +    BUG_ON( (lpae_t *)(unsigned long)(httbr - xen_phys_offset) != xen_pgtable );
> +    first = xen_pgtable;
> +    printk("1ST[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
> +           first_table_offset(addr),
> +           first, first_table_offset(addr),
> +           virt_to_maddr(&first[first_table_offset(addr)]),
> +           first[first_table_offset(addr)].bits);
> +
> +    if ( !first[first_table_offset(addr)].pt.valid ||
> +         !first[first_table_offset(addr)].pt.table )
> +        goto done;

This could probably be a for loop rather than open-coding three
almost-identical printks.

> +
> +    second_ma = (paddr_t)first[first_table_offset(addr)].pt.base << PAGE_SHIFT;
> +    second = (lpae_t *)(unsigned long)(second_ma - xen_phys_offset);
> +    printk("2ND[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
> +           second_table_offset(addr),
> +           second, second_table_offset(addr),
> +           virt_to_maddr(&second[second_table_offset(addr)]),
> +           second[second_table_offset(addr)].bits);
> +    if ( !second[second_table_offset(addr)].pt.valid ||
> +         !second[second_table_offset(addr)].pt.table )
> +        goto done;
> +
> +    third_ma = (paddr_t)second[second_table_offset(addr)].pt.base << PAGE_SHIFT;
> +    third = (lpae_t *)(unsigned long)(third_ma - xen_phys_offset);
> +    printk("3RD[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
> +           third_table_offset(addr),
> +           third, third_table_offset(addr),
> +           virt_to_maddr(&third[third_table_offset(addr)]),
> +           third[third_table_offset(addr)].bits);
> +    if ( !third[third_table_offset(addr)].pt.valid ||
> +         !third[third_table_offset(addr)].pt.table )
> +        goto done;
> +
> +done:
> +    return;

Maybe use return above instead of goto?

> +}
> +
>  /* Map a 4k page in a fixmap entry */
>  void set_fixmap(unsigned map, unsigned long mfn, unsigned attributes)
>  {
> @@ -173,8 +222,8 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
>      flush_xen_data_tlb_va(dest_va);
>  
>      /* Calculate virt-to-phys offset for the new location */
> -    phys_offset = xen_paddr - (unsigned long) _start;
> -
> +    xen_phys_offset = phys_offset = xen_paddr - (unsigned long) _start;
> +    

Just make phys_offset file-scope static; no need to add a new variable
for this.

>      /* Copy */
>      memcpy((void *) dest_va, _start, _end - _start);
>  
> diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
> index b6df64e..22c56b5 100644
> --- a/xen/include/asm-arm/page.h
> +++ b/xen/include/asm-arm/page.h
> @@ -312,6 +312,8 @@ static inline uint64_t gva_to_ipa(uint32_t va)
>  /* Bits in the PAR returned by va_to_par */
>  #define PAR_FAULT 0x1
>  
> +extern void dump_pt_walk(uint32_t addr);
> +

Maybe a comment?

Cheers,

Tim.

Re: [PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565172), Ian Campbell wrote:
> Useful for debug but not actually used in this patch.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/p2m.c         |   34 ++++++++++++++++++++++++++++++++++
>  xen/include/asm-arm/page.h |    1 +
>  2 files changed, 35 insertions(+), 0 deletions(-)
> 
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index 4f624d8..fdbecbc 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -5,6 +5,40 @@
>  #include <xen/domain_page.h>
>  #include <asm/flushtlb.h>
>  
> +void dump_p2m_lookup(struct domain *d, paddr_t addr)
> +{
> +    struct p2m_domain *p2m = &d->arch.p2m;
> +    lpae_t *first = NULL, *second = NULL, *third = NULL;
> +
> +    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
> +
> +    first = __map_domain_page(p2m->first_level);
> +    printk("1ST[%#03llx] = %#016llx\n",
> +           first_table_offset(addr),
> +           first[first_table_offset(addr)].bits);
> +    if ( !first[first_table_offset(addr)].p2m.valid ||
> +         !first[first_table_offset(addr)].p2m.table )
> +        goto done;
> +
> +    second = map_domain_page(first[first_table_offset(addr)].p2m.base);
> +    printk("2ND[%#03llx] = %#016llx\n",
> +           second_table_offset(addr),
> +           second[second_table_offset(addr)].bits);
> +    if ( !second[second_table_offset(addr)].p2m.valid ||
> +         !second[second_table_offset(addr)].p2m.table )
> +        goto done;
> +
> +    third = map_domain_page(second[second_table_offset(addr)].p2m.base);
> +    printk("3RD[%#03llx] = %#016llx\n",
> +           third_table_offset(addr),
> +           third[third_table_offset(addr)].bits);
> +
> +done:
> +    if (third) unmap_domain_page(third);
> +    if (second) unmap_domain_page(second);
> +    if (first) unmap_domain_page(first);
> +}

Can this be unified with dump_pt_walk?  As it happens, the valid, table,
and base fields are the same in pt and p2m entries.

Tim.

Re: [PATCH 11/38] arm: implement p2m lookup

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565180), Ian Campbell wrote:
> +/*
> + * Lookup the MFN corresponding to a domain's PFN.
> + *
> + * There are no processor functions to do a stage 2 only lookup therefore we
> + * do a a software walk.
> + */
> +paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
> +{
> +    struct p2m_domain *p2m = &d->arch.p2m;
> +    lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
> +    paddr_t maddr = INVALID_PADDR;
> +
> +    spin_lock(&p2m->lock);
> +
> +    first = __map_domain_page(p2m->first_level);
> +    if ( !first[first_table_offset(paddr)].p2m.valid )
> +        goto done_err;
> +    if ( !first[first_table_offset(paddr)].p2m.table )
> +    {
> +        pte = first[first_table_offset(paddr)];
> +        goto done;
> +    }

This would be neater as: 
       pte = first[first_table_offset(paddr)];
       if ( !pte.p2m.valid || !pte.p2m.table )
           goto done;

and test for pte.valid at 'done'.

It would be nice to do the three levels in a loop as well, but the weird
way the table bit behaves in third-level entries might make that more
confusing than the straight-line version.

Tim.

Re: [PATCH 16/38] arm: Add simple cpu_{sibling, core}_mask

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565185), Ian Campbell wrote:
> @@ -230,6 +230,13 @@ void __init start_xen(unsigned long boot_phys_offset,
>          }
>      }
>  
> +    if ( !zalloc_cpumask_var(&per_cpu(cpu_sibling_mask, 0)) ||
> +         !zalloc_cpumask_var(&per_cpu(cpu_core_mask, 0)) )
> +        BUG();
> +
> +    cpumask_clear(per_cpu(cpu_sibling_mask, 0));
> +    cpumask_clear(per_cpu(cpu_core_mask, 0));

Aren't these clear()s noops?  

Tim.

Re: [PATCH 18/38] arm: implement vpl011 (UART) emulator.

[Tim Deegan]

> +int domain_uart0_init(struct domain *d)
> +{
> +    int rc;
> +    if ( d->domain_id == 0 )
> +        return 0;

Why?  There's no equivalent gate on the MMIO handlers.

> +    spin_lock_init(&d->arch.uart0.lock);
> +    d->arch.uart0.idx = 0;
> +
> +    rc = -ENOMEM;
> +    d->arch.uart0.buf = xzalloc_array(char, VPL011_BUF_SIZE);
> +    if ( !d->arch.uart0.buf )
> +        goto out;

Just return ENOMEM here - the 'rc=foo; goto out' is overkill.

> +
> +    rc = 0;
> +out:
> +    return rc;
> +}

> +static int uart0_mmio_check(struct vcpu *v, paddr_t addr)
> +{
> +    return v->domain->domain_id && addr >= UART0_BASE_ADDRESS && addr < (UART0_BASE_ADDRESS+65536);
> +}

linewrap?

> +
> +static int uart0_mmio_read(struct vcpu *v, mmio_info_t *info)
> +{
> +    struct hsr_dabt dabt = info->dabt;
> +    struct cpu_user_regs *regs = guest_cpu_user_regs();
> +    uint32_t *r = &regs->r0 + dabt.reg;
> +    int offset = (int)(info->gpa - UART0_BASE_ADDRESS);
> +

Need to check dabt.size != 0 here too?

Tim.

Re: [PATCH 20/38] arm: dump a page table walk when va_to_par fails.

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565189), Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Tim Deegan <tim@xen.org>

> ---
>  xen/include/asm-arm/page.h |   12 ++++++++----
>  1 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
> index bb1729a..f36bf6f 100644
> --- a/xen/include/asm-arm/page.h
> +++ b/xen/include/asm-arm/page.h
> @@ -254,6 +254,9 @@ static inline void flush_guest_tlb(void)
>      WRITE_CP32(r0 /* dummy */, TLBIALLNSNH);
>  }
>  
> +extern void dump_pt_walk(uint32_t addr);
> +extern void dump_p2m_lookup(struct domain *d, paddr_t addr);
> +
>  /* Ask the MMU to translate a VA for us */
>  static inline uint64_t __va_to_par(uint32_t va)
>  {
> @@ -270,7 +273,11 @@ static inline uint64_t va_to_par(uint32_t va)
>  {
>      uint64_t par = __va_to_par(va);
>      /* It is not OK to call this with an invalid VA */
> -    if ( par & PAR_F ) panic_PAR(par, "Hypervisor");
> +    if ( par & PAR_F )
> +    {
> +        dump_pt_walk(va);
> +        panic_PAR(par, "Hypervisor");
> +    }
>      return par;
>  }
>  
> @@ -314,9 +321,6 @@ static inline uint64_t gva_to_ipa(uint32_t va)
>  /* Bits in the PAR returned by va_to_par */
>  #define PAR_FAULT 0x1
>  
> -extern void dump_pt_walk(uint32_t addr);
> -extern void dump_p2m_lookup(struct domain *d, paddr_t addr);
> -
>  #endif /* __ASSEMBLY__ */
>  
>  /* These numbers add up to a 39-bit input address space.  The  ARMv7-A
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH 18/38] arm: implement vpl011 (UART) emulator.

[Ian Campbell]

On Thu, 2012-06-07 at 10:29 +0100, Tim Deegan wrote:
> > +int domain_uart0_init(struct domain *d)
> > +{
> > +    int rc;
> > +    if ( d->domain_id == 0 )
> > +        return 0;
> 
> Why?  There's no equivalent gate on the MMIO handlers.

dom0 has the actual uart mapped at this address, not the emulated one.
Maybe that should be written at the caller instead?

> 
> > +    spin_lock_init(&d->arch.uart0.lock);
> > +    d->arch.uart0.idx = 0;
> > +
> > +    rc = -ENOMEM;
> > +    d->arch.uart0.buf = xzalloc_array(char, VPL011_BUF_SIZE);
> > +    if ( !d->arch.uart0.buf )
> > +        goto out;
> 
> Just return ENOMEM here - the 'rc=foo; goto out' is overkill.

ok.

> > +
> > +    rc = 0;
> > +out:
> > +    return rc;
> > +}
> 
> > +static int uart0_mmio_check(struct vcpu *v, paddr_t addr)
> > +{
> > +    return v->domain->domain_id && addr >= UART0_BASE_ADDRESS && addr < (UART0_BASE_ADDRESS+65536);
> > +}
> 
> linewrap?

yes, Stefano also suggests a better #define than 65536...

> 
> > +
> > +static int uart0_mmio_read(struct vcpu *v, mmio_info_t *info)
> > +{
> > +    struct hsr_dabt dabt = info->dabt;
> > +    struct cpu_user_regs *regs = guest_cpu_user_regs();
> > +    uint32_t *r = &regs->r0 + dabt.reg;
> > +    int offset = (int)(info->gpa - UART0_BASE_ADDRESS);
> > +
> 
> Need to check dabt.size != 0 here too?

IIRC I was seeing reads of different sizes. To be honest I mostly
tailored this for the specific behaviour of the Linux decompression
code, it didn't really want to write a full & correct UART emulation...

Ian.

Re: [PATCH 08/38] arm: allocate and setup a guest vcpu.

[Ian Campbell]

On Wed, 2012-06-06 at 14:55 +0100, Ian Campbell wrote:
> On Wed, 2012-06-06 at 14:46 +0100, Stefano Stabellini wrote:
> > On Fri, 1 Jun 2012, Ian Campbell wrote:
> > > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > > ---
> > >  xen/arch/arm/domain.c         |   68 +++++++++++++++++++++++++++++++++++++++++
> > >  xen/arch/arm/dummy.S          |    3 --
> > >  xen/include/public/arch-arm.h |    9 -----
> > >  3 files changed, 68 insertions(+), 12 deletions(-)
> > > 
> > > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > > index 9339a11..62a2f3a 100644
> > > --- a/xen/arch/arm/domain.c
> > > +++ b/xen/arch/arm/domain.c
> > > @@ -144,6 +144,17 @@ void free_vcpu_struct(struct vcpu *v)
> > >      free_xenheap_page(v);
> > >  }
> > >  
> > > +struct vcpu_guest_context *alloc_vcpu_guest_context(void)
> > > +{
> > > +    return xmalloc(struct vcpu_guest_context);
> > > +
> > > +}
> > > +
> > > +void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
> > > +{
> > > +    xfree(vgc);
> > > +}
> > > +
> > >  int vcpu_initialise(struct vcpu *v)
> > >  {
> > >      int rc = 0;
> > > @@ -182,6 +193,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
> > >      if ( (rc = p2m_init(d)) != 0 )
> > >          goto fail;
> > >  
> > > +    if ( (rc = domain_vgic_init(d)) != 0 )
> > > +        goto fail;
> > > +
> > 
> > there is a call to domain_vgic_init already in arch_domain_create
> 
> So there is!

Rather inexplicably removing either one of those two domain_vgic_init
calls causes:
        (XEN) Unexpected Trap: Data Abort
        (XEN) ----[ Xen-4.2-unstable  x86_64  debug=y  Not tainted ]----
        (XEN) CPU:    0
        (XEN) PC:     00222e7c _spin_lock+0x28/0x6c
        (XEN) CPSR:   600001da MODE:HYP
        (XEN)      R0: 002c4389 R1: 800001da R2: 00000001 R3: 0000ffff
        (XEN)      R4: 002c4381 R5: 00000080 R6: 002c4380 R7: 002c4000
        (XEN)      R8: 002c4380 R9: 4000015a R10:00000080 R11:40017d6c R12:00000000
        (XEN)      SP: 40017d5c LR: 00222e68
        (XEN) 
        (XEN) HTTBR ffec1000
        (XEN) HDFAR 2c4381
        (XEN) HIFAR 0
        (XEN) HPFAR 0
        (XEN) HCR 00000835
        (XEN) HSR   94000021
        (XEN) 
        (XEN) DFSR 817 DFAR 134bc
        (XEN) IFSR 7 IFAR 4024c224
        (XEN) 
        (XEN) Xen stack trace from sp=40017d5c:
        [...]
        (XEN) Xen call trace:
        (XEN)    [<00222e7c>] _spin_lock+0x28/0x6c
        (XEN)    [<00226270>] init_timer+0xbc/0x160
        (XEN)    [<0021fc14>] sched_init_vcpu+0x94/0x200
        (XEN)    [<002061a4>] alloc_vcpu+0x124/0x210
        (XEN)    [<00204890>] do_domctl+0xaa4/0x14e4
        (XEN)    [<00241aec>] do_trap_hypervisor+0x588/0x8cc
        (XEN)    [<0023bbf0>] return_from_trap+0x0/0x4

I'm totally at a loss to explain that. domain_vgic_init allocates two
arrays so it is possible we have some sort of overrun error, although I
can't for the life of me see it in there (it could be elsewhere though).

As an experiment I tried doubling the size of both allocations in that
function (and calling it once) but that didn't help so no hints from
that...

More head scratching required I think!

Ian.

Re: [PATCH 21/38] arm: dump guest s1 walk on data abort which is not a stage 2 issue.

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565190), Ian Campbell wrote:
> +    offset = addr >> (12+10);
> +    printk("1ST[%#03"PRIx32"] (%#"PRIpaddr") = %#010"PRIx32"\n",

Nit: 0x%08 prints '0' as '0x00000000', which is nicer I think.

Otherwise: ack.

Tim.

Re: [PATCH 26/38] arm: fix locking in create_p2m_entries

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565195), Ian Campbell wrote:
> For some reason we were holding the lock over only the unmaps at the end of
> the function, rather than for the whole walk.
> 
> We might want to be more clever in the future, but for now lets just lock for
> the whole walk+create process.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 28/38] arm: map GICV in all domains, not just dom0.

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565197), Ian Campbell wrote:
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index a7fb227..e15c1e8 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -329,13 +329,17 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
>  
>          if ( (rc = p2m_alloc_table(d)) != 0 )
>              goto fail;
> -    }
>  
> -    if ( (rc = domain_vgic_init(d)) != 0 )
> -        goto fail;
> +        if ( (rc = gicv_setup(d)) != 0 )
> +            goto fail;
> +
> +        if ( (rc = domain_vgic_init(d)) != 0 )
> +            goto fail;
> +    }
>  
>      rc = 0;
>  fail:
> +    /*XXX unwind allocations etc */

Ahem. :)

Tim.

Re: [PATCH 18/38] arm: implement vpl011 (UART) emulator.

[Tim Deegan]

At 10:34 +0100 on 07 Jun (1339065291), Ian Campbell wrote:
> On Thu, 2012-06-07 at 10:29 +0100, Tim Deegan wrote:
> > > +int domain_uart0_init(struct domain *d)
> > > +{
> > > +    int rc;
> > > +    if ( d->domain_id == 0 )
> > > +        return 0;
> > 
> > Why?  There's no equivalent gate on the MMIO handlers.
> 
> dom0 has the actual uart mapped at this address, not the emulated one.
> Maybe that should be written at the caller instead?

Yes - ideally in the same place that puts the MMIO hooks in that will
call the other functions in this file. 

Tim.

Re: [PATCH 30/38] arm: Upgrade guest barriers to Outer-Shareable. Enable Protected Table Walk.

[Tim Deegan]

At 15:39 +0000 on 01 Jun (1338565199), Ian Campbell wrote:
> Upgrading barriers is conservative and may not be necessary.
> 
> Protected Table Walk traps stage 1 page tables which refer to device memory
> (per stage 2) using a non-device mapping. This generally indicates a guest
> error but trapping it as a fault for now helps us know if something odd is
> going on.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 31/38] arm: gic.lock can be taken in interrupt context, so lock appropriately.

[Tim Deegan]

At 15:40 +0000 on 01 Jun (1338565200), Ian Campbell wrote:
> In particular it is taken by gic_set_guest_irq which is called by
> vgic_vcpu_inject_irq
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 32/38] arm: context switch virtual timer registers

[Tim Deegan]

At 15:40 +0000 on 01 Jun (1338565201), Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 33/38] arm: the hyp timer seems to work now, default to using it.

[Tim Deegan]

At 15:40 +0000 on 01 Jun (1338565202), Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 35/38] arm: move PSR flag definitions into interface, for tools use.

[Tim Deegan]

At 15:40 +0000 on 01 Jun (1338565204), Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 26/38] arm: fix locking in create_p2m_entries

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> For some reason we were holding the lock over only the unmaps at the end of
> the function, rather than for the whole walk.
> 
> We might want to be more clever in the future, but for now lets just lock for
> the whole walk+create process.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/p2m.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index 46c6f17..c4daf83 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -168,6 +168,8 @@ static int create_p2m_entries(struct domain *d,
>      paddr_t addr;
>      unsigned long cur_first_offset = ~0, cur_second_offset = ~0;
>  
> +    spin_lock(&p2m->lock);
> +
>      /* XXX Don't actually handle 40 bit guest physical addresses */
>      BUG_ON(start_gpaddr & 0x8000000000ULL);
>      BUG_ON(end_gpaddr   & 0x8000000000ULL);
> @@ -249,8 +251,6 @@ static int create_p2m_entries(struct domain *d,
>      rc = 0;
>  
>  out:
> -    spin_lock(&p2m->lock);
> -
>      if (third) unmap_domain_page(third);
>      if (second) unmap_domain_page(second);
>      if (first) unmap_domain_page(first);
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 27/38] arm: split pending SPIs (global) out from pending PPIs and SGIs (per CPU)

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> This tracks SPIs in struct arch_domain and PPIs+SGIs in struct arch_vcpu which
> seems more logical.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/vgic.c          |   17 ++++++++++-------
>  xen/include/asm-arm/domain.h |   10 ++++++++++
>  2 files changed, 20 insertions(+), 7 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 629a0da..91d6166 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -82,9 +82,8 @@ int domain_vgic_init(struct domain *d)
>      d->arch.vgic.shared_irqs =
>          xmalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
>      d->arch.vgic.pending_irqs =
> -        xmalloc_array(struct pending_irq,
> -                d->arch.vgic.nr_lines + (32 * d->max_vcpus));
> -    for (i=0; i<d->arch.vgic.nr_lines + (32 * d->max_vcpus); i++)
> +        xzalloc_array(struct pending_irq, d->arch.vgic.nr_lines);
> +    for (i=0; i<d->arch.vgic.nr_lines; i++)
>          INIT_LIST_HEAD(&d->arch.vgic.pending_irqs[i].inflight);
>      for (i=0; i<DOMAIN_NR_RANKS(d); i++)
>          spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
> @@ -98,6 +97,10 @@ int vcpu_vgic_init(struct vcpu *v)
>  
>      spin_lock_init(&v->arch.vgic.private_irqs.lock);
>  
> +    memset(&v->arch.vgic.pending_irqs, 0, sizeof(v->arch.vgic.pending_irqs));
> +    for (i = 0; i < 32; i++)
> +        INIT_LIST_HEAD(&v->arch.vgic.pending_irqs[i].inflight);
> +
>      /* For SGI and PPI the target is always this CPU */
>      for ( i = 0 ; i < 8 ; i++ )
>          v->arch.vgic.private_irqs.itargets[i] =
> @@ -535,8 +538,7 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
>      /* Pending irqs allocation strategy: the first vgic.nr_lines irqs
>       * are used for SPIs; the rests are used for per cpu irqs */
>      if ( irq < 32 )
> -        n = &v->domain->arch.vgic.pending_irqs[irq + (v->vcpu_id * 32)
> -            + v->domain->arch.vgic.nr_lines];
> +        n = &v->arch.vgic.pending_irqs[irq];
>      else
>          n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>      return n;
> @@ -548,6 +550,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
>      uint8_t priority;
>      struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
>      struct pending_irq *iter, *n = irq_to_pending(v, irq);
> +    unsigned long flags;
>  
>      /* irq still pending */
>      if (!list_empty(&n->inflight))
> @@ -564,7 +567,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
>  
>      gic_set_guest_irq(irq, GICH_LR_PENDING, priority);
>  
> -    spin_lock(&v->arch.vgic.lock);
> +    spin_lock_irqsave(&v->arch.vgic.lock, flags);
>      list_for_each_entry ( iter, &v->arch.vgic.inflight_irqs, inflight )
>      {
>          if ( iter->priority > priority )
> @@ -575,7 +578,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
>          }
>      }
>      list_add_tail(&n->inflight, &v->arch.vgic.inflight_irqs);
> -    spin_unlock(&v->arch.vgic.lock);
> +    spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
>      /* we have a new higher priority irq, inject it into the guest */
>  }

Besides moving PPIs and SGIs to struct vcpu, this patch also turns
spin_lock into spin_lock_irqsave in vgic_vcpu_inject_irq: I think it is
correct because it can be called in IRQ context, but it needs to be
explicitly stated in the commit message.


> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 620b26e..32deb52 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -46,6 +46,10 @@ struct arch_domain
>          int ctlr;
>          int nr_lines;
>          struct vgic_irq_rank *shared_irqs;
> +        /*
> +         * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
> +         * struct arch_vcpu.
> +         */
>          struct pending_irq *pending_irqs;
>      } vgic;
>  
> @@ -114,7 +118,13 @@ struct arch_vcpu
>      uint32_t gic_lr[64];
>  
>      struct {
> +        /*
> +         * SGIs and PPIs are per-VCPU, SPIs are domain global and in
> +         * struct arch_domain.
> +         */
> +        struct pending_irq pending_irqs[32];
>          struct vgic_irq_rank private_irqs;
> +
>          /* This list is ordered by IRQ priority and it is used to keep
>           * track of the IRQs that the VGIC injected into the guest.
>           * Depending on the availability of LR registers, the IRQs might
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 28/38] arm: map GICV in all domains, not just dom0.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> This requires that we allocate all p2m pages from domheap without a particular
> dom because max pages is not setup yet so there is no allocation available to
> us.
> 
> At some point we should create a separate p2m allocation (similar to x86's shadow allocation) and use that.
> 
> Also we seem to have been calling p2m_alloc_table twice for dom0.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/domain.c       |   10 +++++++---
>  xen/arch/arm/domain_build.c |    5 -----
>  xen/arch/arm/gic.c          |    9 ++++-----
>  xen/arch/arm/gic.h          |    2 +-
>  xen/arch/arm/p2m.c          |    3 ++-
>  5 files changed, 14 insertions(+), 15 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index a7fb227..e15c1e8 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -329,13 +329,17 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
>  
>          if ( (rc = p2m_alloc_table(d)) != 0 )
>              goto fail;
> -    }
>  
> -    if ( (rc = domain_vgic_init(d)) != 0 )
> -        goto fail;
> +        if ( (rc = gicv_setup(d)) != 0 )
> +            goto fail;
> +
> +        if ( (rc = domain_vgic_init(d)) != 0 )
> +            goto fail;
> +    }
>  
>      rc = 0;
>  fail:
> +    /*XXX unwind allocations etc */
>      return rc;
>  }
>  
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 72e775c..1b19e54 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -270,9 +270,6 @@ int construct_dom0(struct domain *d)
>  
>      d->max_pages = ~0U;
>  
> -    if ( (rc = p2m_alloc_table(d)) != 0 )
> -        return rc;
> -
>      rc = prepare_dtb(d, &kinfo);
>      if ( rc < 0 )
>          return rc;
> @@ -288,8 +285,6 @@ int construct_dom0(struct domain *d)
>      printk("Map VGIC MMIO regions 1:1 in the P2M %#llx->%#llx\n", 0x2C008000ULL, 0x2DFFFFFFULL);
>      map_mmio_regions(d, 0x2C008000, 0x2DFFFFFF, 0x2C008000);
>  
> -    gicv_setup(d);
> -
>      printk("Routing peripheral interrupts to guest\n");
>      /* TODO Get from device tree */
>      gic_route_irq_to_guest(d, 34, "timer0");
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 339c327..a398f92 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -541,14 +541,13 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
>      do_IRQ(regs, irq, is_fiq);
>  }
>  
> -void gicv_setup(struct domain *d)
> +int gicv_setup(struct domain *d)
>  {
> +    printk("GICV setup for DOM%d\n", d->domain_id);
> +
>      /* map the gic virtual cpu interface in the gic cpu interface region of
>       * the guest */
> -    printk("mapping GICC at %#"PRIx32" to %#"PRIx32"\n",
> -           GIC_BASE_ADDRESS + GIC_CR_OFFSET,
> -           GIC_BASE_ADDRESS + GIC_VR_OFFSET);
> -    map_mmio_regions(d, GIC_BASE_ADDRESS + GIC_CR_OFFSET,
> +    return map_mmio_regions(d, GIC_BASE_ADDRESS + GIC_CR_OFFSET,
>                          GIC_BASE_ADDRESS + GIC_CR_OFFSET + (2 * PAGE_SIZE) - 1,
>                          GIC_BASE_ADDRESS + GIC_VR_OFFSET);
>  }
> diff --git a/xen/arch/arm/gic.h b/xen/arch/arm/gic.h
> index ac9cf3a..018d820 100644
> --- a/xen/arch/arm/gic.h
> +++ b/xen/arch/arm/gic.h
> @@ -148,7 +148,7 @@ extern void gic_init_secondary_cpu(void);
>  /* Take down a CPU's per-CPU GIC interface */
>  extern void gic_disable_cpu(void);
>  /* setup the gic virtual interface for a guest */
> -extern void gicv_setup(struct domain *d);
> +extern int gicv_setup(struct domain *d);
>  
>  /* Context switch */
>  extern void gic_save_state(struct vcpu *v);
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index c4daf83..0665445 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -4,6 +4,7 @@
>  #include <xen/errno.h>
>  #include <xen/domain_page.h>
>  #include <asm/flushtlb.h>
> +#include "gic.h"
>  
>  void dump_p2m_lookup(struct domain *d, paddr_t addr)
>  {
> @@ -138,7 +139,7 @@ static int p2m_create_entry(struct domain *d,
>  
>      BUG_ON(entry->p2m.valid);
>  
> -    page = alloc_domheap_page(d, 0);
> +    page = alloc_domheap_page(NULL, 0);
>      if ( page == NULL )
>          return -ENOMEM;
>  
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 29/38] arm: delay enabling data-cache until paging enabled.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> cached before paging was enabled had been mapped with to inconsistent sets of
> attributes. I'm not convinced that isn't a model issue, nor am I convinced
> this has really fixed anything, but it seems sensible enough.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/head.S |    9 +++++++--
>  1 files changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/head.S b/xen/arch/arm/head.S
> index 9a7714a..71197af 100644
> --- a/xen/arch/arm/head.S
> +++ b/xen/arch/arm/head.S
> @@ -148,10 +148,11 @@ hyp:
>  	 * Exceptions in LE ARM,
>  	 * Low-latency IRQs disabled,
>  	 * Write-implies-XN disabled (for now),
> -	 * I-cache and d-cache enabled,
> +	 * D-cache diabled (for now),
                 ^ misspell
> +	 * I-cache enabled,
>  	 * Alignment checking enabled,
>  	 * MMU translation disabled (for now). */
> -	ldr   r0, =(HSCTLR_BASE|SCTLR_A|SCTLR_C)
> +	ldr   r0, =(HSCTLR_BASE|SCTLR_A)
>  	mcr   CP32(r0, HSCTLR)
>  
>  	/* Write Xen's PT's paddr into the HTTBR */
> @@ -217,6 +218,10 @@ pt_ready:
>  	mov   pc, r1                 /* Get a proper vaddr into PC */
>  paging:
>  
> +	mrc   CP32(r0, HSCTLR)       /* Now enable data cache */
> +	orr   r0, r0, #(SCTLR_C)
> +	mcr   CP32(r0, HSCTLR)
> +
>  #ifdef EARLY_UART_ADDRESS
>  	/* Recover the UART address in the new address space. */
>  	lsl   r11, #11
> -- 
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 31/38] arm: gic.lock can be taken in interrupt context, so lock appropriately.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> In particular it is taken by gic_set_guest_irq which is called by
> vgic_vcpu_inject_irq
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/gic.c |   20 ++++++++++----------
>  1 files changed, 10 insertions(+), 10 deletions(-)
> 
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index a398f92..ededa99 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -329,19 +329,19 @@ int __init gic_init(void)
>  /* Set up the per-CPU parts of the GIC for a secondary CPU */
>  void __cpuinit gic_init_secondary_cpu(void)
>  {
> -    spin_lock(&gic.lock);
> +    spin_lock_irq(&gic.lock);
>      gic_cpu_init();
>      gic_hyp_init();
> -    spin_unlock(&gic.lock);
> +    spin_unlock_irq(&gic.lock);
>  }
>  
>  /* Shut down the per-CPU GIC interface */
>  void gic_disable_cpu(void)
>  {
> -    spin_lock(&gic.lock);
> +    spin_lock_irq(&gic.lock);
>      gic_cpu_disable();
>      gic_hyp_disable();
> -    spin_unlock(&gic.lock);
> +    spin_unlock_irq(&gic.lock);
>  }
>  
>  void gic_route_irqs(void)
> @@ -439,7 +439,7 @@ void gic_set_guest_irq(unsigned int virtual_irq,
>  
>      events_maintenance(current);
>  
> -    spin_lock(&gic.lock);
> +    spin_lock_irq(&gic.lock);
>  
>      if ( list_empty(&gic.lr_pending) )
>      {
> @@ -465,7 +465,7 @@ void gic_set_guest_irq(unsigned int virtual_irq,
>      list_add_tail(&n->lr_queue, &gic.lr_pending);
>  
>  out:
> -    spin_unlock(&gic.lock);
> +    spin_unlock_irq(&gic.lock);
>      return;
>  }
>  
> @@ -559,7 +559,7 @@ static void events_maintenance(struct vcpu *v)
>              (unsigned long *)&vcpu_info(v, evtchn_upcall_pending));
>  
>      if (!already_pending && gic.event_mask != 0) {
> -        spin_lock(&gic.lock);
> +        spin_lock_irq(&gic.lock);
>          while ((i = find_next_bit((const long unsigned int *) &gic.event_mask,
>                          sizeof(uint64_t), i)) < sizeof(uint64_t)) {
>  
> @@ -569,7 +569,7 @@ static void events_maintenance(struct vcpu *v)
>  
>              i++;
>          }
> -        spin_unlock(&gic.lock);
> +        spin_unlock_irq(&gic.lock);
>      }
>  }
>  
> @@ -585,7 +585,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
>                                sizeof(eisr), i)) < sizeof(eisr)) {
>          struct pending_irq *p;
>  
> -        spin_lock(&gic.lock);
> +        spin_lock_irq(&gic.lock);
>          lr = GICH[GICH_LR + i];
>          virq = lr & GICH_LR_VIRTUAL_MASK;
>          GICH[GICH_LR + i] = 0;
> @@ -601,7 +601,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
>          } else {
>              gic_inject_irq_stop();
>          }
> -        spin_unlock(&gic.lock);
> +        spin_unlock_irq(&gic.lock);
>  
>          spin_lock(&current->arch.vgic.lock);
               ^
shouldn't you change this into spin_lock_irq too?

Re: [PATCH 34/38] HACK: arm: initial XENMAPSPACE_gmfn_foreign

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Should use same interface as hybrid x86.
> ---
>  xen/arch/arm/mm.c             |   32 ++++++++++++++++++++++++++------
>  xen/arch/x86/mm.c             |    2 ++
>  xen/include/public/arch-arm.h |    1 +
>  xen/include/public/memory.h   |   12 +++++++-----
>  4 files changed, 36 insertions(+), 11 deletions(-)
> 
> diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> index ab52171..1832e7f 100644
> --- a/xen/arch/arm/mm.c
> +++ b/xen/arch/arm/mm.c
> @@ -480,12 +480,32 @@ static int xenmem_add_to_physmap_once(
>  
>      switch ( xatp->space )
>      {
> -        case XENMAPSPACE_shared_info:
> -            if ( xatp->idx == 0 )
> -                mfn = virt_to_mfn(d->shared_info);
> -            break;
> -        default:
> -            return -ENOSYS;
> +    case XENMAPSPACE_shared_info:
> +        if ( xatp->idx == 0 )
> +            mfn = virt_to_mfn(d->shared_info);
> +        break;
> +    case XENMAPSPACE_gmfn_foreign:
> +    {
> +        paddr_t maddr;
> +        struct domain *od;
> +
> +        rc = rcu_lock_target_domain_by_id(xatp->foreign_domid, &od);
> +        if ( rc < 0 )
> +            return rc;
> +        maddr = p2m_lookup(od, xatp->idx << PAGE_SHIFT);
> +        if ( maddr == INVALID_PADDR )
> +        {
> +            printk("bad p2m lookup\n");
> +            dump_p2m_lookup(od, xatp->idx << PAGE_SHIFT);
> +            rcu_unlock_domain(od);
> +            return -EINVAL;
> +        }
> +        mfn = maddr >> PAGE_SHIFT;
> +        rcu_unlock_domain(od);
> +        break;
> +    }

It is probably a good idea at least to test xatp->size and WARN if it is
not 1 page.


> +    default:
> +        return -ENOSYS;
>      }
>  
>      domain_lock(d);
> diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
> index 876e1ef..d6c90f9 100644
> --- a/xen/arch/x86/mm.c
> +++ b/xen/arch/x86/mm.c
> @@ -4572,6 +4572,8 @@ static int xenmem_add_to_physmap_once(
>              mfn = idx;
>              page = mfn_to_page(mfn);
>              break;
> +        case XENMAPSPACE_gmfn_foreign:
> +            return -ENOSYS;
>          }
>          default:
>              break;
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index e915cbf..b52bfc7 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -121,6 +121,7 @@ typedef uint64_t xen_pfn_t;
>  #define XEN_LEGACY_MAX_VCPUS 1
>  
>  typedef uint32_t xen_ulong_t;
> +#define PRI_xen_ulong PRIx32
>  
>  struct vcpu_guest_context {
>      struct cpu_user_regs user_regs;         /* User-level CPU registers     */

Why did you need to define this here?

Re: [PATCH 38/38] HACK: arm: disable hypercall continuations.

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
>  xen/include/xen/sched.h |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
> index 53804c8..15fa6b4 100644
> --- a/xen/include/xen/sched.h
> +++ b/xen/include/xen/sched.h
> @@ -577,10 +577,14 @@ unsigned long hypercall_create_continuation(
>      unsigned int op, const char *format, ...);
>  void hypercall_cancel_continuation(void);
>  
> +#ifdef CONFIG_ARM
> +#define hypercall_preempt_check() (0)
> +#else
>  #define hypercall_preempt_check() (unlikely(    \
>          softirq_pending(smp_processor_id()) |   \
>          local_events_need_delivery()            \
>      ))
> +#endif


I think it is fine for now, but could we add a TODO comment on it?

Re: [PATCH 16/38] arm: Add simple cpu_{sibling, core}_mask

[Ian Campbell]

(Keir, this slightly touches common code...)

On Thu, 2012-06-07 at 10:08 +0100, Tim Deegan wrote:
> At 15:39 +0000 on 01 Jun (1338565185), Ian Campbell wrote:
> > @@ -230,6 +230,13 @@ void __init start_xen(unsigned long boot_phys_offset,
> >          }
> >      }
> >  
> > +    if ( !zalloc_cpumask_var(&per_cpu(cpu_sibling_mask, 0)) ||
> > +         !zalloc_cpumask_var(&per_cpu(cpu_core_mask, 0)) )
> > +        BUG();
> > +
> > +    cpumask_clear(per_cpu(cpu_sibling_mask, 0));
> > +    cpumask_clear(per_cpu(cpu_core_mask, 0));
> 
> Aren't these clear()s noops?  

Yes, they were also incorrect since a CPU is it's own sibling and shares
a core with itself. Otherwise all manner of weirdness ensues (see commit
message). These also need to be setup on all CPUs.

I replaced this patch with the following.

Ian.

8<------------------------------------------------------

>From e980ca1ec9bf92b2f1255ac5222b1da1292f9f72 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Mon, 14 May 2012 12:25:31 +0100
Subject: [PATCH] arm: Add simple cpu_{sibling,core}_mask

This needs to be done for all cpus. The allocations require smp_prepare_cpus
to be called a bit later on.

In a previous version of this patch these maps were being zeroed (instead of
setting the CPU itself in them). This in turn causes cpumask_first to return
NR_CPUS, which in turn was causing default_vcpu0_location to misbehave and
read off the end of its cnt array. Add a couple of asserts to catch this in
the future.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/dummy.S   |    2 --
 xen/arch/arm/setup.c   |    4 ++--
 xen/arch/arm/smpboot.c |   21 +++++++++++++++++++++
 xen/common/domctl.c    |    2 ++
 4 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
index c001e8d..03f7489 100644
--- a/xen/arch/arm/dummy.S
+++ b/xen/arch/arm/dummy.S
@@ -7,8 +7,6 @@ x:	.word 0xe7f000f0 /* Undefined instruction */
 x:	mov pc, lr
 	
 /* SMP support */
-DUMMY(per_cpu__cpu_core_mask);
-DUMMY(per_cpu__cpu_sibling_mask);
 DUMMY(node_online_map);
 DUMMY(smp_send_state_dump);
 
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index 81ababb..d6c0178 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -173,8 +173,6 @@ void __init start_xen(unsigned long boot_phys_offset,
     set_current((struct vcpu *)0xfffff000); /* debug sanity */
     idle_vcpu[0] = current;
 
-    smp_prepare_cpus(cpus);
-
     init_xen_time();
 
     setup_mm(atag_paddr, fdt_size);
@@ -214,6 +212,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
     local_irq_enable();
 
+    smp_prepare_cpus(cpus);
+
     initialize_keytable();
 
     console_init_postirq();
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index ea05afc..6463a8d 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -52,6 +52,23 @@ unsigned long __initdata ready_cpus = 0;
 
 /* ID of the PCPU we're running on */
 DEFINE_PER_CPU(unsigned int, cpu_id);
+/* XXX these seem awfully x86ish... */
+/* representing HT siblings of each logical CPU */
+DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask);
+/* representing HT and core siblings of each logical CPU */
+DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
+
+static void setup_cpu_sibling_map(int cpu)
+{
+    if ( !zalloc_cpumask_var(&per_cpu(cpu_sibling_mask, cpu)) ||
+         !zalloc_cpumask_var(&per_cpu(cpu_core_mask, cpu)) )
+        panic("No memory for CPU sibling/core maps\n");
+
+    /* A CPU is a sibling with itself and is always on its own core. */
+    cpumask_set_cpu(cpu, per_cpu(cpu_sibling_mask, cpu));
+    cpumask_set_cpu(cpu, per_cpu(cpu_core_mask, cpu));
+}
+
 
 void __init
 smp_prepare_cpus (unsigned int max_cpus)
@@ -65,6 +82,8 @@ smp_prepare_cpus (unsigned int max_cpus)
     for ( i = 0; i < max_cpus; i++ )
         cpumask_set_cpu(i, &cpu_possible_map);
     cpumask_copy(&cpu_present_map, &cpu_possible_map);
+
+    setup_cpu_sibling_map(0);
 }
 
 void __init
@@ -115,6 +134,8 @@ void __cpuinit start_secondary(unsigned long boot_phys_offset,
 
     set_current(idle_vcpu[cpuid]);
 
+    setup_cpu_sibling_map(cpuid);
+
     /* Run local notifiers */
     notify_cpu_starting(cpuid);
     wmb();
diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index 9f1a9ad..c1acd1d 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -190,10 +190,12 @@ static unsigned int default_vcpu0_location(cpumask_t *online)
      */
     cpumask_copy(&cpu_exclude_map, per_cpu(cpu_sibling_mask, 0));
     cpu = cpumask_first(&cpu_exclude_map);
+    ASSERT(cpu < nr_cpus);
     if ( cpumask_weight(&cpu_exclude_map) > 1 )
         cpu = cpumask_next(cpu, &cpu_exclude_map);
     for_each_cpu(i, online)
     {
+        ASSERT(i < nr_cpus);
         if ( cpumask_test_cpu(i, &cpu_exclude_map) )
             continue;
         if ( (i == cpumask_first(per_cpu(cpu_sibling_mask, i))) &&
-- 
1.7.9.1

Re: [PATCH 36/38] libxc: add ARM support to xc_dom (PV domain building)

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Includes ARM zImage support.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  tools/libxc/Makefile                 |    1 +
>  tools/libxc/xc_dom.h                 |    5 +-
>  tools/libxc/xc_dom_arm.c             |  135 +++++++++++++++++++++++++++-
>  tools/libxc/xc_dom_armzimageloader.c |  167 ++++++++++++++++++++++++++++++++++
>  tools/libxc/xc_dom_core.c            |   12 ++-
>  tools/libxc/xg_private.h             |    4 +
>  6 files changed, 315 insertions(+), 9 deletions(-)
>  create mode 100644 tools/libxc/xc_dom_armzimageloader.c
> 
> diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
> index ca38cbd..a01d457 100644
> --- a/tools/libxc/Makefile
> +++ b/tools/libxc/Makefile
> @@ -59,6 +59,7 @@ GUEST_SRCS-y += libelf-dominfo.c libelf-relocate.c
>  GUEST_SRCS-y                 += xc_dom_core.c xc_dom_boot.c
>  GUEST_SRCS-y                 += xc_dom_elfloader.c
>  GUEST_SRCS-$(CONFIG_X86)     += xc_dom_bzimageloader.c
> +GUEST_SRCS-$(CONFIG_ARM)     += xc_dom_armzimageloader.c
>  GUEST_SRCS-y                 += xc_dom_binloader.c
>  GUEST_SRCS-y                 += xc_dom_compat_linux.c
> 
> diff --git a/tools/libxc/xc_dom.h b/tools/libxc/xc_dom.h
> index 2aef64a..4db8fad 100644
> --- a/tools/libxc/xc_dom.h
> +++ b/tools/libxc/xc_dom.h
> @@ -93,6 +93,7 @@ struct xc_dom_image {
>      void *p2m_guest;
> 
>      /* physical memory */
> +    xen_pfn_t rambase_pfn;
>      xen_pfn_t total_pages;
>      struct xc_dom_phys *phys_pages;
>      int realmodearea_log;
> @@ -286,7 +287,7 @@ static inline xen_pfn_t xc_dom_p2m_host(struct xc_dom_image *dom, xen_pfn_t pfn)
>  {
>      if (dom->shadow_enabled)
>          return pfn;
> -    return dom->p2m_host[pfn];
> +    return dom->p2m_host[pfn - dom->rambase_pfn];
>  }
> 
>  static inline xen_pfn_t xc_dom_p2m_guest(struct xc_dom_image *dom,
> @@ -294,7 +295,7 @@ static inline xen_pfn_t xc_dom_p2m_guest(struct xc_dom_image *dom,
>  {
>      if (xc_dom_feature_translated(dom))
>          return pfn;
> -    return dom->p2m_host[pfn];
> +    return dom->p2m_host[pfn - dom->rambase_pfn];
>  }

I take that rambase_pfn is the offset in the guest physical address
space where the ram is located. It would be nice to write it down.


>  /* --- arch bits --------------------------------------------------- */
> diff --git a/tools/libxc/xc_dom_arm.c b/tools/libxc/xc_dom_arm.c
> index 122d0e8..9099cad 100644
> --- a/tools/libxc/xc_dom_arm.c
> +++ b/tools/libxc/xc_dom_arm.c
> @@ -18,14 +18,138 @@
>   * Copyright (c) 2011, Citrix Systems
>   */
>  #include <inttypes.h>
> +
>  #include <xen/xen.h>
> +#include <xen/io/protocols.h>
> +
>  #include "xg_private.h"
>  #include "xc_dom.h"
> 
> +/* ------------------------------------------------------------------------ */
> +/*
> + * arm guests are hybrid and start off with paging disabled, therefore no
> + * pagetables and nothing to do here.
> + */
> +static int count_pgtables_arm(struct xc_dom_image *dom)
> +{
> +    DOMPRINTF_CALLED(dom->xch);
> +    return 0;
> +}
> +
> +static int setup_pgtables_arm(struct xc_dom_image *dom)
> +{
> +    DOMPRINTF_CALLED(dom->xch);
> +    return 0;
> +}
> +
> +/* ------------------------------------------------------------------------ */
> +
> +static int alloc_magic_pages(struct xc_dom_image *dom)
> +{
> +    DOMPRINTF_CALLED(dom->xch);
> +    /* XXX
> +     *   dom->p2m_guest
> +     *   dom->start_info_pfn
> +     *   dom->xenstore_pfn
> +     *   dom->console_pfn
> +     */
> +    return 0;
> +}
> +
> +/* ------------------------------------------------------------------------ */
> +
> +static int start_info_arm(struct xc_dom_image *dom)
> +{
> +    DOMPRINTF_CALLED(dom->xch);
> +    /* XXX */
> +    return 0;
> +}
> +
> +static int shared_info_arm(struct xc_dom_image *dom, void *ptr)
> +{
> +    DOMPRINTF_CALLED(dom->xch);
> +    /* XXX */
> +    return 0;
> +}
> +
> +/* ------------------------------------------------------------------------ */
> +
> +static int vcpu_arm(struct xc_dom_image *dom, void *ptr)
> +{
> +    vcpu_guest_context_t *ctxt = ptr;
> +
> +    DOMPRINTF_CALLED(dom->xch);
> +
> +    /* clear everything */
> +    memset(ctxt, 0, sizeof(*ctxt));
> +
> +    ctxt->user_regs.pc = dom->parms.virt_entry;
> +    ctxt->user_regs.r0 = 0; /* SBZ */
> +    ctxt->user_regs.r1 = 2272; /* Machine NR: Versatile Express */
> +
> +    ctxt->user_regs.r2 = 0xffffffff; //devicetree_seg //dtb_paddr; //atags or dtb /* XXX using APPEND right now */
> +    ctxt->user_regs.r3 = 0xdeadbeef;
> +    ctxt->sctlr = /* #define SCTLR_BASE */0x00c50078;
> +    ctxt->ttbr0 = 0;
> +    ctxt->ttbr1 = 0;
> +    ctxt->ttbcr = 0; /* Defined Reset Value */
> +
> +    ctxt->user_regs.cpsr = PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC;
> +
> +    DOMPRINTF("Initial state CPSR %#"PRIx32" PC %#"PRIx32,
> +           ctxt->user_regs.cpsr, ctxt->user_regs.pc);
> +
> +    return 0;
> +}
> +
> +/* ------------------------------------------------------------------------ */
> +
> +static struct xc_dom_arch xc_dom_32 = {
> +    .guest_type = "xen-3.0-armv7l",
> +    .native_protocol = XEN_IO_PROTO_ABI_ARM,
> +    .page_shift = PAGE_SHIFT_ARM,
> +    .sizeof_pfn = 8,
> +    .alloc_magic_pages = alloc_magic_pages,
> +    .count_pgtables = count_pgtables_arm,
> +    .setup_pgtables = setup_pgtables_arm,
> +    .start_info = start_info_arm,
> +    .shared_info = shared_info_arm,
> +    .vcpu = vcpu_arm,
> +};
> +
> +static void __init register_arch_hooks(void)
> +{
> +    xc_dom_register_arch_hooks(&xc_dom_32);
> +}
> +
>  int arch_setup_meminit(struct xc_dom_image *dom)
>  {
> -    errno = ENOSYS;
> -    return -1;
> +    int rc;
> +    xen_pfn_t pfn, allocsz, i;
> +
> +    dom->shadow_enabled = 1;
> +
> +    dom->p2m_host = xc_dom_malloc(dom, sizeof(xen_pfn_t) * dom->total_pages);
> +
> +    /* setup initial p2m */
> +    for ( pfn = 0; pfn < dom->total_pages; pfn++ )
> +        dom->p2m_host[pfn] = pfn + dom->rambase_pfn;

uhm.. so maybe rambase_pfn is the offset in the machine address space where
the guest ram has been allocated?


> +    /* allocate guest memory */
> +    for ( i = rc = allocsz = 0;
> +          (i < dom->total_pages) && !rc;
> +          i += allocsz )
> +    {
> +        allocsz = dom->total_pages - i;
> +        if ( allocsz > 1024*1024 )
> +            allocsz = 1024*1024;
> +
> +        rc = xc_domain_populate_physmap_exact(
> +            dom->xch, dom->guest_domid, allocsz,
> +            0, 0, &dom->p2m_host[i]);
> +    }
> +
> +    return 0;
>  }
> 
>  int arch_setup_bootearly(struct xc_dom_image *dom)
> @@ -36,9 +160,14 @@ int arch_setup_bootearly(struct xc_dom_image *dom)
> 
>  int arch_setup_bootlate(struct xc_dom_image *dom)
>  {
> -    DOMPRINTF("%s: doing nothing", __FUNCTION__);
> +    /* XXX
> +     *   map shared info
> +     *   map grant tables
> +     *   setup shared info
> +     */
>      return 0;
>  }
> +
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/tools/libxc/xc_dom_armzimageloader.c b/tools/libxc/xc_dom_armzimageloader.c
> new file mode 100644
> index 0000000..220176d
> --- /dev/null
> +++ b/tools/libxc/xc_dom_armzimageloader.c
> @@ -0,0 +1,167 @@
> +/*
> + * Xen domain builder -- ARM zImage bits
> + *
> + * Parse and load ARM zImage kernel images.
> + *
> + * Copyright (C) 2012, Citrix Systems.
> + *
> + * This library is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation;
> + * version 2.1 of the License.
> + *
> + * This library is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with this library; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
> + *
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <inttypes.h>
> +
> +#include "xg_private.h"
> +#include "xc_dom.h"
> +
> +#include <arpa/inet.h> /* XXX ntohl is not the right function... */

Yes, you are right, we should write a be32_to_cpu (the ones in Xen, QEMU
and Linux are GPLv2 rather than LGLPv2).


> +#define ZIMAGE_MAGIC_OFFSET 0x24
> +#define ZIMAGE_START_OFFSET 0x28
> +#define ZIMAGE_END_OFFSET   0x2c
> +
> +#define ZIMAGE_MAGIC 0x016f2818
> +
> +struct minimal_dtb_header {
> +    uint32_t magic;
> +    uint32_t total_size;
> +    /* There are other fields but we don't use them yet. */
> +};
> +
> +#define DTB_MAGIC 0xd00dfeed
> +
> +static int xc_dom_probe_zimage_kernel(struct xc_dom_image *dom)
> +{
> +    uint32_t *zimage;
> +    uint32_t end;
> +
> +    if ( dom->kernel_blob == NULL )
> +    {
> +        xc_dom_panic(dom->xch, XC_INTERNAL_ERROR,
> +                     "%s: no kernel image loaded", __FUNCTION__);
> +        return -EINVAL;
> +    }
> +
> +    if ( dom->kernel_size < 0x30 /*sizeof(struct setup_header)*/ )
> +    {
> +        xc_dom_printf(dom->xch, "%s: kernel image too small", __FUNCTION__);
> +        return -EINVAL;
> +    }
> +
> +    zimage = (uint32_t *)dom->kernel_blob;
> +    if ( zimage[ZIMAGE_MAGIC_OFFSET/4] != ZIMAGE_MAGIC )
> +    {
> +        xc_dom_printf(dom->xch, "%s: kernel is not a bzImage", __FUNCTION__);
> +        return -EINVAL;
> +    }
> +
> +    end = zimage[ZIMAGE_END_OFFSET/4];
> +
> +    /*
> +     * Check for an appended DTB.
> +     */
> +    if ( end + sizeof(struct minimal_dtb_header) < dom->kernel_size ) {
> +        struct minimal_dtb_header *dtb_hdr;
> +        dtb_hdr = (struct minimal_dtb_header *)(dom->kernel_blob + end);
> +        if (ntohl/*be32_to_cpu*/(dtb_hdr->magic) == DTB_MAGIC) {
> +            xc_dom_printf(dom->xch, "%s: found an appended DTB", __FUNCTION__);
> +            end += ntohl/*be32_to_cpu*/(dtb_hdr->total_size);
> +        }
> +    }
> +
> +    dom->kernel_size = end;
> +
> +    return 0;
> +}
> +
> +static int xc_dom_parse_zimage_kernel(struct xc_dom_image *dom)
> +{
> +    uint32_t *zimage;
> +    uint32_t start, entry_addr;
> +    uint64_t v_start, v_end;
> +    uint64_t rambase = 0x80000000; /* XXX */
> +
> +    DOMPRINTF_CALLED(dom->xch);
> +
> +    zimage = (uint32_t *)dom->kernel_blob;
> +
> +    dom->rambase_pfn = rambase >> XC_PAGE_SHIFT;
> +
> +    v_start = rambase + 0x8000; /* XXX */
> +    v_end = v_start + dom->kernel_size;
> +
> +    start = zimage[ZIMAGE_START_OFFSET/4];
> +
> +    if (start == 0)
> +        entry_addr = v_start;
> +    else
> +        entry_addr = start;
> +
> +    /* find kernel segment */
> +    dom->kernel_seg.vstart = v_start;
> +    dom->kernel_seg.vend   = v_end;
> +
> +    dom->parms.virt_entry = entry_addr;
> +
> +    dom->guest_type = "xen-3.0-armv7l";
> +    DOMPRINTF("%s: %s: RAM starts at %"PRI_xen_pfn,
> +              __FUNCTION__, dom->guest_type, dom->rambase_pfn);
> +    DOMPRINTF("%s: %s: 0x%" PRIx64 " -> 0x%" PRIx64 "",
> +              __FUNCTION__, dom->guest_type,
> +              dom->kernel_seg.vstart, dom->kernel_seg.vend);
> +    return 0;
> +}
> +
> +static int xc_dom_load_zimage_kernel(struct xc_dom_image *dom)
> +{
> +    void *dst;
> +
> +    DOMPRINTF_CALLED(dom->xch);
> +
> +    dst = xc_dom_seg_to_ptr(dom, &dom->kernel_seg);
> +
> +    DOMPRINTF("%s: kernel sed %#"PRIx64"-%#"PRIx64,
> +              __func__, dom->kernel_seg.vstart, dom->kernel_seg.vend);
> +    DOMPRINTF("%s: copy %zd bytes from blob %p to dst %p",
> +              __func__, dom->kernel_size, dom->kernel_blob, dst);
> +
> +    memcpy(dst, dom->kernel_blob, dom->kernel_size);
> +
> +    return 0;
> +}
> +
> +static struct xc_dom_loader zimage_loader = {
> +    .name = "Linux zImage (ARM)",
> +    .probe = xc_dom_probe_zimage_kernel,
> +    .parser = xc_dom_parse_zimage_kernel,
> +    .loader = xc_dom_load_zimage_kernel,
> +};
> +
> +static void __init register_loader(void)
> +{
> +    xc_dom_register_loader(&zimage_loader);
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-set-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxc/xc_dom_core.c b/tools/libxc/xc_dom_core.c
> index fea9de5..b0d48d5 100644
> --- a/tools/libxc/xc_dom_core.c
> +++ b/tools/libxc/xc_dom_core.c
> @@ -307,15 +307,17 @@ void *xc_dom_pfn_to_ptr(struct xc_dom_image *dom, xen_pfn_t pfn,
>                          xen_pfn_t count)
>  {
>      struct xc_dom_phys *phys;
> +    xen_pfn_t offset;
>      unsigned int page_shift = XC_DOM_PAGE_SHIFT(dom);
>      char *mode = "unset";
> 
> -    if ( pfn > dom->total_pages ||    /* multiple checks to avoid overflows */
> +    offset = pfn-dom->rambase_pfn;

spaces around the '-' please


> +    if ( offset > dom->total_pages ||    /* multiple checks to avoid overflows */
>           count > dom->total_pages ||
> -         pfn > dom->total_pages - count )
> +         offset > dom->total_pages - count )
>      {
> -        DOMPRINTF("%s: pfn out of range (0x%" PRIpfn " > 0x%" PRIpfn ")",
> -                  __FUNCTION__, pfn, dom->total_pages);
> +        DOMPRINTF("%s: pfn %"PRI_xen_pfn" out of range (0x%" PRIpfn " > 0x%" PRIpfn ")",
> +                  __FUNCTION__, pfn, offset, dom->total_pages);
>          return NULL;
>      }
> 
> @@ -599,6 +601,8 @@ struct xc_dom_image *xc_dom_allocate(xc_interface *xch,
>      dom->parms.virt_hv_start_low = UNSET_ADDR;
>      dom->parms.elf_paddr_offset = UNSET_ADDR;
> 
> +    dom->rambase_pfn = 0;
> +
>      dom->alloc_malloc += sizeof(*dom);
>      return dom;

There is no need to explicitly set rambase_pfn to 0, because the whole
dom struct is memset to 0 few lines above.



> diff --git a/tools/libxc/xg_private.h b/tools/libxc/xg_private.h
> index a29fa26..a271942 100644
> --- a/tools/libxc/xg_private.h
> +++ b/tools/libxc/xg_private.h
> @@ -148,6 +148,10 @@ typedef l4_pgentry_64_t l4_pgentry_t;
>  #define l4_table_offset(_a) l4_table_offset_x86_64(_a)
>  #endif
> 
> +#define PAGE_SHIFT_ARM          12
> +#define PAGE_SIZE_ARM           (1UL << PAGE_SHIFT_ARM)
> +#define PAGE_MASK_ARM           (~(PAGE_SIZE_ARM-1))
> +
>  #define PAGE_SHIFT_X86          12
>  #define PAGE_SIZE_X86           (1UL << PAGE_SHIFT_X86)
>  #define PAGE_MASK_X86           (~(PAGE_SIZE_X86-1))
> --
> 1.7.9.1
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [PATCH 37/38] HACK: add simple xcbuild

[Stefano Stabellini]

On Fri, 1 Jun 2012, Ian Campbell wrote:
> Based on init-xenstore-domain.c.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  tools/xcutils/Makefile  |    6 ++-
>  tools/xcutils/xcbuild.c |  100 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 105 insertions(+), 1 deletions(-)
>  create mode 100644 tools/xcutils/xcbuild.c
> 
> diff --git a/tools/xcutils/Makefile b/tools/xcutils/Makefile
> index 6c502f1..dcd2c84 100644
> --- a/tools/xcutils/Makefile
> +++ b/tools/xcutils/Makefile
> @@ -11,7 +11,7 @@
>  XEN_ROOT	= $(CURDIR)/../..
>  include $(XEN_ROOT)/tools/Rules.mk
>  
> -PROGRAMS = xc_restore xc_save readnotes lsevtchn
> +PROGRAMS = xc_restore xc_save readnotes lsevtchn xcbuild
>  
>  CFLAGS += -Werror
>  
> @@ -19,6 +19,7 @@ CFLAGS_xc_restore.o := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
>  CFLAGS_xc_save.o    := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore)
>  CFLAGS_readnotes.o  := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
>  CFLAGS_lsevtchn.o   := $(CFLAGS_libxenctrl)
> +CFLAGS_xcbuild.o    := $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
>  
>  .PHONY: all
>  all: build
> @@ -32,6 +33,9 @@ xc_restore: xc_restore.o
>  xc_save: xc_save.o
>  	$(CC) $(LDFLAGS) $^ -o $@ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(APPEND_LDFLAGS)
>  
> +xcbuild: xcbuild.o
> +	$(CC) $(LDFLAGS) $^ -o $@ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
> +
>  readnotes: readnotes.o
>  	$(CC) $(LDFLAGS) $^ -o $@ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
>  
> diff --git a/tools/xcutils/xcbuild.c b/tools/xcutils/xcbuild.c
> new file mode 100644
> index 0000000..8f8660e
> --- /dev/null
> +++ b/tools/xcutils/xcbuild.c
> @@ -0,0 +1,100 @@
> +#include <unistd.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +
> +#include <errno.h>
> +
> +#include <xenctrl.h>
> +#include <xentoollog.h>
> +#include <xc_dom.h>
> +
> +int main(int argc, char **argv)
> +{
> +	xentoollog_logger *logger;
> +	xc_interface *xch;
> +	int rv;
> +	const char *image;
> +	uint32_t domid;
> +	xen_domain_handle_t handle;
> +	int maxmem = 128; /* MB */ //atoi(argv[2]);
> +	int memory_kb = 2*(maxmem + 1)*1024; /* bit of slack... */
> +	struct xc_dom_image *dom;
> +
> +	image = (argc < 2) ? "guest.img" : argv[1];
> +	printf("Image: %s\n", image);
> +	printf("Memory: %dKB\n", memory_kb);
> +
> +	logger = (xentoollog_logger*)
> +		xtl_createlogger_stdiostream(stderr, XTL_DEBUG, 0);
> +	if ( logger == NULL )
> +	{
> +		perror("xtl_createlogger_stdiostream");
> +		exit(1);
> +	}
> +
> +	xch = xc_interface_open(logger, logger, 0);
> +	if ( xch == NULL )
> +	{
> +		perror("xc_interface_open");
> +		exit(1);
> +	}
> +
> +	rv = xc_dom_loginit(xch);
> +	if (rv) return rv;
> +
> +	//rv = xc_flask_context_to_sid(xch, argv[3], strlen(argv[3]), &ssid);
> +	//if (rv) return rv;
> +
> +	rv = xc_domain_create(xch, 0 /* ssid */, handle, 0 /* flags */, &domid);
> +	printf("xc_domain_create: %d (%d)\n", rv, errno);
> +	if ( rv < 0 )
> +	{
> +		perror("xc_domain_create");
> +		exit(1);
> +	}
> +
> +	printf("building dom%d\n", domid);
> +
> +	rv = xc_domain_max_vcpus(xch, domid, 1);
> +	if ( rv < 0)
> +	{
> +		perror("xc_domain_max_vcpus");
> +		exit(1);
> +	}
> +
> +	rv = xc_domain_setmaxmem(xch, domid, memory_kb);
> +	if ( rv < 0)
> +	{
> +		perror("xc_domain_setmaxmem");
> +		exit(1);
> +	}
> +
> +	dom = xc_dom_allocate(xch, "", NULL);
> +	rv = xc_dom_kernel_file(dom, image);
> +	if (rv) return rv;
> +	rv = xc_dom_boot_xen_init(dom, xch, domid);
> +	if (rv) return rv;
> +	rv = xc_dom_parse_image(dom);
> +	if (rv) return rv;
> +	rv = xc_dom_mem_init(dom, 2*maxmem);/* XXX */
> +	if (rv) return rv;
> +	rv = xc_dom_boot_mem_init(dom);
> +	if (rv) return rv;
> +	rv = xc_dom_build_image(dom);
> +	if (rv) return rv;
> +	rv = xc_dom_boot_image(dom);
> +	if (rv) return rv;
> +
> +	xc_dom_release(dom);
> +
> +	rv = xc_domain_unpause(xch, domid);
> +	if ( rv )
> +	{
> +		perror("xc_domain_unpause");
> +		exit(1);
> +	}
> +
> +	xc_interface_close(xch);
> +
> +	return 0;
> +}

It is OK but I would remove the commented out code and add a very basic
arguments check.

Re: [PATCH 16/38] arm: Add simple cpu_{sibling, core}_mask

[Tim Deegan]

At 12:35 +0100 on 07 Jun (1339072547), Ian Campbell wrote:
> From e980ca1ec9bf92b2f1255ac5222b1da1292f9f72 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Mon, 14 May 2012 12:25:31 +0100
> Subject: [PATCH] arm: Add simple cpu_{sibling,core}_mask
> 
> This needs to be done for all cpus. The allocations require smp_prepare_cpus
> to be called a bit later on.
> 
> In a previous version of this patch these maps were being zeroed (instead of
> setting the CPU itself in them). This in turn causes cpumask_first to return
> NR_CPUS, which in turn was causing default_vcpu0_location to misbehave and
> read off the end of its cnt array. Add a couple of asserts to catch this in
> the future.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

(ARM stuff) Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 02/38] arm: handy function to print a walk of the hypervisor page tables

[Ian Campbell]

On Thu, 2012-06-07 at 09:45 +0100, Tim Deegan wrote:
> Hi,
> 
> At 15:39 +0000 on 01 Jun (1338565171), Ian Campbell wrote:
> > +void dump_pt_walk(uint32_t addr)
> > +{
> > +    paddr_t second_ma, third_ma;
> > +    lpae_t *first = NULL, *second = NULL, *third = NULL;
> > +    uint64_t httbr = READ_CP64(HTTBR);
> > +
> > +    printk("Walking Hypervisor VA %#"PRIx32" via HTTBR %#"PRIx64"\n",
> > +           addr, httbr);
> > +
> > +    BUG_ON( (lpae_t *)(unsigned long)(httbr - xen_phys_offset) != xen_pgtable );
> > +    first = xen_pgtable;
> > +    printk("1ST[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
> > +           first_table_offset(addr),
> > +           first, first_table_offset(addr),
> > +           virt_to_maddr(&first[first_table_offset(addr)]),
> > +           first[first_table_offset(addr)].bits);
> > +
> > +    if ( !first[first_table_offset(addr)].pt.valid ||
> > +         !first[first_table_offset(addr)].pt.table )
> > +        goto done;
> 
> This could probably be a for loop rather than open-coding three
> almost-identical printks.

The *_table_offsets macro names are different at each stage, I suppose I
could open code the equivalent shifts, but I'd rather keep using the
access macros.

> > +
> > +    second_ma = (paddr_t)first[first_table_offset(addr)].pt.base << PAGE_SHIFT;
> > +    second = (lpae_t *)(unsigned long)(second_ma - xen_phys_offset);
> > +    printk("2ND[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
> > +           second_table_offset(addr),
> > +           second, second_table_offset(addr),
> > +           virt_to_maddr(&second[second_table_offset(addr)]),
> > +           second[second_table_offset(addr)].bits);
> > +    if ( !second[second_table_offset(addr)].pt.valid ||
> > +         !second[second_table_offset(addr)].pt.table )
> > +        goto done;
> > +
> > +    third_ma = (paddr_t)second[second_table_offset(addr)].pt.base << PAGE_SHIFT;
> > +    third = (lpae_t *)(unsigned long)(third_ma - xen_phys_offset);
> > +    printk("3RD[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
> > +           third_table_offset(addr),
> > +           third, third_table_offset(addr),
> > +           virt_to_maddr(&third[third_table_offset(addr)]),
> > +           third[third_table_offset(addr)].bits);
> > +    if ( !third[third_table_offset(addr)].pt.valid ||
> > +         !third[third_table_offset(addr)].pt.table )
> > +        goto done;
> > +
> > +done:
> > +    return;
> 
> Maybe use return above instead of goto?

Yes, good idea.

> 
> > +}
> > +
> >  /* Map a 4k page in a fixmap entry */
> >  void set_fixmap(unsigned map, unsigned long mfn, unsigned attributes)
> >  {
> > @@ -173,8 +222,8 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
> >      flush_xen_data_tlb_va(dest_va);
> >  
> >      /* Calculate virt-to-phys offset for the new location */
> > -    phys_offset = xen_paddr - (unsigned long) _start;
> > -
> > +    xen_phys_offset = phys_offset = xen_paddr - (unsigned long) _start;
> > +    
> 
> Just make phys_offset file-scope static; no need to add a new variable
> for this.

Will do.

> 
> >      /* Copy */
> >      memcpy((void *) dest_va, _start, _end - _start);
> >  
> > diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
> > index b6df64e..22c56b5 100644
> > --- a/xen/include/asm-arm/page.h
> > +++ b/xen/include/asm-arm/page.h
> > @@ -312,6 +312,8 @@ static inline uint64_t gva_to_ipa(uint32_t va)
> >  /* Bits in the PAR returned by va_to_par */
> >  #define PAR_FAULT 0x1
> >  
> > +extern void dump_pt_walk(uint32_t addr);
> > +
> 
> Maybe a comment?

Yes, good idea.

Re: [PATCH 37/38] HACK: add simple xcbuild

[Ian Campbell]

On Thu, 2012-06-07 at 12:42 +0100, Stefano Stabellini wrote:
> It is OK but I would remove the commented out code and add a very basic
> arguments check.

Thanks, but this one was just for testing etc rather than commit

In general the things with HACK in them were just for illustration and
or to allow you to exercise the non-HACK patches.

Ian.

Re: [PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Ian Campbell]

On Thu, 2012-06-07 at 09:49 +0100, Tim Deegan wrote:
> At 15:39 +0000 on 01 Jun (1338565172), Ian Campbell wrote:
> > Useful for debug but not actually used in this patch.
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/p2m.c         |   34 ++++++++++++++++++++++++++++++++++
> >  xen/include/asm-arm/page.h |    1 +
> >  2 files changed, 35 insertions(+), 0 deletions(-)
> > 
> > diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> > index 4f624d8..fdbecbc 100644
> > --- a/xen/arch/arm/p2m.c
> > +++ b/xen/arch/arm/p2m.c
> > @@ -5,6 +5,40 @@
> >  #include <xen/domain_page.h>
> >  #include <asm/flushtlb.h>
> >  
> > +void dump_p2m_lookup(struct domain *d, paddr_t addr)
> > +{
> > +    struct p2m_domain *p2m = &d->arch.p2m;
> > +    lpae_t *first = NULL, *second = NULL, *third = NULL;
> > +
> > +    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
> > +
> > +    first = __map_domain_page(p2m->first_level);
> > +    printk("1ST[%#03llx] = %#016llx\n",
> > +           first_table_offset(addr),
> > +           first[first_table_offset(addr)].bits);
> > +    if ( !first[first_table_offset(addr)].p2m.valid ||
> > +         !first[first_table_offset(addr)].p2m.table )
> > +        goto done;
> > +
> > +    second = map_domain_page(first[first_table_offset(addr)].p2m.base);
> > +    printk("2ND[%#03llx] = %#016llx\n",
> > +           second_table_offset(addr),
> > +           second[second_table_offset(addr)].bits);
> > +    if ( !second[second_table_offset(addr)].p2m.valid ||
> > +         !second[second_table_offset(addr)].p2m.table )
> > +        goto done;
> > +
> > +    third = map_domain_page(second[second_table_offset(addr)].p2m.base);
> > +    printk("3RD[%#03llx] = %#016llx\n",
> > +           third_table_offset(addr),
> > +           third[third_table_offset(addr)].bits);
> > +
> > +done:
> > +    if (third) unmap_domain_page(third);
> > +    if (second) unmap_domain_page(second);
> > +    if (first) unmap_domain_page(first);
> > +}
> 
> Can this be unified with dump_pt_walk?

Not all that easily, mainly because dump_pt_walk walks xenheap pages
(which don't need mapping) while dump_p2m_walk dumps domheap pages
(which need mapping as we go). I probably could write something generic
enough but I fear that it would be a mass of if's.

Ian.

Re: [PATCH 02/38] arm: handy function to print a walk of the hypervisor page tables

[Tim Deegan]

At 12:57 +0100 on 07 Jun (1339073860), Ian Campbell wrote:
> On Thu, 2012-06-07 at 09:45 +0100, Tim Deegan wrote:
> > Hi,
> > 
> > At 15:39 +0000 on 01 Jun (1338565171), Ian Campbell wrote:
> > > +void dump_pt_walk(uint32_t addr)
> > > +{
> > > +    paddr_t second_ma, third_ma;
> > > +    lpae_t *first = NULL, *second = NULL, *third = NULL;
> > > +    uint64_t httbr = READ_CP64(HTTBR);
> > > +
> > > +    printk("Walking Hypervisor VA %#"PRIx32" via HTTBR %#"PRIx64"\n",
> > > +           addr, httbr);
> > > +
> > > +    BUG_ON( (lpae_t *)(unsigned long)(httbr - xen_phys_offset) != xen_pgtable );
> > > +    first = xen_pgtable;
> > > +    printk("1ST[%#03"PRIx32"] = %p[%#03"PRIx32"] = %#llx = %#016llx\n",
> > > +           first_table_offset(addr),
> > > +           first, first_table_offset(addr),
> > > +           virt_to_maddr(&first[first_table_offset(addr)]),
> > > +           first[first_table_offset(addr)].bits);
> > > +
> > > +    if ( !first[first_table_offset(addr)].pt.valid ||
> > > +         !first[first_table_offset(addr)].pt.table )
> > > +        goto done;
> > 
> > This could probably be a for loop rather than open-coding three
> > almost-identical printks.
> 
> The *_table_offsets macro names are different at each stage, I suppose I
> could open code the equivalent shifts, but I'd rather keep using the
> access macros.

Ah, OK.  In that case, open-coded is fine. 

Tim.

Re: [PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Tim Deegan]

At 13:26 +0100 on 07 Jun (1339075605), Ian Campbell wrote:
> On Thu, 2012-06-07 at 09:49 +0100, Tim Deegan wrote:
> > At 15:39 +0000 on 01 Jun (1338565172), Ian Campbell wrote:
> > > Useful for debug but not actually used in this patch.
> > > 
> > > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > > ---
> > >  xen/arch/arm/p2m.c         |   34 ++++++++++++++++++++++++++++++++++
> > >  xen/include/asm-arm/page.h |    1 +
> > >  2 files changed, 35 insertions(+), 0 deletions(-)
> > > 
> > > diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> > > index 4f624d8..fdbecbc 100644
> > > --- a/xen/arch/arm/p2m.c
> > > +++ b/xen/arch/arm/p2m.c
> > > @@ -5,6 +5,40 @@
> > >  #include <xen/domain_page.h>
> > >  #include <asm/flushtlb.h>
> > >  
> > > +void dump_p2m_lookup(struct domain *d, paddr_t addr)
> > > +{
> > > +    struct p2m_domain *p2m = &d->arch.p2m;
> > > +    lpae_t *first = NULL, *second = NULL, *third = NULL;
> > > +
> > > +    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
> > > +
> > > +    first = __map_domain_page(p2m->first_level);
> > > +    printk("1ST[%#03llx] = %#016llx\n",
> > > +           first_table_offset(addr),
> > > +           first[first_table_offset(addr)].bits);
> > > +    if ( !first[first_table_offset(addr)].p2m.valid ||
> > > +         !first[first_table_offset(addr)].p2m.table )
> > > +        goto done;
> > > +
> > > +    second = map_domain_page(first[first_table_offset(addr)].p2m.base);
> > > +    printk("2ND[%#03llx] = %#016llx\n",
> > > +           second_table_offset(addr),
> > > +           second[second_table_offset(addr)].bits);
> > > +    if ( !second[second_table_offset(addr)].p2m.valid ||
> > > +         !second[second_table_offset(addr)].p2m.table )
> > > +        goto done;
> > > +
> > > +    third = map_domain_page(second[second_table_offset(addr)].p2m.base);
> > > +    printk("3RD[%#03llx] = %#016llx\n",
> > > +           third_table_offset(addr),
> > > +           third[third_table_offset(addr)].bits);
> > > +
> > > +done:
> > > +    if (third) unmap_domain_page(third);
> > > +    if (second) unmap_domain_page(second);
> > > +    if (first) unmap_domain_page(first);
> > > +}
> > 
> > Can this be unified with dump_pt_walk?
> 
> Not all that easily, mainly because dump_pt_walk walks xenheap pages
> (which don't need mapping) while dump_p2m_walk dumps domheap pages
> (which need mapping as we go). I probably could write something generic
> enough but I fear that it would be a mass of if's.

Is there any harm in mapping xenheap pages?  Since this is only invoked
on error paths, we don't particularly need it to be fast. 

Tim.

Re: [PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Ian Campbell]

On Thu, 2012-06-07 at 13:40 +0100, Tim Deegan wrote:
> At 13:26 +0100 on 07 Jun (1339075605), Ian Campbell wrote:
> > On Thu, 2012-06-07 at 09:49 +0100, Tim Deegan wrote:
> > > At 15:39 +0000 on 01 Jun (1338565172), Ian Campbell wrote:
> > > > Useful for debug but not actually used in this patch.
> > > > 
> > > > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > > > ---
> > > >  xen/arch/arm/p2m.c         |   34 ++++++++++++++++++++++++++++++++++
> > > >  xen/include/asm-arm/page.h |    1 +
> > > >  2 files changed, 35 insertions(+), 0 deletions(-)
> > > > 
> > > > diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> > > > index 4f624d8..fdbecbc 100644
> > > > --- a/xen/arch/arm/p2m.c
> > > > +++ b/xen/arch/arm/p2m.c
> > > > @@ -5,6 +5,40 @@
> > > >  #include <xen/domain_page.h>
> > > >  #include <asm/flushtlb.h>
> > > >  
> > > > +void dump_p2m_lookup(struct domain *d, paddr_t addr)
> > > > +{
> > > > +    struct p2m_domain *p2m = &d->arch.p2m;
> > > > +    lpae_t *first = NULL, *second = NULL, *third = NULL;
> > > > +
> > > > +    printk("dom%d IPA %#016llx\n", d->domain_id, addr);
> > > > +
> > > > +    first = __map_domain_page(p2m->first_level);
> > > > +    printk("1ST[%#03llx] = %#016llx\n",
> > > > +           first_table_offset(addr),
> > > > +           first[first_table_offset(addr)].bits);
> > > > +    if ( !first[first_table_offset(addr)].p2m.valid ||
> > > > +         !first[first_table_offset(addr)].p2m.table )
> > > > +        goto done;
> > > > +
> > > > +    second = map_domain_page(first[first_table_offset(addr)].p2m.base);
> > > > +    printk("2ND[%#03llx] = %#016llx\n",
> > > > +           second_table_offset(addr),
> > > > +           second[second_table_offset(addr)].bits);
> > > > +    if ( !second[second_table_offset(addr)].p2m.valid ||
> > > > +         !second[second_table_offset(addr)].p2m.table )
> > > > +        goto done;
> > > > +
> > > > +    third = map_domain_page(second[second_table_offset(addr)].p2m.base);
> > > > +    printk("3RD[%#03llx] = %#016llx\n",
> > > > +           third_table_offset(addr),
> > > > +           third[third_table_offset(addr)].bits);
> > > > +
> > > > +done:
> > > > +    if (third) unmap_domain_page(third);
> > > > +    if (second) unmap_domain_page(second);
> > > > +    if (first) unmap_domain_page(first);
> > > > +}
> > > 
> > > Can this be unified with dump_pt_walk?
> > 
> > Not all that easily, mainly because dump_pt_walk walks xenheap pages
> > (which don't need mapping) while dump_p2m_walk dumps domheap pages
> > (which need mapping as we go). I probably could write something generic
> > enough but I fear that it would be a mass of if's.
> 
> Is there any harm in mapping xenheap pages?  Since this is only invoked
> on error paths, we don't particularly need it to be fast. 

For some reason I thought it just didn't work -- I'll give it a go
though.

Ian.

Re: [PATCH 29/38] arm: delay enabling data-cache until paging enabled.

[Tim Deegan]

At 20:04 +0100 on 01 Jun (1338581061), Ian Campbell wrote:
> On Fri, 2012-06-01 at 18:05 +0100, Tim Deegan wrote:
> > At 15:39 +0000 on 01 Jun (1338565198), Ian Campbell wrote:
> > > With enough warnings enabled the model seemed to be complaining that pages
> > > cached before paging was enabled had been mapped with to inconsistent sets of
> > > attributes. I'm not convinced that isn't a model issue, nor am I convinced
> > > this has really fixed anything, but it seems sensible enough.
> > 
> > This might be what breaks secondary CPU bringup: pagetables built by CPU
> > 0 may not have been flushed all the way to RAM when CPU 1 comes up, and
> > CPU 1 isn't participating in cache coherence protocols when it
> > starts to need them.
> 
> The issue here is the lack of the necessary flush, rather than this
> change particularly, right?

It turns out to be much more prosaic - the patch to delete the identity
mapping from the boot pagetables was scuppering non-boot CPUs.

So this is OK, but can I suggest this to tidy it up?

diff --git a/xen/arch/arm/head.S b/xen/arch/arm/head.S
index 71197af..0d8ce0f 100644
--- a/xen/arch/arm/head.S
+++ b/xen/arch/arm/head.S
@@ -211,17 +211,13 @@ pt_ready:
 
 	ldr   r1, =paging            /* Explicit vaddr, not RIP-relative */
 	mrc   CP32(r0, HSCTLR)
-	orr   r0, r0, #0x1           /* Add in the MMU enable bit */
+	orr   r0, r0, #(SCTLR_M|SCTLR_C) /* Enable MMU and D-cache */
 	dsb                          /* Flush PTE writes and finish reads */
 	mcr   CP32(r0, HSCTLR)       /* now paging is enabled */
 	isb                          /* Now, flush the icache */
 	mov   pc, r1                 /* Get a proper vaddr into PC */
 paging:
 
-	mrc   CP32(r0, HSCTLR)       /* Now enable data cache */
-	orr   r0, r0, #(SCTLR_C)
-	mcr   CP32(r0, HSCTLR)
-
 #ifdef EARLY_UART_ADDRESS
 	/* Recover the UART address in the new address space. */
 	lsl   r11, #11

Re: [PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Ian Campbell]

> > > > > diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> > > > > index 4f624d8..fdbecbc 100644
> > > > > --- a/xen/arch/arm/p2m.c
> > > > > +++ b/xen/arch/arm/p2m.c
> > > > > @@ -5,6 +5,40 @@
> > > > >  #include <xen/domain_page.h>
> > > > >  #include <asm/flushtlb.h>
> > > > >  
> > > > > +void dump_p2m_lookup(struct domain *d, paddr_t addr)
> > > > > +{
[...]
> > > > > +}
> > > > 
> > > > Can this be unified with dump_pt_walk?
> > > 
> > > Not all that easily, mainly because dump_pt_walk walks xenheap pages
> > > (which don't need mapping) while dump_p2m_walk dumps domheap pages
> > > (which need mapping as we go). I probably could write something generic
> > > enough but I fear that it would be a mass of if's.
> > 
> > Is there any harm in mapping xenheap pages?  Since this is only invoked
> > on error paths, we don't particularly need it to be fast. 
> 
> For some reason I thought it just didn't work -- I'll give it a go
> though.

Well it works, this means that this patch and the previous patch then
collapse down into a single patch:

8<-------------------------------------------------------

>From 055fe5f4a3a77f292d5a2a6b9f56a4d14dad3519 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Fri, 2 Mar 2012 12:02:42 +0000
Subject: [PATCH] arm: handy function to print a walk of a page table

Include helpers for dumping hypervisor walks and guest p2m walks.

Useful for debug but not actually used in this patch.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/mm.c          |   49 +++++++++++++++++++++++++++++++++++++++++++-
 xen/arch/arm/p2m.c         |   15 +++++++++++++
 xen/include/asm-arm/page.h |   26 +++++++++++++++++++++++
 3 files changed, 89 insertions(+), 1 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 10ff883..715a98a 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -26,6 +26,7 @@
 #include <xen/preempt.h>
 #include <xen/errno.h>
 #include <xen/guest_access.h>
+#include <xen/domain_page.h>
 #include <asm/page.h>
 #include <asm/current.h>
 #include <public/memory.h>
@@ -42,6 +43,8 @@ static lpae_t xen_xenmap[LPAE_ENTRIES] __attribute__((__aligned__(4096)));
 /* Non-boot CPUs use this to find the correct pagetables. */
 uint64_t boot_httbr;
 
+static paddr_t phys_offset;
+
 /* Limits of the Xen heap */
 unsigned long xenheap_mfn_start, xenheap_mfn_end;
 unsigned long xenheap_virt_end;
@@ -53,6 +56,50 @@ unsigned long max_page;
 
 extern char __init_begin[], __init_end[];
 
+void dump_pt_walk(lpae_t *first, paddr_t addr)
+{
+    lpae_t *second = NULL, *third = NULL;
+
+    if ( first_table_offset(addr) >= LPAE_ENTRIES )
+        return;
+
+    printk("1ST[0x%llx] = 0x%"PRIpaddr"\n",
+           first_table_offset(addr),
+           first[first_table_offset(addr)].bits);
+    if ( !first[first_table_offset(addr)].walk.valid ||
+         !first[first_table_offset(addr)].walk.table )
+        goto done;
+
+    second = map_domain_page(first[first_table_offset(addr)].walk.base);
+    printk("2ND[0x%llx] = 0x%"PRIpaddr"\n",
+           second_table_offset(addr),
+           second[second_table_offset(addr)].bits);
+    if ( !second[second_table_offset(addr)].walk.valid ||
+         !second[second_table_offset(addr)].walk.table )
+        goto done;
+
+    third = map_domain_page(second[second_table_offset(addr)].walk.base);
+    printk("3RD[0x%llx] = 0x%"PRIpaddr"\n",
+           third_table_offset(addr),
+           third[third_table_offset(addr)].bits);
+
+done:
+    if (third) unmap_domain_page(third);
+    if (second) unmap_domain_page(second);
+
+}
+
+void dump_hyp_walk(uint32_t addr)
+{
+    uint64_t httbr = READ_CP64(HTTBR);
+
+    printk("Walking Hypervisor VA 0x%08"PRIx32" via HTTBR 0x%016"PRIx64"\n",
+           addr, httbr);
+
+    BUG_ON( (lpae_t *)(unsigned long)(httbr - phys_offset) != xen_pgtable );
+    dump_pt_walk(xen_pgtable, addr);
+}
+
 /* Map a 4k page in a fixmap entry */
 void set_fixmap(unsigned map, unsigned long mfn, unsigned attributes)
 {
@@ -159,7 +206,7 @@ void unmap_domain_page(const void *va)
  * Changes here may need matching changes in head.S */
 void __init setup_pagetables(unsigned long boot_phys_offset)
 {
-    paddr_t xen_paddr, phys_offset;
+    paddr_t xen_paddr;
     unsigned long dest_va;
     lpae_t pte, *p;
     int i;
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 4f624d8..ea385a6 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -5,6 +5,21 @@
 #include <xen/domain_page.h>
 #include <asm/flushtlb.h>
 
+void dump_p2m_lookup(struct domain *d, paddr_t addr)
+{
+    struct p2m_domain *p2m = &d->arch.p2m;
+    lpae_t *first;
+
+    printk("dom%d IPA 0x%"PRIpaddr"\n", d->domain_id, addr);
+
+    printk("P2M @ %p mfn:0x%lx\n",
+           p2m->first_level, page_to_mfn(p2m->first_level));
+
+    first = __map_domain_page(p2m->first_level);
+    dump_pt_walk(first, addr);
+    unmap_domain_page(first);
+}
+
 void p2m_load_VTTBR(struct domain *d)
 {
     struct p2m_domain *p2m = &d->arch.p2m;
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index b6df64e..183ba5f 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -132,10 +132,28 @@ typedef struct {
     unsigned long sbz1:5;
 } __attribute__((__packed__)) lpae_p2m_t;
 
+/*
+ * Walk is the common bits of p2m and pt entries which are needed to
+ * simply walk the table (e.g. for debug).
+ */
+typedef struct {
+    /* These are used in all kinds of entry. */
+    unsigned long valid:1;      /* Valid mapping */
+    unsigned long table:1;      /* == 1 in 4k map entries too */
+
+    unsigned long pad2:10;
+
+    /* The base address must be appropriately aligned for Block entries */
+    unsigned long base:28;      /* Base address of block or next table */
+
+    unsigned long pad1:24;
+} __attribute__((__packed__)) lpae_walk_t;
+
 typedef union {
     uint64_t bits;
     lpae_pt_t pt;
     lpae_p2m_t p2m;
+    lpae_walk_t walk;
 } lpae_t;
 
 /* Standard entry type that we'll use to build Xen's own pagetables.
@@ -252,6 +270,14 @@ static inline void flush_guest_tlb(void)
     WRITE_CP32(r0 /* dummy */, TLBIALLNSNH);
 }
 
+/* Print a walk of an arbitrary page table */
+void dump_pt_walk(lpae_t *table, paddr_t addr);
+
+/* Print a walk of the hypervisor's page tables for a virtual addr. */
+extern void dump_hyp_walk(uint32_t addr);
+/* Print a walk of the p2m for a domain for a physical address. */
+extern void dump_p2m_lookup(struct domain *d, paddr_t addr);
+
 /* Ask the MMU to translate a VA for us */
 static inline uint64_t __va_to_par(uint32_t va)
 {
-- 
1.7.9.1

Re: [PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Ian Campbell]

On Thu, 2012-06-07 at 17:34 +0100, Ian Campbell wrote:

> +    /* The base address must be appropriately aligned for Block entries */

I fixed upo a typo in this in my mail client, but when I went to fix it
in my tree I noticed I'd just cut-n-pasted it from some other places, so
I actually just added to my series:

8<----------------------

>From 7a04d5c045df2afee7a9302c26b007e6f1901e01 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 7 Jun 2012 16:35:12 +0000
Subject: [PATCH] arm: fix typo s/approprately/appropriately/g

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/include/asm-arm/page.h |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index 20de411..3c59923 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -87,7 +87,7 @@ typedef struct {
     unsigned long af:1;         /* Access Flag */
     unsigned long ng:1;         /* Not-Global */
 
-    /* The base address must be approprately aligned for Block entries */
+    /* The base address must be appropriately aligned for Block entries */
     unsigned long base:28;      /* Base address of block or next table */
     unsigned long sbz:12;       /* Must be zero */
 
@@ -122,7 +122,7 @@ typedef struct {
     unsigned long af:1;         /* Access Flag */
     unsigned long sbz4:1;
 
-    /* The base address must be approprately aligned for Block entries */
+    /* The base address must be appropriately aligned for Block entries */
     unsigned long base:28;      /* Base address of block or next table */
     unsigned long sbz3:12;
 
@@ -147,7 +147,7 @@ typedef struct {
 
     unsigned long pad2:10;
 
-    /* The base address must be approprately aligned for Block entries */
+    /* The base address must be appropriately aligned for Block entries */
     unsigned long base:28;      /* Base address of block or next table */
 
     unsigned long pad1:24;
-- 
1.7.9.1

Re: [PATCH 14/38] arm: do not set max_vcpus = 8 in arch_domain_create.

[Ian Campbell]

On Wed, 2012-06-06 at 16:26 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > XEN_DOMCTL_max_vcpus cannot reduce max_vcpus and therefore we can't create a
> > smaller guest.
> > 
> > The limit of 8 (due to GIC limits) should be expressed elsewhere, likely in
> > MAX_VIRT_CPUS -- but making that change caused:
> 
> Are you sure? I made that change and I didn't see the error.
> I think this patch should set MAX_VIRT_CPUS to 8 as well as removing
> max_vcpus = 8.

This was the same heap corruption again as seen in "[PATCH 16/38] arm:
Add simple cpu_{sibling,core}_mask"  and having fixed that I don't see
the crash with MAX_VIRT_CPUS == 8 any more...

The patch becomes:

>From b68c4abe1dec44f3ed87a0d7ae98f4269043cce3 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 7 Jun 2012 16:52:46 +0000
Subject: [PATCH] arm: do not set max_vcpus = 8 in arch_domain_create.

XEN_DOMCTL_max_vcpus cannot reduce max_vcpus and therefore we can't create a
smaller guest.

The limit of 8 (due to GIC limits) should be expressed in MAX_VIRT_CPUS.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c        |    2 --
 xen/include/asm-arm/config.h |    2 +-
 2 files changed, 1 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 1336dc4..040a2ce 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -338,8 +338,6 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
             goto fail;
     }
 
-    d->max_vcpus = 8;
-
     if ( (rc = domain_vgic_init(d)) != 0 )
         goto fail;
 
diff --git a/xen/include/asm-arm/config.h b/xen/include/asm-arm/config.h
index 91e87e1..7d02cc7 100644
--- a/xen/include/asm-arm/config.h
+++ b/xen/include/asm-arm/config.h
@@ -27,7 +27,7 @@
 #define NR_CPUS 128
 #endif
 
-#define MAX_VIRT_CPUS 128 /* XXX */
+#define MAX_VIRT_CPUS 8
 #define MAX_HVM_VCPUS MAX_VIRT_CPUS
 
 #define asmlinkage /* Nothing needed */
-- 
1.7.9.1

Re: [PATCH 14/38] arm: do not set max_vcpus = 8 in arch_domain_create.

[Stefano Stabellini]

On Thu, 7 Jun 2012, Ian Campbell wrote:
> On Wed, 2012-06-06 at 16:26 +0100, Stefano Stabellini wrote:
> > On Fri, 1 Jun 2012, Ian Campbell wrote:
> > > XEN_DOMCTL_max_vcpus cannot reduce max_vcpus and therefore we can't create a
> > > smaller guest.
> > > 
> > > The limit of 8 (due to GIC limits) should be expressed elsewhere, likely in
> > > MAX_VIRT_CPUS -- but making that change caused:
> > 
> > Are you sure? I made that change and I didn't see the error.
> > I think this patch should set MAX_VIRT_CPUS to 8 as well as removing
> > max_vcpus = 8.
> 
> This was the same heap corruption again as seen in "[PATCH 16/38] arm:
> Add simple cpu_{sibling,core}_mask"  and having fixed that I don't see
> the crash with MAX_VIRT_CPUS == 8 any more...
> 
> The patch becomes:
> 
> From b68c4abe1dec44f3ed87a0d7ae98f4269043cce3 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Thu, 7 Jun 2012 16:52:46 +0000
> Subject: [PATCH] arm: do not set max_vcpus = 8 in arch_domain_create.
> 
> XEN_DOMCTL_max_vcpus cannot reduce max_vcpus and therefore we can't create a
> smaller guest.
> 
> The limit of 8 (due to GIC limits) should be expressed in MAX_VIRT_CPUS.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

>  xen/arch/arm/domain.c        |    2 --
>  xen/include/asm-arm/config.h |    2 +-
>  2 files changed, 1 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 1336dc4..040a2ce 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -338,8 +338,6 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
>              goto fail;
>      }
>  
> -    d->max_vcpus = 8;
> -
>      if ( (rc = domain_vgic_init(d)) != 0 )
>          goto fail;
>  
> diff --git a/xen/include/asm-arm/config.h b/xen/include/asm-arm/config.h
> index 91e87e1..7d02cc7 100644
> --- a/xen/include/asm-arm/config.h
> +++ b/xen/include/asm-arm/config.h
> @@ -27,7 +27,7 @@
>  #define NR_CPUS 128
>  #endif
>  
> -#define MAX_VIRT_CPUS 128 /* XXX */
> +#define MAX_VIRT_CPUS 8
>  #define MAX_HVM_VCPUS MAX_VIRT_CPUS
>  
>  #define asmlinkage /* Nothing needed */
> -- 
> 1.7.9.1
> 
> 
> 
> 

Re: [PATCH 08/38] arm: allocate and setup a guest vcpu.

[Ian Campbell]

On Thu, 2012-06-07 at 10:40 +0100, Ian Campbell wrote:
> More head scratching required I think!

This turned out to be the problem with not initialising cpu_sibling_mask
properly, see the thread against "[PATCH 16/38] arm: Add simple
cpu_{sibling,core}_mask".

Having fixed that I updated based on your comments to:

8<-----------------------------------------------------------

>From 75cff29f4645dd19d07175109b5891fd44de9d60 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Fri, 13 Apr 2012 16:07:21 +0100
Subject: [PATCH] arm: allocate and setup a guest vcpu.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c         |   67 +++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/dummy.S          |    3 --
 xen/include/public/arch-arm.h |    9 -----
 3 files changed, 67 insertions(+), 12 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 9339a11..b099d91 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -144,6 +144,17 @@ void free_vcpu_struct(struct vcpu *v)
     free_xenheap_page(v);
 }
 
+struct vcpu_guest_context *alloc_vcpu_guest_context(void)
+{
+    return xmalloc(struct vcpu_guest_context);
+
+}
+
+void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
+{
+    xfree(vgc);
+}
+
 int vcpu_initialise(struct vcpu *v)
 {
     int rc = 0;
@@ -212,6 +223,62 @@ void arch_domain_destroy(struct domain *d)
     /* domain_vgic_destroy */
 }
 
+static int is_guest_psr(uint32_t psr)
+{
+    switch (psr & PSR_MODE_MASK)
+    {
+    case PSR_MODE_USR:
+    case PSR_MODE_FIQ:
+    case PSR_MODE_IRQ:
+    case PSR_MODE_SVC:
+    case PSR_MODE_ABT:
+    case PSR_MODE_UND:
+    case PSR_MODE_SYS:
+        return 1;
+    case PSR_MODE_MON:
+    case PSR_MODE_HYP:
+    default:
+        return 0;
+    }
+}
+
+/*
+ * Initialise VCPU state. The context can be supplied by either the
+ * toolstack (XEN_DOMCTL_setvcpucontext) or the guest
+ * (VCPUOP_initialise) and therefore must be properly validated.
+ */
+int arch_set_info_guest(
+    struct vcpu *v, vcpu_guest_context_u c)
+{
+    struct cpu_user_regs *regs = &c.nat->user_regs;
+
+    if ( !is_guest_psr(regs->cpsr) )
+        return -EINVAL;
+
+    if ( regs->spsr_svc && !is_guest_psr(regs->spsr_svc) )
+        return -EINVAL;
+    if ( regs->spsr_abt && !is_guest_psr(regs->spsr_abt) )
+        return -EINVAL;
+    if ( regs->spsr_und && !is_guest_psr(regs->spsr_und) )
+        return -EINVAL;
+    if ( regs->spsr_irq && !is_guest_psr(regs->spsr_irq) )
+        return -EINVAL;
+    if ( regs->spsr_fiq && !is_guest_psr(regs->spsr_fiq) )
+        return -EINVAL;
+
+    v->arch.cpu_info->guest_cpu_user_regs = *regs;
+
+    /* XXX other state:
+     * - SCTLR
+     * - TTBR0/1
+     * - TTBCR
+     */
+
+    clear_bit(_VPF_down, &v->pause_flags);
+
+    return 0;
+}
+
 void arch_dump_domain_info(struct domain *d)
 {
 }
diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
index 016340c..3b48917 100644
--- a/xen/arch/arm/dummy.S
+++ b/xen/arch/arm/dummy.S
@@ -20,11 +20,8 @@ DUMMY(pirq_guest_unbind);
 DUMMY(pirq_set_affinity);
 
 /* VCPU */
-DUMMY(alloc_vcpu_guest_context);
 DUMMY(arch_get_info_guest);
-DUMMY(arch_set_info_guest);
 DUMMY(arch_vcpu_reset);
-DUMMY(free_vcpu_guest_context);
 DUMMY(sync_vcpu_execstate);
 NOP(update_vcpu_system_time);
 DUMMY(vcpu_show_execution_state);
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 1b1bcf3..e439727 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -124,15 +124,6 @@ typedef uint32_t xen_ulong_t;
 
 struct vcpu_guest_context {
     struct cpu_user_regs user_regs;         /* User-level CPU registers     */
-    union {
-        uint32_t reg[16];
-        struct {
-            uint32_t __pad[12];
-            uint32_t sp; /* r13 */
-            uint32_t lr; /* r14 */
-            uint32_t pc; /* r15 */
-        };
-    };
 };
 typedef struct vcpu_guest_context vcpu_guest_context_t;
 DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
-- 
1.7.9.1

Re: [PATCH 03/38] arm: handy function to print a walk of a domain's p2m.

[Tim Deegan]

At 17:34 +0100 on 07 Jun (1339090467), Ian Campbell wrote:
> Well it works, this means that this patch and the previous patch then
> collapse down into a single patch:
> 
> 8<-------------------------------------------------------
> 
> From 055fe5f4a3a77f292d5a2a6b9f56a4d14dad3519 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Fri, 2 Mar 2012 12:02:42 +0000
> Subject: [PATCH] arm: handy function to print a walk of a page table
> 
> Include helpers for dumping hypervisor walks and guest p2m walks.
> 
> Useful for debug but not actually used in this patch.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Great!  For this and the follow-up typo fix:
Acked-by: Tim Deegan <tim@xen.org>

Re: [PATCH 08/38] arm: allocate and setup a guest vcpu.

[Stefano Stabellini]

On Thu, 7 Jun 2012, Ian Campbell wrote:
> On Thu, 2012-06-07 at 10:40 +0100, Ian Campbell wrote:
> > More head scratching required I think!
> 
> This turned out to be the problem with not initialising cpu_sibling_mask
> properly, see the thread against "[PATCH 16/38] arm: Add simple
> cpu_{sibling,core}_mask".
> 
> Having fixed that I updated based on your comments to:
> 
> 8<-----------------------------------------------------------
> 
> From 75cff29f4645dd19d07175109b5891fd44de9d60 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Fri, 13 Apr 2012 16:07:21 +0100
> Subject: [PATCH] arm: allocate and setup a guest vcpu.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

It looks OK now.


Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>


>  xen/arch/arm/domain.c         |   67 +++++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/dummy.S          |    3 --
>  xen/include/public/arch-arm.h |    9 -----
>  3 files changed, 67 insertions(+), 12 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 9339a11..b099d91 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -144,6 +144,17 @@ void free_vcpu_struct(struct vcpu *v)
>      free_xenheap_page(v);
>  }
>  
> +struct vcpu_guest_context *alloc_vcpu_guest_context(void)
> +{
> +    return xmalloc(struct vcpu_guest_context);
> +
> +}
> +
> +void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
> +{
> +    xfree(vgc);
> +}
> +
>  int vcpu_initialise(struct vcpu *v)
>  {
>      int rc = 0;
> @@ -212,6 +223,62 @@ void arch_domain_destroy(struct domain *d)
>      /* domain_vgic_destroy */
>  }
>  
> +static int is_guest_psr(uint32_t psr)
> +{
> +    switch (psr & PSR_MODE_MASK)
> +    {
> +    case PSR_MODE_USR:
> +    case PSR_MODE_FIQ:
> +    case PSR_MODE_IRQ:
> +    case PSR_MODE_SVC:
> +    case PSR_MODE_ABT:
> +    case PSR_MODE_UND:
> +    case PSR_MODE_SYS:
> +        return 1;
> +    case PSR_MODE_MON:
> +    case PSR_MODE_HYP:
> +    default:
> +        return 0;
> +    }
> +}
> +
> +/*
> + * Initialise VCPU state. The context can be supplied by either the
> + * toolstack (XEN_DOMCTL_setvcpucontext) or the guest
> + * (VCPUOP_initialise) and therefore must be properly validated.
> + */
> +int arch_set_info_guest(
> +    struct vcpu *v, vcpu_guest_context_u c)
> +{
> +    struct cpu_user_regs *regs = &c.nat->user_regs;
> +
> +    if ( !is_guest_psr(regs->cpsr) )
> +        return -EINVAL;
> +
> +    if ( regs->spsr_svc && !is_guest_psr(regs->spsr_svc) )
> +        return -EINVAL;
> +    if ( regs->spsr_abt && !is_guest_psr(regs->spsr_abt) )
> +        return -EINVAL;
> +    if ( regs->spsr_und && !is_guest_psr(regs->spsr_und) )
> +        return -EINVAL;
> +    if ( regs->spsr_irq && !is_guest_psr(regs->spsr_irq) )
> +        return -EINVAL;
> +    if ( regs->spsr_fiq && !is_guest_psr(regs->spsr_fiq) )
> +        return -EINVAL;
> +
> +    v->arch.cpu_info->guest_cpu_user_regs = *regs;
> +
> +    /* XXX other state:
> +     * - SCTLR
> +     * - TTBR0/1
> +     * - TTBCR
> +     */
> +
> +    clear_bit(_VPF_down, &v->pause_flags);
> +
> +    return 0;
> +}
> +
>  void arch_dump_domain_info(struct domain *d)
>  {
>  }
> diff --git a/xen/arch/arm/dummy.S b/xen/arch/arm/dummy.S
> index 016340c..3b48917 100644
> --- a/xen/arch/arm/dummy.S
> +++ b/xen/arch/arm/dummy.S
> @@ -20,11 +20,8 @@ DUMMY(pirq_guest_unbind);
>  DUMMY(pirq_set_affinity);
>  
>  /* VCPU */
> -DUMMY(alloc_vcpu_guest_context);
>  DUMMY(arch_get_info_guest);
> -DUMMY(arch_set_info_guest);
>  DUMMY(arch_vcpu_reset);
> -DUMMY(free_vcpu_guest_context);
>  DUMMY(sync_vcpu_execstate);
>  NOP(update_vcpu_system_time);
>  DUMMY(vcpu_show_execution_state);
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 1b1bcf3..e439727 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -124,15 +124,6 @@ typedef uint32_t xen_ulong_t;
>  
>  struct vcpu_guest_context {
>      struct cpu_user_regs user_regs;         /* User-level CPU registers     */
> -    union {
> -        uint32_t reg[16];
> -        struct {
> -            uint32_t __pad[12];
> -            uint32_t sp; /* r13 */
> -            uint32_t lr; /* r14 */
> -            uint32_t pc; /* r15 */
> -        };
> -    };
>  };
>  typedef struct vcpu_guest_context vcpu_guest_context_t;
>  DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> -- 
> 1.7.9.1
> 
> 
> 
> 

Re: [PATCH 11/38] arm: implement p2m lookup

[Ian Campbell]

On Wed, 2012-06-06 at 15:30 +0100, Ian Campbell wrote:
> > there is no need to introduce p2m_lookup in the same patch you do these
> > unrelated printk adjustments, correct?
> > 
> 
> Right, I think I've just put them in the wrong patch by mistake, I'll
> figure out what I meant to do ...

These ended up in my combined replacement for patches #2 and #3 which
was posted in <1339086867.18523.36.camel@zakaz.uk.xensource.com>.

Ian.

Re: [PATCH 11/38] arm: implement p2m lookup

[Ian Campbell]

On Thu, 2012-06-07 at 10:03 +0100, Tim Deegan wrote:
> At 15:39 +0000 on 01 Jun (1338565180), Ian Campbell wrote:
> > +/*
> > + * Lookup the MFN corresponding to a domain's PFN.
> > + *
> > + * There are no processor functions to do a stage 2 only lookup therefore we
> > + * do a a software walk.
> > + */
> > +paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
> > +{
> > +    struct p2m_domain *p2m = &d->arch.p2m;
> > +    lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
> > +    paddr_t maddr = INVALID_PADDR;
> > +
> > +    spin_lock(&p2m->lock);
> > +
> > +    first = __map_domain_page(p2m->first_level);
> > +    if ( !first[first_table_offset(paddr)].p2m.valid )
> > +        goto done_err;
> > +    if ( !first[first_table_offset(paddr)].p2m.table )
> > +    {
> > +        pte = first[first_table_offset(paddr)];
> > +        goto done;
> > +    }
> 
> This would be neater as: 
>        pte = first[first_table_offset(paddr)];
>        if ( !pte.p2m.valid || !pte.p2m.table )
>            goto done;
> 
> and test for pte.valid at 'done'.

Yes, that looks nice, although you still need a bit of a quirk for the
third level table bit. Patch below.

> It would be nice to do the three levels in a loop as well, but the weird
> way the table bit behaves in third-level entries might make that more
> confusing than the straight-line version.

This also has the same issue as with *_table_offset as the other similar
functions discussed earlier.

8<-----------------------------------------------------------

>From 347855d863303720cbf5ceb0f1e067660108d3f1 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Fri, 13 Apr 2012 16:24:58 +0100
Subject: [PATCH] arm: implement p2m lookup

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c        |   45 +++++++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/p2m.h |    3 +++
 2 files changed, 48 insertions(+), 0 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 6df5b62..145d9fe 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -32,6 +32,51 @@ void p2m_load_VTTBR(struct domain *d)
     isb(); /* Ensure update is visible */
 }
 
+/*
+ * Lookup the MFN corresponding to a domain's PFN.
+ *
+ * There are no processor functions to do a stage 2 only lookup therefore we
+ * do a a software walk.
+ */
+paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
+{
+    struct p2m_domain *p2m = &d->arch.p2m;
+    lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
+    paddr_t maddr = INVALID_PADDR;
+
+    spin_lock(&p2m->lock);
+
+    first = __map_domain_page(p2m->first_level);
+
+    pte = first[first_table_offset(paddr)];
+    if ( !pte.p2m.valid || !pte.p2m.table )
+        goto done;
+
+    second = map_domain_page(first[first_table_offset(paddr)].p2m.base);
+    pte = second[second_table_offset(paddr)];
+    if ( !pte.p2m.valid || !pte.p2m.table )
+        goto done;
+
+    third = map_domain_page(second[second_table_offset(paddr)].p2m.base);
+    pte = third[third_table_offset(paddr)];
+
+    /* This bit must be one in the level 3 entry */
+    if ( !pte.p2m.table )
+        pte.bits = 0;
+
+done:
+    if ( pte.p2m.valid )
+        maddr = (pte.bits & PADDR_MASK & PAGE_MASK) | (paddr & ~PAGE_MASK);
+
+    if (third) unmap_domain_page(third);
+    if (second) unmap_domain_page(second);
+    if (first) unmap_domain_page(first);
+
+    spin_unlock(&p2m->lock);
+
+    return maddr;
+}
+
 int guest_physmap_mark_populate_on_demand(struct domain *d,
                                           unsigned long gfn,
                                           unsigned int order)
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 349923a..1afd5cb 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -32,6 +32,9 @@ int p2m_alloc_table(struct domain *d);
 /* */
 void p2m_load_VTTBR(struct domain *d);
 
+/* */
+paddr_t p2m_lookup(struct domain *d, paddr_t gpfn);
+
 /* Setup p2m RAM mapping for domain d from start-end. */
 int p2m_populate_ram(struct domain *d, paddr_t start, paddr_t end);
 /* Map MMIO regions in the p2m: start_gaddr and end_gaddr is the range
-- 
1.7.9.1

Re: [PATCH 16/38] arm: Add simple cpu_{sibling, core}_mask

[Ian Campbell]

Keir, Jan: Any objection/ack for the generic part of this commit?

> 8<------------------------------------------------------
> 
> From e980ca1ec9bf92b2f1255ac5222b1da1292f9f72 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Mon, 14 May 2012 12:25:31 +0100
> Subject: [PATCH] arm: Add simple cpu_{sibling,core}_mask
> 
> This needs to be done for all cpus. The allocations require smp_prepare_cpus
> to be called a bit later on.
> 
> In a previous version of this patch these maps were being zeroed (instead of
> setting the CPU itself in them). This in turn causes cpumask_first to return
> NR_CPUS, which in turn was causing default_vcpu0_location to misbehave and
> read off the end of its cnt array. Add a couple of asserts to catch this in
> the future.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/arch/arm/dummy.S   |    2 --
>  xen/arch/arm/setup.c   |    4 ++--
>  xen/arch/arm/smpboot.c |   21 +++++++++++++++++++++
>  xen/common/domctl.c    |    2 ++
>  4 files changed, 25 insertions(+), 4 deletions(-)
[...]
> diff --git a/xen/common/domctl.c b/xen/common/domctl.c
> index 9f1a9ad..c1acd1d 100644
> --- a/xen/common/domctl.c
> +++ b/xen/common/domctl.c
> @@ -190,10 +190,12 @@ static unsigned int default_vcpu0_location(cpumask_t *online)
>       */
>      cpumask_copy(&cpu_exclude_map, per_cpu(cpu_sibling_mask, 0));
>      cpu = cpumask_first(&cpu_exclude_map);
> +    ASSERT(cpu < nr_cpus);
>      if ( cpumask_weight(&cpu_exclude_map) > 1 )
>          cpu = cpumask_next(cpu, &cpu_exclude_map);
>      for_each_cpu(i, online)
>      {
> +        ASSERT(i < nr_cpus);
>          if ( cpumask_test_cpu(i, &cpu_exclude_map) )
>              continue;
>          if ( (i == cpumask_first(per_cpu(cpu_sibling_mask, i))) &&

Re: [PATCH 17/38] arm: allow p2m to be created with specific MATTR.

[Ian Campbell]

On Wed, 2012-06-06 at 17:27 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/p2m.c         |   15 ++++++++-------
> >  xen/include/asm-arm/page.h |    6 ++++--
> >  2 files changed, 12 insertions(+), 9 deletions(-)
> > 
> > diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> > index 9b40e93..46c6f17 100644
> > --- a/xen/arch/arm/p2m.c
> > +++ b/xen/arch/arm/p2m.c
> > @@ -148,7 +148,7 @@ static int p2m_create_entry(struct domain *d,
> >      clear_page(p);
> >      unmap_domain_page(p);
> >  
> > -    pte = mfn_to_p2m_entry(page_to_mfn(page));
> > +    pte = mfn_to_p2m_entry(page_to_mfn(page), MATTR_MEM);
> >  
> >      write_pte(entry, pte);
> >  
> 
> This works because p2m_create_entry is always called to create first or
> second level entries only.
> Maybe we should rename p2m_create_entry to p2m_create_table_entry for
> clarity? Or add a comment?

I think p2m_create_table would be a fine name for this function. I've
also added a comment.

8<------------------------------

>From 33d7d69b95ed016542631e2daca55d5cd9969627 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Mon, 14 May 2012 12:30:04 +0100
Subject: [PATCH] arm: allow p2m to be created with specific MATTR.

Rename p2m_create_entry to p2m_create_table since it can now only be used to
insert non-leaf entries into the page table.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c         |   22 ++++++++++++----------
 xen/include/asm-arm/page.h |    6 ++++--
 2 files changed, 16 insertions(+), 12 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 145d9fe..b411fe7 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -91,7 +91,8 @@ int p2m_pod_decrease_reservation(struct domain *d,
     return -ENOSYS;
 }
 
-static int p2m_create_entry(struct domain *d,
+/* Allocate a new page table page and hook it in via the given entry */
+static int p2m_create_table(struct domain *d,
                             lpae_t *entry)
 {
     struct p2m_domain *p2m = &d->arch.p2m;
@@ -111,7 +112,7 @@ static int p2m_create_entry(struct domain *d,
     clear_page(p);
     unmap_domain_page(p);
 
-    pte = mfn_to_p2m_entry(page_to_mfn(page));
+    pte = mfn_to_p2m_entry(page_to_mfn(page), MATTR_MEM);
 
     write_pte(entry, pte);
 
@@ -122,7 +123,8 @@ static int create_p2m_entries(struct domain *d,
                      int alloc,
                      paddr_t start_gpaddr,
                      paddr_t end_gpaddr,
-                     paddr_t maddr)
+                     paddr_t maddr,
+                     int mattr)
 {
     int rc;
     struct p2m_domain *p2m = &d->arch.p2m;
@@ -140,7 +142,7 @@ static int create_p2m_entries(struct domain *d,
     {
         if ( !first[first_table_offset(addr)].p2m.valid )
         {
-            rc = p2m_create_entry(d, &first[first_table_offset(addr)]);
+            rc = p2m_create_table(d, &first[first_table_offset(addr)]);
             if ( rc < 0 ) {
                 printk("p2m_populate_ram: L1 failed\n");
                 goto out;
@@ -159,7 +161,7 @@ static int create_p2m_entries(struct domain *d,
 
         if ( !second[second_table_offset(addr)].p2m.valid )
         {
-            rc = p2m_create_entry(d, &second[second_table_offset(addr)]);
+            rc = p2m_create_table(d, &second[second_table_offset(addr)]);
             if ( rc < 0 ) {
                 printk("p2m_populate_ram: L2 failed\n");
                 goto out;
@@ -198,11 +200,11 @@ static int create_p2m_entries(struct domain *d,
                 goto out;
             }
 
-            pte = mfn_to_p2m_entry(page_to_mfn(page));
+            pte = mfn_to_p2m_entry(page_to_mfn(page), mattr);
 
             write_pte(&third[third_table_offset(addr)], pte);
         } else {
-            lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT);
+            lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT, mattr);
             write_pte(&third[third_table_offset(addr)], pte);
             maddr += PAGE_SIZE;
         }
@@ -226,7 +228,7 @@ int p2m_populate_ram(struct domain *d,
                      paddr_t start,
                      paddr_t end)
 {
-    return create_p2m_entries(d, 1, start, end, 0);
+    return create_p2m_entries(d, 1, start, end, 0, MATTR_MEM);
 }
 
 int map_mmio_regions(struct domain *d,
@@ -234,7 +236,7 @@ int map_mmio_regions(struct domain *d,
                      paddr_t end_gaddr,
                      paddr_t maddr)
 {
-    return create_p2m_entries(d, 0, start_gaddr, end_gaddr, maddr);
+    return create_p2m_entries(d, 0, start_gaddr, end_gaddr, maddr, MATTR_DEV);
 }
 
 int guest_physmap_add_page(struct domain *d,
@@ -244,7 +246,7 @@ int guest_physmap_add_page(struct domain *d,
 {
     return create_p2m_entries(d, 0, gpfn << PAGE_SHIFT,
                               (gpfn + (1<<page_order)) << PAGE_SHIFT,
-                              mfn << PAGE_SHIFT);
+                              mfn << PAGE_SHIFT, MATTR_MEM);
 }
 
 void guest_physmap_remove_page(struct domain *d,
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index 183ba5f..2783c30 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -46,6 +46,8 @@
 #define DEV_WC        BUFFERABLE
 #define DEV_CACHED    WRITEBACK
 
+#define MATTR_DEV     0x1
+#define MATTR_MEM     0xf
 
 #ifndef __ASSEMBLY__
 
@@ -187,7 +189,7 @@ static inline lpae_t mfn_to_xen_entry(unsigned long mfn)
     return e;
 }
 
-static inline lpae_t mfn_to_p2m_entry(unsigned long mfn)
+static inline lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
 {
     paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
     lpae_t e = (lpae_t) {
@@ -196,7 +198,7 @@ static inline lpae_t mfn_to_p2m_entry(unsigned long mfn)
         .p2m.sh = LPAE_SH_OUTER,
         .p2m.write = 1,
         .p2m.read = 1,
-        .p2m.mattr = 0xf,
+        .p2m.mattr = mattr,
         .p2m.table = 1,
         .p2m.valid = 1,
     };
-- 
1.7.9.1

Re: [PATCH 11/38] arm: implement p2m lookup

[Tim Deegan]

At 16:13 +0100 on 13 Jun (1339604016), Ian Campbell wrote:
> Yes, that looks nice, although you still need a bit of a quirk for the
> third level table bit. Patch below.

Much nicer.   One more round of nits below if you feel keen; in any case,
Acked-by: Tim Deegan <tim@xen.org>

> +paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
> +{
> +    struct p2m_domain *p2m = &d->arch.p2m;
> +    lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
> +    paddr_t maddr = INVALID_PADDR;
> +
> +    spin_lock(&p2m->lock);
> +
> +    first = __map_domain_page(p2m->first_level);
> +
> +    pte = first[first_table_offset(paddr)];
> +    if ( !pte.p2m.valid || !pte.p2m.table )
> +        goto done;
> +
> +    second = map_domain_page(first[first_table_offset(paddr)].p2m.base);

second = map_domain_page(pte.p2m.base); ?

> +    pte = second[second_table_offset(paddr)];
> +    if ( !pte.p2m.valid || !pte.p2m.table )
> +        goto done;
> +
> +    third = map_domain_page(second[second_table_offset(paddr)].p2m.base);

likewise, third = map_domain_page(pte.p2m.base); ?

> +    pte = third[third_table_offset(paddr)];
> +
> +    /* This bit must be one in the level 3 entry */
> +    if ( !pte.p2m.table )
> +        pte.bits = 0;
> +
> +done:
> +    if ( pte.p2m.valid )
> +        maddr = (pte.bits & PADDR_MASK & PAGE_MASK) | (paddr & ~PAGE_MASK);
> +
> +    if (third) unmap_domain_page(third);
> +    if (second) unmap_domain_page(second);
> +    if (first) unmap_domain_page(first);
> +
> +    spin_unlock(&p2m->lock);
> +
> +    return maddr;
> +}
> +
>  int guest_physmap_mark_populate_on_demand(struct domain *d,
>                                            unsigned long gfn,
>                                            unsigned int order)
> diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
> index 349923a..1afd5cb 100644
> --- a/xen/include/asm-arm/p2m.h
> +++ b/xen/include/asm-arm/p2m.h
> @@ -32,6 +32,9 @@ int p2m_alloc_table(struct domain *d);
>  /* */
>  void p2m_load_VTTBR(struct domain *d);
>  
> +/* */

 /* Look up the MFN corresponding to a domain's PFN. */

> +paddr_t p2m_lookup(struct domain *d, paddr_t gpfn);
> +
>  /* Setup p2m RAM mapping for domain d from start-end. */
>  int p2m_populate_ram(struct domain *d, paddr_t start, paddr_t end);
>  /* Map MMIO regions in the p2m: start_gaddr and end_gaddr is the range
> -- 
> 1.7.9.1
> 
> 
> 
> 

Re: [PATCH 11/38] arm: implement p2m lookup

[Ian Campbell]

On Thu, 2012-06-14 at 09:47 +0100, Tim Deegan wrote:
> At 16:13 +0100 on 13 Jun (1339604016), Ian Campbell wrote:
> > Yes, that looks nice, although you still need a bit of a quirk for the
> > third level table bit. Patch below.
> 
> Much nicer.   One more round of nits below if you feel keen; in any case,
> Acked-by: Tim Deegan <tim@xen.org>

Thanks. Your nits were sensible so I have folded them in.

Re: [PATCH 18/38] arm: implement vpl011 (UART) emulator.

[Ian Campbell]

On Thu, 2012-06-07 at 11:18 +0100, Tim Deegan wrote:
> At 10:34 +0100 on 07 Jun (1339065291), Ian Campbell wrote:
> > On Thu, 2012-06-07 at 10:29 +0100, Tim Deegan wrote:
> > > > +int domain_uart0_init(struct domain *d)
> > > > +{
> > > > +    int rc;
> > > > +    if ( d->domain_id == 0 )
> > > > +        return 0;
> > > 
> > > Why?  There's no equivalent gate on the MMIO handlers.
> > 
> > dom0 has the actual uart mapped at this address, not the emulated one.
> > Maybe that should be written at the caller instead?
> 
> Yes - ideally in the same place that puts the MMIO hooks in that will
> call the other functions in this file. 

Turns out that there isn't actually such a place, these are part of a
global list of mmio handlers which is traversed for any guest dabt.

This might be something we want to improve (using bits in not-present
p2m entries to identify specific handlers or whatever) but not right now
I think.

I've pulled the dom0 check out into the caller of the init function, but
left an ASSERT behind. I also shortened that long line by using a
#define for the end and removing the dom0 check from there too.

8<------------------------------------------------------------

>From 401e861ff99860d07e758c8d6e3260b8fb847fc5 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Mon, 14 May 2012 12:55:31 +0100
Subject: [PATCH] arm: implement vpl011 (UART) emulator.

This is not interended to provide a full emulation, but rather just enough to
satisfy the use made by Linux' boot time decompressor code (which is too early
for DT etc)

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/Makefile        |    1 +
 xen/arch/arm/domain.c        |    5 ++
 xen/arch/arm/io.c            |    1 +
 xen/arch/arm/io.h            |    1 +
 xen/arch/arm/vpl011.c        |  145 ++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vpl011.h        |   34 ++++++++++
 xen/include/asm-arm/domain.h |    8 ++
 7 files changed, 195 insertions(+), 0 deletions(-)
 create mode 100644 xen/arch/arm/vpl011.c
 create mode 100644 xen/arch/arm/vpl011.h

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 9440a21..5a87ba6 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -25,6 +25,7 @@ obj-y += shutdown.o
 obj-y += traps.o
 obj-y += vgic.o
 obj-y += vtimer.o
+obj-y += vpl011.o
 
 #obj-bin-y += ....o
 
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 63bad07..931261b 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -13,6 +13,7 @@
 
 #include "gic.h"
 #include "vtimer.h"
+#include "vpl011.h"
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
@@ -215,6 +216,10 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
     if ( (rc = domain_vgic_init(d)) != 0 )
         goto fail;
 
+    /* Domain 0 gets a real UART not an emulated one */
+    if ( d->domain_id && (rc = domain_uart0_init(d)) != 0 )
+        goto fail;
+
     rc = 0;
 fail:
     return rc;
diff --git a/xen/arch/arm/io.c b/xen/arch/arm/io.c
index 4461225..18f6164 100644
--- a/xen/arch/arm/io.c
+++ b/xen/arch/arm/io.c
@@ -25,6 +25,7 @@
 static const struct mmio_handler *const mmio_handlers[] =
 {
     &vgic_distr_mmio_handler,
+    &uart0_mmio_handler,
 };
 #define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
 
diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
index 8cc5ca7..9a507f5 100644
--- a/xen/arch/arm/io.h
+++ b/xen/arch/arm/io.h
@@ -40,6 +40,7 @@ struct mmio_handler {
 };
 
 extern const struct mmio_handler vgic_distr_mmio_handler;
+extern const struct mmio_handler uart0_mmio_handler;
 
 extern int handle_mmio(mmio_info_t *info);
 
diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
new file mode 100644
index 0000000..5dc8b28
--- /dev/null
+++ b/xen/arch/arm/vpl011.c
@@ -0,0 +1,145 @@
+/*
+ * xen/arch/arm/vpl011.c
+ *
+ * ARM PL011 UART Emulator (DEBUG)
+ *
+ * Ian Campbell <ian.campbell@citrix.com>
+ * Copyright (c) 2012 Citrix Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+/*
+ * This is not intended to be a full emulation of a PL011
+ * device. Rather it is intended to provide a sufficient veneer of one
+ * that early code (such as Linux's boot time decompressor) which
+ * hardcodes output directly to such a device are able to make progress.
+ *
+ * This device is not intended to be enumerable or exposed to the OS
+ * (e.g. via Device Tree).
+ */
+
+#include <xen/config.h>
+#include <xen/lib.h>
+#include <xen/sched.h>
+#include <xen/errno.h>
+#include <xen/ctype.h>
+
+#include "io.h"
+
+#define UART0_START 0x1c090000
+#define UART0_END   (UART0_START+65536)
+
+#define UARTDR 0x000
+#define UARTFR 0x018
+
+int domain_uart0_init(struct domain *d)
+{
+    ASSERT( d->domain_id );
+
+    spin_lock_init(&d->arch.uart0.lock);
+    d->arch.uart0.idx = 0;
+
+    d->arch.uart0.buf = xzalloc_array(char, VPL011_BUF_SIZE);
+    if ( !d->arch.uart0.buf )
+        return -ENOMEM;
+
+    return 0;
+
+}
+
+static void uart0_print_char(char c)
+{
+    struct vpl011 *uart = &current->domain->arch.uart0;
+
+    /* Accept only printable characters, newline, and horizontal tab. */
+    if ( !isprint(c) && (c != '\n') && (c != '\t') )
+        return ;
+
+    spin_lock(&uart->lock);
+    uart->buf[uart->idx++] = c;
+    if ( (uart->idx == (VPL011_BUF_SIZE - 2)) || (c == '\n') )
+    {
+        if ( c != '\n' )
+            uart->buf[uart->idx++] = '\n';
+        uart->buf[uart->idx] = '\0';
+        printk(XENLOG_G_DEBUG "DOM%u: %s",
+               current->domain->domain_id, uart->buf);
+        uart->idx = 0;
+    }
+    spin_unlock(&uart->lock);
+}
+
+static int uart0_mmio_check(struct vcpu *v, paddr_t addr)
+{
+    return addr >= UART0_START && addr < UART0_END;
+}
+
+static int uart0_mmio_read(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    uint32_t *r = &regs->r0 + dabt.reg;
+    int offset = (int)(info->gpa - UART0_START);
+
+    switch ( offset )
+    {
+    case UARTDR:
+        *r = 0;
+        return 1;
+    case UARTFR:
+        *r = 0x87; /* All holding registers empty, ready to send etc */
+        return 1;
+    default:
+        printk("VPL011: unhandled read r%d offset %#08x\n",
+               dabt.reg, offset);
+        domain_crash_synchronous();
+    }
+}
+
+static int uart0_mmio_write(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    uint32_t *r = &regs->r0 + dabt.reg;
+    int offset = (int)(info->gpa - UART0_START);
+
+    switch ( offset )
+    {
+    case UARTDR:
+        /* ignore any status bits */
+        uart0_print_char((int)((*r) & 0xFF));
+        return 1;
+    case UARTFR:
+        /* Silently ignore */
+        return 1;
+    default:
+        printk("VPL011: unhandled write r%d=%"PRIx32" offset %#08x\n",
+               dabt.reg, *r, offset);
+        domain_crash_synchronous();
+    }
+}
+
+const struct mmio_handler uart0_mmio_handler = {
+    .check_handler = uart0_mmio_check,
+    .read_handler  = uart0_mmio_read,
+    .write_handler = uart0_mmio_write,
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
diff --git a/xen/arch/arm/vpl011.h b/xen/arch/arm/vpl011.h
new file mode 100644
index 0000000..952d812
--- /dev/null
+++ b/xen/arch/arm/vpl011.h
@@ -0,0 +1,34 @@
+/*
+ * xen/arch/arm/vpl011.h
+ *
+ * ARM PL011 Emulation Support
+ *
+ * Ian Campbell <ian.campbell@citrix.com>
+ * Copyright (c) 2012 Citrix Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPL011_H__
+#define __ARCH_ARM_VPL011_H__
+
+extern int domain_uart0_init(struct domain *d);
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 10ed540..f295a82 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -48,6 +48,14 @@ struct arch_domain
         struct vgic_irq_rank *shared_irqs;
         struct pending_irq *pending_irqs;
     } vgic;
+
+    struct vpl011 {
+#define VPL011_BUF_SIZE 128
+        char                  *buf;
+        int                    idx;
+        spinlock_t             lock;
+    } uart0;
+
 }  __cacheline_aligned;
 
 struct arch_vcpu
-- 
1.7.9.1

Re: [PATCH 21/38] arm: dump guest s1 walk on data abort which is not a stage 2 issue.

[Ian Campbell]

On Thu, 2012-06-07 at 10:41 +0100, Tim Deegan wrote:
> At 15:39 +0000 on 01 Jun (1338565190), Ian Campbell wrote:
> > +    offset = addr >> (12+10);
> > +    printk("1ST[%#03"PRIx32"] (%#"PRIpaddr") = %#010"PRIx32"\n",
> 
> Nit: 0x%08 prints '0' as '0x00000000', which is nicer I think.

Agreed, I wasn't aware of this nit of the printf formatting strings
until you told me.

> Otherwise: ack.

Thanks, update patch is below.

8<---------------------------------------------------------------

>From e045b07e1509482e66333a9300668b4d24c33d13 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Mon, 14 May 2012 14:19:41 +0100
Subject: [PATCH] arm: dump guest s1 walk on data abort which is not a stage 2
 issue.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Tim Deegan <tim@xen.org>
---
 xen/arch/arm/traps.c            |   75 +++++++++++++++++++++++++++++++++++---
 xen/include/asm-arm/processor.h |    1 +
 2 files changed, 70 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 40bb375..d8eb5a9 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -28,6 +28,7 @@
 #include <xen/errno.h>
 #include <xen/hypercall.h>
 #include <xen/softirq.h>
+#include <xen/domain_page.h>
 #include <public/xen.h>
 #include <asm/regs.h>
 #include <asm/cpregs.h>
@@ -528,6 +529,62 @@ static void do_cp15_64(struct cpu_user_regs *regs,
 
 }
 
+void dump_guest_s1_walk(struct domain *d, uint32_t addr)
+{
+    uint32_t ttbcr = READ_CP32(TTBCR);
+    uint32_t ttbr0 = READ_CP32(TTBR0);
+    paddr_t paddr;
+    uint32_t offset;
+    uint32_t *first = NULL, *second = NULL;
+
+    printk("dom%d VA 0x%08"PRIx32"\n", d->domain_id, addr);
+    printk("    TTBCR: 0x%08"PRIx32"\n", ttbcr);
+    printk("    TTBR0: 0x%08"PRIx32" = 0x%"PRIpaddr"\n",
+           ttbr0, p2m_lookup(d, ttbr0 & PAGE_MASK));
+
+    if ( ttbcr & TTBCR_EAE )
+    {
+        printk("Cannot handle LPAE guest PT walk\n");
+        return;
+    }
+    if ( (ttbcr & TTBCR_N_MASK) != 0 )
+    {
+        printk("Cannot handle TTBR1 guest walks\n");
+        return;
+    }
+
+    paddr = p2m_lookup(d, ttbr0 & PAGE_MASK);
+    if ( paddr == INVALID_PADDR )
+    {
+        printk("Failed TTBR0 maddr lookup\n");
+        goto done;
+    }
+    first = map_domain_page(paddr>>PAGE_SHIFT);
+
+    offset = addr >> (12+10);
+    printk("1ST[0x%"PRIx32"] (0x%"PRIpaddr") = 0x%08"PRIx32"\n",
+           offset, paddr, first[offset]);
+    if ( !(first[offset] & 0x1) ||
+         !(first[offset] & 0x2) )
+        goto done;
+
+    paddr = p2m_lookup(d, first[offset] & PAGE_MASK);
+
+    if ( paddr == INVALID_PADDR )
+    {
+        printk("Failed L1 entry maddr lookup\n");
+        goto done;
+    }
+    second = map_domain_page(paddr>>PAGE_SHIFT);
+    offset = (addr >> 12) & 0x3FF;
+    printk("2ND[0x%"PRIx32"] (0x%"PRIpaddr") = 0x%08"PRIx32"\n",
+           offset, paddr, second[offset]);
+
+done:
+    if (second) unmap_domain_page(second);
+    if (first) unmap_domain_page(first);
+}
+
 static void do_trap_data_abort_guest(struct cpu_user_regs *regs,
                                      struct hsr_dabt dabt)
 {
@@ -535,11 +592,12 @@ static void do_trap_data_abort_guest(struct cpu_user_regs *regs,
     int level = -1;
     mmio_info_t info;
 
+    info.dabt = dabt;
+    info.gva = READ_CP32(HDFAR);
+
     if (dabt.s1ptw)
         goto bad_data_abort;
 
-    info.dabt = dabt;
-    info.gva = READ_CP32(HDFAR);
     info.gpa = gva_to_ipa(info.gva);
 
     if (handle_mmio(&info))
@@ -553,18 +611,23 @@ bad_data_abort:
     msg = decode_fsc( dabt.dfsc, &level);
 
     printk("Guest data abort: %s%s%s\n"
-           "    gva=%"PRIx32" gpa=%"PRIpaddr"\n",
+           "    gva=%"PRIx32"\n",
            msg, dabt.s1ptw ? " S2 during S1" : "",
            fsc_level_str(level),
-           info.gva, info.gpa);
-    if (dabt.valid)
+           info.gva);
+    if ( !dabt.s1ptw )
+        printk("    gpa=%"PRIpaddr"\n", info.gpa);
+    if ( dabt.valid )
         printk("    size=%d sign=%d write=%d reg=%d\n",
                dabt.size, dabt.sign, dabt.write, dabt.reg);
     else
         printk("    instruction syndrome invalid\n");
     printk("    eat=%d cm=%d s1ptw=%d dfsc=%d\n",
            dabt.eat, dabt.cache, dabt.s1ptw, dabt.dfsc);
-
+    if ( !dabt.s1ptw )
+        dump_p2m_lookup(current->domain, info.gpa);
+    else
+        dump_guest_s1_walk(current->domain, info.gva);
     show_execution_state(regs);
     panic("Unhandled guest data abort\n");
 }
diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h
index ec6fb48..81924a4 100644
--- a/xen/include/asm-arm/processor.h
+++ b/xen/include/asm-arm/processor.h
@@ -25,6 +25,7 @@
 #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
 
 /* TTBCR Translation Table Base Control Register */
+#define TTBCR_EAE    0x80000000
 #define TTBCR_N_MASK 0x07
 #define TTBCR_N_16KB 0x00
 #define TTBCR_N_8KB  0x01
-- 
1.7.9.1

Re: [PATCH 22/38] arm: implement vcpu_show_execution_state

[Ian Campbell]

On Wed, 2012-06-06 at 18:26 +0100, Stefano Stabellini wrote:
> > @@ -334,6 +360,26 @@ void show_execution_state(struct cpu_user_regs *regs)
> >      show_stack(regs);
> >  }
> >  
> > +void vcpu_show_execution_state(struct vcpu *v)
> > +{
> > +    printk("*** Dumping Dom%d vcpu#%d state: ***\n",
> > +           v->domain->domain_id, v->vcpu_id);
> > +
> > +    if ( v == current )
> > +    {
> > +        show_execution_state(guest_cpu_user_regs());
> > +        return;
> > +    }
> > +
> > +    vcpu_pause(v); /* acceptably dangerous */
> > +
> > +    vcpu_show_registers(v);
> > +    if ( !usr_mode(&v->arch.cpu_info->guest_cpu_user_regs) )
> > +        show_guest_stack(&v->arch.cpu_info->guest_cpu_user_regs);
> 
> isn't the if condition inverted?

I think what I'm trying to do here is only dump the stack if the guest
was in one of the privileged modes (i.e. not user mode). 

I don't really recall why -- I guess I figured the usermode stack was
not likely to be all that useful. Unless there's some reason (which I've
forgotten) why we can't walk guest userspace stacks, but given that the
entirety of the implementation of show_guest_stack() is:
        static void show_guest_stack(struct cpu_user_regs *regs)
        {
            printk("GUEST STACK GOES HERE\n");
        }
I guess it doesn't really matter ;-)

Ian.
> 
> > +    vcpu_unpause(v);
> > +}
> > +
> >  static void do_unexpected_trap(const char *msg, struct cpu_user_regs *regs)
> >  {
> >      printk("Unexpected Trap: %s\n", msg);
> 

Re: [PATCH 22/38] arm: implement vcpu_show_execution_state

[Ian Campbell]

On Wed, 2012-06-20 at 14:53 +0100, Ian Campbell wrote:
> On Wed, 2012-06-06 at 18:26 +0100, Stefano Stabellini wrote:
> > > @@ -334,6 +360,26 @@ void show_execution_state(struct cpu_user_regs *regs)
> > >      show_stack(regs);
> > >  }
> > >  
> > > +void vcpu_show_execution_state(struct vcpu *v)
> > > +{
> > > +    printk("*** Dumping Dom%d vcpu#%d state: ***\n",
> > > +           v->domain->domain_id, v->vcpu_id);
> > > +
> > > +    if ( v == current )
> > > +    {
> > > +        show_execution_state(guest_cpu_user_regs());
> > > +        return;
> > > +    }
> > > +
> > > +    vcpu_pause(v); /* acceptably dangerous */
> > > +
> > > +    vcpu_show_registers(v);
> > > +    if ( !usr_mode(&v->arch.cpu_info->guest_cpu_user_regs) )
> > > +        show_guest_stack(&v->arch.cpu_info->guest_cpu_user_regs);
> > 
> > isn't the if condition inverted?
> 
> I think what I'm trying to do here is only dump the stack if the guest
> was in one of the privileged modes (i.e. not user mode). 
> 
> I don't really recall why -- I guess I figured the usermode stack was
> not likely to be all that useful.

It looks like I just copied the behaviour of x86, which has:
   if ( guest_kernel_mode(v, &v->arch.user_regs) )
        show_guest_stack(v, &v->arch.user_regs);

Ian.

Re: [PATCH 23/38] arm: use correct attributes for mappings in copy_from_paddr()

[Ian Campbell]

On Wed, 2012-06-06 at 18:38 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > The DTB is in RAM (hence bufferable), kernel is in flash and therefor requires
> > a device type mapping (hence dev shared).
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/kernel.c       |    8 ++++----
> >  xen/arch/arm/setup.c        |    2 +-
> >  xen/include/asm-arm/setup.h |    2 +-
> >  3 files changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/xen/arch/arm/kernel.c b/xen/arch/arm/kernel.c
> > index 130d488..1a705c9 100644
> > --- a/xen/arch/arm/kernel.c
> > +++ b/xen/arch/arm/kernel.c
> > @@ -39,7 +39,7 @@ struct minimal_dtb_header {
> >   * @paddr: source physical address
> >   * @len: length to copy
> >   */
> > -void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
> > +void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int mattr)
> >  {
> >      void *src = (void *)FIXMAP_ADDR(FIXMAP_MISC);
> >  
> > @@ -51,7 +51,7 @@ void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
> >          s = paddr & (PAGE_SIZE-1);
> >          l = min(PAGE_SIZE - s, len);
> >  
> > -        set_fixmap(FIXMAP_MISC, p, DEV_SHARED);
> > +        set_fixmap(FIXMAP_MISC, p, mattr);
> >          memcpy(dst, src + s, l);
> >  
> >          paddr += l;
> > @@ -111,7 +111,7 @@ static int kernel_try_zimage_prepare(struct kernel_info *info)
> >      /*
> >       * Check for an appended DTB.
> >       */
> > -    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr));
> > +    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr), DEV_SHARED);
> >      if (be32_to_cpu(dtb_hdr.magic) == DTB_MAGIC) {
> >          end += be32_to_cpu(dtb_hdr.total_size);
> >      }
> > @@ -151,7 +151,7 @@ static int kernel_try_elf_prepare(struct kernel_info *info)
> >      if ( info->kernel_img == NULL )
> >          panic("Cannot allocate temporary buffer for kernel.\n");
> >  
> > -    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE);
> > +    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE, DEV_SHARED);
> >  
> >      if ( (rc = elf_init(&info->elf.elf, info->kernel_img, KERNEL_FLASH_SIZE )) != 0 )
> >          return rc;
> > diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> > index b0cfacc..f5473cd 100644
> > --- a/xen/arch/arm/setup.c
> > +++ b/xen/arch/arm/setup.c
> > @@ -122,7 +122,7 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
> >       * TODO: handle other payloads too.
> >       */
> >      device_tree_flattened = mfn_to_virt(alloc_boot_pages(dtb_pages, 1));
> > -    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size);
> > +    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size, BUFFERABLE);
> >  
> >      /* Add non-xenheap memory */
> >      init_boot_pages(pfn_to_paddr(xenheap_mfn_start + xenheap_pages),
> > diff --git a/xen/include/asm-arm/setup.h b/xen/include/asm-arm/setup.h
> > index 05ff89e..faadccc 100644
> > --- a/xen/include/asm-arm/setup.h
> > +++ b/xen/include/asm-arm/setup.h
> > @@ -3,7 +3,7 @@
> >  
> >  #include <public/version.h>
> >  
> > -void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len);
> > +void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int mattr);
> >  
> >  void arch_get_xen_caps(xen_capabilities_info_t *info);
> >  
> 
> The patch is correct, but it shouldn't call the new parameter mattr
> because it can easily be confused with the memattr bits (see
> http://marc.info/?l=xen-devel&m=133856578918985).
> I would call it "int attrindx" instead.

Yes, this is a good point and attrindx is a good name, I'll make that
change.

I also added a comment to the two sets of #defines explaining which they
are. Updated patch is:

8<------------------------------------------------

>From 08f78c5b7e31ed0f6533ff6a378c586c0db58276 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Mon, 14 May 2012 14:02:26 +0100
Subject: [PATCH] arm: use correct attributes for mappings in
 copy_from_paddr()

The DTB is in RAM (hence bufferable), kernel is in flash and therefor requires
a device type mapping (hence dev shared).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: David Vrabel <david.vrabel@citrix.com>
---
 xen/arch/arm/kernel.c       |    8 ++++----
 xen/arch/arm/setup.c        |    2 +-
 xen/include/asm-arm/page.h  |   15 +++++++++++++++
 xen/include/asm-arm/setup.h |    2 +-
 4 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/kernel.c b/xen/arch/arm/kernel.c
index 130d488..2d56130 100644
--- a/xen/arch/arm/kernel.c
+++ b/xen/arch/arm/kernel.c
@@ -39,7 +39,7 @@ struct minimal_dtb_header {
  * @paddr: source physical address
  * @len: length to copy
  */
-void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
+void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int attrindx)
 {
     void *src = (void *)FIXMAP_ADDR(FIXMAP_MISC);
 
@@ -51,7 +51,7 @@ void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len)
         s = paddr & (PAGE_SIZE-1);
         l = min(PAGE_SIZE - s, len);
 
-        set_fixmap(FIXMAP_MISC, p, DEV_SHARED);
+        set_fixmap(FIXMAP_MISC, p, attrindx);
         memcpy(dst, src + s, l);
 
         paddr += l;
@@ -111,7 +111,7 @@ static int kernel_try_zimage_prepare(struct kernel_info *info)
     /*
      * Check for an appended DTB.
      */
-    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr));
+    copy_from_paddr(&dtb_hdr, KERNEL_FLASH_ADDRESS + end - start, sizeof(dtb_hdr), DEV_SHARED);
     if (be32_to_cpu(dtb_hdr.magic) == DTB_MAGIC) {
         end += be32_to_cpu(dtb_hdr.total_size);
     }
@@ -151,7 +151,7 @@ static int kernel_try_elf_prepare(struct kernel_info *info)
     if ( info->kernel_img == NULL )
         panic("Cannot allocate temporary buffer for kernel.\n");
 
-    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE);
+    copy_from_paddr(info->kernel_img, KERNEL_FLASH_ADDRESS, KERNEL_FLASH_SIZE, DEV_SHARED);
 
     if ( (rc = elf_init(&info->elf.elf, info->kernel_img, KERNEL_FLASH_SIZE )) != 0 )
         return rc;
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index d6c0178..fd70553 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -122,7 +122,7 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t dtb_size)
      * TODO: handle other payloads too.
      */
     device_tree_flattened = mfn_to_virt(alloc_boot_pages(dtb_pages, 1));
-    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size);
+    copy_from_paddr(device_tree_flattened, dtb_paddr, dtb_size, BUFFERABLE);
 
     /* Add non-xenheap memory */
     init_boot_pages(pfn_to_paddr(xenheap_mfn_start + xenheap_pages),
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index 6efe23c..2b6c1780 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -36,6 +36,14 @@
 #define MAIR0VAL 0xeeaa4400
 #define MAIR1VAL 0xff000004
 
+/*
+ * Attribute Indexes.
+ *
+ * These are valid in the AttrIndx[2:0] field of an LPAE stage 1 page
+ * table entry. They are indexes into the bytes of the MAIR*
+ * registers, as defined above.
+ *
+ */
 #define UNCACHED      0x0
 #define BUFFERABLE    0x1
 #define WRITETHROUGH  0x2
@@ -46,6 +54,13 @@
 #define DEV_WC        BUFFERABLE
 #define DEV_CACHED    WRITEBACK
 
+/*
+ * Stage 2 Memory Type.
+ *
+ * These are valid in the MemAttr[3:0] field of an LPAE stage 2 page
+ * table entry.
+ *
+ */
 #define MATTR_DEV     0x1
 #define MATTR_MEM     0xf
 
diff --git a/xen/include/asm-arm/setup.h b/xen/include/asm-arm/setup.h
index 05ff89e..6433b4e 100644
--- a/xen/include/asm-arm/setup.h
+++ b/xen/include/asm-arm/setup.h
@@ -3,7 +3,7 @@
 
 #include <public/version.h>
 
-void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len);
+void copy_from_paddr(void *dst, paddr_t paddr, unsigned long len, int attrindx);
 
 void arch_get_xen_caps(xen_capabilities_info_t *info);
 
-- 
1.7.9.1

Re: [PATCH 25/38] arm: remove old identity map of boot paddr when we are done with it.

[Ian Campbell]

On Wed, 2012-06-06 at 19:18 +0100, Tim Deegan wrote:
> At 19:04 +0100 on 06 Jun (1339009473), Stefano Stabellini wrote:
> > On Fri, 1 Jun 2012, Ian Campbell wrote:
> > > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > > ---
> > >  xen/arch/arm/mm.c |    8 ++++++++
> > >  1 files changed, 8 insertions(+), 0 deletions(-)
> > > 
> > > diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> > > index 160a4e9..ab52171 100644
> > > --- a/xen/arch/arm/mm.c
> > > +++ b/xen/arch/arm/mm.c
> > > @@ -214,6 +214,14 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
> > >      lpae_t pte, *p;
> > >      int i;
> > >  
> > > +    if ( boot_phys_offset != 0 )
> > > +    {
> > > +        /* Remove the old identity mapping of the boot paddr */
> > > +        pte.bits = 0;
> > > +        dest_va = (unsigned long)_start + boot_phys_offset;
> > > +        write_pte(xen_second + second_linear_offset(dest_va), pte);
> > > +    }
> > 
> > It looks like we are already doing this few lines below.
> 
> We used to do this here and now we do it futher down, after the copy.
> That way the secondary CPUs can come up on the pre-copy tables where
> the identity map still exists.

I think I wrote this patch before your SMP series went in, but since
there was no textual conflict I didn't notice that I should nuke it.

Hrm... I was about to do the nuking but without it I see:

        At 0x00000000, Error: Pagetable properties were remapped while
        MMU was enabled, may be stale translations

from the model. I use 
        cluster.messages.ignore_warning_level=0x1
in my cfg which I think is a bit more verbose than the default.

I need to re-figure out what lead me to this change...

Ian.

Re: [PATCH 25/38] arm: remove old identity map of boot paddr when we are done with it.

[Ian Campbell]

On Thu, 2012-06-21 at 09:15 +0100, Ian Campbell wrote:
> On Wed, 2012-06-06 at 19:18 +0100, Tim Deegan wrote:
> > At 19:04 +0100 on 06 Jun (1339009473), Stefano Stabellini wrote:
> > > On Fri, 1 Jun 2012, Ian Campbell wrote:
> > > > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > > > ---
> > > >  xen/arch/arm/mm.c |    8 ++++++++
> > > >  1 files changed, 8 insertions(+), 0 deletions(-)
> > > > 
> > > > diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
> > > > index 160a4e9..ab52171 100644
> > > > --- a/xen/arch/arm/mm.c
> > > > +++ b/xen/arch/arm/mm.c
> > > > @@ -214,6 +214,14 @@ void __init setup_pagetables(unsigned long boot_phys_offset)
> > > >      lpae_t pte, *p;
> > > >      int i;
> > > >  
> > > > +    if ( boot_phys_offset != 0 )
> > > > +    {
> > > > +        /* Remove the old identity mapping of the boot paddr */
> > > > +        pte.bits = 0;
> > > > +        dest_va = (unsigned long)_start + boot_phys_offset;
> > > > +        write_pte(xen_second + second_linear_offset(dest_va), pte);
> > > > +    }
> > > 
> > > It looks like we are already doing this few lines below.
> > 
> > We used to do this here and now we do it futher down, after the copy.
> > That way the secondary CPUs can come up on the pre-copy tables where
> > the identity map still exists.
> 
> I think I wrote this patch before your SMP series went in, but since
> there was no textual conflict I didn't notice that I should nuke it.
> 
> Hrm... I was about to do the nuking but without it I see:
> 
>         At 0x00000000, Error: Pagetable properties were remapped while
>         MMU was enabled, may be stale translations

I must've been running a stale hypervisor image because now I don't see
this even with the patch removed.

So I've now nuked the patch from my queue.

Ian.

Re: [PATCH 27/38] arm: split pending SPIs (global) out from pending PPIs and SGIs (per CPU)

[Ian Campbell]

On Thu, 2012-06-07 at 11:35 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > This tracks SPIs in struct arch_domain and PPIs+SGIs in struct arch_vcpu which
> > seems more logical.
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/vgic.c          |   17 ++++++++++-------
> >  xen/include/asm-arm/domain.h |   10 ++++++++++
> >  2 files changed, 20 insertions(+), 7 deletions(-)
> > 
> > diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> > index 629a0da..91d6166 100644
> > --- a/xen/arch/arm/vgic.c
> > +++ b/xen/arch/arm/vgic.c
> > @@ -82,9 +82,8 @@ int domain_vgic_init(struct domain *d)
> >      d->arch.vgic.shared_irqs =
> >          xmalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
> >      d->arch.vgic.pending_irqs =
> > -        xmalloc_array(struct pending_irq,
> > -                d->arch.vgic.nr_lines + (32 * d->max_vcpus));
> > -    for (i=0; i<d->arch.vgic.nr_lines + (32 * d->max_vcpus); i++)
> > +        xzalloc_array(struct pending_irq, d->arch.vgic.nr_lines);
> > +    for (i=0; i<d->arch.vgic.nr_lines; i++)
> >          INIT_LIST_HEAD(&d->arch.vgic.pending_irqs[i].inflight);
> >      for (i=0; i<DOMAIN_NR_RANKS(d); i++)
> >          spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
> > @@ -98,6 +97,10 @@ int vcpu_vgic_init(struct vcpu *v)
> >  
> >      spin_lock_init(&v->arch.vgic.private_irqs.lock);
> >  
> > +    memset(&v->arch.vgic.pending_irqs, 0, sizeof(v->arch.vgic.pending_irqs));
> > +    for (i = 0; i < 32; i++)
> > +        INIT_LIST_HEAD(&v->arch.vgic.pending_irqs[i].inflight);
> > +
> >      /* For SGI and PPI the target is always this CPU */
> >      for ( i = 0 ; i < 8 ; i++ )
> >          v->arch.vgic.private_irqs.itargets[i] =
> > @@ -535,8 +538,7 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
> >      /* Pending irqs allocation strategy: the first vgic.nr_lines irqs
> >       * are used for SPIs; the rests are used for per cpu irqs */
> >      if ( irq < 32 )
> > -        n = &v->domain->arch.vgic.pending_irqs[irq + (v->vcpu_id * 32)
> > -            + v->domain->arch.vgic.nr_lines];
> > +        n = &v->arch.vgic.pending_irqs[irq];
> >      else
> >          n = &v->domain->arch.vgic.pending_irqs[irq - 32];
> >      return n;
> > @@ -548,6 +550,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
> >      uint8_t priority;
> >      struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
> >      struct pending_irq *iter, *n = irq_to_pending(v, irq);
> > +    unsigned long flags;
> >  
> >      /* irq still pending */
> >      if (!list_empty(&n->inflight))
> > @@ -564,7 +567,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
> >  
> >      gic_set_guest_irq(irq, GICH_LR_PENDING, priority);
> >  
> > -    spin_lock(&v->arch.vgic.lock);
> > +    spin_lock_irqsave(&v->arch.vgic.lock, flags);
> >      list_for_each_entry ( iter, &v->arch.vgic.inflight_irqs, inflight )
> >      {
> >          if ( iter->priority > priority )
> > @@ -575,7 +578,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
> >          }
> >      }
> >      list_add_tail(&n->inflight, &v->arch.vgic.inflight_irqs);
> > -    spin_unlock(&v->arch.vgic.lock);
> > +    spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
> >      /* we have a new higher priority irq, inject it into the guest */
> >  }
> 
> Besides moving PPIs and SGIs to struct vcpu, this patch also turns
> spin_lock into spin_lock_irqsave in vgic_vcpu_inject_irq: I think it is
> correct because it can be called in IRQ context,

good point.

> but it needs to be explicitly stated in the commit message.

I've split it into its own patch instead:

8<----------------------------------------

>From 6d4f199127488480f9a9cc2a94bb73734a75ff88 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Tue, 26 Jun 2012 09:00:55 +0000
Subject: [PATCH] arm: use interrupt safe spin locks in vgic_vcpu_inject_irq

This function can be called in both interrupt and regular context.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/vgic.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index af3523f..91d6166 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -550,6 +550,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
     uint8_t priority;
     struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
     struct pending_irq *iter, *n = irq_to_pending(v, irq);
+    unsigned long flags;
 
     /* irq still pending */
     if (!list_empty(&n->inflight))
@@ -566,7 +567,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
 
     gic_set_guest_irq(irq, GICH_LR_PENDING, priority);
 
-    spin_lock(&v->arch.vgic.lock);
+    spin_lock_irqsave(&v->arch.vgic.lock, flags);
     list_for_each_entry ( iter, &v->arch.vgic.inflight_irqs, inflight )
     {
         if ( iter->priority > priority )
@@ -577,7 +578,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
         }
     }
     list_add_tail(&n->inflight, &v->arch.vgic.inflight_irqs);
-    spin_unlock(&v->arch.vgic.lock);
+    spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
     /* we have a new higher priority irq, inject it into the guest */
 }
 
-- 
1.7.9.1

Re: [PATCH 28/38] arm: map GICV in all domains, not just dom0.

[Ian Campbell]

On Thu, 2012-06-07 at 10:51 +0100, Tim Deegan wrote:
> At 15:39 +0000 on 01 Jun (1338565197), Ian Campbell wrote:
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> > index a7fb227..e15c1e8 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -329,13 +329,17 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
> >  
> >          if ( (rc = p2m_alloc_table(d)) != 0 )
> >              goto fail;
> > -    }
> >  
> > -    if ( (rc = domain_vgic_init(d)) != 0 )
> > -        goto fail;
> > +        if ( (rc = gicv_setup(d)) != 0 )
> > +            goto fail;
> > +
> > +        if ( (rc = domain_vgic_init(d)) != 0 )
> > +            goto fail;
> > +    }
> >  
> >      rc = 0;
> >  fail:
> > +    /*XXX unwind allocations etc */
> 
> Ahem. :)

Indeed ;-)

I've added the following to the end of the series:

8<------------------------------------------------

>From 9b12490d654683b1c3c65eb41ab35e20cd7682db Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 7 Jun 2012 16:59:57 +0000
Subject: [PATCH] arm: unwind allocations etc on arch_domain_create_failure

This involves adding the necessary teardown/free functions for some modules.

Don't initialise full arch domain state for the idle domain, it's not needed.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/domain.c     |   42 +++++++++++++++++++++++++-----------------
 xen/arch/arm/gic.h        |    3 +++
 xen/arch/arm/p2m.c        |   15 +++++++++++++++
 xen/arch/arm/vgic.c       |    6 ++++++
 xen/arch/arm/vpl011.c     |    5 +++++
 xen/arch/arm/vpl011.h     |    1 +
 xen/include/asm-arm/p2m.h |    3 +++
 7 files changed, 58 insertions(+), 17 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index c16fa02..f61568b 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -317,37 +317,45 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
 {
     int rc;
 
+    /* Idle domains do not need this setup */
+    if ( is_idle_domain(d) )
+        return 0;
+
     rc = -ENOMEM;
     if ( (rc = p2m_init(d)) != 0 )
         goto fail;
 
-    if ( !is_idle_domain(d) )
-    {
-        rc = -ENOMEM;
-        if ( (d->shared_info = alloc_xenheap_pages(0, 0)) == NULL )
-            goto fail;
+    if ( (d->shared_info = alloc_xenheap_pages(0, 0)) == NULL )
+        goto fail;
 
-        clear_page(d->shared_info);
-        share_xen_page_with_guest(
-                virt_to_page(d->shared_info), d, XENSHARE_writable);
+    clear_page(d->shared_info);
+    share_xen_page_with_guest(
+        virt_to_page(d->shared_info), d, XENSHARE_writable);
 
-        if ( (rc = p2m_alloc_table(d)) != 0 )
-            goto fail;
+    if ( (rc = p2m_alloc_table(d)) != 0 )
+        goto fail;
 
-        if ( (rc = gicv_setup(d)) != 0 )
-            goto fail;
+    if ( (rc = gicv_setup(d)) != 0 )
+        goto fail;
 
-        if ( (rc = domain_vgic_init(d)) != 0 )
-            goto fail;
-    }
+    if ( (rc = domain_vgic_init(d)) != 0 )
+        goto fail;
 
     /* Domain 0 gets a real UART not an emulated one */
     if ( d->domain_id && (rc = domain_uart0_init(d)) != 0 )
         goto fail;
 
-    rc = 0;
+    return 0;
+
 fail:
-    /*XXX unwind allocations etc */
+    d->is_dying = DOMDYING_dead;
+    free_xenheap_page(d->shared_info);
+
+    p2m_teardown(d);
+
+    domain_vgic_free(d);
+    domain_uart0_free(d);
+
     return rc;
 }
 
diff --git a/xen/arch/arm/gic.h b/xen/arch/arm/gic.h
index 018d820..e36d6ad 100644
--- a/xen/arch/arm/gic.h
+++ b/xen/arch/arm/gic.h
@@ -125,7 +125,10 @@
 #define VGIC_IRQ_EVTCHN_CALLBACK 31
 
 extern int domain_vgic_init(struct domain *d);
+extern void domain_vgic_free(struct domain *d);
+
 extern int vcpu_vgic_init(struct vcpu *v);
+
 extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq,int virtual);
 extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
 
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 67bfeba..073216b 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -288,6 +288,21 @@ int p2m_alloc_table(struct domain *d)
     return 0;
 }
 
+void p2m_teardown(struct domain *d)
+{
+    struct p2m_domain *p2m = &d->arch.p2m;
+    struct page_info *pg;
+
+    spin_lock(&p2m->lock);
+
+    while ( (pg = page_list_remove_head(&p2m->pages)) )
+        free_domheap_page(pg);
+
+    p2m->first_level = NULL;
+
+    spin_unlock(&p2m->lock);
+}
+
 int p2m_init(struct domain *d)
 {
     struct p2m_domain *p2m = &d->arch.p2m;
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 91d6166..06bbd0c 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -90,6 +90,12 @@ int domain_vgic_init(struct domain *d)
     return 0;
 }
 
+void domain_vgic_free(struct domain *d)
+{
+    xfree(d->arch.vgic.shared_irqs);
+    xfree(d->arch.vgic.pending_irqs);
+}
+
 int vcpu_vgic_init(struct vcpu *v)
 {
     int i;
diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
index 5dc8b28..1522667 100644
--- a/xen/arch/arm/vpl011.c
+++ b/xen/arch/arm/vpl011.c
@@ -56,6 +56,11 @@ int domain_uart0_init(struct domain *d)
 
 }
 
+void domain_uart0_free(struct domain *d)
+{
+    xfree(d->arch.uart0.buf);
+}
+
 static void uart0_print_char(char c)
 {
     struct vpl011 *uart = &current->domain->arch.uart0;
diff --git a/xen/arch/arm/vpl011.h b/xen/arch/arm/vpl011.h
index 952d812..eabd99d 100644
--- a/xen/arch/arm/vpl011.h
+++ b/xen/arch/arm/vpl011.h
@@ -21,6 +21,7 @@
 #define __ARCH_ARM_VPL011_H__
 
 extern int domain_uart0_init(struct domain *d);
+extern void domain_uart0_free(struct domain *d);
 
 #endif
 
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 666bb88..14e71bf 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -23,6 +23,9 @@ struct p2m_domain {
 /* Init the datastructures for later use by the p2m code */
 int p2m_init(struct domain *d);
 
+/* Return all the p2m resources to Xen. */
+void p2m_teardown(struct domain *d);
+
 /* Allocate a new p2m table for a domain.
  *
  * Returns 0 for success or -errno.
-- 
1.7.9.1

Re: [PATCH 29/38] arm: delay enabling data-cache until paging enabled.

[Ian Campbell]

On Thu, 2012-06-07 at 14:59 +0100, Tim Deegan wrote:
> At 20:04 +0100 on 01 Jun (1338581061), Ian Campbell wrote:
> > On Fri, 2012-06-01 at 18:05 +0100, Tim Deegan wrote:
> > > At 15:39 +0000 on 01 Jun (1338565198), Ian Campbell wrote:
> > > > With enough warnings enabled the model seemed to be complaining that pages
> > > > cached before paging was enabled had been mapped with to inconsistent sets of
> > > > attributes. I'm not convinced that isn't a model issue, nor am I convinced
> > > > this has really fixed anything, but it seems sensible enough.
> > > 
> > > This might be what breaks secondary CPU bringup: pagetables built by CPU
> > > 0 may not have been flushed all the way to RAM when CPU 1 comes up, and
> > > CPU 1 isn't participating in cache coherence protocols when it
> > > starts to need them.
> > 
> > The issue here is the lack of the necessary flush, rather than this
> > change particularly, right?
> 
> It turns out to be much more prosaic - the patch to delete the identity
> mapping from the boot pagetables was scuppering non-boot CPUs.
> 
> So this is OK, but can I suggest this to tidy it up?

Seems sensible. With this and Stefano's spelling fix it becomes:

>From fe5304fb1d1e6e607b125ccfbdc59b25e1d84b34 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Mon, 14 May 2012 15:49:27 +0100
Subject: [PATCH] arm: enable data-cache at the same time as enabling the MMU,
 not before

With enough warnings enabled the model seemed to be complaining that pages
cached before paging was enabled had been mapped with to inconsistent sets of
attributes. I'm not convinced that isn't a model issue, nor am I convinced
this has really fixed anything, but it seems sensible enough.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/head.S |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/head.S b/xen/arch/arm/head.S
index 9a7714a..cdbe011 100644
--- a/xen/arch/arm/head.S
+++ b/xen/arch/arm/head.S
@@ -148,10 +148,11 @@ hyp:
 	 * Exceptions in LE ARM,
 	 * Low-latency IRQs disabled,
 	 * Write-implies-XN disabled (for now),
-	 * I-cache and d-cache enabled,
+	 * D-cache disabled (for now),
+	 * I-cache enabled,
 	 * Alignment checking enabled,
 	 * MMU translation disabled (for now). */
-	ldr   r0, =(HSCTLR_BASE|SCTLR_A|SCTLR_C)
+	ldr   r0, =(HSCTLR_BASE|SCTLR_A)
 	mcr   CP32(r0, HSCTLR)
 
 	/* Write Xen's PT's paddr into the HTTBR */
@@ -210,7 +211,7 @@ pt_ready:
 
 	ldr   r1, =paging            /* Explicit vaddr, not RIP-relative */
 	mrc   CP32(r0, HSCTLR)
-	orr   r0, r0, #0x1           /* Add in the MMU enable bit */
+	orr   r0, r0, #(SCTLR_M|SCTLR_C) /* Enable MMU and D-cache */
 	dsb                          /* Flush PTE writes and finish reads */
 	mcr   CP32(r0, HSCTLR)       /* now paging is enabled */
 	isb                          /* Now, flush the icache */
-- 
1.7.9.1

Re: [PATCH 31/38] arm: gic.lock can be taken in interrupt context, so lock appropriately.

[Ian Campbell]

On Thu, 2012-06-07 at 11:49 +0100, Stefano Stabellini wrote:
> On Fri, 1 Jun 2012, Ian Campbell wrote:
> > In particular it is taken by gic_set_guest_irq which is called by
> > vgic_vcpu_inject_irq
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  xen/arch/arm/gic.c |   20 ++++++++++----------
> >  1 files changed, 10 insertions(+), 10 deletions(-)
> > 
> > diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> > index a398f92..ededa99 100644
> > --- a/xen/arch/arm/gic.c
> > +++ b/xen/arch/arm/gic.c
> > @@ -329,19 +329,19 @@ int __init gic_init(void)
> >  /* Set up the per-CPU parts of the GIC for a secondary CPU */
> >  void __cpuinit gic_init_secondary_cpu(void)
> >  {
> > -    spin_lock(&gic.lock);
> > +    spin_lock_irq(&gic.lock);
> >      gic_cpu_init();
> >      gic_hyp_init();
> > -    spin_unlock(&gic.lock);
> > +    spin_unlock_irq(&gic.lock);
> >  }
> >  
> >  /* Shut down the per-CPU GIC interface */
> >  void gic_disable_cpu(void)
> >  {
> > -    spin_lock(&gic.lock);
> > +    spin_lock_irq(&gic.lock);
> >      gic_cpu_disable();
> >      gic_hyp_disable();
> > -    spin_unlock(&gic.lock);
> > +    spin_unlock_irq(&gic.lock);
> >  }
> >  
> >  void gic_route_irqs(void)
> > @@ -439,7 +439,7 @@ void gic_set_guest_irq(unsigned int virtual_irq,
> >  
> >      events_maintenance(current);
> >  
> > -    spin_lock(&gic.lock);
> > +    spin_lock_irq(&gic.lock);
> >  
> >      if ( list_empty(&gic.lr_pending) )
> >      {
> > @@ -465,7 +465,7 @@ void gic_set_guest_irq(unsigned int virtual_irq,
> >      list_add_tail(&n->lr_queue, &gic.lr_pending);
> >  
> >  out:
> > -    spin_unlock(&gic.lock);
> > +    spin_unlock_irq(&gic.lock);
> >      return;
> >  }
> >  
> > @@ -559,7 +559,7 @@ static void events_maintenance(struct vcpu *v)
> >              (unsigned long *)&vcpu_info(v, evtchn_upcall_pending));
> >  
> >      if (!already_pending && gic.event_mask != 0) {
> > -        spin_lock(&gic.lock);
> > +        spin_lock_irq(&gic.lock);
> >          while ((i = find_next_bit((const long unsigned int *) &gic.event_mask,
> >                          sizeof(uint64_t), i)) < sizeof(uint64_t)) {
> >  
> > @@ -569,7 +569,7 @@ static void events_maintenance(struct vcpu *v)
> >  
> >              i++;
> >          }
> > -        spin_unlock(&gic.lock);
> > +        spin_unlock_irq(&gic.lock);
> >      }
> >  }
> >  
> > @@ -585,7 +585,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
> >                                sizeof(eisr), i)) < sizeof(eisr)) {
> >          struct pending_irq *p;
> >  
> > -        spin_lock(&gic.lock);
> > +        spin_lock_irq(&gic.lock);
> >          lr = GICH[GICH_LR + i];
> >          virq = lr & GICH_LR_VIRTUAL_MASK;
> >          GICH[GICH_LR + i] = 0;
> > @@ -601,7 +601,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
> >          } else {
> >              gic_inject_irq_stop();
> >          }
> > -        spin_unlock(&gic.lock);
> > +        spin_unlock_irq(&gic.lock);
> >  
> >          spin_lock(&current->arch.vgic.lock);
>                ^
> shouldn't you change this into spin_lock_irq too?


If so then that should be in "arm: use interrupt safe spin locks in
vgic_vcpu_inject_irq" rather than here?

I think you've reworked this stuff a bit in one of your follow up series
-- is it worth me changing this here or do you handle it / make it
irrelevant?

Ian.

Re: [PATCH 28/38] arm: map GICV in all domains, not just dom0.

[Ian Campbell]

On Fri, 2012-06-01 at 16:39 +0100, Ian Campbell wrote:
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 339c327..a398f92 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -541,14 +541,13 @@ void gic_interrupt(struct cpu_user_regs *regs,
> int is_fiq)
>      do_IRQ(regs, irq, is_fiq);
>  }
>  
> -void gicv_setup(struct domain *d)
> +int gicv_setup(struct domain *d)
>  {
> +    printk("GICV setup for DOM%d\n", d->domain_id);

This print was a bit verbose/pointless -- I've removed it.

Ian.

Re: [PATCH 31/38] arm: gic.lock can be taken in interrupt context, so lock appropriately.

[Stefano Stabellini]

On Tue, 26 Jun 2012, Ian Campbell wrote:
> > > @@ -601,7 +601,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
> > >          } else {
> > >              gic_inject_irq_stop();
> > >          }
> > > -        spin_unlock(&gic.lock);
> > > +        spin_unlock_irq(&gic.lock);
> > >  
> > >          spin_lock(&current->arch.vgic.lock);
> >                ^
> > shouldn't you change this into spin_lock_irq too?
> 
> 
> If so then that should be in "arm: use interrupt safe spin locks in
> vgic_vcpu_inject_irq" rather than here?
> 
> I think you've reworked this stuff a bit in one of your follow up series
> -- is it worth me changing this here or do you handle it / make it
> irrelevant?
 
No, I am not, I am only removing everything related to
events_maintenance. I think it is worth fixing it in one of your patches
or in a follow up patch.

Re: [PATCH 31/38] arm: gic.lock can be taken in interrupt context, so lock appropriately.

[Ian Campbell]

On Tue, 2012-06-26 at 12:41 +0100, Stefano Stabellini wrote:
> On Tue, 26 Jun 2012, Ian Campbell wrote:
> > > > @@ -601,7 +601,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
> > > >          } else {
> > > >              gic_inject_irq_stop();
> > > >          }
> > > > -        spin_unlock(&gic.lock);
> > > > +        spin_unlock_irq(&gic.lock);
> > > >  
> > > >          spin_lock(&current->arch.vgic.lock);
> > >                ^
> > > shouldn't you change this into spin_lock_irq too?
> > 
> > 
> > If so then that should be in "arm: use interrupt safe spin locks in
> > vgic_vcpu_inject_irq" rather than here?
> > 
> > I think you've reworked this stuff a bit in one of your follow up series
> > -- is it worth me changing this here or do you handle it / make it
> > irrelevant?
>  
> No, I am not, I am only removing everything related to
> events_maintenance. I think it is worth fixing it in one of your patches
> or in a follow up patch.

OK, I'll do it in another patch.

Ian.

Re: [PATCH 31/38] arm: gic.lock can be taken in interrupt context, so lock appropriately.

[Ian Campbell]

On Tue, 2012-06-26 at 13:25 +0100, Ian Campbell wrote:
> On Tue, 2012-06-26 at 12:41 +0100, Stefano Stabellini wrote:
> > On Tue, 26 Jun 2012, Ian Campbell wrote:
> > > > > @@ -601,7 +601,7 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
> > > > >          } else {
> > > > >              gic_inject_irq_stop();
> > > > >          }
> > > > > -        spin_unlock(&gic.lock);
> > > > > +        spin_unlock_irq(&gic.lock);
> > > > >  
> > > > >          spin_lock(&current->arch.vgic.lock);
> > > >                ^
> > > > shouldn't you change this into spin_lock_irq too?
> > > 
> > > 
> > > If so then that should be in "arm: use interrupt safe spin locks in
> > > vgic_vcpu_inject_irq" rather than here?
> > > 
> > > I think you've reworked this stuff a bit in one of your follow up series
> > > -- is it worth me changing this here or do you handle it / make it
> > > irrelevant?
> >  
> > No, I am not, I am only removing everything related to
> > events_maintenance. I think it is worth fixing it in one of your patches
> > or in a follow up patch.
> 
> OK, I'll do it in another patch.

Actually, this should have been in "arm: use interrupt safe spin locks
in vgic_vcpu_inject_irq" after all. I also noticed a bug in that patch
(using spin_unlock_irq not irqrestore in one exit path from
vgic_vcpu_inject_irq).

The new combined patch is (this is against my v2 series and replaces
"arm: use interrupt safe spin locks in vgic_vcpu_inject_irq" there):

8<--------------------------------------------------------------

>From 33832172c77d72cd87505f4926f380ee800ff0c7 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Tue, 26 Jun 2012 09:00:55 +0000
Subject: [PATCH] arm: make vgic lock safe for use in interrupt context.

In particular vgic_vcpu_inject_irq can be called in both interupt and regular
context.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/gic.c  |    4 ++--
 xen/arch/arm/vgic.c |   11 ++++++-----
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 339c327..1e60d3b 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -604,14 +604,14 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
         }
         spin_unlock(&gic.lock);
 
-        spin_lock(&current->arch.vgic.lock);
+        spin_lock_irq(&current->arch.vgic.lock);
         p = irq_to_pending(current, virq);
         if ( p->desc != NULL ) {
             p->desc->status &= ~IRQ_INPROGRESS;
             GICC[GICC_DIR] = virq;
         }
         list_del_init(&p->inflight);
-        spin_unlock(&current->arch.vgic.lock);
+        spin_unlock_irq(&current->arch.vgic.lock);
 
         i++;
     }
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index af3523f..a217151 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -114,8 +114,8 @@ int vcpu_vgic_init(struct vcpu *v)
     return 0;
 }
 
-#define vgic_lock(v)   spin_lock(&(v)->domain->arch.vgic.lock)
-#define vgic_unlock(v) spin_unlock(&(v)->domain->arch.vgic.lock)
+#define vgic_lock(v)   spin_lock_irq(&(v)->domain->arch.vgic.lock)
+#define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
 
 #define vgic_lock_rank(v, r) spin_lock(&(r)->lock)
 #define vgic_unlock_rank(v, r) spin_unlock(&(r)->lock)
@@ -550,6 +550,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
     uint8_t priority;
     struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
     struct pending_irq *iter, *n = irq_to_pending(v, irq);
+    unsigned long flags;
 
     /* irq still pending */
     if (!list_empty(&n->inflight))
@@ -566,18 +567,18 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq, int virtual)
 
     gic_set_guest_irq(irq, GICH_LR_PENDING, priority);
 
-    spin_lock(&v->arch.vgic.lock);
+    spin_lock_irqsave(&v->arch.vgic.lock, flags);
     list_for_each_entry ( iter, &v->arch.vgic.inflight_irqs, inflight )
     {
         if ( iter->priority > priority )
         {
             list_add_tail(&n->inflight, &iter->inflight);
-            spin_unlock(&v->arch.vgic.lock);
+            spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
             return;
         }
     }
     list_add_tail(&n->inflight, &v->arch.vgic.inflight_irqs);
-    spin_unlock(&v->arch.vgic.lock);
+    spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
     /* we have a new higher priority irq, inject it into the guest */
 }
 
-- 
1.7.9.1

Re: [PATCH 36/38] libxc: add ARM support to xc_dom (PV domain building)

[Ian Campbell]

On Thu, 2012-06-07 at 12:38 +0100, Stefano Stabellini wrote:
> > @@ -294,7 +295,7 @@ static inline xen_pfn_t xc_dom_p2m_guest(struct xc_dom_image *dom,
> >  {
> >      if (xc_dom_feature_translated(dom))
> >          return pfn;
> > -    return dom->p2m_host[pfn];
> > +    return dom->p2m_host[pfn - dom->rambase_pfn];
> >  }
> 
> I take that rambase_pfn is the offset in the guest physical address
> space where the ram is located. It would be nice to write it down.

I've added this comment to the struct where rambase_pfn is defined:
     * A PV guest has a single contiguous block of physical RAM,
     * consisting of total_pages starting at rambase_pfn.

[...]

> > +    /* setup initial p2m */
> > +    for ( pfn = 0; pfn < dom->total_pages; pfn++ )
> > +        dom->p2m_host[pfn] = pfn + dom->rambase_pfn;
> 
> uhm.. so maybe rambase_pfn is the offset in the machine address space where
> the guest ram has been allocated?

ARM guests are hybrid so there isn't really a distinction between MFN
and PFN space as seen by the tools. We always deal in guest PFNs in the
tools (only the hypervisor can see the MFNs).

This is a bit confusing for a hybrid guest because the generic xc_dom_*
stuff expects to have a p2m, so we setup this weird 1:1 thing (actually
1:rambase_pfn+1 just to add to the confusion).

I've added a comment next to the definition of p2m_host:
    /*
     * p2m_host maps guest physical addresses an offset from
     * rambase_pfn (see below) into gfns.
     *
     * For a pure PV guest this means that it maps GPFNs into MFNs for
     * a hybrid guest this means that it maps GPFNs to GPFNS.
     *
     * Note that the input is offset by rambase.
     */

I'm not sure this doesn't just add to the confusion.

Also I fully expect Tim to complain that I've got my *FN terminology all
wrong :-P.

> > diff --git a/tools/libxc/xc_dom_armzimageloader.c b/tools/libxc/xc_dom_armzimageloader.c
> > new file mode 100644
> > index 0000000..220176d
> > --- /dev/null
> > +++ b/tools/libxc/xc_dom_armzimageloader.c
[...]
> > +#include "xg_private.h"
> > +#include "xc_dom.h"
> > +
> > +#include <arpa/inet.h> /* XXX ntohl is not the right function... */
> 
> Yes, you are right, we should write a be32_to_cpu (the ones in Xen, QEMU
> and Linux are GPLv2 rather than LGLPv2).

I wonder if we can/should just declare that we handle only native endian
zImages.

> > diff --git a/tools/libxc/xc_dom_core.c b/tools/libxc/xc_dom_core.c
> > index fea9de5..b0d48d5 100644
> > --- a/tools/libxc/xc_dom_core.c
> > +++ b/tools/libxc/xc_dom_core.c
> > @@ -307,15 +307,17 @@ void *xc_dom_pfn_to_ptr(struct xc_dom_image *dom, xen_pfn_t pfn,
> >                          xen_pfn_t count)
> >  {
> >      struct xc_dom_phys *phys;
> > +    xen_pfn_t offset;
> >      unsigned int page_shift = XC_DOM_PAGE_SHIFT(dom);
> >      char *mode = "unset";
> >
> > -    if ( pfn > dom->total_pages ||    /* multiple checks to avoid overflows */
> > +    offset = pfn-dom->rambase_pfn;
> 
> spaces around the '-' please

Done.

> > +    if ( offset > dom->total_pages ||    /* multiple checks to avoid overflows */
> >           count > dom->total_pages ||
> > -         pfn > dom->total_pages - count )
> > +         offset > dom->total_pages - count )
> >      {
> > -        DOMPRINTF("%s: pfn out of range (0x%" PRIpfn " > 0x%" PRIpfn ")",
> > -                  __FUNCTION__, pfn, dom->total_pages);
> > +        DOMPRINTF("%s: pfn %"PRI_xen_pfn" out of range (0x%" PRIpfn " > 0x%" PRIpfn ")",
> > +                  __FUNCTION__, pfn, offset, dom->total_pages);
> >          return NULL;
> >      }
> >
> > @@ -599,6 +601,8 @@ struct xc_dom_image *xc_dom_allocate(xc_interface *xch,
> >      dom->parms.virt_hv_start_low = UNSET_ADDR;
> >      dom->parms.elf_paddr_offset = UNSET_ADDR;
> >
> > +    dom->rambase_pfn = 0;
> > +
> >      dom->alloc_malloc += sizeof(*dom);
> >      return dom;
> 
> There is no need to explicitly set rambase_pfn to 0, because the whole
> dom struct is memset to 0 few lines above.

Removed.

Re: [PATCH 36/38] libxc: add ARM support to xc_dom (PV domain building)

[Ian Campbell]

> > > diff --git a/tools/libxc/xc_dom_armzimageloader.c b/tools/libxc/xc_dom_armzimageloader.c
> > > new file mode 100644
> > > index 0000000..220176d
> > > --- /dev/null
> > > +++ b/tools/libxc/xc_dom_armzimageloader.c
> [...]
> > > +#include "xg_private.h"
> > > +#include "xc_dom.h"
> > > +
> > > +#include <arpa/inet.h> /* XXX ntohl is not the right function... */
> > 
> > Yes, you are right, we should write a be32_to_cpu (the ones in Xen, QEMU
> > and Linux are GPLv2 rather than LGLPv2).
> 
> I wonder if we can/should just declare that we handle only native endian
> zImages.

Except this is used with the DTB and that is what it is. :-(

We have ./tools/blktap2/drivers/bswap.h already which maybe we can make
more generic?

TBH, given this is internal to this one loader I'm very tempted to just
keep using ntohl and friends.

Ian

Re: [PATCH 36/38] libxc: add ARM support to xc_dom (PV domain building)

[David Vrabel]

On 23/07/12 15:11, Ian Campbell wrote:
> 
>>>> diff --git a/tools/libxc/xc_dom_armzimageloader.c b/tools/libxc/xc_dom_armzimageloader.c
>>>> new file mode 100644
>>>> index 0000000..220176d
>>>> --- /dev/null
>>>> +++ b/tools/libxc/xc_dom_armzimageloader.c
>> [...]
>>>> +#include "xg_private.h"
>>>> +#include "xc_dom.h"
>>>> +
>>>> +#include <arpa/inet.h> /* XXX ntohl is not the right function... */
>>>
>>> Yes, you are right, we should write a be32_to_cpu (the ones in Xen, QEMU
>>> and Linux are GPLv2 rather than LGLPv2).
>>
>> I wonder if we can/should just declare that we handle only native endian
>> zImages.
> 
> Except this is used with the DTB and that is what it is. :-(
> 
> We have ./tools/blktap2/drivers/bswap.h already which maybe we can make
> more generic?
> 
> TBH, given this is internal to this one loader I'm very tempted to just
> keep using ntohl and friends.

You will likely need to link to libfdt in the future and you will need
to provide cpu_to_fdt32() and fdt32_to_cpu() etc. implementations (like
include/xen/libfdt/libfdt_env.h for Xen itself).

David

Re: [PATCH 36/38] libxc: add ARM support to xc_dom (PV domain building)

[Ian Campbell]

On Mon, 2012-07-23 at 15:37 +0100, David Vrabel wrote:
> You will likely need to link to libfdt in the future and you will need
> to provide cpu_to_fdt32() and fdt32_to_cpu() etc. implementations (like
> include/xen/libfdt/libfdt_env.h for Xen itself).

Good point.

I'm now more tempted to defer this until then ;-)