[PATCH v5 00/10] xen/arm: Handle correctly foreign mapping

[PATCH v5 00/10] xen/arm: Handle correctly foreign mapping

[Julien Grall]

Hello,

This patch series aims to fix "Failed to unmap" message in dom0 when a guest is
creating. Without this patch series, dom0 will leak memory each time a domain
is created. It should be considered as a blocker for Xen 4.4 release.

Compare to the previous version, I have removed all specific modification
in common code (ie p2m foreing check).

    - Patch #1-2: prepare work for the others patches
    - Patch #3-6: add support for p2m type
    - Patch #7-9: handle correctly foreign mapping.
    - Patch #10: it's not really part of this series. It adds support
        for read-only grant-mapping

For all the changes, see in each patch.

Sincerely yours,
Release-acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Mukesh Rathor <mukesh.rathor@oracle.com>

Julien Grall (10):
  xen/arm: Introduce steps in domain_relinquish_resource
  xen/arm: move mfn_to_p2m_entry in arch/arm/p2m.c
  xen/arm: Implement p2m_type_t as an enum
  xen/arm: Store p2m type in each page of the guest
  xen/arm: p2m: Extend p2m_lookup parameters to retrieve the p2m type
  xen/arm: Retrieve p2m type in get_page_from_gfn
  xen/arm: Handle remove foreign mapping
  xen/arm: Add relinquish_p2m_mapping to remove reference on every
    mapped page
  xen/arm: Set foreign page type to p2m_map_foreign
  xen/arm: grant-table: Support read-only mapping

 xen/arch/arm/domain.c        |   45 +++++++++++--
 xen/arch/arm/mm.c            |   50 +++++++++-----
 xen/arch/arm/p2m.c           |  148 +++++++++++++++++++++++++++++++++++++-----
 xen/arch/arm/traps.c         |    6 +-
 xen/include/asm-arm/domain.h |    9 +++
 xen/include/asm-arm/p2m.h    |   80 ++++++++++++++++++++---
 xen/include/asm-arm/page.h   |   24 +------
 7 files changed, 287 insertions(+), 75 deletions(-)

-- 
1.7.10.4

[PATCH v5 01/10] xen/arm: Introduce steps in domain_relinquish_resource

[Julien Grall]

In a later patch, a new step will be added. It will avoid to check every step
when the function was preempted.

Signed-off-by: Julien Grall <julien.grall@linaro.org>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

---
    Changes in v2:
        - Introduce the patch
---
 xen/arch/arm/domain.c        |   37 ++++++++++++++++++++++++++++++-------
 xen/include/asm-arm/domain.h |    8 ++++++++
 2 files changed, 38 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 67c65c3..1590708 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -497,6 +497,8 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
 {
     int rc;
 
+    d->arch.relmem = RELMEM_not_started;
+
     /* Idle domains do not need this setup */
     if ( is_idle_domain(d) )
         return 0;
@@ -696,15 +698,36 @@ int domain_relinquish_resources(struct domain *d)
 {
     int ret = 0;
 
-    ret = relinquish_memory(d, &d->xenpage_list);
-    if ( ret )
-        return ret;
+    switch ( d->arch.relmem )
+    {
+    case RELMEM_not_started:
+        d->arch.relmem = RELMEM_xen;
+        /* Falltrough */
 
-    ret = relinquish_memory(d, &d->page_list);
-    if ( ret )
-        return ret;
+    case RELMEM_xen:
+        ret = relinquish_memory(d, &d->xenpage_list);
+        if ( ret )
+            return ret;
 
-    return ret;
+        d->arch.relmem = RELMEM_page;
+        /* Fallthrough */
+
+    case RELMEM_page:
+        ret = relinquish_memory(d, &d->page_list);
+        if ( ret )
+            return ret;
+
+        d->arch.relmem = RELMEM_done;
+        /* Fallthrough */
+
+    case RELMEM_done:
+        break;
+
+    default:
+        BUG();
+    }
+
+    return 0;
 }
 
 void arch_dump_domain_info(struct domain *d)
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 0733c5e..53c9895 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -107,6 +107,14 @@ struct arch_domain
     struct hvm_domain hvm_domain;
     xen_pfn_t *grant_table_gpfn;
 
+    /* Continuable domain_relinquish_resources(). */
+    enum {
+        RELMEM_not_started,
+        RELMEM_xen,
+        RELMEM_page,
+        RELMEM_done,
+    } relmem;
+
     /* Virtual CPUID */
     uint32_t vpidr;
 
-- 
1.7.10.4

[PATCH v5 02/10] xen/arm: move mfn_to_p2m_entry in arch/arm/p2m.c

[Julien Grall]

The function mfn_to_p2m_entry will be extended in a following patch to handle
p2m_type_t. It will break compilation because p2m_type_t is not defined
(interdependence between includes).
It's easier to move the function in arch/arm/p2m.c and it's not harmful as the
function is only used in this file.

Signed-off-by: Julien Grall <julien.grall@linaro.org>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c         |   22 ++++++++++++++++++++++
 xen/include/asm-arm/page.h |   22 ----------------------
 2 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 083f8bf..74636df 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -128,6 +128,28 @@ int p2m_pod_decrease_reservation(struct domain *d,
     return -ENOSYS;
 }
 
+static lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
+{
+    paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
+    lpae_t e = (lpae_t) {
+        .p2m.xn = 0,
+        .p2m.af = 1,
+        .p2m.sh = LPAE_SH_OUTER,
+        .p2m.read = 1,
+        .p2m.write = 1,
+        .p2m.mattr = mattr,
+        .p2m.table = 1,
+        .p2m.valid = 1,
+    };
+
+    ASSERT(!(pa & ~PAGE_MASK));
+    ASSERT(!(pa & ~PADDR_MASK));
+
+    e.bits |= pa;
+
+    return e;
+}
+
 /* Allocate a new page table page and hook it in via the given entry */
 static int p2m_create_table(struct domain *d,
                             lpae_t *entry)
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index d468418..0625464 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -213,28 +213,6 @@ static inline lpae_t mfn_to_xen_entry(unsigned long mfn)
     return e;
 }
 
-static inline lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
-{
-    paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
-    lpae_t e = (lpae_t) {
-        .p2m.xn = 0,
-        .p2m.af = 1,
-        .p2m.sh = LPAE_SH_OUTER,
-        .p2m.write = 1,
-        .p2m.read = 1,
-        .p2m.mattr = mattr,
-        .p2m.table = 1,
-        .p2m.valid = 1,
-    };
-
-    ASSERT(!(pa & ~PAGE_MASK));
-    ASSERT(!(pa & ~PADDR_MASK));
-
-    e.bits |= pa;
-
-    return e;
-}
-
 #if defined(CONFIG_ARM_32)
 # include <asm/arm32/page.h>
 #elif defined(CONFIG_ARM_64)
-- 
1.7.10.4

[PATCH v5 03/10] xen/arm: Implement p2m_type_t as an enum

[Julien Grall]

Until now, Xen doesn't know the type of the page (ram, foreign page, mmio,...).
Introduce p2m_type_t with basic types:
    - p2m_invalid: Nothing is mapped here
    - p2m_ram_rw: Normal read/write guest RAM
    - p2m_ram_ro: Read-only guest RAM
    - p2m_mmio_direct: Read/write mapping of device memory
    - p2m_map_foreign: RAM page from foreign guest
    - p2m_grant_map_rw: Read/write grant mapping
    - p2m_grant_map_ro: Read-only grant mapping

Signed-off-by: Julien Grall <julien.grall@linaro.org>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

---
    Changes in v3:
        - s/  / /
        - Replace p2m by either pte or p2m entry
        - Fix compilation (unitialized value)
        - Add BUILD_BUG_ON (from patch #4) and fix it
    Changes in v2:
        - Add comment for future improvement
        - Add p2m_max_real_type. Will be use later to check the size of
        the enum
        - Let the compiler choose the value for each name of the enum
        - Add grant mapping type
---
 xen/arch/arm/p2m.c        |    2 ++
 xen/include/asm-arm/p2m.h |   20 +++++++++++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 74636df..691cdfa 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -142,6 +142,8 @@ static lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
         .p2m.valid = 1,
     };
 
+    BUILD_BUG_ON(p2m_max_real_type > (1 << 4));
+
     ASSERT(!(pa & ~PAGE_MASK));
     ASSERT(!(pa & ~PADDR_MASK));
 
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index c660820..bc86a49 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -20,6 +20,25 @@ struct p2m_domain {
     uint8_t vmid;
 };
 
+/* List of possible type for each page in the p2m entry.
+ * The number of available bit per page in the pte for this purpose is 4 bits.
+ * So it's possible to only have 16 fields. If we run out of value in the
+ * future, it's possible to use higher value for pseudo-type and don't store
+ * them in the p2m entry.
+ */
+typedef enum {
+    p2m_invalid = 0,    /* Nothing mapped here */
+    p2m_ram_rw,         /* Normal read/write guest RAM */
+    p2m_ram_ro,         /* Read-only; writes are silently dropped */
+    p2m_mmio_direct,    /* Read/write mapping of genuine MMIO area */
+    p2m_map_foreign,    /* Ram pages from foreign domain */
+    p2m_grant_map_rw,   /* Read/write grant mapping */
+    p2m_grant_map_ro,   /* Read-only grant mapping */
+    p2m_max_real_type,  /* Types after this won't be store in the p2m */
+} p2m_type_t;
+
+#define p2m_is_foreign(_t)  ((_t) == p2m_map_foreign)
+
 /* Initialise vmid allocator */
 void p2m_vmid_allocator_init(void);
 
@@ -72,7 +91,6 @@ p2m_pod_decrease_reservation(struct domain *d,
                              unsigned int order);
 
 /* Look up a GFN and take a reference count on the backing page. */
-typedef int p2m_type_t;
 typedef unsigned int p2m_query_t;
 #define P2M_ALLOC    (1u<<0)   /* Populate PoD and paged-out entries */
 #define P2M_UNSHARE  (1u<<1)   /* Break CoW sharing */
-- 
1.7.10.4

[PATCH v5 04/10] xen/arm: Store p2m type in each page of the guest

[Julien Grall]

Use the field 'avail' to store the type of the page. Rename it to 'type' for
convenience.
The information stored in this field will be retrieved in a future patch to
change the behaviour when the page is removed.

Also introduce guest_physmap_add_entry to map and set a specific p2m type for
a page.

Signed-off-by: Julien Grall <julien.grall@linaro.org>

---
    Changes in v5:
        - Foreign mapping doesn't need to have execution right
    Changes in v3:
        - Typo in the commit message
        - Rename 'p2mt' field to 'type'
        - Remove default in switch to let the compiler warns
        - Move BUILD_BUG_ON to patch #3
    Changes in v2:
        - Rename 'avail' field to 'p2mt' in the p2m structure
        - Add BUILD_BUG_ON to check if the enum value will fit in the field
        - Implement grant mapping type
---
 xen/arch/arm/p2m.c         |   56 ++++++++++++++++++++++++++++++++------------
 xen/include/asm-arm/p2m.h  |   18 ++++++++++----
 xen/include/asm-arm/page.h |    2 +-
 3 files changed, 56 insertions(+), 20 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 691cdfa..17d6620 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -128,7 +128,8 @@ int p2m_pod_decrease_reservation(struct domain *d,
     return -ENOSYS;
 }
 
-static lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
+static lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr,
+                               p2m_type_t t)
 {
     paddr_t pa = ((paddr_t) mfn) << PAGE_SHIFT;
     lpae_t e = (lpae_t) {
@@ -136,14 +137,34 @@ static lpae_t mfn_to_p2m_entry(unsigned long mfn, unsigned int mattr)
         .p2m.af = 1,
         .p2m.sh = LPAE_SH_OUTER,
         .p2m.read = 1,
-        .p2m.write = 1,
         .p2m.mattr = mattr,
         .p2m.table = 1,
         .p2m.valid = 1,
+        .p2m.type = t,
     };
 
     BUILD_BUG_ON(p2m_max_real_type > (1 << 4));
 
+    switch (t)
+    {
+    case p2m_map_foreign:
+    case p2m_grant_map_rw:
+        e.p2m.xn = 1;
+        /* Fallthrough */
+    case p2m_ram_rw:
+    case p2m_mmio_direct:
+        e.p2m.write = 1;
+        break;
+
+    case p2m_grant_map_ro:
+        e.p2m.xn = 1;
+        /* Fallthrough */
+    case p2m_invalid:
+    case p2m_ram_ro:
+    default:
+        e.p2m.write = 0;
+    }
+
     ASSERT(!(pa & ~PAGE_MASK));
     ASSERT(!(pa & ~PADDR_MASK));
 
@@ -173,7 +194,7 @@ static int p2m_create_table(struct domain *d,
     clear_page(p);
     unmap_domain_page(p);
 
-    pte = mfn_to_p2m_entry(page_to_mfn(page), MATTR_MEM);
+    pte = mfn_to_p2m_entry(page_to_mfn(page), MATTR_MEM, p2m_invalid);
 
     write_pte(entry, pte);
 
@@ -191,7 +212,8 @@ static int create_p2m_entries(struct domain *d,
                      paddr_t start_gpaddr,
                      paddr_t end_gpaddr,
                      paddr_t maddr,
-                     int mattr)
+                     int mattr,
+                     p2m_type_t t)
 {
     int rc, flush;
     struct p2m_domain *p2m = &d->arch.p2m;
@@ -271,14 +293,15 @@ static int create_p2m_entries(struct domain *d,
                         goto out;
                     }
 
-                    pte = mfn_to_p2m_entry(page_to_mfn(page), mattr);
+                    pte = mfn_to_p2m_entry(page_to_mfn(page), mattr, t);
 
                     write_pte(&third[third_table_offset(addr)], pte);
                 }
                 break;
             case INSERT:
                 {
-                    lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT, mattr);
+                    lpae_t pte = mfn_to_p2m_entry(maddr >> PAGE_SHIFT,
+                                                  mattr, t);
                     write_pte(&third[third_table_offset(addr)], pte);
                     maddr += PAGE_SIZE;
                 }
@@ -313,7 +336,8 @@ int p2m_populate_ram(struct domain *d,
                      paddr_t start,
                      paddr_t end)
 {
-    return create_p2m_entries(d, ALLOCATE, start, end, 0, MATTR_MEM);
+    return create_p2m_entries(d, ALLOCATE, start, end,
+                              0, MATTR_MEM, p2m_ram_rw);
 }
 
 int map_mmio_regions(struct domain *d,
@@ -321,18 +345,20 @@ int map_mmio_regions(struct domain *d,
                      paddr_t end_gaddr,
                      paddr_t maddr)
 {
-    return create_p2m_entries(d, INSERT, start_gaddr, end_gaddr, maddr, MATTR_DEV);
+    return create_p2m_entries(d, INSERT, start_gaddr, end_gaddr,
+                              maddr, MATTR_DEV, p2m_mmio_direct);
 }
 
-int guest_physmap_add_page(struct domain *d,
-                           unsigned long gpfn,
-                           unsigned long mfn,
-                           unsigned int page_order)
+int guest_physmap_add_entry(struct domain *d,
+                            unsigned long gpfn,
+                            unsigned long mfn,
+                            unsigned long page_order,
+                            p2m_type_t t)
 {
     return create_p2m_entries(d, INSERT,
                               pfn_to_paddr(gpfn),
-                              pfn_to_paddr(gpfn + (1<<page_order)),
-                              pfn_to_paddr(mfn), MATTR_MEM);
+                              pfn_to_paddr(gpfn + (1 << page_order)),
+                              pfn_to_paddr(mfn), MATTR_MEM, t);
 }
 
 void guest_physmap_remove_page(struct domain *d,
@@ -342,7 +368,7 @@ void guest_physmap_remove_page(struct domain *d,
     create_p2m_entries(d, REMOVE,
                        pfn_to_paddr(gpfn),
                        pfn_to_paddr(gpfn + (1<<page_order)),
-                       pfn_to_paddr(mfn), MATTR_MEM);
+                       pfn_to_paddr(mfn), MATTR_MEM, p2m_invalid);
 }
 
 int p2m_alloc_table(struct domain *d)
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index bc86a49..5742bbc 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -68,11 +68,21 @@ int p2m_populate_ram(struct domain *d, paddr_t start, paddr_t end);
 int map_mmio_regions(struct domain *d, paddr_t start_gaddr,
                      paddr_t end_gaddr, paddr_t maddr);
 
+int guest_physmap_add_entry(struct domain *d,
+                            unsigned long gfn,
+                            unsigned long mfn,
+                            unsigned long page_order,
+                            p2m_type_t t);
+
 /* Untyped version for RAM only, for compatibility */
-int guest_physmap_add_page(struct domain *d,
-                           unsigned long gfn,
-                           unsigned long mfn,
-                           unsigned int page_order);
+static inline int guest_physmap_add_page(struct domain *d,
+                                         unsigned long gfn,
+                                         unsigned long mfn,
+                                         unsigned int page_order)
+{
+    return guest_physmap_add_entry(d, gfn, mfn, page_order, p2m_ram_rw);
+}
+
 void guest_physmap_remove_page(struct domain *d,
                                unsigned long gpfn,
                                unsigned long mfn, unsigned int page_order);
diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
index 0625464..670d4e7 100644
--- a/xen/include/asm-arm/page.h
+++ b/xen/include/asm-arm/page.h
@@ -153,7 +153,7 @@ typedef struct {
     unsigned long contig:1;     /* In a block of 16 contiguous entries */
     unsigned long sbz2:1;
     unsigned long xn:1;         /* eXecute-Never */
-    unsigned long avail:4;      /* Ignored by hardware */
+    unsigned long type:4;       /* Ignore by hardware. Used to store p2m types */
 
     unsigned long sbz1:5;
 } __attribute__((__packed__)) lpae_p2m_t;
-- 
1.7.10.4

[PATCH v5 05/10] xen/arm: p2m: Extend p2m_lookup parameters to retrieve the p2m type

[Julien Grall]

Signed-off-by: Julien Grall <julien.grall@linaro.org>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

---
    Changes in v3:
        - 'p2mt' field was renamed in 'type'
        - missing one p2m_lookup
    Changes in v2:
        - This patch was "Add p2m_get_entry" on the previous version
        - Don't add a new function but only extend p2m_lookup
---
 xen/arch/arm/mm.c         |    2 +-
 xen/arch/arm/p2m.c        |   13 +++++++++++--
 xen/arch/arm/traps.c      |    6 +++---
 xen/include/asm-arm/p2m.h |    2 +-
 4 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 654281a..b1222be 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -1032,7 +1032,7 @@ static int xenmem_add_to_physmap_one(
             return rc;
         }
 
-        maddr = p2m_lookup(od, pfn_to_paddr(idx));
+        maddr = p2m_lookup(od, pfn_to_paddr(idx), NULL);
         if ( maddr == INVALID_PADDR )
         {
             dump_p2m_lookup(od, pfn_to_paddr(idx));
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 17d6620..9bdcacd 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -72,11 +72,17 @@ static lpae_t *p2m_map_first(struct p2m_domain *p2m, paddr_t addr)
  * There are no processor functions to do a stage 2 only lookup therefore we
  * do a a software walk.
  */
-paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
+paddr_t p2m_lookup(struct domain *d, paddr_t paddr, p2m_type_t *t)
 {
     struct p2m_domain *p2m = &d->arch.p2m;
     lpae_t pte, *first = NULL, *second = NULL, *third = NULL;
     paddr_t maddr = INVALID_PADDR;
+    p2m_type_t _t;
+
+    /* Allow t to be NULL */
+    t = t ?: &_t;
+
+    *t = p2m_invalid;
 
     spin_lock(&p2m->lock);
 
@@ -102,7 +108,10 @@ paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
 
 done:
     if ( pte.p2m.valid )
+    {
         maddr = (pte.bits & PADDR_MASK & PAGE_MASK) | (paddr & ~PAGE_MASK);
+        *t = pte.p2m.type;
+    }
 
     if (third) unmap_domain_page(third);
     if (second) unmap_domain_page(second);
@@ -511,7 +520,7 @@ err:
 
 unsigned long gmfn_to_mfn(struct domain *d, unsigned long gpfn)
 {
-    paddr_t p = p2m_lookup(d, pfn_to_paddr(gpfn));
+    paddr_t p = p2m_lookup(d, pfn_to_paddr(gpfn), NULL);
     return p >> PAGE_SHIFT;
 }
 
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index e01045e..7c5ab19 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -1421,7 +1421,7 @@ void dump_guest_s1_walk(struct domain *d, vaddr_t addr)
     printk("dom%d VA 0x%08"PRIvaddr"\n", d->domain_id, addr);
     printk("    TTBCR: 0x%08"PRIregister"\n", ttbcr);
     printk("    TTBR0: 0x%016"PRIx64" = 0x%"PRIpaddr"\n",
-           ttbr0, p2m_lookup(d, ttbr0 & PAGE_MASK));
+           ttbr0, p2m_lookup(d, ttbr0 & PAGE_MASK, NULL));
 
     if ( ttbcr & TTBCR_EAE )
     {
@@ -1434,7 +1434,7 @@ void dump_guest_s1_walk(struct domain *d, vaddr_t addr)
         return;
     }
 
-    paddr = p2m_lookup(d, ttbr0 & PAGE_MASK);
+    paddr = p2m_lookup(d, ttbr0 & PAGE_MASK, NULL);
     if ( paddr == INVALID_PADDR )
     {
         printk("Failed TTBR0 maddr lookup\n");
@@ -1449,7 +1449,7 @@ void dump_guest_s1_walk(struct domain *d, vaddr_t addr)
          !(first[offset] & 0x2) )
         goto done;
 
-    paddr = p2m_lookup(d, first[offset] & PAGE_MASK);
+    paddr = p2m_lookup(d, first[offset] & PAGE_MASK, NULL);
 
     if ( paddr == INVALID_PADDR )
     {
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 5742bbc..9d0d1e7 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -58,7 +58,7 @@ int p2m_alloc_table(struct domain *d);
 void p2m_load_VTTBR(struct domain *d);
 
 /* Look up the MFN corresponding to a domain's PFN. */
-paddr_t p2m_lookup(struct domain *d, paddr_t gpfn);
+paddr_t p2m_lookup(struct domain *d, paddr_t gpfn, p2m_type_t *t);
 
 /* Setup p2m RAM mapping for domain d from start-end. */
 int p2m_populate_ram(struct domain *d, paddr_t start, paddr_t end);
-- 
1.7.10.4

[PATCH v5 06/10] xen/arm: Retrieve p2m type in get_page_from_gfn

[Julien Grall]

Signed-off-by: Julien Grall <julien.grall@linaro.org>

---
    Changes in v3:
        - Return NULL when p2m type is invalid or mmio
    Changes in v2:
        - Use p2m_lookup as p2m_get_entry was removed
---
 xen/include/asm-arm/p2m.h |   12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 9d0d1e7..0eb07a8 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -109,11 +109,17 @@ static inline struct page_info *get_page_from_gfn(
     struct domain *d, unsigned long gfn, p2m_type_t *t, p2m_query_t q)
 {
     struct page_info *page;
-    unsigned long mfn = gmfn_to_mfn(d, gfn);
+    p2m_type_t p2mt;
+    paddr_t maddr = p2m_lookup(d, pfn_to_paddr(gfn), &p2mt);
+    unsigned long mfn = maddr >> PAGE_SHIFT;
 
-    ASSERT(t == NULL);
+    if (t)
+        *t = p2mt;
 
-    if (!mfn_valid(mfn))
+    if ( p2mt == p2m_invalid || p2mt == p2m_mmio_direct )
+        return NULL;
+
+    if ( !mfn_valid(mfn) )
         return NULL;
     page = mfn_to_page(mfn);
     if ( !get_page(page, d) )
-- 
1.7.10.4

[PATCH v5 07/10] xen/arm: Handle remove foreign mapping

[Julien Grall]

Modify get_page_from_gfn to take reference on foreign mapping. This will avoid
specific handling in the common code.

Signed-off-by: Julien Grall <julien.grall@linaro.org>

---
    Changes in v4.2:
        - get_page_from_gfn: move foreign checking before get_page
        - add assert fdom != dom
    Changes in v4.1:
        - Remove specific p2m handling in common code
        - Handle foreign mapping in get_page_from_gfn
    Changes in v4:
        - Split patch #6 from dom0 pvh series v6.2 to retrieve only common
        code.
        - Rework commit title
        - Rename xen_rem_foreign_from_p2m in p2m_remove_foreign
        - Get the mfn from the pte. We are not sure that maddr given in
        parameters is valid
    Changes in v3:
        - Move put_page in create_p2m_entries
        - Move xenmem_rem_foreign_from_p2m in arch/arm/p2m.c
    Changes in v2:
        - Introduce the patch
---
 xen/arch/arm/p2m.c        |   15 +++++++++++++--
 xen/include/asm-arm/p2m.h |   12 ++++++++++++
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 9bdcacd..d05fdff 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -317,10 +317,21 @@ static int create_p2m_entries(struct domain *d,
                 break;
             case REMOVE:
                 {
-                    lpae_t pte;
+                    lpae_t pte = third[third_table_offset(addr)];
+                    unsigned long mfn;
+
+                    maddr = (pte.bits & PADDR_MASK & PAGE_MASK);
+                    mfn = paddr_to_pfn(maddr);
+
+                    /* TODO: Handle other p2m type */
+                    if ( pte.p2m.valid && p2m_is_foreign(pte.p2m.type) )
+                    {
+                        ASSERT(mfn_valid(mfn));
+                        put_page(mfn_to_page(mfn));
+                    }
+
                     memset(&pte, 0x00, sizeof(pte));
                     write_pte(&third[third_table_offset(addr)], pte);
-                    maddr += PAGE_SIZE;
                 }
                 break;
         }
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 0eb07a8..5ccfa7f 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -122,6 +122,18 @@ static inline struct page_info *get_page_from_gfn(
     if ( !mfn_valid(mfn) )
         return NULL;
     page = mfn_to_page(mfn);
+
+    /* get_page won't work on foreign mapping because the page doesn't
+     * belong to the current domain.
+     */
+    if ( p2mt == p2m_map_foreign )
+    {
+        struct domain *fdom = page_get_owner_and_reference(page);
+        ASSERT(fdom != NULL);
+        ASSERT(fdom != d);
+        return page;
+    }
+
     if ( !get_page(page, d) )
         return NULL;
     return page;
-- 
1.7.10.4

[PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Julien Grall]

This function will be called when the domain relinquishes its memory.
It removes refcount on every mapped page to a valid MFN.

Currently, Xen doesn't take reference on every new mapping but only for foreign
mapping. Restrict the function only on foreign mapping.

Signed-off-by: Julien Grall <julien.grall@linaro.org>

---
    Changes in v4:
        - Use LPAE_ENTRIES instead of hardcoded value
    Changes in v3:
        - Rework title
        - Reuse create_p2m_entries to remove reference
        - Don't forget to set relmem!
        - Fix compilation (missing include)
    Changes in v2:
        - Introduce the patch
---
 xen/arch/arm/domain.c        |    8 ++++++++
 xen/arch/arm/p2m.c           |   44 +++++++++++++++++++++++++++++++++++++++++-
 xen/include/asm-arm/domain.h |    1 +
 xen/include/asm-arm/p2m.h    |   15 ++++++++++++++
 4 files changed, 67 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 1590708..4099e88 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -717,6 +717,14 @@ int domain_relinquish_resources(struct domain *d)
         if ( ret )
             return ret;
 
+        d->arch.relmem = RELMEM_mapping;
+        /* Fallthrough */
+
+    case RELMEM_mapping:
+        ret = relinquish_p2m_mapping(d);
+        if ( ret )
+            return ret;
+
         d->arch.relmem = RELMEM_done;
         /* Fallthrough */
 
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index d05fdff..45179f7 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -6,6 +6,8 @@
 #include <xen/bitops.h>
 #include <asm/flushtlb.h>
 #include <asm/gic.h>
+#include <asm/event.h>
+#include <asm/hardirq.h>
 
 /* First level P2M is 2 consecutive pages */
 #define P2M_FIRST_ORDER 1
@@ -213,7 +215,8 @@ static int p2m_create_table(struct domain *d,
 enum p2m_operation {
     INSERT,
     ALLOCATE,
-    REMOVE
+    REMOVE,
+    RELINQUISH,
 };
 
 static int create_p2m_entries(struct domain *d,
@@ -231,6 +234,7 @@ static int create_p2m_entries(struct domain *d,
     unsigned long cur_first_page = ~0,
                   cur_first_offset = ~0,
                   cur_second_offset = ~0;
+    unsigned long count = 0;
 
     spin_lock(&p2m->lock);
 
@@ -315,6 +319,7 @@ static int create_p2m_entries(struct domain *d,
                     maddr += PAGE_SIZE;
                 }
                 break;
+            case RELINQUISH:
             case REMOVE:
                 {
                     lpae_t pte = third[third_table_offset(addr)];
@@ -338,6 +343,29 @@ static int create_p2m_entries(struct domain *d,
 
         if ( flush )
             flush_tlb_all_local();
+
+
+        count++;
+
+        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
+             hypercall_preempt_check() )
+        {
+            p2m->next_gfn_to_relinquish = maddr >> PAGE_SHIFT;
+            rc = -EAGAIN;
+            goto out;
+        }
+    }
+
+    /* When the function will remove mapping, p2m type should always
+     * be p2m_invalid. */
+    if ( (t == p2m_ram_rw) || (t == p2m_ram_ro) || (t == p2m_map_foreign))
+    {
+        unsigned long sgfn = paddr_to_pfn(start_gpaddr);
+        unsigned long egfn = paddr_to_pfn(end_gpaddr);
+
+        p2m->max_mapped_gfn = MAX(p2m->max_mapped_gfn, egfn);
+        /* Use next_gfn_to_relinquish to store the lowest gfn mapped */
+        p2m->next_gfn_to_relinquish = MIN(p2m->next_gfn_to_relinquish, sgfn);
     }
 
     rc = 0;
@@ -523,12 +551,26 @@ int p2m_init(struct domain *d)
 
     p2m->first_level = NULL;
 
+    p2m->max_mapped_gfn = 0;
+    p2m->next_gfn_to_relinquish = ULONG_MAX;
+
 err:
     spin_unlock(&p2m->lock);
 
     return rc;
 }
 
+int relinquish_p2m_mapping(struct domain *d)
+{
+    struct p2m_domain *p2m = &d->arch.p2m;
+
+    return create_p2m_entries(d, RELINQUISH,
+                              pfn_to_paddr(p2m->next_gfn_to_relinquish),
+                              pfn_to_paddr(p2m->max_mapped_gfn),
+                              pfn_to_paddr(INVALID_MFN),
+                              MATTR_MEM, p2m_invalid);
+}
+
 unsigned long gmfn_to_mfn(struct domain *d, unsigned long gpfn)
 {
     paddr_t p = p2m_lookup(d, pfn_to_paddr(gpfn), NULL);
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 53c9895..28d39a0 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -112,6 +112,7 @@ struct arch_domain
         RELMEM_not_started,
         RELMEM_xen,
         RELMEM_page,
+        RELMEM_mapping,
         RELMEM_done,
     } relmem;
 
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 5ccfa7f..ac2b6fa 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -18,6 +18,15 @@ struct p2m_domain {
 
     /* Current VMID in use */
     uint8_t vmid;
+
+    /* Highest guest frame that's ever been mapped in the p2m
+     * Take only into account ram and foreign mapping
+     */
+    unsigned long max_mapped_gfn;
+
+    /* When releasing mapped gfn's in a preemptible manner, recall where
+     * to resume the search */
+    unsigned long next_gfn_to_relinquish;
 };
 
 /* List of possible type for each page in the p2m entry.
@@ -48,6 +57,12 @@ int p2m_init(struct domain *d);
 /* Return all the p2m resources to Xen. */
 void p2m_teardown(struct domain *d);
 
+/* Remove mapping refcount on each mapping page in the p2m
+ *
+ * TODO: For the moment only foreign mapping is handled
+ */
+int relinquish_p2m_mapping(struct domain *d);
+
 /* Allocate a new p2m table for a domain.
  *
  * Returns 0 for success or -errno.
-- 
1.7.10.4

[PATCH v5 09/10] xen/arm: Set foreign page type to p2m_map_foreign

[Julien Grall]

Xen needs to know that the current page belongs to another domain. Also take
a reference to this page.

The current process to add a foreign page is:
   1) get the page from the foreign p2m
   2) take a reference on the page with the foreign domain in parameters
   3) add the page to the current domain p2m

If the foreign domain drops the page:
    - before 2), get_page will return NULL because the page doesn't
    belong anymore to the domain
    - after 2), the current domain already have a reference. Write will
    occur to an old page which is not yet released. It can corrupt the foreign
    domain.

Signed-off-by: Julien Grall <julien.grall@linaro.org>

---
    Changes in v5:
        - Missing rcu_unlock_domain in one path
        - Check if the foreign page is RAM
    Changes in v4:
        - Typo s/release/released/
        - Improve commit message
    Changes in v3:
        - Typoes
        - Check if the foreign domain is different from the current domain

    Changes in v2:
        - Even if gcc is buggy (see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18501)
        define p2m type per mapspace to let the compiler warns about unitialized
        values.
---
 xen/arch/arm/mm.c         |   38 ++++++++++++++++++++++++++++++--------
 xen/include/asm-arm/p2m.h |    1 +
 2 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index b1222be..6829822 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -977,6 +977,7 @@ static int xenmem_add_to_physmap_one(
 {
     unsigned long mfn = 0;
     int rc;
+    p2m_type_t t;
 
     switch ( space )
     {
@@ -1009,22 +1010,33 @@ static int xenmem_add_to_physmap_one(
         
         d->arch.grant_table_gpfn[idx] = gpfn;
 
+        t = p2m_ram_rw;
+
         spin_unlock(&d->grant_table->lock);
         break;
     case XENMAPSPACE_shared_info:
-        if ( idx == 0 )
-            mfn = virt_to_mfn(d->shared_info);
-        else
+        if ( idx != 0 )
             return -EINVAL;
+
+        mfn = virt_to_mfn(d->shared_info);
+        t = p2m_ram_rw;
+
         break;
     case XENMAPSPACE_gmfn_foreign:
     {
-        paddr_t maddr;
         struct domain *od;
+        struct page_info *page;
+        p2m_type_t p2mt;
         od = rcu_lock_domain_by_any_id(foreign_domid);
         if ( od == NULL )
             return -ESRCH;
 
+        if ( od == d )
+        {
+            rcu_unlock_domain(od);
+            return -EINVAL;
+        }
+
         rc = xsm_map_gmfn_foreign(XSM_TARGET, d, od);
         if ( rc )
         {
@@ -1032,15 +1044,25 @@ static int xenmem_add_to_physmap_one(
             return rc;
         }
 
-        maddr = p2m_lookup(od, pfn_to_paddr(idx), NULL);
-        if ( maddr == INVALID_PADDR )
+        /* Take reference to the foreign domain page.
+         * Reference will be released in XENMEM_remove_from_physmap */
+        page = get_page_from_gfn(od, idx, &p2mt, P2M_ALLOC);
+        if ( !page )
         {
             dump_p2m_lookup(od, pfn_to_paddr(idx));
             rcu_unlock_domain(od);
             return -EINVAL;
         }
 
-        mfn = maddr >> PAGE_SHIFT;
+        if ( !p2m_is_ram(p2mt) )
+        {
+            put_page(page);
+            rcu_unlock_domain(od);
+            return -EINVAL;
+        }
+
+        mfn = page_to_mfn(page);
+        t = p2m_map_foreign;
 
         rcu_unlock_domain(od);
         break;
@@ -1051,7 +1073,7 @@ static int xenmem_add_to_physmap_one(
     }
 
     /* Map at new location. */
-    rc = guest_physmap_add_page(d, gpfn, mfn, 0);
+    rc = guest_physmap_add_entry(d, gpfn, mfn, 0, t);
 
     return rc;
 }
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index ac2b6fa..6a185db 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -47,6 +47,7 @@ typedef enum {
 } p2m_type_t;
 
 #define p2m_is_foreign(_t)  ((_t) == p2m_map_foreign)
+#define p2m_is_ram(_t)      ((_t) == p2m_ram_rw || (_t) == p2m_ram_ro)
 
 /* Initialise vmid allocator */
 void p2m_vmid_allocator_init(void);
-- 
1.7.10.4

[PATCH v5 10/10] xen/arm: grant-table: Support read-only mapping

[Julien Grall]

Signed-off-by: Julien Grall <julien.grall@linaro.org>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

---
    Changes in v2:
        - Use p2m grant type to map grant-table mapping
---
 xen/arch/arm/mm.c |   12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 6829822..b1d3c60 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -1307,19 +1307,17 @@ int create_grant_host_mapping(unsigned long addr, unsigned long frame,
                               unsigned int flags, unsigned int cache_flags)
 {
     int rc;
+    p2m_type_t t = p2m_grant_map_rw;
 
     if ( cache_flags  || (flags & ~GNTMAP_readonly) != GNTMAP_host_map )
         return GNTST_general_error;
 
-    /* XXX: read only mappings */
     if ( flags & GNTMAP_readonly )
-    {
-        gdprintk(XENLOG_WARNING, "read only mappings not implemented yet\n");
-        return GNTST_general_error;
-    }
+        t = p2m_grant_map_ro;
+
+    rc = guest_physmap_add_entry(current->domain, addr >> PAGE_SHIFT,
+                                 frame, 0, t);
 
-    rc = guest_physmap_add_page(current->domain,
-                                 addr >> PAGE_SHIFT, frame, 0);
     if ( rc )
         return GNTST_general_error;
     else
-- 
1.7.10.4

Re: [PATCH v5 04/10] xen/arm: Store p2m type in each page of the guest

[Ian Campbell]

On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> Use the field 'avail' to store the type of the page. Rename it to 'type' for
> convenience.
> The information stored in this field will be retrieved in a future patch to
> change the behaviour when the page is removed.
> 
> Also introduce guest_physmap_add_entry to map and set a specific p2m type for
> a page.
> 
> Signed-off-by: Julien Grall <julien.grall@linaro.org>
> 
> ---
>     Changes in v5:
>         - Foreign mapping doesn't need to have execution right

Neither does MMIO for that matter...

>  
> +    switch (t)
> +    {
> +    case p2m_map_foreign:
> +    case p2m_grant_map_rw:
> +        e.p2m.xn = 1;
> +        /* Fallthrough */
> +    case p2m_ram_rw:
> +    case p2m_mmio_direct:

... so move this up.

> +        e.p2m.write = 1;
> +        break;
> +
> +    case p2m_grant_map_ro:
> +        e.p2m.xn = 1;
> +        /* Fallthrough */
> +    case p2m_invalid:
> +    case p2m_ram_ro:
> +    default:

You were going to remove the default case IIRC to let the compiler catch
new type additions.

> +        e.p2m.write = 0;
> +    }
> +

>diff --git a/xen/include/asm-arm/page.h b/xen/include/asm-arm/page.h
> index 0625464..670d4e7 100644
> --- a/xen/include/asm-arm/page.h
> +++ b/xen/include/asm-arm/page.h
> @@ -153,7 +153,7 @@ typedef struct {
>      unsigned long contig:1;     /* In a block of 16 contiguous entries */
>      unsigned long sbz2:1;
>      unsigned long xn:1;         /* eXecute-Never */
> -    unsigned long avail:4;      /* Ignored by hardware */
> +    unsigned long type:4;       /* Ignore by hardware. Used to store p2m types */

"Ignored" was correct.

>  
>      unsigned long sbz1:5;
>  } __attribute__((__packed__)) lpae_p2m_t;

Re: [PATCH v5 04/10] xen/arm: Store p2m type in each page of the guest

[Julien Grall]

On 12/16/2013 05:49 PM, Ian Campbell wrote:
> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
>> Use the field 'avail' to store the type of the page. Rename it to 'type' for
>> convenience.
>> The information stored in this field will be retrieved in a future patch to
>> change the behaviour when the page is removed.
>>
>> Also introduce guest_physmap_add_entry to map and set a specific p2m type for
>> a page.
>>
>> Signed-off-by: Julien Grall <julien.grall@linaro.org>
>>
>> ---
>>      Changes in v5:
>>          - Foreign mapping doesn't need to have execution right
>
> Neither does MMIO for that matter...
>
>>
>> +    switch (t)
>> +    {
>> +    case p2m_map_foreign:
>> +    case p2m_grant_map_rw:
>> +        e.p2m.xn = 1;
>> +        /* Fallthrough */
>> +    case p2m_ram_rw:
>> +    case p2m_mmio_direct:
>
> ... so move this up.

Right.

>
>> +        e.p2m.write = 1;
>> +        break;
>> +
>> +    case p2m_grant_map_ro:
>> +        e.p2m.xn = 1;
>> +        /* Fallthrough */
>> +    case p2m_invalid:
>> +    case p2m_ram_ro:
>> +    default:
>
> You were going to remove the default case IIRC to let the compiler catch
> new type additions.

Sorry, I completely forgot. I will do it for the next version.

-- 
Julien Grall

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Ian Campbell]

On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
> +             hypercall_preempt_check() )
> +        {

If hypercall_preempt_check returns false the first time it is called
then you won't ever try again. I think you want
	(count+1 % LPAE_ENTRIES) == 0
(the +1 avoids preempting without having done any work yet)

(credit to Jan for pointing this out in his review of Mukesh's x86
variant)

Ian .

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Jan Beulich]

>>> On 17.12.13 at 10:26, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
>> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
>> +             hypercall_preempt_check() )
>> +        {
> 
> If hypercall_preempt_check returns false the first time it is called
> then you won't ever try again. I think you want
> 	(count+1 % LPAE_ENTRIES) == 0
> (the +1 avoids preempting without having done any work yet)

And hence needing proper parenthesization:

 	(count+1) % LPAE_ENTRIES == 0

Jan

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Ian Campbell]

On Tue, 2013-12-17 at 10:03 +0000, Jan Beulich wrote:
> >>> On 17.12.13 at 10:26, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> >> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
> >> +             hypercall_preempt_check() )
> >> +        {
> > 
> > If hypercall_preempt_check returns false the first time it is called
> > then you won't ever try again. I think you want
> > 	(count+1 % LPAE_ENTRIES) == 0
> > (the +1 avoids preempting without having done any work yet)
> 
> And hence needing proper parenthesization:
> 
>  	(count+1) % LPAE_ENTRIES == 0

Er, yes, oops!

Re: [PATCH v5 05/10] xen/arm: p2m: Extend p2m_lookup parameters to retrieve the p2m type

[Ian Campbell]

On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> @@ -102,7 +108,10 @@ paddr_t p2m_lookup(struct domain *d, paddr_t paddr)
>  
>  done:
>      if ( pte.p2m.valid )
> +    {
>          maddr = (pte.bits & PADDR_MASK & PAGE_MASK) | (paddr & ~PAGE_MASK);
> +        *t = pte.p2m.type;

This needs to either ASSERT(pte.p2m.type != p2m_invalid) or to
explicitly return INVALID_PADDR if it finds p2m_invalid.

(which one depends on whether p2m_invalid would be a coding bug here or
not)

Ian.

Re: [PATCH v5 06/10] xen/arm: Retrieve p2m type in get_page_from_gfn

[Ian Campbell]

On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> Signed-off-by: Julien Grall <julien.grall@linaro.org>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [PATCH v5 07/10] xen/arm: Handle remove foreign mapping

[Ian Campbell]

On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> Modify get_page_from_gfn to take reference on foreign mapping. This will avoid
> specific handling in the common code.
> 
> Signed-off-by: Julien Grall <julien.grall@linaro.org>
> 
> ---
>     Changes in v4.2:
>         - get_page_from_gfn: move foreign checking before get_page
>         - add assert fdom != dom
>     Changes in v4.1:
>         - Remove specific p2m handling in common code
>         - Handle foreign mapping in get_page_from_gfn
>     Changes in v4:
>         - Split patch #6 from dom0 pvh series v6.2 to retrieve only common
>         code.
>         - Rework commit title
>         - Rename xen_rem_foreign_from_p2m in p2m_remove_foreign
>         - Get the mfn from the pte. We are not sure that maddr given in
>         parameters is valid
>     Changes in v3:
>         - Move put_page in create_p2m_entries
>         - Move xenmem_rem_foreign_from_p2m in arch/arm/p2m.c
>     Changes in v2:
>         - Introduce the patch
> ---
>  xen/arch/arm/p2m.c        |   15 +++++++++++++--
>  xen/include/asm-arm/p2m.h |   12 ++++++++++++
>  2 files changed, 25 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index 9bdcacd..d05fdff 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -317,10 +317,21 @@ static int create_p2m_entries(struct domain *d,
>                  break;
>              case REMOVE:
>                  {
> -                    lpae_t pte;
> +                    lpae_t pte = third[third_table_offset(addr)];
> +                    unsigned long mfn;
> +
> +                    maddr = (pte.bits & PADDR_MASK & PAGE_MASK);
> +                    mfn = paddr_to_pfn(maddr);

FWIW mfn = pte.p2m.base, I think. I'm not sure we use this everywhere we
could.

> +
> +                    /* TODO: Handle other p2m type */
> +                    if ( pte.p2m.valid && p2m_is_foreign(pte.p2m.type) )

How useful is p2m_is_foreign now that this stuff doesn't need to be in
common code?

Both of the above are really just observations rather than requests for
change, so:
Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Ian Campbell]

On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index d05fdff..45179f7 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -6,6 +6,8 @@
>  #include <xen/bitops.h>
>  #include <asm/flushtlb.h>
>  #include <asm/gic.h>
> +#include <asm/event.h>
> +#include <asm/hardirq.h>
>  
>  /* First level P2M is 2 consecutive pages */
>  #define P2M_FIRST_ORDER 1
> @@ -213,7 +215,8 @@ static int p2m_create_table(struct domain *d,
>  enum p2m_operation {
>      INSERT,
>      ALLOCATE,
> -    REMOVE
> +    REMOVE,
> +    RELINQUISH,
>  };
>  
>  static int create_p2m_entries(struct domain *d,
> @@ -231,6 +234,7 @@ static int create_p2m_entries(struct domain *d,

If this function finds any non-present first or second level PTE then it
will stop and exit (goto out), meaning it will miss any mappings which
are higher up after the hole.

e.g. if you have a guest p2m with RAM at 0-2M and 4-6M then
relinquishing 0-6M will only actually free 0-2M, then abort on 4-6M.

Perhaps this could be fixed by making relinquish_p2m_mapping loop over
the address space relinquishing 2M chunks as it goes? This would remove
the need for the if ( op == RELINQUISH && .. && prempt() ) stuff,
because you could add the preempt in that loop.

BTW, Jan's approach in his x86 fix was to preempt every 2MB of present
or 32MB of non-present mappings, which makes sense because you skip
through the non-present ones more quickly. It'd be nice but lets not
hold up the series if it isn't trivial to achieve.

>      unsigned long cur_first_page = ~0,
>                    cur_first_offset = ~0,
>                    cur_second_offset = ~0;
> +    unsigned long count = 0;
>  
>      spin_lock(&p2m->lock);
>  
> @@ -315,6 +319,7 @@ static int create_p2m_entries(struct domain *d,
>                      maddr += PAGE_SIZE;
>                  }
>                  break;
> +            case RELINQUISH:
>              case REMOVE:
>                  {
>                      lpae_t pte = third[third_table_offset(addr)];
> @@ -338,6 +343,29 @@ static int create_p2m_entries(struct domain *d,
>  
>          if ( flush )
>              flush_tlb_all_local();
> +
> +
> +        count++;
> +
> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
> +             hypercall_preempt_check() )
> +        {
> +            p2m->next_gfn_to_relinquish = maddr >> PAGE_SHIFT;
> +            rc = -EAGAIN;
> +            goto out;
> +        }
> +    }
> +
> +    /* When the function will remove mapping, p2m type should always
> +     * be p2m_invalid. */
> +    if ( (t == p2m_ram_rw) || (t == p2m_ram_ro) || (t == p2m_map_foreign))

Is this a slightly odd way of saying either "t != p2m_invalid" or (op ==
CREATE || op == INSERT) ?

> +    {
> +        unsigned long sgfn = paddr_to_pfn(start_gpaddr);
> +        unsigned long egfn = paddr_to_pfn(end_gpaddr);
> +
> +        p2m->max_mapped_gfn = MAX(p2m->max_mapped_gfn, egfn);
> +        /* Use next_gfn_to_relinquish to store the lowest gfn mapped */
> +        p2m->next_gfn_to_relinquish = MIN(p2m->next_gfn_to_relinquish, sgfn);
>      }
>  
>      rc = 0;
> @@ -523,12 +551,26 @@ int p2m_init(struct domain *d)
>  
>      p2m->first_level = NULL;
>  
> +    p2m->max_mapped_gfn = 0;
> +    p2m->next_gfn_to_relinquish = ULONG_MAX;
> +
>  err:
>      spin_unlock(&p2m->lock);
>  
>      return rc;
>  }
>  
> +int relinquish_p2m_mapping(struct domain *d)
> +{
> +    struct p2m_domain *p2m = &d->arch.p2m;
> +
> +    return create_p2m_entries(d, RELINQUISH,
> +                              pfn_to_paddr(p2m->next_gfn_to_relinquish),
> +                              pfn_to_paddr(p2m->max_mapped_gfn),
> +                              pfn_to_paddr(INVALID_MFN),
> +                              MATTR_MEM, p2m_invalid);
> +}
> +
>  unsigned long gmfn_to_mfn(struct domain *d, unsigned long gpfn)
>  {
>      paddr_t p = p2m_lookup(d, pfn_to_paddr(gpfn), NULL);
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 53c9895..28d39a0 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -112,6 +112,7 @@ struct arch_domain
>          RELMEM_not_started,
>          RELMEM_xen,
>          RELMEM_page,
> +        RELMEM_mapping,
>          RELMEM_done,
>      } relmem;
>  
> diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
> index 5ccfa7f..ac2b6fa 100644
> --- a/xen/include/asm-arm/p2m.h
> +++ b/xen/include/asm-arm/p2m.h
> @@ -18,6 +18,15 @@ struct p2m_domain {
>  
>      /* Current VMID in use */
>      uint8_t vmid;
> +
> +    /* Highest guest frame that's ever been mapped in the p2m
> +     * Take only into account ram and foreign mapping

"Only takes into account ... mappings" or "Takes only ... mappings into
account".


> +     */
> +    unsigned long max_mapped_gfn;
> +
> +    /* When releasing mapped gfn's in a preemptible manner, recall where
> +     * to resume the search */
> +    unsigned long next_gfn_to_relinquish;
>  };
>  
>  /* List of possible type for each page in the p2m entry.
> @@ -48,6 +57,12 @@ int p2m_init(struct domain *d);
>  /* Return all the p2m resources to Xen. */
>  void p2m_teardown(struct domain *d);
>  
> +/* Remove mapping refcount on each mapping page in the p2m
> + *
> + * TODO: For the moment only foreign mapping is handled

"only foreign mappings are handled".

> + */
> +int relinquish_p2m_mapping(struct domain *d);
> +
>  /* Allocate a new p2m table for a domain.
>   *
>   * Returns 0 for success or -errno.

Re: [PATCH v5 09/10] xen/arm: Set foreign page type to p2m_map_foreign

[Ian Campbell]

On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> Xen needs to know that the current page belongs to another domain. Also take
> a reference to this page.
> 
> The current process to add a foreign page is:
>    1) get the page from the foreign p2m
>    2) take a reference on the page with the foreign domain in parameters
>    3) add the page to the current domain p2m
> 
> If the foreign domain drops the page:
>     - before 2), get_page will return NULL because the page doesn't
>     belong anymore to the domain
>     - after 2), the current domain already have a reference. Write will
>     occur to an old page which is not yet released. It can corrupt the foreign
>     domain.
> 
> Signed-off-by: Julien Grall <julien.grall@linaro.org>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Ian.

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Julien Grall]

On 12/17/2013 09:26 AM, Ian Campbell wrote:
> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
>> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
>> +             hypercall_preempt_check() )
>> +        {
> 
> If hypercall_preempt_check returns false the first time it is called
> then you won't ever try again. I think you want
> 	(count+1 % LPAE_ENTRIES) == 0
> (the +1 avoids preempting without having done any work yet)
> 
> (credit to Jan for pointing this out in his review of Mukesh's x86
> variant)

The code was copied from relinquish_shared_pages
(arch/x86/mem_sharing.c), so if I'm not mistaken this code is also buggy.

-- 
Julien Grall

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Ian Campbell]

On Tue, 2013-12-17 at 14:08 +0000, Julien Grall wrote:
> On 12/17/2013 09:26 AM, Ian Campbell wrote:
> > On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> >> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
> >> +             hypercall_preempt_check() )
> >> +        {
> > 
> > If hypercall_preempt_check returns false the first time it is called
> > then you won't ever try again. I think you want
> > 	(count+1 % LPAE_ENTRIES) == 0
> > (the +1 avoids preempting without having done any work yet)
> > 
> > (credit to Jan for pointing this out in his review of Mukesh's x86
> > variant)
> 
> The code was copied from relinquish_shared_pages
> (arch/x86/mem_sharing.c), so if I'm not mistaken this code is also buggy.

Yes, Jan sent a fix for it this morning.

Ian.

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Julien Grall]

On 12/17/2013 09:26 AM, Ian Campbell wrote:
> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
>> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
>> +             hypercall_preempt_check() )
>> +        {
> 
> If hypercall_preempt_check returns false the first time it is called
> then you won't ever try again. I think you want
> 	(count+1 % LPAE_ENTRIES) == 0
> (the +1 avoids preempting without having done any work yet)

I don't think we need +1 because count is increment each round just
before the check.

Is a solution as Jan made for x86 would be more appropriate?

-- 
Julien Grall

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Ian Campbell]

On Tue, 2013-12-17 at 14:40 +0000, Julien Grall wrote:
> On 12/17/2013 09:26 AM, Ian Campbell wrote:
> > On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> >> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
> >> +             hypercall_preempt_check() )
> >> +        {
> > 
> > If hypercall_preempt_check returns false the first time it is called
> > then you won't ever try again. I think you want
> > 	(count+1 % LPAE_ENTRIES) == 0
> > (the +1 avoids preempting without having done any work yet)
> 
> I don't think we need +1 because count is increment each round just
> before the check.

Ah yes, if the increment has already happened then good.

> Is a solution as Jan made for x86 would be more appropriate?

It's about equivalent I think? I'd hope the compiler would take a mod
2^N and turn it into a mask, but I suppose you never know...

Ian.

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Julien Grall]

On 12/17/2013 02:42 PM, Ian Campbell wrote:
> On Tue, 2013-12-17 at 14:40 +0000, Julien Grall wrote:
>> On 12/17/2013 09:26 AM, Ian Campbell wrote:
>>> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
>>>> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
>>>> +             hypercall_preempt_check() )
>>>> +        {
>>>
>>> If hypercall_preempt_check returns false the first time it is called
>>> then you won't ever try again. I think you want
>>> 	(count+1 % LPAE_ENTRIES) == 0
>>> (the +1 avoids preempting without having done any work yet)
>>
>> I don't think we need +1 because count is increment each round just
>> before the check.
> 
> Ah yes, if the increment has already happened then good.
> 
>> Is a solution as Jan made for x86 would be more appropriate?
> 
> It's about equivalent I think? I'd hope the compiler would take a mod
> 2^N and turn it into a mask, but I suppose you never know...

I was talking about adding a greater increment when it's a foreign mapping.

-- 
Julien Grall

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Ian Campbell]

On Tue, 2013-12-17 at 14:45 +0000, Julien Grall wrote:
> On 12/17/2013 02:42 PM, Ian Campbell wrote:
> > On Tue, 2013-12-17 at 14:40 +0000, Julien Grall wrote:
> >> On 12/17/2013 09:26 AM, Ian Campbell wrote:
> >>> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> >>>> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
> >>>> +             hypercall_preempt_check() )
> >>>> +        {
> >>>
> >>> If hypercall_preempt_check returns false the first time it is called
> >>> then you won't ever try again. I think you want
> >>> 	(count+1 % LPAE_ENTRIES) == 0
> >>> (the +1 avoids preempting without having done any work yet)
> >>
> >> I don't think we need +1 because count is increment each round just
> >> before the check.
> > 
> > Ah yes, if the increment has already happened then good.
> > 
> >> Is a solution as Jan made for x86 would be more appropriate?
> > 
> > It's about equivalent I think? I'd hope the compiler would take a mod
> > 2^N and turn it into a mask, but I suppose you never know...
> 
> I was talking about adding a greater increment when it's a foreign mapping.

Like I said in the other subthread:

        BTW, Jan's approach in his x86 fix was to preempt every 2MB of present
        or 32MB of non-present mappings, which makes sense because you skip
        through the non-present ones more quickly. It'd be nice but lets not
        hold up the series if it isn't trivial to achieve.
        
Ian

Re: [PATCH v5 08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

[Julien Grall]

On 12/17/2013 02:52 PM, Ian Campbell wrote:
> On Tue, 2013-12-17 at 14:45 +0000, Julien Grall wrote:
>> On 12/17/2013 02:42 PM, Ian Campbell wrote:
>>> On Tue, 2013-12-17 at 14:40 +0000, Julien Grall wrote:
>>>> On 12/17/2013 09:26 AM, Ian Campbell wrote:
>>>>> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
>>>>>> +        if ( op == RELINQUISH && count == LPAE_ENTRIES &&
>>>>>> +             hypercall_preempt_check() )
>>>>>> +        {
>>>>>
>>>>> If hypercall_preempt_check returns false the first time it is called
>>>>> then you won't ever try again. I think you want
>>>>> 	(count+1 % LPAE_ENTRIES) == 0
>>>>> (the +1 avoids preempting without having done any work yet)
>>>>
>>>> I don't think we need +1 because count is increment each round just
>>>> before the check.
>>>
>>> Ah yes, if the increment has already happened then good.
>>>
>>>> Is a solution as Jan made for x86 would be more appropriate?
>>>
>>> It's about equivalent I think? I'd hope the compiler would take a mod
>>> 2^N and turn it into a mask, but I suppose you never know...
>>
>> I was talking about adding a greater increment when it's a foreign mapping.
> 
> Like I said in the other subthread:
> 
>         BTW, Jan's approach in his x86 fix was to preempt every 2MB of present
>         or 32MB of non-present mappings, which makes sense because you skip
>         through the non-present ones more quickly. It'd be nice but lets not
>         hold up the series if it isn't trivial to achieve.

Sorry I didn't yet read this mail. I will try to implement the same
behavior for the next version.

-- 
Julien Grall

Re: [PATCH v5 07/10] xen/arm: Handle remove foreign mapping

[Julien Grall]

On 12/17/2013 11:18 AM, Ian Campbell wrote:
> On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
>> Modify get_page_from_gfn to take reference on foreign mapping. This will avoid
>> specific handling in the common code.
>>
>> Signed-off-by: Julien Grall <julien.grall@linaro.org>
>>
>> ---
>>     Changes in v4.2:
>>         - get_page_from_gfn: move foreign checking before get_page
>>         - add assert fdom != dom
>>     Changes in v4.1:
>>         - Remove specific p2m handling in common code
>>         - Handle foreign mapping in get_page_from_gfn
>>     Changes in v4:
>>         - Split patch #6 from dom0 pvh series v6.2 to retrieve only common
>>         code.
>>         - Rework commit title
>>         - Rename xen_rem_foreign_from_p2m in p2m_remove_foreign
>>         - Get the mfn from the pte. We are not sure that maddr given in
>>         parameters is valid
>>     Changes in v3:
>>         - Move put_page in create_p2m_entries
>>         - Move xenmem_rem_foreign_from_p2m in arch/arm/p2m.c
>>     Changes in v2:
>>         - Introduce the patch
>> ---
>>  xen/arch/arm/p2m.c        |   15 +++++++++++++--
>>  xen/include/asm-arm/p2m.h |   12 ++++++++++++
>>  2 files changed, 25 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
>> index 9bdcacd..d05fdff 100644
>> --- a/xen/arch/arm/p2m.c
>> +++ b/xen/arch/arm/p2m.c
>> @@ -317,10 +317,21 @@ static int create_p2m_entries(struct domain *d,
>>                  break;
>>              case REMOVE:
>>                  {
>> -                    lpae_t pte;
>> +                    lpae_t pte = third[third_table_offset(addr)];
>> +                    unsigned long mfn;
>> +
>> +                    maddr = (pte.bits & PADDR_MASK & PAGE_MASK);
>> +                    mfn = paddr_to_pfn(maddr);
> 
> FWIW mfn = pte.p2m.base, I think. I'm not sure we use this everywhere we
> could.

The patch #8 which relinquish memory doesn't give the right mfn in
parameters.
It's because, you will need to look for each gfn the translated mfn (you
can't assumed the mfn are contiguous). So the loop will go back to the
version 3.

> 
>> +
>> +                    /* TODO: Handle other p2m type */
>> +                    if ( pte.p2m.valid && p2m_is_foreign(pte.p2m.type) )
> 
> How useful is p2m_is_foreign now that this stuff doesn't need to be in
> common code?

I'm not sure to understand. Do you mean the  macro? If so it's only for
convenience.

> Both of the above are really just observations rather than requests for
> change, so:
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks,

-- 
Julien Grall

Re: [PATCH v5 07/10] xen/arm: Handle remove foreign mapping

[Ian Campbell]

On Tue, 2013-12-17 at 15:06 +0000, Julien Grall wrote:
> On 12/17/2013 11:18 AM, Ian Campbell wrote:
> > On Mon, 2013-12-16 at 17:37 +0000, Julien Grall wrote:
> >> Modify get_page_from_gfn to take reference on foreign mapping. This will avoid
> >> specific handling in the common code.
> >>
> >> Signed-off-by: Julien Grall <julien.grall@linaro.org>
> >>
> >> ---
> >>     Changes in v4.2:
> >>         - get_page_from_gfn: move foreign checking before get_page
> >>         - add assert fdom != dom
> >>     Changes in v4.1:
> >>         - Remove specific p2m handling in common code
> >>         - Handle foreign mapping in get_page_from_gfn
> >>     Changes in v4:
> >>         - Split patch #6 from dom0 pvh series v6.2 to retrieve only common
> >>         code.
> >>         - Rework commit title
> >>         - Rename xen_rem_foreign_from_p2m in p2m_remove_foreign
> >>         - Get the mfn from the pte. We are not sure that maddr given in
> >>         parameters is valid
> >>     Changes in v3:
> >>         - Move put_page in create_p2m_entries
> >>         - Move xenmem_rem_foreign_from_p2m in arch/arm/p2m.c
> >>     Changes in v2:
> >>         - Introduce the patch
> >> ---
> >>  xen/arch/arm/p2m.c        |   15 +++++++++++++--
> >>  xen/include/asm-arm/p2m.h |   12 ++++++++++++
> >>  2 files changed, 25 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> >> index 9bdcacd..d05fdff 100644
> >> --- a/xen/arch/arm/p2m.c
> >> +++ b/xen/arch/arm/p2m.c
> >> @@ -317,10 +317,21 @@ static int create_p2m_entries(struct domain *d,
> >>                  break;
> >>              case REMOVE:
> >>                  {
> >> -                    lpae_t pte;
> >> +                    lpae_t pte = third[third_table_offset(addr)];
> >> +                    unsigned long mfn;
> >> +
> >> +                    maddr = (pte.bits & PADDR_MASK & PAGE_MASK);
> >> +                    mfn = paddr_to_pfn(maddr);
> > 
> > FWIW mfn = pte.p2m.base, I think. I'm not sure we use this everywhere we
> > could.
> 
> The patch #8 which relinquish memory doesn't give the right mfn in
> parameters.
> It's because, you will need to look for each gfn the translated mfn (you
> can't assumed the mfn are contiguous). So the loop will go back to the
> version 3.

I'm not sure what you mean.

All I was trying to say is that:
	mfn = paddr_to_pfn(pte.bits & PADDR_MASK & PAGE_MASK)
is (probably) equivalent to:
	mfn = pte.p2m.base

> >> +
> >> +                    /* TODO: Handle other p2m type */
> >> +                    if ( pte.p2m.valid && p2m_is_foreign(pte.p2m.type) )
> > 
> > How useful is p2m_is_foreign now that this stuff doesn't need to be in
> > common code?
> 
> I'm not sure to understand. Do you mean the  macro?

Yes

> If so it's only for convenience.

Convenience is fair enoguh, especially once I noticed p2m_is_ram etc.

> 
> > Both of the above are really just observations rather than requests for
> > change, so:
> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
> 
> Thanks,
> 

Re: [PATCH v5 07/10] xen/arm: Handle remove foreign mapping

[Julien Grall]

On 12/17/2013 03:21 PM, Ian Campbell wrote:
> On Tue, 2013-12-17 at 15:06 +0000, Julien Grall wrote:
>> On 12/17/2013 11:18 AM, Ian Campbell wrote:
>> The patch #8 which relinquish memory doesn't give the right mfn in
>> parameters.
>> It's because, you will need to look for each gfn the translated mfn (you
>> can't assumed the mfn are contiguous). So the loop will go back to the
>> version 3.
> 
> I'm not sure what you mean.
> 
> All I was trying to say is that:
> 	mfn = paddr_to_pfn(pte.bits & PADDR_MASK & PAGE_MASK)
> is (probably) equivalent to:
> 	mfn = pte.p2m.base

We didn't talk about the same thing.

I will use pte.p2m.base for the next version.


-- 
Julien Grall

Re: [PATCH v5 07/10] xen/arm: Handle remove foreign mapping

[Ian Campbell]

On Tue, 2013-12-17 at 15:44 +0000, Julien Grall wrote:
> On 12/17/2013 03:21 PM, Ian Campbell wrote:
> > On Tue, 2013-12-17 at 15:06 +0000, Julien Grall wrote:
> >> On 12/17/2013 11:18 AM, Ian Campbell wrote:
> >> The patch #8 which relinquish memory doesn't give the right mfn in
> >> parameters.
> >> It's because, you will need to look for each gfn the translated mfn (you
> >> can't assumed the mfn are contiguous). So the loop will go back to the
> >> version 3.
> > 
> > I'm not sure what you mean.
> > 
> > All I was trying to say is that:
> > 	mfn = paddr_to_pfn(pte.bits & PADDR_MASK & PAGE_MASK)
> > is (probably) equivalent to:
> > 	mfn = pte.p2m.base
> 
> We didn't talk about the same thing.
> 
> I will use pte.p2m.base for the next version.

Make sure to check that I am right about them being the same ;-)

Ian.