* [PATCH v02 0/7] arm: introduce remoteprocessor iommu module
@ 2014-06-26 11:06 Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 1/7] " Andrii Tseglytskyi
` (6 more replies)
0 siblings, 7 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:06 UTC (permalink / raw)
To: xen-devel
The following patch series introduces IOMMU translation
framework for remote processors. Remote processors are
typically used for graphic rendering (GPUs) and
high quality video decoding (IPUs). They are typically
installed on such multimedia SoCs as OMAP4 / OMAP5.
As soon as remoteprocessor MMU typically works with
pagetables filled by physical addresses, which are
allocated by domU kernel, it is almost impossible to
use them under Xen - intermediate physical addresses
allocated by kernel, need to be translated to machine
addresses which are managed by Xen.
Andrii Tseglytskyi (7):
arm: introduce remoteprocessor iommu module
arm: omap: introduce iommu translation for IPU remoteproc
arm: omap: introduce iommu translation for GPU remoteproc
arm: omap: introduce print pagetable function for IPU remoteproc
arm: omap: introduce print pagetable function for GPU remoteproc
arm: introduce do_translate_pagetable hypercall
arm: add trap for remoteproc mmio accesses
xen/arch/arm/Makefile | 1 +
xen/arch/arm/io.c | 1 +
xen/arch/arm/io.h | 1 +
xen/arch/arm/platforms/Makefile | 1 +
xen/arch/arm/platforms/omap_iommu.c | 458 +++++++++++++++++++++++++++++++++++
xen/arch/arm/remoteproc_iommu.c | 436 +++++++++++++++++++++++++++++++++
xen/arch/arm/traps.c | 1 +
xen/include/public/xen.h | 1 +
xen/include/xen/hypercall.h | 12 +
xen/include/xen/remoteproc_iommu.h | 82 +++++++
10 files changed, 994 insertions(+)
create mode 100644 xen/arch/arm/platforms/omap_iommu.c
create mode 100644 xen/arch/arm/remoteproc_iommu.c
create mode 100644 xen/include/xen/remoteproc_iommu.h
--
1.7.9.5
^ permalink raw reply [flat|nested] 57+ messages in thread
* [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
@ 2014-06-26 11:07 ` Andrii Tseglytskyi
2014-06-29 18:00 ` Julien Grall
` (2 more replies)
2014-06-26 11:07 ` [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc Andrii Tseglytskyi
` (5 subsequent siblings)
6 siblings, 3 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:07 UTC (permalink / raw)
To: xen-devel
This is a fisrst patch from patch series which was
developed to handle remote (external) processors
memory management units. Remote processors are
typically used for graphic rendering (GPUs) and
high quality video decoding (IPUs). They are typically
installed on such multimedia SoCs as OMAP4 / OMAP5.
As soon as remoteprocessor MMU typically works with
pagetables filled by physical addresses, which are
allocated by domU kernel, it is almost impossible to
use them under Xen, intermediate physical addresses
allocated by kernel, need to be translated to machine
addresses.
This patch introduces a simple framework to perform
pfn -> mfn translation for external MMUs.
It introduces basic data structures and algorithms
needed for translation.
Typically, when MMU is configured, some it registers
are updated by new values. Introduced frameworks
uses traps as starting point of remoteproc MMUs
pagetables translation.
Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
---
xen/arch/arm/Makefile | 1 +
xen/arch/arm/remoteproc_iommu.c | 412 ++++++++++++++++++++++++++++++++++++
xen/include/xen/remoteproc_iommu.h | 79 +++++++
3 files changed, 492 insertions(+)
create mode 100644 xen/arch/arm/remoteproc_iommu.c
create mode 100644 xen/include/xen/remoteproc_iommu.h
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index d70f6d5..0204d1c 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -15,6 +15,7 @@ obj-y += io.o
obj-y += irq.o
obj-y += kernel.o
obj-y += mm.o
+obj-y += remoteproc_iommu.o
obj-y += p2m.o
obj-y += percpu.o
obj-y += guestcopy.o
diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
new file mode 100644
index 0000000..b4d22d9
--- /dev/null
+++ b/xen/arch/arm/remoteproc_iommu.c
@@ -0,0 +1,412 @@
+/*
+ * xen/arch/arm/remoteproc_iommu.c
+ *
+ * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
+ * Copyright (c) 2014 GlobalLogic
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/config.h>
+#include <xen/lib.h>
+#include <xen/errno.h>
+#include <xen/mm.h>
+#include <xen/vmap.h>
+#include <xen/init.h>
+#include <xen/sched.h>
+#include <xen/stdbool.h>
+#include <asm/system.h>
+#include <asm/current.h>
+#include <asm/io.h>
+#include <asm/p2m.h>
+
+#include <xen/remoteproc_iommu.h>
+
+#include "io.h"
+
+static struct mmu_info *mmu_list[] = {
+};
+
+#define mmu_for_each(pfunc, data) \
+({ \
+ u32 __i; \
+ int __res = 0; \
+ \
+ for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
+ { \
+ __res |= pfunc(mmu_list[__i], data); \
+ } \
+ __res; \
+})
+
+static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
+{
+ if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start + mmu->mem_size)) )
+ return 1;
+
+ return 0;
+}
+
+static inline struct mmu_info *mmu_lookup(u32 addr)
+{
+ u32 i;
+
+ /* enumerate all registered MMU's and check is address in range */
+ for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
+ {
+ if ( mmu_check_mem_range(mmu_list[i], addr) )
+ return mmu_list[i];
+ }
+
+ return NULL;
+}
+
+static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
+{
+ return mmu_for_each(mmu_check_mem_range, addr);
+}
+
+static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
+{
+ void __iomem *pagetable = NULL;
+ u32 maddr, i;
+
+ ASSERT(mmu);
+ ASSERT(pgt);
+
+ if ( !pgt->paddr )
+ return -EINVAL;
+
+ /* pagetable size can be more than one page */
+ for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
+ {
+ /* lookup address where remoteproc pagetable is stored by kernel */
+ maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE, NULL);
+ if ( !maddr )
+ {
+ pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i * PAGE_SIZE);
+ return -EINVAL;
+ }
+
+ pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
+ if ( !pagetable )
+ {
+ pr_mmu("failed to map pagetable");
+ return -EINVAL;
+ }
+
+ /* copy pagetable to hypervisor memory */
+ clean_and_invalidate_xen_dcache_va_range(pagetable, PAGE_SIZE);
+ memcpy((u32*)((u32)pgt->kern_pagetable + i * PAGE_SIZE), pagetable, PAGE_SIZE);
+
+ iounmap(pagetable);
+ }
+
+ return 0;
+}
+
+struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32 addr, bool is_maddr)
+{
+ struct mmu_pagetable *pgt;
+ u32 pgt_addr;
+
+ list_for_each_entry(pgt, &mmu->pagetables_list, link_node)
+ {
+ if ( is_maddr )
+ pgt_addr = pgt->maddr;
+ else
+ pgt_addr = pgt->paddr;
+
+ if ( pgt_addr == addr )
+ return pgt;
+ }
+
+ return NULL;
+}
+
+static struct mmu_pagetable *mmu_alloc_pagetable(struct mmu_info *mmu, u32 paddr)
+{
+ struct mmu_pagetable *pgt;
+ u32 pgt_size = MMU_PGD_TABLE_SIZE(mmu);
+
+ pgt = xzalloc_bytes(sizeof(struct mmu_pagetable));
+ if ( !pgt )
+ {
+ pr_mmu("failed to alloc pagetable structure");
+ return NULL;
+ }
+
+ /* allocate pagetable managed by hypervisor */
+ pgt->hyp_pagetable = xzalloc_bytes(pgt_size);
+ if ( !pgt->hyp_pagetable )
+ {
+ pr_mmu("failed to alloc private hyp_pagetable");
+ return NULL;
+ }
+
+ /* alocate pagetable for ipa storing */
+ pgt->kern_pagetable = xzalloc_bytes(pgt_size);
+ if ( !pgt->kern_pagetable )
+ {
+ pr_mmu("failed to alloc private kern_pagetable");
+ return NULL;
+ }
+
+ pr_mmu("private pagetables for 0x%08x paddr %u bytes (main 0x%08x, temp 0x%08x)",
+ paddr, pgt_size, (u32)__pa(pgt->hyp_pagetable), (u32)__pa(pgt->kern_pagetable));
+
+ pgt->paddr = paddr;
+
+ list_add(&pgt->link_node, &mmu->pagetables_list);
+
+ return pgt;
+}
+
+static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
+{
+ struct mmu_pagetable *pgt;
+ int res;
+
+ /* lookup using machine address first */
+ pgt = mmu_pagetable_lookup(mmu, paddr, true);
+ if ( !pgt )
+ {
+ /* lookup using kernel physical address */
+ pgt = mmu_pagetable_lookup(mmu, paddr, false);
+ if ( !pgt )
+ {
+ /* if pagetable doesn't exists in lookup list - allocate it */
+ pgt = mmu_alloc_pagetable(mmu, paddr);
+ }
+ }
+
+ pgt->maddr = MMU_INVALID_ADDRESS;
+
+ /* copy pagetable from domain to hypervisor */
+ res = mmu_copy_pagetable(mmu, pgt);
+ if ( res )
+ return res;
+
+ /* translate pagetable */
+ pgt->maddr = mmu->translate_pfunc(mmu, pgt);
+ return pgt->maddr;
+}
+
+static u32 mmu_trap_translate_pagetable(struct mmu_info *mmu, mmio_info_t *info)
+{
+ register_t *reg;
+ bool valid_trap = false;
+ u32 i, paddr;
+
+ reg = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
+
+ ASSERT(reg);
+
+ paddr = *reg;
+ if ( !paddr )
+ return MMU_INVALID_ADDRESS;
+
+ /* check is the register is a valid TTB register */
+ for ( i = 0; i < mmu->num_traps; i++ )
+ {
+ if ( mmu->trap_offsets[i] == (info->gpa - mmu->mem_start) )
+ {
+ valid_trap = true;
+ break;
+ }
+ }
+
+ if ( !valid_trap )
+ return MMU_INVALID_ADDRESS;
+
+ return mmu_translate_pagetable(mmu, paddr);
+}
+
+u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
+ u32 maddr, u32 hyp_addr)
+{
+ u32 *pte_table = NULL, *hyp_pte_table = NULL, pte_table_size = PAGE_SIZE;
+ u32 i;
+
+ /* remap second level translation table */
+ pte_table = ioremap_nocache(maddr, MMU_PTE_TABLE_SIZE(mmu));
+ if ( !pte_table )
+ {
+ pr_mmu("failed to map pte_table");
+ return MMU_INVALID_ADDRESS;
+ }
+
+ /* allocate new second level pagetable once */
+ if ( 0 == hyp_addr )
+ {
+ if ( MMU_PTE_TABLE_SIZE(mmu) > PAGE_SIZE )
+ pte_table_size = MMU_PTE_TABLE_SIZE(mmu);
+
+ hyp_pte_table = xzalloc_bytes(pte_table_size);
+ if ( !hyp_pte_table )
+ {
+ pr_mmu("failed to alloc new iopt");
+ return MMU_INVALID_ADDRESS;
+ }
+ }
+ else
+ {
+ hyp_pte_table = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
+ }
+
+ /* 2-nd level translation */
+ for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
+ {
+ paddr_t pt_maddr, pt_paddr, pt_flags;
+ u32 pt_mask = MMU_SECTION_MASK(mmu->pg_data->pte_shift);
+
+ if ( !pte_table[i] )
+ {
+ /* handle the case when page was removed */
+ if ( unlikely(hyp_pte_table[i]) )
+ {
+ hyp_pte_table[i] = 0;
+ }
+
+ continue;
+ }
+
+ pt_paddr = pte_table[i] & pt_mask;
+ pt_flags = pte_table[i] & ~pt_mask;
+ pt_maddr = p2m_lookup(current->domain, pt_paddr, NULL);
+ ASSERT(pt_maddr != INVALID_PADDR);
+
+ hyp_pte_table[i] = pt_maddr | pt_flags;
+ pgt->page_counter++;
+ }
+
+ iounmap(pte_table);
+
+ clean_and_invalidate_xen_dcache_va_range(hyp_pte_table, MMU_PTE_TABLE_SIZE(mmu));
+ return __pa(hyp_pte_table);
+}
+
+static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
+{
+ struct mmu_info *mmu = NULL;
+ unsigned long flags;
+ register_t *r;
+
+ r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
+
+ ASSERT(r);
+
+ mmu = mmu_lookup(info->gpa);
+ if ( !mmu )
+ {
+ pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
+ return -EINVAL;
+ }
+
+ spin_lock_irqsave(&mmu->lock, flags);
+ *r = readl(mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
+ spin_unlock_irqrestore(&mmu->lock, flags);
+
+ return 1;
+}
+
+static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
+{
+ struct mmu_info *mmu = NULL;
+ unsigned long flags;
+ register_t *r;
+ u32 new_addr, val;
+
+ r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
+
+ ASSERT(r);
+
+ /* dom0 should not access remoteproc MMU */
+ if ( 0 == current->domain->domain_id )
+ return 1;
+
+ /* find corresponding MMU */
+ mmu = mmu_lookup(info->gpa);
+ if ( !mmu )
+ {
+ pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
+ return -EINVAL;
+ }
+
+ ASSERT(v->domain == current->domain);
+
+ spin_lock_irqsave(&mmu->lock, flags);
+
+ /* get new address of translated pagetable */
+ new_addr = mmu_trap_translate_pagetable(mmu, info);
+ if ( MMU_INVALID_ADDRESS != new_addr )
+ val = new_addr;
+ else
+ val = *r;
+
+ writel(val, mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
+ spin_unlock_irqrestore(&mmu->lock, flags);
+
+ return 1;
+}
+
+static int mmu_init(struct mmu_info *mmu, u32 data)
+{
+ ASSERT(mmu);
+ ASSERT(!mmu->mem_map);
+
+ INIT_LIST_HEAD(&mmu->pagetables_list);
+
+ /* map MMU memory */
+ mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
+ if ( !mmu->mem_map )
+ {
+ pr_mmu("failed to map memory");
+ return -EINVAL;
+ }
+
+ pr_mmu("memory map = 0x%pS", _p(mmu->mem_map));
+
+ spin_lock_init(&mmu->lock);
+
+ return 0;
+}
+
+static int mmu_init_all(void)
+{
+ int res;
+
+ res = mmu_for_each(mmu_init, 0);
+ if ( res )
+ {
+ printk("%s error during init %d\n", __func__, res);
+ return res;
+ }
+
+ return 0;
+}
+
+const struct mmio_handler remoteproc_mmio_handler = {
+ .check_handler = mmu_mmio_check,
+ .read_handler = mmu_mmio_read,
+ .write_handler = mmu_mmio_write,
+};
+
+__initcall(mmu_init_all);
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
new file mode 100644
index 0000000..22e2951
--- /dev/null
+++ b/xen/include/xen/remoteproc_iommu.h
@@ -0,0 +1,79 @@
+/*
+ * xen/include/xen/remoteproc_iommu.h
+ *
+ * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
+ * Copyright (c) 2014 GlobalLogic
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _REMOTEPROC_IOMMU_H_
+#define _REMOTEPROC_IOMMU_H_
+
+#include <asm/types.h>
+
+#define MMU_SECTION_SIZE(shift) (1UL << (shift))
+#define MMU_SECTION_MASK(shift) (~(MMU_SECTION_SIZE(shift) - 1))
+
+/* 4096 first level descriptors for "supersection" and "section" */
+#define MMU_PTRS_PER_PGD(mmu) (1UL << (32 - (mmu->pg_data->pgd_shift)))
+#define MMU_PGD_TABLE_SIZE(mmu) (MMU_PTRS_PER_PGD(mmu) * sizeof(u32))
+
+/* 256 second level descriptors for "small" and "large" pages */
+#define MMU_PTRS_PER_PTE(mmu) (1UL << ((mmu->pg_data->pgd_shift) - (mmu->pg_data->pte_shift)))
+#define MMU_PTE_TABLE_SIZE(mmu) (MMU_PTRS_PER_PTE(mmu) * sizeof(u32))
+
+/* 16 sections in supersection */
+#define MMU_SECTION_PER_SUPER(mmu) (1UL << ((mmu->pg_data->super_shift) - (mmu->pg_data->section_shift)))
+
+#define MMU_INVALID_ADDRESS ((u32)(-1))
+
+#define pr_mmu(fmt, ...) \
+ printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
+
+struct pagetable_data {
+ /* 1st level translation */
+ u32 pgd_shift;
+ u32 pte_shift;
+ u32 super_shift;
+ u32 section_shift;
+ /* 2nd level translation */
+ u32 pte_large_shift;
+};
+
+struct mmu_pagetable {
+ u32 *hyp_pagetable;
+ u32 *kern_pagetable;
+ u32 paddr;
+ u32 maddr;
+ struct list_head link_node;
+ u32 page_counter;
+};
+
+struct mmu_info {
+ const char *name;
+ const struct pagetable_data *pg_data;
+ /* register where phys pointer to pagetable is stored */
+ u32 *trap_offsets;
+ paddr_t mem_start;
+ u32 mem_size;
+ spinlock_t lock;
+ struct list_head pagetables_list;
+ u32 num_traps;
+ void __iomem *mem_map;
+ u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
+ void (*print_pagetable_pfunc)(struct mmu_info *);
+};
+
+u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
+ u32 maddr, u32 hyp_addr);
+
+#endif /* _REMOTEPROC_IOMMU_H_ */
--
1.7.9.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 1/7] " Andrii Tseglytskyi
@ 2014-06-26 11:07 ` Andrii Tseglytskyi
2014-07-04 14:01 ` Stefano Stabellini
` (2 more replies)
2014-06-26 11:07 ` [PATCH v02 3/7] arm: omap: introduce iommu translation for GPU remoteproc Andrii Tseglytskyi
` (4 subsequent siblings)
6 siblings, 3 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:07 UTC (permalink / raw)
To: xen-devel
The following patch introduced platform specific MMU data
definitions and pagetable translation function for OMAP5 IPU
remoteproc. This MMU is a bit specific - it typically performs
one level translation and map a big chunks of memory. 16 Mb
supersections and 1 Mb sections are mapped instead of 4 Kb pages.
Introduced algorithm performs internal remapping of big sections
to small 4 Kb pages.
Change-Id: If20449f07e22f780e1fded67fed4f79cbe1fc156
Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
---
xen/arch/arm/platforms/Makefile | 1 +
xen/arch/arm/platforms/omap_iommu.c | 247 +++++++++++++++++++++++++++++++++++
xen/arch/arm/remoteproc_iommu.c | 1 +
xen/include/xen/remoteproc_iommu.h | 2 +
4 files changed, 251 insertions(+)
create mode 100644 xen/arch/arm/platforms/omap_iommu.c
diff --git a/xen/arch/arm/platforms/Makefile b/xen/arch/arm/platforms/Makefile
index 080ea9a..f224f08 100644
--- a/xen/arch/arm/platforms/Makefile
+++ b/xen/arch/arm/platforms/Makefile
@@ -4,4 +4,5 @@ obj-$(CONFIG_ARM_32) += midway.o
obj-$(CONFIG_ARM_32) += omap5.o
obj-$(CONFIG_ARM_32) += dra7xx.o
obj-$(CONFIG_ARM_32) += sunxi.o
+obj-$(CONFIG_ARM_32) += omap_iommu.o
obj-$(CONFIG_ARM_64) += xgene-storm.o
diff --git a/xen/arch/arm/platforms/omap_iommu.c b/xen/arch/arm/platforms/omap_iommu.c
new file mode 100644
index 0000000..e0c4633
--- /dev/null
+++ b/xen/arch/arm/platforms/omap_iommu.c
@@ -0,0 +1,247 @@
+/*
+ * xen/arch/arm/platforms/omap_iommu.c
+ *
+ * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
+ * Copyright (c) 2014 GlobalLogic
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/lib.h>
+#include <xen/errno.h>
+#include <xen/stdbool.h>
+#include <xen/mm.h>
+#include <xen/vmap.h>
+#include <xen/sched.h>
+#include <xen/remoteproc_iommu.h>
+
+#include <asm/p2m.h>
+
+/*
+ * "L2 table" address mask and size definitions.
+ */
+
+/* register where address of pagetable is stored */
+#define MMU_IPU_TTB_OFFSET 0x4c
+
+/* 1st level translation */
+#define MMU_OMAP_PGD_SHIFT 20
+#define MMU_OMAP_SUPER_SHIFT 24 /* "supersection" - 16 Mb */
+#define MMU_OMAP_SECTION_SHIFT 20 /* "section" - 1 Mb */
+#define MMU_OMAP_SECOND_LEVEL_SHIFT 10
+
+/* 2nd level translation */
+#define MMU_OMAP_PTE_SMALL_SHIFT 12 /* "small page" - 4Kb */
+#define MMU_OMAP_PTE_LARGE_SHIFT 16 /* "large page" - 64 Kb */
+
+/*
+ * some descriptor attributes.
+ */
+#define PGD_TABLE (1 << 0)
+#define PGD_SECTION (2 << 0)
+#define PGD_SUPER (1 << 18 | 2 << 0)
+
+#define ipu_pgd_is_table(x) (((x) & 3) == PGD_TABLE)
+#define ipu_pgd_is_section(x) (((x) & (1 << 18 | 3)) == PGD_SECTION)
+#define ipu_pgd_is_super(x) (((x) & (1 << 18 | 3)) == PGD_SUPER)
+
+#define PTE_SMALL (2 << 0)
+#define PTE_LARGE (1 << 0)
+
+#define OMAP_IPU_MMU_MEM_BASE 0x55082000
+
+static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
+
+static u32 ipu_trap_offsets[] = {
+ MMU_IPU_TTB_OFFSET,
+};
+
+static const struct pagetable_data pagetable_ipu_data = {
+ .pgd_shift = MMU_OMAP_PGD_SHIFT,
+ .super_shift = MMU_OMAP_SUPER_SHIFT,
+ .section_shift = MMU_OMAP_SECTION_SHIFT,
+ .pte_shift = MMU_OMAP_PTE_SMALL_SHIFT,
+ .pte_large_shift = MMU_OMAP_PTE_LARGE_SHIFT,
+};
+
+struct mmu_info omap_ipu_mmu = {
+ .name = "IPU_L2_MMU",
+ .pg_data = &pagetable_ipu_data,
+ .trap_offsets = ipu_trap_offsets,
+ .mem_start = OMAP_IPU_MMU_MEM_BASE,
+ .mem_size = 0x1000,
+ .num_traps = ARRAY_SIZE(ipu_trap_offsets),
+ .translate_pfunc = mmu_ipu_translate_pagetable,
+};
+
+static bool translate_supersections_to_pages = true;
+static bool translate_sections_to_pages = true;
+
+static u32 mmu_pte_table_alloc(struct mmu_info *mmu, u32 pgd, u32 sect_num,
+ struct mmu_pagetable *pgt, u32 hyp_addr)
+{
+ u32 *pte = NULL;
+ u32 i;
+
+ /* allocate pte table once */
+ if ( 0 == hyp_addr )
+ {
+ pte = xzalloc_bytes(PAGE_SIZE);
+ if ( !pte )
+ {
+ pr_mmu("failed to alloc 2nd level table");
+ return 0;
+ }
+ }
+ else
+ {
+ pte = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
+ }
+
+ ASSERT(256 == MMU_PTRS_PER_PTE(mmu));
+
+ for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
+ {
+ u32 paddr, maddr;
+
+ paddr = pgd + (i * PAGE_SIZE);
+ maddr = p2m_lookup(current->domain, paddr, NULL);
+ ASSERT(maddr != INVALID_PADDR);
+
+ pte[i] = maddr | PTE_SMALL;
+ pgt->page_counter++;
+ }
+
+ clean_and_invalidate_xen_dcache_va_range(pte, PAGE_SIZE);
+ return __pa(pte) | PGD_TABLE;
+}
+
+static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
+{
+ u32 *kern_pgt, *hyp_pgt;
+ const struct pagetable_data *data;
+ u32 i;
+
+ ASSERT(mmu);
+ ASSERT(pgt);
+
+ data = mmu->pg_data;
+ kern_pgt = pgt->kern_pagetable;
+ hyp_pgt = pgt->hyp_pagetable;
+ pgt->page_counter = 0;
+
+ ASSERT(4096 == MMU_PTRS_PER_PGD(mmu));
+
+ /* 1-st level translation */
+ for ( i = 0; i < MMU_PTRS_PER_PGD(mmu); i++ )
+ {
+ paddr_t pd_maddr, pd_paddr, pd_flags;
+ u32 pd_mask;
+ u32 pgd_tmp, pgd = kern_pgt[i];
+
+ if ( !pgd )
+ {
+ /* handle the case when second level translation table
+ * was removed from kernel */
+ if ( unlikely(hyp_pgt[i]) )
+ {
+ xfree(__va(hyp_pgt[i] & MMU_SECTION_MASK(MMU_OMAP_SECOND_LEVEL_SHIFT)));
+ hyp_pgt[i] = 0;
+ }
+
+ continue;
+ }
+
+ /* first level pointers have different formats, depending on their type */
+ if ( ipu_pgd_is_super(pgd) )
+ pd_mask = MMU_SECTION_MASK(MMU_OMAP_SUPER_SHIFT);
+ else if ( ipu_pgd_is_section(pgd) )
+ pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECTION_SHIFT);
+ else if ( ipu_pgd_is_table(pgd) )
+ pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECOND_LEVEL_SHIFT);
+
+ pd_paddr = pgd & pd_mask;
+ pd_flags = pgd & ~pd_mask;
+ pd_maddr = p2m_lookup(current->domain, pd_paddr, NULL);
+ ASSERT(pd_maddr != INVALID_PADDR);
+
+ /* "supersection" 16 Mb */
+ if ( ipu_pgd_is_super(pgd) )
+ {
+ /* mapping of 16 Mb chunk is fragmented to 4 Kb pages */
+ if( likely(translate_supersections_to_pages) )
+ {
+ u32 j;
+
+ ASSERT(16 == MMU_SECTION_PER_SUPER(mmu));
+ ASSERT(1048576 == MMU_SECTION_SIZE(data->section_shift));
+
+ /* 16 Mb supersection is divided to 16 sections of 1 MB size */
+ for ( j = 0 ; j < MMU_SECTION_PER_SUPER(mmu); j++ )
+ {
+ pgd_tmp = (pgd & ~PGD_SUPER) + (j * MMU_SECTION_SIZE(data->section_shift));
+ hyp_pgt[i + j] = mmu_pte_table_alloc(mmu, pgd_tmp, i, pgt, hyp_pgt[i + j]);
+ }
+
+ /* move counter after supersection is translated */
+ i += (j - 1);
+ }
+ else
+ {
+ hyp_pgt[i] = pd_maddr | pd_flags;
+ }
+
+ /* "section" 1Mb */
+ }
+ else if ( ipu_pgd_is_section(pgd) )
+ {
+ if ( likely(translate_sections_to_pages) )
+ {
+ pgd_tmp = (pgd & ~PGD_SECTION);
+ hyp_pgt[i] = mmu_pte_table_alloc(mmu, pgd_tmp, i, pgt, hyp_pgt[i]);
+ }
+ else
+ {
+ hyp_pgt[i] = pd_maddr | pd_flags;
+ }
+
+ /* "table" */
+ }
+ else if ( unlikely(ipu_pgd_is_table(pgd)) )
+ {
+ ASSERT(4096 == MMU_PTRS_PER_PGD(mmu));
+ ASSERT(256 == MMU_PTRS_PER_PTE(mmu));
+
+ hyp_pgt[i] = mmu_translate_second_level(mmu, pgt, pd_maddr, hyp_pgt[i]);
+ hyp_pgt[i] |= pd_flags;
+
+ /* error */
+ }
+ else
+ {
+ pr_mmu("unknown entry %u: 0x%08x", i, pgd);
+ return MMU_INVALID_ADDRESS;
+ }
+ }
+
+ /* force omap IOMMU to use new pagetable */
+ clean_and_invalidate_xen_dcache_va_range(hyp_pgt, MMU_PGD_TABLE_SIZE(mmu));
+ return __pa(hyp_pgt);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
index b4d22d9..8291f3f 100644
--- a/xen/arch/arm/remoteproc_iommu.c
+++ b/xen/arch/arm/remoteproc_iommu.c
@@ -33,6 +33,7 @@
#include "io.h"
static struct mmu_info *mmu_list[] = {
+ &omap_ipu_mmu,
};
#define mmu_for_each(pfunc, data) \
diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
index 22e2951..ff1c439 100644
--- a/xen/include/xen/remoteproc_iommu.h
+++ b/xen/include/xen/remoteproc_iommu.h
@@ -76,4 +76,6 @@ struct mmu_info {
u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
u32 maddr, u32 hyp_addr);
+extern struct mmu_info omap_ipu_mmu;
+
#endif /* _REMOTEPROC_IOMMU_H_ */
--
1.7.9.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v02 3/7] arm: omap: introduce iommu translation for GPU remoteproc
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 1/7] " Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc Andrii Tseglytskyi
@ 2014-06-26 11:07 ` Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 4/7] arm: omap: introduce print pagetable function for IPU remoteproc Andrii Tseglytskyi
` (3 subsequent siblings)
6 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:07 UTC (permalink / raw)
To: xen-devel
The following patch introduced platform specific MMU data
definitions and pagetable translation function for OMAP5 GPU
remoteproc. Typically GPU MMU performs uses two level address
translation, so algorithm is quite straightforward here -
pagetables are enumerated and all pfns are updated with
corresponding mfns.
Current patch adds functionality, needed for proper handling of
GPU MMU, which is very similar to existing IPU/DSP MMUs.
Change-Id: I129da9485c61cc94801c6b243498e31db33f5d30
Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
---
xen/arch/arm/platforms/omap_iommu.c | 86 +++++++++++++++++++++++++++++++++++
xen/arch/arm/remoteproc_iommu.c | 1 +
xen/include/xen/remoteproc_iommu.h | 1 +
3 files changed, 88 insertions(+)
diff --git a/xen/arch/arm/platforms/omap_iommu.c b/xen/arch/arm/platforms/omap_iommu.c
index e0c4633..bbaa7bc 100644
--- a/xen/arch/arm/platforms/omap_iommu.c
+++ b/xen/arch/arm/platforms/omap_iommu.c
@@ -32,12 +32,23 @@
/* register where address of pagetable is stored */
#define MMU_IPU_TTB_OFFSET 0x4c
+#define MMU_SGX_TTB_OFFSET_00 0xc84
+#define MMU_SGX_TTB_OFFSET_01 0xc38
+#define MMU_SGX_TTB_OFFSET_02 0xc3c
+#define MMU_SGX_TTB_OFFSET_03 0xc40
+#define MMU_SGX_TTB_OFFSET_04 0xc44
+#define MMU_SGX_TTB_OFFSET_05 0xc48
+#define MMU_SGX_TTB_OFFSET_06 0xc4c
+#define MMU_SGX_TTB_OFFSET_07 0xc50
+
/* 1st level translation */
#define MMU_OMAP_PGD_SHIFT 20
#define MMU_OMAP_SUPER_SHIFT 24 /* "supersection" - 16 Mb */
#define MMU_OMAP_SECTION_SHIFT 20 /* "section" - 1 Mb */
#define MMU_OMAP_SECOND_LEVEL_SHIFT 10
+#define MMU_SGX_PGD_SHIFT 22 /* SGX section */
+
/* 2nd level translation */
#define MMU_OMAP_PTE_SMALL_SHIFT 12 /* "small page" - 4Kb */
#define MMU_OMAP_PTE_LARGE_SHIFT 16 /* "large page" - 64 Kb */
@@ -57,13 +68,26 @@
#define PTE_LARGE (1 << 0)
#define OMAP_IPU_MMU_MEM_BASE 0x55082000
+#define OMAP_SGX_MMU_MEM_BASE 0x56000000
static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
+static u32 mmu_sgx_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
static u32 ipu_trap_offsets[] = {
MMU_IPU_TTB_OFFSET,
};
+static u32 sgx_trap_offsets[] = {
+ MMU_SGX_TTB_OFFSET_00,
+ MMU_SGX_TTB_OFFSET_01,
+ MMU_SGX_TTB_OFFSET_02,
+ MMU_SGX_TTB_OFFSET_03,
+ MMU_SGX_TTB_OFFSET_04,
+ MMU_SGX_TTB_OFFSET_05,
+ MMU_SGX_TTB_OFFSET_06,
+ MMU_SGX_TTB_OFFSET_07,
+};
+
static const struct pagetable_data pagetable_ipu_data = {
.pgd_shift = MMU_OMAP_PGD_SHIFT,
.super_shift = MMU_OMAP_SUPER_SHIFT,
@@ -82,6 +106,23 @@ struct mmu_info omap_ipu_mmu = {
.translate_pfunc = mmu_ipu_translate_pagetable,
};
+static const struct pagetable_data pagetable_sgx_data = {
+ .pgd_shift = MMU_SGX_PGD_SHIFT,
+ .super_shift = MMU_SGX_PGD_SHIFT,
+ .section_shift = MMU_SGX_PGD_SHIFT,
+ .pte_shift = MMU_OMAP_PTE_SMALL_SHIFT, /* the same as IPU */
+};
+
+struct mmu_info omap_sgx_mmu = {
+ .name = "SGX_L2_MMU",
+ .pg_data = &pagetable_sgx_data,
+ .trap_offsets = sgx_trap_offsets,
+ .mem_start = OMAP_SGX_MMU_MEM_BASE,
+ .mem_size = 0x1000,
+ .num_traps = ARRAY_SIZE(sgx_trap_offsets),
+ .translate_pfunc = mmu_sgx_translate_pagetable,
+};
+
static bool translate_supersections_to_pages = true;
static bool translate_sections_to_pages = true;
@@ -237,6 +278,51 @@ static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetabl
return __pa(hyp_pgt);
}
+static u32 mmu_sgx_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
+{
+ u32 *kern_pgt, *hyp_pgt;
+ u32 i;
+
+ ASSERT(mmu);
+ ASSERT(pgt);
+
+ kern_pgt = pgt->kern_pagetable;
+ hyp_pgt = pgt->hyp_pagetable;
+ pgt->page_counter = 0;
+
+ /* 1-st level translation */
+ for ( i = 0; i < MMU_PTRS_PER_PGD(mmu); i++ )
+ {
+ paddr_t pd_maddr, pd_paddr, pd_flags;
+ u32 pgd, pd_mask = MMU_SECTION_MASK(mmu->pg_data->pte_shift);
+
+ pgd = kern_pgt[i];
+ if ( !pgd )
+ {
+ /* handle the case when second level translation table
+ * was removed from kernel */
+ if ( unlikely(hyp_pgt[i]) )
+ {
+ xfree(__va(hyp_pgt[i] & pd_mask));
+ hyp_pgt[i] = 0;
+ }
+ continue;
+ }
+
+ pd_paddr = pgd & pd_mask;
+ pd_flags = pgd & ~pd_mask;
+ pd_maddr = p2m_lookup(current->domain, pd_paddr, NULL);
+ ASSERT(pd_maddr != INVALID_PADDR);
+
+ /* 2-nd level translation */
+ hyp_pgt[i] = mmu_translate_second_level(mmu, pgt, pd_maddr, hyp_pgt[i]);
+ hyp_pgt[i] |= pd_flags;
+ }
+
+ clean_and_invalidate_xen_dcache_va_range(hyp_pgt, MMU_PGD_TABLE_SIZE(mmu));
+ return __pa(hyp_pgt);
+}
+
/*
* Local variables:
* mode: C
diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
index 8291f3f..3b3da3b 100644
--- a/xen/arch/arm/remoteproc_iommu.c
+++ b/xen/arch/arm/remoteproc_iommu.c
@@ -34,6 +34,7 @@
static struct mmu_info *mmu_list[] = {
&omap_ipu_mmu,
+ &omap_sgx_mmu,
};
#define mmu_for_each(pfunc, data) \
diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
index ff1c439..d69c85e 100644
--- a/xen/include/xen/remoteproc_iommu.h
+++ b/xen/include/xen/remoteproc_iommu.h
@@ -77,5 +77,6 @@ u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
u32 maddr, u32 hyp_addr);
extern struct mmu_info omap_ipu_mmu;
+extern struct mmu_info omap_sgx_mmu;
#endif /* _REMOTEPROC_IOMMU_H_ */
--
1.7.9.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v02 4/7] arm: omap: introduce print pagetable function for IPU remoteproc
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
` (2 preceding siblings ...)
2014-06-26 11:07 ` [PATCH v02 3/7] arm: omap: introduce iommu translation for GPU remoteproc Andrii Tseglytskyi
@ 2014-06-26 11:07 ` Andrii Tseglytskyi
2014-07-16 15:38 ` Ian Campbell
2014-06-26 11:07 ` [PATCH v02 5/7] arm: omap: introduce print pagetable function for GPU remoteproc Andrii Tseglytskyi
` (2 subsequent siblings)
6 siblings, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:07 UTC (permalink / raw)
To: xen-devel
This patch adds a possibility to dump all pagetables of
IPU remoteproc. The only reason to have this patch - is a
low level debug.
Change-Id: If327ca62c964ff9cba3d0952c80a15c06884470b
Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
---
xen/arch/arm/platforms/omap_iommu.c | 68 ++++++++++++++++++++++++++++++++++-
1 file changed, 67 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/platforms/omap_iommu.c b/xen/arch/arm/platforms/omap_iommu.c
index bbaa7bc..d3a3c15 100644
--- a/xen/arch/arm/platforms/omap_iommu.c
+++ b/xen/arch/arm/platforms/omap_iommu.c
@@ -19,7 +19,6 @@
#include <xen/errno.h>
#include <xen/stdbool.h>
#include <xen/mm.h>
-#include <xen/vmap.h>
#include <xen/sched.h>
#include <xen/remoteproc_iommu.h>
@@ -73,6 +72,8 @@
static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
static u32 mmu_sgx_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
+static void mmu_ipu_print_pagetables(struct mmu_info *mmu);
+
static u32 ipu_trap_offsets[] = {
MMU_IPU_TTB_OFFSET,
};
@@ -104,6 +105,7 @@ struct mmu_info omap_ipu_mmu = {
.mem_size = 0x1000,
.num_traps = ARRAY_SIZE(ipu_trap_offsets),
.translate_pfunc = mmu_ipu_translate_pagetable,
+ .print_pagetable_pfunc = mmu_ipu_print_pagetables,
};
static const struct pagetable_data pagetable_sgx_data = {
@@ -165,6 +167,70 @@ static u32 mmu_pte_table_alloc(struct mmu_info *mmu, u32 pgd, u32 sect_num,
return __pa(pte) | PGD_TABLE;
}
+static void mmu_ipu_print_one_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt, u32 index)
+{
+ u32 i, page_counter = 0;
+ u32 *pagetable;
+
+ ASSERT(pgt);
+ ASSERT(pgt->hyp_pagetable);
+ ASSERT(pgt->paddr);
+ ASSERT(pgt->maddr);
+
+ pagetable = pgt->hyp_pagetable;
+
+ pr_mmu("pgt[%u][0x%08x][0x%08x]", index, pgt->paddr, pgt->maddr);
+ for ( i = 0; i < MMU_PTRS_PER_PGD(mmu); i++ )
+ {
+ u32 pgd = pagetable[i];
+ u32 *pte_table = NULL;
+ u32 j;
+
+ if ( !pgd )
+ continue;
+
+ /* "supersection" 16 Mb */
+ /* "section" 1Mb */
+ if ( ipu_pgd_is_super(pgd) || ipu_pgd_is_section(pgd) )
+ {
+ pr_mmu("pgt[%u][0x%08x][0x%08x] pgd[%u] 0x%08x (max %lu)",
+ index, pgt->paddr, pgt->maddr, i, pgd, MMU_PTRS_PER_PGD(mmu));
+
+ /* "table" */
+ }
+ else if ( ipu_pgd_is_table(pgd) )
+ {
+ pte_table = __va(pgd & MMU_SECTION_MASK(MMU_OMAP_SECOND_LEVEL_SHIFT));
+ if ( !pte_table )
+ {
+ pr_mmu("failed to map pagetable");
+ return;
+ }
+
+ for ( j = 0; j < MMU_PTRS_PER_PTE(mmu); j++ )
+ {
+ if ( !pte_table[j] )
+ continue;
+
+ page_counter++;
+ pr_mmu("pgt[%u][0x%08x][0x%08x] pgd[%u][0x%08x]\t pte[%u][0x%08x] (max %lu)",
+ index, pgt->paddr, pgt->maddr, i, pgd, j, pte_table[j], MMU_PTRS_PER_PTE(mmu));
+ }
+ }
+ }
+ ASSERT(page_counter == pgt->page_counter);
+}
+
+static void mmu_ipu_print_pagetables(struct mmu_info *mmu)
+{
+ struct mmu_pagetable *pgt;
+ u32 i = 0;
+
+ list_for_each_entry(pgt, &mmu->pagetables_list, link_node) {
+ mmu_ipu_print_one_pagetable(mmu, pgt, i++);
+ }
+}
+
static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
{
u32 *kern_pgt, *hyp_pgt;
--
1.7.9.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v02 5/7] arm: omap: introduce print pagetable function for GPU remoteproc
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
` (3 preceding siblings ...)
2014-06-26 11:07 ` [PATCH v02 4/7] arm: omap: introduce print pagetable function for IPU remoteproc Andrii Tseglytskyi
@ 2014-06-26 11:07 ` Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 7/7] arm: add trap for remoteproc mmio accesses Andrii Tseglytskyi
6 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:07 UTC (permalink / raw)
To: xen-devel
This patch adds a possibility to dump all pagetables of
GPU remoteproc. The only reason to have this patch - is a
low level debug.
Change-Id: I9577dda0f3e4a15f596d2bdcc7b6488f91a7c0e0
Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
---
xen/arch/arm/platforms/omap_iommu.c | 59 +++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/xen/arch/arm/platforms/omap_iommu.c b/xen/arch/arm/platforms/omap_iommu.c
index d3a3c15..c92b027 100644
--- a/xen/arch/arm/platforms/omap_iommu.c
+++ b/xen/arch/arm/platforms/omap_iommu.c
@@ -73,6 +73,7 @@ static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetabl
static u32 mmu_sgx_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
static void mmu_ipu_print_pagetables(struct mmu_info *mmu);
+static void mmu_sgx_print_pagetables(struct mmu_info *mmu);
static u32 ipu_trap_offsets[] = {
MMU_IPU_TTB_OFFSET,
@@ -123,6 +124,7 @@ struct mmu_info omap_sgx_mmu = {
.mem_size = 0x1000,
.num_traps = ARRAY_SIZE(sgx_trap_offsets),
.translate_pfunc = mmu_sgx_translate_pagetable,
+ .print_pagetable_pfunc = mmu_sgx_print_pagetables,
};
static bool translate_supersections_to_pages = true;
@@ -344,6 +346,63 @@ static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetabl
return __pa(hyp_pgt);
}
+static void mmu_sgx_print_one_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt, u32 index)
+{
+ u32 i, page_counter = 0;
+ u32 *pagetable;
+
+ ASSERT(pgt);
+ ASSERT(pgt->hyp_pagetable);
+ ASSERT(pgt->paddr);
+ ASSERT(pgt->maddr);
+
+ pagetable = pgt->hyp_pagetable;
+
+ pr_mmu("pgt[%u][0x%08x][0x%08x]", index, pgt->paddr, pgt->maddr);
+ /* 1-st level translation */
+ for ( i = 0; i < MMU_PTRS_PER_PGD(mmu); i++ )
+ {
+ u32 pgd = pagetable[i];
+ u32 *pte_table = NULL;
+ u32 j;
+
+ if ( !pgd )
+ continue;
+
+ pr_mmu("pgt[%u][0x%08x][0x%08x] pgd[%u] 0x%08x (max %lu)",
+ index, pgt->paddr, pgt->maddr, i, pgd, MMU_PTRS_PER_PGD(mmu));
+
+ pte_table = __va(pgd & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
+ if ( !pte_table )
+ {
+ pr_mmu("failed to map pagetable");
+ return;
+ }
+
+ for ( j = 0; j < MMU_PTRS_PER_PTE(mmu); j++ )
+ {
+ if ( !pte_table[j] )
+ continue;
+
+ page_counter++;
+ pr_mmu("pgt[%u][0x%08x][0x%08x] pgd[%u]\t pte_table[%u] 0x%08x (max %lu)",
+ index, pgt->paddr, pgt->maddr, i, j, pte_table[j], MMU_PTRS_PER_PTE(mmu));
+ }
+ }
+ ASSERT(page_counter == pgt->page_counter);
+}
+
+static void mmu_sgx_print_pagetables(struct mmu_info *mmu)
+{
+ struct mmu_pagetable *pgt;
+ u32 i = 0;
+
+ list_for_each_entry(pgt, &mmu->pagetables_list, link_node)
+ {
+ mmu_sgx_print_one_pagetable(mmu, pgt, i++);
+ }
+}
+
static u32 mmu_sgx_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
{
u32 *kern_pgt, *hyp_pgt;
--
1.7.9.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
` (4 preceding siblings ...)
2014-06-26 11:07 ` [PATCH v02 5/7] arm: omap: introduce print pagetable function for GPU remoteproc Andrii Tseglytskyi
@ 2014-06-26 11:07 ` Andrii Tseglytskyi
2014-07-04 14:05 ` Stefano Stabellini
2014-07-04 14:35 ` Julien Grall
2014-06-26 11:07 ` [PATCH v02 7/7] arm: add trap for remoteproc mmio accesses Andrii Tseglytskyi
6 siblings, 2 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:07 UTC (permalink / raw)
To: xen-devel
The reason of the patch is the following - some remoteprocs
are quite complicated, and their MMUs can handle several
pagetables. Good example is a OMAP5 GPU, which allocates
several pagetables during it work. Additional requirement
is that not all pagetable physical addresses are stored
in MMU registers. Some pagetables may be allocated and
then their physical addresses are sent to GPU using private
message loop between GPU kernel driver and GPU remoteproc.
Patch is developed to handle this. At any moment of time
kernel can perform translation of such pagetables, before
sending their addresses to GPU remoteproc.
Change-Id: Ie84012163205c3a2f920dc4cf07327a244647c93
Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
---
xen/arch/arm/remoteproc_iommu.c | 22 ++++++++++++++++++++++
xen/arch/arm/traps.c | 1 +
xen/include/public/xen.h | 1 +
xen/include/xen/hypercall.h | 12 ++++++++++++
4 files changed, 36 insertions(+)
diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
index 3b3da3b..316ceb4 100644
--- a/xen/arch/arm/remoteproc_iommu.c
+++ b/xen/arch/arm/remoteproc_iommu.c
@@ -23,6 +23,8 @@
#include <xen/init.h>
#include <xen/sched.h>
#include <xen/stdbool.h>
+#include <xen/hypercall.h>
+#include <xen/guest_access.h>
#include <asm/system.h>
#include <asm/current.h>
#include <asm/io.h>
@@ -382,6 +384,26 @@ static int mmu_init(struct mmu_info *mmu, u32 data)
return 0;
}
+long do_translate_pagetable(int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
+{
+ struct xen_pagetable_addr pgt;
+ struct mmu_info *mmu = NULL;
+
+ if ( copy_from_guest(&pgt, pgt_addr, 1) )
+ return -EFAULT;
+
+ mmu = mmu_lookup(pgt.reg);
+ if ( !mmu )
+ {
+ pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
+ return -EINVAL;
+ }
+
+ pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
+
+ return copy_to_guest(pgt_addr, &pgt, 1);
+}
+
static int mmu_init_all(void)
{
int res;
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 21c7b26..05b5184 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -1012,6 +1012,7 @@ static arm_hypercall_t arm_hypercall_table[] = {
HYPERCALL(hvm_op, 2),
HYPERCALL(grant_table_op, 3),
HYPERCALL_ARM(vcpu_op, 3),
+ HYPERCALL(translate_pagetable, 2),
};
typedef int (*arm_psci_fn_t)(uint32_t, register_t);
diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
index 8c5697e..91ca6a1 100644
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -101,6 +101,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);
#define __HYPERVISOR_kexec_op 37
#define __HYPERVISOR_tmem_op 38
#define __HYPERVISOR_xc_reserved_op 39 /* reserved for XenClient */
+#define __HYPERVISOR_translate_pagetable 40
/* Architecture-specific hypercall definitions. */
#define __HYPERVISOR_arch_0 48
diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
index a9e5229..a025435 100644
--- a/xen/include/xen/hypercall.h
+++ b/xen/include/xen/hypercall.h
@@ -136,6 +136,18 @@ extern long
do_tmem_op(
XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);
+struct xen_pagetable_addr {
+ u32 reg;
+ u32 paddr;
+ u32 maddr;
+};
+typedef struct xen_pagetable_addr xen_pagetable_addr_t;
+DEFINE_XEN_GUEST_HANDLE(xen_pagetable_addr_t);
+
+extern long
+do_translate_pagetable(
+ int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr);
+
extern long
do_xenoprof_op(int op, XEN_GUEST_HANDLE_PARAM(void) arg);
--
1.7.9.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v02 7/7] arm: add trap for remoteproc mmio accesses
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
` (5 preceding siblings ...)
2014-06-26 11:07 ` [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall Andrii Tseglytskyi
@ 2014-06-26 11:07 ` Andrii Tseglytskyi
2014-06-26 16:52 ` Julien Grall
6 siblings, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-26 11:07 UTC (permalink / raw)
To: xen-devel
The following patch connects already introduced
remoteproc iommu framework with existing trap framework.
Now, when kernel tries to access external MMU registers,
Xen triggers remoteproc iommu, which may perform
proper pfn to mfn translation.
Change-Id: Iffeacfda7783dee1aaa49299cf0a026a481fe6e7
Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
---
xen/arch/arm/io.c | 1 +
xen/arch/arm/io.h | 1 +
2 files changed, 2 insertions(+)
diff --git a/xen/arch/arm/io.c b/xen/arch/arm/io.c
index a6db00b..c1cae5c 100644
--- a/xen/arch/arm/io.c
+++ b/xen/arch/arm/io.c
@@ -26,6 +26,7 @@ static const struct mmio_handler *const mmio_handlers[] =
{
&vgic_distr_mmio_handler,
&vuart_mmio_handler,
+ &remoteproc_mmio_handler,
};
#define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
index 8d252c0..c780335 100644
--- a/xen/arch/arm/io.h
+++ b/xen/arch/arm/io.h
@@ -42,6 +42,7 @@ struct mmio_handler {
extern const struct mmio_handler vgic_distr_mmio_handler;
extern const struct mmio_handler vuart_mmio_handler;
+extern const struct mmio_handler remoteproc_mmio_handler;
extern int handle_mmio(mmio_info_t *info);
--
1.7.9.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* Re: [PATCH v02 7/7] arm: add trap for remoteproc mmio accesses
2014-06-26 11:07 ` [PATCH v02 7/7] arm: add trap for remoteproc mmio accesses Andrii Tseglytskyi
@ 2014-06-26 16:52 ` Julien Grall
2014-06-27 8:36 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-06-26 16:52 UTC (permalink / raw)
To: Andrii Tseglytskyi, xen-devel
Hi Andrii,
Thank you for the patch.
On 06/26/2014 12:07 PM, Andrii Tseglytskyi wrote:
> The following patch connects already introduced
> remoteproc iommu framework with existing trap framework.
> Now, when kernel tries to access external MMU registers,
> Xen triggers remoteproc iommu, which may perform
> proper pfn to mfn translation.
Vijay Kilari has rewritten the MMIO handling in Xen recently. Although
his patch is not yet in tree, I think it will go soon as the patch has
been acked by Ian, Stefano and me.
I advice you to rebase you work on it [1] [2]
Regards,
[1] http://lists.xen.org/archives/html/xen-devel/2014-06/msg03302.html
[2] http://lists.xen.org/archives/html/xen-devel/2014-06/msg03303.html
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 7/7] arm: add trap for remoteproc mmio accesses
2014-06-26 16:52 ` Julien Grall
@ 2014-06-27 8:36 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-06-27 8:36 UTC (permalink / raw)
To: Julien Grall; +Cc: xen-devel
Hi Julien,
On Thu, Jun 26, 2014 at 7:52 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Andrii,
>
> Vijay Kilari has rewritten the MMIO handling in Xen recently. Although
> his patch is not yet in tree, I think it will go soon as the patch has
> been acked by Ian, Stefano and me.
>
> I advice you to rebase you work on it [1] [2]
>
> Regards,
>
> [1] http://lists.xen.org/archives/html/xen-devel/2014-06/msg03302.html
> [2] http://lists.xen.org/archives/html/xen-devel/2014-06/msg03303.html
Thanks a lot. Will rebase my series on top of this.
Regards,
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-06-26 11:07 ` [PATCH v02 1/7] " Andrii Tseglytskyi
@ 2014-06-29 18:00 ` Julien Grall
2014-07-22 15:20 ` Andrii Tseglytskyi
2014-07-04 13:59 ` Stefano Stabellini
2014-07-16 15:29 ` Ian Campbell
2 siblings, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-06-29 18:00 UTC (permalink / raw)
To: Andrii Tseglytskyi, xen-devel, Stefano Stabellini, Ian Campbell,
Tim Deegan
Hi Andrii,
On 26/06/14 12:07, Andrii Tseglytskyi wrote:
> This is a fisrst patch from patch series which was
s/firsrst/first/
Although I don't think you have to say that this is the first patch :).
This is not useful for the commit message.
> developed to handle remote (external) processors
> memory management units. Remote processors are
> typically used for graphic rendering (GPUs) and
> high quality video decoding (IPUs). They are typically
> installed on such multimedia SoCs as OMAP4 / OMAP5.
>
> As soon as remoteprocessor MMU typically works with
> pagetables filled by physical addresses, which are
> allocated by domU kernel, it is almost impossible to
> use them under Xen, intermediate physical addresses
> allocated by kernel, need to be translated to machine
> addresses.
>
> This patch introduces a simple framework to perform
> pfn -> mfn translation for external MMUs.
> It introduces basic data structures and algorithms
> needed for translation.
>
> Typically, when MMU is configured, some it registers
> are updated by new values. Introduced frameworks
> uses traps as starting point of remoteproc MMUs
> pagetables translation.
>
> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
[..]
> +static struct mmu_info *mmu_list[] = {
> +};
> +
> +#define mmu_for_each(pfunc, data) \
> +({ \
> + u32 __i; \
> + int __res = 0; \
> + \
> + for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
> + { \
> + __res |= pfunc(mmu_list[__i], data); \
Using _res |= will result to a wrong errno at the end.
See the usage in iommu mmu_init_all.
I would use
__res = pfunc(...)
if ( !__res )
break;
And therefore modify mmu_check to return 1 if failing, 0 otherwise.
This will also avoid to continue to browse all the MMU for nothing.
> + } \
> + __res; \
> +})
> +
> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
> +{
> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start + mmu->mem_size)) )
> + return 1;
> +
> + return 0;
> +}
> +
> +static inline struct mmu_info *mmu_lookup(u32 addr)
> +{
> + u32 i;
> +
> + /* enumerate all registered MMU's and check is address in range */
> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
> + {
> + if ( mmu_check_mem_range(mmu_list[i], addr) )
> + return mmu_list[i];
> + }
> +
> + return NULL;
> +}
> +
> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
> +{
> + return mmu_for_each(mmu_check_mem_range, addr);
> +}
This solution leads any guest to access to the MMU and therefore program
it. If you plan to use for passthrough, you have to find a way to say
that a specific domain is able to use the MMU, maybe an hypercall.
Otherwise this is a security issue.
IIRC I have already raised this concern on V1 :). It would be nice if
you resolve it ASAP, because I suspect it will need some rework in the
way you handle MMNU.
> +static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
> +{
> + void __iomem *pagetable = NULL;
> + u32 maddr, i;
> +
> + ASSERT(mmu);
> + ASSERT(pgt);
> +
> + if ( !pgt->paddr )
> + return -EINVAL;
> +
> + /* pagetable size can be more than one page */
> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
> + {
> + /* lookup address where remoteproc pagetable is stored by kernel */
> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE, NULL);
> + if ( !maddr )
> + {
> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i * PAGE_SIZE);
> + return -EINVAL;
> + }
> +
> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
ioremap_* should only be used to map device memory. For the guest memory
you have to use copy_from_guest helper.
> +struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32 addr, bool is_maddr)
you have to use paddr_t for addr. The number of bit for the physical
address can be greater than 40 bits.
The remark is the same everywhere within this file.
[..]
> +static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
> +{
> + struct mmu_pagetable *pgt;
> + int res;
> +
> + /* lookup using machine address first */
> + pgt = mmu_pagetable_lookup(mmu, paddr, true);
> + if ( !pgt )
> + {
> + /* lookup using kernel physical address */
> + pgt = mmu_pagetable_lookup(mmu, paddr, false);
> + if ( !pgt )
> + {
> + /* if pagetable doesn't exists in lookup list - allocate it */
> + pgt = mmu_alloc_pagetable(mmu, paddr);
The function mmu_alloc_pagetable can return NULL. But you never check
the return value and dereference it just below.
> + }
> + }
> +
> + pgt->maddr = MMU_INVALID_ADDRESS;
[..]
> +static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
> +{
> + struct mmu_info *mmu = NULL;
> + unsigned long flags;
> + register_t *r;
> +
> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> +
> + ASSERT(r);
The ASSERT is pointless, select_user_reg will never return NULL.
[..]
> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
> +{
> + struct mmu_info *mmu = NULL;
> + unsigned long flags;
> + register_t *r;
> + u32 new_addr, val;
> +
> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> +
> + ASSERT(r);
Same here.
> + /* dom0 should not access remoteproc MMU */
> + if ( 0 == current->domain->domain_id )
> + return 1;
Why this restriction?
> + /* find corresponding MMU */
> + mmu = mmu_lookup(info->gpa);
> + if ( !mmu )
> + {
> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
> + return -EINVAL;
> + }
> +
> + ASSERT(v->domain == current->domain);
I guess this restriction is because you are using current in
mmu_trap_translate_pagetable?
So, the iohandler is usually call on the current VCPU, there is no need
to worry about it. Futhermore, I would pass the vcpu/domain in argument
of the next function.
> +static int mmu_init(struct mmu_info *mmu, u32 data)
> +{
> + ASSERT(mmu);
> + ASSERT(!mmu->mem_map);
> +
> + INIT_LIST_HEAD(&mmu->pagetables_list);
> +
> + /* map MMU memory */
> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
> + if ( !mmu->mem_map )
It looks like there is a hard tab here.
[..]
> +static int mmu_init_all(void)
> +{
> + int res;
> +
> + res = mmu_for_each(mmu_init, 0);
> + if ( res )
> + {
> + printk("%s error during init %d\n", __func__, res);
> + return res;
> + }
Hmmm... do_initcalls doesn't check the return value. How your code
behave we one of the MMU has not been initialized?
I think do_initcalls & co should check the return, but as it's the
common code I don't know how x86 respect this convention to return 0 if
succeded. Ian, Stefano, any thoughs?
[..]
> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
I think this file as
> new file mode 100644
> index 0000000..22e2951
> --- /dev/null
> +++ b/xen/include/xen/remoteproc_iommu.h
The remoteproc things is ARM specific, right? If so, this header should
be moved in include/asm-arm.
> +
> +#define MMU_INVALID_ADDRESS ((u32)(-1))
You should not assume that the MMU ADDRESS is always 32 bits. Please use
paddr_t here.
> +#define pr_mmu(fmt, ...) \
> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
Hmmm, you are assuming that mmu is existing within the function. You
should pass the mmu in parameter of this macro.
Also, most of the usage are for an error (except the one in
mmu_page_alloc). I would prefix it by XENLOG_ERROR.
> +struct pagetable_data {
> + /* 1st level translation */
> + u32 pgd_shift;
> + u32 pte_shift;
> + u32 super_shift;
> + u32 section_shift;
> + /* 2nd level translation */
> + u32 pte_large_shift;
> +};
No hard tab in Xen. Please remove them.
> +
> +struct mmu_pagetable {
> + u32 *hyp_pagetable;
> + u32 *kern_pagetable;
> + u32 paddr;
> + u32 maddr;
> + struct list_head link_node;
> + u32 page_counter;
> +};
Same here.
> +
> +struct mmu_info {
> + const char *name;
> + const struct pagetable_data *pg_data;
> + /* register where phys pointer to pagetable is stored */
> + u32 *trap_offsets;
> + paddr_t mem_start;
> + u32 mem_size;
> + spinlock_t lock;
> + struct list_head pagetables_list;
> + u32 num_traps;
> + void __iomem *mem_map;
> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
> + void (*print_pagetable_pfunc)(struct mmu_info *);
> +};
Same here.
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-06-26 11:07 ` [PATCH v02 1/7] " Andrii Tseglytskyi
2014-06-29 18:00 ` Julien Grall
@ 2014-07-04 13:59 ` Stefano Stabellini
2014-07-16 15:19 ` Ian Campbell
` (2 more replies)
2014-07-16 15:29 ` Ian Campbell
2 siblings, 3 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-07-04 13:59 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Julien Grall, Stefano Stabellini, Ian Campbell, xen-devel
On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> This is a fisrst patch from patch series which was
> developed to handle remote (external) processors
> memory management units. Remote processors are
> typically used for graphic rendering (GPUs) and
> high quality video decoding (IPUs). They are typically
> installed on such multimedia SoCs as OMAP4 / OMAP5.
>
> As soon as remoteprocessor MMU typically works with
> pagetables filled by physical addresses, which are
> allocated by domU kernel, it is almost impossible to
> use them under Xen, intermediate physical addresses
> allocated by kernel, need to be translated to machine
> addresses.
>
> This patch introduces a simple framework to perform
> pfn -> mfn translation for external MMUs.
> It introduces basic data structures and algorithms
> needed for translation.
>
> Typically, when MMU is configured, some it registers
> are updated by new values. Introduced frameworks
> uses traps as starting point of remoteproc MMUs
> pagetables translation.
>
> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
There is one problem with this patch: you need to find a way to "pin"
the p2m entries for the pfns and mfns found in the pagetables translated
by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
mfn mappings for the domain, breaking the pagetables already translated
by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
under the guest feet, but features like memory sharing or swapping,
supported on x86, could cause it to happen.
In the past I tried to introduce a way to "pin the mappings, but never
finished to upstream the patches, because it didn't need it anymore.
Give a look at:
http://marc.info/?l=xen-devel&m=138029864707973
In particular you would need to introduce something like
the pin function and call it from mmu_translate_pagetable.
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/remoteproc_iommu.c | 412 ++++++++++++++++++++++++++++++++++++
> xen/include/xen/remoteproc_iommu.h | 79 +++++++
> 3 files changed, 492 insertions(+)
> create mode 100644 xen/arch/arm/remoteproc_iommu.c
> create mode 100644 xen/include/xen/remoteproc_iommu.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index d70f6d5..0204d1c 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -15,6 +15,7 @@ obj-y += io.o
> obj-y += irq.o
> obj-y += kernel.o
> obj-y += mm.o
> +obj-y += remoteproc_iommu.o
> obj-y += p2m.o
> obj-y += percpu.o
> obj-y += guestcopy.o
> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
> new file mode 100644
> index 0000000..b4d22d9
> --- /dev/null
> +++ b/xen/arch/arm/remoteproc_iommu.c
> @@ -0,0 +1,412 @@
> +/*
> + * xen/arch/arm/remoteproc_iommu.c
> + *
> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> + * Copyright (c) 2014 GlobalLogic
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/config.h>
> +#include <xen/lib.h>
> +#include <xen/errno.h>
> +#include <xen/mm.h>
> +#include <xen/vmap.h>
> +#include <xen/init.h>
> +#include <xen/sched.h>
> +#include <xen/stdbool.h>
> +#include <asm/system.h>
> +#include <asm/current.h>
> +#include <asm/io.h>
> +#include <asm/p2m.h>
> +
> +#include <xen/remoteproc_iommu.h>
> +
> +#include "io.h"
> +
> +static struct mmu_info *mmu_list[] = {
> +};
> +
> +#define mmu_for_each(pfunc, data) \
> +({ \
> + u32 __i; \
> + int __res = 0; \
> + \
> + for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
> + { \
> + __res |= pfunc(mmu_list[__i], data); \
> + } \
> + __res; \
> +})
> +
> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
> +{
> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start + mmu->mem_size)) )
> + return 1;
> +
> + return 0;
> +}
> +
> +static inline struct mmu_info *mmu_lookup(u32 addr)
> +{
> + u32 i;
> +
> + /* enumerate all registered MMU's and check is address in range */
> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
> + {
> + if ( mmu_check_mem_range(mmu_list[i], addr) )
> + return mmu_list[i];
> + }
> +
> + return NULL;
> +}
> +
> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
> +{
> + return mmu_for_each(mmu_check_mem_range, addr);
> +}
> +
> +static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
> +{
> + void __iomem *pagetable = NULL;
> + u32 maddr, i;
> +
> + ASSERT(mmu);
> + ASSERT(pgt);
> +
> + if ( !pgt->paddr )
> + return -EINVAL;
> +
> + /* pagetable size can be more than one page */
> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
> + {
> + /* lookup address where remoteproc pagetable is stored by kernel */
> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE, NULL);
> + if ( !maddr )
> + {
> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i * PAGE_SIZE);
> + return -EINVAL;
> + }
> +
> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
> + if ( !pagetable )
> + {
> + pr_mmu("failed to map pagetable");
> + return -EINVAL;
> + }
> +
> + /* copy pagetable to hypervisor memory */
> + clean_and_invalidate_xen_dcache_va_range(pagetable, PAGE_SIZE);
> + memcpy((u32*)((u32)pgt->kern_pagetable + i * PAGE_SIZE), pagetable, PAGE_SIZE);
> +
> + iounmap(pagetable);
> + }
> +
> + return 0;
> +}
> +
> +struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32 addr, bool is_maddr)
> +{
> + struct mmu_pagetable *pgt;
> + u32 pgt_addr;
> +
> + list_for_each_entry(pgt, &mmu->pagetables_list, link_node)
> + {
> + if ( is_maddr )
> + pgt_addr = pgt->maddr;
> + else
> + pgt_addr = pgt->paddr;
> +
> + if ( pgt_addr == addr )
> + return pgt;
> + }
> +
> + return NULL;
> +}
> +
> +static struct mmu_pagetable *mmu_alloc_pagetable(struct mmu_info *mmu, u32 paddr)
> +{
> + struct mmu_pagetable *pgt;
> + u32 pgt_size = MMU_PGD_TABLE_SIZE(mmu);
> +
> + pgt = xzalloc_bytes(sizeof(struct mmu_pagetable));
> + if ( !pgt )
> + {
> + pr_mmu("failed to alloc pagetable structure");
> + return NULL;
> + }
> +
> + /* allocate pagetable managed by hypervisor */
> + pgt->hyp_pagetable = xzalloc_bytes(pgt_size);
> + if ( !pgt->hyp_pagetable )
> + {
> + pr_mmu("failed to alloc private hyp_pagetable");
> + return NULL;
> + }
> +
> + /* alocate pagetable for ipa storing */
> + pgt->kern_pagetable = xzalloc_bytes(pgt_size);
> + if ( !pgt->kern_pagetable )
> + {
> + pr_mmu("failed to alloc private kern_pagetable");
> + return NULL;
> + }
> +
> + pr_mmu("private pagetables for 0x%08x paddr %u bytes (main 0x%08x, temp 0x%08x)",
> + paddr, pgt_size, (u32)__pa(pgt->hyp_pagetable), (u32)__pa(pgt->kern_pagetable));
> +
> + pgt->paddr = paddr;
> +
> + list_add(&pgt->link_node, &mmu->pagetables_list);
> +
> + return pgt;
> +}
> +
> +static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
> +{
> + struct mmu_pagetable *pgt;
> + int res;
> +
> + /* lookup using machine address first */
> + pgt = mmu_pagetable_lookup(mmu, paddr, true);
> + if ( !pgt )
> + {
> + /* lookup using kernel physical address */
> + pgt = mmu_pagetable_lookup(mmu, paddr, false);
> + if ( !pgt )
> + {
> + /* if pagetable doesn't exists in lookup list - allocate it */
> + pgt = mmu_alloc_pagetable(mmu, paddr);
> + }
> + }
> +
> + pgt->maddr = MMU_INVALID_ADDRESS;
> +
> + /* copy pagetable from domain to hypervisor */
> + res = mmu_copy_pagetable(mmu, pgt);
> + if ( res )
> + return res;
> +
> + /* translate pagetable */
> + pgt->maddr = mmu->translate_pfunc(mmu, pgt);
> + return pgt->maddr;
> +}
> +
> +static u32 mmu_trap_translate_pagetable(struct mmu_info *mmu, mmio_info_t *info)
> +{
> + register_t *reg;
> + bool valid_trap = false;
> + u32 i, paddr;
> +
> + reg = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> +
> + ASSERT(reg);
> +
> + paddr = *reg;
> + if ( !paddr )
> + return MMU_INVALID_ADDRESS;
> +
> + /* check is the register is a valid TTB register */
> + for ( i = 0; i < mmu->num_traps; i++ )
> + {
> + if ( mmu->trap_offsets[i] == (info->gpa - mmu->mem_start) )
> + {
> + valid_trap = true;
> + break;
> + }
> + }
> +
> + if ( !valid_trap )
> + return MMU_INVALID_ADDRESS;
> +
> + return mmu_translate_pagetable(mmu, paddr);
> +}
> +
> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
> + u32 maddr, u32 hyp_addr)
Is this actually used anywhere?
> +{
> + u32 *pte_table = NULL, *hyp_pte_table = NULL, pte_table_size = PAGE_SIZE;
> + u32 i;
> +
> + /* remap second level translation table */
> + pte_table = ioremap_nocache(maddr, MMU_PTE_TABLE_SIZE(mmu));
> + if ( !pte_table )
> + {
> + pr_mmu("failed to map pte_table");
> + return MMU_INVALID_ADDRESS;
> + }
> +
> + /* allocate new second level pagetable once */
> + if ( 0 == hyp_addr )
> + {
> + if ( MMU_PTE_TABLE_SIZE(mmu) > PAGE_SIZE )
> + pte_table_size = MMU_PTE_TABLE_SIZE(mmu);
> +
> + hyp_pte_table = xzalloc_bytes(pte_table_size);
> + if ( !hyp_pte_table )
> + {
> + pr_mmu("failed to alloc new iopt");
> + return MMU_INVALID_ADDRESS;
> + }
> + }
> + else
> + {
> + hyp_pte_table = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
> + }
> +
> + /* 2-nd level translation */
> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
> + {
> + paddr_t pt_maddr, pt_paddr, pt_flags;
> + u32 pt_mask = MMU_SECTION_MASK(mmu->pg_data->pte_shift);
> +
> + if ( !pte_table[i] )
> + {
> + /* handle the case when page was removed */
> + if ( unlikely(hyp_pte_table[i]) )
> + {
> + hyp_pte_table[i] = 0;
> + }
> +
> + continue;
> + }
> +
> + pt_paddr = pte_table[i] & pt_mask;
> + pt_flags = pte_table[i] & ~pt_mask;
> + pt_maddr = p2m_lookup(current->domain, pt_paddr, NULL);
> + ASSERT(pt_maddr != INVALID_PADDR);
> +
> + hyp_pte_table[i] = pt_maddr | pt_flags;
> + pgt->page_counter++;
> + }
> +
> + iounmap(pte_table);
> +
> + clean_and_invalidate_xen_dcache_va_range(hyp_pte_table, MMU_PTE_TABLE_SIZE(mmu));
> + return __pa(hyp_pte_table);
> +}
> +
> +static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
> +{
> + struct mmu_info *mmu = NULL;
> + unsigned long flags;
> + register_t *r;
> +
> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> +
> + ASSERT(r);
If you check on the domid in mmu_mmio_write, I would check here too
> + mmu = mmu_lookup(info->gpa);
> + if ( !mmu )
> + {
> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
> + return -EINVAL;
> + }
> +
> + spin_lock_irqsave(&mmu->lock, flags);
> + *r = readl(mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
> + spin_unlock_irqrestore(&mmu->lock, flags);
> +
> + return 1;
It looks like you are returning the mfns here, is that correct?
> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
> +{
> + struct mmu_info *mmu = NULL;
> + unsigned long flags;
> + register_t *r;
> + u32 new_addr, val;
> +
> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> +
> + ASSERT(r);
> +
> + /* dom0 should not access remoteproc MMU */
> + if ( 0 == current->domain->domain_id )
> + return 1;
This is too specific to one particular configuration.
Would it be possible to generalize this somehow? At the very least you
could introduce an XSM label to access the pagetables, so that you can
dynamically configure the domains the can write to them.
> + /* find corresponding MMU */
> + mmu = mmu_lookup(info->gpa);
> + if ( !mmu )
> + {
> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
> + return -EINVAL;
> + }
> +
> + ASSERT(v->domain == current->domain);
You can remove this assert.
> + spin_lock_irqsave(&mmu->lock, flags);
> +
> + /* get new address of translated pagetable */
> + new_addr = mmu_trap_translate_pagetable(mmu, info);
> + if ( MMU_INVALID_ADDRESS != new_addr )
> + val = new_addr;
> + else
> + val = *r;
> +
> + writel(val, mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
> + spin_unlock_irqrestore(&mmu->lock, flags);
> +
> + return 1;
> +}
> +
> +static int mmu_init(struct mmu_info *mmu, u32 data)
> +{
> + ASSERT(mmu);
> + ASSERT(!mmu->mem_map);
> +
> + INIT_LIST_HEAD(&mmu->pagetables_list);
> +
> + /* map MMU memory */
> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
> + if ( !mmu->mem_map )
> + {
> + pr_mmu("failed to map memory");
> + return -EINVAL;
> + }
> +
> + pr_mmu("memory map = 0x%pS", _p(mmu->mem_map));
> +
> + spin_lock_init(&mmu->lock);
> +
> + return 0;
> +}
> +
> +static int mmu_init_all(void)
> +{
> + int res;
> +
> + res = mmu_for_each(mmu_init, 0);
> + if ( res )
> + {
> + printk("%s error during init %d\n", __func__, res);
> + return res;
> + }
> +
> + return 0;
> +}
> +
> +const struct mmio_handler remoteproc_mmio_handler = {
> + .check_handler = mmu_mmio_check,
> + .read_handler = mmu_mmio_read,
> + .write_handler = mmu_mmio_write,
> +};
> +
> +__initcall(mmu_init_all);
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
> new file mode 100644
> index 0000000..22e2951
> --- /dev/null
> +++ b/xen/include/xen/remoteproc_iommu.h
> @@ -0,0 +1,79 @@
> +/*
> + * xen/include/xen/remoteproc_iommu.h
> + *
> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> + * Copyright (c) 2014 GlobalLogic
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef _REMOTEPROC_IOMMU_H_
> +#define _REMOTEPROC_IOMMU_H_
> +
> +#include <asm/types.h>
> +
> +#define MMU_SECTION_SIZE(shift) (1UL << (shift))
> +#define MMU_SECTION_MASK(shift) (~(MMU_SECTION_SIZE(shift) - 1))
> +
> +/* 4096 first level descriptors for "supersection" and "section" */
> +#define MMU_PTRS_PER_PGD(mmu) (1UL << (32 - (mmu->pg_data->pgd_shift)))
> +#define MMU_PGD_TABLE_SIZE(mmu) (MMU_PTRS_PER_PGD(mmu) * sizeof(u32))
> +
> +/* 256 second level descriptors for "small" and "large" pages */
> +#define MMU_PTRS_PER_PTE(mmu) (1UL << ((mmu->pg_data->pgd_shift) - (mmu->pg_data->pte_shift)))
> +#define MMU_PTE_TABLE_SIZE(mmu) (MMU_PTRS_PER_PTE(mmu) * sizeof(u32))
> +
> +/* 16 sections in supersection */
> +#define MMU_SECTION_PER_SUPER(mmu) (1UL << ((mmu->pg_data->super_shift) - (mmu->pg_data->section_shift)))
> +
> +#define MMU_INVALID_ADDRESS ((u32)(-1))
> +
> +#define pr_mmu(fmt, ...) \
> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
> +
> +struct pagetable_data {
> + /* 1st level translation */
> + u32 pgd_shift;
> + u32 pte_shift;
> + u32 super_shift;
> + u32 section_shift;
> + /* 2nd level translation */
> + u32 pte_large_shift;
> +};
> +
> +struct mmu_pagetable {
> + u32 *hyp_pagetable;
> + u32 *kern_pagetable;
> + u32 paddr;
> + u32 maddr;
> + struct list_head link_node;
> + u32 page_counter;
> +};
> +
> +struct mmu_info {
> + const char *name;
> + const struct pagetable_data *pg_data;
> + /* register where phys pointer to pagetable is stored */
> + u32 *trap_offsets;
> + paddr_t mem_start;
> + u32 mem_size;
> + spinlock_t lock;
> + struct list_head pagetables_list;
> + u32 num_traps;
> + void __iomem *mem_map;
> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
> + void (*print_pagetable_pfunc)(struct mmu_info *);
> +};
> +
> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
> + u32 maddr, u32 hyp_addr);
> +
> +#endif /* _REMOTEPROC_IOMMU_H_ */
> --
> 1.7.9.5
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc
2014-06-26 11:07 ` [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc Andrii Tseglytskyi
@ 2014-07-04 14:01 ` Stefano Stabellini
2014-07-22 16:56 ` Andrii Tseglytskyi
2014-07-04 14:30 ` Julien Grall
2014-07-16 15:36 ` Ian Campbell
2 siblings, 1 reply; 57+ messages in thread
From: Stefano Stabellini @ 2014-07-04 14:01 UTC (permalink / raw)
To: Andrii Tseglytskyi; +Cc: xen-devel
On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> The following patch introduced platform specific MMU data
> definitions and pagetable translation function for OMAP5 IPU
> remoteproc. This MMU is a bit specific - it typically performs
> one level translation and map a big chunks of memory. 16 Mb
> supersections and 1 Mb sections are mapped instead of 4 Kb pages.
> Introduced algorithm performs internal remapping of big sections
> to small 4 Kb pages.
>
> Change-Id: If20449f07e22f780e1fded67fed4f79cbe1fc156
> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> ---
> xen/arch/arm/platforms/Makefile | 1 +
> xen/arch/arm/platforms/omap_iommu.c | 247 +++++++++++++++++++++++++++++++++++
> xen/arch/arm/remoteproc_iommu.c | 1 +
> xen/include/xen/remoteproc_iommu.h | 2 +
> 4 files changed, 251 insertions(+)
> create mode 100644 xen/arch/arm/platforms/omap_iommu.c
>
> diff --git a/xen/arch/arm/platforms/Makefile b/xen/arch/arm/platforms/Makefile
> index 080ea9a..f224f08 100644
> --- a/xen/arch/arm/platforms/Makefile
> +++ b/xen/arch/arm/platforms/Makefile
> @@ -4,4 +4,5 @@ obj-$(CONFIG_ARM_32) += midway.o
> obj-$(CONFIG_ARM_32) += omap5.o
> obj-$(CONFIG_ARM_32) += dra7xx.o
> obj-$(CONFIG_ARM_32) += sunxi.o
> +obj-$(CONFIG_ARM_32) += omap_iommu.o
> obj-$(CONFIG_ARM_64) += xgene-storm.o
> diff --git a/xen/arch/arm/platforms/omap_iommu.c b/xen/arch/arm/platforms/omap_iommu.c
> new file mode 100644
> index 0000000..e0c4633
> --- /dev/null
> +++ b/xen/arch/arm/platforms/omap_iommu.c
> @@ -0,0 +1,247 @@
> +/*
> + * xen/arch/arm/platforms/omap_iommu.c
> + *
> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> + * Copyright (c) 2014 GlobalLogic
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/lib.h>
> +#include <xen/errno.h>
> +#include <xen/stdbool.h>
> +#include <xen/mm.h>
> +#include <xen/vmap.h>
> +#include <xen/sched.h>
> +#include <xen/remoteproc_iommu.h>
> +
> +#include <asm/p2m.h>
> +
> +/*
> + * "L2 table" address mask and size definitions.
> + */
> +
> +/* register where address of pagetable is stored */
> +#define MMU_IPU_TTB_OFFSET 0x4c
> +
> +/* 1st level translation */
> +#define MMU_OMAP_PGD_SHIFT 20
> +#define MMU_OMAP_SUPER_SHIFT 24 /* "supersection" - 16 Mb */
> +#define MMU_OMAP_SECTION_SHIFT 20 /* "section" - 1 Mb */
> +#define MMU_OMAP_SECOND_LEVEL_SHIFT 10
> +
> +/* 2nd level translation */
> +#define MMU_OMAP_PTE_SMALL_SHIFT 12 /* "small page" - 4Kb */
> +#define MMU_OMAP_PTE_LARGE_SHIFT 16 /* "large page" - 64 Kb */
> +
> +/*
> + * some descriptor attributes.
> + */
> +#define PGD_TABLE (1 << 0)
> +#define PGD_SECTION (2 << 0)
> +#define PGD_SUPER (1 << 18 | 2 << 0)
> +
> +#define ipu_pgd_is_table(x) (((x) & 3) == PGD_TABLE)
> +#define ipu_pgd_is_section(x) (((x) & (1 << 18 | 3)) == PGD_SECTION)
> +#define ipu_pgd_is_super(x) (((x) & (1 << 18 | 3)) == PGD_SUPER)
> +
> +#define PTE_SMALL (2 << 0)
> +#define PTE_LARGE (1 << 0)
> +
> +#define OMAP_IPU_MMU_MEM_BASE 0x55082000
> +
> +static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
> +
> +static u32 ipu_trap_offsets[] = {
> + MMU_IPU_TTB_OFFSET,
> +};
> +
> +static const struct pagetable_data pagetable_ipu_data = {
> + .pgd_shift = MMU_OMAP_PGD_SHIFT,
> + .super_shift = MMU_OMAP_SUPER_SHIFT,
> + .section_shift = MMU_OMAP_SECTION_SHIFT,
> + .pte_shift = MMU_OMAP_PTE_SMALL_SHIFT,
> + .pte_large_shift = MMU_OMAP_PTE_LARGE_SHIFT,
> +};
> +
> +struct mmu_info omap_ipu_mmu = {
> + .name = "IPU_L2_MMU",
> + .pg_data = &pagetable_ipu_data,
> + .trap_offsets = ipu_trap_offsets,
> + .mem_start = OMAP_IPU_MMU_MEM_BASE,
> + .mem_size = 0x1000,
> + .num_traps = ARRAY_SIZE(ipu_trap_offsets),
> + .translate_pfunc = mmu_ipu_translate_pagetable,
> +};
> +
> +static bool translate_supersections_to_pages = true;
> +static bool translate_sections_to_pages = true;
> +
> +static u32 mmu_pte_table_alloc(struct mmu_info *mmu, u32 pgd, u32 sect_num,
> + struct mmu_pagetable *pgt, u32 hyp_addr)
> +{
> + u32 *pte = NULL;
> + u32 i;
> +
> + /* allocate pte table once */
> + if ( 0 == hyp_addr )
> + {
> + pte = xzalloc_bytes(PAGE_SIZE);
> + if ( !pte )
> + {
> + pr_mmu("failed to alloc 2nd level table");
> + return 0;
> + }
> + }
> + else
> + {
> + pte = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
> + }
> +
> + ASSERT(256 == MMU_PTRS_PER_PTE(mmu));
> +
> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
> + {
> + u32 paddr, maddr;
> +
> + paddr = pgd + (i * PAGE_SIZE);
> + maddr = p2m_lookup(current->domain, paddr, NULL);
Here is where you would need to make sure that paddr->maddr doesn't
change in the future by pinning the mapping.
> + ASSERT(maddr != INVALID_PADDR);
> +
> + pte[i] = maddr | PTE_SMALL;
> + pgt->page_counter++;
> + }
> +
> + clean_and_invalidate_xen_dcache_va_range(pte, PAGE_SIZE);
> + return __pa(pte) | PGD_TABLE;
> +}
> +
> +static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
> +{
> + u32 *kern_pgt, *hyp_pgt;
> + const struct pagetable_data *data;
> + u32 i;
> +
> + ASSERT(mmu);
> + ASSERT(pgt);
> +
> + data = mmu->pg_data;
> + kern_pgt = pgt->kern_pagetable;
> + hyp_pgt = pgt->hyp_pagetable;
> + pgt->page_counter = 0;
> +
> + ASSERT(4096 == MMU_PTRS_PER_PGD(mmu));
> +
> + /* 1-st level translation */
> + for ( i = 0; i < MMU_PTRS_PER_PGD(mmu); i++ )
> + {
> + paddr_t pd_maddr, pd_paddr, pd_flags;
> + u32 pd_mask;
> + u32 pgd_tmp, pgd = kern_pgt[i];
> +
> + if ( !pgd )
> + {
> + /* handle the case when second level translation table
> + * was removed from kernel */
> + if ( unlikely(hyp_pgt[i]) )
> + {
> + xfree(__va(hyp_pgt[i] & MMU_SECTION_MASK(MMU_OMAP_SECOND_LEVEL_SHIFT)));
> + hyp_pgt[i] = 0;
> + }
> +
> + continue;
> + }
> +
> + /* first level pointers have different formats, depending on their type */
> + if ( ipu_pgd_is_super(pgd) )
> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SUPER_SHIFT);
> + else if ( ipu_pgd_is_section(pgd) )
> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECTION_SHIFT);
> + else if ( ipu_pgd_is_table(pgd) )
> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECOND_LEVEL_SHIFT);
> +
> + pd_paddr = pgd & pd_mask;
> + pd_flags = pgd & ~pd_mask;
> + pd_maddr = p2m_lookup(current->domain, pd_paddr, NULL);
> + ASSERT(pd_maddr != INVALID_PADDR);
> +
> + /* "supersection" 16 Mb */
> + if ( ipu_pgd_is_super(pgd) )
> + {
> + /* mapping of 16 Mb chunk is fragmented to 4 Kb pages */
> + if( likely(translate_supersections_to_pages) )
> + {
> + u32 j;
> +
> + ASSERT(16 == MMU_SECTION_PER_SUPER(mmu));
> + ASSERT(1048576 == MMU_SECTION_SIZE(data->section_shift));
> +
> + /* 16 Mb supersection is divided to 16 sections of 1 MB size */
> + for ( j = 0 ; j < MMU_SECTION_PER_SUPER(mmu); j++ )
> + {
> + pgd_tmp = (pgd & ~PGD_SUPER) + (j * MMU_SECTION_SIZE(data->section_shift));
> + hyp_pgt[i + j] = mmu_pte_table_alloc(mmu, pgd_tmp, i, pgt, hyp_pgt[i + j]);
> + }
> +
> + /* move counter after supersection is translated */
> + i += (j - 1);
> + }
> + else
> + {
> + hyp_pgt[i] = pd_maddr | pd_flags;
> + }
> +
> + /* "section" 1Mb */
> + }
> + else if ( ipu_pgd_is_section(pgd) )
> + {
> + if ( likely(translate_sections_to_pages) )
> + {
> + pgd_tmp = (pgd & ~PGD_SECTION);
> + hyp_pgt[i] = mmu_pte_table_alloc(mmu, pgd_tmp, i, pgt, hyp_pgt[i]);
> + }
> + else
> + {
> + hyp_pgt[i] = pd_maddr | pd_flags;
> + }
> +
> + /* "table" */
> + }
> + else if ( unlikely(ipu_pgd_is_table(pgd)) )
> + {
> + ASSERT(4096 == MMU_PTRS_PER_PGD(mmu));
> + ASSERT(256 == MMU_PTRS_PER_PTE(mmu));
> +
> + hyp_pgt[i] = mmu_translate_second_level(mmu, pgt, pd_maddr, hyp_pgt[i]);
> + hyp_pgt[i] |= pd_flags;
> +
> + /* error */
> + }
> + else
> + {
> + pr_mmu("unknown entry %u: 0x%08x", i, pgd);
> + return MMU_INVALID_ADDRESS;
> + }
> + }
> +
> + /* force omap IOMMU to use new pagetable */
> + clean_and_invalidate_xen_dcache_va_range(hyp_pgt, MMU_PGD_TABLE_SIZE(mmu));
> + return __pa(hyp_pgt);
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
> index b4d22d9..8291f3f 100644
> --- a/xen/arch/arm/remoteproc_iommu.c
> +++ b/xen/arch/arm/remoteproc_iommu.c
> @@ -33,6 +33,7 @@
> #include "io.h"
>
> static struct mmu_info *mmu_list[] = {
> + &omap_ipu_mmu,
> };
>
> #define mmu_for_each(pfunc, data) \
> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
> index 22e2951..ff1c439 100644
> --- a/xen/include/xen/remoteproc_iommu.h
> +++ b/xen/include/xen/remoteproc_iommu.h
> @@ -76,4 +76,6 @@ struct mmu_info {
> u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
> u32 maddr, u32 hyp_addr);
>
> +extern struct mmu_info omap_ipu_mmu;
> +
> #endif /* _REMOTEPROC_IOMMU_H_ */
> --
> 1.7.9.5
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-06-26 11:07 ` [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall Andrii Tseglytskyi
@ 2014-07-04 14:05 ` Stefano Stabellini
2014-07-16 15:42 ` Ian Campbell
2014-07-22 16:37 ` Andrii Tseglytskyi
2014-07-04 14:35 ` Julien Grall
1 sibling, 2 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-07-04 14:05 UTC (permalink / raw)
To: Andrii Tseglytskyi; +Cc: xen-devel
On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> The reason of the patch is the following - some remoteprocs
> are quite complicated, and their MMUs can handle several
> pagetables. Good example is a OMAP5 GPU, which allocates
> several pagetables during it work. Additional requirement
> is that not all pagetable physical addresses are stored
> in MMU registers. Some pagetables may be allocated and
> then their physical addresses are sent to GPU using private
> message loop between GPU kernel driver and GPU remoteproc.
>
> Patch is developed to handle this. At any moment of time
> kernel can perform translation of such pagetables, before
> sending their addresses to GPU remoteproc.
>
> Change-Id: Ie84012163205c3a2f920dc4cf07327a244647c93
> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> ---
> xen/arch/arm/remoteproc_iommu.c | 22 ++++++++++++++++++++++
> xen/arch/arm/traps.c | 1 +
> xen/include/public/xen.h | 1 +
> xen/include/xen/hypercall.h | 12 ++++++++++++
> 4 files changed, 36 insertions(+)
>
> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
> index 3b3da3b..316ceb4 100644
> --- a/xen/arch/arm/remoteproc_iommu.c
> +++ b/xen/arch/arm/remoteproc_iommu.c
> @@ -23,6 +23,8 @@
> #include <xen/init.h>
> #include <xen/sched.h>
> #include <xen/stdbool.h>
> +#include <xen/hypercall.h>
> +#include <xen/guest_access.h>
> #include <asm/system.h>
> #include <asm/current.h>
> #include <asm/io.h>
> @@ -382,6 +384,26 @@ static int mmu_init(struct mmu_info *mmu, u32 data)
> return 0;
> }
>
> +long do_translate_pagetable(int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
> +{
> + struct xen_pagetable_addr pgt;
> + struct mmu_info *mmu = NULL;
> +
> + if ( copy_from_guest(&pgt, pgt_addr, 1) )
> + return -EFAULT;
> +
> + mmu = mmu_lookup(pgt.reg);
> + if ( !mmu )
> + {
> + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
> + return -EINVAL;
> + }
> +
> + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
> +
> + return copy_to_guest(pgt_addr, &pgt, 1);
> +}
> +
> static int mmu_init_all(void)
> {
> int res;
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 21c7b26..05b5184 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -1012,6 +1012,7 @@ static arm_hypercall_t arm_hypercall_table[] = {
> HYPERCALL(hvm_op, 2),
> HYPERCALL(grant_table_op, 3),
> HYPERCALL_ARM(vcpu_op, 3),
> + HYPERCALL(translate_pagetable, 2),
> };
>
> typedef int (*arm_psci_fn_t)(uint32_t, register_t);
> diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> index 8c5697e..91ca6a1 100644
> --- a/xen/include/public/xen.h
> +++ b/xen/include/public/xen.h
> @@ -101,6 +101,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);
> #define __HYPERVISOR_kexec_op 37
> #define __HYPERVISOR_tmem_op 38
> #define __HYPERVISOR_xc_reserved_op 39 /* reserved for XenClient */
> +#define __HYPERVISOR_translate_pagetable 40
It might be best to introduce this hypercall as an arch memory op
(xen/arch/arm/mm.c:arch_memory_op), rather than a full blown new
hypercall. Otherwise you'll need to provide an empty stub implementation
for x86.
> /* Architecture-specific hypercall definitions. */
> #define __HYPERVISOR_arch_0 48
> diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
> index a9e5229..a025435 100644
> --- a/xen/include/xen/hypercall.h
> +++ b/xen/include/xen/hypercall.h
> @@ -136,6 +136,18 @@ extern long
> do_tmem_op(
> XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);
>
> +struct xen_pagetable_addr {
> + u32 reg;
> + u32 paddr;
> + u32 maddr;
Could you please comment what these fields are for exactly?
> +};
> +typedef struct xen_pagetable_addr xen_pagetable_addr_t;
> +DEFINE_XEN_GUEST_HANDLE(xen_pagetable_addr_t);
> +
> +extern long
> +do_translate_pagetable(
> + int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr);
> +
> extern long
> do_xenoprof_op(int op, XEN_GUEST_HANDLE_PARAM(void) arg);
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc
2014-06-26 11:07 ` [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc Andrii Tseglytskyi
2014-07-04 14:01 ` Stefano Stabellini
@ 2014-07-04 14:30 ` Julien Grall
2014-07-22 16:58 ` Andrii Tseglytskyi
2014-07-16 15:36 ` Ian Campbell
2 siblings, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-07-04 14:30 UTC (permalink / raw)
To: Andrii Tseglytskyi, xen-devel
Hi Andrii,
On 26/06/14 12:07, Andrii Tseglytskyi wrote:
> +static u32 mmu_pte_table_alloc(struct mmu_info *mmu, u32 pgd, u32 sect_num,
> + struct mmu_pagetable *pgt, u32 hyp_addr)
> +{
> + u32 *pte = NULL;
> + u32 i;
> +
> + /* allocate pte table once */
> + if ( 0 == hyp_addr )
> + {
> + pte = xzalloc_bytes(PAGE_SIZE);
> + if ( !pte )
> + {
> + pr_mmu("failed to alloc 2nd level table");
> + return 0;
> + }
> + }
> + else
> + {
> + pte = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
> + }
> +
> + ASSERT(256 == MMU_PTRS_PER_PTE(mmu));
> +
> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
> + {
> + u32 paddr, maddr;
> +
> + paddr = pgd + (i * PAGE_SIZE);
> + maddr = p2m_lookup(current->domain, paddr, NULL);
> + ASSERT(maddr != INVALID_PADDR);
Hmmm, what prevents the guest to pass an invalid IPA? On debug build you
will hit the assert, but on non-debug build you will screw the IPU page
table.
[..]
> +static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
> +{
[..]
> + /* first level pointers have different formats, depending on their type */
> + if ( ipu_pgd_is_super(pgd) )
> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SUPER_SHIFT);
> + else if ( ipu_pgd_is_section(pgd) )
> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECTION_SHIFT);
> + else if ( ipu_pgd_is_table(pgd) )
> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECOND_LEVEL_SHIFT);
> +
> + pd_paddr = pgd & pd_mask;
> + pd_flags = pgd & ~pd_mask;
> + pd_maddr = p2m_lookup(current->domain, pd_paddr, NULL);
> + ASSERT(pd_maddr != INVALID_PADDR);
Same remark here.
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-06-26 11:07 ` [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall Andrii Tseglytskyi
2014-07-04 14:05 ` Stefano Stabellini
@ 2014-07-04 14:35 ` Julien Grall
2014-07-16 15:43 ` Ian Campbell
2014-07-22 16:39 ` Andrii Tseglytskyi
1 sibling, 2 replies; 57+ messages in thread
From: Julien Grall @ 2014-07-04 14:35 UTC (permalink / raw)
To: Andrii Tseglytskyi, xen-devel, Ian Campbell, Stefano Stabellini
Hi Andrii,
On 26/06/14 12:07, Andrii Tseglytskyi wrote:
> +long do_translate_pagetable(int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
> +{
> + struct xen_pagetable_addr pgt;
> + struct mmu_info *mmu = NULL;
> +
> + if ( copy_from_guest(&pgt, pgt_addr, 1) )
> + return -EFAULT;
> +
> + mmu = mmu_lookup(pgt.reg);
> + if ( !mmu )
> + {
> + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
> + return -EINVAL;
> + }
> +
> + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
> +
> + return copy_to_guest(pgt_addr, &pgt, 1);
> +}
> +
AFAIU, nothing prevents a malicious guest to call this hypercall and
screw the pagetable of the MMU.
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-04 13:59 ` Stefano Stabellini
@ 2014-07-16 15:19 ` Ian Campbell
2014-07-22 12:42 ` Stefano Stabellini
2014-07-22 15:40 ` Andrii Tseglytskyi
2014-07-22 15:32 ` Andrii Tseglytskyi
2014-08-20 19:40 ` Andrii Tseglytskyi
2 siblings, 2 replies; 57+ messages in thread
From: Ian Campbell @ 2014-07-16 15:19 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Julien Grall, xen-devel, Andrii Tseglytskyi
On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> > This is a fisrst patch from patch series which was
> > developed to handle remote (external) processors
> > memory management units. Remote processors are
> > typically used for graphic rendering (GPUs) and
> > high quality video decoding (IPUs). They are typically
> > installed on such multimedia SoCs as OMAP4 / OMAP5.
> >
> > As soon as remoteprocessor MMU typically works with
> > pagetables filled by physical addresses, which are
> > allocated by domU kernel, it is almost impossible to
> > use them under Xen, intermediate physical addresses
> > allocated by kernel, need to be translated to machine
> > addresses.
> >
> > This patch introduces a simple framework to perform
> > pfn -> mfn translation for external MMUs.
> > It introduces basic data structures and algorithms
> > needed for translation.
> >
> > Typically, when MMU is configured, some it registers
> > are updated by new values. Introduced frameworks
> > uses traps as starting point of remoteproc MMUs
> > pagetables translation.
> >
> > Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
> > Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>
> There is one problem with this patch: you need to find a way to "pin"
> the p2m entries for the pfns and mfns found in the pagetables translated
> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
> mfn mappings for the domain, breaking the pagetables already translated
> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
> under the guest feet, but features like memory sharing or swapping,
> supported on x86, could cause it to happen.
I've not fully grokked this patch but wouldn't it be easier and better
to have a hook to cause the remoteproc to throw away and rebuild its
mappings when this happens? It could be called from e.g. the TLB flush
associated with the p2m changing, I think.
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-06-26 11:07 ` [PATCH v02 1/7] " Andrii Tseglytskyi
2014-06-29 18:00 ` Julien Grall
2014-07-04 13:59 ` Stefano Stabellini
@ 2014-07-16 15:29 ` Ian Campbell
2014-07-16 15:34 ` Ian Campbell
2014-07-22 16:14 ` Andrii Tseglytskyi
2 siblings, 2 replies; 57+ messages in thread
From: Ian Campbell @ 2014-07-16 15:29 UTC (permalink / raw)
To: Andrii Tseglytskyi; +Cc: xen-devel
On Thu, 2014-06-26 at 14:07 +0300, Andrii Tseglytskyi wrote:
> This is a fisrst patch from patch series which was
> developed to handle remote (external) processors
> memory management units. Remote processors are
> typically used for graphic rendering (GPUs) and
> high quality video decoding (IPUs). They are typically
> installed on such multimedia SoCs as OMAP4 / OMAP5.
>
> As soon as remoteprocessor MMU typically works with
> pagetables filled by physical addresses, which are
> allocated by domU kernel, it is almost impossible to
> use them under Xen, intermediate physical addresses
> allocated by kernel, need to be translated to machine
> addresses.
>
> This patch introduces a simple framework to perform
> pfn -> mfn translation for external MMUs.
> It introduces basic data structures and algorithms
> needed for translation.
>
> Typically, when MMU is configured, some it registers
> are updated by new values. Introduced frameworks
> uses traps as starting point of remoteproc MMUs
> pagetables translation.
Just to clarify the model:
This is intended as a framework to enable Xen to provide an SMMU to the
guest. Is the intention that this SMMU be an idealised PV thing or is it
intending to emulate a real SMMU (and therefore the guests existing
drivers)? I think this is the second one.
And the intention here is to let the guest manipulate the page tables
directly and shadow them from IPA->MA as opposed to e.g. exposing the
guest's complete IPA->MA mapping to the remoteproc.
I suppose either way my main concern is going to be the potential for
proliferation of backends (and perhaps frontends) for all sorts of
different H/W MMUs. I suppose that is unavoidable though.
WRT the shadowing, I don't see the bit which causes the guest pages
which it is using as PT page for the remoteproc to get marked R/O to
force a trap. Perhaps that comes later.
Do these remoteproc page tables get frequently updated at run time or
are they mostly just build once and forget?
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-16 15:29 ` Ian Campbell
@ 2014-07-16 15:34 ` Ian Campbell
2014-07-22 16:24 ` Andrii Tseglytskyi
2014-07-22 16:14 ` Andrii Tseglytskyi
1 sibling, 1 reply; 57+ messages in thread
From: Ian Campbell @ 2014-07-16 15:34 UTC (permalink / raw)
To: Andrii Tseglytskyi; +Cc: xen-devel
On Wed, 2014-07-16 at 16:29 +0100, Ian Campbell wrote:
> WRT the shadowing, I don't see the bit which causes the guest pages
> which it is using as PT page for the remoteproc to get marked R/O to
> force a trap. Perhaps that comes later.
>
> Do these remoteproc page tables get frequently updated at run time or
> are they mostly just build once and forget?
Reading the next patch I'm concluding that the translation happens each
time you write to the MMU's base pointer. IOW every update requires you
to rewrite that pointer, is that right?
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc
2014-06-26 11:07 ` [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc Andrii Tseglytskyi
2014-07-04 14:01 ` Stefano Stabellini
2014-07-04 14:30 ` Julien Grall
@ 2014-07-16 15:36 ` Ian Campbell
2014-07-22 17:16 ` Andrii Tseglytskyi
2 siblings, 1 reply; 57+ messages in thread
From: Ian Campbell @ 2014-07-16 15:36 UTC (permalink / raw)
To: Andrii Tseglytskyi; +Cc: xen-devel
On Thu, 2014-06-26 at 14:07 +0300, Andrii Tseglytskyi wrote:
> The following patch introduced platform specific MMU data
> definitions and pagetable translation function for OMAP5 IPU
> remoteproc. This MMU is a bit specific - it typically performs
> one level translation and map a big chunks of memory. 16 Mb
> supersections and 1 Mb sections are mapped instead of 4 Kb pages.
> Introduced algorithm performs internal remapping of big sections
> to small 4 Kb pages.
How does that work if the MMU only supports 1MB/16MB supersections? Or
have I misunderstood?
> Change-Id: If20449f07e22f780e1fded67fed4f79cbe1fc156
> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> ---
> xen/arch/arm/platforms/Makefile | 1 +
> xen/arch/arm/platforms/omap_iommu.c | 247 +++++++++++++++++++++++++++++++++++
I don't think this is the right home for this.
I think either xen/arch/arm/remoteproc/* or xen/drivers/remoteproc/*
would be more appropriate.
> +#define OMAP_IPU_MMU_MEM_BASE 0x55082000
> +
> +static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
> +
> +static u32 ipu_trap_offsets[] = {
> + MMU_IPU_TTB_OFFSET,
How large is this region? I think the machinery needs to be told.
Also if this functionality is to be used by guests then it can't really
use the h/w base address, you'd need to define a region of guest address
map (!= host/dom0 address map) for it.
> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
> index b4d22d9..8291f3f 100644
> --- a/xen/arch/arm/remoteproc_iommu.c
> +++ b/xen/arch/arm/remoteproc_iommu.c
> @@ -33,6 +33,7 @@
> #include "io.h"
>
> static struct mmu_info *mmu_list[] = {
> + &omap_ipu_mmu,
This suggests there is exactly one such and it is exposed to every
domain.
Wouldn't this rather be dynamic and per domain?
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 4/7] arm: omap: introduce print pagetable function for IPU remoteproc
2014-06-26 11:07 ` [PATCH v02 4/7] arm: omap: introduce print pagetable function for IPU remoteproc Andrii Tseglytskyi
@ 2014-07-16 15:38 ` Ian Campbell
2014-07-22 16:55 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Ian Campbell @ 2014-07-16 15:38 UTC (permalink / raw)
To: Andrii Tseglytskyi; +Cc: xen-devel
On Thu, 2014-06-26 at 14:07 +0300, Andrii Tseglytskyi wrote:
> This patch adds a possibility to dump all pagetables of
> IPU remoteproc. The only reason to have this patch - is a
> low level debug.
It dumps the whole thing? Rather than the translation of a specific
(I)PA?
Isn't that rather, er, verbose?
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-04 14:05 ` Stefano Stabellini
@ 2014-07-16 15:42 ` Ian Campbell
2014-07-22 16:47 ` Andrii Tseglytskyi
2014-07-22 16:37 ` Andrii Tseglytskyi
1 sibling, 1 reply; 57+ messages in thread
From: Ian Campbell @ 2014-07-16 15:42 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, Andrii Tseglytskyi
On Fri, 2014-07-04 at 15:05 +0100, Stefano Stabellini wrote:
> > diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> > index 8c5697e..91ca6a1 100644
> > --- a/xen/include/public/xen.h
> > +++ b/xen/include/public/xen.h
> > @@ -101,6 +101,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);
> > #define __HYPERVISOR_kexec_op 37
> > #define __HYPERVISOR_tmem_op 38
> > #define __HYPERVISOR_xc_reserved_op 39 /* reserved for XenClient */
> > +#define __HYPERVISOR_translate_pagetable 40
>
> It might be best to introduce this hypercall as an arch memory op
> (xen/arch/arm/mm.c:arch_memory_op), rather than a full blown new
> hypercall.
Or physdev op perhaps?
> Otherwise you'll need to provide an empty stub implementation
> for x86.
All unused hypercalls -ENOSYS automatically without stubs I think.
Are there any security concerns with exposing machine addresses to
guests? There are certainly "breaking the abstraction" concerns, but
that's unavoidable here I think.
> > /* Architecture-specific hypercall definitions. */
> > #define __HYPERVISOR_arch_0 48
> > diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
> > index a9e5229..a025435 100644
> > --- a/xen/include/xen/hypercall.h
> > +++ b/xen/include/xen/hypercall.h
> > @@ -136,6 +136,18 @@ extern long
> > do_tmem_op(
> > XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);
> >
> > +struct xen_pagetable_addr {
> > + u32 reg;
> > + u32 paddr;
> > + u32 maddr;
>
> Could you please comment what these fields are for exactly?
And mark them as IN/OUT as appropriate please.
Also the physical and machine addresses should always be 64 bits for
compatibility with future larger chips.
Having done that you then need padding after reg or to reorder things to
avoid holes on 64-bit. Or maybe reg should be 64-bit, depends what it
is. Either way the 32-bit and 64-bit layout of this struct should be
identical.
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-04 14:35 ` Julien Grall
@ 2014-07-16 15:43 ` Ian Campbell
2014-07-22 16:50 ` Andrii Tseglytskyi
2014-07-22 16:39 ` Andrii Tseglytskyi
1 sibling, 1 reply; 57+ messages in thread
From: Ian Campbell @ 2014-07-16 15:43 UTC (permalink / raw)
To: Julien Grall; +Cc: xen-devel, Stefano Stabellini, Andrii Tseglytskyi
On Fri, 2014-07-04 at 15:35 +0100, Julien Grall wrote:
> Hi Andrii,
>
> On 26/06/14 12:07, Andrii Tseglytskyi wrote:
> > +long do_translate_pagetable(int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
> > +{
> > + struct xen_pagetable_addr pgt;
> > + struct mmu_info *mmu = NULL;
> > +
> > + if ( copy_from_guest(&pgt, pgt_addr, 1) )
> > + return -EFAULT;
> > +
> > + mmu = mmu_lookup(pgt.reg);
> > + if ( !mmu )
> > + {
> > + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
> > + return -EINVAL;
> > + }
> > +
> > + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
> > +
> > + return copy_to_guest(pgt_addr, &pgt, 1);
> > +}
> > +
>
> AFAIU, nothing prevents a malicious guest to call this hypercall and
> screw the pagetable of the MMU.
It's R/O, isn't it?
(I sure hope so!)
Ian.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-16 15:19 ` Ian Campbell
@ 2014-07-22 12:42 ` Stefano Stabellini
2014-07-22 13:29 ` Julien Grall
2014-07-22 17:22 ` Andrii Tseglytskyi
2014-07-22 15:40 ` Andrii Tseglytskyi
1 sibling, 2 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-07-22 12:42 UTC (permalink / raw)
To: Ian Campbell
Cc: Julien Grall, Andrii Tseglytskyi, xen-devel, Stefano Stabellini
On Wed, 16 Jul 2014, Ian Campbell wrote:
> On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
> > On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> > > This is a fisrst patch from patch series which was
> > > developed to handle remote (external) processors
> > > memory management units. Remote processors are
> > > typically used for graphic rendering (GPUs) and
> > > high quality video decoding (IPUs). They are typically
> > > installed on such multimedia SoCs as OMAP4 / OMAP5.
> > >
> > > As soon as remoteprocessor MMU typically works with
> > > pagetables filled by physical addresses, which are
> > > allocated by domU kernel, it is almost impossible to
> > > use them under Xen, intermediate physical addresses
> > > allocated by kernel, need to be translated to machine
> > > addresses.
> > >
> > > This patch introduces a simple framework to perform
> > > pfn -> mfn translation for external MMUs.
> > > It introduces basic data structures and algorithms
> > > needed for translation.
> > >
> > > Typically, when MMU is configured, some it registers
> > > are updated by new values. Introduced frameworks
> > > uses traps as starting point of remoteproc MMUs
> > > pagetables translation.
> > >
> > > Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
> > > Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> >
> > There is one problem with this patch: you need to find a way to "pin"
> > the p2m entries for the pfns and mfns found in the pagetables translated
> > by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
> > mfn mappings for the domain, breaking the pagetables already translated
> > by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
> > stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
> > under the guest feet, but features like memory sharing or swapping,
> > supported on x86, could cause it to happen.
>
> I've not fully grokked this patch but wouldn't it be easier and better
> to have a hook to cause the remoteproc to throw away and rebuild its
> mappings when this happens? It could be called from e.g. the TLB flush
> associated with the p2m changing, I think.
I guess it depends on how often Xen or the Xen tools are going to change
the p2m for a running domain. For example changing one page mapping a
couple of times a second would be very expensive if we have to rebuild
the remoteproc pagetables every time.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-22 12:42 ` Stefano Stabellini
@ 2014-07-22 13:29 ` Julien Grall
2014-07-22 16:31 ` Andrii Tseglytskyi
2014-07-22 17:22 ` Andrii Tseglytskyi
1 sibling, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-07-22 13:29 UTC (permalink / raw)
To: Stefano Stabellini, Ian Campbell
Cc: Julien Grall, xen-devel, Andrii Tseglytskyi
On 07/22/2014 01:42 PM, Stefano Stabellini wrote:
> On Wed, 16 Jul 2014, Ian Campbell wrote:
>> On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
>>> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>>>> This is a fisrst patch from patch series which was
>>>> developed to handle remote (external) processors
>>>> memory management units. Remote processors are
>>>> typically used for graphic rendering (GPUs) and
>>>> high quality video decoding (IPUs). They are typically
>>>> installed on such multimedia SoCs as OMAP4 / OMAP5.
>>>>
>>>> As soon as remoteprocessor MMU typically works with
>>>> pagetables filled by physical addresses, which are
>>>> allocated by domU kernel, it is almost impossible to
>>>> use them under Xen, intermediate physical addresses
>>>> allocated by kernel, need to be translated to machine
>>>> addresses.
>>>>
>>>> This patch introduces a simple framework to perform
>>>> pfn -> mfn translation for external MMUs.
>>>> It introduces basic data structures and algorithms
>>>> needed for translation.
>>>>
>>>> Typically, when MMU is configured, some it registers
>>>> are updated by new values. Introduced frameworks
>>>> uses traps as starting point of remoteproc MMUs
>>>> pagetables translation.
>>>>
>>>> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>>>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>>>
>>> There is one problem with this patch: you need to find a way to "pin"
>>> the p2m entries for the pfns and mfns found in the pagetables translated
>>> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
>>> mfn mappings for the domain, breaking the pagetables already translated
>>> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
>>> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
>>> under the guest feet, but features like memory sharing or swapping,
>>> supported on x86, could cause it to happen.
>>
>> I've not fully grokked this patch but wouldn't it be easier and better
>> to have a hook to cause the remoteproc to throw away and rebuild its
>> mappings when this happens? It could be called from e.g. the TLB flush
>> associated with the p2m changing, I think.
>
> I guess it depends on how often Xen or the Xen tools are going to change
> the p2m for a running domain. For example changing one page mapping a
> couple of times a second would be very expensive if we have to rebuild
> the remoteproc pagetables every time.
The kernel could also change the p2m mapping via the ballooning driver.
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-06-29 18:00 ` Julien Grall
@ 2014-07-22 15:20 ` Andrii Tseglytskyi
2014-07-22 16:29 ` Julien Grall
2014-07-31 11:59 ` Andrii Tseglytskyi
0 siblings, 2 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 15:20 UTC (permalink / raw)
To: Julien Grall; +Cc: Stefano Stabellini, Tim Deegan, Ian Campbell, xen-devel
Hi Julien,
Thanks a lot for detailed review and sorry for so late reply. Was in
big rush last days.
On Sun, Jun 29, 2014 at 9:00 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Andrii,
>
>
> On 26/06/14 12:07, Andrii Tseglytskyi wrote:
>>
>> This is a fisrst patch from patch series which was
>
>
> s/firsrst/first/
>
> Although I don't think you have to say that this is the first patch :). This
> is not useful for the commit message.
>
>
>> developed to handle remote (external) processors
>> memory management units. Remote processors are
>> typically used for graphic rendering (GPUs) and
>> high quality video decoding (IPUs). They are typically
>> installed on such multimedia SoCs as OMAP4 / OMAP5.
>>
>> As soon as remoteprocessor MMU typically works with
>> pagetables filled by physical addresses, which are
>> allocated by domU kernel, it is almost impossible to
>> use them under Xen, intermediate physical addresses
>> allocated by kernel, need to be translated to machine
>> addresses.
>>
>> This patch introduces a simple framework to perform
>> pfn -> mfn translation for external MMUs.
>> It introduces basic data structures and algorithms
>> needed for translation.
>>
>> Typically, when MMU is configured, some it registers
>> are updated by new values. Introduced frameworks
>> uses traps as starting point of remoteproc MMUs
>> pagetables translation.
>>
>> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>
>
> [..]
>
>
>> +static struct mmu_info *mmu_list[] = {
>> +};
>> +
>> +#define mmu_for_each(pfunc, data) \
>> +({ \
>> + u32 __i; \
>> + int __res = 0; \
>> + \
>> + for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
>> + { \
>> + __res |= pfunc(mmu_list[__i], data); \
>
>
> Using _res |= will result to a wrong errno at the end.
> See the usage in iommu mmu_init_all.
>
> I would use
>
> __res = pfunc(...)
> if ( !__res )
> break;
>
> And therefore modify mmu_check to return 1 if failing, 0 otherwise.
>
> This will also avoid to continue to browse all the MMU for nothing.
>
OK.
>
>> + } \
>> + __res; \
>> +})
>> +
>> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
>> +{
>> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start +
>> mmu->mem_size)) )
>> + return 1;
>> +
>> + return 0;
>> +}
>> +
>> +static inline struct mmu_info *mmu_lookup(u32 addr)
>> +{
>> + u32 i;
>> +
>> + /* enumerate all registered MMU's and check is address in range */
>> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
>> + {
>> + if ( mmu_check_mem_range(mmu_list[i], addr) )
>> + return mmu_list[i];
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
>> +{
>> + return mmu_for_each(mmu_check_mem_range, addr);
>> +}
>
>
> This solution leads any guest to access to the MMU and therefore program it.
> If you plan to use for passthrough, you have to find a way to say that a
> specific domain is able to use the MMU, maybe an hypercall. Otherwise this
> is a security issue.
>
> IIRC I have already raised this concern on V1 :). It would be nice if you
> resolve it ASAP, because I suspect it will need some rework in the way you
> handle MMNU.
>
Agree. I need to think how to solve this. I don't think that the
hypercall is what is needed here.
>
>> +static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable
>> *pgt)
>> +{
>> + void __iomem *pagetable = NULL;
>> + u32 maddr, i;
>> +
>> + ASSERT(mmu);
>> + ASSERT(pgt);
>> +
>> + if ( !pgt->paddr )
>> + return -EINVAL;
>> +
>> + /* pagetable size can be more than one page */
>> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
>> + {
>> + /* lookup address where remoteproc pagetable is stored by kernel
>> */
>> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE,
>> NULL);
>> + if ( !maddr )
>> + {
>> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i
>> * PAGE_SIZE);
>> + return -EINVAL;
>> + }
>> +
>> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
>
>
> ioremap_* should only be used to map device memory. For the guest memory you
> have to use copy_from_guest helper.
>
OK. Just a small question - if I use copy_from_guest(), can I copy
from physical pointer ? Here I have an address, which points to exact
physical memory.
>
>> +struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32
>> addr, bool is_maddr)
>
>
> you have to use paddr_t for addr. The number of bit for the physical address
> can be greater than 40 bits.
>
OK
> The remark is the same everywhere within this file.
>
> [..]
>
>
>> +static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
>> +{
>> + struct mmu_pagetable *pgt;
>> + int res;
>> +
>> + /* lookup using machine address first */
>> + pgt = mmu_pagetable_lookup(mmu, paddr, true);
>> + if ( !pgt )
>> + {
>> + /* lookup using kernel physical address */
>> + pgt = mmu_pagetable_lookup(mmu, paddr, false);
>> + if ( !pgt )
>> + {
>> + /* if pagetable doesn't exists in lookup list - allocate it
>> */
>> + pgt = mmu_alloc_pagetable(mmu, paddr);
>
>
> The function mmu_alloc_pagetable can return NULL. But you never check the
> return value and dereference it just below.
>
OK. Thank you for catching this.
>
>> + }
>> + }
>> +
>> + pgt->maddr = MMU_INVALID_ADDRESS;
>
>
> [..]
>
>
>> +static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
>> +{
>> + struct mmu_info *mmu = NULL;
>> + unsigned long flags;
>> + register_t *r;
>> +
>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(r);
>
>
> The ASSERT is pointless, select_user_reg will never return NULL.
OK
>
> [..]
>
>
>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
>> +{
>> + struct mmu_info *mmu = NULL;
>> + unsigned long flags;
>> + register_t *r;
>> + u32 new_addr, val;
>> +
>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(r);
>
>
> Same here.
>
OK
>
>> + /* dom0 should not access remoteproc MMU */
>> + if ( 0 == current->domain->domain_id )
>> + return 1;
>
>
> Why this restriction?
>
We agreed that remoteproc will be handled by domU, which doesn't has 1
to 1 memory mapping.
If remoteproc belongs to dom0, translation is not needed - it MMU will
be configured with machine pointers.
>
>> + /* find corresponding MMU */
>> + mmu = mmu_lookup(info->gpa);
>> + if ( !mmu )
>> + {
>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>> + return -EINVAL;
>> + }
>> +
>> + ASSERT(v->domain == current->domain);
>
>
> I guess this restriction is because you are using current in
> mmu_trap_translate_pagetable?
Right. I just tried to make sure that p2m_lookup is called with proper
domain pointer.
>
> So, the iohandler is usually call on the current VCPU, there is no need to
> worry about it. Futhermore, I would pass the vcpu/domain in argument of the
> next function.
>
Can be. In first revision of these patches domain passed as a
parameter. But that leaded to one more additional parameter in all
functions below this call. I found that I can reduce arg list if I use
current-> domain pointer.
>
>> +static int mmu_init(struct mmu_info *mmu, u32 data)
>> +{
>> + ASSERT(mmu);
>> + ASSERT(!mmu->mem_map);
>> +
>> + INIT_LIST_HEAD(&mmu->pagetables_list);
>> +
>> + /* map MMU memory */
>> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
>> + if ( !mmu->mem_map )
>
>
> It looks like there is a hard tab here.
>
OK. Thank you for catching.
> [..]
>
>
>> +static int mmu_init_all(void)
>> +{
>> + int res;
>> +
>> + res = mmu_for_each(mmu_init, 0);
>> + if ( res )
>> + {
>> + printk("%s error during init %d\n", __func__, res);
>> + return res;
>> + }
>
>
> Hmmm... do_initcalls doesn't check the return value. How your code behave we
> one of the MMU has not been initialized?
>
> I think do_initcalls & co should check the return, but as it's the common
> code I don't know how x86 respect this convention to return 0 if succeded.
> Ian, Stefano, any thoughs?
>
I would like to make this specific to ARM only if possible.
> [..]
>
>> diff --git a/xen/include/xen/remoteproc_iommu.h
>> b/xen/include/xen/remoteproc_iommu.h
>
>
> I think this file as
>
>
>> new file mode 100644
>> index 0000000..22e2951
>> --- /dev/null
>> +++ b/xen/include/xen/remoteproc_iommu.h
>
>
> The remoteproc things is ARM specific, right? If so, this header should be
> moved in include/asm-arm.
>
OK
>> +
>> +#define MMU_INVALID_ADDRESS ((u32)(-1))
>
>
> You should not assume that the MMU ADDRESS is always 32 bits. Please use
> paddr_t here.
>
Sure. Thank you. Next series will contain paddr_t everywhere, where
u32 is used for address definition.
>
>> +#define pr_mmu(fmt, ...) \
>> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""),
>> ##__VA_ARGS__)
>
>
> Hmmm, you are assuming that mmu is existing within the function. You should
> pass the mmu in parameter of this macro.
>
> Also, most of the usage are for an error (except the one in mmu_page_alloc).
> I would prefix it by XENLOG_ERROR.
>
OK
>
>> +struct pagetable_data {
>> + /* 1st level translation */
>> + u32 pgd_shift;
>> + u32 pte_shift;
>> + u32 super_shift;
>> + u32 section_shift;
>> + /* 2nd level translation */
>> + u32 pte_large_shift;
>> +};
>
>
> No hard tab in Xen. Please remove them.
>
OK
>
>> +
>> +struct mmu_pagetable {
>> + u32 *hyp_pagetable;
>> + u32 *kern_pagetable;
>> + u32 paddr;
>> + u32 maddr;
>> + struct list_head link_node;
>> + u32 page_counter;
>> +};
>
>
> Same here.
>
OK
>
>> +
>> +struct mmu_info {
>> + const char *name;
>> + const struct pagetable_data *pg_data;
>> + /* register where phys pointer to pagetable is stored */
>> + u32 *trap_offsets;
>> + paddr_t mem_start;
>> + u32 mem_size;
>> + spinlock_t lock;
>> + struct list_head pagetables_list;
>> + u32 num_traps;
>> + void __iomem *mem_map;
>> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable
>> *);
>> + void (*print_pagetable_pfunc)(struct mmu_info *);
>> +};
>
>
> Same here.
>
> Regards,
>
> --
> Julien Grall
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-04 13:59 ` Stefano Stabellini
2014-07-16 15:19 ` Ian Campbell
@ 2014-07-22 15:32 ` Andrii Tseglytskyi
2014-08-01 10:06 ` Andrii Tseglytskyi
2014-08-20 19:40 ` Andrii Tseglytskyi
2 siblings, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 15:32 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Julien Grall, Ian Campbell, xen-devel
Hi Stefano,
Thanks a lot for review.
Sorry for so late reply - I was in big rush last days.
On Fri, Jul 4, 2014 at 4:59 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>> This is a fisrst patch from patch series which was
>> developed to handle remote (external) processors
>> memory management units. Remote processors are
>> typically used for graphic rendering (GPUs) and
>> high quality video decoding (IPUs). They are typically
>> installed on such multimedia SoCs as OMAP4 / OMAP5.
>>
>> As soon as remoteprocessor MMU typically works with
>> pagetables filled by physical addresses, which are
>> allocated by domU kernel, it is almost impossible to
>> use them under Xen, intermediate physical addresses
>> allocated by kernel, need to be translated to machine
>> addresses.
>>
>> This patch introduces a simple framework to perform
>> pfn -> mfn translation for external MMUs.
>> It introduces basic data structures and algorithms
>> needed for translation.
>>
>> Typically, when MMU is configured, some it registers
>> are updated by new values. Introduced frameworks
>> uses traps as starting point of remoteproc MMUs
>> pagetables translation.
>>
>> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>
> There is one problem with this patch: you need to find a way to "pin"
> the p2m entries for the pfns and mfns found in the pagetables translated
> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
> mfn mappings for the domain, breaking the pagetables already translated
> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
> under the guest feet, but features like memory sharing or swapping,
> supported on x86, could cause it to happen.
>
> In the past I tried to introduce a way to "pin the mappings, but never
> finished to upstream the patches, because it didn't need it anymore.
>
Thank you for this comment. I never thought that Xen may change
mappings, while domain is running. For sure it must be handled in my
code. Need to investigate this more deeply.
>
> Give a look at:
>
> http://marc.info/?l=xen-devel&m=138029864707973
>
> In particular you would need to introduce something like
> the pin function and call it from mmu_translate_pagetable.
>
>
OK. Thank you.
>
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/remoteproc_iommu.c | 412 ++++++++++++++++++++++++++++++++++++
>> xen/include/xen/remoteproc_iommu.h | 79 +++++++
>> 3 files changed, 492 insertions(+)
>> create mode 100644 xen/arch/arm/remoteproc_iommu.c
>> create mode 100644 xen/include/xen/remoteproc_iommu.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index d70f6d5..0204d1c 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -15,6 +15,7 @@ obj-y += io.o
>> obj-y += irq.o
>> obj-y += kernel.o
>> obj-y += mm.o
>> +obj-y += remoteproc_iommu.o
>> obj-y += p2m.o
>> obj-y += percpu.o
>> obj-y += guestcopy.o
>> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
>> new file mode 100644
>> index 0000000..b4d22d9
>> --- /dev/null
>> +++ b/xen/arch/arm/remoteproc_iommu.c
>> @@ -0,0 +1,412 @@
>> +/*
>> + * xen/arch/arm/remoteproc_iommu.c
>> + *
>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> + * Copyright (c) 2014 GlobalLogic
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <xen/config.h>
>> +#include <xen/lib.h>
>> +#include <xen/errno.h>
>> +#include <xen/mm.h>
>> +#include <xen/vmap.h>
>> +#include <xen/init.h>
>> +#include <xen/sched.h>
>> +#include <xen/stdbool.h>
>> +#include <asm/system.h>
>> +#include <asm/current.h>
>> +#include <asm/io.h>
>> +#include <asm/p2m.h>
>> +
>> +#include <xen/remoteproc_iommu.h>
>> +
>> +#include "io.h"
>> +
>> +static struct mmu_info *mmu_list[] = {
>> +};
>> +
>> +#define mmu_for_each(pfunc, data) \
>> +({ \
>> + u32 __i; \
>> + int __res = 0; \
>> + \
>> + for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
>> + { \
>> + __res |= pfunc(mmu_list[__i], data); \
>> + } \
>> + __res; \
>> +})
>> +
>> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
>> +{
>> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start + mmu->mem_size)) )
>> + return 1;
>> +
>> + return 0;
>> +}
>> +
>> +static inline struct mmu_info *mmu_lookup(u32 addr)
>> +{
>> + u32 i;
>> +
>> + /* enumerate all registered MMU's and check is address in range */
>> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
>> + {
>> + if ( mmu_check_mem_range(mmu_list[i], addr) )
>> + return mmu_list[i];
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
>> +{
>> + return mmu_for_each(mmu_check_mem_range, addr);
>> +}
>> +
>> +static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
>> +{
>> + void __iomem *pagetable = NULL;
>> + u32 maddr, i;
>> +
>> + ASSERT(mmu);
>> + ASSERT(pgt);
>> +
>> + if ( !pgt->paddr )
>> + return -EINVAL;
>> +
>> + /* pagetable size can be more than one page */
>> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
>> + {
>> + /* lookup address where remoteproc pagetable is stored by kernel */
>> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE, NULL);
>> + if ( !maddr )
>> + {
>> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i * PAGE_SIZE);
>> + return -EINVAL;
>> + }
>> +
>> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
>> + if ( !pagetable )
>> + {
>> + pr_mmu("failed to map pagetable");
>> + return -EINVAL;
>> + }
>> +
>> + /* copy pagetable to hypervisor memory */
>> + clean_and_invalidate_xen_dcache_va_range(pagetable, PAGE_SIZE);
>> + memcpy((u32*)((u32)pgt->kern_pagetable + i * PAGE_SIZE), pagetable, PAGE_SIZE);
>> +
>> + iounmap(pagetable);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32 addr, bool is_maddr)
>> +{
>> + struct mmu_pagetable *pgt;
>> + u32 pgt_addr;
>> +
>> + list_for_each_entry(pgt, &mmu->pagetables_list, link_node)
>> + {
>> + if ( is_maddr )
>> + pgt_addr = pgt->maddr;
>> + else
>> + pgt_addr = pgt->paddr;
>> +
>> + if ( pgt_addr == addr )
>> + return pgt;
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static struct mmu_pagetable *mmu_alloc_pagetable(struct mmu_info *mmu, u32 paddr)
>> +{
>> + struct mmu_pagetable *pgt;
>> + u32 pgt_size = MMU_PGD_TABLE_SIZE(mmu);
>> +
>> + pgt = xzalloc_bytes(sizeof(struct mmu_pagetable));
>> + if ( !pgt )
>> + {
>> + pr_mmu("failed to alloc pagetable structure");
>> + return NULL;
>> + }
>> +
>> + /* allocate pagetable managed by hypervisor */
>> + pgt->hyp_pagetable = xzalloc_bytes(pgt_size);
>> + if ( !pgt->hyp_pagetable )
>> + {
>> + pr_mmu("failed to alloc private hyp_pagetable");
>> + return NULL;
>> + }
>> +
>> + /* alocate pagetable for ipa storing */
>> + pgt->kern_pagetable = xzalloc_bytes(pgt_size);
>> + if ( !pgt->kern_pagetable )
>> + {
>> + pr_mmu("failed to alloc private kern_pagetable");
>> + return NULL;
>> + }
>> +
>> + pr_mmu("private pagetables for 0x%08x paddr %u bytes (main 0x%08x, temp 0x%08x)",
>> + paddr, pgt_size, (u32)__pa(pgt->hyp_pagetable), (u32)__pa(pgt->kern_pagetable));
>> +
>> + pgt->paddr = paddr;
>> +
>> + list_add(&pgt->link_node, &mmu->pagetables_list);
>> +
>> + return pgt;
>> +}
>> +
>> +static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
>> +{
>> + struct mmu_pagetable *pgt;
>> + int res;
>> +
>> + /* lookup using machine address first */
>> + pgt = mmu_pagetable_lookup(mmu, paddr, true);
>> + if ( !pgt )
>> + {
>> + /* lookup using kernel physical address */
>> + pgt = mmu_pagetable_lookup(mmu, paddr, false);
>> + if ( !pgt )
>> + {
>> + /* if pagetable doesn't exists in lookup list - allocate it */
>> + pgt = mmu_alloc_pagetable(mmu, paddr);
>> + }
>> + }
>> +
>> + pgt->maddr = MMU_INVALID_ADDRESS;
>> +
>> + /* copy pagetable from domain to hypervisor */
>> + res = mmu_copy_pagetable(mmu, pgt);
>> + if ( res )
>> + return res;
>> +
>> + /* translate pagetable */
>> + pgt->maddr = mmu->translate_pfunc(mmu, pgt);
>> + return pgt->maddr;
>> +}
>> +
>> +static u32 mmu_trap_translate_pagetable(struct mmu_info *mmu, mmio_info_t *info)
>> +{
>> + register_t *reg;
>> + bool valid_trap = false;
>> + u32 i, paddr;
>> +
>> + reg = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(reg);
>> +
>> + paddr = *reg;
>> + if ( !paddr )
>> + return MMU_INVALID_ADDRESS;
>> +
>> + /* check is the register is a valid TTB register */
>> + for ( i = 0; i < mmu->num_traps; i++ )
>> + {
>> + if ( mmu->trap_offsets[i] == (info->gpa - mmu->mem_start) )
>> + {
>> + valid_trap = true;
>> + break;
>> + }
>> + }
>> +
>> + if ( !valid_trap )
>> + return MMU_INVALID_ADDRESS;
>> +
>> + return mmu_translate_pagetable(mmu, paddr);
>> +}
>> +
>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
>> + u32 maddr, u32 hyp_addr)
>
> Is this actually used anywhere?
>
>
>> +{
>> + u32 *pte_table = NULL, *hyp_pte_table = NULL, pte_table_size = PAGE_SIZE;
>> + u32 i;
>> +
>> + /* remap second level translation table */
>> + pte_table = ioremap_nocache(maddr, MMU_PTE_TABLE_SIZE(mmu));
>> + if ( !pte_table )
>> + {
>> + pr_mmu("failed to map pte_table");
>> + return MMU_INVALID_ADDRESS;
>> + }
>> +
>> + /* allocate new second level pagetable once */
>> + if ( 0 == hyp_addr )
>> + {
>> + if ( MMU_PTE_TABLE_SIZE(mmu) > PAGE_SIZE )
>> + pte_table_size = MMU_PTE_TABLE_SIZE(mmu);
>> +
>> + hyp_pte_table = xzalloc_bytes(pte_table_size);
>> + if ( !hyp_pte_table )
>> + {
>> + pr_mmu("failed to alloc new iopt");
>> + return MMU_INVALID_ADDRESS;
>> + }
>> + }
>> + else
>> + {
>> + hyp_pte_table = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
>> + }
>> +
>> + /* 2-nd level translation */
>> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
>> + {
>> + paddr_t pt_maddr, pt_paddr, pt_flags;
>> + u32 pt_mask = MMU_SECTION_MASK(mmu->pg_data->pte_shift);
>> +
>> + if ( !pte_table[i] )
>> + {
>> + /* handle the case when page was removed */
>> + if ( unlikely(hyp_pte_table[i]) )
>> + {
>> + hyp_pte_table[i] = 0;
>> + }
>> +
>> + continue;
>> + }
>> +
>> + pt_paddr = pte_table[i] & pt_mask;
>> + pt_flags = pte_table[i] & ~pt_mask;
>> + pt_maddr = p2m_lookup(current->domain, pt_paddr, NULL);
>> + ASSERT(pt_maddr != INVALID_PADDR);
>> +
>> + hyp_pte_table[i] = pt_maddr | pt_flags;
>> + pgt->page_counter++;
>> + }
>> +
>> + iounmap(pte_table);
>> +
>> + clean_and_invalidate_xen_dcache_va_range(hyp_pte_table, MMU_PTE_TABLE_SIZE(mmu));
>> + return __pa(hyp_pte_table);
>> +}
>> +
>> +static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
>> +{
>> + struct mmu_info *mmu = NULL;
>> + unsigned long flags;
>> + register_t *r;
>> +
>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(r);
>
> If you check on the domid in mmu_mmio_write, I would check here too
>
>
>> + mmu = mmu_lookup(info->gpa);
>> + if ( !mmu )
>> + {
>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>> + return -EINVAL;
>> + }
>> +
>> + spin_lock_irqsave(&mmu->lock, flags);
>> + *r = readl(mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
>> + spin_unlock_irqrestore(&mmu->lock, flags);
>> +
>> + return 1;
>
> It looks like you are returning the mfns here, is that correct?
>
No. This callback due to following reasons:
- to trap MMU registers I need to unmap memory, where they are mapped.
- I call table translation logic if only one specific register is
accessed for write
- I can't unmap not less than one page of memory
- if I unmap 1 page and more - I need to pass through all kernel read
/ write calls to this page, otherwise kernel code fails
>
>
>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
>> +{
>> + struct mmu_info *mmu = NULL;
>> + unsigned long flags;
>> + register_t *r;
>> + u32 new_addr, val;
>> +
>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(r);
>> +
>> + /* dom0 should not access remoteproc MMU */
>> + if ( 0 == current->domain->domain_id )
>> + return 1;
>
> This is too specific to one particular configuration.
> Would it be possible to generalize this somehow? At the very least you
> could introduce an XSM label to access the pagetables, so that you can
> dynamically configure the domains the can write to them.
>
I need to think about this. Sounds reasonable.
>
>> + /* find corresponding MMU */
>> + mmu = mmu_lookup(info->gpa);
>> + if ( !mmu )
>> + {
>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>> + return -EINVAL;
>> + }
>> +
>> + ASSERT(v->domain == current->domain);
>
> You can remove this assert.
>
OK.
>
>> + spin_lock_irqsave(&mmu->lock, flags);
>> +
>> + /* get new address of translated pagetable */
>> + new_addr = mmu_trap_translate_pagetable(mmu, info);
>> + if ( MMU_INVALID_ADDRESS != new_addr )
>> + val = new_addr;
>> + else
>> + val = *r;
>> +
>> + writel(val, mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
>> + spin_unlock_irqrestore(&mmu->lock, flags);
>> +
>> + return 1;
>> +}
>> +
>> +static int mmu_init(struct mmu_info *mmu, u32 data)
>> +{
>> + ASSERT(mmu);
>> + ASSERT(!mmu->mem_map);
>> +
>> + INIT_LIST_HEAD(&mmu->pagetables_list);
>> +
>> + /* map MMU memory */
>> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
>> + if ( !mmu->mem_map )
>> + {
>> + pr_mmu("failed to map memory");
>> + return -EINVAL;
>> + }
>> +
>> + pr_mmu("memory map = 0x%pS", _p(mmu->mem_map));
>> +
>> + spin_lock_init(&mmu->lock);
>> +
>> + return 0;
>> +}
>> +
>> +static int mmu_init_all(void)
>> +{
>> + int res;
>> +
>> + res = mmu_for_each(mmu_init, 0);
>> + if ( res )
>> + {
>> + printk("%s error during init %d\n", __func__, res);
>> + return res;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +const struct mmio_handler remoteproc_mmio_handler = {
>> + .check_handler = mmu_mmio_check,
>> + .read_handler = mmu_mmio_read,
>> + .write_handler = mmu_mmio_write,
>> +};
>> +
>> +__initcall(mmu_init_all);
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
>> new file mode 100644
>> index 0000000..22e2951
>> --- /dev/null
>> +++ b/xen/include/xen/remoteproc_iommu.h
>> @@ -0,0 +1,79 @@
>> +/*
>> + * xen/include/xen/remoteproc_iommu.h
>> + *
>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> + * Copyright (c) 2014 GlobalLogic
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef _REMOTEPROC_IOMMU_H_
>> +#define _REMOTEPROC_IOMMU_H_
>> +
>> +#include <asm/types.h>
>> +
>> +#define MMU_SECTION_SIZE(shift) (1UL << (shift))
>> +#define MMU_SECTION_MASK(shift) (~(MMU_SECTION_SIZE(shift) - 1))
>> +
>> +/* 4096 first level descriptors for "supersection" and "section" */
>> +#define MMU_PTRS_PER_PGD(mmu) (1UL << (32 - (mmu->pg_data->pgd_shift)))
>> +#define MMU_PGD_TABLE_SIZE(mmu) (MMU_PTRS_PER_PGD(mmu) * sizeof(u32))
>> +
>> +/* 256 second level descriptors for "small" and "large" pages */
>> +#define MMU_PTRS_PER_PTE(mmu) (1UL << ((mmu->pg_data->pgd_shift) - (mmu->pg_data->pte_shift)))
>> +#define MMU_PTE_TABLE_SIZE(mmu) (MMU_PTRS_PER_PTE(mmu) * sizeof(u32))
>> +
>> +/* 16 sections in supersection */
>> +#define MMU_SECTION_PER_SUPER(mmu) (1UL << ((mmu->pg_data->super_shift) - (mmu->pg_data->section_shift)))
>> +
>> +#define MMU_INVALID_ADDRESS ((u32)(-1))
>> +
>> +#define pr_mmu(fmt, ...) \
>> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
>> +
>> +struct pagetable_data {
>> + /* 1st level translation */
>> + u32 pgd_shift;
>> + u32 pte_shift;
>> + u32 super_shift;
>> + u32 section_shift;
>> + /* 2nd level translation */
>> + u32 pte_large_shift;
>> +};
>> +
>> +struct mmu_pagetable {
>> + u32 *hyp_pagetable;
>> + u32 *kern_pagetable;
>> + u32 paddr;
>> + u32 maddr;
>> + struct list_head link_node;
>> + u32 page_counter;
>> +};
>> +
>> +struct mmu_info {
>> + const char *name;
>> + const struct pagetable_data *pg_data;
>> + /* register where phys pointer to pagetable is stored */
>> + u32 *trap_offsets;
>> + paddr_t mem_start;
>> + u32 mem_size;
>> + spinlock_t lock;
>> + struct list_head pagetables_list;
>> + u32 num_traps;
>> + void __iomem *mem_map;
>> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
>> + void (*print_pagetable_pfunc)(struct mmu_info *);
>> +};
>> +
>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
>> + u32 maddr, u32 hyp_addr);
>> +
>> +#endif /* _REMOTEPROC_IOMMU_H_ */
>> --
>> 1.7.9.5
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-16 15:19 ` Ian Campbell
2014-07-22 12:42 ` Stefano Stabellini
@ 2014-07-22 15:40 ` Andrii Tseglytskyi
1 sibling, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 15:40 UTC (permalink / raw)
To: Ian Campbell; +Cc: Julien Grall, xen-devel, Stefano Stabellini
Hi Ian,
Sorry for late reply,
On Wed, Jul 16, 2014 at 6:19 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
>> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>> > This is a fisrst patch from patch series which was
>> > developed to handle remote (external) processors
>> > memory management units. Remote processors are
>> > typically used for graphic rendering (GPUs) and
>> > high quality video decoding (IPUs). They are typically
>> > installed on such multimedia SoCs as OMAP4 / OMAP5.
>> >
>> > As soon as remoteprocessor MMU typically works with
>> > pagetables filled by physical addresses, which are
>> > allocated by domU kernel, it is almost impossible to
>> > use them under Xen, intermediate physical addresses
>> > allocated by kernel, need to be translated to machine
>> > addresses.
>> >
>> > This patch introduces a simple framework to perform
>> > pfn -> mfn translation for external MMUs.
>> > It introduces basic data structures and algorithms
>> > needed for translation.
>> >
>> > Typically, when MMU is configured, some it registers
>> > are updated by new values. Introduced frameworks
>> > uses traps as starting point of remoteproc MMUs
>> > pagetables translation.
>> >
>> > Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>> > Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>>
>> There is one problem with this patch: you need to find a way to "pin"
>> the p2m entries for the pfns and mfns found in the pagetables translated
>> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
>> mfn mappings for the domain, breaking the pagetables already translated
>> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
>> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
>> under the guest feet, but features like memory sharing or swapping,
>> supported on x86, could cause it to happen.
>
> I've not fully grokked this patch but wouldn't it be easier and better
> to have a hook to cause the remoteproc to throw away and rebuild its
> mappings when this happens? It could be called from e.g. the TLB flush
> associated with the p2m changing, I think.
>
Yes., you are right. And this series contains a hypercall definition,
which is designed to do this.
But let me explained a bit:
Pagetables are translated in 2 cases:
- specific register access
- direct hypercall
Specific register access trap was designed to hide pagetables
translation completely. Kernel won't know anything that addresses were
changed. And this works fine for 1 of OMAP remoteprocs - IPU. This is
the only reason why hypercall is used together with memory trap
algorithm, which is much more complicated than hypercall.
Regards,
Andrii
> Ian.
>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-16 15:29 ` Ian Campbell
2014-07-16 15:34 ` Ian Campbell
@ 2014-07-22 16:14 ` Andrii Tseglytskyi
1 sibling, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:14 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
On Wed, Jul 16, 2014 at 6:29 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-06-26 at 14:07 +0300, Andrii Tseglytskyi wrote:
>> This is a fisrst patch from patch series which was
>> developed to handle remote (external) processors
>> memory management units. Remote processors are
>> typically used for graphic rendering (GPUs) and
>> high quality video decoding (IPUs). They are typically
>> installed on such multimedia SoCs as OMAP4 / OMAP5.
>>
>> As soon as remoteprocessor MMU typically works with
>> pagetables filled by physical addresses, which are
>> allocated by domU kernel, it is almost impossible to
>> use them under Xen, intermediate physical addresses
>> allocated by kernel, need to be translated to machine
>> addresses.
>>
>> This patch introduces a simple framework to perform
>> pfn -> mfn translation for external MMUs.
>> It introduces basic data structures and algorithms
>> needed for translation.
>>
>> Typically, when MMU is configured, some it registers
>> are updated by new values. Introduced frameworks
>> uses traps as starting point of remoteproc MMUs
>> pagetables translation.
>
> Just to clarify the model:
>
> This is intended as a framework to enable Xen to provide an SMMU to the
> guest. Is the intention that this SMMU be an idealised PV thing or is it
> intending to emulate a real SMMU (and therefore the guests existing
> drivers)? I think this is the second one.
>
For sure it is a second one.
> And the intention here is to let the guest manipulate the page tables
> directly and shadow them from IPA->MA as opposed to e.g. exposing the
> guest's complete IPA->MA mapping to the remoteproc.
>
Right.
> I suppose either way my main concern is going to be the potential for
> proliferation of backends (and perhaps frontends) for all sorts of
> different H/W MMUs. I suppose that is unavoidable though.
>
Right. The only comment - code is separated between files.
*omap_iommu.c* contains H/W specific definitions for existing IPU and GPU
*remoteproc_iommu.c* is designed to contain generic code. In ideal
world any platform with any remoteprocs, which need pagetables
translation will add it's own file with specific HW definitions and
hooks to remoteproc_iommu.c. Of course this is true only for "ideal
world". I never tried to run this code on other platforms than OMAP5,
and I would expect that generic part may need adjustments for other
platforms.
> WRT the shadowing, I don't see the bit which causes the guest pages
> which it is using as PT page for the remoteproc to get marked R/O to
> force a trap. Perhaps that comes later.
>
Hmm. thank you for this comment. To make this framework working I
unmap MMU device memory pages in domain config file. On my setup IPU
and SGX belongs to Android domU kernel and all their HW such as
interrupts and iomem are forwarded as is in config file.
Due to this the only I need - is to unmap their MMU device memory. Do
you mean that it can be done in a different way in hypervisor code?
> Do these remoteproc page tables get frequently updated at run time or
> are they mostly just build once and forget?
>
IPU - boot time and that's it. It allocates a big (about 256 Mb) chunk
of memory. In my kernel, driver which handles remoteproc is moved to
kernel module which starts after graphic - therefore user desn't see
*any* delays on boot and any performance drop.
GPU - allocates every time when rendering is performed. Also it
creates new standalone MMU pagetable on each new screen. And this was
the most painful case - to handle dynamic page allocations /
translations without any performance drop. But looks like an attempt
was successful - framework does not change complete pagetabe every
time. It handles difference between previous pagetable and current
kernel pagetable and adds / deletes only that pages, which were
allocated / deleted. Of course, taking in account what Stefano saying
about Xen mfns mapping changes - it will need modifications.
Regards,
Andrii
> Ian.
>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-16 15:34 ` Ian Campbell
@ 2014-07-22 16:24 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:24 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Hi Ian
On Wed, Jul 16, 2014 at 6:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2014-07-16 at 16:29 +0100, Ian Campbell wrote:
>> WRT the shadowing, I don't see the bit which causes the guest pages
>> which it is using as PT page for the remoteproc to get marked R/O to
>> force a trap. Perhaps that comes later.
>>
>> Do these remoteproc page tables get frequently updated at run time or
>> are they mostly just build once and forget?
>
> Reading the next patch I'm concluding that the translation happens each
> time you write to the MMU's base pointer. IOW every update requires you
> to rewrite that pointer, is that right?
>
Not exactly. Continuing an explanation from prev mail - this just an
entry point for translation. IPU and GPU work in a bit different way.
IPU is quite straight forward
Boot -> Allocate all memory -> Update register -> Xen performs translation
GPU:
Step one:
Reset -> Allocate several pages of memory -> Update register -> Xen
performas translation
Step two:
Allocate several new pagetables -> Flush caches -> call
do_translate_pagetable hypercall
Here I don't have any registers update. GPU kernel driver uses
interprocess messaging mechanism to pass physical pointers of newly
allocated pagetables directly to GPU device. So, I use hypercall which
translate pagetable during device creation.
Regards,
Andrii
> Ian.
>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-22 15:20 ` Andrii Tseglytskyi
@ 2014-07-22 16:29 ` Julien Grall
2014-07-31 11:59 ` Andrii Tseglytskyi
1 sibling, 0 replies; 57+ messages in thread
From: Julien Grall @ 2014-07-22 16:29 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Stefano Stabellini, Tim Deegan, Ian Campbell, xen-devel
On 07/22/2014 04:20 PM, Andrii Tseglytskyi wrote:
> Hi Julien,
Hi Andrii,
>>
>>> + } \
>>> + __res; \
>>> +})
>>> +
>>> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
>>> +{
>>> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start +
>>> mmu->mem_size)) )
>>> + return 1;
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static inline struct mmu_info *mmu_lookup(u32 addr)
>>> +{
>>> + u32 i;
>>> +
>>> + /* enumerate all registered MMU's and check is address in range */
>>> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
>>> + {
>>> + if ( mmu_check_mem_range(mmu_list[i], addr) )
>>> + return mmu_list[i];
>>> + }
>>> +
>>> + return NULL;
>>> +}
>>> +
>>> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
>>> +{
>>> + return mmu_for_each(mmu_check_mem_range, addr);
>>> +}
>>
>>
>> This solution leads any guest to access to the MMU and therefore program it.
>> If you plan to use for passthrough, you have to find a way to say that a
>> specific domain is able to use the MMU, maybe an hypercall. Otherwise this
>> is a security issue.
>>
>> IIRC I have already raised this concern on V1 :). It would be nice if you
>> resolve it ASAP, because I suspect it will need some rework in the way you
>> handle MMNU.
>>
>
> Agree. I need to think how to solve this. I don't think that the
> hypercall is what is needed here.
IHMO, only the toolstack is able to know which domain will use the
remote processor or not. Adding a new DOMCTL looks like the best solution.
>> ioremap_* should only be used to map device memory. For the guest memory you
>> have to use copy_from_guest helper.
>>
>
> OK. Just a small question - if I use copy_from_guest(), can I copy
> from physical pointer ? Here I have an address, which points to exact
> physical memory.
copy_from_guest only cope with virtual guest address. You have to use
directly p2m_lookup and map_domain_page.
There is a ongoing discussion about it. See:
http://lists.xen.org/archives/html/xen-devel/2014-07/msg02096.html
>>
>>> + /* dom0 should not access remoteproc MMU */
>>> + if ( 0 == current->domain->domain_id )
>>> + return 1;
>>
>>
>> Why this restriction?
>>
>
> We agreed that remoteproc will be handled by domU, which doesn't has 1
> to 1 memory mapping.
> If remoteproc belongs to dom0, translation is not needed - it MMU will
> be configured with machine pointers.
In this case, the io ops are not registered for DOM0. So we should never
reach this case.
>>
>> So, the iohandler is usually call on the current VCPU, there is no need to
>> worry about it. Futhermore, I would pass the vcpu/domain in argument of the
>> next function.
>>
>
> Can be. In first revision of these patches domain passed as a
> parameter. But that leaded to one more additional parameter in all
> functions below this call. I found that I can reduce arg list if I use
> current-> domain pointer.
Ok.
>>> +static int mmu_init_all(void)
>>> +{
>>> + int res;
>>> +
>>> + res = mmu_for_each(mmu_init, 0);
>>> + if ( res )
>>> + {
>>> + printk("%s error during init %d\n", __func__, res);
>>> + return res;
>>> + }
>>
>>
>> Hmmm... do_initcalls doesn't check the return value. How your code behave we
>> one of the MMU has not been initialized?
>>
>> I think do_initcalls & co should check the return, but as it's the common
>> code I don't know how x86 respect this convention to return 0 if succeded.
>> Ian, Stefano, any thoughs?
>>
>
> I would like to make this specific to ARM only if possible.
It looks like Ian and Stefano doesn't answer to this question. If we
can't check the return of the init call in do_initcalls, I would replace
by a panic.
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-22 13:29 ` Julien Grall
@ 2014-07-22 16:31 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:31 UTC (permalink / raw)
To: Julien Grall; +Cc: Julien Grall, xen-devel, Ian Campbell, Stefano Stabellini
On Tue, Jul 22, 2014 at 4:29 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 07/22/2014 01:42 PM, Stefano Stabellini wrote:
>> On Wed, 16 Jul 2014, Ian Campbell wrote:
>>> On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
>>>> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>>>>> This is a fisrst patch from patch series which was
>>>>> developed to handle remote (external) processors
>>>>> memory management units. Remote processors are
>>>>> typically used for graphic rendering (GPUs) and
>>>>> high quality video decoding (IPUs). They are typically
>>>>> installed on such multimedia SoCs as OMAP4 / OMAP5.
>>>>>
>>>>> As soon as remoteprocessor MMU typically works with
>>>>> pagetables filled by physical addresses, which are
>>>>> allocated by domU kernel, it is almost impossible to
>>>>> use them under Xen, intermediate physical addresses
>>>>> allocated by kernel, need to be translated to machine
>>>>> addresses.
>>>>>
>>>>> This patch introduces a simple framework to perform
>>>>> pfn -> mfn translation for external MMUs.
>>>>> It introduces basic data structures and algorithms
>>>>> needed for translation.
>>>>>
>>>>> Typically, when MMU is configured, some it registers
>>>>> are updated by new values. Introduced frameworks
>>>>> uses traps as starting point of remoteproc MMUs
>>>>> pagetables translation.
>>>>>
>>>>> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>>>>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>>>>
>>>> There is one problem with this patch: you need to find a way to "pin"
>>>> the p2m entries for the pfns and mfns found in the pagetables translated
>>>> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
>>>> mfn mappings for the domain, breaking the pagetables already translated
>>>> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
>>>> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
>>>> under the guest feet, but features like memory sharing or swapping,
>>>> supported on x86, could cause it to happen.
>>>
>>> I've not fully grokked this patch but wouldn't it be easier and better
>>> to have a hook to cause the remoteproc to throw away and rebuild its
>>> mappings when this happens? It could be called from e.g. the TLB flush
>>> associated with the p2m changing, I think.
>>
>> I guess it depends on how often Xen or the Xen tools are going to change
>> the p2m for a running domain. For example changing one page mapping a
>> couple of times a second would be very expensive if we have to rebuild
>> the remoteproc pagetables every time.
>
> The kernel could also change the p2m mapping via the ballooning driver.
>
I think that kernel may call hypercall to update needed pagetables in
this case. But I need to look deeply to this code.
> Regards,
>
> --
> Julien Grall
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-04 14:05 ` Stefano Stabellini
2014-07-16 15:42 ` Ian Campbell
@ 2014-07-22 16:37 ` Andrii Tseglytskyi
1 sibling, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:37 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel
Hi Stefano,
On Fri, Jul 4, 2014 at 5:05 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>> The reason of the patch is the following - some remoteprocs
>> are quite complicated, and their MMUs can handle several
>> pagetables. Good example is a OMAP5 GPU, which allocates
>> several pagetables during it work. Additional requirement
>> is that not all pagetable physical addresses are stored
>> in MMU registers. Some pagetables may be allocated and
>> then their physical addresses are sent to GPU using private
>> message loop between GPU kernel driver and GPU remoteproc.
>>
>> Patch is developed to handle this. At any moment of time
>> kernel can perform translation of such pagetables, before
>> sending their addresses to GPU remoteproc.
>>
>> Change-Id: Ie84012163205c3a2f920dc4cf07327a244647c93
>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> ---
>> xen/arch/arm/remoteproc_iommu.c | 22 ++++++++++++++++++++++
>> xen/arch/arm/traps.c | 1 +
>> xen/include/public/xen.h | 1 +
>> xen/include/xen/hypercall.h | 12 ++++++++++++
>> 4 files changed, 36 insertions(+)
>>
>> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
>> index 3b3da3b..316ceb4 100644
>> --- a/xen/arch/arm/remoteproc_iommu.c
>> +++ b/xen/arch/arm/remoteproc_iommu.c
>> @@ -23,6 +23,8 @@
>> #include <xen/init.h>
>> #include <xen/sched.h>
>> #include <xen/stdbool.h>
>> +#include <xen/hypercall.h>
>> +#include <xen/guest_access.h>
>> #include <asm/system.h>
>> #include <asm/current.h>
>> #include <asm/io.h>
>> @@ -382,6 +384,26 @@ static int mmu_init(struct mmu_info *mmu, u32 data)
>> return 0;
>> }
>>
>> +long do_translate_pagetable(int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
>> +{
>> + struct xen_pagetable_addr pgt;
>> + struct mmu_info *mmu = NULL;
>> +
>> + if ( copy_from_guest(&pgt, pgt_addr, 1) )
>> + return -EFAULT;
>> +
>> + mmu = mmu_lookup(pgt.reg);
>> + if ( !mmu )
>> + {
>> + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
>> + return -EINVAL;
>> + }
>> +
>> + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
>> +
>> + return copy_to_guest(pgt_addr, &pgt, 1);
>> +}
>> +
>> static int mmu_init_all(void)
>> {
>> int res;
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index 21c7b26..05b5184 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -1012,6 +1012,7 @@ static arm_hypercall_t arm_hypercall_table[] = {
>> HYPERCALL(hvm_op, 2),
>> HYPERCALL(grant_table_op, 3),
>> HYPERCALL_ARM(vcpu_op, 3),
>> + HYPERCALL(translate_pagetable, 2),
>> };
>>
>> typedef int (*arm_psci_fn_t)(uint32_t, register_t);
>> diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
>> index 8c5697e..91ca6a1 100644
>> --- a/xen/include/public/xen.h
>> +++ b/xen/include/public/xen.h
>> @@ -101,6 +101,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);
>> #define __HYPERVISOR_kexec_op 37
>> #define __HYPERVISOR_tmem_op 38
>> #define __HYPERVISOR_xc_reserved_op 39 /* reserved for XenClient */
>> +#define __HYPERVISOR_translate_pagetable 40
>
> It might be best to introduce this hypercall as an arch memory op
> (xen/arch/arm/mm.c:arch_memory_op), rather than a full blown new
> hypercall. Otherwise you'll need to provide an empty stub implementation
> for x86.
>
I would prefer not to touch x86 code. arch_memory_op may be the point
here. Will try this.
>
>> /* Architecture-specific hypercall definitions. */
>> #define __HYPERVISOR_arch_0 48
>> diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
>> index a9e5229..a025435 100644
>> --- a/xen/include/xen/hypercall.h
>> +++ b/xen/include/xen/hypercall.h
>> @@ -136,6 +136,18 @@ extern long
>> do_tmem_op(
>> XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);
>>
>> +struct xen_pagetable_addr {
>> + u32 reg;
Base address of iomem, associated with corresponding MMU. Used as a search key
>> + u32 paddr;
Physical address of MMU pagetable, which is allocated in kernel.
>> + u32 maddr;
Output parameter. Machine address of translated pagetable.
>
> Could you please comment what these fields are for exactly?
>
Regards,
Andrii
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-04 14:35 ` Julien Grall
2014-07-16 15:43 ` Ian Campbell
@ 2014-07-22 16:39 ` Andrii Tseglytskyi
2014-07-22 16:44 ` Julien Grall
1 sibling, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:39 UTC (permalink / raw)
To: Julien Grall; +Cc: Stefano Stabellini, Ian Campbell, xen-devel
Hi Julien,
On Fri, Jul 4, 2014 at 5:35 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Andrii,
>
>
> On 26/06/14 12:07, Andrii Tseglytskyi wrote:
>>
>> +long do_translate_pagetable(int cmd,
>> XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
>> +{
>> + struct xen_pagetable_addr pgt;
>> + struct mmu_info *mmu = NULL;
>> +
>> + if ( copy_from_guest(&pgt, pgt_addr, 1) )
>> + return -EFAULT;
>> +
>> + mmu = mmu_lookup(pgt.reg);
>> + if ( !mmu )
>> + {
>> + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
>> + return -EINVAL;
>> + }
>> +
>> + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
>> +
>> + return copy_to_guest(pgt_addr, &pgt, 1);
>> +}
>> +
>
>
> AFAIU, nothing prevents a malicious guest to call this hypercall and screw
> the pagetable of the MMU.
Right. Do you think that some kind of security checks needed here?
>
> Regards,
>
> --
> Julien Grall
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-22 16:39 ` Andrii Tseglytskyi
@ 2014-07-22 16:44 ` Julien Grall
2014-07-22 16:48 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-07-22 16:44 UTC (permalink / raw)
To: Andrii Tseglytskyi; +Cc: Stefano Stabellini, Ian Campbell, xen-devel
On 07/22/2014 05:39 PM, Andrii Tseglytskyi wrote:
> Hi Julien,
Hi Andrii,
> On Fri, Jul 4, 2014 at 5:35 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> Hi Andrii,
>>
>>
>> On 26/06/14 12:07, Andrii Tseglytskyi wrote:
>>>
>>> +long do_translate_pagetable(int cmd,
>>> XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
>>> +{
>>> + struct xen_pagetable_addr pgt;
>>> + struct mmu_info *mmu = NULL;
>>> +
>>> + if ( copy_from_guest(&pgt, pgt_addr, 1) )
>>> + return -EFAULT;
>>> +
>>> + mmu = mmu_lookup(pgt.reg);
>>> + if ( !mmu )
>>> + {
>>> + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
>>> +
>>> + return copy_to_guest(pgt_addr, &pgt, 1);
>>> +}
>>> +
>>
>>
>> AFAIU, nothing prevents a malicious guest to call this hypercall and screw
>> the pagetable of the MMU.
>
> Right. Do you think that some kind of security checks needed here?
You need at least ot check that the domain is allowed to access to the
remote processor.
It may be implemented via the solution we were talking on patch #1.
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-16 15:42 ` Ian Campbell
@ 2014-07-22 16:47 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:47 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel, Stefano Stabellini
Hi Ian,
On Wed, Jul 16, 2014 at 6:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-07-04 at 15:05 +0100, Stefano Stabellini wrote:
>> > diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
>> > index 8c5697e..91ca6a1 100644
>> > --- a/xen/include/public/xen.h
>> > +++ b/xen/include/public/xen.h
>> > @@ -101,6 +101,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);
>> > #define __HYPERVISOR_kexec_op 37
>> > #define __HYPERVISOR_tmem_op 38
>> > #define __HYPERVISOR_xc_reserved_op 39 /* reserved for XenClient */
>> > +#define __HYPERVISOR_translate_pagetable 40
>>
>> It might be best to introduce this hypercall as an arch memory op
>> (xen/arch/arm/mm.c:arch_memory_op), rather than a full blown new
>> hypercall.
>
> Or physdev op perhaps?
>
>> Otherwise you'll need to provide an empty stub implementation
>> for x86.
>
> All unused hypercalls -ENOSYS automatically without stubs I think.
Taking this in account - do I need to redefine this hypercall in other
place? Or leave it as is?
>
> Are there any security concerns with exposing machine addresses to
> guests? There are certainly "breaking the abstraction" concerns, but
> that's unavoidable here I think.
Right. This is not good for security. But unfortunately on of OMAP5
remoteprocs (GPU) uses very specific pagetable allocations.
I described this in previous mails - it allocates pagetable and sends
it physical address directly for GPU, using messaging mechanism.
This address is not written to any MMU device iomem, and therefore I
can't avoid exposing of pagetable mfn.
>
>> > /* Architecture-specific hypercall definitions. */
>> > #define __HYPERVISOR_arch_0 48
>> > diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
>> > index a9e5229..a025435 100644
>> > --- a/xen/include/xen/hypercall.h
>> > +++ b/xen/include/xen/hypercall.h
>> > @@ -136,6 +136,18 @@ extern long
>> > do_tmem_op(
>> > XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);
>> >
>> > +struct xen_pagetable_addr {
>> > + u32 reg;
>> > + u32 paddr;
>> > + u32 maddr;
>>
>> Could you please comment what these fields are for exactly?
>
> And mark them as IN/OUT as appropriate please.
>
OK
> Also the physical and machine addresses should always be 64 bits for
> compatibility with future larger chips.
OK. Will update.
>
> Having done that you then need padding after reg or to reorder things to
> avoid holes on 64-bit. Or maybe reg should be 64-bit, depends what it
> is. Either way the 32-bit and 64-bit layout of this struct should be
> identical.
I think reg will be 64 bit in this case. I would prefer this way.
Regards,
Andrii
>
> Ian.
>
>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-22 16:44 ` Julien Grall
@ 2014-07-22 16:48 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:48 UTC (permalink / raw)
To: Julien Grall; +Cc: Stefano Stabellini, Ian Campbell, xen-devel
Hi Julien,
On Tue, Jul 22, 2014 at 7:44 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 07/22/2014 05:39 PM, Andrii Tseglytskyi wrote:
>> Hi Julien,
>
> Hi Andrii,
>
>> On Fri, Jul 4, 2014 at 5:35 PM, Julien Grall <julien.grall@linaro.org> wrote:
>>> Hi Andrii,
>>>
>>>
>>> On 26/06/14 12:07, Andrii Tseglytskyi wrote:
>>>>
>>>> +long do_translate_pagetable(int cmd,
>>>> XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
>>>> +{
>>>> + struct xen_pagetable_addr pgt;
>>>> + struct mmu_info *mmu = NULL;
>>>> +
>>>> + if ( copy_from_guest(&pgt, pgt_addr, 1) )
>>>> + return -EFAULT;
>>>> +
>>>> + mmu = mmu_lookup(pgt.reg);
>>>> + if ( !mmu )
>>>> + {
>>>> + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
>>>> +
>>>> + return copy_to_guest(pgt_addr, &pgt, 1);
>>>> +}
>>>> +
>>>
>>>
>>> AFAIU, nothing prevents a malicious guest to call this hypercall and screw
>>> the pagetable of the MMU.
>>
>> Right. Do you think that some kind of security checks needed here?
>
> You need at least ot check that the domain is allowed to access to the
> remote processor.
>
> It may be implemented via the solution we were talking on patch #1.
>
Agree.
> Regards,
>
> --
> Julien Grall
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall
2014-07-16 15:43 ` Ian Campbell
@ 2014-07-22 16:50 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:50 UTC (permalink / raw)
To: Ian Campbell; +Cc: Julien Grall, Stefano Stabellini, xen-devel
Hi Ian,
On Wed, Jul 16, 2014 at 6:43 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-07-04 at 15:35 +0100, Julien Grall wrote:
>> Hi Andrii,
>>
>> On 26/06/14 12:07, Andrii Tseglytskyi wrote:
>> > +long do_translate_pagetable(int cmd, XEN_GUEST_HANDLE_PARAM(xen_pagetable_addr_t) pgt_addr)
>> > +{
>> > + struct xen_pagetable_addr pgt;
>> > + struct mmu_info *mmu = NULL;
>> > +
>> > + if ( copy_from_guest(&pgt, pgt_addr, 1) )
>> > + return -EFAULT;
>> > +
>> > + mmu = mmu_lookup(pgt.reg);
>> > + if ( !mmu )
>> > + {
>> > + pr_mmu("can't get mmu for addr 0x%08x", pgt.reg);
>> > + return -EINVAL;
>> > + }
>> > +
>> > + pgt.maddr = mmu_translate_pagetable(mmu, pgt.paddr);
>> > +
>> > + return copy_to_guest(pgt_addr, &pgt, 1);
>> > +}
>> > +
>>
>> AFAIU, nothing prevents a malicious guest to call this hypercall and
>> screw the pagetable of the MMU.
>
> It's R/O, isn't it?
>
> (I sure hope so!)
>
I'll implement security check here. For now it is a hole.
> Ian.
>
>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 4/7] arm: omap: introduce print pagetable function for IPU remoteproc
2014-07-16 15:38 ` Ian Campbell
@ 2014-07-22 16:55 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:55 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Hi Ian,
On Wed, Jul 16, 2014 at 6:38 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-06-26 at 14:07 +0300, Andrii Tseglytskyi wrote:
>> This patch adds a possibility to dump all pagetables of
>> IPU remoteproc. The only reason to have this patch - is a
>> low level debug.
>
> It dumps the whole thing? Rather than the translation of a specific
> (I)PA?
>
> Isn't that rather, er, verbose?
>
Yes. Just dumping. I'm using this patch for debug purposes only.
I would say - it is a very very verbose. This particular patch can be
dropped from patch series, but I would like to keep *print_pagetables*
callback.
What is your suggestion?
> Ian.
>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc
2014-07-04 14:01 ` Stefano Stabellini
@ 2014-07-22 16:56 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:56 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel
On Fri, Jul 4, 2014 at 5:01 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>> The following patch introduced platform specific MMU data
>> definitions and pagetable translation function for OMAP5 IPU
>> remoteproc. This MMU is a bit specific - it typically performs
>> one level translation and map a big chunks of memory. 16 Mb
>> supersections and 1 Mb sections are mapped instead of 4 Kb pages.
>> Introduced algorithm performs internal remapping of big sections
>> to small 4 Kb pages.
>>
>> Change-Id: If20449f07e22f780e1fded67fed4f79cbe1fc156
>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> ---
>> xen/arch/arm/platforms/Makefile | 1 +
>> xen/arch/arm/platforms/omap_iommu.c | 247 +++++++++++++++++++++++++++++++++++
>> xen/arch/arm/remoteproc_iommu.c | 1 +
>> xen/include/xen/remoteproc_iommu.h | 2 +
>> 4 files changed, 251 insertions(+)
>> create mode 100644 xen/arch/arm/platforms/omap_iommu.c
>>
>> diff --git a/xen/arch/arm/platforms/Makefile b/xen/arch/arm/platforms/Makefile
>> index 080ea9a..f224f08 100644
>> --- a/xen/arch/arm/platforms/Makefile
>> +++ b/xen/arch/arm/platforms/Makefile
>> @@ -4,4 +4,5 @@ obj-$(CONFIG_ARM_32) += midway.o
>> obj-$(CONFIG_ARM_32) += omap5.o
>> obj-$(CONFIG_ARM_32) += dra7xx.o
>> obj-$(CONFIG_ARM_32) += sunxi.o
>> +obj-$(CONFIG_ARM_32) += omap_iommu.o
>> obj-$(CONFIG_ARM_64) += xgene-storm.o
>> diff --git a/xen/arch/arm/platforms/omap_iommu.c b/xen/arch/arm/platforms/omap_iommu.c
>> new file mode 100644
>> index 0000000..e0c4633
>> --- /dev/null
>> +++ b/xen/arch/arm/platforms/omap_iommu.c
>> @@ -0,0 +1,247 @@
>> +/*
>> + * xen/arch/arm/platforms/omap_iommu.c
>> + *
>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> + * Copyright (c) 2014 GlobalLogic
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <xen/lib.h>
>> +#include <xen/errno.h>
>> +#include <xen/stdbool.h>
>> +#include <xen/mm.h>
>> +#include <xen/vmap.h>
>> +#include <xen/sched.h>
>> +#include <xen/remoteproc_iommu.h>
>> +
>> +#include <asm/p2m.h>
>> +
>> +/*
>> + * "L2 table" address mask and size definitions.
>> + */
>> +
>> +/* register where address of pagetable is stored */
>> +#define MMU_IPU_TTB_OFFSET 0x4c
>> +
>> +/* 1st level translation */
>> +#define MMU_OMAP_PGD_SHIFT 20
>> +#define MMU_OMAP_SUPER_SHIFT 24 /* "supersection" - 16 Mb */
>> +#define MMU_OMAP_SECTION_SHIFT 20 /* "section" - 1 Mb */
>> +#define MMU_OMAP_SECOND_LEVEL_SHIFT 10
>> +
>> +/* 2nd level translation */
>> +#define MMU_OMAP_PTE_SMALL_SHIFT 12 /* "small page" - 4Kb */
>> +#define MMU_OMAP_PTE_LARGE_SHIFT 16 /* "large page" - 64 Kb */
>> +
>> +/*
>> + * some descriptor attributes.
>> + */
>> +#define PGD_TABLE (1 << 0)
>> +#define PGD_SECTION (2 << 0)
>> +#define PGD_SUPER (1 << 18 | 2 << 0)
>> +
>> +#define ipu_pgd_is_table(x) (((x) & 3) == PGD_TABLE)
>> +#define ipu_pgd_is_section(x) (((x) & (1 << 18 | 3)) == PGD_SECTION)
>> +#define ipu_pgd_is_super(x) (((x) & (1 << 18 | 3)) == PGD_SUPER)
>> +
>> +#define PTE_SMALL (2 << 0)
>> +#define PTE_LARGE (1 << 0)
>> +
>> +#define OMAP_IPU_MMU_MEM_BASE 0x55082000
>> +
>> +static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
>> +
>> +static u32 ipu_trap_offsets[] = {
>> + MMU_IPU_TTB_OFFSET,
>> +};
>> +
>> +static const struct pagetable_data pagetable_ipu_data = {
>> + .pgd_shift = MMU_OMAP_PGD_SHIFT,
>> + .super_shift = MMU_OMAP_SUPER_SHIFT,
>> + .section_shift = MMU_OMAP_SECTION_SHIFT,
>> + .pte_shift = MMU_OMAP_PTE_SMALL_SHIFT,
>> + .pte_large_shift = MMU_OMAP_PTE_LARGE_SHIFT,
>> +};
>> +
>> +struct mmu_info omap_ipu_mmu = {
>> + .name = "IPU_L2_MMU",
>> + .pg_data = &pagetable_ipu_data,
>> + .trap_offsets = ipu_trap_offsets,
>> + .mem_start = OMAP_IPU_MMU_MEM_BASE,
>> + .mem_size = 0x1000,
>> + .num_traps = ARRAY_SIZE(ipu_trap_offsets),
>> + .translate_pfunc = mmu_ipu_translate_pagetable,
>> +};
>> +
>> +static bool translate_supersections_to_pages = true;
>> +static bool translate_sections_to_pages = true;
>> +
>> +static u32 mmu_pte_table_alloc(struct mmu_info *mmu, u32 pgd, u32 sect_num,
>> + struct mmu_pagetable *pgt, u32 hyp_addr)
>> +{
>> + u32 *pte = NULL;
>> + u32 i;
>> +
>> + /* allocate pte table once */
>> + if ( 0 == hyp_addr )
>> + {
>> + pte = xzalloc_bytes(PAGE_SIZE);
>> + if ( !pte )
>> + {
>> + pr_mmu("failed to alloc 2nd level table");
>> + return 0;
>> + }
>> + }
>> + else
>> + {
>> + pte = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
>> + }
>> +
>> + ASSERT(256 == MMU_PTRS_PER_PTE(mmu));
>> +
>> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
>> + {
>> + u32 paddr, maddr;
>> +
>> + paddr = pgd + (i * PAGE_SIZE);
>> + maddr = p2m_lookup(current->domain, paddr, NULL);
>
> Here is where you would need to make sure that paddr->maddr doesn't
> change in the future by pinning the mapping.
>
OK.
Regards,
Andrii
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc
2014-07-04 14:30 ` Julien Grall
@ 2014-07-22 16:58 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 16:58 UTC (permalink / raw)
To: Julien Grall; +Cc: xen-devel
Hi Julien,
On Fri, Jul 4, 2014 at 5:30 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Andrii,
>
>
> On 26/06/14 12:07, Andrii Tseglytskyi wrote:
>>
>> +static u32 mmu_pte_table_alloc(struct mmu_info *mmu, u32 pgd, u32
>> sect_num,
>> + struct mmu_pagetable *pgt, u32 hyp_addr)
>> +{
>> + u32 *pte = NULL;
>> + u32 i;
>> +
>> + /* allocate pte table once */
>> + if ( 0 == hyp_addr )
>> + {
>> + pte = xzalloc_bytes(PAGE_SIZE);
>> + if ( !pte )
>> + {
>> + pr_mmu("failed to alloc 2nd level table");
>> + return 0;
>> + }
>> + }
>> + else
>> + {
>> + pte = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
>> + }
>> +
>> + ASSERT(256 == MMU_PTRS_PER_PTE(mmu));
>> +
>> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
>> + {
>> + u32 paddr, maddr;
>> +
>> + paddr = pgd + (i * PAGE_SIZE);
>> + maddr = p2m_lookup(current->domain, paddr, NULL);
>> + ASSERT(maddr != INVALID_PADDR);
>
>
> Hmmm, what prevents the guest to pass an invalid IPA? On debug build you
> will hit the assert, but on non-debug build you will screw the IPU page
> table.
>
Agree. Will replace ASSERTS with proper if - else logic.
> [..]
>
>> +static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct
>> mmu_pagetable *pgt)
>> +{
>
>
> [..]
>
>
>> + /* first level pointers have different formats, depending on
>> their type */
>> + if ( ipu_pgd_is_super(pgd) )
>> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SUPER_SHIFT);
>> + else if ( ipu_pgd_is_section(pgd) )
>> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECTION_SHIFT);
>> + else if ( ipu_pgd_is_table(pgd) )
>> + pd_mask = MMU_SECTION_MASK(MMU_OMAP_SECOND_LEVEL_SHIFT);
>> +
>> + pd_paddr = pgd & pd_mask;
>> + pd_flags = pgd & ~pd_mask;
>> + pd_maddr = p2m_lookup(current->domain, pd_paddr, NULL);
>> + ASSERT(pd_maddr != INVALID_PADDR);
>
>
> Same remark here.
>
OK
> --
> Julien Grall
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc
2014-07-16 15:36 ` Ian Campbell
@ 2014-07-22 17:16 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 17:16 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Hi Ian,
On Wed, Jul 16, 2014 at 6:36 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-06-26 at 14:07 +0300, Andrii Tseglytskyi wrote:
>> The following patch introduced platform specific MMU data
>> definitions and pagetable translation function for OMAP5 IPU
>> remoteproc. This MMU is a bit specific - it typically performs
>> one level translation and map a big chunks of memory. 16 Mb
>> supersections and 1 Mb sections are mapped instead of 4 Kb pages.
>> Introduced algorithm performs internal remapping of big sections
>> to small 4 Kb pages.
>
> How does that work if the MMU only supports 1MB/16MB supersections? Or
> have I misunderstood?
>
In my case MMU supports all - 4K, 1MB and 16MB. But for performance
reasons only 1MB and 16MB mappings are used during
IPU boot - using 4K pages mapping increases boot time for about 40
seconds, which is not acceptable for Android.
Taking in account, that MMU supports 4K pages - I can overwrite
pagetables changing 1MB and 16 MB mappings to corresponding subsets of
4K pages mappings. Kernel code will not know anything about this
change, and MMU will continue working fine.
>> Change-Id: If20449f07e22f780e1fded67fed4f79cbe1fc156
>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> ---
>> xen/arch/arm/platforms/Makefile | 1 +
>> xen/arch/arm/platforms/omap_iommu.c | 247 +++++++++++++++++++++++++++++++++++
>
> I don't think this is the right home for this.
>
> I think either xen/arch/arm/remoteproc/* or xen/drivers/remoteproc/*
> would be more appropriate.
OK.
>
>> +#define OMAP_IPU_MMU_MEM_BASE 0x55082000
>> +
>> +static u32 mmu_ipu_translate_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt);
>> +
>> +static u32 ipu_trap_offsets[] = {
>> + MMU_IPU_TTB_OFFSET,
>
> How large is this region? I think the machinery needs to be told.
>
Not sure that I get your question correctly.
To make a trap - I unmap 1 page of MMU device iomem, where register
which contains pagetable address is stored.
> Also if this functionality is to be used by guests then it can't really
> use the h/w base address, you'd need to define a region of guest address
> map (!= host/dom0 address map) for it.
>
This is just a register offset and it is used to define is guest
accessing a proper iomem register. IOW if offset is the same as offset
where pagetable address is stored - than translation is performed. Do
I need to define this in a different way?
>> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
>> index b4d22d9..8291f3f 100644
>> --- a/xen/arch/arm/remoteproc_iommu.c
>> +++ b/xen/arch/arm/remoteproc_iommu.c
>> @@ -33,6 +33,7 @@
>> #include "io.h"
>>
>> static struct mmu_info *mmu_list[] = {
>> + &omap_ipu_mmu,
>
> This suggests there is exactly one such and it is exposed to every
> domain.
>
> Wouldn't this rather be dynamic and per domain?
>
Yes, this is what Julien already suggested. I will investigate how to
implement this in the right way.
> Ian.
>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-22 12:42 ` Stefano Stabellini
2014-07-22 13:29 ` Julien Grall
@ 2014-07-22 17:22 ` Andrii Tseglytskyi
2014-07-23 10:32 ` Stefano Stabellini
1 sibling, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-22 17:22 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Julien Grall, Ian Campbell, xen-devel
Hi Stefano,
On Tue, Jul 22, 2014 at 3:42 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Wed, 16 Jul 2014, Ian Campbell wrote:
>> On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
>> > On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>> > > This is a fisrst patch from patch series which was
>> > > developed to handle remote (external) processors
>> > > memory management units. Remote processors are
>> > > typically used for graphic rendering (GPUs) and
>> > > high quality video decoding (IPUs). They are typically
>> > > installed on such multimedia SoCs as OMAP4 / OMAP5.
>> > >
>> > > As soon as remoteprocessor MMU typically works with
>> > > pagetables filled by physical addresses, which are
>> > > allocated by domU kernel, it is almost impossible to
>> > > use them under Xen, intermediate physical addresses
>> > > allocated by kernel, need to be translated to machine
>> > > addresses.
>> > >
>> > > This patch introduces a simple framework to perform
>> > > pfn -> mfn translation for external MMUs.
>> > > It introduces basic data structures and algorithms
>> > > needed for translation.
>> > >
>> > > Typically, when MMU is configured, some it registers
>> > > are updated by new values. Introduced frameworks
>> > > uses traps as starting point of remoteproc MMUs
>> > > pagetables translation.
>> > >
>> > > Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>> > > Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> >
>> > There is one problem with this patch: you need to find a way to "pin"
>> > the p2m entries for the pfns and mfns found in the pagetables translated
>> > by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
>> > mfn mappings for the domain, breaking the pagetables already translated
>> > by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
>> > stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
>> > under the guest feet, but features like memory sharing or swapping,
>> > supported on x86, could cause it to happen.
>>
>> I've not fully grokked this patch but wouldn't it be easier and better
>> to have a hook to cause the remoteproc to throw away and rebuild its
>> mappings when this happens? It could be called from e.g. the TLB flush
>> associated with the p2m changing, I think.
>
> I guess it depends on how often Xen or the Xen tools are going to change
> the p2m for a running domain. For example changing one page mapping a
> couple of times a second would be very expensive if we have to rebuild
> the remoteproc pagetables every time.
Sorry, I didn't get the point here. Looks like you are talking about
TLB flush initiated by Xen? Am I right?
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-22 17:22 ` Andrii Tseglytskyi
@ 2014-07-23 10:32 ` Stefano Stabellini
2014-07-23 10:54 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Stefano Stabellini @ 2014-07-23 10:32 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Julien Grall, xen-devel, Ian Campbell, Stefano Stabellini
On Tue, 22 Jul 2014, Andrii Tseglytskyi wrote:
> Hi Stefano,
>
> On Tue, Jul 22, 2014 at 3:42 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Wed, 16 Jul 2014, Ian Campbell wrote:
> >> On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
> >> > On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> >> > > This is a fisrst patch from patch series which was
> >> > > developed to handle remote (external) processors
> >> > > memory management units. Remote processors are
> >> > > typically used for graphic rendering (GPUs) and
> >> > > high quality video decoding (IPUs). They are typically
> >> > > installed on such multimedia SoCs as OMAP4 / OMAP5.
> >> > >
> >> > > As soon as remoteprocessor MMU typically works with
> >> > > pagetables filled by physical addresses, which are
> >> > > allocated by domU kernel, it is almost impossible to
> >> > > use them under Xen, intermediate physical addresses
> >> > > allocated by kernel, need to be translated to machine
> >> > > addresses.
> >> > >
> >> > > This patch introduces a simple framework to perform
> >> > > pfn -> mfn translation for external MMUs.
> >> > > It introduces basic data structures and algorithms
> >> > > needed for translation.
> >> > >
> >> > > Typically, when MMU is configured, some it registers
> >> > > are updated by new values. Introduced frameworks
> >> > > uses traps as starting point of remoteproc MMUs
> >> > > pagetables translation.
> >> > >
> >> > > Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
> >> > > Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> >> >
> >> > There is one problem with this patch: you need to find a way to "pin"
> >> > the p2m entries for the pfns and mfns found in the pagetables translated
> >> > by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
> >> > mfn mappings for the domain, breaking the pagetables already translated
> >> > by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
> >> > stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
> >> > under the guest feet, but features like memory sharing or swapping,
> >> > supported on x86, could cause it to happen.
> >>
> >> I've not fully grokked this patch but wouldn't it be easier and better
> >> to have a hook to cause the remoteproc to throw away and rebuild its
> >> mappings when this happens? It could be called from e.g. the TLB flush
> >> associated with the p2m changing, I think.
> >
> > I guess it depends on how often Xen or the Xen tools are going to change
> > the p2m for a running domain. For example changing one page mapping a
> > couple of times a second would be very expensive if we have to rebuild
> > the remoteproc pagetables every time.
>
> Sorry, I didn't get the point here. Looks like you are talking about
> TLB flush initiated by Xen? Am I right?
p2m stands for physical to machine (mappings) or IPA to PA using ARM
terminology. Xen is free to change IPA to PA mappings of a VM without
the VM knowing, while the VM is running. Nothing in Xen on ARM does it
yet, but a few features do it today on Xen on x86. For example memory
sharing and swapping.
My first suggestion was to "pin" the p2m mappings of pages used in
remoteproc pagetables, so that their IPA to PA mappings wouldn't change.
Ian suggested to rebuild the remoteproc pagetables if one of the IPA to
PA mappings change.
My reply here is that the performance could suffer if IPA to PA mappings
are changed often enough.
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-23 10:32 ` Stefano Stabellini
@ 2014-07-23 10:54 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-23 10:54 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Julien Grall, Ian Campbell, xen-devel
Oh, Thank you. Clear enough, I would prefer to "pin" remoteproc
mappings, I would say that this is not possible to rewrite pagetable
at any moment of time. For example - during video playback IPU
allocates and frees buffers. The whole amount of memory, mapped to its
MMU is about 256 Mb. If Xen decides to change p2m mappings at this
moment of time, this will lead to significant playback lags. More than
- playback may crash.
Regards,
Andrii
On Wed, Jul 23, 2014 at 1:32 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 22 Jul 2014, Andrii Tseglytskyi wrote:
>> Hi Stefano,
>>
>> On Tue, Jul 22, 2014 at 3:42 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Wed, 16 Jul 2014, Ian Campbell wrote:
>> >> On Fri, 2014-07-04 at 14:59 +0100, Stefano Stabellini wrote:
>> >> > On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>> >> > > This is a fisrst patch from patch series which was
>> >> > > developed to handle remote (external) processors
>> >> > > memory management units. Remote processors are
>> >> > > typically used for graphic rendering (GPUs) and
>> >> > > high quality video decoding (IPUs). They are typically
>> >> > > installed on such multimedia SoCs as OMAP4 / OMAP5.
>> >> > >
>> >> > > As soon as remoteprocessor MMU typically works with
>> >> > > pagetables filled by physical addresses, which are
>> >> > > allocated by domU kernel, it is almost impossible to
>> >> > > use them under Xen, intermediate physical addresses
>> >> > > allocated by kernel, need to be translated to machine
>> >> > > addresses.
>> >> > >
>> >> > > This patch introduces a simple framework to perform
>> >> > > pfn -> mfn translation for external MMUs.
>> >> > > It introduces basic data structures and algorithms
>> >> > > needed for translation.
>> >> > >
>> >> > > Typically, when MMU is configured, some it registers
>> >> > > are updated by new values. Introduced frameworks
>> >> > > uses traps as starting point of remoteproc MMUs
>> >> > > pagetables translation.
>> >> > >
>> >> > > Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>> >> > > Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> >> >
>> >> > There is one problem with this patch: you need to find a way to "pin"
>> >> > the p2m entries for the pfns and mfns found in the pagetables translated
>> >> > by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
>> >> > mfn mappings for the domain, breaking the pagetables already translated
>> >> > by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
>> >> > stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
>> >> > under the guest feet, but features like memory sharing or swapping,
>> >> > supported on x86, could cause it to happen.
>> >>
>> >> I've not fully grokked this patch but wouldn't it be easier and better
>> >> to have a hook to cause the remoteproc to throw away and rebuild its
>> >> mappings when this happens? It could be called from e.g. the TLB flush
>> >> associated with the p2m changing, I think.
>> >
>> > I guess it depends on how often Xen or the Xen tools are going to change
>> > the p2m for a running domain. For example changing one page mapping a
>> > couple of times a second would be very expensive if we have to rebuild
>> > the remoteproc pagetables every time.
>>
>> Sorry, I didn't get the point here. Looks like you are talking about
>> TLB flush initiated by Xen? Am I right?
>
> p2m stands for physical to machine (mappings) or IPA to PA using ARM
> terminology. Xen is free to change IPA to PA mappings of a VM without
> the VM knowing, while the VM is running. Nothing in Xen on ARM does it
> yet, but a few features do it today on Xen on x86. For example memory
> sharing and swapping.
>
> My first suggestion was to "pin" the p2m mappings of pages used in
> remoteproc pagetables, so that their IPA to PA mappings wouldn't change.
> Ian suggested to rebuild the remoteproc pagetables if one of the IPA to
> PA mappings change.
> My reply here is that the performance could suffer if IPA to PA mappings
> are changed often enough.
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-22 15:20 ` Andrii Tseglytskyi
2014-07-22 16:29 ` Julien Grall
@ 2014-07-31 11:59 ` Andrii Tseglytskyi
2014-07-31 12:11 ` Julien Grall
1 sibling, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-31 11:59 UTC (permalink / raw)
To: Julien Grall; +Cc: Stefano Stabellini, Tim Deegan, Ian Campbell, xen-devel
Hi Julien,
Could you please clarify:
>
> >> + /* pagetable size can be more than one page */
> >> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
> >> + {
> >> + /* lookup address where remoteproc pagetable is stored by kernel
> >> */
> >> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE,
> >> NULL);
> >> + if ( !maddr )
> >> + {
> >> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i
> >> * PAGE_SIZE);
> >> + return -EINVAL;
> >> + }
> >> +
> >> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
> >
> >
> > ioremap_* should only be used to map device memory. For the guest memory you
> > have to use copy_from_guest helper.
> >
>
> OK. Just a small question - if I use copy_from_guest(), can I copy
> from physical pointer ? Here I have an address, which points to exact
> physical memory.
>
I'm not sure that this works in my case. I see that copy_from_user /
copy_to_user deals with domain virtual addresses and it is widely
used in hypercalls. Here I need to copy from a pointer to physical
memory - iow from intermediate physical address. Can I use this macro
in this case? Maybe copy_from_user / copy_to_user can be slightly
modified to work with IPAs ?
Regards,
Andrii
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-31 11:59 ` Andrii Tseglytskyi
@ 2014-07-31 12:11 ` Julien Grall
2014-07-31 12:49 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-07-31 12:11 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Stefano Stabellini, Tim Deegan, Ian Campbell, xen-devel
On 07/31/2014 12:59 PM, Andrii Tseglytskyi wrote:
> Hi Julien,
Hi Andrii,
> Could you please clarify:
>
>>
>>>> + /* pagetable size can be more than one page */
>>>> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
>>>> + {
>>>> + /* lookup address where remoteproc pagetable is stored by kernel
>>>> */
>>>> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE,
>>>> NULL);
>>>> + if ( !maddr )
>>>> + {
>>>> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i
>>>> * PAGE_SIZE);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
>>>
>>>
>>> ioremap_* should only be used to map device memory. For the guest memory you
>>> have to use copy_from_guest helper.
>>>
>>
>> OK. Just a small question - if I use copy_from_guest(), can I copy
>> from physical pointer ? Here I have an address, which points to exact
>> physical memory.
>>
>
> I'm not sure that this works in my case. I see that copy_from_user /
> copy_to_user deals with domain virtual addresses and it is widely
> used in hypercalls. Here I need to copy from a pointer to physical
> memory - iow from intermediate physical address. Can I use this macro
> in this case? Maybe copy_from_user / copy_to_user can be slightly
> modified to work with IPAs ?
Right, copy_*_guest helpers are only able to cope with guest virtual
address.
The behavior of those function can't be modified because they are used
in common code.
You will have to introduce new helpers based on p2m_lookup and
map_domain_page.
There is an ongoing discussion about it here:
http://lists.xen.org/archives/html/xen-devel/2014-07/msg02096.html
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-31 12:11 ` Julien Grall
@ 2014-07-31 12:49 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-07-31 12:49 UTC (permalink / raw)
To: Julien Grall; +Cc: Stefano Stabellini, Tim Deegan, Ian Campbell, xen-devel
Hi
On Thu, Jul 31, 2014 at 3:11 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 07/31/2014 12:59 PM, Andrii Tseglytskyi wrote:
>> Hi Julien,
>
> Hi Andrii,
>
>
>> Could you please clarify:
>>
>>>
>>>>> + /* pagetable size can be more than one page */
>>>>> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
>>>>> + {
>>>>> + /* lookup address where remoteproc pagetable is stored by kernel
>>>>> */
>>>>> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE,
>>>>> NULL);
>>>>> + if ( !maddr )
>>>>> + {
>>>>> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i
>>>>> * PAGE_SIZE);
>>>>> + return -EINVAL;
>>>>> + }
>>>>> +
>>>>> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
>>>>
>>>>
>>>> ioremap_* should only be used to map device memory. For the guest memory you
>>>> have to use copy_from_guest helper.
>>>>
>>>
>>> OK. Just a small question - if I use copy_from_guest(), can I copy
>>> from physical pointer ? Here I have an address, which points to exact
>>> physical memory.
>>>
>>
>> I'm not sure that this works in my case. I see that copy_from_user /
>> copy_to_user deals with domain virtual addresses and it is widely
>> used in hypercalls. Here I need to copy from a pointer to physical
>> memory - iow from intermediate physical address. Can I use this macro
>> in this case? Maybe copy_from_user / copy_to_user can be slightly
>> modified to work with IPAs ?
>
> Right, copy_*_guest helpers are only able to cope with guest virtual
> address.
>
> The behavior of those function can't be modified because they are used
> in common code.
>
> You will have to introduce new helpers based on p2m_lookup and
> map_domain_page.
map_domain_page works fine for me, thank you for detailed explanation.
Regards,
Andrii
>
> There is an ongoing discussion about it here:
>
> http://lists.xen.org/archives/html/xen-devel/2014-07/msg02096.html
>
> Regards,
>
> --
> Julien Grall
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-22 15:32 ` Andrii Tseglytskyi
@ 2014-08-01 10:06 ` Andrii Tseglytskyi
2014-08-01 10:32 ` Julien Grall
0 siblings, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-08-01 10:06 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Julien Grall, Ian Campbell, xen-devel
Hi Stefano,
Could you lease clarify:
>>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
>>> +{
>>> + struct mmu_info *mmu = NULL;
>>> + unsigned long flags;
>>> + register_t *r;
>>> + u32 new_addr, val;
>>> +
>>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>>> +
>>> + ASSERT(r);
>>> +
>>> + /* dom0 should not access remoteproc MMU */
>>> + if ( 0 == current->domain->domain_id )
>>> + return 1;
>>
>> This is too specific to one particular configuration.
>> Would it be possible to generalize this somehow? At the very least you
>> could introduce an XSM label to access the pagetables, so that you can
>> dynamically configure the domains the can write to them.
>>
>
> I need to think about this. Sounds reasonable.
>
I tried to compile Xen with XSM support and got an error with
including msi.h file:
arm-linux-gnueabi-gcc -O1 -fno-omit-frame-pointer -marm -g
-fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes
-Wdeclaration-after-statement -Wno-unused-but-set-variable
-I/home/x0174653/xen/dra7/core_dra7/hypervisor/xen/include -nopie
-fno-stack-protector -fno-exceptions -Wnested-externs -msoft-float
-mcpu=cortex-a15 -DGCC_HAS_VISIBILITY_ATTRIBUTE -fno-builtin
-fno-common -Werror -Wredundant-decls -Wno-pointer-arith -pipe -g
-D__XEN__ -include
/home/x0174653/xen/dra7/core_dra7/hypervisor/xen/include/xen/config.h
-nostdinc -fno-optimize-sibling-calls -DXSM_ENABLE -DFLASK_ENABLE
-DXSM_MAGIC=0xf97cff8c -DFLASK_DEVELOP -DFLASK_BOOTPARAM
-DFLASK_AVC_STATS -DVERBOSE -DHAS_DEVICE_TREE -fno-omit-frame-pointer
-DCONFIG_FRAME_POINTER -MMD -MF .hooks.o.d -I./include -c hooks.c -o
hooks.o
hooks.c:22:21: fatal error: asm/msi.h: No such file or directory
I see that msi.h is available only for x86 platforms. So, the question
is - is XSM supported on ARMs ?
I used this link to started working with XSM
http://wiki.xen.org/wiki/Xen_Security_Modules_:_XSM-FLASK
>>
>>> + /* find corresponding MMU */
>>> + mmu = mmu_lookup(info->gpa);
>>> + if ( !mmu )
>>> + {
>>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + ASSERT(v->domain == current->domain);
>>
>> You can remove this assert.
>>
>
> OK.
>
>>
>>> + spin_lock_irqsave(&mmu->lock, flags);
>>> +
>>> + /* get new address of translated pagetable */
>>> + new_addr = mmu_trap_translate_pagetable(mmu, info);
>>> + if ( MMU_INVALID_ADDRESS != new_addr )
>>> + val = new_addr;
>>> + else
>>> + val = *r;
>>> +
>>> + writel(val, mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
>>> + spin_unlock_irqrestore(&mmu->lock, flags);
>>> +
>>> + return 1;
>>> +}
>>> +
>>> +static int mmu_init(struct mmu_info *mmu, u32 data)
>>> +{
>>> + ASSERT(mmu);
>>> + ASSERT(!mmu->mem_map);
>>> +
>>> + INIT_LIST_HEAD(&mmu->pagetables_list);
>>> +
>>> + /* map MMU memory */
>>> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
>>> + if ( !mmu->mem_map )
>>> + {
>>> + pr_mmu("failed to map memory");
>>> + return -EINVAL;
>>> + }
>>> +
>>> + pr_mmu("memory map = 0x%pS", _p(mmu->mem_map));
>>> +
>>> + spin_lock_init(&mmu->lock);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int mmu_init_all(void)
>>> +{
>>> + int res;
>>> +
>>> + res = mmu_for_each(mmu_init, 0);
>>> + if ( res )
>>> + {
>>> + printk("%s error during init %d\n", __func__, res);
>>> + return res;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +const struct mmio_handler remoteproc_mmio_handler = {
>>> + .check_handler = mmu_mmio_check,
>>> + .read_handler = mmu_mmio_read,
>>> + .write_handler = mmu_mmio_write,
>>> +};
>>> +
>>> +__initcall(mmu_init_all);
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
>>> new file mode 100644
>>> index 0000000..22e2951
>>> --- /dev/null
>>> +++ b/xen/include/xen/remoteproc_iommu.h
>>> @@ -0,0 +1,79 @@
>>> +/*
>>> + * xen/include/xen/remoteproc_iommu.h
>>> + *
>>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>>> + * Copyright (c) 2014 GlobalLogic
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; either version 2 of the License, or
>>> + * (at your option) any later version.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + */
>>> +
>>> +#ifndef _REMOTEPROC_IOMMU_H_
>>> +#define _REMOTEPROC_IOMMU_H_
>>> +
>>> +#include <asm/types.h>
>>> +
>>> +#define MMU_SECTION_SIZE(shift) (1UL << (shift))
>>> +#define MMU_SECTION_MASK(shift) (~(MMU_SECTION_SIZE(shift) - 1))
>>> +
>>> +/* 4096 first level descriptors for "supersection" and "section" */
>>> +#define MMU_PTRS_PER_PGD(mmu) (1UL << (32 - (mmu->pg_data->pgd_shift)))
>>> +#define MMU_PGD_TABLE_SIZE(mmu) (MMU_PTRS_PER_PGD(mmu) * sizeof(u32))
>>> +
>>> +/* 256 second level descriptors for "small" and "large" pages */
>>> +#define MMU_PTRS_PER_PTE(mmu) (1UL << ((mmu->pg_data->pgd_shift) - (mmu->pg_data->pte_shift)))
>>> +#define MMU_PTE_TABLE_SIZE(mmu) (MMU_PTRS_PER_PTE(mmu) * sizeof(u32))
>>> +
>>> +/* 16 sections in supersection */
>>> +#define MMU_SECTION_PER_SUPER(mmu) (1UL << ((mmu->pg_data->super_shift) - (mmu->pg_data->section_shift)))
>>> +
>>> +#define MMU_INVALID_ADDRESS ((u32)(-1))
>>> +
>>> +#define pr_mmu(fmt, ...) \
>>> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
>>> +
>>> +struct pagetable_data {
>>> + /* 1st level translation */
>>> + u32 pgd_shift;
>>> + u32 pte_shift;
>>> + u32 super_shift;
>>> + u32 section_shift;
>>> + /* 2nd level translation */
>>> + u32 pte_large_shift;
>>> +};
>>> +
>>> +struct mmu_pagetable {
>>> + u32 *hyp_pagetable;
>>> + u32 *kern_pagetable;
>>> + u32 paddr;
>>> + u32 maddr;
>>> + struct list_head link_node;
>>> + u32 page_counter;
>>> +};
>>> +
>>> +struct mmu_info {
>>> + const char *name;
>>> + const struct pagetable_data *pg_data;
>>> + /* register where phys pointer to pagetable is stored */
>>> + u32 *trap_offsets;
>>> + paddr_t mem_start;
>>> + u32 mem_size;
>>> + spinlock_t lock;
>>> + struct list_head pagetables_list;
>>> + u32 num_traps;
>>> + void __iomem *mem_map;
>>> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
>>> + void (*print_pagetable_pfunc)(struct mmu_info *);
>>> +};
>>> +
>>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
>>> + u32 maddr, u32 hyp_addr);
>>> +
>>> +#endif /* _REMOTEPROC_IOMMU_H_ */
>>> --
>>> 1.7.9.5
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>>>
>
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-08-01 10:06 ` Andrii Tseglytskyi
@ 2014-08-01 10:32 ` Julien Grall
2014-08-01 10:34 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-08-01 10:32 UTC (permalink / raw)
To: Andrii Tseglytskyi, Stefano Stabellini
Cc: Julien Grall, Ian Campbell, xen-devel
Hi Andrii,
On 01/08/14 11:06, Andrii Tseglytskyi wrote:
>>>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
>>>> +{
>>>> + struct mmu_info *mmu = NULL;
>>>> + unsigned long flags;
>>>> + register_t *r;
>>>> + u32 new_addr, val;
>>>> +
>>>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>>>> +
>>>> + ASSERT(r);
>>>> +
>>>> + /* dom0 should not access remoteproc MMU */
>>>> + if ( 0 == current->domain->domain_id )
>>>> + return 1;
>>>
>>> This is too specific to one particular configuration.
>>> Would it be possible to generalize this somehow? At the very least you
>>> could introduce an XSM label to access the pagetables, so that you can
>>> dynamically configure the domains the can write to them.
>>>
>>
>> I need to think about this. Sounds reasonable.
>>
>
> I tried to compile Xen with XSM support and got an error with
> including msi.h file:
>
> arm-linux-gnueabi-gcc -O1 -fno-omit-frame-pointer -marm -g
> -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes
> -Wdeclaration-after-statement -Wno-unused-but-set-variable
> -I/home/x0174653/xen/dra7/core_dra7/hypervisor/xen/include -nopie
> -fno-stack-protector -fno-exceptions -Wnested-externs -msoft-float
> -mcpu=cortex-a15 -DGCC_HAS_VISIBILITY_ATTRIBUTE -fno-builtin
> -fno-common -Werror -Wredundant-decls -Wno-pointer-arith -pipe -g
> -D__XEN__ -include
> /home/x0174653/xen/dra7/core_dra7/hypervisor/xen/include/xen/config.h
> -nostdinc -fno-optimize-sibling-calls -DXSM_ENABLE -DFLASK_ENABLE
> -DXSM_MAGIC=0xf97cff8c -DFLASK_DEVELOP -DFLASK_BOOTPARAM
> -DFLASK_AVC_STATS -DVERBOSE -DHAS_DEVICE_TREE -fno-omit-frame-pointer
> -DCONFIG_FRAME_POINTER -MMD -MF .hooks.o.d -I./include -c hooks.c -o
> hooks.o
> hooks.c:22:21: fatal error: asm/msi.h: No such file or directory
>
> I see that msi.h is available only for x86 platforms. So, the question
> is - is XSM supported on ARMs ?
It's supported only on Xen unstable since last march. On which branch
are you working on?
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-08-01 10:32 ` Julien Grall
@ 2014-08-01 10:34 ` Andrii Tseglytskyi
2014-08-01 10:37 ` Julien Grall
0 siblings, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-08-01 10:34 UTC (permalink / raw)
To: Julien Grall; +Cc: Julien Grall, xen-devel, Ian Campbell, Stefano Stabellini
Hi,
I'm working with Xen 4.4 release, looks like I need to switch to
latest Xen, right ?
Regards,
Andrii
On Fri, Aug 1, 2014 at 1:32 PM, Julien Grall <julien.grall@linaro.org> wrote:
>
> Hi Andrii,
>
>
> On 01/08/14 11:06, Andrii Tseglytskyi wrote:
>>>>>
>>>>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
>>>>> +{
>>>>> + struct mmu_info *mmu = NULL;
>>>>> + unsigned long flags;
>>>>> + register_t *r;
>>>>> + u32 new_addr, val;
>>>>> +
>>>>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>>>>> +
>>>>> + ASSERT(r);
>>>>> +
>>>>> + /* dom0 should not access remoteproc MMU */
>>>>> + if ( 0 == current->domain->domain_id )
>>>>> + return 1;
>>>>
>>>>
>>>> This is too specific to one particular configuration.
>>>> Would it be possible to generalize this somehow? At the very least you
>>>> could introduce an XSM label to access the pagetables, so that you can
>>>> dynamically configure the domains the can write to them.
>>>>
>>>
>>> I need to think about this. Sounds reasonable.
>>>
>>
>> I tried to compile Xen with XSM support and got an error with
>> including msi.h file:
>>
>> arm-linux-gnueabi-gcc -O1 -fno-omit-frame-pointer -marm -g
>> -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes
>> -Wdeclaration-after-statement -Wno-unused-but-set-variable
>> -I/home/x0174653/xen/dra7/core_dra7/hypervisor/xen/include -nopie
>> -fno-stack-protector -fno-exceptions -Wnested-externs -msoft-float
>> -mcpu=cortex-a15 -DGCC_HAS_VISIBILITY_ATTRIBUTE -fno-builtin
>> -fno-common -Werror -Wredundant-decls -Wno-pointer-arith -pipe -g
>> -D__XEN__ -include
>> /home/x0174653/xen/dra7/core_dra7/hypervisor/xen/include/xen/config.h
>> -nostdinc -fno-optimize-sibling-calls -DXSM_ENABLE -DFLASK_ENABLE
>> -DXSM_MAGIC=0xf97cff8c -DFLASK_DEVELOP -DFLASK_BOOTPARAM
>> -DFLASK_AVC_STATS -DVERBOSE -DHAS_DEVICE_TREE -fno-omit-frame-pointer
>> -DCONFIG_FRAME_POINTER -MMD -MF .hooks.o.d -I./include -c hooks.c -o
>> hooks.o
>> hooks.c:22:21: fatal error: asm/msi.h: No such file or directory
>>
>> I see that msi.h is available only for x86 platforms. So, the question
>> is - is XSM supported on ARMs ?
>
>
> It's supported only on Xen unstable since last march. On which branch are
> you working on?
>
> Regards,
>
> --
> Julien Grall
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-08-01 10:34 ` Andrii Tseglytskyi
@ 2014-08-01 10:37 ` Julien Grall
2014-08-01 10:43 ` Andrii Tseglytskyi
0 siblings, 1 reply; 57+ messages in thread
From: Julien Grall @ 2014-08-01 10:37 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Julien Grall, xen-devel, Ian Campbell, Stefano Stabellini
On 01/08/14 11:34, Andrii Tseglytskyi wrote:
> Hi,
Hi,
> I'm working with Xen 4.4 release, looks like I need to switch to
> latest Xen, right ?
Yes. In general when you send a patch series, you have to test it on Xen
4.5. The internal API may have changes,...
In case of ARM, we made changes on the TLBs and cache API.
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-08-01 10:37 ` Julien Grall
@ 2014-08-01 10:43 ` Andrii Tseglytskyi
0 siblings, 0 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-08-01 10:43 UTC (permalink / raw)
To: Julien Grall; +Cc: Julien Grall, xen-devel, Ian Campbell, Stefano Stabellini
Hi Julien,
On Fri, Aug 1, 2014 at 1:37 PM, Julien Grall <julien.grall@linaro.org> wrote:
>
>
> On 01/08/14 11:34, Andrii Tseglytskyi wrote:
>>
>> Hi,
>
>
> Hi,
>
>> I'm working with Xen 4.4 release, looks like I need to switch to
>> latest Xen, right ?
>
>
> Yes. In general when you send a patch series, you have to test it on Xen
> 4.5. The internal API may have changes,...
>
> In case of ARM, we made changes on the TLBs and cache API.
>
OK. Thank you for clarification.
> Regards,
>
> --
> Julien Grall
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-07-04 13:59 ` Stefano Stabellini
2014-07-16 15:19 ` Ian Campbell
2014-07-22 15:32 ` Andrii Tseglytskyi
@ 2014-08-20 19:40 ` Andrii Tseglytskyi
2014-08-21 15:30 ` Andrii Tseglytskyi
2014-08-21 23:43 ` Stefano Stabellini
2 siblings, 2 replies; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-08-20 19:40 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Julien Grall, Ian Campbell, xen-devel
Hi Stefano,
On Fri, Jul 4, 2014 at 4:59 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>> This is a fisrst patch from patch series which was
>> developed to handle remote (external) processors
>> memory management units. Remote processors are
>> typically used for graphic rendering (GPUs) and
>> high quality video decoding (IPUs). They are typically
>> installed on such multimedia SoCs as OMAP4 / OMAP5.
>>
>> As soon as remoteprocessor MMU typically works with
>> pagetables filled by physical addresses, which are
>> allocated by domU kernel, it is almost impossible to
>> use them under Xen, intermediate physical addresses
>> allocated by kernel, need to be translated to machine
>> addresses.
>>
>> This patch introduces a simple framework to perform
>> pfn -> mfn translation for external MMUs.
>> It introduces basic data structures and algorithms
>> needed for translation.
>>
>> Typically, when MMU is configured, some it registers
>> are updated by new values. Introduced frameworks
>> uses traps as starting point of remoteproc MMUs
>> pagetables translation.
>>
>> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>
> There is one problem with this patch: you need to find a way to "pin"
> the p2m entries for the pfns and mfns found in the pagetables translated
> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
> mfn mappings for the domain, breaking the pagetables already translated
> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
> under the guest feet, but features like memory sharing or swapping,
> supported on x86, could cause it to happen.
>
> In the past I tried to introduce a way to "pin the mappings, but never
> finished to upstream the patches, because it didn't need it anymore.
>
>
> Give a look at:
>
> http://marc.info/?l=xen-devel&m=138029864707973
>
> In particular you would need to introduce something like
> the pin function and call it from mmu_translate_pagetable.
>
What if I try your series and re-send it if helpful for me? Can you
link me to its latest code?
>
>
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/remoteproc_iommu.c | 412 ++++++++++++++++++++++++++++++++++++
>> xen/include/xen/remoteproc_iommu.h | 79 +++++++
>> 3 files changed, 492 insertions(+)
>> create mode 100644 xen/arch/arm/remoteproc_iommu.c
>> create mode 100644 xen/include/xen/remoteproc_iommu.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index d70f6d5..0204d1c 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -15,6 +15,7 @@ obj-y += io.o
>> obj-y += irq.o
>> obj-y += kernel.o
>> obj-y += mm.o
>> +obj-y += remoteproc_iommu.o
>> obj-y += p2m.o
>> obj-y += percpu.o
>> obj-y += guestcopy.o
>> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
>> new file mode 100644
>> index 0000000..b4d22d9
>> --- /dev/null
>> +++ b/xen/arch/arm/remoteproc_iommu.c
>> @@ -0,0 +1,412 @@
>> +/*
>> + * xen/arch/arm/remoteproc_iommu.c
>> + *
>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> + * Copyright (c) 2014 GlobalLogic
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <xen/config.h>
>> +#include <xen/lib.h>
>> +#include <xen/errno.h>
>> +#include <xen/mm.h>
>> +#include <xen/vmap.h>
>> +#include <xen/init.h>
>> +#include <xen/sched.h>
>> +#include <xen/stdbool.h>
>> +#include <asm/system.h>
>> +#include <asm/current.h>
>> +#include <asm/io.h>
>> +#include <asm/p2m.h>
>> +
>> +#include <xen/remoteproc_iommu.h>
>> +
>> +#include "io.h"
>> +
>> +static struct mmu_info *mmu_list[] = {
>> +};
>> +
>> +#define mmu_for_each(pfunc, data) \
>> +({ \
>> + u32 __i; \
>> + int __res = 0; \
>> + \
>> + for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
>> + { \
>> + __res |= pfunc(mmu_list[__i], data); \
>> + } \
>> + __res; \
>> +})
>> +
>> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
>> +{
>> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start + mmu->mem_size)) )
>> + return 1;
>> +
>> + return 0;
>> +}
>> +
>> +static inline struct mmu_info *mmu_lookup(u32 addr)
>> +{
>> + u32 i;
>> +
>> + /* enumerate all registered MMU's and check is address in range */
>> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
>> + {
>> + if ( mmu_check_mem_range(mmu_list[i], addr) )
>> + return mmu_list[i];
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
>> +{
>> + return mmu_for_each(mmu_check_mem_range, addr);
>> +}
>> +
>> +static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
>> +{
>> + void __iomem *pagetable = NULL;
>> + u32 maddr, i;
>> +
>> + ASSERT(mmu);
>> + ASSERT(pgt);
>> +
>> + if ( !pgt->paddr )
>> + return -EINVAL;
>> +
>> + /* pagetable size can be more than one page */
>> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
>> + {
>> + /* lookup address where remoteproc pagetable is stored by kernel */
>> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE, NULL);
>> + if ( !maddr )
>> + {
>> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i * PAGE_SIZE);
>> + return -EINVAL;
>> + }
>> +
>> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
>> + if ( !pagetable )
>> + {
>> + pr_mmu("failed to map pagetable");
>> + return -EINVAL;
>> + }
>> +
>> + /* copy pagetable to hypervisor memory */
>> + clean_and_invalidate_xen_dcache_va_range(pagetable, PAGE_SIZE);
>> + memcpy((u32*)((u32)pgt->kern_pagetable + i * PAGE_SIZE), pagetable, PAGE_SIZE);
>> +
>> + iounmap(pagetable);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32 addr, bool is_maddr)
>> +{
>> + struct mmu_pagetable *pgt;
>> + u32 pgt_addr;
>> +
>> + list_for_each_entry(pgt, &mmu->pagetables_list, link_node)
>> + {
>> + if ( is_maddr )
>> + pgt_addr = pgt->maddr;
>> + else
>> + pgt_addr = pgt->paddr;
>> +
>> + if ( pgt_addr == addr )
>> + return pgt;
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static struct mmu_pagetable *mmu_alloc_pagetable(struct mmu_info *mmu, u32 paddr)
>> +{
>> + struct mmu_pagetable *pgt;
>> + u32 pgt_size = MMU_PGD_TABLE_SIZE(mmu);
>> +
>> + pgt = xzalloc_bytes(sizeof(struct mmu_pagetable));
>> + if ( !pgt )
>> + {
>> + pr_mmu("failed to alloc pagetable structure");
>> + return NULL;
>> + }
>> +
>> + /* allocate pagetable managed by hypervisor */
>> + pgt->hyp_pagetable = xzalloc_bytes(pgt_size);
>> + if ( !pgt->hyp_pagetable )
>> + {
>> + pr_mmu("failed to alloc private hyp_pagetable");
>> + return NULL;
>> + }
>> +
>> + /* alocate pagetable for ipa storing */
>> + pgt->kern_pagetable = xzalloc_bytes(pgt_size);
>> + if ( !pgt->kern_pagetable )
>> + {
>> + pr_mmu("failed to alloc private kern_pagetable");
>> + return NULL;
>> + }
>> +
>> + pr_mmu("private pagetables for 0x%08x paddr %u bytes (main 0x%08x, temp 0x%08x)",
>> + paddr, pgt_size, (u32)__pa(pgt->hyp_pagetable), (u32)__pa(pgt->kern_pagetable));
>> +
>> + pgt->paddr = paddr;
>> +
>> + list_add(&pgt->link_node, &mmu->pagetables_list);
>> +
>> + return pgt;
>> +}
>> +
>> +static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
>> +{
>> + struct mmu_pagetable *pgt;
>> + int res;
>> +
>> + /* lookup using machine address first */
>> + pgt = mmu_pagetable_lookup(mmu, paddr, true);
>> + if ( !pgt )
>> + {
>> + /* lookup using kernel physical address */
>> + pgt = mmu_pagetable_lookup(mmu, paddr, false);
>> + if ( !pgt )
>> + {
>> + /* if pagetable doesn't exists in lookup list - allocate it */
>> + pgt = mmu_alloc_pagetable(mmu, paddr);
>> + }
>> + }
>> +
>> + pgt->maddr = MMU_INVALID_ADDRESS;
>> +
>> + /* copy pagetable from domain to hypervisor */
>> + res = mmu_copy_pagetable(mmu, pgt);
>> + if ( res )
>> + return res;
>> +
>> + /* translate pagetable */
>> + pgt->maddr = mmu->translate_pfunc(mmu, pgt);
>> + return pgt->maddr;
>> +}
>> +
>> +static u32 mmu_trap_translate_pagetable(struct mmu_info *mmu, mmio_info_t *info)
>> +{
>> + register_t *reg;
>> + bool valid_trap = false;
>> + u32 i, paddr;
>> +
>> + reg = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(reg);
>> +
>> + paddr = *reg;
>> + if ( !paddr )
>> + return MMU_INVALID_ADDRESS;
>> +
>> + /* check is the register is a valid TTB register */
>> + for ( i = 0; i < mmu->num_traps; i++ )
>> + {
>> + if ( mmu->trap_offsets[i] == (info->gpa - mmu->mem_start) )
>> + {
>> + valid_trap = true;
>> + break;
>> + }
>> + }
>> +
>> + if ( !valid_trap )
>> + return MMU_INVALID_ADDRESS;
>> +
>> + return mmu_translate_pagetable(mmu, paddr);
>> +}
>> +
>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
>> + u32 maddr, u32 hyp_addr)
>
> Is this actually used anywhere?
>
>
>> +{
>> + u32 *pte_table = NULL, *hyp_pte_table = NULL, pte_table_size = PAGE_SIZE;
>> + u32 i;
>> +
>> + /* remap second level translation table */
>> + pte_table = ioremap_nocache(maddr, MMU_PTE_TABLE_SIZE(mmu));
>> + if ( !pte_table )
>> + {
>> + pr_mmu("failed to map pte_table");
>> + return MMU_INVALID_ADDRESS;
>> + }
>> +
>> + /* allocate new second level pagetable once */
>> + if ( 0 == hyp_addr )
>> + {
>> + if ( MMU_PTE_TABLE_SIZE(mmu) > PAGE_SIZE )
>> + pte_table_size = MMU_PTE_TABLE_SIZE(mmu);
>> +
>> + hyp_pte_table = xzalloc_bytes(pte_table_size);
>> + if ( !hyp_pte_table )
>> + {
>> + pr_mmu("failed to alloc new iopt");
>> + return MMU_INVALID_ADDRESS;
>> + }
>> + }
>> + else
>> + {
>> + hyp_pte_table = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
>> + }
>> +
>> + /* 2-nd level translation */
>> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
>> + {
>> + paddr_t pt_maddr, pt_paddr, pt_flags;
>> + u32 pt_mask = MMU_SECTION_MASK(mmu->pg_data->pte_shift);
>> +
>> + if ( !pte_table[i] )
>> + {
>> + /* handle the case when page was removed */
>> + if ( unlikely(hyp_pte_table[i]) )
>> + {
>> + hyp_pte_table[i] = 0;
>> + }
>> +
>> + continue;
>> + }
>> +
>> + pt_paddr = pte_table[i] & pt_mask;
>> + pt_flags = pte_table[i] & ~pt_mask;
>> + pt_maddr = p2m_lookup(current->domain, pt_paddr, NULL);
>> + ASSERT(pt_maddr != INVALID_PADDR);
>> +
>> + hyp_pte_table[i] = pt_maddr | pt_flags;
>> + pgt->page_counter++;
>> + }
>> +
>> + iounmap(pte_table);
>> +
>> + clean_and_invalidate_xen_dcache_va_range(hyp_pte_table, MMU_PTE_TABLE_SIZE(mmu));
>> + return __pa(hyp_pte_table);
>> +}
>> +
>> +static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
>> +{
>> + struct mmu_info *mmu = NULL;
>> + unsigned long flags;
>> + register_t *r;
>> +
>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(r);
>
> If you check on the domid in mmu_mmio_write, I would check here too
>
>
>> + mmu = mmu_lookup(info->gpa);
>> + if ( !mmu )
>> + {
>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>> + return -EINVAL;
>> + }
>> +
>> + spin_lock_irqsave(&mmu->lock, flags);
>> + *r = readl(mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
>> + spin_unlock_irqrestore(&mmu->lock, flags);
>> +
>> + return 1;
>
> It looks like you are returning the mfns here, is that correct?
>
>
>
>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
>> +{
>> + struct mmu_info *mmu = NULL;
>> + unsigned long flags;
>> + register_t *r;
>> + u32 new_addr, val;
>> +
>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>> +
>> + ASSERT(r);
>> +
>> + /* dom0 should not access remoteproc MMU */
>> + if ( 0 == current->domain->domain_id )
>> + return 1;
>
> This is too specific to one particular configuration.
> Would it be possible to generalize this somehow? At the very least you
> could introduce an XSM label to access the pagetables, so that you can
> dynamically configure the domains the can write to them.
>
I succeeded to enable XSM on our platform and enforce FLASK on boot,
but I still have some questions:
- What is this new XSM label ? Do you mean
*seclabel='system_u:system_r:domU_t'* and change value of seclabel to
something specific?
As described in
http://wiki.xen.org/wiki/Xen_Security_Modules_:_XSM-FLASK#Creating_domains_with_XSM_security_labels
- Do I need to introduce a new policy for this? As far as I understand
from previous link - it is obligatory for FLASK, therefore it is
obligatory for XSM. In other words - all XSM related checks, which I
may introduce won't work correctly without installing a policy, am I
right ?
Regards,
Andrii
>
>> + /* find corresponding MMU */
>> + mmu = mmu_lookup(info->gpa);
>> + if ( !mmu )
>> + {
>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>> + return -EINVAL;
>> + }
>> +
>> + ASSERT(v->domain == current->domain);
>
> You can remove this assert.
>
>
>> + spin_lock_irqsave(&mmu->lock, flags);
>> +
>> + /* get new address of translated pagetable */
>> + new_addr = mmu_trap_translate_pagetable(mmu, info);
>> + if ( MMU_INVALID_ADDRESS != new_addr )
>> + val = new_addr;
>> + else
>> + val = *r;
>> +
>> + writel(val, mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
>> + spin_unlock_irqrestore(&mmu->lock, flags);
>> +
>> + return 1;
>> +}
>> +
>> +static int mmu_init(struct mmu_info *mmu, u32 data)
>> +{
>> + ASSERT(mmu);
>> + ASSERT(!mmu->mem_map);
>> +
>> + INIT_LIST_HEAD(&mmu->pagetables_list);
>> +
>> + /* map MMU memory */
>> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
>> + if ( !mmu->mem_map )
>> + {
>> + pr_mmu("failed to map memory");
>> + return -EINVAL;
>> + }
>> +
>> + pr_mmu("memory map = 0x%pS", _p(mmu->mem_map));
>> +
>> + spin_lock_init(&mmu->lock);
>> +
>> + return 0;
>> +}
>> +
>> +static int mmu_init_all(void)
>> +{
>> + int res;
>> +
>> + res = mmu_for_each(mmu_init, 0);
>> + if ( res )
>> + {
>> + printk("%s error during init %d\n", __func__, res);
>> + return res;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +const struct mmio_handler remoteproc_mmio_handler = {
>> + .check_handler = mmu_mmio_check,
>> + .read_handler = mmu_mmio_read,
>> + .write_handler = mmu_mmio_write,
>> +};
>> +
>> +__initcall(mmu_init_all);
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
>> new file mode 100644
>> index 0000000..22e2951
>> --- /dev/null
>> +++ b/xen/include/xen/remoteproc_iommu.h
>> @@ -0,0 +1,79 @@
>> +/*
>> + * xen/include/xen/remoteproc_iommu.h
>> + *
>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>> + * Copyright (c) 2014 GlobalLogic
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef _REMOTEPROC_IOMMU_H_
>> +#define _REMOTEPROC_IOMMU_H_
>> +
>> +#include <asm/types.h>
>> +
>> +#define MMU_SECTION_SIZE(shift) (1UL << (shift))
>> +#define MMU_SECTION_MASK(shift) (~(MMU_SECTION_SIZE(shift) - 1))
>> +
>> +/* 4096 first level descriptors for "supersection" and "section" */
>> +#define MMU_PTRS_PER_PGD(mmu) (1UL << (32 - (mmu->pg_data->pgd_shift)))
>> +#define MMU_PGD_TABLE_SIZE(mmu) (MMU_PTRS_PER_PGD(mmu) * sizeof(u32))
>> +
>> +/* 256 second level descriptors for "small" and "large" pages */
>> +#define MMU_PTRS_PER_PTE(mmu) (1UL << ((mmu->pg_data->pgd_shift) - (mmu->pg_data->pte_shift)))
>> +#define MMU_PTE_TABLE_SIZE(mmu) (MMU_PTRS_PER_PTE(mmu) * sizeof(u32))
>> +
>> +/* 16 sections in supersection */
>> +#define MMU_SECTION_PER_SUPER(mmu) (1UL << ((mmu->pg_data->super_shift) - (mmu->pg_data->section_shift)))
>> +
>> +#define MMU_INVALID_ADDRESS ((u32)(-1))
>> +
>> +#define pr_mmu(fmt, ...) \
>> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
>> +
>> +struct pagetable_data {
>> + /* 1st level translation */
>> + u32 pgd_shift;
>> + u32 pte_shift;
>> + u32 super_shift;
>> + u32 section_shift;
>> + /* 2nd level translation */
>> + u32 pte_large_shift;
>> +};
>> +
>> +struct mmu_pagetable {
>> + u32 *hyp_pagetable;
>> + u32 *kern_pagetable;
>> + u32 paddr;
>> + u32 maddr;
>> + struct list_head link_node;
>> + u32 page_counter;
>> +};
>> +
>> +struct mmu_info {
>> + const char *name;
>> + const struct pagetable_data *pg_data;
>> + /* register where phys pointer to pagetable is stored */
>> + u32 *trap_offsets;
>> + paddr_t mem_start;
>> + u32 mem_size;
>> + spinlock_t lock;
>> + struct list_head pagetables_list;
>> + u32 num_traps;
>> + void __iomem *mem_map;
>> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
>> + void (*print_pagetable_pfunc)(struct mmu_info *);
>> +};
>> +
>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
>> + u32 maddr, u32 hyp_addr);
>> +
>> +#endif /* _REMOTEPROC_IOMMU_H_ */
>> --
>> 1.7.9.5
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-08-20 19:40 ` Andrii Tseglytskyi
@ 2014-08-21 15:30 ` Andrii Tseglytskyi
2014-08-21 23:41 ` Stefano Stabellini
2014-08-21 23:43 ` Stefano Stabellini
1 sibling, 1 reply; 57+ messages in thread
From: Andrii Tseglytskyi @ 2014-08-21 15:30 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: Julien Grall, Ian Campbell, xen-devel
On Wed, Aug 20, 2014 at 10:40 PM, Andrii Tseglytskyi
<andrii.tseglytskyi@globallogic.com> wrote:
> Hi Stefano,
>
>
> On Fri, Jul 4, 2014 at 4:59 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
>>> This is a fisrst patch from patch series which was
>>> developed to handle remote (external) processors
>>> memory management units. Remote processors are
>>> typically used for graphic rendering (GPUs) and
>>> high quality video decoding (IPUs). They are typically
>>> installed on such multimedia SoCs as OMAP4 / OMAP5.
>>>
>>> As soon as remoteprocessor MMU typically works with
>>> pagetables filled by physical addresses, which are
>>> allocated by domU kernel, it is almost impossible to
>>> use them under Xen, intermediate physical addresses
>>> allocated by kernel, need to be translated to machine
>>> addresses.
>>>
>>> This patch introduces a simple framework to perform
>>> pfn -> mfn translation for external MMUs.
>>> It introduces basic data structures and algorithms
>>> needed for translation.
>>>
>>> Typically, when MMU is configured, some it registers
>>> are updated by new values. Introduced frameworks
>>> uses traps as starting point of remoteproc MMUs
>>> pagetables translation.
>>>
>>> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
>>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>>
>> There is one problem with this patch: you need to find a way to "pin"
>> the p2m entries for the pfns and mfns found in the pagetables translated
>> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
>> mfn mappings for the domain, breaking the pagetables already translated
>> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
>> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
>> under the guest feet, but features like memory sharing or swapping,
>> supported on x86, could cause it to happen.
>>
>> In the past I tried to introduce a way to "pin the mappings, but never
>> finished to upstream the patches, because it didn't need it anymore.
>>
>>
>> Give a look at:
>>
>> http://marc.info/?l=xen-devel&m=138029864707973
>>
>> In particular you would need to introduce something like
>> the pin function and call it from mmu_translate_pagetable.
>>
>
> What if I try your series and re-send it if helpful for me? Can you
> link me to its latest code?
>
>>
>>
>>> xen/arch/arm/Makefile | 1 +
>>> xen/arch/arm/remoteproc_iommu.c | 412 ++++++++++++++++++++++++++++++++++++
>>> xen/include/xen/remoteproc_iommu.h | 79 +++++++
>>> 3 files changed, 492 insertions(+)
>>> create mode 100644 xen/arch/arm/remoteproc_iommu.c
>>> create mode 100644 xen/include/xen/remoteproc_iommu.h
>>>
>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>> index d70f6d5..0204d1c 100644
>>> --- a/xen/arch/arm/Makefile
>>> +++ b/xen/arch/arm/Makefile
>>> @@ -15,6 +15,7 @@ obj-y += io.o
>>> obj-y += irq.o
>>> obj-y += kernel.o
>>> obj-y += mm.o
>>> +obj-y += remoteproc_iommu.o
>>> obj-y += p2m.o
>>> obj-y += percpu.o
>>> obj-y += guestcopy.o
>>> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
>>> new file mode 100644
>>> index 0000000..b4d22d9
>>> --- /dev/null
>>> +++ b/xen/arch/arm/remoteproc_iommu.c
>>> @@ -0,0 +1,412 @@
>>> +/*
>>> + * xen/arch/arm/remoteproc_iommu.c
>>> + *
>>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>>> + * Copyright (c) 2014 GlobalLogic
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; either version 2 of the License, or
>>> + * (at your option) any later version.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + */
>>> +
>>> +#include <xen/config.h>
>>> +#include <xen/lib.h>
>>> +#include <xen/errno.h>
>>> +#include <xen/mm.h>
>>> +#include <xen/vmap.h>
>>> +#include <xen/init.h>
>>> +#include <xen/sched.h>
>>> +#include <xen/stdbool.h>
>>> +#include <asm/system.h>
>>> +#include <asm/current.h>
>>> +#include <asm/io.h>
>>> +#include <asm/p2m.h>
>>> +
>>> +#include <xen/remoteproc_iommu.h>
>>> +
>>> +#include "io.h"
>>> +
>>> +static struct mmu_info *mmu_list[] = {
>>> +};
>>> +
>>> +#define mmu_for_each(pfunc, data) \
>>> +({ \
>>> + u32 __i; \
>>> + int __res = 0; \
>>> + \
>>> + for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
>>> + { \
>>> + __res |= pfunc(mmu_list[__i], data); \
>>> + } \
>>> + __res; \
>>> +})
>>> +
>>> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
>>> +{
>>> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start + mmu->mem_size)) )
>>> + return 1;
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static inline struct mmu_info *mmu_lookup(u32 addr)
>>> +{
>>> + u32 i;
>>> +
>>> + /* enumerate all registered MMU's and check is address in range */
>>> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
>>> + {
>>> + if ( mmu_check_mem_range(mmu_list[i], addr) )
>>> + return mmu_list[i];
>>> + }
>>> +
>>> + return NULL;
>>> +}
>>> +
>>> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
>>> +{
>>> + return mmu_for_each(mmu_check_mem_range, addr);
>>> +}
>>> +
>>> +static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
>>> +{
>>> + void __iomem *pagetable = NULL;
>>> + u32 maddr, i;
>>> +
>>> + ASSERT(mmu);
>>> + ASSERT(pgt);
>>> +
>>> + if ( !pgt->paddr )
>>> + return -EINVAL;
>>> +
>>> + /* pagetable size can be more than one page */
>>> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
>>> + {
>>> + /* lookup address where remoteproc pagetable is stored by kernel */
>>> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE, NULL);
>>> + if ( !maddr )
>>> + {
>>> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i * PAGE_SIZE);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
>>> + if ( !pagetable )
>>> + {
>>> + pr_mmu("failed to map pagetable");
>>> + return -EINVAL;
>>> + }
>>> +
>>> + /* copy pagetable to hypervisor memory */
>>> + clean_and_invalidate_xen_dcache_va_range(pagetable, PAGE_SIZE);
>>> + memcpy((u32*)((u32)pgt->kern_pagetable + i * PAGE_SIZE), pagetable, PAGE_SIZE);
>>> +
>>> + iounmap(pagetable);
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32 addr, bool is_maddr)
>>> +{
>>> + struct mmu_pagetable *pgt;
>>> + u32 pgt_addr;
>>> +
>>> + list_for_each_entry(pgt, &mmu->pagetables_list, link_node)
>>> + {
>>> + if ( is_maddr )
>>> + pgt_addr = pgt->maddr;
>>> + else
>>> + pgt_addr = pgt->paddr;
>>> +
>>> + if ( pgt_addr == addr )
>>> + return pgt;
>>> + }
>>> +
>>> + return NULL;
>>> +}
>>> +
>>> +static struct mmu_pagetable *mmu_alloc_pagetable(struct mmu_info *mmu, u32 paddr)
>>> +{
>>> + struct mmu_pagetable *pgt;
>>> + u32 pgt_size = MMU_PGD_TABLE_SIZE(mmu);
>>> +
>>> + pgt = xzalloc_bytes(sizeof(struct mmu_pagetable));
>>> + if ( !pgt )
>>> + {
>>> + pr_mmu("failed to alloc pagetable structure");
>>> + return NULL;
>>> + }
>>> +
>>> + /* allocate pagetable managed by hypervisor */
>>> + pgt->hyp_pagetable = xzalloc_bytes(pgt_size);
>>> + if ( !pgt->hyp_pagetable )
>>> + {
>>> + pr_mmu("failed to alloc private hyp_pagetable");
>>> + return NULL;
>>> + }
>>> +
>>> + /* alocate pagetable for ipa storing */
>>> + pgt->kern_pagetable = xzalloc_bytes(pgt_size);
>>> + if ( !pgt->kern_pagetable )
>>> + {
>>> + pr_mmu("failed to alloc private kern_pagetable");
>>> + return NULL;
>>> + }
>>> +
>>> + pr_mmu("private pagetables for 0x%08x paddr %u bytes (main 0x%08x, temp 0x%08x)",
>>> + paddr, pgt_size, (u32)__pa(pgt->hyp_pagetable), (u32)__pa(pgt->kern_pagetable));
>>> +
>>> + pgt->paddr = paddr;
>>> +
>>> + list_add(&pgt->link_node, &mmu->pagetables_list);
>>> +
>>> + return pgt;
>>> +}
>>> +
>>> +static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
>>> +{
>>> + struct mmu_pagetable *pgt;
>>> + int res;
>>> +
>>> + /* lookup using machine address first */
>>> + pgt = mmu_pagetable_lookup(mmu, paddr, true);
>>> + if ( !pgt )
>>> + {
>>> + /* lookup using kernel physical address */
>>> + pgt = mmu_pagetable_lookup(mmu, paddr, false);
>>> + if ( !pgt )
>>> + {
>>> + /* if pagetable doesn't exists in lookup list - allocate it */
>>> + pgt = mmu_alloc_pagetable(mmu, paddr);
>>> + }
>>> + }
>>> +
>>> + pgt->maddr = MMU_INVALID_ADDRESS;
>>> +
>>> + /* copy pagetable from domain to hypervisor */
>>> + res = mmu_copy_pagetable(mmu, pgt);
>>> + if ( res )
>>> + return res;
>>> +
>>> + /* translate pagetable */
>>> + pgt->maddr = mmu->translate_pfunc(mmu, pgt);
>>> + return pgt->maddr;
>>> +}
>>> +
>>> +static u32 mmu_trap_translate_pagetable(struct mmu_info *mmu, mmio_info_t *info)
>>> +{
>>> + register_t *reg;
>>> + bool valid_trap = false;
>>> + u32 i, paddr;
>>> +
>>> + reg = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>>> +
>>> + ASSERT(reg);
>>> +
>>> + paddr = *reg;
>>> + if ( !paddr )
>>> + return MMU_INVALID_ADDRESS;
>>> +
>>> + /* check is the register is a valid TTB register */
>>> + for ( i = 0; i < mmu->num_traps; i++ )
>>> + {
>>> + if ( mmu->trap_offsets[i] == (info->gpa - mmu->mem_start) )
>>> + {
>>> + valid_trap = true;
>>> + break;
>>> + }
>>> + }
>>> +
>>> + if ( !valid_trap )
>>> + return MMU_INVALID_ADDRESS;
>>> +
>>> + return mmu_translate_pagetable(mmu, paddr);
>>> +}
>>> +
>>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
>>> + u32 maddr, u32 hyp_addr)
>>
>> Is this actually used anywhere?
>>
>>
>>> +{
>>> + u32 *pte_table = NULL, *hyp_pte_table = NULL, pte_table_size = PAGE_SIZE;
>>> + u32 i;
>>> +
>>> + /* remap second level translation table */
>>> + pte_table = ioremap_nocache(maddr, MMU_PTE_TABLE_SIZE(mmu));
>>> + if ( !pte_table )
>>> + {
>>> + pr_mmu("failed to map pte_table");
>>> + return MMU_INVALID_ADDRESS;
>>> + }
>>> +
>>> + /* allocate new second level pagetable once */
>>> + if ( 0 == hyp_addr )
>>> + {
>>> + if ( MMU_PTE_TABLE_SIZE(mmu) > PAGE_SIZE )
>>> + pte_table_size = MMU_PTE_TABLE_SIZE(mmu);
>>> +
>>> + hyp_pte_table = xzalloc_bytes(pte_table_size);
>>> + if ( !hyp_pte_table )
>>> + {
>>> + pr_mmu("failed to alloc new iopt");
>>> + return MMU_INVALID_ADDRESS;
>>> + }
>>> + }
>>> + else
>>> + {
>>> + hyp_pte_table = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
>>> + }
>>> +
>>> + /* 2-nd level translation */
>>> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
>>> + {
>>> + paddr_t pt_maddr, pt_paddr, pt_flags;
>>> + u32 pt_mask = MMU_SECTION_MASK(mmu->pg_data->pte_shift);
>>> +
>>> + if ( !pte_table[i] )
>>> + {
>>> + /* handle the case when page was removed */
>>> + if ( unlikely(hyp_pte_table[i]) )
>>> + {
>>> + hyp_pte_table[i] = 0;
>>> + }
>>> +
>>> + continue;
>>> + }
>>> +
>>> + pt_paddr = pte_table[i] & pt_mask;
>>> + pt_flags = pte_table[i] & ~pt_mask;
>>> + pt_maddr = p2m_lookup(current->domain, pt_paddr, NULL);
>>> + ASSERT(pt_maddr != INVALID_PADDR);
>>> +
>>> + hyp_pte_table[i] = pt_maddr | pt_flags;
>>> + pgt->page_counter++;
>>> + }
>>> +
>>> + iounmap(pte_table);
>>> +
>>> + clean_and_invalidate_xen_dcache_va_range(hyp_pte_table, MMU_PTE_TABLE_SIZE(mmu));
>>> + return __pa(hyp_pte_table);
>>> +}
>>> +
>>> +static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
>>> +{
>>> + struct mmu_info *mmu = NULL;
>>> + unsigned long flags;
>>> + register_t *r;
>>> +
>>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>>> +
>>> + ASSERT(r);
>>
>> If you check on the domid in mmu_mmio_write, I would check here too
>>
>>
>>> + mmu = mmu_lookup(info->gpa);
>>> + if ( !mmu )
>>> + {
>>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + spin_lock_irqsave(&mmu->lock, flags);
>>> + *r = readl(mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
>>> + spin_unlock_irqrestore(&mmu->lock, flags);
>>> +
>>> + return 1;
>>
>> It looks like you are returning the mfns here, is that correct?
>>
>>
>>
>>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
>>> +{
>>> + struct mmu_info *mmu = NULL;
>>> + unsigned long flags;
>>> + register_t *r;
>>> + u32 new_addr, val;
>>> +
>>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
>>> +
>>> + ASSERT(r);
>>> +
>>> + /* dom0 should not access remoteproc MMU */
>>> + if ( 0 == current->domain->domain_id )
>>> + return 1;
>>
>> This is too specific to one particular configuration.
>> Would it be possible to generalize this somehow? At the very least you
>> could introduce an XSM label to access the pagetables, so that you can
>> dynamically configure the domains the can write to them.
>>
>
> I succeeded to enable XSM on our platform and enforce FLASK on boot,
> but I still have some questions:
> - What is this new XSM label ? Do you mean
> *seclabel='system_u:system_r:domU_t'* and change value of seclabel to
> something specific?
> As described in
> http://wiki.xen.org/wiki/Xen_Security_Modules_:_XSM-FLASK#Creating_domains_with_XSM_security_labels
>
> - Do I need to introduce a new policy for this? As far as I understand
> from previous link - it is obligatory for FLASK, therefore it is
> obligatory for XSM. In other words - all XSM related checks, which I
> may introduce won't work correctly without installing a policy, am I
> right ?
>
Well. Looks like answer is 'yes'. I tried to play with XSM policy and
see how it works. Will try to introduce a new label for domain which
can access remoteprocessor MMU
> Regards,
> Andrii
>
>>
>>> + /* find corresponding MMU */
>>> + mmu = mmu_lookup(info->gpa);
>>> + if ( !mmu )
>>> + {
>>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + ASSERT(v->domain == current->domain);
>>
>> You can remove this assert.
>>
>>
>>> + spin_lock_irqsave(&mmu->lock, flags);
>>> +
>>> + /* get new address of translated pagetable */
>>> + new_addr = mmu_trap_translate_pagetable(mmu, info);
>>> + if ( MMU_INVALID_ADDRESS != new_addr )
>>> + val = new_addr;
>>> + else
>>> + val = *r;
>>> +
>>> + writel(val, mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
>>> + spin_unlock_irqrestore(&mmu->lock, flags);
>>> +
>>> + return 1;
>>> +}
>>> +
>>> +static int mmu_init(struct mmu_info *mmu, u32 data)
>>> +{
>>> + ASSERT(mmu);
>>> + ASSERT(!mmu->mem_map);
>>> +
>>> + INIT_LIST_HEAD(&mmu->pagetables_list);
>>> +
>>> + /* map MMU memory */
>>> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
>>> + if ( !mmu->mem_map )
>>> + {
>>> + pr_mmu("failed to map memory");
>>> + return -EINVAL;
>>> + }
>>> +
>>> + pr_mmu("memory map = 0x%pS", _p(mmu->mem_map));
>>> +
>>> + spin_lock_init(&mmu->lock);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int mmu_init_all(void)
>>> +{
>>> + int res;
>>> +
>>> + res = mmu_for_each(mmu_init, 0);
>>> + if ( res )
>>> + {
>>> + printk("%s error during init %d\n", __func__, res);
>>> + return res;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +const struct mmio_handler remoteproc_mmio_handler = {
>>> + .check_handler = mmu_mmio_check,
>>> + .read_handler = mmu_mmio_read,
>>> + .write_handler = mmu_mmio_write,
>>> +};
>>> +
>>> +__initcall(mmu_init_all);
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
>>> new file mode 100644
>>> index 0000000..22e2951
>>> --- /dev/null
>>> +++ b/xen/include/xen/remoteproc_iommu.h
>>> @@ -0,0 +1,79 @@
>>> +/*
>>> + * xen/include/xen/remoteproc_iommu.h
>>> + *
>>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
>>> + * Copyright (c) 2014 GlobalLogic
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; either version 2 of the License, or
>>> + * (at your option) any later version.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + */
>>> +
>>> +#ifndef _REMOTEPROC_IOMMU_H_
>>> +#define _REMOTEPROC_IOMMU_H_
>>> +
>>> +#include <asm/types.h>
>>> +
>>> +#define MMU_SECTION_SIZE(shift) (1UL << (shift))
>>> +#define MMU_SECTION_MASK(shift) (~(MMU_SECTION_SIZE(shift) - 1))
>>> +
>>> +/* 4096 first level descriptors for "supersection" and "section" */
>>> +#define MMU_PTRS_PER_PGD(mmu) (1UL << (32 - (mmu->pg_data->pgd_shift)))
>>> +#define MMU_PGD_TABLE_SIZE(mmu) (MMU_PTRS_PER_PGD(mmu) * sizeof(u32))
>>> +
>>> +/* 256 second level descriptors for "small" and "large" pages */
>>> +#define MMU_PTRS_PER_PTE(mmu) (1UL << ((mmu->pg_data->pgd_shift) - (mmu->pg_data->pte_shift)))
>>> +#define MMU_PTE_TABLE_SIZE(mmu) (MMU_PTRS_PER_PTE(mmu) * sizeof(u32))
>>> +
>>> +/* 16 sections in supersection */
>>> +#define MMU_SECTION_PER_SUPER(mmu) (1UL << ((mmu->pg_data->super_shift) - (mmu->pg_data->section_shift)))
>>> +
>>> +#define MMU_INVALID_ADDRESS ((u32)(-1))
>>> +
>>> +#define pr_mmu(fmt, ...) \
>>> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
>>> +
>>> +struct pagetable_data {
>>> + /* 1st level translation */
>>> + u32 pgd_shift;
>>> + u32 pte_shift;
>>> + u32 super_shift;
>>> + u32 section_shift;
>>> + /* 2nd level translation */
>>> + u32 pte_large_shift;
>>> +};
>>> +
>>> +struct mmu_pagetable {
>>> + u32 *hyp_pagetable;
>>> + u32 *kern_pagetable;
>>> + u32 paddr;
>>> + u32 maddr;
>>> + struct list_head link_node;
>>> + u32 page_counter;
>>> +};
>>> +
>>> +struct mmu_info {
>>> + const char *name;
>>> + const struct pagetable_data *pg_data;
>>> + /* register where phys pointer to pagetable is stored */
>>> + u32 *trap_offsets;
>>> + paddr_t mem_start;
>>> + u32 mem_size;
>>> + spinlock_t lock;
>>> + struct list_head pagetables_list;
>>> + u32 num_traps;
>>> + void __iomem *mem_map;
>>> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
>>> + void (*print_pagetable_pfunc)(struct mmu_info *);
>>> +};
>>> +
>>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
>>> + u32 maddr, u32 hyp_addr);
>>> +
>>> +#endif /* _REMOTEPROC_IOMMU_H_ */
>>> --
>>> 1.7.9.5
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>>>
>
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
--
Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-08-21 15:30 ` Andrii Tseglytskyi
@ 2014-08-21 23:41 ` Stefano Stabellini
0 siblings, 0 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-08-21 23:41 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Julien Grall, xen-devel, Ian Campbell, Stefano Stabellini
On Thu, 21 Aug 2014, Andrii Tseglytskyi wrote:
> On Wed, Aug 20, 2014 at 10:40 PM, Andrii Tseglytskyi
> <andrii.tseglytskyi@globallogic.com> wrote:
> > Hi Stefano,
> >
> >
> > On Fri, Jul 4, 2014 at 4:59 PM, Stefano Stabellini
> > <stefano.stabellini@eu.citrix.com> wrote:
> >> On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> >>> This is a fisrst patch from patch series which was
> >>> developed to handle remote (external) processors
> >>> memory management units. Remote processors are
> >>> typically used for graphic rendering (GPUs) and
> >>> high quality video decoding (IPUs). They are typically
> >>> installed on such multimedia SoCs as OMAP4 / OMAP5.
> >>>
> >>> As soon as remoteprocessor MMU typically works with
> >>> pagetables filled by physical addresses, which are
> >>> allocated by domU kernel, it is almost impossible to
> >>> use them under Xen, intermediate physical addresses
> >>> allocated by kernel, need to be translated to machine
> >>> addresses.
> >>>
> >>> This patch introduces a simple framework to perform
> >>> pfn -> mfn translation for external MMUs.
> >>> It introduces basic data structures and algorithms
> >>> needed for translation.
> >>>
> >>> Typically, when MMU is configured, some it registers
> >>> are updated by new values. Introduced frameworks
> >>> uses traps as starting point of remoteproc MMUs
> >>> pagetables translation.
> >>>
> >>> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
> >>> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> >>
> >> There is one problem with this patch: you need to find a way to "pin"
> >> the p2m entries for the pfns and mfns found in the pagetables translated
> >> by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
> >> mfn mappings for the domain, breaking the pagetables already translated
> >> by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
> >> stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
> >> under the guest feet, but features like memory sharing or swapping,
> >> supported on x86, could cause it to happen.
> >>
> >> In the past I tried to introduce a way to "pin the mappings, but never
> >> finished to upstream the patches, because it didn't need it anymore.
> >>
> >>
> >> Give a look at:
> >>
> >> http://marc.info/?l=xen-devel&m=138029864707973
> >>
> >> In particular you would need to introduce something like
> >> the pin function and call it from mmu_translate_pagetable.
> >>
> >
> > What if I try your series and re-send it if helpful for me? Can you
> > link me to its latest code?
> >
> >>
> >>
> >>> xen/arch/arm/Makefile | 1 +
> >>> xen/arch/arm/remoteproc_iommu.c | 412 ++++++++++++++++++++++++++++++++++++
> >>> xen/include/xen/remoteproc_iommu.h | 79 +++++++
> >>> 3 files changed, 492 insertions(+)
> >>> create mode 100644 xen/arch/arm/remoteproc_iommu.c
> >>> create mode 100644 xen/include/xen/remoteproc_iommu.h
> >>>
> >>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> >>> index d70f6d5..0204d1c 100644
> >>> --- a/xen/arch/arm/Makefile
> >>> +++ b/xen/arch/arm/Makefile
> >>> @@ -15,6 +15,7 @@ obj-y += io.o
> >>> obj-y += irq.o
> >>> obj-y += kernel.o
> >>> obj-y += mm.o
> >>> +obj-y += remoteproc_iommu.o
> >>> obj-y += p2m.o
> >>> obj-y += percpu.o
> >>> obj-y += guestcopy.o
> >>> diff --git a/xen/arch/arm/remoteproc_iommu.c b/xen/arch/arm/remoteproc_iommu.c
> >>> new file mode 100644
> >>> index 0000000..b4d22d9
> >>> --- /dev/null
> >>> +++ b/xen/arch/arm/remoteproc_iommu.c
> >>> @@ -0,0 +1,412 @@
> >>> +/*
> >>> + * xen/arch/arm/remoteproc_iommu.c
> >>> + *
> >>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> >>> + * Copyright (c) 2014 GlobalLogic
> >>> + *
> >>> + * This program is free software; you can redistribute it and/or modify
> >>> + * it under the terms of the GNU General Public License as published by
> >>> + * the Free Software Foundation; either version 2 of the License, or
> >>> + * (at your option) any later version.
> >>> + *
> >>> + * This program is distributed in the hope that it will be useful,
> >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >>> + * GNU General Public License for more details.
> >>> + */
> >>> +
> >>> +#include <xen/config.h>
> >>> +#include <xen/lib.h>
> >>> +#include <xen/errno.h>
> >>> +#include <xen/mm.h>
> >>> +#include <xen/vmap.h>
> >>> +#include <xen/init.h>
> >>> +#include <xen/sched.h>
> >>> +#include <xen/stdbool.h>
> >>> +#include <asm/system.h>
> >>> +#include <asm/current.h>
> >>> +#include <asm/io.h>
> >>> +#include <asm/p2m.h>
> >>> +
> >>> +#include <xen/remoteproc_iommu.h>
> >>> +
> >>> +#include "io.h"
> >>> +
> >>> +static struct mmu_info *mmu_list[] = {
> >>> +};
> >>> +
> >>> +#define mmu_for_each(pfunc, data) \
> >>> +({ \
> >>> + u32 __i; \
> >>> + int __res = 0; \
> >>> + \
> >>> + for ( __i = 0; __i < ARRAY_SIZE(mmu_list); __i++ ) \
> >>> + { \
> >>> + __res |= pfunc(mmu_list[__i], data); \
> >>> + } \
> >>> + __res; \
> >>> +})
> >>> +
> >>> +static int mmu_check_mem_range(struct mmu_info *mmu, paddr_t addr)
> >>> +{
> >>> + if ( (addr >= mmu->mem_start) && (addr < (mmu->mem_start + mmu->mem_size)) )
> >>> + return 1;
> >>> +
> >>> + return 0;
> >>> +}
> >>> +
> >>> +static inline struct mmu_info *mmu_lookup(u32 addr)
> >>> +{
> >>> + u32 i;
> >>> +
> >>> + /* enumerate all registered MMU's and check is address in range */
> >>> + for ( i = 0; i < ARRAY_SIZE(mmu_list); i++ )
> >>> + {
> >>> + if ( mmu_check_mem_range(mmu_list[i], addr) )
> >>> + return mmu_list[i];
> >>> + }
> >>> +
> >>> + return NULL;
> >>> +}
> >>> +
> >>> +static int mmu_mmio_check(struct vcpu *v, paddr_t addr)
> >>> +{
> >>> + return mmu_for_each(mmu_check_mem_range, addr);
> >>> +}
> >>> +
> >>> +static int mmu_copy_pagetable(struct mmu_info *mmu, struct mmu_pagetable *pgt)
> >>> +{
> >>> + void __iomem *pagetable = NULL;
> >>> + u32 maddr, i;
> >>> +
> >>> + ASSERT(mmu);
> >>> + ASSERT(pgt);
> >>> +
> >>> + if ( !pgt->paddr )
> >>> + return -EINVAL;
> >>> +
> >>> + /* pagetable size can be more than one page */
> >>> + for ( i = 0; i < MMU_PGD_TABLE_SIZE(mmu) / PAGE_SIZE; i++ )
> >>> + {
> >>> + /* lookup address where remoteproc pagetable is stored by kernel */
> >>> + maddr = p2m_lookup(current->domain, pgt->paddr + i * PAGE_SIZE, NULL);
> >>> + if ( !maddr )
> >>> + {
> >>> + pr_mmu("failed to translate 0x%08lx to maddr", pgt->paddr + i * PAGE_SIZE);
> >>> + return -EINVAL;
> >>> + }
> >>> +
> >>> + pagetable = ioremap_nocache(maddr, MMU_PGD_TABLE_SIZE(mmu));
> >>> + if ( !pagetable )
> >>> + {
> >>> + pr_mmu("failed to map pagetable");
> >>> + return -EINVAL;
> >>> + }
> >>> +
> >>> + /* copy pagetable to hypervisor memory */
> >>> + clean_and_invalidate_xen_dcache_va_range(pagetable, PAGE_SIZE);
> >>> + memcpy((u32*)((u32)pgt->kern_pagetable + i * PAGE_SIZE), pagetable, PAGE_SIZE);
> >>> +
> >>> + iounmap(pagetable);
> >>> + }
> >>> +
> >>> + return 0;
> >>> +}
> >>> +
> >>> +struct mmu_pagetable *mmu_pagetable_lookup(struct mmu_info *mmu, u32 addr, bool is_maddr)
> >>> +{
> >>> + struct mmu_pagetable *pgt;
> >>> + u32 pgt_addr;
> >>> +
> >>> + list_for_each_entry(pgt, &mmu->pagetables_list, link_node)
> >>> + {
> >>> + if ( is_maddr )
> >>> + pgt_addr = pgt->maddr;
> >>> + else
> >>> + pgt_addr = pgt->paddr;
> >>> +
> >>> + if ( pgt_addr == addr )
> >>> + return pgt;
> >>> + }
> >>> +
> >>> + return NULL;
> >>> +}
> >>> +
> >>> +static struct mmu_pagetable *mmu_alloc_pagetable(struct mmu_info *mmu, u32 paddr)
> >>> +{
> >>> + struct mmu_pagetable *pgt;
> >>> + u32 pgt_size = MMU_PGD_TABLE_SIZE(mmu);
> >>> +
> >>> + pgt = xzalloc_bytes(sizeof(struct mmu_pagetable));
> >>> + if ( !pgt )
> >>> + {
> >>> + pr_mmu("failed to alloc pagetable structure");
> >>> + return NULL;
> >>> + }
> >>> +
> >>> + /* allocate pagetable managed by hypervisor */
> >>> + pgt->hyp_pagetable = xzalloc_bytes(pgt_size);
> >>> + if ( !pgt->hyp_pagetable )
> >>> + {
> >>> + pr_mmu("failed to alloc private hyp_pagetable");
> >>> + return NULL;
> >>> + }
> >>> +
> >>> + /* alocate pagetable for ipa storing */
> >>> + pgt->kern_pagetable = xzalloc_bytes(pgt_size);
> >>> + if ( !pgt->kern_pagetable )
> >>> + {
> >>> + pr_mmu("failed to alloc private kern_pagetable");
> >>> + return NULL;
> >>> + }
> >>> +
> >>> + pr_mmu("private pagetables for 0x%08x paddr %u bytes (main 0x%08x, temp 0x%08x)",
> >>> + paddr, pgt_size, (u32)__pa(pgt->hyp_pagetable), (u32)__pa(pgt->kern_pagetable));
> >>> +
> >>> + pgt->paddr = paddr;
> >>> +
> >>> + list_add(&pgt->link_node, &mmu->pagetables_list);
> >>> +
> >>> + return pgt;
> >>> +}
> >>> +
> >>> +static u32 mmu_translate_pagetable(struct mmu_info *mmu, u32 paddr)
> >>> +{
> >>> + struct mmu_pagetable *pgt;
> >>> + int res;
> >>> +
> >>> + /* lookup using machine address first */
> >>> + pgt = mmu_pagetable_lookup(mmu, paddr, true);
> >>> + if ( !pgt )
> >>> + {
> >>> + /* lookup using kernel physical address */
> >>> + pgt = mmu_pagetable_lookup(mmu, paddr, false);
> >>> + if ( !pgt )
> >>> + {
> >>> + /* if pagetable doesn't exists in lookup list - allocate it */
> >>> + pgt = mmu_alloc_pagetable(mmu, paddr);
> >>> + }
> >>> + }
> >>> +
> >>> + pgt->maddr = MMU_INVALID_ADDRESS;
> >>> +
> >>> + /* copy pagetable from domain to hypervisor */
> >>> + res = mmu_copy_pagetable(mmu, pgt);
> >>> + if ( res )
> >>> + return res;
> >>> +
> >>> + /* translate pagetable */
> >>> + pgt->maddr = mmu->translate_pfunc(mmu, pgt);
> >>> + return pgt->maddr;
> >>> +}
> >>> +
> >>> +static u32 mmu_trap_translate_pagetable(struct mmu_info *mmu, mmio_info_t *info)
> >>> +{
> >>> + register_t *reg;
> >>> + bool valid_trap = false;
> >>> + u32 i, paddr;
> >>> +
> >>> + reg = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> >>> +
> >>> + ASSERT(reg);
> >>> +
> >>> + paddr = *reg;
> >>> + if ( !paddr )
> >>> + return MMU_INVALID_ADDRESS;
> >>> +
> >>> + /* check is the register is a valid TTB register */
> >>> + for ( i = 0; i < mmu->num_traps; i++ )
> >>> + {
> >>> + if ( mmu->trap_offsets[i] == (info->gpa - mmu->mem_start) )
> >>> + {
> >>> + valid_trap = true;
> >>> + break;
> >>> + }
> >>> + }
> >>> +
> >>> + if ( !valid_trap )
> >>> + return MMU_INVALID_ADDRESS;
> >>> +
> >>> + return mmu_translate_pagetable(mmu, paddr);
> >>> +}
> >>> +
> >>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
> >>> + u32 maddr, u32 hyp_addr)
> >>
> >> Is this actually used anywhere?
> >>
> >>
> >>> +{
> >>> + u32 *pte_table = NULL, *hyp_pte_table = NULL, pte_table_size = PAGE_SIZE;
> >>> + u32 i;
> >>> +
> >>> + /* remap second level translation table */
> >>> + pte_table = ioremap_nocache(maddr, MMU_PTE_TABLE_SIZE(mmu));
> >>> + if ( !pte_table )
> >>> + {
> >>> + pr_mmu("failed to map pte_table");
> >>> + return MMU_INVALID_ADDRESS;
> >>> + }
> >>> +
> >>> + /* allocate new second level pagetable once */
> >>> + if ( 0 == hyp_addr )
> >>> + {
> >>> + if ( MMU_PTE_TABLE_SIZE(mmu) > PAGE_SIZE )
> >>> + pte_table_size = MMU_PTE_TABLE_SIZE(mmu);
> >>> +
> >>> + hyp_pte_table = xzalloc_bytes(pte_table_size);
> >>> + if ( !hyp_pte_table )
> >>> + {
> >>> + pr_mmu("failed to alloc new iopt");
> >>> + return MMU_INVALID_ADDRESS;
> >>> + }
> >>> + }
> >>> + else
> >>> + {
> >>> + hyp_pte_table = __va(hyp_addr & MMU_SECTION_MASK(mmu->pg_data->pte_shift));
> >>> + }
> >>> +
> >>> + /* 2-nd level translation */
> >>> + for ( i = 0; i < MMU_PTRS_PER_PTE(mmu); i++ )
> >>> + {
> >>> + paddr_t pt_maddr, pt_paddr, pt_flags;
> >>> + u32 pt_mask = MMU_SECTION_MASK(mmu->pg_data->pte_shift);
> >>> +
> >>> + if ( !pte_table[i] )
> >>> + {
> >>> + /* handle the case when page was removed */
> >>> + if ( unlikely(hyp_pte_table[i]) )
> >>> + {
> >>> + hyp_pte_table[i] = 0;
> >>> + }
> >>> +
> >>> + continue;
> >>> + }
> >>> +
> >>> + pt_paddr = pte_table[i] & pt_mask;
> >>> + pt_flags = pte_table[i] & ~pt_mask;
> >>> + pt_maddr = p2m_lookup(current->domain, pt_paddr, NULL);
> >>> + ASSERT(pt_maddr != INVALID_PADDR);
> >>> +
> >>> + hyp_pte_table[i] = pt_maddr | pt_flags;
> >>> + pgt->page_counter++;
> >>> + }
> >>> +
> >>> + iounmap(pte_table);
> >>> +
> >>> + clean_and_invalidate_xen_dcache_va_range(hyp_pte_table, MMU_PTE_TABLE_SIZE(mmu));
> >>> + return __pa(hyp_pte_table);
> >>> +}
> >>> +
> >>> +static int mmu_mmio_read(struct vcpu *v, mmio_info_t *info)
> >>> +{
> >>> + struct mmu_info *mmu = NULL;
> >>> + unsigned long flags;
> >>> + register_t *r;
> >>> +
> >>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> >>> +
> >>> + ASSERT(r);
> >>
> >> If you check on the domid in mmu_mmio_write, I would check here too
> >>
> >>
> >>> + mmu = mmu_lookup(info->gpa);
> >>> + if ( !mmu )
> >>> + {
> >>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
> >>> + return -EINVAL;
> >>> + }
> >>> +
> >>> + spin_lock_irqsave(&mmu->lock, flags);
> >>> + *r = readl(mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
> >>> + spin_unlock_irqrestore(&mmu->lock, flags);
> >>> +
> >>> + return 1;
> >>
> >> It looks like you are returning the mfns here, is that correct?
> >>
> >>
> >>
> >>> +static int mmu_mmio_write(struct vcpu *v, mmio_info_t *info)
> >>> +{
> >>> + struct mmu_info *mmu = NULL;
> >>> + unsigned long flags;
> >>> + register_t *r;
> >>> + u32 new_addr, val;
> >>> +
> >>> + r = select_user_reg(guest_cpu_user_regs(), info->dabt.reg);
> >>> +
> >>> + ASSERT(r);
> >>> +
> >>> + /* dom0 should not access remoteproc MMU */
> >>> + if ( 0 == current->domain->domain_id )
> >>> + return 1;
> >>
> >> This is too specific to one particular configuration.
> >> Would it be possible to generalize this somehow? At the very least you
> >> could introduce an XSM label to access the pagetables, so that you can
> >> dynamically configure the domains the can write to them.
> >>
> >
> > I succeeded to enable XSM on our platform and enforce FLASK on boot,
> > but I still have some questions:
> > - What is this new XSM label ? Do you mean
> > *seclabel='system_u:system_r:domU_t'* and change value of seclabel to
> > something specific?
> > As described in
> > http://wiki.xen.org/wiki/Xen_Security_Modules_:_XSM-FLASK#Creating_domains_with_XSM_security_labels
> >
> > - Do I need to introduce a new policy for this? As far as I understand
> > from previous link - it is obligatory for FLASK, therefore it is
> > obligatory for XSM. In other words - all XSM related checks, which I
> > may introduce won't work correctly without installing a policy, am I
> > right ?
> >
>
> Well. Looks like answer is 'yes'. I tried to play with XSM policy and
> see how it works. Will try to introduce a new label for domain which
> can access remoteprocessor MMU
Yep, that's right. That is going to make your code much more flexible
too.
> > Regards,
> > Andrii
> >
> >>
> >>> + /* find corresponding MMU */
> >>> + mmu = mmu_lookup(info->gpa);
> >>> + if ( !mmu )
> >>> + {
> >>> + pr_mmu("can't get mmu for addr 0x%08x", (u32)info->gpa);
> >>> + return -EINVAL;
> >>> + }
> >>> +
> >>> + ASSERT(v->domain == current->domain);
> >>
> >> You can remove this assert.
> >>
> >>
> >>> + spin_lock_irqsave(&mmu->lock, flags);
> >>> +
> >>> + /* get new address of translated pagetable */
> >>> + new_addr = mmu_trap_translate_pagetable(mmu, info);
> >>> + if ( MMU_INVALID_ADDRESS != new_addr )
> >>> + val = new_addr;
> >>> + else
> >>> + val = *r;
> >>> +
> >>> + writel(val, mmu->mem_map + ((u32)(info->gpa) - mmu->mem_start));
> >>> + spin_unlock_irqrestore(&mmu->lock, flags);
> >>> +
> >>> + return 1;
> >>> +}
> >>> +
> >>> +static int mmu_init(struct mmu_info *mmu, u32 data)
> >>> +{
> >>> + ASSERT(mmu);
> >>> + ASSERT(!mmu->mem_map);
> >>> +
> >>> + INIT_LIST_HEAD(&mmu->pagetables_list);
> >>> +
> >>> + /* map MMU memory */
> >>> + mmu->mem_map = ioremap_nocache(mmu->mem_start, mmu->mem_size);
> >>> + if ( !mmu->mem_map )
> >>> + {
> >>> + pr_mmu("failed to map memory");
> >>> + return -EINVAL;
> >>> + }
> >>> +
> >>> + pr_mmu("memory map = 0x%pS", _p(mmu->mem_map));
> >>> +
> >>> + spin_lock_init(&mmu->lock);
> >>> +
> >>> + return 0;
> >>> +}
> >>> +
> >>> +static int mmu_init_all(void)
> >>> +{
> >>> + int res;
> >>> +
> >>> + res = mmu_for_each(mmu_init, 0);
> >>> + if ( res )
> >>> + {
> >>> + printk("%s error during init %d\n", __func__, res);
> >>> + return res;
> >>> + }
> >>> +
> >>> + return 0;
> >>> +}
> >>> +
> >>> +const struct mmio_handler remoteproc_mmio_handler = {
> >>> + .check_handler = mmu_mmio_check,
> >>> + .read_handler = mmu_mmio_read,
> >>> + .write_handler = mmu_mmio_write,
> >>> +};
> >>> +
> >>> +__initcall(mmu_init_all);
> >>> +
> >>> +/*
> >>> + * Local variables:
> >>> + * mode: C
> >>> + * c-file-style: "BSD"
> >>> + * c-basic-offset: 4
> >>> + * indent-tabs-mode: nil
> >>> + * End:
> >>> + */
> >>> diff --git a/xen/include/xen/remoteproc_iommu.h b/xen/include/xen/remoteproc_iommu.h
> >>> new file mode 100644
> >>> index 0000000..22e2951
> >>> --- /dev/null
> >>> +++ b/xen/include/xen/remoteproc_iommu.h
> >>> @@ -0,0 +1,79 @@
> >>> +/*
> >>> + * xen/include/xen/remoteproc_iommu.h
> >>> + *
> >>> + * Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> >>> + * Copyright (c) 2014 GlobalLogic
> >>> + *
> >>> + * This program is free software; you can redistribute it and/or modify
> >>> + * it under the terms of the GNU General Public License as published by
> >>> + * the Free Software Foundation; either version 2 of the License, or
> >>> + * (at your option) any later version.
> >>> + *
> >>> + * This program is distributed in the hope that it will be useful,
> >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >>> + * GNU General Public License for more details.
> >>> + */
> >>> +
> >>> +#ifndef _REMOTEPROC_IOMMU_H_
> >>> +#define _REMOTEPROC_IOMMU_H_
> >>> +
> >>> +#include <asm/types.h>
> >>> +
> >>> +#define MMU_SECTION_SIZE(shift) (1UL << (shift))
> >>> +#define MMU_SECTION_MASK(shift) (~(MMU_SECTION_SIZE(shift) - 1))
> >>> +
> >>> +/* 4096 first level descriptors for "supersection" and "section" */
> >>> +#define MMU_PTRS_PER_PGD(mmu) (1UL << (32 - (mmu->pg_data->pgd_shift)))
> >>> +#define MMU_PGD_TABLE_SIZE(mmu) (MMU_PTRS_PER_PGD(mmu) * sizeof(u32))
> >>> +
> >>> +/* 256 second level descriptors for "small" and "large" pages */
> >>> +#define MMU_PTRS_PER_PTE(mmu) (1UL << ((mmu->pg_data->pgd_shift) - (mmu->pg_data->pte_shift)))
> >>> +#define MMU_PTE_TABLE_SIZE(mmu) (MMU_PTRS_PER_PTE(mmu) * sizeof(u32))
> >>> +
> >>> +/* 16 sections in supersection */
> >>> +#define MMU_SECTION_PER_SUPER(mmu) (1UL << ((mmu->pg_data->super_shift) - (mmu->pg_data->section_shift)))
> >>> +
> >>> +#define MMU_INVALID_ADDRESS ((u32)(-1))
> >>> +
> >>> +#define pr_mmu(fmt, ...) \
> >>> + printk("%s: %s: "fmt"\n", __func__, ((mmu) ? (mmu)->name : ""), ##__VA_ARGS__)
> >>> +
> >>> +struct pagetable_data {
> >>> + /* 1st level translation */
> >>> + u32 pgd_shift;
> >>> + u32 pte_shift;
> >>> + u32 super_shift;
> >>> + u32 section_shift;
> >>> + /* 2nd level translation */
> >>> + u32 pte_large_shift;
> >>> +};
> >>> +
> >>> +struct mmu_pagetable {
> >>> + u32 *hyp_pagetable;
> >>> + u32 *kern_pagetable;
> >>> + u32 paddr;
> >>> + u32 maddr;
> >>> + struct list_head link_node;
> >>> + u32 page_counter;
> >>> +};
> >>> +
> >>> +struct mmu_info {
> >>> + const char *name;
> >>> + const struct pagetable_data *pg_data;
> >>> + /* register where phys pointer to pagetable is stored */
> >>> + u32 *trap_offsets;
> >>> + paddr_t mem_start;
> >>> + u32 mem_size;
> >>> + spinlock_t lock;
> >>> + struct list_head pagetables_list;
> >>> + u32 num_traps;
> >>> + void __iomem *mem_map;
> >>> + u32 (*translate_pfunc)(struct mmu_info *, struct mmu_pagetable *);
> >>> + void (*print_pagetable_pfunc)(struct mmu_info *);
> >>> +};
> >>> +
> >>> +u32 mmu_translate_second_level(struct mmu_info *mmu, struct mmu_pagetable *pgt,
> >>> + u32 maddr, u32 hyp_addr);
> >>> +
> >>> +#endif /* _REMOTEPROC_IOMMU_H_ */
> >>> --
> >>> 1.7.9.5
> >>>
> >>>
> >>> _______________________________________________
> >>> Xen-devel mailing list
> >>> Xen-devel@lists.xen.org
> >>> http://lists.xen.org/xen-devel
> >>>
> >
> >
> >
> > --
> >
> > Andrii Tseglytskyi | Embedded Dev
> > GlobalLogic
> > www.globallogic.com
>
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v02 1/7] arm: introduce remoteprocessor iommu module
2014-08-20 19:40 ` Andrii Tseglytskyi
2014-08-21 15:30 ` Andrii Tseglytskyi
@ 2014-08-21 23:43 ` Stefano Stabellini
1 sibling, 0 replies; 57+ messages in thread
From: Stefano Stabellini @ 2014-08-21 23:43 UTC (permalink / raw)
To: Andrii Tseglytskyi
Cc: Julien Grall, xen-devel, Ian Campbell, Stefano Stabellini
On Wed, 20 Aug 2014, Andrii Tseglytskyi wrote:
> Hi Stefano,
>
>
> On Fri, Jul 4, 2014 at 4:59 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Thu, 26 Jun 2014, Andrii Tseglytskyi wrote:
> >> This is a fisrst patch from patch series which was
> >> developed to handle remote (external) processors
> >> memory management units. Remote processors are
> >> typically used for graphic rendering (GPUs) and
> >> high quality video decoding (IPUs). They are typically
> >> installed on such multimedia SoCs as OMAP4 / OMAP5.
> >>
> >> As soon as remoteprocessor MMU typically works with
> >> pagetables filled by physical addresses, which are
> >> allocated by domU kernel, it is almost impossible to
> >> use them under Xen, intermediate physical addresses
> >> allocated by kernel, need to be translated to machine
> >> addresses.
> >>
> >> This patch introduces a simple framework to perform
> >> pfn -> mfn translation for external MMUs.
> >> It introduces basic data structures and algorithms
> >> needed for translation.
> >>
> >> Typically, when MMU is configured, some it registers
> >> are updated by new values. Introduced frameworks
> >> uses traps as starting point of remoteproc MMUs
> >> pagetables translation.
> >>
> >> Change-Id: Ia4d311a40289df46a003f5ae8706c150bee1885d
> >> Signed-off-by: Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com>
> >
> > There is one problem with this patch: you need to find a way to "pin"
> > the p2m entries for the pfns and mfns found in the pagetables translated
> > by remoteproc_iommu.c. Otherwise Xen might decide to change the pfn to
> > mfn mappings for the domain, breaking the pagetables already translated
> > by remoteproc_iommu.c. pfn to mfn mappings are not guaranteed to be
> > stable. At the moment Xen on ARM wouldn't change pfn to mfn mappings
> > under the guest feet, but features like memory sharing or swapping,
> > supported on x86, could cause it to happen.
> >
> > In the past I tried to introduce a way to "pin the mappings, but never
> > finished to upstream the patches, because it didn't need it anymore.
> >
> >
> > Give a look at:
> >
> > http://marc.info/?l=xen-devel&m=138029864707973
> >
> > In particular you would need to introduce something like
> > the pin function and call it from mmu_translate_pagetable.
> >
>
> What if I try your series and re-send it if helpful for me? Can you
> link me to its latest code?
Yeah, it might be useful to update my old series and integrate it into
yours. You probably don't need the two hypercalls, just the pin and
unpin functions. The link above points to the latest version of it.
^ permalink raw reply [flat|nested] 57+ messages in thread
end of thread, other threads:[~2014-08-21 23:43 UTC | newest]
Thread overview: 57+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-06-26 11:06 [PATCH v02 0/7] arm: introduce remoteprocessor iommu module Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 1/7] " Andrii Tseglytskyi
2014-06-29 18:00 ` Julien Grall
2014-07-22 15:20 ` Andrii Tseglytskyi
2014-07-22 16:29 ` Julien Grall
2014-07-31 11:59 ` Andrii Tseglytskyi
2014-07-31 12:11 ` Julien Grall
2014-07-31 12:49 ` Andrii Tseglytskyi
2014-07-04 13:59 ` Stefano Stabellini
2014-07-16 15:19 ` Ian Campbell
2014-07-22 12:42 ` Stefano Stabellini
2014-07-22 13:29 ` Julien Grall
2014-07-22 16:31 ` Andrii Tseglytskyi
2014-07-22 17:22 ` Andrii Tseglytskyi
2014-07-23 10:32 ` Stefano Stabellini
2014-07-23 10:54 ` Andrii Tseglytskyi
2014-07-22 15:40 ` Andrii Tseglytskyi
2014-07-22 15:32 ` Andrii Tseglytskyi
2014-08-01 10:06 ` Andrii Tseglytskyi
2014-08-01 10:32 ` Julien Grall
2014-08-01 10:34 ` Andrii Tseglytskyi
2014-08-01 10:37 ` Julien Grall
2014-08-01 10:43 ` Andrii Tseglytskyi
2014-08-20 19:40 ` Andrii Tseglytskyi
2014-08-21 15:30 ` Andrii Tseglytskyi
2014-08-21 23:41 ` Stefano Stabellini
2014-08-21 23:43 ` Stefano Stabellini
2014-07-16 15:29 ` Ian Campbell
2014-07-16 15:34 ` Ian Campbell
2014-07-22 16:24 ` Andrii Tseglytskyi
2014-07-22 16:14 ` Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 2/7] arm: omap: introduce iommu translation for IPU remoteproc Andrii Tseglytskyi
2014-07-04 14:01 ` Stefano Stabellini
2014-07-22 16:56 ` Andrii Tseglytskyi
2014-07-04 14:30 ` Julien Grall
2014-07-22 16:58 ` Andrii Tseglytskyi
2014-07-16 15:36 ` Ian Campbell
2014-07-22 17:16 ` Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 3/7] arm: omap: introduce iommu translation for GPU remoteproc Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 4/7] arm: omap: introduce print pagetable function for IPU remoteproc Andrii Tseglytskyi
2014-07-16 15:38 ` Ian Campbell
2014-07-22 16:55 ` Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 5/7] arm: omap: introduce print pagetable function for GPU remoteproc Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 6/7] arm: introduce do_translate_pagetable hypercall Andrii Tseglytskyi
2014-07-04 14:05 ` Stefano Stabellini
2014-07-16 15:42 ` Ian Campbell
2014-07-22 16:47 ` Andrii Tseglytskyi
2014-07-22 16:37 ` Andrii Tseglytskyi
2014-07-04 14:35 ` Julien Grall
2014-07-16 15:43 ` Ian Campbell
2014-07-22 16:50 ` Andrii Tseglytskyi
2014-07-22 16:39 ` Andrii Tseglytskyi
2014-07-22 16:44 ` Julien Grall
2014-07-22 16:48 ` Andrii Tseglytskyi
2014-06-26 11:07 ` [PATCH v02 7/7] arm: add trap for remoteproc mmio accesses Andrii Tseglytskyi
2014-06-26 16:52 ` Julien Grall
2014-06-27 8:36 ` Andrii Tseglytskyi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.