[Qemu-devel] [RFC PATCH v3 00/18] spapr: vfio: Enable Dynamic DMA windows (DDW)

[Qemu-devel] [RFC PATCH v3 00/18] spapr: vfio: Enable Dynamic DMA windows (DDW)

[Alexey Kardashevskiy]

At the moment sPAPR PHB supports only a single 32bit window
which is normally 1..2GB which is not enough for high performance devices.

PAPR spec enables creating an additional window(s) to support 64bit
DMA and bigger page sizes.

This patchset adds DDW support for pseries. The host kernel changes are
required, posted earlier today as:
[PATCH 00/13] powerpc/iommu/vfio: Enable Dynamic DMA windows

This was tested on POWER8 system which allows one additional DMA window
which is mapped at 0x800.0000.0000.0000 and supports 16MB pages.
Existing guests check for DDW capabilities in PHB's device tree and if it
is present, they request for an additional window and map entire guest RAM
using H_PUT_TCE/... hypercalls once at boot time and switch to direct DMA
operations.

TCE tables still may be big enough for guests backed with 64K pages but they
are reasonably small for guests backed by 16MB pages.

Please comment. Thanks!

Changes:
v3:
* removed "reset" from API now
* reworked machine versions
* applied multiple comments
* includes David's machine QOM rework as this patchset adds a new machine type

v2:
* tested on emulated PHB
* removed "ddw" machine property, now it is PHB property
* disabled by default
* defined "pseries-2.2" machine which enables DDW by default
* fixed reset() and reference counting




Alexey Kardashevskiy (17):
  qom: Make object_child_foreach safe for objects removal
  spapr_iommu: Disable in-kernel IOMMU tables for >4GB windows
  spapr_pci: Introduce a liobn number generating macros
  spapr_vio: Introduce a liobn number generating macros
  spapr_pci: Make find_phb()/find_dev() public
  spapr_iommu: Make spapr_tce_find_by_liobn() public
  spapr_iommu: Implement free_table() helper
  spapr_rtas: Reserve DDW RTAS token numbers
  spapr_pci: Define DDW callbacks
  spapr_pci: Add windows counter
  spapr_rtas: Add Dynamic DMA windows (DDW) RTAS calls support
  spapr_pci: Enable DDW
  spapr_pci_vfio: Call spapr_pci::reset on reset
  linux headers update for SPAPR (DDW + EEH)
  spapr_pci_vfio: Enable DDW
  vfio: Enable DDW ioctls to VFIO IOMMU driver
  spapr: Add pseries-2.2 machine

David Gibson (1):
  spapr: Make machine naming conventions closer to those for PC

 hw/misc/vfio.c              |   2 +
 hw/ppc/Makefile.objs        |   3 +
 hw/ppc/spapr.c              |  29 +++++-
 hw/ppc/spapr_iommu.c        |  18 +++-
 hw/ppc/spapr_pci.c          | 127 +++++++++++++++++++++---
 hw/ppc/spapr_pci_vfio.c     |  78 ++++++++++++++-
 hw/ppc/spapr_rtas_ddw.c     | 234 ++++++++++++++++++++++++++++++++++++++++++++
 hw/ppc/spapr_vio.c          |   2 +-
 include/hw/pci-host/spapr.h |  30 ++++++
 include/hw/ppc/spapr.h      |  13 ++-
 linux-headers/linux/vfio.h  |  58 ++++++++++-
 qom/object.c                |   4 +-
 trace-events                |   3 +
 13 files changed, 569 insertions(+), 32 deletions(-)
 create mode 100644 hw/ppc/spapr_rtas_ddw.c

-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 01/18] spapr: Make machine naming conventions closer to those for PC

[Alexey Kardashevskiy]

From: David Gibson <david@gibson.dropbear.id.au>

As of qemu-2.1, spapr/pseries, has a set of versioned machine classes to
represent the machine type as it appeared to the guest in different qemu
versions.  This allows for safe migration of guests between current and
future qemu versions.

In PC, however, the default plain "pc" machine type is just an alias for
the most recent versioned machine type.  In sPAPR at the moment, it names
the base machine class from which the versioned types are derived.

The PC approach is preferable; it makes it clearer which explicit version
is the current one.  Additionally updating the "current" machine as the
base class makes it even more likely than otherwise to incorrectly alter
the versioned machines' behaviour when updating the current machine.

Therefore this patch changes sPAPR to the PC approach - the base class
becomes abstract, and plain "pseries" becomes an alias for the most
recent versioned machine class.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
[aik: removed "pseries" name from the parent class]
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 647c383..d2d3c27 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1643,9 +1643,6 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
     MachineClass *mc = MACHINE_CLASS(oc);
     FWPathProviderClass *fwc = FW_PATH_PROVIDER_CLASS(oc);
 
-    mc->name = "pseries";
-    mc->desc = "pSeries Logical Partition (PAPR compliant)";
-    mc->is_default = 1;
     mc->init = ppc_spapr_init;
     mc->reset = ppc_spapr_reset;
     mc->block_default_type = IF_SCSI;
@@ -1660,6 +1657,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data)
 static const TypeInfo spapr_machine_info = {
     .name          = TYPE_SPAPR_MACHINE,
     .parent        = TYPE_MACHINE,
+    .abstract      = true,
     .instance_size = sizeof(sPAPRMachineState),
     .instance_init = spapr_machine_initfn,
     .class_init    = spapr_machine_class_init,
@@ -1675,7 +1673,8 @@ static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
 
     mc->name = "pseries-2.1";
     mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
-    mc->is_default = 0;
+    mc->alias = "pseries";
+    mc->is_default = 1;
 }
 
 static const TypeInfo spapr_machine_2_1_info = {
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 02/18] qom: Make object_child_foreach safe for objects removal

[Alexey Kardashevskiy]

Current object_child_foreach() uses QTAILQ_FOREACH() to walk
through children and that makes children removal from the callback
impossible.

This makes object_child_foreach() use QTAILQ_FOREACH_SAFE().

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---

This went to Andreas's qom-next tree, it is here for the reference only.
---
 qom/object.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/qom/object.c b/qom/object.c
index 1b00831..012548c 100644
--- a/qom/object.c
+++ b/qom/object.c
@@ -668,10 +668,10 @@ void object_class_foreach(void (*fn)(ObjectClass *klass, void *opaque),
 int object_child_foreach(Object *obj, int (*fn)(Object *child, void *opaque),
                          void *opaque)
 {
-    ObjectProperty *prop;
+    ObjectProperty *prop, *next;
     int ret = 0;
 
-    QTAILQ_FOREACH(prop, &obj->properties, node) {
+    QTAILQ_FOREACH_SAFE(prop, &obj->properties, node, next) {
         if (object_property_is_child(prop)) {
             ret = fn(prop->opaque, opaque);
             if (ret != 0) {
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 03/18] spapr_iommu: Disable in-kernel IOMMU tables for >4GB windows

[Alexey Kardashevskiy]

The existing KVM_CREATE_SPAPR_TCE ioctl only support 4G windows max as
the window size parameter to the kernel ioctl() is 32-bit so
there's no way of expressing a TCE window > 4GB.

We are going to add huge DMA windows support so this will create small
window and unexpectedly fail later.

This disables KVM_CREATE_SPAPR_TCE for windows bigger that 4GB. Since
those windows are normally mapped at the boot time, there will be no
performance impact.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v3:
* fixed commit log
* added cast to uint64_t
---
 hw/ppc/spapr_iommu.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/hw/ppc/spapr_iommu.c b/hw/ppc/spapr_iommu.c
index f6e32a4..7f9a862 100644
--- a/hw/ppc/spapr_iommu.c
+++ b/hw/ppc/spapr_iommu.c
@@ -113,11 +113,11 @@ static MemoryRegionIOMMUOps spapr_iommu_ops = {
 static int spapr_tce_table_realize(DeviceState *dev)
 {
     sPAPRTCETable *tcet = SPAPR_TCE_TABLE(dev);
+    uint64_t window_size = (uint64_t)tcet->nb_table << tcet->page_shift;
 
-    if (kvm_enabled()) {
+    if (kvm_enabled() && !(window_size >> 32)) {
         tcet->table = kvmppc_create_spapr_tce(tcet->liobn,
-                                              tcet->nb_table <<
-                                              tcet->page_shift,
+                                              window_size,
                                               &tcet->fd,
                                               tcet->vfio_accel);
     }
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 04/18] spapr_pci: Introduce a liobn number generating macros

[Alexey Kardashevskiy]

We are going to have multiple DMA windows per PHB and we want them to
migrate so we need a predictable way of assigning LIOBNs.

This introduces a macro which makes up a LIOBN from fixed prefix,
PHB index (unique PHB id) and window number.

This introduces a SPAPR_PCI_DMA_WINDOW_NUM() to know the window number
from LIOBN. It is used to distinguish the default 32bit windows from
dynamic windows and avoid picking default DMA window properties from
a wrong TCE table.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr_pci.c     | 4 ++--
 include/hw/ppc/spapr.h | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index ad0da7f..c6cf5c1 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -502,7 +502,7 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp)
         }
 
         sphb->buid = SPAPR_PCI_BASE_BUID + sphb->index;
-        sphb->dma_liobn = SPAPR_PCI_BASE_LIOBN + sphb->index;
+        sphb->dma_liobn = SPAPR_PCI_LIOBN(sphb->index, 0);
 
         windows_base = SPAPR_PCI_WINDOW_BASE
             + sphb->index * SPAPR_PCI_WINDOW_SPACING;
@@ -837,7 +837,7 @@ static int spapr_phb_children_dt(Object *child, void *opaque)
     sPAPRTCETable *tcet;
 
     tcet = (sPAPRTCETable *) object_dynamic_cast(child, TYPE_SPAPR_TCE_TABLE);
-    if (!tcet) {
+    if (!tcet || SPAPR_PCI_DMA_WINDOW_NUM(tcet->liobn)) {
         return 0;
     }
 
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 749daf4..c24fa96 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -441,7 +441,8 @@ int spapr_rtas_device_tree_setup(void *fdt, hwaddr rtas_addr,
 #define SPAPR_TCE_PAGE_MASK    (SPAPR_TCE_PAGE_SIZE - 1)
 
 #define SPAPR_VIO_BASE_LIOBN    0x00000000
-#define SPAPR_PCI_BASE_LIOBN    0x80000000
+#define SPAPR_PCI_LIOBN(i, n)   (0x80000000 | ((i) << 8) | (n))
+#define SPAPR_PCI_DMA_WINDOW_NUM(liobn) ((liobn) & 0xff)
 
 #define RTAS_ERROR_LOG_MAX      2048
 
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 05/18] spapr_vio: Introduce a liobn number generating macros

[Alexey Kardashevskiy]

This introduces a macro which makes up a LIOBN from fixed prefix and
VIO device address (@reg property).

This is to keep LIOBN macros rendering consistent - the same macro for
PCI has been added by the previous patch.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr_vio.c     | 2 +-
 include/hw/ppc/spapr.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/hw/ppc/spapr_vio.c b/hw/ppc/spapr_vio.c
index dc9e46a..4404334 100644
--- a/hw/ppc/spapr_vio.c
+++ b/hw/ppc/spapr_vio.c
@@ -455,7 +455,7 @@ static int spapr_vio_busdev_init(DeviceState *qdev)
     }
 
     if (pc->rtce_window_size) {
-        uint32_t liobn = SPAPR_VIO_BASE_LIOBN | dev->reg;
+        uint32_t liobn = SPAPR_VIO_LIOBN(dev->reg);
         dev->tcet = spapr_tce_new_table(qdev, liobn,
                                         0,
                                         SPAPR_TCE_PAGE_SHIFT,
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index c24fa96..8c89c18 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -440,7 +440,7 @@ int spapr_rtas_device_tree_setup(void *fdt, hwaddr rtas_addr,
 #define SPAPR_TCE_PAGE_SIZE    (1ULL << SPAPR_TCE_PAGE_SHIFT)
 #define SPAPR_TCE_PAGE_MASK    (SPAPR_TCE_PAGE_SIZE - 1)
 
-#define SPAPR_VIO_BASE_LIOBN    0x00000000
+#define SPAPR_VIO_LIOBN(reg)    (0x00000000 | (reg))
 #define SPAPR_PCI_LIOBN(i, n)   (0x80000000 | ((i) << 8) | (n))
 #define SPAPR_PCI_DMA_WINDOW_NUM(liobn) ((liobn) & 0xff)
 
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 06/18] spapr_pci: Make find_phb()/find_dev() public

[Alexey Kardashevskiy]

This makes find_phb()/find_dev() public and changed its names
to spapr_pci_find_phb()/spapr_pci_find_dev() as they are going to
be used from other parts of QEMU such as VFIO DDW (dynamic DMA window)
or VFIO PCI error injection or VFIO EEH handling - in all these
cases there are RTAS calls which are addressed to BUID+config_addr
in IEEE1275 format.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 hw/ppc/spapr_pci.c          | 22 +++++++++++-----------
 include/hw/pci-host/spapr.h |  4 ++++
 2 files changed, 15 insertions(+), 11 deletions(-)

diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index c6cf5c1..727d1fe 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -47,7 +47,7 @@
 #define RTAS_TYPE_MSI           1
 #define RTAS_TYPE_MSIX          2
 
-static sPAPRPHBState *find_phb(sPAPREnvironment *spapr, uint64_t buid)
+sPAPRPHBState *spapr_pci_find_phb(sPAPREnvironment *spapr, uint64_t buid)
 {
     sPAPRPHBState *sphb;
 
@@ -61,10 +61,10 @@ static sPAPRPHBState *find_phb(sPAPREnvironment *spapr, uint64_t buid)
     return NULL;
 }
 
-static PCIDevice *find_dev(sPAPREnvironment *spapr, uint64_t buid,
-                           uint32_t config_addr)
+PCIDevice *spapr_pci_find_dev(sPAPREnvironment *spapr, uint64_t buid,
+                              uint32_t config_addr)
 {
-    sPAPRPHBState *sphb = find_phb(spapr, buid);
+    sPAPRPHBState *sphb = spapr_pci_find_phb(spapr, buid);
     PCIHostState *phb = PCI_HOST_BRIDGE(sphb);
     int bus_num = (config_addr >> 16) & 0xFF;
     int devfn = (config_addr >> 8) & 0xFF;
@@ -95,7 +95,7 @@ static void finish_read_pci_config(sPAPREnvironment *spapr, uint64_t buid,
         return;
     }
 
-    pci_dev = find_dev(spapr, buid, addr);
+    pci_dev = spapr_pci_find_dev(spapr, buid, addr);
     addr = rtas_pci_cfgaddr(addr);
 
     if (!pci_dev || (addr % size) || (addr >= pci_config_size(pci_dev))) {
@@ -162,7 +162,7 @@ static void finish_write_pci_config(sPAPREnvironment *spapr, uint64_t buid,
         return;
     }
 
-    pci_dev = find_dev(spapr, buid, addr);
+    pci_dev = spapr_pci_find_dev(spapr, buid, addr);
     addr = rtas_pci_cfgaddr(addr);
 
     if (!pci_dev || (addr % size) || (addr >= pci_config_size(pci_dev))) {
@@ -280,9 +280,9 @@ static void rtas_ibm_change_msi(PowerPCCPU *cpu, sPAPREnvironment *spapr,
     }
 
     /* Fins sPAPRPHBState */
-    phb = find_phb(spapr, buid);
+    phb = spapr_pci_find_phb(spapr, buid);
     if (phb) {
-        pdev = find_dev(spapr, buid, config_addr);
+        pdev = spapr_pci_find_dev(spapr, buid, config_addr);
     }
     if (!phb || !pdev) {
         rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
@@ -381,9 +381,9 @@ static void rtas_ibm_query_interrupt_source_number(PowerPCCPU *cpu,
     spapr_pci_msi *msi;
 
     /* Find sPAPRPHBState */
-    phb = find_phb(spapr, buid);
+    phb = spapr_pci_find_phb(spapr, buid);
     if (phb) {
-        pdev = find_dev(spapr, buid, config_addr);
+        pdev = spapr_pci_find_dev(spapr, buid, config_addr);
     }
     if (!phb || !pdev) {
         rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
@@ -530,7 +530,7 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp)
         return;
     }
 
-    if (find_phb(spapr, sphb->buid)) {
+    if (spapr_pci_find_phb(spapr, sphb->buid)) {
         error_setg(errp, "PCI host bridges must have unique BUIDs");
         return;
     }
diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index 4ea2a0d..3892f1a 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -122,4 +122,8 @@ void spapr_pci_msi_init(sPAPREnvironment *spapr, hwaddr addr);
 
 void spapr_pci_rtas_init(void);
 
+sPAPRPHBState *spapr_pci_find_phb(sPAPREnvironment *spapr, uint64_t buid);
+PCIDevice *spapr_pci_find_dev(sPAPREnvironment *spapr, uint64_t buid,
+                              uint32_t config_addr);
+
 #endif /* __HW_SPAPR_PCI_H__ */
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 07/18] spapr_iommu: Make spapr_tce_find_by_liobn() public

[Alexey Kardashevskiy]

At the moment spapr_tce_find_by_liobn() is used by H_PUT_TCE/...
handlers to find an IOMMU by LIOBN.

We are going to implement Dynamic DMA windows (DDW), new code
will go to a new file and we will use spapr_tce_find_by_liobn()
there too so let's make it public.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
 hw/ppc/spapr_iommu.c   | 2 +-
 include/hw/ppc/spapr.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr_iommu.c b/hw/ppc/spapr_iommu.c
index 7f9a862..a789ce1 100644
--- a/hw/ppc/spapr_iommu.c
+++ b/hw/ppc/spapr_iommu.c
@@ -40,7 +40,7 @@ enum sPAPRTCEAccess {
 
 static QLIST_HEAD(spapr_tce_tables, sPAPRTCETable) spapr_tce_tables;
 
-static sPAPRTCETable *spapr_tce_find_by_liobn(uint32_t liobn)
+sPAPRTCETable *spapr_tce_find_by_liobn(uint32_t liobn)
 {
     sPAPRTCETable *tcet;
 
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 8c89c18..f3911a0 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -466,6 +466,7 @@ struct sPAPRTCETable {
     QLIST_ENTRY(sPAPRTCETable) list;
 };
 
+sPAPRTCETable *spapr_tce_find_by_liobn(uint32_t liobn);
 void spapr_events_init(sPAPREnvironment *spapr);
 void spapr_events_fdt_skel(void *fdt, uint32_t epow_irq);
 int spapr_h_cas_compose_response(target_ulong addr, target_ulong size);
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 08/18] spapr_iommu: Implement free_table() helper

[Alexey Kardashevskiy]

Every sPAPRTCETable object holds an IOMMU memory region which holds
a referenced to the sPAPRTCETable instance. So if we want to free
an sPAPRTCETable instance, calling object_unref() will not be enough
as embedded memory region will hold the reference and we need to break
the loop.

This adds a spapr_tce_free_table() helper which destroys the embedded
memory region and then calls object_unref() on the sPAPRTCETable instance
which will succeed now. The helper adds a new child under unique name
derived from LIOBN.

As we are here, fix spapr_tce_new_table() callers.
At the moment spapr_tce_new_table() references sPAPRTCETable twice -
once in object_new() and second time in object_property_add_child().
The callers of spapr_tce_new_table() should take care of correct
referencing.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr_iommu.c    | 10 +++++++++-
 hw/ppc/spapr_pci.c      |  2 ++
 hw/ppc/spapr_pci_vfio.c |  2 ++
 include/hw/ppc/spapr.h  |  1 +
 4 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr_iommu.c b/hw/ppc/spapr_iommu.c
index a789ce1..bde4c1c 100644
--- a/hw/ppc/spapr_iommu.c
+++ b/hw/ppc/spapr_iommu.c
@@ -147,6 +147,7 @@ sPAPRTCETable *spapr_tce_new_table(DeviceState *owner, uint32_t liobn,
                                    bool vfio_accel)
 {
     sPAPRTCETable *tcet;
+    char buf[32];
 
     if (spapr_tce_find_by_liobn(liobn)) {
         fprintf(stderr, "Attempted to create TCE table with duplicate"
@@ -165,13 +166,20 @@ sPAPRTCETable *spapr_tce_new_table(DeviceState *owner, uint32_t liobn,
     tcet->nb_table = nb_table;
     tcet->vfio_accel = vfio_accel;
 
-    object_property_add_child(OBJECT(owner), "tce-table", OBJECT(tcet), NULL);
+    snprintf(buf, sizeof(buf) - 1, "tce-table-%08X", tcet->liobn);
+    object_property_add_child(OBJECT(owner), buf, OBJECT(tcet), NULL);
 
     object_property_set_bool(OBJECT(tcet), true, "realized", NULL);
 
     return tcet;
 }
 
+void spapr_tce_free_table(sPAPRTCETable *tcet)
+{
+    object_unparent(OBJECT(&tcet->iommu));
+    object_unparent(OBJECT(tcet));
+}
+
 static void spapr_tce_table_finalize(Object *obj)
 {
     sPAPRTCETable *tcet = SPAPR_TCE_TABLE(obj);
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index 727d1fe..ea4b1bb 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -649,6 +649,8 @@ static void spapr_phb_finish_realize(sPAPRPHBState *sphb, Error **errp)
     /* Register default 32bit DMA window */
     memory_region_add_subregion(&sphb->iommu_root, 0,
                                 spapr_tce_get_iommu(tcet));
+
+    object_unref(OBJECT(tcet));
 }
 
 static int spapr_phb_children_reset(Object *child, void *opaque)
diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
index d3bddf2..51b4314 100644
--- a/hw/ppc/spapr_pci_vfio.c
+++ b/hw/ppc/spapr_pci_vfio.c
@@ -69,6 +69,8 @@ static void spapr_phb_vfio_finish_realize(sPAPRPHBState *sphb, Error **errp)
     /* Register default 32bit DMA window */
     memory_region_add_subregion(&sphb->iommu_root, tcet->bus_offset,
                                 spapr_tce_get_iommu(tcet));
+
+    object_unref(OBJECT(tcet));
 }
 
 static void spapr_phb_vfio_reset(DeviceState *qdev)
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index f3911a0..5a2a49f 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -475,6 +475,7 @@ sPAPRTCETable *spapr_tce_new_table(DeviceState *owner, uint32_t liobn,
                                    uint32_t page_shift,
                                    uint32_t nb_table,
                                    bool vfio_accel);
+void spapr_tce_free_table(sPAPRTCETable *tcet);
 MemoryRegion *spapr_tce_get_iommu(sPAPRTCETable *tcet);
 void spapr_tce_set_bypass(sPAPRTCETable *tcet, bool bypass);
 int spapr_dma_dt(void *fdt, int node_off, const char *propname,
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 09/18] spapr_rtas: Reserve DDW RTAS token numbers

[Alexey Kardashevskiy]

Will be squashed later.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 include/hw/ppc/spapr.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 5a2a49f..0e8c0e7 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -381,8 +381,12 @@ int spapr_allocate_irq_block(int num, bool lsi, bool msi);
 #define RTAS_GET_SENSOR_STATE                   (RTAS_TOKEN_BASE + 0x1D)
 #define RTAS_IBM_CONFIGURE_CONNECTOR            (RTAS_TOKEN_BASE + 0x1E)
 #define RTAS_IBM_OS_TERM                        (RTAS_TOKEN_BASE + 0x1F)
+#define RTAS_IBM_QUERY_PE_DMA_WINDOW            (RTAS_TOKEN_BASE + 0x20)
+#define RTAS_IBM_CREATE_PE_DMA_WINDOW           (RTAS_TOKEN_BASE + 0x21)
+#define RTAS_IBM_REMOVE_PE_DMA_WINDOW           (RTAS_TOKEN_BASE + 0x22)
+#define RTAS_IBM_RESET_PE_DMA_WINDOW            (RTAS_TOKEN_BASE + 0x23)
 
-#define RTAS_TOKEN_MAX                          (RTAS_TOKEN_BASE + 0x20)
+#define RTAS_TOKEN_MAX                          (RTAS_TOKEN_BASE + 0x24)
 
 /* RTAS ibm,get-system-parameter token values */
 #define RTAS_SYSPARM_SPLPAR_CHARACTERISTICS      20
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 10/18] spapr_pci: Define DDW callbacks

[Alexey Kardashevskiy]

This adds callbacks definitions which PHB needs to implement in order to
support dynamic DMA windows (DDW).

Will be squashed later.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 include/hw/pci-host/spapr.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index 3892f1a..6658b9c 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -49,6 +49,24 @@ struct sPAPRPHBClass {
     PCIHostBridgeClass parent_class;
 
     void (*finish_realize)(sPAPRPHBState *sphb, Error **errp);
+
+/* sPAPR spec defined pagesize mask values */
+#define DDW_PGSIZE_4K       0x01
+#define DDW_PGSIZE_64K      0x02
+#define DDW_PGSIZE_16M      0x04
+#define DDW_PGSIZE_32M      0x08
+#define DDW_PGSIZE_64M      0x10
+#define DDW_PGSIZE_128M     0x20
+#define DDW_PGSIZE_256M     0x40
+#define DDW_PGSIZE_16G      0x80
+
+    int (*ddw_query)(sPAPRPHBState *sphb, uint32_t *windows_available,
+                     uint32_t *page_size_mask);
+    int (*ddw_create)(sPAPRPHBState *sphb, uint32_t page_shift,
+                      uint32_t window_shift, uint32_t liobn,
+                      sPAPRTCETable **ptcet);
+    int (*ddw_remove)(sPAPRPHBState *sphb, sPAPRTCETable *tcet);
+    int (*ddw_reset)(sPAPRPHBState *sphb);
 };
 
 typedef struct spapr_pci_msi {
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 11/18] spapr_pci: Add windows counter

[Alexey Kardashevskiy]

We are goint to have more than one DMA window and this is going to
be dynamic. This adds a counter which will be used to get a new DMA
window LIOBN. It is set to 1 by default as window #0 is there by default.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr_pci.c          | 5 ++++-
 hw/ppc/spapr_pci_vfio.c     | 2 ++
 include/hw/pci-host/spapr.h | 1 +
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index ea4b1bb..2968b39 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -651,6 +651,8 @@ static void spapr_phb_finish_realize(sPAPRPHBState *sphb, Error **errp)
                                 spapr_tce_get_iommu(tcet));
 
     object_unref(OBJECT(tcet));
+
+    sphb->windows_num = 1;
 }
 
 static int spapr_phb_children_reset(Object *child, void *opaque)
@@ -754,7 +756,7 @@ static int spapr_pci_post_load(void *opaque, int version_id)
 
 static const VMStateDescription vmstate_spapr_pci = {
     .name = "spapr_pci",
-    .version_id = 2,
+    .version_id = 3,
     .minimum_version_id = 2,
     .pre_save = spapr_pci_pre_save,
     .post_load = spapr_pci_post_load,
@@ -770,6 +772,7 @@ static const VMStateDescription vmstate_spapr_pci = {
         VMSTATE_INT32(msi_devs_num, sPAPRPHBState),
         VMSTATE_STRUCT_VARRAY_ALLOC(msi_devs, sPAPRPHBState, msi_devs_num, 0,
                                     vmstate_spapr_pci_msi, spapr_pci_msi_mig),
+        VMSTATE_UINT32_V(windows_num, sPAPRPHBState, 3),
         VMSTATE_END_OF_LIST()
     },
 };
diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
index 51b4314..efed23f 100644
--- a/hw/ppc/spapr_pci_vfio.c
+++ b/hw/ppc/spapr_pci_vfio.c
@@ -71,6 +71,8 @@ static void spapr_phb_vfio_finish_realize(sPAPRPHBState *sphb, Error **errp)
                                 spapr_tce_get_iommu(tcet));
 
     object_unref(OBJECT(tcet));
+
+    sphb->windows_num = 1;
 }
 
 static void spapr_phb_vfio_reset(DeviceState *qdev)
diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index 6658b9c..07b60c3 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -91,6 +91,7 @@ struct sPAPRPHBState {
     MemoryRegion memwindow, iowindow, msiwindow;
 
     uint32_t dma_liobn;
+    uint32_t windows_num;
     AddressSpace iommu_as;
     MemoryRegion iommu_root;
 
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 12/18] spapr_rtas: Add Dynamic DMA windows (DDW) RTAS calls support

[Alexey Kardashevskiy]

NO RESET

This adds support for Dynamic DMA Windows (DDW) option defined by
the SPAPR specification which allows to have additional DMA window(s)
which can support page sizes other than 4K.

The existing implementation of DDW in the guest tries to create one huge
DMA window with 64K or 16MB pages and map the entire guest RAM to. If it
succeeds, the guest switches to dma_direct_ops and never calls
TCE hypercalls (H_PUT_TCE,...) again. This enables VFIO devices to use
the entire RAM and not waste time on map/unmap.

This adds 4 RTAS handlers:
* ibm,query-pe-dma-window
* ibm,create-pe-dma-window
* ibm,remove-pe-dma-window
* ibm,reset-pe-dma-window
These are registered from type_init() callback.

These RTAS handlers are implemented in a separate file to avoid polluting
spapr_iommu.c with PHB.

Since no PHB class implements new callback in this patch, no functional
change is expected.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v2:
* double loop squashed to spapr_iommu_fixmask() helper
* added @ddw_num counter to PHB, it is used to generate LIOBN for new
window; it is reset on ddw-reset event
* added ULL to constants used in shift operations
* rtas_ibm_reset_pe_dma_window() and rtas_ibm_remove_pe_dma_window()
do not remove windows anymore, the PHB callback has to as it will reuse
the same code in case of guest reboot as well
---
 hw/ppc/Makefile.objs    |   3 +
 hw/ppc/spapr_rtas_ddw.c | 234 ++++++++++++++++++++++++++++++++++++++++++++++++
 trace-events            |   3 +
 3 files changed, 240 insertions(+)
 create mode 100644 hw/ppc/spapr_rtas_ddw.c

diff --git a/hw/ppc/Makefile.objs b/hw/ppc/Makefile.objs
index edd44d0..9773294 100644
--- a/hw/ppc/Makefile.objs
+++ b/hw/ppc/Makefile.objs
@@ -7,6 +7,9 @@ obj-$(CONFIG_PSERIES) += spapr_pci.o
 ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES)$(CONFIG_LINUX), yyy)
 obj-y += spapr_pci_vfio.o
 endif
+ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES), yy)
+obj-y += spapr_rtas_ddw.o
+endif
 # PowerPC 4xx boards
 obj-y += ppc405_boards.o ppc4xx_devs.o ppc405_uc.o ppc440_bamboo.o
 obj-y += ppc4xx_pci.o
diff --git a/hw/ppc/spapr_rtas_ddw.c b/hw/ppc/spapr_rtas_ddw.c
new file mode 100644
index 0000000..9ed15dc
--- /dev/null
+++ b/hw/ppc/spapr_rtas_ddw.c
@@ -0,0 +1,234 @@
+/*
+ * QEMU sPAPR Dynamic DMA windows support
+ *
+ * Copyright (c) 2014 Alexey Kardashevskiy, IBM Corporation.
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License,
+ *  or (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "hw/ppc/spapr.h"
+#include "hw/pci-host/spapr.h"
+#include "trace.h"
+
+static uint32_t spapr_iommu_fixmask(struct ppc_one_seg_page_size *sps,
+                                    uint32_t query_mask)
+{
+    int i, j;
+    uint32_t mask = 0;
+    const struct { int shift; uint32_t mask; } masks[] = {
+        { 12, DDW_PGSIZE_4K },
+        { 16, DDW_PGSIZE_64K },
+        { 24, DDW_PGSIZE_16M },
+        { 25, DDW_PGSIZE_32M },
+        { 26, DDW_PGSIZE_64M },
+        { 27, DDW_PGSIZE_128M },
+        { 28, DDW_PGSIZE_256M },
+        { 34, DDW_PGSIZE_16G },
+    };
+
+    for (i = 0; i < PPC_PAGE_SIZES_MAX_SZ; i++) {
+        for (j = 0; j < ARRAY_SIZE(masks); ++j) {
+            if ((sps[i].page_shift == masks[j].shift) &&
+                    (query_mask & masks[j].mask)) {
+                mask |= masks[j].mask;
+            }
+        }
+    }
+
+    return mask;
+}
+
+static void rtas_ibm_query_pe_dma_window(PowerPCCPU *cpu,
+                                         sPAPREnvironment *spapr,
+                                         uint32_t token, uint32_t nargs,
+                                         target_ulong args,
+                                         uint32_t nret, target_ulong rets)
+{
+    CPUPPCState *env = &cpu->env;
+    sPAPRPHBState *sphb;
+    sPAPRPHBClass *spc;
+    uint64_t buid;
+    uint32_t addr, pgmask = 0;
+    uint32_t windows_available = 0, page_size_mask = 0;
+    long ret;
+
+    if ((nargs != 3) || (nret != 5)) {
+        goto param_error_exit;
+    }
+
+    buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    addr = rtas_ld(args, 0);
+    sphb = spapr_pci_find_phb(spapr, buid);
+    if (!sphb) {
+        goto param_error_exit;
+    }
+
+    spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+    if (!spc->ddw_query) {
+        goto hw_error_exit;
+    }
+
+    ret = spc->ddw_query(sphb, &windows_available, &page_size_mask);
+    trace_spapr_iommu_ddw_query(buid, addr, windows_available,
+                                page_size_mask, pgmask, ret);
+    if (ret) {
+        goto hw_error_exit;
+    }
+
+    /* Work out supported page masks */
+    pgmask = spapr_iommu_fixmask(env->sps.sps, page_size_mask);
+
+    rtas_st(rets, 0, RTAS_OUT_SUCCESS);
+    rtas_st(rets, 1, windows_available);
+
+    /*
+     * This is "Largest contiguous block of TCEs allocated specifically
+     * for (that is, are reserved for) this PE".
+     * Return the maximum number as all RAM was in 4K pages.
+     */
+    rtas_st(rets, 2, ram_size >> SPAPR_TCE_PAGE_SHIFT);
+    rtas_st(rets, 3, pgmask);
+    rtas_st(rets, 4, pgmask); /* DMA migration mask */
+    return;
+
+hw_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_HW_ERROR);
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+static void rtas_ibm_create_pe_dma_window(PowerPCCPU *cpu,
+                                          sPAPREnvironment *spapr,
+                                          uint32_t token, uint32_t nargs,
+                                          target_ulong args,
+                                          uint32_t nret, target_ulong rets)
+{
+    sPAPRPHBState *sphb;
+    sPAPRPHBClass *spc;
+    sPAPRTCETable *tcet = NULL;
+    uint32_t addr, page_shift, window_shift, liobn;
+    uint64_t buid;
+    long ret;
+
+    if ((nargs != 5) || (nret != 4)) {
+        goto param_error_exit;
+    }
+
+    buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
+    addr = rtas_ld(args, 0);
+    sphb = spapr_pci_find_phb(spapr, buid);
+    if (!sphb) {
+        goto param_error_exit;
+    }
+
+    spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+    if (!spc->ddw_create) {
+        goto hw_error_exit;
+    }
+
+    page_shift = rtas_ld(args, 3);
+    window_shift = rtas_ld(args, 4);
+    /* Default 32bit window#0 is always there so +1 */
+    liobn = SPAPR_PCI_LIOBN(sphb->index, sphb->windows_num);
+
+    ret = spc->ddw_create(sphb, page_shift, window_shift, liobn, &tcet);
+    trace_spapr_iommu_ddw_create(buid, addr, 1ULL << page_shift,
+                                 1ULL << window_shift,
+                                 tcet ? tcet->bus_offset : 0xbaadf00d,
+                                 liobn, ret);
+    if (ret || !tcet) {
+        goto hw_error_exit;
+    }
+
+    rtas_st(rets, 0, RTAS_OUT_SUCCESS);
+    rtas_st(rets, 1, liobn);
+    rtas_st(rets, 2, tcet->bus_offset >> 32);
+    rtas_st(rets, 3, tcet->bus_offset & ((uint32_t) -1));
+
+    object_unref(OBJECT(tcet));
+    return;
+
+hw_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_HW_ERROR);
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+static void rtas_ibm_remove_pe_dma_window(PowerPCCPU *cpu,
+                                          sPAPREnvironment *spapr,
+                                          uint32_t token, uint32_t nargs,
+                                          target_ulong args,
+                                          uint32_t nret, target_ulong rets)
+{
+    sPAPRPHBState *sphb;
+    sPAPRPHBClass *spc;
+    sPAPRTCETable *tcet;
+    uint32_t liobn;
+    long ret;
+
+    if ((nargs != 1) || (nret != 1)) {
+        goto param_error_exit;
+    }
+
+    liobn = rtas_ld(args, 0);
+    tcet = spapr_tce_find_by_liobn(liobn);
+    if (!tcet) {
+        goto param_error_exit;
+    }
+
+    sphb = SPAPR_PCI_HOST_BRIDGE(OBJECT(tcet)->parent);
+    if (!sphb) {
+        goto param_error_exit;
+    }
+
+    spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+    if (!spc->ddw_remove) {
+        goto hw_error_exit;
+    }
+
+    ret = spc->ddw_remove(sphb, tcet);
+    trace_spapr_iommu_ddw_remove(liobn, ret);
+    if (ret) {
+        goto hw_error_exit;
+    }
+
+    rtas_st(rets, 0, RTAS_OUT_SUCCESS);
+    return;
+
+hw_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_HW_ERROR);
+    return;
+
+param_error_exit:
+    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
+}
+
+static void spapr_rtas_ddw_init(void)
+{
+    spapr_rtas_register(RTAS_IBM_QUERY_PE_DMA_WINDOW,
+                        "ibm,query-pe-dma-window",
+                        rtas_ibm_query_pe_dma_window);
+    spapr_rtas_register(RTAS_IBM_CREATE_PE_DMA_WINDOW,
+                        "ibm,create-pe-dma-window",
+                        rtas_ibm_create_pe_dma_window);
+    spapr_rtas_register(RTAS_IBM_REMOVE_PE_DMA_WINDOW,
+                        "ibm,remove-pe-dma-window",
+                        rtas_ibm_remove_pe_dma_window);
+}
+
+type_init(spapr_rtas_ddw_init)
diff --git a/trace-events b/trace-events
index 81bc915..0d8de94 100644
--- a/trace-events
+++ b/trace-events
@@ -1218,6 +1218,9 @@ spapr_iommu_indirect(uint64_t liobn, uint64_t ioba, uint64_t tce, uint64_t iobaN
 spapr_iommu_stuff(uint64_t liobn, uint64_t ioba, uint64_t tce_value, uint64_t npages, uint64_t ret) "liobn=%"PRIx64" ioba=0x%"PRIx64" tcevalue=0x%"PRIx64" npages=%"PRId64" ret=%"PRId64
 spapr_iommu_xlate(uint64_t liobn, uint64_t ioba, uint64_t tce, unsigned perm, unsigned pgsize) "liobn=%"PRIx64" 0x%"PRIx64" -> 0x%"PRIx64" perm=%u mask=%x"
 spapr_iommu_new_table(uint64_t liobn, void *tcet, void *table, int fd) "liobn=%"PRIx64" tcet=%p table=%p fd=%d"
+spapr_iommu_ddw_query(uint64_t buid, uint32_t cfgaddr, uint32_t wa, uint32_t pgz, uint32_t pgz_fixed, long ret) "buid=%"PRIx64" addr=%"PRIx32", %u windows available, sizes %"PRIx32", fixed %"PRIx32", ret = %ld"
+spapr_iommu_ddw_create(uint64_t buid, uint32_t cfgaddr, unsigned long long pg_size, unsigned long long req_size, uint64_t start, uint32_t liobn, long ret) "buid=%"PRIx64" addr=%"PRIx32", page size=0x%llx, requested=0x%llx, start addr=%"PRIx64", liobn=%"PRIx32", ret = %ld"
+spapr_iommu_ddw_remove(uint32_t liobn, long ret) "liobn=%"PRIx32", ret = %ld"
 
 # hw/ppc/ppc.c
 ppc_tb_adjust(uint64_t offs1, uint64_t offs2, int64_t diff, int64_t seconds) "adjusted from 0x%"PRIx64" to 0x%"PRIx64", diff %"PRId64" (%"PRId64"s)"
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 13/18] spapr_pci: Enable DDW

[Alexey Kardashevskiy]

This implements DDW for emulated PHB.

This advertises the query/create/remove RTAS tokens in device tree.
This does not advertise the reset RTAS token though, will be added later.

The "ddw" property is enabled by default on a PHB but for compatibility
pseries-2.1 machine disables it.

Since QEMU does not implement any 64bit DMA capable device, this hack
has been used to enable 64bit DMA on E1000:

diff --git a/hw/net/e1000.c b/hw/net/e1000.c
index 0fc29a0..131f80a 100644
--- a/hw/net/e1000.c
+++ b/hw/net/e1000.c
@@ -240,6 +240,7 @@ static const uint32_t mac_reg_init[] = {
     [STATUS] =  0x80000000 | E1000_STATUS_GIO_MASTER_ENABLE |
                 E1000_STATUS_ASDV | E1000_STATUS_MTXCKOK |
                 E1000_STATUS_SPEED_1000 | E1000_STATUS_FD |
+                E1000_STATUS_PCIX_MODE |
                 E1000_STATUS_LU,
     [MANC] =    E1000_MANC_EN_MNG2HOST | E1000_MANC_RCV_TCO_EN |
                 E1000_MANC_ARP_EN | E1000_MANC_0298_EN |

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v3:
* removed reset
* windows_num is now 1 or bigger rather than 0-based value and it is only
changed in PHB code, not in RTAS
* added page mask check in create()

v2:
* tested on hacked emulated E1000
* implemented DDW reset on the PHB reset
* spapr_pci_ddw_remove/spapr_pci_ddw_reset are public for reuse by VFIO
---
 hw/ppc/spapr.c              |  9 +++++
 hw/ppc/spapr_pci.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci-host/spapr.h |  7 ++++
 3 files changed, 110 insertions(+)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index d2d3c27..663cb75 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1670,11 +1670,20 @@ static const TypeInfo spapr_machine_info = {
 static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
+    static GlobalProperty compat_props[] = {
+        {
+            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,
+            .property = "ddw",
+            .value    = stringify(off),
+        },
+        { /* end of list */ }
+    };
 
     mc->name = "pseries-2.1";
     mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
     mc->alias = "pseries";
     mc->is_default = 1;
+    mc->compat_props = compat_props;
 }
 
 static const TypeInfo spapr_machine_2_1_info = {
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index 2968b39..04ee1dc 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -470,6 +470,76 @@ static const MemoryRegionOps spapr_msi_ops = {
 };
 
 /*
+ * Dynamic DMA windows
+ */
+static int spapr_pci_ddw_query(sPAPRPHBState *sphb,
+                               uint32_t *windows_available,
+                               uint32_t *page_size_mask)
+{
+    *windows_available = 1;
+    *page_size_mask = DDW_PGSIZE_64K | DDW_PGSIZE_16M;
+
+    return 0;
+}
+
+static int spapr_pci_ddw_create(sPAPRPHBState *sphb, uint32_t page_shift,
+                                uint32_t window_shift, uint32_t liobn,
+                                sPAPRTCETable **ptcet)
+{
+    if ((page_shift != 16) && (page_shift != 24)) {
+        return -1;
+    }
+
+    *ptcet = spapr_tce_new_table(DEVICE(sphb), liobn,
+                                 SPAPR_PCI_TCE64_START, page_shift,
+                                 1ULL << (window_shift - page_shift),
+                                 true);
+    if (!*ptcet) {
+        return -1;
+    }
+    memory_region_add_subregion(&sphb->iommu_root, (*ptcet)->bus_offset,
+                                spapr_tce_get_iommu(*ptcet));
+
+    ++sphb->windows_num;
+
+    return 0;
+}
+
+int spapr_pci_ddw_remove(sPAPRPHBState *sphb, sPAPRTCETable *tcet)
+{
+    memory_region_del_subregion(&sphb->iommu_root,
+                                spapr_tce_get_iommu(tcet));
+    spapr_tce_free_table(tcet);
+
+    return 0;
+}
+
+static int spapr_pci_remove_ddw_cb(Object *child, void *opaque)
+{
+    sPAPRTCETable *tcet;
+
+    tcet = (sPAPRTCETable *) object_dynamic_cast(child, TYPE_SPAPR_TCE_TABLE);
+
+    /* Delete all dynamic windows, i.e. every except the default one with #0 */
+    if (tcet && SPAPR_PCI_DMA_WINDOW_NUM(tcet->liobn)) {
+        sPAPRPHBState *sphb = opaque;
+        sPAPRPHBClass *spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
+
+        spc->ddw_remove(sphb, tcet);
+    }
+
+    return 0;
+}
+
+int spapr_pci_ddw_reset(sPAPRPHBState *sphb)
+{
+    object_child_foreach(OBJECT(sphb), spapr_pci_remove_ddw_cb, sphb);
+    sphb->windows_num = 1;
+
+    return 0;
+}
+
+/*
  * PHB PCI device
  */
 static AddressSpace *spapr_pci_dma_iommu(PCIBus *bus, void *opaque, int devfn)
@@ -668,6 +738,12 @@ static int spapr_phb_children_reset(Object *child, void *opaque)
 
 static void spapr_phb_reset(DeviceState *qdev)
 {
+    sPAPRPHBClass *spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(qdev);
+
+    if (spc->ddw_reset) {
+        spc->ddw_reset(SPAPR_PCI_HOST_BRIDGE(qdev));
+    }
+
     /* Reset the IOMMU state */
     object_child_foreach(OBJECT(qdev), spapr_phb_children_reset, NULL);
 }
@@ -682,6 +758,7 @@ static Property spapr_phb_properties[] = {
     DEFINE_PROP_UINT64("io_win_addr", sPAPRPHBState, io_win_addr, -1),
     DEFINE_PROP_UINT64("io_win_size", sPAPRPHBState, io_win_size,
                        SPAPR_PCI_IO_WIN_SIZE),
+    DEFINE_PROP_BOOL("ddw", sPAPRPHBState, ddw_enabled, true),
     DEFINE_PROP_END_OF_LIST(),
 };
 
@@ -799,6 +876,10 @@ static void spapr_phb_class_init(ObjectClass *klass, void *data)
     set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
     dc->cannot_instantiate_with_device_add_yet = false;
     spc->finish_realize = spapr_phb_finish_realize;
+    spc->ddw_query = spapr_pci_ddw_query;
+    spc->ddw_create = spapr_pci_ddw_create;
+    spc->ddw_remove = spapr_pci_ddw_remove;
+    spc->ddw_reset = spapr_pci_ddw_reset;
 }
 
 static const TypeInfo spapr_phb_info = {
@@ -882,6 +963,12 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
     uint32_t interrupt_map_mask[] = {
         cpu_to_be32(b_ddddd(-1)|b_fff(0)), 0x0, 0x0, cpu_to_be32(-1)};
     uint32_t interrupt_map[PCI_SLOT_MAX * PCI_NUM_PINS][7];
+    uint32_t ddw_applicable[] = {
+        RTAS_IBM_QUERY_PE_DMA_WINDOW,
+        RTAS_IBM_CREATE_PE_DMA_WINDOW,
+        RTAS_IBM_REMOVE_PE_DMA_WINDOW
+    };
+    sPAPRPHBClass *spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(phb);
 
     /* Start populating the FDT */
     sprintf(nodename, "pci@%" PRIx64, phb->buid);
@@ -911,6 +998,13 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
     _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1));
     _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS));
 
+    /* Dynamic DMA window */
+    if (phb->ddw_enabled &&
+        spc->ddw_query && spc->ddw_create && spc->ddw_remove) {
+        _FDT(fdt_setprop(fdt, bus_off, "ibm,ddw-applicable", &ddw_applicable,
+                         sizeof(ddw_applicable)));
+    }
+
     /* Build the interrupt-map, this must matches what is done
      * in pci_spapr_map_irq
      */
diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index 07b60c3..01b3d25 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -104,6 +104,8 @@ struct sPAPRPHBState {
     int32_t msi_devs_num;
     spapr_pci_msi_mig *msi_devs;
 
+    bool ddw_enabled;
+
     QLIST_ENTRY(sPAPRPHBState) list;
 };
 
@@ -126,6 +128,9 @@ struct sPAPRPHBVFIOState {
 
 #define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL
 
+/* Default 64bit dynamic window offset */
+#define SPAPR_PCI_TCE64_START        0x8000000000000000ULL
+
 static inline qemu_irq spapr_phb_lsi_qirq(struct sPAPRPHBState *phb, int pin)
 {
     return xics_get_qirq(spapr->icp, phb->lsi_table[pin].irq);
@@ -144,5 +149,7 @@ void spapr_pci_rtas_init(void);
 sPAPRPHBState *spapr_pci_find_phb(sPAPREnvironment *spapr, uint64_t buid);
 PCIDevice *spapr_pci_find_dev(sPAPREnvironment *spapr, uint64_t buid,
                               uint32_t config_addr);
+int spapr_pci_ddw_remove(sPAPRPHBState *sphb, sPAPRTCETable *tcet);
+int spapr_pci_ddw_reset(sPAPRPHBState *sphb);
 
 #endif /* __HW_SPAPR_PCI_H__ */
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 14/18] spapr_pci_vfio: Call spapr_pci::reset on reset

[Alexey Kardashevskiy]

This enables use of the parent class rest() callback in VFIO.

This effectively enables reset of PHBs TCE IOMMU object(s) which are not
bus devices and do not get reset by qemu_devices_reset() called from
a machine reset callback.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr_pci_vfio.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
index efed23f..e53880e 100644
--- a/hw/ppc/spapr_pci_vfio.c
+++ b/hw/ppc/spapr_pci_vfio.c
@@ -75,18 +75,12 @@ static void spapr_phb_vfio_finish_realize(sPAPRPHBState *sphb, Error **errp)
     sphb->windows_num = 1;
 }
 
-static void spapr_phb_vfio_reset(DeviceState *qdev)
-{
-    /* Do nothing */
-}
-
 static void spapr_phb_vfio_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
     sPAPRPHBClass *spc = SPAPR_PCI_HOST_BRIDGE_CLASS(klass);
 
     dc->props = spapr_phb_vfio_properties;
-    dc->reset = spapr_phb_vfio_reset;
     spc->finish_realize = spapr_phb_vfio_finish_realize;
 }
 
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 15/18] linux headers update for SPAPR (DDW + EEH)

[Alexey Kardashevskiy]

Since the changes are not in upstream yet, no tag or branch is specified here.

EEH is included to have a chance of catching problems with some shifted
IOCTL number or other random stuff like that.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 linux-headers/linux/vfio.h | 58 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h
index 26c218e..b3c3e9e 100644
--- a/linux-headers/linux/vfio.h
+++ b/linux-headers/linux/vfio.h
@@ -30,6 +30,9 @@
  */
 #define VFIO_DMA_CC_IOMMU		4
 
+/* Check if EEH is supported */
+#define VFIO_EEH			5
+
 /*
  * The IOCTL interface is designed for extensibility by embedding the
  * structure length (argsz) and flags into structures passed between
@@ -448,13 +451,66 @@ struct vfio_iommu_type1_dma_unmap {
  */
 struct vfio_iommu_spapr_tce_info {
 	__u32 argsz;
-	__u32 flags;			/* reserved for future use */
+	__u32 flags;
+#define VFIO_IOMMU_SPAPR_TCE_FLAG_DDW	1 /* Support dynamic windows */
 	__u32 dma32_window_start;	/* 32 bit window start (bytes) */
 	__u32 dma32_window_size;	/* 32 bit window size (bytes) */
+	__u32 current_windows;
+	__u32 windows_available;
+	__u32 page_size_mask;
 };
 
 #define VFIO_IOMMU_SPAPR_TCE_GET_INFO	_IO(VFIO_TYPE, VFIO_BASE + 12)
 
+/*
+ * EEH PE operation struct provides ways to:
+ * - enable/disable EEH functionality;
+ * - unfreeze IO/DMA for frozen PE;
+ * - read PE state;
+ * - reset PE;
+ * - configure PE.
+ */
+struct vfio_eeh_pe_op {
+	__u32 argsz;
+	__u32 flags;
+	__u32 op;
+};
+
+#define VFIO_EEH_PE_DISABLE		0	/* Disable EEH functionality */
+#define VFIO_EEH_PE_ENABLE		1	/* Enable EEH functionality  */
+#define VFIO_EEH_PE_UNFREEZE_IO		2	/* Enable IO for frozen PE   */
+#define VFIO_EEH_PE_UNFREEZE_DMA	3	/* Enable DMA for frozen PE  */
+#define VFIO_EEH_PE_GET_STATE		4	/* PE state retrieval        */
+#define  VFIO_EEH_PE_STATE_NORMAL	0	/* PE in functional state    */
+#define  VFIO_EEH_PE_STATE_RESET	1	/* PE reset in progress      */
+#define  VFIO_EEH_PE_STATE_STOPPED	2	/* Stopped DMA and IO        */
+#define  VFIO_EEH_PE_STATE_STOPPED_DMA	4	/* Stopped DMA only          */
+#define  VFIO_EEH_PE_STATE_UNAVAIL	5	/* State unavailable         */
+#define VFIO_EEH_PE_RESET_DEACTIVATE	5	/* Deassert PE reset         */
+#define VFIO_EEH_PE_RESET_HOT		6	/* Assert hot reset          */
+#define VFIO_EEH_PE_RESET_FUNDAMENTAL	7	/* Assert fundamental reset  */
+#define VFIO_EEH_PE_CONFIGURE		8	/* PE configuration          */
+
+#define VFIO_EEH_PE_OP			_IO(VFIO_TYPE, VFIO_BASE + 21)
+
+struct vfio_iommu_spapr_tce_create {
+	__u32 argsz;
+	__u32 flags;
+	/* in */
+	__u32 page_shift;
+	__u32 window_shift;
+	/* out */
+	__u64 start_addr;
+};
+#define VFIO_IOMMU_SPAPR_TCE_CREATE	_IO(VFIO_TYPE, VFIO_BASE + 18)
+
+struct vfio_iommu_spapr_tce_remove {
+	__u32 argsz;
+	__u32 flags;
+	/* in */
+	__u64 start_addr;
+};
+#define VFIO_IOMMU_SPAPR_TCE_REMOVE	_IO(VFIO_TYPE, VFIO_BASE + 19)
 /* ***************************************************************** */
 
 #endif /* VFIO_H */
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 16/18] spapr_pci_vfio: Enable DDW

[Alexey Kardashevskiy]

This implements DDW for VFIO. Host kernel support is required for this.

After this patch DDW will be enabled on all machines but pseries-2.1.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v2:
* remove()/reset() callbacks use spapr_pci's ones
---
 hw/ppc/spapr_pci_vfio.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)

diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
index e53880e..94e1361 100644
--- a/hw/ppc/spapr_pci_vfio.c
+++ b/hw/ppc/spapr_pci_vfio.c
@@ -73,6 +73,77 @@ static void spapr_phb_vfio_finish_realize(sPAPRPHBState *sphb, Error **errp)
     object_unref(OBJECT(tcet));
 
     sphb->windows_num = 1;
+
+    if (sphb->ddw_enabled) {
+        sphb->ddw_enabled = !!(info.flags & VFIO_IOMMU_SPAPR_TCE_FLAG_DDW);
+    }
+}
+
+static int spapr_pci_vfio_ddw_query(sPAPRPHBState *sphb,
+                                    uint32_t *windows_available,
+                                    uint32_t *page_size_mask)
+{
+    sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
+    struct vfio_iommu_spapr_tce_info info = { .argsz = sizeof(info) };
+    int ret;
+
+    ret = vfio_container_ioctl(&sphb->iommu_as, svphb->iommugroupid,
+                               VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+    if (ret) {
+        return ret;
+    }
+
+    *windows_available = info.windows_available;
+    *page_size_mask = info.page_size_mask;
+
+    return ret;
+}
+
+static int spapr_pci_vfio_ddw_create(sPAPRPHBState *sphb, uint32_t page_shift,
+                                     uint32_t window_shift, uint32_t liobn,
+                                     sPAPRTCETable **ptcet)
+{
+    sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
+    struct vfio_iommu_spapr_tce_create create = {
+        .argsz = sizeof(create),
+        .page_shift = page_shift,
+        .window_shift = window_shift,
+        .start_addr = 0
+    };
+    int ret;
+
+    ret = vfio_container_ioctl(&sphb->iommu_as, svphb->iommugroupid,
+                               VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+    if (ret) {
+        return ret;
+    }
+
+    *ptcet = spapr_tce_new_table(DEVICE(sphb), liobn,
+                                 create.start_addr, page_shift,
+                                 1ULL << (window_shift - page_shift),
+                                 true);
+    memory_region_add_subregion(&sphb->iommu_root, (*ptcet)->bus_offset,
+                                spapr_tce_get_iommu(*ptcet));
+
+    ++sphb->windows_num;
+
+    return ret;
+}
+
+static int spapr_pci_vfio_ddw_remove(sPAPRPHBState *sphb, sPAPRTCETable *tcet)
+{
+    sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
+    struct vfio_iommu_spapr_tce_remove remove = {
+        .argsz = sizeof(remove),
+        .start_addr = tcet->bus_offset
+    };
+    int ret;
+
+    spapr_pci_ddw_remove(sphb, tcet);
+    ret = vfio_container_ioctl(&sphb->iommu_as, svphb->iommugroupid,
+                               VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+
+    return ret;
 }
 
 static void spapr_phb_vfio_class_init(ObjectClass *klass, void *data)
@@ -82,6 +153,9 @@ static void spapr_phb_vfio_class_init(ObjectClass *klass, void *data)
 
     dc->props = spapr_phb_vfio_properties;
     spc->finish_realize = spapr_phb_vfio_finish_realize;
+    spc->ddw_query = spapr_pci_vfio_ddw_query;
+    spc->ddw_create = spapr_pci_vfio_ddw_create;
+    spc->ddw_remove = spapr_pci_vfio_ddw_remove;
 }
 
 static const TypeInfo spapr_phb_vfio_info = {
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 17/18] vfio: Enable DDW ioctls to VFIO IOMMU driver

[Alexey Kardashevskiy]

This enables DDW RTAS-related ioctls in VFIO.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/misc/vfio.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
index 40dcaa6..fa90c93 100644
--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -4450,6 +4450,8 @@ int vfio_container_ioctl(AddressSpace *as, int32_t groupid,
     switch (req) {
     case VFIO_CHECK_EXTENSION:
     case VFIO_IOMMU_SPAPR_TCE_GET_INFO:
+    case VFIO_IOMMU_SPAPR_TCE_CREATE:
+    case VFIO_IOMMU_SPAPR_TCE_REMOVE:
         break;
     default:
         /* Return an error on unknown requests */
-- 
2.0.0

[Qemu-devel] [RFC PATCH v3 18/18] spapr: Add pseries-2.2 machine

[Alexey Kardashevskiy]

This defines new "pseries" machine version 2.2 and make it a default
machine.

Since "pseries-2.2" does not have "ddw" compat properties set to off and
"pseries" becomes an alias of "presies-2.2" instead of 2.1, this
effectively enables DDW on both emulated and VFIO PHBs.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 663cb75..9dfa96c 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1681,8 +1681,7 @@ static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
 
     mc->name = "pseries-2.1";
     mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
-    mc->alias = "pseries";
-    mc->is_default = 1;
+    mc->is_default = 0;
     mc->compat_props = compat_props;
 }
 
@@ -1692,10 +1691,26 @@ static const TypeInfo spapr_machine_2_1_info = {
     .class_init    = spapr_machine_2_1_class_init,
 };
 
+static void spapr_machine_2_2_class_init(ObjectClass *oc, void *data)
+{
+    MachineClass *mc = MACHINE_CLASS(oc);
+    mc->name = "pseries-2.2";
+    mc->desc = "pSeries Logical Partition (PAPR compliant) v2.2";
+    mc->alias = "pseries";
+    mc->is_default = 1;
+}
+
+static const TypeInfo spapr_machine_2_2_info = {
+    .name          = TYPE_SPAPR_MACHINE "2.2",
+    .parent        = TYPE_SPAPR_MACHINE,
+    .class_init    = spapr_machine_2_2_class_init,
+};
+
 static void spapr_machine_register_types(void)
 {
     type_register_static(&spapr_machine_info);
     type_register_static(&spapr_machine_2_1_info);
+    type_register_static(&spapr_machine_2_2_info);
 }
 
 type_init(spapr_machine_register_types)
-- 
2.0.0

Re: [Qemu-devel] [RFC PATCH v3 03/18] spapr_iommu: Disable in-kernel IOMMU tables for >4GB windows

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

On Fri, Aug 29, 2014 at 08:12:07PM +1000, Alexey Kardashevskiy wrote:
> The existing KVM_CREATE_SPAPR_TCE ioctl only support 4G windows max as
> the window size parameter to the kernel ioctl() is 32-bit so
> there's no way of expressing a TCE window > 4GB.
> 
> We are going to add huge DMA windows support so this will create small
> window and unexpectedly fail later.
> 
> This disables KVM_CREATE_SPAPR_TCE for windows bigger that 4GB. Since
> those windows are normally mapped at the boot time, there will be no
> performance impact.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 04/18] spapr_pci: Introduce a liobn number generating macros

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

On Fri, Aug 29, 2014 at 08:12:08PM +1000, Alexey Kardashevskiy wrote:
> We are going to have multiple DMA windows per PHB and we want them to
> migrate so we need a predictable way of assigning LIOBNs.
> 
> This introduces a macro which makes up a LIOBN from fixed prefix,
> PHB index (unique PHB id) and window number.
> 
> This introduces a SPAPR_PCI_DMA_WINDOW_NUM() to know the window number
> from LIOBN. It is used to distinguish the default 32bit windows from
> dynamic windows and avoid picking default DMA window properties from
> a wrong TCE table.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 05/18] spapr_vio: Introduce a liobn number generating macros

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

On Fri, Aug 29, 2014 at 08:12:09PM +1000, Alexey Kardashevskiy wrote:
> This introduces a macro which makes up a LIOBN from fixed prefix and
> VIO device address (@reg property).
> 
> This is to keep LIOBN macros rendering consistent - the same macro for
> PCI has been added by the previous patch.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 09/18] spapr_rtas: Reserve DDW RTAS token numbers

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

On Fri, Aug 29, 2014 at 08:12:13PM +1000, Alexey Kardashevskiy wrote:
> Will be squashed later.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 10/18] spapr_pci: Define DDW callbacks

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 528 bytes --]

On Fri, Aug 29, 2014 at 08:12:14PM +1000, Alexey Kardashevskiy wrote:
> This adds callbacks definitions which PHB needs to implement in order to
> support dynamic DMA windows (DDW).
> 
> Will be squashed later.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 11/18] spapr_pci: Add windows counter

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

On Fri, Aug 29, 2014 at 08:12:15PM +1000, Alexey Kardashevskiy wrote:
> We are goint to have more than one DMA window and this is going to
> be dynamic. This adds a counter which will be used to get a new DMA
> window LIOBN. It is set to 1 by default as window #0 is there by default.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 13/18] spapr_pci: Enable DDW

[Alexander Graf]

On 29.08.14 12:12, Alexey Kardashevskiy wrote:
> This implements DDW for emulated PHB.
> 
> This advertises the query/create/remove RTAS tokens in device tree.
> This does not advertise the reset RTAS token though, will be added later.
> 
> The "ddw" property is enabled by default on a PHB but for compatibility
> pseries-2.1 machine disables it.
> 
> Since QEMU does not implement any 64bit DMA capable device, this hack
> has been used to enable 64bit DMA on E1000:
> 
> diff --git a/hw/net/e1000.c b/hw/net/e1000.c
> index 0fc29a0..131f80a 100644
> --- a/hw/net/e1000.c
> +++ b/hw/net/e1000.c
> @@ -240,6 +240,7 @@ static const uint32_t mac_reg_init[] = {
>      [STATUS] =  0x80000000 | E1000_STATUS_GIO_MASTER_ENABLE |
>                  E1000_STATUS_ASDV | E1000_STATUS_MTXCKOK |
>                  E1000_STATUS_SPEED_1000 | E1000_STATUS_FD |
> +                E1000_STATUS_PCIX_MODE |
>                  E1000_STATUS_LU,
>      [MANC] =    E1000_MANC_EN_MNG2HOST | E1000_MANC_RCV_TCO_EN |
>                  E1000_MANC_ARP_EN | E1000_MANC_0298_EN |
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> Changes:
> v3:
> * removed reset
> * windows_num is now 1 or bigger rather than 0-based value and it is only
> changed in PHB code, not in RTAS
> * added page mask check in create()
> 
> v2:
> * tested on hacked emulated E1000
> * implemented DDW reset on the PHB reset
> * spapr_pci_ddw_remove/spapr_pci_ddw_reset are public for reuse by VFIO
> ---
>  hw/ppc/spapr.c              |  9 +++++
>  hw/ppc/spapr_pci.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++
>  include/hw/pci-host/spapr.h |  7 ++++
>  3 files changed, 110 insertions(+)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index d2d3c27..663cb75 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1670,11 +1670,20 @@ static const TypeInfo spapr_machine_info = {
>  static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
>  {
>      MachineClass *mc = MACHINE_CLASS(oc);
> +    static GlobalProperty compat_props[] = {
> +        {
> +            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,
> +            .property = "ddw",
> +            .value    = stringify(off),
> +        },
> +        { /* end of list */ }
> +    };
>  
>      mc->name = "pseries-2.1";
>      mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
>      mc->alias = "pseries";
>      mc->is_default = 1;
> +    mc->compat_props = compat_props;
>  }
>  
>  static const TypeInfo spapr_machine_2_1_info = {
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index 2968b39..04ee1dc 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -470,6 +470,76 @@ static const MemoryRegionOps spapr_msi_ops = {
>  };
>  
>  /*
> + * Dynamic DMA windows
> + */
> +static int spapr_pci_ddw_query(sPAPRPHBState *sphb,
> +                               uint32_t *windows_available,
> +                               uint32_t *page_size_mask)
> +{
> +    *windows_available = 1;
> +    *page_size_mask = DDW_PGSIZE_64K | DDW_PGSIZE_16M;
> +
> +    return 0;

What exactly does this return? The number of still available windows?
The total number of windows?


Alex

Re: [Qemu-devel] [RFC PATCH v3 16/18] spapr_pci_vfio: Enable DDW

[Alexander Graf]

On 29.08.14 12:12, Alexey Kardashevskiy wrote:
> This implements DDW for VFIO. Host kernel support is required for this.
> 
> After this patch DDW will be enabled on all machines but pseries-2.1.

I fail to see the code that restricts DDW to pseries-2.1. Can you please
point me to it?


Alex

Re: [Qemu-devel] [RFC PATCH v3 13/18] spapr_pci: Enable DDW

[Alexey Kardashevskiy]

On 09/10/2014 11:01 PM, Alexander Graf wrote:
> 
> 
> On 29.08.14 12:12, Alexey Kardashevskiy wrote:
>> This implements DDW for emulated PHB.
>>
>> This advertises the query/create/remove RTAS tokens in device tree.
>> This does not advertise the reset RTAS token though, will be added later.
>>
>> The "ddw" property is enabled by default on a PHB but for compatibility
>> pseries-2.1 machine disables it.
>>
>> Since QEMU does not implement any 64bit DMA capable device, this hack
>> has been used to enable 64bit DMA on E1000:
>>
>> diff --git a/hw/net/e1000.c b/hw/net/e1000.c
>> index 0fc29a0..131f80a 100644
>> --- a/hw/net/e1000.c
>> +++ b/hw/net/e1000.c
>> @@ -240,6 +240,7 @@ static const uint32_t mac_reg_init[] = {
>>      [STATUS] =  0x80000000 | E1000_STATUS_GIO_MASTER_ENABLE |
>>                  E1000_STATUS_ASDV | E1000_STATUS_MTXCKOK |
>>                  E1000_STATUS_SPEED_1000 | E1000_STATUS_FD |
>> +                E1000_STATUS_PCIX_MODE |
>>                  E1000_STATUS_LU,
>>      [MANC] =    E1000_MANC_EN_MNG2HOST | E1000_MANC_RCV_TCO_EN |
>>                  E1000_MANC_ARP_EN | E1000_MANC_0298_EN |
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>> Changes:
>> v3:
>> * removed reset
>> * windows_num is now 1 or bigger rather than 0-based value and it is only
>> changed in PHB code, not in RTAS
>> * added page mask check in create()
>>
>> v2:
>> * tested on hacked emulated E1000
>> * implemented DDW reset on the PHB reset
>> * spapr_pci_ddw_remove/spapr_pci_ddw_reset are public for reuse by VFIO
>> ---
>>  hw/ppc/spapr.c              |  9 +++++
>>  hw/ppc/spapr_pci.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++
>>  include/hw/pci-host/spapr.h |  7 ++++
>>  3 files changed, 110 insertions(+)
>>
>> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
>> index d2d3c27..663cb75 100644
>> --- a/hw/ppc/spapr.c
>> +++ b/hw/ppc/spapr.c
>> @@ -1670,11 +1670,20 @@ static const TypeInfo spapr_machine_info = {
>>  static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
>>  {
>>      MachineClass *mc = MACHINE_CLASS(oc);
>> +    static GlobalProperty compat_props[] = {
>> +        {
>> +            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,
>> +            .property = "ddw",
>> +            .value    = stringify(off),
>> +        },
>> +        { /* end of list */ }
>> +    };
>>  
>>      mc->name = "pseries-2.1";
>>      mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
>>      mc->alias = "pseries";
>>      mc->is_default = 1;
>> +    mc->compat_props = compat_props;
>>  }
>>  
>>  static const TypeInfo spapr_machine_2_1_info = {
>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>> index 2968b39..04ee1dc 100644
>> --- a/hw/ppc/spapr_pci.c
>> +++ b/hw/ppc/spapr_pci.c
>> @@ -470,6 +470,76 @@ static const MemoryRegionOps spapr_msi_ops = {
>>  };
>>  
>>  /*
>> + * Dynamic DMA windows
>> + */
>> +static int spapr_pci_ddw_query(sPAPRPHBState *sphb,
>> +                               uint32_t *windows_available,
>> +                               uint32_t *page_size_mask)
>> +{
>> +    *windows_available = 1;
>> +    *page_size_mask = DDW_PGSIZE_64K | DDW_PGSIZE_16M;
>> +
>> +    return 0;
> 
> What exactly does this return? The number of still available windows?
> The total number of windows?

1. Number of available windows
2. page masks supported for new windows.

This is what RTAS's counterpart does. We can change it to something better
if we want but I cannot think of anything better.



-- 
Alexey

Re: [Qemu-devel] [RFC PATCH v3 16/18] spapr_pci_vfio: Enable DDW

[Alexey Kardashevskiy]

On 09/10/2014 11:03 PM, Alexander Graf wrote:
> 
> 
> On 29.08.14 12:12, Alexey Kardashevskiy wrote:
>> This implements DDW for VFIO. Host kernel support is required for this.
>>
>> After this patch DDW will be enabled on all machines but pseries-2.1.
> 
> I fail to see the code that restricts DDW to pseries-2.1. Can you please
> point me to it?

Patch "[RFC PATCH v3 13/18] spapr_pci: Enable DDW" which enables DDW on
spapr-pci disables it for the current machine by default.

Since spapr-pci-vfio inherits from spapr-pci, this property is off for
pseries-2.1 as well (I checked).


-- 
Alexey

Re: [Qemu-devel] [RFC PATCH v3 13/18] spapr_pci: Enable DDW

[Alexander Graf]

On 10.09.14 16:58, Alexey Kardashevskiy wrote:
> On 09/10/2014 11:01 PM, Alexander Graf wrote:
>>
>>
>> On 29.08.14 12:12, Alexey Kardashevskiy wrote:
>>> This implements DDW for emulated PHB.
>>>
>>> This advertises the query/create/remove RTAS tokens in device tree.
>>> This does not advertise the reset RTAS token though, will be added later.
>>>
>>> The "ddw" property is enabled by default on a PHB but for compatibility
>>> pseries-2.1 machine disables it.
>>>
>>> Since QEMU does not implement any 64bit DMA capable device, this hack
>>> has been used to enable 64bit DMA on E1000:
>>>
>>> diff --git a/hw/net/e1000.c b/hw/net/e1000.c
>>> index 0fc29a0..131f80a 100644
>>> --- a/hw/net/e1000.c
>>> +++ b/hw/net/e1000.c
>>> @@ -240,6 +240,7 @@ static const uint32_t mac_reg_init[] = {
>>>      [STATUS] =  0x80000000 | E1000_STATUS_GIO_MASTER_ENABLE |
>>>                  E1000_STATUS_ASDV | E1000_STATUS_MTXCKOK |
>>>                  E1000_STATUS_SPEED_1000 | E1000_STATUS_FD |
>>> +                E1000_STATUS_PCIX_MODE |
>>>                  E1000_STATUS_LU,
>>>      [MANC] =    E1000_MANC_EN_MNG2HOST | E1000_MANC_RCV_TCO_EN |
>>>                  E1000_MANC_ARP_EN | E1000_MANC_0298_EN |
>>>
>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>> ---
>>> Changes:
>>> v3:
>>> * removed reset
>>> * windows_num is now 1 or bigger rather than 0-based value and it is only
>>> changed in PHB code, not in RTAS
>>> * added page mask check in create()
>>>
>>> v2:
>>> * tested on hacked emulated E1000
>>> * implemented DDW reset on the PHB reset
>>> * spapr_pci_ddw_remove/spapr_pci_ddw_reset are public for reuse by VFIO
>>> ---
>>>  hw/ppc/spapr.c              |  9 +++++
>>>  hw/ppc/spapr_pci.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++
>>>  include/hw/pci-host/spapr.h |  7 ++++
>>>  3 files changed, 110 insertions(+)
>>>
>>> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
>>> index d2d3c27..663cb75 100644
>>> --- a/hw/ppc/spapr.c
>>> +++ b/hw/ppc/spapr.c
>>> @@ -1670,11 +1670,20 @@ static const TypeInfo spapr_machine_info = {
>>>  static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
>>>  {
>>>      MachineClass *mc = MACHINE_CLASS(oc);
>>> +    static GlobalProperty compat_props[] = {
>>> +        {
>>> +            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,
>>> +            .property = "ddw",
>>> +            .value    = stringify(off),
>>> +        },
>>> +        { /* end of list */ }
>>> +    };
>>>  
>>>      mc->name = "pseries-2.1";
>>>      mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
>>>      mc->alias = "pseries";
>>>      mc->is_default = 1;
>>> +    mc->compat_props = compat_props;
>>>  }
>>>  
>>>  static const TypeInfo spapr_machine_2_1_info = {
>>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>>> index 2968b39..04ee1dc 100644
>>> --- a/hw/ppc/spapr_pci.c
>>> +++ b/hw/ppc/spapr_pci.c
>>> @@ -470,6 +470,76 @@ static const MemoryRegionOps spapr_msi_ops = {
>>>  };
>>>  
>>>  /*
>>> + * Dynamic DMA windows
>>> + */
>>> +static int spapr_pci_ddw_query(sPAPRPHBState *sphb,
>>> +                               uint32_t *windows_available,
>>> +                               uint32_t *page_size_mask)
>>> +{
>>> +    *windows_available = 1;
>>> +    *page_size_mask = DDW_PGSIZE_64K | DDW_PGSIZE_16M;
>>> +
>>> +    return 0;
>>
>> What exactly does this return? The number of still available windows?
>> The total number of windows?
> 
> 1. Number of available windows
> 2. page masks supported for new windows.
> 
> This is what RTAS's counterpart does. We can change it to something better
> if we want but I cannot think of anything better.

The reason I'm asking is that I don't quite grasp why we're always
returning 1 available window.

1 window is already taken by the default window, no? So "1" naturally
would mean no ddw window. If we ignore that one, what if I create 1 ddw
window? The call would still return 1, so could I create another one? If
I can, why didn't the call return 2 in the first place?


Alex

Re: [Qemu-devel] [RFC PATCH v3 16/18] spapr_pci_vfio: Enable DDW

[Alexander Graf]

On 10.09.14 17:00, Alexey Kardashevskiy wrote:
> On 09/10/2014 11:03 PM, Alexander Graf wrote:
>>
>>
>> On 29.08.14 12:12, Alexey Kardashevskiy wrote:
>>> This implements DDW for VFIO. Host kernel support is required for this.
>>>
>>> After this patch DDW will be enabled on all machines but pseries-2.1.
>>
>> I fail to see the code that restricts DDW to pseries-2.1. Can you please
>> point me to it?
> 
> Patch "[RFC PATCH v3 13/18] spapr_pci: Enable DDW" which enables DDW on
> spapr-pci disables it for the current machine by default.
> 
> Since spapr-pci-vfio inherits from spapr-pci, this property is off for
> pseries-2.1 as well (I checked).

Ah, I see. The logic is still reversed. Old machine types get compat
options, new machine types don't. So the default should be "ddw enabled"
and pseries-2.1 gets a compat field saying "disable ddw".


Alex

Re: [Qemu-devel] [RFC PATCH v3 13/18] spapr_pci: Enable DDW

[Alexey Kardashevskiy]

On 09/11/2014 07:16 AM, Alexander Graf wrote:
> 
> 
> On 10.09.14 16:58, Alexey Kardashevskiy wrote:
>> On 09/10/2014 11:01 PM, Alexander Graf wrote:
>>>
>>>
>>> On 29.08.14 12:12, Alexey Kardashevskiy wrote:
>>>> This implements DDW for emulated PHB.
>>>>
>>>> This advertises the query/create/remove RTAS tokens in device tree.
>>>> This does not advertise the reset RTAS token though, will be added later.
>>>>
>>>> The "ddw" property is enabled by default on a PHB but for compatibility
>>>> pseries-2.1 machine disables it.
>>>>
>>>> Since QEMU does not implement any 64bit DMA capable device, this hack
>>>> has been used to enable 64bit DMA on E1000:
>>>>
>>>> diff --git a/hw/net/e1000.c b/hw/net/e1000.c
>>>> index 0fc29a0..131f80a 100644
>>>> --- a/hw/net/e1000.c
>>>> +++ b/hw/net/e1000.c
>>>> @@ -240,6 +240,7 @@ static const uint32_t mac_reg_init[] = {
>>>>      [STATUS] =  0x80000000 | E1000_STATUS_GIO_MASTER_ENABLE |
>>>>                  E1000_STATUS_ASDV | E1000_STATUS_MTXCKOK |
>>>>                  E1000_STATUS_SPEED_1000 | E1000_STATUS_FD |
>>>> +                E1000_STATUS_PCIX_MODE |
>>>>                  E1000_STATUS_LU,
>>>>      [MANC] =    E1000_MANC_EN_MNG2HOST | E1000_MANC_RCV_TCO_EN |
>>>>                  E1000_MANC_ARP_EN | E1000_MANC_0298_EN |
>>>>
>>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>>> ---
>>>> Changes:
>>>> v3:
>>>> * removed reset
>>>> * windows_num is now 1 or bigger rather than 0-based value and it is only
>>>> changed in PHB code, not in RTAS
>>>> * added page mask check in create()
>>>>
>>>> v2:
>>>> * tested on hacked emulated E1000
>>>> * implemented DDW reset on the PHB reset
>>>> * spapr_pci_ddw_remove/spapr_pci_ddw_reset are public for reuse by VFIO
>>>> ---
>>>>  hw/ppc/spapr.c              |  9 +++++
>>>>  hw/ppc/spapr_pci.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++
>>>>  include/hw/pci-host/spapr.h |  7 ++++
>>>>  3 files changed, 110 insertions(+)
>>>>
>>>> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
>>>> index d2d3c27..663cb75 100644
>>>> --- a/hw/ppc/spapr.c
>>>> +++ b/hw/ppc/spapr.c
>>>> @@ -1670,11 +1670,20 @@ static const TypeInfo spapr_machine_info = {
>>>>  static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
>>>>  {
>>>>      MachineClass *mc = MACHINE_CLASS(oc);
>>>> +    static GlobalProperty compat_props[] = {
>>>> +        {
>>>> +            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,
>>>> +            .property = "ddw",
>>>> +            .value    = stringify(off),
>>>> +        },
>>>> +        { /* end of list */ }
>>>> +    };
>>>>  
>>>>      mc->name = "pseries-2.1";
>>>>      mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
>>>>      mc->alias = "pseries";
>>>>      mc->is_default = 1;
>>>> +    mc->compat_props = compat_props;
>>>>  }
>>>>  
>>>>  static const TypeInfo spapr_machine_2_1_info = {
>>>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>>>> index 2968b39..04ee1dc 100644
>>>> --- a/hw/ppc/spapr_pci.c
>>>> +++ b/hw/ppc/spapr_pci.c
>>>> @@ -470,6 +470,76 @@ static const MemoryRegionOps spapr_msi_ops = {
>>>>  };
>>>>  
>>>>  /*
>>>> + * Dynamic DMA windows
>>>> + */
>>>> +static int spapr_pci_ddw_query(sPAPRPHBState *sphb,
>>>> +                               uint32_t *windows_available,
>>>> +                               uint32_t *page_size_mask)
>>>> +{
>>>> +    *windows_available = 1;
>>>> +    *page_size_mask = DDW_PGSIZE_64K | DDW_PGSIZE_16M;
>>>> +
>>>> +    return 0;
>>>
>>> What exactly does this return? The number of still available windows?
>>> The total number of windows?
>>
>> 1. Number of available windows
>> 2. page masks supported for new windows.
>>
>> This is what RTAS's counterpart does. We can change it to something better
>> if we want but I cannot think of anything better.
> 
> The reason I'm asking is that I don't quite grasp why we're always
> returning 1 available window.


Oh. I see it now. Bug. Should 1 or 0 depending on whether big window was
created or not.

> 1 window is already taken by the default window, no? So "1" naturally
> would mean no ddw window. If we ignore that one, what if I create 1 ddw
> window? The call would still return 1, so could I create another one? If
> I can, why didn't the call return 2 in the first place?

"1" means 1 additional window can be created. It does not return "2"
because the default window is always there. When/If I enable "pe-dma-reset"
RTAS call, then the guest will be able to remove default DMA window and
then spapr_pci_ddw_query() will start returning "2" after the default
window is removed.


-- 
Alexey

Re: [Qemu-devel] [RFC PATCH v3 16/18] spapr_pci_vfio: Enable DDW

[Alexey Kardashevskiy]

On 09/11/2014 07:17 AM, Alexander Graf wrote:
> 
> 
> On 10.09.14 17:00, Alexey Kardashevskiy wrote:
>> On 09/10/2014 11:03 PM, Alexander Graf wrote:
>>>
>>>
>>> On 29.08.14 12:12, Alexey Kardashevskiy wrote:
>>>> This implements DDW for VFIO. Host kernel support is required for this.
>>>>
>>>> After this patch DDW will be enabled on all machines but pseries-2.1.
>>>
>>> I fail to see the code that restricts DDW to pseries-2.1. Can you please
>>> point me to it?
>>
>> Patch "[RFC PATCH v3 13/18] spapr_pci: Enable DDW" which enables DDW on
>> spapr-pci disables it for the current machine by default.
>>
>> Since spapr-pci-vfio inherits from spapr-pci, this property is off for
>> pseries-2.1 as well (I checked).
> 
> Ah, I see. The logic is still reversed. Old machine types get compat
> options, new machine types don't. So the default should be "ddw enabled"
> and pseries-2.1 gets a compat field saying "disable ddw".

I lost you here, sorry. "[RFC PATCH v3 13/18] spapr_pci: Enable DDW" once
again:

 static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
+    static GlobalProperty compat_props[] = {
+        {
+            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,
+            .property = "ddw",
+            .value    = stringify(off),
+        },
+        { /* end of list */ }
+    };

[...]

@@ -682,6 +758,7 @@ static Property spapr_phb_properties[] = {
     DEFINE_PROP_UINT64("io_win_addr", sPAPRPHBState, io_win_addr, -1),
     DEFINE_PROP_UINT64("io_win_size", sPAPRPHBState, io_win_size,
                        SPAPR_PCI_IO_WIN_SIZE),
+    DEFINE_PROP_BOOL("ddw", sPAPRPHBState, ddw_enabled, true),
     DEFINE_PROP_END_OF_LIST(),
 };


What is wrong here?


-- 
Alexey

Re: [Qemu-devel] [RFC PATCH v3 16/18] spapr_pci_vfio: Enable DDW

[Alexander Graf]

On 11.09.14 04:55, Alexey Kardashevskiy wrote:
> On 09/11/2014 07:17 AM, Alexander Graf wrote:
>>
>>
>> On 10.09.14 17:00, Alexey Kardashevskiy wrote:
>>> On 09/10/2014 11:03 PM, Alexander Graf wrote:
>>>>
>>>>
>>>> On 29.08.14 12:12, Alexey Kardashevskiy wrote:
>>>>> This implements DDW for VFIO. Host kernel support is required for this.
>>>>>
>>>>> After this patch DDW will be enabled on all machines but pseries-2.1.
>>>>
>>>> I fail to see the code that restricts DDW to pseries-2.1. Can you please
>>>> point me to it?
>>>
>>> Patch "[RFC PATCH v3 13/18] spapr_pci: Enable DDW" which enables DDW on
>>> spapr-pci disables it for the current machine by default.
>>>
>>> Since spapr-pci-vfio inherits from spapr-pci, this property is off for
>>> pseries-2.1 as well (I checked).
>>
>> Ah, I see. The logic is still reversed. Old machine types get compat
>> options, new machine types don't. So the default should be "ddw enabled"
>> and pseries-2.1 gets a compat field saying "disable ddw".
> 
> I lost you here, sorry. "[RFC PATCH v3 13/18] spapr_pci: Enable DDW" once
> again:

Ugh, sorry, I misread :). Looks perfect.


Alex

Re: [Qemu-devel] [RFC PATCH v3 12/18] spapr_rtas: Add Dynamic DMA windows (DDW) RTAS calls support

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 9251 bytes --]

On Fri, Aug 29, 2014 at 08:12:16PM +1000, Alexey Kardashevskiy wrote:
> NO RESET
> 
> This adds support for Dynamic DMA Windows (DDW) option defined by
> the SPAPR specification which allows to have additional DMA window(s)
> which can support page sizes other than 4K.
> 
> The existing implementation of DDW in the guest tries to create one huge
> DMA window with 64K or 16MB pages and map the entire guest RAM to. If it
> succeeds, the guest switches to dma_direct_ops and never calls
> TCE hypercalls (H_PUT_TCE,...) again. This enables VFIO devices to use
> the entire RAM and not waste time on map/unmap.
> 
> This adds 4 RTAS handlers:
> * ibm,query-pe-dma-window
> * ibm,create-pe-dma-window
> * ibm,remove-pe-dma-window
> * ibm,reset-pe-dma-window
> These are registered from type_init() callback.
> 
> These RTAS handlers are implemented in a separate file to avoid polluting
> spapr_iommu.c with PHB.
> 
> Since no PHB class implements new callback in this patch, no functional
> change is expected.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> Changes:
> v2:
> * double loop squashed to spapr_iommu_fixmask() helper
> * added @ddw_num counter to PHB, it is used to generate LIOBN for new
> window; it is reset on ddw-reset event
> * added ULL to constants used in shift operations
> * rtas_ibm_reset_pe_dma_window() and rtas_ibm_remove_pe_dma_window()
> do not remove windows anymore, the PHB callback has to as it will reuse
> the same code in case of guest reboot as well
> ---
>  hw/ppc/Makefile.objs    |   3 +
>  hw/ppc/spapr_rtas_ddw.c | 234 ++++++++++++++++++++++++++++++++++++++++++++++++
>  trace-events            |   3 +
>  3 files changed, 240 insertions(+)
>  create mode 100644 hw/ppc/spapr_rtas_ddw.c
> 
> diff --git a/hw/ppc/Makefile.objs b/hw/ppc/Makefile.objs
> index edd44d0..9773294 100644
> --- a/hw/ppc/Makefile.objs
> +++ b/hw/ppc/Makefile.objs
> @@ -7,6 +7,9 @@ obj-$(CONFIG_PSERIES) += spapr_pci.o
>  ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES)$(CONFIG_LINUX), yyy)
>  obj-y += spapr_pci_vfio.o
>  endif
> +ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES), yy)
> +obj-y += spapr_rtas_ddw.o
> +endif
>  # PowerPC 4xx boards
>  obj-y += ppc405_boards.o ppc4xx_devs.o ppc405_uc.o ppc440_bamboo.o
>  obj-y += ppc4xx_pci.o
> diff --git a/hw/ppc/spapr_rtas_ddw.c b/hw/ppc/spapr_rtas_ddw.c
> new file mode 100644
> index 0000000..9ed15dc
> --- /dev/null
> +++ b/hw/ppc/spapr_rtas_ddw.c
> @@ -0,0 +1,234 @@
> +/*
> + * QEMU sPAPR Dynamic DMA windows support
> + *
> + * Copyright (c) 2014 Alexey Kardashevskiy, IBM Corporation.
> + *
> + *  This program is free software; you can redistribute it and/or modify
> + *  it under the terms of the GNU General Public License as published by
> + *  the Free Software Foundation; either version 2 of the License,
> + *  or (at your option) any later version.
> + *
> + *  This program is distributed in the hope that it will be useful,
> + *  but WITHOUT ANY WARRANTY; without even the implied warranty of
> + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + *  GNU General Public License for more details.
> + *
> + *  You should have received a copy of the GNU General Public License
> + *  along with this program; if not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include "hw/ppc/spapr.h"
> +#include "hw/pci-host/spapr.h"
> +#include "trace.h"
> +
> +static uint32_t spapr_iommu_fixmask(struct ppc_one_seg_page_size *sps,
> +                                    uint32_t query_mask)
> +{
> +    int i, j;
> +    uint32_t mask = 0;
> +    const struct { int shift; uint32_t mask; } masks[] = {
> +        { 12, DDW_PGSIZE_4K },
> +        { 16, DDW_PGSIZE_64K },
> +        { 24, DDW_PGSIZE_16M },
> +        { 25, DDW_PGSIZE_32M },
> +        { 26, DDW_PGSIZE_64M },
> +        { 27, DDW_PGSIZE_128M },
> +        { 28, DDW_PGSIZE_256M },
> +        { 34, DDW_PGSIZE_16G },
> +    };
> +
> +    for (i = 0; i < PPC_PAGE_SIZES_MAX_SZ; i++) {
> +        for (j = 0; j < ARRAY_SIZE(masks); ++j) {
> +            if ((sps[i].page_shift == masks[j].shift) &&
> +                    (query_mask & masks[j].mask)) {
> +                mask |= masks[j].mask;
> +            }
> +        }
> +    }
> +
> +    return mask;
> +}
> +
> +static void rtas_ibm_query_pe_dma_window(PowerPCCPU *cpu,
> +                                         sPAPREnvironment *spapr,
> +                                         uint32_t token, uint32_t nargs,
> +                                         target_ulong args,
> +                                         uint32_t nret, target_ulong rets)
> +{
> +    CPUPPCState *env = &cpu->env;
> +    sPAPRPHBState *sphb;
> +    sPAPRPHBClass *spc;
> +    uint64_t buid;
> +    uint32_t addr, pgmask = 0;
> +    uint32_t windows_available = 0, page_size_mask = 0;
> +    long ret;
> +
> +    if ((nargs != 3) || (nret != 5)) {
> +        goto param_error_exit;
> +    }
> +
> +    buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
> +    addr = rtas_ld(args, 0);
> +    sphb = spapr_pci_find_phb(spapr, buid);
> +    if (!sphb) {
> +        goto param_error_exit;
> +    }
> +
> +    spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
> +    if (!spc->ddw_query) {
> +        goto hw_error_exit;
> +    }
> +
> +    ret = spc->ddw_query(sphb, &windows_available, &page_size_mask);
> +    trace_spapr_iommu_ddw_query(buid, addr, windows_available,
> +                                page_size_mask, pgmask, ret);
> +    if (ret) {
> +        goto hw_error_exit;
> +    }
> +
> +    /* Work out supported page masks */
> +    pgmask = spapr_iommu_fixmask(env->sps.sps, page_size_mask);
> +
> +    rtas_st(rets, 0, RTAS_OUT_SUCCESS);
> +    rtas_st(rets, 1, windows_available);
> +
> +    /*
> +     * This is "Largest contiguous block of TCEs allocated specifically
> +     * for (that is, are reserved for) this PE".
> +     * Return the maximum number as all RAM was in 4K pages.
> +     */
> +    rtas_st(rets, 2, ram_size >> SPAPR_TCE_PAGE_SHIFT);

I still think this value should be supplied by the PHB callback, just
in case it needs to apply some constraint to the TCE table size.

> +    rtas_st(rets, 3, pgmask);
> +    rtas_st(rets, 4, pgmask); /* DMA migration mask */

What's the difference between these two copies of the pgmask?

> +    return;
> +
> +hw_error_exit:
> +    rtas_st(rets, 0, RTAS_OUT_HW_ERROR);
> +    return;
> +
> +param_error_exit:
> +    rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR);
> +}
> +
> +static void rtas_ibm_create_pe_dma_window(PowerPCCPU *cpu,
> +                                          sPAPREnvironment *spapr,
> +                                          uint32_t token, uint32_t nargs,
> +                                          target_ulong args,
> +                                          uint32_t nret, target_ulong rets)
> +{
> +    sPAPRPHBState *sphb;
> +    sPAPRPHBClass *spc;
> +    sPAPRTCETable *tcet = NULL;
> +    uint32_t addr, page_shift, window_shift, liobn;
> +    uint64_t buid;
> +    long ret;
> +
> +    if ((nargs != 5) || (nret != 4)) {
> +        goto param_error_exit;
> +    }
> +
> +    buid = ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2);
> +    addr = rtas_ld(args, 0);
> +    sphb = spapr_pci_find_phb(spapr, buid);
> +    if (!sphb) {
> +        goto param_error_exit;
> +    }
> +
> +    spc = SPAPR_PCI_HOST_BRIDGE_GET_CLASS(sphb);
> +    if (!spc->ddw_create) {
> +        goto hw_error_exit;
> +    }
> +
> +    page_shift = rtas_ld(args, 3);
> +    window_shift = rtas_ld(args, 4);
> +    /* Default 32bit window#0 is always there so +1 */
> +    liobn = SPAPR_PCI_LIOBN(sphb->index, sphb->windows_num);
> +
> +    ret = spc->ddw_create(sphb, page_shift, window_shift, liobn, &tcet);
> +    trace_spapr_iommu_ddw_create(buid, addr, 1ULL << page_shift,
> +                                 1ULL << window_shift,
> +                                 tcet ? tcet->bus_offset : 0xbaadf00d,
> +                                 liobn, ret);
> +    if (ret || !tcet) {
> +        goto hw_error_exit;

This isn't strictly always accurate.  For instance if the callback
rejects this because window_shift or page_shift is out of bounds, then
this should really be RTAS_OUT_PARAM_ERROR rather than
RTAS_OUT_HW_ERROR.

I'm not sure it matters enough to fix though.


More generally, one way of fixing the migration problems with this
code we've discussed occurred to me:

Instead of actually dynamically creating the TCE tables, allocate stub
tce table objects for as many DDW windows as permitted as soon as the
PHB initializes, but mark them as disabled.  When the
create-pe-dma-window call comes through enable it and allocate to the
correct size.  That way a stub will exist on the destination for
migration, and the pre/post load hooks should be able to handle the
necessary allocation and transfer.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 13/18] spapr_pci: Enable DDW

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3777 bytes --]

On Fri, Aug 29, 2014 at 08:12:17PM +1000, Alexey Kardashevskiy wrote:
> This implements DDW for emulated PHB.
> 
> This advertises the query/create/remove RTAS tokens in device tree.
> This does not advertise the reset RTAS token though, will be added later.
> 
> The "ddw" property is enabled by default on a PHB but for compatibility
> pseries-2.1 machine disables it.
> 
> Since QEMU does not implement any 64bit DMA capable device, this hack
> has been used to enable 64bit DMA on E1000:
> 
> diff --git a/hw/net/e1000.c b/hw/net/e1000.c
> index 0fc29a0..131f80a 100644
> --- a/hw/net/e1000.c
> +++ b/hw/net/e1000.c
> @@ -240,6 +240,7 @@ static const uint32_t mac_reg_init[] = {
>      [STATUS] =  0x80000000 | E1000_STATUS_GIO_MASTER_ENABLE |
>                  E1000_STATUS_ASDV | E1000_STATUS_MTXCKOK |
>                  E1000_STATUS_SPEED_1000 | E1000_STATUS_FD |
> +                E1000_STATUS_PCIX_MODE |
>                  E1000_STATUS_LU,
>      [MANC] =    E1000_MANC_EN_MNG2HOST | E1000_MANC_RCV_TCO_EN |
>                  E1000_MANC_ARP_EN | E1000_MANC_0298_EN |
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> Changes:
> v3:
> * removed reset
> * windows_num is now 1 or bigger rather than 0-based value and it is only
> changed in PHB code, not in RTAS
> * added page mask check in create()
> 
> v2:
> * tested on hacked emulated E1000
> * implemented DDW reset on the PHB reset
> * spapr_pci_ddw_remove/spapr_pci_ddw_reset are public for reuse by VFIO
> ---
>  hw/ppc/spapr.c              |  9 +++++
>  hw/ppc/spapr_pci.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++
>  include/hw/pci-host/spapr.h |  7 ++++
>  3 files changed, 110 insertions(+)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index d2d3c27..663cb75 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1670,11 +1670,20 @@ static const TypeInfo spapr_machine_info = {
>  static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
>  {
>      MachineClass *mc = MACHINE_CLASS(oc);
> +    static GlobalProperty compat_props[] = {
> +        {
> +            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,
> +            .property = "ddw",
> +            .value    = stringify(off),
> +        },
> +        { /* end of list */ }
> +    };
>  
>      mc->name = "pseries-2.1";
>      mc->desc = "pSeries Logical Partition (PAPR compliant) v2.1";
>      mc->alias = "pseries";
>      mc->is_default = 1;
> +    mc->compat_props = compat_props;
>  }
>  
>  static const TypeInfo spapr_machine_2_1_info = {
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index 2968b39..04ee1dc 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -470,6 +470,76 @@ static const MemoryRegionOps spapr_msi_ops = {
>  };
>  
>  /*
> + * Dynamic DMA windows
> + */
> +static int spapr_pci_ddw_query(sPAPRPHBState *sphb,
> +                               uint32_t *windows_available,
> +                               uint32_t *page_size_mask)
> +{
> +    *windows_available = 1;
> +    *page_size_mask = DDW_PGSIZE_64K | DDW_PGSIZE_16M;
> +
> +    return 0;
> +}
> +
> +static int spapr_pci_ddw_create(sPAPRPHBState *sphb, uint32_t page_shift,
> +                                uint32_t window_shift, uint32_t liobn,
> +                                sPAPRTCETable **ptcet)
> +{

I think the callbacks should check the ddw_enabled parameter and fail
if ddw is disabled, rather than just relying on the guest not calling
them if they're not advertised.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 14/18] spapr_pci_vfio: Call spapr_pci::reset on reset

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Fri, Aug 29, 2014 at 08:12:18PM +1000, Alexey Kardashevskiy wrote:
> This enables use of the parent class rest() callback in VFIO.
> 
> This effectively enables reset of PHBs TCE IOMMU object(s) which are not
> bus devices and do not get reset by qemu_devices_reset() called from
> a machine reset callback.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 15/18] linux headers update for SPAPR (DDW + EEH)

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

On Fri, Aug 29, 2014 at 08:12:19PM +1000, Alexey Kardashevskiy wrote:
> Since the changes are not in upstream yet, no tag or branch is specified here.
> 
> EEH is included to have a chance of catching problems with some shifted
> IOCTL number or other random stuff like that.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

It would be good to give the SHA of the kernel version you're updating
these to match.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 16/18] spapr_pci_vfio: Enable DDW

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4344 bytes --]

On Fri, Aug 29, 2014 at 08:12:20PM +1000, Alexey Kardashevskiy wrote:
> This implements DDW for VFIO. Host kernel support is required for this.
> 
> After this patch DDW will be enabled on all machines but pseries-2.1.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> Changes:
> v2:
> * remove()/reset() callbacks use spapr_pci's ones
> ---
>  hw/ppc/spapr_pci_vfio.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 74 insertions(+)
> 
> diff --git a/hw/ppc/spapr_pci_vfio.c b/hw/ppc/spapr_pci_vfio.c
> index e53880e..94e1361 100644
> --- a/hw/ppc/spapr_pci_vfio.c
> +++ b/hw/ppc/spapr_pci_vfio.c
> @@ -73,6 +73,77 @@ static void spapr_phb_vfio_finish_realize(sPAPRPHBState *sphb, Error **errp)
>      object_unref(OBJECT(tcet));
>  
>      sphb->windows_num = 1;
> +
> +    if (sphb->ddw_enabled) {
> +        sphb->ddw_enabled = !!(info.flags & VFIO_IOMMU_SPAPR_TCE_FLAG_DDW);
> +    }
> +}
> +
> +static int spapr_pci_vfio_ddw_query(sPAPRPHBState *sphb,
> +                                    uint32_t *windows_available,
> +                                    uint32_t *page_size_mask)
> +{
> +    sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
> +    struct vfio_iommu_spapr_tce_info info = { .argsz = sizeof(info) };
> +    int ret;

Again, I think you should check ddw_enabled for the explicitly
disabled case.

> +    ret = vfio_container_ioctl(&sphb->iommu_as, svphb->iommugroupid,
> +                               VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
> +    if (ret) {
> +        return ret;
> +    }

Btw, is the kernel/VFIO side stuff for DDW merged yet?  If not could
you give me a pointer to somewhere the patches are posted, or a git
tree?

> +
> +    *windows_available = info.windows_available;
> +    *page_size_mask = info.page_size_mask;
> +
> +    return ret;
> +}
> +
> +static int spapr_pci_vfio_ddw_create(sPAPRPHBState *sphb, uint32_t page_shift,
> +                                     uint32_t window_shift, uint32_t liobn,
> +                                     sPAPRTCETable **ptcet)
> +{
> +    sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
> +    struct vfio_iommu_spapr_tce_create create = {
> +        .argsz = sizeof(create),
> +        .page_shift = page_shift,
> +        .window_shift = window_shift,
> +        .start_addr = 0
> +    };
> +    int ret;
> +
> +    ret = vfio_container_ioctl(&sphb->iommu_as, svphb->iommugroupid,
> +                               VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
> +    if (ret) {
> +        return ret;
> +    }
> +
> +    *ptcet = spapr_tce_new_table(DEVICE(sphb), liobn,
> +                                 create.start_addr, page_shift,
> +                                 1ULL << (window_shift - page_shift),
> +                                 true);
> +    memory_region_add_subregion(&sphb->iommu_root, (*ptcet)->bus_offset,
> +                                spapr_tce_get_iommu(*ptcet));
> +
> +    ++sphb->windows_num;
> +
> +    return ret;
> +}
> +
> +static int spapr_pci_vfio_ddw_remove(sPAPRPHBState *sphb, sPAPRTCETable *tcet)
> +{
> +    sPAPRPHBVFIOState *svphb = SPAPR_PCI_VFIO_HOST_BRIDGE(sphb);
> +    struct vfio_iommu_spapr_tce_remove remove = {
> +        .argsz = sizeof(remove),
> +        .start_addr = tcet->bus_offset
> +    };
> +    int ret;
> +
> +    spapr_pci_ddw_remove(sphb, tcet);
> +    ret = vfio_container_ioctl(&sphb->iommu_as, svphb->iommugroupid,
> +                               VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
> +
> +    return ret;
>  }
>  
>  static void spapr_phb_vfio_class_init(ObjectClass *klass, void *data)
> @@ -82,6 +153,9 @@ static void spapr_phb_vfio_class_init(ObjectClass *klass, void *data)
>  
>      dc->props = spapr_phb_vfio_properties;
>      spc->finish_realize = spapr_phb_vfio_finish_realize;
> +    spc->ddw_query = spapr_pci_vfio_ddw_query;
> +    spc->ddw_create = spapr_pci_vfio_ddw_create;
> +    spc->ddw_remove = spapr_pci_vfio_ddw_remove;
>  }
>  
>  static const TypeInfo spapr_phb_vfio_info = {

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 17/18] vfio: Enable DDW ioctls to VFIO IOMMU driver

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

On Fri, Aug 29, 2014 at 08:12:21PM +1000, Alexey Kardashevskiy wrote:
> This enables DDW RTAS-related ioctls in VFIO.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [RFC PATCH v3 15/18] linux headers update for SPAPR (DDW + EEH)

[Alexey Kardashevskiy]

On 09/26/2014 03:40 PM, David Gibson wrote:
> On Fri, Aug 29, 2014 at 08:12:19PM +1000, Alexey Kardashevskiy wrote:
>> Since the changes are not in upstream yet, no tag or branch is specified here.
>>
>> EEH is included to have a chance of catching problems with some shifted
>> IOCTL number or other random stuff like that.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> 
> It would be good to give the SHA of the kernel version you're updating
> these to match.
> 

As the commit log states, the changes are not in upstream, only in mail :(



-- 
Alexey