From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35939) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bFYlq-0002xW-J2 for qemu-devel@nongnu.org; Tue, 21 Jun 2016 23:24:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bFYll-0007xp-7j for qemu-devel@nongnu.org; Tue, 21 Jun 2016 23:24:05 -0400 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]:34423) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bFYlk-0007w5-Og for qemu-devel@nongnu.org; Tue, 21 Jun 2016 23:24:01 -0400 Received: by mail-pf0-x243.google.com with SMTP id 66so2782358pfy.1 for ; Tue, 21 Jun 2016 20:23:59 -0700 (PDT) References: <1466471645-5396-1-git-send-email-aik@ozlabs.ru> <1466471645-5396-6-git-send-email-aik@ozlabs.ru> <20160622023523.GJ17957@voom.fritz.box> From: Alexey Kardashevskiy Message-ID: Date: Wed, 22 Jun 2016 13:23:51 +1000 MIME-Version: 1.0 In-Reply-To: <20160622023523.GJ17957@voom.fritz.box> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="NGiDbCfPUbWN7p1MdCQa23U5eVDgUCaDv" Subject: Re: [Qemu-devel] [PATCH qemu v18 5/5] spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: David Gibson Cc: qemu-devel@nongnu.org, qemu-ppc@nongnu.org, Alex Williamson This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --NGiDbCfPUbWN7p1MdCQa23U5eVDgUCaDv From: Alexey Kardashevskiy To: David Gibson Cc: qemu-devel@nongnu.org, qemu-ppc@nongnu.org, Alex Williamson Message-ID: Subject: Re: [PATCH qemu v18 5/5] spapr_pci/spapr_pci_vfio: Support Dynamic DMA Windows (DDW) References: <1466471645-5396-1-git-send-email-aik@ozlabs.ru> <1466471645-5396-6-git-send-email-aik@ozlabs.ru> <20160622023523.GJ17957@voom.fritz.box> In-Reply-To: <20160622023523.GJ17957@voom.fritz.box> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: quoted-printable On 22/06/16 12:35, David Gibson wrote: > On Tue, Jun 21, 2016 at 11:14:05AM +1000, Alexey Kardashevskiy wrote: >> This adds support for Dynamic DMA Windows (DDW) option defined by >> the SPAPR specification which allows to have additional DMA window(s) >> >> The "ddw" property is enabled by default on a PHB but for compatibilit= y >> the pseries-2.6 machine and older disable it. >> This also creates a single DMA window for the older machines to >> maintain backward migration. >> >> This implements DDW for PHB with emulated and VFIO devices. The host >> kernel support is required. The advertised IOMMU page sizes are 4K and= >> 64K; 16M pages are supported but not advertised by default, in order t= o >> enable them, the user has to specify "pgsz" property for PHB and >> enable huge pages for RAM. >> >> The existing linux guests try creating one additional huge DMA window >> with 64K or 16MB pages and map the entire guest RAM to. If succeeded, >> the guest switches to dma_direct_ops and never calls TCE hypercalls >> (H_PUT_TCE,...) again. This enables VFIO devices to use the entire RAM= >> and not waste time on map/unmap later. This adds a "dma64_win_addr" >> property which is a bus address for the 64bit window and by default >> set to 0x800.0000.0000.0000 as this is what the modern POWER8 hardware= >> uses and this allows having emulated and VFIO devices on the same bus.= >> >> This adds 4 RTAS handlers: >> * ibm,query-pe-dma-window >> * ibm,create-pe-dma-window >> * ibm,remove-pe-dma-window >> * ibm,reset-pe-dma-window >> These are registered from type_init() callback. >> >> These RTAS handlers are implemented in a separate file to avoid pollut= ing >> spapr_iommu.c with PCI. >> >> This changes sPAPRPHBState::dma_liobn to an array to allow 2 LIOBNs. >> >> Signed-off-by: Alexey Kardashevskiy >=20 > A few queries below. Not sure if they'll require code changes or just > explanation. >=20 >> --- >> Changes: >> v18: >> * fixed bug when ddw-create rtas call was always creating window at 1<= <59 >> offset >> * update minimum supported machine version >> * s/dma64_window_addr/dma_win_addr/ to match dma_win_addr >> >> v17: >> * fixed: "query" did return non-page-shifted value when memory hotplug= is enabled >> >> v16: >> * s/dma_liobn/dma_liobn[SPAPR_PCI_DMA_MAX_WINDOWS]/ >> * s/SPAPR_PCI_LIOBN()/dma_liobn[]/ >> >> v15: >> * moved page mask filtering to PHB realize(), use "-mempath" to know >> if there are huge pages >> * fixed error reporting in RTAS handlers >> * max window size accounts now hotpluggable memory boundaries >> --- >> hw/ppc/Makefile.objs | 1 + >> hw/ppc/spapr.c | 7 +- >> hw/ppc/spapr_pci.c | 77 +++++++++--- >> hw/ppc/spapr_rtas_ddw.c | 295 +++++++++++++++++++++++++++++++++++= +++++++++ >> include/hw/pci-host/spapr.h | 8 +- >> include/hw/ppc/spapr.h | 16 ++- >> trace-events | 4 + >> 7 files changed, 386 insertions(+), 22 deletions(-) >> create mode 100644 hw/ppc/spapr_rtas_ddw.c >> >> diff --git a/hw/ppc/Makefile.objs b/hw/ppc/Makefile.objs >> index 5cc6608..91a3420 100644 >> --- a/hw/ppc/Makefile.objs >> +++ b/hw/ppc/Makefile.objs >> @@ -8,6 +8,7 @@ obj-$(CONFIG_PSERIES) +=3D spapr_cpu_core.o >> ifeq ($(CONFIG_PCI)$(CONFIG_PSERIES)$(CONFIG_LINUX), yyy) >> obj-y +=3D spapr_pci_vfio.o >> endif >> +obj-$(CONFIG_PSERIES) +=3D spapr_rtas_ddw.o >> # PowerPC 4xx boards >> obj-y +=3D ppc405_boards.o ppc4xx_devs.o ppc405_uc.o ppc440_bamboo.o >> obj-y +=3D ppc4xx_pci.o >> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c >> index 778fa25..f7cff27 100644 >> --- a/hw/ppc/spapr.c >> +++ b/hw/ppc/spapr.c >> @@ -2485,7 +2485,12 @@ DEFINE_SPAPR_MACHINE(2_7, "2.7", true); >> * pseries-2.6 >> */ >> #define SPAPR_COMPAT_2_6 \ >> - HW_COMPAT_2_6 >> + HW_COMPAT_2_6 \ >> + { \ >> + .driver =3D TYPE_SPAPR_PCI_HOST_BRIDGE,\ >> + .property =3D "ddw",\ >> + .value =3D stringify(off),\ >> + }, >> =20 >> static void spapr_machine_2_6_instance_options(MachineState *machine)= >> { >> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c >> index 9f28fb3..0cb51dd 100644 >> --- a/hw/ppc/spapr_pci.c >> +++ b/hw/ppc/spapr_pci.c >> @@ -35,6 +35,7 @@ >> #include "hw/ppc/spapr.h" >> #include "hw/pci-host/spapr.h" >> #include "exec/address-spaces.h" >> +#include "exec/ram_addr.h" >> #include >> #include "trace.h" >> #include "qemu/error-report.h" >> @@ -45,6 +46,7 @@ >> #include "hw/ppc/spapr_drc.h" >> #include "sysemu/device_tree.h" >> #include "sysemu/kvm.h" >> +#include "sysemu/hostmem.h" >> =20 >> #include "hw/vfio/vfio.h" >> =20 >> @@ -1088,7 +1090,7 @@ static void spapr_phb_add_pci_device(sPAPRDRConn= ector *drc, >> int fdt_start_offset =3D 0, fdt_size; >> =20 >> if (object_dynamic_cast(OBJECT(pdev), "vfio-pci")) { >> - sPAPRTCETable *tcet =3D spapr_tce_find_by_liobn(phb->dma_liob= n); >> + sPAPRTCETable *tcet =3D spapr_tce_find_by_liobn(phb->dma_liob= n[0]); >> =20 >> spapr_tce_set_need_vfio(tcet, true); >=20 > Now that Alex took your notifier on/off patches, can you remove this > chunk?=20 It will stop compiling as dma_liobn is an array now. > If it's still necessary, don't you need to loop over all the > possible liobns, rather than just acting on liobn[0]? Ah, right. Forgot about it. That was the reason why I wanted those notifi= er callbacks in this series, lost it in respins. I do need a loop here which= I'll have to remove soon though. >=20 >> } >> @@ -1310,11 +1312,14 @@ static void spapr_phb_realize(DeviceState *dev= , Error **errp) >> PCIBus *bus; >> uint64_t msi_window_size =3D 4096; >> sPAPRTCETable *tcet; >> + const unsigned windows_supported =3D >> + sphb->ddw_enabled ? SPAPR_PCI_DMA_MAX_WINDOWS : 1; >> =20 >> if (sphb->index !=3D (uint32_t)-1) { >> hwaddr windows_base; >> =20 >> - if ((sphb->buid !=3D (uint64_t)-1) || (sphb->dma_liobn !=3D (= uint32_t)-1) >> + if ((sphb->buid !=3D (uint64_t)-1) || (sphb->dma_liobn[0] !=3D= (uint32_t)-1) >> + || (sphb->dma_liobn[1] !=3D (uint32_t)-1 && windows_suppo= rted =3D=3D 2) >> || (sphb->mem_win_addr !=3D (hwaddr)-1) >> || (sphb->io_win_addr !=3D (hwaddr)-1)) { >> error_setg(errp, "Either \"index\" or other parameters mu= st" >> @@ -1329,7 +1334,9 @@ static void spapr_phb_realize(DeviceState *dev, = Error **errp) >> } >> =20 >> sphb->buid =3D SPAPR_PCI_BASE_BUID + sphb->index; >> - sphb->dma_liobn =3D SPAPR_PCI_LIOBN(sphb->index, 0); >> + for (i =3D 0; i < windows_supported; ++i) { >> + sphb->dma_liobn[i] =3D SPAPR_PCI_LIOBN(sphb->index, i); >> + } >> =20 >> windows_base =3D SPAPR_PCI_WINDOW_BASE >> + sphb->index * SPAPR_PCI_WINDOW_SPACING; >> @@ -1342,8 +1349,9 @@ static void spapr_phb_realize(DeviceState *dev, = Error **errp) >> return; >> } >> =20 >> - if (sphb->dma_liobn =3D=3D (uint32_t)-1) { >> - error_setg(errp, "LIOBN not specified for PHB"); >> + if ((sphb->dma_liobn[0] =3D=3D (uint32_t)-1) || >> + ((sphb->dma_liobn[1] =3D=3D (uint32_t)-1) && (windows_support= ed > 1))) { >> + error_setg(errp, "LIOBN(s) not specified for PHB"); >> return; >> } >> =20 >> @@ -1461,16 +1469,18 @@ static void spapr_phb_realize(DeviceState *dev= , Error **errp) >> } >> } >> =20 >> - tcet =3D spapr_tce_new_table(DEVICE(sphb), sphb->dma_liobn); >> - if (!tcet) { >> - error_setg(errp, "Unable to create TCE table for %s", >> - sphb->dtbusname); >> - return; >> + /* DMA setup */ >> + for (i =3D 0; i < windows_supported; ++i) { >> + tcet =3D spapr_tce_new_table(DEVICE(sphb), sphb->dma_liobn[i]= ); >> + if (!tcet) { >> + error_setg(errp, "Creating window#%d failed for %s", >> + i, sphb->dtbusname); >> + return; >> + } >> + memory_region_add_subregion_overlap(&sphb->iommu_root, 0, >> + spapr_tce_get_iommu(tcet)= , 0); >> } >> =20 >> - memory_region_add_subregion_overlap(&sphb->iommu_root, 0, >> - spapr_tce_get_iommu(tcet), 0)= ; >> - >> sphb->msi =3D g_hash_table_new_full(g_int_hash, g_int_equal, g_fr= ee, g_free); >> } >> =20 >> @@ -1487,13 +1497,19 @@ static int spapr_phb_children_reset(Object *ch= ild, void *opaque) >> =20 >> void spapr_phb_dma_reset(sPAPRPHBState *sphb) >> { >> - sPAPRTCETable *tcet =3D spapr_tce_find_by_liobn(sphb->dma_liobn);= >> + int i; >> + sPAPRTCETable *tcet; >> =20 >> - if (tcet && tcet->nb_table) { >> - spapr_tce_table_disable(tcet); >> + for (i =3D 0; i < SPAPR_PCI_DMA_MAX_WINDOWS; ++i) { >> + tcet =3D spapr_tce_find_by_liobn(sphb->dma_liobn[i]); >> + >> + if (tcet && tcet->nb_table) { >> + spapr_tce_table_disable(tcet); >> + } >> } >> =20 >> /* Register default 32bit DMA window */ >> + tcet =3D spapr_tce_find_by_liobn(sphb->dma_liobn[0]); >> spapr_tce_table_enable(tcet, SPAPR_TCE_PAGE_SHIFT, sphb->dma_win_= addr, >> sphb->dma_win_size >> SPAPR_TCE_PAGE_SHIFT= ); >> } >> @@ -1515,7 +1531,8 @@ static void spapr_phb_reset(DeviceState *qdev) >> static Property spapr_phb_properties[] =3D { >> DEFINE_PROP_UINT32("index", sPAPRPHBState, index, -1), >> DEFINE_PROP_UINT64("buid", sPAPRPHBState, buid, -1), >> - DEFINE_PROP_UINT32("liobn", sPAPRPHBState, dma_liobn, -1), >> + DEFINE_PROP_UINT32("liobn", sPAPRPHBState, dma_liobn[0], -1), >> + DEFINE_PROP_UINT32("liobn64", sPAPRPHBState, dma_liobn[1], -1), >> DEFINE_PROP_UINT64("mem_win_addr", sPAPRPHBState, mem_win_addr, -= 1), >> DEFINE_PROP_UINT64("mem_win_size", sPAPRPHBState, mem_win_size, >> SPAPR_PCI_MMIO_WIN_SIZE), >> @@ -1527,6 +1544,11 @@ static Property spapr_phb_properties[] =3D { >> /* Default DMA window is 0..1GB */ >> DEFINE_PROP_UINT64("dma_win_addr", sPAPRPHBState, dma_win_addr, 0= ), >> DEFINE_PROP_UINT64("dma_win_size", sPAPRPHBState, dma_win_size, 0= x40000000), >> + DEFINE_PROP_UINT64("dma64_win_addr", sPAPRPHBState, dma64_win_add= r, >> + 0x800000000000000ULL), >> + DEFINE_PROP_BOOL("ddw", sPAPRPHBState, ddw_enabled, true), >> + DEFINE_PROP_UINT64("pgsz", sPAPRPHBState, page_size_mask, >> + (1ULL << 12) | (1ULL << 16)), >> DEFINE_PROP_END_OF_LIST(), >> }; >> =20 >> @@ -1603,7 +1625,7 @@ static const VMStateDescription vmstate_spapr_pc= i =3D { >> .post_load =3D spapr_pci_post_load, >> .fields =3D (VMStateField[]) { >> VMSTATE_UINT64_EQUAL(buid, sPAPRPHBState), >> - VMSTATE_UINT32_EQUAL(dma_liobn, sPAPRPHBState), >> + VMSTATE_UNUSED(4), /* dma_liobn */ >=20 > It's not obvious to me why this change is necessary. It is not. But I was touching liobn and this is a proper cleanup which needs to be done anyway as _EQUAL() macros are sort of deprecated and rather pointless. Since I am adding a new 64bit LIOBN in this patch, shou= ld I add it in VMSTATE as 32bit one and bump the vmstate version? Or not add= it (leaving some inconsistency)? >=20 >> VMSTATE_UINT64_EQUAL(mem_win_addr, sPAPRPHBState), >> VMSTATE_UINT64_EQUAL(mem_win_size, sPAPRPHBState), >> VMSTATE_UINT64_EQUAL(io_win_addr, sPAPRPHBState), >> @@ -1779,6 +1801,15 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, >> uint32_t interrupt_map_mask[] =3D { >> cpu_to_be32(b_ddddd(-1)|b_fff(0)), 0x0, 0x0, cpu_to_be32(-1)}= ; >> uint32_t interrupt_map[PCI_SLOT_MAX * PCI_NUM_PINS][7]; >> + uint32_t ddw_applicable[] =3D { >> + cpu_to_be32(RTAS_IBM_QUERY_PE_DMA_WINDOW), >> + cpu_to_be32(RTAS_IBM_CREATE_PE_DMA_WINDOW), >> + cpu_to_be32(RTAS_IBM_REMOVE_PE_DMA_WINDOW) >> + }; >> + uint32_t ddw_extensions[] =3D { >> + cpu_to_be32(1), >> + cpu_to_be32(RTAS_IBM_RESET_PE_DMA_WINDOW) >> + }; >> sPAPRTCETable *tcet; >> PCIBus *bus =3D PCI_HOST_BRIDGE(phb)->bus; >> sPAPRFDT s_fdt; >> @@ -1803,6 +1834,14 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, >> _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", = 0x1)); >> _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQ= S)); >> =20 >> + /* Dynamic DMA window */ >> + if (phb->ddw_enabled) { >> + _FDT(fdt_setprop(fdt, bus_off, "ibm,ddw-applicable", &ddw_app= licable, >> + sizeof(ddw_applicable))); >> + _FDT(fdt_setprop(fdt, bus_off, "ibm,ddw-extensions", >> + &ddw_extensions, sizeof(ddw_extensions))); >> + } >> + >> /* Build the interrupt-map, this must matches what is done >> * in pci_spapr_map_irq >> */ >> @@ -1826,7 +1865,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb, >> _FDT(fdt_setprop(fdt, bus_off, "interrupt-map", &interrupt_map, >> sizeof(interrupt_map))); >> =20 >> - tcet =3D spapr_tce_find_by_liobn(phb->dma_liobn); >> + tcet =3D spapr_tce_find_by_liobn(phb->dma_liobn[0]); >> if (!tcet) { >> return -1; >> } >> diff --git a/hw/ppc/spapr_rtas_ddw.c b/hw/ppc/spapr_rtas_ddw.c >> new file mode 100644 >> index 0000000..177dcff >> --- /dev/null >> +++ b/hw/ppc/spapr_rtas_ddw.c >> @@ -0,0 +1,295 @@ >> +/* >> + * QEMU sPAPR Dynamic DMA windows support >> + * >> + * Copyright (c) 2015 Alexey Kardashevskiy, IBM Corporation. >> + * >> + * This program is free software; you can redistribute it and/or mod= ify >> + * it under the terms of the GNU General Public License as published= by >> + * the Free Software Foundation; either version 2 of the License, >> + * or (at your option) any later version. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License= >> + * along with this program; if not, see . >> + */ >> + >> +#include "qemu/osdep.h" >> +#include "cpu.h" >> +#include "qemu/error-report.h" >> +#include "hw/ppc/spapr.h" >> +#include "hw/pci-host/spapr.h" >> +#include "trace.h" >> + >> +static int spapr_phb_get_active_win_num_cb(Object *child, void *opaqu= e) >> +{ >> + sPAPRTCETable *tcet; >> + >> + tcet =3D (sPAPRTCETable *) object_dynamic_cast(child, TYPE_SPAPR_= TCE_TABLE); >> + if (tcet && tcet->nb_table) { >> + ++*(unsigned *)opaque; >> + } >> + return 0; >> +} >> + >> +static unsigned spapr_phb_get_active_win_num(sPAPRPHBState *sphb) >> +{ >> + unsigned ret =3D 0; >> + >> + object_child_foreach(OBJECT(sphb), spapr_phb_get_active_win_num_c= b, &ret); >> + >> + return ret; >> +} >> + >> +static int spapr_phb_get_free_liobn_cb(Object *child, void *opaque) >> +{ >> + sPAPRTCETable *tcet; >> + >> + tcet =3D (sPAPRTCETable *) object_dynamic_cast(child, TYPE_SPAPR_= TCE_TABLE); >> + if (tcet && !tcet->nb_table) { >> + *(uint32_t *)opaque =3D tcet->liobn; >> + return 1; >> + } >> + return 0; >> +} >> + >> +static unsigned spapr_phb_get_free_liobn(sPAPRPHBState *sphb) >> +{ >> + uint32_t liobn =3D 0; >> + >> + object_child_foreach(OBJECT(sphb), spapr_phb_get_free_liobn_cb, &= liobn); >> + >> + return liobn; >> +} >> + >> +static uint32_t spapr_page_mask_to_query_mask(uint64_t page_mask) >> +{ >> + int i; >> + uint32_t mask =3D 0; >> + const struct { int shift; uint32_t mask; } masks[] =3D { >> + { 12, RTAS_DDW_PGSIZE_4K }, >> + { 16, RTAS_DDW_PGSIZE_64K }, >> + { 24, RTAS_DDW_PGSIZE_16M }, >> + { 25, RTAS_DDW_PGSIZE_32M }, >> + { 26, RTAS_DDW_PGSIZE_64M }, >> + { 27, RTAS_DDW_PGSIZE_128M }, >> + { 28, RTAS_DDW_PGSIZE_256M }, >> + { 34, RTAS_DDW_PGSIZE_16G }, >> + }; >> + >> + for (i =3D 0; i < ARRAY_SIZE(masks); ++i) { >> + if (page_mask & (1ULL << masks[i].shift)) { >> + mask |=3D masks[i].mask; >> + } >> + } >> + >> + return mask; >> +} >> + >> +static void rtas_ibm_query_pe_dma_window(PowerPCCPU *cpu, >> + sPAPRMachineState *spapr, >> + uint32_t token, uint32_t nar= gs, >> + target_ulong args, >> + uint32_t nret, target_ulong = rets) >> +{ >> + sPAPRPHBState *sphb; >> + uint64_t buid, max_window_size; >> + uint32_t avail, addr, pgmask =3D 0; >> + MachineState *machine =3D MACHINE(spapr); >> + >> + if ((nargs !=3D 3) || (nret !=3D 5)) { >> + goto param_error_exit; >> + } >> + >> + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); >> + addr =3D rtas_ld(args, 0); >> + sphb =3D spapr_pci_find_phb(spapr, buid); >> + if (!sphb || !sphb->ddw_enabled) { >> + goto param_error_exit; >> + } >> + >> + /* Translate page mask to LoPAPR format */ >> + pgmask =3D spapr_page_mask_to_query_mask(sphb->page_size_mask); >> + >> + /* >> + * This is "Largest contiguous block of TCEs allocated specifical= ly >> + * for (that is, are reserved for) this PE". >> + * Return the maximum number as maximum supported RAM size was in= 4K pages. >> + */ >> + if (machine->ram_size =3D=3D machine->maxram_size) { >> + max_window_size =3D machine->ram_size; >> + } else { >> + MemoryHotplugState *hpms =3D &spapr->hotplug_memory; >> + >> + max_window_size =3D hpms->base + memory_region_size(&hpms->mr= ); >> + } >> + >> + avail =3D SPAPR_PCI_DMA_MAX_WINDOWS - spapr_phb_get_active_win_nu= m(sphb); >> + >> + rtas_st(rets, 0, RTAS_OUT_SUCCESS); >> + rtas_st(rets, 1, avail); >> + rtas_st(rets, 2, max_window_size >> SPAPR_TCE_PAGE_SHIFT); >> + rtas_st(rets, 3, pgmask); >> + rtas_st(rets, 4, 0); /* DMA migration mask, not supported */ >> + >> + trace_spapr_iommu_ddw_query(buid, addr, avail, max_window_size, p= gmask); >> + return; >> + >> +param_error_exit: >> + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); >> +} >> + >> +static void rtas_ibm_create_pe_dma_window(PowerPCCPU *cpu, >> + sPAPRMachineState *spapr, >> + uint32_t token, uint32_t na= rgs, >> + target_ulong args, >> + uint32_t nret, target_ulong= rets) >> +{ >> + sPAPRPHBState *sphb; >> + sPAPRTCETable *tcet =3D NULL; >> + uint32_t addr, page_shift, window_shift, liobn; >> + uint64_t buid, win_addr; >> + int windows; >> + >> + if ((nargs !=3D 5) || (nret !=3D 4)) { >> + goto param_error_exit; >> + } >> + >> + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); >> + addr =3D rtas_ld(args, 0); >> + sphb =3D spapr_pci_find_phb(spapr, buid); >> + if (!sphb || !sphb->ddw_enabled) { >> + goto param_error_exit; >> + } >> + >> + page_shift =3D rtas_ld(args, 3); >> + window_shift =3D rtas_ld(args, 4); >> + liobn =3D spapr_phb_get_free_liobn(sphb); >> + windows =3D spapr_phb_get_active_win_num(sphb); >> + >> + if (!(sphb->page_size_mask & (1ULL << page_shift)) || >> + (window_shift < page_shift)) { >> + goto param_error_exit; >> + } >> + >> + if (!liobn || !sphb->ddw_enabled || windows =3D=3D SPAPR_PCI_DMA_= MAX_WINDOWS) { >> + goto hw_error_exit; >> + } >> + >> + tcet =3D spapr_tce_find_by_liobn(liobn); >> + if (!tcet) { >> + goto hw_error_exit; >> + } >> + >> + win_addr =3D (windows =3D=3D 0) ? sphb->dma_win_addr : sphb->dma6= 4_win_addr; >=20 > If the guest delets the default 32-bit window, then requests a really > big 64-bit DMA window, will that work ok with the big window at 0 > instead of the usual 64-bit window address? There is no valid guest to try that as they keep 32bit window. There was a relatively short period of time in v3.0-ish era (sles11 did have it and sles11sp3 did not if I remember correctly) when the guest wou= ld remove all windows and create one huge window but for some reason it expected the window to start non from zero (perhaps pHyp implementation detail) so it would fail. I did an experiment and removed that particular= check and it worked just fine. Today guests always keep a 32bit window as the platform cannot tell if al= l the drivers on a specific PHB will request 64bit DMA. >> + spapr_tce_table_enable(tcet, page_shift, win_addr, >> + 1ULL << (window_shift - page_shift)); >> + if (!tcet->nb_table) { >> + goto hw_error_exit; >> + } >> + >> + trace_spapr_iommu_ddw_create(buid, addr, 1ULL << page_shift, >> + 1ULL << window_shift, tcet->bus_offs= et, liobn); >> + >> + rtas_st(rets, 0, RTAS_OUT_SUCCESS); >> + rtas_st(rets, 1, liobn); >> + rtas_st(rets, 2, tcet->bus_offset >> 32); >> + rtas_st(rets, 3, tcet->bus_offset & ((uint32_t) -1)); >> + >> + return; >> + >> +hw_error_exit: >> + rtas_st(rets, 0, RTAS_OUT_HW_ERROR); >> + return; >> + >> +param_error_exit: >> + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); >> +} >> + >> +static void rtas_ibm_remove_pe_dma_window(PowerPCCPU *cpu, >> + sPAPRMachineState *spapr, >> + uint32_t token, uint32_t na= rgs, >> + target_ulong args, >> + uint32_t nret, target_ulong= rets) >> +{ >> + sPAPRPHBState *sphb; >> + sPAPRTCETable *tcet; >> + uint32_t liobn; >> + >> + if ((nargs !=3D 1) || (nret !=3D 1)) { >> + goto param_error_exit; >> + } >> + >> + liobn =3D rtas_ld(args, 0); >> + tcet =3D spapr_tce_find_by_liobn(liobn); >> + if (!tcet) { >> + goto param_error_exit; >> + } >> + >> + sphb =3D SPAPR_PCI_HOST_BRIDGE(OBJECT(tcet)->parent); >> + if (!sphb || !sphb->ddw_enabled || !tcet->nb_table) { >> + goto param_error_exit; >> + } >> + >> + spapr_tce_table_disable(tcet); >> + trace_spapr_iommu_ddw_remove(liobn); >> + >> + rtas_st(rets, 0, RTAS_OUT_SUCCESS); >> + return; >> + >> +param_error_exit: >> + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); >> +} >> + >> +static void rtas_ibm_reset_pe_dma_window(PowerPCCPU *cpu, >> + sPAPRMachineState *spapr, >> + uint32_t token, uint32_t nar= gs, >> + target_ulong args, >> + uint32_t nret, target_ulong = rets) >> +{ >> + sPAPRPHBState *sphb; >> + uint64_t buid; >> + uint32_t addr; >> + >> + if ((nargs !=3D 3) || (nret !=3D 1)) { >> + goto param_error_exit; >> + } >> + >> + buid =3D ((uint64_t)rtas_ld(args, 1) << 32) | rtas_ld(args, 2); >> + addr =3D rtas_ld(args, 0); >> + sphb =3D spapr_pci_find_phb(spapr, buid); >> + if (!sphb || !sphb->ddw_enabled) { >> + goto param_error_exit; >> + } >> + >> + spapr_phb_dma_reset(sphb); >> + trace_spapr_iommu_ddw_reset(buid, addr); >> + >> + rtas_st(rets, 0, RTAS_OUT_SUCCESS); >> + >> + return; >> + >> +param_error_exit: >> + rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); >> +} >> + >> +static void spapr_rtas_ddw_init(void) >> +{ >> + spapr_rtas_register(RTAS_IBM_QUERY_PE_DMA_WINDOW, >> + "ibm,query-pe-dma-window", >> + rtas_ibm_query_pe_dma_window); >> + spapr_rtas_register(RTAS_IBM_CREATE_PE_DMA_WINDOW, >> + "ibm,create-pe-dma-window", >> + rtas_ibm_create_pe_dma_window); >> + spapr_rtas_register(RTAS_IBM_REMOVE_PE_DMA_WINDOW, >> + "ibm,remove-pe-dma-window", >> + rtas_ibm_remove_pe_dma_window); >> + spapr_rtas_register(RTAS_IBM_RESET_PE_DMA_WINDOW, >> + "ibm,reset-pe-dma-window", >> + rtas_ibm_reset_pe_dma_window); >> +} >> + >> +type_init(spapr_rtas_ddw_init) >> diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h= >> index 7848366..92aa610 100644 >> --- a/include/hw/pci-host/spapr.h >> +++ b/include/hw/pci-host/spapr.h >> @@ -32,6 +32,8 @@ >> #define SPAPR_PCI_HOST_BRIDGE(obj) \ >> OBJECT_CHECK(sPAPRPHBState, (obj), TYPE_SPAPR_PCI_HOST_BRIDGE) >> =20 >> +#define SPAPR_PCI_DMA_MAX_WINDOWS 2 >> + >> typedef struct sPAPRPHBState sPAPRPHBState; >> =20 >> typedef struct spapr_pci_msi { >> @@ -56,7 +58,7 @@ struct sPAPRPHBState { >> hwaddr mem_win_addr, mem_win_size, io_win_addr, io_win_size; >> MemoryRegion memwindow, iowindow, msiwindow; >> =20 >> - uint32_t dma_liobn; >> + uint32_t dma_liobn[SPAPR_PCI_DMA_MAX_WINDOWS]; >> hwaddr dma_win_addr, dma_win_size; >> AddressSpace iommu_as; >> MemoryRegion iommu_root; >> @@ -71,6 +73,10 @@ struct sPAPRPHBState { >> spapr_pci_msi_mig *msi_devs; >> =20 >> QLIST_ENTRY(sPAPRPHBState) list; >> + >> + bool ddw_enabled; >> + uint64_t page_size_mask; >> + uint64_t dma64_win_addr; >> }; >> =20 >> #define SPAPR_PCI_MAX_INDEX 255 >> diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h >> index e1f8274..36d1748 100644 >> --- a/include/hw/ppc/spapr.h >> +++ b/include/hw/ppc/spapr.h >> @@ -416,6 +416,16 @@ int spapr_allocate_irq_block(int num, bool lsi, b= ool msi); >> #define RTAS_OUT_NOT_AUTHORIZED -9002 >> #define RTAS_OUT_SYSPARM_PARAM_ERROR -9999 >> =20 >> +/* DDW pagesize mask values from ibm,query-pe-dma-window */ >> +#define RTAS_DDW_PGSIZE_4K 0x01 >> +#define RTAS_DDW_PGSIZE_64K 0x02 >> +#define RTAS_DDW_PGSIZE_16M 0x04 >> +#define RTAS_DDW_PGSIZE_32M 0x08 >> +#define RTAS_DDW_PGSIZE_64M 0x10 >> +#define RTAS_DDW_PGSIZE_128M 0x20 >> +#define RTAS_DDW_PGSIZE_256M 0x40 >> +#define RTAS_DDW_PGSIZE_16G 0x80 >> + >> /* RTAS tokens */ >> #define RTAS_TOKEN_BASE 0x2000 >> =20 >> @@ -457,8 +467,12 @@ int spapr_allocate_irq_block(int num, bool lsi, b= ool msi); >> #define RTAS_IBM_SET_SLOT_RESET (RTAS_TOKEN_BASE + 0x= 23) >> #define RTAS_IBM_CONFIGURE_PE (RTAS_TOKEN_BASE + 0x= 24) >> #define RTAS_IBM_SLOT_ERROR_DETAIL (RTAS_TOKEN_BASE + 0x= 25) >> +#define RTAS_IBM_QUERY_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x= 26) >> +#define RTAS_IBM_CREATE_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x= 27) >> +#define RTAS_IBM_REMOVE_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x= 28) >> +#define RTAS_IBM_RESET_PE_DMA_WINDOW (RTAS_TOKEN_BASE + 0x= 29) >> =20 >> -#define RTAS_TOKEN_MAX (RTAS_TOKEN_BASE + 0x= 26) >> +#define RTAS_TOKEN_MAX (RTAS_TOKEN_BASE + 0x= 2A) >> =20 >> /* RTAS ibm,get-system-parameter token values */ >> #define RTAS_SYSPARM_SPLPAR_CHARACTERISTICS 20 >> diff --git a/trace-events b/trace-events >> index 7e94d92..5b52634 100644 >> --- a/trace-events >> +++ b/trace-events >> @@ -1435,6 +1435,10 @@ spapr_iommu_xlate(uint64_t liobn, uint64_t ioba= , uint64_t tce, unsigned perm, un >> spapr_iommu_new_table(uint64_t liobn, void *table, int fd) "liobn=3D%= "PRIx64" table=3D%p fd=3D%d" >> spapr_iommu_pre_save(uint64_t liobn, uint32_t nb, uint64_t offs, uint= 32_t ps) "liobn=3D%"PRIx64" %"PRIx32" bus_offset=3D%"PRIx64" ps=3D%"PRIu3= 2 >> spapr_iommu_post_load(uint64_t liobn, uint32_t pre_nb, uint32_t post_= nb, uint64_t offs, uint32_t ps) "liobn=3D%"PRIx64" %"PRIx32" =3D> %"PRIx3= 2" bus_offset=3D%"PRIx64" ps=3D%"PRIu32 >> +spapr_iommu_ddw_query(uint64_t buid, uint32_t cfgaddr, unsigned wa, u= int64_t win_size, uint32_t pgmask) "buid=3D%"PRIx64" addr=3D%"PRIx32", %u= windows available, max window size=3D%"PRIx64", mask=3D%"PRIx32 >> +spapr_iommu_ddw_create(uint64_t buid, uint32_t cfgaddr, uint64_t pg_s= ize, uint64_t req_size, uint64_t start, uint32_t liobn) "buid=3D%"PRIx64"= addr=3D%"PRIx32", page size=3D0x%"PRIx64", requested=3D0x%"PRIx64", star= t addr=3D%"PRIx64", liobn=3D%"PRIx32 >> +spapr_iommu_ddw_remove(uint32_t liobn) "liobn=3D%"PRIx32 >> +spapr_iommu_ddw_reset(uint64_t buid, uint32_t cfgaddr) "buid=3D%"PRIx= 64" addr=3D%"PRIx32 >> =20 >> # hw/ppc/ppc.c >> ppc_tb_adjust(uint64_t offs1, uint64_t offs2, int64_t diff, int64_t s= econds) "adjusted from 0x%"PRIx64" to 0x%"PRIx64", diff %"PRId64" (%"PRId= 64"s)" >=20 --=20 Alexey --NGiDbCfPUbWN7p1MdCQa23U5eVDgUCaDv Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXagTHAAoJEIYTPdgrwSC5w5QP/jWWtlqwWfGmRlt6eNEOsvnX KxhCBuumhNpJd7RVt1/IoJLLcH5ZHBxtNYP9joHPTCxzvj0xQH3RiDE9u4cUhuHB s1WCO/6qGTvs/81e97yCb2AS841Cl9YrVaXj3NZFf1TmdcUBhemJpVJN1e1sVeLj hzYHFIe5lH8kXV7eBP5SDZ4KHCtbrjK7dbHCzvYpKpfNUkBF2+qmhS9c+z76rGbB XqsaW4mqXnDbqlmYTDR3HRopIZZxtxsyaAPEdA3rwZ7U/XJaZMb6lbxT/p7hYXxm kX/E9IVgLxcJqWi8DNqSFEubtyLVOyfL7O1lqsQ/IeD9j9EvbCnePJp7LGiQOnj2 5in1j7FfbV/PwVg0jKqlMV4wE43zkHpVL4g1vPEWFrcauqWBkajNqTTTtm/IYmrO Mz6myvgmXBSSg6HZZw1Mpo9EwgMBU8g8RxfvrYhF4OklYH91hdQfYm1hJVeGoQKt rjW/5XFr5eq7HpU1UUh25jd6OEAQ+PvqydKEPXW/9dv+77+KuHlzOXo5WO5JShpY f+i8y2S3KC+aQMvnAB9FWr89BoBCJN4y7pHOB2xxQVGTLr7WV5De9VtylnyjNhlL EnhpHd6Jhdd09jow3FWcNa/6XPHsnCFWyuKs2IIzoRcc1GK481KE9+c62EmQBoxw 8vPBmHbCdPbNlmvMlyWA =j1DW -----END PGP SIGNATURE----- --NGiDbCfPUbWN7p1MdCQa23U5eVDgUCaDv--