* [PATCH v3 0/2] VMD endpoint passthrough support
@ 2020-05-28 3:02 Jon Derrick
2020-05-28 3:02 ` [PATCH v3 FOR QEMU v3] hw/vfio: Add VMD Passthrough Quirk Jon Derrick
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Jon Derrick @ 2020-05-28 3:02 UTC (permalink / raw)
To: linux-pci, qemu-devel
Cc: Lorenzo Pieralisi, Andrzej Jakowski, virtualization,
Alex Williamson, Bjorn Helgaas, Christoph Hellwig, Jon Derrick
This set contains 2 patches for Linux and 1 for QEMU. VMD device
8086:28C0 contains information in registers to assist with direct
assignment passthrough. Several other VMD devices don't have this
information, but hypervisors can easily provide the guest with this
information through various means.
The QEMU patch provides the information in an emulated vendor-specific
PCI capability. Existing VMD devices don't conflict with the offset
chosen for the capability.
The Linux patch allows guest kernels to use the passthrough information
emulated by the QEMU patch, by matching against the vendor-specific PCI
capability if it exists.
V2 Ref:
https://lore.kernel.org/linux-pci/20200511190129.9313-1-jonathan.derrick@intel.com/
Changes from v2:
Uses vendor-specific PCI capability rather than emulating the 28C0
MEMBAR/VMLOCK modes.
Changes from v1:
v1 changed the VMD Subsystem ID to QEMU's so that the guest driver could
match against it. This was unnecessary as the VMLOCK register and shadow
membar registers could be safely emulated. Future VMDs will be aligned
on these register bits.
Jon Derrick (2):
PCI: vmd: Filter resource type bits from shadow register
PCI: vmd: Use Shadow MEMBAR registers for QEMU/KVM guests
drivers/pci/controller/vmd.c | 50 +++++++++++++++++++++++++++++++++++++-------
1 file changed, 42 insertions(+), 8 deletions(-)
--
1.8.3.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [PATCH v3 FOR QEMU v3] hw/vfio: Add VMD Passthrough Quirk 2020-05-28 3:02 [PATCH v3 0/2] VMD endpoint passthrough support Jon Derrick @ 2020-05-28 3:02 ` Jon Derrick 2020-05-28 3:02 ` [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register Jon Derrick 2020-05-28 3:02 ` [PATCH v3 2/2] PCI: vmd: Use Shadow MEMBAR registers for QEMU/KVM guests Jon Derrick 2 siblings, 0 replies; 11+ messages in thread From: Jon Derrick @ 2020-05-28 3:02 UTC (permalink / raw) To: linux-pci, qemu-devel Cc: Lorenzo Pieralisi, Andrzej Jakowski, virtualization, Alex Williamson, Bjorn Helgaas, Christoph Hellwig, Jon Derrick The VMD endpoint provides a real PCIe domain to the guest, including bridges and endpoints. Because the VMD domain is enumerated by the guest kernel, the guest kernel will assign Guest Physical Addresses to the downstream endpoint BARs and bridge windows. When the guest kernel performs MMIO to VMD sub-devices, MMU will translate from the guest address space to the physical address space. Because the bridges have been programmed with guest addresses, the bridges will reject the transaction containing physical addresses. VMD device 28C0 natively assists passthrough by providing the Host Physical Address in shadow registers accessible to the guest for bridge window assignment. The shadow registers are valid if bit 1 is set in VMD VMLOCK config register 0x70. In order to support existing VMDs, this quirk provides the shadow registers in a vendor-specific PCI capability to the vfio-passthrough device for all VMD device ids which don't natively assist with passthrough. The Linux VMD driver is updated to check for this new vendor-specific capability. Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> --- hw/vfio/pci-quirks.c | 84 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 72 insertions(+), 12 deletions(-) diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index 3bd05fed12..6b8c1edfd5 100644 --- a/hw/vfio/pci-quirks.c +++ b/hw/vfio/pci-quirks.c @@ -1567,18 +1567,6 @@ static int vfio_add_nv_gpudirect_cap(VFIOPCIDevice *vdev, Error **errp) return 0; } -int vfio_add_virt_caps(VFIOPCIDevice *vdev, Error **errp) -{ - int ret; - - ret = vfio_add_nv_gpudirect_cap(vdev, errp); - if (ret) { - return ret; - } - - return 0; -} - static void vfio_pci_nvlink2_get_tgt(Object *obj, Visitor *v, const char *name, void *opaque, Error **errp) @@ -1709,3 +1697,75 @@ free_exit: return ret; } + +/* + * The VMD endpoint provides a real PCIe domain to the guest and the guest + * kernel performs enumeration of the VMD sub-device domain. Guest transactions + * to VMD sub-devices go through MMU translation from guest addresses to + * physical addresses. When MMIO goes to an endpoint after being translated to + * physical addresses, the bridge rejects the transaction because the window + * has been programmed with guest addresses. + * + * VMD can use the Host Physical Address in order to correctly program the + * bridge windows in its PCIe domain. VMD device 28C0 has HPA shadow registers + * located at offset 0x2000 in MEMBAR2 (BAR 4). This quirk provides the HPA + * shadow registers in a vendor-specific capability register for devices + * without native support. The position of 0xE8-0xFF is in the reserved range + * of the VMD device capability space following the Power Management + * Capability. + */ +#define VMD_SHADOW_CAP_VER 1 +#define VMD_SHADOW_CAP_LEN 24 +static int vfio_add_vmd_shadow_cap(VFIOPCIDevice *vdev, Error **errp) +{ + uint8_t membar_phys[16]; + int ret, pos = 0xE8; + + if (!(vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, 0x201D) || + vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, 0x467F) || + vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, 0x4C3D) || + vfio_pci_is(vdev, PCI_VENDOR_ID_INTEL, 0x9A0B))) { + return 0; + } + + ret = pread(vdev->vbasedev.fd, membar_phys, 16, + vdev->config_offset + PCI_BASE_ADDRESS_2); + if (ret != 16) { + error_report("VMD %s cannot read MEMBARs (%d)", + vdev->vbasedev.name, ret); + return -EFAULT; + } + + ret = pci_add_capability(&vdev->pdev, PCI_CAP_ID_VNDR, pos, + VMD_SHADOW_CAP_LEN, errp); + if (ret < 0) { + error_prepend(errp, "Failed to add VMD MEMBAR Shadow cap: "); + return ret; + } + + memset(vdev->emulated_config_bits + pos, 0xFF, VMD_SHADOW_CAP_LEN); + pos += PCI_CAP_FLAGS; + pci_set_byte(vdev->pdev.config + pos++, VMD_SHADOW_CAP_LEN); + pci_set_byte(vdev->pdev.config + pos++, VMD_SHADOW_CAP_VER); + pci_set_long(vdev->pdev.config + pos, 0x53484457); /* SHDW */ + memcpy(vdev->pdev.config + pos + 4, membar_phys, 16); + + return 0; +} + +int vfio_add_virt_caps(VFIOPCIDevice *vdev, Error **errp) +{ + int ret; + + ret = vfio_add_nv_gpudirect_cap(vdev, errp); + if (ret) { + return ret; + } + + ret = vfio_add_vmd_shadow_cap(vdev, errp); + if (ret) { + return ret; + } + + return 0; +} -- 2.18.1 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register 2020-05-28 3:02 [PATCH v3 0/2] VMD endpoint passthrough support Jon Derrick 2020-05-28 3:02 ` [PATCH v3 FOR QEMU v3] hw/vfio: Add VMD Passthrough Quirk Jon Derrick @ 2020-05-28 3:02 ` Jon Derrick 2020-05-29 10:33 ` Lorenzo Pieralisi 2020-05-28 3:02 ` [PATCH v3 2/2] PCI: vmd: Use Shadow MEMBAR registers for QEMU/KVM guests Jon Derrick 2 siblings, 1 reply; 11+ messages in thread From: Jon Derrick @ 2020-05-28 3:02 UTC (permalink / raw) To: linux-pci, qemu-devel Cc: Lorenzo Pieralisi, Andrzej Jakowski, virtualization, Alex Williamson, Bjorn Helgaas, Christoph Hellwig, Jon Derrick Versions of VMD with the Host Physical Address shadow register use this register to calculate the bus address offset needed to do guest passthrough of the domain. This register shadows the Host Physical Address registers including the resource type bits. After calculating the offset, the extra resource type bits lead to the VMD resources being over-provisioned at the front and under-provisioned at the back. Example: pci 10000:80:02.0: reg 0x10: [mem 0xf801fffc-0xf803fffb 64bit] Expected: pci 10000:80:02.0: reg 0x10: [mem 0xf8020000-0xf803ffff 64bit] If other devices are mapped in the over-provisioned front, it could lead to resource conflict issues with VMD or those devices. Fixes: a1a30170138c9 ("PCI: vmd: Fix shadow offsets to reflect spec changes") Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> --- drivers/pci/controller/vmd.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index dac91d6..e386d4e 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -445,9 +445,11 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) if (!membar2) return -ENOMEM; offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - - readq(membar2 + MB2_SHADOW_OFFSET); + (readq(membar2 + MB2_SHADOW_OFFSET) & + PCI_BASE_ADDRESS_MEM_MASK); offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - - readq(membar2 + MB2_SHADOW_OFFSET + 8); + (readq(membar2 + MB2_SHADOW_OFFSET + 8) & + PCI_BASE_ADDRESS_MEM_MASK); pci_iounmap(vmd->dev, membar2); } } -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register 2020-05-28 3:02 ` [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register Jon Derrick @ 2020-05-29 10:33 ` Lorenzo Pieralisi 2020-05-29 15:53 ` Derrick, Jonathan 0 siblings, 1 reply; 11+ messages in thread From: Lorenzo Pieralisi @ 2020-05-29 10:33 UTC (permalink / raw) To: Jon Derrick Cc: Alex Williamson, linux-pci, qemu-devel, virtualization, Andrzej Jakowski, Bjorn Helgaas, Christoph Hellwig On Wed, May 27, 2020 at 11:02:39PM -0400, Jon Derrick wrote: > Versions of VMD with the Host Physical Address shadow register use this > register to calculate the bus address offset needed to do guest > passthrough of the domain. This register shadows the Host Physical > Address registers including the resource type bits. After calculating > the offset, the extra resource type bits lead to the VMD resources being > over-provisioned at the front and under-provisioned at the back. > > Example: > pci 10000:80:02.0: reg 0x10: [mem 0xf801fffc-0xf803fffb 64bit] > > Expected: > pci 10000:80:02.0: reg 0x10: [mem 0xf8020000-0xf803ffff 64bit] > > If other devices are mapped in the over-provisioned front, it could lead > to resource conflict issues with VMD or those devices. > > Fixes: a1a30170138c9 ("PCI: vmd: Fix shadow offsets to reflect spec changes") > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > --- > drivers/pci/controller/vmd.c | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) Hi Jon, it looks like I can take this patch for v5.8 whereas patch 2 depends on the QEMU changes acceptance and should probably wait. Please let me know your thoughts asap and I will try to at least squeeze this patch in. Lorenzo > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > index dac91d6..e386d4e 100644 > --- a/drivers/pci/controller/vmd.c > +++ b/drivers/pci/controller/vmd.c > @@ -445,9 +445,11 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > if (!membar2) > return -ENOMEM; > offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - > - readq(membar2 + MB2_SHADOW_OFFSET); > + (readq(membar2 + MB2_SHADOW_OFFSET) & > + PCI_BASE_ADDRESS_MEM_MASK); > offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - > - readq(membar2 + MB2_SHADOW_OFFSET + 8); > + (readq(membar2 + MB2_SHADOW_OFFSET + 8) & > + PCI_BASE_ADDRESS_MEM_MASK); > pci_iounmap(vmd->dev, membar2); > } > } > -- > 1.8.3.1 > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register 2020-05-29 10:33 ` Lorenzo Pieralisi @ 2020-05-29 15:53 ` Derrick, Jonathan 2020-05-29 16:18 ` Lorenzo Pieralisi 0 siblings, 1 reply; 11+ messages in thread From: Derrick, Jonathan @ 2020-05-29 15:53 UTC (permalink / raw) To: lorenzo.pieralisi Cc: alex.williamson, linux-pci, qemu-devel, virtualization, andrzej.jakowski, helgaas, hch On Fri, 2020-05-29 at 11:33 +0100, Lorenzo Pieralisi wrote: > On Wed, May 27, 2020 at 11:02:39PM -0400, Jon Derrick wrote: > > Versions of VMD with the Host Physical Address shadow register use this > > register to calculate the bus address offset needed to do guest > > passthrough of the domain. This register shadows the Host Physical > > Address registers including the resource type bits. After calculating > > the offset, the extra resource type bits lead to the VMD resources being > > over-provisioned at the front and under-provisioned at the back. > > > > Example: > > pci 10000:80:02.0: reg 0x10: [mem 0xf801fffc-0xf803fffb 64bit] > > > > Expected: > > pci 10000:80:02.0: reg 0x10: [mem 0xf8020000-0xf803ffff 64bit] > > > > If other devices are mapped in the over-provisioned front, it could lead > > to resource conflict issues with VMD or those devices. > > > > Fixes: a1a30170138c9 ("PCI: vmd: Fix shadow offsets to reflect spec changes") > > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > > --- > > drivers/pci/controller/vmd.c | 6 ++++-- > > 1 file changed, 4 insertions(+), 2 deletions(-) > > Hi Jon, > > it looks like I can take this patch for v5.8 whereas patch 2 depends > on the QEMU changes acceptance and should probably wait. > > Please let me know your thoughts asap and I will try to at least > squeeze this patch in. > > Lorenzo Hi Lorenzo, This is fine. Please take Patch 1. Patch 2 is harmless without the QEMU changes, but may always need a different approach. Best, jon > > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > > index dac91d6..e386d4e 100644 > > --- a/drivers/pci/controller/vmd.c > > +++ b/drivers/pci/controller/vmd.c > > @@ -445,9 +445,11 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > if (!membar2) > > return -ENOMEM; > > offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - > > - readq(membar2 + MB2_SHADOW_OFFSET); > > + (readq(membar2 + MB2_SHADOW_OFFSET) & > > + PCI_BASE_ADDRESS_MEM_MASK); > > offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - > > - readq(membar2 + MB2_SHADOW_OFFSET + 8); > > + (readq(membar2 + MB2_SHADOW_OFFSET + 8) & > > + PCI_BASE_ADDRESS_MEM_MASK); > > pci_iounmap(vmd->dev, membar2); > > } > > } > > -- > > 1.8.3.1 > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register 2020-05-29 15:53 ` Derrick, Jonathan @ 2020-05-29 16:18 ` Lorenzo Pieralisi 2020-06-11 21:16 ` Derrick, Jonathan 0 siblings, 1 reply; 11+ messages in thread From: Lorenzo Pieralisi @ 2020-05-29 16:18 UTC (permalink / raw) To: Derrick, Jonathan Cc: alex.williamson, linux-pci, qemu-devel, virtualization, andrzej.jakowski, helgaas, hch On Fri, May 29, 2020 at 03:53:37PM +0000, Derrick, Jonathan wrote: > On Fri, 2020-05-29 at 11:33 +0100, Lorenzo Pieralisi wrote: > > On Wed, May 27, 2020 at 11:02:39PM -0400, Jon Derrick wrote: > > > Versions of VMD with the Host Physical Address shadow register use this > > > register to calculate the bus address offset needed to do guest > > > passthrough of the domain. This register shadows the Host Physical > > > Address registers including the resource type bits. After calculating > > > the offset, the extra resource type bits lead to the VMD resources being > > > over-provisioned at the front and under-provisioned at the back. > > > > > > Example: > > > pci 10000:80:02.0: reg 0x10: [mem 0xf801fffc-0xf803fffb 64bit] > > > > > > Expected: > > > pci 10000:80:02.0: reg 0x10: [mem 0xf8020000-0xf803ffff 64bit] > > > > > > If other devices are mapped in the over-provisioned front, it could lead > > > to resource conflict issues with VMD or those devices. > > > > > > Fixes: a1a30170138c9 ("PCI: vmd: Fix shadow offsets to reflect spec changes") > > > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > > > --- > > > drivers/pci/controller/vmd.c | 6 ++++-- > > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > Hi Jon, > > > > it looks like I can take this patch for v5.8 whereas patch 2 depends > > on the QEMU changes acceptance and should probably wait. > > > > Please let me know your thoughts asap and I will try to at least > > squeeze this patch in. > > > > Lorenzo > > Hi Lorenzo, > > This is fine. Please take Patch 1. > Patch 2 is harmless without the QEMU changes, but may always need a > different approach. Pulled patch 1 into pci/vmd, thanks. Lorenzo ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register 2020-05-29 16:18 ` Lorenzo Pieralisi @ 2020-06-11 21:16 ` Derrick, Jonathan 2020-06-12 13:54 ` Lorenzo Pieralisi 0 siblings, 1 reply; 11+ messages in thread From: Derrick, Jonathan @ 2020-06-11 21:16 UTC (permalink / raw) To: lorenzo.pieralisi Cc: alex.williamson, linux-pci, qemu-devel, virtualization, andrzej.jakowski, helgaas, hch On Fri, 2020-05-29 at 17:18 +0100, Lorenzo Pieralisi wrote: > On Fri, May 29, 2020 at 03:53:37PM +0000, Derrick, Jonathan wrote: > > On Fri, 2020-05-29 at 11:33 +0100, Lorenzo Pieralisi wrote: > > > On Wed, May 27, 2020 at 11:02:39PM -0400, Jon Derrick wrote: > > > > Versions of VMD with the Host Physical Address shadow register use this > > > > register to calculate the bus address offset needed to do guest > > > > passthrough of the domain. This register shadows the Host Physical > > > > Address registers including the resource type bits. After calculating > > > > the offset, the extra resource type bits lead to the VMD resources being > > > > over-provisioned at the front and under-provisioned at the back. > > > > > > > > Example: > > > > pci 10000:80:02.0: reg 0x10: [mem 0xf801fffc-0xf803fffb 64bit] > > > > > > > > Expected: > > > > pci 10000:80:02.0: reg 0x10: [mem 0xf8020000-0xf803ffff 64bit] > > > > > > > > If other devices are mapped in the over-provisioned front, it could lead > > > > to resource conflict issues with VMD or those devices. > > > > > > > > Fixes: a1a30170138c9 ("PCI: vmd: Fix shadow offsets to reflect spec changes") > > > > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > > > > --- > > > > drivers/pci/controller/vmd.c | 6 ++++-- > > > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > > > Hi Jon, > > > > > > it looks like I can take this patch for v5.8 whereas patch 2 depends > > > on the QEMU changes acceptance and should probably wait. > > > > > > Please let me know your thoughts asap and I will try to at least > > > squeeze this patch in. > > > > > > Lorenzo > > > > Hi Lorenzo, > > > > This is fine. Please take Patch 1. > > Patch 2 is harmless without the QEMU changes, but may always need a > > different approach. > > Pulled patch 1 into pci/vmd, thanks. > > Lorenzo Hi Lorenzo, Alex has pr-ed the QEMU patch [1] Is it too late to pull patch 2/2 for v5.8? [1] https://github.com/awilliam/qemu-vfio/releases/tag/vfio-update-20200611.0 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register 2020-06-11 21:16 ` Derrick, Jonathan @ 2020-06-12 13:54 ` Lorenzo Pieralisi 2020-06-12 15:11 ` Derrick, Jonathan 0 siblings, 1 reply; 11+ messages in thread From: Lorenzo Pieralisi @ 2020-06-12 13:54 UTC (permalink / raw) To: Derrick, Jonathan, Bjorn Helgaas Cc: alex.williamson, linux-pci, qemu-devel, virtualization, andrzej.jakowski, helgaas, hch On Thu, Jun 11, 2020 at 09:16:48PM +0000, Derrick, Jonathan wrote: [...] > > > > Hi Jon, > > > > > > > > it looks like I can take this patch for v5.8 whereas patch 2 depends > > > > on the QEMU changes acceptance and should probably wait. > > > > > > > > Please let me know your thoughts asap and I will try to at least > > > > squeeze this patch in. > > > > > > > > Lorenzo > > > > > > Hi Lorenzo, > > > > > > This is fine. Please take Patch 1. > > > Patch 2 is harmless without the QEMU changes, but may always need a > > > different approach. > > > > Pulled patch 1 into pci/vmd, thanks. > > > > Lorenzo > > Hi Lorenzo, > > Alex has pr-ed the QEMU patch [1] > Is it too late to pull patch 2/2 for v5.8? I think it is - I don't know if Bjorn planned a second PR for this merge window, if not it is v5.9 material I am afraid. Thanks, Lorenzo > [1] > https://github.com/awilliam/qemu-vfio/releases/tag/vfio-update-20200611.0 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register 2020-06-12 13:54 ` Lorenzo Pieralisi @ 2020-06-12 15:11 ` Derrick, Jonathan 0 siblings, 0 replies; 11+ messages in thread From: Derrick, Jonathan @ 2020-06-12 15:11 UTC (permalink / raw) To: lorenzo.pieralisi, bhelgaas Cc: alex.williamson, linux-pci, qemu-devel, virtualization, andrzej.jakowski, helgaas, hch On Fri, 2020-06-12 at 14:54 +0100, Lorenzo Pieralisi wrote: > On Thu, Jun 11, 2020 at 09:16:48PM +0000, Derrick, Jonathan wrote: > > [...] > > > > > > Hi Jon, > > > > > > > > > > it looks like I can take this patch for v5.8 whereas patch 2 depends > > > > > on the QEMU changes acceptance and should probably wait. > > > > > > > > > > Please let me know your thoughts asap and I will try to at least > > > > > squeeze this patch in. > > > > > > > > > > Lorenzo > > > > > > > > Hi Lorenzo, > > > > > > > > This is fine. Please take Patch 1. > > > > Patch 2 is harmless without the QEMU changes, but may always need a > > > > different approach. > > > > > > Pulled patch 1 into pci/vmd, thanks. > > > > > > Lorenzo > > > > Hi Lorenzo, > > > > Alex has pr-ed the QEMU patch [1] > > Is it too late to pull patch 2/2 for v5.8? > > I think it is - I don't know if Bjorn planned a second PR for this > merge window, if not it is v5.9 material I am afraid. > > Thanks, > Lorenzo > > > [1] > > https://github.com/awilliam/qemu-vfio/releases/tag/vfio-update-20200611.0 No problem Jon ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v3 2/2] PCI: vmd: Use Shadow MEMBAR registers for QEMU/KVM guests 2020-05-28 3:02 [PATCH v3 0/2] VMD endpoint passthrough support Jon Derrick 2020-05-28 3:02 ` [PATCH v3 FOR QEMU v3] hw/vfio: Add VMD Passthrough Quirk Jon Derrick 2020-05-28 3:02 ` [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register Jon Derrick @ 2020-05-28 3:02 ` Jon Derrick 2020-07-06 9:16 ` Lorenzo Pieralisi 2 siblings, 1 reply; 11+ messages in thread From: Jon Derrick @ 2020-05-28 3:02 UTC (permalink / raw) To: linux-pci, qemu-devel Cc: Lorenzo Pieralisi, Andrzej Jakowski, virtualization, Alex Williamson, Bjorn Helgaas, Christoph Hellwig, Jon Derrick VMD device 28C0 natively assists guest passthrough of the VMD endpoint through the use of shadow registers that provide Host Physical Addresses to correctly assign bridge windows. These shadow registers are only available if VMD config space register 0x70, bit 1 is set. In order to support this mode in existing VMD devices which don't natively support the shadow register, it was decided that the hypervisor could offer the shadow registers in a vendor-specific PCI capability. QEMU has been modified to create this vendor-specific capability and supply the shadow membar registers for VMDs which don't natively support this feature. This patch adds this mode and updates the supported device list to allow this feature to be used on these VMDs. Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> --- drivers/pci/controller/vmd.c | 44 ++++++++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index e386d4e..76d8acb 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -40,13 +40,19 @@ enum vmd_features { * membars, in order to allow proper address translation during * resource assignment to enable guest virtualization */ - VMD_FEAT_HAS_MEMBAR_SHADOW = (1 << 0), + VMD_FEAT_HAS_MEMBAR_SHADOW = (1 << 0), /* * Device may provide root port configuration information which limits * bus numbering */ - VMD_FEAT_HAS_BUS_RESTRICTIONS = (1 << 1), + VMD_FEAT_HAS_BUS_RESTRICTIONS = (1 << 1), + + /* + * Device contains physical location shadow registers in + * vendor-specific capability space + */ + VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP = (1 << 2), }; /* @@ -454,6 +460,28 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) } } + if (features & VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP) { + int pos = pci_find_capability(vmd->dev, PCI_CAP_ID_VNDR); + u32 reg, regu; + + pci_read_config_dword(vmd->dev, pos + 4, ®); + + /* "SHDW" */ + if (pos && reg == 0x53484457) { + pci_read_config_dword(vmd->dev, pos + 8, ®); + pci_read_config_dword(vmd->dev, pos + 12, ®u); + offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - + (((u64) regu << 32 | reg) & + PCI_BASE_ADDRESS_MEM_MASK); + + pci_read_config_dword(vmd->dev, pos + 16, ®); + pci_read_config_dword(vmd->dev, pos + 20, ®u); + offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - + (((u64) regu << 32 | reg) & + PCI_BASE_ADDRESS_MEM_MASK); + } + } + /* * Certain VMD devices may have a root port configuration option which * limits the bus range to between 0-127, 128-255, or 224-255 @@ -716,16 +744,20 @@ static int vmd_resume(struct device *dev) static SIMPLE_DEV_PM_OPS(vmd_dev_pm_ops, vmd_suspend, vmd_resume); static const struct pci_device_id vmd_ids[] = { - {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_201D),}, + {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_201D), + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP,}, {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_28C0), .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW | VMD_FEAT_HAS_BUS_RESTRICTIONS,}, {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x467f), - .driver_data = VMD_FEAT_HAS_BUS_RESTRICTIONS,}, + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | + VMD_FEAT_HAS_BUS_RESTRICTIONS,}, {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x4c3d), - .driver_data = VMD_FEAT_HAS_BUS_RESTRICTIONS,}, + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | + VMD_FEAT_HAS_BUS_RESTRICTIONS,}, {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_9A0B), - .driver_data = VMD_FEAT_HAS_BUS_RESTRICTIONS,}, + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | + VMD_FEAT_HAS_BUS_RESTRICTIONS,}, {0,} }; MODULE_DEVICE_TABLE(pci, vmd_ids); -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v3 2/2] PCI: vmd: Use Shadow MEMBAR registers for QEMU/KVM guests 2020-05-28 3:02 ` [PATCH v3 2/2] PCI: vmd: Use Shadow MEMBAR registers for QEMU/KVM guests Jon Derrick @ 2020-07-06 9:16 ` Lorenzo Pieralisi 0 siblings, 0 replies; 11+ messages in thread From: Lorenzo Pieralisi @ 2020-07-06 9:16 UTC (permalink / raw) To: Jon Derrick Cc: Alex Williamson, linux-pci, qemu-devel, virtualization, Andrzej Jakowski, Bjorn Helgaas, Christoph Hellwig On Wed, May 27, 2020 at 11:02:40PM -0400, Jon Derrick wrote: > VMD device 28C0 natively assists guest passthrough of the VMD endpoint > through the use of shadow registers that provide Host Physical Addresses > to correctly assign bridge windows. These shadow registers are only > available if VMD config space register 0x70, bit 1 is set. > > In order to support this mode in existing VMD devices which don't > natively support the shadow register, it was decided that the hypervisor > could offer the shadow registers in a vendor-specific PCI capability. > > QEMU has been modified to create this vendor-specific capability and > supply the shadow membar registers for VMDs which don't natively support > this feature. This patch adds this mode and updates the supported device > list to allow this feature to be used on these VMDs. > > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > --- > drivers/pci/controller/vmd.c | 44 ++++++++++++++++++++++++++++++++++++++------ > 1 file changed, 38 insertions(+), 6 deletions(-) Applied to pci/vmd, thanks. Lorenzo > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > index e386d4e..76d8acb 100644 > --- a/drivers/pci/controller/vmd.c > +++ b/drivers/pci/controller/vmd.c > @@ -40,13 +40,19 @@ enum vmd_features { > * membars, in order to allow proper address translation during > * resource assignment to enable guest virtualization > */ > - VMD_FEAT_HAS_MEMBAR_SHADOW = (1 << 0), > + VMD_FEAT_HAS_MEMBAR_SHADOW = (1 << 0), > > /* > * Device may provide root port configuration information which limits > * bus numbering > */ > - VMD_FEAT_HAS_BUS_RESTRICTIONS = (1 << 1), > + VMD_FEAT_HAS_BUS_RESTRICTIONS = (1 << 1), > + > + /* > + * Device contains physical location shadow registers in > + * vendor-specific capability space > + */ > + VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP = (1 << 2), > }; > > /* > @@ -454,6 +460,28 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > } > } > > + if (features & VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP) { > + int pos = pci_find_capability(vmd->dev, PCI_CAP_ID_VNDR); > + u32 reg, regu; > + > + pci_read_config_dword(vmd->dev, pos + 4, ®); > + > + /* "SHDW" */ > + if (pos && reg == 0x53484457) { > + pci_read_config_dword(vmd->dev, pos + 8, ®); > + pci_read_config_dword(vmd->dev, pos + 12, ®u); > + offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - > + (((u64) regu << 32 | reg) & > + PCI_BASE_ADDRESS_MEM_MASK); > + > + pci_read_config_dword(vmd->dev, pos + 16, ®); > + pci_read_config_dword(vmd->dev, pos + 20, ®u); > + offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - > + (((u64) regu << 32 | reg) & > + PCI_BASE_ADDRESS_MEM_MASK); > + } > + } > + > /* > * Certain VMD devices may have a root port configuration option which > * limits the bus range to between 0-127, 128-255, or 224-255 > @@ -716,16 +744,20 @@ static int vmd_resume(struct device *dev) > static SIMPLE_DEV_PM_OPS(vmd_dev_pm_ops, vmd_suspend, vmd_resume); > > static const struct pci_device_id vmd_ids[] = { > - {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_201D),}, > + {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_201D), > + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP,}, > {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_28C0), > .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW | > VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x467f), > - .driver_data = VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | > + VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x4c3d), > - .driver_data = VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | > + VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_9A0B), > - .driver_data = VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > + .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | > + VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > {0,} > }; > MODULE_DEVICE_TABLE(pci, vmd_ids); > -- > 1.8.3.1 > ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2020-07-06 9:20 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-05-28 3:02 [PATCH v3 0/2] VMD endpoint passthrough support Jon Derrick 2020-05-28 3:02 ` [PATCH v3 FOR QEMU v3] hw/vfio: Add VMD Passthrough Quirk Jon Derrick 2020-05-28 3:02 ` [PATCH v3 1/2] PCI: vmd: Filter resource type bits from shadow register Jon Derrick 2020-05-29 10:33 ` Lorenzo Pieralisi 2020-05-29 15:53 ` Derrick, Jonathan 2020-05-29 16:18 ` Lorenzo Pieralisi 2020-06-11 21:16 ` Derrick, Jonathan 2020-06-12 13:54 ` Lorenzo Pieralisi 2020-06-12 15:11 ` Derrick, Jonathan 2020-05-28 3:02 ` [PATCH v3 2/2] PCI: vmd: Use Shadow MEMBAR registers for QEMU/KVM guests Jon Derrick 2020-07-06 9:16 ` Lorenzo Pieralisi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).