* [PATCH v5 00/11] PCI devices passthrough on Arm
@ 2021-10-06 17:40 Rahul Singh
2021-10-06 17:40 ` [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM Rahul Singh
` (11 more replies)
0 siblings, 12 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara, Ian Jackson,
Wei Liu, Juergen Gross, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Jan Beulich,
Roger Pau Monné,
Christian Lindig, David Scott, Paul Durrant, Anthony PERARD
Hello All,
The purpose of this patch series is to add PCI passthrough support to Xen on
Arm. PCI passthrough support on ARM is the collaboration work between EPAM and
ARM. ARM submitted the partial RFC [1][2] last year to get early feedback. We
tried to fix all the comments and added more features to this patch series.
Working POC with all the features can be found at [3]. Working POC is tested
on x86 so that there will be no regression on x86. Design presentation can be
found at [4]
PCI passthrough support is divided into different patches. This patch series
includes following features:
Preparatory work to implement the PCI passthrough support for the ARM:
- Refactor MSI code.
- Fixed compilation error when HAS_PCI enabled for ARM.
Discovering PCI Host Bridge in XEN:
- PCI init to initialize the PCI driver.
- PCI host bridge discovery in XEN and map the PCI ECAM configuration space to
the XEN memory.
- PCI access functions.
Discovering PCI devices:
- To support the PCI passthrough, XEN should be aware of the PCI
devices.
- Hardware domain is in charge of doing the PCI enumeration and will discover
the PCI devices and then communicate to the XEN via a hypercall to add the
PCI devices in XEN.
Enable the existing x86 virtual PCI support for ARM:
- Add VPCI trap handler for each of the PCI device added for config space
access.
- Register the trap handler in XEN for each of the host bridge PCI ECAM config
space access.
Emulated PCI device tree node in libxl:
- Create a virtual PCI device tree node in libxl to enable the guest OS to
discover the virtual PCI during guest boot.
This patch series does not inlcude the following features. Following features
will be send for review in the next version of the patch series once initial
patch series merged.
- VPCI support for DOMU guests (Non-identity mappings guest view of the BARs)
- Virtual bus topology implementation
- IOMMU related changes (generic, SMMUv2, SMMUv3)
- MSI support for DOMU guests.
- Virual ITS support for DOMU guests
[1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg01184.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2020-07/threads.html#01184
[3] https://gitlab.com/rahsingh/xen-integration/-/tree/arm_pci_v4_integrate
[4] https://static.sched.com/hosted_files/xen2021/e4/PCI_Device_Passthrough_On_Arm.pdf
Oleksandr Andrushchenko (1):
xen/arm: Add support for Xilinx ZynqMP PCI host controller
Rahul Singh (10):
xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM
xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
xen/arm: PCI host bridge discovery within XEN on ARM
xen/arm: Implement pci access functions
xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
xen/arm: Enable the existing x86 virtual PCI support for ARM.
xen/arm: Transitional change to build HAS_VPCI on ARM.
arm/libxl: Emulated PCI device tree node in libxl
xen/arm: Add linux,pci-domain property for hwdom if not available.
docs/misc/xen-command-line.pandoc | 7 +
tools/include/libxl.h | 6 +
tools/libs/ctrl/xc_domain.c | 9 +
tools/libs/light/libxl_arm.c | 111 +++++++++++
tools/libs/light/libxl_types.idl | 1 +
tools/ocaml/libs/xc/xenctrl.ml | 1 +
tools/ocaml/libs/xc/xenctrl.mli | 1 +
xen/arch/arm/Makefile | 1 +
xen/arch/arm/domain.c | 8 +-
xen/arch/arm/domain_build.c | 19 ++
xen/arch/arm/pci/Makefile | 5 +
xen/arch/arm/pci/ecam.c | 61 ++++++
xen/arch/arm/pci/pci-access.c | 140 ++++++++++++++
xen/arch/arm/pci/pci-host-common.c | 287 ++++++++++++++++++++++++++++
xen/arch/arm/pci/pci-host-generic.c | 48 +++++
xen/arch/arm/pci/pci-host-zynqmp.c | 65 +++++++
xen/arch/arm/pci/pci.c | 12 ++
xen/arch/arm/physdev.c | 6 +-
xen/arch/arm/vpci.c | 102 ++++++++++
xen/arch/arm/vpci.h | 36 ++++
xen/arch/x86/domain.c | 6 +
xen/arch/x86/physdev.c | 52 +----
xen/arch/x86/x86_64/physdev.c | 2 +-
xen/common/domain.c | 2 +-
xen/drivers/passthrough/pci.c | 18 ++
xen/drivers/pci/Makefile | 1 +
xen/drivers/pci/physdev.c | 86 +++++++++
xen/drivers/vpci/Makefile | 3 +-
xen/drivers/vpci/header.c | 2 +
xen/include/asm-arm/domain.h | 7 +-
xen/include/asm-arm/pci.h | 85 ++++++++
xen/include/asm-x86/pci.h | 8 +-
xen/include/public/arch-arm.h | 21 +-
xen/include/public/domctl.h | 4 +-
xen/include/xen/hypercall.h | 4 +
xen/include/xen/pci.h | 2 +
36 files changed, 1168 insertions(+), 61 deletions(-)
create mode 100644 xen/arch/arm/pci/ecam.c
create mode 100644 xen/arch/arm/pci/pci-access.c
create mode 100644 xen/arch/arm/pci/pci-host-common.c
create mode 100644 xen/arch/arm/pci/pci-host-generic.c
create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
create mode 100644 xen/arch/arm/vpci.c
create mode 100644 xen/arch/arm/vpci.h
create mode 100644 xen/drivers/pci/physdev.c
--
2.25.1
^ permalink raw reply [flat|nested] 190+ messages in thread
* [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-11 11:47 ` Roger Pau Monné
2021-10-06 17:40 ` [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM Rahul Singh
` (10 subsequent siblings)
11 siblings, 1 reply; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara, Ian Jackson,
Wei Liu, Juergen Gross, Stefano Stabellini
ARM architecture does not implement I/O ports. Ignore this call on ARM
to avoid the overhead of making a hypercall just for Xen to return
-ENOSYS.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Change in v5: none
Change in v4:
- Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Change in v3: Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Change in v2:
- Instead of returning success in XEN, ignored the call in xl.
---
---
tools/libs/ctrl/xc_domain.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
index 23322b70b5..25c95f6596 100644
--- a/tools/libs/ctrl/xc_domain.c
+++ b/tools/libs/ctrl/xc_domain.c
@@ -1348,6 +1348,14 @@ int xc_domain_ioport_permission(xc_interface *xch,
uint32_t nr_ports,
uint32_t allow_access)
{
+#if defined(__arm__) || defined(__aarch64__)
+ /*
+ * The ARM architecture does not implement I/O ports.
+ * Avoid the overhead of making a hypercall just for Xen to return -ENOSYS.
+ * It is safe to ignore this call on ARM so we just return 0.
+ */
+ return 0;
+#else
DECLARE_DOMCTL;
domctl.cmd = XEN_DOMCTL_ioport_permission;
@@ -1357,6 +1365,7 @@ int xc_domain_ioport_permission(xc_interface *xch,
domctl.u.ioport_permission.allow_access = allow_access;
return do_domctl(xch, &domctl);
+#endif
}
int xc_availheap(xc_interface *xch,
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
2021-10-06 17:40 ` [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-07 0:05 ` Stefano Stabellini
2021-10-21 9:28 ` xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM) Julien Grall
2021-10-06 17:40 ` [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>" Rahul Singh
` (9 subsequent siblings)
11 siblings, 2 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Ian Jackson, Jan Beulich, Wei Liu,
Roger Pau Monné
Hardware domain is in charge of doing the PCI enumeration and will
discover the PCI devices and then will communicate to XEN via hyper
call PHYSDEVOP_pci_device_add(..) to add the PCI devices in XEN.
Also implement PHYSDEVOP_pci_device_remove(..) to remove the PCI device.
As most of the code for PHYSDEVOP_pci_device_* is the same between x86
and ARM, move the code to a common file to avoid duplication.
There are other PHYSDEVOP_pci_device_* operations to add PCI devices.
Currently implemented PHYSDEVOP_pci_device_remove(..) and
PHYSDEVOP_pci_device_add(..) only as those are minimum required to
support PCI passthrough on ARM.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
Change in v5:
- Move the pci_physdev_op() stub to xen/arch/arm/physdev.c.
Change in v4:
- Move file commom/physdev.c to drivers/pci/physdev.c
- minor comments.
Change in v3: Fixed minor comment.
Change in v2:
- Add support for PHYSDEVOP_pci_device_remove()
- Move code to common code
---
---
xen/arch/arm/physdev.c | 6 ++-
xen/arch/x86/physdev.c | 52 +----------------------
xen/arch/x86/x86_64/physdev.c | 2 +-
xen/drivers/pci/Makefile | 1 +
xen/drivers/pci/physdev.c | 80 +++++++++++++++++++++++++++++++++++
xen/include/public/arch-arm.h | 4 +-
xen/include/xen/hypercall.h | 4 ++
7 files changed, 96 insertions(+), 53 deletions(-)
create mode 100644 xen/drivers/pci/physdev.c
diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
index e91355fe22..f9aa274dda 100644
--- a/xen/arch/arm/physdev.c
+++ b/xen/arch/arm/physdev.c
@@ -8,13 +8,17 @@
#include <xen/lib.h>
#include <xen/errno.h>
#include <xen/sched.h>
-#include <asm/hypercall.h>
+#include <xen/hypercall.h>
int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
{
+#ifdef CONFIG_HAS_PCI
+ return pci_physdev_op(cmd, arg);
+#else
gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
return -ENOSYS;
+#endif
}
/*
diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
index 23465bcd00..ea38be8b79 100644
--- a/xen/arch/x86/physdev.c
+++ b/xen/arch/x86/physdev.c
@@ -12,7 +12,7 @@
#include <asm/io_apic.h>
#include <asm/msi.h>
#include <asm/hvm/irq.h>
-#include <asm/hypercall.h>
+#include <xen/hypercall.h>
#include <public/xen.h>
#include <public/physdev.h>
#include <xsm/xsm.h>
@@ -480,54 +480,6 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
break;
}
- case PHYSDEVOP_pci_device_add: {
- struct physdev_pci_device_add add;
- struct pci_dev_info pdev_info;
- nodeid_t node;
-
- ret = -EFAULT;
- if ( copy_from_guest(&add, arg, 1) != 0 )
- break;
-
- pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
- if ( add.flags & XEN_PCI_DEV_VIRTFN )
- {
- pdev_info.is_virtfn = 1;
- pdev_info.physfn.bus = add.physfn.bus;
- pdev_info.physfn.devfn = add.physfn.devfn;
- }
- else
- pdev_info.is_virtfn = 0;
-
- if ( add.flags & XEN_PCI_DEV_PXM )
- {
- uint32_t pxm;
- size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
- sizeof(add.optarr[0]);
-
- if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
- break;
-
- node = pxm_to_node(pxm);
- }
- else
- node = NUMA_NO_NODE;
-
- ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
- break;
- }
-
- case PHYSDEVOP_pci_device_remove: {
- struct physdev_pci_device dev;
-
- ret = -EFAULT;
- if ( copy_from_guest(&dev, arg, 1) != 0 )
- break;
-
- ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
- break;
- }
-
case PHYSDEVOP_prepare_msix:
case PHYSDEVOP_release_msix: {
struct physdev_pci_device dev;
@@ -663,7 +615,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
}
default:
- ret = -ENOSYS;
+ ret = pci_physdev_op(cmd, arg);
break;
}
diff --git a/xen/arch/x86/x86_64/physdev.c b/xen/arch/x86/x86_64/physdev.c
index 0a50cbd4d8..e3cbd5ebcb 100644
--- a/xen/arch/x86/x86_64/physdev.c
+++ b/xen/arch/x86/x86_64/physdev.c
@@ -9,7 +9,7 @@ EMIT_FILE;
#include <compat/xen.h>
#include <compat/event_channel.h>
#include <compat/physdev.h>
-#include <asm/hypercall.h>
+#include <xen/hypercall.h>
#define do_physdev_op compat_physdev_op
diff --git a/xen/drivers/pci/Makefile b/xen/drivers/pci/Makefile
index a98035df4c..972c923db0 100644
--- a/xen/drivers/pci/Makefile
+++ b/xen/drivers/pci/Makefile
@@ -1 +1,2 @@
obj-y += pci.o
+obj-y += physdev.o
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
new file mode 100644
index 0000000000..4f3e1a96c0
--- /dev/null
+++ b/xen/drivers/pci/physdev.c
@@ -0,0 +1,80 @@
+
+#include <xen/guest_access.h>
+#include <xen/hypercall.h>
+#include <xen/init.h>
+
+#ifndef COMPAT
+typedef long ret_t;
+#endif
+
+ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
+{
+ ret_t ret;
+
+ switch ( cmd )
+ {
+ case PHYSDEVOP_pci_device_add: {
+ struct physdev_pci_device_add add;
+ struct pci_dev_info pdev_info;
+ nodeid_t node = NUMA_NO_NODE;
+
+ ret = -EFAULT;
+ if ( copy_from_guest(&add, arg, 1) != 0 )
+ break;
+
+ pdev_info.is_extfn = (add.flags & XEN_PCI_DEV_EXTFN);
+ if ( add.flags & XEN_PCI_DEV_VIRTFN )
+ {
+ pdev_info.is_virtfn = true;
+ pdev_info.physfn.bus = add.physfn.bus;
+ pdev_info.physfn.devfn = add.physfn.devfn;
+ }
+ else
+ pdev_info.is_virtfn = false;
+
+#ifdef CONFIG_NUMA
+ if ( add.flags & XEN_PCI_DEV_PXM )
+ {
+ uint32_t pxm;
+ size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
+ sizeof(add.optarr[0]);
+
+ if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
+ break;
+
+ node = pxm_to_node(pxm);
+ }
+#endif
+
+ ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
+ break;
+ }
+
+ case PHYSDEVOP_pci_device_remove: {
+ struct physdev_pci_device dev;
+
+ ret = -EFAULT;
+ if ( copy_from_guest(&dev, arg, 1) != 0 )
+ break;
+
+ ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
+ break;
+ }
+
+ default:
+ ret = -ENOSYS;
+ break;
+ }
+
+ return ret;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 6b5a5f818a..d46c61fca9 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -107,7 +107,9 @@
* All generic sub-operations
*
* HYPERVISOR_physdev_op
- * No sub-operations are currenty supported
+ * Exactly these sub-operations are supported:
+ * PHYSDEVOP_pci_device_add
+ * PHYSDEVOP_pci_device_remove
*
* HYPERVISOR_sysctl
* All generic sub-operations, with the exception of:
diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
index 3771487a30..07b10ec230 100644
--- a/xen/include/xen/hypercall.h
+++ b/xen/include/xen/hypercall.h
@@ -45,6 +45,10 @@ extern long
do_platform_op(
XEN_GUEST_HANDLE_PARAM(xen_platform_op_t) u_xenpf_op);
+extern long
+pci_physdev_op(
+ int cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
+
/*
* To allow safe resume of do_memory_op() after preemption, we need to know
* at what point in the page list to resume. For this purpose I steal the
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
2021-10-06 17:40 ` [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM Rahul Singh
2021-10-06 17:40 ` [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-07 8:27 ` Jan Beulich
2021-10-07 12:59 ` Jan Beulich
2021-10-06 17:40 ` [PATCH v5 04/11] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
` (8 subsequent siblings)
11 siblings, 2 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara, Andrew Cooper,
George Dunlap, Ian Jackson, Jan Beulich, Julien Grall,
Stefano Stabellini, Wei Liu, Volodymyr Babchuk,
Roger Pau Monné
Add cmdline boot option "pci-passthrough = = <boolean>" to enable or
disable the PCI passthrough support on ARM.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Change in v5: none
Change in v4:
- Fixe minor comments
- Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Change in v3:
- Remove "define pci_passthrough_enabled (false)"
- Fixed coding style and minor comment
Change in v2:
- Add option in xen-command-line.pandoc
- Change pci option to pci-passthrough
- modify option from custom_param to boolean param
---
---
docs/misc/xen-command-line.pandoc | 7 +++++++
xen/arch/arm/pci/pci.c | 12 ++++++++++++
xen/drivers/pci/physdev.c | 6 ++++++
xen/include/asm-arm/pci.h | 11 +++++++++++
xen/include/asm-x86/pci.h | 6 ++++++
5 files changed, 42 insertions(+)
diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index 5cae4adc58..b1f7978aa4 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -1819,6 +1819,13 @@ All numbers specified must be hexadecimal ones.
This option can be specified more than once (up to 8 times at present).
+### pci-passthrough (arm)
+> `= <boolean>`
+
+> Default: `false`
+
+Flag to enable or disable support for PCI passthrough
+
### pcid (x86)
> `= <boolean> | xpti=<bool>`
diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index e359bab9ea..73540045d1 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -16,6 +16,7 @@
#include <xen/device_tree.h>
#include <xen/errno.h>
#include <xen/init.h>
+#include <xen/param.h>
#include <xen/pci.h>
/*
@@ -62,8 +63,19 @@ static int __init acpi_pci_init(void)
}
#endif
+/* By default pci passthrough is disabled. */
+bool __read_mostly pci_passthrough_enabled;
+boolean_param("pci-passthrough", pci_passthrough_enabled);
+
static int __init pci_init(void)
{
+ /*
+ * Enable PCI passthrough when has been enabled explicitly
+ * (pci-passthrough=on).
+ */
+ if ( !pci_passthrough_enabled )
+ return 0;
+
pci_segments_init();
if ( acpi_disabled )
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index 4f3e1a96c0..42db3e6d13 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -18,6 +18,9 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
struct pci_dev_info pdev_info;
nodeid_t node = NUMA_NO_NODE;
+ if ( !is_pci_passthrough_enabled() )
+ return -EOPNOTSUPP;
+
ret = -EFAULT;
if ( copy_from_guest(&add, arg, 1) != 0 )
break;
@@ -53,6 +56,9 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
case PHYSDEVOP_pci_device_remove: {
struct physdev_pci_device dev;
+ if ( !is_pci_passthrough_enabled() )
+ return -EOPNOTSUPP;
+
ret = -EFAULT;
if ( copy_from_guest(&dev, arg, 1) != 0 )
break;
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 7dd9eb3dba..0cf849e26f 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -19,14 +19,25 @@
#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+extern bool_t pci_passthrough_enabled;
+
/* Arch pci dev struct */
struct arch_pci_dev {
struct device dev;
};
+static always_inline bool is_pci_passthrough_enabled(void)
+{
+ return pci_passthrough_enabled;
+}
#else /*!CONFIG_HAS_PCI*/
struct arch_pci_dev { };
+static always_inline bool is_pci_passthrough_enabled(void)
+{
+ return false;
+}
+
#endif /*!CONFIG_HAS_PCI*/
#endif /* __ARM_PCI_H__ */
diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
index cc05045e9c..e076951032 100644
--- a/xen/include/asm-x86/pci.h
+++ b/xen/include/asm-x86/pci.h
@@ -32,4 +32,10 @@ bool_t pci_ro_mmcfg_decode(unsigned long mfn, unsigned int *seg,
extern int pci_mmcfg_config_num;
extern struct acpi_mcfg_allocation *pci_mmcfg_config;
+/* Unlike ARM, PCI passthrough is always enabled for x86. */
+static always_inline bool is_pci_passthrough_enabled(void)
+{
+ return true;
+}
+
#endif /* __X86_PCI_H__ */
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 04/11] xen/arm: PCI host bridge discovery within XEN on ARM
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (2 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>" Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-06 17:40 ` [PATCH v5 05/11] xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh
` (7 subsequent siblings)
11 siblings, 0 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk
XEN during boot will read the PCI device tree node “reg” property
and will map the PCI config space to the XEN memory.
As of now only "pci-host-ecam-generic" compatible board is supported.
"linux,pci-domain" device tree property assigns a fixed PCI domain
number to a host bridge, otherwise an unstable (across boots) unique
number will be assigned by Linux. XEN access the PCI devices based on
Segment:Bus:Device:Function. A Segment number in the XEN is same as a
domain number in Linux. Segment number and domain number has to be in
sync to access the correct PCI devices.
XEN will read the “linux,pci-domain” property from the device tree node
and configure the host bridge segment number accordingly. If this
property is not available XEN will allocate the unique segment number
to the host bridge.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
Chane in v5:
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Chane in v4:
- Move "#include <asm/pci.h>" in "xen/pci.h" after pci_sbdf_t
sbdf declaration change to separate patch.
- Make gen_pci_dt_match as __initconstrel
- Make pci_host_generic_probe as __init
Change in v3:
- Modify commit msg based on received comments.
- Remove added struct match_table{} struct in struct device{}
- Replace uint32_t sbdf to pci_sbdf_t sbdf to avoid typecast
- Remove bus_start,bus_end and void *sysdata from struct pci_host_bridge{}
- Move "#include <asm/pci.h>" in "xen/pci.h" after pci_sbdf_t sbdf declaration
- Add pci_host_generic_probe() function
Change in v2:
- Add more info in commit msg
- Add callback to parse register index.
- Merge patch pci_ecam_operation into this patch to avoid confusion
- Add new struct in struct device for match table
---
---
xen/arch/arm/pci/Makefile | 4 +
xen/arch/arm/pci/ecam.c | 61 +++++++
xen/arch/arm/pci/pci-access.c | 83 ++++++++++
xen/arch/arm/pci/pci-host-common.c | 247 ++++++++++++++++++++++++++++
xen/arch/arm/pci/pci-host-generic.c | 48 ++++++
xen/include/asm-arm/pci.h | 56 +++++++
6 files changed, 499 insertions(+)
create mode 100644 xen/arch/arm/pci/ecam.c
create mode 100644 xen/arch/arm/pci/pci-access.c
create mode 100644 xen/arch/arm/pci/pci-host-common.c
create mode 100644 xen/arch/arm/pci/pci-host-generic.c
diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
index a98035df4c..6f32fbbe67 100644
--- a/xen/arch/arm/pci/Makefile
+++ b/xen/arch/arm/pci/Makefile
@@ -1 +1,5 @@
obj-y += pci.o
+obj-y += pci-access.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += ecam.o
diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c
new file mode 100644
index 0000000000..602d00799c
--- /dev/null
+++ b/xen/arch/arm/pci/ecam.c
@@ -0,0 +1,61 @@
+/*
+ * Based on Linux drivers/pci/ecam.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/pci.h>
+#include <xen/sched.h>
+
+/*
+ * Function to implement the pci_ops->map_bus method.
+ */
+void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
+ pci_sbdf_t sbdf, uint32_t where)
+{
+ const struct pci_config_window *cfg = bridge->cfg;
+ struct pci_ecam_ops *ops =
+ container_of(bridge->ops, struct pci_ecam_ops, pci_ops);
+ unsigned int devfn_shift = ops->bus_shift - 8;
+ void __iomem *base;
+
+ unsigned int busn = PCI_BUS(sbdf.bdf);
+
+ if ( busn < cfg->busn_start || busn > cfg->busn_end )
+ return NULL;
+
+ busn -= cfg->busn_start;
+ base = cfg->win + (busn << ops->bus_shift);
+
+ return base + (PCI_DEVFN2(sbdf.bdf) << devfn_shift) + where;
+}
+
+/* ECAM ops */
+const struct pci_ecam_ops pci_generic_ecam_ops = {
+ .bus_shift = 20,
+ .pci_ops = {
+ .map_bus = pci_ecam_map_bus,
+ .read = pci_generic_config_read,
+ .write = pci_generic_config_write,
+ }
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
new file mode 100644
index 0000000000..3cd14a4b87
--- /dev/null
+++ b/xen/arch/arm/pci/pci-access.c
@@ -0,0 +1,83 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/pci.h>
+#include <asm/io.h>
+
+#define INVALID_VALUE (~0U)
+
+int pci_generic_config_read(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
+ uint32_t reg, uint32_t len, uint32_t *value)
+{
+ void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
+
+ if ( !addr )
+ {
+ *value = INVALID_VALUE;
+ return -ENODEV;
+ }
+
+ switch ( len )
+ {
+ case 1:
+ *value = readb(addr);
+ break;
+ case 2:
+ *value = readw(addr);
+ break;
+ case 4:
+ *value = readl(addr);
+ break;
+ default:
+ ASSERT_UNREACHABLE();
+ }
+
+ return 0;
+}
+
+int pci_generic_config_write(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
+ uint32_t reg, uint32_t len, uint32_t value)
+{
+ void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
+
+ if ( !addr )
+ return -ENODEV;
+
+ switch ( len )
+ {
+ case 1:
+ writeb(value, addr);
+ break;
+ case 2:
+ writew(value, addr);
+ break;
+ case 4:
+ writel(value, addr);
+ break;
+ default:
+ ASSERT_UNREACHABLE();
+ }
+
+ return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
new file mode 100644
index 0000000000..a08e06cea1
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -0,0 +1,247 @@
+/*
+ * Based on Linux drivers/pci/ecam.c
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/rwlock.h>
+#include <xen/sched.h>
+#include <xen/vmap.h>
+
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static atomic_t domain_nr = ATOMIC_INIT(-1);
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+ return ioremap_nocache(start, len);
+}
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+ if ( cfg->win )
+ iounmap(cfg->win);
+
+ xfree(cfg);
+}
+
+static struct pci_config_window * __init
+gen_pci_init(struct dt_device_node *dev, const struct pci_ecam_ops *ops)
+{
+ int err, cfg_reg_idx;
+ u32 bus_range[2];
+ paddr_t addr, size;
+ struct pci_config_window *cfg;
+
+ cfg = xzalloc(struct pci_config_window);
+ if ( !cfg )
+ return NULL;
+
+ err = dt_property_read_u32_array(dev, "bus-range", bus_range,
+ ARRAY_SIZE(bus_range));
+ if ( err ) {
+ cfg->busn_start = 0;
+ cfg->busn_end = 0xff;
+ printk(XENLOG_INFO "%s: No bus range found for pci controller\n",
+ dt_node_full_name(dev));
+ } else {
+ cfg->busn_start = bus_range[0];
+ cfg->busn_end = bus_range[1];
+ if ( cfg->busn_end > cfg->busn_start + 0xff )
+ cfg->busn_end = cfg->busn_start + 0xff;
+ }
+
+ if ( ops->cfg_reg_index )
+ {
+ cfg_reg_idx = ops->cfg_reg_index(dev);
+ if ( cfg_reg_idx < 0 )
+ goto err_exit;
+ }
+ else
+ cfg_reg_idx = 0;
+
+ /* Parse our PCI ecam register address */
+ err = dt_device_get_address(dev, cfg_reg_idx, &addr, &size);
+ if ( err )
+ goto err_exit;
+
+ cfg->phys_addr = addr;
+ cfg->size = size;
+
+ /*
+ * On 64-bit systems, we do a single ioremap for the whole config space
+ * since we have enough virtual address range available. On 32-bit, we
+ * ioremap the config space for each bus individually.
+ * As of now only 64-bit is supported 32-bit is not supported.
+ *
+ * TODO: For 32-bit implement the ioremap/iounmap of config space
+ * dynamically for each read/write call.
+ */
+ cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
+ if ( !cfg->win )
+ {
+ printk(XENLOG_ERR "ECAM ioremap failed\n");
+ goto err_exit;
+ }
+ printk("ECAM at [mem 0x%"PRIpaddr"-0x%"PRIpaddr"] for [bus %x-%x] \n",
+ cfg->phys_addr, cfg->phys_addr + cfg->size - 1,
+ cfg->busn_start, cfg->busn_end);
+
+ if ( ops->init )
+ {
+ err = ops->init(cfg);
+ if ( err )
+ goto err_exit;
+ }
+
+ return cfg;
+
+err_exit:
+ pci_ecam_free(cfg);
+
+ return NULL;
+}
+
+struct pci_host_bridge *pci_alloc_host_bridge(void)
+{
+ struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+ if ( !bridge )
+ return NULL;
+
+ INIT_LIST_HEAD(&bridge->node);
+
+ return bridge;
+}
+
+void pci_add_host_bridge(struct pci_host_bridge *bridge)
+{
+ list_add_tail(&bridge->node, &pci_host_bridges);
+}
+
+static int pci_get_new_domain_nr(void)
+{
+ return atomic_inc_return(&domain_nr);
+}
+
+static int pci_bus_find_domain_nr(struct dt_device_node *dev)
+{
+ static int use_dt_domains = -1;
+ int domain;
+
+ domain = dt_get_pci_domain_nr(dev);
+
+ /*
+ * Check DT domain and use_dt_domains values.
+ *
+ * If DT domain property is valid (domain >= 0) and
+ * use_dt_domains != 0, the DT assignment is valid since this means
+ * we have not previously allocated a domain number by using
+ * pci_get_new_domain_nr(); we should also update use_dt_domains to
+ * 1, to indicate that we have just assigned a domain number from
+ * DT.
+ *
+ * If DT domain property value is not valid (ie domain < 0), and we
+ * have not previously assigned a domain number from DT
+ * (use_dt_domains != 1) we should assign a domain number by
+ * using the:
+ *
+ * pci_get_new_domain_nr()
+ *
+ * API and update the use_dt_domains value to keep track of method we
+ * are using to assign domain numbers (use_dt_domains = 0).
+ *
+ * All other combinations imply we have a platform that is trying
+ * to mix domain numbers obtained from DT and pci_get_new_domain_nr(),
+ * which is a recipe for domain mishandling and it is prevented by
+ * invalidating the domain value (domain = -1) and printing a
+ * corresponding error.
+ */
+ if ( domain >= 0 && use_dt_domains )
+ {
+ use_dt_domains = 1;
+ }
+ else if ( domain < 0 && use_dt_domains != 1 )
+ {
+ use_dt_domains = 0;
+ domain = pci_get_new_domain_nr();
+ }
+ else
+ {
+ domain = -1;
+ }
+
+ return domain;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev, const void *data)
+{
+ struct pci_host_bridge *bridge;
+ struct pci_config_window *cfg;
+ struct pci_ecam_ops *ops;
+ int err;
+
+ if ( dt_device_for_passthrough(dev) )
+ return 0;
+
+ ops = (struct pci_ecam_ops *) data;
+
+ bridge = pci_alloc_host_bridge();
+ if ( !bridge )
+ return -ENOMEM;
+
+ /* Parse and map our Configuration Space windows */
+ cfg = gen_pci_init(dev, ops);
+ if ( !cfg )
+ {
+ err = -ENOMEM;
+ goto err_exit;
+ }
+
+ bridge->dt_node = dev;
+ bridge->cfg = cfg;
+ bridge->ops = &ops->pci_ops;
+
+ bridge->segment = pci_bus_find_domain_nr(dev);
+ if ( bridge->segment < 0 )
+ {
+ printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n");
+ BUG();
+ }
+ pci_add_host_bridge(bridge);
+
+ return 0;
+
+err_exit:
+ xfree(bridge);
+
+ return err;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
new file mode 100644
index 0000000000..33457fbe96
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-generic.c
@@ -0,0 +1,48 @@
+/*
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/device.h>
+#include <asm/pci.h>
+
+static const struct dt_device_match __initconstrel gen_pci_dt_match[] =
+{
+ { .compatible = "pci-host-ecam-generic" },
+ { },
+};
+
+static int __init pci_host_generic_probe(struct dt_device_node *dev,
+ const void *data)
+{
+ return pci_host_common_probe(dev, &pci_generic_ecam_ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = pci_host_generic_probe,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 0cf849e26f..bb7eda6705 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -26,6 +26,62 @@ struct arch_pci_dev {
struct device dev;
};
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+ paddr_t phys_addr;
+ paddr_t size;
+ uint8_t busn_start;
+ uint8_t busn_end;
+ void __iomem *win;
+};
+
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+ struct dt_device_node *dt_node; /* Pointer to the associated DT node */
+ struct list_head node; /* Node in list of host bridges */
+ uint16_t segment; /* Segment number */
+ struct pci_config_window* cfg; /* Pointer to the bridge config window */
+ struct pci_ops *ops;
+};
+
+struct pci_ops {
+ void __iomem *(*map_bus)(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
+ uint32_t offset);
+ int (*read)(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
+ uint32_t reg, uint32_t len, uint32_t *value);
+ int (*write)(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
+ uint32_t reg, uint32_t len, uint32_t value);
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+ unsigned int bus_shift;
+ struct pci_ops pci_ops;
+ int (*cfg_reg_index)(struct dt_device_node *dev);
+ int (*init)(struct pci_config_window *);
+};
+
+/* Default ECAM ops */
+extern const struct pci_ecam_ops pci_generic_ecam_ops;
+
+int pci_host_common_probe(struct dt_device_node *dev, const void *data);
+int pci_generic_config_read(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
+ uint32_t reg, uint32_t len, uint32_t *value);
+int pci_generic_config_write(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
+ uint32_t reg, uint32_t len, uint32_t value);
+void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
+ pci_sbdf_t sbdf, uint32_t where);
+
static always_inline bool is_pci_passthrough_enabled(void)
{
return pci_passthrough_enabled;
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 05/11] xen/arm: Add support for Xilinx ZynqMP PCI host controller
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (3 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 04/11] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-06 17:40 ` [PATCH v5 06/11] xen/arm: Implement pci access functions Rahul Singh
` (6 subsequent siblings)
11 siblings, 0 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara,
Oleksandr Andrushchenko, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk
From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Add support for Xilinx ZynqMP PCI host controller to map the PCI config
space to the XEN memory.
Patch helps to understand how the generic infrastructure for PCI
host-bridge discovery will be used for future references.
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
Change in v5:
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Change in v4:
- Add __initconstrel and __init for struct and functions
- Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Change in v3:
- nwl_cfg_reg_index(..) as static function
- Add support for pci_host_generic_probe()
Change in v2:
- Add more info in commit msg
---
---
xen/arch/arm/pci/Makefile | 1 +
xen/arch/arm/pci/pci-host-zynqmp.c | 65 ++++++++++++++++++++++++++++++
2 files changed, 66 insertions(+)
create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
index 6f32fbbe67..1d045ade01 100644
--- a/xen/arch/arm/pci/Makefile
+++ b/xen/arch/arm/pci/Makefile
@@ -3,3 +3,4 @@ obj-y += pci-access.o
obj-y += pci-host-generic.o
obj-y += pci-host-common.o
obj-y += ecam.o
+obj-y += pci-host-zynqmp.o
diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
new file mode 100644
index 0000000000..61a9807d3d
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-zynqmp.c
@@ -0,0 +1,65 @@
+/*
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Based on xen/arch/arm/pci/pci-host-generic.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/device.h>
+#include <asm/pci.h>
+
+static int __init nwl_cfg_reg_index(struct dt_device_node *np)
+{
+ return dt_property_match_string(np, "reg-names", "cfg");
+}
+
+/* ECAM ops */
+const struct pci_ecam_ops nwl_pcie_ops = {
+ .bus_shift = 20,
+ .cfg_reg_index = nwl_cfg_reg_index,
+ .pci_ops = {
+ .map_bus = pci_ecam_map_bus,
+ .read = pci_generic_config_read,
+ .write = pci_generic_config_write,
+ }
+};
+
+static const struct dt_device_match __initconstrel nwl_pcie_dt_match[] =
+{
+ { .compatible = "xlnx,nwl-pcie-2.11" },
+ { },
+};
+
+static int __init pci_host_generic_probe(struct dt_device_node *dev,
+ const void *data)
+{
+ return pci_host_common_probe(dev, &nwl_pcie_ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI)
+.dt_match = nwl_pcie_dt_match,
+.init = pci_host_generic_probe,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 06/11] xen/arm: Implement pci access functions
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (4 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 05/11] xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-06 17:40 ` [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag Rahul Singh
` (5 subsequent siblings)
11 siblings, 0 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk
Implement generic pci access functions to read/write the configuration
space.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
Change in v5: none
Change in v4:
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
- Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Change in v3:
- Remove PRI_pci as not used.
- Replace uint32_t sbdf to pci_sbdf_t sbdf to avoid typecast
Change in v2: Fixed comments
---
---
xen/arch/arm/pci/pci-access.c | 57 ++++++++++++++++++++++++++++++
xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++
xen/include/asm-arm/pci.h | 1 +
3 files changed, 77 insertions(+)
diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
index 3cd14a4b87..9f9aac43d7 100644
--- a/xen/arch/arm/pci/pci-access.c
+++ b/xen/arch/arm/pci/pci-access.c
@@ -16,6 +16,7 @@
#include <asm/io.h>
#define INVALID_VALUE (~0U)
+#define PCI_ERR_VALUE(len) GENMASK(0, len * 8)
int pci_generic_config_read(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
uint32_t reg, uint32_t len, uint32_t *value)
@@ -72,6 +73,62 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
return 0;
}
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+ unsigned int len)
+{
+ uint32_t val = PCI_ERR_VALUE(len);
+ struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+ if ( unlikely(!bridge) )
+ return val;
+
+ if ( unlikely(!bridge->ops->read) )
+ return val;
+
+ bridge->ops->read(bridge, sbdf, reg, len, &val);
+
+ return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+ unsigned int len, uint32_t val)
+{
+ struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+ if ( unlikely(!bridge) )
+ return;
+
+ if ( unlikely(!bridge->ops->write) )
+ return;
+
+ bridge->ops->write(bridge, sbdf, reg, len, val);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+ void pci_conf_write##size(pci_sbdf_t sbdf, \
+ unsigned int reg, type val) \
+{ \
+ pci_config_write(sbdf, reg, size / 8, val); \
+}
+
+#define PCI_OP_READ(size, type) \
+ type pci_conf_read##size(pci_sbdf_t sbdf, \
+ unsigned int reg) \
+{ \
+ return pci_config_read(sbdf, reg, size / 8); \
+}
+
+PCI_OP_READ(8, uint8_t)
+PCI_OP_READ(16, uint16_t)
+PCI_OP_READ(32, uint32_t)
+PCI_OP_WRITE(8, uint8_t)
+PCI_OP_WRITE(16, uint16_t)
+PCI_OP_WRITE(32, uint32_t)
+
/*
* Local variables:
* mode: C
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
index a08e06cea1..c5941b10e9 100644
--- a/xen/arch/arm/pci/pci-host-common.c
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -236,6 +236,25 @@ err_exit:
return err;
}
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+ struct pci_host_bridge *bridge;
+
+ list_for_each_entry( bridge, &pci_host_bridges, node )
+ {
+ if ( bridge->segment != segment )
+ continue;
+ if ( (bus < bridge->cfg->busn_start) || (bus > bridge->cfg->busn_end) )
+ continue;
+ return bridge;
+ }
+
+ return NULL;
+}
/*
* Local variables:
* mode: C
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index bb7eda6705..49c9622902 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -81,6 +81,7 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
uint32_t reg, uint32_t len, uint32_t value);
void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
pci_sbdf_t sbdf, uint32_t where);
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
static always_inline bool is_pci_passthrough_enabled(void)
{
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (5 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 06/11] xen/arm: Implement pci access functions Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-07 13:08 ` Jan Beulich
` (2 more replies)
2021-10-06 17:40 ` [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
` (4 subsequent siblings)
11 siblings, 3 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara, Christian Lindig,
David Scott, Ian Jackson, Wei Liu, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Volodymyr Babchuk,
Roger Pau Monné
Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
Reject the use of this new flag for x86 as VPCI is not supported for
DOMU guests for x86.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Acked-by: Christian Lindig <christian.lindig@citrix.com>
---
Change in v5:
- Added Acked-by: Christian Lindig <christian.lindig@citrix.com>
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Change in v4: Added in this version
---
---
tools/ocaml/libs/xc/xenctrl.ml | 1 +
tools/ocaml/libs/xc/xenctrl.mli | 1 +
xen/arch/arm/domain.c | 4 ++--
xen/arch/x86/domain.c | 6 ++++++
xen/common/domain.c | 2 +-
xen/include/public/domctl.h | 4 +++-
6 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/tools/ocaml/libs/xc/xenctrl.ml b/tools/ocaml/libs/xc/xenctrl.ml
index a5588c643f..7ed1c00e47 100644
--- a/tools/ocaml/libs/xc/xenctrl.ml
+++ b/tools/ocaml/libs/xc/xenctrl.ml
@@ -69,6 +69,7 @@ type domain_create_flag =
| CDF_XS_DOMAIN
| CDF_IOMMU
| CDF_NESTED_VIRT
+ | CDF_VPCI
type domain_create_iommu_opts =
| IOMMU_NO_SHAREPT
diff --git a/tools/ocaml/libs/xc/xenctrl.mli b/tools/ocaml/libs/xc/xenctrl.mli
index 6e94940a8a..391d4abdf8 100644
--- a/tools/ocaml/libs/xc/xenctrl.mli
+++ b/tools/ocaml/libs/xc/xenctrl.mli
@@ -62,6 +62,7 @@ type domain_create_flag =
| CDF_XS_DOMAIN
| CDF_IOMMU
| CDF_NESTED_VIRT
+ | CDF_VPCI
type domain_create_iommu_opts =
| IOMMU_NO_SHAREPT
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 19c756ac3d..36138c1b2e 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -622,8 +622,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
{
unsigned int max_vcpus;
- /* HVM and HAP must be set. IOMMU may or may not be */
- if ( (config->flags & ~XEN_DOMCTL_CDF_iommu) !=
+ /* HVM and HAP must be set. IOMMU and VPCI may or may not be */
+ if ( (config->flags & ~XEN_DOMCTL_CDF_iommu & ~XEN_DOMCTL_CDF_vpci) !=
(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap) )
{
dprintk(XENLOG_INFO, "Unsupported configuration %#x\n",
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index ef1812dc14..79c2aa4636 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -662,6 +662,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
return -EINVAL;
}
+ if ( config->flags & XEN_DOMCTL_CDF_vpci )
+ {
+ dprintk(XENLOG_INFO, "vPCI cannot be enabled yet\n");
+ return -EINVAL;
+ }
+
if ( config->vmtrace_size )
{
unsigned int size = config->vmtrace_size;
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 6ee5d033b0..40d67ec342 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -483,7 +483,7 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
- XEN_DOMCTL_CDF_nested_virt) )
+ XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpci) )
{
dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
return -EINVAL;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 96696e3842..4245da3f45 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -70,9 +70,11 @@ struct xen_domctl_createdomain {
#define XEN_DOMCTL_CDF_iommu (1U<<_XEN_DOMCTL_CDF_iommu)
#define _XEN_DOMCTL_CDF_nested_virt 6
#define XEN_DOMCTL_CDF_nested_virt (1U << _XEN_DOMCTL_CDF_nested_virt)
+#define _XEN_DOMCTL_CDF_vpci 7
+#define XEN_DOMCTL_CDF_vpci (1U << _XEN_DOMCTL_CDF_vpci)
/* Max XEN_DOMCTL_CDF_* constant. Used for ABI checking. */
-#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_nested_virt
+#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpci
uint32_t flags;
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (6 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-07 13:43 ` Jan Beulich
` (3 more replies)
2021-10-06 17:40 ` [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM Rahul Singh
` (3 subsequent siblings)
11 siblings, 4 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Ian Jackson, Jan Beulich, Wei Liu,
Paul Durrant, Roger Pau Monné
The existing VPCI support available for X86 is adapted for Arm.
When the device is added to XEN via the hyper call
“PHYSDEVOP_pci_device_add”, VPCI handler for the config space
access is added to the Xen to emulate the PCI devices config space.
A MMIO trap handler for the PCI ECAM space is registered in XEN
so that when guest is trying to access the PCI config space,XEN
will trap the access and emulate read/write using the VPCI and
not the real PCI hardware.
For Dom0less systems scan_pci_devices() would be used to discover the
PCI device in XEN and VPCI handler will be added during XEN boots.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
Change in v5:
- Add pci_cleanup_msi(pdev) in cleanup part.
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Change in v4:
- Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
Change in v3:
- Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
- Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
- Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
Change in v2:
- Add new XEN_DOMCTL_CDF_vpci flag
- modify has_vpci() to include XEN_DOMCTL_CDF_vpci
- enable vpci support when pci-passthough option is enabled.
---
---
xen/arch/arm/Makefile | 1 +
xen/arch/arm/domain.c | 4 ++
xen/arch/arm/domain_build.c | 3 +
xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
xen/arch/arm/vpci.h | 36 ++++++++++++
xen/drivers/passthrough/pci.c | 18 ++++++
xen/include/asm-arm/domain.h | 7 ++-
xen/include/asm-x86/pci.h | 2 -
xen/include/public/arch-arm.h | 7 +++
xen/include/xen/pci.h | 2 +
10 files changed, 179 insertions(+), 3 deletions(-)
create mode 100644 xen/arch/arm/vpci.c
create mode 100644 xen/arch/arm/vpci.h
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 44d7cc81fa..fb9c976ea2 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_HAS_VPCI) += vpci.o
obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 36138c1b2e..fbb52f78f1 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -39,6 +39,7 @@
#include <asm/vgic.h>
#include <asm/vtimer.h>
+#include "vpci.h"
#include "vuart.h"
DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
@@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
goto fail;
+ if ( (rc = domain_vpci_init(d)) != 0 )
+ goto fail;
+
return 0;
fail:
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index c5afbe2e05..f4c89bde8c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3053,6 +3053,9 @@ void __init create_dom0(void)
if ( iommu_enabled )
dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
+ if ( is_pci_passthrough_enabled() )
+ dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
+
dom0 = domain_create(0, &dom0_cfg, true);
if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
panic("Error creating domain 0\n");
diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
new file mode 100644
index 0000000000..76c12b9281
--- /dev/null
+++ b/xen/arch/arm/vpci.c
@@ -0,0 +1,102 @@
+/*
+ * xen/arch/arm/vpci.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <xen/sched.h>
+
+#include <asm/mmio.h>
+
+#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
+
+/* Do some sanity checks. */
+static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
+{
+ /* Check access size. */
+ if ( len > 8 )
+ return false;
+
+ /* Check that access is size aligned. */
+ if ( (reg & (len - 1)) )
+ return false;
+
+ return true;
+}
+
+static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
+ register_t *r, void *p)
+{
+ unsigned int reg;
+ pci_sbdf_t sbdf;
+ unsigned long data = ~0UL;
+ unsigned int size = 1U << info->dabt.size;
+
+ sbdf.sbdf = MMCFG_BDF(info->gpa);
+ reg = REGISTER_OFFSET(info->gpa);
+
+ if ( !vpci_mmio_access_allowed(reg, size) )
+ return 0;
+
+ data = vpci_read(sbdf, reg, min(4u, size));
+ if ( size == 8 )
+ data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
+
+ *r = data;
+
+ return 1;
+}
+
+static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
+ register_t r, void *p)
+{
+ unsigned int reg;
+ pci_sbdf_t sbdf;
+ unsigned long data = r;
+ unsigned int size = 1U << info->dabt.size;
+
+ sbdf.sbdf = MMCFG_BDF(info->gpa);
+ reg = REGISTER_OFFSET(info->gpa);
+
+ if ( !vpci_mmio_access_allowed(reg, size) )
+ return 0;
+
+ vpci_write(sbdf, reg, min(4u, size), data);
+ if ( size == 8 )
+ vpci_write(sbdf, reg + 4, 4, data >> 32);
+
+ return 1;
+}
+
+static const struct mmio_handler_ops vpci_mmio_handler = {
+ .read = vpci_mmio_read,
+ .write = vpci_mmio_write,
+};
+
+int domain_vpci_init(struct domain *d)
+{
+ if ( !has_vpci(d) )
+ return 0;
+
+ register_mmio_handler(d, &vpci_mmio_handler,
+ GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
+
+ return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
new file mode 100644
index 0000000000..d8a7b0e3e8
--- /dev/null
+++ b/xen/arch/arm/vpci.h
@@ -0,0 +1,36 @@
+/*
+ * xen/arch/arm/vpci.h
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPCI_H__
+#define __ARCH_ARM_VPCI_H__
+
+#ifdef CONFIG_HAS_VPCI
+int domain_vpci_init(struct domain *d);
+#else
+static inline int domain_vpci_init(struct domain *d)
+{
+ return 0;
+}
+#endif
+
+#endif /* __ARCH_ARM_VPCI_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index e1b735d9e8..e568457e60 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
else
iommu_enable_device(pdev);
+#ifdef CONFIG_ARM
+ /*
+ * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
+ * Dom0 inform XEN to add the PCI devices in XEN.
+ */
+ ret = vpci_add_handlers(pdev);
+ if ( ret )
+ {
+ printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
+ pci_cleanup_msi(pdev);
+ ret = iommu_remove_device(pdev);
+ if ( pdev->domain )
+ list_del(&pdev->domain_list);
+ free_pdev(pseg, pdev);
+ goto out;
+ }
+#endif
+
pci_enable_acs(pdev);
out:
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index c9277b5c6d..91d614b37e 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -2,6 +2,7 @@
#define __ASM_DOMAIN_H__
#include <xen/cache.h>
+#include <xen/nospec.h>
#include <xen/timer.h>
#include <asm/page.h>
#include <asm/p2m.h>
@@ -262,7 +263,11 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
#define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
-#define has_vpci(d) ({ (void)(d); false; })
+/*
+ * For X86 VPCI is enabled and tested for PVH DOM0 only but
+ * for ARM we enable support VPCI for guest domain also.
+ */
+#define has_vpci(d) evaluate_nospec((d)->options & XEN_DOMCTL_CDF_vpci)
#endif /* __ASM_DOMAIN_H__ */
diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
index e076951032..c4a4fdcbc2 100644
--- a/xen/include/asm-x86/pci.h
+++ b/xen/include/asm-x86/pci.h
@@ -6,8 +6,6 @@
#define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
#define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
-#define MMCFG_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
-
#define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
|| id == 0x01268086 || id == 0x01028086 \
|| id == 0x01128086 || id == 0x01228086 \
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index d46c61fca9..44be337dec 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
#define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
#define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
+/*
+ * 256 MB is reserved for VPCI configuration space based on calculation
+ * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
+ */
+#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
+#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
+
/* ACPI tables physical address */
#define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
#define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index 70ac25345c..43b8a08170 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -40,6 +40,8 @@
#define PCI_SBDF3(s,b,df) \
((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
+#define MMCFG_BDF(addr) (((addr) & 0x0ffff000) >> 12)
+
typedef union {
uint32_t sbdf;
struct {
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM.
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (7 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-11 11:43 ` Roger Pau Monné
2021-10-06 17:40 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
` (2 subsequent siblings)
11 siblings, 1 reply; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara,
Roger Pau Monné,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk
This patch will be reverted once we add support for VPCI MSI/MSIX
support on ARM.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Change in v5: none
Change in v4:
- Added Acked-by: Stefano Stabellini <sstabellini@kernel.org>
- Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Change in v3: none
Change in v2: Patch introduced in v2
---
---
xen/drivers/vpci/Makefile | 3 ++-
| 2 ++
xen/include/asm-arm/pci.h | 8 ++++++++
3 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
index 55d1bdfda0..1a1413b93e 100644
--- a/xen/drivers/vpci/Makefile
+++ b/xen/drivers/vpci/Makefile
@@ -1 +1,2 @@
-obj-y += vpci.o header.o msi.o msix.o
+obj-y += vpci.o header.o
+obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
--git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
index ba9a036202..f8cd55e7c0 100644
--- a/xen/drivers/vpci/header.c
+++ b/xen/drivers/vpci/header.c
@@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
* FIXME: punching holes after the p2m has been set up might be racy for
* DomU usage, needs to be revisited.
*/
+#ifdef CONFIG_HAS_PCI_MSI
if ( map && !rom_only && vpci_make_msix_hole(pdev) )
return;
+#endif
for ( i = 0; i < ARRAY_SIZE(header->bars); i++ )
{
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 49c9622902..5532ce3977 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -26,6 +26,14 @@ struct arch_pci_dev {
struct device dev;
};
+/* Arch-specific MSI data for vPCI. */
+struct vpci_arch_msi {
+};
+
+/* Arch-specific MSI-X entry data for vPCI. */
+struct vpci_arch_msix_entry {
+};
+
/*
* struct to hold the mappings of a config space window. This
* is expected to be used as sysdata for PCI controllers that
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (8 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-06 18:01 ` Julien Grall
2021-10-12 15:03 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Ian Jackson
2021-10-06 17:40 ` [PATCH v5 11/11] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
2021-10-07 19:54 ` [PATCH v5 00/11] PCI devices passthrough on Arm Stefano Stabellini
11 siblings, 2 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara, Ian Jackson,
Wei Liu, Anthony PERARD, Juergen Gross, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk
libxl will create an emulated PCI device tree node in the device tree to
enable the guest OS to discover the virtual PCI during guest boot.
Emulated PCI device tree node will only be created when there is any
device assigned to guest.
A new area has been reserved in the arm guest physical map at
which the VPCI bus is declared in the device tree (reg and ranges
parameters of the node).
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
Change in v5:
- Move setting the arch_arm.vpci and XEN_DOMCTL_CDF_vpci to libxl_arm.c
Change in v4:
- Gate code for x86 for setting the XEN_DOMCTL_CDF_vpci for x86.
Change in v3:
- Make GUEST_VPCI_MEM_ADDR address 2MB aligned
Change in v2:
- enable doamin_vpci_init() when XEN_DOMCTL_CDF_vpci is set for domain.
---
---
tools/include/libxl.h | 6 ++
tools/libs/light/libxl_arm.c | 111 +++++++++++++++++++++++++++++++
tools/libs/light/libxl_types.idl | 1 +
xen/include/public/arch-arm.h | 10 +++
4 files changed, 128 insertions(+)
diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index b9ba16d698..3362073b21 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -358,6 +358,12 @@
*/
#define LIBXL_HAVE_BUILDINFO_ARM_VUART 1
+/*
+ * LIBXL_HAVE_BUILDINFO_ARM_VPCI indicates that the toolstack supports virtual
+ * PCI for ARM.
+ */
+#define LIBXL_HAVE_BUILDINFO_ARM_VPCI 1
+
/*
* LIBXL_HAVE_BUILDINFO_GRANT_LIMITS indicates that libxl_domain_build_info
* has the max_grant_frames and max_maptrack_frames fields.
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index e3140a6e00..2be208b99b 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -101,6 +101,12 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
return ERROR_FAIL;
}
+ /* Enable VPCI support. */
+ if (d_config->num_pcidevs) {
+ config->flags |= XEN_DOMCTL_CDF_vpci;
+ libxl_defbool_set(&d_config->b_info.arch_arm.vpci, true);
+ }
+
return 0;
}
@@ -269,6 +275,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
return fdt_property(fdt, "reg", regs, sizeof(regs));
}
+static int fdt_property_values(libxl__gc *gc, void *fdt,
+ const char *name, unsigned num_cells, ...)
+{
+ uint32_t prop[num_cells];
+ be32 *cells = &prop[0];
+ int i;
+ va_list ap;
+ uint32_t arg;
+
+ va_start(ap, num_cells);
+ for (i = 0 ; i < num_cells; i++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1, arg);
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, name, prop, sizeof(prop));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+ unsigned addr_cells,
+ unsigned size_cells,
+ unsigned num_regs, ...)
+{
+ uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
+ be32 *cells = ®s[0];
+ int i;
+ va_list ap;
+ uint64_t arg;
+
+ va_start(ap, num_regs);
+ for (i = 0 ; i < num_regs; i++) {
+ /* Set the memory bit field */
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1, arg);
+
+ /* Set the vpci bus address */
+ arg = addr_cells ? va_arg(ap, uint64_t) : 0;
+ set_cell(&cells, addr_cells , arg);
+
+ /* Set the cpu bus address where vpci address is mapped */
+ set_cell(&cells, addr_cells, arg);
+
+ /* Set the vpci size requested */
+ arg = size_cells ? va_arg(ap, uint64_t) : 0;
+ set_cell(&cells, size_cells, arg);
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
static int make_root_properties(libxl__gc *gc,
const libxl_version_info *vers,
void *fdt)
@@ -668,6 +726,53 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
return 0;
}
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+ const struct arch_info *ainfo,
+ struct xc_dom_image *dom)
+{
+ int res;
+ const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+ const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+ const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+ res = fdt_begin_node(fdt, name);
+ if (res) return res;
+
+ res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+ if (res) return res;
+
+ res = fdt_property_string(fdt, "device_type", "pci");
+ if (res) return res;
+
+ res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+ GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+ if (res) return res;
+
+ res = fdt_property_values(gc, fdt, "bus-range", 2, 0, 255);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#address-cells", 3);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#size-cells", 2);
+ if (res) return res;
+
+ res = fdt_property_string(fdt, "status", "okay");
+ if (res) return res;
+
+ res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+ GUEST_ROOT_SIZE_CELLS, 2,
+ GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE,
+ GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR,
+ GUEST_VPCI_PREFETCH_MEM_SIZE);
+ if (res) return res;
+
+ res = fdt_end_node(fdt);
+ if (res) return res;
+
+ return 0;
+}
+
static const struct arch_info *get_arch_info(libxl__gc *gc,
const struct xc_dom_image *dom)
{
@@ -971,6 +1076,9 @@ next_resize:
if (info->tee == LIBXL_TEE_TYPE_OPTEE)
FDT( make_optee_node(gc, fdt) );
+ if (libxl_defbool_val(info->arch_arm.vpci))
+ FDT( make_vpci_node(gc, fdt, ainfo, dom) );
+
if (pfdt)
FDT( copy_partial_fdt(gc, fdt, pfdt) );
@@ -1189,6 +1297,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
/* ACPI is disabled by default */
libxl_defbool_setdefault(&b_info->acpi, false);
+ /* VPCI is disabled by default */
+ libxl_defbool_setdefault(&b_info->arch_arm.vpci, false);
+
if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
return;
diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index 3f9fff653a..78b1ddf0b8 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
("vuart", libxl_vuart_type),
+ ("vpci", libxl_defbool),
])),
("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
])),
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 44be337dec..45aac5d18f 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -433,6 +433,11 @@ typedef uint64_t xen_callback_t;
#define GUEST_PL011_BASE xen_mk_ullong(0x22000000)
#define GUEST_PL011_SIZE xen_mk_ullong(0x00001000)
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000)
+#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x23000000)
+#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x10000000)
+
/*
* 16MB == 4096 pages reserved for guest to use as a region to map its
* grant table in.
@@ -448,6 +453,11 @@ typedef uint64_t xen_callback_t;
#define GUEST_RAM0_BASE xen_mk_ullong(0x40000000) /* 3GB of low RAM @ 1GB */
#define GUEST_RAM0_SIZE xen_mk_ullong(0xc0000000)
+/* 4GB @ 4GB Prefetch Memory for VPCI */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x100000000)
+#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000)
+
#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 1016GB of RAM @ 8GB */
#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000)
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v5 11/11] xen/arm: Add linux,pci-domain property for hwdom if not available.
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (9 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
@ 2021-10-06 17:40 ` Rahul Singh
2021-10-13 20:54 ` Stefano Stabellini
2021-10-07 19:54 ` [PATCH v5 00/11] PCI devices passthrough on Arm Stefano Stabellini
11 siblings, 1 reply; 190+ messages in thread
From: Rahul Singh @ 2021-10-06 17:40 UTC (permalink / raw)
To: xen-devel
Cc: bertrand.marquis, rahul.singh, Andre.Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk
If the property is not present in the device tree node for host bridge,
XEN while creating the dtb for hwdom will create this property and
assigns the already allocated segment to the host bridge
so that XEN and linux will have the same segment for the host bridges.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
Change in v5: none
Change in v4:
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
- Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Change in v3:
- Use is_pci_passthrough_enabled()
Change in v2:
- Add linux,pci-domain only when pci-passthrough command line option is enabeld
---
---
xen/arch/arm/domain_build.c | 16 ++++++++++++++++
xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++
xen/include/asm-arm/pci.h | 9 +++++++++
3 files changed, 46 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index f4c89bde8c..8739e3285e 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -747,6 +747,22 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
return res;
}
+ if ( is_pci_passthrough_enabled() && dt_device_type_is_equal(node, "pci") )
+ {
+ if ( !dt_find_property(node, "linux,pci-domain", NULL) )
+ {
+ uint16_t segment;
+
+ res = pci_get_host_bridge_segment(node, &segment);
+ if ( res < 0 )
+ return res;
+
+ res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment);
+ if ( res )
+ return res;
+ }
+ }
+
/*
* Override the property "status" to disable the device when it's
* marked for passthrough.
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
index c5941b10e9..593beeb48c 100644
--- a/xen/arch/arm/pci/pci-host-common.c
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -255,6 +255,27 @@ struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
return NULL;
}
+
+/*
+ * This function will lookup an hostbridge based on config space address.
+ */
+int pci_get_host_bridge_segment(const struct dt_device_node *node,
+ uint16_t *segment)
+{
+ struct pci_host_bridge *bridge;
+
+ list_for_each_entry( bridge, &pci_host_bridges, node )
+ {
+ if ( bridge->dt_node != node )
+ continue;
+
+ *segment = bridge->segment;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
/*
* Local variables:
* mode: C
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 5532ce3977..7cb2e2f1ed 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -90,6 +90,8 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
pci_sbdf_t sbdf, uint32_t where);
struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
+int pci_get_host_bridge_segment(const struct dt_device_node *node,
+ uint16_t *segment);
static always_inline bool is_pci_passthrough_enabled(void)
{
@@ -104,5 +106,12 @@ static always_inline bool is_pci_passthrough_enabled(void)
return false;
}
+static inline int pci_get_host_bridge_segment(const struct dt_device_node *node,
+ uint16_t *segment)
+{
+ ASSERT_UNREACHABLE();
+ return -EINVAL;
+}
+
#endif /*!CONFIG_HAS_PCI*/
#endif /* __ARM_PCI_H__ */
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-06 17:40 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
@ 2021-10-06 18:01 ` Julien Grall
2021-10-07 0:26 ` Stefano Stabellini
2021-10-07 10:53 ` Ian Jackson
2021-10-12 15:03 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Ian Jackson
1 sibling, 2 replies; 190+ messages in thread
From: Julien Grall @ 2021-10-06 18:01 UTC (permalink / raw)
To: Rahul Singh, xen-devel
Cc: bertrand.marquis, Andre.Przywara, Ian Jackson, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk
Hi Rahul,
On 06/10/2021 19:40, Rahul Singh wrote:
> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> index 3f9fff653a..78b1ddf0b8 100644
> --- a/tools/libs/light/libxl_types.idl
> +++ b/tools/libs/light/libxl_types.idl
> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>
> ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
> ("vuart", libxl_vuart_type),
> + ("vpci", libxl_defbool),
I have posted some comments regarding the field in v4. To summarize,
AFAICT, this option is meant to be only set by libxl but you still let
the toolstack (e.g. xl, libvirt) to set it.
If you still want to expose to the toolstack, then I think the option
should be outside of arch_arm. Otherwise, this should be moved in an
internal structure (Ian, do you have any suggestion?).
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM
2021-10-06 17:40 ` [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM Rahul Singh
@ 2021-10-07 0:05 ` Stefano Stabellini
2021-10-07 12:58 ` Jan Beulich
2021-10-21 9:28 ` xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM) Julien Grall
1 sibling, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-07 0:05 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Roger Pau Monné
On Wed, 6 Oct 2021, Rahul Singh wrote:
> Hardware domain is in charge of doing the PCI enumeration and will
> discover the PCI devices and then will communicate to XEN via hyper
> call PHYSDEVOP_pci_device_add(..) to add the PCI devices in XEN.
>
> Also implement PHYSDEVOP_pci_device_remove(..) to remove the PCI device.
>
> As most of the code for PHYSDEVOP_pci_device_* is the same between x86
> and ARM, move the code to a common file to avoid duplication.
>
> There are other PHYSDEVOP_pci_device_* operations to add PCI devices.
> Currently implemented PHYSDEVOP_pci_device_remove(..) and
> PHYSDEVOP_pci_device_add(..) only as those are minimum required to
> support PCI passthrough on ARM.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> Change in v5:
> - Move the pci_physdev_op() stub to xen/arch/arm/physdev.c.
> Change in v4:
> - Move file commom/physdev.c to drivers/pci/physdev.c
> - minor comments.
> Change in v3: Fixed minor comment.
> Change in v2:
> - Add support for PHYSDEVOP_pci_device_remove()
> - Move code to common code
> ---
> ---
> xen/arch/arm/physdev.c | 6 ++-
> xen/arch/x86/physdev.c | 52 +----------------------
> xen/arch/x86/x86_64/physdev.c | 2 +-
> xen/drivers/pci/Makefile | 1 +
> xen/drivers/pci/physdev.c | 80 +++++++++++++++++++++++++++++++++++
> xen/include/public/arch-arm.h | 4 +-
> xen/include/xen/hypercall.h | 4 ++
> 7 files changed, 96 insertions(+), 53 deletions(-)
> create mode 100644 xen/drivers/pci/physdev.c
>
> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
> index e91355fe22..f9aa274dda 100644
> --- a/xen/arch/arm/physdev.c
> +++ b/xen/arch/arm/physdev.c
> @@ -8,13 +8,17 @@
> #include <xen/lib.h>
> #include <xen/errno.h>
> #include <xen/sched.h>
> -#include <asm/hypercall.h>
> +#include <xen/hypercall.h>
>
>
> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> {
> +#ifdef CONFIG_HAS_PCI
> + return pci_physdev_op(cmd, arg);
> +#else
> gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> return -ENOSYS;
> +#endif
> }
>
> /*
> diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
> index 23465bcd00..ea38be8b79 100644
> --- a/xen/arch/x86/physdev.c
> +++ b/xen/arch/x86/physdev.c
> @@ -12,7 +12,7 @@
> #include <asm/io_apic.h>
> #include <asm/msi.h>
> #include <asm/hvm/irq.h>
> -#include <asm/hypercall.h>
> +#include <xen/hypercall.h>
> #include <public/xen.h>
> #include <public/physdev.h>
> #include <xsm/xsm.h>
> @@ -480,54 +480,6 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> break;
> }
>
> - case PHYSDEVOP_pci_device_add: {
> - struct physdev_pci_device_add add;
> - struct pci_dev_info pdev_info;
> - nodeid_t node;
> -
> - ret = -EFAULT;
> - if ( copy_from_guest(&add, arg, 1) != 0 )
> - break;
> -
> - pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
> - if ( add.flags & XEN_PCI_DEV_VIRTFN )
> - {
> - pdev_info.is_virtfn = 1;
> - pdev_info.physfn.bus = add.physfn.bus;
> - pdev_info.physfn.devfn = add.physfn.devfn;
> - }
> - else
> - pdev_info.is_virtfn = 0;
> -
> - if ( add.flags & XEN_PCI_DEV_PXM )
> - {
> - uint32_t pxm;
> - size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
> - sizeof(add.optarr[0]);
> -
> - if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
> - break;
> -
> - node = pxm_to_node(pxm);
> - }
> - else
> - node = NUMA_NO_NODE;
> -
> - ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
> - break;
> - }
> -
> - case PHYSDEVOP_pci_device_remove: {
> - struct physdev_pci_device dev;
> -
> - ret = -EFAULT;
> - if ( copy_from_guest(&dev, arg, 1) != 0 )
> - break;
> -
> - ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
> - break;
> - }
> -
> case PHYSDEVOP_prepare_msix:
> case PHYSDEVOP_release_msix: {
> struct physdev_pci_device dev;
> @@ -663,7 +615,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> }
>
> default:
> - ret = -ENOSYS;
> + ret = pci_physdev_op(cmd, arg);
> break;
> }
>
> diff --git a/xen/arch/x86/x86_64/physdev.c b/xen/arch/x86/x86_64/physdev.c
> index 0a50cbd4d8..e3cbd5ebcb 100644
> --- a/xen/arch/x86/x86_64/physdev.c
> +++ b/xen/arch/x86/x86_64/physdev.c
> @@ -9,7 +9,7 @@ EMIT_FILE;
> #include <compat/xen.h>
> #include <compat/event_channel.h>
> #include <compat/physdev.h>
> -#include <asm/hypercall.h>
> +#include <xen/hypercall.h>
>
> #define do_physdev_op compat_physdev_op
>
> diff --git a/xen/drivers/pci/Makefile b/xen/drivers/pci/Makefile
> index a98035df4c..972c923db0 100644
> --- a/xen/drivers/pci/Makefile
> +++ b/xen/drivers/pci/Makefile
> @@ -1 +1,2 @@
> obj-y += pci.o
> +obj-y += physdev.o
> diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
> new file mode 100644
> index 0000000000..4f3e1a96c0
> --- /dev/null
> +++ b/xen/drivers/pci/physdev.c
> @@ -0,0 +1,80 @@
> +
> +#include <xen/guest_access.h>
> +#include <xen/hypercall.h>
> +#include <xen/init.h>
> +
> +#ifndef COMPAT
> +typedef long ret_t;
> +#endif
> +
> +ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> +{
> + ret_t ret;
> +
> + switch ( cmd )
> + {
> + case PHYSDEVOP_pci_device_add: {
> + struct physdev_pci_device_add add;
> + struct pci_dev_info pdev_info;
> + nodeid_t node = NUMA_NO_NODE;
> +
> + ret = -EFAULT;
> + if ( copy_from_guest(&add, arg, 1) != 0 )
> + break;
> +
> + pdev_info.is_extfn = (add.flags & XEN_PCI_DEV_EXTFN);
> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
> + {
> + pdev_info.is_virtfn = true;
> + pdev_info.physfn.bus = add.physfn.bus;
> + pdev_info.physfn.devfn = add.physfn.devfn;
> + }
> + else
> + pdev_info.is_virtfn = false;
> +
> +#ifdef CONFIG_NUMA
> + if ( add.flags & XEN_PCI_DEV_PXM )
> + {
> + uint32_t pxm;
> + size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
> + sizeof(add.optarr[0]);
> +
> + if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
> + break;
> +
> + node = pxm_to_node(pxm);
> + }
> +#endif
> +
> + ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
> + break;
> + }
> +
> + case PHYSDEVOP_pci_device_remove: {
> + struct physdev_pci_device dev;
> +
> + ret = -EFAULT;
> + if ( copy_from_guest(&dev, arg, 1) != 0 )
> + break;
> +
> + ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
> + break;
> + }
> +
> + default:
> + ret = -ENOSYS;
> + break;
> + }
> +
> + return ret;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 6b5a5f818a..d46c61fca9 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -107,7 +107,9 @@
> * All generic sub-operations
> *
> * HYPERVISOR_physdev_op
> - * No sub-operations are currenty supported
> + * Exactly these sub-operations are supported:
> + * PHYSDEVOP_pci_device_add
> + * PHYSDEVOP_pci_device_remove
> *
> * HYPERVISOR_sysctl
> * All generic sub-operations, with the exception of:
> diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
> index 3771487a30..07b10ec230 100644
> --- a/xen/include/xen/hypercall.h
> +++ b/xen/include/xen/hypercall.h
> @@ -45,6 +45,10 @@ extern long
> do_platform_op(
> XEN_GUEST_HANDLE_PARAM(xen_platform_op_t) u_xenpf_op);
>
> +extern long
> +pci_physdev_op(
> + int cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
> +
> /*
> * To allow safe resume of do_memory_op() after preemption, we need to know
> * at what point in the page list to resume. For this purpose I steal the
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-06 18:01 ` Julien Grall
@ 2021-10-07 0:26 ` Stefano Stabellini
2021-10-07 15:31 ` Rahul Singh
2021-10-07 10:53 ` Ian Jackson
1 sibling, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-07 0:26 UTC (permalink / raw)
To: Julien Grall, iwj
Cc: Rahul Singh, xen-devel, bertrand.marquis, Andre.Przywara,
Wei Liu, Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk
On Wed, 6 Oct 2021, Julien Grall wrote:
> Hi Rahul,
>
> On 06/10/2021 19:40, Rahul Singh wrote:
> > diff --git a/tools/libs/light/libxl_types.idl
> > b/tools/libs/light/libxl_types.idl
> > index 3f9fff653a..78b1ddf0b8 100644
> > --- a/tools/libs/light/libxl_types.idl
> > +++ b/tools/libs/light/libxl_types.idl
> > @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
> > ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
> > ("vuart", libxl_vuart_type),
> > + ("vpci", libxl_defbool),
>
> I have posted some comments regarding the field in v4. To summarize, AFAICT,
> this option is meant to be only set by libxl but you still let the toolstack
> (e.g. xl, libvirt) to set it.
>
> If you still want to expose to the toolstack, then I think the option should
> be outside of arch_arm. Otherwise, this should be moved in an internal
> structure (Ian, do you have any suggestion?).
First let me premise that the patch is much better already and Rahul
addessed my request well. However, Julien's point about not wanting to
make a potentially breaking ABI change in libxl is a good one. FYI we
had a few libvirt breakages due to this kind of changes in the past and
I would certainly be happier if we didn't cause another one. And in
general, it is better to avoid changes to the libxl ABI if we can.
I think in this case we can: I looked at the patch and
b_info.arch_arm.vpci is only used within tools/libs/light/libxl_arm.c.
Also, we don't need b_info.arch_arm.vpci if we can access
d_config->num_pcidevs given that the check is just based on
d_config->num_pcidevs.
So the only issue is how to check on d_config->num_pcidevs in
libxl__prepare_dtb. libxl__prepare_dtb takes libxl_domain_build_info as
parameter but with container_of we can retrieve libxl_domain_config and
from there check on num_pcidevs.
Something like the appended (untested). It doesn't need any libxl struct
changes but it requires the introduction of container_of (which is a
simple macro). Alternatively, it would be just as simple to change
libxl__arch_domain_init_hw_description and libxl__prepare_dtb to take a
libxl_domain_config *d_config parameter instead of a
libxl_domain_build_info *info parameter.
Ian, any thoughts?
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index 2be208b99b..ee1176519c 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -102,10 +102,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
}
/* Enable VPCI support. */
- if (d_config->num_pcidevs) {
+ if (d_config->num_pcidevs)
config->flags |= XEN_DOMCTL_CDF_vpci;
- libxl_defbool_set(&d_config->b_info.arch_arm.vpci, true);
- }
return 0;
}
@@ -976,6 +974,7 @@ static int libxl__prepare_dtb(libxl__gc *gc, libxl_domain_build_info *info,
const libxl_version_info *vers;
const struct arch_info *ainfo;
+ libxl_domain_config *d_config = container_of(info, libxl_domain_config, b_info);
vers = libxl_get_version_info(CTX);
if (vers == NULL) return ERROR_FAIL;
@@ -1076,7 +1075,7 @@ next_resize:
if (info->tee == LIBXL_TEE_TYPE_OPTEE)
FDT( make_optee_node(gc, fdt) );
- if (libxl_defbool_val(info->arch_arm.vpci))
+ if (d_config->num_pcidevs)
FDT( make_vpci_node(gc, fdt, ainfo, dom) );
if (pfdt)
^ permalink raw reply related [flat|nested] 190+ messages in thread
* Re: [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
2021-10-06 17:40 ` [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>" Rahul Singh
@ 2021-10-07 8:27 ` Jan Beulich
2021-10-07 8:32 ` Rahul Singh
2021-10-07 12:59 ` Jan Beulich
1 sibling, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-07 8:27 UTC (permalink / raw)
To: Rahul Singh
Cc: bertrand.marquis, Andre.Przywara, Andrew Cooper, George Dunlap,
Ian Jackson, Julien Grall, Stefano Stabellini, Wei Liu,
Volodymyr Babchuk, Roger Pau Monné,
xen-devel
On 06.10.2021 19:40, Rahul Singh wrote:
> @@ -62,8 +63,19 @@ static int __init acpi_pci_init(void)
> }
> #endif
>
> +/* By default pci passthrough is disabled. */
> +bool __read_mostly pci_passthrough_enabled;
You properly use "bool " here. Why ...
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -19,14 +19,25 @@
>
> #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
>
> +extern bool_t pci_passthrough_enabled;
... the being phased out "bool_t" here?
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
2021-10-07 8:27 ` Jan Beulich
@ 2021-10-07 8:32 ` Rahul Singh
0 siblings, 0 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-07 8:32 UTC (permalink / raw)
To: Jan Beulich
Cc: Bertrand Marquis, Andre Przywara, Andrew Cooper, George Dunlap,
Ian Jackson, Julien Grall, Stefano Stabellini, Wei Liu,
Volodymyr Babchuk, Roger Pau Monné,
xen-devel
Hi Jan,
> On 7 Oct 2021, at 9:27 am, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 06.10.2021 19:40, Rahul Singh wrote:
>> @@ -62,8 +63,19 @@ static int __init acpi_pci_init(void)
>> }
>> #endif
>>
>> +/* By default pci passthrough is disabled. */
>> +bool __read_mostly pci_passthrough_enabled;
>
> You properly use "bool " here. Why ...
>
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -19,14 +19,25 @@
>>
>> #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
>>
>> +extern bool_t pci_passthrough_enabled;
>
> ... the being phased out "bool_t" here?
I missed that I will fix this.
Regards,
Rahul
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-06 18:01 ` Julien Grall
2021-10-07 0:26 ` Stefano Stabellini
@ 2021-10-07 10:53 ` Ian Jackson
2021-10-07 15:29 ` Rahul Singh
1 sibling, 1 reply; 190+ messages in thread
From: Ian Jackson @ 2021-10-07 10:53 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, xen-devel, bertrand.marquis, Andre.Przywara,
Wei Liu, Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk
Julien Grall writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
> On 06/10/2021 19:40, Rahul Singh wrote:
> > diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> > index 3f9fff653a..78b1ddf0b8 100644
> > --- a/tools/libs/light/libxl_types.idl
> > +++ b/tools/libs/light/libxl_types.idl
> > @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
> >
> > ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
> > ("vuart", libxl_vuart_type),
> > + ("vpci", libxl_defbool),
>
> I have posted some comments regarding the field in v4. To summarize,
> AFAICT, this option is meant to be only set by libxl but you still let
> the toolstack (e.g. xl, libvirt) to set it.
>
> If you still want to expose to the toolstack, then I think the option
> should be outside of arch_arm. Otherwise, this should be moved in an
> internal structure (Ian, do you have any suggestion?).
If it should be in an internal structure, probably the libxl create
context.
But I'm not convinced yet. In particular, if enabling VPCI is
necessary on ARM for hotplugged PCI devices[1], then there has to be
a way for the admin to say "while this domain may not have any PCI
devices right now, I may wish to hotplug some". That's what the
"passthrough=" option is for.
See my other mail.
[1] I think this is all true even if PCI hotplug for ARM is not
currently implemented.
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM
2021-10-07 0:05 ` Stefano Stabellini
@ 2021-10-07 12:58 ` Jan Beulich
0 siblings, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-07 12:58 UTC (permalink / raw)
To: Stefano Stabellini, Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Roger Pau Monné
On 07.10.2021 02:05, Stefano Stabellini wrote:
> On Wed, 6 Oct 2021, Rahul Singh wrote:
>> Hardware domain is in charge of doing the PCI enumeration and will
>> discover the PCI devices and then will communicate to XEN via hyper
>> call PHYSDEVOP_pci_device_add(..) to add the PCI devices in XEN.
>>
>> Also implement PHYSDEVOP_pci_device_remove(..) to remove the PCI device.
>>
>> As most of the code for PHYSDEVOP_pci_device_* is the same between x86
>> and ARM, move the code to a common file to avoid duplication.
>>
>> There are other PHYSDEVOP_pci_device_* operations to add PCI devices.
>> Currently implemented PHYSDEVOP_pci_device_remove(..) and
>> PHYSDEVOP_pci_device_add(..) only as those are minimum required to
>> support PCI passthrough on ARM.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Acked-by: Jan Beulich <jbeulich@suse.com>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
2021-10-06 17:40 ` [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>" Rahul Singh
2021-10-07 8:27 ` Jan Beulich
@ 2021-10-07 12:59 ` Jan Beulich
1 sibling, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-07 12:59 UTC (permalink / raw)
To: Rahul Singh
Cc: bertrand.marquis, Andre.Przywara, Andrew Cooper, George Dunlap,
Ian Jackson, Julien Grall, Stefano Stabellini, Wei Liu,
Volodymyr Babchuk, Roger Pau Monné,
xen-devel
On 06.10.2021 19:40, Rahul Singh wrote:
> Add cmdline boot option "pci-passthrough = = <boolean>" to enable or
> disable the PCI passthrough support on ARM.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-06 17:40 ` [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag Rahul Singh
@ 2021-10-07 13:08 ` Jan Beulich
2021-10-08 18:06 ` Andrew Cooper
2021-10-11 9:27 ` Roger Pau Monné
2 siblings, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-07 13:08 UTC (permalink / raw)
To: Rahul Singh
Cc: bertrand.marquis, Andre.Przywara, Christian Lindig, David Scott,
Ian Jackson, Wei Liu, Andrew Cooper, George Dunlap, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk, Roger Pau Monné,
xen-devel
On 06.10.2021 19:40, Rahul Singh wrote:
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -70,9 +70,11 @@ struct xen_domctl_createdomain {
> #define XEN_DOMCTL_CDF_iommu (1U<<_XEN_DOMCTL_CDF_iommu)
> #define _XEN_DOMCTL_CDF_nested_virt 6
> #define XEN_DOMCTL_CDF_nested_virt (1U << _XEN_DOMCTL_CDF_nested_virt)
> +#define _XEN_DOMCTL_CDF_vpci 7
> +#define XEN_DOMCTL_CDF_vpci (1U << _XEN_DOMCTL_CDF_vpci)
Like said in [1] here I similarly don't see the need for two constants.
Preferably with the former of the two dropped
Acked-by: Jan Beulich <jbeulich@suse.com>
Jan
[1] https://lists.xen.org/archives/html/xen-devel/2021-10/msg00266.html
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-06 17:40 ` [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
@ 2021-10-07 13:43 ` Jan Beulich
2021-10-11 12:41 ` Bertrand Marquis
2021-10-11 10:51 ` Roger Pau Monné
` (2 subsequent siblings)
3 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-07 13:43 UTC (permalink / raw)
To: Rahul Singh
Cc: bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 06.10.2021 19:40, Rahul Singh wrote:
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,102 @@
> +/*
> + * xen/arch/arm/vpci.c
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +
> +#include <asm/mmio.h>
> +
> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
Nit: Stray blank (like you had in an earlier version for MMCFG_BDF()).
Also isn't this effectively part of the public interface (along with
MMCFG_BDF()), alongside GUEST_VPCI_ECAM_{BASE,SIZE}?
> +/* Do some sanity checks. */
> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> +{
> + /* Check access size. */
> + if ( len > 8 )
> + return false;
struct hsr_dabt's size field doesn't allow len to be above 8. I could
see that you may want to sanity check things, but that's not helpful
if done incompletely. Elsewhere you assume the value to be non-zero,
and ...
> + /* Check that access is size aligned. */
> + if ( (reg & (len - 1)) )
... right here you assume the value to be a power of 2. While I'm not
a maintainer, I'd still like to suggest consistency: Do all pertinent
checks or none of them (relying on the caller).
Independent of this - is bare metal Arm enforcing this same
alignment restriction (unconditionally)? Iirc on x86 we felt we'd
better synthesize misaligned accesses.
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> + register_t *r, void *p)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + unsigned long data = ~0UL;
What use is this initializer? The error path further down doesn't
forward the value into *r, and subsequently the value gets fully
overwritten.
> + unsigned int size = 1U << info->dabt.size;
> +
> + sbdf.sbdf = MMCFG_BDF(info->gpa);
This implies segment to be zero. While probably fine for now, I
wonder if this wouldn't warrant a comment.
> + reg = REGISTER_OFFSET(info->gpa);
> +
> + if ( !vpci_mmio_access_allowed(reg, size) )
> + return 0;
> +
> + data = vpci_read(sbdf, reg, min(4u, size));
> + if ( size == 8 )
> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
Throughout this series I haven't been able to spot where the HAS_VPCI
Kconfig symbol would get selected. Hence I cannot tell whether all of
this is Arm64-specific. Otherwise I wonder whether size 8 actually
can occur on Arm32.
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> + register_t r, void *p)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + unsigned long data = r;
A little like in the read function - what use is this local variable?
Can't you use r directly?
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> else
> iommu_enable_device(pdev);
Please note the context above; ...
> +#ifdef CONFIG_ARM
> + /*
> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> + * Dom0 inform XEN to add the PCI devices in XEN.
> + */
> + ret = vpci_add_handlers(pdev);
> + if ( ret )
> + {
> + printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
> + pci_cleanup_msi(pdev);
> + ret = iommu_remove_device(pdev);
> + if ( pdev->domain )
> + list_del(&pdev->domain_list);
> + free_pdev(pseg, pdev);
... you unconditionally undo the if() part of the earlier conditional;
is there any guarantee that the other path can't ever be taken, now
and forever? If it can't be taken now (which I think is the case as
long as Dom0 wouldn't report the same device twice), then at least some
precaution wants taking. Maybe moving your addition into that if()
could be an option.
Furthermore I continue to wonder whether this ordering is indeed
preferable over doing software setup before hardware arrangements. This
would address the above issue as well as long as vpci_add_handlers() is
idempotent. And it would likely simplify error cleanup.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-07 10:53 ` Ian Jackson
@ 2021-10-07 15:29 ` Rahul Singh
2021-10-07 16:11 ` Ian Jackson
0 siblings, 1 reply; 190+ messages in thread
From: Rahul Singh @ 2021-10-07 15:29 UTC (permalink / raw)
To: Ian Jackson
Cc: Julien Grall, xen-devel, Bertrand Marquis, Andre Przywara,
Wei Liu, Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk
Hi Ian,
> On 7 Oct 2021, at 11:53 am, Ian Jackson <iwj@xenproject.org> wrote:
>
> Julien Grall writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
>> On 06/10/2021 19:40, Rahul Singh wrote:
>>> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
>>> index 3f9fff653a..78b1ddf0b8 100644
>>> --- a/tools/libs/light/libxl_types.idl
>>> +++ b/tools/libs/light/libxl_types.idl
>>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>>>
>>> ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>>> ("vuart", libxl_vuart_type),
>>> + ("vpci", libxl_defbool),
>>
>> I have posted some comments regarding the field in v4. To summarize,
>> AFAICT, this option is meant to be only set by libxl but you still let
>> the toolstack (e.g. xl, libvirt) to set it.
>>
>> If you still want to expose to the toolstack, then I think the option
>> should be outside of arch_arm. Otherwise, this should be moved in an
>> internal structure (Ian, do you have any suggestion?).
>
> If it should be in an internal structure, probably the libxl create
> context.
As Stefano suggested in another email that we can remove the vpci option, if we
reach to conclusion that we need vpci option I will move it to internal structure.
>
> But I'm not convinced yet. In particular, if enabling VPCI is
> necessary on ARM for hotplugged PCI devices[1], then there has to be
> a way for the admin to say "while this domain may not have any PCI
> devices right now, I may wish to hotplug some". That's what the
> "passthrough=" option is for.
Yes I agree with you VPCI is necessary for hot plugged PCI device and once we
implement the hotplug in future we will use the passthrough= option to enable VPCI.
Regards,
Rahul
>
> See my other mail.
>
> [1] I think this is all true even if PCI hotplug for ARM is not
> currently implemented.
>
> Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-07 0:26 ` Stefano Stabellini
@ 2021-10-07 15:31 ` Rahul Singh
0 siblings, 0 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-07 15:31 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, Ian Jackson, xen-devel, Bertrand Marquis,
Andre Przywara, Wei Liu, Anthony PERARD, Juergen Gross,
Volodymyr Babchuk
Hi Stefano,
> On 7 Oct 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Wed, 6 Oct 2021, Julien Grall wrote:
>> Hi Rahul,
>>
>> On 06/10/2021 19:40, Rahul Singh wrote:
>>> diff --git a/tools/libs/light/libxl_types.idl
>>> b/tools/libs/light/libxl_types.idl
>>> index 3f9fff653a..78b1ddf0b8 100644
>>> --- a/tools/libs/light/libxl_types.idl
>>> +++ b/tools/libs/light/libxl_types.idl
>>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>>> ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>>> ("vuart", libxl_vuart_type),
>>> + ("vpci", libxl_defbool),
>>
>> I have posted some comments regarding the field in v4. To summarize, AFAICT,
>> this option is meant to be only set by libxl but you still let the toolstack
>> (e.g. xl, libvirt) to set it.
>>
>> If you still want to expose to the toolstack, then I think the option should
>> be outside of arch_arm. Otherwise, this should be moved in an internal
>> structure (Ian, do you have any suggestion?).
>
>
> First let me premise that the patch is much better already and Rahul
> addessed my request well. However, Julien's point about not wanting to
> make a potentially breaking ABI change in libxl is a good one. FYI we
> had a few libvirt breakages due to this kind of changes in the past and
> I would certainly be happier if we didn't cause another one. And in
> general, it is better to avoid changes to the libxl ABI if we can.
>
> I think in this case we can: I looked at the patch and
> b_info.arch_arm.vpci is only used within tools/libs/light/libxl_arm.c.
> Also, we don't need b_info.arch_arm.vpci if we can access
> d_config->num_pcidevs given that the check is just based on
> d_config->num_pcidevs.
>
> So the only issue is how to check on d_config->num_pcidevs in
> libxl__prepare_dtb. libxl__prepare_dtb takes libxl_domain_build_info as
> parameter but with container_of we can retrieve libxl_domain_config and
> from there check on num_pcidevs.
>
> Something like the appended (untested). It doesn't need any libxl struct
> changes but it requires the introduction of container_of (which is a
> simple macro). Alternatively, it would be just as simple to change
> libxl__arch_domain_init_hw_description and libxl__prepare_dtb to take a
> libxl_domain_config *d_config parameter instead of a
> libxl_domain_build_info *info parameter.
Thanks for sharing the ideas to remove the arch_arm.vpci field.
I am ok with any options, but I feel second option is simple and better to avoid
introduction of container_of(). I tested the second option and is working fine.
If everyone agree that we don’t need vpci option I will send the patch for review
In next version.
Regards,
Rahul
>
> Ian, any thoughts?
>
>
> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> index 2be208b99b..ee1176519c 100644
> --- a/tools/libs/light/libxl_arm.c
> +++ b/tools/libs/light/libxl_arm.c
> @@ -102,10 +102,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
> }
>
> /* Enable VPCI support. */
> - if (d_config->num_pcidevs) {
> + if (d_config->num_pcidevs)
> config->flags |= XEN_DOMCTL_CDF_vpci;
> - libxl_defbool_set(&d_config->b_info.arch_arm.vpci, true);
> - }
>
> return 0;
> }
> @@ -976,6 +974,7 @@ static int libxl__prepare_dtb(libxl__gc *gc, libxl_domain_build_info *info,
>
> const libxl_version_info *vers;
> const struct arch_info *ainfo;
> + libxl_domain_config *d_config = container_of(info, libxl_domain_config, b_info);
>
> vers = libxl_get_version_info(CTX);
> if (vers == NULL) return ERROR_FAIL;
> @@ -1076,7 +1075,7 @@ next_resize:
> if (info->tee == LIBXL_TEE_TYPE_OPTEE)
> FDT( make_optee_node(gc, fdt) );
>
> - if (libxl_defbool_val(info->arch_arm.vpci))
> + if (d_config->num_pcidevs)
> FDT( make_vpci_node(gc, fdt, ainfo, dom) );
>
> if (pfdt)
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-07 15:29 ` Rahul Singh
@ 2021-10-07 16:11 ` Ian Jackson
2021-10-11 13:46 ` Roger Pau Monné
2021-10-14 17:16 ` Bertrand Marquis
0 siblings, 2 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-07 16:11 UTC (permalink / raw)
To: Rahul Singh
Cc: Ian Jackson, Julien Grall, xen-devel, Bertrand Marquis,
Andre Przywara, Wei Liu, Anthony PERARD, Juergen Gross,
Stefano Stabellini, Volodymyr Babchuk
Rahul Singh writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
> As Stefano suggested in another email that we can remove the vpci
> option, if we reach to conclusion that we need vpci option I will
> move it to internal structure.
...
> Yes I agree with you VPCI is necessary for hot plugged PCI device
> and once we implement the hotplug in future we will use the
> passthrough= option to enable VPCI.
So, to summarise, I think the situation is:
* VCPI is necessry for passthrough on ARM, whether coldplug or
hotplug. It's part of the way that PCI-PT works on ARM.
* Hotplug is not yet implemented.
* VPCI is not necessary on x86 (evidently, since we don't have it
there but we do have passthrough).
So when hotplug is added, vpci will need to be turned on when
passthrough=yes is selected. I don't fully understand the other
possible values for passthrough= but maybe we can defer the question
of whether they apply to ARM ?
I think that means that yes, this should be an internal variable.
Probably in libxl__domain_create_state. We don't currently arrange to
elide arch-specific state in there, so perhaps it's fine just to
invent a member called `arm_vpci`.
Maybe you could leave a comment somewhere so that if and when PCI PT
hotplug is implemented for ARM, the implementor remembers to wire this
up.
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 00/11] PCI devices passthrough on Arm
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
` (10 preceding siblings ...)
2021-10-06 17:40 ` [PATCH v5 11/11] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
@ 2021-10-07 19:54 ` Stefano Stabellini
2021-10-07 21:29 ` Rahul Singh
11 siblings, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-07 19:54 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Ian Jackson,
Wei Liu, Juergen Gross, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Jan Beulich,
Roger Pau Monné,
Christian Lindig, David Scott, Paul Durrant, Anthony PERARD
I committed patches 2-7.
I made two small adjustments on commit:
- patch #3: bool_t/bool
- patch #7: drop _XEN_DOMCTL_CDF_vpci
Cheers,
Stefano
On Wed, 6 Oct 2021, Rahul Singh wrote:
> Hello All,
>
> The purpose of this patch series is to add PCI passthrough support to Xen on
> Arm. PCI passthrough support on ARM is the collaboration work between EPAM and
> ARM. ARM submitted the partial RFC [1][2] last year to get early feedback. We
> tried to fix all the comments and added more features to this patch series.
>
> Working POC with all the features can be found at [3]. Working POC is tested
> on x86 so that there will be no regression on x86. Design presentation can be
> found at [4]
>
> PCI passthrough support is divided into different patches. This patch series
> includes following features:
>
> Preparatory work to implement the PCI passthrough support for the ARM:
> - Refactor MSI code.
> - Fixed compilation error when HAS_PCI enabled for ARM.
>
> Discovering PCI Host Bridge in XEN:
> - PCI init to initialize the PCI driver.
> - PCI host bridge discovery in XEN and map the PCI ECAM configuration space to
> the XEN memory.
> - PCI access functions.
>
> Discovering PCI devices:
> - To support the PCI passthrough, XEN should be aware of the PCI
> devices.
> - Hardware domain is in charge of doing the PCI enumeration and will discover
> the PCI devices and then communicate to the XEN via a hypercall to add the
> PCI devices in XEN.
>
> Enable the existing x86 virtual PCI support for ARM:
> - Add VPCI trap handler for each of the PCI device added for config space
> access.
> - Register the trap handler in XEN for each of the host bridge PCI ECAM config
> space access.
>
> Emulated PCI device tree node in libxl:
> - Create a virtual PCI device tree node in libxl to enable the guest OS to
> discover the virtual PCI during guest boot.
>
> This patch series does not inlcude the following features. Following features
> will be send for review in the next version of the patch series once initial
> patch series merged.
>
> - VPCI support for DOMU guests (Non-identity mappings guest view of the BARs)
> - Virtual bus topology implementation
> - IOMMU related changes (generic, SMMUv2, SMMUv3)
> - MSI support for DOMU guests.
> - Virual ITS support for DOMU guests
>
> [1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg01184.html
> [2] https://lists.xenproject.org/archives/html/xen-devel/2020-07/threads.html#01184
> [3] https://gitlab.com/rahsingh/xen-integration/-/tree/arm_pci_v4_integrate
> [4] https://static.sched.com/hosted_files/xen2021/e4/PCI_Device_Passthrough_On_Arm.pdf
>
> Oleksandr Andrushchenko (1):
> xen/arm: Add support for Xilinx ZynqMP PCI host controller
>
> Rahul Singh (10):
> xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
> xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM
> xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
> xen/arm: PCI host bridge discovery within XEN on ARM
> xen/arm: Implement pci access functions
> xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
> xen/arm: Enable the existing x86 virtual PCI support for ARM.
> xen/arm: Transitional change to build HAS_VPCI on ARM.
> arm/libxl: Emulated PCI device tree node in libxl
> xen/arm: Add linux,pci-domain property for hwdom if not available.
>
> docs/misc/xen-command-line.pandoc | 7 +
> tools/include/libxl.h | 6 +
> tools/libs/ctrl/xc_domain.c | 9 +
> tools/libs/light/libxl_arm.c | 111 +++++++++++
> tools/libs/light/libxl_types.idl | 1 +
> tools/ocaml/libs/xc/xenctrl.ml | 1 +
> tools/ocaml/libs/xc/xenctrl.mli | 1 +
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 8 +-
> xen/arch/arm/domain_build.c | 19 ++
> xen/arch/arm/pci/Makefile | 5 +
> xen/arch/arm/pci/ecam.c | 61 ++++++
> xen/arch/arm/pci/pci-access.c | 140 ++++++++++++++
> xen/arch/arm/pci/pci-host-common.c | 287 ++++++++++++++++++++++++++++
> xen/arch/arm/pci/pci-host-generic.c | 48 +++++
> xen/arch/arm/pci/pci-host-zynqmp.c | 65 +++++++
> xen/arch/arm/pci/pci.c | 12 ++
> xen/arch/arm/physdev.c | 6 +-
> xen/arch/arm/vpci.c | 102 ++++++++++
> xen/arch/arm/vpci.h | 36 ++++
> xen/arch/x86/domain.c | 6 +
> xen/arch/x86/physdev.c | 52 +----
> xen/arch/x86/x86_64/physdev.c | 2 +-
> xen/common/domain.c | 2 +-
> xen/drivers/passthrough/pci.c | 18 ++
> xen/drivers/pci/Makefile | 1 +
> xen/drivers/pci/physdev.c | 86 +++++++++
> xen/drivers/vpci/Makefile | 3 +-
> xen/drivers/vpci/header.c | 2 +
> xen/include/asm-arm/domain.h | 7 +-
> xen/include/asm-arm/pci.h | 85 ++++++++
> xen/include/asm-x86/pci.h | 8 +-
> xen/include/public/arch-arm.h | 21 +-
> xen/include/public/domctl.h | 4 +-
> xen/include/xen/hypercall.h | 4 +
> xen/include/xen/pci.h | 2 +
> 36 files changed, 1168 insertions(+), 61 deletions(-)
> create mode 100644 xen/arch/arm/pci/ecam.c
> create mode 100644 xen/arch/arm/pci/pci-access.c
> create mode 100644 xen/arch/arm/pci/pci-host-common.c
> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
> create mode 100644 xen/arch/arm/vpci.c
> create mode 100644 xen/arch/arm/vpci.h
> create mode 100644 xen/drivers/pci/physdev.c
>
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 00/11] PCI devices passthrough on Arm
2021-10-07 19:54 ` [PATCH v5 00/11] PCI devices passthrough on Arm Stefano Stabellini
@ 2021-10-07 21:29 ` Rahul Singh
0 siblings, 0 replies; 190+ messages in thread
From: Rahul Singh @ 2021-10-07 21:29 UTC (permalink / raw)
To: Stefano Stabellini
Cc: xen-devel, Bertrand Marquis, Andre Przywara, Ian Jackson,
Wei Liu, Juergen Gross, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Jan Beulich, Roger Pau Monné,
Christian Lindig, David Scott, Paul Durrant, Anthony PERARD
Hi Stefano,
> On 7 Oct 2021, at 8:54 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> I committed patches 2-7.
>
> I made two small adjustments on commit:
> - patch #3: bool_t/bool
> - patch #7: drop _XEN_DOMCTL_CDF_vpci
Thank you very much. I will be on leave from tomorrow. Someone from my
team will take care of the rest of the patch and reply to the comments.
Regards,
Rahul
>
> Cheers,
>
> Stefano
>
>
> On Wed, 6 Oct 2021, Rahul Singh wrote:
>> Hello All,
>>
>> The purpose of this patch series is to add PCI passthrough support to Xen on
>> Arm. PCI passthrough support on ARM is the collaboration work between EPAM and
>> ARM. ARM submitted the partial RFC [1][2] last year to get early feedback. We
>> tried to fix all the comments and added more features to this patch series.
>>
>> Working POC with all the features can be found at [3]. Working POC is tested
>> on x86 so that there will be no regression on x86. Design presentation can be
>> found at [4]
>>
>> PCI passthrough support is divided into different patches. This patch series
>> includes following features:
>>
>> Preparatory work to implement the PCI passthrough support for the ARM:
>> - Refactor MSI code.
>> - Fixed compilation error when HAS_PCI enabled for ARM.
>>
>> Discovering PCI Host Bridge in XEN:
>> - PCI init to initialize the PCI driver.
>> - PCI host bridge discovery in XEN and map the PCI ECAM configuration space to
>> the XEN memory.
>> - PCI access functions.
>>
>> Discovering PCI devices:
>> - To support the PCI passthrough, XEN should be aware of the PCI
>> devices.
>> - Hardware domain is in charge of doing the PCI enumeration and will discover
>> the PCI devices and then communicate to the XEN via a hypercall to add the
>> PCI devices in XEN.
>>
>> Enable the existing x86 virtual PCI support for ARM:
>> - Add VPCI trap handler for each of the PCI device added for config space
>> access.
>> - Register the trap handler in XEN for each of the host bridge PCI ECAM config
>> space access.
>>
>> Emulated PCI device tree node in libxl:
>> - Create a virtual PCI device tree node in libxl to enable the guest OS to
>> discover the virtual PCI during guest boot.
>>
>> This patch series does not inlcude the following features. Following features
>> will be send for review in the next version of the patch series once initial
>> patch series merged.
>>
>> - VPCI support for DOMU guests (Non-identity mappings guest view of the BARs)
>> - Virtual bus topology implementation
>> - IOMMU related changes (generic, SMMUv2, SMMUv3)
>> - MSI support for DOMU guests.
>> - Virual ITS support for DOMU guests
>>
>> [1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg01184.html
>> [2] https://lists.xenproject.org/archives/html/xen-devel/2020-07/threads.html#01184
>> [3] https://gitlab.com/rahsingh/xen-integration/-/tree/arm_pci_v4_integrate
>> [4] https://static.sched.com/hosted_files/xen2021/e4/PCI_Device_Passthrough_On_Arm.pdf
>>
>> Oleksandr Andrushchenko (1):
>> xen/arm: Add support for Xilinx ZynqMP PCI host controller
>>
>> Rahul Singh (10):
>> xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
>> xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM
>> xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
>> xen/arm: PCI host bridge discovery within XEN on ARM
>> xen/arm: Implement pci access functions
>> xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
>> xen/arm: Enable the existing x86 virtual PCI support for ARM.
>> xen/arm: Transitional change to build HAS_VPCI on ARM.
>> arm/libxl: Emulated PCI device tree node in libxl
>> xen/arm: Add linux,pci-domain property for hwdom if not available.
>>
>> docs/misc/xen-command-line.pandoc | 7 +
>> tools/include/libxl.h | 6 +
>> tools/libs/ctrl/xc_domain.c | 9 +
>> tools/libs/light/libxl_arm.c | 111 +++++++++++
>> tools/libs/light/libxl_types.idl | 1 +
>> tools/ocaml/libs/xc/xenctrl.ml | 1 +
>> tools/ocaml/libs/xc/xenctrl.mli | 1 +
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/domain.c | 8 +-
>> xen/arch/arm/domain_build.c | 19 ++
>> xen/arch/arm/pci/Makefile | 5 +
>> xen/arch/arm/pci/ecam.c | 61 ++++++
>> xen/arch/arm/pci/pci-access.c | 140 ++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c | 287 ++++++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-host-generic.c | 48 +++++
>> xen/arch/arm/pci/pci-host-zynqmp.c | 65 +++++++
>> xen/arch/arm/pci/pci.c | 12 ++
>> xen/arch/arm/physdev.c | 6 +-
>> xen/arch/arm/vpci.c | 102 ++++++++++
>> xen/arch/arm/vpci.h | 36 ++++
>> xen/arch/x86/domain.c | 6 +
>> xen/arch/x86/physdev.c | 52 +----
>> xen/arch/x86/x86_64/physdev.c | 2 +-
>> xen/common/domain.c | 2 +-
>> xen/drivers/passthrough/pci.c | 18 ++
>> xen/drivers/pci/Makefile | 1 +
>> xen/drivers/pci/physdev.c | 86 +++++++++
>> xen/drivers/vpci/Makefile | 3 +-
>> xen/drivers/vpci/header.c | 2 +
>> xen/include/asm-arm/domain.h | 7 +-
>> xen/include/asm-arm/pci.h | 85 ++++++++
>> xen/include/asm-x86/pci.h | 8 +-
>> xen/include/public/arch-arm.h | 21 +-
>> xen/include/public/domctl.h | 4 +-
>> xen/include/xen/hypercall.h | 4 +
>> xen/include/xen/pci.h | 2 +
>> 36 files changed, 1168 insertions(+), 61 deletions(-)
>> create mode 100644 xen/arch/arm/pci/ecam.c
>> create mode 100644 xen/arch/arm/pci/pci-access.c
>> create mode 100644 xen/arch/arm/pci/pci-host-common.c
>> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>> create mode 100644 xen/drivers/pci/physdev.c
>>
>> --
>> 2.25.1
>>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-06 17:40 ` [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag Rahul Singh
2021-10-07 13:08 ` Jan Beulich
@ 2021-10-08 18:06 ` Andrew Cooper
2021-10-08 21:12 ` Julien Grall
2021-10-11 9:48 ` Ian Jackson
2021-10-11 9:27 ` Roger Pau Monné
2 siblings, 2 replies; 190+ messages in thread
From: Andrew Cooper @ 2021-10-08 18:06 UTC (permalink / raw)
To: Rahul Singh, xen-devel
Cc: bertrand.marquis, Andre.Przywara, Christian Lindig, David Scott,
Ian Jackson, Wei Liu, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk, Roger Pau Monné
On 06/10/2021 18:40, Rahul Singh wrote:
> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> Reject the use of this new flag for x86 as VPCI is not supported for
> DOMU guests for x86.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> Acked-by: Christian Lindig <christian.lindig@citrix.com>
I'm afraid this logic is broken.
There's no matching feature to indicate to the toolstack whether
XEN_DOMCTL_CDF_vpci will be accepted or not. The usual way of doing
this is with a physinfo_cap field.
This flag needs using in Patch 10 to reject attempts to create a VM with
devices when passthrough support is unavailable.
Ian, for the 4.16 release, this series either needs completing with the
additional flag implemented, or this patch needs reverting to avoid us
shipping a broken interface.
~Andrew
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-08 18:06 ` Andrew Cooper
@ 2021-10-08 21:12 ` Julien Grall
2021-10-08 21:46 ` Stefano Stabellini
2021-10-11 9:48 ` Ian Jackson
1 sibling, 1 reply; 190+ messages in thread
From: Julien Grall @ 2021-10-08 21:12 UTC (permalink / raw)
To: Andrew Cooper
Cc: Rahul Singh, xen-devel, Bertrand Marquis, Andre Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
George Dunlap, Jan Beulich, Stefano Stabellini,
Volodymyr Babchuk, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]
Hi Andrew,
On Fri, 8 Oct 2021, 20:07 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:
> On 06/10/2021 18:40, Rahul Singh wrote:
> > Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> > Reject the use of this new flag for x86 as VPCI is not supported for
> > DOMU guests for x86.
> >
> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> > Acked-by: Christian Lindig <christian.lindig@citrix.com>
>
> I'm afraid this logic is broken.
>
> There's no matching feature to indicate to the toolstack whether
> XEN_DOMCTL_CDF_vpci will be accepted or not. The usual way of doing
> this is with a physinfo_cap field.
>
I am slightly puzzled by this. I am assuming you are referring to
XENVER_get_features which AFAICT is a stable interface. So why should we
use it to describe the presence of an unstable interface?
> This flag needs using in Patch 10 to reject attempts to create a VM with
> devices when passthrough support is unavailable.
>
May I ask why we can't rely on the hypervisor to reject the flag when
suitable?
> Ian, for the 4.16 release, this series either needs completing with the
> additional flag implemented, or this patch needs reverting to avoid us
> shipping a broken interface.
I fail to see how the interface would be broken... Would you mind to
describe a bit more what could go wrong with this interface?
Cheers,
> ~Andrew
>
>
[-- Attachment #2: Type: text/html, Size: 2994 bytes --]
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-08 21:12 ` Julien Grall
@ 2021-10-08 21:46 ` Stefano Stabellini
2021-10-11 9:24 ` Julien Grall
2021-10-11 11:29 ` Michal Orzel
0 siblings, 2 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-08 21:46 UTC (permalink / raw)
To: Julien Grall
Cc: Andrew Cooper, Rahul Singh, xen-devel, Bertrand Marquis,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, George Dunlap, Jan Beulich, Stefano Stabellini,
Volodymyr Babchuk, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]
On Fri, 8 Oct 2021, Julien Grall wrote:
> Hi Andrew,
>
> On Fri, 8 Oct 2021, 20:07 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:
> On 06/10/2021 18:40, Rahul Singh wrote:
> > Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> > Reject the use of this new flag for x86 as VPCI is not supported for
> > DOMU guests for x86.
> >
> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> > Acked-by: Christian Lindig <christian.lindig@citrix.com>
>
> I'm afraid this logic is broken.
>
> There's no matching feature to indicate to the toolstack whether
> XEN_DOMCTL_CDF_vpci will be accepted or not. The usual way of doing
> this is with a physinfo_cap field.
>
>
> I am slightly puzzled by this. I am assuming you are referring to XENVER_get_features which AFAICT is a stable interface. So why should we
> use it to describe the presence of an unstable interface?
>
>
> This flag needs using in Patch 10 to reject attempts to create a VM with
> devices when passthrough support is unavailable.
>
>
> May I ask why we can't rely on the hypervisor to reject the flag when suitable?
>
>
> Ian, for the 4.16 release, this series either needs completing with the
> additional flag implemented, or this patch needs reverting to avoid us
> shipping a broken interface.
>
>
> I fail to see how the interface would be broken... Would you mind to describe a bit more what could go wrong with this interface?
After chatting with Andrew on IRC, this is my understanding.
Today if pci=[] is specified in the VM config file then
XEN_DOMCTL_CDF_vpci is added. If Xen doesn't support it, Xen returns
an error but libxl/xl won't be able to tell exactly why it fails. So xl
will end up printing a generic VM creation failure. Andrew would like to
see something like the following in libxl:
if ( PCI_devices && !cap.vcpi )
error("Sorry - PCI not supported")
So that the user gets a clear informative error back rather than a
generic VM creation failure. Also, this is a requirement for the stable
hypercall interface.
I think that's fine and we can implement this request easily by adding
XEN_SYSCTL_PHYSCAP_vpci. Rahul or Bertrand, are you guys happy with
doing that? Otherwise I could take it on.
As a side note, given that PCI passthrough support is actually not yet
complete on ARM, we could even just do the following in xl/libxl:
if ( PCI_devices )
error("Sorry - PCI not supported")
or temporarily remove XEN_DOMCTL_CDF_vpci until PCI passthrough gets
finalized.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-08 21:46 ` Stefano Stabellini
@ 2021-10-11 9:24 ` Julien Grall
2021-10-11 11:29 ` Michal Orzel
1 sibling, 0 replies; 190+ messages in thread
From: Julien Grall @ 2021-10-11 9:24 UTC (permalink / raw)
To: Stefano Stabellini, Julien Grall
Cc: Andrew Cooper, Rahul Singh, xen-devel, Bertrand Marquis,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, George Dunlap, Jan Beulich, Volodymyr Babchuk,
Roger Pau Monné
Hi Stefano,
On 08/10/2021 22:46, Stefano Stabellini wrote:
> On Fri, 8 Oct 2021, Julien Grall wrote:
>> Hi Andrew,
>>
>> On Fri, 8 Oct 2021, 20:07 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:
>> On 06/10/2021 18:40, Rahul Singh wrote:
>> > Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>> > Reject the use of this new flag for x86 as VPCI is not supported for
>> > DOMU guests for x86.
>> >
>> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> > Acked-by: Christian Lindig <christian.lindig@citrix.com>
>>
>> I'm afraid this logic is broken.
>>
>> There's no matching feature to indicate to the toolstack whether
>> XEN_DOMCTL_CDF_vpci will be accepted or not. The usual way of doing
>> this is with a physinfo_cap field.
>>
>>
>> I am slightly puzzled by this. I am assuming you are referring to XENVER_get_features which AFAICT is a stable interface. So why should we
>> use it to describe the presence of an unstable interface?
>>
>>
>> This flag needs using in Patch 10 to reject attempts to create a VM with
>> devices when passthrough support is unavailable.
>>
>>
>> May I ask why we can't rely on the hypervisor to reject the flag when suitable?
>>
>>
>> Ian, for the 4.16 release, this series either needs completing with the
>> additional flag implemented, or this patch needs reverting to avoid us
>> shipping a broken interface.
>>
>>
>> I fail to see how the interface would be broken... Would you mind to describe a bit more what could go wrong with this interface?
>
>
> After chatting with Andrew on IRC, this is my understanding.
>
> Today if pci=[] is specified in the VM config file then
> XEN_DOMCTL_CDF_vpci is added. If Xen doesn't support it, Xen returns
> an error but libxl/xl won't be able to tell exactly why it fails. So xl
> will end up printing a generic VM creation failure. Andrew would like to
> see something like the following in libxl:
>
> if ( PCI_devices && !cap.vcpi )
> error("Sorry - PCI not supported")
>
> So that the user gets a clear informative error back rather than a
> generic VM creation failure.
I can understand this request. However...
Also, this is a requirement for the stable
> hypercall interface.
... leaving aside the fact that domctl is clearly not stable today, the
flag would be rejected on hypervisor not supporting vPCI. So I don't
really see how this is a requirement for the stable hypercall interface.
Would you mind providing more details?
>
>
> I think that's fine and we can implement this request easily by adding
> XEN_SYSCTL_PHYSCAP_vpci. Rahul or Bertrand, are you guys happy with
> doing that? Otherwise I could take it on.
>
>
> As a side note, given that PCI passthrough support is actually not yet
> complete on ARM, we could even just do the following in xl/libxl:
>
> if ( PCI_devices )
> error("Sorry - PCI not supported")
>
> or temporarily remove XEN_DOMCTL_CDF_vpci until PCI passthrough gets
> finalized.
Cheers,
>
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-06 17:40 ` [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag Rahul Singh
2021-10-07 13:08 ` Jan Beulich
2021-10-08 18:06 ` Andrew Cooper
@ 2021-10-11 9:27 ` Roger Pau Monné
2021-10-11 12:06 ` Michal Orzel
` (2 more replies)
2 siblings, 3 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 9:27 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Christian Lindig,
David Scott, Ian Jackson, Wei Liu, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Volodymyr Babchuk
On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> Reject the use of this new flag for x86 as VPCI is not supported for
> DOMU guests for x86.
I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
PVH dom0, like we do for any other CDF flags when Xen builds dom0.
Things like PVH vs PV get translated into CDF flags by create_dom0,
and processed normally by the sanitise_domain_config logic, vPCI
should be handled that way.
Do you think you could see about fixing this?
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-08 18:06 ` Andrew Cooper
2021-10-08 21:12 ` Julien Grall
@ 2021-10-11 9:48 ` Ian Jackson
1 sibling, 0 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-11 9:48 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, Andrew Cooper, bertrand.marquis, Andre.Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
George Dunlap, Jan Beulich, Julien Grall, Stefano Stabellini,
Volodymyr Babchuk, Roger Pau Monné
Andrew Cooper writes ("Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag"):
> Ian, for the 4.16 release, this series either needs completing with the
> additional flag implemented, or this patch needs reverting to avoid us
> shipping a broken interface.
I have caught up on this thread. I think (hope?) it's converging.
If not please let me know and maybe I can help.
Can I ask to please be CC'd on the whole series for the patch(es) to
sort this out. Please also make sure that those who commented are
CC'd. I want the fixes that ultimately get committed to be the final
fixes (probably that means they should have consensus).
FTAOD, from a formal release management point of view: I regard those
putative fixes as bugfixes so they can go in after the feature freeze
(which is this Friday). But if suitable fixes don't make it in within
the first few weeks of the freeze (and, as I expect, the maintainers
or I still regard this as an RC bug) then a revert of the new feature
will be the only option.
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-06 17:40 ` [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
2021-10-07 13:43 ` Jan Beulich
@ 2021-10-11 10:51 ` Roger Pau Monné
2021-10-11 16:12 ` Bertrand Marquis
2021-10-12 14:32 ` Julien Grall
2021-10-13 8:45 ` Roger Pau Monné
3 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 10:51 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Paul Durrant
On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the Xen to emulate the PCI devices config space.
>
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
>
> For Dom0less systems scan_pci_devices() would be used to discover the
> PCI device in XEN and VPCI handler will be added during XEN boots.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> Change in v5:
> - Add pci_cleanup_msi(pdev) in cleanup part.
> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> Change in v4:
> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
> Change in v3:
> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
> Change in v2:
> - Add new XEN_DOMCTL_CDF_vpci flag
> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
> - enable vpci support when pci-passthough option is enabled.
> ---
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 4 ++
> xen/arch/arm/domain_build.c | 3 +
> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
> xen/arch/arm/vpci.h | 36 ++++++++++++
> xen/drivers/passthrough/pci.c | 18 ++++++
> xen/include/asm-arm/domain.h | 7 ++-
> xen/include/asm-x86/pci.h | 2 -
> xen/include/public/arch-arm.h | 7 +++
> xen/include/xen/pci.h | 2 +
> 10 files changed, 179 insertions(+), 3 deletions(-)
> create mode 100644 xen/arch/arm/vpci.c
> create mode 100644 xen/arch/arm/vpci.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 44d7cc81fa..fb9c976ea2 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 36138c1b2e..fbb52f78f1 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -39,6 +39,7 @@
> #include <asm/vgic.h>
> #include <asm/vtimer.h>
>
> +#include "vpci.h"
> #include "vuart.h"
>
> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> goto fail;
>
> + if ( (rc = domain_vpci_init(d)) != 0 )
> + goto fail;
> +
> return 0;
>
> fail:
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index c5afbe2e05..f4c89bde8c 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
> if ( iommu_enabled )
> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>
> + if ( is_pci_passthrough_enabled() )
> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
I think I'm confused with this. You seem to enable vPCI for dom0, but
then domain_vpci_init will setup traps for the guest virtual ECAM
layout, not the native one that dom0 will be using.
> +
> dom0 = domain_create(0, &dom0_cfg, true);
> if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
> panic("Error creating domain 0\n");
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..76c12b9281
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,102 @@
> +/*
> + * xen/arch/arm/vpci.c
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +
> +#include <asm/mmio.h>
> +
> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
> +
> +/* Do some sanity checks. */
> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> +{
> + /* Check access size. */
> + if ( len > 8 )
> + return false;
> +
> + /* Check that access is size aligned. */
> + if ( (reg & (len - 1)) )
> + return false;
> +
> + return true;
> +}
There's a vpci_access_allowed which I think you could generalize and
use here, there's no need to have this duplicated code.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-08 21:46 ` Stefano Stabellini
2021-10-11 9:24 ` Julien Grall
@ 2021-10-11 11:29 ` Michal Orzel
2021-10-11 11:35 ` Jan Beulich
1 sibling, 1 reply; 190+ messages in thread
From: Michal Orzel @ 2021-10-11 11:29 UTC (permalink / raw)
To: Stefano Stabellini, Julien Grall
Cc: Andrew Cooper, Rahul Singh, xen-devel, Bertrand Marquis,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, George Dunlap, Jan Beulich, Volodymyr Babchuk,
Roger Pau Monné
Hi Stefano,
On 08.10.2021 23:46, Stefano Stabellini wrote:
> On Fri, 8 Oct 2021, Julien Grall wrote:
>> Hi Andrew,
>>
>> On Fri, 8 Oct 2021, 20:07 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:
>> On 06/10/2021 18:40, Rahul Singh wrote:
>> > Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>> > Reject the use of this new flag for x86 as VPCI is not supported for
>> > DOMU guests for x86.
>> >
>> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> > Acked-by: Christian Lindig <christian.lindig@citrix.com>
>>
>> I'm afraid this logic is broken.
>>
>> There's no matching feature to indicate to the toolstack whether
>> XEN_DOMCTL_CDF_vpci will be accepted or not. The usual way of doing
>> this is with a physinfo_cap field.
>>
>>
>> I am slightly puzzled by this. I am assuming you are referring to XENVER_get_features which AFAICT is a stable interface. So why should we
>> use it to describe the presence of an unstable interface?
>>
>>
>> This flag needs using in Patch 10 to reject attempts to create a VM with
>> devices when passthrough support is unavailable.
>>
>>
>> May I ask why we can't rely on the hypervisor to reject the flag when suitable?
>>
>>
>> Ian, for the 4.16 release, this series either needs completing with the
>> additional flag implemented, or this patch needs reverting to avoid us
>> shipping a broken interface.
>>
>>
>> I fail to see how the interface would be broken... Would you mind to describe a bit more what could go wrong with this interface?
>
>
> After chatting with Andrew on IRC, this is my understanding.
>
> Today if pci=[] is specified in the VM config file then
> XEN_DOMCTL_CDF_vpci is added. If Xen doesn't support it, Xen returns
> an error but libxl/xl won't be able to tell exactly why it fails. So xl
> will end up printing a generic VM creation failure. Andrew would like to
> see something like the following in libxl:
>
> if ( PCI_devices && !cap.vcpi )
> error("Sorry - PCI not supported")
>
> So that the user gets a clear informative error back rather than a
> generic VM creation failure. Also, this is a requirement for the stable
> hypercall interface.
>
>
> I think that's fine and we can implement this request easily by adding
> XEN_SYSCTL_PHYSCAP_vpci. Rahul or Bertrand, are you guys happy with
> doing that? Otherwise I could take it on.
>
>
> As a side note, given that PCI passthrough support is actually not yet
> complete on ARM, we could even just do the following in xl/libxl:
>
> if ( PCI_devices )
> error("Sorry - PCI not supported")
>
> or temporarily remove XEN_DOMCTL_CDF_vpci until PCI passthrough gets
> finalized.
>
As Rahul is on leave:
I'm ok to introduce XEN_SYSCTL_PHYSCAP_vpci. I did the same for vpmu so it's ok.
However the problem I have is about setting this cap.
On arm it is easy as we are not supporting vpci at the moment so the cap can be set to false.
But how to deal with this cap in x86 code? I'm not familiar with x86 so I'm asking for advice.
Cheers,
Michal
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-11 11:29 ` Michal Orzel
@ 2021-10-11 11:35 ` Jan Beulich
2021-10-11 13:17 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-11 11:35 UTC (permalink / raw)
To: Michal Orzel, Roger Pau Monné
Cc: Andrew Cooper, Rahul Singh, xen-devel, Bertrand Marquis,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, George Dunlap, Volodymyr Babchuk, Julien Grall,
Stefano Stabellini
On 11.10.2021 13:29, Michal Orzel wrote:
> On 08.10.2021 23:46, Stefano Stabellini wrote:
>> On Fri, 8 Oct 2021, Julien Grall wrote:
>>> On Fri, 8 Oct 2021, 20:07 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:
>>> On 06/10/2021 18:40, Rahul Singh wrote:
>>> > Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>>> > Reject the use of this new flag for x86 as VPCI is not supported for
>>> > DOMU guests for x86.
>>> >
>>> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>> > Acked-by: Christian Lindig <christian.lindig@citrix.com>
>>>
>>> I'm afraid this logic is broken.
>>>
>>> There's no matching feature to indicate to the toolstack whether
>>> XEN_DOMCTL_CDF_vpci will be accepted or not. The usual way of doing
>>> this is with a physinfo_cap field.
>>>
>>>
>>> I am slightly puzzled by this. I am assuming you are referring to XENVER_get_features which AFAICT is a stable interface. So why should we
>>> use it to describe the presence of an unstable interface?
>>>
>>>
>>> This flag needs using in Patch 10 to reject attempts to create a VM with
>>> devices when passthrough support is unavailable.
>>>
>>>
>>> May I ask why we can't rely on the hypervisor to reject the flag when suitable?
>>>
>>>
>>> Ian, for the 4.16 release, this series either needs completing with the
>>> additional flag implemented, or this patch needs reverting to avoid us
>>> shipping a broken interface.
>>>
>>>
>>> I fail to see how the interface would be broken... Would you mind to describe a bit more what could go wrong with this interface?
>>
>>
>> After chatting with Andrew on IRC, this is my understanding.
>>
>> Today if pci=[] is specified in the VM config file then
>> XEN_DOMCTL_CDF_vpci is added. If Xen doesn't support it, Xen returns
>> an error but libxl/xl won't be able to tell exactly why it fails. So xl
>> will end up printing a generic VM creation failure. Andrew would like to
>> see something like the following in libxl:
>>
>> if ( PCI_devices && !cap.vcpi )
>> error("Sorry - PCI not supported")
>>
>> So that the user gets a clear informative error back rather than a
>> generic VM creation failure. Also, this is a requirement for the stable
>> hypercall interface.
>>
>>
>> I think that's fine and we can implement this request easily by adding
>> XEN_SYSCTL_PHYSCAP_vpci. Rahul or Bertrand, are you guys happy with
>> doing that? Otherwise I could take it on.
>>
>>
>> As a side note, given that PCI passthrough support is actually not yet
>> complete on ARM, we could even just do the following in xl/libxl:
>>
>> if ( PCI_devices )
>> error("Sorry - PCI not supported")
>>
>> or temporarily remove XEN_DOMCTL_CDF_vpci until PCI passthrough gets
>> finalized.
>>
> As Rahul is on leave:
> I'm ok to introduce XEN_SYSCTL_PHYSCAP_vpci. I did the same for vpmu so it's ok.
> However the problem I have is about setting this cap.
> On arm it is easy as we are not supporting vpci at the moment so the cap can be set to false.
> But how to deal with this cap in x86 code? I'm not familiar with x86 so I'm asking for advice.
As the sysctl is mainly from tool stacks to drive guests (DomU-s), I'd
set it to false for x86 as well. Roger - do you see any reason this
could be needed to express anything Dom0-related?
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM.
2021-10-06 17:40 ` [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM Rahul Singh
@ 2021-10-11 11:43 ` Roger Pau Monné
2021-10-11 12:15 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 11:43 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk
On Wed, Oct 06, 2021 at 06:40:35PM +0100, Rahul Singh wrote:
> This patch will be reverted once we add support for VPCI MSI/MSIX
> support on ARM.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Acked-by: Stefano Stabellini <sstabellini@kernel.org>
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
Reviewed-by: Roger Pau Monné <rogewr.pau@citrix.com>
> ---
> Change in v5: none
> Change in v4:
> - Added Acked-by: Stefano Stabellini <sstabellini@kernel.org>
> - Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
> Change in v3: none
> Change in v2: Patch introduced in v2
> ---
> ---
> xen/drivers/vpci/Makefile | 3 ++-
> xen/drivers/vpci/header.c | 2 ++
> xen/include/asm-arm/pci.h | 8 ++++++++
> 3 files changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
> index 55d1bdfda0..1a1413b93e 100644
> --- a/xen/drivers/vpci/Makefile
> +++ b/xen/drivers/vpci/Makefile
> @@ -1 +1,2 @@
> -obj-y += vpci.o header.o msi.o msix.o
> +obj-y += vpci.o header.o
> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
> index ba9a036202..f8cd55e7c0 100644
> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
> * FIXME: punching holes after the p2m has been set up might be racy for
> * DomU usage, needs to be revisited.
> */
> +#ifdef CONFIG_HAS_PCI_MSI
> if ( map && !rom_only && vpci_make_msix_hole(pdev) )
> return;
> +#endif
FWIW, I would also be fine with providing a dummy inline function for
vpci_make_msix_hole when !CONFIG_HAS_PCI_MSI, but I assume this is a
temporary workaround until MSI is implemented for Arm.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-06 17:40 ` [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM Rahul Singh
@ 2021-10-11 11:47 ` Roger Pau Monné
2021-10-11 12:11 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 11:47 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Ian Jackson,
Wei Liu, Juergen Gross, Stefano Stabellini
On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
> ARM architecture does not implement I/O ports. Ignore this call on ARM
> to avoid the overhead of making a hypercall just for Xen to return
> -ENOSYS.
What is the cal trace of this function actually on Arm?
AFAICT libxl will only call xc_domain_ioport_permission if there are
IO ports explicitly defined in the guest configuration, or if any of
the BARs of the PCI device is in the IO space, which is not possible
on Arm.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-11 9:27 ` Roger Pau Monné
@ 2021-10-11 12:06 ` Michal Orzel
2021-10-12 10:38 ` Michal Orzel
2021-10-12 21:48 ` Stefano Stabellini
2 siblings, 0 replies; 190+ messages in thread
From: Michal Orzel @ 2021-10-11 12:06 UTC (permalink / raw)
To: Roger Pau Monné, Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Christian Lindig,
David Scott, Ian Jackson, Wei Liu, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Volodymyr Babchuk,
Ian Jackson
Hi Roger,
On 11.10.2021 11:27, Roger Pau Monné wrote:
> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>> Reject the use of this new flag for x86 as VPCI is not supported for
>> DOMU guests for x86.
>
> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
>
> Things like PVH vs PV get translated into CDF flags by create_dom0,
> and processed normally by the sanitise_domain_config logic, vPCI
> should be handled that way.
>
> Do you think you could see about fixing this?
>
> Thanks, Roger.
>
As Rahul is on leave:
I agree with you. XEN_DOMCTL_CDF_vpci should be set in dom0_cfg.flags in create_dom0 for pvh.
We can do such fix in a new patch together with adding physcap for vpci.
Cheers,
Michal
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 11:47 ` Roger Pau Monné
@ 2021-10-11 12:11 ` Bertrand Marquis
2021-10-11 13:20 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 12:11 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini
Hi Roger,
> On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
>> ARM architecture does not implement I/O ports. Ignore this call on ARM
>> to avoid the overhead of making a hypercall just for Xen to return
>> -ENOSYS.
>
> What is the cal trace of this function actually on Arm?
>
> AFAICT libxl will only call xc_domain_ioport_permission if there are
> IO ports explicitly defined in the guest configuration, or if any of
> the BARs of the PCI device is in the IO space, which is not possible
> on Arm.
PCI devices BARs can be in the IO space as the PCI devices are not
Arm specific. There is not ioports on arm so to be used those can be
in some cases remapped and accessed as MMIOs or are not possible
to use at all.
But the IO space does appear when BARs are listed even on Arm.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM.
2021-10-11 11:43 ` Roger Pau Monné
@ 2021-10-11 12:15 ` Bertrand Marquis
2021-10-12 1:32 ` Stefano Stabellini
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 12:15 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Rahul Singh, xen-devel, Andre Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk
Hi Roger,
> On 11 Oct 2021, at 12:43, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Wed, Oct 06, 2021 at 06:40:35PM +0100, Rahul Singh wrote:
>> This patch will be reverted once we add support for VPCI MSI/MSIX
>> support on ARM.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> Acked-by: Stefano Stabellini <sstabellini@kernel.org>
>> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
>
> Reviewed-by: Roger Pau Monné <rogewr.pau@citrix.com>
Thanks
>
>> ---
>> Change in v5: none
>> Change in v4:
>> - Added Acked-by: Stefano Stabellini <sstabellini@kernel.org>
>> - Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
>> Change in v3: none
>> Change in v2: Patch introduced in v2
>> ---
>> ---
>> xen/drivers/vpci/Makefile | 3 ++-
>> xen/drivers/vpci/header.c | 2 ++
>> xen/include/asm-arm/pci.h | 8 ++++++++
>> 3 files changed, 12 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
>> index 55d1bdfda0..1a1413b93e 100644
>> --- a/xen/drivers/vpci/Makefile
>> +++ b/xen/drivers/vpci/Makefile
>> @@ -1 +1,2 @@
>> -obj-y += vpci.o header.o msi.o msix.o
>> +obj-y += vpci.o header.o
>> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
>> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
>> index ba9a036202..f8cd55e7c0 100644
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>> * FIXME: punching holes after the p2m has been set up might be racy for
>> * DomU usage, needs to be revisited.
>> */
>> +#ifdef CONFIG_HAS_PCI_MSI
>> if ( map && !rom_only && vpci_make_msix_hole(pdev) )
>> return;
>> +#endif
>
> FWIW, I would also be fine with providing a dummy inline function for
> vpci_make_msix_hole when !CONFIG_HAS_PCI_MSI, but I assume this is a
> temporary workaround until MSI is implemented for Arm.
Yes this is temporary and MSI support will be added on Arm.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-07 13:43 ` Jan Beulich
@ 2021-10-11 12:41 ` Bertrand Marquis
2021-10-11 13:09 ` Jan Beulich
2021-10-12 15:04 ` Julien Grall
0 siblings, 2 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 12:41 UTC (permalink / raw)
To: Jan Beulich
Cc: Rahul Singh, Andre Przywara, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
Hi Jan,
As Rahul is on leave, I will answer you and make the changes needed.
> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 06.10.2021 19:40, Rahul Singh wrote:
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,102 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +
>> +#include <asm/mmio.h>
>> +
>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>
> Nit: Stray blank (like you had in an earlier version for MMCFG_BDF()).
> Also isn't this effectively part of the public interface (along with
> MMCFG_BDF()), alongside GUEST_VPCI_ECAM_{BASE,SIZE}?
I will move that in the next version to xen/pci.h and rename it MMCFG_REG_OFFSET.
Would that be ok ?
>
>> +/* Do some sanity checks. */
>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> + /* Check access size. */
>> + if ( len > 8 )
>> + return false;
>
> struct hsr_dabt's size field doesn't allow len to be above 8. I could
> see that you may want to sanity check things, but that's not helpful
> if done incompletely. Elsewhere you assume the value to be non-zero,
> and ...
>
>> + /* Check that access is size aligned. */
>> + if ( (reg & (len - 1)) )
>
> ... right here you assume the value to be a power of 2. While I'm not
> a maintainer, I'd still like to suggest consistency: Do all pertinent
> checks or none of them (relying on the caller).
I will remove the check for len > 8 as dabt.size cannot have a value
greater than 3.
But I will have to introduce a check for len > 4 on 32 bit systems (see after).
>
> Independent of this - is bare metal Arm enforcing this same
> alignment restriction (unconditionally)? Iirc on x86 we felt we'd
> better synthesize misaligned accesses.
Unaligned IO access could be synthesise also on arm to but I would
rather not make such a change now without testing it (and there is
also a question of it making sense).
So if it is ok with you I will keep that check and discuss it with Rahul
when he is back. I will add a comment in the code to make that clear.
>
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> + register_t *r, void *p)
>> +{
>> + unsigned int reg;
>> + pci_sbdf_t sbdf;
>> + unsigned long data = ~0UL;
>
> What use is this initializer? The error path further down doesn't
> forward the value into *r, and subsequently the value gets fully
> overwritten.
Right I will remove it.
>
>> + unsigned int size = 1U << info->dabt.size;
>> +
>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>
> This implies segment to be zero. While probably fine for now, I
> wonder if this wouldn't warrant a comment.
I will add the following comment just before:
/* We ignore segment part and always handle segment 0 */
>
>> + reg = REGISTER_OFFSET(info->gpa);
>> +
>> + if ( !vpci_mmio_access_allowed(reg, size) )
>> + return 0;
>> +
>> + data = vpci_read(sbdf, reg, min(4u, size));
>> + if ( size == 8 )
>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>
> Throughout this series I haven't been able to spot where the HAS_VPCI
> Kconfig symbol would get selected. Hence I cannot tell whether all of
> this is Arm64-specific. Otherwise I wonder whether size 8 actually
> can occur on Arm32.
Dabt.size could be 3 even on ARM32 but we should not allow 64bit
access on mmio regions on arm32.
So I will surround this code with ifdef CONFIG_ARM_64 and add a test
for len > 4 to prevent this case on 32bit.
To be completely right we should disable this also for 32bit guests but
this change would be a bit more invasive.
>
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> + register_t r, void *p)
>> +{
>> + unsigned int reg;
>> + pci_sbdf_t sbdf;
>> + unsigned long data = r;
>
> A little like in the read function - what use is this local variable?
> Can't you use r directly?
We can and I will remove the data variable.
>
>> --- a/xen/drivers/passthrough/pci.c
>> +++ b/xen/drivers/passthrough/pci.c
>> @@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>> else
>> iommu_enable_device(pdev);
>
> Please note the context above; ...
>
>> +#ifdef CONFIG_ARM
>> + /*
>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>> + * Dom0 inform XEN to add the PCI devices in XEN.
>> + */
>> + ret = vpci_add_handlers(pdev);
>> + if ( ret )
>> + {
>> + printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
>> + pci_cleanup_msi(pdev);
>> + ret = iommu_remove_device(pdev);
>> + if ( pdev->domain )
>> + list_del(&pdev->domain_list);
>> + free_pdev(pseg, pdev);
>
> ... you unconditionally undo the if() part of the earlier conditional;
> is there any guarantee that the other path can't ever be taken, now
> and forever? If it can't be taken now (which I think is the case as
> long as Dom0 wouldn't report the same device twice), then at least some
> precaution wants taking. Maybe moving your addition into that if()
> could be an option.
>
> Furthermore I continue to wonder whether this ordering is indeed
> preferable over doing software setup before hardware arrangements. This
> would address the above issue as well as long as vpci_add_handlers() is
> idempotent. And it would likely simplify error cleanup.
I agree with you so I will move this code block before iommu part.
But digging deeper into this, I would have 2 questions:
- msi_cleanup was done there after a request from Stefano, but is not
done in case or iommu error, is there an issue to solve here ?
Same could also go for the free_pdev ?
- cleanup code was exactly the same as pci_remove_device code.
Should the question about the path also be checked there ?
Regards
Bertrand
>
> Jan
>
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 12:41 ` Bertrand Marquis
@ 2021-10-11 13:09 ` Jan Beulich
2021-10-11 13:34 ` Bertrand Marquis
2021-10-12 15:04 ` Julien Grall
1 sibling, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-11 13:09 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Rahul Singh, Andre Przywara, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 11.10.2021 14:41, Bertrand Marquis wrote:
>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>> On 06.10.2021 19:40, Rahul Singh wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vpci.c
>>> @@ -0,0 +1,102 @@
>>> +/*
>>> + * xen/arch/arm/vpci.c
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; either version 2 of the License, or
>>> + * (at your option) any later version.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + */
>>> +#include <xen/sched.h>
>>> +
>>> +#include <asm/mmio.h>
>>> +
>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>
>> Nit: Stray blank (like you had in an earlier version for MMCFG_BDF()).
>> Also isn't this effectively part of the public interface (along with
>> MMCFG_BDF()), alongside GUEST_VPCI_ECAM_{BASE,SIZE}?
>
> I will move that in the next version to xen/pci.h and rename it MMCFG_REG_OFFSET.
> Would that be ok ?
That would be okay and make sense when put next to MMCFG_BDF(), but
it would not address my comment: That still wouldn't be the public
interface. Otoh you only mimic hardware behavior, so perhaps the
splitting of the address isn't as relevant to put there as would be
GUEST_VPCI_ECAM_{BASE,SIZE}.
>>> --- a/xen/drivers/passthrough/pci.c
>>> +++ b/xen/drivers/passthrough/pci.c
>>> @@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>> else
>>> iommu_enable_device(pdev);
>>
>> Please note the context above; ...
>>
>>> +#ifdef CONFIG_ARM
>>> + /*
>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>> + */
>>> + ret = vpci_add_handlers(pdev);
>>> + if ( ret )
>>> + {
>>> + printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
>>> + pci_cleanup_msi(pdev);
>>> + ret = iommu_remove_device(pdev);
>>> + if ( pdev->domain )
>>> + list_del(&pdev->domain_list);
>>> + free_pdev(pseg, pdev);
>>
>> ... you unconditionally undo the if() part of the earlier conditional;
>> is there any guarantee that the other path can't ever be taken, now
>> and forever? If it can't be taken now (which I think is the case as
>> long as Dom0 wouldn't report the same device twice), then at least some
>> precaution wants taking. Maybe moving your addition into that if()
>> could be an option.
>>
>> Furthermore I continue to wonder whether this ordering is indeed
>> preferable over doing software setup before hardware arrangements. This
>> would address the above issue as well as long as vpci_add_handlers() is
>> idempotent. And it would likely simplify error cleanup.
>
> I agree with you so I will move this code block before iommu part.
>
> But digging deeper into this, I would have 2 questions:
>
> - msi_cleanup was done there after a request from Stefano, but is not
> done in case or iommu error, is there an issue to solve here ?
Maybe, but I'm not sure. This very much depends on what a domain
could in principle do with a partly set-up device. Plus let's
not forget that we're talking of only Dom0 here (for now at least,
i.e. not considering the dom0less case).
But I'd also like to further defer to Stefano.
> Same could also go for the free_pdev ?
I think it's wrong to free_pdev() here. We want to internally keep
record of the device, even if the device ends up unusable. The only
place where free_pdev() ought to be called is imo pci_remove_device().
> - cleanup code was exactly the same as pci_remove_device code.
> Should the question about the path also be checked there ?
I'm sorry, but I'm afraid I don't see what "the path" refers to
here. You can't mean the conditional in pci_add_device() selecting
between iommu_add_device() and iommu_enable_device(), as "remove"
can only mean "remove", never "disable".
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-11 11:35 ` Jan Beulich
@ 2021-10-11 13:17 ` Roger Pau Monné
0 siblings, 0 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 13:17 UTC (permalink / raw)
To: Jan Beulich
Cc: Michal Orzel, Andrew Cooper, Rahul Singh, xen-devel,
Bertrand Marquis, Andre Przywara, Christian Lindig, David Scott,
Ian Jackson, Wei Liu, George Dunlap, Volodymyr Babchuk,
Julien Grall, Stefano Stabellini
On Mon, Oct 11, 2021 at 01:35:18PM +0200, Jan Beulich wrote:
> On 11.10.2021 13:29, Michal Orzel wrote:
> > On 08.10.2021 23:46, Stefano Stabellini wrote:
> >> On Fri, 8 Oct 2021, Julien Grall wrote:
> >>> On Fri, 8 Oct 2021, 20:07 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:
> >>> On 06/10/2021 18:40, Rahul Singh wrote:
> >>> > Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> >>> > Reject the use of this new flag for x86 as VPCI is not supported for
> >>> > DOMU guests for x86.
> >>> >
> >>> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> >>> > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> >>> > Acked-by: Christian Lindig <christian.lindig@citrix.com>
> >>>
> >>> I'm afraid this logic is broken.
> >>>
> >>> There's no matching feature to indicate to the toolstack whether
> >>> XEN_DOMCTL_CDF_vpci will be accepted or not. The usual way of doing
> >>> this is with a physinfo_cap field.
> >>>
> >>>
> >>> I am slightly puzzled by this. I am assuming you are referring to XENVER_get_features which AFAICT is a stable interface. So why should we
> >>> use it to describe the presence of an unstable interface?
> >>>
> >>>
> >>> This flag needs using in Patch 10 to reject attempts to create a VM with
> >>> devices when passthrough support is unavailable.
> >>>
> >>>
> >>> May I ask why we can't rely on the hypervisor to reject the flag when suitable?
> >>>
> >>>
> >>> Ian, for the 4.16 release, this series either needs completing with the
> >>> additional flag implemented, or this patch needs reverting to avoid us
> >>> shipping a broken interface.
> >>>
> >>>
> >>> I fail to see how the interface would be broken... Would you mind to describe a bit more what could go wrong with this interface?
> >>
> >>
> >> After chatting with Andrew on IRC, this is my understanding.
> >>
> >> Today if pci=[] is specified in the VM config file then
> >> XEN_DOMCTL_CDF_vpci is added. If Xen doesn't support it, Xen returns
> >> an error but libxl/xl won't be able to tell exactly why it fails. So xl
> >> will end up printing a generic VM creation failure. Andrew would like to
> >> see something like the following in libxl:
> >>
> >> if ( PCI_devices && !cap.vcpi )
> >> error("Sorry - PCI not supported")
> >>
> >> So that the user gets a clear informative error back rather than a
> >> generic VM creation failure. Also, this is a requirement for the stable
> >> hypercall interface.
> >>
> >>
> >> I think that's fine and we can implement this request easily by adding
> >> XEN_SYSCTL_PHYSCAP_vpci. Rahul or Bertrand, are you guys happy with
> >> doing that? Otherwise I could take it on.
> >>
> >>
> >> As a side note, given that PCI passthrough support is actually not yet
> >> complete on ARM, we could even just do the following in xl/libxl:
> >>
> >> if ( PCI_devices )
> >> error("Sorry - PCI not supported")
> >>
> >> or temporarily remove XEN_DOMCTL_CDF_vpci until PCI passthrough gets
> >> finalized.
> >>
> > As Rahul is on leave:
> > I'm ok to introduce XEN_SYSCTL_PHYSCAP_vpci. I did the same for vpmu so it's ok.
> > However the problem I have is about setting this cap.
> > On arm it is easy as we are not supporting vpci at the moment so the cap can be set to false.
> > But how to deal with this cap in x86 code? I'm not familiar with x86 so I'm asking for advice.
>
> As the sysctl is mainly from tool stacks to drive guests (DomU-s), I'd
> set it to false for x86 as well. Roger - do you see any reason this
> could be needed to express anything Dom0-related?
I agree, I don't think we should set it to true unless we also support
vPCI for domUs on x86, or else it's just going to confuse the
toolstack.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 12:11 ` Bertrand Marquis
@ 2021-10-11 13:20 ` Roger Pau Monné
2021-10-11 13:40 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 13:20 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini
On Mon, Oct 11, 2021 at 12:11:04PM +0000, Bertrand Marquis wrote:
> Hi Roger,
>
> > On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
> >> ARM architecture does not implement I/O ports. Ignore this call on ARM
> >> to avoid the overhead of making a hypercall just for Xen to return
> >> -ENOSYS.
> >
> > What is the cal trace of this function actually on Arm?
> >
> > AFAICT libxl will only call xc_domain_ioport_permission if there are
> > IO ports explicitly defined in the guest configuration, or if any of
> > the BARs of the PCI device is in the IO space, which is not possible
> > on Arm.
>
> PCI devices BARs can be in the IO space as the PCI devices are not
> Arm specific. There is not ioports on arm so to be used those can be
> in some cases remapped and accessed as MMIOs or are not possible
> to use at all.
>
> But the IO space does appear when BARs are listed even on Arm.
Urg, I wonder whether those devices with IO BARs will work correctly
under Arm then.
How do you know whether the BAR has been remapped from IO space into
MMIO?
IMO instead of faking a successful return value from
xc_domain_ioport_permission we should avoid the call completely in the
first place, specially if we need to instead issue a call to
xc_domain_iomem_permission.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 13:09 ` Jan Beulich
@ 2021-10-11 13:34 ` Bertrand Marquis
2021-10-11 14:10 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 13:34 UTC (permalink / raw)
To: Jan Beulich
Cc: Rahul Singh, Andre Przywara, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
Hi Jan,
> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 11.10.2021 14:41, Bertrand Marquis wrote:
>>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 06.10.2021 19:40, Rahul Singh wrote:
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/vpci.c
>>>> @@ -0,0 +1,102 @@
>>>> +/*
>>>> + * xen/arch/arm/vpci.c
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify
>>>> + * it under the terms of the GNU General Public License as published by
>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>> + * (at your option) any later version.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>> + * GNU General Public License for more details.
>>>> + */
>>>> +#include <xen/sched.h>
>>>> +
>>>> +#include <asm/mmio.h>
>>>> +
>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>
>>> Nit: Stray blank (like you had in an earlier version for MMCFG_BDF()).
>>> Also isn't this effectively part of the public interface (along with
>>> MMCFG_BDF()), alongside GUEST_VPCI_ECAM_{BASE,SIZE}?
>>
>> I will move that in the next version to xen/pci.h and rename itMMCFG_REG_OFFSET.
>> Would that be ok ?
>
> That would be okay and make sense when put next to MMCFG_BDF(), but
> it would not address my comment: That still wouldn't be the public
> interface. Otoh you only mimic hardware behavior, so perhaps the
> splitting of the address isn't as relevant to put there as would be
> GUEST_VPCI_ECAM_{BASE,SIZE}.
Ok now I get what you wanted.
You would actually like both MMCFG_BDF and MMCFG_REG_OFFSET to
be moved to arch-arm.h.
Then I am not quite sure to follow why.
Those are not macros coming out of a way we have to define this but from
how it works in standard PCI.
The base and size are needed to know where the PCI bus will be.
So why should those be needed in public headers ?
>
>>>> --- a/xen/drivers/passthrough/pci.c
>>>> +++ b/xen/drivers/passthrough/pci.c
>>>> @@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>> else
>>>> iommu_enable_device(pdev);
>>>
>>> Please note the context above; ...
>>>
>>>> +#ifdef CONFIG_ARM
>>>> + /*
>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>> + */
>>>> + ret = vpci_add_handlers(pdev);
>>>> + if ( ret )
>>>> + {
>>>> + printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
>>>> + pci_cleanup_msi(pdev);
>>>> + ret = iommu_remove_device(pdev);
>>>> + if ( pdev->domain )
>>>> + list_del(&pdev->domain_list);
>>>> + free_pdev(pseg, pdev);
>>>
>>> ... you unconditionally undo the if() part of the earlier conditional;
>>> is there any guarantee that the other path can't ever be taken, now
>>> and forever? If it can't be taken now (which I think is the case as
>>> long as Dom0 wouldn't report the same device twice), then at least some
>>> precaution wants taking. Maybe moving your addition into that if()
>>> could be an option.
>>>
>>> Furthermore I continue to wonder whether this ordering is indeed
>>> preferable over doing software setup before hardware arrangements. This
>>> would address the above issue as well as long as vpci_add_handlers() is
>>> idempotent. And it would likely simplify error cleanup.
>>
>> I agree with you so I will move this code block before iommu part.
>>
>> But digging deeper into this, I would have 2 questions:
>>
>> - msi_cleanup was done there after a request from Stefano, but is not
>> done in case or iommu error, is there an issue to solve here ?
>
> Maybe, but I'm not sure. This very much depends on what a domain
> could in principle do with a partly set-up device. Plus let's
> not forget that we're talking of only Dom0 here (for now at least,
> i.e. not considering the dom0less case).
>
> But I'd also like to further defer to Stefano.
Ok, I must admit I do not really see at that stage why doing an MSI cleanup
could be needed so I will wait for Stefano to know if I need to keep this when
moving the block up (at the end it is theoretical right now as this is empty).
>
>> Same could also go for the free_pdev ?
>
> I think it's wrong to free_pdev() here. We want to internally keep
> record of the device, even if the device ends up unusable. The only
> place where free_pdev() ought to be called is imo pci_remove_device().
Ok.
>
>> - cleanup code was exactly the same as pci_remove_device code.
>> Should the question about the path also be checked there ?
>
> I'm sorry, but I'm afraid I don't see what "the path" refers to
> here. You can't mean the conditional in pci_add_device() selecting
> between iommu_add_device() and iommu_enable_device(), as "remove"
> can only mean "remove", never "disable".
I will try to explain: when we just enable we do not add an entry in the list but
we still remove an entry from the list every time (as the condition becomes
always true because pdev->domain is at the end always set)
Regards
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 13:20 ` Roger Pau Monné
@ 2021-10-11 13:40 ` Bertrand Marquis
2021-10-11 13:57 ` Roger Pau Monné
2021-10-11 14:16 ` Oleksandr Andrushchenko
0 siblings, 2 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 13:40 UTC (permalink / raw)
To: Roger Pau Monné, Oleksandr Andrushchenko
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini
Hi Roger,
+ Oleksandr to have a better PCI expert then me.
> On 11 Oct 2021, at 14:20, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Mon, Oct 11, 2021 at 12:11:04PM +0000, Bertrand Marquis wrote:
>> Hi Roger,
>>
>>> On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
>>>> ARM architecture does not implement I/O ports. Ignore this call on ARM
>>>> to avoid the overhead of making a hypercall just for Xen to return
>>>> -ENOSYS.
>>>
>>> What is the cal trace of this function actually on Arm?
>>>
>>> AFAICT libxl will only call xc_domain_ioport_permission if there are
>>> IO ports explicitly defined in the guest configuration, or if any of
>>> the BARs of the PCI device is in the IO space, which is not possible
>>> on Arm.
>>
>> PCI devices BARs can be in the IO space as the PCI devices are not
>> Arm specific. There is not ioports on arm so to be used those can be
>> in some cases remapped and accessed as MMIOs or are not possible
>> to use at all.
>>
>> But the IO space does appear when BARs are listed even on Arm.
>
> Urg, I wonder whether those devices with IO BARs will work correctly
> under Arm then.
>
> How do you know whether the BAR has been remapped from IO space into
> MMIO?
We cannot, I think the platform will define if this is the case and where.
@oleksandr: I remember that this was discussed during some of our
meetings but I have no idea of the details here, can you help ?
>
> IMO instead of faking a successful return value from
> xc_domain_ioport_permission we should avoid the call completely in the
> first place, specially if we need to instead issue a call to
> xc_domain_iomem_permission.
At the end we will never have to issue this because this will never be a matter
of “iomem” permission as there would not be any way to cut on something under
the page. If this is to be supported one day, it will probably have to be fully emulated
to keep the isolation.
Right now on arm you can just make the more simple assumption that ioports are
just not supported.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-07 16:11 ` Ian Jackson
@ 2021-10-11 13:46 ` Roger Pau Monné
2021-10-14 17:16 ` Bertrand Marquis
1 sibling, 0 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 13:46 UTC (permalink / raw)
To: Ian Jackson
Cc: Rahul Singh, Julien Grall, xen-devel, Bertrand Marquis,
Andre Przywara, Wei Liu, Anthony PERARD, Juergen Gross,
Stefano Stabellini, Volodymyr Babchuk
On Thu, Oct 07, 2021 at 05:11:47PM +0100, Ian Jackson wrote:
> Rahul Singh writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
> > As Stefano suggested in another email that we can remove the vpci
> > option, if we reach to conclusion that we need vpci option I will
> > move it to internal structure.
> ...
> > Yes I agree with you VPCI is necessary for hot plugged PCI device
> > and once we implement the hotplug in future we will use the
> > passthrough= option to enable VPCI.
>
> So, to summarise, I think the situation is:
>
> * VCPI is necessry for passthrough on ARM, whether coldplug or
> hotplug. It's part of the way that PCI-PT works on ARM.
>
> * Hotplug is not yet implemented.
>
> * VPCI is not necessary on x86 (evidently, since we don't have it
> there but we do have passthrough).
>
> So when hotplug is added, vpci will need to be turned on when
> passthrough=yes is selected. I don't fully understand the other
> possible values for passthrough= but maybe we can defer the question
> of whether they apply to ARM ?
>
> I think that means that yes, this should be an internal variable.
> Probably in libxl__domain_create_state. We don't currently arrange to
> elide arch-specific state in there, so perhaps it's fine just to
> invent a member called `arm_vpci`.
Seeing as we might want to also use this on x86, I wonder whether we
should allow to specify a backend to use for each passthrough device,
ie:
pci = [ '36:00.0,backend=vpci',
'36:01.0,backend=qemu',
... ]
In principle we should support different backends on a per-device
basis, albeit if not currently possible.
Iff we have to introduce a new struct member for vPCI it should be
shared between all arches, as it's likely x86 will also want to at
least have the option to use vPCI for passthrough.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 13:40 ` Bertrand Marquis
@ 2021-10-11 13:57 ` Roger Pau Monné
2021-10-11 14:16 ` Bertrand Marquis
2021-10-11 14:16 ` Oleksandr Andrushchenko
1 sibling, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 13:57 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini
On Mon, Oct 11, 2021 at 01:40:30PM +0000, Bertrand Marquis wrote:
> Hi Roger,
>
> + Oleksandr to have a better PCI expert then me.
>
> > On 11 Oct 2021, at 14:20, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Mon, Oct 11, 2021 at 12:11:04PM +0000, Bertrand Marquis wrote:
> >> Hi Roger,
> >>
> >>> On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>
> >>> On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
> >>>> ARM architecture does not implement I/O ports. Ignore this call on ARM
> >>>> to avoid the overhead of making a hypercall just for Xen to return
> >>>> -ENOSYS.
> >>>
> >>> What is the cal trace of this function actually on Arm?
> >>>
> >>> AFAICT libxl will only call xc_domain_ioport_permission if there are
> >>> IO ports explicitly defined in the guest configuration, or if any of
> >>> the BARs of the PCI device is in the IO space, which is not possible
> >>> on Arm.
> >>
> >> PCI devices BARs can be in the IO space as the PCI devices are not
> >> Arm specific. There is not ioports on arm so to be used those can be
> >> in some cases remapped and accessed as MMIOs or are not possible
> >> to use at all.
> >>
> >> But the IO space does appear when BARs are listed even on Arm.
> >
> > Urg, I wonder whether those devices with IO BARs will work correctly
> > under Arm then.
> >
> > How do you know whether the BAR has been remapped from IO space into
> > MMIO?
>
> We cannot, I think the platform will define if this is the case and where.
> @oleksandr: I remember that this was discussed during some of our
> meetings but I have no idea of the details here, can you help ?
>
> >
> > IMO instead of faking a successful return value from
> > xc_domain_ioport_permission we should avoid the call completely in the
> > first place, specially if we need to instead issue a call to
> > xc_domain_iomem_permission.
>
> At the end we will never have to issue this because this will never be a matter
> of “iomem” permission as there would not be any way to cut on something under
> the page. If this is to be supported one day, it will probably have to be fully emulated
> to keep the isolation.
So you have a set of memory pages that map accesses from
MMIO into IO space but it's not possible to isolate specific IO port
regions as they are all contiguous in the same page(s).
> Right now on arm you can just make the more simple assumption that ioports are
> just not supported.
Would it make sense in the future to provide a memory region to guests
in order to use for IO port accesses, and call
xc_domain_ioport_permission to set which ports would be allowed?
I think the commit message needs to at least be expanded in order to
contain the information provided here. It might also be helpful to
figure out whether we would have to handle IO port accesses in the
future on Arm, or if it's fine to just ignore them.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 13:34 ` Bertrand Marquis
@ 2021-10-11 14:10 ` Jan Beulich
2021-10-11 14:52 ` Bertrand Marquis
2021-10-11 18:18 ` Stefano Stabellini
0 siblings, 2 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-11 14:10 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Rahul Singh, Andre Przywara, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 11.10.2021 15:34, Bertrand Marquis wrote:
>> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
>> On 11.10.2021 14:41, Bertrand Marquis wrote:
>>>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>>>> On 06.10.2021 19:40, Rahul Singh wrote:
>>>>> --- /dev/null
>>>>> +++ b/xen/arch/arm/vpci.c
>>>>> @@ -0,0 +1,102 @@
>>>>> +/*
>>>>> + * xen/arch/arm/vpci.c
>>>>> + *
>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>> + * it under the terms of the GNU General Public License as published by
>>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>>> + * (at your option) any later version.
>>>>> + *
>>>>> + * This program is distributed in the hope that it will be useful,
>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>>> + * GNU General Public License for more details.
>>>>> + */
>>>>> +#include <xen/sched.h>
>>>>> +
>>>>> +#include <asm/mmio.h>
>>>>> +
>>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>>
>>>> Nit: Stray blank (like you had in an earlier version for MMCFG_BDF()).
>>>> Also isn't this effectively part of the public interface (along with
>>>> MMCFG_BDF()), alongside GUEST_VPCI_ECAM_{BASE,SIZE}?
>>>
>>> I will move that in the next version to xen/pci.h and rename itMMCFG_REG_OFFSET.
>>> Would that be ok ?
>>
>> That would be okay and make sense when put next to MMCFG_BDF(), but
>> it would not address my comment: That still wouldn't be the public
>> interface. Otoh you only mimic hardware behavior, so perhaps the
>> splitting of the address isn't as relevant to put there as would be
>> GUEST_VPCI_ECAM_{BASE,SIZE}.
>
> Ok now I get what you wanted.
>
> You would actually like both MMCFG_BDF and MMCFG_REG_OFFSET to
> be moved to arch-arm.h.
>
> Then I am not quite sure to follow why.
> Those are not macros coming out of a way we have to define this but from
> how it works in standard PCI.
> The base and size are needed to know where the PCI bus will be.
>
> So why should those be needed in public headers ?
Well, see my "Otoh ..." in the earlier reply. Keeping the two
address splitting macros out of there is okay.
>>>>> --- a/xen/drivers/passthrough/pci.c
>>>>> +++ b/xen/drivers/passthrough/pci.c
>>>>> @@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>> else
>>>>> iommu_enable_device(pdev);
>>>>
>>>> Please note the context above; ...
>>>>
>>>>> +#ifdef CONFIG_ARM
>>>>> + /*
>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>>> + */
>>>>> + ret = vpci_add_handlers(pdev);
>>>>> + if ( ret )
>>>>> + {
>>>>> + printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
>>>>> + pci_cleanup_msi(pdev);
>>>>> + ret = iommu_remove_device(pdev);
>>>>> + if ( pdev->domain )
>>>>> + list_del(&pdev->domain_list);
>>>>> + free_pdev(pseg, pdev);
>>>>
>>>> ... you unconditionally undo the if() part of the earlier conditional;
>>>> is there any guarantee that the other path can't ever be taken, now
>>>> and forever? If it can't be taken now (which I think is the case as
>>>> long as Dom0 wouldn't report the same device twice), then at least some
>>>> precaution wants taking. Maybe moving your addition into that if()
>>>> could be an option.
>>>>
>>>> Furthermore I continue to wonder whether this ordering is indeed
>>>> preferable over doing software setup before hardware arrangements. This
>>>> would address the above issue as well as long as vpci_add_handlers() is
>>>> idempotent. And it would likely simplify error cleanup.
>>>
>>> I agree with you so I will move this code block before iommu part.
>>>
>>> But digging deeper into this, I would have 2 questions:
>>>
>>> - msi_cleanup was done there after a request from Stefano, but is not
>>> done in case or iommu error, is there an issue to solve here ?
>>
>> Maybe, but I'm not sure. This very much depends on what a domain
>> could in principle do with a partly set-up device. Plus let's
>> not forget that we're talking of only Dom0 here (for now at least,
>> i.e. not considering the dom0less case).
>>
>> But I'd also like to further defer to Stefano.
>
> Ok, I must admit I do not really see at that stage why doing an MSI cleanup
> could be needed so I will wait for Stefano to know if I need to keep this when
> moving the block up (at the end it is theoretical right now as this is empty).
>
>>
>>> Same could also go for the free_pdev ?
>>
>> I think it's wrong to free_pdev() here. We want to internally keep
>> record of the device, even if the device ends up unusable. The only
>> place where free_pdev() ought to be called is imo pci_remove_device().
>
> Ok.
>
>>
>>> - cleanup code was exactly the same as pci_remove_device code.
>>> Should the question about the path also be checked there ?
>>
>> I'm sorry, but I'm afraid I don't see what "the path" refers to
>> here. You can't mean the conditional in pci_add_device() selecting
>> between iommu_add_device() and iommu_enable_device(), as "remove"
>> can only mean "remove", never "disable".
>
> I will try to explain: when we just enable we do not add an entry in the list but
> we still remove an entry from the list every time (as the condition becomes
> always true because pdev->domain is at the end always set)
Well, that anomaly is what I did point out in my review remarks to
Rahul. We shouldn't remove an entry from the list if we didn't add
one. But quite likely, if we don't free the pdev, we shouldn't be
removing the list entry in either case.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 13:40 ` Bertrand Marquis
2021-10-11 13:57 ` Roger Pau Monné
@ 2021-10-11 14:16 ` Oleksandr Andrushchenko
1 sibling, 0 replies; 190+ messages in thread
From: Oleksandr Andrushchenko @ 2021-10-11 14:16 UTC (permalink / raw)
To: Bertrand Marquis, Roger Pau Monné
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini
Hi, all
On 11.10.21 16:40, Bertrand Marquis wrote:
> Hi Roger,
>
> + Oleksandr to have a better PCI expert then me.
>
>> On 11 Oct 2021, at 14:20, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>
>> On Mon, Oct 11, 2021 at 12:11:04PM +0000, Bertrand Marquis wrote:
>>> Hi Roger,
>>>
>>>> On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>
>>>> On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
>>>>> ARM architecture does not implement I/O ports. Ignore this call on ARM
>>>>> to avoid the overhead of making a hypercall just for Xen to return
>>>>> -ENOSYS.
>>>> What is the cal trace of this function actually on Arm?
>>>>
>>>> AFAICT libxl will only call xc_domain_ioport_permission if there are
>>>> IO ports explicitly defined in the guest configuration, or if any of
>>>> the BARs of the PCI device is in the IO space, which is not possible
>>>> on Arm.
>>> PCI devices BARs can be in the IO space as the PCI devices are not
>>> Arm specific. There is not ioports on arm so to be used those can be
>>> in some cases remapped and accessed as MMIOs or are not possible
>>> to use at all.
>>>
>>> But the IO space does appear when BARs are listed even on Arm.
>> Urg, I wonder whether those devices with IO BARs will work correctly
>> under Arm then.
>>
>> How do you know whether the BAR has been remapped from IO space into
>> MMIO?
> We cannot, I think the platform will define if this is the case and where.
> @oleksandr: I remember that this was discussed during some of our
> meetings but I have no idea of the details here, can you help ?
>
For the guest domains we emulate a host bridge without IO, so the
guest won't be able to assign any IO memory at all.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 13:57 ` Roger Pau Monné
@ 2021-10-11 14:16 ` Bertrand Marquis
2021-10-11 16:32 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 14:16 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini
Hi Roger,
> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Mon, Oct 11, 2021 at 01:40:30PM +0000, Bertrand Marquis wrote:
>> Hi Roger,
>>
>> + Oleksandr to have a better PCI expert then me.
>>
>>> On 11 Oct 2021, at 14:20, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Mon, Oct 11, 2021 at 12:11:04PM +0000, Bertrand Marquis wrote:
>>>> Hi Roger,
>>>>
>>>>> On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>
>>>>> On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
>>>>>> ARM architecture does not implement I/O ports. Ignore this call on ARM
>>>>>> to avoid the overhead of making a hypercall just for Xen to return
>>>>>> -ENOSYS.
>>>>>
>>>>> What is the cal trace of this function actually on Arm?
>>>>>
>>>>> AFAICT libxl will only call xc_domain_ioport_permission if there are
>>>>> IO ports explicitly defined in the guest configuration, or if any of
>>>>> the BARs of the PCI device is in the IO space, which is not possible
>>>>> on Arm.
>>>>
>>>> PCI devices BARs can be in the IO space as the PCI devices are not
>>>> Arm specific. There is not ioports on arm so to be used those can be
>>>> in some cases remapped and accessed as MMIOs or are not possible
>>>> to use at all.
>>>>
>>>> But the IO space does appear when BARs are listed even on Arm.
>>>
>>> Urg, I wonder whether those devices with IO BARs will work correctly
>>> under Arm then.
>>>
>>> How do you know whether the BAR has been remapped from IO space into
>>> MMIO?
>>
>> We cannot, I think the platform will define if this is the case and where.
>> @oleksandr: I remember that this was discussed during some of our
>> meetings but I have no idea of the details here, can you help ?
>>
>>>
>>> IMO instead of faking a successful return value from
>>> xc_domain_ioport_permission we should avoid the call completely in the
>>> first place, specially if we need to instead issue a call to
>>> xc_domain_iomem_permission.
>>
>> At the end we will never have to issue this because this will never be a matter
>> of “iomem” permission as there would not be any way to cut on something under
>> the page. If this is to be supported one day, it will probably have to be fully emulated
>> to keep the isolation.
>
> So you have a set of memory pages that map accesses from
> MMIO into IO space but it's not possible to isolate specific IO port
> regions as they are all contiguous in the same page(s).
Exact.
>
>> Right now on arm you can just make the more simple assumption that ioports are
>> just not supported.
>
> Would it make sense in the future to provide a memory region to guests
> in order to use for IO port accesses, and call
> xc_domain_ioport_permission to set which ports would be allowed?
Right now we do not plan to support this at all and we will have to
figure this out if we do this one day.
>
> I think the commit message needs to at least be expanded in order to
> contain the information provided here. It might also be helpful to
> figure out whether we would have to handle IO port accesses in the
> future on Arm, or if it's fine to just ignore them.
All our investigations and tests have been done without supporting it
without any issues so this is not a critical feature (most devices can
be operated without using the I/O ports).
I can add the following to the commit message:
I/O ports accessible through MMIO are currently not supported by Xen.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 14:10 ` Jan Beulich
@ 2021-10-11 14:52 ` Bertrand Marquis
2021-10-11 18:18 ` Stefano Stabellini
1 sibling, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 14:52 UTC (permalink / raw)
To: Jan Beulich
Cc: Rahul Singh, Andre Przywara, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
Hi Jan,
> On 11 Oct 2021, at 15:10, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 11.10.2021 15:34, Bertrand Marquis wrote:
>>> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 11.10.2021 14:41, Bertrand Marquis wrote:
>>>>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> On 06.10.2021 19:40, Rahul Singh wrote:
>>>>>> --- /dev/null
>>>>>> +++ b/xen/arch/arm/vpci.c
>>>>>> @@ -0,0 +1,102 @@
>>>>>> +/*
>>>>>> + * xen/arch/arm/vpci.c
>>>>>> + *
>>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>>> + * it under the terms of the GNU General Public License as published by
>>>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>>>> + * (at your option) any later version.
>>>>>> + *
>>>>>> + * This program is distributed in the hope that it will be useful,
>>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>>>> + * GNU General Public License for more details.
>>>>>> + */
>>>>>> +#include <xen/sched.h>
>>>>>> +
>>>>>> +#include <asm/mmio.h>
>>>>>> +
>>>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>>>
>>>>> Nit: Stray blank (like you had in an earlier version for MMCFG_BDF()).
>>>>> Also isn't this effectively part of the public interface (along with
>>>>> MMCFG_BDF()), alongside GUEST_VPCI_ECAM_{BASE,SIZE}?
>>>>
>>>> I will move that in the next version to xen/pci.h and rename itMMCFG_REG_OFFSET.
>>>> Would that be ok ?
>>>
>>> That would be okay and make sense when put next to MMCFG_BDF(), but
>>> it would not address my comment: That still wouldn't be the public
>>> interface. Otoh you only mimic hardware behavior, so perhaps the
>>> splitting of the address isn't as relevant to put there as would be
>>> GUEST_VPCI_ECAM_{BASE,SIZE}.
>>
>> Ok now I get what you wanted.
>>
>> You would actually like both MMCFG_BDF and MMCFG_REG_OFFSET to
>> be moved to arch-arm.h.
>>
>> Then I am not quite sure to follow why.
>> Those are not macros coming out of a way we have to define this but from
>> how it works in standard PCI.
>> The base and size are needed to know where the PCI bus will be.
>>
>> So why should those be needed in public headers ?
>
> Well, see my "Otoh ..." in the earlier reply. Keeping the two
> address splitting macros out of there is okay.
Ok.
>
>>>>>> --- a/xen/drivers/passthrough/pci.c
>>>>>> +++ b/xen/drivers/passthrough/pci.c
>>>>>> @@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>>> else
>>>>>> iommu_enable_device(pdev);
>>>>>
>>>>> Please note the context above; ...
>>>>>
>>>>>> +#ifdef CONFIG_ARM
>>>>>> + /*
>>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>>>> + */
>>>>>> + ret = vpci_add_handlers(pdev);
>>>>>> + if ( ret )
>>>>>> + {
>>>>>> + printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
>>>>>> + pci_cleanup_msi(pdev);
>>>>>> + ret = iommu_remove_device(pdev);
>>>>>> + if ( pdev->domain )
>>>>>> + list_del(&pdev->domain_list);
>>>>>> + free_pdev(pseg, pdev);
>>>>>
>>>>> ... you unconditionally undo the if() part of the earlier conditional;
>>>>> is there any guarantee that the other path can't ever be taken, now
>>>>> and forever? If it can't be taken now (which I think is the case as
>>>>> long as Dom0 wouldn't report the same device twice), then at least some
>>>>> precaution wants taking. Maybe moving your addition into that if()
>>>>> could be an option.
>>>>>
>>>>> Furthermore I continue to wonder whether this ordering is indeed
>>>>> preferable over doing software setup before hardware arrangements. This
>>>>> would address the above issue as well as long as vpci_add_handlers() is
>>>>> idempotent. And it would likely simplify error cleanup.
>>>>
>>>> I agree with you so I will move this code block before iommu part.
>>>>
>>>> But digging deeper into this, I would have 2 questions:
>>>>
>>>> - msi_cleanup was done there after a request from Stefano, but is not
>>>> done in case or iommu error, is there an issue to solve here ?
>>>
>>> Maybe, but I'm not sure. This very much depends on what a domain
>>> could in principle do with a partly set-up device. Plus let's
>>> not forget that we're talking of only Dom0 here (for now at least,
>>> i.e. not considering the dom0less case).
>>>
>>> But I'd also like to further defer to Stefano.
>>
>> Ok, I must admit I do not really see at that stage why doing an MSI cleanup
>> could be needed so I will wait for Stefano to know if I need to keep this when
>> moving the block up (at the end it is theoretical right now as this is empty).
>>
>>>
>>>> Same could also go for the free_pdev ?
>>>
>>> I think it's wrong to free_pdev() here. We want to internally keep
>>> record of the device, even if the device ends up unusable. The only
>>> place where free_pdev() ought to be called is imo pci_remove_device().
>>
>> Ok.
>>
>>>
>>>> - cleanup code was exactly the same as pci_remove_device code.
>>>> Should the question about the path also be checked there ?
>>>
>>> I'm sorry, but I'm afraid I don't see what "the path" refers to
>>> here. You can't mean the conditional in pci_add_device() selecting
>>> between iommu_add_device() and iommu_enable_device(), as "remove"
>>> can only mean "remove", never "disable".
>>
>> I will try to explain: when we just enable we do not add an entry in the list but
>> we still remove an entry from the list every time (as the condition becomes
>> always true because pdev->domain is at the end always set)
>
> Well, that anomaly is what I did point out in my review remarks to
> Rahul. We shouldn't remove an entry from the list if we didn't add
> one. But quite likely, if we don't free the pdev, we shouldn't be
> removing the list entry in either case.
This problem will not exist anymore when I will move the code up but I will add to adapt the error case in iommu to also remove the vpci handlers.
To be coherent I will do the same in the pci_remove_device code.
I will do all those in the v6 of the serie.
Thanks a lot for the answers.
Cheers
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 10:51 ` Roger Pau Monné
@ 2021-10-11 16:12 ` Bertrand Marquis
2021-10-11 16:20 ` Oleksandr Andrushchenko
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 16:12 UTC (permalink / raw)
To: Roger Pau Monné, Oleksandr Andrushchenko
Cc: Rahul Singh, xen-devel, Andre Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Paul Durrant
Hi Roger,
> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
>>
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>>
>> For Dom0less systems scan_pci_devices() would be used to discover the
>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> ---
>> Change in v5:
>> - Add pci_cleanup_msi(pdev) in cleanup part.
>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> Change in v4:
>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>> Change in v3:
>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>> Change in v2:
>> - Add new XEN_DOMCTL_CDF_vpci flag
>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>> - enable vpci support when pci-passthough option is enabled.
>> ---
>> ---
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/domain.c | 4 ++
>> xen/arch/arm/domain_build.c | 3 +
>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>> xen/arch/arm/vpci.h | 36 ++++++++++++
>> xen/drivers/passthrough/pci.c | 18 ++++++
>> xen/include/asm-arm/domain.h | 7 ++-
>> xen/include/asm-x86/pci.h | 2 -
>> xen/include/public/arch-arm.h | 7 +++
>> xen/include/xen/pci.h | 2 +
>> 10 files changed, 179 insertions(+), 3 deletions(-)
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 44d7cc81fa..fb9c976ea2 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index 36138c1b2e..fbb52f78f1 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -39,6 +39,7 @@
>> #include <asm/vgic.h>
>> #include <asm/vtimer.h>
>>
>> +#include "vpci.h"
>> #include "vuart.h"
>>
>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>> goto fail;
>>
>> + if ( (rc = domain_vpci_init(d)) != 0 )
>> + goto fail;
>> +
>> return 0;
>>
>> fail:
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index c5afbe2e05..f4c89bde8c 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>> if ( iommu_enabled )
>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>
>> + if ( is_pci_passthrough_enabled() )
>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>
> I think I'm confused with this. You seem to enable vPCI for dom0, but
> then domain_vpci_init will setup traps for the guest virtual ECAM
> layout, not the native one that dom0 will be using.
I think after the last discussions, it was decided to also installed the vpci
handler for dom0. I will have to look into this and come back to you.
@Oleksandr: Could you comment on this.
>
>> +
>> dom0 = domain_create(0, &dom0_cfg, true);
>> if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
>> panic("Error creating domain 0\n");
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..76c12b9281
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,102 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +
>> +#include <asm/mmio.h>
>> +
>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>> +
>> +/* Do some sanity checks. */
>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> + /* Check access size. */
>> + if ( len > 8 )
>> + return false;
>> +
>> + /* Check that access is size aligned. */
>> + if ( (reg & (len - 1)) )
>> + return false;
>> +
>> + return true;
>> +}
>
> There's a vpci_access_allowed which I think you could generalize and
> use here, there's no need to have this duplicated code.
It definitely looks exactly like what we want to do.
I would need to turn this into a static inline and put it in some global header.
It is currently in arch/x86/hvm/io.c, any suggestion on the header to move it to ?
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 16:12 ` Bertrand Marquis
@ 2021-10-11 16:20 ` Oleksandr Andrushchenko
2021-10-11 16:43 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Oleksandr Andrushchenko @ 2021-10-11 16:20 UTC (permalink / raw)
To: Bertrand Marquis, Roger Pau Monné
Cc: Rahul Singh, xen-devel, Andre Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Paul Durrant,
Oleksandr Andrushchenko
On 11.10.21 19:12, Bertrand Marquis wrote:
> Hi Roger,
>
>> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>
>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>> The existing VPCI support available for X86 is adapted for Arm.
>>> When the device is added to XEN via the hyper call
>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>> access is added to the Xen to emulate the PCI devices config space.
>>>
>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>> so that when guest is trying to access the PCI config space,XEN
>>> will trap the access and emulate read/write using the VPCI and
>>> not the real PCI hardware.
>>>
>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>> ---
>>> Change in v5:
>>> - Add pci_cleanup_msi(pdev) in cleanup part.
>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>> Change in v4:
>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>> Change in v3:
>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>> Change in v2:
>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>> - enable vpci support when pci-passthough option is enabled.
>>> ---
>>> ---
>>> xen/arch/arm/Makefile | 1 +
>>> xen/arch/arm/domain.c | 4 ++
>>> xen/arch/arm/domain_build.c | 3 +
>>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>>> xen/arch/arm/vpci.h | 36 ++++++++++++
>>> xen/drivers/passthrough/pci.c | 18 ++++++
>>> xen/include/asm-arm/domain.h | 7 ++-
>>> xen/include/asm-x86/pci.h | 2 -
>>> xen/include/public/arch-arm.h | 7 +++
>>> xen/include/xen/pci.h | 2 +
>>> 10 files changed, 179 insertions(+), 3 deletions(-)
>>> create mode 100644 xen/arch/arm/vpci.c
>>> create mode 100644 xen/arch/arm/vpci.h
>>>
>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>> index 44d7cc81fa..fb9c976ea2 100644
>>> --- a/xen/arch/arm/Makefile
>>> +++ b/xen/arch/arm/Makefile
>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>> obj-y += platforms/
>>> endif
>>> obj-$(CONFIG_TEE) += tee/
>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>
>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>> obj-y += bootfdt.init.o
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index 36138c1b2e..fbb52f78f1 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -39,6 +39,7 @@
>>> #include <asm/vgic.h>
>>> #include <asm/vtimer.h>
>>>
>>> +#include "vpci.h"
>>> #include "vuart.h"
>>>
>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>> goto fail;
>>>
>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>> + goto fail;
>>> +
>>> return 0;
>>>
>>> fail:
>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>> index c5afbe2e05..f4c89bde8c 100644
>>> --- a/xen/arch/arm/domain_build.c
>>> +++ b/xen/arch/arm/domain_build.c
>>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>>> if ( iommu_enabled )
>>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>>
>>> + if ( is_pci_passthrough_enabled() )
>>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>> I think I'm confused with this. You seem to enable vPCI for dom0, but
>> then domain_vpci_init will setup traps for the guest virtual ECAM
>> layout, not the native one that dom0 will be using.
> I think after the last discussions, it was decided to also installed the vpci
> handler for dom0. I will have to look into this and come back to you.
> @Oleksandr: Could you comment on this.
Yes, we do trap Dom0 as well. The Dom0 traps are not in this series, but
are in mine as it needs more preparatory work for that. Please see [1]
Thank you,
Oleksandr
[1] https://patchwork.kernel.org/project/xen-devel/patch/20211008055535.337436-9-andr2000@gmail.com/
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 14:16 ` Bertrand Marquis
@ 2021-10-11 16:32 ` Roger Pau Monné
2021-10-11 17:11 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 16:32 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini
On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
> Hi Roger,
>
> > On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Mon, Oct 11, 2021 at 01:40:30PM +0000, Bertrand Marquis wrote:
> >> Hi Roger,
> >>
> >> + Oleksandr to have a better PCI expert then me.
> >>
> >>> On 11 Oct 2021, at 14:20, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>
> >>> On Mon, Oct 11, 2021 at 12:11:04PM +0000, Bertrand Marquis wrote:
> >>>> Hi Roger,
> >>>>
> >>>>> On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>>>
> >>>>> On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
> >>>>>> ARM architecture does not implement I/O ports. Ignore this call on ARM
> >>>>>> to avoid the overhead of making a hypercall just for Xen to return
> >>>>>> -ENOSYS.
> >>>>>
> >>>>> What is the cal trace of this function actually on Arm?
> >>>>>
> >>>>> AFAICT libxl will only call xc_domain_ioport_permission if there are
> >>>>> IO ports explicitly defined in the guest configuration, or if any of
> >>>>> the BARs of the PCI device is in the IO space, which is not possible
> >>>>> on Arm.
> >>>>
> >>>> PCI devices BARs can be in the IO space as the PCI devices are not
> >>>> Arm specific. There is not ioports on arm so to be used those can be
> >>>> in some cases remapped and accessed as MMIOs or are not possible
> >>>> to use at all.
> >>>>
> >>>> But the IO space does appear when BARs are listed even on Arm.
> >>>
> >>> Urg, I wonder whether those devices with IO BARs will work correctly
> >>> under Arm then.
> >>>
> >>> How do you know whether the BAR has been remapped from IO space into
> >>> MMIO?
> >>
> >> We cannot, I think the platform will define if this is the case and where.
> >> @oleksandr: I remember that this was discussed during some of our
> >> meetings but I have no idea of the details here, can you help ?
> >>
> >>>
> >>> IMO instead of faking a successful return value from
> >>> xc_domain_ioport_permission we should avoid the call completely in the
> >>> first place, specially if we need to instead issue a call to
> >>> xc_domain_iomem_permission.
> >>
> >> At the end we will never have to issue this because this will never be a matter
> >> of “iomem” permission as there would not be any way to cut on something under
> >> the page. If this is to be supported one day, it will probably have to be fully emulated
> >> to keep the isolation.
> >
> > So you have a set of memory pages that map accesses from
> > MMIO into IO space but it's not possible to isolate specific IO port
> > regions as they are all contiguous in the same page(s).
>
> Exact.
>
> >
> >> Right now on arm you can just make the more simple assumption that ioports are
> >> just not supported.
> >
> > Would it make sense in the future to provide a memory region to guests
> > in order to use for IO port accesses, and call
> > xc_domain_ioport_permission to set which ports would be allowed?
>
> Right now we do not plan to support this at all and we will have to
> figure this out if we do this one day.
>
> >
> > I think the commit message needs to at least be expanded in order to
> > contain the information provided here. It might also be helpful to
> > figure out whether we would have to handle IO port accesses in the
> > future on Arm, or if it's fine to just ignore them.
>
> All our investigations and tests have been done without supporting it
> without any issues so this is not a critical feature (most devices can
> be operated without using the I/O ports).
IMO we should let the users know they attempted to use a device with
BARs in the IO space, and that those BARs won't be accessible which
could make the device not function as expected.
Do you think it would be reasonable to attempt the hypercall on Arm
also, and in case of error (on Arm) just print a warning message and
continue operations as normal?
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 16:20 ` Oleksandr Andrushchenko
@ 2021-10-11 16:43 ` Roger Pau Monné
2021-10-11 17:15 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-11 16:43 UTC (permalink / raw)
To: Oleksandr Andrushchenko
Cc: Bertrand Marquis, Rahul Singh, xen-devel, Andre Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Ian Jackson, Jan Beulich, Wei Liu,
Paul Durrant
On Mon, Oct 11, 2021 at 04:20:14PM +0000, Oleksandr Andrushchenko wrote:
>
>
> On 11.10.21 19:12, Bertrand Marquis wrote:
> > Hi Roger,
> >
> >> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>
> >> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
> >>> The existing VPCI support available for X86 is adapted for Arm.
> >>> When the device is added to XEN via the hyper call
> >>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> >>> access is added to the Xen to emulate the PCI devices config space.
> >>>
> >>> A MMIO trap handler for the PCI ECAM space is registered in XEN
> >>> so that when guest is trying to access the PCI config space,XEN
> >>> will trap the access and emulate read/write using the VPCI and
> >>> not the real PCI hardware.
> >>>
> >>> For Dom0less systems scan_pci_devices() would be used to discover the
> >>> PCI device in XEN and VPCI handler will be added during XEN boots.
> >>>
> >>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> >>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> >>> ---
> >>> Change in v5:
> >>> - Add pci_cleanup_msi(pdev) in cleanup part.
> >>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> >>> Change in v4:
> >>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
> >>> Change in v3:
> >>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
> >>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
> >>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
> >>> Change in v2:
> >>> - Add new XEN_DOMCTL_CDF_vpci flag
> >>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
> >>> - enable vpci support when pci-passthough option is enabled.
> >>> ---
> >>> ---
> >>> xen/arch/arm/Makefile | 1 +
> >>> xen/arch/arm/domain.c | 4 ++
> >>> xen/arch/arm/domain_build.c | 3 +
> >>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
> >>> xen/arch/arm/vpci.h | 36 ++++++++++++
> >>> xen/drivers/passthrough/pci.c | 18 ++++++
> >>> xen/include/asm-arm/domain.h | 7 ++-
> >>> xen/include/asm-x86/pci.h | 2 -
> >>> xen/include/public/arch-arm.h | 7 +++
> >>> xen/include/xen/pci.h | 2 +
> >>> 10 files changed, 179 insertions(+), 3 deletions(-)
> >>> create mode 100644 xen/arch/arm/vpci.c
> >>> create mode 100644 xen/arch/arm/vpci.h
> >>>
> >>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> >>> index 44d7cc81fa..fb9c976ea2 100644
> >>> --- a/xen/arch/arm/Makefile
> >>> +++ b/xen/arch/arm/Makefile
> >>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> >>> obj-y += platforms/
> >>> endif
> >>> obj-$(CONFIG_TEE) += tee/
> >>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
> >>>
> >>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> >>> obj-y += bootfdt.init.o
> >>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>> index 36138c1b2e..fbb52f78f1 100644
> >>> --- a/xen/arch/arm/domain.c
> >>> +++ b/xen/arch/arm/domain.c
> >>> @@ -39,6 +39,7 @@
> >>> #include <asm/vgic.h>
> >>> #include <asm/vtimer.h>
> >>>
> >>> +#include "vpci.h"
> >>> #include "vuart.h"
> >>>
> >>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> >>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
> >>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> >>> goto fail;
> >>>
> >>> + if ( (rc = domain_vpci_init(d)) != 0 )
> >>> + goto fail;
> >>> +
> >>> return 0;
> >>>
> >>> fail:
> >>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> >>> index c5afbe2e05..f4c89bde8c 100644
> >>> --- a/xen/arch/arm/domain_build.c
> >>> +++ b/xen/arch/arm/domain_build.c
> >>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
> >>> if ( iommu_enabled )
> >>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
> >>>
> >>> + if ( is_pci_passthrough_enabled() )
> >>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
> >> I think I'm confused with this. You seem to enable vPCI for dom0, but
> >> then domain_vpci_init will setup traps for the guest virtual ECAM
> >> layout, not the native one that dom0 will be using.
> > I think after the last discussions, it was decided to also installed the vpci
> > handler for dom0. I will have to look into this and come back to you.
> > @Oleksandr: Could you comment on this.
> Yes, we do trap Dom0 as well. The Dom0 traps are not in this series, but
> are in mine as it needs more preparatory work for that. Please see [1]
Then I don't think we should set XEN_DOMCTL_CDF_vpci for dom0 here, it
should instead be done in the patch where dom0 support is introduced.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 16:32 ` Roger Pau Monné
@ 2021-10-11 17:11 ` Bertrand Marquis
2021-10-12 8:29 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 17:11 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini
Hi Roger,
> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>> Hi Roger,
>>
>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Mon, Oct 11, 2021 at 01:40:30PM +0000, Bertrand Marquis wrote:
>>>> Hi Roger,
>>>>
>>>> + Oleksandr to have a better PCI expert then me.
>>>>
>>>>> On 11 Oct 2021, at 14:20, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>
>>>>> On Mon, Oct 11, 2021 at 12:11:04PM +0000, Bertrand Marquis wrote:
>>>>>> Hi Roger,
>>>>>>
>>>>>>> On 11 Oct 2021, at 12:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>>
>>>>>>> On Wed, Oct 06, 2021 at 06:40:27PM +0100, Rahul Singh wrote:
>>>>>>>> ARM architecture does not implement I/O ports. Ignore this call on ARM
>>>>>>>> to avoid the overhead of making a hypercall just for Xen to return
>>>>>>>> -ENOSYS.
>>>>>>>
>>>>>>> What is the cal trace of this function actually on Arm?
>>>>>>>
>>>>>>> AFAICT libxl will only call xc_domain_ioport_permission if there are
>>>>>>> IO ports explicitly defined in the guest configuration, or if any of
>>>>>>> the BARs of the PCI device is in the IO space, which is not possible
>>>>>>> on Arm.
>>>>>>
>>>>>> PCI devices BARs can be in the IO space as the PCI devices are not
>>>>>> Arm specific. There is not ioports on arm so to be used those can be
>>>>>> in some cases remapped and accessed as MMIOs or are not possible
>>>>>> to use at all.
>>>>>>
>>>>>> But the IO space does appear when BARs are listed even on Arm.
>>>>>
>>>>> Urg, I wonder whether those devices with IO BARs will work correctly
>>>>> under Arm then.
>>>>>
>>>>> How do you know whether the BAR has been remapped from IO space into
>>>>> MMIO?
>>>>
>>>> We cannot, I think the platform will define if this is the case and where.
>>>> @oleksandr: I remember that this was discussed during some of our
>>>> meetings but I have no idea of the details here, can you help ?
>>>>
>>>>>
>>>>> IMO instead of faking a successful return value from
>>>>> xc_domain_ioport_permission we should avoid the call completely in the
>>>>> first place, specially if we need to instead issue a call to
>>>>> xc_domain_iomem_permission.
>>>>
>>>> At the end we will never have to issue this because this will never be a matter
>>>> of “iomem” permission as there would not be any way to cut on something under
>>>> the page. If this is to be supported one day, it will probably have to be fully emulated
>>>> to keep the isolation.
>>>
>>> So you have a set of memory pages that map accesses from
>>> MMIO into IO space but it's not possible to isolate specific IO port
>>> regions as they are all contiguous in the same page(s).
>>
>> Exact.
>>
>>>
>>>> Right now on arm you can just make the more simple assumption that ioports are
>>>> just not supported.
>>>
>>> Would it make sense in the future to provide a memory region to guests
>>> in order to use for IO port accesses, and call
>>> xc_domain_ioport_permission to set which ports would be allowed?
>>
>> Right now we do not plan to support this at all and we will have to
>> figure this out if we do this one day.
>>
>>>
>>> I think the commit message needs to at least be expanded in order to
>>> contain the information provided here. It might also be helpful to
>>> figure out whether we would have to handle IO port accesses in the
>>> future on Arm, or if it's fine to just ignore them.
>>
>> All our investigations and tests have been done without supporting it
>> without any issues so this is not a critical feature (most devices can
>> be operated without using the I/O ports).
>
> IMO we should let the users know they attempted to use a device with
> BARs in the IO space, and that those BARs won't be accessible which
> could make the device not function as expected.
>
> Do you think it would be reasonable to attempt the hypercall on Arm
> also, and in case of error (on Arm) just print a warning message and
> continue operations as normal?
I think this would lead to a warning printed on lots of devices where in
fact there would be no issues.
If this is an issue for a device driver because it cannot operate without
I/O ports, this will be raised by the driver inside the guest.
So in the current state I think the way to do it is right.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 16:43 ` Roger Pau Monné
@ 2021-10-11 17:15 ` Bertrand Marquis
2021-10-11 18:30 ` Oleksandr Andrushchenko
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-11 17:15 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Ian Jackson, Jan Beulich, Wei Liu,
Paul Durrant
Hi Roger,
> On 11 Oct 2021, at 17:43, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Mon, Oct 11, 2021 at 04:20:14PM +0000, Oleksandr Andrushchenko wrote:
>>
>>
>> On 11.10.21 19:12, Bertrand Marquis wrote:
>>> Hi Roger,
>>>
>>>> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>
>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>> The existing VPCI support available for X86 is adapted for Arm.
>>>>> When the device is added to XEN via the hyper call
>>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>>>> access is added to the Xen to emulate the PCI devices config space.
>>>>>
>>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>>>> so that when guest is trying to access the PCI config space,XEN
>>>>> will trap the access and emulate read/write using the VPCI and
>>>>> not the real PCI hardware.
>>>>>
>>>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>>>
>>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>>>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>> ---
>>>>> Change in v5:
>>>>> - Add pci_cleanup_msi(pdev) in cleanup part.
>>>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>> Change in v4:
>>>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>>>> Change in v3:
>>>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>>>> Change in v2:
>>>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>>>> - enable vpci support when pci-passthough option is enabled.
>>>>> ---
>>>>> ---
>>>>> xen/arch/arm/Makefile | 1 +
>>>>> xen/arch/arm/domain.c | 4 ++
>>>>> xen/arch/arm/domain_build.c | 3 +
>>>>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>>>>> xen/arch/arm/vpci.h | 36 ++++++++++++
>>>>> xen/drivers/passthrough/pci.c | 18 ++++++
>>>>> xen/include/asm-arm/domain.h | 7 ++-
>>>>> xen/include/asm-x86/pci.h | 2 -
>>>>> xen/include/public/arch-arm.h | 7 +++
>>>>> xen/include/xen/pci.h | 2 +
>>>>> 10 files changed, 179 insertions(+), 3 deletions(-)
>>>>> create mode 100644 xen/arch/arm/vpci.c
>>>>> create mode 100644 xen/arch/arm/vpci.h
>>>>>
>>>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>>>> index 44d7cc81fa..fb9c976ea2 100644
>>>>> --- a/xen/arch/arm/Makefile
>>>>> +++ b/xen/arch/arm/Makefile
>>>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>>>> obj-y += platforms/
>>>>> endif
>>>>> obj-$(CONFIG_TEE) += tee/
>>>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>>>
>>>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>>>> obj-y += bootfdt.init.o
>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>> index 36138c1b2e..fbb52f78f1 100644
>>>>> --- a/xen/arch/arm/domain.c
>>>>> +++ b/xen/arch/arm/domain.c
>>>>> @@ -39,6 +39,7 @@
>>>>> #include <asm/vgic.h>
>>>>> #include <asm/vtimer.h>
>>>>>
>>>>> +#include "vpci.h"
>>>>> #include "vuart.h"
>>>>>
>>>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>>>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>>>> goto fail;
>>>>>
>>>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>>>> + goto fail;
>>>>> +
>>>>> return 0;
>>>>>
>>>>> fail:
>>>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>>> index c5afbe2e05..f4c89bde8c 100644
>>>>> --- a/xen/arch/arm/domain_build.c
>>>>> +++ b/xen/arch/arm/domain_build.c
>>>>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>>>>> if ( iommu_enabled )
>>>>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>>>>
>>>>> + if ( is_pci_passthrough_enabled() )
>>>>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>>>> I think I'm confused with this. You seem to enable vPCI for dom0, but
>>>> then domain_vpci_init will setup traps for the guest virtual ECAM
>>>> layout, not the native one that dom0 will be using.
>>> I think after the last discussions, it was decided to also installed the vpci
>>> handler for dom0. I will have to look into this and come back to you.
>>> @Oleksandr: Could you comment on this.
>> Yes, we do trap Dom0 as well. The Dom0 traps are not in this series, but
>> are in mine as it needs more preparatory work for that. Please see [1]
>
> Then I don't think we should set XEN_DOMCTL_CDF_vpci for dom0 here, it
> should instead be done in the patch where dom0 support is introduced.
Ok I will check to remove this and include the change in v6.
Cheers
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 14:10 ` Jan Beulich
2021-10-11 14:52 ` Bertrand Marquis
@ 2021-10-11 18:18 ` Stefano Stabellini
2021-10-12 8:04 ` Jan Beulich
1 sibling, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-11 18:18 UTC (permalink / raw)
To: Jan Beulich
Cc: Bertrand Marquis, Rahul Singh, Andre Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Ian Jackson, Wei Liu, Paul Durrant,
Roger Pau Monné,
xen-devel
On Mon, 11 Oct 2021, Jan Beulich wrote:
> On 11.10.2021 15:34, Bertrand Marquis wrote:
> >> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
> >> On 11.10.2021 14:41, Bertrand Marquis wrote:
> >>>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
> >>>> On 06.10.2021 19:40, Rahul Singh wrote:
> >>>>> --- /dev/null
> >>>>> +++ b/xen/arch/arm/vpci.c
> >>>>> @@ -0,0 +1,102 @@
> >>>>> +/*
> >>>>> + * xen/arch/arm/vpci.c
> >>>>> + *
> >>>>> + * This program is free software; you can redistribute it and/or modify
> >>>>> + * it under the terms of the GNU General Public License as published by
> >>>>> + * the Free Software Foundation; either version 2 of the License, or
> >>>>> + * (at your option) any later version.
> >>>>> + *
> >>>>> + * This program is distributed in the hope that it will be useful,
> >>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >>>>> + * GNU General Public License for more details.
> >>>>> + */
> >>>>> +#include <xen/sched.h>
> >>>>> +
> >>>>> +#include <asm/mmio.h>
> >>>>> +
> >>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
> >>>>
> >>>> Nit: Stray blank (like you had in an earlier version for MMCFG_BDF()).
> >>>> Also isn't this effectively part of the public interface (along with
> >>>> MMCFG_BDF()), alongside GUEST_VPCI_ECAM_{BASE,SIZE}?
> >>>
> >>> I will move that in the next version to xen/pci.h and rename itMMCFG_REG_OFFSET.
> >>> Would that be ok ?
> >>
> >> That would be okay and make sense when put next to MMCFG_BDF(), but
> >> it would not address my comment: That still wouldn't be the public
> >> interface. Otoh you only mimic hardware behavior, so perhaps the
> >> splitting of the address isn't as relevant to put there as would be
> >> GUEST_VPCI_ECAM_{BASE,SIZE}.
> >
> > Ok now I get what you wanted.
> >
> > You would actually like both MMCFG_BDF and MMCFG_REG_OFFSET to
> > be moved to arch-arm.h.
> >
> > Then I am not quite sure to follow why.
> > Those are not macros coming out of a way we have to define this but from
> > how it works in standard PCI.
> > The base and size are needed to know where the PCI bus will be.
> >
> > So why should those be needed in public headers ?
>
> Well, see my "Otoh ..." in the earlier reply. Keeping the two
> address splitting macros out of there is okay.
>
> >>>>> --- a/xen/drivers/passthrough/pci.c
> >>>>> +++ b/xen/drivers/passthrough/pci.c
> >>>>> @@ -766,6 +766,24 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> >>>>> else
> >>>>> iommu_enable_device(pdev);
> >>>>
> >>>> Please note the context above; ...
> >>>>
> >>>>> +#ifdef CONFIG_ARM
> >>>>> + /*
> >>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> >>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
> >>>>> + */
> >>>>> + ret = vpci_add_handlers(pdev);
> >>>>> + if ( ret )
> >>>>> + {
> >>>>> + printk(XENLOG_ERR "setup of vPCI failed: %d\n", ret);
> >>>>> + pci_cleanup_msi(pdev);
> >>>>> + ret = iommu_remove_device(pdev);
> >>>>> + if ( pdev->domain )
> >>>>> + list_del(&pdev->domain_list);
> >>>>> + free_pdev(pseg, pdev);
> >>>>
> >>>> ... you unconditionally undo the if() part of the earlier conditional;
> >>>> is there any guarantee that the other path can't ever be taken, now
> >>>> and forever? If it can't be taken now (which I think is the case as
> >>>> long as Dom0 wouldn't report the same device twice), then at least some
> >>>> precaution wants taking. Maybe moving your addition into that if()
> >>>> could be an option.
> >>>>
> >>>> Furthermore I continue to wonder whether this ordering is indeed
> >>>> preferable over doing software setup before hardware arrangements. This
> >>>> would address the above issue as well as long as vpci_add_handlers() is
> >>>> idempotent. And it would likely simplify error cleanup.
> >>>
> >>> I agree with you so I will move this code block before iommu part.
> >>>
> >>> But digging deeper into this, I would have 2 questions:
> >>>
> >>> - msi_cleanup was done there after a request from Stefano, but is not
> >>> done in case or iommu error, is there an issue to solve here ?
> >>
> >> Maybe, but I'm not sure. This very much depends on what a domain
> >> could in principle do with a partly set-up device. Plus let's
> >> not forget that we're talking of only Dom0 here (for now at least,
> >> i.e. not considering the dom0less case).
> >>
> >> But I'd also like to further defer to Stefano.
> >
> > Ok, I must admit I do not really see at that stage why doing an MSI cleanup
> > could be needed so I will wait for Stefano to know if I need to keep this when
> > moving the block up (at the end it is theoretical right now as this is empty).
I know that MSIs are not supported yet on ARM (pci_cleanup_msi does
nothing). But I wanted to make sure that the pci_cleanup_msi() calls are
present anywhere necessary, especially on the error paths. So that once
we add MSI support, we don't need to search through the code to find all
the error paths missing a pci_cleanup_msi() call.
To answer your first question: you are right, we are also missing a
pci_cleanup_msi() call in the case of IOMMU error. So it might be better
to move the call to pci_cleanup_msi() under the "out" label so that we
can do it once for both cases.
To answer your second point about whether it is necessary at all: if
MSIs and MSI-Xs cannot be already setup at this point at all (not even
the enable bit), then we don't need any call to pci_cleanup_msi() in
pci_add_device.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 17:15 ` Bertrand Marquis
@ 2021-10-11 18:30 ` Oleksandr Andrushchenko
2021-10-11 19:27 ` Stefano Stabellini
0 siblings, 1 reply; 190+ messages in thread
From: Oleksandr Andrushchenko @ 2021-10-11 18:30 UTC (permalink / raw)
To: Bertrand Marquis, Roger Pau Monné
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Ian Jackson, Jan Beulich, Wei Liu,
Paul Durrant
On 11.10.21 20:15, Bertrand Marquis wrote:
> Hi Roger,
>
>> On 11 Oct 2021, at 17:43, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>
>> On Mon, Oct 11, 2021 at 04:20:14PM +0000, Oleksandr Andrushchenko wrote:
>>>
>>> On 11.10.21 19:12, Bertrand Marquis wrote:
>>>> Hi Roger,
>>>>
>>>>> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>
>>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>>> The existing VPCI support available for X86 is adapted for Arm.
>>>>>> When the device is added to XEN via the hyper call
>>>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>>>>> access is added to the Xen to emulate the PCI devices config space.
>>>>>>
>>>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>>>>> so that when guest is trying to access the PCI config space,XEN
>>>>>> will trap the access and emulate read/write using the VPCI and
>>>>>> not the real PCI hardware.
>>>>>>
>>>>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>>>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>>>>
>>>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>>>>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>>> ---
>>>>>> Change in v5:
>>>>>> - Add pci_cleanup_msi(pdev) in cleanup part.
>>>>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>>> Change in v4:
>>>>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>>>>> Change in v3:
>>>>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>>>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>>>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>>>>> Change in v2:
>>>>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>>>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>>>>> - enable vpci support when pci-passthough option is enabled.
>>>>>> ---
>>>>>> ---
>>>>>> xen/arch/arm/Makefile | 1 +
>>>>>> xen/arch/arm/domain.c | 4 ++
>>>>>> xen/arch/arm/domain_build.c | 3 +
>>>>>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>>>>>> xen/arch/arm/vpci.h | 36 ++++++++++++
>>>>>> xen/drivers/passthrough/pci.c | 18 ++++++
>>>>>> xen/include/asm-arm/domain.h | 7 ++-
>>>>>> xen/include/asm-x86/pci.h | 2 -
>>>>>> xen/include/public/arch-arm.h | 7 +++
>>>>>> xen/include/xen/pci.h | 2 +
>>>>>> 10 files changed, 179 insertions(+), 3 deletions(-)
>>>>>> create mode 100644 xen/arch/arm/vpci.c
>>>>>> create mode 100644 xen/arch/arm/vpci.h
>>>>>>
>>>>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>>>>> index 44d7cc81fa..fb9c976ea2 100644
>>>>>> --- a/xen/arch/arm/Makefile
>>>>>> +++ b/xen/arch/arm/Makefile
>>>>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>>>>> obj-y += platforms/
>>>>>> endif
>>>>>> obj-$(CONFIG_TEE) += tee/
>>>>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>>>>
>>>>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>>>>> obj-y += bootfdt.init.o
>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>> index 36138c1b2e..fbb52f78f1 100644
>>>>>> --- a/xen/arch/arm/domain.c
>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>> @@ -39,6 +39,7 @@
>>>>>> #include <asm/vgic.h>
>>>>>> #include <asm/vtimer.h>
>>>>>>
>>>>>> +#include "vpci.h"
>>>>>> #include "vuart.h"
>>>>>>
>>>>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>>>>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>>>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>>>>> goto fail;
>>>>>>
>>>>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>>>>> + goto fail;
>>>>>> +
>>>>>> return 0;
>>>>>>
>>>>>> fail:
>>>>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>>>> index c5afbe2e05..f4c89bde8c 100644
>>>>>> --- a/xen/arch/arm/domain_build.c
>>>>>> +++ b/xen/arch/arm/domain_build.c
>>>>>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>>>>>> if ( iommu_enabled )
>>>>>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>>>>>
>>>>>> + if ( is_pci_passthrough_enabled() )
>>>>>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>>>>> I think I'm confused with this. You seem to enable vPCI for dom0, but
>>>>> then domain_vpci_init will setup traps for the guest virtual ECAM
>>>>> layout, not the native one that dom0 will be using.
>>>> I think after the last discussions, it was decided to also installed the vpci
>>>> handler for dom0. I will have to look into this and come back to you.
>>>> @Oleksandr: Could you comment on this.
>>> Yes, we do trap Dom0 as well. The Dom0 traps are not in this series, but
>>> are in mine as it needs more preparatory work for that. Please see [1]
>> Then I don't think we should set XEN_DOMCTL_CDF_vpci for dom0 here, it
>> should instead be done in the patch where dom0 support is introduced.
> Ok I will check to remove this and include the change in v6.
Just to make it clear: do we want to remove this piece from this patch
and instead have a dedicated patch on top of my series, so it is enabled
right after we have the code that sets up the trap handlers for Dom0?
If so, then do we want that patch to be chained in my series or sent as
a follow up right after it separately?
Thanks,
Oleksandr
>
> Cheers
> Bertrand
>
>> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 18:30 ` Oleksandr Andrushchenko
@ 2021-10-11 19:27 ` Stefano Stabellini
2021-10-12 5:34 ` Oleksandr Andrushchenko
2021-10-12 7:44 ` Bertrand Marquis
0 siblings, 2 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-11 19:27 UTC (permalink / raw)
To: Oleksandr Andrushchenko
Cc: Bertrand Marquis, Roger Pau Monné,
Rahul Singh, xen-devel, Andre Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Paul Durrant
[-- Attachment #1: Type: text/plain, Size: 5899 bytes --]
On Mon, 11 Oct 2021, Oleksandr Andrushchenko wrote:
> On 11.10.21 20:15, Bertrand Marquis wrote:
> > Hi Roger,
> >
> >> On 11 Oct 2021, at 17:43, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>
> >> On Mon, Oct 11, 2021 at 04:20:14PM +0000, Oleksandr Andrushchenko wrote:
> >>>
> >>> On 11.10.21 19:12, Bertrand Marquis wrote:
> >>>> Hi Roger,
> >>>>
> >>>>> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>>>
> >>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
> >>>>>> The existing VPCI support available for X86 is adapted for Arm.
> >>>>>> When the device is added to XEN via the hyper call
> >>>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> >>>>>> access is added to the Xen to emulate the PCI devices config space.
> >>>>>>
> >>>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
> >>>>>> so that when guest is trying to access the PCI config space,XEN
> >>>>>> will trap the access and emulate read/write using the VPCI and
> >>>>>> not the real PCI hardware.
> >>>>>>
> >>>>>> For Dom0less systems scan_pci_devices() would be used to discover the
> >>>>>> PCI device in XEN and VPCI handler will be added during XEN boots.
> >>>>>>
> >>>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> >>>>>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> >>>>>> ---
> >>>>>> Change in v5:
> >>>>>> - Add pci_cleanup_msi(pdev) in cleanup part.
> >>>>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> >>>>>> Change in v4:
> >>>>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
> >>>>>> Change in v3:
> >>>>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
> >>>>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
> >>>>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
> >>>>>> Change in v2:
> >>>>>> - Add new XEN_DOMCTL_CDF_vpci flag
> >>>>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
> >>>>>> - enable vpci support when pci-passthough option is enabled.
> >>>>>> ---
> >>>>>> ---
> >>>>>> xen/arch/arm/Makefile | 1 +
> >>>>>> xen/arch/arm/domain.c | 4 ++
> >>>>>> xen/arch/arm/domain_build.c | 3 +
> >>>>>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
> >>>>>> xen/arch/arm/vpci.h | 36 ++++++++++++
> >>>>>> xen/drivers/passthrough/pci.c | 18 ++++++
> >>>>>> xen/include/asm-arm/domain.h | 7 ++-
> >>>>>> xen/include/asm-x86/pci.h | 2 -
> >>>>>> xen/include/public/arch-arm.h | 7 +++
> >>>>>> xen/include/xen/pci.h | 2 +
> >>>>>> 10 files changed, 179 insertions(+), 3 deletions(-)
> >>>>>> create mode 100644 xen/arch/arm/vpci.c
> >>>>>> create mode 100644 xen/arch/arm/vpci.h
> >>>>>>
> >>>>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> >>>>>> index 44d7cc81fa..fb9c976ea2 100644
> >>>>>> --- a/xen/arch/arm/Makefile
> >>>>>> +++ b/xen/arch/arm/Makefile
> >>>>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> >>>>>> obj-y += platforms/
> >>>>>> endif
> >>>>>> obj-$(CONFIG_TEE) += tee/
> >>>>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
> >>>>>>
> >>>>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> >>>>>> obj-y += bootfdt.init.o
> >>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >>>>>> index 36138c1b2e..fbb52f78f1 100644
> >>>>>> --- a/xen/arch/arm/domain.c
> >>>>>> +++ b/xen/arch/arm/domain.c
> >>>>>> @@ -39,6 +39,7 @@
> >>>>>> #include <asm/vgic.h>
> >>>>>> #include <asm/vtimer.h>
> >>>>>>
> >>>>>> +#include "vpci.h"
> >>>>>> #include "vuart.h"
> >>>>>>
> >>>>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> >>>>>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
> >>>>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> >>>>>> goto fail;
> >>>>>>
> >>>>>> + if ( (rc = domain_vpci_init(d)) != 0 )
> >>>>>> + goto fail;
> >>>>>> +
> >>>>>> return 0;
> >>>>>>
> >>>>>> fail:
> >>>>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> >>>>>> index c5afbe2e05..f4c89bde8c 100644
> >>>>>> --- a/xen/arch/arm/domain_build.c
> >>>>>> +++ b/xen/arch/arm/domain_build.c
> >>>>>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
> >>>>>> if ( iommu_enabled )
> >>>>>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
> >>>>>>
> >>>>>> + if ( is_pci_passthrough_enabled() )
> >>>>>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
> >>>>> I think I'm confused with this. You seem to enable vPCI for dom0, but
> >>>>> then domain_vpci_init will setup traps for the guest virtual ECAM
> >>>>> layout, not the native one that dom0 will be using.
> >>>> I think after the last discussions, it was decided to also installed the vpci
> >>>> handler for dom0. I will have to look into this and come back to you.
> >>>> @Oleksandr: Could you comment on this.
> >>> Yes, we do trap Dom0 as well. The Dom0 traps are not in this series, but
> >>> are in mine as it needs more preparatory work for that. Please see [1]
> >> Then I don't think we should set XEN_DOMCTL_CDF_vpci for dom0 here, it
> >> should instead be done in the patch where dom0 support is introduced.
> > Ok I will check to remove this and include the change in v6.
> Just to make it clear: do we want to remove this piece from this patch
> and instead have a dedicated patch on top of my series, so it is enabled
> right after we have the code that sets up the trap handlers for Dom0?
> If so, then do we want that patch to be chained in my series or sent as
> a follow up right after it separately?
I think we want to remove the XEN_DOMCTL_CDF_vpci chunk from this patch.
Where exactly it should be introduced, I am not sure. I think it would
be OK as a separate single patch at the end. I doesn't have to be part
of the outstanding series, considering that we are also missing the
patch to add "select HAS_PCI" for ARM.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM.
2021-10-11 12:15 ` Bertrand Marquis
@ 2021-10-12 1:32 ` Stefano Stabellini
0 siblings, 0 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-12 1:32 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Roger Pau Monné,
Rahul Singh, xen-devel, Andre Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk
[-- Attachment #1: Type: text/plain, Size: 587 bytes --]
On Mon, 11 Oct 2021, Bertrand Marquis wrote:
> Hi Roger,
>
> > On 11 Oct 2021, at 12:43, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Wed, Oct 06, 2021 at 06:40:35PM +0100, Rahul Singh wrote:
> >> This patch will be reverted once we add support for VPCI MSI/MSIX
> >> support on ARM.
> >>
> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> >> Acked-by: Stefano Stabellini <sstabellini@kernel.org>
> >> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
> >
> > Reviewed-by: Roger Pau Monné <rogewr.pau@citrix.com>
>
> Thanks
Thanks! I committed this patch.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 19:27 ` Stefano Stabellini
@ 2021-10-12 5:34 ` Oleksandr Andrushchenko
2021-10-12 7:44 ` Bertrand Marquis
1 sibling, 0 replies; 190+ messages in thread
From: Oleksandr Andrushchenko @ 2021-10-12 5:34 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Bertrand Marquis, Roger Pau Monné,
Rahul Singh, xen-devel, Andre Przywara, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Paul Durrant, Oleksandr Andrushchenko
On 11.10.21 22:27, Stefano Stabellini wrote:
> On Mon, 11 Oct 2021, Oleksandr Andrushchenko wrote:
>> On 11.10.21 20:15, Bertrand Marquis wrote:
>>> Hi Roger,
>>>
>>>> On 11 Oct 2021, at 17:43, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>
>>>> On Mon, Oct 11, 2021 at 04:20:14PM +0000, Oleksandr Andrushchenko wrote:
>>>>> On 11.10.21 19:12, Bertrand Marquis wrote:
>>>>>> Hi Roger,
>>>>>>
>>>>>>> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>>
>>>>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>>>>> The existing VPCI support available for X86 is adapted for Arm.
>>>>>>>> When the device is added to XEN via the hyper call
>>>>>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>>>>>>> access is added to the Xen to emulate the PCI devices config space.
>>>>>>>>
>>>>>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>>>>>>> so that when guest is trying to access the PCI config space,XEN
>>>>>>>> will trap the access and emulate read/write using the VPCI and
>>>>>>>> not the real PCI hardware.
>>>>>>>>
>>>>>>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>>>>>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>>>>>>
>>>>>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>>>>>>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>>>>> ---
>>>>>>>> Change in v5:
>>>>>>>> - Add pci_cleanup_msi(pdev) in cleanup part.
>>>>>>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>>>>> Change in v4:
>>>>>>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>>>>>>> Change in v3:
>>>>>>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>>>>>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>>>>>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>>>>>>> Change in v2:
>>>>>>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>>>>>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>>>>>>> - enable vpci support when pci-passthough option is enabled.
>>>>>>>> ---
>>>>>>>> ---
>>>>>>>> xen/arch/arm/Makefile | 1 +
>>>>>>>> xen/arch/arm/domain.c | 4 ++
>>>>>>>> xen/arch/arm/domain_build.c | 3 +
>>>>>>>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>>>>>>>> xen/arch/arm/vpci.h | 36 ++++++++++++
>>>>>>>> xen/drivers/passthrough/pci.c | 18 ++++++
>>>>>>>> xen/include/asm-arm/domain.h | 7 ++-
>>>>>>>> xen/include/asm-x86/pci.h | 2 -
>>>>>>>> xen/include/public/arch-arm.h | 7 +++
>>>>>>>> xen/include/xen/pci.h | 2 +
>>>>>>>> 10 files changed, 179 insertions(+), 3 deletions(-)
>>>>>>>> create mode 100644 xen/arch/arm/vpci.c
>>>>>>>> create mode 100644 xen/arch/arm/vpci.h
>>>>>>>>
>>>>>>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>>>>>>> index 44d7cc81fa..fb9c976ea2 100644
>>>>>>>> --- a/xen/arch/arm/Makefile
>>>>>>>> +++ b/xen/arch/arm/Makefile
>>>>>>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>>>>>>> obj-y += platforms/
>>>>>>>> endif
>>>>>>>> obj-$(CONFIG_TEE) += tee/
>>>>>>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>>>>>>
>>>>>>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>>>>>>> obj-y += bootfdt.init.o
>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>> index 36138c1b2e..fbb52f78f1 100644
>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>> @@ -39,6 +39,7 @@
>>>>>>>> #include <asm/vgic.h>
>>>>>>>> #include <asm/vtimer.h>
>>>>>>>>
>>>>>>>> +#include "vpci.h"
>>>>>>>> #include "vuart.h"
>>>>>>>>
>>>>>>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>>>>>>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>>>>>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>>>>>>> goto fail;
>>>>>>>>
>>>>>>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>>>>>>> + goto fail;
>>>>>>>> +
>>>>>>>> return 0;
>>>>>>>>
>>>>>>>> fail:
>>>>>>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>>>>>> index c5afbe2e05..f4c89bde8c 100644
>>>>>>>> --- a/xen/arch/arm/domain_build.c
>>>>>>>> +++ b/xen/arch/arm/domain_build.c
>>>>>>>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>>>>>>>> if ( iommu_enabled )
>>>>>>>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>>>>>>>
>>>>>>>> + if ( is_pci_passthrough_enabled() )
>>>>>>>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>>>>>>> I think I'm confused with this. You seem to enable vPCI for dom0, but
>>>>>>> then domain_vpci_init will setup traps for the guest virtual ECAM
>>>>>>> layout, not the native one that dom0 will be using.
>>>>>> I think after the last discussions, it was decided to also installed the vpci
>>>>>> handler for dom0. I will have to look into this and come back to you.
>>>>>> @Oleksandr: Could you comment on this.
>>>>> Yes, we do trap Dom0 as well. The Dom0 traps are not in this series, but
>>>>> are in mine as it needs more preparatory work for that. Please see [1]
>>>> Then I don't think we should set XEN_DOMCTL_CDF_vpci for dom0 here, it
>>>> should instead be done in the patch where dom0 support is introduced.
>>> Ok I will check to remove this and include the change in v6.
>> Just to make it clear: do we want to remove this piece from this patch
>> and instead have a dedicated patch on top of my series, so it is enabled
>> right after we have the code that sets up the trap handlers for Dom0?
>> If so, then do we want that patch to be chained in my series or sent as
>> a follow up right after it separately?
> I think we want to remove the XEN_DOMCTL_CDF_vpci chunk from this patch.
>
> Where exactly it should be introduced, I am not sure. I think it would
> be OK as a separate single patch at the end. I doesn't have to be part
> of the outstanding series, considering that we are also missing the
> patch to add "select HAS_PCI" for ARM.
Yes, makes sense
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 19:27 ` Stefano Stabellini
2021-10-12 5:34 ` Oleksandr Andrushchenko
@ 2021-10-12 7:44 ` Bertrand Marquis
1 sibling, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 7:44 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Oleksandr Andrushchenko, Roger Pau Monné,
Rahul Singh, xen-devel, Andre Przywara, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Paul Durrant
Hi Stefano,
> On 11 Oct 2021, at 20:27, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Mon, 11 Oct 2021, Oleksandr Andrushchenko wrote:
>> On 11.10.21 20:15, Bertrand Marquis wrote:
>>> Hi Roger,
>>>
>>>> On 11 Oct 2021, at 17:43, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>
>>>> On Mon, Oct 11, 2021 at 04:20:14PM +0000, Oleksandr Andrushchenko wrote:
>>>>>
>>>>> On 11.10.21 19:12, Bertrand Marquis wrote:
>>>>>> Hi Roger,
>>>>>>
>>>>>>> On 11 Oct 2021, at 11:51, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>>
>>>>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>>>>> The existing VPCI support available for X86 is adapted for Arm.
>>>>>>>> When the device is added to XEN via the hyper call
>>>>>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>>>>>>> access is added to the Xen to emulate the PCI devices config space.
>>>>>>>>
>>>>>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>>>>>>> so that when guest is trying to access the PCI config space,XEN
>>>>>>>> will trap the access and emulate read/write using the VPCI and
>>>>>>>> not the real PCI hardware.
>>>>>>>>
>>>>>>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>>>>>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>>>>>>
>>>>>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>>>>>>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>>>>> ---
>>>>>>>> Change in v5:
>>>>>>>> - Add pci_cleanup_msi(pdev) in cleanup part.
>>>>>>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>>>>>> Change in v4:
>>>>>>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>>>>>>> Change in v3:
>>>>>>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>>>>>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>>>>>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>>>>>>> Change in v2:
>>>>>>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>>>>>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>>>>>>> - enable vpci support when pci-passthough option is enabled.
>>>>>>>> ---
>>>>>>>> ---
>>>>>>>> xen/arch/arm/Makefile | 1 +
>>>>>>>> xen/arch/arm/domain.c | 4 ++
>>>>>>>> xen/arch/arm/domain_build.c | 3 +
>>>>>>>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>>>>>>>> xen/arch/arm/vpci.h | 36 ++++++++++++
>>>>>>>> xen/drivers/passthrough/pci.c | 18 ++++++
>>>>>>>> xen/include/asm-arm/domain.h | 7 ++-
>>>>>>>> xen/include/asm-x86/pci.h | 2 -
>>>>>>>> xen/include/public/arch-arm.h | 7 +++
>>>>>>>> xen/include/xen/pci.h | 2 +
>>>>>>>> 10 files changed, 179 insertions(+), 3 deletions(-)
>>>>>>>> create mode 100644 xen/arch/arm/vpci.c
>>>>>>>> create mode 100644 xen/arch/arm/vpci.h
>>>>>>>>
>>>>>>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>>>>>>> index 44d7cc81fa..fb9c976ea2 100644
>>>>>>>> --- a/xen/arch/arm/Makefile
>>>>>>>> +++ b/xen/arch/arm/Makefile
>>>>>>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>>>>>>> obj-y += platforms/
>>>>>>>> endif
>>>>>>>> obj-$(CONFIG_TEE) += tee/
>>>>>>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>>>>>>
>>>>>>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>>>>>>> obj-y += bootfdt.init.o
>>>>>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>>>>>> index 36138c1b2e..fbb52f78f1 100644
>>>>>>>> --- a/xen/arch/arm/domain.c
>>>>>>>> +++ b/xen/arch/arm/domain.c
>>>>>>>> @@ -39,6 +39,7 @@
>>>>>>>> #include <asm/vgic.h>
>>>>>>>> #include <asm/vtimer.h>
>>>>>>>>
>>>>>>>> +#include "vpci.h"
>>>>>>>> #include "vuart.h"
>>>>>>>>
>>>>>>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>>>>>>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>>>>>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>>>>>>> goto fail;
>>>>>>>>
>>>>>>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>>>>>>> + goto fail;
>>>>>>>> +
>>>>>>>> return 0;
>>>>>>>>
>>>>>>>> fail:
>>>>>>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>>>>>> index c5afbe2e05..f4c89bde8c 100644
>>>>>>>> --- a/xen/arch/arm/domain_build.c
>>>>>>>> +++ b/xen/arch/arm/domain_build.c
>>>>>>>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>>>>>>>> if ( iommu_enabled )
>>>>>>>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>>>>>>>
>>>>>>>> + if ( is_pci_passthrough_enabled() )
>>>>>>>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>>>>>>> I think I'm confused with this. You seem to enable vPCI for dom0, but
>>>>>>> then domain_vpci_init will setup traps for the guest virtual ECAM
>>>>>>> layout, not the native one that dom0 will be using.
>>>>>> I think after the last discussions, it was decided to also installed the vpci
>>>>>> handler for dom0. I will have to look into this and come back to you.
>>>>>> @Oleksandr: Could you comment on this.
>>>>> Yes, we do trap Dom0 as well. The Dom0 traps are not in this series, but
>>>>> are in mine as it needs more preparatory work for that. Please see [1]
>>>> Then I don't think we should set XEN_DOMCTL_CDF_vpci for dom0 here, it
>>>> should instead be done in the patch where dom0 support is introduced.
>>> Ok I will check to remove this and include the change in v6.
>> Just to make it clear: do we want to remove this piece from this patch
>> and instead have a dedicated patch on top of my series, so it is enabled
>> right after we have the code that sets up the trap handlers for Dom0?
>> If so, then do we want that patch to be chained in my series or sent as
>> a follow up right after it separately?
>
> I think we want to remove the XEN_DOMCTL_CDF_vpci chunk from this patch.
>
> Where exactly it should be introduced, I am not sure. I think it would
> be OK as a separate single patch at the end. I doesn't have to be part
> of the outstanding series, considering that we are also missing the
> patch to add "select HAS_PCI" for ARM.
Agree, I will remove that from v6.
Cheers
Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 18:18 ` Stefano Stabellini
@ 2021-10-12 8:04 ` Jan Beulich
2021-10-12 21:37 ` Stefano Stabellini
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-12 8:04 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Bertrand Marquis, Rahul Singh, Andre Przywara, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 11.10.2021 20:18, Stefano Stabellini wrote:
> On Mon, 11 Oct 2021, Jan Beulich wrote:
>> On 11.10.2021 15:34, Bertrand Marquis wrote:
>>>> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
>>>> On 11.10.2021 14:41, Bertrand Marquis wrote:
>>>>> But digging deeper into this, I would have 2 questions:
>>>>>
>>>>> - msi_cleanup was done there after a request from Stefano, but is not
>>>>> done in case or iommu error, is there an issue to solve here ?
>>>>
>>>> Maybe, but I'm not sure. This very much depends on what a domain
>>>> could in principle do with a partly set-up device. Plus let's
>>>> not forget that we're talking of only Dom0 here (for now at least,
>>>> i.e. not considering the dom0less case).
>>>>
>>>> But I'd also like to further defer to Stefano.
>>>
>>> Ok, I must admit I do not really see at that stage why doing an MSI cleanup
>>> could be needed so I will wait for Stefano to know if I need to keep this when
>>> moving the block up (at the end it is theoretical right now as this is empty).
>
> I know that MSIs are not supported yet on ARM (pci_cleanup_msi does
> nothing). But I wanted to make sure that the pci_cleanup_msi() calls are
> present anywhere necessary, especially on the error paths. So that once
> we add MSI support, we don't need to search through the code to find all
> the error paths missing a pci_cleanup_msi() call.
>
> To answer your first question: you are right, we are also missing a
> pci_cleanup_msi() call in the case of IOMMU error. So it might be better
> to move the call to pci_cleanup_msi() under the "out" label so that we
> can do it once for both cases.
>
> To answer your second point about whether it is necessary at all: if
> MSIs and MSI-Xs cannot be already setup at this point at all (not even
> the enable bit), then we don't need any call to pci_cleanup_msi() in
> pci_add_device.
Well, at the very least MSI can't be set up ahead of the traps getting
put in place. Whether partial success of putting traps in place may
allow a cunning guest to set up MSI may depend on further aspects.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-11 17:11 ` Bertrand Marquis
@ 2021-10-12 8:29 ` Jan Beulich
2021-10-12 8:41 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-12 8:29 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
On 11.10.2021 19:11, Bertrand Marquis wrote:
>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>> I think the commit message needs to at least be expanded in order to
>>>> contain the information provided here. It might also be helpful to
>>>> figure out whether we would have to handle IO port accesses in the
>>>> future on Arm, or if it's fine to just ignore them.
>>>
>>> All our investigations and tests have been done without supporting it
>>> without any issues so this is not a critical feature (most devices can
>>> be operated without using the I/O ports).
>>
>> IMO we should let the users know they attempted to use a device with
>> BARs in the IO space, and that those BARs won't be accessible which
>> could make the device not function as expected.
>>
>> Do you think it would be reasonable to attempt the hypercall on Arm
>> also, and in case of error (on Arm) just print a warning message and
>> continue operations as normal?
>
> I think this would lead to a warning printed on lots of devices where in
> fact there would be no issues.
>
> If this is an issue for a device driver because it cannot operate without
> I/O ports, this will be raised by the driver inside the guest.
On what basis would the driver complain? The kernel might know of
the MMIO equivalent for ports, and hence might allow the driver
to properly obtain whatever is needed to later access the ports.
Just that the port accesses then wouldn't work (possibly crashing
the guest, or making it otherwise misbehave).
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 8:29 ` Jan Beulich
@ 2021-10-12 8:41 ` Bertrand Marquis
2021-10-12 9:32 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 8:41 UTC (permalink / raw)
To: Jan Beulich
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
Hi Jan,
> On 12 Oct 2021, at 09:29, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 11.10.2021 19:11, Bertrand Marquis wrote:
>>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>> I think the commit message needs to at least be expanded in order to
>>>>> contain the information provided here. It might also be helpful to
>>>>> figure out whether we would have to handle IO port accesses in the
>>>>> future on Arm, or if it's fine to just ignore them.
>>>>
>>>> All our investigations and tests have been done without supporting it
>>>> without any issues so this is not a critical feature (most devices can
>>>> be operated without using the I/O ports).
>>>
>>> IMO we should let the users know they attempted to use a device with
>>> BARs in the IO space, and that those BARs won't be accessible which
>>> could make the device not function as expected.
>>>
>>> Do you think it would be reasonable to attempt the hypercall on Arm
>>> also, and in case of error (on Arm) just print a warning message and
>>> continue operations as normal?
>>
>> I think this would lead to a warning printed on lots of devices where in
>> fact there would be no issues.
>>
>> If this is an issue for a device driver because it cannot operate without
>> I/O ports, this will be raised by the driver inside the guest.
>
> On what basis would the driver complain? The kernel might know of
> the MMIO equivalent for ports, and hence might allow the driver
> to properly obtain whatever is needed to later access the ports.
> Just that the port accesses then wouldn't work (possibly crashing
> the guest, or making it otherwise misbehave).
As ECAM and Arm does not support I/O ports, a driver requesting access
to them would get an error back.
So in practice it is not possible to try to access the ioports as there is no
way on arm to use them (no instructions).
A driver could misbehave by ignoring the fact that ioports are not there but
I am not quite sure how we could solve that as it would be a bug in the driver.
Regards
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 8:41 ` Bertrand Marquis
@ 2021-10-12 9:32 ` Jan Beulich
2021-10-12 9:38 ` Oleksandr Andrushchenko
2021-10-12 9:40 ` Bertrand Marquis
0 siblings, 2 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-12 9:32 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
On 12.10.2021 10:41, Bertrand Marquis wrote:
> Hi Jan,
>
>> On 12 Oct 2021, at 09:29, Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 11.10.2021 19:11, Bertrand Marquis wrote:
>>>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>> I think the commit message needs to at least be expanded in order to
>>>>>> contain the information provided here. It might also be helpful to
>>>>>> figure out whether we would have to handle IO port accesses in the
>>>>>> future on Arm, or if it's fine to just ignore them.
>>>>>
>>>>> All our investigations and tests have been done without supporting it
>>>>> without any issues so this is not a critical feature (most devices can
>>>>> be operated without using the I/O ports).
>>>>
>>>> IMO we should let the users know they attempted to use a device with
>>>> BARs in the IO space, and that those BARs won't be accessible which
>>>> could make the device not function as expected.
>>>>
>>>> Do you think it would be reasonable to attempt the hypercall on Arm
>>>> also, and in case of error (on Arm) just print a warning message and
>>>> continue operations as normal?
>>>
>>> I think this would lead to a warning printed on lots of devices where in
>>> fact there would be no issues.
>>>
>>> If this is an issue for a device driver because it cannot operate without
>>> I/O ports, this will be raised by the driver inside the guest.
>>
>> On what basis would the driver complain? The kernel might know of
>> the MMIO equivalent for ports, and hence might allow the driver
>> to properly obtain whatever is needed to later access the ports.
>> Just that the port accesses then wouldn't work (possibly crashing
>> the guest, or making it otherwise misbehave).
>
> As ECAM and Arm does not support I/O ports, a driver requesting access
> to them would get an error back.
> So in practice it is not possible to try to access the ioports as there is no
> way on arm to use them (no instructions).
>
> A driver could misbehave by ignoring the fact that ioports are not there but
> I am not quite sure how we could solve that as it would be a bug in the driver.
The minimal thing I'd suggest (or maybe you're doing this already)
would be to expose such BARs to the guest as r/o zero, rather than
letting their port nature "shine through".
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 9:32 ` Jan Beulich
@ 2021-10-12 9:38 ` Oleksandr Andrushchenko
2021-10-12 10:01 ` Jan Beulich
2021-10-12 9:40 ` Bertrand Marquis
1 sibling, 1 reply; 190+ messages in thread
From: Oleksandr Andrushchenko @ 2021-10-12 9:38 UTC (permalink / raw)
To: Jan Beulich, Bertrand Marquis
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini, Roger Pau Monné,
Oleksandr Andrushchenko
On 12.10.21 12:32, Jan Beulich wrote:
> On 12.10.2021 10:41, Bertrand Marquis wrote:
>> Hi Jan,
>>
>>> On 12 Oct 2021, at 09:29, Jan Beulich <jbeulich@suse.com> wrote:
>>>
>>> On 11.10.2021 19:11, Bertrand Marquis wrote:
>>>>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>> I think the commit message needs to at least be expanded in order to
>>>>>>> contain the information provided here. It might also be helpful to
>>>>>>> figure out whether we would have to handle IO port accesses in the
>>>>>>> future on Arm, or if it's fine to just ignore them.
>>>>>> All our investigations and tests have been done without supporting it
>>>>>> without any issues so this is not a critical feature (most devices can
>>>>>> be operated without using the I/O ports).
>>>>> IMO we should let the users know they attempted to use a device with
>>>>> BARs in the IO space, and that those BARs won't be accessible which
>>>>> could make the device not function as expected.
>>>>>
>>>>> Do you think it would be reasonable to attempt the hypercall on Arm
>>>>> also, and in case of error (on Arm) just print a warning message and
>>>>> continue operations as normal?
>>>> I think this would lead to a warning printed on lots of devices where in
>>>> fact there would be no issues.
>>>>
>>>> If this is an issue for a device driver because it cannot operate without
>>>> I/O ports, this will be raised by the driver inside the guest.
>>> On what basis would the driver complain? The kernel might know of
>>> the MMIO equivalent for ports, and hence might allow the driver
>>> to properly obtain whatever is needed to later access the ports.
>>> Just that the port accesses then wouldn't work (possibly crashing
>>> the guest, or making it otherwise misbehave).
>> As ECAM and Arm does not support I/O ports, a driver requesting access
>> to them would get an error back.
>> So in practice it is not possible to try to access the ioports as there is no
>> way on arm to use them (no instructions).
>>
>> A driver could misbehave by ignoring the fact that ioports are not there but
>> I am not quite sure how we could solve that as it would be a bug in the driver.
> The minimal thing I'd suggest (or maybe you're doing this already)
> would be to expose such BARs to the guest as r/o zero, rather than
> letting their port nature "shine through".
If we have the same, but baremetal then which entity disallows
those BARs to shine? I mean that if guest wants to crash... why
should we stop it and try emulating something special for it?
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 9:32 ` Jan Beulich
2021-10-12 9:38 ` Oleksandr Andrushchenko
@ 2021-10-12 9:40 ` Bertrand Marquis
2021-10-12 10:03 ` Jan Beulich
1 sibling, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 9:40 UTC (permalink / raw)
To: Jan Beulich
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
Hi Jan,
> On 12 Oct 2021, at 10:32, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 12.10.2021 10:41, Bertrand Marquis wrote:
>> Hi Jan,
>>
>>> On 12 Oct 2021, at 09:29, Jan Beulich <jbeulich@suse.com> wrote:
>>>
>>> On 11.10.2021 19:11, Bertrand Marquis wrote:
>>>>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>> I think the commit message needs to at least be expanded in order to
>>>>>>> contain the information provided here. It might also be helpful to
>>>>>>> figure out whether we would have to handle IO port accesses in the
>>>>>>> future on Arm, or if it's fine to just ignore them.
>>>>>>
>>>>>> All our investigations and tests have been done without supporting it
>>>>>> without any issues so this is not a critical feature (most devices can
>>>>>> be operated without using the I/O ports).
>>>>>
>>>>> IMO we should let the users know they attempted to use a device with
>>>>> BARs in the IO space, and that those BARs won't be accessible which
>>>>> could make the device not function as expected.
>>>>>
>>>>> Do you think it would be reasonable to attempt the hypercall on Arm
>>>>> also, and in case of error (on Arm) just print a warning message and
>>>>> continue operations as normal?
>>>>
>>>> I think this would lead to a warning printed on lots of devices where in
>>>> fact there would be no issues.
>>>>
>>>> If this is an issue for a device driver because it cannot operate without
>>>> I/O ports, this will be raised by the driver inside the guest.
>>>
>>> On what basis would the driver complain? The kernel might know of
>>> the MMIO equivalent for ports, and hence might allow the driver
>>> to properly obtain whatever is needed to later access the ports.
>>> Just that the port accesses then wouldn't work (possibly crashing
>>> the guest, or making it otherwise misbehave).
>>
>> As ECAM and Arm does not support I/O ports, a driver requesting access
>> to them would get an error back.
>> So in practice it is not possible to try to access the ioports as there is no
>> way on arm to use them (no instructions).
>>
>> A driver could misbehave by ignoring the fact that ioports are not there but
>> I am not quite sure how we could solve that as it would be a bug in the driver.
>
> The minimal thing I'd suggest (or maybe you're doing this already)
> would be to expose such BARs to the guest as r/o zero, rather than
> letting their port nature "shine through".
We are emulating an ECAM PCI which does not support I/O ports so I do not
think we are (and can) expose those to guests.
Anyway I will mark this as a point to check for Rahul when he is back.
Cheers
Bertrand
>
> Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 9:38 ` Oleksandr Andrushchenko
@ 2021-10-12 10:01 ` Jan Beulich
2021-10-12 10:06 ` Oleksandr Andrushchenko
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-12 10:01 UTC (permalink / raw)
To: Oleksandr Andrushchenko, Bertrand Marquis
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini, Roger Pau Monné
On 12.10.2021 11:38, Oleksandr Andrushchenko wrote:
> On 12.10.21 12:32, Jan Beulich wrote:
>> On 12.10.2021 10:41, Bertrand Marquis wrote:
>>>> On 12 Oct 2021, at 09:29, Jan Beulich <jbeulich@suse.com> wrote:
>>>> On 11.10.2021 19:11, Bertrand Marquis wrote:
>>>>>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>>>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>>> I think the commit message needs to at least be expanded in order to
>>>>>>>> contain the information provided here. It might also be helpful to
>>>>>>>> figure out whether we would have to handle IO port accesses in the
>>>>>>>> future on Arm, or if it's fine to just ignore them.
>>>>>>> All our investigations and tests have been done without supporting it
>>>>>>> without any issues so this is not a critical feature (most devices can
>>>>>>> be operated without using the I/O ports).
>>>>>> IMO we should let the users know they attempted to use a device with
>>>>>> BARs in the IO space, and that those BARs won't be accessible which
>>>>>> could make the device not function as expected.
>>>>>>
>>>>>> Do you think it would be reasonable to attempt the hypercall on Arm
>>>>>> also, and in case of error (on Arm) just print a warning message and
>>>>>> continue operations as normal?
>>>>> I think this would lead to a warning printed on lots of devices where in
>>>>> fact there would be no issues.
>>>>>
>>>>> If this is an issue for a device driver because it cannot operate without
>>>>> I/O ports, this will be raised by the driver inside the guest.
>>>> On what basis would the driver complain? The kernel might know of
>>>> the MMIO equivalent for ports, and hence might allow the driver
>>>> to properly obtain whatever is needed to later access the ports.
>>>> Just that the port accesses then wouldn't work (possibly crashing
>>>> the guest, or making it otherwise misbehave).
>>> As ECAM and Arm does not support I/O ports, a driver requesting access
>>> to them would get an error back.
>>> So in practice it is not possible to try to access the ioports as there is no
>>> way on arm to use them (no instructions).
>>>
>>> A driver could misbehave by ignoring the fact that ioports are not there but
>>> I am not quite sure how we could solve that as it would be a bug in the driver.
>> The minimal thing I'd suggest (or maybe you're doing this already)
>> would be to expose such BARs to the guest as r/o zero, rather than
>> letting their port nature "shine through".
> If we have the same, but baremetal then which entity disallows
> those BARs to shine?
I'm sorry, but I don't understand what you're trying to say.
> I mean that if guest wants to crash... why
> should we stop it and try emulating something special for it?
This isn't about a guest "wanting to crash", but a driver potentially
getting mislead into thinking that it can driver a device a certain
way.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 9:40 ` Bertrand Marquis
@ 2021-10-12 10:03 ` Jan Beulich
0 siblings, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-12 10:03 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
On 12.10.2021 11:40, Bertrand Marquis wrote:
> Hi Jan,
>
>> On 12 Oct 2021, at 10:32, Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 12.10.2021 10:41, Bertrand Marquis wrote:
>>> Hi Jan,
>>>
>>>> On 12 Oct 2021, at 09:29, Jan Beulich <jbeulich@suse.com> wrote:
>>>>
>>>> On 11.10.2021 19:11, Bertrand Marquis wrote:
>>>>>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>>>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>>> I think the commit message needs to at least be expanded in order to
>>>>>>>> contain the information provided here. It might also be helpful to
>>>>>>>> figure out whether we would have to handle IO port accesses in the
>>>>>>>> future on Arm, or if it's fine to just ignore them.
>>>>>>>
>>>>>>> All our investigations and tests have been done without supporting it
>>>>>>> without any issues so this is not a critical feature (most devices can
>>>>>>> be operated without using the I/O ports).
>>>>>>
>>>>>> IMO we should let the users know they attempted to use a device with
>>>>>> BARs in the IO space, and that those BARs won't be accessible which
>>>>>> could make the device not function as expected.
>>>>>>
>>>>>> Do you think it would be reasonable to attempt the hypercall on Arm
>>>>>> also, and in case of error (on Arm) just print a warning message and
>>>>>> continue operations as normal?
>>>>>
>>>>> I think this would lead to a warning printed on lots of devices where in
>>>>> fact there would be no issues.
>>>>>
>>>>> If this is an issue for a device driver because it cannot operate without
>>>>> I/O ports, this will be raised by the driver inside the guest.
>>>>
>>>> On what basis would the driver complain? The kernel might know of
>>>> the MMIO equivalent for ports, and hence might allow the driver
>>>> to properly obtain whatever is needed to later access the ports.
>>>> Just that the port accesses then wouldn't work (possibly crashing
>>>> the guest, or making it otherwise misbehave).
>>>
>>> As ECAM and Arm does not support I/O ports, a driver requesting access
>>> to them would get an error back.
>>> So in practice it is not possible to try to access the ioports as there is no
>>> way on arm to use them (no instructions).
>>>
>>> A driver could misbehave by ignoring the fact that ioports are not there but
>>> I am not quite sure how we could solve that as it would be a bug in the driver.
>>
>> The minimal thing I'd suggest (or maybe you're doing this already)
>> would be to expose such BARs to the guest as r/o zero, rather than
>> letting their port nature "shine through".
>
> We are emulating an ECAM PCI which does not support I/O ports so I do not
> think we are (and can) expose those to guests.
I'm afraid I don't follow: What do you mean by "expose to guests" in
relation to what I've said? I did specifically suggest to hide the real
nature of such BARs, and instead surface them such that the guest would
have to conclude the BAR is actually not in use at all.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 10:01 ` Jan Beulich
@ 2021-10-12 10:06 ` Oleksandr Andrushchenko
2021-10-12 10:20 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Oleksandr Andrushchenko @ 2021-10-12 10:06 UTC (permalink / raw)
To: Jan Beulich, Bertrand Marquis
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini, Roger Pau Monné,
Oleksandr Andrushchenko
On 12.10.21 13:01, Jan Beulich wrote:
> On 12.10.2021 11:38, Oleksandr Andrushchenko wrote:
>> On 12.10.21 12:32, Jan Beulich wrote:
>>> On 12.10.2021 10:41, Bertrand Marquis wrote:
>>>>> On 12 Oct 2021, at 09:29, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> On 11.10.2021 19:11, Bertrand Marquis wrote:
>>>>>>> On 11 Oct 2021, at 17:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>> On Mon, Oct 11, 2021 at 02:16:19PM +0000, Bertrand Marquis wrote:
>>>>>>>>> On 11 Oct 2021, at 14:57, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>>>> I think the commit message needs to at least be expanded in order to
>>>>>>>>> contain the information provided here. It might also be helpful to
>>>>>>>>> figure out whether we would have to handle IO port accesses in the
>>>>>>>>> future on Arm, or if it's fine to just ignore them.
>>>>>>>> All our investigations and tests have been done without supporting it
>>>>>>>> without any issues so this is not a critical feature (most devices can
>>>>>>>> be operated without using the I/O ports).
>>>>>>> IMO we should let the users know they attempted to use a device with
>>>>>>> BARs in the IO space, and that those BARs won't be accessible which
>>>>>>> could make the device not function as expected.
>>>>>>>
>>>>>>> Do you think it would be reasonable to attempt the hypercall on Arm
>>>>>>> also, and in case of error (on Arm) just print a warning message and
>>>>>>> continue operations as normal?
>>>>>> I think this would lead to a warning printed on lots of devices where in
>>>>>> fact there would be no issues.
>>>>>>
>>>>>> If this is an issue for a device driver because it cannot operate without
>>>>>> I/O ports, this will be raised by the driver inside the guest.
>>>>> On what basis would the driver complain? The kernel might know of
>>>>> the MMIO equivalent for ports, and hence might allow the driver
>>>>> to properly obtain whatever is needed to later access the ports.
>>>>> Just that the port accesses then wouldn't work (possibly crashing
>>>>> the guest, or making it otherwise misbehave).
>>>> As ECAM and Arm does not support I/O ports, a driver requesting access
>>>> to them would get an error back.
>>>> So in practice it is not possible to try to access the ioports as there is no
>>>> way on arm to use them (no instructions).
>>>>
>>>> A driver could misbehave by ignoring the fact that ioports are not there but
>>>> I am not quite sure how we could solve that as it would be a bug in the driver.
>>> The minimal thing I'd suggest (or maybe you're doing this already)
>>> would be to expose such BARs to the guest as r/o zero, rather than
>>> letting their port nature "shine through".
>> If we have the same, but baremetal then which entity disallows
>> those BARs to shine?
> I'm sorry, but I don't understand what you're trying to say.
>
>> I mean that if guest wants to crash... why
>> should we stop it and try emulating something special for it?
> This isn't about a guest "wanting to crash", but a driver potentially
> getting mislead into thinking that it can driver a device a certain
> way.
Well, for the guest, as we do not advertise IO in the emulated host
bridge, the driver won't be able to allocate any IO at all. Thus, even
if we have a BAR with PCI_BASE_ADDRESS_SPACE_IO bit set, the
driver won't get anything. So, I think this is equivalent to a baremetal
use-case where we have no IO supported by the host bridge and
a device with IO BAR.
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 10:06 ` Oleksandr Andrushchenko
@ 2021-10-12 10:20 ` Jan Beulich
2021-10-12 10:41 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-12 10:20 UTC (permalink / raw)
To: Oleksandr Andrushchenko, Bertrand Marquis
Cc: Rahul Singh, xen-devel, Andre Przywara, Ian Jackson, Wei Liu,
Juergen Gross, Stefano Stabellini, Roger Pau Monné
On 12.10.2021 12:06, Oleksandr Andrushchenko wrote:
> On 12.10.21 13:01, Jan Beulich wrote:
>> On 12.10.2021 11:38, Oleksandr Andrushchenko wrote:
>>> On 12.10.21 12:32, Jan Beulich wrote:
>>>> The minimal thing I'd suggest (or maybe you're doing this already)
>>>> would be to expose such BARs to the guest as r/o zero, rather than
>>>> letting their port nature "shine through".
>>> If we have the same, but baremetal then which entity disallows
>>> those BARs to shine?
>> I'm sorry, but I don't understand what you're trying to say.
>>
>>> I mean that if guest wants to crash... why
>>> should we stop it and try emulating something special for it?
>> This isn't about a guest "wanting to crash", but a driver potentially
>> getting mislead into thinking that it can driver a device a certain
>> way.
> Well, for the guest, as we do not advertise IO in the emulated host
> bridge, the driver won't be able to allocate any IO at all. Thus, even
> if we have a BAR with PCI_BASE_ADDRESS_SPACE_IO bit set, the
> driver won't get anything. So, I think this is equivalent to a baremetal
> use-case where we have no IO supported by the host bridge and
> a device with IO BAR.
Oh, now I follow. Fair enough.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-11 9:27 ` Roger Pau Monné
2021-10-11 12:06 ` Michal Orzel
@ 2021-10-12 10:38 ` Michal Orzel
2021-10-13 8:30 ` Roger Pau Monné
2021-10-12 21:48 ` Stefano Stabellini
2 siblings, 1 reply; 190+ messages in thread
From: Michal Orzel @ 2021-10-12 10:38 UTC (permalink / raw)
To: Roger Pau Monné, Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Christian Lindig,
David Scott, Ian Jackson, Wei Liu, Andrew Cooper, George Dunlap,
Jan Beulich, Julien Grall, Stefano Stabellini, Volodymyr Babchuk
Hi Roger,
On 11.10.2021 11:27, Roger Pau Monné wrote:
> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>> Reject the use of this new flag for x86 as VPCI is not supported for
>> DOMU guests for x86.
>
> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
>
> Things like PVH vs PV get translated into CDF flags by create_dom0,
> and processed normally by the sanitise_domain_config logic, vPCI
> should be handled that way.
>
> Do you think you could see about fixing this?
>
> Thanks, Roger.
>
I have one question about this fix.
If I set XEN_DOMCTL_CDF_vpci for dom0 pvh in create_dom0, then in
sanitise_domain_config or arch_sanitise_domain_config I have no
knowledge on whether I am dom0 or not. I can check if I'm PVH but not if dom0.
This would be needed to add a warning if this flag is set but we are not dom0 pvh.
Any ideas?
Cheers,
Michal
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 10:20 ` Jan Beulich
@ 2021-10-12 10:41 ` Bertrand Marquis
2021-10-12 10:44 ` Jan Beulich
2021-10-12 14:53 ` Ian Jackson
0 siblings, 2 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 10:41 UTC (permalink / raw)
To: Jan Beulich
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
Hi Jan,
> On 12 Oct 2021, at 11:20, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 12.10.2021 12:06, Oleksandr Andrushchenko wrote:
>> On 12.10.21 13:01, Jan Beulich wrote:
>>> On 12.10.2021 11:38, Oleksandr Andrushchenko wrote:
>>>> On 12.10.21 12:32, Jan Beulich wrote:
>>>>> The minimal thing I'd suggest (or maybe you're doing this already)
>>>>> would be to expose such BARs to the guest as r/o zero, rather than
>>>>> letting their port nature "shine through".
>>>> If we have the same, but baremetal then which entity disallows
>>>> those BARs to shine?
>>> I'm sorry, but I don't understand what you're trying to say.
>>>
>>>> I mean that if guest wants to crash... why
>>>> should we stop it and try emulating something special for it?
>>> This isn't about a guest "wanting to crash", but a driver potentially
>>> getting mislead into thinking that it can driver a device a certain
>>> way.
>> Well, for the guest, as we do not advertise IO in the emulated host
>> bridge, the driver won't be able to allocate any IO at all. Thus, even
>> if we have a BAR with PCI_BASE_ADDRESS_SPACE_IO bit set, the
>> driver won't get anything. So, I think this is equivalent to a baremetal
>> use-case where we have no IO supported by the host bridge and
>> a device with IO BAR.
>
> Oh, now I follow. Fair enough.
So there is no comment remaining on this patch ?
Would it possible to get an ack on it ?
Thanks
Bertrand
>
> Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 10:41 ` Bertrand Marquis
@ 2021-10-12 10:44 ` Jan Beulich
2021-10-12 14:53 ` Ian Jackson
1 sibling, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-12 10:44 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Ian Jackson, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
On 12.10.2021 12:41, Bertrand Marquis wrote:
> So there is no comment remaining on this patch ?
No, but the comments weren't on the patch anyway, but on a question
raised in the course of the discussion.
> Would it possible to get an ack on it ?
No, as I can't ack this patch: I'm not a maintainer of the file being
changed. And I don't feel qualified to give an R-b.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-06 17:40 ` [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
2021-10-07 13:43 ` Jan Beulich
2021-10-11 10:51 ` Roger Pau Monné
@ 2021-10-12 14:32 ` Julien Grall
2021-10-12 14:34 ` Bertrand Marquis
2021-10-13 8:45 ` Roger Pau Monné
3 siblings, 1 reply; 190+ messages in thread
From: Julien Grall @ 2021-10-12 14:32 UTC (permalink / raw)
To: Rahul Singh, xen-devel
Cc: bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Paul Durrant, Roger Pau Monné
Hi Rahul,
On 06/10/2021 18:40, Rahul Singh wrote:
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index c9277b5c6d..91d614b37e 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -2,6 +2,7 @@
> #define __ASM_DOMAIN_H__
>
> #include <xen/cache.h>
> +#include <xen/nospec.h>
> #include <xen/timer.h>
> #include <asm/page.h>
> #include <asm/p2m.h>
> @@ -262,7 +263,11 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>
> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>
> -#define has_vpci(d) ({ (void)(d); false; })
> +/*
> + * For X86 VPCI is enabled and tested for PVH DOM0 only but
> + * for ARM we enable support VPCI for guest domain also.
> + */
This is the sort of comment that will easily get rot if we change the
behavior on x86. But I find a bit odd to justify the implementation
based on x86. Can we simply avoid to mention x86?
> +#define has_vpci(d) evaluate_nospec((d)->options & XEN_DOMCTL_CDF_vpci)
>
> #endif /* __ASM_DOMAIN_H__ */
>
> diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
> index e076951032..c4a4fdcbc2 100644
> --- a/xen/include/asm-x86/pci.h
> +++ b/xen/include/asm-x86/pci.h
> @@ -6,8 +6,6 @@
> #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
> #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
>
> -#define MMCFG_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
> -
> #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
> || id == 0x01268086 || id == 0x01028086 \
> || id == 0x01128086 || id == 0x01228086 \
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index d46c61fca9..44be337dec 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>
> +/*
> + * 256 MB is reserved for VPCI configuration space based on calculation
> + * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
> + */
> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
> +
> /* ACPI tables physical address */
> #define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
> #define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
> diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
> index 70ac25345c..43b8a08170 100644
> --- a/xen/include/xen/pci.h
> +++ b/xen/include/xen/pci.h
> @@ -40,6 +40,8 @@
> #define PCI_SBDF3(s,b,df) \
> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>
> +#define MMCFG_BDF(addr) (((addr) & 0x0ffff000) >> 12)
> +
> typedef union {
> uint32_t sbdf;
> struct {
>
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-12 14:32 ` Julien Grall
@ 2021-10-12 14:34 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 14:34 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, Xen-devel, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Paul Durrant, Roger Pau Monné
Hi Julien,
> On 12 Oct 2021, at 15:32, Julien Grall <julien@xen.org> wrote:
>
> Hi Rahul,
>
> On 06/10/2021 18:40, Rahul Singh wrote:
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index c9277b5c6d..91d614b37e 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -2,6 +2,7 @@
>> #define __ASM_DOMAIN_H__
>> #include <xen/cache.h>
>> +#include <xen/nospec.h>
>> #include <xen/timer.h>
>> #include <asm/page.h>
>> #include <asm/p2m.h>
>> @@ -262,7 +263,11 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>> -#define has_vpci(d) ({ (void)(d); false; })
>> +/*
>> + * For X86 VPCI is enabled and tested for PVH DOM0 only but
>> + * for ARM we enable support VPCI for guest domain also.
>> + */
>
> This is the sort of comment that will easily get rot if we change the behavior on x86. But I find a bit odd to justify the implementation based on x86. Can we simply avoid to mention x86?
Yes I will remove the x86 part of it.
Cheers
Bertrand
>
>> +#define has_vpci(d) evaluate_nospec((d)->options & XEN_DOMCTL_CDF_vpci)
>> #endif /* __ASM_DOMAIN_H__ */
>> diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
>> index e076951032..c4a4fdcbc2 100644
>> --- a/xen/include/asm-x86/pci.h
>> +++ b/xen/include/asm-x86/pci.h
>> @@ -6,8 +6,6 @@
>> #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
>> #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
>> -#define MMCFG_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
>> -
>> #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
>> || id == 0x01268086 || id == 0x01028086 \
>> || id == 0x01128086 || id == 0x01228086 \
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index d46c61fca9..44be337dec 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
>> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>> +/*
>> + * 256 MB is reserved for VPCI configuration space based on calculation
>> + * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
>> + */
>> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
>> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
>> +
>> /* ACPI tables physical address */
>> #define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
>> #define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
>> diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
>> index 70ac25345c..43b8a08170 100644
>> --- a/xen/include/xen/pci.h
>> +++ b/xen/include/xen/pci.h
>> @@ -40,6 +40,8 @@
>> #define PCI_SBDF3(s,b,df) \
>> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>> +#define MMCFG_BDF(addr) (((addr) & 0x0ffff000) >> 12)
>> +
>> typedef union {
>> uint32_t sbdf;
>> struct {
>
> Cheers,
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 10:41 ` Bertrand Marquis
2021-10-12 10:44 ` Jan Beulich
@ 2021-10-12 14:53 ` Ian Jackson
2021-10-12 16:15 ` Bertrand Marquis
1 sibling, 1 reply; 190+ messages in thread
From: Ian Jackson @ 2021-10-12 14:53 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Jan Beulich, Oleksandr Andrushchenko, Rahul Singh, xen-devel,
Andre Przywara, Ian Jackson, Wei Liu, Juergen Gross,
Stefano Stabellini, Roger Pau Monné
Bertrand Marquis writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
> So there is no comment remaining on this patch ?
I have been following this thread. I think I have a notion of why
this is needed but I'm not 100% clear on it. The commit message says
this:
> ARM architecture does not implement I/O ports. Ignore this call on ARM
> to avoid the overhead of making a hypercall just for Xen to return
> -ENOSYS.
which implies it's a performance improvement. But the change also
suppresses an error return, so this commit message is false. I think
that the thread has concluded something different, but it should be
explained in the commit message. The purpose of a commit message is
precisely to capture the kind of considerations and discussion that
occurred in this thread.
If the overall outcome implied by this patch is correct (as I *think*
the thread has concluded) then I don't think the #ifdefery is
appropriate. This should be done with a new arch-specific function in
libxl_x86.c and libxl_arm.c. I'm not sure precisely what that
function should be called, but maybe something like
libxl_ioports_supported
?
I see that the fact that we avoid #ifdefs wasn't documented in
CODING_STYLE, so I have sent a patch to add that. Sorry about that.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-06 17:40 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
2021-10-06 18:01 ` Julien Grall
@ 2021-10-12 15:03 ` Ian Jackson
1 sibling, 0 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-12 15:03 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk
Rahul Singh writes ("[PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
> libxl will create an emulated PCI device tree node in the device tree to
> enable the guest OS to discover the virtual PCI during guest boot.
> Emulated PCI device tree node will only be created when there is any
> device assigned to guest.
>
> A new area has been reserved in the arm guest physical map at
> which the VPCI bus is declared in the device tree (reg and ranges
> parameters of the node).
I think this series is targeted for 4.16. Stefano drew this patch to
my attention. I have read the thread on this patch and it is not
clear to me that it has converged. Code freeze is imminent.
Is there some way I can help here ?
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-11 12:41 ` Bertrand Marquis
2021-10-11 13:09 ` Jan Beulich
@ 2021-10-12 15:04 ` Julien Grall
2021-10-12 16:12 ` Bertrand Marquis
1 sibling, 1 reply; 190+ messages in thread
From: Julien Grall @ 2021-10-12 15:04 UTC (permalink / raw)
To: Bertrand Marquis, Jan Beulich
Cc: Rahul Singh, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 11/10/2021 13:41, Bertrand Marquis wrote:
> Hi Jan,
Hi Bertrand,
> As Rahul is on leave, I will answer you and make the changes needed.
>
>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>> Independent of this - is bare metal Arm enforcing this same
>> alignment restriction (unconditionally)? Iirc on x86 we felt we'd
>> better synthesize misaligned accesses.
>
> Unaligned IO access could be synthesise also on arm to but I would
> rather not make such a change now without testing it (and there is
> also a question of it making sense).
Yes it makes sense. I actually have an item in my TODO list to forbid
unaligned access because they should not happen on any device we
currently emulate.
Although, I am not aware of any issue other than the guest would shoot
itself in the foot if this happens.
>
> So if it is ok with you I will keep that check and discuss it with Rahul
> when he is back. I will add a comment in the code to make that clear.
I am OK with it.
[...]
>> Throughout this series I haven't been able to spot where the HAS_VPCI
>> Kconfig symbol would get selected. Hence I cannot tell whether all of
>> this is Arm64-specific. Otherwise I wonder whether size 8 actually
>> can occur on Arm32.
>
> Dabt.size could be 3 even on ARM32 but we should not allow 64bit
> access on mmio regions on arm32.
Hmmm... Looking at the Armv7 and Armv8 spec, ldrd/strd (64-bit read)
would not present a valid ISV. So I think it is not be possible to have
dabt.size = 3 for 32-bit domain. But I agree we probably want to harden
the code.
>
> So I will surround this code with ifdef CONFIG_ARM_64 and add a test
> for len > 4 to prevent this case on 32bit.
>
> To be completely right we should disable this also for 32bit guests but
> this change would be a bit more invasive.
I think the following should be sufficient:
if ( is_domain_32bit_domain() && len > 4 )
return ...;
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-12 15:04 ` Julien Grall
@ 2021-10-12 16:12 ` Bertrand Marquis
2021-10-12 16:20 ` Julien Grall
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 16:12 UTC (permalink / raw)
To: Julien Grall
Cc: Jan Beulich, Rahul Singh, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
Hi Julien,
> On 12 Oct 2021, at 16:04, Julien Grall <julien@xen.org> wrote:
>
> On 11/10/2021 13:41, Bertrand Marquis wrote:
>> Hi Jan,
>
> Hi Bertrand,
>
>> As Rahul is on leave, I will answer you and make the changes needed.
>>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>>> Independent of this - is bare metal Arm enforcing this same
>>> alignment restriction (unconditionally)? Iirc on x86 we felt we'd
>>> better synthesize misaligned accesses.
>> Unaligned IO access could be synthesise also on arm to but I would
>> rather not make such a change now without testing it (and there is
>> also a question of it making sense).
>
> Yes it makes sense. I actually have an item in my TODO list to forbid unaligned access because they should not happen on any device we currently emulate.
>
> Although, I am not aware of any issue other than the guest would shoot itself in the foot if this happens.
>
>> So if it is ok with you I will keep that check and discuss it with Rahul
>> when he is back. I will add a comment in the code to make that clear.
>
> I am OK with it.
>
> [...]
>
>>> Throughout this series I haven't been able to spot where the HAS_VPCI
>>> Kconfig symbol would get selected. Hence I cannot tell whether all of
>>> this is Arm64-specific. Otherwise I wonder whether size 8 actually
>>> can occur on Arm32.
>> Dabt.size could be 3 even on ARM32 but we should not allow 64bit
>> access on mmio regions on arm32.
>
> Hmmm... Looking at the Armv7 and Armv8 spec, ldrd/strd (64-bit read) would not present a valid ISV. So I think it is not be possible to have dabt.size = 3 for 32-bit domain. But I agree we probably want to harden the code.
>
>> So I will surround this code with ifdef CONFIG_ARM_64 and add a test
>> for len > 4 to prevent this case on 32bit.
>> To be completely right we should disable this also for 32bit guests but
>> this change would be a bit more invasive.
>
> I think the following should be sufficient:
>
> if ( is_domain_32bit_domain() && len > 4 )
> return ...;
With the last request from Roger to use the function implemented in arch/x86/hw/io.c, the function will move to vpci.h so using is_32bit_domain will not be possible without ifdefery CONFIG_ARM.
Also I have no access to the domain there.
So the best I can do for now would be something like:
#ifdef CONFIG_ARM_32
If (len == 8)
return false
#endif
A 32bit guest on 64bit xen would not be checked.
Would that be ok for now ?
I could add a comment in the code to warn about the limitation.
Cheers
Bertrand
>
> Cheers,
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 14:53 ` Ian Jackson
@ 2021-10-12 16:15 ` Bertrand Marquis
2021-10-12 16:29 ` Ian Jackson
2021-10-13 8:02 ` Roger Pau Monné
0 siblings, 2 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 16:15 UTC (permalink / raw)
To: Ian Jackson
Cc: Jan Beulich, Oleksandr Andrushchenko, Rahul Singh, xen-devel,
Andre Przywara, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
Hi Ian,
> On 12 Oct 2021, at 15:53, Ian Jackson <iwj@xenproject.org> wrote:
>
> Bertrand Marquis writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
>> So there is no comment remaining on this patch ?
>
> I have been following this thread. I think I have a notion of why
> this is needed but I'm not 100% clear on it. The commit message says
> this:
>
>> ARM architecture does not implement I/O ports. Ignore this call on ARM
>> to avoid the overhead of making a hypercall just for Xen to return
>> -ENOSYS.
>
> which implies it's a performance improvement. But the change also
> suppresses an error return, so this commit message is false. I think
> that the thread has concluded something different, but it should be
> explained in the commit message. The purpose of a commit message is
> precisely to capture the kind of considerations and discussion that
> occurred in this thread.
I can add something in the commit message about the fact that we improve
performance and prevent to do a call that is and will not be supported in Xen.
>
> If the overall outcome implied by this patch is correct (as I *think*
> the thread has concluded) then I don't think the #ifdefery is
> appropriate. This should be done with a new arch-specific function in
> libxl_x86.c and libxl_arm.c. I'm not sure precisely what that
> function should be called, but maybe something like
> libxl_ioports_supported
> ?
>
> I see that the fact that we avoid #ifdefs wasn't documented in
> CODING_STYLE, so I have sent a patch to add that. Sorry about that.
I saw your change in CODING_STYLE and I understand the request.
I will try to see if we can handle this change before the feature freeze.
Regards
Bertrand
>
> Thanks,
> Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-12 16:12 ` Bertrand Marquis
@ 2021-10-12 16:20 ` Julien Grall
2021-10-12 17:50 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Julien Grall @ 2021-10-12 16:20 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Jan Beulich, Rahul Singh, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 12/10/2021 17:12, Bertrand Marquis wrote:
> Hi Julien,
>
>> On 12 Oct 2021, at 16:04, Julien Grall <julien@xen.org> wrote:
>>
>> On 11/10/2021 13:41, Bertrand Marquis wrote:
>>> Hi Jan,
>>
>> Hi Bertrand,
>>
>>> As Rahul is on leave, I will answer you and make the changes needed.
>>>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>>>> Independent of this - is bare metal Arm enforcing this same
>>>> alignment restriction (unconditionally)? Iirc on x86 we felt we'd
>>>> better synthesize misaligned accesses.
>>> Unaligned IO access could be synthesise also on arm to but I would
>>> rather not make such a change now without testing it (and there is
>>> also a question of it making sense).
>>
>> Yes it makes sense. I actually have an item in my TODO list to forbid unaligned access because they should not happen on any device we currently emulate.
>>
>> Although, I am not aware of any issue other than the guest would shoot itself in the foot if this happens.
>>
>>> So if it is ok with you I will keep that check and discuss it with Rahul
>>> when he is back. I will add a comment in the code to make that clear.
>>
>> I am OK with it.
>>
>> [...]
>>
>>>> Throughout this series I haven't been able to spot where the HAS_VPCI
>>>> Kconfig symbol would get selected. Hence I cannot tell whether all of
>>>> this is Arm64-specific. Otherwise I wonder whether size 8 actually
>>>> can occur on Arm32.
>>> Dabt.size could be 3 even on ARM32 but we should not allow 64bit
>>> access on mmio regions on arm32.
>>
>> Hmmm... Looking at the Armv7 and Armv8 spec, ldrd/strd (64-bit read) would not present a valid ISV. So I think it is not be possible to have dabt.size = 3 for 32-bit domain. But I agree we probably want to harden the code.
>>
>>> So I will surround this code with ifdef CONFIG_ARM_64 and add a test
>>> for len > 4 to prevent this case on 32bit.
>>> To be completely right we should disable this also for 32bit guests but
>>> this change would be a bit more invasive.
>>
>> I think the following should be sufficient:
>>
>> if ( is_domain_32bit_domain() && len > 4 )
>> return ...;
>
> With the last request from Roger to use the function implemented in arch/x86/hw/io.c, the function will move to vpci.h so using is_32bit_domain will not be possible without ifdefery CONFIG_ARM.
> Also I have no access to the domain there.
>
> So the best I can do for now would be something like:
> #ifdef CONFIG_ARM_32
> If (len == 8)
> return false
> #endif
>
> A 32bit guest on 64bit xen would not be checked.
>
> Would that be ok for now ?
I think the #ifdef is a bit pointless. My preference would be to not add
the #ifdef but instead add...
>
> I could add a comment in the code to warn about the limitation.
.. a comment for now as this is only an hardening problem.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 16:15 ` Bertrand Marquis
@ 2021-10-12 16:29 ` Ian Jackson
2021-10-12 20:42 ` Stefano Stabellini
2021-10-13 8:02 ` Roger Pau Monné
1 sibling, 1 reply; 190+ messages in thread
From: Ian Jackson @ 2021-10-12 16:29 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Jan Beulich, Oleksandr Andrushchenko, Rahul Singh, xen-devel,
Andre Przywara, Wei Liu, Juergen Gross, Stefano Stabellini,
Roger Pau Monné
Bertrand Marquis writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
> I can add something in the commit message about the fact that we improve
> performance and prevent to do a call that is and will not be supported in Xen.
Thanks but I'm afraid I don't think that is a correct summary of the
thread. Nor would it be an adequate justification for the change. At
least, not unless you plan to write something considerably longer (and
more precise).
Firstly, I'm not convinced this change would be justified by the
performance impact. This is a small number of hypercalls during
domain startup. Usually none, I think ? If someone wants to optimise
domain startup speed then I am very open to that but I think this
change will make negligible change in practice. Unless someone wants
to tell me I'm wrong about that ? And if I am wrong about that then
an explanation of why my suppositions are wrong ought to go in the
commit message.
Secondly, there is no justification there for the change in error
status.
Why is this change needed ? (What goes wrong if it is omitted ?)
That is what the commit message ought to answer.
Plus, given that it stubs out a function to make it into a no-op, that
itself requires an explanation. Why is it OK for this function which
is supposed to do a thing, to in fact not do anything at all and
return successfully saying "yes I did that" ?
I think (having read the thread) that I know the answers to these
questions but it needs to be clearly and explicitly written down.
> I saw your change in CODING_STYLE and I understand the request.
> I will try to see if we can handle this change before the feature freeze.
Thanks. I doubt that this will be hard. I am more worried about the
commit message.
Indeed, since we haven't had the rationale for this change explicitly
written down, there is a risk that when we do so, we will discover
some problem with the approach that we had previously overlooked.
Discovering that kind of thing is one reason to explicitly write down
why we are doing what we are doing, but this situation does mean we
shouldn't feel we've yet achieved confidence that this patch is right.
Sorry,
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-12 16:20 ` Julien Grall
@ 2021-10-12 17:50 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-12 17:50 UTC (permalink / raw)
To: Julien Grall
Cc: Jan Beulich, Rahul Singh, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
Hi Julien,
> On 12 Oct 2021, at 17:20, Julien Grall <julien@xen.org> wrote:
>
>
>
> On 12/10/2021 17:12, Bertrand Marquis wrote:
>> Hi Julien,
>>> On 12 Oct 2021, at 16:04, Julien Grall <julien@xen.org> wrote:
>>>
>>> On 11/10/2021 13:41, Bertrand Marquis wrote:
>>>> Hi Jan,
>>>
>>> Hi Bertrand,
>>>
>>>> As Rahul is on leave, I will answer you and make the changes needed.
>>>>> On 7 Oct 2021, at 14:43, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> Independent of this - is bare metal Arm enforcing this same
>>>>> alignment restriction (unconditionally)? Iirc on x86 we felt we'd
>>>>> better synthesize misaligned accesses.
>>>> Unaligned IO access could be synthesise also on arm to but I would
>>>> rather not make such a change now without testing it (and there is
>>>> also a question of it making sense).
>>>
>>> Yes it makes sense. I actually have an item in my TODO list to forbid unaligned access because they should not happen on any device we currently emulate.
>>>
>>> Although, I am not aware of any issue other than the guest would shoot itself in the foot if this happens.
>>>
>>>> So if it is ok with you I will keep that check and discuss it with Rahul
>>>> when he is back. I will add a comment in the code to make that clear.
>>>
>>> I am OK with it.
>>>
>>> [...]
>>>
>>>>> Throughout this series I haven't been able to spot where the HAS_VPCI
>>>>> Kconfig symbol would get selected. Hence I cannot tell whether all of
>>>>> this is Arm64-specific. Otherwise I wonder whether size 8 actually
>>>>> can occur on Arm32.
>>>> Dabt.size could be 3 even on ARM32 but we should not allow 64bit
>>>> access on mmio regions on arm32.
>>>
>>> Hmmm... Looking at the Armv7 and Armv8 spec, ldrd/strd (64-bit read) would not present a valid ISV. So I think it is not be possible to have dabt.size = 3 for 32-bit domain. But I agree we probably want to harden the code.
>>>
>>>> So I will surround this code with ifdef CONFIG_ARM_64 and add a test
>>>> for len > 4 to prevent this case on 32bit.
>>>> To be completely right we should disable this also for 32bit guests but
>>>> this change would be a bit more invasive.
>>>
>>> I think the following should be sufficient:
>>>
>>> if ( is_domain_32bit_domain() && len > 4 )
>>> return ...;
>> With the last request from Roger to use the function implemented in arch/x86/hw/io.c, the function will move to vpci.h so using is_32bit_domain will not be possible without ifdefery CONFIG_ARM.
>> Also I have no access to the domain there.
>> So the best I can do for now would be something like:
>> #ifdef CONFIG_ARM_32
>> If (len == 8)
>> return false
>> #endif
>> A 32bit guest on 64bit xen would not be checked.
>> Would that be ok for now ?
>
> I think the #ifdef is a bit pointless. My preference would be to not add the #ifdef but instead add...
>
>> I could add a comment in the code to warn about the limitation.
>
> .. a comment for now as this is only an hardening problem.
Agree I will do that.
Cheers
Bertrand
>
> Cheers,
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 16:29 ` Ian Jackson
@ 2021-10-12 20:42 ` Stefano Stabellini
2021-10-13 8:07 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-12 20:42 UTC (permalink / raw)
To: Ian Jackson
Cc: Bertrand Marquis, Jan Beulich, Oleksandr Andrushchenko,
Rahul Singh, xen-devel, Andre Przywara, Wei Liu, Juergen Gross,
Stefano Stabellini, Roger Pau Monné
On Tue, 12 Oct 2021, Ian Jackson wrote:
> Bertrand Marquis writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
> > I can add something in the commit message about the fact that we improve
> > performance and prevent to do a call that is and will not be supported in Xen.
>
> Thanks but I'm afraid I don't think that is a correct summary of the
> thread. Nor would it be an adequate justification for the change. At
> least, not unless you plan to write something considerably longer (and
> more precise).
>
> Firstly, I'm not convinced this change would be justified by the
> performance impact. This is a small number of hypercalls during
> domain startup. Usually none, I think ? If someone wants to optimise
> domain startup speed then I am very open to that but I think this
> change will make negligible change in practice. Unless someone wants
> to tell me I'm wrong about that ? And if I am wrong about that then
> an explanation of why my suppositions are wrong ought to go in the
> commit message.
>
> Secondly, there is no justification there for the change in error
> status.
>
> Why is this change needed ? (What goes wrong if it is omitted ?)
> That is what the commit message ought to answer.
>
> Plus, given that it stubs out a function to make it into a no-op, that
> itself requires an explanation. Why is it OK for this function which
> is supposed to do a thing, to in fact not do anything at all and
> return successfully saying "yes I did that" ?
>
> I think (having read the thread) that I know the answers to these
> questions but it needs to be clearly and explicitly written down.
>
> > I saw your change in CODING_STYLE and I understand the request.
> > I will try to see if we can handle this change before the feature freeze.
>
> Thanks. I doubt that this will be hard. I am more worried about the
> commit message.
>
> Indeed, since we haven't had the rationale for this change explicitly
> written down, there is a risk that when we do so, we will discover
> some problem with the approach that we had previously overlooked.
>
> Discovering that kind of thing is one reason to explicitly write down
> why we are doing what we are doing, but this situation does mean we
> shouldn't feel we've yet achieved confidence that this patch is right.
I don't think it is about performance. From a performance point of view,
we could make as many (unneeded) hypercalls as required. It is mostly
about minimizing unwanted changes to common libxl code. Let me explain.
IO ports on ARM don't exist so all IO ports related hypercalls are going
to fail. This is expected. Today, a failure of
xc_domain_ioport_permission would turn into a critical failure at domain
creation. We need to avoid this outcome; instead we want to continue
with domain creation as normal even if xc_domain_ioport_permission
fails. (FYI the underlying hypercall XEN_DOMCTL_ioport_permission is not
implemented on ARM so it would return -ENOSYS.)
We have a few options to achieve this goal:
1) No xc_domain_ioport_permission calls on ARM
Use #ifdefs or similar checks in libxl_pci.c to avoid calling
xc_domain_ioport_permission on ARM. This could be best but it would
cause some churn in arch-neutral libxl code.
2) Handle xc_domain_ioport_permission errors in libxl
Introduce checks on the return value of xc_domain_ioport_permission
and ignore specific errors on ARM in libxl_pci.c.
For instance: if (ARM && rc == -ENOSYS) continue.
This might cause less churn than 1) but still requires a few changes
in arch-neutral libxl code.
3) Force XEN_DOMCTL_ioport_permission to return zero on ARM
Force the hypercall to return success even if it did nothing.
Currently it returns -ENOSYS.
This is possible but it wasn't chosen for the implementation as we
felt that the hypercall should reflect what was actually done
(nothing) and it should be userspace to handle the error. I guess
this could be argued either way.
4) Force xc_domain_ioport_permission to return zero on ARM
Force xc_domain_ioport_permission to return success even if the
hypercall would return -ENOSYS. This way there are no changes to
libxl.
This is what the patch currently implements by using #ifdef in
xc_domain_ioport_permission. It could also have achieved the same
goal by making the implementation of xc_domain_ioport_permission
arch-specific, and in the ARM implementation returning 0.
All options above achieve the goal of a successful domain creation with
PCI device assigned on ARM. You might be able to think of other options
as well. I think noone here is really set on using one option over the
other -- as long as xc_domain_ioport_permission failures don't turn into
domain creation failures on ARM we are good.
Let us know what you think.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-12 8:04 ` Jan Beulich
@ 2021-10-12 21:37 ` Stefano Stabellini
2021-10-13 6:10 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-12 21:37 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, Bertrand Marquis, Rahul Singh,
Andre Przywara, Julien Grall, Volodymyr Babchuk, Andrew Cooper,
George Dunlap, Ian Jackson, Wei Liu, Paul Durrant,
Roger Pau Monné,
xen-devel
On Tue, 12 Oct 2021, Jan Beulich wrote:
> On 11.10.2021 20:18, Stefano Stabellini wrote:
> > On Mon, 11 Oct 2021, Jan Beulich wrote:
> >> On 11.10.2021 15:34, Bertrand Marquis wrote:
> >>>> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
> >>>> On 11.10.2021 14:41, Bertrand Marquis wrote:
> >>>>> But digging deeper into this, I would have 2 questions:
> >>>>>
> >>>>> - msi_cleanup was done there after a request from Stefano, but is not
> >>>>> done in case or iommu error, is there an issue to solve here ?
> >>>>
> >>>> Maybe, but I'm not sure. This very much depends on what a domain
> >>>> could in principle do with a partly set-up device. Plus let's
> >>>> not forget that we're talking of only Dom0 here (for now at least,
> >>>> i.e. not considering the dom0less case).
> >>>>
> >>>> But I'd also like to further defer to Stefano.
> >>>
> >>> Ok, I must admit I do not really see at that stage why doing an MSI cleanup
> >>> could be needed so I will wait for Stefano to know if I need to keep this when
> >>> moving the block up (at the end it is theoretical right now as this is empty).
> >
> > I know that MSIs are not supported yet on ARM (pci_cleanup_msi does
> > nothing). But I wanted to make sure that the pci_cleanup_msi() calls are
> > present anywhere necessary, especially on the error paths. So that once
> > we add MSI support, we don't need to search through the code to find all
> > the error paths missing a pci_cleanup_msi() call.
> >
> > To answer your first question: you are right, we are also missing a
> > pci_cleanup_msi() call in the case of IOMMU error. So it might be better
> > to move the call to pci_cleanup_msi() under the "out" label so that we
> > can do it once for both cases.
> >
> > To answer your second point about whether it is necessary at all: if
> > MSIs and MSI-Xs cannot be already setup at this point at all (not even
> > the enable bit), then we don't need any call to pci_cleanup_msi() in
> > pci_add_device.
>
> Well, at the very least MSI can't be set up ahead of the traps getting
> put in place. Whether partial success of putting traps in place may
> allow a cunning guest to set up MSI may depend on further aspects.
Good point about MSIs not being setup before the traps. We should remove
the call to pci_cleanup_msi() in the error path then.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-11 9:27 ` Roger Pau Monné
2021-10-11 12:06 ` Michal Orzel
2021-10-12 10:38 ` Michal Orzel
@ 2021-10-12 21:48 ` Stefano Stabellini
2021-10-13 6:18 ` Jan Beulich
2 siblings, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-12 21:48 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Rahul Singh, xen-devel, bertrand.marquis, Andre.Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk
[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]
On Mon, 11 Oct 2021, Roger Pau Monné wrote:
> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
> > Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> > Reject the use of this new flag for x86 as VPCI is not supported for
> > DOMU guests for x86.
>
> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
>
> Things like PVH vs PV get translated into CDF flags by create_dom0,
> and processed normally by the sanitise_domain_config logic, vPCI
> should be handled that way.
>
> Do you think you could see about fixing this?
Andrew suggested to use XEN_SYSCTL_PHYSCAP_vpci to check whether we can
set XEN_DOMCTL_CDF_vpci in libxl and it looks like we have consensus on
this approach. [1][2]
So it makes sense that XEN_DOMCTL_CDF_vpci is only set when
XEN_SYSCTL_PHYSCAP_vpci is also set, i.e. XEN_SYSCTL_PHYSCAP_vpci ==
XEN_DOMCTL_CDF_vpci.
From [2], XEN_SYSCTL_PHYSCAP_vpci is not going to be set on x86, so then
XEN_DOMCTL_CDF_vpci should also be left unset?
If you think XEN_DOMCTL_CDF_vpci should be set for x86 PVH Dom0, then
XEN_SYSCTL_PHYSCAP_vpci should also be set for x86 PVH Dom0.
[1] https://marc.info/?l=xen-devel&m=163372953907637
[2] https://marc.info/?l=xen-devel&m=163395821428850
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-12 21:37 ` Stefano Stabellini
@ 2021-10-13 6:10 ` Jan Beulich
2021-10-13 10:02 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-13 6:10 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Bertrand Marquis, Rahul Singh, Andre Przywara, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 12.10.2021 23:37, Stefano Stabellini wrote:
> On Tue, 12 Oct 2021, Jan Beulich wrote:
>> On 11.10.2021 20:18, Stefano Stabellini wrote:
>>> On Mon, 11 Oct 2021, Jan Beulich wrote:
>>>> On 11.10.2021 15:34, Bertrand Marquis wrote:
>>>>>> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>> On 11.10.2021 14:41, Bertrand Marquis wrote:
>>>>>>> But digging deeper into this, I would have 2 questions:
>>>>>>>
>>>>>>> - msi_cleanup was done there after a request from Stefano, but is not
>>>>>>> done in case or iommu error, is there an issue to solve here ?
>>>>>>
>>>>>> Maybe, but I'm not sure. This very much depends on what a domain
>>>>>> could in principle do with a partly set-up device. Plus let's
>>>>>> not forget that we're talking of only Dom0 here (for now at least,
>>>>>> i.e. not considering the dom0less case).
>>>>>>
>>>>>> But I'd also like to further defer to Stefano.
>>>>>
>>>>> Ok, I must admit I do not really see at that stage why doing an MSI cleanup
>>>>> could be needed so I will wait for Stefano to know if I need to keep this when
>>>>> moving the block up (at the end it is theoretical right now as this is empty).
>>>
>>> I know that MSIs are not supported yet on ARM (pci_cleanup_msi does
>>> nothing). But I wanted to make sure that the pci_cleanup_msi() calls are
>>> present anywhere necessary, especially on the error paths. So that once
>>> we add MSI support, we don't need to search through the code to find all
>>> the error paths missing a pci_cleanup_msi() call.
>>>
>>> To answer your first question: you are right, we are also missing a
>>> pci_cleanup_msi() call in the case of IOMMU error. So it might be better
>>> to move the call to pci_cleanup_msi() under the "out" label so that we
>>> can do it once for both cases.
>>>
>>> To answer your second point about whether it is necessary at all: if
>>> MSIs and MSI-Xs cannot be already setup at this point at all (not even
>>> the enable bit), then we don't need any call to pci_cleanup_msi() in
>>> pci_add_device.
>>
>> Well, at the very least MSI can't be set up ahead of the traps getting
>> put in place. Whether partial success of putting traps in place may
>> allow a cunning guest to set up MSI may depend on further aspects.
>
> Good point about MSIs not being setup before the traps. We should remove
> the call to pci_cleanup_msi() in the error path then.
Your reply makes me fear you didn't pay enough attention to the "partial"
in my earlier reply. The traps for the various registers can't all be set
up atomically, so there may be a transient period where enough traps are
already in place for a cunning guest to arrange for setup. Unless, as
said, there are further setup steps needed before a guest could succeed
in doing so.
But even if partial trap setup alone was sufficient, I think the cleaning
up of MSI then might still better go on the error path there than on that
of pci_add_device().
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-12 21:48 ` Stefano Stabellini
@ 2021-10-13 6:18 ` Jan Beulich
2021-10-13 7:11 ` Michal Orzel
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-13 6:18 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Rahul Singh, xen-devel, bertrand.marquis, Andre.Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Julien Grall, Volodymyr Babchuk,
Roger Pau Monné
On 12.10.2021 23:48, Stefano Stabellini wrote:
> On Mon, 11 Oct 2021, Roger Pau Monné wrote:
>> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
>>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>>> Reject the use of this new flag for x86 as VPCI is not supported for
>>> DOMU guests for x86.
>>
>> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
>> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
>>
>> Things like PVH vs PV get translated into CDF flags by create_dom0,
>> and processed normally by the sanitise_domain_config logic, vPCI
>> should be handled that way.
>>
>> Do you think you could see about fixing this?
>
> Andrew suggested to use XEN_SYSCTL_PHYSCAP_vpci to check whether we can
> set XEN_DOMCTL_CDF_vpci in libxl and it looks like we have consensus on
> this approach. [1][2]
>
> So it makes sense that XEN_DOMCTL_CDF_vpci is only set when
> XEN_SYSCTL_PHYSCAP_vpci is also set, i.e. XEN_SYSCTL_PHYSCAP_vpci ==
> XEN_DOMCTL_CDF_vpci.
>
> From [2], XEN_SYSCTL_PHYSCAP_vpci is not going to be set on x86, so then
> XEN_DOMCTL_CDF_vpci should also be left unset?
>
> If you think XEN_DOMCTL_CDF_vpci should be set for x86 PVH Dom0, then
> XEN_SYSCTL_PHYSCAP_vpci should also be set for x86 PVH Dom0.
Except that XEN_SYSCTL_PHYSCAP_vpci is not a domain specific attribute,
but a host-wide one.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 6:18 ` Jan Beulich
@ 2021-10-13 7:11 ` Michal Orzel
0 siblings, 0 replies; 190+ messages in thread
From: Michal Orzel @ 2021-10-13 7:11 UTC (permalink / raw)
To: Jan Beulich, Stefano Stabellini
Cc: Rahul Singh, xen-devel, bertrand.marquis, Andre.Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Julien Grall, Volodymyr Babchuk,
Roger Pau Monné
On 13.10.2021 08:18, Jan Beulich wrote:
> On 12.10.2021 23:48, Stefano Stabellini wrote:
>> On Mon, 11 Oct 2021, Roger Pau Monné wrote:
>>> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
>>>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>>>> Reject the use of this new flag for x86 as VPCI is not supported for
>>>> DOMU guests for x86.
>>>
>>> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
>>> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
>>>
>>> Things like PVH vs PV get translated into CDF flags by create_dom0,
>>> and processed normally by the sanitise_domain_config logic, vPCI
>>> should be handled that way.
>>>
>>> Do you think you could see about fixing this?
>>
>> Andrew suggested to use XEN_SYSCTL_PHYSCAP_vpci to check whether we can
>> set XEN_DOMCTL_CDF_vpci in libxl and it looks like we have consensus on
>> this approach. [1][2]
>>
>> So it makes sense that XEN_DOMCTL_CDF_vpci is only set when
>> XEN_SYSCTL_PHYSCAP_vpci is also set, i.e. XEN_SYSCTL_PHYSCAP_vpci ==
>> XEN_DOMCTL_CDF_vpci.
>>
>> From [2], XEN_SYSCTL_PHYSCAP_vpci is not going to be set on x86, so then
>> XEN_DOMCTL_CDF_vpci should also be left unset?
>>
>> If you think XEN_DOMCTL_CDF_vpci should be set for x86 PVH Dom0, then
>> XEN_SYSCTL_PHYSCAP_vpci should also be set for x86 PVH Dom0.
>
> Except that XEN_SYSCTL_PHYSCAP_vpci is not a domain specific attribute,
> but a host-wide one.
>
> Jan
>
>
I already prepared a patch introducing XEN_SYSCTL_PHYSCAP_vpci. We agreed
that the cap_vpci should not be set neither for x86 nor ARM. This means that
the flag vpci_is_available, which tells us about vPCI platform support (and is used
in condition to set cap_vpci) is set to false by default. Later on it should be set by vPCI driver.
For me it does not make sense for XEN_SYSCTL_PHYSCAP_vpci saying that platform does not support vPCI
but setting XEN_DOMCTL_CDF_vpci for dom0 pvh.
I would prefer not setting XEN_DOMCTL_CDF_vpci for now at all. This way we have a chance
to merge Rahul's series until friday.
Cheers,
Michal
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 16:15 ` Bertrand Marquis
2021-10-12 16:29 ` Ian Jackson
@ 2021-10-13 8:02 ` Roger Pau Monné
2021-10-13 12:02 ` Ian Jackson
1 sibling, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-13 8:02 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Ian Jackson, Jan Beulich, Oleksandr Andrushchenko, Rahul Singh,
xen-devel, Andre Przywara, Wei Liu, Juergen Gross,
Stefano Stabellini
On Tue, Oct 12, 2021 at 04:15:20PM +0000, Bertrand Marquis wrote:
> Hi Ian,
>
> > On 12 Oct 2021, at 15:53, Ian Jackson <iwj@xenproject.org> wrote:
> >
> > Bertrand Marquis writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
> >> So there is no comment remaining on this patch ?
> >
> > I have been following this thread. I think I have a notion of why
> > this is needed but I'm not 100% clear on it. The commit message says
> > this:
> >
> >> ARM architecture does not implement I/O ports. Ignore this call on ARM
> >> to avoid the overhead of making a hypercall just for Xen to return
> >> -ENOSYS.
> >
> > which implies it's a performance improvement. But the change also
> > suppresses an error return, so this commit message is false. I think
> > that the thread has concluded something different, but it should be
> > explained in the commit message. The purpose of a commit message is
> > precisely to capture the kind of considerations and discussion that
> > occurred in this thread.
>
> I can add something in the commit message about the fact that we improve
> performance and prevent to do a call that is and will not be supported in Xen.
IMO it would be good to modify the commit message so it covers the
fact that the emulated host bridge on Arm does not advertise IO port
support, so the guest is capable of realizing IO BARs are not
supported.
Otherwise it seems like the toolstack is ignoring a failure which
could cause a device to malfunction when passed though (which is still
the case, but the guest will be able to notice).
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-12 20:42 ` Stefano Stabellini
@ 2021-10-13 8:07 ` Roger Pau Monné
2021-10-13 11:52 ` Ian Jackson
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-13 8:07 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Ian Jackson, Bertrand Marquis, Jan Beulich,
Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Wei Liu, Juergen Gross
On Tue, Oct 12, 2021 at 01:42:22PM -0700, Stefano Stabellini wrote:
> On Tue, 12 Oct 2021, Ian Jackson wrote:
> > Bertrand Marquis writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
> > > I can add something in the commit message about the fact that we improve
> > > performance and prevent to do a call that is and will not be supported in Xen.
> >
> > Thanks but I'm afraid I don't think that is a correct summary of the
> > thread. Nor would it be an adequate justification for the change. At
> > least, not unless you plan to write something considerably longer (and
> > more precise).
> >
> > Firstly, I'm not convinced this change would be justified by the
> > performance impact. This is a small number of hypercalls during
> > domain startup. Usually none, I think ? If someone wants to optimise
> > domain startup speed then I am very open to that but I think this
> > change will make negligible change in practice. Unless someone wants
> > to tell me I'm wrong about that ? And if I am wrong about that then
> > an explanation of why my suppositions are wrong ought to go in the
> > commit message.
> >
> > Secondly, there is no justification there for the change in error
> > status.
> >
> > Why is this change needed ? (What goes wrong if it is omitted ?)
> > That is what the commit message ought to answer.
> >
> > Plus, given that it stubs out a function to make it into a no-op, that
> > itself requires an explanation. Why is it OK for this function which
> > is supposed to do a thing, to in fact not do anything at all and
> > return successfully saying "yes I did that" ?
> >
> > I think (having read the thread) that I know the answers to these
> > questions but it needs to be clearly and explicitly written down.
> >
> > > I saw your change in CODING_STYLE and I understand the request.
> > > I will try to see if we can handle this change before the feature freeze.
> >
> > Thanks. I doubt that this will be hard. I am more worried about the
> > commit message.
> >
> > Indeed, since we haven't had the rationale for this change explicitly
> > written down, there is a risk that when we do so, we will discover
> > some problem with the approach that we had previously overlooked.
> >
> > Discovering that kind of thing is one reason to explicitly write down
> > why we are doing what we are doing, but this situation does mean we
> > shouldn't feel we've yet achieved confidence that this patch is right.
>
>
> I don't think it is about performance. From a performance point of view,
> we could make as many (unneeded) hypercalls as required. It is mostly
> about minimizing unwanted changes to common libxl code. Let me explain.
>
>
> IO ports on ARM don't exist so all IO ports related hypercalls are going
> to fail. This is expected. Today, a failure of
> xc_domain_ioport_permission would turn into a critical failure at domain
> creation. We need to avoid this outcome; instead we want to continue
> with domain creation as normal even if xc_domain_ioport_permission
> fails. (FYI the underlying hypercall XEN_DOMCTL_ioport_permission is not
> implemented on ARM so it would return -ENOSYS.)
>
>
> We have a few options to achieve this goal:
>
>
> 1) No xc_domain_ioport_permission calls on ARM
>
> Use #ifdefs or similar checks in libxl_pci.c to avoid calling
> xc_domain_ioport_permission on ARM. This could be best but it would
> cause some churn in arch-neutral libxl code.
>
>
> 2) Handle xc_domain_ioport_permission errors in libxl
>
> Introduce checks on the return value of xc_domain_ioport_permission
> and ignore specific errors on ARM in libxl_pci.c.
> For instance: if (ARM && rc == -ENOSYS) continue.
>
> This might cause less churn than 1) but still requires a few changes
> in arch-neutral libxl code.
>
>
> 3) Force XEN_DOMCTL_ioport_permission to return zero on ARM
>
> Force the hypercall to return success even if it did nothing.
> Currently it returns -ENOSYS.
>
> This is possible but it wasn't chosen for the implementation as we
> felt that the hypercall should reflect what was actually done
> (nothing) and it should be userspace to handle the error. I guess
> this could be argued either way.
>
>
> 4) Force xc_domain_ioport_permission to return zero on ARM
>
> Force xc_domain_ioport_permission to return success even if the
> hypercall would return -ENOSYS. This way there are no changes to
> libxl.
>
> This is what the patch currently implements by using #ifdef in
> xc_domain_ioport_permission. It could also have achieved the same
> goal by making the implementation of xc_domain_ioport_permission
> arch-specific, and in the ARM implementation returning 0.
>
>
> All options above achieve the goal of a successful domain creation with
> PCI device assigned on ARM. You might be able to think of other options
> as well. I think noone here is really set on using one option over the
> other -- as long as xc_domain_ioport_permission failures don't turn into
> domain creation failures on ARM we are good.
>
I think having a libxl_arch_io_ports_supported helper could be the
cleaner way to do this. For x86 it will unconditionally return true,
while for Arm you could consider poking at
XEN_DOMCTL_ioport_permission and see if it returns ENOSYS or
otherwise.
I guess it's possible that in the future we allow IO ports access on
Arm guests using some kind of emulated mechanism if the need arises,
at which point the hypercall will be implemented.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-12 10:38 ` Michal Orzel
@ 2021-10-13 8:30 ` Roger Pau Monné
2021-10-13 9:36 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-13 8:30 UTC (permalink / raw)
To: Michal Orzel
Cc: Rahul Singh, xen-devel, bertrand.marquis, Andre.Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk
On Tue, Oct 12, 2021 at 12:38:35PM +0200, Michal Orzel wrote:
> Hi Roger,
>
> On 11.10.2021 11:27, Roger Pau Monné wrote:
> > On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
> >> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> >> Reject the use of this new flag for x86 as VPCI is not supported for
> >> DOMU guests for x86.
> >
> > I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
> > PVH dom0, like we do for any other CDF flags when Xen builds dom0.
> >
> > Things like PVH vs PV get translated into CDF flags by create_dom0,
> > and processed normally by the sanitise_domain_config logic, vPCI
> > should be handled that way.
> >
> > Do you think you could see about fixing this?
> >
> > Thanks, Roger.
> >
>
> I have one question about this fix.
> If I set XEN_DOMCTL_CDF_vpci for dom0 pvh in create_dom0, then in
> sanitise_domain_config or arch_sanitise_domain_config I have no
> knowledge on whether I am dom0 or not. I can check if I'm PVH but not if dom0.
> This would be needed to add a warning if this flag is set but we are not dom0 pvh.
>
> Any ideas?
I've just realized this is more wrong that I thought. vPCI is
signaled on x86 in xen_arch_domainconfig.emulation_flags, so
introducing a top level option for it without removing the arch
specific one is wrong, as then on x86 we have a duplicated option.
Then I'm also not sure whether we want to move it from
emulation_flags, it seems like the more natural place for it to live
on x86.
If we really want to make vPCI a CDF option we must deal with the
removal of XEN_X86_EMU_VPCI, or else you could introduce an arch
specific flag for vPCI on Arm.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-06 17:40 ` [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
` (2 preceding siblings ...)
2021-10-12 14:32 ` Julien Grall
@ 2021-10-13 8:45 ` Roger Pau Monné
2021-10-13 9:48 ` Bertrand Marquis
2021-10-13 13:00 ` Jan Beulich
3 siblings, 2 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-13 8:45 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Paul Durrant
On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the Xen to emulate the PCI devices config space.
>
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
>
> For Dom0less systems scan_pci_devices() would be used to discover the
> PCI device in XEN and VPCI handler will be added during XEN boots.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> Change in v5:
> - Add pci_cleanup_msi(pdev) in cleanup part.
> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> Change in v4:
> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
> Change in v3:
> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
> Change in v2:
> - Add new XEN_DOMCTL_CDF_vpci flag
> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
> - enable vpci support when pci-passthough option is enabled.
> ---
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 4 ++
> xen/arch/arm/domain_build.c | 3 +
> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
> xen/arch/arm/vpci.h | 36 ++++++++++++
> xen/drivers/passthrough/pci.c | 18 ++++++
> xen/include/asm-arm/domain.h | 7 ++-
> xen/include/asm-x86/pci.h | 2 -
> xen/include/public/arch-arm.h | 7 +++
> xen/include/xen/pci.h | 2 +
> 10 files changed, 179 insertions(+), 3 deletions(-)
> create mode 100644 xen/arch/arm/vpci.c
> create mode 100644 xen/arch/arm/vpci.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 44d7cc81fa..fb9c976ea2 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 36138c1b2e..fbb52f78f1 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -39,6 +39,7 @@
> #include <asm/vgic.h>
> #include <asm/vtimer.h>
>
> +#include "vpci.h"
> #include "vuart.h"
>
> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> goto fail;
>
> + if ( (rc = domain_vpci_init(d)) != 0 )
> + goto fail;
> +
> return 0;
>
> fail:
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index c5afbe2e05..f4c89bde8c 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
> if ( iommu_enabled )
> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>
> + if ( is_pci_passthrough_enabled() )
> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
> +
> dom0 = domain_create(0, &dom0_cfg, true);
> if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
> panic("Error creating domain 0\n");
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..76c12b9281
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,102 @@
> +/*
> + * xen/arch/arm/vpci.c
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +
> +#include <asm/mmio.h>
> +
> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
> +
> +/* Do some sanity checks. */
> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> +{
> + /* Check access size. */
> + if ( len > 8 )
> + return false;
> +
> + /* Check that access is size aligned. */
> + if ( (reg & (len - 1)) )
> + return false;
> +
> + return true;
> +}
> +
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> + register_t *r, void *p)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + unsigned long data = ~0UL;
> + unsigned int size = 1U << info->dabt.size;
> +
> + sbdf.sbdf = MMCFG_BDF(info->gpa);
> + reg = REGISTER_OFFSET(info->gpa);
> +
> + if ( !vpci_mmio_access_allowed(reg, size) )
> + return 0;
> +
> + data = vpci_read(sbdf, reg, min(4u, size));
> + if ( size == 8 )
> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> +
> + *r = data;
> +
> + return 1;
> +}
> +
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> + register_t r, void *p)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + unsigned long data = r;
> + unsigned int size = 1U << info->dabt.size;
> +
> + sbdf.sbdf = MMCFG_BDF(info->gpa);
> + reg = REGISTER_OFFSET(info->gpa);
> +
> + if ( !vpci_mmio_access_allowed(reg, size) )
> + return 0;
> +
> + vpci_write(sbdf, reg, min(4u, size), data);
> + if ( size == 8 )
> + vpci_write(sbdf, reg + 4, 4, data >> 32);
I think those two helpers (and vpci_mmio_access_allowed) are very
similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
the point where I would consider moving the shared code to vpci.c as
vpci_ecam_{read,write} and call them from the arch specific trap
handlers.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 8:30 ` Roger Pau Monné
@ 2021-10-13 9:36 ` Bertrand Marquis
2021-10-13 10:56 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-13 9:36 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Michal Orzel, Rahul Singh, xen-devel, Andre Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk
Hi Roger,
> On 13 Oct 2021, at 09:30, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Tue, Oct 12, 2021 at 12:38:35PM +0200, Michal Orzel wrote:
>> Hi Roger,
>>
>> On 11.10.2021 11:27, Roger Pau Monné wrote:
>>> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
>>>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>>>> Reject the use of this new flag for x86 as VPCI is not supported for
>>>> DOMU guests for x86.
>>>
>>> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
>>> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
>>>
>>> Things like PVH vs PV get translated into CDF flags by create_dom0,
>>> and processed normally by the sanitise_domain_config logic, vPCI
>>> should be handled that way.
>>>
>>> Do you think you could see about fixing this?
>>>
>>> Thanks, Roger.
>>>
>>
>> I have one question about this fix.
>> If I set XEN_DOMCTL_CDF_vpci for dom0 pvh in create_dom0, then in
>> sanitise_domain_config or arch_sanitise_domain_config I have no
>> knowledge on whether I am dom0 or not. I can check if I'm PVH but not if dom0.
>> This would be needed to add a warning if this flag is set but we are not dom0 pvh.
>>
>> Any ideas?
>
> I've just realized this is more wrong that I thought. vPCI is
> signaled on x86 in xen_arch_domainconfig.emulation_flags, so
> introducing a top level option for it without removing the arch
> specific one is wrong, as then on x86 we have a duplicated option.
>
> Then I'm also not sure whether we want to move it from
> emulation_flags, it seems like the more natural place for it to live
> on x86.
>
> If we really want to make vPCI a CDF option we must deal with the
> removal of XEN_X86_EMU_VPCI, or else you could introduce an arch
> specific flag for vPCI on Arm.
First issue that we have here is that there is no emulation_flags right now on arm.
So I think there are 2 solutions:
- introduce PHYSCAP. The problem here is that it is not a physical capacity and
I think that will hit us in the future for example if we want to use vpci for VIRTIO
even if there is not physical PCI on the platform.
- introduce an emulation flag on arm. The problem here is that there is no emulation
flag right now on arm so adding this feature will require a change of interface in
arch-arm.h and in xen domctl interface. But if we introduce one on Arm, it would allow
the tools to check if CDF_vpci can be set or not which would solve current issues.
I think the second solution is the right one but it cannot be done so near from the
feature freeze.
The CDF flag as introduced right now is not creating any issue and will be used once
the emulation flag will be introduce. We will be able at this stage to set this properly
also on x86 (for dom0 pvh).
Moreover keeping it will allow to continue to merge the remaining part of the PCI
passthrough which are otherwise not possible to be done as they are dependent on this flag.
Can we agree on keep the DOMCTL_CDF_vpci flag and introduce the emulation
flag on Arm after 4.16 release ?
Cheers
Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 8:45 ` Roger Pau Monné
@ 2021-10-13 9:48 ` Bertrand Marquis
2021-10-13 10:33 ` Roger Pau Monné
2021-10-13 13:00 ` Jan Beulich
1 sibling, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-13 9:48 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Rahul Singh, xen-devel, Andre Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Paul Durrant
Hi Roger,
> On 13 Oct 2021, at 09:45, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
>>
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>>
>> For Dom0less systems scan_pci_devices() would be used to discover the
>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> ---
>> Change in v5:
>> - Add pci_cleanup_msi(pdev) in cleanup part.
>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> Change in v4:
>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>> Change in v3:
>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>> Change in v2:
>> - Add new XEN_DOMCTL_CDF_vpci flag
>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>> - enable vpci support when pci-passthough option is enabled.
>> ---
>> ---
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/domain.c | 4 ++
>> xen/arch/arm/domain_build.c | 3 +
>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>> xen/arch/arm/vpci.h | 36 ++++++++++++
>> xen/drivers/passthrough/pci.c | 18 ++++++
>> xen/include/asm-arm/domain.h | 7 ++-
>> xen/include/asm-x86/pci.h | 2 -
>> xen/include/public/arch-arm.h | 7 +++
>> xen/include/xen/pci.h | 2 +
>> 10 files changed, 179 insertions(+), 3 deletions(-)
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 44d7cc81fa..fb9c976ea2 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index 36138c1b2e..fbb52f78f1 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -39,6 +39,7 @@
>> #include <asm/vgic.h>
>> #include <asm/vtimer.h>
>>
>> +#include "vpci.h"
>> #include "vuart.h"
>>
>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>> goto fail;
>>
>> + if ( (rc = domain_vpci_init(d)) != 0 )
>> + goto fail;
>> +
>> return 0;
>>
>> fail:
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index c5afbe2e05..f4c89bde8c 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>> if ( iommu_enabled )
>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>
>> + if ( is_pci_passthrough_enabled() )
>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>> +
>> dom0 = domain_create(0, &dom0_cfg, true);
>> if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
>> panic("Error creating domain 0\n");
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..76c12b9281
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,102 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +
>> +#include <asm/mmio.h>
>> +
>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>> +
>> +/* Do some sanity checks. */
>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> + /* Check access size. */
>> + if ( len > 8 )
>> + return false;
>> +
>> + /* Check that access is size aligned. */
>> + if ( (reg & (len - 1)) )
>> + return false;
>> +
>> + return true;
>> +}
>> +
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> + register_t *r, void *p)
>> +{
>> + unsigned int reg;
>> + pci_sbdf_t sbdf;
>> + unsigned long data = ~0UL;
>> + unsigned int size = 1U << info->dabt.size;
>> +
>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>> + reg = REGISTER_OFFSET(info->gpa);
>> +
>> + if ( !vpci_mmio_access_allowed(reg, size) )
>> + return 0;
>> +
>> + data = vpci_read(sbdf, reg, min(4u, size));
>> + if ( size == 8 )
>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>> +
>> + *r = data;
>> +
>> + return 1;
>> +}
>> +
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> + register_t r, void *p)
>> +{
>> + unsigned int reg;
>> + pci_sbdf_t sbdf;
>> + unsigned long data = r;
>> + unsigned int size = 1U << info->dabt.size;
>> +
>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>> + reg = REGISTER_OFFSET(info->gpa);
>> +
>> + if ( !vpci_mmio_access_allowed(reg, size) )
>> + return 0;
>> +
>> + vpci_write(sbdf, reg, min(4u, size), data);
>> + if ( size == 8 )
>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>
> I think those two helpers (and vpci_mmio_access_allowed) are very
> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
> the point where I would consider moving the shared code to vpci.c as
> vpci_ecam_{read,write} and call them from the arch specific trap
> handlers.
Would it be ok to have ecam specific code in the vpci common code ?
This is an optimisation and we could do that later on as this is an other
change to be done and tested which will start to make things very
challenging with the friday deadline.
Cheers
Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 6:10 ` Jan Beulich
@ 2021-10-13 10:02 ` Bertrand Marquis
2021-10-13 12:21 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-13 10:02 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, Rahul Singh, Andre Przywara, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
Hi Jan,
> On 13 Oct 2021, at 07:10, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 12.10.2021 23:37, Stefano Stabellini wrote:
>> On Tue, 12 Oct 2021, Jan Beulich wrote:
>>> On 11.10.2021 20:18, Stefano Stabellini wrote:
>>>> On Mon, 11 Oct 2021, Jan Beulich wrote:
>>>>> On 11.10.2021 15:34, Bertrand Marquis wrote:
>>>>>>> On 11 Oct 2021, at 14:09, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>>> On 11.10.2021 14:41, Bertrand Marquis wrote:
>>>>>>>> But digging deeper into this, I would have 2 questions:
>>>>>>>>
>>>>>>>> - msi_cleanup was done there after a request from Stefano, but is not
>>>>>>>> done in case or iommu error, is there an issue to solve here ?
>>>>>>>
>>>>>>> Maybe, but I'm not sure. This very much depends on what a domain
>>>>>>> could in principle do with a partly set-up device. Plus let's
>>>>>>> not forget that we're talking of only Dom0 here (for now at least,
>>>>>>> i.e. not considering the dom0less case).
>>>>>>>
>>>>>>> But I'd also like to further defer to Stefano.
>>>>>>
>>>>>> Ok, I must admit I do not really see at that stage why doing an MSI cleanup
>>>>>> could be needed so I will wait for Stefano to know if I need to keep this when
>>>>>> moving the block up (at the end it is theoretical right now as this is empty).
>>>>
>>>> I know that MSIs are not supported yet on ARM (pci_cleanup_msi does
>>>> nothing). But I wanted to make sure that the pci_cleanup_msi() calls are
>>>> present anywhere necessary, especially on the error paths. So that once
>>>> we add MSI support, we don't need to search through the code to find all
>>>> the error paths missing a pci_cleanup_msi() call.
>>>>
>>>> To answer your first question: you are right, we are also missing a
>>>> pci_cleanup_msi() call in the case of IOMMU error. So it might be better
>>>> to move the call to pci_cleanup_msi() under the "out" label so that we
>>>> can do it once for both cases.
>>>>
>>>> To answer your second point about whether it is necessary at all: if
>>>> MSIs and MSI-Xs cannot be already setup at this point at all (not even
>>>> the enable bit), then we don't need any call to pci_cleanup_msi() in
>>>> pci_add_device.
>>>
>>> Well, at the very least MSI can't be set up ahead of the traps getting
>>> put in place. Whether partial success of putting traps in place may
>>> allow a cunning guest to set up MSI may depend on further aspects.
>>
>> Good point about MSIs not being setup before the traps. We should remove
>> the call to pci_cleanup_msi() in the error path then.
>
> Your reply makes me fear you didn't pay enough attention to the "partial"
> in my earlier reply. The traps for the various registers can't all be set
> up atomically, so there may be a transient period where enough traps are
> already in place for a cunning guest to arrange for setup. Unless, as
> said, there are further setup steps needed before a guest could succeed
> in doing so.
>
> But even if partial trap setup alone was sufficient, I think the cleaning
> up of MSI then might still better go on the error path there than on that
> of pci_add_device().
I think I should put the msi_cleanup in the exit path if pdev is not null but
we got a non null ret (in an else if ( pdev ) ).
This would cover all exit paths, especially as I will move the add_handler
before the iommu init.
Would that be ok for everyone ?
Cheers
Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 9:48 ` Bertrand Marquis
@ 2021-10-13 10:33 ` Roger Pau Monné
0 siblings, 0 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-13 10:33 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Rahul Singh, xen-devel, Andre Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Jan Beulich, Wei Liu, Paul Durrant
On Wed, Oct 13, 2021 at 09:48:42AM +0000, Bertrand Marquis wrote:
> Hi Roger,
>
> > On 13 Oct 2021, at 09:45, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
> >> The existing VPCI support available for X86 is adapted for Arm.
> >> When the device is added to XEN via the hyper call
> >> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> >> access is added to the Xen to emulate the PCI devices config space.
> >>
> >> A MMIO trap handler for the PCI ECAM space is registered in XEN
> >> so that when guest is trying to access the PCI config space,XEN
> >> will trap the access and emulate read/write using the VPCI and
> >> not the real PCI hardware.
> >>
> >> For Dom0less systems scan_pci_devices() would be used to discover the
> >> PCI device in XEN and VPCI handler will be added during XEN boots.
> >>
> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> >> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> >> ---
> >> Change in v5:
> >> - Add pci_cleanup_msi(pdev) in cleanup part.
> >> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> >> Change in v4:
> >> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
> >> Change in v3:
> >> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
> >> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
> >> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
> >> Change in v2:
> >> - Add new XEN_DOMCTL_CDF_vpci flag
> >> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
> >> - enable vpci support when pci-passthough option is enabled.
> >> ---
> >> ---
> >> xen/arch/arm/Makefile | 1 +
> >> xen/arch/arm/domain.c | 4 ++
> >> xen/arch/arm/domain_build.c | 3 +
> >> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
> >> xen/arch/arm/vpci.h | 36 ++++++++++++
> >> xen/drivers/passthrough/pci.c | 18 ++++++
> >> xen/include/asm-arm/domain.h | 7 ++-
> >> xen/include/asm-x86/pci.h | 2 -
> >> xen/include/public/arch-arm.h | 7 +++
> >> xen/include/xen/pci.h | 2 +
> >> 10 files changed, 179 insertions(+), 3 deletions(-)
> >> create mode 100644 xen/arch/arm/vpci.c
> >> create mode 100644 xen/arch/arm/vpci.h
> >>
> >> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> >> index 44d7cc81fa..fb9c976ea2 100644
> >> --- a/xen/arch/arm/Makefile
> >> +++ b/xen/arch/arm/Makefile
> >> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> >> obj-y += platforms/
> >> endif
> >> obj-$(CONFIG_TEE) += tee/
> >> +obj-$(CONFIG_HAS_VPCI) += vpci.o
> >>
> >> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> >> obj-y += bootfdt.init.o
> >> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> >> index 36138c1b2e..fbb52f78f1 100644
> >> --- a/xen/arch/arm/domain.c
> >> +++ b/xen/arch/arm/domain.c
> >> @@ -39,6 +39,7 @@
> >> #include <asm/vgic.h>
> >> #include <asm/vtimer.h>
> >>
> >> +#include "vpci.h"
> >> #include "vuart.h"
> >>
> >> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> >> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
> >> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> >> goto fail;
> >>
> >> + if ( (rc = domain_vpci_init(d)) != 0 )
> >> + goto fail;
> >> +
> >> return 0;
> >>
> >> fail:
> >> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> >> index c5afbe2e05..f4c89bde8c 100644
> >> --- a/xen/arch/arm/domain_build.c
> >> +++ b/xen/arch/arm/domain_build.c
> >> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
> >> if ( iommu_enabled )
> >> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
> >>
> >> + if ( is_pci_passthrough_enabled() )
> >> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
> >> +
> >> dom0 = domain_create(0, &dom0_cfg, true);
> >> if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
> >> panic("Error creating domain 0\n");
> >> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> >> new file mode 100644
> >> index 0000000000..76c12b9281
> >> --- /dev/null
> >> +++ b/xen/arch/arm/vpci.c
> >> @@ -0,0 +1,102 @@
> >> +/*
> >> + * xen/arch/arm/vpci.c
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License as published by
> >> + * the Free Software Foundation; either version 2 of the License, or
> >> + * (at your option) any later version.
> >> + *
> >> + * This program is distributed in the hope that it will be useful,
> >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >> + * GNU General Public License for more details.
> >> + */
> >> +#include <xen/sched.h>
> >> +
> >> +#include <asm/mmio.h>
> >> +
> >> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
> >> +
> >> +/* Do some sanity checks. */
> >> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> >> +{
> >> + /* Check access size. */
> >> + if ( len > 8 )
> >> + return false;
> >> +
> >> + /* Check that access is size aligned. */
> >> + if ( (reg & (len - 1)) )
> >> + return false;
> >> +
> >> + return true;
> >> +}
> >> +
> >> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> >> + register_t *r, void *p)
> >> +{
> >> + unsigned int reg;
> >> + pci_sbdf_t sbdf;
> >> + unsigned long data = ~0UL;
> >> + unsigned int size = 1U << info->dabt.size;
> >> +
> >> + sbdf.sbdf = MMCFG_BDF(info->gpa);
> >> + reg = REGISTER_OFFSET(info->gpa);
> >> +
> >> + if ( !vpci_mmio_access_allowed(reg, size) )
> >> + return 0;
> >> +
> >> + data = vpci_read(sbdf, reg, min(4u, size));
> >> + if ( size == 8 )
> >> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> >> +
> >> + *r = data;
> >> +
> >> + return 1;
> >> +}
> >> +
> >> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> >> + register_t r, void *p)
> >> +{
> >> + unsigned int reg;
> >> + pci_sbdf_t sbdf;
> >> + unsigned long data = r;
> >> + unsigned int size = 1U << info->dabt.size;
> >> +
> >> + sbdf.sbdf = MMCFG_BDF(info->gpa);
> >> + reg = REGISTER_OFFSET(info->gpa);
> >> +
> >> + if ( !vpci_mmio_access_allowed(reg, size) )
> >> + return 0;
> >> +
> >> + vpci_write(sbdf, reg, min(4u, size), data);
> >> + if ( size == 8 )
> >> + vpci_write(sbdf, reg + 4, 4, data >> 32);
> >
> > I think those two helpers (and vpci_mmio_access_allowed) are very
> > similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
> > the point where I would consider moving the shared code to vpci.c as
> > vpci_ecam_{read,write} and call them from the arch specific trap
> > handlers.
>
> Would it be ok to have ecam specific code in the vpci common code ?
I think so, ECAM is part of the PCI specification and architecture
agnostic, so fine to place in common code.
> This is an optimisation and we could do that later on as this is an other
> change to be done and tested which will start to make things very
> challenging with the friday deadline.
I guess this could be done, by adding a TODO note that the handlers
should be unified with the x86 ones and placed in common code.
I however get the feeling that such work would ultimately be ignored,
as there's always more pressing stuff to be done, and people tend to
forget about those cleanups once things get committed.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 9:36 ` Bertrand Marquis
@ 2021-10-13 10:56 ` Roger Pau Monné
2021-10-13 12:11 ` Bertrand Marquis
2021-10-13 20:53 ` Stefano Stabellini
0 siblings, 2 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-13 10:56 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Michal Orzel, Rahul Singh, xen-devel, Andre Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk
On Wed, Oct 13, 2021 at 09:36:01AM +0000, Bertrand Marquis wrote:
> Hi Roger,
>
> > On 13 Oct 2021, at 09:30, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Tue, Oct 12, 2021 at 12:38:35PM +0200, Michal Orzel wrote:
> >> Hi Roger,
> >>
> >> On 11.10.2021 11:27, Roger Pau Monné wrote:
> >>> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
> >>>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> >>>> Reject the use of this new flag for x86 as VPCI is not supported for
> >>>> DOMU guests for x86.
> >>>
> >>> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
> >>> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
> >>>
> >>> Things like PVH vs PV get translated into CDF flags by create_dom0,
> >>> and processed normally by the sanitise_domain_config logic, vPCI
> >>> should be handled that way.
> >>>
> >>> Do you think you could see about fixing this?
> >>>
> >>> Thanks, Roger.
> >>>
> >>
> >> I have one question about this fix.
> >> If I set XEN_DOMCTL_CDF_vpci for dom0 pvh in create_dom0, then in
> >> sanitise_domain_config or arch_sanitise_domain_config I have no
> >> knowledge on whether I am dom0 or not. I can check if I'm PVH but not if dom0.
> >> This would be needed to add a warning if this flag is set but we are not dom0 pvh.
> >>
> >> Any ideas?
> >
> > I've just realized this is more wrong that I thought. vPCI is
> > signaled on x86 in xen_arch_domainconfig.emulation_flags, so
> > introducing a top level option for it without removing the arch
> > specific one is wrong, as then on x86 we have a duplicated option.
> >
> > Then I'm also not sure whether we want to move it from
> > emulation_flags, it seems like the more natural place for it to live
> > on x86.
> >
> > If we really want to make vPCI a CDF option we must deal with the
> > removal of XEN_X86_EMU_VPCI, or else you could introduce an arch
> > specific flag for vPCI on Arm.
>
> First issue that we have here is that there is no emulation_flags right now on arm.
You don't explicitly need an emulation_flags field, you could add a
uint8_t vpci or some such to xen_arch_domainconfig for Arm if you
don't think there's a need to select more emulation. That's up to Arm
folks.
> So I think there are 2 solutions:
>
> - introduce PHYSCAP. The problem here is that it is not a physical capacity and
> I think that will hit us in the future for example if we want to use vpci for VIRTIO
> even if there is not physical PCI on the platform.
Hm, PHYSCAP is a bit weird, for example Xen signals shadow paging
support using PHYSCAP which doesn't depend on any hardware feature.
I think you could signal vPCI regardless of whether the underlying
platform has PCI devices or not, as vPCI is purely a software
component.
Regarding VirtIO, won't it be implemented using ioreqs in user-space,
and thus won't depend on vPCI?
> - introduce an emulation flag on arm. The problem here is that there is no emulation
> flag right now on arm so adding this feature will require a change of interface in
> arch-arm.h and in xen domctl interface. But if we introduce one on Arm, it would allow
> the tools to check if CDF_vpci can be set or not which would solve current issues.
I'm not sure I follow. If we go the emulation flags route this will all
be set in xen_arch_domainconfig, and CDF_vpci will completely go away.
> I think the second solution is the right one but it cannot be done so near from the
> feature freeze.
>
> The CDF flag as introduced right now is not creating any issue and will be used once
> the emulation flag will be introduce. We will be able at this stage to set this properly
> also on x86 (for dom0 pvh).
> Moreover keeping it will allow to continue to merge the remaining part of the PCI
> passthrough which are otherwise not possible to be done as they are dependent on this flag.
>
> Can we agree on keep the DOMCTL_CDF_vpci flag and introduce the emulation
> flag on Arm after 4.16 release ?
If vPCI for Arm on 4.16 is not going to be functional, why so much
pressure in pushing those patches so fast? I understand the need to
remove stuff from the queue, but I don't think it's worth the cost of
introducing a broken interface deliberately on a release.
I think we need to at least settle on whether we want to keep
CDF_vpci or use an arch specific signal mechanism in order to decide
what to do regarding the release.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-13 8:07 ` Roger Pau Monné
@ 2021-10-13 11:52 ` Ian Jackson
0 siblings, 0 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-13 11:52 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Stefano Stabellini, Bertrand Marquis, Jan Beulich,
Oleksandr Andrushchenko, Rahul Singh, xen-devel, Andre Przywara,
Wei Liu, Juergen Gross
Roger Pau Monné writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
> On Tue, Oct 12, 2021 at 01:42:22PM -0700, Stefano Stabellini wrote:
> > I don't think it is about performance. From a performance point of view,
> > we could make as many (unneeded) hypercalls as required. It is mostly
> > about minimizing unwanted changes to common libxl code. Let me explain.
Thanks. This summary is helpful And, pleasingly, it matches what I
had thought I had gleaned from the thread.
> > All options above achieve the goal of a successful domain creation with
> > PCI device assigned on ARM. You might be able to think of other options
> > as well. I think noone here is really set on using one option over the
> > other -- as long as xc_domain_ioport_permission failures don't turn into
> > domain creation failures on ARM we are good.
>
> I think having a libxl_arch_io_ports_supported helper could be the
> cleaner way to do this. For x86 it will unconditionally return true,
> while for Arm you could consider poking at
> XEN_DOMCTL_ioport_permission and see if it returns ENOSYS or
> otherwise.
> I guess it's possible that in the future we allow IO ports access on
> Arm guests using some kind of emulated mechanism if the need arises,
> at which point the hypercall will be implemented.
I agree with Roger.
So I think I would like to see a version of this patch which
* Introduces libxl_arch_io_ports_supported. (I am fine with it just
returning false, unconditionally on Arm, ie in libxl_arm.c.)
* Has a commit message explaining what is actually going on.
Cutting and pasting liberally from your email seems like it would
be a very good starting point. Even discussion of rejected
alternatives is fine, if it seems like it fits. I'm quite
unlikely to object to a commit message on grounds that it's too
long.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
2021-10-13 8:02 ` Roger Pau Monné
@ 2021-10-13 12:02 ` Ian Jackson
0 siblings, 0 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-13 12:02 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Bertrand Marquis, Jan Beulich, Oleksandr Andrushchenko,
Rahul Singh, xen-devel, Andre Przywara, Wei Liu, Juergen Gross,
Stefano Stabellini
Roger Pau Monné writes ("Re: [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM."):
> IMO it would be good to modify the commit message so it covers the
> fact that the emulated host bridge on Arm does not advertise IO port
> support, so the guest is capable of realizing IO BARs are not
> supported.
Yes.
> Otherwise it seems like the toolstack is ignoring a failure which
> could cause a device to malfunction when passed though (which is still
> the case, but the guest will be able to notice).
Quite.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 10:56 ` Roger Pau Monné
@ 2021-10-13 12:11 ` Bertrand Marquis
2021-10-13 12:57 ` Jan Beulich
2021-10-13 14:28 ` Roger Pau Monné
2021-10-13 20:53 ` Stefano Stabellini
1 sibling, 2 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-13 12:11 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Michal Orzel, Rahul Singh, xen-devel, Andre Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk
Hi Roger,
> On 13 Oct 2021, at 11:56, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Wed, Oct 13, 2021 at 09:36:01AM +0000, Bertrand Marquis wrote:
>> Hi Roger,
>>
>>> On 13 Oct 2021, at 09:30, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Tue, Oct 12, 2021 at 12:38:35PM +0200, Michal Orzel wrote:
>>>> Hi Roger,
>>>>
>>>> On 11.10.2021 11:27, Roger Pau Monné wrote:
>>>>> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
>>>>>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
>>>>>> Reject the use of this new flag for x86 as VPCI is not supported for
>>>>>> DOMU guests for x86.
>>>>>
>>>>> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
>>>>> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
>>>>>
>>>>> Things like PVH vs PV get translated into CDF flags by create_dom0,
>>>>> and processed normally by the sanitise_domain_config logic, vPCI
>>>>> should be handled that way.
>>>>>
>>>>> Do you think you could see about fixing this?
>>>>>
>>>>> Thanks, Roger.
>>>>>
>>>>
>>>> I have one question about this fix.
>>>> If I set XEN_DOMCTL_CDF_vpci for dom0 pvh in create_dom0, then in
>>>> sanitise_domain_config or arch_sanitise_domain_config I have no
>>>> knowledge on whether I am dom0 or not. I can check if I'm PVH but not if dom0.
>>>> This would be needed to add a warning if this flag is set but we are not dom0 pvh.
>>>>
>>>> Any ideas?
>>>
>>> I've just realized this is more wrong that I thought. vPCI is
>>> signaled on x86 in xen_arch_domainconfig.emulation_flags, so
>>> introducing a top level option for it without removing the arch
>>> specific one is wrong, as then on x86 we have a duplicated option.
>>>
>>> Then I'm also not sure whether we want to move it from
>>> emulation_flags, it seems like the more natural place for it to live
>>> on x86.
>>>
>>> If we really want to make vPCI a CDF option we must deal with the
>>> removal of XEN_X86_EMU_VPCI, or else you could introduce an arch
>>> specific flag for vPCI on Arm.
>>
>> First issue that we have here is that there is no emulation_flags right now on arm.
>
> You don't explicitly need an emulation_flags field, you could add a
> uint8_t vpci or some such to xen_arch_domainconfig for Arm if you
> don't think there's a need to select more emulation. That's up to Arm
> folks.
One way or an other it is still changing the interface.
>
>> So I think there are 2 solutions:
>>
>> - introduce PHYSCAP. The problem here is that it is not a physical capacity and
>> I think that will hit us in the future for example if we want to use vpci for VIRTIO
>> even if there is not physical PCI on the platform.
>
> Hm, PHYSCAP is a bit weird, for example Xen signals shadow paging
> support using PHYSCAP which doesn't depend on any hardware feature.
>
> I think you could signal vPCI regardless of whether the underlying
> platform has PCI devices or not, as vPCI is purely a software
> component.
But in the x86 case this is something not supported in all cases and actually only by dom0 pvh.
So having something like that defined as a PHYSCAP is a bit weird.
>
> Regarding VirtIO, won't it be implemented using ioreqs in user-space,
> and thus won't depend on vPCI?
Yes that’s a solution but you can list them through a PCI interface to let a guest discover them.
I am not saying that we will do that but that was an example of case where we could use vPCI without hardware PCI present.
>
>> - introduce an emulation flag on arm. The problem here is that there is no emulation
>> flag right now on arm so adding this feature will require a change of interface in
>> arch-arm.h and in xen domctl interface. But if we introduce one on Arm, it would allow
>> the tools to check if CDF_vpci can be set or not which would solve current issues.
>
> I'm not sure I follow. If we go the emulation flags route this will all
> be set in xen_arch_domainconfig, and CDF_vpci will completely go away.
This is a mistake on my side. You are right using emulation flags will require to remove CDF_vpci.
>
>> I think the second solution is the right one but it cannot be done so near from the
>> feature freeze.
>>
>> The CDF flag as introduced right now is not creating any issue and will be used once
>> the emulation flag will be introduce. We will be able at this stage to set this properly
>> also on x86 (for dom0 pvh).
>> Moreover keeping it will allow to continue to merge the remaining part of the PCI
>> passthrough which are otherwise not possible to be done as they are dependent on this flag.
>>
>> Can we agree on keep the DOMCTL_CDF_vpci flag and introduce the emulation
>> flag on Arm after 4.16 release ?
>
> If vPCI for Arm on 4.16 is not going to be functional, why so much
> pressure in pushing those patches so fast? I understand the need to
> remove stuff from the queue, but I don't think it's worth the cost of
> introducing a broken interface deliberately on a release.
We did not push that fast, those have been on the mailing list for a while (this is the 5th version of the serie).
PCI passthrough is a big change requiring lots of patches and we decided to push it piece by piece to make
the review easier.
>
> I think we need to at least settle on whether we want to keep
> CDF_vpci or use an arch specific signal mechanism in order to decide
> what to do regarding the release.
Agree and I have no definitive idea on this so but switching will require to revert the patch adding CDF_vpci
and change the subsequent patches.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 10:02 ` Bertrand Marquis
@ 2021-10-13 12:21 ` Jan Beulich
0 siblings, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-13 12:21 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Stefano Stabellini, Rahul Singh, Andre Przywara, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Wei Liu, Paul Durrant, Roger Pau Monné,
xen-devel
On 13.10.2021 12:02, Bertrand Marquis wrote:
>> On 13 Oct 2021, at 07:10, Jan Beulich <jbeulich@suse.com> wrote:
>> On 12.10.2021 23:37, Stefano Stabellini wrote:
>>> Good point about MSIs not being setup before the traps. We should remove
>>> the call to pci_cleanup_msi() in the error path then.
>>
>> Your reply makes me fear you didn't pay enough attention to the "partial"
>> in my earlier reply. The traps for the various registers can't all be set
>> up atomically, so there may be a transient period where enough traps are
>> already in place for a cunning guest to arrange for setup. Unless, as
>> said, there are further setup steps needed before a guest could succeed
>> in doing so.
>>
>> But even if partial trap setup alone was sufficient, I think the cleaning
>> up of MSI then might still better go on the error path there than on that
>> of pci_add_device().
>
> I think I should put the msi_cleanup in the exit path if pdev is not null but
> we got a non null ret (in an else if ( pdev ) ).
> This would cover all exit paths, especially as I will move the add_handler
> before the iommu init.
>
> Would that be ok for everyone ?
Sounds reasonable at the first glance.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 12:11 ` Bertrand Marquis
@ 2021-10-13 12:57 ` Jan Beulich
2021-10-13 20:41 ` Stefano Stabellini
2021-10-13 14:28 ` Roger Pau Monné
1 sibling, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-13 12:57 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Michal Orzel, Rahul Singh, xen-devel, Andre Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Julien Grall, Stefano Stabellini,
Volodymyr Babchuk, Roger Pau Monné
On 13.10.2021 14:11, Bertrand Marquis wrote:
>> On 13 Oct 2021, at 11:56, Roger Pau Monné <roger.pau@citrix.com> wrote:
>> If vPCI for Arm on 4.16 is not going to be functional, why so much
>> pressure in pushing those patches so fast? I understand the need to
>> remove stuff from the queue, but I don't think it's worth the cost of
>> introducing a broken interface deliberately on a release.
+1
> We did not push that fast, those have been on the mailing list for a while (this is the 5th version of the serie).
> PCI passthrough is a big change requiring lots of patches and we decided to push it piece by piece to make
> the review easier.
5 versions for a series like this one was to be expected. Imo it has
been wrong in the past to rush in new features (even if experimental
ones) very close to the freeze, and it has mislead people to think
they can delay work until very (too?) late a point in time.
In fact I'm getting somewhat angry seeing how much effort is put into
getting this work in (including myself doing reviews there), when at
the same time far older series of mine aren't given any chance to
make any progress. Not your fault, sure, but an observation.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 8:45 ` Roger Pau Monné
2021-10-13 9:48 ` Bertrand Marquis
@ 2021-10-13 13:00 ` Jan Beulich
2021-10-13 14:51 ` Oleksandr Andrushchenko
1 sibling, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-13 13:00 UTC (permalink / raw)
To: Roger Pau Monné, Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Wei Liu, Paul Durrant
On 13.10.2021 10:45, Roger Pau Monné wrote:
> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
>>
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>>
>> For Dom0less systems scan_pci_devices() would be used to discover the
>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> ---
>> Change in v5:
>> - Add pci_cleanup_msi(pdev) in cleanup part.
>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> Change in v4:
>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>> Change in v3:
>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>> Change in v2:
>> - Add new XEN_DOMCTL_CDF_vpci flag
>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>> - enable vpci support when pci-passthough option is enabled.
>> ---
>> ---
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/domain.c | 4 ++
>> xen/arch/arm/domain_build.c | 3 +
>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>> xen/arch/arm/vpci.h | 36 ++++++++++++
>> xen/drivers/passthrough/pci.c | 18 ++++++
>> xen/include/asm-arm/domain.h | 7 ++-
>> xen/include/asm-x86/pci.h | 2 -
>> xen/include/public/arch-arm.h | 7 +++
>> xen/include/xen/pci.h | 2 +
>> 10 files changed, 179 insertions(+), 3 deletions(-)
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 44d7cc81fa..fb9c976ea2 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index 36138c1b2e..fbb52f78f1 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -39,6 +39,7 @@
>> #include <asm/vgic.h>
>> #include <asm/vtimer.h>
>>
>> +#include "vpci.h"
>> #include "vuart.h"
>>
>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>> goto fail;
>>
>> + if ( (rc = domain_vpci_init(d)) != 0 )
>> + goto fail;
>> +
>> return 0;
>>
>> fail:
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index c5afbe2e05..f4c89bde8c 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>> if ( iommu_enabled )
>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>
>> + if ( is_pci_passthrough_enabled() )
>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>> +
>> dom0 = domain_create(0, &dom0_cfg, true);
>> if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
>> panic("Error creating domain 0\n");
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..76c12b9281
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,102 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +
>> +#include <asm/mmio.h>
>> +
>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>> +
>> +/* Do some sanity checks. */
>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> + /* Check access size. */
>> + if ( len > 8 )
>> + return false;
>> +
>> + /* Check that access is size aligned. */
>> + if ( (reg & (len - 1)) )
>> + return false;
>> +
>> + return true;
>> +}
>> +
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> + register_t *r, void *p)
>> +{
>> + unsigned int reg;
>> + pci_sbdf_t sbdf;
>> + unsigned long data = ~0UL;
>> + unsigned int size = 1U << info->dabt.size;
>> +
>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>> + reg = REGISTER_OFFSET(info->gpa);
>> +
>> + if ( !vpci_mmio_access_allowed(reg, size) )
>> + return 0;
>> +
>> + data = vpci_read(sbdf, reg, min(4u, size));
>> + if ( size == 8 )
>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>> +
>> + *r = data;
>> +
>> + return 1;
>> +}
>> +
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> + register_t r, void *p)
>> +{
>> + unsigned int reg;
>> + pci_sbdf_t sbdf;
>> + unsigned long data = r;
>> + unsigned int size = 1U << info->dabt.size;
>> +
>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>> + reg = REGISTER_OFFSET(info->gpa);
>> +
>> + if ( !vpci_mmio_access_allowed(reg, size) )
>> + return 0;
>> +
>> + vpci_write(sbdf, reg, min(4u, size), data);
>> + if ( size == 8 )
>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>
> I think those two helpers (and vpci_mmio_access_allowed) are very
> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
> the point where I would consider moving the shared code to vpci.c as
> vpci_ecam_{read,write} and call them from the arch specific trap
> handlers.
Except that please can we stick to mcfg or mmcfg instead of ecam
in names, as that's how the thing has been named in Xen from its
introduction? I've just grep-ed the code base (case insensitively)
and found no mention of ECAM. There are only a few "became".
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 12:11 ` Bertrand Marquis
2021-10-13 12:57 ` Jan Beulich
@ 2021-10-13 14:28 ` Roger Pau Monné
1 sibling, 0 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-13 14:28 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Michal Orzel, Rahul Singh, xen-devel, Andre Przywara,
Christian Lindig, David Scott, Ian Jackson, Wei Liu,
Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk
On Wed, Oct 13, 2021 at 12:11:30PM +0000, Bertrand Marquis wrote:
> Hi Roger,
>
> > On 13 Oct 2021, at 11:56, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Wed, Oct 13, 2021 at 09:36:01AM +0000, Bertrand Marquis wrote:
> >> Hi Roger,
> >>
> >>> On 13 Oct 2021, at 09:30, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>
> >>> On Tue, Oct 12, 2021 at 12:38:35PM +0200, Michal Orzel wrote:
> >>>> Hi Roger,
> >>>>
> >>>> On 11.10.2021 11:27, Roger Pau Monné wrote:
> >>>>> On Wed, Oct 06, 2021 at 06:40:33PM +0100, Rahul Singh wrote:
> >>>>>> Introduce XEN_DOMCTL_CDF_vpci flag to enable VPCI support in XEN.
> >>>>>> Reject the use of this new flag for x86 as VPCI is not supported for
> >>>>>> DOMU guests for x86.
> >>>>>
> >>>>> I don't like this approach, XEN_DOMCTL_CDF_vpci should be set for x86
> >>>>> PVH dom0, like we do for any other CDF flags when Xen builds dom0.
> >>>>>
> >>>>> Things like PVH vs PV get translated into CDF flags by create_dom0,
> >>>>> and processed normally by the sanitise_domain_config logic, vPCI
> >>>>> should be handled that way.
> >>>>>
> >>>>> Do you think you could see about fixing this?
> >>>>>
> >>>>> Thanks, Roger.
> >>>>>
> >>>>
> >>>> I have one question about this fix.
> >>>> If I set XEN_DOMCTL_CDF_vpci for dom0 pvh in create_dom0, then in
> >>>> sanitise_domain_config or arch_sanitise_domain_config I have no
> >>>> knowledge on whether I am dom0 or not. I can check if I'm PVH but not if dom0.
> >>>> This would be needed to add a warning if this flag is set but we are not dom0 pvh.
> >>>>
> >>>> Any ideas?
> >>>
> >>> I've just realized this is more wrong that I thought. vPCI is
> >>> signaled on x86 in xen_arch_domainconfig.emulation_flags, so
> >>> introducing a top level option for it without removing the arch
> >>> specific one is wrong, as then on x86 we have a duplicated option.
> >>>
> >>> Then I'm also not sure whether we want to move it from
> >>> emulation_flags, it seems like the more natural place for it to live
> >>> on x86.
> >>>
> >>> If we really want to make vPCI a CDF option we must deal with the
> >>> removal of XEN_X86_EMU_VPCI, or else you could introduce an arch
> >>> specific flag for vPCI on Arm.
> >>
> >> First issue that we have here is that there is no emulation_flags right now on arm.
> >
> > You don't explicitly need an emulation_flags field, you could add a
> > uint8_t vpci or some such to xen_arch_domainconfig for Arm if you
> > don't think there's a need to select more emulation. That's up to Arm
> > folks.
>
> One way or an other it is still changing the interface.
Well, I don't want to sound rude, but you already changed the
interface first and very recently by introducing CDF_vpci. Complaining
that you don't want to change it anymore just after that seems
slightly weird.
> >
> >> So I think there are 2 solutions:
> >>
> >> - introduce PHYSCAP. The problem here is that it is not a physical capacity and
> >> I think that will hit us in the future for example if we want to use vpci for VIRTIO
> >> even if there is not physical PCI on the platform.
> >
> > Hm, PHYSCAP is a bit weird, for example Xen signals shadow paging
> > support using PHYSCAP which doesn't depend on any hardware feature.
> >
> > I think you could signal vPCI regardless of whether the underlying
> > platform has PCI devices or not, as vPCI is purely a software
> > component.
>
> But in the x86 case this is something not supported in all cases and actually only by dom0 pvh.
> So having something like that defined as a PHYSCAP is a bit weird.
Well, even if vPCI was fully supported on x86 for both domU and dom0
it would only apply to PVH and maybe HVM guests, but classic PV will
never get vPCI. It's kind of trying to create a PV guest with HAP
support.
One option would be to defer the introduction of the CDF_vpci flag
until the vpci work has been finished - then we could decide if the
current code can also be used by x86 so maybe it could be enabled for
both arches, and there would be no problem in setting the PHYSCAP.
> >> I think the second solution is the right one but it cannot be done so near from the
> >> feature freeze.
> >>
> >> The CDF flag as introduced right now is not creating any issue and will be used once
> >> the emulation flag will be introduce. We will be able at this stage to set this properly
> >> also on x86 (for dom0 pvh).
> >> Moreover keeping it will allow to continue to merge the remaining part of the PCI
> >> passthrough which are otherwise not possible to be done as they are dependent on this flag.
> >>
> >> Can we agree on keep the DOMCTL_CDF_vpci flag and introduce the emulation
> >> flag on Arm after 4.16 release ?
> >
> > If vPCI for Arm on 4.16 is not going to be functional, why so much
> > pressure in pushing those patches so fast? I understand the need to
> > remove stuff from the queue, but I don't think it's worth the cost of
> > introducing a broken interface deliberately on a release.
>
> We did not push that fast, those have been on the mailing list for a while (this is the 5th version of the serie).
> PCI passthrough is a big change requiring lots of patches and we decided to push it piece by piece to make
> the review easier.
First version of this patch I have on my inbox is from 23rd of
September, that's less than 3 weeks ago.
> >
> > I think we need to at least settle on whether we want to keep
> > CDF_vpci or use an arch specific signal mechanism in order to decide
> > what to do regarding the release.
>
> Agree and I have no definitive idea on this so but switching will require to revert the patch adding CDF_vpci
> and change the subsequent patches.
I think it's fair to expect changes to the interface when things are
reviewed, that's part of the review process, changes to subsequent
patches can be painful, but we have all been there and dealt with
them.
I'm not saying we must remove the CDF_vpci flag. I have to admit I was
fine using emulation_flags, but I could also be fine using CDF_vpci if
the x86 side is fixed and XEN_X86_EMU_VPCI is removed.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 13:00 ` Jan Beulich
@ 2021-10-13 14:51 ` Oleksandr Andrushchenko
2021-10-13 15:15 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Oleksandr Andrushchenko @ 2021-10-13 14:51 UTC (permalink / raw)
To: Jan Beulich, Roger Pau Monné, Rahul Singh
Cc: xen-devel, Bertrand Marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Ian Jackson, Wei Liu, Paul Durrant, Oleksandr Andrushchenko
On 13.10.21 16:00, Jan Beulich wrote:
> On 13.10.2021 10:45, Roger Pau Monné wrote:
>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>> The existing VPCI support available for X86 is adapted for Arm.
>>> When the device is added to XEN via the hyper call
>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>> access is added to the Xen to emulate the PCI devices config space.
>>>
>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>> so that when guest is trying to access the PCI config space,XEN
>>> will trap the access and emulate read/write using the VPCI and
>>> not the real PCI hardware.
>>>
>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>> ---
>>> Change in v5:
>>> - Add pci_cleanup_msi(pdev) in cleanup part.
>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>> Change in v4:
>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>> Change in v3:
>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>> Change in v2:
>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>> - enable vpci support when pci-passthough option is enabled.
>>> ---
>>> ---
>>> xen/arch/arm/Makefile | 1 +
>>> xen/arch/arm/domain.c | 4 ++
>>> xen/arch/arm/domain_build.c | 3 +
>>> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
>>> xen/arch/arm/vpci.h | 36 ++++++++++++
>>> xen/drivers/passthrough/pci.c | 18 ++++++
>>> xen/include/asm-arm/domain.h | 7 ++-
>>> xen/include/asm-x86/pci.h | 2 -
>>> xen/include/public/arch-arm.h | 7 +++
>>> xen/include/xen/pci.h | 2 +
>>> 10 files changed, 179 insertions(+), 3 deletions(-)
>>> create mode 100644 xen/arch/arm/vpci.c
>>> create mode 100644 xen/arch/arm/vpci.h
>>>
>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>> index 44d7cc81fa..fb9c976ea2 100644
>>> --- a/xen/arch/arm/Makefile
>>> +++ b/xen/arch/arm/Makefile
>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>> obj-y += platforms/
>>> endif
>>> obj-$(CONFIG_TEE) += tee/
>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>
>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>> obj-y += bootfdt.init.o
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index 36138c1b2e..fbb52f78f1 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -39,6 +39,7 @@
>>> #include <asm/vgic.h>
>>> #include <asm/vtimer.h>
>>>
>>> +#include "vpci.h"
>>> #include "vuart.h"
>>>
>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>> goto fail;
>>>
>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>> + goto fail;
>>> +
>>> return 0;
>>>
>>> fail:
>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>> index c5afbe2e05..f4c89bde8c 100644
>>> --- a/xen/arch/arm/domain_build.c
>>> +++ b/xen/arch/arm/domain_build.c
>>> @@ -3053,6 +3053,9 @@ void __init create_dom0(void)
>>> if ( iommu_enabled )
>>> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>>>
>>> + if ( is_pci_passthrough_enabled() )
>>> + dom0_cfg.flags |= XEN_DOMCTL_CDF_vpci;
>>> +
>>> dom0 = domain_create(0, &dom0_cfg, true);
>>> if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
>>> panic("Error creating domain 0\n");
>>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>>> new file mode 100644
>>> index 0000000000..76c12b9281
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vpci.c
>>> @@ -0,0 +1,102 @@
>>> +/*
>>> + * xen/arch/arm/vpci.c
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; either version 2 of the License, or
>>> + * (at your option) any later version.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + */
>>> +#include <xen/sched.h>
>>> +
>>> +#include <asm/mmio.h>
>>> +
>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>> +
>>> +/* Do some sanity checks. */
>>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>>> +{
>>> + /* Check access size. */
>>> + if ( len > 8 )
>>> + return false;
>>> +
>>> + /* Check that access is size aligned. */
>>> + if ( (reg & (len - 1)) )
>>> + return false;
>>> +
>>> + return true;
>>> +}
>>> +
>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>> + register_t *r, void *p)
>>> +{
>>> + unsigned int reg;
>>> + pci_sbdf_t sbdf;
>>> + unsigned long data = ~0UL;
>>> + unsigned int size = 1U << info->dabt.size;
>>> +
>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>> + reg = REGISTER_OFFSET(info->gpa);
>>> +
>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>> + return 0;
>>> +
>>> + data = vpci_read(sbdf, reg, min(4u, size));
>>> + if ( size == 8 )
>>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>> +
>>> + *r = data;
>>> +
>>> + return 1;
>>> +}
>>> +
>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>> + register_t r, void *p)
>>> +{
>>> + unsigned int reg;
>>> + pci_sbdf_t sbdf;
>>> + unsigned long data = r;
>>> + unsigned int size = 1U << info->dabt.size;
>>> +
>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>> + reg = REGISTER_OFFSET(info->gpa);
>>> +
>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>> + return 0;
>>> +
>>> + vpci_write(sbdf, reg, min(4u, size), data);
>>> + if ( size == 8 )
>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>> I think those two helpers (and vpci_mmio_access_allowed) are very
>> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
>> the point where I would consider moving the shared code to vpci.c as
>> vpci_ecam_{read,write} and call them from the arch specific trap
>> handlers.
> Except that please can we stick to mcfg or mmcfg instead of ecam
> in names, as that's how the thing has been named in Xen from its
> introduction? I've just grep-ed the code base (case insensitively)
> and found no mention of ECAM. There are only a few "became".
I do understand that this is historically that we do not have ECAM in Xen,
but PCI is not about Xen. Thus, I think it is also acceptable to use
a commonly known ECAM for the code that works with ECAM.
At least for the new code
> Jan
>
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 14:51 ` Oleksandr Andrushchenko
@ 2021-10-13 15:15 ` Jan Beulich
2021-10-13 19:27 ` Stefano Stabellini
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-13 15:15 UTC (permalink / raw)
To: Oleksandr Andrushchenko
Cc: xen-devel, Bertrand Marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, Rahul Singh,
George Dunlap, Ian Jackson, Wei Liu, Paul Durrant,
Roger Pau Monné
On 13.10.2021 16:51, Oleksandr Andrushchenko wrote:
> On 13.10.21 16:00, Jan Beulich wrote:
>> On 13.10.2021 10:45, Roger Pau Monné wrote:
>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/vpci.c
>>>> @@ -0,0 +1,102 @@
>>>> +/*
>>>> + * xen/arch/arm/vpci.c
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify
>>>> + * it under the terms of the GNU General Public License as published by
>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>> + * (at your option) any later version.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>> + * GNU General Public License for more details.
>>>> + */
>>>> +#include <xen/sched.h>
>>>> +
>>>> +#include <asm/mmio.h>
>>>> +
>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>> +
>>>> +/* Do some sanity checks. */
>>>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>>>> +{
>>>> + /* Check access size. */
>>>> + if ( len > 8 )
>>>> + return false;
>>>> +
>>>> + /* Check that access is size aligned. */
>>>> + if ( (reg & (len - 1)) )
>>>> + return false;
>>>> +
>>>> + return true;
>>>> +}
>>>> +
>>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>>> + register_t *r, void *p)
>>>> +{
>>>> + unsigned int reg;
>>>> + pci_sbdf_t sbdf;
>>>> + unsigned long data = ~0UL;
>>>> + unsigned int size = 1U << info->dabt.size;
>>>> +
>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>> +
>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>> + return 0;
>>>> +
>>>> + data = vpci_read(sbdf, reg, min(4u, size));
>>>> + if ( size == 8 )
>>>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>> +
>>>> + *r = data;
>>>> +
>>>> + return 1;
>>>> +}
>>>> +
>>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>>> + register_t r, void *p)
>>>> +{
>>>> + unsigned int reg;
>>>> + pci_sbdf_t sbdf;
>>>> + unsigned long data = r;
>>>> + unsigned int size = 1U << info->dabt.size;
>>>> +
>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>> +
>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>> + return 0;
>>>> +
>>>> + vpci_write(sbdf, reg, min(4u, size), data);
>>>> + if ( size == 8 )
>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>> I think those two helpers (and vpci_mmio_access_allowed) are very
>>> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
>>> the point where I would consider moving the shared code to vpci.c as
>>> vpci_ecam_{read,write} and call them from the arch specific trap
>>> handlers.
>> Except that please can we stick to mcfg or mmcfg instead of ecam
>> in names, as that's how the thing has been named in Xen from its
>> introduction? I've just grep-ed the code base (case insensitively)
>> and found no mention of ECAM. There are only a few "became".
> I do understand that this is historically that we do not have ECAM in Xen,
> but PCI is not about Xen. Thus, I think it is also acceptable to use
> a commonly known ECAM for the code that works with ECAM.
ACPI, afaik, also doesn't call this ECAM. That's where MCFG / MMCFG
actually come from, I believe.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 15:15 ` Jan Beulich
@ 2021-10-13 19:27 ` Stefano Stabellini
2021-10-14 6:33 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-13 19:27 UTC (permalink / raw)
To: Jan Beulich
Cc: Oleksandr Andrushchenko, xen-devel, Bertrand Marquis,
Andre.Przywara, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, Rahul Singh, George Dunlap,
Ian Jackson, Wei Liu, Paul Durrant, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 4098 bytes --]
On Wed, 13 Oct 2021, Jan Beulich wrote:
> On 13.10.2021 16:51, Oleksandr Andrushchenko wrote:
> > On 13.10.21 16:00, Jan Beulich wrote:
> >> On 13.10.2021 10:45, Roger Pau Monné wrote:
> >>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
> >>>> --- /dev/null
> >>>> +++ b/xen/arch/arm/vpci.c
> >>>> @@ -0,0 +1,102 @@
> >>>> +/*
> >>>> + * xen/arch/arm/vpci.c
> >>>> + *
> >>>> + * This program is free software; you can redistribute it and/or modify
> >>>> + * it under the terms of the GNU General Public License as published by
> >>>> + * the Free Software Foundation; either version 2 of the License, or
> >>>> + * (at your option) any later version.
> >>>> + *
> >>>> + * This program is distributed in the hope that it will be useful,
> >>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >>>> + * GNU General Public License for more details.
> >>>> + */
> >>>> +#include <xen/sched.h>
> >>>> +
> >>>> +#include <asm/mmio.h>
> >>>> +
> >>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
> >>>> +
> >>>> +/* Do some sanity checks. */
> >>>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> >>>> +{
> >>>> + /* Check access size. */
> >>>> + if ( len > 8 )
> >>>> + return false;
> >>>> +
> >>>> + /* Check that access is size aligned. */
> >>>> + if ( (reg & (len - 1)) )
> >>>> + return false;
> >>>> +
> >>>> + return true;
> >>>> +}
> >>>> +
> >>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> >>>> + register_t *r, void *p)
> >>>> +{
> >>>> + unsigned int reg;
> >>>> + pci_sbdf_t sbdf;
> >>>> + unsigned long data = ~0UL;
> >>>> + unsigned int size = 1U << info->dabt.size;
> >>>> +
> >>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
> >>>> + reg = REGISTER_OFFSET(info->gpa);
> >>>> +
> >>>> + if ( !vpci_mmio_access_allowed(reg, size) )
> >>>> + return 0;
> >>>> +
> >>>> + data = vpci_read(sbdf, reg, min(4u, size));
> >>>> + if ( size == 8 )
> >>>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> >>>> +
> >>>> + *r = data;
> >>>> +
> >>>> + return 1;
> >>>> +}
> >>>> +
> >>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> >>>> + register_t r, void *p)
> >>>> +{
> >>>> + unsigned int reg;
> >>>> + pci_sbdf_t sbdf;
> >>>> + unsigned long data = r;
> >>>> + unsigned int size = 1U << info->dabt.size;
> >>>> +
> >>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
> >>>> + reg = REGISTER_OFFSET(info->gpa);
> >>>> +
> >>>> + if ( !vpci_mmio_access_allowed(reg, size) )
> >>>> + return 0;
> >>>> +
> >>>> + vpci_write(sbdf, reg, min(4u, size), data);
> >>>> + if ( size == 8 )
> >>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
> >>> I think those two helpers (and vpci_mmio_access_allowed) are very
> >>> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
> >>> the point where I would consider moving the shared code to vpci.c as
> >>> vpci_ecam_{read,write} and call them from the arch specific trap
> >>> handlers.
> >> Except that please can we stick to mcfg or mmcfg instead of ecam
> >> in names, as that's how the thing has been named in Xen from its
> >> introduction? I've just grep-ed the code base (case insensitively)
> >> and found no mention of ECAM. There are only a few "became".
> > I do understand that this is historically that we do not have ECAM in Xen,
> > but PCI is not about Xen. Thus, I think it is also acceptable to use
> > a commonly known ECAM for the code that works with ECAM.
>
> ACPI, afaik, also doesn't call this ECAM. That's where MCFG / MMCFG
> actually come from, I believe.
My understanding is that "MCFG" is the name of the ACPI table that
describes the PCI config space [1]. The underlying PCI standard for the
memory mapped layout of the PCI config space is called ECAM. Here, it
makes sense to call it ECAM as it is firmware independent.
[1] https://wiki.osdev.org/PCI_Express
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 12:57 ` Jan Beulich
@ 2021-10-13 20:41 ` Stefano Stabellini
2021-10-14 6:23 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-13 20:41 UTC (permalink / raw)
To: Jan Beulich
Cc: Bertrand Marquis, Michal Orzel, Rahul Singh, xen-devel,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, Andrew Cooper, George Dunlap, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 5724 bytes --]
On Wed, 13 Oct 2021, Jan Beulich wrote:
> On 13.10.2021 14:11, Bertrand Marquis wrote:
> >> On 13 Oct 2021, at 11:56, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >> If vPCI for Arm on 4.16 is not going to be functional, why so much
> >> pressure in pushing those patches so fast? I understand the need to
> >> remove stuff from the queue, but I don't think it's worth the cost of
> >> introducing a broken interface deliberately on a release.
>
> +1
>
> > We did not push that fast, those have been on the mailing list for a while (this is the 5th version of the serie).
> > PCI passthrough is a big change requiring lots of patches and we decided to push it piece by piece to make
> > the review easier.
>
> 5 versions for a series like this one was to be expected. Imo it has
> been wrong in the past to rush in new features (even if experimental
> ones) very close to the freeze, and it has mislead people to think
> they can delay work until very (too?) late a point in time.
Hi Roger, Jan,
Let me take a few minutes to clarify and provide context for this work.
I don't think anyone "pushed hard" any of the ARM series close to the
release. I sent "one" email in public as a reminder of things that need
reviewing:
https://marc.info/?l=xen-devel&m=163373776611154
I did send a couple of private emails to Jan but they were to synchronize
ourselves rather than push; Jan, I hope you didn't take them the wrong
way.
At the same time it is certainly true that all the people involved here
worked very hard to get these series ready for 4.16. Oct 15 is the Xen
Project feature freeze. It is also the deadline for many of us working
with upstream Xen Project to upstream features and report back to our
management. I know it doesn't affect you guys directly, but you can
imagine that as employees of our respective companies we feel pressure
to complete our objectives in the given timeframe. Of course we need to
make sure to do that without compromising quality and without
overextending upstream review capacity.
The ARM PCI series are public since Dec 2020, pushed to a gitlab branch
for testing: https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
I helped creating, managing and testing the branch.
So, I can see why Bertrand feels like they have been around for a while:
we have been dealing with these patches for almost a year, even if from
a xen-devel perspective we are "only" at version 5.
In the past there have been contributors dropping a large half-broken
series at the last minute before the code freeze, asking for it to be
committed, promising it will be improved later, and then disappearing
right after the release, leaving the code half-broken forever.
It is easy to draw comparisons between that kind of behavior and what is
happening in this thread. We have all been burned by it at some point.
However, that is really *not* what is happening this week.
PCI Passthrough for ARM is a difficult feature to implement. We have
been discussing it for 5+ years, close to 10 years. I know it sounds
unbelievable but it is true. For one reason or the other it never
materialized. Until now.
Even with all the outstanding series committed, PCI Passthrough for ARM
is not fully functional, but it is a lot closer to the objective.
Rahul, Oleksandr and their teams have been already "sticking around" for
almost a year making improvements and without their help going forward
it is not going to get finished anyway. We still don't have MSI support.
Given that Rahul, Oleksandr and the others are required to finish the
work anyway, and they are not at risk of disappearing, I think we can
deal with one or two TODO left by Oct 15 [1].
My suggestion is to accept the TODO for patch #8 [1] but I agree with
Roger that we shouldn't introduce bad interfaces, especially as they
impact x86 which is not "tech preview". So I think we should revert
patch #7 (this patch.) I'll reply with more details in separate shorter
email.
[1] https://marc.info/?l=xen-devel&m=163412120531248
> In fact I'm getting somewhat angry seeing how much effort is put into
> getting this work in (including myself doing reviews there), when at
> the same time far older series of mine aren't given any chance to
> make any progress. Not your fault, sure, but an observation.
Jan,
Let me start by pointing out that without your help we would never have
managed to get so many ARM patches committed for 4.16. Not just PCI
related, but also EFI too. You should really get credit for that.
I can imagine it is frustrating helping so much somebody else but not
getting help for your own series. In a way it happened to me too as the
only Xilinx series outstanding didn't make any progress:
https://marc.info/?l=xen-devel&m=163056545414965
The only way to solve this problem is by growing the review capacity of
the community. You'll be happy to know that (without naming names) I
have heard from multiple people, new members of the community, that they
are going to commit time to do reviews during the next months. We are
growing the community again and some of the new contributors will become
good reviewers and maintainers over time. I am positive that the next 12
months will be a lot better than the last 12 months from that
perspective.
Thanks Jan, Roger, the two Oleksandr, Rahul, Michal, Bertrand, Luca and
everyone else that stayed up late these last few weeks to push Xen
forward with new groundbreaking features and pushing the project forward
in new directions. I do think 4.16 is going to be a great release and
the both CI-loops are still fully working with staging very close to
master, further proof of the great work of this community.
Cheers,
Stefano
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 10:56 ` Roger Pau Monné
2021-10-13 12:11 ` Bertrand Marquis
@ 2021-10-13 20:53 ` Stefano Stabellini
2021-10-13 23:21 ` Stefano Stabellini
1 sibling, 1 reply; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-13 20:53 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Bertrand Marquis, Michal Orzel, Rahul Singh, xen-devel,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Stefano Stabellini, Volodymyr Babchuk
[-- Attachment #1: Type: text/plain, Size: 2646 bytes --]
On Wed, 13 Oct 2021, Roger Pau Monné wrote:
> > I think the second solution is the right one but it cannot be done so near from the
> > feature freeze.
> >
> > The CDF flag as introduced right now is not creating any issue and will be used once
> > the emulation flag will be introduce. We will be able at this stage to set this properly
> > also on x86 (for dom0 pvh).
> > Moreover keeping it will allow to continue to merge the remaining part of the PCI
> > passthrough which are otherwise not possible to be done as they are dependent on this flag.
> >
> > Can we agree on keep the DOMCTL_CDF_vpci flag and introduce the emulation
> > flag on Arm after 4.16 release ?
>
> If vPCI for Arm on 4.16 is not going to be functional, why so much
> pressure in pushing those patches so fast? I understand the need to
> remove stuff from the queue, but I don't think it's worth the cost of
> introducing a broken interface deliberately on a release.
>
> I think we need to at least settle on whether we want to keep
> CDF_vpci or use an arch specific signal mechanism in order to decide
> what to do regarding the release.
I wrote a longer separate email to provide more context about the
"pushing fast" comment.
I agree that we don't want to introduce a bad interface.
In regards to a way forward for 4.16, my suggestion is the following:
- revert this patch: do not change the interface in this series
- do not change anything related to CDF_vpci for x86
- for ARM, leave has_vpci(d) to { false } and do not set
XEN_DOMCTL_CDF_vpci
- we can discuss it in depth later on, no rush
- in patch #10, in libxl_arm.c:libxl__arch_domain_prepare_config
- do not set XEN_DOMCTL_CDF_vpci
- do not set b_info.arch_arm.vpci
- do not define LIBXL_HAVE_BUILDINFO_ARM_VPCI in libxl.h
- keep make_vpci_node and arch_arm.vpci
The other patches (#1, #8, #10), which don't change any interfaces, can
still make it for 4.16 if the review feedback is addressed on time, with
one open TODO item [1].
This way, we get all the essential infrastructure we are trying to
introduce without making any compromises on the external interfaces.
Still it is good to have patches #1 #8 #10 so that with a trival
oneliner patch on top of 4.16 we can enable PCI for ARM and do testing
in the community, in gitlab-ci, and OSSTest too. (We have been
discussing special OSSTest flights to valide PCI passthrough as we
complete development.)
If we think we need a snap decision on this topic, I am available for a
quick sync-up call or IRC meeting between 8AM and 10AM tomorrow (Oct
14).
[1] https://marc.info/?l=xen-devel&m=163412120531248
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 11/11] xen/arm: Add linux,pci-domain property for hwdom if not available.
2021-10-06 17:40 ` [PATCH v5 11/11] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
@ 2021-10-13 20:54 ` Stefano Stabellini
0 siblings, 0 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-13 20:54 UTC (permalink / raw)
To: Rahul Singh
Cc: xen-devel, bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk
On Wed, 6 Oct 2021, Rahul Singh wrote:
> If the property is not present in the device tree node for host bridge,
> XEN while creating the dtb for hwdom will create this property and
> assigns the already allocated segment to the host bridge
> so that XEN and linux will have the same segment for the host bridges.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
I committed this patch
> ---
> Change in v5: none
> Change in v4:
> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> - Added Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>
> Change in v3:
> - Use is_pci_passthrough_enabled()
> Change in v2:
> - Add linux,pci-domain only when pci-passthrough command line option is enabeld
> ---
> ---
> xen/arch/arm/domain_build.c | 16 ++++++++++++++++
> xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++
> xen/include/asm-arm/pci.h | 9 +++++++++
> 3 files changed, 46 insertions(+)
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index f4c89bde8c..8739e3285e 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -747,6 +747,22 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
> return res;
> }
>
> + if ( is_pci_passthrough_enabled() && dt_device_type_is_equal(node, "pci") )
> + {
> + if ( !dt_find_property(node, "linux,pci-domain", NULL) )
> + {
> + uint16_t segment;
> +
> + res = pci_get_host_bridge_segment(node, &segment);
> + if ( res < 0 )
> + return res;
> +
> + res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment);
> + if ( res )
> + return res;
> + }
> + }
> +
> /*
> * Override the property "status" to disable the device when it's
> * marked for passthrough.
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> index c5941b10e9..593beeb48c 100644
> --- a/xen/arch/arm/pci/pci-host-common.c
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -255,6 +255,27 @@ struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
>
> return NULL;
> }
> +
> +/*
> + * This function will lookup an hostbridge based on config space address.
> + */
> +int pci_get_host_bridge_segment(const struct dt_device_node *node,
> + uint16_t *segment)
> +{
> + struct pci_host_bridge *bridge;
> +
> + list_for_each_entry( bridge, &pci_host_bridges, node )
> + {
> + if ( bridge->dt_node != node )
> + continue;
> +
> + *segment = bridge->segment;
> + return 0;
> + }
> +
> + return -EINVAL;
> +}
> +
> /*
> * Local variables:
> * mode: C
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 5532ce3977..7cb2e2f1ed 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -90,6 +90,8 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, pci_sbdf_t sbdf,
> void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
> pci_sbdf_t sbdf, uint32_t where);
> struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
> +int pci_get_host_bridge_segment(const struct dt_device_node *node,
> + uint16_t *segment);
>
> static always_inline bool is_pci_passthrough_enabled(void)
> {
> @@ -104,5 +106,12 @@ static always_inline bool is_pci_passthrough_enabled(void)
> return false;
> }
>
> +static inline int pci_get_host_bridge_segment(const struct dt_device_node *node,
> + uint16_t *segment)
> +{
> + ASSERT_UNREACHABLE();
> + return -EINVAL;
> +}
> +
> #endif /*!CONFIG_HAS_PCI*/
> #endif /* __ARM_PCI_H__ */
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 20:53 ` Stefano Stabellini
@ 2021-10-13 23:21 ` Stefano Stabellini
0 siblings, 0 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-13 23:21 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Roger Pau Monné,
Bertrand Marquis, Michal Orzel, Rahul Singh, xen-devel,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, Andrew Cooper, George Dunlap, Jan Beulich, Julien Grall,
Volodymyr Babchuk
[-- Attachment #1: Type: text/plain, Size: 2857 bytes --]
On Wed, 13 Oct 2021, Stefano Stabellini wrote:
> On Wed, 13 Oct 2021, Roger Pau Monné wrote:
> > > I think the second solution is the right one but it cannot be done so near from the
> > > feature freeze.
> > >
> > > The CDF flag as introduced right now is not creating any issue and will be used once
> > > the emulation flag will be introduce. We will be able at this stage to set this properly
> > > also on x86 (for dom0 pvh).
> > > Moreover keeping it will allow to continue to merge the remaining part of the PCI
> > > passthrough which are otherwise not possible to be done as they are dependent on this flag.
> > >
> > > Can we agree on keep the DOMCTL_CDF_vpci flag and introduce the emulation
> > > flag on Arm after 4.16 release ?
> >
> > If vPCI for Arm on 4.16 is not going to be functional, why so much
> > pressure in pushing those patches so fast? I understand the need to
> > remove stuff from the queue, but I don't think it's worth the cost of
> > introducing a broken interface deliberately on a release.
> >
> > I think we need to at least settle on whether we want to keep
> > CDF_vpci or use an arch specific signal mechanism in order to decide
> > what to do regarding the release.
>
> I wrote a longer separate email to provide more context about the
> "pushing fast" comment.
>
> I agree that we don't want to introduce a bad interface.
>
> In regards to a way forward for 4.16, my suggestion is the following:
>
> - revert this patch: do not change the interface in this series
> - do not change anything related to CDF_vpci for x86
> - for ARM, leave has_vpci(d) to { false } and do not set
> XEN_DOMCTL_CDF_vpci
> - we can discuss it in depth later on, no rush
>
> - in patch #10, in libxl_arm.c:libxl__arch_domain_prepare_config
> - do not set XEN_DOMCTL_CDF_vpci
> - do not set b_info.arch_arm.vpci
> - do not define LIBXL_HAVE_BUILDINFO_ARM_VPCI in libxl.h
> - keep make_vpci_node and arch_arm.vpci
>
>
> The other patches (#1, #8, #10), which don't change any interfaces, can
> still make it for 4.16 if the review feedback is addressed on time, with
> one open TODO item [1].
>
> This way, we get all the essential infrastructure we are trying to
> introduce without making any compromises on the external interfaces.
> Still it is good to have patches #1 #8 #10 so that with a trival
> oneliner patch on top of 4.16 we can enable PCI for ARM and do testing
> in the community, in gitlab-ci, and OSSTest too. (We have been
> discussing special OSSTest flights to valide PCI passthrough as we
> complete development.)
One more thing: I would really like to get at least patch #8 committed
because it would help with making progress with part 2 and part 3 of the
PCI enablement series. My preference would also be to get #1 and #10
in as well but I feel less strongly about it.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-13 20:41 ` Stefano Stabellini
@ 2021-10-14 6:23 ` Jan Beulich
2021-10-14 7:53 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-14 6:23 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Bertrand Marquis, Michal Orzel, Rahul Singh, xen-devel,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, Andrew Cooper, George Dunlap, Julien Grall,
Volodymyr Babchuk, Roger Pau Monné
On 13.10.2021 22:41, Stefano Stabellini wrote:
> On Wed, 13 Oct 2021, Jan Beulich wrote:
>> On 13.10.2021 14:11, Bertrand Marquis wrote:
>>>> On 13 Oct 2021, at 11:56, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>> If vPCI for Arm on 4.16 is not going to be functional, why so much
>>>> pressure in pushing those patches so fast? I understand the need to
>>>> remove stuff from the queue, but I don't think it's worth the cost of
>>>> introducing a broken interface deliberately on a release.
>>
>> +1
>>
>>> We did not push that fast, those have been on the mailing list for a while (this is the 5th version of the serie).
>>> PCI passthrough is a big change requiring lots of patches and we decided to push it piece by piece to make
>>> the review easier.
>>
>> 5 versions for a series like this one was to be expected. Imo it has
>> been wrong in the past to rush in new features (even if experimental
>> ones) very close to the freeze, and it has mislead people to think
>> they can delay work until very (too?) late a point in time.
>
>
> Hi Roger, Jan,
>
> Let me take a few minutes to clarify and provide context for this work.
>
>
> I don't think anyone "pushed hard" any of the ARM series close to the
> release. I sent "one" email in public as a reminder of things that need
> reviewing:
> https://marc.info/?l=xen-devel&m=163373776611154
>
> I did send a couple of private emails to Jan but they were to synchronize
> ourselves rather than push; Jan, I hope you didn't take them the wrong
> way.
Definitely not, no worries.
> At the same time it is certainly true that all the people involved here
> worked very hard to get these series ready for 4.16. Oct 15 is the Xen
> Project feature freeze. It is also the deadline for many of us working
> with upstream Xen Project to upstream features and report back to our
> management. I know it doesn't affect you guys directly, but you can
> imagine that as employees of our respective companies we feel pressure
> to complete our objectives in the given timeframe. Of course we need to
> make sure to do that without compromising quality and without
> overextending upstream review capacity.
>
>
> The ARM PCI series are public since Dec 2020, pushed to a gitlab branch
> for testing: https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
> I helped creating, managing and testing the branch.
>
> So, I can see why Bertrand feels like they have been around for a while:
> we have been dealing with these patches for almost a year, even if from
> a xen-devel perspective we are "only" at version 5.
I'm afraid that anything not on xen-devel doesn't really count; the list
is the only "official" channel. ISTR that for certain aspects there's a
plan George has to make this more pronounced / formal in some of the docs
we have.
Making internally set deadlines is a fully understandable aspect. But
starting to post in September for a mid-October deadline is putting
oneself at risk, I would say, for a (set of) series this involved. I see
no problem with anyone doing so, as long as they accept that risk rather
than expecting to get away by either extraordinary efforts by others
(reviewers) or relaxation of what is permitted to go in.
Of course there's the opposite problem of feedback taking unusually (I'm
inclined to say unduly) long to arrive, like with the gpaddr_bits
addition still facing vague / unclear opposition by Andrew.
> My suggestion is to accept the TODO for patch #8 [1] but I agree with
> Roger that we shouldn't introduce bad interfaces, especially as they
> impact x86 which is not "tech preview". So I think we should revert
> patch #7 (this patch.) I'll reply with more details in separate shorter
> email.
>
> [1] https://marc.info/?l=xen-devel&m=163412120531248
FWIW I agree with both proposals (acceptance of TODO and revert).
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-13 19:27 ` Stefano Stabellini
@ 2021-10-14 6:33 ` Jan Beulich
2021-10-14 7:53 ` Bertrand Marquis
2021-10-14 9:03 ` Bertrand Marquis
0 siblings, 2 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-14 6:33 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Oleksandr Andrushchenko, xen-devel, Bertrand Marquis,
Andre.Przywara, Julien Grall, Volodymyr Babchuk, Andrew Cooper,
Rahul Singh, George Dunlap, Ian Jackson, Wei Liu, Paul Durrant,
Roger Pau Monné
On 13.10.2021 21:27, Stefano Stabellini wrote:
> On Wed, 13 Oct 2021, Jan Beulich wrote:
>> On 13.10.2021 16:51, Oleksandr Andrushchenko wrote:
>>> On 13.10.21 16:00, Jan Beulich wrote:
>>>> On 13.10.2021 10:45, Roger Pau Monné wrote:
>>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>>> --- /dev/null
>>>>>> +++ b/xen/arch/arm/vpci.c
>>>>>> @@ -0,0 +1,102 @@
>>>>>> +/*
>>>>>> + * xen/arch/arm/vpci.c
>>>>>> + *
>>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>>> + * it under the terms of the GNU General Public License as published by
>>>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>>>> + * (at your option) any later version.
>>>>>> + *
>>>>>> + * This program is distributed in the hope that it will be useful,
>>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>>>> + * GNU General Public License for more details.
>>>>>> + */
>>>>>> +#include <xen/sched.h>
>>>>>> +
>>>>>> +#include <asm/mmio.h>
>>>>>> +
>>>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>>>> +
>>>>>> +/* Do some sanity checks. */
>>>>>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>>>>>> +{
>>>>>> + /* Check access size. */
>>>>>> + if ( len > 8 )
>>>>>> + return false;
>>>>>> +
>>>>>> + /* Check that access is size aligned. */
>>>>>> + if ( (reg & (len - 1)) )
>>>>>> + return false;
>>>>>> +
>>>>>> + return true;
>>>>>> +}
>>>>>> +
>>>>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>>>>> + register_t *r, void *p)
>>>>>> +{
>>>>>> + unsigned int reg;
>>>>>> + pci_sbdf_t sbdf;
>>>>>> + unsigned long data = ~0UL;
>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>> +
>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>> +
>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>> + return 0;
>>>>>> +
>>>>>> + data = vpci_read(sbdf, reg, min(4u, size));
>>>>>> + if ( size == 8 )
>>>>>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>>> +
>>>>>> + *r = data;
>>>>>> +
>>>>>> + return 1;
>>>>>> +}
>>>>>> +
>>>>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>>>>> + register_t r, void *p)
>>>>>> +{
>>>>>> + unsigned int reg;
>>>>>> + pci_sbdf_t sbdf;
>>>>>> + unsigned long data = r;
>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>> +
>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>> +
>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>> + return 0;
>>>>>> +
>>>>>> + vpci_write(sbdf, reg, min(4u, size), data);
>>>>>> + if ( size == 8 )
>>>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>>> I think those two helpers (and vpci_mmio_access_allowed) are very
>>>>> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
>>>>> the point where I would consider moving the shared code to vpci.c as
>>>>> vpci_ecam_{read,write} and call them from the arch specific trap
>>>>> handlers.
>>>> Except that please can we stick to mcfg or mmcfg instead of ecam
>>>> in names, as that's how the thing has been named in Xen from its
>>>> introduction? I've just grep-ed the code base (case insensitively)
>>>> and found no mention of ECAM. There are only a few "became".
>>> I do understand that this is historically that we do not have ECAM in Xen,
>>> but PCI is not about Xen. Thus, I think it is also acceptable to use
>>> a commonly known ECAM for the code that works with ECAM.
>>
>> ACPI, afaik, also doesn't call this ECAM. That's where MCFG / MMCFG
>> actually come from, I believe.
>
> My understanding is that "MCFG" is the name of the ACPI table that
> describes the PCI config space [1]. The underlying PCI standard for the
> memory mapped layout of the PCI config space is called ECAM. Here, it
> makes sense to call it ECAM as it is firmware independent.
>
> [1] https://wiki.osdev.org/PCI_Express
Okay, I can accept this, but then I'd expect all existing uses of
MCFG / MMCFG in the code which aren't directly ACPI-related to get
replaced. Otherwise it's needlessly harder to identify that one
piece of code relates to certain other pieces.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag
2021-10-14 6:23 ` Jan Beulich
@ 2021-10-14 7:53 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 7:53 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, Michal Orzel, Rahul Singh, xen-devel,
Andre Przywara, Christian Lindig, David Scott, Ian Jackson,
Wei Liu, Andrew Cooper, George Dunlap, Julien Grall,
Volodymyr Babchuk, Roger Pau Monné
Hi,
> On 14 Oct 2021, at 07:23, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 13.10.2021 22:41, Stefano Stabellini wrote:
>> On Wed, 13 Oct 2021, Jan Beulich wrote:
>>> On 13.10.2021 14:11, Bertrand Marquis wrote:
>>>>> On 13 Oct 2021, at 11:56, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>> If vPCI for Arm on 4.16 is not going to be functional, why so much
>>>>> pressure in pushing those patches so fast? I understand the need to
>>>>> remove stuff from the queue, but I don't think it's worth the cost of
>>>>> introducing a broken interface deliberately on a release.
>>>
>>> +1
>>>
>>>> We did not push that fast, those have been on the mailing list for a while (this is the 5th version of the serie).
>>>> PCI passthrough is a big change requiring lots of patches and we decided to push it piece by piece to make
>>>> the review easier.
>>>
>>> 5 versions for a series like this one was to be expected. Imo it has
>>> been wrong in the past to rush in new features (even if experimental
>>> ones) very close to the freeze, and it has mislead people to think
>>> they can delay work until very (too?) late a point in time.
>>
>>
>> Hi Roger, Jan,
>>
>> Let me take a few minutes to clarify and provide context for this work.
>>
>>
>> I don't think anyone "pushed hard" any of the ARM series close to the
>> release. I sent "one" email in public as a reminder of things that need
>> reviewing:
>> https://marc.info/?l=xen-devel&m=163373776611154
>>
>> I did send a couple of private emails to Jan but they were to synchronize
>> ourselves rather than push; Jan, I hope you didn't take them the wrong
>> way.
>
> Definitely not, no worries.
>
>> At the same time it is certainly true that all the people involved here
>> worked very hard to get these series ready for 4.16. Oct 15 is the Xen
>> Project feature freeze. It is also the deadline for many of us working
>> with upstream Xen Project to upstream features and report back to our
>> management. I know it doesn't affect you guys directly, but you can
>> imagine that as employees of our respective companies we feel pressure
>> to complete our objectives in the given timeframe. Of course we need to
>> make sure to do that without compromising quality and without
>> overextending upstream review capacity.
>>
>>
>> The ARM PCI series are public since Dec 2020, pushed to a gitlab branch
>> for testing: https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
>> I helped creating, managing and testing the branch.
>>
>> So, I can see why Bertrand feels like they have been around for a while:
>> we have been dealing with these patches for almost a year, even if from
>> a xen-devel perspective we are "only" at version 5.
>
> I'm afraid that anything not on xen-devel doesn't really count; the list
> is the only "official" channel. ISTR that for certain aspects there's a
> plan George has to make this more pronounced / formal in some of the docs
> we have.
>
> Making internally set deadlines is a fully understandable aspect. But
> starting to post in September for a mid-October deadline is putting
> oneself at risk, I would say, for a (set of) series this involved. I see
> no problem with anyone doing so, as long as they accept that risk rather
> than expecting to get away by either extraordinary efforts by others
> (reviewers) or relaxation of what is permitted to go in.
>
> Of course there's the opposite problem of feedback taking unusually (I'm
> inclined to say unduly) long to arrive, like with the gpaddr_bits
> addition still facing vague / unclear opposition by Andrew.
>
>> My suggestion is to accept the TODO for patch #8 [1] but I agree with
>> Roger that we shouldn't introduce bad interfaces, especially as they
>> impact x86 which is not "tech preview". So I think we should revert
>> patch #7 (this patch.) I'll reply with more details in separate shorter
>> email.
>>
>> [1] https://marc.info/?l=xen-devel&m=163412120531248
>
> FWIW I agree with both proposals (acceptance of TODO and revert).
I want to say again that nothing I said was meant as a critic and I am very
grateful of all reviews and comments that we have (even if some of them
are implying work or pushing stuff in the future) as we want to do this right.
We will do the following before the end of the day today:
- send a reverting patch for patch 7
- send updated version of patch 8 handling all pending comments
We will try to (not sure as some changes might be more complex)
- send an updated version of patch 10
- try to find a solution acceptable for patch 1
Thanks everyone for the support here and the huge amount of time
spent on reviewing all patches that was pushed (and there was a lot of them).
Over the next month Arm is committed to put a lot more effort in reviewing and
testing patches pushed to xen-devel. We now have an internal CI that will help
us a lot and all people in Arm Xen team will start to have dedicated time to review
upstream patches and help the community.
Xen is a big part of Arm strategy around automotive as the reference Type 1
hypervisor and we will not disappear soon :-)
If there are any outstanding serie or patches for which you would like us to help
On review, please do not hesitate to ping me and I will see how we can help (even
if it is only to do some testing).
Thanks again everyone for the help.
Cheers
Bertrand
PS: I will stay connected on IRC if someone needs to contact me.
>
> Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 6:33 ` Jan Beulich
@ 2021-10-14 7:53 ` Bertrand Marquis
2021-10-14 9:03 ` Bertrand Marquis
1 sibling, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 7:53 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, Oleksandr Andrushchenko, xen-devel,
Andre Przywara, Julien Grall, Volodymyr Babchuk, Andrew Cooper,
Rahul Singh, George Dunlap, Ian Jackson, Wei Liu, Paul Durrant,
Roger Pau Monné
Hi,
> On 14 Oct 2021, at 07:33, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 13.10.2021 21:27, Stefano Stabellini wrote:
>> On Wed, 13 Oct 2021, Jan Beulich wrote:
>>> On 13.10.2021 16:51, Oleksandr Andrushchenko wrote:
>>>> On 13.10.21 16:00, Jan Beulich wrote:
>>>>> On 13.10.2021 10:45, Roger Pau Monné wrote:
>>>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>>>> --- /dev/null
>>>>>>> +++ b/xen/arch/arm/vpci.c
>>>>>>> @@ -0,0 +1,102 @@
>>>>>>> +/*
>>>>>>> + * xen/arch/arm/vpci.c
>>>>>>> + *
>>>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>>>> + * it under the terms of the GNU General Public License as published by
>>>>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>>>>> + * (at your option) any later version.
>>>>>>> + *
>>>>>>> + * This program is distributed in the hope that it will be useful,
>>>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>>>>> + * GNU General Public License for more details.
>>>>>>> + */
>>>>>>> +#include <xen/sched.h>
>>>>>>> +
>>>>>>> +#include <asm/mmio.h>
>>>>>>> +
>>>>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>>>>> +
>>>>>>> +/* Do some sanity checks. */
>>>>>>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>>>>>>> +{
>>>>>>> + /* Check access size. */
>>>>>>> + if ( len > 8 )
>>>>>>> + return false;
>>>>>>> +
>>>>>>> + /* Check that access is size aligned. */
>>>>>>> + if ( (reg & (len - 1)) )
>>>>>>> + return false;
>>>>>>> +
>>>>>>> + return true;
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>>>>>> + register_t *r, void *p)
>>>>>>> +{
>>>>>>> + unsigned int reg;
>>>>>>> + pci_sbdf_t sbdf;
>>>>>>> + unsigned long data = ~0UL;
>>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>>> +
>>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>>> +
>>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>>> + return 0;
>>>>>>> +
>>>>>>> + data = vpci_read(sbdf, reg, min(4u, size));
>>>>>>> + if ( size == 8 )
>>>>>>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>>>> +
>>>>>>> + *r = data;
>>>>>>> +
>>>>>>> + return 1;
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>>>>>> + register_t r, void *p)
>>>>>>> +{
>>>>>>> + unsigned int reg;
>>>>>>> + pci_sbdf_t sbdf;
>>>>>>> + unsigned long data = r;
>>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>>> +
>>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>>> +
>>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>>> + return 0;
>>>>>>> +
>>>>>>> + vpci_write(sbdf, reg, min(4u, size), data);
>>>>>>> + if ( size == 8 )
>>>>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>>>> I think those two helpers (and vpci_mmio_access_allowed) are very
>>>>>> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
>>>>>> the point where I would consider moving the shared code to vpci.c as
>>>>>> vpci_ecam_{read,write} and call them from the arch specific trap
>>>>>> handlers.
>>>>> Except that please can we stick to mcfg or mmcfg instead of ecam
>>>>> in names, as that's how the thing has been named in Xen from its
>>>>> introduction? I've just grep-ed the code base (case insensitively)
>>>>> and found no mention of ECAM. There are only a few "became".
>>>> I do understand that this is historically that we do not have ECAM in Xen,
>>>> but PCI is not about Xen. Thus, I think it is also acceptable to use
>>>> a commonly known ECAM for the code that works with ECAM.
>>>
>>> ACPI, afaik, also doesn't call this ECAM. That's where MCFG / MMCFG
>>> actually come from, I believe.
>>
>> My understanding is that "MCFG" is the name of the ACPI table that
>> describes the PCI config space [1]. The underlying PCI standard for the
>> memory mapped layout of the PCI config space is called ECAM. Here, it
>> makes sense to call it ECAM as it is firmware independent.
>>
>> [1] https://wiki.osdev.org/PCI_Express
>
> Okay, I can accept this, but then I'd expect all existing uses of
> MCFG / MMCFG in the code which aren't directly ACPI-related to get
> replaced. Otherwise it's needlessly harder to identify that one
> piece of code relates to certain other pieces.
To sum up on this subject, here what I will do in version 6:
- move vpci_ecam_{read/write} to vpci.c and rename them
- same for access allowed and I will rename it to vpci_ecam_access_allowed
I will work on this and try to push v6 before the end of the day (also handling other remarks on this patch).
Cheers
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 6:33 ` Jan Beulich
2021-10-14 7:53 ` Bertrand Marquis
@ 2021-10-14 9:03 ` Bertrand Marquis
2021-10-14 9:24 ` Jan Beulich
1 sibling, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 9:03 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, Oleksandr Andrushchenko, xen-devel,
Andre Przywara, Julien Grall, Volodymyr Babchuk, Andrew Cooper,
Rahul Singh, George Dunlap, Ian Jackson, Wei Liu, Paul Durrant,
Roger Pau Monné
Hi Jan,
> On 14 Oct 2021, at 07:33, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 13.10.2021 21:27, Stefano Stabellini wrote:
>> On Wed, 13 Oct 2021, Jan Beulich wrote:
>>> On 13.10.2021 16:51, Oleksandr Andrushchenko wrote:
>>>> On 13.10.21 16:00, Jan Beulich wrote:
>>>>> On 13.10.2021 10:45, Roger Pau Monné wrote:
>>>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>>>> --- /dev/null
>>>>>>> +++ b/xen/arch/arm/vpci.c
>>>>>>> @@ -0,0 +1,102 @@
>>>>>>> +/*
>>>>>>> + * xen/arch/arm/vpci.c
>>>>>>> + *
>>>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>>>> + * it under the terms of the GNU General Public License as published by
>>>>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>>>>> + * (at your option) any later version.
>>>>>>> + *
>>>>>>> + * This program is distributed in the hope that it will be useful,
>>>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>>>>> + * GNU General Public License for more details.
>>>>>>> + */
>>>>>>> +#include <xen/sched.h>
>>>>>>> +
>>>>>>> +#include <asm/mmio.h>
>>>>>>> +
>>>>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>>>>> +
>>>>>>> +/* Do some sanity checks. */
>>>>>>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>>>>>>> +{
>>>>>>> + /* Check access size. */
>>>>>>> + if ( len > 8 )
>>>>>>> + return false;
>>>>>>> +
>>>>>>> + /* Check that access is size aligned. */
>>>>>>> + if ( (reg & (len - 1)) )
>>>>>>> + return false;
>>>>>>> +
>>>>>>> + return true;
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>>>>>> + register_t *r, void *p)
>>>>>>> +{
>>>>>>> + unsigned int reg;
>>>>>>> + pci_sbdf_t sbdf;
>>>>>>> + unsigned long data = ~0UL;
>>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>>> +
>>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>>> +
>>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>>> + return 0;
>>>>>>> +
>>>>>>> + data = vpci_read(sbdf, reg, min(4u, size));
>>>>>>> + if ( size == 8 )
>>>>>>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>>>> +
>>>>>>> + *r = data;
>>>>>>> +
>>>>>>> + return 1;
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>>>>>> + register_t r, void *p)
>>>>>>> +{
>>>>>>> + unsigned int reg;
>>>>>>> + pci_sbdf_t sbdf;
>>>>>>> + unsigned long data = r;
>>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>>> +
>>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>>> +
>>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>>> + return 0;
>>>>>>> +
>>>>>>> + vpci_write(sbdf, reg, min(4u, size), data);
>>>>>>> + if ( size == 8 )
>>>>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>>>> I think those two helpers (and vpci_mmio_access_allowed) are very
>>>>>> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
>>>>>> the point where I would consider moving the shared code to vpci.c as
>>>>>> vpci_ecam_{read,write} and call them from the arch specific trap
>>>>>> handlers.
>>>>> Except that please can we stick to mcfg or mmcfg instead of ecam
>>>>> in names, as that's how the thing has been named in Xen from its
>>>>> introduction? I've just grep-ed the code base (case insensitively)
>>>>> and found no mention of ECAM. There are only a few "became".
>>>> I do understand that this is historically that we do not have ECAM in Xen,
>>>> but PCI is not about Xen. Thus, I think it is also acceptable to use
>>>> a commonly known ECAM for the code that works with ECAM.
>>>
>>> ACPI, afaik, also doesn't call this ECAM. That's where MCFG / MMCFG
>>> actually come from, I believe.
>>
>> My understanding is that "MCFG" is the name of the ACPI table that
>> describes the PCI config space [1]. The underlying PCI standard for the
>> memory mapped layout of the PCI config space is called ECAM. Here, it
>> makes sense to call it ECAM as it is firmware independent.
>>
>> [1] https://wiki.osdev.org/PCI_Express
>
> Okay, I can accept this, but then I'd expect all existing uses of
> MCFG / MMCFG in the code which aren't directly ACPI-related to get
> replaced. Otherwise it's needlessly harder to identify that one
> piece of code relates to certain other pieces.
If that is ok with I will:
- move function from x86/hw/io.c to vpci.c renaming them to ECAM
- rename MMCFG_{BDF/REG_OFFSET) to ECAM_{BDF/REG_OFFSET}
For the rest of the occurrences in x86 I will not modify any of them as some
are related to ACPI so using (M)MCFG is right and for the others I am not 100%
sure.
Do you agree ?
Cheers
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 9:03 ` Bertrand Marquis
@ 2021-10-14 9:24 ` Jan Beulich
0 siblings, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-14 9:24 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Stefano Stabellini, Oleksandr Andrushchenko, xen-devel,
Andre Przywara, Julien Grall, Volodymyr Babchuk, Andrew Cooper,
Rahul Singh, George Dunlap, Ian Jackson, Wei Liu, Paul Durrant,
Roger Pau Monné
On 14.10.2021 11:03, Bertrand Marquis wrote:
> Hi Jan,
>
>> On 14 Oct 2021, at 07:33, Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 13.10.2021 21:27, Stefano Stabellini wrote:
>>> On Wed, 13 Oct 2021, Jan Beulich wrote:
>>>> On 13.10.2021 16:51, Oleksandr Andrushchenko wrote:
>>>>> On 13.10.21 16:00, Jan Beulich wrote:
>>>>>> On 13.10.2021 10:45, Roger Pau Monné wrote:
>>>>>>> On Wed, Oct 06, 2021 at 06:40:34PM +0100, Rahul Singh wrote:
>>>>>>>> --- /dev/null
>>>>>>>> +++ b/xen/arch/arm/vpci.c
>>>>>>>> @@ -0,0 +1,102 @@
>>>>>>>> +/*
>>>>>>>> + * xen/arch/arm/vpci.c
>>>>>>>> + *
>>>>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>>>>> + * it under the terms of the GNU General Public License as published by
>>>>>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>>>>>> + * (at your option) any later version.
>>>>>>>> + *
>>>>>>>> + * This program is distributed in the hope that it will be useful,
>>>>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>>>>>> + * GNU General Public License for more details.
>>>>>>>> + */
>>>>>>>> +#include <xen/sched.h>
>>>>>>>> +
>>>>>>>> +#include <asm/mmio.h>
>>>>>>>> +
>>>>>>>> +#define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff)
>>>>>>>> +
>>>>>>>> +/* Do some sanity checks. */
>>>>>>>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>>>>>>>> +{
>>>>>>>> + /* Check access size. */
>>>>>>>> + if ( len > 8 )
>>>>>>>> + return false;
>>>>>>>> +
>>>>>>>> + /* Check that access is size aligned. */
>>>>>>>> + if ( (reg & (len - 1)) )
>>>>>>>> + return false;
>>>>>>>> +
>>>>>>>> + return true;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>>>>>>> + register_t *r, void *p)
>>>>>>>> +{
>>>>>>>> + unsigned int reg;
>>>>>>>> + pci_sbdf_t sbdf;
>>>>>>>> + unsigned long data = ~0UL;
>>>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>>>> +
>>>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>>>> +
>>>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>>>> + return 0;
>>>>>>>> +
>>>>>>>> + data = vpci_read(sbdf, reg, min(4u, size));
>>>>>>>> + if ( size == 8 )
>>>>>>>> + data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>>>>> +
>>>>>>>> + *r = data;
>>>>>>>> +
>>>>>>>> + return 1;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>>>>>>> + register_t r, void *p)
>>>>>>>> +{
>>>>>>>> + unsigned int reg;
>>>>>>>> + pci_sbdf_t sbdf;
>>>>>>>> + unsigned long data = r;
>>>>>>>> + unsigned int size = 1U << info->dabt.size;
>>>>>>>> +
>>>>>>>> + sbdf.sbdf = MMCFG_BDF(info->gpa);
>>>>>>>> + reg = REGISTER_OFFSET(info->gpa);
>>>>>>>> +
>>>>>>>> + if ( !vpci_mmio_access_allowed(reg, size) )
>>>>>>>> + return 0;
>>>>>>>> +
>>>>>>>> + vpci_write(sbdf, reg, min(4u, size), data);
>>>>>>>> + if ( size == 8 )
>>>>>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>>>>> I think those two helpers (and vpci_mmio_access_allowed) are very
>>>>>>> similar to the existing x86 ones (see vpci_mmcfg_{read,write}), up to
>>>>>>> the point where I would consider moving the shared code to vpci.c as
>>>>>>> vpci_ecam_{read,write} and call them from the arch specific trap
>>>>>>> handlers.
>>>>>> Except that please can we stick to mcfg or mmcfg instead of ecam
>>>>>> in names, as that's how the thing has been named in Xen from its
>>>>>> introduction? I've just grep-ed the code base (case insensitively)
>>>>>> and found no mention of ECAM. There are only a few "became".
>>>>> I do understand that this is historically that we do not have ECAM in Xen,
>>>>> but PCI is not about Xen. Thus, I think it is also acceptable to use
>>>>> a commonly known ECAM for the code that works with ECAM.
>>>>
>>>> ACPI, afaik, also doesn't call this ECAM. That's where MCFG / MMCFG
>>>> actually come from, I believe.
>>>
>>> My understanding is that "MCFG" is the name of the ACPI table that
>>> describes the PCI config space [1]. The underlying PCI standard for the
>>> memory mapped layout of the PCI config space is called ECAM. Here, it
>>> makes sense to call it ECAM as it is firmware independent.
>>>
>>> [1] https://wiki.osdev.org/PCI_Express
>>
>> Okay, I can accept this, but then I'd expect all existing uses of
>> MCFG / MMCFG in the code which aren't directly ACPI-related to get
>> replaced. Otherwise it's needlessly harder to identify that one
>> piece of code relates to certain other pieces.
>
> If that is ok with I will:
> - move function from x86/hw/io.c to vpci.c renaming them to ECAM
> - rename MMCFG_{BDF/REG_OFFSET) to ECAM_{BDF/REG_OFFSET}
>
> For the rest of the occurrences in x86 I will not modify any of them as some
> are related to ACPI so using (M)MCFG is right and for the others I am not 100%
> sure.
>
> Do you agree ?
Probably not, but I don't want to put time into checking existing
names right now. Yet I'm getting the impression that I'm being
overruled here anyway, so there may be little point in investing any
time here in the first place.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* [PATCH v6 0/3] PCI devices passthrough on Arm
@ 2021-10-14 14:49 ` Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code Bertrand Marquis
` (2 more replies)
0 siblings, 3 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 14:49 UTC (permalink / raw)
To: xen-devel
Cc: iwj, Paul Durrant, Jan Beulich, Andrew Cooper,
Roger Pau Monné,
Wei Liu, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
George Dunlap, Anthony PERARD, Juergen Gross
Hello All,
This serie is a follow up on Rahul serie where we included various fixes
required after review on the mailing list and a new patch to move some
of the x86 ecam related code to the common vpci code.
Most of the patches of the original serie have been merged and this
serie includes only 2 of the original patches reworked and 1 new patch:
Move some ECAM related functions from x86 to generic vpci
implementation:
- move vcpi mmcfg_{read/write} and vpci_access_allowed to common vpci.c.
- use ecam instead of mmcfg in common code.
Enable the existing x86 virtual PCI support for ARM:
- Add VPCI trap handler for each of the PCI device added for config
space access.
- Register the trap handler in XEN for each of the host bridge PCI ECAM
config space access.
Emulated PCI device tree node in libxl:
- Create a virtual PCI device tree node in libxl to enable the guest OS
to discover the virtual PCI during guest boot.
The patch modifying xc_domain_ioport_permission has been removed from
the serie.
Bertrand Marquis (1):
xen/vpci: Move ecam access functions to common code
Rahul Singh (2):
xen/arm: Enable the existing x86 virtual PCI support for ARM.
arm/libxl: Emulated PCI device tree node in libxl
tools/libs/light/libxl_arm.c | 105 +++++++++++++++++++++++++++++++
tools/libs/light/libxl_types.idl | 1 +
xen/arch/arm/Makefile | 1 +
xen/arch/arm/domain.c | 4 ++
xen/arch/arm/vpci.c | 74 ++++++++++++++++++++++
xen/arch/arm/vpci.h | 36 +++++++++++
xen/arch/x86/hvm/io.c | 50 +++------------
xen/drivers/passthrough/pci.c | 18 +++++-
xen/drivers/vpci/header.c | 3 +-
xen/drivers/vpci/vpci.c | 60 ++++++++++++++++++
xen/include/asm-arm/domain.h | 1 +
xen/include/asm-x86/pci.h | 2 -
xen/include/public/arch-arm.h | 17 +++++
xen/include/xen/pci.h | 3 +
xen/include/xen/vpci.h | 10 +++
15 files changed, 338 insertions(+), 47 deletions(-)
create mode 100644 xen/arch/arm/vpci.c
create mode 100644 xen/arch/arm/vpci.h
--
2.25.1
^ permalink raw reply [flat|nested] 190+ messages in thread
* [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-14 14:49 ` [PATCH v6 0/3] PCI devices passthrough on Arm Bertrand Marquis
@ 2021-10-14 14:49 ` Bertrand Marquis
2021-10-14 16:06 ` Jan Beulich
` (2 more replies)
2021-10-14 14:49 ` [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl Bertrand Marquis
2 siblings, 3 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 14:49 UTC (permalink / raw)
To: xen-devel
Cc: iwj, Paul Durrant, Jan Beulich, Andrew Cooper,
Roger Pau Monné,
Wei Liu
PCI standard is using ECAM and not MCFG which is coming from ACPI[1].
Use ECAM/ecam instead of MCFG in common code and in new functions added
in common vpci code by this patch.
Rename vpci_access_allowed to vpci_ecam_access_allowed and move it
from arch/x86/hvm/io.c to drivers/vpci/vpci.c.
Create vpci_ecam_mmio_{read,write} in drivers/vpci/vpci.c that
contains the common code to perform these operations, changed
vpci_mmcfg_{read,write} accordingly to make use of these functions.
The vpci_ecam_mmio_{read,write} functions are returning 0 on error and 1
on success. As the x86 code was previously always returning X86EMUL_OKAY
the return code is ignored. A comment has been added in the code to show
that this is intentional.
Those functions will be used in a following patch inside by arm vpci
implementation.
Rename MMCFG_SBDF to ECAM_SBDF.
Not functional change intended with this patch.
[1] https://wiki.osdev.org/PCI_Express
Suggested-by: Roger Pau Monné <roger.pau@citrix.com>
Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Changes in v6: Patch added
---
xen/arch/x86/hvm/io.c | 50 +++++---------------------------
xen/drivers/vpci/vpci.c | 60 +++++++++++++++++++++++++++++++++++++++
xen/include/asm-x86/pci.h | 2 +-
xen/include/xen/vpci.h | 10 +++++++
4 files changed, 78 insertions(+), 44 deletions(-)
diff --git a/xen/arch/x86/hvm/io.c b/xen/arch/x86/hvm/io.c
index 046a8eb4ed..340b8c8b0e 100644
--- a/xen/arch/x86/hvm/io.c
+++ b/xen/arch/x86/hvm/io.c
@@ -260,20 +260,6 @@ unsigned int hvm_pci_decode_addr(unsigned int cf8, unsigned int addr,
return CF8_ADDR_LO(cf8) | (addr & 3);
}
-/* Do some sanity checks. */
-static bool vpci_access_allowed(unsigned int reg, unsigned int len)
-{
- /* Check access size. */
- if ( len != 1 && len != 2 && len != 4 && len != 8 )
- return false;
-
- /* Check that access is size aligned. */
- if ( (reg & (len - 1)) )
- return false;
-
- return true;
-}
-
/* vPCI config space IO ports handlers (0xcf8/0xcfc). */
static bool vpci_portio_accept(const struct hvm_io_handler *handler,
const ioreq_t *p)
@@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
- if ( !vpci_access_allowed(reg, size) )
+ if ( !vpci_ecam_access_allowed(reg, size) )
return X86EMUL_OKAY;
*data = vpci_read(sbdf, reg, size);
@@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
- if ( !vpci_access_allowed(reg, size) )
+ if ( !vpci_ecam_access_allowed(reg, size) )
return X86EMUL_OKAY;
vpci_write(sbdf, reg, size, data);
@@ -394,7 +380,7 @@ static unsigned int vpci_mmcfg_decode_addr(const struct hvm_mmcfg *mmcfg,
paddr_t addr, pci_sbdf_t *sbdf)
{
addr -= mmcfg->addr;
- sbdf->bdf = MMCFG_BDF(addr);
+ sbdf->bdf = ECAM_BDF(addr);
sbdf->bus += mmcfg->start_bus;
sbdf->seg = mmcfg->segment;
@@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
read_unlock(&d->arch.hvm.mmcfg_lock);
- if ( !vpci_access_allowed(reg, len) ||
- (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
- return X86EMUL_OKAY;
-
- /*
- * According to the PCIe 3.1A specification:
- * - Configuration Reads and Writes must usually be DWORD or smaller
- * in size.
- * - Because Root Complex implementations are not required to support
- * accesses to a RCRB that cross DW boundaries [...] software
- * should take care not to cause the generation of such accesses
- * when accessing a RCRB unless the Root Complex will support the
- * access.
- * Xen however supports 8byte accesses by splitting them into two
- * 4byte accesses.
- */
- *data = vpci_read(sbdf, reg, min(4u, len));
- if ( len == 8 )
- *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
+ /* Ignore return code */
+ vpci_ecam_mmio_read(sbdf, reg, len, data);
return X86EMUL_OKAY;
}
@@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
read_unlock(&d->arch.hvm.mmcfg_lock);
- if ( !vpci_access_allowed(reg, len) ||
- (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
- return X86EMUL_OKAY;
-
- vpci_write(sbdf, reg, min(4u, len), data);
- if ( len == 8 )
- vpci_write(sbdf, reg + 4, 4, data >> 32);
+ /* Ignore return code */
+ vpci_ecam_mmio_write(sbdf, reg, len, data);
return X86EMUL_OKAY;
}
diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
index cbd1bac7fc..c0853176d7 100644
--- a/xen/drivers/vpci/vpci.c
+++ b/xen/drivers/vpci/vpci.c
@@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
spin_unlock(&pdev->vpci->lock);
}
+/* Helper function to check an access size and alignment on vpci space. */
+bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
+{
+ /*
+ * Check access size.
+ *
+ * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
+ * but as for those platform ISV register, which gives the access size,
+ * cannot have a value 3, checking this would just harden the code.
+ */
+ if ( len != 1 && len != 2 && len != 4 && len != 8 )
+ return false;
+
+ /* Check that access is size aligned. */
+ if ( (reg & (len - 1)) )
+ return false;
+
+ return true;
+}
+
+int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
+ unsigned long data)
+{
+ if ( !vpci_ecam_access_allowed(reg, len) ||
+ (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
+ return 0;
+
+ vpci_write(sbdf, reg, min(4u, len), data);
+ if ( len == 8 )
+ vpci_write(sbdf, reg + 4, 4, data >> 32);
+
+ return 1;
+}
+
+int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
+ unsigned long *data)
+{
+ if ( !vpci_ecam_access_allowed(reg, len) ||
+ (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
+ return 0;
+
+ /*
+ * According to the PCIe 3.1A specification:
+ * - Configuration Reads and Writes must usually be DWORD or smaller
+ * in size.
+ * - Because Root Complex implementations are not required to support
+ * accesses to a RCRB that cross DW boundaries [...] software
+ * should take care not to cause the generation of such accesses
+ * when accessing a RCRB unless the Root Complex will support the
+ * access.
+ * Xen however supports 8byte accesses by splitting them into two
+ * 4byte accesses.
+ */
+ *data = vpci_read(sbdf, reg, min(4u, len));
+ if ( len == 8 )
+ *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
+
+ return 1;
+}
+
/*
* Local variables:
* mode: C
diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
index edd7c3e71a..a0df5c1279 100644
--- a/xen/include/asm-x86/pci.h
+++ b/xen/include/asm-x86/pci.h
@@ -6,7 +6,7 @@
#define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
#define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
-#define MMCFG_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
+#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
#define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
|| id == 0x01268086 || id == 0x01028086 \
diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h
index 9f5b5d52e1..4a0c3d77c9 100644
--- a/xen/include/xen/vpci.h
+++ b/xen/include/xen/vpci.h
@@ -208,6 +208,16 @@ static inline unsigned int vmsix_entry_nr(const struct vpci_msix *msix,
{
return entry - msix->entries;
}
+
+/* ECAM mmio read/write helpers */
+bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len);
+
+int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
+ unsigned long data);
+
+int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
+ unsigned long *data);
+
#endif /* __XEN__ */
#else /* !CONFIG_HAS_VPCI */
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 14:49 ` [PATCH v6 0/3] PCI devices passthrough on Arm Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code Bertrand Marquis
@ 2021-10-14 14:49 ` Bertrand Marquis
2021-10-14 23:49 ` Stefano Stabellini
` (2 more replies)
2021-10-14 14:49 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl Bertrand Marquis
2 siblings, 3 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 14:49 UTC (permalink / raw)
To: xen-devel
Cc: iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Jan Beulich,
Wei Liu, Paul Durrant, Roger Pau Monné
From: Rahul Singh <rahul.singh@arm.com>
The existing VPCI support available for X86 is adapted for Arm.
When the device is added to XEN via the hyper call
“PHYSDEVOP_pci_device_add”, VPCI handler for the config space
access is added to the Xen to emulate the PCI devices config space.
A MMIO trap handler for the PCI ECAM space is registered in XEN
so that when guest is trying to access the PCI config space,XEN
will trap the access and emulate read/write using the VPCI and
not the real PCI hardware.
For Dom0less systems scan_pci_devices() would be used to discover the
PCI device in XEN and VPCI handler will be added during XEN boots.
This patch is also doing some small fixes to fix compilation errors on
arm32 of vpci:
- add a cast to unsigned long in print in header.c
- add a cast to uint64_t in vpci_ecam_mmio_write
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
---
Changes in v6:
- Use new vpci_ecam_ helpers for PCI access
- Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
future patch once everything is ready)
- rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
- remove not needed local variables in vpci_mmio_write, the one in read
has been kept to ensure proper compilation on arm32
- move call to vpci_add_handlers before iommu init to simplify exit path
- move call to pci_cleanup_msi in the out section of pci_add_device if
pdev is not NULL and on error
- initialize pdev to NULL to handle properly exit path call of
pci_cleanup_msi
- keep has_vpci to return false for now as CFG_vpci has been removed.
Added a comment on top of the definition.
- fix compilation errors on arm32 (print in header.c, cast missing in
mmio_write.
- local variable was kept in vpci_mmio_read on arm to prevent a cast
error in arm32.
Change in v5:
- Add pci_cleanup_msi(pdev) incleanup part.
- Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Change in v4:
- Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
Change in v3:
- Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
- Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
- Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
Change in v2:
- Add new XEN_DOMCTL_CDF_vpci flag
- modify has_vpci() to include XEN_DOMCTL_CDF_vpci
- enable vpci support when pci-passthough option is enabled.
---
---
xen/arch/arm/Makefile | 1 +
xen/arch/arm/domain.c | 4 ++
xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
xen/arch/arm/vpci.h | 36 +++++++++++++++++
xen/drivers/passthrough/pci.c | 18 ++++++++-
xen/drivers/vpci/header.c | 3 +-
xen/drivers/vpci/vpci.c | 2 +-
xen/include/asm-arm/domain.h | 1 +
xen/include/asm-x86/pci.h | 2 -
xen/include/public/arch-arm.h | 7 ++++
xen/include/xen/pci.h | 3 ++
11 files changed, 146 insertions(+), 5 deletions(-)
create mode 100644 xen/arch/arm/vpci.c
create mode 100644 xen/arch/arm/vpci.h
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 64518848b2..07f634508e 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_HAS_VPCI) += vpci.o
obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index eef0661beb..96e1b23550 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -39,6 +39,7 @@
#include <asm/vgic.h>
#include <asm/vtimer.h>
+#include "vpci.h"
#include "vuart.h"
DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
@@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
goto fail;
+ if ( (rc = domain_vpci_init(d)) != 0 )
+ goto fail;
+
return 0;
fail:
diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
new file mode 100644
index 0000000000..7c3552b65d
--- /dev/null
+++ b/xen/arch/arm/vpci.c
@@ -0,0 +1,74 @@
+/*
+ * xen/arch/arm/vpci.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <xen/sched.h>
+#include <xen/vpci.h>
+
+#include <asm/mmio.h>
+
+static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
+ register_t *r, void *p)
+{
+ pci_sbdf_t sbdf;
+ /* data is needed to prevent a pointer cast on 32bit */
+ unsigned long data = ~0ul;
+ int ret;
+
+ /* We ignore segment part and always handle segment 0 */
+ sbdf.sbdf = ECAM_BDF(info->gpa);
+
+ ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
+ 1U << info->dabt.size, &data);
+
+ *r = data;
+
+ return ret;
+}
+
+static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
+ register_t r, void *p)
+{
+ pci_sbdf_t sbdf;
+
+ /* We ignore segment part and always handle segment 0 */
+ sbdf.sbdf = ECAM_BDF(info->gpa);
+
+ return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
+ 1U << info->dabt.size, r);
+}
+
+static const struct mmio_handler_ops vpci_mmio_handler = {
+ .read = vpci_mmio_read,
+ .write = vpci_mmio_write,
+};
+
+int domain_vpci_init(struct domain *d)
+{
+ if ( !has_vpci(d) )
+ return 0;
+
+ register_mmio_handler(d, &vpci_mmio_handler,
+ GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
+
+ return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
new file mode 100644
index 0000000000..d8a7b0e3e8
--- /dev/null
+++ b/xen/arch/arm/vpci.h
@@ -0,0 +1,36 @@
+/*
+ * xen/arch/arm/vpci.h
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPCI_H__
+#define __ARCH_ARM_VPCI_H__
+
+#ifdef CONFIG_HAS_VPCI
+int domain_vpci_init(struct domain *d);
+#else
+static inline int domain_vpci_init(struct domain *d)
+{
+ return 0;
+}
+#endif
+
+#endif /* __ARCH_ARM_VPCI_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 3aa8c3175f..8cc529ecec 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
const struct pci_dev_info *info, nodeid_t node)
{
struct pci_seg *pseg;
- struct pci_dev *pdev;
+ struct pci_dev *pdev = NULL;
unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
const char *pdev_type;
int ret;
@@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
check_pdev(pdev);
+#ifdef CONFIG_ARM
+ /*
+ * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
+ * Dom0 inform XEN to add the PCI devices in XEN.
+ */
+ ret = vpci_add_handlers(pdev);
+ if ( ret )
+ {
+ printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
+ goto out;
+ }
+#endif
+
ret = 0;
if ( !pdev->domain )
{
@@ -784,6 +797,9 @@ out:
&PCI_SBDF(seg, bus, slot, func));
}
}
+ else if ( pdev )
+ pci_cleanup_msi(pdev);
+
return ret;
}
diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
index f8cd55e7c0..c5b025b88b 100644
--- a/xen/drivers/vpci/header.c
+++ b/xen/drivers/vpci/header.c
@@ -374,7 +374,8 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
if ( val != (uint32_t)(bar->addr >> (hi ? 32 : 0)) )
gprintk(XENLOG_WARNING,
"%pp: ignored BAR %lu write with memory decoding enabled\n",
- &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
+ &pdev->sbdf,
+ (unsigned long)(bar - pdev->vpci->header.bars + hi));
return;
}
diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
index c0853176d7..2bd67fc27a 100644
--- a/xen/drivers/vpci/vpci.c
+++ b/xen/drivers/vpci/vpci.c
@@ -507,7 +507,7 @@ int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
vpci_write(sbdf, reg, min(4u, len), data);
if ( len == 8 )
- vpci_write(sbdf, reg + 4, 4, data >> 32);
+ vpci_write(sbdf, reg + 4, 4, (uint64_t)data >> 32);
return 1;
}
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 14e575288f..9b3647587a 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -263,6 +263,7 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
#define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
+/* vPCI is not available on Arm */
#define has_vpci(d) ({ (void)(d); false; })
#endif /* __ASM_DOMAIN_H__ */
diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
index a0df5c1279..443f25347d 100644
--- a/xen/include/asm-x86/pci.h
+++ b/xen/include/asm-x86/pci.h
@@ -6,8 +6,6 @@
#define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
#define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
-#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
-
#define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
|| id == 0x01268086 || id == 0x01028086 \
|| id == 0x01128086 || id == 0x01228086 \
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index d46c61fca9..44be337dec 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
#define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
#define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
+/*
+ * 256 MB is reserved for VPCI configuration space based on calculation
+ * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
+ */
+#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
+#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
+
/* ACPI tables physical address */
#define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
#define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index 70ac25345c..db18cb7639 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -40,6 +40,9 @@
#define PCI_SBDF3(s,b,df) \
((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
+#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
+#define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
+
typedef union {
uint32_t sbdf;
struct {
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl
2021-10-14 14:49 ` [PATCH v6 0/3] PCI devices passthrough on Arm Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM Bertrand Marquis
@ 2021-10-14 14:49 ` Bertrand Marquis
2021-10-14 17:54 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages] Ian Jackson
2 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 14:49 UTC (permalink / raw)
To: xen-devel
Cc: iwj, Rahul Singh, Wei Liu, Anthony PERARD, Juergen Gross,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Michal Orzel
From: Rahul Singh <rahul.singh@arm.com>
libxl will create an emulated PCI device tree node in the device tree to
enable the guest OS to discover the virtual PCI during guest boot.
Emulated PCI device tree node will only be created when there is any
device assigned to guest.
A new area has been reserved in the arm guest physical map at
which the VPCI bus is declared in the device tree (reg and ranges
parameters of the node).
b_info.arch_arm.vpci is set by default to false
and it is not set to true anywhere. This way the
vpci node is not going to be created and we are only
introducing functions to create vpci DT node to be used
in the future.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Michal Orzel <michal.orzel@arm.com>
---
Changes in v6:
According to https://marc.info/?l=xen-devel&m=163415838129479&w=2:
-do not set XEN_DOMCTL_CDF_vpci
-do not enable VPCI support (by setting b_info.arch_arm.vpci)
-do not define LIBXL_HAVE_BUILDINFO_ARM_VPCI
-keep b_info.arch_arm.vpci, make_vpci_node and its helpers
Change in v5:
- Move setting the arch_arm.vpci and XEN_DOMCTL_CDF_vpci to libxl_arm.c
Change in v4:
- Gate code for x86 for setting the XEN_DOMCTL_CDF_vpci for x86.
Change in v3:
- Make GUEST_VPCI_MEM_ADDR address 2MB aligned
Change in v2:
- enable doamin_vpci_init() when XEN_DOMCTL_CDF_vpci is set for domain.
---
tools/libs/light/libxl_arm.c | 105 +++++++++++++++++++++++++++++++
tools/libs/light/libxl_types.idl | 1 +
xen/include/public/arch-arm.h | 10 +++
3 files changed, 116 insertions(+)
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index e3140a6e00..52f1ddce48 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
return fdt_property(fdt, "reg", regs, sizeof(regs));
}
+static int fdt_property_values(libxl__gc *gc, void *fdt,
+ const char *name, unsigned num_cells, ...)
+{
+ uint32_t prop[num_cells];
+ be32 *cells = &prop[0];
+ int i;
+ va_list ap;
+ uint32_t arg;
+
+ va_start(ap, num_cells);
+ for (i = 0 ; i < num_cells; i++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1, arg);
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, name, prop, sizeof(prop));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+ unsigned addr_cells,
+ unsigned size_cells,
+ unsigned num_regs, ...)
+{
+ uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
+ be32 *cells = ®s[0];
+ int i;
+ va_list ap;
+ uint64_t arg;
+
+ va_start(ap, num_regs);
+ for (i = 0 ; i < num_regs; i++) {
+ /* Set the memory bit field */
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1, arg);
+
+ /* Set the vpci bus address */
+ arg = addr_cells ? va_arg(ap, uint64_t) : 0;
+ set_cell(&cells, addr_cells , arg);
+
+ /* Set the cpu bus address where vpci address is mapped */
+ set_cell(&cells, addr_cells, arg);
+
+ /* Set the vpci size requested */
+ arg = size_cells ? va_arg(ap, uint64_t) : 0;
+ set_cell(&cells, size_cells, arg);
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
static int make_root_properties(libxl__gc *gc,
const libxl_version_info *vers,
void *fdt)
@@ -668,6 +720,53 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
return 0;
}
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+ const struct arch_info *ainfo,
+ struct xc_dom_image *dom)
+{
+ int res;
+ const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+ const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+ const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+ res = fdt_begin_node(fdt, name);
+ if (res) return res;
+
+ res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+ if (res) return res;
+
+ res = fdt_property_string(fdt, "device_type", "pci");
+ if (res) return res;
+
+ res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+ GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+ if (res) return res;
+
+ res = fdt_property_values(gc, fdt, "bus-range", 2, 0, 255);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#address-cells", 3);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#size-cells", 2);
+ if (res) return res;
+
+ res = fdt_property_string(fdt, "status", "okay");
+ if (res) return res;
+
+ res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+ GUEST_ROOT_SIZE_CELLS, 2,
+ GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE,
+ GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR,
+ GUEST_VPCI_PREFETCH_MEM_SIZE);
+ if (res) return res;
+
+ res = fdt_end_node(fdt);
+ if (res) return res;
+
+ return 0;
+}
+
static const struct arch_info *get_arch_info(libxl__gc *gc,
const struct xc_dom_image *dom)
{
@@ -971,6 +1070,9 @@ next_resize:
if (info->tee == LIBXL_TEE_TYPE_OPTEE)
FDT( make_optee_node(gc, fdt) );
+ if (libxl_defbool_val(info->arch_arm.vpci))
+ FDT( make_vpci_node(gc, fdt, ainfo, dom) );
+
if (pfdt)
FDT( copy_partial_fdt(gc, fdt, pfdt) );
@@ -1189,6 +1291,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
/* ACPI is disabled by default */
libxl_defbool_setdefault(&b_info->acpi, false);
+ /* VPCI is disabled by default */
+ libxl_defbool_setdefault(&b_info->arch_arm.vpci, false);
+
if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
return;
diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index b96fb5c47e..61418ef6eb 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
("vuart", libxl_vuart_type),
+ ("vpci", libxl_defbool),
])),
("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
])),
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 44be337dec..45aac5d18f 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -433,6 +433,11 @@ typedef uint64_t xen_callback_t;
#define GUEST_PL011_BASE xen_mk_ullong(0x22000000)
#define GUEST_PL011_SIZE xen_mk_ullong(0x00001000)
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000)
+#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x23000000)
+#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x10000000)
+
/*
* 16MB == 4096 pages reserved for guest to use as a region to map its
* grant table in.
@@ -448,6 +453,11 @@ typedef uint64_t xen_callback_t;
#define GUEST_RAM0_BASE xen_mk_ullong(0x40000000) /* 3GB of low RAM @ 1GB */
#define GUEST_RAM0_SIZE xen_mk_ullong(0xc0000000)
+/* 4GB @ 4GB Prefetch Memory for VPCI */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x100000000)
+#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000)
+
#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 1016GB of RAM @ 8GB */
#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000)
--
2.25.1
^ permalink raw reply related [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-14 14:49 ` [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code Bertrand Marquis
@ 2021-10-14 16:06 ` Jan Beulich
2021-10-14 17:09 ` Bertrand Marquis
2021-10-14 23:47 ` Stefano Stabellini
2021-10-15 7:44 ` Roger Pau Monné
2 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-14 16:06 UTC (permalink / raw)
To: Bertrand Marquis
Cc: iwj, Paul Durrant, Andrew Cooper, Roger Pau Monné,
Wei Liu, xen-devel
On 14.10.2021 16:49, Bertrand Marquis wrote:
> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>
> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>
> - if ( !vpci_access_allowed(reg, size) )
> + if ( !vpci_ecam_access_allowed(reg, size) )
> return X86EMUL_OKAY;
>
> *data = vpci_read(sbdf, reg, size);
> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>
> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>
> - if ( !vpci_access_allowed(reg, size) )
> + if ( !vpci_ecam_access_allowed(reg, size) )
> return X86EMUL_OKAY;
>
> vpci_write(sbdf, reg, size, data);
Why would port I/O functions call an ECAM helper? And in how far is
that helper actually ECAM-specific?
> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
> read_unlock(&d->arch.hvm.mmcfg_lock);
>
> - if ( !vpci_access_allowed(reg, len) ||
> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> - return X86EMUL_OKAY;
While I assume this earlier behavior is the reason for ...
> - /*
> - * According to the PCIe 3.1A specification:
> - * - Configuration Reads and Writes must usually be DWORD or smaller
> - * in size.
> - * - Because Root Complex implementations are not required to support
> - * accesses to a RCRB that cross DW boundaries [...] software
> - * should take care not to cause the generation of such accesses
> - * when accessing a RCRB unless the Root Complex will support the
> - * access.
> - * Xen however supports 8byte accesses by splitting them into two
> - * 4byte accesses.
> - */
> - *data = vpci_read(sbdf, reg, min(4u, len));
> - if ( len == 8 )
> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> + /* Ignore return code */
> + vpci_ecam_mmio_read(sbdf, reg, len, data);
... the commented-upon ignoring of the return value, I don't think
that's a good way to deal with things anymore. Instead I think
*data should be written to ~0 upon failure, unless it is intended
for vpci_ecam_mmio_read() to take care of that case (in which case
I'm not sure I would see why it needs to return an error indicator
in the first place).
> @@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
> read_unlock(&d->arch.hvm.mmcfg_lock);
>
> - if ( !vpci_access_allowed(reg, len) ||
> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> - return X86EMUL_OKAY;
> -
> - vpci_write(sbdf, reg, min(4u, len), data);
> - if ( len == 8 )
> - vpci_write(sbdf, reg + 4, 4, data >> 32);
> + /* Ignore return code */
> + vpci_ecam_mmio_write(sbdf, reg, len, data);
Here ignoring is fine imo, but if you feel it is important to
comment on this, then I think you need to prefer "why" over "what".
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
> spin_unlock(&pdev->vpci->lock);
> }
>
> +/* Helper function to check an access size and alignment on vpci space. */
> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
> +{
> + /*
> + * Check access size.
> + *
> + * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
> + * but as for those platform ISV register, which gives the access size,
> + * cannot have a value 3, checking this would just harden the code.
> + */
> + if ( len != 1 && len != 2 && len != 4 && len != 8 )
> + return false;
I'm not convinced talking about Arm specifically here is
warranted, unless there's something there that's clearly
different from all other architectures. Otherwise the comment
should imo be written in more general terms.
> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
> + unsigned long data)
> +{
> + if ( !vpci_ecam_access_allowed(reg, len) ||
> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> + return 0;
> +
> + vpci_write(sbdf, reg, min(4u, len), data);
> + if ( len == 8 )
> + vpci_write(sbdf, reg + 4, 4, data >> 32);
> +
> + return 1;
> +}
> +
> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
> + unsigned long *data)
> +{
> + if ( !vpci_ecam_access_allowed(reg, len) ||
> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> + return 0;
> +
> + /*
> + * According to the PCIe 3.1A specification:
> + * - Configuration Reads and Writes must usually be DWORD or smaller
> + * in size.
> + * - Because Root Complex implementations are not required to support
> + * accesses to a RCRB that cross DW boundaries [...] software
> + * should take care not to cause the generation of such accesses
> + * when accessing a RCRB unless the Root Complex will support the
> + * access.
> + * Xen however supports 8byte accesses by splitting them into two
> + * 4byte accesses.
> + */
> + *data = vpci_read(sbdf, reg, min(4u, len));
> + if ( len == 8 )
> + *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> +
> + return 1;
> +}
Why do these two functions return int/0/1 instead of
bool/false/true (assuming, as per above, that them returning non-
void is warranted at all)?
Also both of these functions will silently misbehave on 32-bit due to
the use of unsigned long in the parameter types. I think you want e.g.
CONFIG_64BIT conditionals here as well as in vpci_access_allowed()
(omitting the questionable "ecam" part of the name) to reject len == 8
there in that case.
Finally, to me, having both "ecam" and "mmio" in the names feels
redundant - the PCI spec doesn't mention any non-MMIO mechanism there
afaics.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-14 16:06 ` Jan Beulich
@ 2021-10-14 17:09 ` Bertrand Marquis
2021-10-15 6:29 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 17:09 UTC (permalink / raw)
To: Jan Beulich
Cc: Ian Jackson, Paul Durrant, Andrew Cooper, Roger Pau Monné,
Wei Liu, xen-devel
Hi Jan,
> On 14 Oct 2021, at 17:06, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 14.10.2021 16:49, Bertrand Marquis wrote:
>> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>>
>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>
>> - if ( !vpci_access_allowed(reg, size) )
>> + if ( !vpci_ecam_access_allowed(reg, size) )
>> return X86EMUL_OKAY;
>>
>> *data = vpci_read(sbdf, reg, size);
>> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>>
>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>
>> - if ( !vpci_access_allowed(reg, size) )
>> + if ( !vpci_ecam_access_allowed(reg, size) )
>> return X86EMUL_OKAY;
>>
>> vpci_write(sbdf, reg, size, data);
>
> Why would port I/O functions call an ECAM helper? And in how far is
> that helper actually ECAM-specific?
The function was global before.
>
>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>
>> - if ( !vpci_access_allowed(reg, len) ||
>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>> - return X86EMUL_OKAY;
>
> While I assume this earlier behavior is the reason for ...
Yes :-)
>
>> - /*
>> - * According to the PCIe 3.1A specification:
>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>> - * in size.
>> - * - Because Root Complex implementations are not required to support
>> - * accesses to a RCRB that cross DW boundaries [...] software
>> - * should take care not to cause the generation of such accesses
>> - * when accessing a RCRB unless the Root Complex will support the
>> - * access.
>> - * Xen however supports 8byte accesses by splitting them into two
>> - * 4byte accesses.
>> - */
>> - *data = vpci_read(sbdf, reg, min(4u, len));
>> - if ( len == 8 )
>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>> + /* Ignore return code */
>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>
> ... the commented-upon ignoring of the return value, I don't think
> that's a good way to deal with things anymore. Instead I think
> *data should be written to ~0 upon failure, unless it is intended
> for vpci_ecam_mmio_read() to take care of that case (in which case
> I'm not sure I would see why it needs to return an error indicator
> in the first place).
I am not sure in the first place why this is actually ignored and just
returning a -1 value.
If an access is not right, an exception should be generated to the
Guest instead.
When we do that on arm the function is returning an error to the upper
layer in that case, that’s why I did keep a generic function informing the
caller.
So I think it is right for the function to return an error if the access is not allowed but I agree the comment on x86 could get a better justification.
@Roger: could you help finding one here as I do not quite understand why it is ok to ignore this case ?
>
>> @@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>
>> - if ( !vpci_access_allowed(reg, len) ||
>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>> - return X86EMUL_OKAY;
>> -
>> - vpci_write(sbdf, reg, min(4u, len), data);
>> - if ( len == 8 )
>> - vpci_write(sbdf, reg + 4, 4, data >> 32);
>> + /* Ignore return code */
>> + vpci_ecam_mmio_write(sbdf, reg, len, data);
>
> Here ignoring is fine imo, but if you feel it is important to
> comment on this, then I think you need to prefer "why" over "what".
Agree I would just need some help on the why.
Now there was no comment before to explain why so I could also
remove the comment altogether.
>
>> --- a/xen/drivers/vpci/vpci.c
>> +++ b/xen/drivers/vpci/vpci.c
>> @@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
>> spin_unlock(&pdev->vpci->lock);
>> }
>>
>> +/* Helper function to check an access size and alignment on vpci space. */
>> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> + /*
>> + * Check access size.
>> + *
>> + * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
>> + * but as for those platform ISV register, which gives the access size,
>> + * cannot have a value 3, checking this would just harden the code.
>> + */
>> + if ( len != 1 && len != 2 && len != 4 && len != 8 )
>> + return false;
>
> I'm not convinced talking about Arm specifically here is
> warranted, unless there's something there that's clearly
> different from all other architectures. Otherwise the comment
> should imo be written in more general terms.
Other architectures might allow this case. So this is specific to Arm.
>
>> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>> + unsigned long data)
>> +{
>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>> + return 0;
>> +
>> + vpci_write(sbdf, reg, min(4u, len), data);
>> + if ( len == 8 )
>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>> +
>> + return 1;
>> +}
>> +
>> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>> + unsigned long *data)
>> +{
>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>> + return 0;
>> +
>> + /*
>> + * According to the PCIe 3.1A specification:
>> + * - Configuration Reads and Writes must usually be DWORD or smaller
>> + * in size.
>> + * - Because Root Complex implementations are not required to support
>> + * accesses to a RCRB that cross DW boundaries [...] software
>> + * should take care not to cause the generation of such accesses
>> + * when accessing a RCRB unless the Root Complex will support the
>> + * access.
>> + * Xen however supports 8byte accesses by splitting them into two
>> + * 4byte accesses.
>> + */
>> + *data = vpci_read(sbdf, reg, min(4u, len));
>> + if ( len == 8 )
>> + *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>> +
>> + return 1;
>> +}
>
> Why do these two functions return int/0/1 instead of
> bool/false/true (assuming, as per above, that them returning non-
> void is warranted at all)?
This is what the mmio handlers should return to say that an access
was ok or not so the function stick to this standard.
>
> Also both of these functions will silently misbehave on 32-bit due to
> the use of unsigned long in the parameter types. I think you want e.g.
> CONFIG_64BIT conditionals here as well as in vpci_access_allowed()
> (omitting the questionable "ecam" part of the name) to reject len == 8
> there in that case.
Right using CONFIG_64BIT is a good idea here in fact.
I can do that in the next version.
>
> Finally, to me, having both "ecam" and "mmio" in the names feels
> redundant - the PCI spec doesn't mention any non-MMIO mechanism there
> afaics.
I was thinking the mmio here stands more for “mmio handler” as to function
handling emulation through mmio access.
I have no objection to remove the mmio part though.
Regards
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl
2021-10-07 16:11 ` Ian Jackson
2021-10-11 13:46 ` Roger Pau Monné
@ 2021-10-14 17:16 ` Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 0/3] PCI devices passthrough on Arm Bertrand Marquis
1 sibling, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-14 17:16 UTC (permalink / raw)
To: Ian Jackson
Cc: Rahul Singh, Julien Grall, xen-devel, Andre Przywara, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk
Hi Ian,
> On 7 Oct 2021, at 17:11, Ian Jackson <iwj@xenproject.org> wrote:
>
> Rahul Singh writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
>> As Stefano suggested in another email that we can remove the vpci
>> option, if we reach to conclusion that we need vpci option I will
>> move it to internal structure.
> ...
>> Yes I agree with you VPCI is necessary for hot plugged PCI device
>> and once we implement the hotplug in future we will use the
>> passthrough= option to enable VPCI.
>
> So, to summarise, I think the situation is:
>
> * VCPI is necessry for passthrough on ARM, whether coldplug or
> hotplug. It's part of the way that PCI-PT works on ARM.
>
> * Hotplug is not yet implemented.
>
> * VPCI is not necessary on x86 (evidently, since we don't have it
> there but we do have passthrough).
>
> So when hotplug is added, vpci will need to be turned on when
> passthrough=yes is selected. I don't fully understand the other
> possible values for passthrough= but maybe we can defer the question
> of whether they apply to ARM ?
>
> I think that means that yes, this should be an internal variable.
> Probably in libxl__domain_create_state. We don't currently arrange to
> elide arch-specific state in there, so perhaps it's fine just to
> invent a member called `arm_vpci`.
>
> Maybe you could leave a comment somewhere so that if and when PCI PT
> hotplug is implemented for ARM, the implementor remembers to wire this
> up.
Sorry for missing this on the v6 serie.
Now you suggest to add a new field arm_vpci in libxl__domain_create_state.
Once we have done that I will need to access this structure to know if I need
to add the DT part and somehow to give it a value depending something which
for now would the number of pcidevs as there will be no user parameter anymore.
I had quite an understanding of the solution using libxl_domain_config and changing
The arguments of libxl__arch_domain_init_hw_description and libxl__prepare_dtb
Suggested by Stefano but I am a bit lost in this solution.
The following might be a stupid question but I did not dig a lot in libxl so:
If we add a parameter in the state structure how should we access it ?
Thank
Bertrand
>
> Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-14 14:49 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl Bertrand Marquis
@ 2021-10-14 17:54 ` Ian Jackson
2021-10-14 23:50 ` Stefano Stabellini
2021-10-15 7:28 ` Julien Grall
0 siblings, 2 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-14 17:54 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, iwj, Rahul Singh, Wei Liu, Anthony PERARD,
Juergen Gross, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Michal Orzel, Andre Przywara
(Replying to both the earlier subthread on v5 and to the new v6
patch.)
Bertrand Marquis writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
> Now you suggest to add a new field arm_vpci in libxl__domain_create_state.
Hi. I was handwaving, hence "probably" :-). I hadn't actually looked
at the existing code to see precisely how it would fit.
> Once we have done that I will need to access this structure to know if I need
> to add the DT part and somehow to give it a value depending something which
> for now would the number of pcidevs as there will be no user parameter anymore.
Right.
I looked at libxl_arm.c again:
It seems that the main entrypoint to all this is libxl__prepare_dtb,
and it is that function you want to do new stuff. That gets
libxl_domain_build_info (which is specified by the IDL and comes from
outside libxl, subject to libxl's default-filling machinery) and
libxl__domain_build_state (which is the general state struct).
The information you need is is libxl_domain_create_info.
libxl__domain_config_setdefault already arranges to set
c_info->passthrough based on the number of PCI PT devices
(search for `need_pt` in libxl_create.c).
That is, as I understand it on ARM vpci should be enabled if
passthrough is enabled and not otherwise. That is precisely what
the variable c_info->passthrough is.
There is a slight issue because of the distinction between create_info
and build_info and domain_config (and, the corresponding _state)
structs. Currently the DT code ony gets b_info, not the whole
libxl_domain_config. These distinctions largely historical nowadays.
Certainly there is no reason not to pass a pointer to the whole
libxl_domain_config, rather than just libxl_domain_build_info, into
libxl__arch_domain_init_hw_description.
So I think the right approach for this looks something like this:
1. Change libxl__arch_domain_init_hw_description to take
libxl_domain_config* rather than libxl_domain_build_info*.
libxl_domain_config contains libxl_domain_build_info so
this is easy.
If you put in a convenience alias variable for the
libxl_domain_build_info* you can avoid extra typing in the function
body. (If you call the convenience alias `info` you won't need to
change the body at all, but maybe `info` isn't the best name so you
could rename it to `b_info` maybe; up to you.)
2. Make the same change to libxl__prepare_dtb.
3. Now you can use d_config->c_info.passthrough to gate the addition
of the additional stuff to the DT. Ie, that is a respin of this
patch 3/3.
1 and 2 need to be separated out from 3, in a different patch (or two
different patches) added to this series before what is now 3/3. Ie
1+2, or 1 and 2 separately, should be pure plumbing changes with no
functional change.
I hope this is helpful.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-14 14:49 ` [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code Bertrand Marquis
2021-10-14 16:06 ` Jan Beulich
@ 2021-10-14 23:47 ` Stefano Stabellini
2021-10-15 7:44 ` Roger Pau Monné
2 siblings, 0 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-14 23:47 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, iwj, Paul Durrant, Jan Beulich, Andrew Cooper,
Roger Pau Monné,
Wei Liu
[-- Attachment #1: Type: text/plain, Size: 9108 bytes --]
On Thu, 14 Oct 2021, Bertrand Marquis wrote:
> PCI standard is using ECAM and not MCFG which is coming from ACPI[1].
> Use ECAM/ecam instead of MCFG in common code and in new functions added
> in common vpci code by this patch.
>
> Rename vpci_access_allowed to vpci_ecam_access_allowed and move it
> from arch/x86/hvm/io.c to drivers/vpci/vpci.c.
>
> Create vpci_ecam_mmio_{read,write} in drivers/vpci/vpci.c that
> contains the common code to perform these operations, changed
> vpci_mmcfg_{read,write} accordingly to make use of these functions.
>
> The vpci_ecam_mmio_{read,write} functions are returning 0 on error and 1
> on success. As the x86 code was previously always returning X86EMUL_OKAY
> the return code is ignored. A comment has been added in the code to show
> that this is intentional.
>
> Those functions will be used in a following patch inside by arm vpci
> implementation.
>
> Rename MMCFG_SBDF to ECAM_SBDF.
>
> Not functional change intended with this patch.
>
> [1] https://wiki.osdev.org/PCI_Express
>
> Suggested-by: Roger Pau Monné <roger.pau@citrix.com>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
Everything checks out:
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
FYI I like the new naming scheme and I think it is the right one for us
to use, but either way for me it wouldn't be a blocker for acceptance.
> ---
> Changes in v6: Patch added
> ---
> xen/arch/x86/hvm/io.c | 50 +++++---------------------------
> xen/drivers/vpci/vpci.c | 60 +++++++++++++++++++++++++++++++++++++++
> xen/include/asm-x86/pci.h | 2 +-
> xen/include/xen/vpci.h | 10 +++++++
> 4 files changed, 78 insertions(+), 44 deletions(-)
>
> diff --git a/xen/arch/x86/hvm/io.c b/xen/arch/x86/hvm/io.c
> index 046a8eb4ed..340b8c8b0e 100644
> --- a/xen/arch/x86/hvm/io.c
> +++ b/xen/arch/x86/hvm/io.c
> @@ -260,20 +260,6 @@ unsigned int hvm_pci_decode_addr(unsigned int cf8, unsigned int addr,
> return CF8_ADDR_LO(cf8) | (addr & 3);
> }
>
> -/* Do some sanity checks. */
> -static bool vpci_access_allowed(unsigned int reg, unsigned int len)
> -{
> - /* Check access size. */
> - if ( len != 1 && len != 2 && len != 4 && len != 8 )
> - return false;
> -
> - /* Check that access is size aligned. */
> - if ( (reg & (len - 1)) )
> - return false;
> -
> - return true;
> -}
> -
> /* vPCI config space IO ports handlers (0xcf8/0xcfc). */
> static bool vpci_portio_accept(const struct hvm_io_handler *handler,
> const ioreq_t *p)
> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>
> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>
> - if ( !vpci_access_allowed(reg, size) )
> + if ( !vpci_ecam_access_allowed(reg, size) )
> return X86EMUL_OKAY;
>
> *data = vpci_read(sbdf, reg, size);
> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>
> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>
> - if ( !vpci_access_allowed(reg, size) )
> + if ( !vpci_ecam_access_allowed(reg, size) )
> return X86EMUL_OKAY;
>
> vpci_write(sbdf, reg, size, data);
> @@ -394,7 +380,7 @@ static unsigned int vpci_mmcfg_decode_addr(const struct hvm_mmcfg *mmcfg,
> paddr_t addr, pci_sbdf_t *sbdf)
> {
> addr -= mmcfg->addr;
> - sbdf->bdf = MMCFG_BDF(addr);
> + sbdf->bdf = ECAM_BDF(addr);
> sbdf->bus += mmcfg->start_bus;
> sbdf->seg = mmcfg->segment;
>
> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
> read_unlock(&d->arch.hvm.mmcfg_lock);
>
> - if ( !vpci_access_allowed(reg, len) ||
> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> - return X86EMUL_OKAY;
> -
> - /*
> - * According to the PCIe 3.1A specification:
> - * - Configuration Reads and Writes must usually be DWORD or smaller
> - * in size.
> - * - Because Root Complex implementations are not required to support
> - * accesses to a RCRB that cross DW boundaries [...] software
> - * should take care not to cause the generation of such accesses
> - * when accessing a RCRB unless the Root Complex will support the
> - * access.
> - * Xen however supports 8byte accesses by splitting them into two
> - * 4byte accesses.
> - */
> - *data = vpci_read(sbdf, reg, min(4u, len));
> - if ( len == 8 )
> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> + /* Ignore return code */
> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>
> return X86EMUL_OKAY;
> }
> @@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
> read_unlock(&d->arch.hvm.mmcfg_lock);
>
> - if ( !vpci_access_allowed(reg, len) ||
> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> - return X86EMUL_OKAY;
> -
> - vpci_write(sbdf, reg, min(4u, len), data);
> - if ( len == 8 )
> - vpci_write(sbdf, reg + 4, 4, data >> 32);
> + /* Ignore return code */
> + vpci_ecam_mmio_write(sbdf, reg, len, data);
>
> return X86EMUL_OKAY;
> }
> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
> index cbd1bac7fc..c0853176d7 100644
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
> spin_unlock(&pdev->vpci->lock);
> }
>
> +/* Helper function to check an access size and alignment on vpci space. */
> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
> +{
> + /*
> + * Check access size.
> + *
> + * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
> + * but as for those platform ISV register, which gives the access size,
> + * cannot have a value 3, checking this would just harden the code.
> + */
> + if ( len != 1 && len != 2 && len != 4 && len != 8 )
> + return false;
> +
> + /* Check that access is size aligned. */
> + if ( (reg & (len - 1)) )
> + return false;
> +
> + return true;
> +}
> +
> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
> + unsigned long data)
> +{
> + if ( !vpci_ecam_access_allowed(reg, len) ||
> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> + return 0;
> +
> + vpci_write(sbdf, reg, min(4u, len), data);
> + if ( len == 8 )
> + vpci_write(sbdf, reg + 4, 4, data >> 32);
> +
> + return 1;
> +}
> +
> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
> + unsigned long *data)
> +{
> + if ( !vpci_ecam_access_allowed(reg, len) ||
> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> + return 0;
> +
> + /*
> + * According to the PCIe 3.1A specification:
> + * - Configuration Reads and Writes must usually be DWORD or smaller
> + * in size.
> + * - Because Root Complex implementations are not required to support
> + * accesses to a RCRB that cross DW boundaries [...] software
> + * should take care not to cause the generation of such accesses
> + * when accessing a RCRB unless the Root Complex will support the
> + * access.
> + * Xen however supports 8byte accesses by splitting them into two
> + * 4byte accesses.
> + */
> + *data = vpci_read(sbdf, reg, min(4u, len));
> + if ( len == 8 )
> + *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> +
> + return 1;
> +}
> +
> /*
> * Local variables:
> * mode: C
> diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
> index edd7c3e71a..a0df5c1279 100644
> --- a/xen/include/asm-x86/pci.h
> +++ b/xen/include/asm-x86/pci.h
> @@ -6,7 +6,7 @@
> #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
> #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
>
> -#define MMCFG_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
>
> #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
> || id == 0x01268086 || id == 0x01028086 \
> diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h
> index 9f5b5d52e1..4a0c3d77c9 100644
> --- a/xen/include/xen/vpci.h
> +++ b/xen/include/xen/vpci.h
> @@ -208,6 +208,16 @@ static inline unsigned int vmsix_entry_nr(const struct vpci_msix *msix,
> {
> return entry - msix->entries;
> }
> +
> +/* ECAM mmio read/write helpers */
> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len);
> +
> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
> + unsigned long data);
> +
> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
> + unsigned long *data);
> +
> #endif /* __XEN__ */
>
> #else /* !CONFIG_HAS_VPCI */
> --
> 2.25.1
>
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 14:49 ` [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM Bertrand Marquis
@ 2021-10-14 23:49 ` Stefano Stabellini
2021-10-15 6:40 ` Jan Beulich
2021-10-15 10:10 ` Bertrand Marquis
2021-10-15 8:00 ` Jan Beulich
2021-10-15 8:32 ` Roger Pau Monné
2 siblings, 2 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-14 23:49 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Jan Beulich,
Wei Liu, Paul Durrant, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 13895 bytes --]
On Thu, 14 Oct 2021, Bertrand Marquis wrote:
> From: Rahul Singh <rahul.singh@arm.com>
>
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the Xen to emulate the PCI devices config space.
>
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
>
> For Dom0less systems scan_pci_devices() would be used to discover the
> PCI device in XEN and VPCI handler will be added during XEN boots.
>
> This patch is also doing some small fixes to fix compilation errors on
> arm32 of vpci:
> - add a cast to unsigned long in print in header.c
> - add a cast to uint64_t in vpci_ecam_mmio_write
Thank you for these! Strictly speaking they are not required now but
they are welcome. I would also be OK if they were removed from this
patch but it is fine to have them in here.
There is an issues with this patch, see below at the bottom
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>
> ---
> Changes in v6:
> - Use new vpci_ecam_ helpers for PCI access
> - Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
> future patch once everything is ready)
> - rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
> - remove not needed local variables in vpci_mmio_write, the one in read
> has been kept to ensure proper compilation on arm32
> - move call to vpci_add_handlers before iommu init to simplify exit path
> - move call to pci_cleanup_msi in the out section of pci_add_device if
> pdev is not NULL and on error
> - initialize pdev to NULL to handle properly exit path call of
> pci_cleanup_msi
> - keep has_vpci to return false for now as CFG_vpci has been removed.
> Added a comment on top of the definition.
> - fix compilation errors on arm32 (print in header.c, cast missing in
> mmio_write.
> - local variable was kept in vpci_mmio_read on arm to prevent a cast
> error in arm32.
> Change in v5:
> - Add pci_cleanup_msi(pdev) incleanup part.
> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> Change in v4:
> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
> Change in v3:
> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
> Change in v2:
> - Add new XEN_DOMCTL_CDF_vpci flag
> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
> - enable vpci support when pci-passthough option is enabled.
> ---
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 4 ++
> xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
> xen/arch/arm/vpci.h | 36 +++++++++++++++++
> xen/drivers/passthrough/pci.c | 18 ++++++++-
> xen/drivers/vpci/header.c | 3 +-
> xen/drivers/vpci/vpci.c | 2 +-
> xen/include/asm-arm/domain.h | 1 +
> xen/include/asm-x86/pci.h | 2 -
> xen/include/public/arch-arm.h | 7 ++++
> xen/include/xen/pci.h | 3 ++
> 11 files changed, 146 insertions(+), 5 deletions(-)
> create mode 100644 xen/arch/arm/vpci.c
> create mode 100644 xen/arch/arm/vpci.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 64518848b2..07f634508e 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index eef0661beb..96e1b23550 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -39,6 +39,7 @@
> #include <asm/vgic.h>
> #include <asm/vtimer.h>
>
> +#include "vpci.h"
> #include "vuart.h"
>
> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> @@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> goto fail;
>
> + if ( (rc = domain_vpci_init(d)) != 0 )
> + goto fail;
> +
> return 0;
>
> fail:
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..7c3552b65d
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,74 @@
> +/*
> + * xen/arch/arm/vpci.c
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +#include <xen/vpci.h>
> +
> +#include <asm/mmio.h>
> +
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> + register_t *r, void *p)
> +{
> + pci_sbdf_t sbdf;
> + /* data is needed to prevent a pointer cast on 32bit */
> + unsigned long data = ~0ul;
> + int ret;
> +
> + /* We ignore segment part and always handle segment 0 */
> + sbdf.sbdf = ECAM_BDF(info->gpa);
> +
> + ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
> + 1U << info->dabt.size, &data);
> +
> + *r = data;
> +
> + return ret;
> +}
> +
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> + register_t r, void *p)
> +{
> + pci_sbdf_t sbdf;
> +
> + /* We ignore segment part and always handle segment 0 */
> + sbdf.sbdf = ECAM_BDF(info->gpa);
> +
> + return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
> + 1U << info->dabt.size, r);
> +}
> +
> +static const struct mmio_handler_ops vpci_mmio_handler = {
> + .read = vpci_mmio_read,
> + .write = vpci_mmio_write,
> +};
> +
> +int domain_vpci_init(struct domain *d)
> +{
> + if ( !has_vpci(d) )
> + return 0;
> +
> + register_mmio_handler(d, &vpci_mmio_handler,
> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
> +
> + return 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> +
> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
> new file mode 100644
> index 0000000000..d8a7b0e3e8
> --- /dev/null
> +++ b/xen/arch/arm/vpci.h
> @@ -0,0 +1,36 @@
> +/*
> + * xen/arch/arm/vpci.h
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef __ARCH_ARM_VPCI_H__
> +#define __ARCH_ARM_VPCI_H__
> +
> +#ifdef CONFIG_HAS_VPCI
> +int domain_vpci_init(struct domain *d);
> +#else
> +static inline int domain_vpci_init(struct domain *d)
> +{
> + return 0;
> +}
> +#endif
> +
> +#endif /* __ARCH_ARM_VPCI_H__ */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> index 3aa8c3175f..8cc529ecec 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> const struct pci_dev_info *info, nodeid_t node)
> {
> struct pci_seg *pseg;
> - struct pci_dev *pdev;
> + struct pci_dev *pdev = NULL;
> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
> const char *pdev_type;
> int ret;
> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>
> check_pdev(pdev);
>
> +#ifdef CONFIG_ARM
> + /*
> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> + * Dom0 inform XEN to add the PCI devices in XEN.
> + */
> + ret = vpci_add_handlers(pdev);
> + if ( ret )
> + {
> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
> + goto out;
> + }
> +#endif
> +
> ret = 0;
> if ( !pdev->domain )
> {
> @@ -784,6 +797,9 @@ out:
> &PCI_SBDF(seg, bus, slot, func));
> }
> }
> + else if ( pdev )
> + pci_cleanup_msi(pdev);
> +
> return ret;
> }
>
> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
> index f8cd55e7c0..c5b025b88b 100644
> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -374,7 +374,8 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
> if ( val != (uint32_t)(bar->addr >> (hi ? 32 : 0)) )
> gprintk(XENLOG_WARNING,
> "%pp: ignored BAR %lu write with memory decoding enabled\n",
> - &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
> + &pdev->sbdf,
> + (unsigned long)(bar - pdev->vpci->header.bars + hi));
> return;
> }
>
> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
> index c0853176d7..2bd67fc27a 100644
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -507,7 +507,7 @@ int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>
> vpci_write(sbdf, reg, min(4u, len), data);
> if ( len == 8 )
> - vpci_write(sbdf, reg + 4, 4, data >> 32);
> + vpci_write(sbdf, reg + 4, 4, (uint64_t)data >> 32);
>
> return 1;
> }
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 14e575288f..9b3647587a 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -263,6 +263,7 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>
> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>
> +/* vPCI is not available on Arm */
> #define has_vpci(d) ({ (void)(d); false; })
>
> #endif /* __ASM_DOMAIN_H__ */
> diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
> index a0df5c1279..443f25347d 100644
> --- a/xen/include/asm-x86/pci.h
> +++ b/xen/include/asm-x86/pci.h
> @@ -6,8 +6,6 @@
> #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
> #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
>
> -#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
> -
> #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
> || id == 0x01268086 || id == 0x01028086 \
> || id == 0x01128086 || id == 0x01228086 \
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index d46c61fca9..44be337dec 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>
> +/*
> + * 256 MB is reserved for VPCI configuration space based on calculation
> + * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
> + */
Somehow 3 non-ascii characters sneaked into this patch. The 'x' are not
'x' but are 0xc3 and cause the following errors in a few gitlab-ci
builds:
python3 mkheader.py arm32 arm32.h.tmp /builds/xen-project/people/sstabellini/xen/tools/include/xen-foreign/../../../xen/include/public/arch-arm.h /builds/xen-project/people/sstabellini/xen/tools/include/xen-foreign/../../../xen/include/public/xen.h
Traceback (most recent call last):
File "mkheader.py", line 120, in <module>
input += f.read();
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14641: ordinal not in range(128)
Makefile:28: recipe for target 'arm32.h' failed
make[2]: *** [arm32.h] Error 1
Full logs here:
https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/38855078
https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/16810108756
Replacing the 3 characters with 'x' solves the problem.
https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/388721262
With the three 'x' changed to ascii:
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
> +
> /* ACPI tables physical address */
> #define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
> #define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
> diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
> index 70ac25345c..db18cb7639 100644
> --- a/xen/include/xen/pci.h
> +++ b/xen/include/xen/pci.h
> @@ -40,6 +40,9 @@
> #define PCI_SBDF3(s,b,df) \
> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>
> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
> +#define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
> +
> typedef union {
> uint32_t sbdf;
> struct {
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-14 17:54 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages] Ian Jackson
@ 2021-10-14 23:50 ` Stefano Stabellini
2021-10-15 7:28 ` Julien Grall
1 sibling, 0 replies; 190+ messages in thread
From: Stefano Stabellini @ 2021-10-14 23:50 UTC (permalink / raw)
To: Ian Jackson
Cc: Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Michal Orzel, Andre Przywara
On Thu, 14 Oct 2021, Ian Jackson wrote:
> (Replying to both the earlier subthread on v5 and to the new v6
> patch.)
>
> Bertrand Marquis writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
> > Now you suggest to add a new field arm_vpci in libxl__domain_create_state.
>
> Hi. I was handwaving, hence "probably" :-). I hadn't actually looked
> at the existing code to see precisely how it would fit.
>
> > Once we have done that I will need to access this structure to know if I need
> > to add the DT part and somehow to give it a value depending something which
> > for now would the number of pcidevs as there will be no user parameter anymore.
>
> Right.
>
> I looked at libxl_arm.c again:
>
> It seems that the main entrypoint to all this is libxl__prepare_dtb,
> and it is that function you want to do new stuff. That gets
> libxl_domain_build_info (which is specified by the IDL and comes from
> outside libxl, subject to libxl's default-filling machinery) and
> libxl__domain_build_state (which is the general state struct).
>
> The information you need is is libxl_domain_create_info.
> libxl__domain_config_setdefault already arranges to set
> c_info->passthrough based on the number of PCI PT devices
> (search for `need_pt` in libxl_create.c).
>
> That is, as I understand it on ARM vpci should be enabled if
> passthrough is enabled and not otherwise. That is precisely what
> the variable c_info->passthrough is.
>
> There is a slight issue because of the distinction between create_info
> and build_info and domain_config (and, the corresponding _state)
> structs. Currently the DT code ony gets b_info, not the whole
> libxl_domain_config. These distinctions largely historical nowadays.
> Certainly there is no reason not to pass a pointer to the whole
> libxl_domain_config, rather than just libxl_domain_build_info, into
> libxl__arch_domain_init_hw_description.
>
> So I think the right approach for this looks something like this:
>
> 1. Change libxl__arch_domain_init_hw_description to take
> libxl_domain_config* rather than libxl_domain_build_info*.
> libxl_domain_config contains libxl_domain_build_info so
> this is easy.
>
> If you put in a convenience alias variable for the
> libxl_domain_build_info* you can avoid extra typing in the function
> body. (If you call the convenience alias `info` you won't need to
> change the body at all, but maybe `info` isn't the best name so you
> could rename it to `b_info` maybe; up to you.)
>
> 2. Make the same change to libxl__prepare_dtb.
>
> 3. Now you can use d_config->c_info.passthrough to gate the addition
> of the additional stuff to the DT. Ie, that is a respin of this
> patch 3/3.
>
> 1 and 2 need to be separated out from 3, in a different patch (or two
> different patches) added to this series before what is now 3/3. Ie
> 1+2, or 1 and 2 separately, should be pure plumbing changes with no
> functional change.
>
> I hope this is helpful.
The explanation is clear and makes sense to me too.
FYI I suggested something similar and Rahul agreed on it; in fact, he
seemed to have already made the change and tested it:
https://marc.info/?l=xen-devel&m=163362066215155
I am just mentioning it in case Bertrand might be able to find Rahul's
updated patch somewhere and it might be useful as a base for his work.
The rest looks good, including the new changes.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-14 17:09 ` Bertrand Marquis
@ 2021-10-15 6:29 ` Jan Beulich
2021-10-15 7:37 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 6:29 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Ian Jackson, Paul Durrant, Andrew Cooper, Roger Pau Monné,
Wei Liu, xen-devel
On 14.10.2021 19:09, Bertrand Marquis wrote:
>> On 14 Oct 2021, at 17:06, Jan Beulich <jbeulich@suse.com> wrote:
>> On 14.10.2021 16:49, Bertrand Marquis wrote:
>>> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>>>
>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>
>>> - if ( !vpci_access_allowed(reg, size) )
>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>> return X86EMUL_OKAY;
>>>
>>> *data = vpci_read(sbdf, reg, size);
>>> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>>>
>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>
>>> - if ( !vpci_access_allowed(reg, size) )
>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>> return X86EMUL_OKAY;
>>>
>>> vpci_write(sbdf, reg, size, data);
>>
>> Why would port I/O functions call an ECAM helper? And in how far is
>> that helper actually ECAM-specific?
>
> The function was global before.
I'm not objecting to the function being global, but to the "ecam" in
its name.
>>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>
>>> - if ( !vpci_access_allowed(reg, len) ||
>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>> - return X86EMUL_OKAY;
>>
>> While I assume this earlier behavior is the reason for ...
>
> Yes :-)
>
>>
>>> - /*
>>> - * According to the PCIe 3.1A specification:
>>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>>> - * in size.
>>> - * - Because Root Complex implementations are not required to support
>>> - * accesses to a RCRB that cross DW boundaries [...] software
>>> - * should take care not to cause the generation of such accesses
>>> - * when accessing a RCRB unless the Root Complex will support the
>>> - * access.
>>> - * Xen however supports 8byte accesses by splitting them into two
>>> - * 4byte accesses.
>>> - */
>>> - *data = vpci_read(sbdf, reg, min(4u, len));
>>> - if ( len == 8 )
>>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>> + /* Ignore return code */
>>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>>
>> ... the commented-upon ignoring of the return value, I don't think
>> that's a good way to deal with things anymore. Instead I think
>> *data should be written to ~0 upon failure, unless it is intended
>> for vpci_ecam_mmio_read() to take care of that case (in which case
>> I'm not sure I would see why it needs to return an error indicator
>> in the first place).
>
> I am not sure in the first place why this is actually ignored and just
> returning a -1 value.
> If an access is not right, an exception should be generated to the
> Guest instead.
No. That's also not what happens on bare metal, at least not on x86.
Faults cannot be raised for reasons outside of the CPU; such errors
(if these are errors in the first place) need to be dealt with
differently. Signaling an error on the PCI bus would be possible,
but would leave open how that's actually to be dealt with. Instead
bad reads return all ones, while bad writes simply get dropped.
> When we do that on arm the function is returning an error to the upper
> layer in that case, that’s why I did keep a generic function informing the
> caller.
While you're the Arm expert, with the above in mind I wonder what
the actual action in that case ought to be there. Would you explain
to me how, say, a misaligned 2-byte read that the CPU permits but
the PCI subsystem doesn't like would be dealt with by bare metal?
>>> @@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>
>>> - if ( !vpci_access_allowed(reg, len) ||
>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>> - return X86EMUL_OKAY;
>>> -
>>> - vpci_write(sbdf, reg, min(4u, len), data);
>>> - if ( len == 8 )
>>> - vpci_write(sbdf, reg + 4, 4, data >> 32);
>>> + /* Ignore return code */
>>> + vpci_ecam_mmio_write(sbdf, reg, len, data);
>>
>> Here ignoring is fine imo, but if you feel it is important to
>> comment on this, then I think you need to prefer "why" over "what".
>
> Agree I would just need some help on the why.
> Now there was no comment before to explain why so I could also
> remove the comment altogether.
The latter would be my preference.
>>> --- a/xen/drivers/vpci/vpci.c
>>> +++ b/xen/drivers/vpci/vpci.c
>>> @@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
>>> spin_unlock(&pdev->vpci->lock);
>>> }
>>>
>>> +/* Helper function to check an access size and alignment on vpci space. */
>>> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
>>> +{
>>> + /*
>>> + * Check access size.
>>> + *
>>> + * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
>>> + * but as for those platform ISV register, which gives the access size,
>>> + * cannot have a value 3, checking this would just harden the code.
>>> + */
>>> + if ( len != 1 && len != 2 && len != 4 && len != 8 )
>>> + return false;
>>
>> I'm not convinced talking about Arm specifically here is
>> warranted, unless there's something there that's clearly
>> different from all other architectures. Otherwise the comment
>> should imo be written in more general terms.
>
> Other architectures might allow this case. So this is specific to Arm.
If it really is, I consider it wrong to live in common code. If
per-arch tweaking is necessary, and if earlier handling of the
intercepted access doesn't already exclude "bad" cases, then a
per-arch hook would imo be the way to go here. Given the size
of the function I would then wonder why it doesn't remain per-
arch in the first place.
>>> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>> + unsigned long data)
>>> +{
>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>> + return 0;
>>> +
>>> + vpci_write(sbdf, reg, min(4u, len), data);
>>> + if ( len == 8 )
>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>> +
>>> + return 1;
>>> +}
>>> +
>>> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>> + unsigned long *data)
>>> +{
>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>> + return 0;
>>> +
>>> + /*
>>> + * According to the PCIe 3.1A specification:
>>> + * - Configuration Reads and Writes must usually be DWORD or smaller
>>> + * in size.
>>> + * - Because Root Complex implementations are not required to support
>>> + * accesses to a RCRB that cross DW boundaries [...] software
>>> + * should take care not to cause the generation of such accesses
>>> + * when accessing a RCRB unless the Root Complex will support the
>>> + * access.
>>> + * Xen however supports 8byte accesses by splitting them into two
>>> + * 4byte accesses.
>>> + */
>>> + *data = vpci_read(sbdf, reg, min(4u, len));
>>> + if ( len == 8 )
>>> + *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>> +
>>> + return 1;
>>> +}
>>
>> Why do these two functions return int/0/1 instead of
>> bool/false/true (assuming, as per above, that them returning non-
>> void is warranted at all)?
>
> This is what the mmio handlers should return to say that an access
> was ok or not so the function stick to this standard.
Sticking to this would be okay if the functions here needed their
address taken, such that they can be installed as hooks for a
more general framework to invoke. The functions, however, only get
called directly. Hence there's no reason to mirror what is in need
of cleaning up elsewhere. I'm sure you're aware there we're in the
(slow going) process of improving which types get used where.
While the functions you refer to may not have undergone such
cleanup yet, we generally expect new code to conform to the new
model.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 23:49 ` Stefano Stabellini
@ 2021-10-15 6:40 ` Jan Beulich
2021-10-15 9:59 ` Ian Jackson
2021-10-15 10:10 ` Bertrand Marquis
1 sibling, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 6:40 UTC (permalink / raw)
To: Stefano Stabellini, Bertrand Marquis
Cc: xen-devel, iwj, Rahul Singh, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Wei Liu, Paul Durrant,
Roger Pau Monné
On 15.10.2021 01:49, Stefano Stabellini wrote:
> On Thu, 14 Oct 2021, Bertrand Marquis wrote:
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
>> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>>
>> +/*
>> + * 256 MB is reserved for VPCI configuration space based on calculation
>> + * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
>> + */
>
> Somehow 3 non-ascii characters sneaked into this patch. The 'x' are not
> 'x' but are 0xc3 and cause the following errors in a few gitlab-ci
> builds:
>
> python3 mkheader.py arm32 arm32.h.tmp /builds/xen-project/people/sstabellini/xen/tools/include/xen-foreign/../../../xen/include/public/arch-arm.h /builds/xen-project/people/sstabellini/xen/tools/include/xen-foreign/../../../xen/include/public/xen.h
> Traceback (most recent call last):
> File "mkheader.py", line 120, in <module>
> input += f.read();
> File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14641: ordinal not in range(128)
> Makefile:28: recipe for target 'arm32.h' failed
> make[2]: *** [arm32.h] Error 1
>
> Full logs here:
> https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/38855078
> https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/16810108756
>
>
> Replacing the 3 characters with 'x' solves the problem.
> https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/388721262
Interesting. I thought we permit UTF-8 in the sources; see e.g.
tools/tests/x86_emulator/simd-sha.c using Σ. Is mkheader.py in need
of adjustment? Of course I can see that right now the easiest is to
use ASCII x, but I think it was deliberate to use × here.
Then again, with the goal of the public headers being usable with
pretty old compilers as well (C89 being the assumed baseline),
excluding them from the permission to use UTF-8 may also be quite
reasonable.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-14 17:54 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages] Ian Jackson
2021-10-14 23:50 ` Stefano Stabellini
@ 2021-10-15 7:28 ` Julien Grall
2021-10-15 7:41 ` Michal Orzel
2021-10-15 10:02 ` Ian Jackson
1 sibling, 2 replies; 190+ messages in thread
From: Julien Grall @ 2021-10-15 7:28 UTC (permalink / raw)
To: Ian Jackson, Bertrand Marquis
Cc: xen-devel, Rahul Singh, Wei Liu, Anthony PERARD, Juergen Gross,
Stefano Stabellini, Volodymyr Babchuk, Michal Orzel,
Andre Przywara
Hi Ian,
On 14/10/2021 18:54, Ian Jackson wrote:
> (Replying to both the earlier subthread on v5 and to the new v6
> patch.)
>
> Bertrand Marquis writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
>> Now you suggest to add a new field arm_vpci in libxl__domain_create_state.
>
> Hi. I was handwaving, hence "probably" :-). I hadn't actually looked
> at the existing code to see precisely how it would fit.
>
>> Once we have done that I will need to access this structure to know if I need
>> to add the DT part and somehow to give it a value depending something which
>> for now would the number of pcidevs as there will be no user parameter anymore.
>
> Right.
>
> I looked at libxl_arm.c again:
>
> It seems that the main entrypoint to all this is libxl__prepare_dtb,
> and it is that function you want to do new stuff. That gets
> libxl_domain_build_info (which is specified by the IDL and comes from
> outside libxl, subject to libxl's default-filling machinery) and
> libxl__domain_build_state (which is the general state struct).
>
> The information you need is is libxl_domain_create_info.
> libxl__domain_config_setdefault already arranges to set
> c_info->passthrough based on the number of PCI PT devices
> (search for `need_pt` in libxl_create.c).
>
> That is, as I understand it on ARM vpci should be enabled if
> passthrough is enabled and not otherwise. That is precisely what
> the variable c_info->passthrough is.
On Arm, c_info->passthrough is also set when assigning platform devives
(e.g. a non-PCI network card). At least for now, we don't want to create
a node for vCPI in the Device-Tree because we don't enable the
emulation. So...
>
> There is a slight issue because of the distinction between create_info
> and build_info and domain_config (and, the corresponding _state)
> structs. Currently the DT code ony gets b_info, not the whole
> libxl_domain_config. These distinctions largely historical nowadays.
> Certainly there is no reason not to pass a pointer to the whole
> libxl_domain_config, rather than just libxl_domain_build_info, into
> libxl__arch_domain_init_hw_description.
>
> So I think the right approach for this looks something like this:
>
> 1. Change libxl__arch_domain_init_hw_description to take
> libxl_domain_config* rather than libxl_domain_build_info*.
> libxl_domain_config contains libxl_domain_build_info so
> this is easy.
>
> If you put in a convenience alias variable for the
> libxl_domain_build_info* you can avoid extra typing in the function
> body. (If you call the convenience alias `info` you won't need to
> change the body at all, but maybe `info` isn't the best name so you
> could rename it to `b_info` maybe; up to you.)
>
> 2. Make the same change to libxl__prepare_dtb.
>
> 3. Now you can use d_config->c_info.passthrough to gate the addition
> of the additional stuff to the DT. Ie, that is a respin of this
> patch 3/3.
... we will need to check d_config->num_pcidevs for the time being.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 6:29 ` Jan Beulich
@ 2021-10-15 7:37 ` Bertrand Marquis
2021-10-15 8:13 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 7:37 UTC (permalink / raw)
To: Jan Beulich
Cc: Ian Jackson, Paul Durrant, Andrew Cooper, Roger Pau Monné,
Wei Liu, xen-devel, Julien Grall
Hi Jan,
> On 15 Oct 2021, at 07:29, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 14.10.2021 19:09, Bertrand Marquis wrote:
>>> On 14 Oct 2021, at 17:06, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 14.10.2021 16:49, Bertrand Marquis wrote:
>>>> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>>>>
>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>>
>>>> - if ( !vpci_access_allowed(reg, size) )
>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>>> return X86EMUL_OKAY;
>>>>
>>>> *data = vpci_read(sbdf, reg, size);
>>>> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>>>>
>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>>
>>>> - if ( !vpci_access_allowed(reg, size) )
>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>>> return X86EMUL_OKAY;
>>>>
>>>> vpci_write(sbdf, reg, size, data);
>>>
>>> Why would port I/O functions call an ECAM helper? And in how far is
>>> that helper actually ECAM-specific?
>>
>> The function was global before.
>
> I'm not objecting to the function being global, but to the "ecam" in
> its name.
Adding ecam in the name was a request from Roger.
This is just a consequence of this.
One suggestion here could be to turn vpci_ecam_access_allowed into vpci_access_allowed and maybe put this into vpci.h as a static inline ?
>
>>>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>>
>>>> - if ( !vpci_access_allowed(reg, len) ||
>>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>> - return X86EMUL_OKAY;
>>>
>>> While I assume this earlier behavior is the reason for ...
>>
>> Yes :-)
>>
>>>
>>>> - /*
>>>> - * According to the PCIe 3.1A specification:
>>>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>>>> - * in size.
>>>> - * - Because Root Complex implementations are not required to support
>>>> - * accesses to a RCRB that cross DW boundaries [...] software
>>>> - * should take care not to cause the generation of such accesses
>>>> - * when accessing a RCRB unless the Root Complex will support the
>>>> - * access.
>>>> - * Xen however supports 8byte accesses by splitting them into two
>>>> - * 4byte accesses.
>>>> - */
>>>> - *data = vpci_read(sbdf, reg, min(4u, len));
>>>> - if ( len == 8 )
>>>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>> + /* Ignore return code */
>>>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>>>
>>> ... the commented-upon ignoring of the return value, I don't think
>>> that's a good way to deal with things anymore. Instead I think
>>> *data should be written to ~0 upon failure, unless it is intended
>>> for vpci_ecam_mmio_read() to take care of that case (in which case
>>> I'm not sure I would see why it needs to return an error indicator
>>> in the first place).
>>
>> I am not sure in the first place why this is actually ignored and just
>> returning a -1 value.
>> If an access is not right, an exception should be generated to the
>> Guest instead.
>
> No. That's also not what happens on bare metal, at least not on x86.
> Faults cannot be raised for reasons outside of the CPU; such errors
> (if these are errors in the first place) need to be dealt with
> differently. Signaling an error on the PCI bus would be possible,
> but would leave open how that's actually to be dealt with. Instead
> bad reads return all ones, while bad writes simply get dropped.
So that behaviour is kept here on x86 and I think as the function is
generic it is right for it to return an error here. It is up to the caller to
ignore it or not.
To make this more generic I could return 0 on success and -EACCESS,
the caller would then handle it as he wants.
>
>> When we do that on arm the function is returning an error to the upper
>> layer in that case, that’s why I did keep a generic function informing the
>> caller.
>
> While you're the Arm expert, with the above in mind I wonder what
> the actual action in that case ought to be there. Would you explain
> to me how, say, a misaligned 2-byte read that the CPU permits but
> the PCI subsystem doesn't like would be dealt with by bare metal?
The hardware will probably return a BUS error as the access did not
work. But the CPU might just prevent unaligned mmio access on the
first place.
@julien: Maybe you know that one ? Otherwise I can dig to make sure
I answer that right.
>
>>>> @@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
>>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>>
>>>> - if ( !vpci_access_allowed(reg, len) ||
>>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>> - return X86EMUL_OKAY;
>>>> -
>>>> - vpci_write(sbdf, reg, min(4u, len), data);
>>>> - if ( len == 8 )
>>>> - vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>> + /* Ignore return code */
>>>> + vpci_ecam_mmio_write(sbdf, reg, len, data);
>>>
>>> Here ignoring is fine imo, but if you feel it is important to
>>> comment on this, then I think you need to prefer "why" over "what".
>>
>> Agree I would just need some help on the why.
>> Now there was no comment before to explain why so I could also
>> remove the comment altogether.
>
> The latter would be my preference.
Ok I will do that.
>
>>>> --- a/xen/drivers/vpci/vpci.c
>>>> +++ b/xen/drivers/vpci/vpci.c
>>>> @@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
>>>> spin_unlock(&pdev->vpci->lock);
>>>> }
>>>>
>>>> +/* Helper function to check an access size and alignment on vpci space. */
>>>> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
>>>> +{
>>>> + /*
>>>> + * Check access size.
>>>> + *
>>>> + * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
>>>> + * but as for those platform ISV register, which gives the access size,
>>>> + * cannot have a value 3, checking this would just harden the code.
>>>> + */
>>>> + if ( len != 1 && len != 2 && len != 4 && len != 8 )
>>>> + return false;
>>>
>>> I'm not convinced talking about Arm specifically here is
>>> warranted, unless there's something there that's clearly
>>> different from all other architectures. Otherwise the comment
>>> should imo be written in more general terms.
>>
>> Other architectures might allow this case. So this is specific to Arm.
>
> If it really is, I consider it wrong to live in common code. If
> per-arch tweaking is necessary, and if earlier handling of the
> intercepted access doesn't already exclude "bad" cases, then a
> per-arch hook would imo be the way to go here. Given the size
> of the function I would then wonder why it doesn't remain per-
> arch in the first place.
To have this in common code was a request from Roger and as
The code is the same I think that is ok.
I suggested before to turn this into a static inline and remove “ecam”
from the name.
For the comment this was a request from Julien in the first place but
before this was moved to the common code.
I can remove the comment from the common code and put it in the
Arm vpci code instead.
@Julien: would that be acceptable for you now ?
Otherwise I can remove it all together.
>
>>>> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>>> + unsigned long data)
>>>> +{
>>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>> + return 0;
>>>> +
>>>> + vpci_write(sbdf, reg, min(4u, len), data);
>>>> + if ( len == 8 )
>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>> +
>>>> + return 1;
>>>> +}
>>>> +
>>>> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>>> + unsigned long *data)
>>>> +{
>>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>> + return 0;
>>>> +
>>>> + /*
>>>> + * According to the PCIe 3.1A specification:
>>>> + * - Configuration Reads and Writes must usually be DWORD or smaller
>>>> + * in size.
>>>> + * - Because Root Complex implementations are not required to support
>>>> + * accesses to a RCRB that cross DW boundaries [...] software
>>>> + * should take care not to cause the generation of such accesses
>>>> + * when accessing a RCRB unless the Root Complex will support the
>>>> + * access.
>>>> + * Xen however supports 8byte accesses by splitting them into two
>>>> + * 4byte accesses.
>>>> + */
>>>> + *data = vpci_read(sbdf, reg, min(4u, len));
>>>> + if ( len == 8 )
>>>> + *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>> +
>>>> + return 1;
>>>> +}
>>>
>>> Why do these two functions return int/0/1 instead of
>>> bool/false/true (assuming, as per above, that them returning non-
>>> void is warranted at all)?
>>
>> This is what the mmio handlers should return to say that an access
>> was ok or not so the function stick to this standard.
>
> Sticking to this would be okay if the functions here needed their
> address taken, such that they can be installed as hooks for a
> more general framework to invoke. The functions, however, only get
> called directly. Hence there's no reason to mirror what is in need
> of cleaning up elsewhere. I'm sure you're aware there we're in the
> (slow going) process of improving which types get used where.
> While the functions you refer to may not have undergone such
> cleanup yet, we generally expect new code to conform to the new
> model.
I am ok to rename those to vpci_ecam_{read/write}.
Is it what you want ?
Regards
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 7:28 ` Julien Grall
@ 2021-10-15 7:41 ` Michal Orzel
2021-10-15 9:01 ` Julien Grall
2021-10-15 10:02 ` Ian Jackson
1 sibling, 1 reply; 190+ messages in thread
From: Michal Orzel @ 2021-10-15 7:41 UTC (permalink / raw)
To: Julien Grall, Ian Jackson, Bertrand Marquis
Cc: xen-devel, Rahul Singh, Wei Liu, Anthony PERARD, Juergen Gross,
Stefano Stabellini, Volodymyr Babchuk, Andre Przywara
Hi guys,
On 15.10.2021 09:28, Julien Grall wrote:
> Hi Ian,
>
> On 14/10/2021 18:54, Ian Jackson wrote:
>> (Replying to both the earlier subthread on v5 and to the new v6
>> patch.)
>>
>> Bertrand Marquis writes ("Re: [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl"):
>>> Now you suggest to add a new field arm_vpci in libxl__domain_create_state.
>>
>> Hi. I was handwaving, hence "probably" :-). I hadn't actually looked
>> at the existing code to see precisely how it would fit.
>>
>>> Once we have done that I will need to access this structure to know if I need
>>> to add the DT part and somehow to give it a value depending something which
>>> for now would the number of pcidevs as there will be no user parameter anymore.
>>
>> Right.
>>
>> I looked at libxl_arm.c again:
>>
>> It seems that the main entrypoint to all this is libxl__prepare_dtb,
>> and it is that function you want to do new stuff. That gets
>> libxl_domain_build_info (which is specified by the IDL and comes from
>> outside libxl, subject to libxl's default-filling machinery) and
>> libxl__domain_build_state (which is the general state struct).
>>
>> The information you need is is libxl_domain_create_info.
>> libxl__domain_config_setdefault already arranges to set
>> c_info->passthrough based on the number of PCI PT devices
>> (search for `need_pt` in libxl_create.c).
>>
>> That is, as I understand it on ARM vpci should be enabled if
>> passthrough is enabled and not otherwise. That is precisely what
>> the variable c_info->passthrough is.
>
> On Arm, c_info->passthrough is also set when assigning platform devives (e.g. a non-PCI network card). At least for now, we don't want to create a node for vCPI in the Device-Tree because we don't enable the emulation. So...
>
>>
>> There is a slight issue because of the distinction between create_info
>> and build_info and domain_config (and, the corresponding _state)
>> structs. Currently the DT code ony gets b_info, not the whole
>> libxl_domain_config. These distinctions largely historical nowadays.
>> Certainly there is no reason not to pass a pointer to the whole
>> libxl_domain_config, rather than just libxl_domain_build_info, into
>> libxl__arch_domain_init_hw_description.
>>
>> So I think the right approach for this looks something like this:
>>
>> 1. Change libxl__arch_domain_init_hw_description to take
>> libxl_domain_config* rather than libxl_domain_build_info*.
>> libxl_domain_config contains libxl_domain_build_info so
>> this is easy.
>>
>> If you put in a convenience alias variable for the
>> libxl_domain_build_info* you can avoid extra typing in the function
>> body. (If you call the convenience alias `info` you won't need to
>> change the body at all, but maybe `info` isn't the best name so you
>> could rename it to `b_info` maybe; up to you.)
>>
>> 2. Make the same change to libxl__prepare_dtb.
>>
>> 3. Now you can use d_config->c_info.passthrough to gate the addition
>> of the additional stuff to the DT. Ie, that is a respin of this
>> patch 3/3.
>
> ... we will need to check d_config->num_pcidevs for the time being.
>
> Cheers,
>
Thanks Julien for a good catch.
Do you agree on the following approach:
1. Modify argument of libxl__arch_domain_init_hw_description to libxl_domain_config (patch 1)
2. Modify argument of libxl__prepare_dtb to libxl_domain_config (patch 2)
3. Remove arch_arm.vpci completely and in libxl__prepare_dtb add a check (patch 3):
if (d_config->num_pcidevs)
FDT( make_vpci_node(gc, fdt, ainfo, dom) );
+ make_vpci_node implementation
Does it sound ok for you?
Cheers,
Michal
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-14 14:49 ` [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code Bertrand Marquis
2021-10-14 16:06 ` Jan Beulich
2021-10-14 23:47 ` Stefano Stabellini
@ 2021-10-15 7:44 ` Roger Pau Monné
2021-10-15 7:53 ` Bertrand Marquis
2 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 7:44 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, iwj, Paul Durrant, Jan Beulich, Andrew Cooper, Wei Liu
On Thu, Oct 14, 2021 at 03:49:49PM +0100, Bertrand Marquis wrote:
> PCI standard is using ECAM and not MCFG which is coming from ACPI[1].
> Use ECAM/ecam instead of MCFG in common code and in new functions added
> in common vpci code by this patch.
>
> Rename vpci_access_allowed to vpci_ecam_access_allowed and move it
> from arch/x86/hvm/io.c to drivers/vpci/vpci.c.
>
> Create vpci_ecam_mmio_{read,write} in drivers/vpci/vpci.c that
> contains the common code to perform these operations, changed
> vpci_mmcfg_{read,write} accordingly to make use of these functions.
>
> The vpci_ecam_mmio_{read,write} functions are returning 0 on error and 1
> on success. As the x86 code was previously always returning X86EMUL_OKAY
> the return code is ignored. A comment has been added in the code to show
> that this is intentional.
>
> Those functions will be used in a following patch inside by arm vpci
> implementation.
>
> Rename MMCFG_SBDF to ECAM_SBDF.
>
> Not functional change intended with this patch.
>
> [1] https://wiki.osdev.org/PCI_Express
>
> Suggested-by: Roger Pau Monné <roger.pau@citrix.com>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
> ---
> Changes in v6: Patch added
> ---
> xen/arch/x86/hvm/io.c | 50 +++++---------------------------
> xen/drivers/vpci/vpci.c | 60 +++++++++++++++++++++++++++++++++++++++
> xen/include/asm-x86/pci.h | 2 +-
> xen/include/xen/vpci.h | 10 +++++++
> 4 files changed, 78 insertions(+), 44 deletions(-)
>
> diff --git a/xen/arch/x86/hvm/io.c b/xen/arch/x86/hvm/io.c
> index 046a8eb4ed..340b8c8b0e 100644
> --- a/xen/arch/x86/hvm/io.c
> +++ b/xen/arch/x86/hvm/io.c
> @@ -260,20 +260,6 @@ unsigned int hvm_pci_decode_addr(unsigned int cf8, unsigned int addr,
> return CF8_ADDR_LO(cf8) | (addr & 3);
> }
>
> -/* Do some sanity checks. */
> -static bool vpci_access_allowed(unsigned int reg, unsigned int len)
> -{
> - /* Check access size. */
> - if ( len != 1 && len != 2 && len != 4 && len != 8 )
> - return false;
> -
> - /* Check that access is size aligned. */
> - if ( (reg & (len - 1)) )
> - return false;
> -
> - return true;
> -}
> -
> /* vPCI config space IO ports handlers (0xcf8/0xcfc). */
> static bool vpci_portio_accept(const struct hvm_io_handler *handler,
> const ioreq_t *p)
> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>
> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>
> - if ( !vpci_access_allowed(reg, size) )
> + if ( !vpci_ecam_access_allowed(reg, size) )
> return X86EMUL_OKAY;
>
> *data = vpci_read(sbdf, reg, size);
> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>
> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>
> - if ( !vpci_access_allowed(reg, size) )
> + if ( !vpci_ecam_access_allowed(reg, size) )
> return X86EMUL_OKAY;
>
> vpci_write(sbdf, reg, size, data);
> @@ -394,7 +380,7 @@ static unsigned int vpci_mmcfg_decode_addr(const struct hvm_mmcfg *mmcfg,
> paddr_t addr, pci_sbdf_t *sbdf)
> {
> addr -= mmcfg->addr;
> - sbdf->bdf = MMCFG_BDF(addr);
> + sbdf->bdf = ECAM_BDF(addr);
> sbdf->bus += mmcfg->start_bus;
> sbdf->seg = mmcfg->segment;
>
> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
> read_unlock(&d->arch.hvm.mmcfg_lock);
>
> - if ( !vpci_access_allowed(reg, len) ||
> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> - return X86EMUL_OKAY;
> -
> - /*
> - * According to the PCIe 3.1A specification:
> - * - Configuration Reads and Writes must usually be DWORD or smaller
> - * in size.
> - * - Because Root Complex implementations are not required to support
> - * accesses to a RCRB that cross DW boundaries [...] software
> - * should take care not to cause the generation of such accesses
> - * when accessing a RCRB unless the Root Complex will support the
> - * access.
> - * Xen however supports 8byte accesses by splitting them into two
> - * 4byte accesses.
> - */
> - *data = vpci_read(sbdf, reg, min(4u, len));
> - if ( len == 8 )
> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> + /* Ignore return code */
> + vpci_ecam_mmio_read(sbdf, reg, len, data);
I think it would be better for vpci_ecam_mmio_read to just return the
read value, or ~0 in case of error, at least that interface would be
simpler and suitable for x86.
Also I would drop the mmio part from the function name.
>
> return X86EMUL_OKAY;
> }
> @@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
> read_unlock(&d->arch.hvm.mmcfg_lock);
>
> - if ( !vpci_access_allowed(reg, len) ||
> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> - return X86EMUL_OKAY;
> -
> - vpci_write(sbdf, reg, min(4u, len), data);
> - if ( len == 8 )
> - vpci_write(sbdf, reg + 4, 4, data >> 32);
> + /* Ignore return code */
> + vpci_ecam_mmio_write(sbdf, reg, len, data);
Kind of likely here, x86 would be fine with this function return type
being void.
If that's not good for Arm, I think the comment can be dropped as it's
clear the return code is ignored. It would better to maybe add:
/* Failed writes are not propagated to the caller */
>
> return X86EMUL_OKAY;
> }
> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
> index cbd1bac7fc..c0853176d7 100644
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
> spin_unlock(&pdev->vpci->lock);
> }
>
> +/* Helper function to check an access size and alignment on vpci space. */
> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
> +{
> + /*
> + * Check access size.
> + *
> + * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
> + * but as for those platform ISV register, which gives the access size,
> + * cannot have a value 3, checking this would just harden the code.
It feels kind of weird to have an Arm specific comment in common code,
but I guess there's no better place for it to live?
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 7:44 ` Roger Pau Monné
@ 2021-10-15 7:53 ` Bertrand Marquis
2021-10-15 9:53 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 7:53 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Xen-devel, iwj, Paul Durrant, Jan Beulich, Andrew Cooper, Wei Liu
Hi Roger,
> On 15 Oct 2021, at 08:44, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Thu, Oct 14, 2021 at 03:49:49PM +0100, Bertrand Marquis wrote:
>> PCI standard is using ECAM and not MCFG which is coming from ACPI[1].
>> Use ECAM/ecam instead of MCFG in common code and in new functions added
>> in common vpci code by this patch.
>>
>> Rename vpci_access_allowed to vpci_ecam_access_allowed and move it
>> from arch/x86/hvm/io.c to drivers/vpci/vpci.c.
>>
>> Create vpci_ecam_mmio_{read,write} in drivers/vpci/vpci.c that
>> contains the common code to perform these operations, changed
>> vpci_mmcfg_{read,write} accordingly to make use of these functions.
>>
>> The vpci_ecam_mmio_{read,write} functions are returning 0 on error and 1
>> on success. As the x86 code was previously always returning X86EMUL_OKAY
>> the return code is ignored. A comment has been added in the code to show
>> that this is intentional.
>>
>> Those functions will be used in a following patch inside by arm vpci
>> implementation.
>>
>> Rename MMCFG_SBDF to ECAM_SBDF.
>>
>> Not functional change intended with this patch.
>>
>> [1] https://wiki.osdev.org/PCI_Express
>>
>> Suggested-by: Roger Pau Monné <roger.pau@citrix.com>
>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>> ---
>> Changes in v6: Patch added
>> ---
>> xen/arch/x86/hvm/io.c | 50 +++++---------------------------
>> xen/drivers/vpci/vpci.c | 60 +++++++++++++++++++++++++++++++++++++++
>> xen/include/asm-x86/pci.h | 2 +-
>> xen/include/xen/vpci.h | 10 +++++++
>> 4 files changed, 78 insertions(+), 44 deletions(-)
>>
>> diff --git a/xen/arch/x86/hvm/io.c b/xen/arch/x86/hvm/io.c
>> index 046a8eb4ed..340b8c8b0e 100644
>> --- a/xen/arch/x86/hvm/io.c
>> +++ b/xen/arch/x86/hvm/io.c
>> @@ -260,20 +260,6 @@ unsigned int hvm_pci_decode_addr(unsigned int cf8, unsigned int addr,
>> return CF8_ADDR_LO(cf8) | (addr & 3);
>> }
>>
>> -/* Do some sanity checks. */
>> -static bool vpci_access_allowed(unsigned int reg, unsigned int len)
>> -{
>> - /* Check access size. */
>> - if ( len != 1 && len != 2 && len != 4 && len != 8 )
>> - return false;
>> -
>> - /* Check that access is size aligned. */
>> - if ( (reg & (len - 1)) )
>> - return false;
>> -
>> - return true;
>> -}
>> -
>> /* vPCI config space IO ports handlers (0xcf8/0xcfc). */
>> static bool vpci_portio_accept(const struct hvm_io_handler *handler,
>> const ioreq_t *p)
>> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>>
>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>
>> - if ( !vpci_access_allowed(reg, size) )
>> + if ( !vpci_ecam_access_allowed(reg, size) )
>> return X86EMUL_OKAY;
>>
>> *data = vpci_read(sbdf, reg, size);
>> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>>
>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>
>> - if ( !vpci_access_allowed(reg, size) )
>> + if ( !vpci_ecam_access_allowed(reg, size) )
>> return X86EMUL_OKAY;
>>
>> vpci_write(sbdf, reg, size, data);
>> @@ -394,7 +380,7 @@ static unsigned int vpci_mmcfg_decode_addr(const struct hvm_mmcfg *mmcfg,
>> paddr_t addr, pci_sbdf_t *sbdf)
>> {
>> addr -= mmcfg->addr;
>> - sbdf->bdf = MMCFG_BDF(addr);
>> + sbdf->bdf = ECAM_BDF(addr);
>> sbdf->bus += mmcfg->start_bus;
>> sbdf->seg = mmcfg->segment;
>>
>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>
>> - if ( !vpci_access_allowed(reg, len) ||
>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>> - return X86EMUL_OKAY;
>> -
>> - /*
>> - * According to the PCIe 3.1A specification:
>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>> - * in size.
>> - * - Because Root Complex implementations are not required to support
>> - * accesses to a RCRB that cross DW boundaries [...] software
>> - * should take care not to cause the generation of such accesses
>> - * when accessing a RCRB unless the Root Complex will support the
>> - * access.
>> - * Xen however supports 8byte accesses by splitting them into two
>> - * 4byte accesses.
>> - */
>> - *data = vpci_read(sbdf, reg, min(4u, len));
>> - if ( len == 8 )
>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>> + /* Ignore return code */
>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>
> I think it would be better for vpci_ecam_mmio_read to just return the
> read value, or ~0 in case of error, at least that interface would be
> simpler and suitable for x86.
I am not quite sure on this as on absolute to read ~0 is possible so the
caller cannot distinguish between properly reading ~0 or an access allowed error.
>
> Also I would drop the mmio part from the function name.
Ok I will do that.
>
>>
>> return X86EMUL_OKAY;
>> }
>> @@ -476,13 +445,8 @@ static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr,
>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>
>> - if ( !vpci_access_allowed(reg, len) ||
>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>> - return X86EMUL_OKAY;
>> -
>> - vpci_write(sbdf, reg, min(4u, len), data);
>> - if ( len == 8 )
>> - vpci_write(sbdf, reg + 4, 4, data >> 32);
>> + /* Ignore return code */
>> + vpci_ecam_mmio_write(sbdf, reg, len, data);
>
> Kind of likely here, x86 would be fine with this function return type
> being void.
>
> If that's not good for Arm, I think the comment can be dropped as it's
> clear the return code is ignored. It would better to maybe add:
>
> /* Failed writes are not propagated to the caller */
I will replace the current comment by this one.
>
>>
>> return X86EMUL_OKAY;
>> }
>> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
>> index cbd1bac7fc..c0853176d7 100644
>> --- a/xen/drivers/vpci/vpci.c
>> +++ b/xen/drivers/vpci/vpci.c
>> @@ -478,6 +478,66 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
>> spin_unlock(&pdev->vpci->lock);
>> }
>>
>> +/* Helper function to check an access size and alignment on vpci space. */
>> +bool vpci_ecam_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> + /*
>> + * Check access size.
>> + *
>> + * On arm32 or for 32bit guests on arm, 64bit accesses should be forbidden
>> + * but as for those platform ISV register, which gives the access size,
>> + * cannot have a value 3, checking this would just harden the code.
>
> It feels kind of weird to have an Arm specific comment in common code,
> but I guess there's no better place for it to live?
For now I think I can move it directly into arm vpci code even though it is not perfect.
Thanks for the comments
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 14:49 ` [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM Bertrand Marquis
2021-10-14 23:49 ` Stefano Stabellini
@ 2021-10-15 8:00 ` Jan Beulich
2021-10-15 10:09 ` Bertrand Marquis
2021-10-15 8:32 ` Roger Pau Monné
2 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 8:00 UTC (permalink / raw)
To: Bertrand Marquis, Roger Pau Monné
Cc: iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
On 14.10.2021 16:49, Bertrand Marquis wrote:
> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>
> check_pdev(pdev);
>
> +#ifdef CONFIG_ARM
> + /*
> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> + * Dom0 inform XEN to add the PCI devices in XEN.
> + */
> + ret = vpci_add_handlers(pdev);
> + if ( ret )
> + {
> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
> + goto out;
> + }
> +#endif
> +
> ret = 0;
> if ( !pdev->domain )
Elsewhere I did point out that you need to be careful wrt pdev->domain
being NULL. As the code in context clearly documents, you're now
adding handlers before that field was set. In comments to a prior
version I did already suggest to consider placing the new code inside
the if() in question (albeit at the time this was mainly to make clear
that error cleanup may not fiddle with things not put in place by the
iommu_enable_device() alternative path). This would have the further
benefit of making is crystal clear that here only handlers for Dom0
would get put in place (and hence their installing for DomU-s would
need to be looked for elsewhere).
> @@ -784,6 +797,9 @@ out:
> &PCI_SBDF(seg, bus, slot, func));
> }
> }
> + else if ( pdev )
> + pci_cleanup_msi(pdev);
Have you thoroughly checked that this call is benign on x86? There's
no wording to that effect in the description afaics, and I can't
easily convince myself that it would be correct when the
iommu_enable_device() path was taken. (I'm also not going to
exclude that it should have been there even prior to your work,
albeit then adding this would want to be a separate bugfix patch.)
> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -374,7 +374,8 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
> if ( val != (uint32_t)(bar->addr >> (hi ? 32 : 0)) )
> gprintk(XENLOG_WARNING,
> "%pp: ignored BAR %lu write with memory decoding enabled\n",
> - &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
> + &pdev->sbdf,
> + (unsigned long)(bar - pdev->vpci->header.bars + hi));
This looks like an entirely unrelated change which I'm afraid I don't
even understand why it needs making. The description says this is for
Arm32, but it remains unclear what the compilation error would have
been. My best guess is that perhaps you really mean to change the
format specifier to %zu (really this should be %td, but our vsprintf()
doesn't support 't' for whatever reason).
Please recall that we try to avoid casts where possible.
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -507,7 +507,7 @@ int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>
> vpci_write(sbdf, reg, min(4u, len), data);
> if ( len == 8 )
> - vpci_write(sbdf, reg + 4, 4, data >> 32);
> + vpci_write(sbdf, reg + 4, 4, (uint64_t)data >> 32);
I assume the need for this change will go away with the use of
CONFIG_64BIT in the earlier patch.
> --- a/xen/include/xen/pci.h
> +++ b/xen/include/xen/pci.h
> @@ -40,6 +40,9 @@
> #define PCI_SBDF3(s,b,df) \
> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>
> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
> +#define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
The latter is fine to be put here (i.e. FTAOD I'm fine with it
staying here). For the former I even question its original placement
in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
the bus portion of the address can be anywhere from 1 to 8 bits. And
in fact there is a reason why this macro was/is used in only a
single place, but not e.g. in x86'es handling of physical MCFG. It
is merely an implementation choice in vPCI that the entire segment 0
has a linear address range covering all 256 buses. Hence I think
this wants to move to xen/vpci.h and then perhaps also be named
VPCI_ECAM_BDF().
Also, as already pointed out on a much earlier version, while the
blank following the opening parenthesis was warranted in
asm-x86/pci.h for alignment reasons, it is no longer warranted here.
It was certainly gone in v4 and v5.
And one final nit: I don't think we commonly use full stops in patch
titles. (Personally I therefore also think titles shouldn't start
with a capital letter, but I know others think differently.)
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 7:37 ` Bertrand Marquis
@ 2021-10-15 8:13 ` Jan Beulich
2021-10-15 8:20 ` Bertrand Marquis
2021-10-15 9:49 ` Roger Pau Monné
0 siblings, 2 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 8:13 UTC (permalink / raw)
To: Bertrand Marquis, Roger Pau Monné
Cc: Ian Jackson, Paul Durrant, Andrew Cooper, Wei Liu, xen-devel,
Julien Grall
On 15.10.2021 09:37, Bertrand Marquis wrote:
>> On 15 Oct 2021, at 07:29, Jan Beulich <jbeulich@suse.com> wrote:
>> On 14.10.2021 19:09, Bertrand Marquis wrote:
>>>> On 14 Oct 2021, at 17:06, Jan Beulich <jbeulich@suse.com> wrote:
>>>> On 14.10.2021 16:49, Bertrand Marquis wrote:
>>>>> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>>>>>
>>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>>>
>>>>> - if ( !vpci_access_allowed(reg, size) )
>>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>>>> return X86EMUL_OKAY;
>>>>>
>>>>> *data = vpci_read(sbdf, reg, size);
>>>>> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>>>>>
>>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>>>
>>>>> - if ( !vpci_access_allowed(reg, size) )
>>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>>>> return X86EMUL_OKAY;
>>>>>
>>>>> vpci_write(sbdf, reg, size, data);
>>>>
>>>> Why would port I/O functions call an ECAM helper? And in how far is
>>>> that helper actually ECAM-specific?
>>>
>>> The function was global before.
>>
>> I'm not objecting to the function being global, but to the "ecam" in
>> its name.
>
> Adding ecam in the name was a request from Roger.
> This is just a consequence of this.
Roger - did you have in mind the uses here when asking for the addition
of "ecam"?
> One suggestion here could be to turn vpci_ecam_access_allowed into
> vpci_access_allowed
That's what I'm asking for.
> and maybe put this into vpci.h as a static inline ?
I'm not overly fussed here.
>>>>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>>>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>>>
>>>>> - if ( !vpci_access_allowed(reg, len) ||
>>>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>>> - return X86EMUL_OKAY;
>>>>
>>>> While I assume this earlier behavior is the reason for ...
>>>
>>> Yes :-)
>>>
>>>>
>>>>> - /*
>>>>> - * According to the PCIe 3.1A specification:
>>>>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>>>>> - * in size.
>>>>> - * - Because Root Complex implementations are not required to support
>>>>> - * accesses to a RCRB that cross DW boundaries [...] software
>>>>> - * should take care not to cause the generation of such accesses
>>>>> - * when accessing a RCRB unless the Root Complex will support the
>>>>> - * access.
>>>>> - * Xen however supports 8byte accesses by splitting them into two
>>>>> - * 4byte accesses.
>>>>> - */
>>>>> - *data = vpci_read(sbdf, reg, min(4u, len));
>>>>> - if ( len == 8 )
>>>>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>> + /* Ignore return code */
>>>>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>>>>
>>>> ... the commented-upon ignoring of the return value, I don't think
>>>> that's a good way to deal with things anymore. Instead I think
>>>> *data should be written to ~0 upon failure, unless it is intended
>>>> for vpci_ecam_mmio_read() to take care of that case (in which case
>>>> I'm not sure I would see why it needs to return an error indicator
>>>> in the first place).
>>>
>>> I am not sure in the first place why this is actually ignored and just
>>> returning a -1 value.
>>> If an access is not right, an exception should be generated to the
>>> Guest instead.
>>
>> No. That's also not what happens on bare metal, at least not on x86.
>> Faults cannot be raised for reasons outside of the CPU; such errors
>> (if these are errors in the first place) need to be dealt with
>> differently. Signaling an error on the PCI bus would be possible,
>> but would leave open how that's actually to be dealt with. Instead
>> bad reads return all ones, while bad writes simply get dropped.
>
> So that behaviour is kept here on x86 and I think as the function is
> generic it is right for it to return an error here. It is up to the caller to
> ignore it or not.
> To make this more generic I could return 0 on success and -EACCESS,
> the caller would then handle it as he wants.
I think boolean is sufficient here, but I wouldn't object to errno-
style return values. All I do object to is int when boolean is meant.
>>>>> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>>>> + unsigned long data)
>>>>> +{
>>>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>>> + return 0;
>>>>> +
>>>>> + vpci_write(sbdf, reg, min(4u, len), data);
>>>>> + if ( len == 8 )
>>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>>> +
>>>>> + return 1;
>>>>> +}
>>>>> +
>>>>> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>>>> + unsigned long *data)
>>>>> +{
>>>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>>> + return 0;
>>>>> +
>>>>> + /*
>>>>> + * According to the PCIe 3.1A specification:
>>>>> + * - Configuration Reads and Writes must usually be DWORD or smaller
>>>>> + * in size.
>>>>> + * - Because Root Complex implementations are not required to support
>>>>> + * accesses to a RCRB that cross DW boundaries [...] software
>>>>> + * should take care not to cause the generation of such accesses
>>>>> + * when accessing a RCRB unless the Root Complex will support the
>>>>> + * access.
>>>>> + * Xen however supports 8byte accesses by splitting them into two
>>>>> + * 4byte accesses.
>>>>> + */
>>>>> + *data = vpci_read(sbdf, reg, min(4u, len));
>>>>> + if ( len == 8 )
>>>>> + *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>> +
>>>>> + return 1;
>>>>> +}
>>>>
>>>> Why do these two functions return int/0/1 instead of
>>>> bool/false/true (assuming, as per above, that them returning non-
>>>> void is warranted at all)?
>>>
>>> This is what the mmio handlers should return to say that an access
>>> was ok or not so the function stick to this standard.
>>
>> Sticking to this would be okay if the functions here needed their
>> address taken, such that they can be installed as hooks for a
>> more general framework to invoke. The functions, however, only get
>> called directly. Hence there's no reason to mirror what is in need
>> of cleaning up elsewhere. I'm sure you're aware there we're in the
>> (slow going) process of improving which types get used where.
>> While the functions you refer to may not have undergone such
>> cleanup yet, we generally expect new code to conform to the new
>> model.
>
> I am ok to rename those to vpci_ecam_{read/write}.
> Is it what you want ?
Yes, that's what I've been asking for, and I just saw Roger requesting
the same. (I'm a little puzzled about the context though, as you reply
looks disconnected here.)
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 8:13 ` Jan Beulich
@ 2021-10-15 8:20 ` Bertrand Marquis
2021-10-15 8:24 ` Jan Beulich
2021-10-15 9:49 ` Roger Pau Monné
1 sibling, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 8:20 UTC (permalink / raw)
To: Jan Beulich
Cc: Roger Pau Monné,
Ian Jackson, Paul Durrant, Andrew Cooper, Wei Liu, xen-devel,
Julien Grall
Hi Jan,
> On 15 Oct 2021, at 09:13, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 15.10.2021 09:37, Bertrand Marquis wrote:
>>> On 15 Oct 2021, at 07:29, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 14.10.2021 19:09, Bertrand Marquis wrote:
>>>>> On 14 Oct 2021, at 17:06, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> On 14.10.2021 16:49, Bertrand Marquis wrote:
>>>>>> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
>>>>>>
>>>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>>>>
>>>>>> - if ( !vpci_access_allowed(reg, size) )
>>>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>>>>> return X86EMUL_OKAY;
>>>>>>
>>>>>> *data = vpci_read(sbdf, reg, size);
>>>>>> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
>>>>>>
>>>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
>>>>>>
>>>>>> - if ( !vpci_access_allowed(reg, size) )
>>>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
>>>>>> return X86EMUL_OKAY;
>>>>>>
>>>>>> vpci_write(sbdf, reg, size, data);
>>>>>
>>>>> Why would port I/O functions call an ECAM helper? And in how far is
>>>>> that helper actually ECAM-specific?
>>>>
>>>> The function was global before.
>>>
>>> I'm not objecting to the function being global, but to the "ecam" in
>>> its name.
>>
>> Adding ecam in the name was a request from Roger.
>> This is just a consequence of this.
>
> Roger - did you have in mind the uses here when asking for the addition
> of "ecam"?
>
>> One suggestion here could be to turn vpci_ecam_access_allowed into
>> vpci_access_allowed
>
> That's what I'm asking for.
Will do
>
>> and maybe put this into vpci.h as a static inline ?
>
> I'm not overly fussed here.
Was just a suggestion, I am ok to just rename it and keep it where it is.
>
>>>>>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>>>>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>>>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>>>>
>>>>>> - if ( !vpci_access_allowed(reg, len) ||
>>>>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>>>> - return X86EMUL_OKAY;
>>>>>
>>>>> While I assume this earlier behavior is the reason for ...
>>>>
>>>> Yes :-)
>>>>
>>>>>
>>>>>> - /*
>>>>>> - * According to the PCIe 3.1A specification:
>>>>>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>>>>>> - * in size.
>>>>>> - * - Because Root Complex implementations are not required to support
>>>>>> - * accesses to a RCRB that cross DW boundaries [...] software
>>>>>> - * should take care not to cause the generation of such accesses
>>>>>> - * when accessing a RCRB unless the Root Complex will support the
>>>>>> - * access.
>>>>>> - * Xen however supports 8byte accesses by splitting them into two
>>>>>> - * 4byte accesses.
>>>>>> - */
>>>>>> - *data = vpci_read(sbdf, reg, min(4u, len));
>>>>>> - if ( len == 8 )
>>>>>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>>> + /* Ignore return code */
>>>>>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>>>>>
>>>>> ... the commented-upon ignoring of the return value, I don't think
>>>>> that's a good way to deal with things anymore. Instead I think
>>>>> *data should be written to ~0 upon failure, unless it is intended
>>>>> for vpci_ecam_mmio_read() to take care of that case (in which case
>>>>> I'm not sure I would see why it needs to return an error indicator
>>>>> in the first place).
>>>>
>>>> I am not sure in the first place why this is actually ignored and just
>>>> returning a -1 value.
>>>> If an access is not right, an exception should be generated to the
>>>> Guest instead.
>>>
>>> No. That's also not what happens on bare metal, at least not on x86.
>>> Faults cannot be raised for reasons outside of the CPU; such errors
>>> (if these are errors in the first place) need to be dealt with
>>> differently. Signaling an error on the PCI bus would be possible,
>>> but would leave open how that's actually to be dealt with. Instead
>>> bad reads return all ones, while bad writes simply get dropped.
>>
>> So that behaviour is kept here on x86 and I think as the function is
>> generic it is right for it to return an error here. It is up to the caller to
>> ignore it or not.
>> To make this more generic I could return 0 on success and -EACCESS,
>> the caller would then handle it as he wants.
>
> I think boolean is sufficient here, but I wouldn't object to errno-
> style return values. All I do object to is int when boolean is meant.
Boolean sounds right as there is only one error case.
I will use that.
>
>>>>>> +int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>>>>> + unsigned long data)
>>>>>> +{
>>>>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>>>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>>>> + return 0;
>>>>>> +
>>>>>> + vpci_write(sbdf, reg, min(4u, len), data);
>>>>>> + if ( len == 8 )
>>>>>> + vpci_write(sbdf, reg + 4, 4, data >> 32);
>>>>>> +
>>>>>> + return 1;
>>>>>> +}
>>>>>> +
>>>>>> +int vpci_ecam_mmio_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>>>>> + unsigned long *data)
>>>>>> +{
>>>>>> + if ( !vpci_ecam_access_allowed(reg, len) ||
>>>>>> + (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>>>> + return 0;
>>>>>> +
>>>>>> + /*
>>>>>> + * According to the PCIe 3.1A specification:
>>>>>> + * - Configuration Reads and Writes must usually be DWORD or smaller
>>>>>> + * in size.
>>>>>> + * - Because Root Complex implementations are not required to support
>>>>>> + * accesses to a RCRB that cross DW boundaries [...] software
>>>>>> + * should take care not to cause the generation of such accesses
>>>>>> + * when accessing a RCRB unless the Root Complex will support the
>>>>>> + * access.
>>>>>> + * Xen however supports 8byte accesses by splitting them into two
>>>>>> + * 4byte accesses.
>>>>>> + */
>>>>>> + *data = vpci_read(sbdf, reg, min(4u, len));
>>>>>> + if ( len == 8 )
>>>>>> + *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>>>> +
>>>>>> + return 1;
>>>>>> +}
>>>>>
>>>>> Why do these two functions return int/0/1 instead of
>>>>> bool/false/true (assuming, as per above, that them returning non-
>>>>> void is warranted at all)?
>>>>
>>>> This is what the mmio handlers should return to say that an access
>>>> was ok or not so the function stick to this standard.
>>>
>>> Sticking to this would be okay if the functions here needed their
>>> address taken, such that they can be installed as hooks for a
>>> more general framework to invoke. The functions, however, only get
>>> called directly. Hence there's no reason to mirror what is in need
>>> of cleaning up elsewhere. I'm sure you're aware there we're in the
>>> (slow going) process of improving which types get used where.
>>> While the functions you refer to may not have undergone such
>>> cleanup yet, we generally expect new code to conform to the new
>>> model.
>>
>> I am ok to rename those to vpci_ecam_{read/write}.
>> Is it what you want ?
>
> Yes, that's what I've been asking for, and I just saw Roger requesting
> the same. (I'm a little puzzled about the context though, as you reply
> looks disconnected here.)
Oups sorry.
Anyway if we agree on naming scheme and bool return type I think all
your comments here are covered ?
Cheers
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 8:20 ` Bertrand Marquis
@ 2021-10-15 8:24 ` Jan Beulich
0 siblings, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 8:24 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Roger Pau Monné,
Ian Jackson, Paul Durrant, Andrew Cooper, Wei Liu, xen-devel,
Julien Grall
On 15.10.2021 10:20, Bertrand Marquis wrote:
> Anyway if we agree on naming scheme and bool return type I think all
> your comments here are covered ?
Probably, but I have to admit I've lost track ...
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 14:49 ` [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM Bertrand Marquis
2021-10-14 23:49 ` Stefano Stabellini
2021-10-15 8:00 ` Jan Beulich
@ 2021-10-15 8:32 ` Roger Pau Monné
2021-10-15 8:42 ` Michal Orzel
2021-10-15 9:52 ` Bertrand Marquis
2 siblings, 2 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 8:32 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel, iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Jan Beulich,
Wei Liu, Paul Durrant
On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
> From: Rahul Singh <rahul.singh@arm.com>
>
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the Xen to emulate the PCI devices config space.
>
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
>
> For Dom0less systems scan_pci_devices() would be used to discover the
> PCI device in XEN and VPCI handler will be added during XEN boots.
>
> This patch is also doing some small fixes to fix compilation errors on
> arm32 of vpci:
> - add a cast to unsigned long in print in header.c
> - add a cast to uint64_t in vpci_ecam_mmio_write
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
> ---
> Changes in v6:
> - Use new vpci_ecam_ helpers for PCI access
> - Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
> future patch once everything is ready)
Isn't the series missing a revert of XEN_DOMCTL_CDF_vpci? I think
that's what we agreed would be the way forward.
> - rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
> - remove not needed local variables in vpci_mmio_write, the one in read
> has been kept to ensure proper compilation on arm32
> - move call to vpci_add_handlers before iommu init to simplify exit path
> - move call to pci_cleanup_msi in the out section of pci_add_device if
> pdev is not NULL and on error
> - initialize pdev to NULL to handle properly exit path call of
> pci_cleanup_msi
> - keep has_vpci to return false for now as CFG_vpci has been removed.
> Added a comment on top of the definition.
> - fix compilation errors on arm32 (print in header.c, cast missing in
> mmio_write.
> - local variable was kept in vpci_mmio_read on arm to prevent a cast
> error in arm32.
> Change in v5:
> - Add pci_cleanup_msi(pdev) incleanup part.
> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> Change in v4:
> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
> Change in v3:
> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
> Change in v2:
> - Add new XEN_DOMCTL_CDF_vpci flag
> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
> - enable vpci support when pci-passthough option is enabled.
> ---
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 4 ++
> xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
> xen/arch/arm/vpci.h | 36 +++++++++++++++++
> xen/drivers/passthrough/pci.c | 18 ++++++++-
> xen/drivers/vpci/header.c | 3 +-
> xen/drivers/vpci/vpci.c | 2 +-
> xen/include/asm-arm/domain.h | 1 +
> xen/include/asm-x86/pci.h | 2 -
> xen/include/public/arch-arm.h | 7 ++++
> xen/include/xen/pci.h | 3 ++
> 11 files changed, 146 insertions(+), 5 deletions(-)
> create mode 100644 xen/arch/arm/vpci.c
> create mode 100644 xen/arch/arm/vpci.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 64518848b2..07f634508e 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index eef0661beb..96e1b23550 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -39,6 +39,7 @@
> #include <asm/vgic.h>
> #include <asm/vtimer.h>
>
> +#include "vpci.h"
> #include "vuart.h"
>
> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> @@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> goto fail;
>
> + if ( (rc = domain_vpci_init(d)) != 0 )
> + goto fail;
> +
> return 0;
>
> fail:
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..7c3552b65d
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,74 @@
> +/*
> + * xen/arch/arm/vpci.c
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +#include <xen/vpci.h>
> +
> +#include <asm/mmio.h>
> +
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> + register_t *r, void *p)
> +{
> + pci_sbdf_t sbdf;
> + /* data is needed to prevent a pointer cast on 32bit */
> + unsigned long data = ~0ul;
> + int ret;
> +
> + /* We ignore segment part and always handle segment 0 */
> + sbdf.sbdf = ECAM_BDF(info->gpa);
> +
> + ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
> + 1U << info->dabt.size, &data);
> +
> + *r = data;
> +
> + return ret;
> +}
> +
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> + register_t r, void *p)
> +{
> + pci_sbdf_t sbdf;
> +
> + /* We ignore segment part and always handle segment 0 */
> + sbdf.sbdf = ECAM_BDF(info->gpa);
> +
> + return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
> + 1U << info->dabt.size, r);
> +}
I'm not sure returning an error value here is helpful, as I'm not sure
how native Arm behaves and how this error is propagated into the
guest. FWIWreturning ~0 or dropping writes is what we do in x86 when
vPCI is not capable of handling the access.
> +
> +static const struct mmio_handler_ops vpci_mmio_handler = {
> + .read = vpci_mmio_read,
> + .write = vpci_mmio_write,
> +};
> +
> +int domain_vpci_init(struct domain *d)
> +{
> + if ( !has_vpci(d) )
> + return 0;
> +
> + register_mmio_handler(d, &vpci_mmio_handler,
> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
> +
> + return 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> +
> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
> new file mode 100644
> index 0000000000..d8a7b0e3e8
> --- /dev/null
> +++ b/xen/arch/arm/vpci.h
> @@ -0,0 +1,36 @@
> +/*
> + * xen/arch/arm/vpci.h
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef __ARCH_ARM_VPCI_H__
> +#define __ARCH_ARM_VPCI_H__
> +
> +#ifdef CONFIG_HAS_VPCI
> +int domain_vpci_init(struct domain *d);
> +#else
> +static inline int domain_vpci_init(struct domain *d)
> +{
> + return 0;
> +}
> +#endif
> +
> +#endif /* __ARCH_ARM_VPCI_H__ */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> index 3aa8c3175f..8cc529ecec 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> const struct pci_dev_info *info, nodeid_t node)
> {
> struct pci_seg *pseg;
> - struct pci_dev *pdev;
> + struct pci_dev *pdev = NULL;
> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
> const char *pdev_type;
> int ret;
> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>
> check_pdev(pdev);
>
> +#ifdef CONFIG_ARM
> + /*
> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> + * Dom0 inform XEN to add the PCI devices in XEN.
> + */
> + ret = vpci_add_handlers(pdev);
> + if ( ret )
> + {
> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
> + goto out;
> + }
> +#endif
I think vpci_add_handlers should be called after checking that
pdev->domain is != NULL, so I would move this chunk a bit below.
> +
> ret = 0;
> if ( !pdev->domain )
> {
> @@ -784,6 +797,9 @@ out:
> &PCI_SBDF(seg, bus, slot, func));
> }
> }
> + else if ( pdev )
> + pci_cleanup_msi(pdev);
I'm slightly lost at why you add this chunk, is this strictly related
to the patch?
> return ret;
> }
>
> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
> index f8cd55e7c0..c5b025b88b 100644
> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -374,7 +374,8 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
> if ( val != (uint32_t)(bar->addr >> (hi ? 32 : 0)) )
> gprintk(XENLOG_WARNING,
> "%pp: ignored BAR %lu write with memory decoding enabled\n",
> - &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
> + &pdev->sbdf,
> + (unsigned long)(bar - pdev->vpci->header.bars + hi));
> return;
> }
>
> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
> index c0853176d7..2bd67fc27a 100644
> --- a/xen/drivers/vpci/vpci.c
> +++ b/xen/drivers/vpci/vpci.c
> @@ -507,7 +507,7 @@ int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>
> vpci_write(sbdf, reg, min(4u, len), data);
> if ( len == 8 )
> - vpci_write(sbdf, reg + 4, 4, data >> 32);
> + vpci_write(sbdf, reg + 4, 4, (uint64_t)data >> 32);
>
> return 1;
> }
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 14e575288f..9b3647587a 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -263,6 +263,7 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>
> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>
> +/* vPCI is not available on Arm */
> #define has_vpci(d) ({ (void)(d); false; })
>
> #endif /* __ASM_DOMAIN_H__ */
> diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
> index a0df5c1279..443f25347d 100644
> --- a/xen/include/asm-x86/pci.h
> +++ b/xen/include/asm-x86/pci.h
> @@ -6,8 +6,6 @@
> #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
> #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
>
> -#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
> -
> #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
> || id == 0x01268086 || id == 0x01028086 \
> || id == 0x01128086 || id == 0x01228086 \
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index d46c61fca9..44be337dec 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>
> +/*
> + * 256 MB is reserved for VPCI configuration space based on calculation
> + * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
> + */
> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
> +
> /* ACPI tables physical address */
> #define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
> #define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
> diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
> index 70ac25345c..db18cb7639 100644
> --- a/xen/include/xen/pci.h
> +++ b/xen/include/xen/pci.h
> @@ -40,6 +40,9 @@
> #define PCI_SBDF3(s,b,df) \
> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>
> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
^ extra space?
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 8:32 ` Roger Pau Monné
@ 2021-10-15 8:42 ` Michal Orzel
2021-10-15 9:52 ` Bertrand Marquis
1 sibling, 0 replies; 190+ messages in thread
From: Michal Orzel @ 2021-10-15 8:42 UTC (permalink / raw)
To: Roger Pau Monné, Bertrand Marquis
Cc: xen-devel, iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Jan Beulich,
Wei Liu, Paul Durrant
On 15.10.2021 10:32, Roger Pau Monné wrote:
> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>> From: Rahul Singh <rahul.singh@arm.com>
>>
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
>>
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>>
>> For Dom0less systems scan_pci_devices() would be used to discover the
>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>
>> This patch is also doing some small fixes to fix compilation errors on
>> arm32 of vpci:
>> - add a cast to unsigned long in print in header.c
>> - add a cast to uint64_t in vpci_ecam_mmio_write
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>> ---
>> Changes in v6:
>> - Use new vpci_ecam_ helpers for PCI access
>> - Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
>> future patch once everything is ready)
>
> Isn't the series missing a revert of XEN_DOMCTL_CDF_vpci? I think
> that's what we agreed would be the way forward.
>
The revert patch has already been merged.
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9516d01ac3015f522528ed6dafb3f584eaa7ed2c
>> - rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
>> - remove not needed local variables in vpci_mmio_write, the one in read
>> has been kept to ensure proper compilation on arm32
>> - move call to vpci_add_handlers before iommu init to simplify exit path
>> - move call to pci_cleanup_msi in the out section of pci_add_device if
>> pdev is not NULL and on error
>> - initialize pdev to NULL to handle properly exit path call of
>> pci_cleanup_msi
>> - keep has_vpci to return false for now as CFG_vpci has been removed.
>> Added a comment on top of the definition.
>> - fix compilation errors on arm32 (print in header.c, cast missing in
>> mmio_write.
>> - local variable was kept in vpci_mmio_read on arm to prevent a cast
>> error in arm32.
>> Change in v5:
>> - Add pci_cleanup_msi(pdev) incleanup part.
>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> Change in v4:
>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>> Change in v3:
>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>> Change in v2:
>> - Add new XEN_DOMCTL_CDF_vpci flag
>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>> - enable vpci support when pci-passthough option is enabled.
>> ---
>> ---
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/domain.c | 4 ++
>> xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
>> xen/arch/arm/vpci.h | 36 +++++++++++++++++
>> xen/drivers/passthrough/pci.c | 18 ++++++++-
>> xen/drivers/vpci/header.c | 3 +-
>> xen/drivers/vpci/vpci.c | 2 +-
>> xen/include/asm-arm/domain.h | 1 +
>> xen/include/asm-x86/pci.h | 2 -
>> xen/include/public/arch-arm.h | 7 ++++
>> xen/include/xen/pci.h | 3 ++
>> 11 files changed, 146 insertions(+), 5 deletions(-)
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 64518848b2..07f634508e 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index eef0661beb..96e1b23550 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -39,6 +39,7 @@
>> #include <asm/vgic.h>
>> #include <asm/vtimer.h>
>>
>> +#include "vpci.h"
>> #include "vuart.h"
>>
>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>> @@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>> goto fail;
>>
>> + if ( (rc = domain_vpci_init(d)) != 0 )
>> + goto fail;
>> +
>> return 0;
>>
>> fail:
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..7c3552b65d
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,74 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +#include <xen/vpci.h>
>> +
>> +#include <asm/mmio.h>
>> +
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> + register_t *r, void *p)
>> +{
>> + pci_sbdf_t sbdf;
>> + /* data is needed to prevent a pointer cast on 32bit */
>> + unsigned long data = ~0ul;
>> + int ret;
>> +
>> + /* We ignore segment part and always handle segment 0 */
>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>> +
>> + ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
>> + 1U << info->dabt.size, &data);
>> +
>> + *r = data;
>> +
>> + return ret;
>> +}
>> +
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> + register_t r, void *p)
>> +{
>> + pci_sbdf_t sbdf;
>> +
>> + /* We ignore segment part and always handle segment 0 */
>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>> +
>> + return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
>> + 1U << info->dabt.size, r);
>> +}
>
> I'm not sure returning an error value here is helpful, as I'm not sure
> how native Arm behaves and how this error is propagated into the
> guest. FWIWreturning ~0 or dropping writes is what we do in x86 when
> vPCI is not capable of handling the access.
>
>> +
>> +static const struct mmio_handler_ops vpci_mmio_handler = {
>> + .read = vpci_mmio_read,
>> + .write = vpci_mmio_write,
>> +};
>> +
>> +int domain_vpci_init(struct domain *d)
>> +{
>> + if ( !has_vpci(d) )
>> + return 0;
>> +
>> + register_mmio_handler(d, &vpci_mmio_handler,
>> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> +
>> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
>> new file mode 100644
>> index 0000000000..d8a7b0e3e8
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.h
>> @@ -0,0 +1,36 @@
>> +/*
>> + * xen/arch/arm/vpci.h
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef __ARCH_ARM_VPCI_H__
>> +#define __ARCH_ARM_VPCI_H__
>> +
>> +#ifdef CONFIG_HAS_VPCI
>> +int domain_vpci_init(struct domain *d);
>> +#else
>> +static inline int domain_vpci_init(struct domain *d)
>> +{
>> + return 0;
>> +}
>> +#endif
>> +
>> +#endif /* __ARCH_ARM_VPCI_H__ */
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>> index 3aa8c3175f..8cc529ecec 100644
>> --- a/xen/drivers/passthrough/pci.c
>> +++ b/xen/drivers/passthrough/pci.c
>> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>> const struct pci_dev_info *info, nodeid_t node)
>> {
>> struct pci_seg *pseg;
>> - struct pci_dev *pdev;
>> + struct pci_dev *pdev = NULL;
>> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
>> const char *pdev_type;
>> int ret;
>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>
>> check_pdev(pdev);
>>
>> +#ifdef CONFIG_ARM
>> + /*
>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>> + * Dom0 inform XEN to add the PCI devices in XEN.
>> + */
>> + ret = vpci_add_handlers(pdev);
>> + if ( ret )
>> + {
>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>> + goto out;
>> + }
>> +#endif
>
> I think vpci_add_handlers should be called after checking that
> pdev->domain is != NULL, so I would move this chunk a bit below.
>
>> +
>> ret = 0;
>> if ( !pdev->domain )
>> {
>> @@ -784,6 +797,9 @@ out:
>> &PCI_SBDF(seg, bus, slot, func));
>> }
>> }
>> + else if ( pdev )
>> + pci_cleanup_msi(pdev);
>
> I'm slightly lost at why you add this chunk, is this strictly related
> to the patch?
>
>> return ret;
>> }
>>
>> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
>> index f8cd55e7c0..c5b025b88b 100644
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -374,7 +374,8 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
>> if ( val != (uint32_t)(bar->addr >> (hi ? 32 : 0)) )
>> gprintk(XENLOG_WARNING,
>> "%pp: ignored BAR %lu write with memory decoding enabled\n",
>> - &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
>> + &pdev->sbdf,
>> + (unsigned long)(bar - pdev->vpci->header.bars + hi));
>> return;
>> }
>>
>> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
>> index c0853176d7..2bd67fc27a 100644
>> --- a/xen/drivers/vpci/vpci.c
>> +++ b/xen/drivers/vpci/vpci.c
>> @@ -507,7 +507,7 @@ int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>
>> vpci_write(sbdf, reg, min(4u, len), data);
>> if ( len == 8 )
>> - vpci_write(sbdf, reg + 4, 4, data >> 32);
>> + vpci_write(sbdf, reg + 4, 4, (uint64_t)data >> 32);
>>
>> return 1;
>> }
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 14e575288f..9b3647587a 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -263,6 +263,7 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>>
>> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>>
>> +/* vPCI is not available on Arm */
>> #define has_vpci(d) ({ (void)(d); false; })
>>
>> #endif /* __ASM_DOMAIN_H__ */
>> diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
>> index a0df5c1279..443f25347d 100644
>> --- a/xen/include/asm-x86/pci.h
>> +++ b/xen/include/asm-x86/pci.h
>> @@ -6,8 +6,6 @@
>> #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
>> #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
>>
>> -#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
>> -
>> #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
>> || id == 0x01268086 || id == 0x01028086 \
>> || id == 0x01128086 || id == 0x01228086 \
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index d46c61fca9..44be337dec 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
>> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>>
>> +/*
>> + * 256 MB is reserved for VPCI configuration space based on calculation
>> + * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
>> + */
>> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
>> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
>> +
>> /* ACPI tables physical address */
>> #define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
>> #define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
>> diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
>> index 70ac25345c..db18cb7639 100644
>> --- a/xen/include/xen/pci.h
>> +++ b/xen/include/xen/pci.h
>> @@ -40,6 +40,9 @@
>> #define PCI_SBDF3(s,b,df) \
>> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>>
>> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
> ^ extra space?
>
> Thanks, Roger.
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 7:41 ` Michal Orzel
@ 2021-10-15 9:01 ` Julien Grall
0 siblings, 0 replies; 190+ messages in thread
From: Julien Grall @ 2021-10-15 9:01 UTC (permalink / raw)
To: Michal Orzel, Ian Jackson, Bertrand Marquis
Cc: xen-devel, Rahul Singh, Wei Liu, Anthony PERARD, Juergen Gross,
Stefano Stabellini, Volodymyr Babchuk, Andre Przywara
Hi Michal,
On 15/10/2021 08:41, Michal Orzel wrote:
> Do you agree on the following approach:
> 1. Modify argument of libxl__arch_domain_init_hw_description to libxl_domain_config (patch 1)
> 2. Modify argument of libxl__prepare_dtb to libxl_domain_config (patch 2)
> 3. Remove arch_arm.vpci completely and in libxl__prepare_dtb add a check (patch 3):
> if (d_config->num_pcidevs)
> FDT( make_vpci_node(gc, fdt, ainfo, dom) );
> + make_vpci_node implementation
>
> Does it sound ok for you?
This sounds ok to me.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 8:13 ` Jan Beulich
2021-10-15 8:20 ` Bertrand Marquis
@ 2021-10-15 9:49 ` Roger Pau Monné
1 sibling, 0 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 9:49 UTC (permalink / raw)
To: Jan Beulich
Cc: Bertrand Marquis, Ian Jackson, Paul Durrant, Andrew Cooper,
Wei Liu, xen-devel, Julien Grall
On Fri, Oct 15, 2021 at 10:13:24AM +0200, Jan Beulich wrote:
> On 15.10.2021 09:37, Bertrand Marquis wrote:
> >> On 15 Oct 2021, at 07:29, Jan Beulich <jbeulich@suse.com> wrote:
> >> On 14.10.2021 19:09, Bertrand Marquis wrote:
> >>>> On 14 Oct 2021, at 17:06, Jan Beulich <jbeulich@suse.com> wrote:
> >>>> On 14.10.2021 16:49, Bertrand Marquis wrote:
> >>>>> @@ -305,7 +291,7 @@ static int vpci_portio_read(const struct hvm_io_handler *handler,
> >>>>>
> >>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
> >>>>>
> >>>>> - if ( !vpci_access_allowed(reg, size) )
> >>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
> >>>>> return X86EMUL_OKAY;
> >>>>>
> >>>>> *data = vpci_read(sbdf, reg, size);
> >>>>> @@ -335,7 +321,7 @@ static int vpci_portio_write(const struct hvm_io_handler *handler,
> >>>>>
> >>>>> reg = hvm_pci_decode_addr(cf8, addr, &sbdf);
> >>>>>
> >>>>> - if ( !vpci_access_allowed(reg, size) )
> >>>>> + if ( !vpci_ecam_access_allowed(reg, size) )
> >>>>> return X86EMUL_OKAY;
> >>>>>
> >>>>> vpci_write(sbdf, reg, size, data);
> >>>>
> >>>> Why would port I/O functions call an ECAM helper? And in how far is
> >>>> that helper actually ECAM-specific?
> >>>
> >>> The function was global before.
> >>
> >> I'm not objecting to the function being global, but to the "ecam" in
> >> its name.
> >
> > Adding ecam in the name was a request from Roger.
> > This is just a consequence of this.
>
> Roger - did you have in mind the uses here when asking for the addition
> of "ecam"?
I didn't realize it was also used by the IO ports accessors. Strictly
speaking 64bit accesses (lenght == 8) shouldn't be possible/allowed
from IO ports, but that's a red herring.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 8:32 ` Roger Pau Monné
2021-10-15 8:42 ` Michal Orzel
@ 2021-10-15 9:52 ` Bertrand Marquis
2021-10-15 10:13 ` Luca Fancellu
` (2 more replies)
1 sibling, 3 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 9:52 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Jan Beulich, Wei Liu, Paul Durrant, Oleksandr Andrushchenko
Hi Roger,
> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>> From: Rahul Singh <rahul.singh@arm.com>
>>
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
>>
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>>
>> For Dom0less systems scan_pci_devices() would be used to discover the
>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>
>> This patch is also doing some small fixes to fix compilation errors on
>> arm32 of vpci:
>> - add a cast to unsigned long in print in header.c
>> - add a cast to uint64_t in vpci_ecam_mmio_write
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>> ---
>> Changes in v6:
>> - Use new vpci_ecam_ helpers for PCI access
>> - Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
>> future patch once everything is ready)
>
> Isn't the series missing a revert of XEN_DOMCTL_CDF_vpci? I think
> that's what we agreed would be the way forward.
A separate reverse patch for that has already been merged:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9516d01ac3015f522528ed6dafb3f584eaa7ed2c
>
>> - rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
>> - remove not needed local variables in vpci_mmio_write, the one in read
>> has been kept to ensure proper compilation on arm32
>> - move call to vpci_add_handlers before iommu init to simplify exit path
>> - move call to pci_cleanup_msi in the out section of pci_add_device if
>> pdev is not NULL and on error
>> - initialize pdev to NULL to handle properly exit path call of
>> pci_cleanup_msi
>> - keep has_vpci to return false for now as CFG_vpci has been removed.
>> Added a comment on top of the definition.
>> - fix compilation errors on arm32 (print in header.c, cast missing in
>> mmio_write.
>> - local variable was kept in vpci_mmio_read on arm to prevent a cast
>> error in arm32.
>> Change in v5:
>> - Add pci_cleanup_msi(pdev) incleanup part.
>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> Change in v4:
>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>> Change in v3:
>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>> Change in v2:
>> - Add new XEN_DOMCTL_CDF_vpci flag
>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>> - enable vpci support when pci-passthough option is enabled.
>> ---
>> ---
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/domain.c | 4 ++
>> xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
>> xen/arch/arm/vpci.h | 36 +++++++++++++++++
>> xen/drivers/passthrough/pci.c | 18 ++++++++-
>> xen/drivers/vpci/header.c | 3 +-
>> xen/drivers/vpci/vpci.c | 2 +-
>> xen/include/asm-arm/domain.h | 1 +
>> xen/include/asm-x86/pci.h | 2 -
>> xen/include/public/arch-arm.h | 7 ++++
>> xen/include/xen/pci.h | 3 ++
>> 11 files changed, 146 insertions(+), 5 deletions(-)
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 64518848b2..07f634508e 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index eef0661beb..96e1b23550 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -39,6 +39,7 @@
>> #include <asm/vgic.h>
>> #include <asm/vtimer.h>
>>
>> +#include "vpci.h"
>> #include "vuart.h"
>>
>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>> @@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>> goto fail;
>>
>> + if ( (rc = domain_vpci_init(d)) != 0 )
>> + goto fail;
>> +
>> return 0;
>>
>> fail:
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..7c3552b65d
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,74 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +#include <xen/vpci.h>
>> +
>> +#include <asm/mmio.h>
>> +
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> + register_t *r, void *p)
>> +{
>> + pci_sbdf_t sbdf;
>> + /* data is needed to prevent a pointer cast on 32bit */
>> + unsigned long data = ~0ul;
>> + int ret;
>> +
>> + /* We ignore segment part and always handle segment 0 */
>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>> +
>> + ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
>> + 1U << info->dabt.size, &data);
>> +
>> + *r = data;
>> +
>> + return ret;
>> +}
>> +
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> + register_t r, void *p)
>> +{
>> + pci_sbdf_t sbdf;
>> +
>> + /* We ignore segment part and always handle segment 0 */
>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>> +
>> + return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
>> + 1U << info->dabt.size, r);
>> +}
>
> I'm not sure returning an error value here is helpful, as I'm not sure
> how native Arm behaves and how this error is propagated into the
> guest. FWIWreturning ~0 or dropping writes is what we do in x86 when
> vPCI is not capable of handling the access.
Mmio handlers can take a return code on arm if something did not work
so I think this is the right way to do it on arm.
Now has agreed with Jan, we will change the return type of
vpci_ecam_write (also renamed) to be a boolean.
>
>> +
>> +static const struct mmio_handler_ops vpci_mmio_handler = {
>> + .read = vpci_mmio_read,
>> + .write = vpci_mmio_write,
>> +};
>> +
>> +int domain_vpci_init(struct domain *d)
>> +{
>> + if ( !has_vpci(d) )
>> + return 0;
>> +
>> + register_mmio_handler(d, &vpci_mmio_handler,
>> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> +
>> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
>> new file mode 100644
>> index 0000000000..d8a7b0e3e8
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.h
>> @@ -0,0 +1,36 @@
>> +/*
>> + * xen/arch/arm/vpci.h
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef __ARCH_ARM_VPCI_H__
>> +#define __ARCH_ARM_VPCI_H__
>> +
>> +#ifdef CONFIG_HAS_VPCI
>> +int domain_vpci_init(struct domain *d);
>> +#else
>> +static inline int domain_vpci_init(struct domain *d)
>> +{
>> + return 0;
>> +}
>> +#endif
>> +
>> +#endif /* __ARCH_ARM_VPCI_H__ */
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>> index 3aa8c3175f..8cc529ecec 100644
>> --- a/xen/drivers/passthrough/pci.c
>> +++ b/xen/drivers/passthrough/pci.c
>> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>> const struct pci_dev_info *info, nodeid_t node)
>> {
>> struct pci_seg *pseg;
>> - struct pci_dev *pdev;
>> + struct pci_dev *pdev = NULL;
>> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
>> const char *pdev_type;
>> int ret;
>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>
>> check_pdev(pdev);
>>
>> +#ifdef CONFIG_ARM
>> + /*
>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>> + * Dom0 inform XEN to add the PCI devices in XEN.
>> + */
>> + ret = vpci_add_handlers(pdev);
>> + if ( ret )
>> + {
>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>> + goto out;
>> + }
>> +#endif
>
> I think vpci_add_handlers should be called after checking that
> pdev->domain is != NULL, so I would move this chunk a bit below.
On arm this would prevent the dom0less use case or to have the PCI
bus enumerated from an other domain.
@oleksandr: can you comment on this one, you might have a better
answer than me on this ?
>
>> +
>> ret = 0;
>> if ( !pdev->domain )
>> {
>> @@ -784,6 +797,9 @@ out:
>> &PCI_SBDF(seg, bus, slot, func));
>> }
>> }
>> + else if ( pdev )
>> + pci_cleanup_msi(pdev);
>
> I'm slightly lost at why you add this chunk, is this strictly related
> to the patch?
This was discussed a lot in previous version of the patch and
requested by Stefano. The idea here is that as soon as handlers
are added some bits might be modified in the PCI config space
leading possibly to msi interrupts. So it is safer to cleanup on the
error path. For references please see discussion on v4 and v5 where
this was actually added (to much references as the discussion was
long so here [1] and [2] are the patchwork thread).
[1] https://patchwork.kernel.org/project/xen-devel/patch/9bdca2cda5d2e83f94dc2423e55714273539760a.1633540842.git.rahul.singh@arm.com/
[2] https://patchwork.kernel.org/project/xen-devel/patch/f093de681c2560a7196895bcd666ef8840885c1d.1633340795.git.rahul.singh@arm.com/
Regards
Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 7:53 ` Bertrand Marquis
@ 2021-10-15 9:53 ` Roger Pau Monné
2021-10-15 10:12 ` Bertrand Marquis
2021-10-15 10:14 ` Jan Beulich
0 siblings, 2 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 9:53 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Xen-devel, iwj, Paul Durrant, Jan Beulich, Andrew Cooper, Wei Liu
On Fri, Oct 15, 2021 at 07:53:38AM +0000, Bertrand Marquis wrote:
> > On 15 Oct 2021, at 08:44, Roger Pau Monné <roger.pau@citrix.com> wrote:
> > On Thu, Oct 14, 2021 at 03:49:49PM +0100, Bertrand Marquis wrote:
> >> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
> >> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
> >> read_unlock(&d->arch.hvm.mmcfg_lock);
> >>
> >> - if ( !vpci_access_allowed(reg, len) ||
> >> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
> >> - return X86EMUL_OKAY;
> >> -
> >> - /*
> >> - * According to the PCIe 3.1A specification:
> >> - * - Configuration Reads and Writes must usually be DWORD or smaller
> >> - * in size.
> >> - * - Because Root Complex implementations are not required to support
> >> - * accesses to a RCRB that cross DW boundaries [...] software
> >> - * should take care not to cause the generation of such accesses
> >> - * when accessing a RCRB unless the Root Complex will support the
> >> - * access.
> >> - * Xen however supports 8byte accesses by splitting them into two
> >> - * 4byte accesses.
> >> - */
> >> - *data = vpci_read(sbdf, reg, min(4u, len));
> >> - if ( len == 8 )
> >> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
> >> + /* Ignore return code */
> >> + vpci_ecam_mmio_read(sbdf, reg, len, data);
> >
> > I think it would be better for vpci_ecam_mmio_read to just return the
> > read value, or ~0 in case of error, at least that interface would be
> > simpler and suitable for x86.
>
> I am not quite sure on this as on absolute to read ~0 is possible so the
> caller cannot distinguish between properly reading ~0 or an access allowed error.
How do you report an access allowed error on Arm for the PCI config
space?
At least on x86 I don't think we currently have a way to propagate
such errors, neither a plan to do so that I'm aware.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 6:40 ` Jan Beulich
@ 2021-10-15 9:59 ` Ian Jackson
0 siblings, 0 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-15 9:59 UTC (permalink / raw)
To: Jan Beulich
Cc: Stefano Stabellini, Bertrand Marquis, xen-devel, Rahul Singh,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, Roger Pau Monné
Jan Beulich writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
> On 15.10.2021 01:49, Stefano Stabellini wrote:
> > Replacing the 3 characters with 'x' solves the problem.
> > https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/388721262
>
> Interesting. I thought we permit UTF-8 in the sources; see e.g.
> tools/tests/x86_emulator/simd-sha.c using Σ. Is mkheader.py in need
> of adjustment? Of course I can see that right now the easiest is to
> use ASCII x, but I think it was deliberate to use × here.
I think in general we should permit UTF-8 in sources.
But:
> Then again, with the goal of the public headers being usable with
> pretty old compilers as well (C89 being the assumed baseline),
> excluding them from the permission to use UTF-8 may also be quite
> reasonable.
This is a good reason to do otherwise for the public headers.
Maybe this should be documented in CODING_STYLE (can be done after
feature freeze obviously).
That python has such botched unicode handling is a bug of course but
we could decide we think it's a feature :-). In which case maybe
mkheaders should be adjusted to explicitly set IO to ascii so that
this will fail in local builds too.
Anyway, for now we need to replace the UTF-8 in this patch.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 7:28 ` Julien Grall
2021-10-15 7:41 ` Michal Orzel
@ 2021-10-15 10:02 ` Ian Jackson
2021-10-15 10:58 ` Michal Orzel
1 sibling, 1 reply; 190+ messages in thread
From: Ian Jackson @ 2021-10-15 10:02 UTC (permalink / raw)
To: Julien Grall
Cc: Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Michal Orzel, Andre Przywara
Julien Grall writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
> On 14/10/2021 18:54, Ian Jackson wrote:
...
> > That is, as I understand it on ARM vpci should be enabled if
> > passthrough is enabled and not otherwise. That is precisely what
> > the variable c_info->passthrough is.
>
> On Arm, c_info->passthrough is also set when assigning platform devives
> (e.g. a non-PCI network card). At least for now, we don't want to create
> a node for vCPI in the Device-Tree because we don't enable the
> emulation. So...
Oh.
> > 3. Now you can use d_config->c_info.passthrough to gate the addition
> > of the additional stuff to the DT. Ie, that is a respin of this
> > patch 3/3.
>
> ... we will need to check d_config->num_pcidevs for the time being.
OK.
Can you leave a comment somewhere (near where c_info.passthrough is
set) pointing to this use of num_pcidevs ? That might save someone
some future confusion.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 8:00 ` Jan Beulich
@ 2021-10-15 10:09 ` Bertrand Marquis
2021-10-15 10:14 ` Ian Jackson
2021-10-15 10:38 ` Jan Beulich
0 siblings, 2 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 10:09 UTC (permalink / raw)
To: Jan Beulich
Cc: Roger Pau Monné,
iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
Hi Jan,
> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 14.10.2021 16:49, Bertrand Marquis wrote:
>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>
>> check_pdev(pdev);
>>
>> +#ifdef CONFIG_ARM
>> + /*
>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>> + * Dom0 inform XEN to add the PCI devices in XEN.
>> + */
>> + ret = vpci_add_handlers(pdev);
>> + if ( ret )
>> + {
>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>> + goto out;
>> + }
>> +#endif
>> +
>> ret = 0;
>> if ( !pdev->domain )
>
> Elsewhere I did point out that you need to be careful wrt pdev->domain
> being NULL. As the code in context clearly documents, you're now
> adding handlers before that field was set. In comments to a prior
> version I did already suggest to consider placing the new code inside
> the if() in question (albeit at the time this was mainly to make clear
> that error cleanup may not fiddle with things not put in place by the
> iommu_enable_device() alternative path). This would have the further
> benefit of making is crystal clear that here only handlers for Dom0
> would get put in place (and hence their installing for DomU-s would
> need to be looked for elsewhere).
I asked Oleksandr for confirmation here but on arm there will be 2 other use cases:
- PCI own by a DomD so device enumeration done from there
- dom0less with devices detection done inside Xen
>
>> @@ -784,6 +797,9 @@ out:
>> &PCI_SBDF(seg, bus, slot, func));
>> }
>> }
>> + else if ( pdev )
>> + pci_cleanup_msi(pdev);
>
> Have you thoroughly checked that this call is benign on x86? There's
> no wording to that effect in the description afaics, and I can't
> easily convince myself that it would be correct when the
> iommu_enable_device() path was taken. (I'm also not going to
> exclude that it should have been there even prior to your work,
> albeit then adding this would want to be a separate bugfix patch.)
This was not in the original serie and requested by Stefano. I must admit
I am not completely sure on the details here so I am really ok to remove this
but this would go against what was requested on the previous versions (4 and 5).
>
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -374,7 +374,8 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
>> if ( val != (uint32_t)(bar->addr >> (hi ? 32 : 0)) )
>> gprintk(XENLOG_WARNING,
>> "%pp: ignored BAR %lu write with memory decoding enabled\n",
>> - &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
>> + &pdev->sbdf,
>> + (unsigned long)(bar - pdev->vpci->header.bars + hi));
>
> This looks like an entirely unrelated change which I'm afraid I don't
> even understand why it needs making. The description says this is for
> Arm32, but it remains unclear what the compilation error would have
> been. My best guess is that perhaps you really mean to change the
> format specifier to %zu (really this should be %td, but our vsprintf()
> doesn't support 't' for whatever reason).
Compilation error is about an invalid %lu print for an unsigned int value
when compiled for arm32.
I will use zu instead in the next version and remove the cast.
>
> Please recall that we try to avoid casts where possible.
Sure I will.
>
>> --- a/xen/drivers/vpci/vpci.c
>> +++ b/xen/drivers/vpci/vpci.c
>> @@ -507,7 +507,7 @@ int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>
>> vpci_write(sbdf, reg, min(4u, len), data);
>> if ( len == 8 )
>> - vpci_write(sbdf, reg + 4, 4, data >> 32);
>> + vpci_write(sbdf, reg + 4, 4, (uint64_t)data >> 32);
>
> I assume the need for this change will go away with the use of
> CONFIG_64BIT in the earlier patch.
Yes
>
>> --- a/xen/include/xen/pci.h
>> +++ b/xen/include/xen/pci.h
>> @@ -40,6 +40,9 @@
>> #define PCI_SBDF3(s,b,df) \
>> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>>
>> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
>> +#define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
>
> The latter is fine to be put here (i.e. FTAOD I'm fine with it
> staying here). For the former I even question its original placement
> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
> the bus portion of the address can be anywhere from 1 to 8 bits. And
> in fact there is a reason why this macro was/is used in only a
> single place, but not e.g. in x86'es handling of physical MCFG. It
> is merely an implementation choice in vPCI that the entire segment 0
> has a linear address range covering all 256 buses. Hence I think
> this wants to move to xen/vpci.h and then perhaps also be named
> VPCI_ECAM_BDF().
On previous version it was request to renamed this to ECAM and agreed
to put is here. Now you want me to rename it to VPCI and move it again.
I would like to have a confirmation that this is ok and the final move if possible.
@Roger can you confirm this is what is wanted ?
Also if I have to do this I will do the same for REG_OFFSET of course.
>
> Also, as already pointed out on a much earlier version, while the
> blank following the opening parenthesis was warranted in
> asm-x86/pci.h for alignment reasons, it is no longer warranted here.
> It was certainly gone in v4 and v5.
Yes my mistake during the process I did a copy and paste from the original
one and did not modify. I will fix that.
>
> And one final nit: I don't think we commonly use full stops in patch
> titles. (Personally I therefore also think titles shouldn't start
> with a capital letter, but I know others think differently.)
Ok I will fix the patch name.
Cheers
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-14 23:49 ` Stefano Stabellini
2021-10-15 6:40 ` Jan Beulich
@ 2021-10-15 10:10 ` Bertrand Marquis
1 sibling, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 10:10 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, iwj, Rahul Singh, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Jan Beulich, Wei Liu, Paul Durrant,
Roger Pau Monné
Hi Stefano,
> On 15 Oct 2021, at 00:49, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 14 Oct 2021, Bertrand Marquis wrote:
>> From: Rahul Singh <rahul.singh@arm.com>
>>
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
>>
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>>
>> For Dom0less systems scan_pci_devices() would be used to discover the
>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>
>> This patch is also doing some small fixes to fix compilation errors on
>> arm32 of vpci:
>> - add a cast to unsigned long in print in header.c
>> - add a cast to uint64_t in vpci_ecam_mmio_write
>
> Thank you for these! Strictly speaking they are not required now but
> they are welcome. I would also be OK if they were removed from this
> patch but it is fine to have them in here.
>
> There is an issues with this patch, see below at the bottom
>
>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>>
>> ---
>> Changes in v6:
>> - Use new vpci_ecam_ helpers for PCI access
>> - Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
>> future patch once everything is ready)
>> - rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
>> - remove not needed local variables in vpci_mmio_write, the one in read
>> has been kept to ensure proper compilation on arm32
>> - move call to vpci_add_handlers before iommu init to simplify exit path
>> - move call to pci_cleanup_msi in the out section of pci_add_device if
>> pdev is not NULL and on error
>> - initialize pdev to NULL to handle properly exit path call of
>> pci_cleanup_msi
>> - keep has_vpci to return false for now as CFG_vpci has been removed.
>> Added a comment on top of the definition.
>> - fix compilation errors on arm32 (print in header.c, cast missing in
>> mmio_write.
>> - local variable was kept in vpci_mmio_read on arm to prevent a cast
>> error in arm32.
>> Change in v5:
>> - Add pci_cleanup_msi(pdev) incleanup part.
>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>> Change in v4:
>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>> Change in v3:
>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>> Change in v2:
>> - Add new XEN_DOMCTL_CDF_vpci flag
>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>> - enable vpci support when pci-passthough option is enabled.
>> ---
>> ---
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/domain.c | 4 ++
>> xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
>> xen/arch/arm/vpci.h | 36 +++++++++++++++++
>> xen/drivers/passthrough/pci.c | 18 ++++++++-
>> xen/drivers/vpci/header.c | 3 +-
>> xen/drivers/vpci/vpci.c | 2 +-
>> xen/include/asm-arm/domain.h | 1 +
>> xen/include/asm-x86/pci.h | 2 -
>> xen/include/public/arch-arm.h | 7 ++++
>> xen/include/xen/pci.h | 3 ++
>> 11 files changed, 146 insertions(+), 5 deletions(-)
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 64518848b2..07f634508e 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index eef0661beb..96e1b23550 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -39,6 +39,7 @@
>> #include <asm/vgic.h>
>> #include <asm/vtimer.h>
>>
>> +#include "vpci.h"
>> #include "vuart.h"
>>
>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>> @@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>> goto fail;
>>
>> + if ( (rc = domain_vpci_init(d)) != 0 )
>> + goto fail;
>> +
>> return 0;
>>
>> fail:
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..7c3552b65d
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,74 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +#include <xen/vpci.h>
>> +
>> +#include <asm/mmio.h>
>> +
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> + register_t *r, void *p)
>> +{
>> + pci_sbdf_t sbdf;
>> + /* data is needed to prevent a pointer cast on 32bit */
>> + unsigned long data = ~0ul;
>> + int ret;
>> +
>> + /* We ignore segment part and always handle segment 0 */
>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>> +
>> + ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
>> + 1U << info->dabt.size, &data);
>> +
>> + *r = data;
>> +
>> + return ret;
>> +}
>> +
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> + register_t r, void *p)
>> +{
>> + pci_sbdf_t sbdf;
>> +
>> + /* We ignore segment part and always handle segment 0 */
>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>> +
>> + return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
>> + 1U << info->dabt.size, r);
>> +}
>> +
>> +static const struct mmio_handler_ops vpci_mmio_handler = {
>> + .read = vpci_mmio_read,
>> + .write = vpci_mmio_write,
>> +};
>> +
>> +int domain_vpci_init(struct domain *d)
>> +{
>> + if ( !has_vpci(d) )
>> + return 0;
>> +
>> + register_mmio_handler(d, &vpci_mmio_handler,
>> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> +
>> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
>> new file mode 100644
>> index 0000000000..d8a7b0e3e8
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.h
>> @@ -0,0 +1,36 @@
>> +/*
>> + * xen/arch/arm/vpci.h
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef __ARCH_ARM_VPCI_H__
>> +#define __ARCH_ARM_VPCI_H__
>> +
>> +#ifdef CONFIG_HAS_VPCI
>> +int domain_vpci_init(struct domain *d);
>> +#else
>> +static inline int domain_vpci_init(struct domain *d)
>> +{
>> + return 0;
>> +}
>> +#endif
>> +
>> +#endif /* __ARCH_ARM_VPCI_H__ */
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>> index 3aa8c3175f..8cc529ecec 100644
>> --- a/xen/drivers/passthrough/pci.c
>> +++ b/xen/drivers/passthrough/pci.c
>> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>> const struct pci_dev_info *info, nodeid_t node)
>> {
>> struct pci_seg *pseg;
>> - struct pci_dev *pdev;
>> + struct pci_dev *pdev = NULL;
>> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
>> const char *pdev_type;
>> int ret;
>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>
>> check_pdev(pdev);
>>
>> +#ifdef CONFIG_ARM
>> + /*
>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>> + * Dom0 inform XEN to add the PCI devices in XEN.
>> + */
>> + ret = vpci_add_handlers(pdev);
>> + if ( ret )
>> + {
>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>> + goto out;
>> + }
>> +#endif
>> +
>> ret = 0;
>> if ( !pdev->domain )
>> {
>> @@ -784,6 +797,9 @@ out:
>> &PCI_SBDF(seg, bus, slot, func));
>> }
>> }
>> + else if ( pdev )
>> + pci_cleanup_msi(pdev);
>> +
>> return ret;
>> }
>>
>> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
>> index f8cd55e7c0..c5b025b88b 100644
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -374,7 +374,8 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg,
>> if ( val != (uint32_t)(bar->addr >> (hi ? 32 : 0)) )
>> gprintk(XENLOG_WARNING,
>> "%pp: ignored BAR %lu write with memory decoding enabled\n",
>> - &pdev->sbdf, bar - pdev->vpci->header.bars + hi);
>> + &pdev->sbdf,
>> + (unsigned long)(bar - pdev->vpci->header.bars + hi));
>> return;
>> }
>>
>> diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c
>> index c0853176d7..2bd67fc27a 100644
>> --- a/xen/drivers/vpci/vpci.c
>> +++ b/xen/drivers/vpci/vpci.c
>> @@ -507,7 +507,7 @@ int vpci_ecam_mmio_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len,
>>
>> vpci_write(sbdf, reg, min(4u, len), data);
>> if ( len == 8 )
>> - vpci_write(sbdf, reg + 4, 4, data >> 32);
>> + vpci_write(sbdf, reg + 4, 4, (uint64_t)data >> 32);
>>
>> return 1;
>> }
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 14e575288f..9b3647587a 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -263,6 +263,7 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>>
>> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>>
>> +/* vPCI is not available on Arm */
>> #define has_vpci(d) ({ (void)(d); false; })
>>
>> #endif /* __ASM_DOMAIN_H__ */
>> diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h
>> index a0df5c1279..443f25347d 100644
>> --- a/xen/include/asm-x86/pci.h
>> +++ b/xen/include/asm-x86/pci.h
>> @@ -6,8 +6,6 @@
>> #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16)
>> #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000))
>>
>> -#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
>> -
>> #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \
>> || id == 0x01268086 || id == 0x01028086 \
>> || id == 0x01128086 || id == 0x01228086 \
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index d46c61fca9..44be337dec 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -418,6 +418,13 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
>> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>>
>> +/*
>> + * 256 MB is reserved for VPCI configuration space based on calculation
>> + * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB
>> + */
>
> Somehow 3 non-ascii characters sneaked into this patch. The 'x' are not
> 'x' but are 0xc3 and cause the following errors in a few gitlab-ci
> builds:
>
> python3 mkheader.py arm32 arm32.h.tmp /builds/xen-project/people/sstabellini/xen/tools/include/xen-foreign/../../../xen/include/public/arch-arm.h /builds/xen-project/people/sstabellini/xen/tools/include/xen-foreign/../../../xen/include/public/xen.h
> Traceback (most recent call last):
> File "mkheader.py", line 120, in <module>
> input += f.read();
> File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14641: ordinal not in range(128)
> Makefile:28: recipe for target 'arm32.h' failed
> make[2]: *** [arm32.h] Error 1
>
> Full logs here:
> https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/38855078
> https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/16810108756
>
>
> Replacing the 3 characters with 'x' solves the problem.
> https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/388721262
>
>
> With the three 'x' changed to ascii:
>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
I will fix that but sadly seeing all other changes to be done I do not think I can keep your R-b.
Thanks
Bertrand
>
>
>
>> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
>> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
>> +
>> /* ACPI tables physical address */
>> #define GUEST_ACPI_BASE xen_mk_ullong(0x20000000)
>> #define GUEST_ACPI_SIZE xen_mk_ullong(0x02000000)
>> diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
>> index 70ac25345c..db18cb7639 100644
>> --- a/xen/include/xen/pci.h
>> +++ b/xen/include/xen/pci.h
>> @@ -40,6 +40,9 @@
>> #define PCI_SBDF3(s,b,df) \
>> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>>
>> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
>> +#define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
>> +
>> typedef union {
>> uint32_t sbdf;
>> struct {
>> --
>> 2.25.1
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 9:53 ` Roger Pau Monné
@ 2021-10-15 10:12 ` Bertrand Marquis
2021-10-15 10:14 ` Jan Beulich
1 sibling, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 10:12 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Xen-devel, iwj, Paul Durrant, Jan Beulich, Andrew Cooper, Wei Liu
Hi Roger,
> On 15 Oct 2021, at 10:53, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Fri, Oct 15, 2021 at 07:53:38AM +0000, Bertrand Marquis wrote:
>>> On 15 Oct 2021, at 08:44, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>> On Thu, Oct 14, 2021 at 03:49:49PM +0100, Bertrand Marquis wrote:
>>>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>>
>>>> - if ( !vpci_access_allowed(reg, len) ||
>>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>> - return X86EMUL_OKAY;
>>>> -
>>>> - /*
>>>> - * According to the PCIe 3.1A specification:
>>>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>>>> - * in size.
>>>> - * - Because Root Complex implementations are not required to support
>>>> - * accesses to a RCRB that cross DW boundaries [...] software
>>>> - * should take care not to cause the generation of such accesses
>>>> - * when accessing a RCRB unless the Root Complex will support the
>>>> - * access.
>>>> - * Xen however supports 8byte accesses by splitting them into two
>>>> - * 4byte accesses.
>>>> - */
>>>> - *data = vpci_read(sbdf, reg, min(4u, len));
>>>> - if ( len == 8 )
>>>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>> + /* Ignore return code */
>>>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>>>
>>> I think it would be better for vpci_ecam_mmio_read to just return the
>>> read value, or ~0 in case of error, at least that interface would be
>>> simpler and suitable for x86.
>>
>> I am not quite sure on this as on absolute to read ~0 is possible so the
>> caller cannot distinguish between properly reading ~0 or an access allowed error.
>
> How do you report an access allowed error on Arm for the PCI config
> space?
This does not go that far. In the current case this is actually an unaligned
access to mmio which will trigger and exception before reaching the PCI config space.
>
> At least on x86 I don't think we currently have a way to propagate
> such errors, neither a plan to do so that I'm aware.
The behaviour on x86 will not change as we will ignore this.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 9:52 ` Bertrand Marquis
@ 2021-10-15 10:13 ` Luca Fancellu
2021-10-15 10:17 ` Bertrand Marquis
2021-10-15 10:19 ` Roger Pau Monné
2021-10-15 10:24 ` Jan Beulich
2 siblings, 1 reply; 190+ messages in thread
From: Luca Fancellu @ 2021-10-15 10:13 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Roger Pau Monné,
Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Jan Beulich, Wei Liu, Paul Durrant, Oleksandr Andrushchenko
> On 15 Oct 2021, at 10:52, Bertrand Marquis <Bertrand.Marquis@arm.com> wrote:
>
> Hi Roger,
>
>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>
>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>> From: Rahul Singh <rahul.singh@arm.com>
>>>
>>> The existing VPCI support available for X86 is adapted for Arm.
>>> When the device is added to XEN via the hyper call
>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>> access is added to the Xen to emulate the PCI devices config space.
>>>
>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>> so that when guest is trying to access the PCI config space,XEN
>>> will trap the access and emulate read/write using the VPCI and
>>> not the real PCI hardware.
>>>
>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>
>>> This patch is also doing some small fixes to fix compilation errors on
>>> arm32 of vpci:
>>> - add a cast to unsigned long in print in header.c
>>> - add a cast to uint64_t in vpci_ecam_mmio_write
>>>
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>>> ---
>>> Changes in v6:
>>> - Use new vpci_ecam_ helpers for PCI access
>>> - Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
>>> future patch once everything is ready)
>>
>> Isn't the series missing a revert of XEN_DOMCTL_CDF_vpci? I think
>> that's what we agreed would be the way forward.
>
> A separate reverse patch for that has already been merged:
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9516d01ac3015f522528ed6dafb3f584eaa7ed2c
>
>>
>>> - rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
>>> - remove not needed local variables in vpci_mmio_write, the one in read
>>> has been kept to ensure proper compilation on arm32
>>> - move call to vpci_add_handlers before iommu init to simplify exit path
>>> - move call to pci_cleanup_msi in the out section of pci_add_device if
>>> pdev is not NULL and on error
>>> - initialize pdev to NULL to handle properly exit path call of
>>> pci_cleanup_msi
>>> - keep has_vpci to return false for now as CFG_vpci has been removed.
>>> Added a comment on top of the definition.
>>> - fix compilation errors on arm32 (print in header.c, cast missing in
>>> mmio_write.
>>> - local variable was kept in vpci_mmio_read on arm to prevent a cast
>>> error in arm32.
>>> Change in v5:
>>> - Add pci_cleanup_msi(pdev) incleanup part.
>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>> Change in v4:
>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>> Change in v3:
>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>> Change in v2:
>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>> - enable vpci support when pci-passthough option is enabled.
>>> ---
>>> ---
>>> xen/arch/arm/Makefile | 1 +
>>> xen/arch/arm/domain.c | 4 ++
>>> xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
>>> xen/arch/arm/vpci.h | 36 +++++++++++++++++
>>> xen/drivers/passthrough/pci.c | 18 ++++++++-
>>> xen/drivers/vpci/header.c | 3 +-
>>> xen/drivers/vpci/vpci.c | 2 +-
>>> xen/include/asm-arm/domain.h | 1 +
>>> xen/include/asm-x86/pci.h | 2 -
>>> xen/include/public/arch-arm.h | 7 ++++
>>> xen/include/xen/pci.h | 3 ++
>>> 11 files changed, 146 insertions(+), 5 deletions(-)
>>> create mode 100644 xen/arch/arm/vpci.c
>>> create mode 100644 xen/arch/arm/vpci.h
>>>
>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>> index 64518848b2..07f634508e 100644
>>> --- a/xen/arch/arm/Makefile
>>> +++ b/xen/arch/arm/Makefile
>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>> obj-y += platforms/
>>> endif
>>> obj-$(CONFIG_TEE) += tee/
>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>
>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>> obj-y += bootfdt.init.o
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index eef0661beb..96e1b23550 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -39,6 +39,7 @@
>>> #include <asm/vgic.h>
>>> #include <asm/vtimer.h>
>>>
>>> +#include "vpci.h"
>>> #include "vuart.h"
>>>
>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>> @@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>> goto fail;
>>>
>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>> + goto fail;
>>> +
>>> return 0;
>>>
>>> fail:
>>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>>> new file mode 100644
>>> index 0000000000..7c3552b65d
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vpci.c
>>> @@ -0,0 +1,74 @@
>>> +/*
>>> + * xen/arch/arm/vpci.c
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; either version 2 of the License, or
>>> + * (at your option) any later version.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + */
>>> +#include <xen/sched.h>
>>> +#include <xen/vpci.h>
>>> +
>>> +#include <asm/mmio.h>
>>> +
>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>> + register_t *r, void *p)
>>> +{
>>> + pci_sbdf_t sbdf;
>>> + /* data is needed to prevent a pointer cast on 32bit */
>>> + unsigned long data = ~0ul;
>>> + int ret;
>>> +
>>> + /* We ignore segment part and always handle segment 0 */
>>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>>> +
>>> + ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
>>> + 1U << info->dabt.size, &data);
>>> +
>>> + *r = data;
>>> +
>>> + return ret;
>>> +}
>>> +
>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>> + register_t r, void *p)
>>> +{
>>> + pci_sbdf_t sbdf;
>>> +
>>> + /* We ignore segment part and always handle segment 0 */
>>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>>> +
>>> + return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
>>> + 1U << info->dabt.size, r);
>>> +}
>>
>> I'm not sure returning an error value here is helpful, as I'm not sure
>> how native Arm behaves and how this error is propagated into the
>> guest. FWIWreturning ~0 or dropping writes is what we do in x86 when
>> vPCI is not capable of handling the access.
>
> Mmio handlers can take a return code on arm if something did not work
> so I think this is the right way to do it on arm.
> Now has agreed with Jan, we will change the return type of
> vpci_ecam_write (also renamed) to be a boolean.
>
>>
>>> +
>>> +static const struct mmio_handler_ops vpci_mmio_handler = {
>>> + .read = vpci_mmio_read,
>>> + .write = vpci_mmio_write,
>>> +};
>>> +
>>> +int domain_vpci_init(struct domain *d)
>>> +{
>>> + if ( !has_vpci(d) )
>>> + return 0;
>>> +
>>> + register_mmio_handler(d, &vpci_mmio_handler,
>>> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> +
>>> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
>>> new file mode 100644
>>> index 0000000000..d8a7b0e3e8
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vpci.h
>>> @@ -0,0 +1,36 @@
>>> +/*
>>> + * xen/arch/arm/vpci.h
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; either version 2 of the License, or
>>> + * (at your option) any later version.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + */
>>> +
>>> +#ifndef __ARCH_ARM_VPCI_H__
>>> +#define __ARCH_ARM_VPCI_H__
>>> +
>>> +#ifdef CONFIG_HAS_VPCI
>>> +int domain_vpci_init(struct domain *d);
>>> +#else
>>> +static inline int domain_vpci_init(struct domain *d)
>>> +{
>>> + return 0;
>>> +}
>>> +#endif
>>> +
>>> +#endif /* __ARCH_ARM_VPCI_H__ */
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>>> index 3aa8c3175f..8cc529ecec 100644
>>> --- a/xen/drivers/passthrough/pci.c
>>> +++ b/xen/drivers/passthrough/pci.c
>>> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>> const struct pci_dev_info *info, nodeid_t node)
>>> {
>>> struct pci_seg *pseg;
>>> - struct pci_dev *pdev;
>>> + struct pci_dev *pdev = NULL;
>>> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
>>> const char *pdev_type;
>>> int ret;
>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>
>>> check_pdev(pdev);
>>>
>>> +#ifdef CONFIG_ARM
>>> + /*
>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>> + */
>>> + ret = vpci_add_handlers(pdev);
>>> + if ( ret )
>>> + {
>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>> + goto out;
>>> + }
>>> +#endif
>>
>> I think vpci_add_handlers should be called after checking that
>> pdev->domain is != NULL, so I would move this chunk a bit below.
>
> On arm this would prevent the dom0less use case or to have the PCI
> bus enumerated from an other domain.
> @oleksandr: can you comment on this one, you might have a better
> answer than me on this ?
For my understanding, as also Jan pointed out, this should be put inside the
If ( !pdev->domain ) because also as the comment suggests, this operation
should be performed only when there is a Dom0 and when we are using
a dom0less setup the pdev->domain should be NULL if I’m not wrong.
@oleksandr maybe can confirm
>
>>
>>> +
>>> ret = 0;
>>> if ( !pdev->domain )
>>> {
>>> @@ -784,6 +797,9 @@ out:
>>> &PCI_SBDF(seg, bus, slot, func));
>>> }
>>> }
>>> + else if ( pdev )
>>> + pci_cleanup_msi(pdev);
>>
>> I'm slightly lost at why you add this chunk, is this strictly related
>> to the patch?
>
> This was discussed a lot in previous version of the patch and
> requested by Stefano. The idea here is that as soon as handlers
> are added some bits might be modified in the PCI config space
> leading possibly to msi interrupts. So it is safer to cleanup on the
> error path. For references please see discussion on v4 and v5 where
> this was actually added (to much references as the discussion was
> long so here [1] and [2] are the patchwork thread).
>
> [1] https://patchwork.kernel.org/project/xen-devel/patch/9bdca2cda5d2e83f94dc2423e55714273539760a.1633540842.git.rahul.singh@arm.com/
> [2] https://patchwork.kernel.org/project/xen-devel/patch/f093de681c2560a7196895bcd666ef8840885c1d.1633340795.git.rahul.singh@arm.com/
>
> Regards
> Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:09 ` Bertrand Marquis
@ 2021-10-15 10:14 ` Ian Jackson
2021-10-15 10:18 ` Jan Beulich
2021-10-15 10:38 ` Jan Beulich
1 sibling, 1 reply; 190+ messages in thread
From: Ian Jackson @ 2021-10-15 10:14 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Jan Beulich, Roger Pau Monné,
iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
> > On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
> > The latter is fine to be put here (i.e. FTAOD I'm fine with it
> > staying here). For the former I even question its original placement
> > in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
> > the bus portion of the address can be anywhere from 1 to 8 bits. And
> > in fact there is a reason why this macro was/is used in only a
> > single place, but not e.g. in x86'es handling of physical MCFG. It
> > is merely an implementation choice in vPCI that the entire segment 0
> > has a linear address range covering all 256 buses. Hence I think
> > this wants to move to xen/vpci.h and then perhaps also be named
> > VPCI_ECAM_BDF().
>
> On previous version it was request to renamed this to ECAM and agreed
> to put is here. Now you want me to rename it to VPCI and move it again.
> I would like to have a confirmation that this is ok and the final move if possible.
>
> @Roger can you confirm this is what is wanted ?
I think Roger is not available today I'm afraid.
Bertrand, can you give me a link to the comment from Roger ?
Assuming that it says what I think it will say:
I think the best thing to do will be to leave the name as it was in
the most recent version of your series. I don't think it makes sense
to block this patch over a naming disagreement. And it would be best
to minimise unnecessary churn.
I would be happy to release-ack a name change (perhaps proposed bo Jan
or Roger) supposing that that is the ultimate maintainer consensus.
Jan, would that approach be OK with you ?
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code
2021-10-15 9:53 ` Roger Pau Monné
2021-10-15 10:12 ` Bertrand Marquis
@ 2021-10-15 10:14 ` Jan Beulich
1 sibling, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 10:14 UTC (permalink / raw)
To: Roger Pau Monné, Bertrand Marquis
Cc: Xen-devel, iwj, Paul Durrant, Andrew Cooper, Wei Liu
On 15.10.2021 11:53, Roger Pau Monné wrote:
> On Fri, Oct 15, 2021 at 07:53:38AM +0000, Bertrand Marquis wrote:
>>> On 15 Oct 2021, at 08:44, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>> On Thu, Oct 14, 2021 at 03:49:49PM +0100, Bertrand Marquis wrote:
>>>> @@ -434,25 +420,8 @@ static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr,
>>>> reg = vpci_mmcfg_decode_addr(mmcfg, addr, &sbdf);
>>>> read_unlock(&d->arch.hvm.mmcfg_lock);
>>>>
>>>> - if ( !vpci_access_allowed(reg, len) ||
>>>> - (reg + len) > PCI_CFG_SPACE_EXP_SIZE )
>>>> - return X86EMUL_OKAY;
>>>> -
>>>> - /*
>>>> - * According to the PCIe 3.1A specification:
>>>> - * - Configuration Reads and Writes must usually be DWORD or smaller
>>>> - * in size.
>>>> - * - Because Root Complex implementations are not required to support
>>>> - * accesses to a RCRB that cross DW boundaries [...] software
>>>> - * should take care not to cause the generation of such accesses
>>>> - * when accessing a RCRB unless the Root Complex will support the
>>>> - * access.
>>>> - * Xen however supports 8byte accesses by splitting them into two
>>>> - * 4byte accesses.
>>>> - */
>>>> - *data = vpci_read(sbdf, reg, min(4u, len));
>>>> - if ( len == 8 )
>>>> - *data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32;
>>>> + /* Ignore return code */
>>>> + vpci_ecam_mmio_read(sbdf, reg, len, data);
>>>
>>> I think it would be better for vpci_ecam_mmio_read to just return the
>>> read value, or ~0 in case of error, at least that interface would be
>>> simpler and suitable for x86.
>>
>> I am not quite sure on this as on absolute to read ~0 is possible so the
>> caller cannot distinguish between properly reading ~0 or an access allowed error.
>
> How do you report an access allowed error on Arm for the PCI config
> space?
>
> At least on x86 I don't think we currently have a way to propagate
> such errors, neither a plan to do so that I'm aware.
I have to admit I can't even think of how such a plan could look like,
give x86'es history.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:13 ` Luca Fancellu
@ 2021-10-15 10:17 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 10:17 UTC (permalink / raw)
To: Luca Fancellu
Cc: Roger Pau Monné,
Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Jan Beulich, Wei Liu, Paul Durrant, Oleksandr Andrushchenko
Hi Luca,
> On 15 Oct 2021, at 11:13, Luca Fancellu <Luca.Fancellu@arm.com> wrote:
>
>
>
>> On 15 Oct 2021, at 10:52, Bertrand Marquis <Bertrand.Marquis@arm.com> wrote:
>>
>> Hi Roger,
>>
>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>> From: Rahul Singh <rahul.singh@arm.com>
>>>>
>>>> The existing VPCI support available for X86 is adapted for Arm.
>>>> When the device is added to XEN via the hyper call
>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>>> access is added to the Xen to emulate the PCI devices config space.
>>>>
>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>>>> so that when guest is trying to access the PCI config space,XEN
>>>> will trap the access and emulate read/write using the VPCI and
>>>> not the real PCI hardware.
>>>>
>>>> For Dom0less systems scan_pci_devices() would be used to discover the
>>>> PCI device in XEN and VPCI handler will be added during XEN boots.
>>>>
>>>> This patch is also doing some small fixes to fix compilation errors on
>>>> arm32 of vpci:
>>>> - add a cast to unsigned long in print in header.c
>>>> - add a cast to uint64_t in vpci_ecam_mmio_write
>>>>
>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>>> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com>
>>>> ---
>>>> Changes in v6:
>>>> - Use new vpci_ecam_ helpers for PCI access
>>>> - Do not set XEN_DOMCTL_CDF_vpci for dom0 for now (will be done in a
>>>> future patch once everything is ready)
>>>
>>> Isn't the series missing a revert of XEN_DOMCTL_CDF_vpci? I think
>>> that's what we agreed would be the way forward.
>>
>> A separate reverse patch for that has already been merged:
>> https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9516d01ac3015f522528ed6dafb3f584eaa7ed2c
>>
>>>
>>>> - rename REGISTER_OFFSET to ECAM_REG_OFFSET and move it to pci.h
>>>> - remove not needed local variables in vpci_mmio_write, the one in read
>>>> has been kept to ensure proper compilation on arm32
>>>> - move call to vpci_add_handlers before iommu init to simplify exit path
>>>> - move call to pci_cleanup_msi in the out section of pci_add_device if
>>>> pdev is not NULL and on error
>>>> - initialize pdev to NULL to handle properly exit path call of
>>>> pci_cleanup_msi
>>>> - keep has_vpci to return false for now as CFG_vpci has been removed.
>>>> Added a comment on top of the definition.
>>>> - fix compilation errors on arm32 (print in header.c, cast missing in
>>>> mmio_write.
>>>> - local variable was kept in vpci_mmio_read on arm to prevent a cast
>>>> error in arm32.
>>>> Change in v5:
>>>> - Add pci_cleanup_msi(pdev) incleanup part.
>>>> - Added Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
>>>> Change in v4:
>>>> - Move addition of XEN_DOMCTL_CDF_vpci flag to separate patch
>>>> Change in v3:
>>>> - Use is_pci_passthrough_enabled() in place of pci_passthrough_enabled variable
>>>> - Reject XEN_DOMCTL_CDF_vpci for x86 in arch_sanitise_domain_config()
>>>> - Remove IS_ENABLED(CONFIG_HAS_VPCI) from has_vpci()
>>>> Change in v2:
>>>> - Add new XEN_DOMCTL_CDF_vpci flag
>>>> - modify has_vpci() to include XEN_DOMCTL_CDF_vpci
>>>> - enable vpci support when pci-passthough option is enabled.
>>>> ---
>>>> ---
>>>> xen/arch/arm/Makefile | 1 +
>>>> xen/arch/arm/domain.c | 4 ++
>>>> xen/arch/arm/vpci.c | 74 +++++++++++++++++++++++++++++++++++
>>>> xen/arch/arm/vpci.h | 36 +++++++++++++++++
>>>> xen/drivers/passthrough/pci.c | 18 ++++++++-
>>>> xen/drivers/vpci/header.c | 3 +-
>>>> xen/drivers/vpci/vpci.c | 2 +-
>>>> xen/include/asm-arm/domain.h | 1 +
>>>> xen/include/asm-x86/pci.h | 2 -
>>>> xen/include/public/arch-arm.h | 7 ++++
>>>> xen/include/xen/pci.h | 3 ++
>>>> 11 files changed, 146 insertions(+), 5 deletions(-)
>>>> create mode 100644 xen/arch/arm/vpci.c
>>>> create mode 100644 xen/arch/arm/vpci.h
>>>>
>>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>>> index 64518848b2..07f634508e 100644
>>>> --- a/xen/arch/arm/Makefile
>>>> +++ b/xen/arch/arm/Makefile
>>>> @@ -7,6 +7,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>>> obj-y += platforms/
>>>> endif
>>>> obj-$(CONFIG_TEE) += tee/
>>>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>>>
>>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>>> obj-y += bootfdt.init.o
>>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>>> index eef0661beb..96e1b23550 100644
>>>> --- a/xen/arch/arm/domain.c
>>>> +++ b/xen/arch/arm/domain.c
>>>> @@ -39,6 +39,7 @@
>>>> #include <asm/vgic.h>
>>>> #include <asm/vtimer.h>
>>>>
>>>> +#include "vpci.h"
>>>> #include "vuart.h"
>>>>
>>>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>>> @@ -773,6 +774,9 @@ int arch_domain_create(struct domain *d,
>>>> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>>> goto fail;
>>>>
>>>> + if ( (rc = domain_vpci_init(d)) != 0 )
>>>> + goto fail;
>>>> +
>>>> return 0;
>>>>
>>>> fail:
>>>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>>>> new file mode 100644
>>>> index 0000000000..7c3552b65d
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/vpci.c
>>>> @@ -0,0 +1,74 @@
>>>> +/*
>>>> + * xen/arch/arm/vpci.c
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify
>>>> + * it under the terms of the GNU General Public License as published by
>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>> + * (at your option) any later version.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>> + * GNU General Public License for more details.
>>>> + */
>>>> +#include <xen/sched.h>
>>>> +#include <xen/vpci.h>
>>>> +
>>>> +#include <asm/mmio.h>
>>>> +
>>>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>>>> + register_t *r, void *p)
>>>> +{
>>>> + pci_sbdf_t sbdf;
>>>> + /* data is needed to prevent a pointer cast on 32bit */
>>>> + unsigned long data = ~0ul;
>>>> + int ret;
>>>> +
>>>> + /* We ignore segment part and always handle segment 0 */
>>>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>>>> +
>>>> + ret = vpci_ecam_mmio_read(sbdf, ECAM_REG_OFFSET(info->gpa),
>>>> + 1U << info->dabt.size, &data);
>>>> +
>>>> + *r = data;
>>>> +
>>>> + return ret;
>>>> +}
>>>> +
>>>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>>>> + register_t r, void *p)
>>>> +{
>>>> + pci_sbdf_t sbdf;
>>>> +
>>>> + /* We ignore segment part and always handle segment 0 */
>>>> + sbdf.sbdf = ECAM_BDF(info->gpa);
>>>> +
>>>> + return vpci_ecam_mmio_write(sbdf, ECAM_REG_OFFSET(info->gpa),
>>>> + 1U << info->dabt.size, r);
>>>> +}
>>>
>>> I'm not sure returning an error value here is helpful, as I'm not sure
>>> how native Arm behaves and how this error is propagated into the
>>> guest. FWIWreturning ~0 or dropping writes is what we do in x86 when
>>> vPCI is not capable of handling the access.
>>
>> Mmio handlers can take a return code on arm if something did not work
>> so I think this is the right way to do it on arm.
>> Now has agreed with Jan, we will change the return type of
>> vpci_ecam_write (also renamed) to be a boolean.
>>
>>>
>>>> +
>>>> +static const struct mmio_handler_ops vpci_mmio_handler = {
>>>> + .read = vpci_mmio_read,
>>>> + .write = vpci_mmio_write,
>>>> +};
>>>> +
>>>> +int domain_vpci_init(struct domain *d)
>>>> +{
>>>> + if ( !has_vpci(d) )
>>>> + return 0;
>>>> +
>>>> + register_mmio_handler(d, &vpci_mmio_handler,
>>>> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +/*
>>>> + * Local variables:
>>>> + * mode: C
>>>> + * c-file-style: "BSD"
>>>> + * c-basic-offset: 4
>>>> + * indent-tabs-mode: nil
>>>> + * End:
>>>> + */
>>>> +
>>>> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
>>>> new file mode 100644
>>>> index 0000000000..d8a7b0e3e8
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/vpci.h
>>>> @@ -0,0 +1,36 @@
>>>> +/*
>>>> + * xen/arch/arm/vpci.h
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify
>>>> + * it under the terms of the GNU General Public License as published by
>>>> + * the Free Software Foundation; either version 2 of the License, or
>>>> + * (at your option) any later version.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>> + * GNU General Public License for more details.
>>>> + */
>>>> +
>>>> +#ifndef __ARCH_ARM_VPCI_H__
>>>> +#define __ARCH_ARM_VPCI_H__
>>>> +
>>>> +#ifdef CONFIG_HAS_VPCI
>>>> +int domain_vpci_init(struct domain *d);
>>>> +#else
>>>> +static inline int domain_vpci_init(struct domain *d)
>>>> +{
>>>> + return 0;
>>>> +}
>>>> +#endif
>>>> +
>>>> +#endif /* __ARCH_ARM_VPCI_H__ */
>>>> +
>>>> +/*
>>>> + * Local variables:
>>>> + * mode: C
>>>> + * c-file-style: "BSD"
>>>> + * c-basic-offset: 4
>>>> + * indent-tabs-mode: nil
>>>> + * End:
>>>> + */
>>>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>>>> index 3aa8c3175f..8cc529ecec 100644
>>>> --- a/xen/drivers/passthrough/pci.c
>>>> +++ b/xen/drivers/passthrough/pci.c
>>>> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>> const struct pci_dev_info *info, nodeid_t node)
>>>> {
>>>> struct pci_seg *pseg;
>>>> - struct pci_dev *pdev;
>>>> + struct pci_dev *pdev = NULL;
>>>> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
>>>> const char *pdev_type;
>>>> int ret;
>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>
>>>> check_pdev(pdev);
>>>>
>>>> +#ifdef CONFIG_ARM
>>>> + /*
>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>> + */
>>>> + ret = vpci_add_handlers(pdev);
>>>> + if ( ret )
>>>> + {
>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>> + goto out;
>>>> + }
>>>> +#endif
>>>
>>> I think vpci_add_handlers should be called after checking that
>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>
>> On arm this would prevent the dom0less use case or to have the PCI
>> bus enumerated from an other domain.
>> @oleksandr: can you comment on this one, you might have a better
>> answer than me on this ?
>
> For my understanding, as also Jan pointed out, this should be put inside the
> If ( !pdev->domain ) because also as the comment suggests, this operation
> should be performed only when there is a Dom0 and when we are using
> a dom0less setup the pdev->domain should be NULL if I’m not wrong.
> @oleksandr maybe can confirm
You are right.
I will fix that and move that into an if.
Regards
Bertrand
>
>>
>>>
>>>> +
>>>> ret = 0;
>>>> if ( !pdev->domain )
>>>> {
>>>> @@ -784,6 +797,9 @@ out:
>>>> &PCI_SBDF(seg, bus, slot, func));
>>>> }
>>>> }
>>>> + else if ( pdev )
>>>> + pci_cleanup_msi(pdev);
>>>
>>> I'm slightly lost at why you add this chunk, is this strictly related
>>> to the patch?
>>
>> This was discussed a lot in previous version of the patch and
>> requested by Stefano. The idea here is that as soon as handlers
>> are added some bits might be modified in the PCI config space
>> leading possibly to msi interrupts. So it is safer to cleanup on the
>> error path. For references please see discussion on v4 and v5 where
>> this was actually added (to much references as the discussion was
>> long so here [1] and [2] are the patchwork thread).
>>
>> [1] https://patchwork.kernel.org/project/xen-devel/patch/9bdca2cda5d2e83f94dc2423e55714273539760a.1633540842.git.rahul.singh@arm.com/
>> [2] https://patchwork.kernel.org/project/xen-devel/patch/f093de681c2560a7196895bcd666ef8840885c1d.1633340795.git.rahul.singh@arm.com/
>>
>> Regards
>> Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:14 ` Ian Jackson
@ 2021-10-15 10:18 ` Jan Beulich
2021-10-15 11:35 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 10:18 UTC (permalink / raw)
To: Ian Jackson, Bertrand Marquis
Cc: Roger Pau Monné,
Rahul Singh, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Wei Liu, Paul Durrant, xen-devel
On 15.10.2021 12:14, Ian Jackson wrote:
> Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
>>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
>>> staying here). For the former I even question its original placement
>>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
>>> the bus portion of the address can be anywhere from 1 to 8 bits. And
>>> in fact there is a reason why this macro was/is used in only a
>>> single place, but not e.g. in x86'es handling of physical MCFG. It
>>> is merely an implementation choice in vPCI that the entire segment 0
>>> has a linear address range covering all 256 buses. Hence I think
>>> this wants to move to xen/vpci.h and then perhaps also be named
>>> VPCI_ECAM_BDF().
>>
>> On previous version it was request to renamed this to ECAM and agreed
>> to put is here. Now you want me to rename it to VPCI and move it again.
>> I would like to have a confirmation that this is ok and the final move if possible.
>>
>> @Roger can you confirm this is what is wanted ?
>
> I think Roger is not available today I'm afraid.
>
> Bertrand, can you give me a link to the comment from Roger ?
> Assuming that it says what I think it will say:
>
> I think the best thing to do will be to leave the name as it was in
> the most recent version of your series. I don't think it makes sense
> to block this patch over a naming disagreement. And it would be best
> to minimise unnecessary churn.
>
> I would be happy to release-ack a name change (perhaps proposed bo Jan
> or Roger) supposing that that is the ultimate maintainer consensus.
>
> Jan, would that approach be OK with you ?
Well, yes, if a subsequent name change is okay, then I could live with
that. I'd still find it odd to rename a function immediately after it
already got renamed. As expressed elsewhere, I suspect in his request
Roger did not pay attention to a use of the function in non-ECAM code.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 9:52 ` Bertrand Marquis
2021-10-15 10:13 ` Luca Fancellu
@ 2021-10-15 10:19 ` Roger Pau Monné
2021-10-15 10:31 ` Bertrand Marquis
2021-10-15 10:24 ` Jan Beulich
2 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 10:19 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Jan Beulich, Wei Liu, Paul Durrant, Oleksandr Andrushchenko
On Fri, Oct 15, 2021 at 09:52:28AM +0000, Bertrand Marquis wrote:
> Hi Roger,
>
> > On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
> >> From: Rahul Singh <rahul.singh@arm.com>
> >> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> >> index 3aa8c3175f..8cc529ecec 100644
> >> --- a/xen/drivers/passthrough/pci.c
> >> +++ b/xen/drivers/passthrough/pci.c
> >> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> >> const struct pci_dev_info *info, nodeid_t node)
> >> {
> >> struct pci_seg *pseg;
> >> - struct pci_dev *pdev;
> >> + struct pci_dev *pdev = NULL;
> >> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
> >> const char *pdev_type;
> >> int ret;
> >> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> >>
> >> check_pdev(pdev);
> >>
> >> +#ifdef CONFIG_ARM
> >> + /*
> >> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> >> + * Dom0 inform XEN to add the PCI devices in XEN.
> >> + */
> >> + ret = vpci_add_handlers(pdev);
> >> + if ( ret )
> >> + {
> >> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
> >> + goto out;
> >> + }
> >> +#endif
> >
> > I think vpci_add_handlers should be called after checking that
> > pdev->domain is != NULL, so I would move this chunk a bit below.
>
> On arm this would prevent the dom0less use case or to have the PCI
> bus enumerated from an other domain.
vpci_add_handlers will try to access pdev->domain, so passing a pdev
without domain being set is wrong.
> @oleksandr: can you comment on this one, you might have a better
> answer than me on this ?
>
> >
> >> +
> >> ret = 0;
> >> if ( !pdev->domain )
> >> {
> >> @@ -784,6 +797,9 @@ out:
> >> &PCI_SBDF(seg, bus, slot, func));
> >> }
> >> }
> >> + else if ( pdev )
> >> + pci_cleanup_msi(pdev);
> >
> > I'm slightly lost at why you add this chunk, is this strictly related
> > to the patch?
>
> This was discussed a lot in previous version of the patch and
> requested by Stefano. The idea here is that as soon as handlers
> are added some bits might be modified in the PCI config space
> leading possibly to msi interrupts. So it is safer to cleanup on the
> error path. For references please see discussion on v4 and v5 where
> this was actually added (to much references as the discussion was
> long so here [1] and [2] are the patchwork thread).
pci_add_device being solely used by trusted domains, I think it would
be fine to require that the domain doesn't poke the PCI config space
until the call to pci_add_device has finished. Else it's likely to get
inconsistent results, or mess up with the device state.
In any case, such addition needs some kind of reasoning added to the
commit message if it's really required.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 9:52 ` Bertrand Marquis
2021-10-15 10:13 ` Luca Fancellu
2021-10-15 10:19 ` Roger Pau Monné
@ 2021-10-15 10:24 ` Jan Beulich
2021-10-15 10:33 ` Bertrand Marquis
2 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 10:24 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, Oleksandr Andrushchenko,
Roger Pau Monné
On 15.10.2021 11:52, Bertrand Marquis wrote:
>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>
>>> check_pdev(pdev);
>>>
>>> +#ifdef CONFIG_ARM
>>> + /*
>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>> + */
>>> + ret = vpci_add_handlers(pdev);
>>> + if ( ret )
>>> + {
>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>> + goto out;
>>> + }
>>> +#endif
>>
>> I think vpci_add_handlers should be called after checking that
>> pdev->domain is != NULL, so I would move this chunk a bit below.
>
> On arm this would prevent the dom0less use case or to have the PCI
> bus enumerated from an other domain.
> @oleksandr: can you comment on this one, you might have a better
> answer than me on this ?
Well, without Xen doing the enumeration, some other entity would need
to do so, including the reporting to Xen. Obviously without a Dom0 it
would be ambiguous which domain to assign the device to; perhaps it
should be the caller in this case? That would make that caller domain
a pseudo-hwdom though, as far as PCI is concerned, which may not be
desirable according to my (limited) understanding of dom0less.
>>> @@ -784,6 +797,9 @@ out:
>>> &PCI_SBDF(seg, bus, slot, func));
>>> }
>>> }
>>> + else if ( pdev )
>>> + pci_cleanup_msi(pdev);
>>
>> I'm slightly lost at why you add this chunk, is this strictly related
>> to the patch?
>
> This was discussed a lot in previous version of the patch and
> requested by Stefano. The idea here is that as soon as handlers
> are added some bits might be modified in the PCI config space
> leading possibly to msi interrupts. So it is safer to cleanup on the
> error path. For references please see discussion on v4 and v5 where
> this was actually added (to much references as the discussion was
> long so here [1] and [2] are the patchwork thread).
>
> [1] https://patchwork.kernel.org/project/xen-devel/patch/9bdca2cda5d2e83f94dc2423e55714273539760a.1633540842.git.rahul.singh@arm.com/
> [2] https://patchwork.kernel.org/project/xen-devel/patch/f093de681c2560a7196895bcd666ef8840885c1d.1633340795.git.rahul.singh@arm.com/
The addition of this call has repeatedly raised questions. This is a
good indication that sufficient discussion thereof has been lacking
from the patch description.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:19 ` Roger Pau Monné
@ 2021-10-15 10:31 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 10:31 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Jan Beulich, Wei Liu, Paul Durrant, Oleksandr Andrushchenko
Hi,
> On 15 Oct 2021, at 11:19, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Fri, Oct 15, 2021 at 09:52:28AM +0000, Bertrand Marquis wrote:
>> Hi Roger,
>>
>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>> From: Rahul Singh <rahul.singh@arm.com>
>>>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>>>> index 3aa8c3175f..8cc529ecec 100644
>>>> --- a/xen/drivers/passthrough/pci.c
>>>> +++ b/xen/drivers/passthrough/pci.c
>>>> @@ -658,7 +658,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>> const struct pci_dev_info *info, nodeid_t node)
>>>> {
>>>> struct pci_seg *pseg;
>>>> - struct pci_dev *pdev;
>>>> + struct pci_dev *pdev = NULL;
>>>> unsigned int slot = PCI_SLOT(devfn), func = PCI_FUNC(devfn);
>>>> const char *pdev_type;
>>>> int ret;
>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>
>>>> check_pdev(pdev);
>>>>
>>>> +#ifdef CONFIG_ARM
>>>> + /*
>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>> + */
>>>> + ret = vpci_add_handlers(pdev);
>>>> + if ( ret )
>>>> + {
>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>> + goto out;
>>>> + }
>>>> +#endif
>>>
>>> I think vpci_add_handlers should be called after checking that
>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>
>> On arm this would prevent the dom0less use case or to have the PCI
>> bus enumerated from an other domain.
>
> vpci_add_handlers will try to access pdev->domain, so passing a pdev
> without domain being set is wrong.
Right and the initial comment from Rahul in the code is saying so.
I will move the block inside the if.
>
>> @oleksandr: can you comment on this one, you might have a better
>> answer than me on this ?
>>
>>>
>>>> +
>>>> ret = 0;
>>>> if ( !pdev->domain )
>>>> {
>>>> @@ -784,6 +797,9 @@ out:
>>>> &PCI_SBDF(seg, bus, slot, func));
>>>> }
>>>> }
>>>> + else if ( pdev )
>>>> + pci_cleanup_msi(pdev);
>>>
>>> I'm slightly lost at why you add this chunk, is this strictly related
>>> to the patch?
>>
>> This was discussed a lot in previous version of the patch and
>> requested by Stefano. The idea here is that as soon as handlers
>> are added some bits might be modified in the PCI config space
>> leading possibly to msi interrupts. So it is safer to cleanup on the
>> error path. For references please see discussion on v4 and v5 where
>> this was actually added (to much references as the discussion was
>> long so here [1] and [2] are the patchwork thread).
>
> pci_add_device being solely used by trusted domains, I think it would
> be fine to require that the domain doesn't poke the PCI config space
> until the call to pci_add_device has finished. Else it's likely to get
> inconsistent results, or mess up with the device state.
Ack.
>
> In any case, such addition needs some kind of reasoning added to the
> commit message if it's really required.
With the code moving inside the following else and pci_cleanup_msi being
empty for now on arm, I will remove the pci_cleanup_msi as it does not
related directly to this change for now and might change the behaviour on x86.
If this is needed at some point due to arm, this will be done once msi support will be added.
Regards
Bertrand
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:24 ` Jan Beulich
@ 2021-10-15 10:33 ` Bertrand Marquis
2021-10-15 10:41 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 10:33 UTC (permalink / raw)
To: Jan Beulich
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, Oleksandr Andrushchenko,
Roger Pau Monné
> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 15.10.2021 11:52, Bertrand Marquis wrote:
>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>
>>>> check_pdev(pdev);
>>>>
>>>> +#ifdef CONFIG_ARM
>>>> + /*
>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>> + */
>>>> + ret = vpci_add_handlers(pdev);
>>>> + if ( ret )
>>>> + {
>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>> + goto out;
>>>> + }
>>>> +#endif
>>>
>>> I think vpci_add_handlers should be called after checking that
>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>
>> On arm this would prevent the dom0less use case or to have the PCI
>> bus enumerated from an other domain.
>> @oleksandr: can you comment on this one, you might have a better
>> answer than me on this ?
>
> Well, without Xen doing the enumeration, some other entity would need
> to do so, including the reporting to Xen. Obviously without a Dom0 it
> would be ambiguous which domain to assign the device to; perhaps it
> should be the caller in this case? That would make that caller domain
> a pseudo-hwdom though, as far as PCI is concerned, which may not be
> desirable according to my (limited) understanding of dom0less.
This is not really related to this patch but the plan is the following:
- enumeration would have to be done by the firmware or boot loader before
- xen will have some code to detect PCI devices
- dom0less can be used to assign PCI devices to guest
Anyway does not change the fact that this must be called when domain is not NULL and I will fix that.
>
>>>> @@ -784,6 +797,9 @@ out:
>>>> &PCI_SBDF(seg, bus, slot, func));
>>>> }
>>>> }
>>>> + else if ( pdev )
>>>> + pci_cleanup_msi(pdev);
>>>
>>> I'm slightly lost at why you add this chunk, is this strictly related
>>> to the patch?
>>
>> This was discussed a lot in previous version of the patch and
>> requested by Stefano. The idea here is that as soon as handlers
>> are added some bits might be modified in the PCI config space
>> leading possibly to msi interrupts. So it is safer to cleanup on the
>> error path. For references please see discussion on v4 and v5 where
>> this was actually added (to much references as the discussion was
>> long so here [1] and [2] are the patchwork thread).
>>
>> [1] https://patchwork.kernel.org/project/xen-devel/patch/9bdca2cda5d2e83f94dc2423e55714273539760a.1633540842.git.rahul.singh@arm.com/
>> [2] https://patchwork.kernel.org/project/xen-devel/patch/f093de681c2560a7196895bcd666ef8840885c1d.1633340795.git.rahul.singh@arm.com/
>
> The addition of this call has repeatedly raised questions. This is a
> good indication that sufficient discussion thereof has been lacking
> from the patch description.
Yes and I will remove it as it only impacts x86 right now.
If this is needed, we will have to do it while adding MSI support on Arm.
Regards
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:09 ` Bertrand Marquis
2021-10-15 10:14 ` Ian Jackson
@ 2021-10-15 10:38 ` Jan Beulich
1 sibling, 0 replies; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 10:38 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Roger Pau Monné,
iwj, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
On 15.10.2021 12:09, Bertrand Marquis wrote:
>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>> On 14.10.2021 16:49, Bertrand Marquis wrote:
>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>
>>> check_pdev(pdev);
>>>
>>> +#ifdef CONFIG_ARM
>>> + /*
>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>> + */
>>> + ret = vpci_add_handlers(pdev);
>>> + if ( ret )
>>> + {
>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>> + goto out;
>>> + }
>>> +#endif
>>> +
>>> ret = 0;
>>> if ( !pdev->domain )
>>
>> Elsewhere I did point out that you need to be careful wrt pdev->domain
>> being NULL. As the code in context clearly documents, you're now
>> adding handlers before that field was set. In comments to a prior
>> version I did already suggest to consider placing the new code inside
>> the if() in question (albeit at the time this was mainly to make clear
>> that error cleanup may not fiddle with things not put in place by the
>> iommu_enable_device() alternative path). This would have the further
>> benefit of making is crystal clear that here only handlers for Dom0
>> would get put in place (and hence their installing for DomU-s would
>> need to be looked for elsewhere).
>
> I asked Oleksandr for confirmation here but on arm there will be 2 other use cases:
> - PCI own by a DomD so device enumeration done from there
> - dom0less with devices detection done inside Xen
Question is whether at this time you mean to handle all these cases.
Installing handlers when device detection happens in Xen might need
to be done differently, for example - e.g. more along the lines of
what x86 PVH Dom0 has done.
Anything you don't mean to handle (and is safe to be left out, i.e.
without breaking existing cases) will want spelling out in the
description.
>>> @@ -784,6 +797,9 @@ out:
>>> &PCI_SBDF(seg, bus, slot, func));
>>> }
>>> }
>>> + else if ( pdev )
>>> + pci_cleanup_msi(pdev);
>>
>> Have you thoroughly checked that this call is benign on x86? There's
>> no wording to that effect in the description afaics, and I can't
>> easily convince myself that it would be correct when the
>> iommu_enable_device() path was taken. (I'm also not going to
>> exclude that it should have been there even prior to your work,
>> albeit then adding this would want to be a separate bugfix patch.)
>
> This was not in the original serie and requested by Stefano. I must admit
> I am not completely sure on the details here so I am really ok to remove this
> but this would go against what was requested on the previous versions (4 and 5).
I understand this, but a request to add something still requires that
it be checked that the addition can't have bad effects in other cases.
A compromise here (which I wouldn't like very much) might be to add
another #ifdef CONFIG_ARM, solely to leave the existing x86 case
undisturbed. Yet then, with the called function doing nothing on Arm,
not adding the code here might as well be an option (which it looked
like Stefano would be amenable to). A TODO item would then perhaps
best be left here in a comment.
>>> --- a/xen/include/xen/pci.h
>>> +++ b/xen/include/xen/pci.h
>>> @@ -40,6 +40,9 @@
>>> #define PCI_SBDF3(s,b,df) \
>>> ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
>>>
>>> +#define ECAM_BDF(addr) ( ((addr) & 0x0ffff000) >> 12)
>>> +#define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
>>
>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
>> staying here). For the former I even question its original placement
>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
>> the bus portion of the address can be anywhere from 1 to 8 bits. And
>> in fact there is a reason why this macro was/is used in only a
>> single place, but not e.g. in x86'es handling of physical MCFG. It
>> is merely an implementation choice in vPCI that the entire segment 0
>> has a linear address range covering all 256 buses. Hence I think
>> this wants to move to xen/vpci.h and then perhaps also be named
>> VPCI_ECAM_BDF().
>
> On previous version it was request to renamed this to ECAM and agreed
> to put is here. Now you want me to rename it to VPCI and move it again.
> I would like to have a confirmation that this is ok and the final move
> if possible.
A final confirmation is, unfortunately, only as final as it can be at
the very moment it is given. It was the MCFG vs ECAM naming discussion
which made me pull out again the section in the spec, reminding me of
aspects I didn't previously take into consideration. I'm sorry for
this, but it's an iterative process on all sides.
So, FTAOD, as a maintainer of this headed I will continue to object to
a non-spec-compliant construct to be put into here. It'll be Roger,
being the vPCI maintainer, to confirm that putting it in xen/vpci.h is
fine.
> Also if I have to do this I will do the same for REG_OFFSET of course.
As said - that one, being in line with the PCI spec, is fine to remain
as and where you have it.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:33 ` Bertrand Marquis
@ 2021-10-15 10:41 ` Jan Beulich
2021-10-15 10:48 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 10:41 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, Oleksandr Andrushchenko,
Roger Pau Monné
On 15.10.2021 12:33, Bertrand Marquis wrote:
>> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
>> On 15.10.2021 11:52, Bertrand Marquis wrote:
>>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>>
>>>>> check_pdev(pdev);
>>>>>
>>>>> +#ifdef CONFIG_ARM
>>>>> + /*
>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>>> + */
>>>>> + ret = vpci_add_handlers(pdev);
>>>>> + if ( ret )
>>>>> + {
>>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>>> + goto out;
>>>>> + }
>>>>> +#endif
>>>>
>>>> I think vpci_add_handlers should be called after checking that
>>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>>
>>> On arm this would prevent the dom0less use case or to have the PCI
>>> bus enumerated from an other domain.
>>> @oleksandr: can you comment on this one, you might have a better
>>> answer than me on this ?
>>
>> Well, without Xen doing the enumeration, some other entity would need
>> to do so, including the reporting to Xen. Obviously without a Dom0 it
>> would be ambiguous which domain to assign the device to; perhaps it
>> should be the caller in this case? That would make that caller domain
>> a pseudo-hwdom though, as far as PCI is concerned, which may not be
>> desirable according to my (limited) understanding of dom0less.
>
> This is not really related to this patch but the plan is the following:
> - enumeration would have to be done by the firmware or boot loader before
> - xen will have some code to detect PCI devices
> - dom0less can be used to assign PCI devices to guest
>
> Anyway does not change the fact that this must be called when domain is
> not NULL and I will fix that.
Since we now all seem to agree that the NULL would have been a problem,
may I ask in how far any of this has actually been tested?
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:41 ` Jan Beulich
@ 2021-10-15 10:48 ` Bertrand Marquis
2021-10-15 10:51 ` Jan Beulich
2021-10-15 13:47 ` Roger Pau Monné
0 siblings, 2 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 10:48 UTC (permalink / raw)
To: Jan Beulich
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, Oleksandr Andrushchenko,
Roger Pau Monné
Hi Jan,
> On 15 Oct 2021, at 11:41, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 15.10.2021 12:33, Bertrand Marquis wrote:
>>> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 15.10.2021 11:52, Bertrand Marquis wrote:
>>>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>>>
>>>>>> check_pdev(pdev);
>>>>>>
>>>>>> +#ifdef CONFIG_ARM
>>>>>> + /*
>>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>>>> + */
>>>>>> + ret = vpci_add_handlers(pdev);
>>>>>> + if ( ret )
>>>>>> + {
>>>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>>>> + goto out;
>>>>>> + }
>>>>>> +#endif
>>>>>
>>>>> I think vpci_add_handlers should be called after checking that
>>>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>>>
>>>> On arm this would prevent the dom0less use case or to have the PCI
>>>> bus enumerated from an other domain.
>>>> @oleksandr: can you comment on this one, you might have a better
>>>> answer than me on this ?
>>>
>>> Well, without Xen doing the enumeration, some other entity would need
>>> to do so, including the reporting to Xen. Obviously without a Dom0 it
>>> would be ambiguous which domain to assign the device to; perhaps it
>>> should be the caller in this case? That would make that caller domain
>>> a pseudo-hwdom though, as far as PCI is concerned, which may not be
>>> desirable according to my (limited) understanding of dom0less.
>>
>> This is not really related to this patch but the plan is the following:
>> - enumeration would have to be done by the firmware or boot loader before
>> - xen will have some code to detect PCI devices
>> - dom0less can be used to assign PCI devices to guest
>>
>> Anyway does not change the fact that this must be called when domain is
>> not NULL and I will fix that.
>
> Since we now all seem to agree that the NULL would have been a problem,
> may I ask in how far any of this has actually been tested?
With the whole serie currently on gitlab we have extensively tested passing
through PCI devices on Arm in several configuration (number of device, MSI,
MSI-X) and check that PCI was still functional on x86.
With the patches pushed to Xen right now it was checked that:
- xen compiles properly on arm32, arm64 and x86
- xen compiles properly with VPCI activated (using a patch) on arm32 and arm64
- xen on x86 is functionnal (using basic test on QEMU)
- xen on arm64 is functionnal (with some extensive tests on different targets)
We are only lacking some actual testing on arm32 internally.
Regards
Bertrand
>
> Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:48 ` Bertrand Marquis
@ 2021-10-15 10:51 ` Jan Beulich
2021-10-15 11:08 ` Bertrand Marquis
2021-10-15 13:47 ` Roger Pau Monné
1 sibling, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 10:51 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, Oleksandr Andrushchenko,
Roger Pau Monné
On 15.10.2021 12:48, Bertrand Marquis wrote:
> Hi Jan,
>
>> On 15 Oct 2021, at 11:41, Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 15.10.2021 12:33, Bertrand Marquis wrote:
>>>> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
>>>> On 15.10.2021 11:52, Bertrand Marquis wrote:
>>>>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>>>>
>>>>>>> check_pdev(pdev);
>>>>>>>
>>>>>>> +#ifdef CONFIG_ARM
>>>>>>> + /*
>>>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>>>>> + */
>>>>>>> + ret = vpci_add_handlers(pdev);
>>>>>>> + if ( ret )
>>>>>>> + {
>>>>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>>>>> + goto out;
>>>>>>> + }
>>>>>>> +#endif
>>>>>>
>>>>>> I think vpci_add_handlers should be called after checking that
>>>>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>>>>
>>>>> On arm this would prevent the dom0less use case or to have the PCI
>>>>> bus enumerated from an other domain.
>>>>> @oleksandr: can you comment on this one, you might have a better
>>>>> answer than me on this ?
>>>>
>>>> Well, without Xen doing the enumeration, some other entity would need
>>>> to do so, including the reporting to Xen. Obviously without a Dom0 it
>>>> would be ambiguous which domain to assign the device to; perhaps it
>>>> should be the caller in this case? That would make that caller domain
>>>> a pseudo-hwdom though, as far as PCI is concerned, which may not be
>>>> desirable according to my (limited) understanding of dom0less.
>>>
>>> This is not really related to this patch but the plan is the following:
>>> - enumeration would have to be done by the firmware or boot loader before
>>> - xen will have some code to detect PCI devices
>>> - dom0less can be used to assign PCI devices to guest
>>>
>>> Anyway does not change the fact that this must be called when domain is
>>> not NULL and I will fix that.
>>
>> Since we now all seem to agree that the NULL would have been a problem,
>> may I ask in how far any of this has actually been tested?
>
> With the whole serie currently on gitlab we have extensively tested passing
> through PCI devices on Arm in several configuration (number of device, MSI,
> MSI-X) and check that PCI was still functional on x86.
>
> With the patches pushed to Xen right now it was checked that:
> - xen compiles properly on arm32, arm64 and x86
> - xen compiles properly with VPCI activated (using a patch) on arm32 and arm64
> - xen on x86 is functionnal (using basic test on QEMU)
> - xen on arm64 is functionnal (with some extensive tests on different targets)
But somehow in the testing done you must have avoided the code path
in question, or else you would have observed a crash.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 10:02 ` Ian Jackson
@ 2021-10-15 10:58 ` Michal Orzel
2021-10-15 11:04 ` Michal Orzel
2021-10-15 11:46 ` Ian Jackson
0 siblings, 2 replies; 190+ messages in thread
From: Michal Orzel @ 2021-10-15 10:58 UTC (permalink / raw)
To: Ian Jackson, Julien Grall
Cc: Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Andre Przywara
Hi,
On 15.10.2021 12:02, Ian Jackson wrote:
> Julien Grall writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
>> On 14/10/2021 18:54, Ian Jackson wrote:
> ...
>>> That is, as I understand it on ARM vpci should be enabled if
>>> passthrough is enabled and not otherwise. That is precisely what
>>> the variable c_info->passthrough is.
>>
>> On Arm, c_info->passthrough is also set when assigning platform devives
>> (e.g. a non-PCI network card). At least for now, we don't want to create
>> a node for vCPI in the Device-Tree because we don't enable the
>> emulation. So...
>
> Oh.
>
>>> 3. Now you can use d_config->c_info.passthrough to gate the addition
>>> of the additional stuff to the DT. Ie, that is a respin of this
>>> patch 3/3.
>>
>> ... we will need to check d_config->num_pcidevs for the time being.
>
> OK.
>
> Can you leave a comment somewhere (near where c_info.passthrough is
> set) pointing to this use of num_pcidevs ? That might save someone
> some future confusion.
>
Here, c_info->passthrough is set to enabled if either d_config->num_pcidevs or
d_config->num_dtdevs is set. Do you think we need to add there additional comment?
If so can you please help with what should I write there?
BTW. None of the patch I'm preparing with regards to this discussion modifies libxl_create
where c_info.passthrough i set. Do you still want me to add some comment there?
> Thanks,
> Ian.
>
Cheers,
Michal
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 10:58 ` Michal Orzel
@ 2021-10-15 11:04 ` Michal Orzel
2021-10-15 11:46 ` Ian Jackson
1 sibling, 0 replies; 190+ messages in thread
From: Michal Orzel @ 2021-10-15 11:04 UTC (permalink / raw)
To: Ian Jackson, Julien Grall
Cc: Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Andre Przywara
On 15.10.2021 12:58, Michal Orzel wrote:
> Hi,
>
> On 15.10.2021 12:02, Ian Jackson wrote:
>> Julien Grall writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
>>> On 14/10/2021 18:54, Ian Jackson wrote:
>> ...
>>>> That is, as I understand it on ARM vpci should be enabled if
>>>> passthrough is enabled and not otherwise. That is precisely what
>>>> the variable c_info->passthrough is.
>>>
>>> On Arm, c_info->passthrough is also set when assigning platform devives
>>> (e.g. a non-PCI network card). At least for now, we don't want to create
>>> a node for vCPI in the Device-Tree because we don't enable the
>>> emulation. So...
>>
>> Oh.
>>
>>>> 3. Now you can use d_config->c_info.passthrough to gate the addition
>>>> of the additional stuff to the DT. Ie, that is a respin of this
>>>> patch 3/3.
>>>
>>> ... we will need to check d_config->num_pcidevs for the time being.
>>
>> OK.
>>
>> Can you leave a comment somewhere (near where c_info.passthrough is
>> set) pointing to this use of num_pcidevs ? That might save someone
>> some future confusion.
>>
> Here, c_info->passthrough is set to enabled if either d_config->num_pcidevs or
> d_config->num_dtdevs is set. Do you think we need to add there additional comment?
> If so can you please help with what should I write there?
>
I forgot to add a link:
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libs/light/libxl_create.c;h=6ebb2bfc768d060fb898619be907fc973375cce5;hb=HEAD#l1099
> BTW. None of the patch I'm preparing with regards to this discussion modifies libxl_create
> where c_info.passthrough i set. Do you still want me to add some comment there?
>> Thanks,
>> Ian.
>>
> Cheers,
> Michal
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:51 ` Jan Beulich
@ 2021-10-15 11:08 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 11:08 UTC (permalink / raw)
To: Jan Beulich
Cc: Xen-devel, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, Oleksandr Andrushchenko,
Roger Pau Monné
Hi,
> On 15 Oct 2021, at 11:51, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 15.10.2021 12:48, Bertrand Marquis wrote:
>> Hi Jan,
>>
>>> On 15 Oct 2021, at 11:41, Jan Beulich <jbeulich@suse.com> wrote:
>>>
>>> On 15.10.2021 12:33, Bertrand Marquis wrote:
>>>>> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> On 15.10.2021 11:52, Bertrand Marquis wrote:
>>>>>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>>>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>>>>>
>>>>>>>> check_pdev(pdev);
>>>>>>>>
>>>>>>>> +#ifdef CONFIG_ARM
>>>>>>>> + /*
>>>>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>>>>>> + */
>>>>>>>> + ret = vpci_add_handlers(pdev);
>>>>>>>> + if ( ret )
>>>>>>>> + {
>>>>>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>>>>>> + goto out;
>>>>>>>> + }
>>>>>>>> +#endif
>>>>>>>
>>>>>>> I think vpci_add_handlers should be called after checking that
>>>>>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>>>>>
>>>>>> On arm this would prevent the dom0less use case or to have the PCI
>>>>>> bus enumerated from an other domain.
>>>>>> @oleksandr: can you comment on this one, you might have a better
>>>>>> answer than me on this ?
>>>>>
>>>>> Well, without Xen doing the enumeration, some other entity would need
>>>>> to do so, including the reporting to Xen. Obviously without a Dom0 it
>>>>> would be ambiguous which domain to assign the device to; perhaps it
>>>>> should be the caller in this case? That would make that caller domain
>>>>> a pseudo-hwdom though, as far as PCI is concerned, which may not be
>>>>> desirable according to my (limited) understanding of dom0less.
>>>>
>>>> This is not really related to this patch but the plan is the following:
>>>> - enumeration would have to be done by the firmware or boot loader before
>>>> - xen will have some code to detect PCI devices
>>>> - dom0less can be used to assign PCI devices to guest
>>>>
>>>> Anyway does not change the fact that this must be called when domain is
>>>> not NULL and I will fix that.
>>>
>>> Since we now all seem to agree that the NULL would have been a problem,
>>> may I ask in how far any of this has actually been tested?
>>
>> With the whole serie currently on gitlab we have extensively tested passing
>> through PCI devices on Arm in several configuration (number of device, MSI,
>> MSI-X) and check that PCI was still functional on x86.
>>
>> With the patches pushed to Xen right now it was checked that:
>> - xen compiles properly on arm32, arm64 and x86
>> - xen compiles properly with VPCI activated (using a patch) on arm32 and arm64
>> - xen on x86 is functionnal (using basic test on QEMU)
>> - xen on arm64 is functionnal (with some extensive tests on different targets)
>
> But somehow in the testing done you must have avoided the code path
> in question, or else you would have observed a crash.
Device_add is always called by dom0 so we did not have issues here.
Dom0less PCI passthrough implementation is not done right now.
So yes we did not go through this path.
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:18 ` Jan Beulich
@ 2021-10-15 11:35 ` Roger Pau Monné
2021-10-15 12:13 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 11:35 UTC (permalink / raw)
To: Jan Beulich
Cc: Ian Jackson, Bertrand Marquis, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, xen-devel
On Fri, Oct 15, 2021 at 12:18:59PM +0200, Jan Beulich wrote:
> On 15.10.2021 12:14, Ian Jackson wrote:
> > Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
> >>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
> >>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
> >>> staying here). For the former I even question its original placement
> >>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
> >>> the bus portion of the address can be anywhere from 1 to 8 bits. And
> >>> in fact there is a reason why this macro was/is used in only a
> >>> single place, but not e.g. in x86'es handling of physical MCFG. It
> >>> is merely an implementation choice in vPCI that the entire segment 0
> >>> has a linear address range covering all 256 buses. Hence I think
> >>> this wants to move to xen/vpci.h and then perhaps also be named
> >>> VPCI_ECAM_BDF().
> >>
> >> On previous version it was request to renamed this to ECAM and agreed
> >> to put is here. Now you want me to rename it to VPCI and move it again.
> >> I would like to have a confirmation that this is ok and the final move if possible.
> >>
> >> @Roger can you confirm this is what is wanted ?
> >
> > I think Roger is not available today I'm afraid.
> >
> > Bertrand, can you give me a link to the comment from Roger ?
> > Assuming that it says what I think it will say:
> >
> > I think the best thing to do will be to leave the name as it was in
> > the most recent version of your series. I don't think it makes sense
> > to block this patch over a naming disagreement. And it would be best
> > to minimise unnecessary churn.
> >
> > I would be happy to release-ack a name change (perhaps proposed bo Jan
> > or Roger) supposing that that is the ultimate maintainer consensus.
> >
> > Jan, would that approach be OK with you ?
>
> Well, yes, if a subsequent name change is okay, then I could live with
> that. I'd still find it odd to rename a function immediately after it
> already got renamed. As expressed elsewhere, I suspect in his request
> Roger did not pay attention to a use of the function in non-ECAM code.
Using MMCFG_BDF was original requested by Julien, not myself I think:
https://lore.kernel.org/xen-devel/a868e1e7-8400-45df-6eaa-69f1e2c99383@xen.org/
I'm slightly loss in so many messages. On x86 we subtract the MCFG
start address from the passed one before getting the BDF, and then we
add the startting bus address passed in the ACPI table. This is so far
not need on Arm AFAICT because of the fixed nature of the selected
virtual ECAM region.
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 10:58 ` Michal Orzel
2021-10-15 11:04 ` Michal Orzel
@ 2021-10-15 11:46 ` Ian Jackson
2021-10-15 11:53 ` Michal Orzel
1 sibling, 1 reply; 190+ messages in thread
From: Ian Jackson @ 2021-10-15 11:46 UTC (permalink / raw)
To: Michal Orzel
Cc: Julien Grall, Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Andre Przywara
Michal Orzel writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
> On 15.10.2021 12:02, Ian Jackson wrote:
> > Can you leave a comment somewhere (near where c_info.passthrough is
> > set) pointing to this use of num_pcidevs ? That might save someone
> > some future confusion.
>
> Here, c_info->passthrough is set to enabled if either d_config->num_pcidevs or
> d_config->num_dtdevs is set. Do you think we need to add there additional comment?
> If so can you please help with what should I write there?
>
> BTW. None of the patch I'm preparing with regards to this discussion modifies libxl_create
> where c_info.passthrough i set. Do you still want me to add some comment there?
Yes, I think so. I won't insisting on it if you feel it doesn't make
sense.
Maybe something like
// NB, on ARM, libxl__arch_blah directly examines num_pcidevs to
// decide whether to enable vpci, rather than using c_info->passthrough.
// This will be insufficient if and when ARM does PCI hotplug.
?
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 11:46 ` Ian Jackson
@ 2021-10-15 11:53 ` Michal Orzel
2021-10-15 12:10 ` Julien Grall
2021-10-15 12:13 ` Ian Jackson
0 siblings, 2 replies; 190+ messages in thread
From: Michal Orzel @ 2021-10-15 11:53 UTC (permalink / raw)
To: Ian Jackson
Cc: Julien Grall, Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Andre Przywara
On 15.10.2021 13:46, Ian Jackson wrote:
> Michal Orzel writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
>> On 15.10.2021 12:02, Ian Jackson wrote:
>>> Can you leave a comment somewhere (near where c_info.passthrough is
>>> set) pointing to this use of num_pcidevs ? That might save someone
>>> some future confusion.
>>
>> Here, c_info->passthrough is set to enabled if either d_config->num_pcidevs or
>> d_config->num_dtdevs is set. Do you think we need to add there additional comment?
>> If so can you please help with what should I write there?
>>
>> BTW. None of the patch I'm preparing with regards to this discussion modifies libxl_create
>> where c_info.passthrough i set. Do you still want me to add some comment there?
>
> Yes, I think so. I won't insisting on it if you feel it doesn't make
> sense.
>
> Maybe something like
>
> // NB, on ARM, libxl__arch_blah directly examines num_pcidevs to
> // decide whether to enable vpci, rather than using c_info->passthrough.
> // This will be insufficient if and when ARM does PCI hotplug.
>
Well we are not enabling vpci. We are creating a DT node for it.
So either I will write:
/*
* Note: libxl_arm directly examines num_pcidevs to decide whether
* to create a vPCI DT node, rather than using c_info->passthrough.
* This will be insufficient if and when ARM does PCI hotplug.
*/
or I will not add any comment (it can be add in the future when "enabling" vpci).
@Julien?
> ?
>
> Ian.
>
Cheers,
Michal
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 11:53 ` Michal Orzel
@ 2021-10-15 12:10 ` Julien Grall
2021-10-15 12:14 ` Ian Jackson
2021-10-15 12:13 ` Ian Jackson
1 sibling, 1 reply; 190+ messages in thread
From: Julien Grall @ 2021-10-15 12:10 UTC (permalink / raw)
To: Michal Orzel, Ian Jackson
Cc: Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Andre Przywara
Hi Michal,
On 15/10/2021 12:53, Michal Orzel wrote:
>
>
> On 15.10.2021 13:46, Ian Jackson wrote:
>> Michal Orzel writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
>>> On 15.10.2021 12:02, Ian Jackson wrote:
>>>> Can you leave a comment somewhere (near where c_info.passthrough is
>>>> set) pointing to this use of num_pcidevs ? That might save someone
>>>> some future confusion.
>>>
>>> Here, c_info->passthrough is set to enabled if either d_config->num_pcidevs or
>>> d_config->num_dtdevs is set. Do you think we need to add there additional comment?
>>> If so can you please help with what should I write there?
>>>
>>> BTW. None of the patch I'm preparing with regards to this discussion modifies libxl_create
>>> where c_info.passthrough i set. Do you still want me to add some comment there?
>>
>> Yes, I think so. I won't insisting on it if you feel it doesn't make
>> sense.
>>
>> Maybe something like
>>
>> // NB, on ARM, libxl__arch_blah directly examines num_pcidevs to
>> // decide whether to enable vpci, rather than using c_info->passthrough.
>> // This will be insufficient if and when ARM does PCI hotplug.
>>
> Well we are not enabling vpci. We are creating a DT node for it.
> So either I will write:
> /*
> * Note: libxl_arm directly examines num_pcidevs to decide whether
> * to create a vPCI DT node, rather than using c_info->passthrough.
> * This will be insufficient if and when ARM does PCI hotplug.
> */
>
> or I will not add any comment (it can be add in the future when "enabling" vpci).
> @Julien?
I would prefer if we had a comment. Your proposal makes more sense as we
only create the DT.
Long term, I would expect a similar check to be necessary to set the
vCPI flag at the domain creation. So it would be best to introduce an
internal field 'vpci' to avoid duplicating that check. (Note I am not
requesting this change for Xen 4.16).
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 11:53 ` Michal Orzel
2021-10-15 12:10 ` Julien Grall
@ 2021-10-15 12:13 ` Ian Jackson
1 sibling, 0 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-15 12:13 UTC (permalink / raw)
To: Michal Orzel
Cc: Julien Grall, Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Andre Przywara
Michal Orzel writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
> On 15.10.2021 13:46, Ian Jackson wrote:
> > Maybe something like
> >
> > // NB, on ARM, libxl__arch_blah directly examines num_pcidevs to
> > // decide whether to enable vpci, rather than using c_info->passthrough.
> > // This will be insufficient if and when ARM does PCI hotplug.
>
> Well we are not enabling vpci. We are creating a DT node for it.
> So either I will write:
> /*
> * Note: libxl_arm directly examines num_pcidevs to decide whether
> * to create a vPCI DT node, rather than using c_info->passthrough.
> * This will be insufficient if and when ARM does PCI hotplug.
> */
That sounds good to me. Thank you for correcting me :-).
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 11:35 ` Roger Pau Monné
@ 2021-10-15 12:13 ` Bertrand Marquis
2021-10-15 12:18 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 12:13 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Jan Beulich, Ian Jackson, Rahul Singh, Stefano Stabellini,
Julien Grall, Volodymyr Babchuk, Andrew Cooper, George Dunlap,
Wei Liu, Paul Durrant, xen-devel
Hi Roger,
> On 15 Oct 2021, at 12:35, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Fri, Oct 15, 2021 at 12:18:59PM +0200, Jan Beulich wrote:
>> On 15.10.2021 12:14, Ian Jackson wrote:
>>> Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
>>>>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
>>>>> staying here). For the former I even question its original placement
>>>>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
>>>>> the bus portion of the address can be anywhere from 1 to 8 bits. And
>>>>> in fact there is a reason why this macro was/is used in only a
>>>>> single place, but not e.g. in x86'es handling of physical MCFG. It
>>>>> is merely an implementation choice in vPCI that the entire segment 0
>>>>> has a linear address range covering all 256 buses. Hence I think
>>>>> this wants to move to xen/vpci.h and then perhaps also be named
>>>>> VPCI_ECAM_BDF().
>>>>
>>>> On previous version it was request to renamed this to ECAM and agreed
>>>> to put is here. Now you want me to rename it to VPCI and move it again.
>>>> I would like to have a confirmation that this is ok and the final move if possible.
>>>>
>>>> @Roger can you confirm this is what is wanted ?
>>>
>>> I think Roger is not available today I'm afraid.
>>>
>>> Bertrand, can you give me a link to the comment from Roger ?
>>> Assuming that it says what I think it will say:
>>>
>>> I think the best thing to do will be to leave the name as it was in
>>> the most recent version of your series. I don't think it makes sense
>>> to block this patch over a naming disagreement. And it would be best
>>> to minimise unnecessary churn.
>>>
>>> I would be happy to release-ack a name change (perhaps proposed bo Jan
>>> or Roger) supposing that that is the ultimate maintainer consensus.
>>>
>>> Jan, would that approach be OK with you ?
>>
>> Well, yes, if a subsequent name change is okay, then I could live with
>> that. I'd still find it odd to rename a function immediately after it
>> already got renamed. As expressed elsewhere, I suspect in his request
>> Roger did not pay attention to a use of the function in non-ECAM code.
>
> Using MMCFG_BDF was original requested by Julien, not myself I think:
>
> https://lore.kernel.org/xen-devel/a868e1e7-8400-45df-6eaa-69f1e2c99383@xen.org/
>
> I'm slightly loss in so many messages. On x86 we subtract the MCFG
> start address from the passed one before getting the BDF, and then we
> add the startting bus address passed in the ACPI table. This is so far
> not need on Arm AFAICT because of the fixed nature of the selected
> virtual ECAM region.
At the end my patch will add in xen/pci.h:
#define ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12)
#define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
Now seeing the comment the question is should those be renamed with a VPCI
prefix and be moved to xen/vpci.h.
So far ECAM_BDF is only used in vpci_mmcfg_decode_addr which is only called
before calling vpci_ecam_{read/write}.
ECAM_REG_OFFSET is only used in arm vpci code.
Do you think the current state is ok of the renaming + moving should be done ?
Cheers
Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]
2021-10-15 12:10 ` Julien Grall
@ 2021-10-15 12:14 ` Ian Jackson
0 siblings, 0 replies; 190+ messages in thread
From: Ian Jackson @ 2021-10-15 12:14 UTC (permalink / raw)
To: Julien Grall
Cc: Michal Orzel, Bertrand Marquis, xen-devel, Rahul Singh, Wei Liu,
Anthony PERARD, Juergen Gross, Stefano Stabellini,
Volodymyr Babchuk, Andre Przywara
Julien Grall writes ("Re: [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages]"):
> Long term, I would expect a similar check to be necessary to set the
> vCPI flag at the domain creation. So it would be best to introduce an
> internal field 'vpci' to avoid duplicating that check. (Note I am not
> requesting this change for Xen 4.16).
Yes, I agree with all of that.
Ian.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 12:13 ` Bertrand Marquis
@ 2021-10-15 12:18 ` Jan Beulich
2021-10-15 12:28 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 12:18 UTC (permalink / raw)
To: Bertrand Marquis, Roger Pau Monné
Cc: Ian Jackson, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
On 15.10.2021 14:13, Bertrand Marquis wrote:
> Hi Roger,
>
>> On 15 Oct 2021, at 12:35, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>
>> On Fri, Oct 15, 2021 at 12:18:59PM +0200, Jan Beulich wrote:
>>> On 15.10.2021 12:14, Ian Jackson wrote:
>>>> Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
>>>>>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
>>>>>> staying here). For the former I even question its original placement
>>>>>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
>>>>>> the bus portion of the address can be anywhere from 1 to 8 bits. And
>>>>>> in fact there is a reason why this macro was/is used in only a
>>>>>> single place, but not e.g. in x86'es handling of physical MCFG. It
>>>>>> is merely an implementation choice in vPCI that the entire segment 0
>>>>>> has a linear address range covering all 256 buses. Hence I think
>>>>>> this wants to move to xen/vpci.h and then perhaps also be named
>>>>>> VPCI_ECAM_BDF().
>>>>>
>>>>> On previous version it was request to renamed this to ECAM and agreed
>>>>> to put is here. Now you want me to rename it to VPCI and move it again.
>>>>> I would like to have a confirmation that this is ok and the final move if possible.
>>>>>
>>>>> @Roger can you confirm this is what is wanted ?
>>>>
>>>> I think Roger is not available today I'm afraid.
>>>>
>>>> Bertrand, can you give me a link to the comment from Roger ?
>>>> Assuming that it says what I think it will say:
>>>>
>>>> I think the best thing to do will be to leave the name as it was in
>>>> the most recent version of your series. I don't think it makes sense
>>>> to block this patch over a naming disagreement. And it would be best
>>>> to minimise unnecessary churn.
>>>>
>>>> I would be happy to release-ack a name change (perhaps proposed bo Jan
>>>> or Roger) supposing that that is the ultimate maintainer consensus.
>>>>
>>>> Jan, would that approach be OK with you ?
>>>
>>> Well, yes, if a subsequent name change is okay, then I could live with
>>> that. I'd still find it odd to rename a function immediately after it
>>> already got renamed. As expressed elsewhere, I suspect in his request
>>> Roger did not pay attention to a use of the function in non-ECAM code.
>>
>> Using MMCFG_BDF was original requested by Julien, not myself I think:
>>
>> https://lore.kernel.org/xen-devel/a868e1e7-8400-45df-6eaa-69f1e2c99383@xen.org/
>>
>> I'm slightly loss in so many messages. On x86 we subtract the MCFG
>> start address from the passed one before getting the BDF, and then we
>> add the startting bus address passed in the ACPI table. This is so far
>> not need on Arm AFAICT because of the fixed nature of the selected
>> virtual ECAM region.
>
> At the end my patch will add in xen/pci.h:
> #define ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12)
Since you still make this proposal, once again: I'm not going to
accept such a macro in this header, whatever the name. Its prior
placement was wrong as well. Only ...
> #define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
... this one is fine to live here (and presumably it could gain uses
elsewhere).
Jan
> Now seeing the comment the question is should those be renamed with a VPCI
> prefix and be moved to xen/vpci.h.
>
> So far ECAM_BDF is only used in vpci_mmcfg_decode_addr which is only called
> before calling vpci_ecam_{read/write}.
>
> ECAM_REG_OFFSET is only used in arm vpci code.
>
> Do you think the current state is ok of the renaming + moving should be done ?
>
> Cheers
> Bertrand
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 12:18 ` Jan Beulich
@ 2021-10-15 12:28 ` Bertrand Marquis
2021-10-15 13:00 ` Jan Beulich
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 12:28 UTC (permalink / raw)
To: Jan Beulich
Cc: Roger Pau Monné,
Ian Jackson, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
Hi Jan,
> On 15 Oct 2021, at 13:18, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 15.10.2021 14:13, Bertrand Marquis wrote:
>> Hi Roger,
>>
>>> On 15 Oct 2021, at 12:35, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>
>>> On Fri, Oct 15, 2021 at 12:18:59PM +0200, Jan Beulich wrote:
>>>> On 15.10.2021 12:14, Ian Jackson wrote:
>>>>> Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
>>>>>>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
>>>>>>> staying here). For the former I even question its original placement
>>>>>>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
>>>>>>> the bus portion of the address can be anywhere from 1 to 8 bits. And
>>>>>>> in fact there is a reason why this macro was/is used in only a
>>>>>>> single place, but not e.g. in x86'es handling of physical MCFG. It
>>>>>>> is merely an implementation choice in vPCI that the entire segment 0
>>>>>>> has a linear address range covering all 256 buses. Hence I think
>>>>>>> this wants to move to xen/vpci.h and then perhaps also be named
>>>>>>> VPCI_ECAM_BDF().
>>>>>>
>>>>>> On previous version it was request to renamed this to ECAM and agreed
>>>>>> to put is here. Now you want me to rename it to VPCI and move it again.
>>>>>> I would like to have a confirmation that this is ok and the final move if possible.
>>>>>>
>>>>>> @Roger can you confirm this is what is wanted ?
>>>>>
>>>>> I think Roger is not available today I'm afraid.
>>>>>
>>>>> Bertrand, can you give me a link to the comment from Roger ?
>>>>> Assuming that it says what I think it will say:
>>>>>
>>>>> I think the best thing to do will be to leave the name as it was in
>>>>> the most recent version of your series. I don't think it makes sense
>>>>> to block this patch over a naming disagreement. And it would be best
>>>>> to minimise unnecessary churn.
>>>>>
>>>>> I would be happy to release-ack a name change (perhaps proposed bo Jan
>>>>> or Roger) supposing that that is the ultimate maintainer consensus.
>>>>>
>>>>> Jan, would that approach be OK with you ?
>>>>
>>>> Well, yes, if a subsequent name change is okay, then I could live with
>>>> that. I'd still find it odd to rename a function immediately after it
>>>> already got renamed. As expressed elsewhere, I suspect in his request
>>>> Roger did not pay attention to a use of the function in non-ECAM code.
>>>
>>> Using MMCFG_BDF was original requested by Julien, not myself I think:
>>>
>>> https://lore.kernel.org/xen-devel/a868e1e7-8400-45df-6eaa-69f1e2c99383@xen.org/
>>>
>>> I'm slightly loss in so many messages. On x86 we subtract the MCFG
>>> start address from the passed one before getting the BDF, and then we
>>> add the startting bus address passed in the ACPI table. This is so far
>>> not need on Arm AFAICT because of the fixed nature of the selected
>>> virtual ECAM region.
>>
>> At the end my patch will add in xen/pci.h:
>> #define ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12)
>
> Since you still make this proposal, once again: I'm not going to
> accept such a macro in this header, whatever the name. Its prior
> placement was wrong as well. Only ...
>
>> #define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
>
> ... this one is fine to live here (and presumably it could gain uses
> elsewhere).
So you would agree if they are both moved to vpci.h with a VPCI_ prefix ?
Bertrand
>
> Jan
>
>> Now seeing the comment the question is should those be renamed with a VPCI
>> prefix and be moved to xen/vpci.h.
>>
>> So far ECAM_BDF is only used in vpci_mmcfg_decode_addr which is only called
>> before calling vpci_ecam_{read/write}.
>>
>> ECAM_REG_OFFSET is only used in arm vpci code.
>>
>> Do you think the current state is ok of the renaming + moving should be done ?
>>
>> Cheers
>> Bertrand
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 12:28 ` Bertrand Marquis
@ 2021-10-15 13:00 ` Jan Beulich
2021-10-15 13:10 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Jan Beulich @ 2021-10-15 13:00 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Roger Pau Monné,
Ian Jackson, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
On 15.10.2021 14:28, Bertrand Marquis wrote:
>> On 15 Oct 2021, at 13:18, Jan Beulich <jbeulich@suse.com> wrote:
>> On 15.10.2021 14:13, Bertrand Marquis wrote:
>>>> On 15 Oct 2021, at 12:35, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>> On Fri, Oct 15, 2021 at 12:18:59PM +0200, Jan Beulich wrote:
>>>>> On 15.10.2021 12:14, Ian Jackson wrote:
>>>>>> Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
>>>>>>>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>>>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
>>>>>>>> staying here). For the former I even question its original placement
>>>>>>>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
>>>>>>>> the bus portion of the address can be anywhere from 1 to 8 bits. And
>>>>>>>> in fact there is a reason why this macro was/is used in only a
>>>>>>>> single place, but not e.g. in x86'es handling of physical MCFG. It
>>>>>>>> is merely an implementation choice in vPCI that the entire segment 0
>>>>>>>> has a linear address range covering all 256 buses. Hence I think
>>>>>>>> this wants to move to xen/vpci.h and then perhaps also be named
>>>>>>>> VPCI_ECAM_BDF().
>>>>>>>
>>>>>>> On previous version it was request to renamed this to ECAM and agreed
>>>>>>> to put is here. Now you want me to rename it to VPCI and move it again.
>>>>>>> I would like to have a confirmation that this is ok and the final move if possible.
>>>>>>>
>>>>>>> @Roger can you confirm this is what is wanted ?
>>>>>>
>>>>>> I think Roger is not available today I'm afraid.
>>>>>>
>>>>>> Bertrand, can you give me a link to the comment from Roger ?
>>>>>> Assuming that it says what I think it will say:
>>>>>>
>>>>>> I think the best thing to do will be to leave the name as it was in
>>>>>> the most recent version of your series. I don't think it makes sense
>>>>>> to block this patch over a naming disagreement. And it would be best
>>>>>> to minimise unnecessary churn.
>>>>>>
>>>>>> I would be happy to release-ack a name change (perhaps proposed bo Jan
>>>>>> or Roger) supposing that that is the ultimate maintainer consensus.
>>>>>>
>>>>>> Jan, would that approach be OK with you ?
>>>>>
>>>>> Well, yes, if a subsequent name change is okay, then I could live with
>>>>> that. I'd still find it odd to rename a function immediately after it
>>>>> already got renamed. As expressed elsewhere, I suspect in his request
>>>>> Roger did not pay attention to a use of the function in non-ECAM code.
>>>>
>>>> Using MMCFG_BDF was original requested by Julien, not myself I think:
>>>>
>>>> https://lore.kernel.org/xen-devel/a868e1e7-8400-45df-6eaa-69f1e2c99383@xen.org/
>>>>
>>>> I'm slightly loss in so many messages. On x86 we subtract the MCFG
>>>> start address from the passed one before getting the BDF, and then we
>>>> add the startting bus address passed in the ACPI table. This is so far
>>>> not need on Arm AFAICT because of the fixed nature of the selected
>>>> virtual ECAM region.
>>>
>>> At the end my patch will add in xen/pci.h:
>>> #define ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12)
>>
>> Since you still make this proposal, once again: I'm not going to
>> accept such a macro in this header, whatever the name. Its prior
>> placement was wrong as well. Only ...
>>
>>> #define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
>>
>> ... this one is fine to live here (and presumably it could gain uses
>> elsewhere).
>
> So you would agree if they are both moved to vpci.h with a VPCI_ prefix ?
I wouldn't object, but as said several times before I see no reason
to also move and rename ECAM_REG_OFFSET(). If you moved it and if
later we find a use for it outside of vPCI, we'd need to rename and
move it again.
Jan
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 13:00 ` Jan Beulich
@ 2021-10-15 13:10 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-15 13:10 UTC (permalink / raw)
To: Jan Beulich
Cc: Roger Pau Monné,
Ian Jackson, Rahul Singh, Stefano Stabellini, Julien Grall,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Wei Liu,
Paul Durrant, xen-devel
Hi Jan,
> On 15 Oct 2021, at 14:00, Jan Beulich <jbeulich@suse.com> wrote:
>
> On 15.10.2021 14:28, Bertrand Marquis wrote:
>>> On 15 Oct 2021, at 13:18, Jan Beulich <jbeulich@suse.com> wrote:
>>> On 15.10.2021 14:13, Bertrand Marquis wrote:
>>>>> On 15 Oct 2021, at 12:35, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>> On Fri, Oct 15, 2021 at 12:18:59PM +0200, Jan Beulich wrote:
>>>>>> On 15.10.2021 12:14, Ian Jackson wrote:
>>>>>>> Bertrand Marquis writes ("Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM."):
>>>>>>>>> On 15 Oct 2021, at 09:00, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>>>>> The latter is fine to be put here (i.e. FTAOD I'm fine with it
>>>>>>>>> staying here). For the former I even question its original placement
>>>>>>>>> in asm-x86/pci.h: It's not generally correct as per the PCI spec, as
>>>>>>>>> the bus portion of the address can be anywhere from 1 to 8 bits. And
>>>>>>>>> in fact there is a reason why this macro was/is used in only a
>>>>>>>>> single place, but not e.g. in x86'es handling of physical MCFG. It
>>>>>>>>> is merely an implementation choice in vPCI that the entire segment 0
>>>>>>>>> has a linear address range covering all 256 buses. Hence I think
>>>>>>>>> this wants to move to xen/vpci.h and then perhaps also be named
>>>>>>>>> VPCI_ECAM_BDF().
>>>>>>>>
>>>>>>>> On previous version it was request to renamed this to ECAM and agreed
>>>>>>>> to put is here. Now you want me to rename it to VPCI and move it again.
>>>>>>>> I would like to have a confirmation that this is ok and the final move if possible.
>>>>>>>>
>>>>>>>> @Roger can you confirm this is what is wanted ?
>>>>>>>
>>>>>>> I think Roger is not available today I'm afraid.
>>>>>>>
>>>>>>> Bertrand, can you give me a link to the comment from Roger ?
>>>>>>> Assuming that it says what I think it will say:
>>>>>>>
>>>>>>> I think the best thing to do will be to leave the name as it was in
>>>>>>> the most recent version of your series. I don't think it makes sense
>>>>>>> to block this patch over a naming disagreement. And it would be best
>>>>>>> to minimise unnecessary churn.
>>>>>>>
>>>>>>> I would be happy to release-ack a name change (perhaps proposed bo Jan
>>>>>>> or Roger) supposing that that is the ultimate maintainer consensus.
>>>>>>>
>>>>>>> Jan, would that approach be OK with you ?
>>>>>>
>>>>>> Well, yes, if a subsequent name change is okay, then I could live with
>>>>>> that. I'd still find it odd to rename a function immediately after it
>>>>>> already got renamed. As expressed elsewhere, I suspect in his request
>>>>>> Roger did not pay attention to a use of the function in non-ECAM code.
>>>>>
>>>>> Using MMCFG_BDF was original requested by Julien, not myself I think:
>>>>>
>>>>> https://lore.kernel.org/xen-devel/a868e1e7-8400-45df-6eaa-69f1e2c99383@xen.org/
>>>>>
>>>>> I'm slightly loss in so many messages. On x86 we subtract the MCFG
>>>>> start address from the passed one before getting the BDF, and then we
>>>>> add the startting bus address passed in the ACPI table. This is so far
>>>>> not need on Arm AFAICT because of the fixed nature of the selected
>>>>> virtual ECAM region.
>>>>
>>>> At the end my patch will add in xen/pci.h:
>>>> #define ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12)
>>>
>>> Since you still make this proposal, once again: I'm not going to
>>> accept such a macro in this header, whatever the name. Its prior
>>> placement was wrong as well. Only ...
>>>
>>>> #define ECAM_REG_OFFSET(addr) ((addr) & 0x00000fff)
>>>
>>> ... this one is fine to live here (and presumably it could gain uses
>>> elsewhere).
>>
>> So you would agree if they are both moved to vpci.h with a VPCI_ prefix ?
>
> I wouldn't object, but as said several times before I see no reason
> to also move and rename ECAM_REG_OFFSET(). If you moved it and if
> later we find a use for it outside of vPCI, we'd need to rename and
> move it again.
I will move BDF to vpci.h and had VPCI prefix and keep REG_OFFSET as
it is and where it is then.
Cheers
Bertrand
>
> Jan
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 10:48 ` Bertrand Marquis
2021-10-15 10:51 ` Jan Beulich
@ 2021-10-15 13:47 ` Roger Pau Monné
2021-10-15 14:00 ` Luca Fancellu
1 sibling, 1 reply; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 13:47 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Jan Beulich, Xen-devel, Ian Jackson, Rahul Singh,
Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Wei Liu, Paul Durrant,
Oleksandr Andrushchenko
On Fri, Oct 15, 2021 at 10:48:41AM +0000, Bertrand Marquis wrote:
> Hi Jan,
>
> > On 15 Oct 2021, at 11:41, Jan Beulich <jbeulich@suse.com> wrote:
> >
> > On 15.10.2021 12:33, Bertrand Marquis wrote:
> >>> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
> >>> On 15.10.2021 11:52, Bertrand Marquis wrote:
> >>>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
> >>>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> >>>>>>
> >>>>>> check_pdev(pdev);
> >>>>>>
> >>>>>> +#ifdef CONFIG_ARM
> >>>>>> + /*
> >>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> >>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
> >>>>>> + */
> >>>>>> + ret = vpci_add_handlers(pdev);
> >>>>>> + if ( ret )
> >>>>>> + {
> >>>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
> >>>>>> + goto out;
> >>>>>> + }
> >>>>>> +#endif
> >>>>>
> >>>>> I think vpci_add_handlers should be called after checking that
> >>>>> pdev->domain is != NULL, so I would move this chunk a bit below.
> >>>>
> >>>> On arm this would prevent the dom0less use case or to have the PCI
> >>>> bus enumerated from an other domain.
> >>>> @oleksandr: can you comment on this one, you might have a better
> >>>> answer than me on this ?
> >>>
> >>> Well, without Xen doing the enumeration, some other entity would need
> >>> to do so, including the reporting to Xen. Obviously without a Dom0 it
> >>> would be ambiguous which domain to assign the device to; perhaps it
> >>> should be the caller in this case? That would make that caller domain
> >>> a pseudo-hwdom though, as far as PCI is concerned, which may not be
> >>> desirable according to my (limited) understanding of dom0less.
> >>
> >> This is not really related to this patch but the plan is the following:
> >> - enumeration would have to be done by the firmware or boot loader before
> >> - xen will have some code to detect PCI devices
> >> - dom0less can be used to assign PCI devices to guest
> >>
> >> Anyway does not change the fact that this must be called when domain is
> >> not NULL and I will fix that.
> >
> > Since we now all seem to agree that the NULL would have been a problem,
> > may I ask in how far any of this has actually been tested?
>
> With the whole serie currently on gitlab we have extensively tested passing
> through PCI devices on Arm in several configuration (number of device, MSI,
> MSI-X) and check that PCI was still functional on x86.
>
> With the patches pushed to Xen right now it was checked that:
> - xen compiles properly on arm32, arm64 and x86
> - xen compiles properly with VPCI activated (using a patch) on arm32 and arm64
> - xen on x86 is functionnal (using basic test on QEMU)
> - xen on arm64 is functionnal (with some extensive tests on different targets)
I thinks it's unlikely, but since I haven't checked myself, could you
see if the vpci user-space test harness (tools/tests/vpci) still
builds and functions properly?
Thanks, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 13:47 ` Roger Pau Monné
@ 2021-10-15 14:00 ` Luca Fancellu
2021-10-15 14:32 ` Roger Pau Monné
0 siblings, 1 reply; 190+ messages in thread
From: Luca Fancellu @ 2021-10-15 14:00 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Bertrand Marquis, Jan Beulich, Xen-devel, Ian Jackson,
Rahul Singh, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Wei Liu, Paul Durrant,
Oleksandr Andrushchenko
> On 15 Oct 2021, at 14:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Fri, Oct 15, 2021 at 10:48:41AM +0000, Bertrand Marquis wrote:
>> Hi Jan,
>>
>>> On 15 Oct 2021, at 11:41, Jan Beulich <jbeulich@suse.com> wrote:
>>>
>>> On 15.10.2021 12:33, Bertrand Marquis wrote:
>>>>> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> On 15.10.2021 11:52, Bertrand Marquis wrote:
>>>>>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
>>>>>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>>>>>>
>>>>>>>> check_pdev(pdev);
>>>>>>>>
>>>>>>>> +#ifdef CONFIG_ARM
>>>>>>>> + /*
>>>>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
>>>>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
>>>>>>>> + */
>>>>>>>> + ret = vpci_add_handlers(pdev);
>>>>>>>> + if ( ret )
>>>>>>>> + {
>>>>>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
>>>>>>>> + goto out;
>>>>>>>> + }
>>>>>>>> +#endif
>>>>>>>
>>>>>>> I think vpci_add_handlers should be called after checking that
>>>>>>> pdev->domain is != NULL, so I would move this chunk a bit below.
>>>>>>
>>>>>> On arm this would prevent the dom0less use case or to have the PCI
>>>>>> bus enumerated from an other domain.
>>>>>> @oleksandr: can you comment on this one, you might have a better
>>>>>> answer than me on this ?
>>>>>
>>>>> Well, without Xen doing the enumeration, some other entity would need
>>>>> to do so, including the reporting to Xen. Obviously without a Dom0 it
>>>>> would be ambiguous which domain to assign the device to; perhaps it
>>>>> should be the caller in this case? That would make that caller domain
>>>>> a pseudo-hwdom though, as far as PCI is concerned, which may not be
>>>>> desirable according to my (limited) understanding of dom0less.
>>>>
>>>> This is not really related to this patch but the plan is the following:
>>>> - enumeration would have to be done by the firmware or boot loader before
>>>> - xen will have some code to detect PCI devices
>>>> - dom0less can be used to assign PCI devices to guest
>>>>
>>>> Anyway does not change the fact that this must be called when domain is
>>>> not NULL and I will fix that.
>>>
>>> Since we now all seem to agree that the NULL would have been a problem,
>>> may I ask in how far any of this has actually been tested?
>>
>> With the whole serie currently on gitlab we have extensively tested passing
>> through PCI devices on Arm in several configuration (number of device, MSI,
>> MSI-X) and check that PCI was still functional on x86.
>>
>> With the patches pushed to Xen right now it was checked that:
>> - xen compiles properly on arm32, arm64 and x86
>> - xen compiles properly with VPCI activated (using a patch) on arm32 and arm64
>> - xen on x86 is functionnal (using basic test on QEMU)
>> - xen on arm64 is functionnal (with some extensive tests on different targets)
>
> I thinks it's unlikely, but since I haven't checked myself, could you
> see if the vpci user-space test harness (tools/tests/vpci) still
> builds and functions properly?
Hi Roger,
Bertrand is going to send the v7, however I’m keen to do this test but unfortunately
I’ve never used it before and I don’t have an x86 environment, do you have any
resource that I can read or some guidance?
Cheers,
Luca
>
> Thanks, Roger.
>
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2021-10-15 14:00 ` Luca Fancellu
@ 2021-10-15 14:32 ` Roger Pau Monné
0 siblings, 0 replies; 190+ messages in thread
From: Roger Pau Monné @ 2021-10-15 14:32 UTC (permalink / raw)
To: Luca Fancellu
Cc: Bertrand Marquis, Jan Beulich, Xen-devel, Ian Jackson,
Rahul Singh, Stefano Stabellini, Julien Grall, Volodymyr Babchuk,
Andrew Cooper, George Dunlap, Wei Liu, Paul Durrant,
Oleksandr Andrushchenko
On Fri, Oct 15, 2021 at 03:00:21PM +0100, Luca Fancellu wrote:
>
>
> > On 15 Oct 2021, at 14:47, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Fri, Oct 15, 2021 at 10:48:41AM +0000, Bertrand Marquis wrote:
> >> Hi Jan,
> >>
> >>> On 15 Oct 2021, at 11:41, Jan Beulich <jbeulich@suse.com> wrote:
> >>>
> >>> On 15.10.2021 12:33, Bertrand Marquis wrote:
> >>>>> On 15 Oct 2021, at 11:24, Jan Beulich <jbeulich@suse.com> wrote:
> >>>>> On 15.10.2021 11:52, Bertrand Marquis wrote:
> >>>>>>> On 15 Oct 2021, at 09:32, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>>>>> On Thu, Oct 14, 2021 at 03:49:50PM +0100, Bertrand Marquis wrote:
> >>>>>>>> @@ -752,6 +752,19 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> >>>>>>>>
> >>>>>>>> check_pdev(pdev);
> >>>>>>>>
> >>>>>>>> +#ifdef CONFIG_ARM
> >>>>>>>> + /*
> >>>>>>>> + * On ARM PCI devices discovery will be done by Dom0. Add vpci handler when
> >>>>>>>> + * Dom0 inform XEN to add the PCI devices in XEN.
> >>>>>>>> + */
> >>>>>>>> + ret = vpci_add_handlers(pdev);
> >>>>>>>> + if ( ret )
> >>>>>>>> + {
> >>>>>>>> + printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
> >>>>>>>> + goto out;
> >>>>>>>> + }
> >>>>>>>> +#endif
> >>>>>>>
> >>>>>>> I think vpci_add_handlers should be called after checking that
> >>>>>>> pdev->domain is != NULL, so I would move this chunk a bit below.
> >>>>>>
> >>>>>> On arm this would prevent the dom0less use case or to have the PCI
> >>>>>> bus enumerated from an other domain.
> >>>>>> @oleksandr: can you comment on this one, you might have a better
> >>>>>> answer than me on this ?
> >>>>>
> >>>>> Well, without Xen doing the enumeration, some other entity would need
> >>>>> to do so, including the reporting to Xen. Obviously without a Dom0 it
> >>>>> would be ambiguous which domain to assign the device to; perhaps it
> >>>>> should be the caller in this case? That would make that caller domain
> >>>>> a pseudo-hwdom though, as far as PCI is concerned, which may not be
> >>>>> desirable according to my (limited) understanding of dom0less.
> >>>>
> >>>> This is not really related to this patch but the plan is the following:
> >>>> - enumeration would have to be done by the firmware or boot loader before
> >>>> - xen will have some code to detect PCI devices
> >>>> - dom0less can be used to assign PCI devices to guest
> >>>>
> >>>> Anyway does not change the fact that this must be called when domain is
> >>>> not NULL and I will fix that.
> >>>
> >>> Since we now all seem to agree that the NULL would have been a problem,
> >>> may I ask in how far any of this has actually been tested?
> >>
> >> With the whole serie currently on gitlab we have extensively tested passing
> >> through PCI devices on Arm in several configuration (number of device, MSI,
> >> MSI-X) and check that PCI was still functional on x86.
> >>
> >> With the patches pushed to Xen right now it was checked that:
> >> - xen compiles properly on arm32, arm64 and x86
> >> - xen compiles properly with VPCI activated (using a patch) on arm32 and arm64
> >> - xen on x86 is functionnal (using basic test on QEMU)
> >> - xen on arm64 is functionnal (with some extensive tests on different targets)
> >
> > I thinks it's unlikely, but since I haven't checked myself, could you
> > see if the vpci user-space test harness (tools/tests/vpci) still
> > builds and functions properly?
>
> Hi Roger,
>
> Bertrand is going to send the v7, however I’m keen to do this test but unfortunately
> I’ve never used it before and I don’t have an x86 environment, do you have any
> resource that I can read or some guidance?
Oh, you just need to build it (make) and then run it (make run). If
it returns success everything is fine.
Regards, Roger.
^ permalink raw reply [flat|nested] 190+ messages in thread
* xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM)
2021-10-06 17:40 ` [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM Rahul Singh
2021-10-07 0:05 ` Stefano Stabellini
@ 2021-10-21 9:28 ` Julien Grall
2021-10-21 13:15 ` Bertrand Marquis
1 sibling, 1 reply; 190+ messages in thread
From: Julien Grall @ 2021-10-21 9:28 UTC (permalink / raw)
To: Rahul Singh, xen-devel
Cc: bertrand.marquis, Andre.Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Roger Pau Monné
Hi all,
While going through the passthrough code. I noticed that we don't have a
common lock for the IOMMU between the PCI and DT code.
This is going to be an issue given it would technically be possible to
add a PCI device while assigning a DT.
Rahul, Bertrand, Oleksandr, can you have a look at the issue?
Cheers,
On 06/10/2021 18:40, Rahul Singh wrote:
> Hardware domain is in charge of doing the PCI enumeration and will
> discover the PCI devices and then will communicate to XEN via hyper
> call PHYSDEVOP_pci_device_add(..) to add the PCI devices in XEN.
>
> Also implement PHYSDEVOP_pci_device_remove(..) to remove the PCI device.
>
> As most of the code for PHYSDEVOP_pci_device_* is the same between x86
> and ARM, move the code to a common file to avoid duplication.
>
> There are other PHYSDEVOP_pci_device_* operations to add PCI devices.
> Currently implemented PHYSDEVOP_pci_device_remove(..) and
> PHYSDEVOP_pci_device_add(..) only as those are minimum required to
> support PCI passthrough on ARM.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> Change in v5:
> - Move the pci_physdev_op() stub to xen/arch/arm/physdev.c.
> Change in v4:
> - Move file commom/physdev.c to drivers/pci/physdev.c
> - minor comments.
> Change in v3: Fixed minor comment.
> Change in v2:
> - Add support for PHYSDEVOP_pci_device_remove()
> - Move code to common code
> ---
> ---
> xen/arch/arm/physdev.c | 6 ++-
> xen/arch/x86/physdev.c | 52 +----------------------
> xen/arch/x86/x86_64/physdev.c | 2 +-
> xen/drivers/pci/Makefile | 1 +
> xen/drivers/pci/physdev.c | 80 +++++++++++++++++++++++++++++++++++
> xen/include/public/arch-arm.h | 4 +-
> xen/include/xen/hypercall.h | 4 ++
> 7 files changed, 96 insertions(+), 53 deletions(-)
> create mode 100644 xen/drivers/pci/physdev.c
>
> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
> index e91355fe22..f9aa274dda 100644
> --- a/xen/arch/arm/physdev.c
> +++ b/xen/arch/arm/physdev.c
> @@ -8,13 +8,17 @@
> #include <xen/lib.h>
> #include <xen/errno.h>
> #include <xen/sched.h>
> -#include <asm/hypercall.h>
> +#include <xen/hypercall.h>
>
>
> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> {
> +#ifdef CONFIG_HAS_PCI
> + return pci_physdev_op(cmd, arg);
> +#else
> gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> return -ENOSYS;
> +#endif
> }
>
> /*
> diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
> index 23465bcd00..ea38be8b79 100644
> --- a/xen/arch/x86/physdev.c
> +++ b/xen/arch/x86/physdev.c
> @@ -12,7 +12,7 @@
> #include <asm/io_apic.h>
> #include <asm/msi.h>
> #include <asm/hvm/irq.h>
> -#include <asm/hypercall.h>
> +#include <xen/hypercall.h>
> #include <public/xen.h>
> #include <public/physdev.h>
> #include <xsm/xsm.h>
> @@ -480,54 +480,6 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> break;
> }
>
> - case PHYSDEVOP_pci_device_add: {
> - struct physdev_pci_device_add add;
> - struct pci_dev_info pdev_info;
> - nodeid_t node;
> -
> - ret = -EFAULT;
> - if ( copy_from_guest(&add, arg, 1) != 0 )
> - break;
> -
> - pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
> - if ( add.flags & XEN_PCI_DEV_VIRTFN )
> - {
> - pdev_info.is_virtfn = 1;
> - pdev_info.physfn.bus = add.physfn.bus;
> - pdev_info.physfn.devfn = add.physfn.devfn;
> - }
> - else
> - pdev_info.is_virtfn = 0;
> -
> - if ( add.flags & XEN_PCI_DEV_PXM )
> - {
> - uint32_t pxm;
> - size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
> - sizeof(add.optarr[0]);
> -
> - if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
> - break;
> -
> - node = pxm_to_node(pxm);
> - }
> - else
> - node = NUMA_NO_NODE;
> -
> - ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
> - break;
> - }
> -
> - case PHYSDEVOP_pci_device_remove: {
> - struct physdev_pci_device dev;
> -
> - ret = -EFAULT;
> - if ( copy_from_guest(&dev, arg, 1) != 0 )
> - break;
> -
> - ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
> - break;
> - }
> -
> case PHYSDEVOP_prepare_msix:
> case PHYSDEVOP_release_msix: {
> struct physdev_pci_device dev;
> @@ -663,7 +615,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> }
>
> default:
> - ret = -ENOSYS;
> + ret = pci_physdev_op(cmd, arg);
> break;
> }
>
> diff --git a/xen/arch/x86/x86_64/physdev.c b/xen/arch/x86/x86_64/physdev.c
> index 0a50cbd4d8..e3cbd5ebcb 100644
> --- a/xen/arch/x86/x86_64/physdev.c
> +++ b/xen/arch/x86/x86_64/physdev.c
> @@ -9,7 +9,7 @@ EMIT_FILE;
> #include <compat/xen.h>
> #include <compat/event_channel.h>
> #include <compat/physdev.h>
> -#include <asm/hypercall.h>
> +#include <xen/hypercall.h>
>
> #define do_physdev_op compat_physdev_op
>
> diff --git a/xen/drivers/pci/Makefile b/xen/drivers/pci/Makefile
> index a98035df4c..972c923db0 100644
> --- a/xen/drivers/pci/Makefile
> +++ b/xen/drivers/pci/Makefile
> @@ -1 +1,2 @@
> obj-y += pci.o
> +obj-y += physdev.o
> diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
> new file mode 100644
> index 0000000000..4f3e1a96c0
> --- /dev/null
> +++ b/xen/drivers/pci/physdev.c
> @@ -0,0 +1,80 @@
> +
> +#include <xen/guest_access.h>
> +#include <xen/hypercall.h>
> +#include <xen/init.h>
> +
> +#ifndef COMPAT
> +typedef long ret_t;
> +#endif
> +
> +ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> +{
> + ret_t ret;
> +
> + switch ( cmd )
> + {
> + case PHYSDEVOP_pci_device_add: {
> + struct physdev_pci_device_add add;
> + struct pci_dev_info pdev_info;
> + nodeid_t node = NUMA_NO_NODE;
> +
> + ret = -EFAULT;
> + if ( copy_from_guest(&add, arg, 1) != 0 )
> + break;
> +
> + pdev_info.is_extfn = (add.flags & XEN_PCI_DEV_EXTFN);
> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
> + {
> + pdev_info.is_virtfn = true;
> + pdev_info.physfn.bus = add.physfn.bus;
> + pdev_info.physfn.devfn = add.physfn.devfn;
> + }
> + else
> + pdev_info.is_virtfn = false;
> +
> +#ifdef CONFIG_NUMA
> + if ( add.flags & XEN_PCI_DEV_PXM )
> + {
> + uint32_t pxm;
> + size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
> + sizeof(add.optarr[0]);
> +
> + if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
> + break;
> +
> + node = pxm_to_node(pxm);
> + }
> +#endif
> +
> + ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
> + break;
> + }
> +
> + case PHYSDEVOP_pci_device_remove: {
> + struct physdev_pci_device dev;
> +
> + ret = -EFAULT;
> + if ( copy_from_guest(&dev, arg, 1) != 0 )
> + break;
> +
> + ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
> + break;
> + }
> +
> + default:
> + ret = -ENOSYS;
> + break;
> + }
> +
> + return ret;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 6b5a5f818a..d46c61fca9 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -107,7 +107,9 @@
> * All generic sub-operations
> *
> * HYPERVISOR_physdev_op
> - * No sub-operations are currenty supported
> + * Exactly these sub-operations are supported:
> + * PHYSDEVOP_pci_device_add
> + * PHYSDEVOP_pci_device_remove
> *
> * HYPERVISOR_sysctl
> * All generic sub-operations, with the exception of:
> diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
> index 3771487a30..07b10ec230 100644
> --- a/xen/include/xen/hypercall.h
> +++ b/xen/include/xen/hypercall.h
> @@ -45,6 +45,10 @@ extern long
> do_platform_op(
> XEN_GUEST_HANDLE_PARAM(xen_platform_op_t) u_xenpf_op);
>
> +extern long
> +pci_physdev_op(
> + int cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
> +
> /*
> * To allow safe resume of do_memory_op() after preemption, we need to know
> * at what point in the page list to resume. For this purpose I steal the
>
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM)
2021-10-21 9:28 ` xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM) Julien Grall
@ 2021-10-21 13:15 ` Bertrand Marquis
2021-10-21 13:47 ` Julien Grall
0 siblings, 1 reply; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-21 13:15 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, Xen-devel, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Roger Pau Monné
Hi Julien,
> On 21 Oct 2021, at 10:28, Julien Grall <julien@xen.org> wrote:
>
> Hi all,
>
> While going through the passthrough code. I noticed that we don't have a common lock for the IOMMU between the PCI and DT code.
>
> This is going to be an issue given it would technically be possible to add a PCI device while assigning a DT.
>
> Rahul, Bertrand, Oleksandr, can you have a look at the issue?
Yes we can have a look at this.
Right now pci device add is done by dom0 so I do not think we have an issue in practice unless I wrongly understood something.
But for sure in theory yes we need to look at this.
Cheers
Bertrand
>
> Cheers,
>
> On 06/10/2021 18:40, Rahul Singh wrote:
>> Hardware domain is in charge of doing the PCI enumeration and will
>> discover the PCI devices and then will communicate to XEN via hyper
>> call PHYSDEVOP_pci_device_add(..) to add the PCI devices in XEN.
>> Also implement PHYSDEVOP_pci_device_remove(..) to remove the PCI device.
>> As most of the code for PHYSDEVOP_pci_device_* is the same between x86
>> and ARM, move the code to a common file to avoid duplication.
>> There are other PHYSDEVOP_pci_device_* operations to add PCI devices.
>> Currently implemented PHYSDEVOP_pci_device_remove(..) and
>> PHYSDEVOP_pci_device_add(..) only as those are minimum required to
>> support PCI passthrough on ARM.
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> Change in v5:
>> - Move the pci_physdev_op() stub to xen/arch/arm/physdev.c.
>> Change in v4:
>> - Move file commom/physdev.c to drivers/pci/physdev.c
>> - minor comments.
>> Change in v3: Fixed minor comment.
>> Change in v2:
>> - Add support for PHYSDEVOP_pci_device_remove()
>> - Move code to common code
>> ---
>> ---
>> xen/arch/arm/physdev.c | 6 ++-
>> xen/arch/x86/physdev.c | 52 +----------------------
>> xen/arch/x86/x86_64/physdev.c | 2 +-
>> xen/drivers/pci/Makefile | 1 +
>> xen/drivers/pci/physdev.c | 80 +++++++++++++++++++++++++++++++++++
>> xen/include/public/arch-arm.h | 4 +-
>> xen/include/xen/hypercall.h | 4 ++
>> 7 files changed, 96 insertions(+), 53 deletions(-)
>> create mode 100644 xen/drivers/pci/physdev.c
>> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
>> index e91355fe22..f9aa274dda 100644
>> --- a/xen/arch/arm/physdev.c
>> +++ b/xen/arch/arm/physdev.c
>> @@ -8,13 +8,17 @@
>> #include <xen/lib.h>
>> #include <xen/errno.h>
>> #include <xen/sched.h>
>> -#include <asm/hypercall.h>
>> +#include <xen/hypercall.h>
>> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>> {
>> +#ifdef CONFIG_HAS_PCI
>> + return pci_physdev_op(cmd, arg);
>> +#else
>> gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>> return -ENOSYS;
>> +#endif
>> }
>> /*
>> diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
>> index 23465bcd00..ea38be8b79 100644
>> --- a/xen/arch/x86/physdev.c
>> +++ b/xen/arch/x86/physdev.c
>> @@ -12,7 +12,7 @@
>> #include <asm/io_apic.h>
>> #include <asm/msi.h>
>> #include <asm/hvm/irq.h>
>> -#include <asm/hypercall.h>
>> +#include <xen/hypercall.h>
>> #include <public/xen.h>
>> #include <public/physdev.h>
>> #include <xsm/xsm.h>
>> @@ -480,54 +480,6 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>> break;
>> }
>> - case PHYSDEVOP_pci_device_add: {
>> - struct physdev_pci_device_add add;
>> - struct pci_dev_info pdev_info;
>> - nodeid_t node;
>> -
>> - ret = -EFAULT;
>> - if ( copy_from_guest(&add, arg, 1) != 0 )
>> - break;
>> -
>> - pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
>> - if ( add.flags & XEN_PCI_DEV_VIRTFN )
>> - {
>> - pdev_info.is_virtfn = 1;
>> - pdev_info.physfn.bus = add.physfn.bus;
>> - pdev_info.physfn.devfn = add.physfn.devfn;
>> - }
>> - else
>> - pdev_info.is_virtfn = 0;
>> -
>> - if ( add.flags & XEN_PCI_DEV_PXM )
>> - {
>> - uint32_t pxm;
>> - size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
>> - sizeof(add.optarr[0]);
>> -
>> - if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
>> - break;
>> -
>> - node = pxm_to_node(pxm);
>> - }
>> - else
>> - node = NUMA_NO_NODE;
>> -
>> - ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
>> - break;
>> - }
>> -
>> - case PHYSDEVOP_pci_device_remove: {
>> - struct physdev_pci_device dev;
>> -
>> - ret = -EFAULT;
>> - if ( copy_from_guest(&dev, arg, 1) != 0 )
>> - break;
>> -
>> - ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
>> - break;
>> - }
>> -
>> case PHYSDEVOP_prepare_msix:
>> case PHYSDEVOP_release_msix: {
>> struct physdev_pci_device dev;
>> @@ -663,7 +615,7 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>> }
>> default:
>> - ret = -ENOSYS;
>> + ret = pci_physdev_op(cmd, arg);
>> break;
>> }
>> diff --git a/xen/arch/x86/x86_64/physdev.c b/xen/arch/x86/x86_64/physdev.c
>> index 0a50cbd4d8..e3cbd5ebcb 100644
>> --- a/xen/arch/x86/x86_64/physdev.c
>> +++ b/xen/arch/x86/x86_64/physdev.c
>> @@ -9,7 +9,7 @@ EMIT_FILE;
>> #include <compat/xen.h>
>> #include <compat/event_channel.h>
>> #include <compat/physdev.h>
>> -#include <asm/hypercall.h>
>> +#include <xen/hypercall.h>
>> #define do_physdev_op compat_physdev_op
>> diff --git a/xen/drivers/pci/Makefile b/xen/drivers/pci/Makefile
>> index a98035df4c..972c923db0 100644
>> --- a/xen/drivers/pci/Makefile
>> +++ b/xen/drivers/pci/Makefile
>> @@ -1 +1,2 @@
>> obj-y += pci.o
>> +obj-y += physdev.o
>> diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
>> new file mode 100644
>> index 0000000000..4f3e1a96c0
>> --- /dev/null
>> +++ b/xen/drivers/pci/physdev.c
>> @@ -0,0 +1,80 @@
>> +
>> +#include <xen/guest_access.h>
>> +#include <xen/hypercall.h>
>> +#include <xen/init.h>
>> +
>> +#ifndef COMPAT
>> +typedef long ret_t;
>> +#endif
>> +
>> +ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>> +{
>> + ret_t ret;
>> +
>> + switch ( cmd )
>> + {
>> + case PHYSDEVOP_pci_device_add: {
>> + struct physdev_pci_device_add add;
>> + struct pci_dev_info pdev_info;
>> + nodeid_t node = NUMA_NO_NODE;
>> +
>> + ret = -EFAULT;
>> + if ( copy_from_guest(&add, arg, 1) != 0 )
>> + break;
>> +
>> + pdev_info.is_extfn = (add.flags & XEN_PCI_DEV_EXTFN);
>> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
>> + {
>> + pdev_info.is_virtfn = true;
>> + pdev_info.physfn.bus = add.physfn.bus;
>> + pdev_info.physfn.devfn = add.physfn.devfn;
>> + }
>> + else
>> + pdev_info.is_virtfn = false;
>> +
>> +#ifdef CONFIG_NUMA
>> + if ( add.flags & XEN_PCI_DEV_PXM )
>> + {
>> + uint32_t pxm;
>> + size_t optarr_off = offsetof(struct physdev_pci_device_add, optarr) /
>> + sizeof(add.optarr[0]);
>> +
>> + if ( copy_from_guest_offset(&pxm, arg, optarr_off, 1) )
>> + break;
>> +
>> + node = pxm_to_node(pxm);
>> + }
>> +#endif
>> +
>> + ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
>> + break;
>> + }
>> +
>> + case PHYSDEVOP_pci_device_remove: {
>> + struct physdev_pci_device dev;
>> +
>> + ret = -EFAULT;
>> + if ( copy_from_guest(&dev, arg, 1) != 0 )
>> + break;
>> +
>> + ret = pci_remove_device(dev.seg, dev.bus, dev.devfn);
>> + break;
>> + }
>> +
>> + default:
>> + ret = -ENOSYS;
>> + break;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index 6b5a5f818a..d46c61fca9 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -107,7 +107,9 @@
>> * All generic sub-operations
>> *
>> * HYPERVISOR_physdev_op
>> - * No sub-operations are currenty supported
>> + * Exactly these sub-operations are supported:
>> + * PHYSDEVOP_pci_device_add
>> + * PHYSDEVOP_pci_device_remove
>> *
>> * HYPERVISOR_sysctl
>> * All generic sub-operations, with the exception of:
>> diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h
>> index 3771487a30..07b10ec230 100644
>> --- a/xen/include/xen/hypercall.h
>> +++ b/xen/include/xen/hypercall.h
>> @@ -45,6 +45,10 @@ extern long
>> do_platform_op(
>> XEN_GUEST_HANDLE_PARAM(xen_platform_op_t) u_xenpf_op);
>> +extern long
>> +pci_physdev_op(
>> + int cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
>> +
>> /*
>> * To allow safe resume of do_memory_op() after preemption, we need to know
>> * at what point in the page list to resume. For this purpose I steal the
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM)
2021-10-21 13:15 ` Bertrand Marquis
@ 2021-10-21 13:47 ` Julien Grall
2021-10-21 13:52 ` Bertrand Marquis
0 siblings, 1 reply; 190+ messages in thread
From: Julien Grall @ 2021-10-21 13:47 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Rahul Singh, Xen-devel, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Roger Pau Monné
On 21/10/2021 14:15, Bertrand Marquis wrote:
> Hi Julien,
Hi Bertand,
>> On 21 Oct 2021, at 10:28, Julien Grall <julien@xen.org> wrote:
>>
>> Hi all,
>>
>> While going through the passthrough code. I noticed that we don't have a common lock for the IOMMU between the PCI and DT code.
>>
>> This is going to be an issue given it would technically be possible to add a PCI device while assigning a DT.
>>
>> Rahul, Bertrand, Oleksandr, can you have a look at the issue?
>
> Yes we can have a look at this.
>
> Right now pci device add is done by dom0 so I do not think we have an issue in practice unless I wrongly understood something
This will depend on the XSM policy. With the default one, then yes I
agree that only dom0 (we don't support hardware domain) can add PCI device.
However, this restriction doesn't really matter here. You would be
relying on dom0 to do the locking and AFAIK this doesn't exist. Instead,
the admin would have to ensure that two don't happen at the same time.
Anyway, I think Xen should take care of preventing concurrent IOMMU
operations rather than relying on external subsystem (e.g. dom0) to do
it. At least the Arm SMMU driver will rely the generic locking to modify
atomically internal list.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
* Re: xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM)
2021-10-21 13:47 ` Julien Grall
@ 2021-10-21 13:52 ` Bertrand Marquis
0 siblings, 0 replies; 190+ messages in thread
From: Bertrand Marquis @ 2021-10-21 13:52 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, Xen-devel, Andre Przywara, Stefano Stabellini,
Volodymyr Babchuk, Andrew Cooper, George Dunlap, Ian Jackson,
Jan Beulich, Wei Liu, Roger Pau Monné
Hi Julien,
> On 21 Oct 2021, at 14:47, Julien Grall <julien@xen.org> wrote:
>
>
>
> On 21/10/2021 14:15, Bertrand Marquis wrote:
>> Hi Julien,
>
> Hi Bertand,
>
>>> On 21 Oct 2021, at 10:28, Julien Grall <julien@xen.org> wrote:
>>>
>>> Hi all,
>>>
>>> While going through the passthrough code. I noticed that we don't have a common lock for the IOMMU between the PCI and DT code.
>>>
>>> This is going to be an issue given it would technically be possible to add a PCI device while assigning a DT.
>>>
>>> Rahul, Bertrand, Oleksandr, can you have a look at the issue?
>> Yes we can have a look at this.
>> Right now pci device add is done by dom0 so I do not think we have an issue in practice unless I wrongly understood something
> This will depend on the XSM policy. With the default one, then yes I agree that only dom0 (we don't support hardware domain) can add PCI device.
>
> However, this restriction doesn't really matter here. You would be relying on dom0 to do the locking and AFAIK this doesn't exist. Instead, the admin would have to ensure that two don't happen at the same time.
>
> Anyway, I think Xen should take care of preventing concurrent IOMMU operations rather than relying on external subsystem (e.g. dom0) to do it. At least the Arm SMMU driver will rely the generic locking to modify atomically internal list.
Agree, was just trying to make sure I understood the problem correctly.
We will check that.
Cheers
Bertrand
>
> Cheers,
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 190+ messages in thread
end of thread, other threads:[~2021-10-21 13:53 UTC | newest]
Thread overview: 190+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-06 17:40 [PATCH v5 00/11] PCI devices passthrough on Arm Rahul Singh
2021-10-06 17:40 ` [PATCH v5 01/11] xen/arm: xc_domain_ioport_permission(..) not supported on ARM Rahul Singh
2021-10-11 11:47 ` Roger Pau Monné
2021-10-11 12:11 ` Bertrand Marquis
2021-10-11 13:20 ` Roger Pau Monné
2021-10-11 13:40 ` Bertrand Marquis
2021-10-11 13:57 ` Roger Pau Monné
2021-10-11 14:16 ` Bertrand Marquis
2021-10-11 16:32 ` Roger Pau Monné
2021-10-11 17:11 ` Bertrand Marquis
2021-10-12 8:29 ` Jan Beulich
2021-10-12 8:41 ` Bertrand Marquis
2021-10-12 9:32 ` Jan Beulich
2021-10-12 9:38 ` Oleksandr Andrushchenko
2021-10-12 10:01 ` Jan Beulich
2021-10-12 10:06 ` Oleksandr Andrushchenko
2021-10-12 10:20 ` Jan Beulich
2021-10-12 10:41 ` Bertrand Marquis
2021-10-12 10:44 ` Jan Beulich
2021-10-12 14:53 ` Ian Jackson
2021-10-12 16:15 ` Bertrand Marquis
2021-10-12 16:29 ` Ian Jackson
2021-10-12 20:42 ` Stefano Stabellini
2021-10-13 8:07 ` Roger Pau Monné
2021-10-13 11:52 ` Ian Jackson
2021-10-13 8:02 ` Roger Pau Monné
2021-10-13 12:02 ` Ian Jackson
2021-10-12 9:40 ` Bertrand Marquis
2021-10-12 10:03 ` Jan Beulich
2021-10-11 14:16 ` Oleksandr Andrushchenko
2021-10-06 17:40 ` [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM Rahul Singh
2021-10-07 0:05 ` Stefano Stabellini
2021-10-07 12:58 ` Jan Beulich
2021-10-21 9:28 ` xen/arm: Missing appropriate locking for the IOMMU (WAS Re: [PATCH v5 02/11] xen/arm: Add PHYSDEVOP_pci_device_(*add/remove) support for ARM) Julien Grall
2021-10-21 13:15 ` Bertrand Marquis
2021-10-21 13:47 ` Julien Grall
2021-10-21 13:52 ` Bertrand Marquis
2021-10-06 17:40 ` [PATCH v5 03/11] xen/arm: Add cmdline boot option "pci-passthrough = <boolean>" Rahul Singh
2021-10-07 8:27 ` Jan Beulich
2021-10-07 8:32 ` Rahul Singh
2021-10-07 12:59 ` Jan Beulich
2021-10-06 17:40 ` [PATCH v5 04/11] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
2021-10-06 17:40 ` [PATCH v5 05/11] xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh
2021-10-06 17:40 ` [PATCH v5 06/11] xen/arm: Implement pci access functions Rahul Singh
2021-10-06 17:40 ` [PATCH v5 07/11] xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag Rahul Singh
2021-10-07 13:08 ` Jan Beulich
2021-10-08 18:06 ` Andrew Cooper
2021-10-08 21:12 ` Julien Grall
2021-10-08 21:46 ` Stefano Stabellini
2021-10-11 9:24 ` Julien Grall
2021-10-11 11:29 ` Michal Orzel
2021-10-11 11:35 ` Jan Beulich
2021-10-11 13:17 ` Roger Pau Monné
2021-10-11 9:48 ` Ian Jackson
2021-10-11 9:27 ` Roger Pau Monné
2021-10-11 12:06 ` Michal Orzel
2021-10-12 10:38 ` Michal Orzel
2021-10-13 8:30 ` Roger Pau Monné
2021-10-13 9:36 ` Bertrand Marquis
2021-10-13 10:56 ` Roger Pau Monné
2021-10-13 12:11 ` Bertrand Marquis
2021-10-13 12:57 ` Jan Beulich
2021-10-13 20:41 ` Stefano Stabellini
2021-10-14 6:23 ` Jan Beulich
2021-10-14 7:53 ` Bertrand Marquis
2021-10-13 14:28 ` Roger Pau Monné
2021-10-13 20:53 ` Stefano Stabellini
2021-10-13 23:21 ` Stefano Stabellini
2021-10-12 21:48 ` Stefano Stabellini
2021-10-13 6:18 ` Jan Beulich
2021-10-13 7:11 ` Michal Orzel
2021-10-06 17:40 ` [PATCH v5 08/11] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
2021-10-07 13:43 ` Jan Beulich
2021-10-11 12:41 ` Bertrand Marquis
2021-10-11 13:09 ` Jan Beulich
2021-10-11 13:34 ` Bertrand Marquis
2021-10-11 14:10 ` Jan Beulich
2021-10-11 14:52 ` Bertrand Marquis
2021-10-11 18:18 ` Stefano Stabellini
2021-10-12 8:04 ` Jan Beulich
2021-10-12 21:37 ` Stefano Stabellini
2021-10-13 6:10 ` Jan Beulich
2021-10-13 10:02 ` Bertrand Marquis
2021-10-13 12:21 ` Jan Beulich
2021-10-12 15:04 ` Julien Grall
2021-10-12 16:12 ` Bertrand Marquis
2021-10-12 16:20 ` Julien Grall
2021-10-12 17:50 ` Bertrand Marquis
2021-10-11 10:51 ` Roger Pau Monné
2021-10-11 16:12 ` Bertrand Marquis
2021-10-11 16:20 ` Oleksandr Andrushchenko
2021-10-11 16:43 ` Roger Pau Monné
2021-10-11 17:15 ` Bertrand Marquis
2021-10-11 18:30 ` Oleksandr Andrushchenko
2021-10-11 19:27 ` Stefano Stabellini
2021-10-12 5:34 ` Oleksandr Andrushchenko
2021-10-12 7:44 ` Bertrand Marquis
2021-10-12 14:32 ` Julien Grall
2021-10-12 14:34 ` Bertrand Marquis
2021-10-13 8:45 ` Roger Pau Monné
2021-10-13 9:48 ` Bertrand Marquis
2021-10-13 10:33 ` Roger Pau Monné
2021-10-13 13:00 ` Jan Beulich
2021-10-13 14:51 ` Oleksandr Andrushchenko
2021-10-13 15:15 ` Jan Beulich
2021-10-13 19:27 ` Stefano Stabellini
2021-10-14 6:33 ` Jan Beulich
2021-10-14 7:53 ` Bertrand Marquis
2021-10-14 9:03 ` Bertrand Marquis
2021-10-14 9:24 ` Jan Beulich
2021-10-06 17:40 ` [PATCH v5 09/11] xen/arm: Transitional change to build HAS_VPCI on ARM Rahul Singh
2021-10-11 11:43 ` Roger Pau Monné
2021-10-11 12:15 ` Bertrand Marquis
2021-10-12 1:32 ` Stefano Stabellini
2021-10-06 17:40 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
2021-10-06 18:01 ` Julien Grall
2021-10-07 0:26 ` Stefano Stabellini
2021-10-07 15:31 ` Rahul Singh
2021-10-07 10:53 ` Ian Jackson
2021-10-07 15:29 ` Rahul Singh
2021-10-07 16:11 ` Ian Jackson
2021-10-11 13:46 ` Roger Pau Monné
2021-10-14 17:16 ` Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 0/3] PCI devices passthrough on Arm Bertrand Marquis
2021-10-14 14:49 ` [PATCH v6 1/3] xen/vpci: Move ecam access functions to common code Bertrand Marquis
2021-10-14 16:06 ` Jan Beulich
2021-10-14 17:09 ` Bertrand Marquis
2021-10-15 6:29 ` Jan Beulich
2021-10-15 7:37 ` Bertrand Marquis
2021-10-15 8:13 ` Jan Beulich
2021-10-15 8:20 ` Bertrand Marquis
2021-10-15 8:24 ` Jan Beulich
2021-10-15 9:49 ` Roger Pau Monné
2021-10-14 23:47 ` Stefano Stabellini
2021-10-15 7:44 ` Roger Pau Monné
2021-10-15 7:53 ` Bertrand Marquis
2021-10-15 9:53 ` Roger Pau Monné
2021-10-15 10:12 ` Bertrand Marquis
2021-10-15 10:14 ` Jan Beulich
2021-10-14 14:49 ` [PATCH v6 2/3] xen/arm: Enable the existing x86 virtual PCI support for ARM Bertrand Marquis
2021-10-14 23:49 ` Stefano Stabellini
2021-10-15 6:40 ` Jan Beulich
2021-10-15 9:59 ` Ian Jackson
2021-10-15 10:10 ` Bertrand Marquis
2021-10-15 8:00 ` Jan Beulich
2021-10-15 10:09 ` Bertrand Marquis
2021-10-15 10:14 ` Ian Jackson
2021-10-15 10:18 ` Jan Beulich
2021-10-15 11:35 ` Roger Pau Monné
2021-10-15 12:13 ` Bertrand Marquis
2021-10-15 12:18 ` Jan Beulich
2021-10-15 12:28 ` Bertrand Marquis
2021-10-15 13:00 ` Jan Beulich
2021-10-15 13:10 ` Bertrand Marquis
2021-10-15 10:38 ` Jan Beulich
2021-10-15 8:32 ` Roger Pau Monné
2021-10-15 8:42 ` Michal Orzel
2021-10-15 9:52 ` Bertrand Marquis
2021-10-15 10:13 ` Luca Fancellu
2021-10-15 10:17 ` Bertrand Marquis
2021-10-15 10:19 ` Roger Pau Monné
2021-10-15 10:31 ` Bertrand Marquis
2021-10-15 10:24 ` Jan Beulich
2021-10-15 10:33 ` Bertrand Marquis
2021-10-15 10:41 ` Jan Beulich
2021-10-15 10:48 ` Bertrand Marquis
2021-10-15 10:51 ` Jan Beulich
2021-10-15 11:08 ` Bertrand Marquis
2021-10-15 13:47 ` Roger Pau Monné
2021-10-15 14:00 ` Luca Fancellu
2021-10-15 14:32 ` Roger Pau Monné
2021-10-14 14:49 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl Bertrand Marquis
2021-10-14 17:54 ` [PATCH v6 3/3] arm/libxl: Emulated PCI device tree node in libxl [and 1 more messages] Ian Jackson
2021-10-14 23:50 ` Stefano Stabellini
2021-10-15 7:28 ` Julien Grall
2021-10-15 7:41 ` Michal Orzel
2021-10-15 9:01 ` Julien Grall
2021-10-15 10:02 ` Ian Jackson
2021-10-15 10:58 ` Michal Orzel
2021-10-15 11:04 ` Michal Orzel
2021-10-15 11:46 ` Ian Jackson
2021-10-15 11:53 ` Michal Orzel
2021-10-15 12:10 ` Julien Grall
2021-10-15 12:14 ` Ian Jackson
2021-10-15 12:13 ` Ian Jackson
2021-10-12 15:03 ` [PATCH v5 10/11] arm/libxl: Emulated PCI device tree node in libxl Ian Jackson
2021-10-06 17:40 ` [PATCH v5 11/11] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
2021-10-13 20:54 ` Stefano Stabellini
2021-10-07 19:54 ` [PATCH v5 00/11] PCI devices passthrough on Arm Stefano Stabellini
2021-10-07 21:29 ` Rahul Singh
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.