* [RFC PATCH v1 0/4] PCI devices passthrough on Arm
@ 2020-07-23 15:40 Rahul Singh
2020-07-23 15:40 ` [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM Rahul Singh
` (3 more replies)
0 siblings, 4 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-23 15:40 UTC (permalink / raw)
To: xen-devel
Cc: rahul.singh, Julien Grall, Wei Liu, Paul Durrant, Ian Jackson,
Bertrand.Marquis, Stefano Stabellini, Jan Beulich,
Anthony PERARD, nd, Volodymyr Babchuk
Following up on the discussion on PCI devices passthrough support on Arm
design proposal.Please feel free to give you feedback.
We are submitting the code that we developed to get the early feedback. PCI
passthrough support on ARM is not fully implemented in this patch series for
that reason we are not enabling the HAS_PCI and HAS_VPCI flags for ARM.
We will work on design document that we submitted for feedback on mailing
list and we will submit the next design document to address all the comments.
This patch series is based on v1 of the design document that we submitted for
review. Any comments in the design will be addressed in the later version of
the design document and will subsequently implemented in the code in next
patch series.
PCI passthrough support is divided into different patches:
Discovering PCI Host Bridge in XEN:
- PCI host bridge discovery in XEN and map the PCI ECAM configuration
space to the XEN memory.
Discovering PCI devices:
- In order to support the PCI passthrough, XEN should be aware of the PCI
devices.
- Hardware domain is in charge of doing the PCI enumeration and will discover
the PCI devices and then communicate to the XEN via hyper call to add the
PCI devices in XEN.
Enable the existing x86 virtual PCI support for ARM:
- Add VPCI trap handler for each of the PCI device added for config space
access.
- Register the trap handler in XEN for each of the host bridge PCI ECAM config
space access.
Emulated PCI device tree node in libxl:
- Create a virtual PCI device tree node in libxl to enable the guest OS to
discover the virtual PCI during guest boot.
This patch series does not implemented the following features. Following
features will be implemented in the next version of the patch series.
- MSI support for interrupt.
- ACPI support for PCI host bridge discovery within XEN on ARM.
- SMMU modification to support PCI devices.
- Use already defined config option "pci=[]" in place of new "vpci=ecam" config
option to create VPCI bus.
- Map the assigned device PCI BAR values and interrupt to the guest when device
is assigned by xl during domain creation.Currently we are using "iomem=[]"
config options to map the value to the guest.
Rahul Singh (4):
arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
xen/arm: Discovering PCI devices and add the PCI devices in XEN.
xen/arm: Enable the existing x86 virtual PCI support for ARM.
arm/libxl: Emulated PCI device tree node in libxl
tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 6 +
tools/xl/xl_parse.c | 7 +
xen/arch/arm/Kconfig | 7 +
xen/arch/arm/Makefile | 2 +
xen/arch/arm/domain.c | 4 +
xen/arch/arm/pci/Makefile | 4 +
xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
xen/arch/arm/pci/pci-host-common.c | 198 +++++++++++++++++++++++++++
xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
xen/arch/arm/physdev.c | 42 +++++-
xen/arch/arm/setup.c | 2 +
xen/arch/arm/vpci.c | 102 ++++++++++++++
xen/arch/arm/vpci.h | 37 +++++
xen/drivers/passthrough/pci.c | 7 +
xen/include/asm-arm/device.h | 7 +-
xen/include/asm-arm/domain.h | 5 +
xen/include/asm-arm/pci.h | 97 +++++++++++++-
xen/include/public/arch-arm.h | 32 +++++
20 files changed, 1094 insertions(+), 9 deletions(-)
create mode 100644 xen/arch/arm/pci/Makefile
create mode 100644 xen/arch/arm/pci/pci-access.c
create mode 100644 xen/arch/arm/pci/pci-host-common.c
create mode 100644 xen/arch/arm/pci/pci-host-generic.c
create mode 100644 xen/arch/arm/pci/pci.c
create mode 100644 xen/arch/arm/vpci.c
create mode 100644 xen/arch/arm/vpci.h
--
2.17.1
^ permalink raw reply [flat|nested] 48+ messages in thread
* [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-23 15:40 [RFC PATCH v1 0/4] PCI devices passthrough on Arm Rahul Singh
@ 2020-07-23 15:40 ` Rahul Singh
2020-07-23 23:38 ` Stefano Stabellini
` (2 more replies)
2020-07-23 15:40 ` [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
` (2 subsequent siblings)
3 siblings, 3 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-23 15:40 UTC (permalink / raw)
To: xen-devel
Cc: rahul.singh, Julien Grall, Bertrand.Marquis, Stefano Stabellini,
nd, Volodymyr Babchuk
XEN during boot will read the PCI device tree node “reg” property
and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree
node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
xen/arch/arm/Kconfig | 7 +
xen/arch/arm/Makefile | 1 +
xen/arch/arm/pci/Makefile | 4 +
xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
xen/arch/arm/setup.c | 2 +
xen/include/asm-arm/device.h | 7 +-
xen/include/asm-arm/pci.h | 97 +++++++++++++-
10 files changed, 654 insertions(+), 6 deletions(-)
create mode 100644 xen/arch/arm/pci/Makefile
create mode 100644 xen/arch/arm/pci/pci-access.c
create mode 100644 xen/arch/arm/pci/pci-host-common.c
create mode 100644 xen/arch/arm/pci/pci-host-generic.c
create mode 100644 xen/arch/arm/pci/pci.c
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 2777388265..ee13339ae9 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -31,6 +31,13 @@ menu "Architecture Features"
source "arch/Kconfig"
+config ARM_PCI
+ bool "PCI Passthrough Support"
+ depends on ARM_64
+ ---help---
+
+ PCI passthrough support for Xen on ARM64.
+
config ACPI
bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
depends on ARM_64
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 7e82b2178c..345cb83eed 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/
obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
new file mode 100644
index 0000000000..358508b787
--- /dev/null
+++ b/xen/arch/arm/pci/Makefile
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o
diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
new file mode 100644
index 0000000000..c53ef58336
--- /dev/null
+++ b/xen/arch/arm/pci/pci-access.c
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+ unsigned int len)
+{
+ int rc;
+ uint32_t val = GENMASK(0, len * 8);
+
+ struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+ if ( unlikely(!bridge) )
+ {
+ printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+ sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+ return val;
+ }
+
+ if ( unlikely(!bridge->ops->read) )
+ return val;
+
+ rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
+ if ( rc )
+ printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
+ reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+
+ return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+ unsigned int len, uint32_t val)
+{
+ int rc;
+ struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+ if ( unlikely(!bridge) )
+ {
+ printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+ sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+ return;
+ }
+
+ if ( unlikely(!bridge->ops->write) )
+ return;
+
+ rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
+ if ( rc )
+ printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
+ reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+ void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
+{ \
+ pci_config_write(sbdf, reg, size / 8, val); \
+}
+
+#define PCI_OP_READ(size, type) \
+ type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
+{ \
+ return pci_config_read(sbdf, reg, size / 8); \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
new file mode 100644
index 0000000000..c5f98be698
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/vmap.h>
+
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+ struct pci_config_window *cfg)
+{
+ const __be32 *cells;
+ uint32_t len;
+
+ cells = dt_get_property(dev, "bus-range", &len);
+ /* bus-range should at least be 2 cells */
+ if ( !cells || (len < (sizeof(*cells) * 2)) )
+ return false;
+
+ cfg->busn_start = dt_next_cell(1, &cells);
+ cfg->busn_end = dt_next_cell(1, &cells);
+
+ return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+ return ioremap_nocache(start, len);
+}
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+ if ( cfg->win )
+ iounmap(cfg->win);
+
+ xfree(cfg);
+}
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+ struct pci_ecam_ops *ops)
+{
+ int err;
+ struct pci_config_window *cfg;
+ paddr_t addr, size;
+
+ cfg = xzalloc(struct pci_config_window);
+ if ( !cfg )
+ return NULL;
+
+ err = dt_pci_parse_bus_range(dev, cfg);
+ if ( !err ) {
+ cfg->busn_start = 0;
+ cfg->busn_end = 0xff;
+ printk(XENLOG_ERR "No bus range found for pci controller\n");
+ } else {
+ if ( cfg->busn_end > cfg->busn_start + 0xff )
+ cfg->busn_end = cfg->busn_start + 0xff;
+ }
+
+ /* Parse our PCI ecam register address*/
+ err = dt_device_get_address(dev, 0, &addr, &size);
+ if ( err )
+ goto err_exit;
+
+ cfg->phys_addr = addr;
+ cfg->size = size;
+ cfg->ops = ops;
+
+ /*
+ * On 64-bit systems, we do a single ioremap for the whole config space
+ * since we have enough virtual address range available. On 32-bit, we
+ * ioremap the config space for each bus individually.
+ *
+ * As of now only 64-bit is supported 32-bit is not supported.
+ */
+ cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
+ if ( !cfg->win )
+ goto err_exit_remap;
+
+ printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+ cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
+
+ if ( ops->init ) {
+ err = ops->init(cfg);
+ if (err)
+ goto err_exit;
+ }
+
+ return cfg;
+
+err_exit_remap:
+ printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+ pci_ecam_free(cfg);
+ return NULL;
+}
+
+static struct pci_host_bridge * pci_alloc_host_bridge(void)
+{
+ struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+ if ( !bridge )
+ return NULL;
+
+ INIT_LIST_HEAD(&bridge->node);
+ return bridge;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+ struct pci_ecam_ops *ops)
+{
+ struct pci_host_bridge *bridge;
+ struct pci_config_window *cfg;
+ u32 segment;
+
+ bridge = pci_alloc_host_bridge();
+ if ( !bridge )
+ return -ENOMEM;
+
+ /* Parse and map our Configuration Space windows */
+ cfg = gen_pci_init(dev, ops);
+ if ( !cfg )
+ return -ENOMEM;
+
+ bridge->dt_node = dev;
+ bridge->sysdata = cfg;
+ bridge->ops = &ops->pci_ops;
+
+ if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+ {
+ printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+ return -ENODEV;
+ }
+
+ bridge->segment = (u16)segment;
+
+ list_add_tail(&bridge->node, &pci_host_bridges);
+
+ return 0;
+}
+
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+ struct pci_host_bridge *bridge;
+ bool found = false;
+
+ list_for_each_entry( bridge, &pci_host_bridges, node )
+ {
+ if ( bridge->segment != segment )
+ continue;
+
+ found = true;
+ break;
+ }
+
+ return (found) ? bridge : NULL;
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
new file mode 100644
index 0000000000..cd67b3dec6
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-generic.c
@@ -0,0 +1,131 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <asm/device.h>
+#include <asm/io.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+/*
+ * Function to get the config space base.
+ */
+static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
+ uint32_t sbdf, int where)
+{
+ struct pci_config_window *cfg = bridge->sysdata;
+ unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+
+ pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+
+ unsigned int busn = sbdf_t.bus;
+ void __iomem *base;
+
+ if ( busn < cfg->busn_start || busn > cfg->busn_end )
+ return NULL;
+
+ base = cfg->win + (busn << cfg->ops->bus_shift);
+
+ return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
+int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+ int where, int size, u32 val)
+{
+ void __iomem *addr;
+
+ addr = pci_config_base(bridge, sbdf, where);
+ if ( !addr )
+ return -ENODEV;
+
+ if ( size == 1 )
+ writeb(val, addr);
+ else if ( size == 2 )
+ writew(val, addr);
+ else
+ writel(val, addr);
+
+ return 0;
+}
+
+int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+ int where, int size, u32 *val)
+{
+ void __iomem *addr;
+
+ addr = pci_config_base(bridge, sbdf, where);
+ if ( !addr ) {
+ *val = ~0;
+ return -ENODEV;
+ }
+
+ if ( size == 1 )
+ *val = readb(addr);
+ else if ( size == 2 )
+ *val = readw(addr);
+ else
+ *val = readl(addr);
+
+ return 0;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {
+ .bus_shift = 20,
+ .pci_ops = {
+ .read = pci_ecam_config_read,
+ .write = pci_ecam_config_write,
+ }
+};
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+ { .compatible = "pci-host-ecam-generic",
+ .data = &pci_generic_ecam_ops },
+
+ { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+ const struct dt_device_match *of_id;
+ struct pci_ecam_ops *ops;
+
+ of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+ ops = (struct pci_ecam_ops *) of_id->data;
+
+ printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+ dt_node_full_name(dev), of_id->compatible);
+
+ return pci_host_common_probe(dev, ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
new file mode 100644
index 0000000000..f8cbb99591
--- /dev/null
+++ b/xen/arch/arm/pci/pci.c
@@ -0,0 +1,112 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/param.h>
+
+static int __init dt_pci_init(void)
+{
+ struct dt_device_node *np;
+ int rc;
+
+ dt_for_each_device_node(dt_host, np)
+ {
+ rc = device_init(np, DEVICE_PCI, NULL);
+ if( !rc )
+ continue;
+ /*
+ * Ignore the following error codes:
+ * - EBADF: Indicate the current is not an pci
+ * - ENODEV: The pci device is not present or cannot be used by
+ * Xen.
+ */
+ else if ( rc != -EBADF && rc != -ENODEV )
+ {
+ printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+ return rc;
+ }
+ }
+
+ return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+ printk(XENLOG_ERR "ACPI pci init not supported \n");
+ return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static bool __initdata param_pci_enable;
+static int __init parse_pci_param(const char *arg)
+{
+ if ( !arg )
+ {
+ param_pci_enable = false;
+ return 0;
+ }
+
+ switch ( parse_bool(arg, NULL) )
+ {
+ case 0:
+ param_pci_enable = false;
+ return 0;
+ case 1:
+ param_pci_enable = true;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+custom_param("pci", parse_pci_param);
+
+void __init pci_init(void)
+{
+ /*
+ * Enable PCI when has been enabled explicitly (pci=on)
+ */
+ if ( !param_pci_enable)
+ goto disable;
+
+ if ( acpi_disabled )
+ dt_pci_init();
+ else
+ acpi_pci_init();
+
+#ifdef CONFIG_HAS_PCI
+ pci_segments_init();
+#endif
+
+disable:
+ return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index 7968cee47d..2d7f1db44f 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
setup_virt_paging();
+ pci_init();
+
do_initcalls();
/*
diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
index ee7cff2d44..28f8049cfd 100644
--- a/xen/include/asm-arm/device.h
+++ b/xen/include/asm-arm/device.h
@@ -4,6 +4,7 @@
enum device_type
{
DEV_DT,
+ DEV_PCI,
};
struct dev_archdata {
@@ -25,15 +26,15 @@ typedef struct device device_t;
#include <xen/device_tree.h>
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)
-#define dev_is_dt(dev) ((dev->type == DEV_DT)
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev) (dev->type == DEV_DT)
enum device_class
{
DEVICE_SERIAL,
DEVICE_IOMMU,
DEVICE_GIC,
+ DEVICE_PCI,
/* Use for error */
DEVICE_UNKNOWN,
};
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index de13359f65..94fd00360a 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
+/* Arch pci dev struct */
struct arch_pci_dev {
+ struct device dev;
+};
+
+#define PRI_pci "%04x:%02x:%02x.%u"
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+ paddr_t phys_addr;
+ paddr_t size;
+ uint8_t busn_start;
+ uint8_t busn_end;
+ struct pci_ecam_ops *ops;
+ void __iomem *win;
+};
+
+/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
+struct pci_host_bridge;
+
+struct pci_ops {
+ int (*read)(struct pci_host_bridge *bridge,
+ uint32_t sbdf, int where, int size, u32 *val);
+ int (*write)(struct pci_host_bridge *bridge,
+ uint32_t sbdf, int where, int size, u32 val);
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+ unsigned int bus_shift;
+ struct pci_ops pci_ops;
+ int (*init)(struct pci_config_window *);
+};
+
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+ struct dt_device_node *dt_node; /* Pointer to the associated DT node */
+ struct list_head node; /* Node in list of host bridges */
+ uint16_t segment; /* Segment number */
+ void *sysdata; /* Pointer to the config space window*/
+ const struct pci_ops *ops;
};
-#endif /* __X86_PCI_H__ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
+
+int pci_host_common_probe(struct dt_device_node *dev,
+ struct pci_ecam_ops *ops);
+
+void pci_init(void);
+
+#else /*!CONFIG_ARM_PCI*/
+struct arch_pci_dev { };
+static inline void pci_init(void) { }
+#endif /*!CONFIG_ARM_PCI*/
+#endif /* __ARM_PCI_H__ */
--
2.17.1
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
2020-07-23 15:40 [RFC PATCH v1 0/4] PCI devices passthrough on Arm Rahul Singh
2020-07-23 15:40 ` [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM Rahul Singh
@ 2020-07-23 15:40 ` Rahul Singh
2020-07-23 20:44 ` Stefano Stabellini
2020-07-23 15:40 ` [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
2020-07-23 15:40 ` [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
3 siblings, 1 reply; 48+ messages in thread
From: Rahul Singh @ 2020-07-23 15:40 UTC (permalink / raw)
To: xen-devel
Cc: rahul.singh, Julien Grall, Bertrand.Marquis, Stefano Stabellini,
nd, Volodymyr Babchuk
Hardware domain is in charge of doing the PCI enumeration and will
discover the PCI devices and then will communicate to XEN via hyper
call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
index e91355fe22..274720f98a 100644
--- a/xen/arch/arm/physdev.c
+++ b/xen/arch/arm/physdev.c
@@ -9,12 +9,48 @@
#include <xen/errno.h>
#include <xen/sched.h>
#include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
{
- gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
- return -ENOSYS;
+ int ret = 0;
+
+ switch ( cmd )
+ {
+#ifdef CONFIG_HAS_PCI
+ case PHYSDEVOP_pci_device_add:
+ {
+ struct physdev_pci_device_add add;
+ struct pci_dev_info pdev_info;
+ nodeid_t node = NUMA_NO_NODE;
+
+ ret = -EFAULT;
+ if ( copy_from_guest(&add, arg, 1) != 0 )
+ break;
+
+ pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+ if ( add.flags & XEN_PCI_DEV_VIRTFN )
+ {
+ pdev_info.is_virtfn = 1;
+ pdev_info.physfn.bus = add.physfn.bus;
+ pdev_info.physfn.devfn = add.physfn.devfn;
+ }
+ else
+ pdev_info.is_virtfn = 0;
+
+ ret = pci_add_device(add.seg, add.bus, add.devfn,
+ &pdev_info, node);
+
+ break;
+ }
+#endif
+ default:
+ gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+ ret = -ENOSYS;
+ }
+
+ return ret;
}
/*
--
2.17.1
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2020-07-23 15:40 [RFC PATCH v1 0/4] PCI devices passthrough on Arm Rahul Singh
2020-07-23 15:40 ` [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM Rahul Singh
2020-07-23 15:40 ` [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
@ 2020-07-23 15:40 ` Rahul Singh
2020-07-23 23:39 ` Stefano Stabellini
2020-07-24 15:08 ` Roger Pau Monné
2020-07-23 15:40 ` [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
3 siblings, 2 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-23 15:40 UTC (permalink / raw)
To: xen-devel
Cc: rahul.singh, Julien Grall, Paul Durrant, Bertrand.Marquis,
Stefano Stabellini, Jan Beulich, nd, Volodymyr Babchuk
The existing VPCI support available for X86 is adapted for Arm.
When the device is added to XEN via the hyper call
“PHYSDEVOP_pci_device_add”, VPCI handler for the config space
access is added to the PCI device to emulate the PCI devices.
A MMIO trap handler for the PCI ECAM space is registered in XEN
so that when guest is trying to access the PCI config space,XEN
will trap the access and emulate read/write using the VPCI and
not the real PCI hardware.
VPCI MSI support is disable for ARM as it is not tested on ARM.
Change-Id: I5501db2781f8064640403fecce53713091cd9ab4
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
xen/arch/arm/Makefile | 1 +
xen/arch/arm/domain.c | 4 ++
xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
xen/arch/arm/vpci.h | 37 ++++++++++++
xen/drivers/passthrough/pci.c | 7 +++
xen/include/asm-arm/domain.h | 5 ++
xen/include/public/arch-arm.h | 4 ++
7 files changed, 160 insertions(+)
create mode 100644 xen/arch/arm/vpci.c
create mode 100644 xen/arch/arm/vpci.h
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 345cb83eed..5a23ec5cc0 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -7,6 +7,7 @@ obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
obj-$(CONFIG_ARM_PCI) += pci/
+obj-$(CONFIG_HAS_VPCI) += vpci.o
obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 31169326b2..23098ffd02 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -39,6 +39,7 @@
#include <asm/vtimer.h>
#include "vuart.h"
+#include "vpci.h"
DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
@@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,
if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
goto fail;
+ if ( (rc = domain_vpci_init(d)) != 0 )
+ goto fail;
+
return 0;
fail:
diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
new file mode 100644
index 0000000000..49e473ab0d
--- /dev/null
+++ b/xen/arch/arm/vpci.c
@@ -0,0 +1,102 @@
+/*
+ * xen/arch/arm/vpci.c
+ * Copyright (c) 2020 Arm Ltd.
+ *
+ * Based on arch/x86/hvm/io.c
+ * Copyright (c) 2004, Intel Corporation.
+ * Copyright (c) 2005, International Business Machines Corporation.
+ * Copyright (c) 2008, Citrix Systems, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <xen/sched.h>
+#include <asm/mmio.h>
+
+/* Do some sanity checks. */
+static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
+{
+ /* Check access size. */
+ if ( len != 1 && len != 2 && len != 4 && len != 8 )
+ return false;
+
+ /* Check that access is size aligned. */
+ if ( (reg & (len - 1)) )
+ return false;
+
+ return true;
+}
+
+static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
+ register_t *r, void *priv)
+{
+ unsigned int reg;
+ pci_sbdf_t sbdf;
+ uint32_t data = 0;
+ unsigned int size = 1U << info->dabt.size;
+
+ sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
+ reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+ if ( !vpci_mmio_access_allowed(reg, size) )
+ return 1;
+
+ data = vpci_read(sbdf, reg, size);
+
+ memcpy(r, &data, size);
+
+ return 1;
+}
+
+static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
+ register_t r, void *priv)
+{
+ unsigned int reg;
+ pci_sbdf_t sbdf;
+ uint32_t data = r;
+ unsigned int size = 1U << info->dabt.size;
+
+ sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
+ reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+ if ( !vpci_mmio_access_allowed(reg, size) )
+ return 1;
+
+ vpci_write(sbdf, reg, size, data);
+
+ return 1;
+}
+
+static const struct mmio_handler_ops vpci_mmio_handler = {
+ .read = vpci_mmio_read,
+ .write = vpci_mmio_write,
+};
+
+int domain_vpci_init(struct domain *d)
+{
+ if ( !has_vpci(d) || is_hardware_domain(d) )
+ return 0;
+
+ register_mmio_handler(d, &vpci_mmio_handler,
+ GUEST_VPCI_ECAM_BASE,GUEST_VPCI_ECAM_SIZE,NULL);
+
+ return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
new file mode 100644
index 0000000000..20dce1f4c4
--- /dev/null
+++ b/xen/arch/arm/vpci.h
@@ -0,0 +1,37 @@
+/*
+ * xen/arch/arm/vpci.h
+ * Copyright (c) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPCI_H__
+#define __ARCH_ARM_VPCI_H__
+
+#ifdef CONFIG_HAS_VPCI
+int domain_vpci_init(struct domain *d);
+#else
+static inline int domain_vpci_init(struct domain *d)
+{
+ return 0;
+}
+#endif
+
+#endif /* __ARCH_ARM_VPCI_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 5846978890..28511eb641 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
else
iommu_enable_device(pdev);
+#ifdef CONFIG_ARM
+ ret = vpci_add_handlers(pdev);
+ if ( ret ) {
+ printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
+ goto out;
+ }
+#endif
pci_enable_acs(pdev);
out:
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 4e2f582006..ad70610226 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -34,6 +34,11 @@ enum domain_type {
/* The hardware domain has always its memory direct mapped. */
#define is_domain_direct_mapped(d) ((d) == hardware_domain)
+/* For X86 VPCI is enabled and tested for PVH DOM0 only but
+ * for ARM we enable support VPCI for guest domain also.
+ */
+#define has_vpci(d) (true)
+
struct vtimer {
struct vcpu *v;
int irq;
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index c365b1b39e..7364a07362 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
#define GUEST_PL011_BASE xen_mk_ullong(0x22000000)
#define GUEST_PL011_SIZE xen_mk_ullong(0x00001000)
+/* VPCI ECAM mappings */
+#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
+#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
+
/*
* 16MB == 4096 pages reserved for guest to use as a region to map its
* grant table in.
--
2.17.1
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
2020-07-23 15:40 [RFC PATCH v1 0/4] PCI devices passthrough on Arm Rahul Singh
` (2 preceding siblings ...)
2020-07-23 15:40 ` [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
@ 2020-07-23 15:40 ` Rahul Singh
2020-07-23 23:39 ` Stefano Stabellini
3 siblings, 1 reply; 48+ messages in thread
From: Rahul Singh @ 2020-07-23 15:40 UTC (permalink / raw)
To: xen-devel
Cc: rahul.singh, Julien Grall, Wei Liu, Ian Jackson,
Bertrand.Marquis, Stefano Stabellini, Anthony PERARD, nd,
Volodymyr Babchuk
libxl will create an emulated PCI device tree node in the
device tree to enable the guest OS to discover the virtual
PCI during guest boot.
We introduced the new config option [vpci="ecam"] for guests.
When this config option is enabled in a guest configuration,
a PCI device tree node will be created in the guest device tree.
A new area has been reserved in the arm guest physical map at
which the VPCI bus is declared in the device tree (reg and ranges
parameters of the node).
Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 6 +
tools/xl/xl_parse.c | 7 ++
xen/include/public/arch-arm.h | 28 +++++
4 files changed, 241 insertions(+)
diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
index 34f8a29056..84568e9dc9 100644
--- a/tools/libxl/libxl_arm.c
+++ b/tools/libxl/libxl_arm.c
@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
return fdt_property(fdt, "reg", regs, sizeof(regs));
}
+static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
+ unsigned num_cells, ...)
+{
+ uint32_t bus_range[num_cells];
+ be32 *cells = &bus_range[0];
+ int i;
+ va_list ap;
+ uint32_t arg;
+
+ va_start(ap, num_cells);
+ for (i = 0 ; i < num_cells; i++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1, arg);
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
+}
+
+static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
+ unsigned num_cells, ...)
+{
+ uint32_t interrupt_map_mask[num_cells];
+ be32 *cells = &interrupt_map_mask[0];
+ int i;
+ va_list ap;
+ uint32_t arg;
+
+ va_start(ap, num_cells);
+ for (i = 0 ; i < num_cells; i++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1, arg);
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
+ sizeof(interrupt_map_mask));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+ unsigned vpci_addr_cells,
+ unsigned cpu_addr_cells,
+ unsigned vpci_size_cells,
+ unsigned num_regs, ...)
+{
+ uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
+ be32 *cells = ®s[0];
+ int i;
+ va_list ap;
+ uint64_t arg;
+
+ va_start(ap, num_regs);
+ for (i = 0 ; i < num_regs; i++) {
+ /* Set the memory bit field */
+ arg = va_arg(ap, uint64_t);
+ set_cell(&cells, 1, arg);
+
+ /* Set the vpci bus address */
+ arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
+ set_cell(&cells, 2 , arg);
+
+ /* Set the cpu bus address where vpci address is mapped */
+ arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
+ set_cell(&cells, cpu_addr_cells, arg);
+
+ /* Set the vpci size requested */
+ arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
+ set_cell(&cells, vpci_size_cells,arg);
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
+static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
+ unsigned child_unit_addr_cells,
+ unsigned child_interrupt_specifier_cells,
+ unsigned parent_unit_addr_cells,
+ unsigned parent_interrupt_specifier_cells,
+ unsigned num_regs, ...)
+{
+ uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
+ child_interrupt_specifier_cells + parent_unit_addr_cells
+ + parent_interrupt_specifier_cells + 1)];
+ be32 *cells = &interrupt_map[0];
+ int i,j;
+ va_list ap;
+ uint64_t arg;
+
+ va_start(ap, num_regs);
+ for (i = 0 ; i < num_regs; i++) {
+ /* Set the child unit address*/
+ for (j = 0 ; j < child_unit_addr_cells; j++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1 , arg);
+ }
+
+ /* Set the child interrupt specifier*/
+ for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1 , arg);
+ }
+
+ /* Set the interrupt-parent*/
+ set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
+
+ /* Set the parent unit address*/
+ for (j = 0 ; j < parent_unit_addr_cells; j++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1 , arg);
+ }
+
+ /* Set the parent interrupt specifier*/
+ for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
+ arg = va_arg(ap, uint32_t);
+ set_cell(&cells, 1 , arg);
+ }
+ }
+ va_end(ap);
+
+ return fdt_property(fdt, "interrupt-map", interrupt_map,
+ sizeof(interrupt_map));
+}
+
static int make_root_properties(libxl__gc *gc,
const libxl_version_info *vers,
void *fdt)
@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
return 0;
}
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+ const struct arch_info *ainfo,
+ struct xc_dom_image *dom)
+{
+ int res;
+ const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+ const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+ const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+ res = fdt_begin_node(fdt, name);
+ if (res) return res;
+
+ res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+ if (res) return res;
+
+ res = fdt_property_string(fdt, "device_type", "pci");
+ if (res) return res;
+
+ res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+ GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+ if (res) return res;
+
+ res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "linux,pci-domain", 0);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#address-cells", 3);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#size-cells", 2);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#interrupt-cells", 1);
+ if (res) return res;
+
+ res = fdt_property_string(fdt, "status", "okay");
+ if (res) return res;
+
+ res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
+ GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,
+ 3,
+ GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
+ GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
+ GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
+ GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
+ GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
+ GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
+ if (res) return res;
+
+ res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);
+ if (res) return res;
+
+ /*
+ * Legacy interrupt is forced and assigned to the guest.
+ * This will be removed once we have implementation for MSI support.
+ *
+ */
+ res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
+ 4,
+ 0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
+ 0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
+ 0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
+ 0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
+ if (res) return res;
+
+ res = fdt_end_node(fdt);
+ if (res) return res;
+
+ return 0;
+}
+
static const struct arch_info *get_arch_info(libxl__gc *gc,
const struct xc_dom_image *dom)
{
@@ -962,6 +1159,9 @@ next_resize:
if (info->tee == LIBXL_TEE_TYPE_OPTEE)
FDT( make_optee_node(gc, fdt) );
+ if (info->arch_arm.vpci == LIBXL_VPCI_TYPE_ECAM)
+ FDT( make_vpci_node(gc, fdt, ainfo, dom) );
+
if (pfdt)
FDT( copy_partial_fdt(gc, fdt, pfdt) );
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 9d3f05f399..d493637705 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -257,6 +257,11 @@ libxl_vuart_type = Enumeration("vuart_type", [
(1, "sbsa_uart"),
])
+libxl_vpci_type = Enumeration("vpci_type", [
+ (0, "unknown"),
+ (1, "ecam"),
+ ])
+
libxl_vkb_backend = Enumeration("vkb_backend", [
(0, "UNKNOWN"),
(1, "QEMU"),
@@ -640,6 +645,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
("vuart", libxl_vuart_type),
+ ("vpci", libxl_vpci_type),
])),
# Alternate p2m is not bound to any architecture or guest type, as it is
# supported by x86 HVM and ARM support is planned.
diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
index 61b4ef7b7e..58b7e6f56a 100644
--- a/tools/xl/xl_parse.c
+++ b/tools/xl/xl_parse.c
@@ -1386,6 +1386,13 @@ void parse_config_data(const char *config_source,
}
}
+ if (!xlu_cfg_get_string(config, "vpci", &buf, 0)) {
+ if (libxl_vpci_type_from_string(buf, &b_info->arch_arm.vpci)) {
+ fprintf(stderr, "ERROR: invalid value \"%s\" for \"vpci\"\n",
+ buf);
+ exit(1);
+ }
+ }
parse_vnuma_config(config, b_info);
/* Set max_memkb to target_memkb and max_vcpus to avail_vcpus if
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 7364a07362..4e19c62948 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
+#define GUEST_PCI_ADDRESS_CELLS 3
+#define GUEST_PCI_SIZE_CELLS 2
+
+/* PCI-PCIe memory space types */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000)
+#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000)
+
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000)
+#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000)
+#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800)
+
+/*
+ * This is hardcoded values for the real PCI physical addresses.
+ * This will be removed once we read the real PCI-PCIe physical
+ * addresses form the config space and map to the guest memory map
+ * when assigning the device to guest via VPCI.
+ *
+ */
+#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000)
+#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000)
+#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000)
+
+#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000)
+#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000)
+#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000)
+
/*
* 16MB == 4096 pages reserved for guest to use as a region to map its
* grant table in.
--
2.17.1
^ permalink raw reply related [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
2020-07-23 15:40 ` [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
@ 2020-07-23 20:44 ` Stefano Stabellini
2020-07-24 7:14 ` Oleksandr Andrushchenko
` (2 more replies)
0 siblings, 3 replies; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-23 20:44 UTC (permalink / raw)
To: Rahul Singh
Cc: Stefano Stabellini, Julien Grall, andrew.cooper3,
Bertrand.Marquis, jbeulich, xen-devel, nd, Volodymyr Babchuk,
roger.pau
On Thu, 23 Jul 2020, Rahul Singh wrote:
> Hardware domain is in charge of doing the PCI enumeration and will
> discover the PCI devices and then will communicate to XEN via hyper
> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
>
> Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Same question about Change-Id
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 39 insertions(+), 3 deletions(-)
>
> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
> index e91355fe22..274720f98a 100644
> --- a/xen/arch/arm/physdev.c
> +++ b/xen/arch/arm/physdev.c
> @@ -9,12 +9,48 @@
> #include <xen/errno.h>
> #include <xen/sched.h>
> #include <asm/hypercall.h>
> -
> +#include <xen/guest_access.h>
> +#include <xsm/xsm.h>
>
> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> {
> - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> - return -ENOSYS;
> + int ret = 0;
> +
> + switch ( cmd )
> + {
> +#ifdef CONFIG_HAS_PCI
> + case PHYSDEVOP_pci_device_add:
> + {
> + struct physdev_pci_device_add add;
> + struct pci_dev_info pdev_info;
> + nodeid_t node = NUMA_NO_NODE;
> +
> + ret = -EFAULT;
> + if ( copy_from_guest(&add, arg, 1) != 0 )
> + break;
> +
> + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
> + {
> + pdev_info.is_virtfn = 1;
> + pdev_info.physfn.bus = add.physfn.bus;
> + pdev_info.physfn.devfn = add.physfn.devfn;
> + }
> + else
> + pdev_info.is_virtfn = 0;
> +
> + ret = pci_add_device(add.seg, add.bus, add.devfn,
> + &pdev_info, node);
> +
> + break;
> + }
> +#endif
> + default:
> + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> + ret = -ENOSYS;
> + }
I think we should make the implementation common between arm and x86 by
creating xen/common/physdev.c:do_physdev_op as a shared entry point for
PHYSDEVOP hypercalls implementations. See for instance:
xen/common/sysctl.c:do_sysctl
and
xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl
Jan, Andrew, Roger, any opinions?
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-23 15:40 ` [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM Rahul Singh
@ 2020-07-23 23:38 ` Stefano Stabellini
2020-07-24 7:03 ` Oleksandr Andrushchenko
` (2 more replies)
2020-07-24 8:23 ` Julien Grall
2020-07-24 14:44 ` Roger Pau Monné
2 siblings, 3 replies; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-23 23:38 UTC (permalink / raw)
To: Rahul Singh
Cc: Stefano Stabellini, Julien Grall, andrew.cooper3,
Bertrand.Marquis, jbeulich, xen-devel, nd, Volodymyr Babchuk,
roger.pau
[-- Attachment #1: Type: text/plain, Size: 25520 bytes --]
+ Jan, Andrew, Roger
Please have a look at my comment on whether we should share the MMCFG
code below, feel free to ignore the rest :-)
On Thu, 23 Jul 2020, Rahul Singh wrote:
> XEN during boot will read the PCI device tree node “reg” property
> and will map the PCI config space to the XEN memory.
>
> XEN will read the “linux, pci-domain” property from the device tree
> node and configure the host bridge segment number accordingly.
>
> As of now "pci-host-ecam-generic" compatible board is supported.
>
> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
What is this Change-Id property?
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/Kconfig | 7 +
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/pci/Makefile | 4 +
> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
> xen/arch/arm/setup.c | 2 +
> xen/include/asm-arm/device.h | 7 +-
> xen/include/asm-arm/pci.h | 97 +++++++++++++-
> 10 files changed, 654 insertions(+), 6 deletions(-)
> create mode 100644 xen/arch/arm/pci/Makefile
> create mode 100644 xen/arch/arm/pci/pci-access.c
> create mode 100644 xen/arch/arm/pci/pci-host-common.c
> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
> create mode 100644 xen/arch/arm/pci/pci.c
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 2777388265..ee13339ae9 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -31,6 +31,13 @@ menu "Architecture Features"
>
> source "arch/Kconfig"
>
> +config ARM_PCI
> + bool "PCI Passthrough Support"
> + depends on ARM_64
> + ---help---
> +
> + PCI passthrough support for Xen on ARM64.
> +
> config ACPI
> bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
> depends on ARM_64
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 7e82b2178c..345cb83eed 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_ARM_PCI) += pci/
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> new file mode 100644
> index 0000000000..358508b787
> --- /dev/null
> +++ b/xen/arch/arm/pci/Makefile
> @@ -0,0 +1,4 @@
> +obj-y += pci.o
> +obj-y += pci-host-generic.o
> +obj-y += pci-host-common.o
> +obj-y += pci-access.o
> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
> new file mode 100644
> index 0000000000..c53ef58336
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-access.c
> @@ -0,0 +1,101 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +#include <xen/rwlock.h>
> +
> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
> + unsigned int len)
> +{
> + int rc;
> + uint32_t val = GENMASK(0, len * 8);
> +
> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> +
> + if ( unlikely(!bridge) )
> + {
> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> + return val;
> + }
> +
> + if ( unlikely(!bridge->ops->read) )
> + return val;
> +
> + rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
> + if ( rc )
> + printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> +
> + return val;
> +}
> +
> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
> + unsigned int len, uint32_t val)
> +{
> + int rc;
> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> +
> + if ( unlikely(!bridge) )
> + {
> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> + return;
> + }
> +
> + if ( unlikely(!bridge->ops->write) )
> + return;
> +
> + rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
> + if ( rc )
> + printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> +}
> +
> +/*
> + * Wrappers for all PCI configuration access functions.
> + */
> +
> +#define PCI_OP_WRITE(size, type) \
> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
> +{ \
> + pci_config_write(sbdf, reg, size / 8, val); \
> +}
> +
> +#define PCI_OP_READ(size, type) \
> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
> +{ \
> + return pci_config_read(sbdf, reg, size / 8); \
> +}
> +
> +PCI_OP_READ(8, u8)
> +PCI_OP_READ(16, u16)
> +PCI_OP_READ(32, u32)
> +PCI_OP_WRITE(8, u8)
> +PCI_OP_WRITE(16, u16)
> +PCI_OP_WRITE(32, u32)
This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?
MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not
mistaken. Is there any value in sharing the code with x86? It is OK if
we don't, but I would like to understand the reasoning.
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> new file mode 100644
> index 0000000000..c5f98be698
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -0,0 +1,198 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +#include <xen/rwlock.h>
> +#include <xen/vmap.h>
> +
> +/*
> + * List for all the pci host bridges.
> + */
> +
> +static LIST_HEAD(pci_host_bridges);
> +
> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
> + struct pci_config_window *cfg)
> +{
> + const __be32 *cells;
> + uint32_t len;
> +
> + cells = dt_get_property(dev, "bus-range", &len);
> + /* bus-range should at least be 2 cells */
> + if ( !cells || (len < (sizeof(*cells) * 2)) )
> + return false;
> +
> + cfg->busn_start = dt_next_cell(1, &cells);
> + cfg->busn_end = dt_next_cell(1, &cells);
> +
> + return true;
> +}
> +
> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
> +{
> + return ioremap_nocache(start, len);
> +}
> +
> +static void pci_ecam_free(struct pci_config_window *cfg)
> +{
> + if ( cfg->win )
> + iounmap(cfg->win);
> +
> + xfree(cfg);
> +}
> +
> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> + struct pci_ecam_ops *ops)
> +{
> + int err;
> + struct pci_config_window *cfg;
> + paddr_t addr, size;
> +
> + cfg = xzalloc(struct pci_config_window);
> + if ( !cfg )
> + return NULL;
> +
> + err = dt_pci_parse_bus_range(dev, cfg);
> + if ( !err ) {
> + cfg->busn_start = 0;
> + cfg->busn_end = 0xff;
> + printk(XENLOG_ERR "No bus range found for pci controller\n");
> + } else {
> + if ( cfg->busn_end > cfg->busn_start + 0xff )
> + cfg->busn_end = cfg->busn_start + 0xff;
> + }
> +
> + /* Parse our PCI ecam register address*/
> + err = dt_device_get_address(dev, 0, &addr, &size);
> + if ( err )
> + goto err_exit;
Shouldn't we handle the possibility of multiple addresses? Is it
possible to have more than one range for an ECAM compliant host bridge?
> + cfg->phys_addr = addr;
> + cfg->size = size;
> + cfg->ops = ops;
> +
> + /*
> + * On 64-bit systems, we do a single ioremap for the whole config space
> + * since we have enough virtual address range available. On 32-bit, we
> + * ioremap the config space for each bus individually.
> + *
> + * As of now only 64-bit is supported 32-bit is not supported.
> + */
> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
> + if ( !cfg->win )
> + goto err_exit_remap;
> +
> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
> + cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
> +
> + if ( ops->init ) {
> + err = ops->init(cfg);
> + if (err)
> + goto err_exit;
> + }
> +
> + return cfg;
> +
> +err_exit_remap:
> + printk(XENLOG_ERR "ECAM ioremap failed\n");
> +err_exit:
> + pci_ecam_free(cfg);
> + return NULL;
> +}
> +
> +static struct pci_host_bridge * pci_alloc_host_bridge(void)
> +{
> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
> +
> + if ( !bridge )
> + return NULL;
> +
> + INIT_LIST_HEAD(&bridge->node);
> + return bridge;
> +}
> +
> +int pci_host_common_probe(struct dt_device_node *dev,
> + struct pci_ecam_ops *ops)
> +{
> + struct pci_host_bridge *bridge;
> + struct pci_config_window *cfg;
> + u32 segment;
> +
> + bridge = pci_alloc_host_bridge();
> + if ( !bridge )
> + return -ENOMEM;
> +
> + /* Parse and map our Configuration Space windows */
> + cfg = gen_pci_init(dev, ops);
> + if ( !cfg )
> + return -ENOMEM;
In case of errors the allocated bridge is not freed.
> + bridge->dt_node = dev;
> + bridge->sysdata = cfg;
> + bridge->ops = &ops->pci_ops;
> +
> + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
> + {
> + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
> + return -ENODEV;
> + }
> +
> + bridge->segment = (u16)segment;
My understanding is that a Linux pci-domain doesn't correspond exactly
to a PCI segment. See for instance:
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
Do we need to care about the difference? If we mean pci-domain here,
should we just call them as such instead of calling them "segments" in
Xen (if they are not segments)?
> + list_add_tail(&bridge->node, &pci_host_bridges);
It looks like &pci_host_bridges should be an ordered list, ordered by
segment number?
> + return 0;
> +}
> +
> +/*
> + * This function will lookup an hostbridge based on the segment and bus
> + * number.
> + */
> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
> +{
> + struct pci_host_bridge *bridge;
> + bool found = false;
> +
> + list_for_each_entry( bridge, &pci_host_bridges, node )
> + {
> + if ( bridge->segment != segment )
> + continue;
> +
> + found = true;
> + break;
> + }
> +
> + return (found) ? bridge : NULL;
> +}
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
> new file mode 100644
> index 0000000000..cd67b3dec6
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-generic.c
> @@ -0,0 +1,131 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/device.h>
> +#include <asm/io.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +
> +/*
> + * Function to get the config space base.
> + */
> +static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
> + uint32_t sbdf, int where)
I think the function is misnamed because reading the code below it looks
like it is not just returning the base config space address but also the
specific address we need to read/write (adding the device offset,
"where", and everything).
Maybe pci_config_get_address or something like that?
> +{
> + struct pci_config_window *cfg = bridge->sysdata;
> + unsigned int devfn_shift = cfg->ops->bus_shift - 8;
> +
> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
> +
> + unsigned int busn = sbdf_t.bus;
> + void __iomem *base;
> +
> + if ( busn < cfg->busn_start || busn > cfg->busn_end )
> + return NULL;
> +
> + base = cfg->win + (busn << cfg->ops->bus_shift);
> +
> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
> +}
> +
> +int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
> + int where, int size, u32 val)
> +{
> + void __iomem *addr;
> +
> + addr = pci_config_base(bridge, sbdf, where);
> + if ( !addr )
> + return -ENODEV;
> +
> + if ( size == 1 )
> + writeb(val, addr);
> + else if ( size == 2 )
> + writew(val, addr);
> + else
> + writel(val, addr);
please use a switch
> + return 0;
> +}
> +
> +int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
> + int where, int size, u32 *val)
> +{
> + void __iomem *addr;
> +
> + addr = pci_config_base(bridge, sbdf, where);
> + if ( !addr ) {
> + *val = ~0;
> + return -ENODEV;
> + }
> +
> + if ( size == 1 )
> + *val = readb(addr);
> + else if ( size == 2 )
> + *val = readw(addr);
> + else
> + *val = readl(addr);
please use a switch
> + return 0;
> +}
> +
> +/* ECAM ops */
> +struct pci_ecam_ops pci_generic_ecam_ops = {
> + .bus_shift = 20,
> + .pci_ops = {
> + .read = pci_ecam_config_read,
> + .write = pci_ecam_config_write,
> + }
> +};
> +
> +static const struct dt_device_match gen_pci_dt_match[] = {
> + { .compatible = "pci-host-ecam-generic",
> + .data = &pci_generic_ecam_ops },
spurious blank line
> + { },
> +};
> +
> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
> +{
> + const struct dt_device_match *of_id;
> + struct pci_ecam_ops *ops;
> +
> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> + ops = (struct pci_ecam_ops *) of_id->data;
> +
> + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
> + dt_node_full_name(dev), of_id->compatible);
> +
> + return pci_host_common_probe(dev, ops);
> +}
> +
> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
> +.dt_match = gen_pci_dt_match,
> +.init = gen_pci_dt_init,
> +DT_DEVICE_END
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
> new file mode 100644
> index 0000000000..f8cbb99591
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci.c
> @@ -0,0 +1,112 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/acpi.h>
> +#include <xen/device_tree.h>
> +#include <xen/errno.h>
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <xen/param.h>
> +
> +static int __init dt_pci_init(void)
> +{
> + struct dt_device_node *np;
> + int rc;
> +
> + dt_for_each_device_node(dt_host, np)
> + {
> + rc = device_init(np, DEVICE_PCI, NULL);
> + if( !rc )
> + continue;
> + /*
> + * Ignore the following error codes:
> + * - EBADF: Indicate the current is not an pci
> + * - ENODEV: The pci device is not present or cannot be used by
> + * Xen.
> + */
> + else if ( rc != -EBADF && rc != -ENODEV )
> + {
> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
> + return rc;
> + }
> + }
> +
> + return 0;
> +}
> +
> +#ifdef CONFIG_ACPI
> +static void __init acpi_pci_init(void)
> +{
> + printk(XENLOG_ERR "ACPI pci init not supported \n");
> + return;
> +}
> +#else
> +static inline void __init acpi_pci_init(void) { }
> +#endif
> +
> +static bool __initdata param_pci_enable;
> +static int __init parse_pci_param(const char *arg)
> +{
> + if ( !arg )
> + {
> + param_pci_enable = false;
> + return 0;
> + }
> +
> + switch ( parse_bool(arg, NULL) )
> + {
> + case 0:
> + param_pci_enable = false;
> + return 0;
> + case 1:
> + param_pci_enable = true;
> + return 0;
> + }
> +
> + return -EINVAL;
> +}
> +custom_param("pci", parse_pci_param);
When adding new command line parameters please also add its
documentation (docs/misc/xen-command-line.pandoc) in the same patch,
unless this is meant to be just transient and we'll get removed before
the final commit of the series.
> +void __init pci_init(void)
> +{
> + /*
> + * Enable PCI when has been enabled explicitly (pci=on)
> + */
> + if ( !param_pci_enable)
> + goto disable;
> +
> + if ( acpi_disabled )
> + dt_pci_init();
> + else
> + acpi_pci_init();
> +
> +#ifdef CONFIG_HAS_PCI
> + pci_segments_init();
> +#endif
> +
> +disable:
> + return;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index 7968cee47d..2d7f1db44f 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>
> setup_virt_paging();
>
> + pci_init();
pci_init should probably be an initcall
> do_initcalls();
>
> /*
> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
> index ee7cff2d44..28f8049cfd 100644
> --- a/xen/include/asm-arm/device.h
> +++ b/xen/include/asm-arm/device.h
> @@ -4,6 +4,7 @@
> enum device_type
> {
> DEV_DT,
> + DEV_PCI,
> };
>
> struct dev_archdata {
> @@ -25,15 +26,15 @@ typedef struct device device_t;
>
> #include <xen/device_tree.h>
>
> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
> -#define dev_is_pci(dev) ((void)(dev), 0)
> -#define dev_is_dt(dev) ((dev->type == DEV_DT)
> +#define dev_is_pci(dev) (dev->type == DEV_PCI)
> +#define dev_is_dt(dev) (dev->type == DEV_DT)
>
> enum device_class
> {
> DEVICE_SERIAL,
> DEVICE_IOMMU,
> DEVICE_GIC,
> + DEVICE_PCI,
> /* Use for error */
> DEVICE_UNKNOWN,
> };
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index de13359f65..94fd00360a 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -1,7 +1,98 @@
> -#ifndef __X86_PCI_H__
> -#define __X86_PCI_H__
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
>
> +#ifndef __ARM_PCI_H__
> +#define __ARM_PCI_H__
> +
> +#include <xen/pci.h>
> +#include <xen/device_tree.h>
> +#include <asm/device.h>
> +
> +#ifdef CONFIG_ARM_PCI
> +
> +/* Arch pci dev struct */
> struct arch_pci_dev {
> + struct device dev;
> +};
Are you actually using struct device in struct arch_pci_dev?
struct device is already part of struct dt_device_node and a pointer to
it is stored in bridge->dt_node.
> +#define PRI_pci "%04x:%02x:%02x.%u"
> +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
> +
> +/*
> + * struct to hold the mappings of a config space window. This
> + * is expected to be used as sysdata for PCI controllers that
> + * use ECAM.
> + */
> +struct pci_config_window {
> + paddr_t phys_addr;
> + paddr_t size;
> + uint8_t busn_start;
> + uint8_t busn_end;
> + struct pci_ecam_ops *ops;
> + void __iomem *win;
> +};
> +
> +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
> +struct pci_host_bridge;
> +
> +struct pci_ops {
> + int (*read)(struct pci_host_bridge *bridge,
> + uint32_t sbdf, int where, int size, u32 *val);
> + int (*write)(struct pci_host_bridge *bridge,
> + uint32_t sbdf, int where, int size, u32 val);
I'd prefer if we could use explicitly-sized integers for "where" and
"size" too. Also, should they be unsigned rather than signed?
Can we use pci_sbdf_t for the sbdf parameter?
> +};
> +
> +/*
> + * struct to hold pci ops and bus shift of the config window
> + * for a PCI controller.
> + */
> +struct pci_ecam_ops {
> + unsigned int bus_shift;
> + struct pci_ops pci_ops;
> + int (*init)(struct pci_config_window *);
> +};
Although I realize that we are only targeting ECAM now, and the
implementation is based on ECAM, the interface doesn't seem to have
anything ECAM-specific in it. I would rename pci_ecam_ops to something
else, maybe simply "pci_ops".
> +/*
> + * struct to hold pci host bridge information
> + * for a PCI controller.
> + */
> +struct pci_host_bridge {
> + struct dt_device_node *dt_node; /* Pointer to the associated DT node */
> + struct list_head node; /* Node in list of host bridges */
> + uint16_t segment; /* Segment number */
> + void *sysdata; /* Pointer to the config space window*/
> + const struct pci_ops *ops;
> };
>
> -#endif /* __X86_PCI_H__ */
> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
> +
> +int pci_host_common_probe(struct dt_device_node *dev,
> + struct pci_ecam_ops *ops);
> +
> +void pci_init(void);
> +
> +#else /*!CONFIG_ARM_PCI*/
> +struct arch_pci_dev { };
> +static inline void pci_init(void) { }
> +#endif /*!CONFIG_ARM_PCI*/
> +#endif /* __ARM_PCI_H__ */
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2020-07-23 15:40 ` [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
@ 2020-07-23 23:39 ` Stefano Stabellini
2020-07-24 15:08 ` Roger Pau Monné
1 sibling, 0 replies; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-23 23:39 UTC (permalink / raw)
To: Rahul Singh
Cc: Stefano Stabellini, Julien Grall, Paul Durrant, andrew.cooper3,
Bertrand.Marquis, Jan Beulich, xen-devel, nd, Volodymyr Babchuk,
roger.pau
[-- Attachment #1: Type: text/plain, Size: 8442 bytes --]
On Thu, 23 Jul 2020, Rahul Singh wrote:
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the PCI device to emulate the PCI devices.
>
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
>
> VPCI MSI support is disable for ARM as it is not tested on ARM.
>
> Change-Id: I5501db2781f8064640403fecce53713091cd9ab4
Same question
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 4 ++
> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
> xen/arch/arm/vpci.h | 37 ++++++++++++
> xen/drivers/passthrough/pci.c | 7 +++
> xen/include/asm-arm/domain.h | 5 ++
> xen/include/public/arch-arm.h | 4 ++
> 7 files changed, 160 insertions(+)
> create mode 100644 xen/arch/arm/vpci.c
> create mode 100644 xen/arch/arm/vpci.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 345cb83eed..5a23ec5cc0 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> obj-$(CONFIG_ARM_PCI) += pci/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 31169326b2..23098ffd02 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -39,6 +39,7 @@
> #include <asm/vtimer.h>
>
> #include "vuart.h"
> +#include "vpci.h"
>
> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>
> @@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,
> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> goto fail;
>
> + if ( (rc = domain_vpci_init(d)) != 0 )
> + goto fail;
> +
> return 0;
>
> fail:
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..49e473ab0d
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,102 @@
> +/*
> + * xen/arch/arm/vpci.c
> + * Copyright (c) 2020 Arm Ltd.
> + *
> + * Based on arch/x86/hvm/io.c
> + * Copyright (c) 2004, Intel Corporation.
> + * Copyright (c) 2005, International Business Machines Corporation.
> + * Copyright (c) 2008, Citrix Systems, Inc.
> + *
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +#include <asm/mmio.h>
> +
> +/* Do some sanity checks. */
> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> +{
> + /* Check access size. */
> + if ( len != 1 && len != 2 && len != 4 && len != 8 )
> + return false;
> +
> + /* Check that access is size aligned. */
> + if ( (reg & (len - 1)) )
> + return false;
> +
> + return true;
> +}
> +
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> + register_t *r, void *priv)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + uint32_t data = 0;
> + unsigned int size = 1U << info->dabt.size;
> +
> + sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> +
> + if ( !vpci_mmio_access_allowed(reg, size) )
> + return 1;
> +
> + data = vpci_read(sbdf, reg, size);
> +
> + memcpy(r, &data, size);
> +
> + return 1;
> +}
> +
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> + register_t r, void *priv)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + uint32_t data = r;
> + unsigned int size = 1U << info->dabt.size;
> +
> + sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> +
> + if ( !vpci_mmio_access_allowed(reg, size) )
> + return 1;
> +
> + vpci_write(sbdf, reg, size, data);
> +
> + return 1;
> +}
I wonder if we could share vpci_mmcfg_read/write. Again, it is OK if we
can't, or if it is not worth the effort. just want to make sure we
thought about it :-)
> +static const struct mmio_handler_ops vpci_mmio_handler = {
> + .read = vpci_mmio_read,
> + .write = vpci_mmio_write,
> +};
> +
> +int domain_vpci_init(struct domain *d)
> +{
> + if ( !has_vpci(d) || is_hardware_domain(d) )
> + return 0;
> +
> + register_mmio_handler(d, &vpci_mmio_handler,
> + GUEST_VPCI_ECAM_BASE,GUEST_VPCI_ECAM_SIZE,NULL);
> +
> + return 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> +
> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
> new file mode 100644
> index 0000000000..20dce1f4c4
> --- /dev/null
> +++ b/xen/arch/arm/vpci.h
> @@ -0,0 +1,37 @@
> +/*
> + * xen/arch/arm/vpci.h
> + * Copyright (c) 2020 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef __ARCH_ARM_VPCI_H__
> +#define __ARCH_ARM_VPCI_H__
> +
> +#ifdef CONFIG_HAS_VPCI
> +int domain_vpci_init(struct domain *d);
> +#else
> +static inline int domain_vpci_init(struct domain *d)
> +{
> + return 0;
> +}
> +#endif
> +
> +#endif /* __ARCH_ARM_VPCI_H__ */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> index 5846978890..28511eb641 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> else
> iommu_enable_device(pdev);
>
> +#ifdef CONFIG_ARM
> + ret = vpci_add_handlers(pdev);
> + if ( ret ) {
> + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
> + goto out;
> + }
> +#endif
> pci_enable_acs(pdev);
>
> out:
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 4e2f582006..ad70610226 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -34,6 +34,11 @@ enum domain_type {
> /* The hardware domain has always its memory direct mapped. */
> #define is_domain_direct_mapped(d) ((d) == hardware_domain)
>
> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
> + * for ARM we enable support VPCI for guest domain also.
> + */
> +#define has_vpci(d) (true)
As mentioned we could make this configurabled based on the presence of
pci=[] or something similar in device tree for dom0less guests.
> struct vtimer {
> struct vcpu *v;
> int irq;
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index c365b1b39e..7364a07362 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
> #define GUEST_PL011_BASE xen_mk_ullong(0x22000000)
> #define GUEST_PL011_SIZE xen_mk_ullong(0x00001000)
>
> +/* VPCI ECAM mappings */
> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
Is 256MB in size part of the ECAM standard?
> /*
> * 16MB == 4096 pages reserved for guest to use as a region to map its
> * grant table in.
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
2020-07-23 15:40 ` [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
@ 2020-07-23 23:39 ` Stefano Stabellini
2020-07-24 7:55 ` Oleksandr Andrushchenko
` (2 more replies)
0 siblings, 3 replies; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-23 23:39 UTC (permalink / raw)
To: Rahul Singh
Cc: Stefano Stabellini, Julien Grall, Wei Liu, Ian Jackson,
Bertrand.Marquis, Anthony PERARD, xen-devel, nd,
Volodymyr Babchuk
On Thu, 23 Jul 2020, Rahul Singh wrote:
> libxl will create an emulated PCI device tree node in the
> device tree to enable the guest OS to discover the virtual
> PCI during guest boot.
>
> We introduced the new config option [vpci="ecam"] for guests.
> When this config option is enabled in a guest configuration,
> a PCI device tree node will be created in the guest device tree.
>
> A new area has been reserved in the arm guest physical map at
> which the VPCI bus is declared in the device tree (reg and ranges
> parameters of the node).
>
> Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd
Same question
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++
> tools/libxl/libxl_types.idl | 6 +
> tools/xl/xl_parse.c | 7 ++
> xen/include/public/arch-arm.h | 28 +++++
> 4 files changed, 241 insertions(+)
>
> diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
> index 34f8a29056..84568e9dc9 100644
> --- a/tools/libxl/libxl_arm.c
> +++ b/tools/libxl/libxl_arm.c
> @@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
> return fdt_property(fdt, "reg", regs, sizeof(regs));
> }
>
> +static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
> + unsigned num_cells, ...)
> +{
> + uint32_t bus_range[num_cells];
> + be32 *cells = &bus_range[0];
> + int i;
> + va_list ap;
> + uint32_t arg;
> +
> + va_start(ap, num_cells);
> + for (i = 0 ; i < num_cells; i++) {
> + arg = va_arg(ap, uint32_t);
> + set_cell(&cells, 1, arg);
> + }
> + va_end(ap);
> +
> + return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
> +}
> +
> +static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
> + unsigned num_cells, ...)
> +{
> + uint32_t interrupt_map_mask[num_cells];
> + be32 *cells = &interrupt_map_mask[0];
> + int i;
> + va_list ap;
> + uint32_t arg;
> +
> + va_start(ap, num_cells);
> + for (i = 0 ; i < num_cells; i++) {
> + arg = va_arg(ap, uint32_t);
> + set_cell(&cells, 1, arg);
> + }
> + va_end(ap);
> +
> + return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
> + sizeof(interrupt_map_mask));
> +}
> +
> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
> + unsigned vpci_addr_cells,
> + unsigned cpu_addr_cells,
> + unsigned vpci_size_cells,
> + unsigned num_regs, ...)
> +{
> + uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
> + be32 *cells = ®s[0];
> + int i;
> + va_list ap;
> + uint64_t arg;
> +
> + va_start(ap, num_regs);
> + for (i = 0 ; i < num_regs; i++) {
> + /* Set the memory bit field */
> + arg = va_arg(ap, uint64_t);
> + set_cell(&cells, 1, arg);
> +
> + /* Set the vpci bus address */
> + arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
> + set_cell(&cells, 2 , arg);
> +
> + /* Set the cpu bus address where vpci address is mapped */
> + arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
> + set_cell(&cells, cpu_addr_cells, arg);
> +
> + /* Set the vpci size requested */
> + arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
> + set_cell(&cells, vpci_size_cells,arg);
> + }
> + va_end(ap);
> +
> + return fdt_property(fdt, "ranges", regs, sizeof(regs));
> +}
> +
> +static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
> + unsigned child_unit_addr_cells,
> + unsigned child_interrupt_specifier_cells,
> + unsigned parent_unit_addr_cells,
> + unsigned parent_interrupt_specifier_cells,
> + unsigned num_regs, ...)
> +{
> + uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
> + child_interrupt_specifier_cells + parent_unit_addr_cells
> + + parent_interrupt_specifier_cells + 1)];
> + be32 *cells = &interrupt_map[0];
> + int i,j;
> + va_list ap;
> + uint64_t arg;
> +
> + va_start(ap, num_regs);
> + for (i = 0 ; i < num_regs; i++) {
> + /* Set the child unit address*/
> + for (j = 0 ; j < child_unit_addr_cells; j++) {
> + arg = va_arg(ap, uint32_t);
> + set_cell(&cells, 1 , arg);
> + }
> +
> + /* Set the child interrupt specifier*/
> + for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
> + arg = va_arg(ap, uint32_t);
> + set_cell(&cells, 1 , arg);
> + }
> +
> + /* Set the interrupt-parent*/
> + set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
> +
> + /* Set the parent unit address*/
> + for (j = 0 ; j < parent_unit_addr_cells; j++) {
> + arg = va_arg(ap, uint32_t);
> + set_cell(&cells, 1 , arg);
> + }
> +
> + /* Set the parent interrupt specifier*/
> + for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
> + arg = va_arg(ap, uint32_t);
> + set_cell(&cells, 1 , arg);
> + }
> + }
> + va_end(ap);
> +
> + return fdt_property(fdt, "interrupt-map", interrupt_map,
> + sizeof(interrupt_map));
> +}
> +
> static int make_root_properties(libxl__gc *gc,
> const libxl_version_info *vers,
> void *fdt)
> @@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
> return 0;
> }
>
> +static int make_vpci_node(libxl__gc *gc, void *fdt,
> + const struct arch_info *ainfo,
> + struct xc_dom_image *dom)
> +{
> + int res;
> + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
> + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
> + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
> +
> + res = fdt_begin_node(fdt, name);
> + if (res) return res;
> +
> + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
> + if (res) return res;
> +
> + res = fdt_property_string(fdt, "device_type", "pci");
> + if (res) return res;
> +
> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
> + if (res) return res;
> +
> + res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
> + if (res) return res;
> +
> + res = fdt_property_cell(fdt, "linux,pci-domain", 0);
> + if (res) return res;
> +
> + res = fdt_property_cell(fdt, "#address-cells", 3);
> + if (res) return res;
> +
> + res = fdt_property_cell(fdt, "#size-cells", 2);
> + if (res) return res;
> +
> + res = fdt_property_cell(fdt, "#interrupt-cells", 1);
> + if (res) return res;
> +
> + res = fdt_property_string(fdt, "status", "okay");
> + if (res) return res;
> +
> + res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
> + GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,
> + 3,
> + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
> + GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
> + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
> + GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
> + GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
> + GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
> + if (res) return res;
> +
> + res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);
it would make sense to separate out child_unit_addr_cells and
child_interrupt_specifier_cells here like we do below with
fdt_property_vpci_interrupt_map
> + if (res) return res;
> +
> + /*
> + * Legacy interrupt is forced and assigned to the guest.
> + * This will be removed once we have implementation for MSI support.
> + *
> + */
> + res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
> + 4,
> + 0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
> + 0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
> + 0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
> + 0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
The 4 interrupt allocated for this need to be defined in
xen/include/public/arch-arm.h as well. Also, why would we want to get
rid of the legacy interrupts completely? It would be possible to still
find device or software that rely on them.
> + if (res) return res;
> +
> + res = fdt_end_node(fdt);
> + if (res) return res;
> +
> + return 0;
> +}
> +
> static const struct arch_info *get_arch_info(libxl__gc *gc,
> const struct xc_dom_image *dom)
> {
[...]
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 7364a07362..4e19c62948 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
> #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
> #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
>
> +#define GUEST_PCI_ADDRESS_CELLS 3
> +#define GUEST_PCI_SIZE_CELLS 2
> +
> +/* PCI-PCIe memory space types */
> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
> +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000)
> +#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000)
> +
> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
> +#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000)
It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?
> +#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000)
> +#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800)
0xC0200800 looks like it could conflict with
GUEST_RAM0_BASE+GUEST_RAM0_SIZE?
> +/*
> + * This is hardcoded values for the real PCI physical addresses.
> + * This will be removed once we read the real PCI-PCIe physical
> + * addresses form the config space and map to the guest memory map
> + * when assigning the device to guest via VPCI.
> + *
> + */
> +#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000)
> +#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000)
> +#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000)
> +
> +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000)
> +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000)
How did you chose these sizes? GUEST_VPCI_MEM_SIZE and/or
GUEST_VPCI_PREFETCH_MEM_SIZE are supposed to potentially cover all the
PCI BARs, including potential future hotplug devices, right?
If so, maybe we need to increase GUEST_VPCI_MEM_SIZE to a couple of GB
and GUEST_VPCI_PREFETCH_MEM_SIZE to even more?
> +#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000)
> +
> /*
> * 16MB == 4096 pages reserved for guest to use as a region to map its
> * grant table in.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-23 23:38 ` Stefano Stabellini
@ 2020-07-24 7:03 ` Oleksandr Andrushchenko
2020-07-24 8:05 ` Julien Grall
2020-07-27 15:20 ` Rahul Singh
2020-07-24 8:44 ` Julien Grall
2020-07-27 13:27 ` Rahul Singh
2 siblings, 2 replies; 48+ messages in thread
From: Oleksandr Andrushchenko @ 2020-07-24 7:03 UTC (permalink / raw)
To: Stefano Stabellini, Rahul Singh
Cc: Julien Grall, andrew.cooper3, Bertrand.Marquis, jbeulich,
xen-devel, nd, Volodymyr Babchuk, roger.pau
On 7/24/20 2:38 AM, Stefano Stabellini wrote:
> + Jan, Andrew, Roger
>
> Please have a look at my comment on whether we should share the MMCFG
> code below, feel free to ignore the rest :-)
>
>
> On Thu, 23 Jul 2020, Rahul Singh wrote:
>> XEN during boot will read the PCI device tree node “reg” property
>> and will map the PCI config space to the XEN memory.
>>
>> XEN will read the “linux, pci-domain” property from the device tree
>> node and configure the host bridge segment number accordingly.
>>
>> As of now "pci-host-ecam-generic" compatible board is supported.
>>
>> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
> What is this Change-Id property?
Gerrit ;)
>
>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/Kconfig | 7 +
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/pci/Makefile | 4 +
>> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
>> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
>> xen/arch/arm/setup.c | 2 +
>> xen/include/asm-arm/device.h | 7 +-
>> xen/include/asm-arm/pci.h | 97 +++++++++++++-
>> 10 files changed, 654 insertions(+), 6 deletions(-)
>> create mode 100644 xen/arch/arm/pci/Makefile
>> create mode 100644 xen/arch/arm/pci/pci-access.c
>> create mode 100644 xen/arch/arm/pci/pci-host-common.c
>> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
>> create mode 100644 xen/arch/arm/pci/pci.c
>>
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 2777388265..ee13339ae9 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -31,6 +31,13 @@ menu "Architecture Features"
>>
>> source "arch/Kconfig"
>>
>> +config ARM_PCI
>> + bool "PCI Passthrough Support"
>> + depends on ARM_64
>> + ---help---
>> +
>> + PCI passthrough support for Xen on ARM64.
>> +
>> config ACPI
>> bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
>> depends on ARM_64
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 7e82b2178c..345cb83eed 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_ARM_PCI) += pci/
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> new file mode 100644
>> index 0000000000..358508b787
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -0,0 +1,4 @@
>> +obj-y += pci.o
>> +obj-y += pci-host-generic.o
>> +obj-y += pci-host-common.o
>> +obj-y += pci-access.o
>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>> new file mode 100644
>> index 0000000000..c53ef58336
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -0,0 +1,101 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
I think SPDX will fit better in any new code.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +#include <xen/rwlock.h>
>> +
>> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>> + unsigned int len)
>> +{
>> + int rc;
>> + uint32_t val = GENMASK(0, len * 8);
>> +
>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> + if ( unlikely(!bridge) )
>> + {
>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> + return val;
>> + }
>> +
>> + if ( unlikely(!bridge->ops->read) )
>> + return val;
>> +
>> + rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
>> + if ( rc )
>> + printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> +
>> + return val;
>> +}
>> +
>> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>> + unsigned int len, uint32_t val)
>> +{
>> + int rc;
>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> + if ( unlikely(!bridge) )
>> + {
>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> + return;
>> + }
>> +
>> + if ( unlikely(!bridge->ops->write) )
>> + return;
>> +
>> + rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
>> + if ( rc )
>> + printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> +}
>> +
>> +/*
>> + * Wrappers for all PCI configuration access functions.
>> + */
>> +
>> +#define PCI_OP_WRITE(size, type) \
>> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
>> +{ \
>> + pci_config_write(sbdf, reg, size / 8, val); \
>> +}
>> +
>> +#define PCI_OP_READ(size, type) \
>> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
>> +{ \
>> + return pci_config_read(sbdf, reg, size / 8); \
>> +}
>> +
>> +PCI_OP_READ(8, u8)
>> +PCI_OP_READ(16, u16)
>> +PCI_OP_READ(32, u32)
>> +PCI_OP_WRITE(8, u8)
>> +PCI_OP_WRITE(16, u16)
>> +PCI_OP_WRITE(32, u32)
> This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?
>
> MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not
> mistaken. Is there any value in sharing the code with x86? It is OK if
> we don't, but I would like to understand the reasoning.
>
>
>
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
>> new file mode 100644
>> index 0000000000..c5f98be698
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-common.c
>> @@ -0,0 +1,198 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +#include <xen/rwlock.h>
>> +#include <xen/vmap.h>
>> +
>> +/*
>> + * List for all the pci host bridges.
>> + */
>> +
>> +static LIST_HEAD(pci_host_bridges);
>> +
>> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
>> + struct pci_config_window *cfg)
>> +{
>> + const __be32 *cells;
>> + uint32_t len;
>> +
>> + cells = dt_get_property(dev, "bus-range", &len);
>> + /* bus-range should at least be 2 cells */
>> + if ( !cells || (len < (sizeof(*cells) * 2)) )
>> + return false;
>> +
>> + cfg->busn_start = dt_next_cell(1, &cells);
>> + cfg->busn_end = dt_next_cell(1, &cells);
>> +
>> + return true;
>> +}
>> +
>> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
>> +{
>> + return ioremap_nocache(start, len);
>> +}
>> +
>> +static void pci_ecam_free(struct pci_config_window *cfg)
>> +{
>> + if ( cfg->win )
>> + iounmap(cfg->win);
>> +
>> + xfree(cfg);
>> +}
The two functions above seem to deal with the same resources, e.g. cfg->win
and map/unmap. Would it make sense to align those, something like
s/pci_remap_cfgspace/pci_ecam_alloc and pci_ecam_alloc handles cfg->win?
Or anything which makes them look init/fini style?
>> +
>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops)
>> +{
>> + int err;
>> + struct pci_config_window *cfg;
>> + paddr_t addr, size;
>> +
>> + cfg = xzalloc(struct pci_config_window);
>> + if ( !cfg )
>> + return NULL;
>> +
>> + err = dt_pci_parse_bus_range(dev, cfg);
>> + if ( !err ) {
>> + cfg->busn_start = 0;
>> + cfg->busn_end = 0xff;
>> + printk(XENLOG_ERR "No bus range found for pci controller\n");
>> + } else {
>> + if ( cfg->busn_end > cfg->busn_start + 0xff )
>> + cfg->busn_end = cfg->busn_start + 0xff;
So, if bus start is, for example, 0x10 then we'll end up with bus end at (0x10 + 0xff) > 0xff
which doesn't seem to be what we want
>> + }
>> +
>> + /* Parse our PCI ecam register address*/
>> + err = dt_device_get_address(dev, 0, &addr, &size);
>> + if ( err )
>> + goto err_exit;
> Shouldn't we handle the possibility of multiple addresses? Is it
> possible to have more than one range for an ECAM compliant host bridge?
>
>
>> + cfg->phys_addr = addr;
>> + cfg->size = size;
>> + cfg->ops = ops;
>> +
>> + /*
>> + * On 64-bit systems, we do a single ioremap for the whole config space
>> + * since we have enough virtual address range available. On 32-bit, we
>> + * ioremap the config space for each bus individually.
>> + *
>> + * As of now only 64-bit is supported 32-bit is not supported.
>> + */
>> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
I am fine with "win", but can we think of something that tells us that
"win" is actually ECAM base address, so one doesn't need to map "win" to "ECAM"
while reading?
>> + if ( !cfg->win )
>> + goto err_exit_remap;
>> +
>> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>> + cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
>> +
>> + if ( ops->init ) {
>> + err = ops->init(cfg);
>> + if (err)
>> + goto err_exit;
>> + }
>> +
>> + return cfg;
>> +
>> +err_exit_remap:
>> + printk(XENLOG_ERR "ECAM ioremap failed\n");
>> +err_exit:
>> + pci_ecam_free(cfg);
>> + return NULL;
>> +}
>> +
>> +static struct pci_host_bridge * pci_alloc_host_bridge(void)
>> +{
>> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
>> +
>> + if ( !bridge )
>> + return NULL;
>> +
>> + INIT_LIST_HEAD(&bridge->node);
>> + return bridge;
>> +}
>> +
>> +int pci_host_common_probe(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops)
>> +{
>> + struct pci_host_bridge *bridge;
>> + struct pci_config_window *cfg;
>> + u32 segment;
>> +
>> + bridge = pci_alloc_host_bridge();
>> + if ( !bridge )
>> + return -ENOMEM;
>> +
>> + /* Parse and map our Configuration Space windows */
Do you expect multiple windows here as the comment says?
>> + cfg = gen_pci_init(dev, ops);
>> + if ( !cfg )
>> + return -ENOMEM;
> In case of errors the allocated bridge is not freed.
>
>
>> + bridge->dt_node = dev;
>> + bridge->sysdata = cfg;
>> + bridge->ops = &ops->pci_ops;
Can we have some sort of dummy ops so we don't have to check for ops != NULL every time
we read/write config? Is it really possible that we have ops set to NULL after we have
the development finished?
>> +
>> + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
>> + {
>> + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
>> + return -ENODEV;
>> + }
>> +
>> + bridge->segment = (u16)segment;
> My understanding is that a Linux pci-domain doesn't correspond exactly
> to a PCI segment. See for instance:
>
> https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
>
> Do we need to care about the difference? If we mean pci-domain here,
> should we just call them as such instead of calling them "segments" in
> Xen (if they are not segments)?
>
>
>> + list_add_tail(&bridge->node, &pci_host_bridges);
> It looks like &pci_host_bridges should be an ordered list, ordered by
> segment number?
Why? Do you expect bridge access in some specific order so ordered
list will make it faster?
>
>
>> + return 0;
>> +}
>> +
>> +/*
>> + * This function will lookup an hostbridge based on the segment and bus
>> + * number.
>> + */
>> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
>> +{
>> + struct pci_host_bridge *bridge;
>> + bool found = false;
>> +
>> + list_for_each_entry( bridge, &pci_host_bridges, node )
>> + {
>> + if ( bridge->segment != segment )
>> + continue;
>> +
>> + found = true;
>> + break;
>> + }
>> +
>> + return (found) ? bridge : NULL;
>> +}
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
>> new file mode 100644
>> index 0000000000..cd67b3dec6
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-generic.c
>> @@ -0,0 +1,131 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/device.h>
>> +#include <asm/io.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +
>> +/*
>> + * Function to get the config space base.
>> + */
>> +static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where)
> I think the function is misnamed because reading the code below it looks
> like it is not just returning the base config space address but also the
> specific address we need to read/write (adding the device offset,
> "where", and everything).
>
> Maybe pci_config_get_address or something like that?
>
>
>> +{
>> + struct pci_config_window *cfg = bridge->sysdata;
I am a bit confused of the naming ;)
We already have 2 maps: win -> ECAM base and now sysdata -> cfg.
Can we please have that aligned somehow so it is easier to follow?
>> + unsigned int devfn_shift = cfg->ops->bus_shift - 8;
We are not checking cfg->ops for NULL, so probably we do not want bridges
with NULL ops as well?
>> +
>> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
>> +
>> + unsigned int busn = sbdf_t.bus;
>> + void __iomem *base;
>> +
>> + if ( busn < cfg->busn_start || busn > cfg->busn_end )
>> + return NULL;
>> +
>> + base = cfg->win + (busn << cfg->ops->bus_shift);
>> +
>> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
>> +}
>> +
>> +int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>> + int where, int size, u32 val)
>> +{
>> + void __iomem *addr;
>> +
>> + addr = pci_config_base(bridge, sbdf, where);
>> + if ( !addr )
>> + return -ENODEV;
>> +
>> + if ( size == 1 )
>> + writeb(val, addr);
>> + else if ( size == 2 )
>> + writew(val, addr);
>> + else
>> + writel(val, addr);
> please use a switch
>
>
>> + return 0;
>> +}
>> +
>> +int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
>> + int where, int size, u32 *val)
>> +{
>> + void __iomem *addr;
>> +
>> + addr = pci_config_base(bridge, sbdf, where);
>> + if ( !addr ) {
>> + *val = ~0;
>> + return -ENODEV;
>> + }
>> +
>> + if ( size == 1 )
>> + *val = readb(addr);
>> + else if ( size == 2 )
>> + *val = readw(addr);
>> + else
>> + *val = readl(addr);
> please use a switch
>
>
>> + return 0;
>> +}
>> +
>> +/* ECAM ops */
>> +struct pci_ecam_ops pci_generic_ecam_ops = {
>> + .bus_shift = 20,
>> + .pci_ops = {
>> + .read = pci_ecam_config_read,
>> + .write = pci_ecam_config_write,
>> + }
>> +};
>> +
>> +static const struct dt_device_match gen_pci_dt_match[] = {
>> + { .compatible = "pci-host-ecam-generic",
>> + .data = &pci_generic_ecam_ops },
> spurious blank line
>
>
>> + { },
>> +};
>> +
>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>> +{
>> + const struct dt_device_match *of_id;
>> + struct pci_ecam_ops *ops;
>> +
>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>> + ops = (struct pci_ecam_ops *) of_id->data;
>> +
>> + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>> + dt_node_full_name(dev), of_id->compatible);
>> +
>> + return pci_host_common_probe(dev, ops);
>> +}
>> +
>> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
>> +.dt_match = gen_pci_dt_match,
>> +.init = gen_pci_dt_init,
>> +DT_DEVICE_END
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>> new file mode 100644
>> index 0000000000..f8cbb99591
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci.c
>> @@ -0,0 +1,112 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/acpi.h>
>> +#include <xen/device_tree.h>
>> +#include <xen/errno.h>
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <xen/param.h>
>> +
>> +static int __init dt_pci_init(void)
>> +{
>> + struct dt_device_node *np;
>> + int rc;
>> +
>> + dt_for_each_device_node(dt_host, np)
>> + {
>> + rc = device_init(np, DEVICE_PCI, NULL);
>> + if( !rc )
>> + continue;
>> + /*
>> + * Ignore the following error codes:
>> + * - EBADF: Indicate the current is not an pci
>> + * - ENODEV: The pci device is not present or cannot be used by
>> + * Xen.
>> + */
>> + else if ( rc != -EBADF && rc != -ENODEV )
>> + {
>> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
>> + return rc;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +#ifdef CONFIG_ACPI
>> +static void __init acpi_pci_init(void)
>> +{
>> + printk(XENLOG_ERR "ACPI pci init not supported \n");
>> + return;
>> +}
>> +#else
>> +static inline void __init acpi_pci_init(void) { }
>> +#endif
>> +
>> +static bool __initdata param_pci_enable;
>> +static int __init parse_pci_param(const char *arg)
>> +{
>> + if ( !arg )
>> + {
>> + param_pci_enable = false;
>> + return 0;
>> + }
>> +
>> + switch ( parse_bool(arg, NULL) )
>> + {
>> + case 0:
>> + param_pci_enable = false;
>> + return 0;
>> + case 1:
>> + param_pci_enable = true;
>> + return 0;
>> + }
>> +
>> + return -EINVAL;
>> +}
>> +custom_param("pci", parse_pci_param);
> When adding new command line parameters please also add its
> documentation (docs/misc/xen-command-line.pandoc) in the same patch,
> unless this is meant to be just transient and we'll get removed before
> the final commit of the series.
>
>
>> +void __init pci_init(void)
>> +{
>> + /*
>> + * Enable PCI when has been enabled explicitly (pci=on)
>> + */
>> + if ( !param_pci_enable)
>> + goto disable;
>> +
>> + if ( acpi_disabled )
>> + dt_pci_init();
>> + else
>> + acpi_pci_init();
>> +
>> +#ifdef CONFIG_HAS_PCI
>> + pci_segments_init();
>> +#endif
>> +
>> +disable:
>> + return;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>> index 7968cee47d..2d7f1db44f 100644
>> --- a/xen/arch/arm/setup.c
>> +++ b/xen/arch/arm/setup.c
>> @@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>>
>> setup_virt_paging();
>>
>> + pci_init();
> pci_init should probably be an initcall
>
>
>> do_initcalls();
>>
>> /*
>> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
>> index ee7cff2d44..28f8049cfd 100644
>> --- a/xen/include/asm-arm/device.h
>> +++ b/xen/include/asm-arm/device.h
>> @@ -4,6 +4,7 @@
>> enum device_type
>> {
>> DEV_DT,
>> + DEV_PCI,
>> };
>>
>> struct dev_archdata {
>> @@ -25,15 +26,15 @@ typedef struct device device_t;
>>
>> #include <xen/device_tree.h>
>>
>> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
>> -#define dev_is_pci(dev) ((void)(dev), 0)
>> -#define dev_is_dt(dev) ((dev->type == DEV_DT)
Didn't we have a patch for that recently or talked about?
>> +#define dev_is_pci(dev) (dev->type == DEV_PCI)
>> +#define dev_is_dt(dev) (dev->type == DEV_DT)
>>
>> enum device_class
>> {
>> DEVICE_SERIAL,
>> DEVICE_IOMMU,
>> DEVICE_GIC,
>> + DEVICE_PCI,
>> /* Use for error */
>> DEVICE_UNKNOWN,
>> };
>> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>> index de13359f65..94fd00360a 100644
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -1,7 +1,98 @@
>> -#ifndef __X86_PCI_H__
>> -#define __X86_PCI_H__
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>>
>> +#ifndef __ARM_PCI_H__
>> +#define __ARM_PCI_H__
>> +
>> +#include <xen/pci.h>
>> +#include <xen/device_tree.h>
>> +#include <asm/device.h>
>> +
>> +#ifdef CONFIG_ARM_PCI
>> +
>> +/* Arch pci dev struct */
>> struct arch_pci_dev {
>> + struct device dev;
>> +};
> Are you actually using struct device in struct arch_pci_dev?
> struct device is already part of struct dt_device_node and a pointer to
> it is stored in bridge->dt_node.
>
>
>> +#define PRI_pci "%04x:%02x:%02x.%u"
>> +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
>> +
>> +/*
>> + * struct to hold the mappings of a config space window. This
>> + * is expected to be used as sysdata for PCI controllers that
>> + * use ECAM.
>> + */
>> +struct pci_config_window {
>> + paddr_t phys_addr;
>> + paddr_t size;
>> + uint8_t busn_start;
>> + uint8_t busn_end;
>> + struct pci_ecam_ops *ops;
>> + void __iomem *win;
>> +};
>> +
>> +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
>> +struct pci_host_bridge;
>> +
>> +struct pci_ops {
>> + int (*read)(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where, int size, u32 *val);
>> + int (*write)(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where, int size, u32 val);
> I'd prefer if we could use explicitly-sized integers for "where" and
> "size" too. Also, should they be unsigned rather than signed?
>
> Can we use pci_sbdf_t for the sbdf parameter?
>
>
>> +};
>> +
>> +/*
>> + * struct to hold pci ops and bus shift of the config window
>> + * for a PCI controller.
>> + */
>> +struct pci_ecam_ops {
>> + unsigned int bus_shift;
>> + struct pci_ops pci_ops;
>> + int (*init)(struct pci_config_window *);
>> +};
> Although I realize that we are only targeting ECAM now, and the
> implementation is based on ECAM, the interface doesn't seem to have
> anything ECAM-specific in it. I would rename pci_ecam_ops to something
> else, maybe simply "pci_ops".
Yes, please, bear in mind that we are about to work with this code on
non-ECAM HW from the very beginning, so this is something that we would
like to see from the ground up
>
>
>> +/*
>> + * struct to hold pci host bridge information
>> + * for a PCI controller.
>> + */
>> +struct pci_host_bridge {
>> + struct dt_device_node *dt_node; /* Pointer to the associated DT node */
>> + struct list_head node; /* Node in list of host bridges */
>> + uint16_t segment; /* Segment number */
>> + void *sysdata; /* Pointer to the config space window*/
>> + const struct pci_ops *ops;
>> };
>>
>> -#endif /* __X86_PCI_H__ */
>> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
>> +
>> +int pci_host_common_probe(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops);
>> +
>> +void pci_init(void);
>> +
>> +#else /*!CONFIG_ARM_PCI*/
>> +struct arch_pci_dev { };
>> +static inline void pci_init(void) { }
>> +#endif /*!CONFIG_ARM_PCI*/
>> +#endif /* __ARM_PCI_H__ */
>> --
>> 2.17.1
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
2020-07-23 20:44 ` Stefano Stabellini
@ 2020-07-24 7:14 ` Oleksandr Andrushchenko
2020-07-24 8:19 ` Julien Grall
2020-07-27 16:10 ` Rahul Singh
2020-07-24 14:49 ` Roger Pau Monné
2020-07-27 8:40 ` Rahul Singh
2 siblings, 2 replies; 48+ messages in thread
From: Oleksandr Andrushchenko @ 2020-07-24 7:14 UTC (permalink / raw)
To: Stefano Stabellini, Rahul Singh
Cc: Julien Grall, andrew.cooper3, Bertrand.Marquis, jbeulich,
xen-devel, nd, Volodymyr Babchuk, roger.pau
On 7/23/20 11:44 PM, Stefano Stabellini wrote:
> On Thu, 23 Jul 2020, Rahul Singh wrote:
>> Hardware domain is in charge of doing the PCI enumeration and will
>> discover the PCI devices and then will communicate to XEN via hyper
>> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
>>
>> Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
> Same question about Change-Id
>
>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
>> 1 file changed, 39 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
>> index e91355fe22..274720f98a 100644
>> --- a/xen/arch/arm/physdev.c
>> +++ b/xen/arch/arm/physdev.c
>> @@ -9,12 +9,48 @@
>> #include <xen/errno.h>
>> #include <xen/sched.h>
>> #include <asm/hypercall.h>
>> -
>> +#include <xen/guest_access.h>
>> +#include <xsm/xsm.h>
>>
>> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>> {
>> - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>> - return -ENOSYS;
>> + int ret = 0;
>> +
>> + switch ( cmd )
>> + {
>> +#ifdef CONFIG_HAS_PCI
In the cover letter you were saying "we are not enabling the HAS_PCI and HAS_VPCI flags for ARM".
Is this still valid?
>> + case PHYSDEVOP_pci_device_add:
>> + {
>> + struct physdev_pci_device_add add;
>> + struct pci_dev_info pdev_info;
>> + nodeid_t node = NUMA_NO_NODE;
>> +
>> + ret = -EFAULT;
>> + if ( copy_from_guest(&add, arg, 1) != 0 )
>> + break;
>> +
>> + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
>> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
>> + {
>> + pdev_info.is_virtfn = 1;
>> + pdev_info.physfn.bus = add.physfn.bus;
>> + pdev_info.physfn.devfn = add.physfn.devfn;
>> + }
>> + else
>> + pdev_info.is_virtfn = 0;
>> +
>> + ret = pci_add_device(add.seg, add.bus, add.devfn,
>> + &pdev_info, node);
>> +
>> + break;
>> + }
>> +#endif
>> + default:
>> + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>> + ret = -ENOSYS;
>> + }
> I think we should make the implementation common between arm and x86 by
> creating xen/common/physdev.c:do_physdev_op as a shared entry point for
> PHYSDEVOP hypercalls implementations. See for instance:
>
> xen/common/sysctl.c:do_sysctl
>
> and
>
> xen/arch/arm/sysctl.c:arch_do_sysctl
> xen/arch/x86/sysctl.c:arch_do_sysctl
>
>
> Jan, Andrew, Roger, any opinions?
>
>
I think we can also have a look at [1] by Julien. That implementation,
IMO, had some thoughts on making Arm/x86 code common where possible
[1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
2020-07-23 23:39 ` Stefano Stabellini
@ 2020-07-24 7:55 ` Oleksandr Andrushchenko
2020-07-24 9:11 ` Julien Grall
2020-07-27 13:40 ` Rahul Singh
2 siblings, 0 replies; 48+ messages in thread
From: Oleksandr Andrushchenko @ 2020-07-24 7:55 UTC (permalink / raw)
To: Stefano Stabellini, Rahul Singh
Cc: Julien Grall, Wei Liu, Ian Jackson, Bertrand.Marquis,
Anthony PERARD, xen-devel, nd, Volodymyr Babchuk
On 7/24/20 2:39 AM, Stefano Stabellini wrote:
> On Thu, 23 Jul 2020, Rahul Singh wrote:
>> libxl will create an emulated PCI device tree node in the
>> device tree to enable the guest OS to discover the virtual
>> PCI during guest boot.
>>
>> We introduced the new config option [vpci="ecam"] for guests.
>> When this config option is enabled in a guest configuration,
>> a PCI device tree node will be created in the guest device tree.
>>
>> A new area has been reserved in the arm guest physical map at
>> which the VPCI bus is declared in the device tree (reg and ranges
>> parameters of the node).
>>
>> Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd
> Same question
>
>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++
>> tools/libxl/libxl_types.idl | 6 +
>> tools/xl/xl_parse.c | 7 ++
>> xen/include/public/arch-arm.h | 28 +++++
>> 4 files changed, 241 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
>> index 34f8a29056..84568e9dc9 100644
>> --- a/tools/libxl/libxl_arm.c
>> +++ b/tools/libxl/libxl_arm.c
>> @@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>> return fdt_property(fdt, "reg", regs, sizeof(regs));
>> }
>>
>> +static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
>> + unsigned num_cells, ...)
>> +{
>> + uint32_t bus_range[num_cells];
>> + be32 *cells = &bus_range[0];
>> + int i;
>> + va_list ap;
>> + uint32_t arg;
>> +
>> + va_start(ap, num_cells);
>> + for (i = 0 ; i < num_cells; i++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1, arg);
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
>> +}
>> +
>> +static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
>> + unsigned num_cells, ...)
>> +{
>> + uint32_t interrupt_map_mask[num_cells];
>> + be32 *cells = &interrupt_map_mask[0];
>> + int i;
>> + va_list ap;
>> + uint32_t arg;
>> +
>> + va_start(ap, num_cells);
>> + for (i = 0 ; i < num_cells; i++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1, arg);
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
>> + sizeof(interrupt_map_mask));
>> +}
>> +
>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
>> + unsigned vpci_addr_cells,
>> + unsigned cpu_addr_cells,
>> + unsigned vpci_size_cells,
>> + unsigned num_regs, ...)
>> +{
>> + uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
>> + be32 *cells = ®s[0];
>> + int i;
>> + va_list ap;
>> + uint64_t arg;
>> +
>> + va_start(ap, num_regs);
>> + for (i = 0 ; i < num_regs; i++) {
>> + /* Set the memory bit field */
>> + arg = va_arg(ap, uint64_t);
>> + set_cell(&cells, 1, arg);
>> +
>> + /* Set the vpci bus address */
>> + arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
>> + set_cell(&cells, 2 , arg);
>> +
>> + /* Set the cpu bus address where vpci address is mapped */
>> + arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
>> + set_cell(&cells, cpu_addr_cells, arg);
>> +
>> + /* Set the vpci size requested */
>> + arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
>> + set_cell(&cells, vpci_size_cells,arg);
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "ranges", regs, sizeof(regs));
>> +}
>> +
>> +static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
>> + unsigned child_unit_addr_cells,
>> + unsigned child_interrupt_specifier_cells,
>> + unsigned parent_unit_addr_cells,
>> + unsigned parent_interrupt_specifier_cells,
>> + unsigned num_regs, ...)
>> +{
>> + uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
>> + child_interrupt_specifier_cells + parent_unit_addr_cells
>> + + parent_interrupt_specifier_cells + 1)];
>> + be32 *cells = &interrupt_map[0];
>> + int i,j;
>> + va_list ap;
>> + uint64_t arg;
>> +
>> + va_start(ap, num_regs);
>> + for (i = 0 ; i < num_regs; i++) {
>> + /* Set the child unit address*/
>> + for (j = 0 ; j < child_unit_addr_cells; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> +
>> + /* Set the child interrupt specifier*/
>> + for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> +
>> + /* Set the interrupt-parent*/
>> + set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
>> +
>> + /* Set the parent unit address*/
>> + for (j = 0 ; j < parent_unit_addr_cells; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> +
>> + /* Set the parent interrupt specifier*/
>> + for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "interrupt-map", interrupt_map,
>> + sizeof(interrupt_map));
>> +}
>> +
>> static int make_root_properties(libxl__gc *gc,
>> const libxl_version_info *vers,
>> void *fdt)
>> @@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>> return 0;
>> }
>>
>> +static int make_vpci_node(libxl__gc *gc, void *fdt,
>> + const struct arch_info *ainfo,
>> + struct xc_dom_image *dom)
>> +{
>> + int res;
>> + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
>> + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
>> + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
>> +
>> + res = fdt_begin_node(fdt, name);
>> + if (res) return res;
>> +
>> + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
>> + if (res) return res;
>> +
>> + res = fdt_property_string(fdt, "device_type", "pci");
>> + if (res) return res;
>> +
>> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>> + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
>> + if (res) return res;
>> +
>> + res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "linux,pci-domain", 0);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "#address-cells", 3);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "#size-cells", 2);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "#interrupt-cells", 1);
>> + if (res) return res;
>> +
>> + res = fdt_property_string(fdt, "status", "okay");
>> + if (res) return res;
>> +
>> + res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
>> + GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,
>> + 3,
>> + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
>> + GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
>> + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
>> + GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
>> + GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
>> + GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
>> + if (res) return res;
>> +
>> + res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);
> it would make sense to separate out child_unit_addr_cells and
> child_interrupt_specifier_cells here like we do below with
> fdt_property_vpci_interrupt_map
>
>
>> + if (res) return res;
>> +
>> + /*
>> + * Legacy interrupt is forced and assigned to the guest.
>> + * This will be removed once we have implementation for MSI support.
>> + *
>> + */
>> + res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
>> + 4,
>> + 0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
> The 4 interrupt allocated for this need to be defined in
> xen/include/public/arch-arm.h as well. Also, why would we want to get
> rid of the legacy interrupts completely? It would be possible to still
> find device or software that rely on them.
I think this is more about shared interrupts support complexity
>
>
>> + if (res) return res;
>> +
>> + res = fdt_end_node(fdt);
>> + if (res) return res;
>> +
>> + return 0;
>> +}
>> +
>> static const struct arch_info *get_arch_info(libxl__gc *gc,
>> const struct xc_dom_image *dom)
>> {
> [...]
>
>
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index 7364a07362..4e19c62948 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
>> #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
>>
>> +#define GUEST_PCI_ADDRESS_CELLS 3
>> +#define GUEST_PCI_SIZE_CELLS 2
>> +
>> +/* PCI-PCIe memory space types */
>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
>> +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000)
>> +#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000)
>> +
>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
>> +#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000)
> It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?
>
>
>> +#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000)
>> +#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800)
> 0xC0200800 looks like it could conflict with
> GUEST_RAM0_BASE+GUEST_RAM0_SIZE?
>
>
>> +/*
>> + * This is hardcoded values for the real PCI physical addresses.
>> + * This will be removed once we read the real PCI-PCIe physical
>> + * addresses form the config space and map to the guest memory map
>> + * when assigning the device to guest via VPCI.
>> + *
>> + */
>> +#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000)
>> +#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000)
>> +#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000)
>> +
>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000)
>> +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000)
> How did you chose these sizes? GUEST_VPCI_MEM_SIZE and/or
> GUEST_VPCI_PREFETCH_MEM_SIZE are supposed to potentially cover all the
> PCI BARs, including potential future hotplug devices, right?
>
> If so, maybe we need to increase GUEST_VPCI_MEM_SIZE to a couple of GB
> and GUEST_VPCI_PREFETCH_MEM_SIZE to even more?
>
>
>
>
>> +#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000)
>> +
>> /*
>> * 16MB == 4096 pages reserved for guest to use as a region to map its
>> * grant table in.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 7:03 ` Oleksandr Andrushchenko
@ 2020-07-24 8:05 ` Julien Grall
2020-07-24 17:47 ` Stefano Stabellini
2020-07-27 15:27 ` Rahul Singh
2020-07-27 15:20 ` Rahul Singh
1 sibling, 2 replies; 48+ messages in thread
From: Julien Grall @ 2020-07-24 8:05 UTC (permalink / raw)
To: Oleksandr Andrushchenko, Stefano Stabellini, Rahul Singh
Cc: andrew.cooper3, Bertrand.Marquis, jbeulich, xen-devel, nd,
Volodymyr Babchuk, roger.pau
Hi,
On 24/07/2020 08:03, Oleksandr Andrushchenko wrote:
>>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>>> new file mode 100644
>>> index 0000000000..c53ef58336
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/pci-access.c
>>> @@ -0,0 +1,101 @@
>>> +/*
>>> + * Copyright (C) 2020 Arm Ltd.
> I think SPDX will fit better in any new code.
While I would love to use SPDX in Xen, there was some push back in the
past to use it. So the new code should use the full-blown copyright
until there is an agreement to use it.
>>
>>> + list_add_tail(&bridge->node, &pci_host_bridges);
>> It looks like &pci_host_bridges should be an ordered list, ordered by
>> segment number?
>
> Why? Do you expect bridge access in some specific order so ordered
>
> list will make it faster?
Access to the configure space will be pretty random. So I don't think
ordering the list will make anything better.
However, looking up for the bridge for every config spec access is
pretty slow. When I was working on the PCI passthrough, I wanted to look
whether it would be possible to have a pointer to the PCI host bridge
passed in argument to the helpers (rather than the segment).
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
2020-07-24 7:14 ` Oleksandr Andrushchenko
@ 2020-07-24 8:19 ` Julien Grall
2020-07-27 16:10 ` Rahul Singh
1 sibling, 0 replies; 48+ messages in thread
From: Julien Grall @ 2020-07-24 8:19 UTC (permalink / raw)
To: Oleksandr Andrushchenko, Stefano Stabellini, Rahul Singh
Cc: andrew.cooper3, Bertrand.Marquis, jbeulich, xen-devel, nd,
Volodymyr Babchuk, roger.pau
Hi,
On 24/07/2020 08:14, Oleksandr Andrushchenko wrote:
>
> On 7/23/20 11:44 PM, Stefano Stabellini wrote:
>> On Thu, 23 Jul 2020, Rahul Singh wrote:
>>> Hardware domain is in charge of doing the PCI enumeration and will
>>> discover the PCI devices and then will communicate to XEN via hyper
>>> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
>>>
>>> Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
>> Same question about Change-Id
>>
>>
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> ---
>>> xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
>>> 1 file changed, 39 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
>>> index e91355fe22..274720f98a 100644
>>> --- a/xen/arch/arm/physdev.c
>>> +++ b/xen/arch/arm/physdev.c
>>> @@ -9,12 +9,48 @@
>>> #include <xen/errno.h>
>>> #include <xen/sched.h>
>>> #include <asm/hypercall.h>
>>> -
>>> +#include <xen/guest_access.h>
>>> +#include <xsm/xsm.h>
>>>
>>> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>>> {
>>> - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>>> - return -ENOSYS;
>>> + int ret = 0;
>>> +
>>> + switch ( cmd )
>>> + {
>>> +#ifdef CONFIG_HAS_PCI
>
> In the cover letter you were saying "we are not enabling the HAS_PCI and HAS_VPCI flags for ARM".
>
> Is this still valid?
>
>>> + case PHYSDEVOP_pci_device_add:
>>> + {
>>> + struct physdev_pci_device_add add;
>>> + struct pci_dev_info pdev_info;
>>> + nodeid_t node = NUMA_NO_NODE;
>>> +
>>> + ret = -EFAULT;
>>> + if ( copy_from_guest(&add, arg, 1) != 0 )
>>> + break;
>>> +
>>> + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
>>> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
>>> + {
>>> + pdev_info.is_virtfn = 1;
>>> + pdev_info.physfn.bus = add.physfn.bus;
>>> + pdev_info.physfn.devfn = add.physfn.devfn;
>>> + }
>>> + else
>>> + pdev_info.is_virtfn = 0;
>>> +
>>> + ret = pci_add_device(add.seg, add.bus, add.devfn,
>>> + &pdev_info, node);
>>> +
>>> + break;
>>> + }
>>> +#endif
>>> + default:
>>> + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>>> + ret = -ENOSYS;
>>> + }
>> I think we should make the implementation common between arm and x86 by
>> creating xen/common/physdev.c:do_physdev_op as a shared entry point for
>> PHYSDEVOP hypercalls implementations. See for instance:
>>
>> xen/common/sysctl.c:do_sysctl
>>
>> and
>>
>> xen/arch/arm/sysctl.c:arch_do_sysctl
>> xen/arch/x86/sysctl.c:arch_do_sysctl
>>
>>
>> Jan, Andrew, Roger, any opinions?
>>
>>
> I think we can also have a look at [1] by Julien. That implementation,
>
> IMO, had some thoughts on making Arm/x86 code common where possible
There are some ideas on how I would like to see the split, although they
need some cleanup. :)
In particular, I was expecting some preparatory work to get the existing
PCI code non-x86 specific.
Regarding the hypercall Stefano mentionned, I think it should be
possible to make it common if we abstract the NUMA node lookup part.
Cheers,
>
>
> [1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci
>
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-23 15:40 ` [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM Rahul Singh
2020-07-23 23:38 ` Stefano Stabellini
@ 2020-07-24 8:23 ` Julien Grall
2020-07-27 15:29 ` Rahul Singh
2020-07-24 14:44 ` Roger Pau Monné
2 siblings, 1 reply; 48+ messages in thread
From: Julien Grall @ 2020-07-24 8:23 UTC (permalink / raw)
To: Rahul Singh, xen-devel
Cc: nd, Volodymyr Babchuk, Stefano Stabellini, Bertrand.Marquis
Hi Rahul,
On 23/07/2020 16:40, Rahul Singh wrote:
> XEN during boot will read the PCI device tree node “reg” property
> and will map the PCI config space to the XEN memory.
>
> XEN will read the “linux, pci-domain” property from the device tree
> node and configure the host bridge segment number accordingly.
>
> As of now "pci-host-ecam-generic" compatible board is supported.
>
> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/Kconfig | 7 +
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/pci/Makefile | 4 +
> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
> xen/arch/arm/setup.c | 2 +
> xen/include/asm-arm/device.h | 7 +-
> xen/include/asm-arm/pci.h | 97 +++++++++++++-
> 10 files changed, 654 insertions(+), 6 deletions(-)
As a general comment, I would suggest to split the patch in smaller
chunk. This would help the review and also allow to provide more
explanation on what is done.
For instance, I think it is possible to a split looking like:
- Add framework to access an hostbridge
- Add support for ECAM
- Add code to initialize the PCI subsystem
There is also some small fixes in this code that probably can move in
there own patches.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-23 23:38 ` Stefano Stabellini
2020-07-24 7:03 ` Oleksandr Andrushchenko
@ 2020-07-24 8:44 ` Julien Grall
2020-07-24 17:41 ` Stefano Stabellini
2020-07-27 13:27 ` Rahul Singh
2 siblings, 1 reply; 48+ messages in thread
From: Julien Grall @ 2020-07-24 8:44 UTC (permalink / raw)
To: Stefano Stabellini, Rahul Singh
Cc: andrew.cooper3, Bertrand.Marquis, jbeulich, xen-devel, nd,
Volodymyr Babchuk, roger.pau
Hi,
On 24/07/2020 00:38, Stefano Stabellini wrote:
>> + bridge->dt_node = dev;
>> + bridge->sysdata = cfg;
>> + bridge->ops = &ops->pci_ops;
>> +
>> + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
>> + {
>> + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
>> + return -ENODEV;
>> + }
>> +
>> + bridge->segment = (u16)segment;
>
> My understanding is that a Linux pci-domain doesn't correspond exactly
> to a PCI segment. See for instance:
>
> https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
>
> Do we need to care about the difference? If we mean pci-domain here,
> should we just call them as such instead of calling them "segments" in
> Xen (if they are not segments)?
So we definitely need a segment number in hand because this is what the
admin will use to assign a PCI device to a guest.
The segment number is just a value defined by the software. So as long
as Linux and Xen agrees with the number, then we should be ok.
Looking at the code, I think Linux is using 'segment' as 'domain'. So
they should be interchangeably. I am not sure what other OS does though.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
2020-07-23 23:39 ` Stefano Stabellini
2020-07-24 7:55 ` Oleksandr Andrushchenko
@ 2020-07-24 9:11 ` Julien Grall
2020-07-27 13:40 ` Rahul Singh
2 siblings, 0 replies; 48+ messages in thread
From: Julien Grall @ 2020-07-24 9:11 UTC (permalink / raw)
To: Stefano Stabellini, Rahul Singh
Cc: Wei Liu, Ian Jackson, Bertrand.Marquis, Anthony PERARD,
xen-devel, nd, Volodymyr Babchuk
Hi,
On 24/07/2020 00:39, Stefano Stabellini wrote:
> On Thu, 23 Jul 2020, Rahul Singh wrote:
>> + if (res) return res;
>> +
>> + /*
>> + * Legacy interrupt is forced and assigned to the guest.
>> + * This will be removed once we have implementation for MSI support.
>> + *
>> + */
>> + res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
>> + 4,
>> + 0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
>
> The 4 interrupt allocated for this need to be defined in
> xen/include/public/arch-arm.h as well. Also, why would we want to get
> rid of the legacy interrupts completely?
With legacy interrupts, there are a few cases to take into account:
1) Two PCI devices from the same hostbridge are assigned to
different cases. As SPIs (used for legacy interrupts) can only be routed
to one guest, we would need to now be able to share them. This raises
the question when to EOI the physical interrupts. AFAICT, Linux has some
code to deal with it using a timer if it takes too long.
2) Two PCI devices from two distinct hostbridge are assigned to the
same virtual hostbridge. Legacy interrupts would need to be virtual and
we would possibly need to merge multiple physical interrupts into one
virtual.
3) A mix of virtual and physical PCI devices are assigned to the
same host bridges. Same as 2) legacy interrupts would need to be virtual.
Given the complexity of handling interrupts legacy and the fact they
will largely be slower compare to MSIs, I would rather focus on MSIs at
first.
We can then decide to implement legacy interrupts if there is a real need.
> It would be possible to still
> find device or software that rely on them.
PCI devices without MSI support is getting extremely rare. There are
actually Arm HW that doesn't support legacy interrupts at all (such as
Thunder-X). I can't tell for the software.
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-23 15:40 ` [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM Rahul Singh
2020-07-23 23:38 ` Stefano Stabellini
2020-07-24 8:23 ` Julien Grall
@ 2020-07-24 14:44 ` Roger Pau Monné
2020-07-24 15:15 ` Julien Grall
2020-07-28 8:06 ` Rahul Singh
2 siblings, 2 replies; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-24 14:44 UTC (permalink / raw)
To: Rahul Singh
Cc: Stefano Stabellini, Julien Grall, Bertrand.Marquis, xen-devel,
nd, Volodymyr Babchuk
On Thu, Jul 23, 2020 at 04:40:21PM +0100, Rahul Singh wrote:
> XEN during boot will read the PCI device tree node “reg” property
> and will map the PCI config space to the XEN memory.
>
> XEN will read the “linux, pci-domain” property from the device tree
> node and configure the host bridge segment number accordingly.
>
> As of now "pci-host-ecam-generic" compatible board is supported.
>
> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/Kconfig | 7 +
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/pci/Makefile | 4 +
> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
> xen/arch/arm/setup.c | 2 +
> xen/include/asm-arm/device.h | 7 +-
> xen/include/asm-arm/pci.h | 97 +++++++++++++-
> 10 files changed, 654 insertions(+), 6 deletions(-)
> create mode 100644 xen/arch/arm/pci/Makefile
> create mode 100644 xen/arch/arm/pci/pci-access.c
> create mode 100644 xen/arch/arm/pci/pci-host-common.c
> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
> create mode 100644 xen/arch/arm/pci/pci.c
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 2777388265..ee13339ae9 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -31,6 +31,13 @@ menu "Architecture Features"
>
> source "arch/Kconfig"
>
> +config ARM_PCI
> + bool "PCI Passthrough Support"
> + depends on ARM_64
> + ---help---
> +
> + PCI passthrough support for Xen on ARM64.
> +
> config ACPI
> bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
> depends on ARM_64
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 7e82b2178c..345cb83eed 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_ARM_PCI) += pci/
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> new file mode 100644
> index 0000000000..358508b787
> --- /dev/null
> +++ b/xen/arch/arm/pci/Makefile
> @@ -0,0 +1,4 @@
> +obj-y += pci.o
> +obj-y += pci-host-generic.o
> +obj-y += pci-host-common.o
> +obj-y += pci-access.o
The Kconfig option mentions the support being explicitly for ARM64,
would it be better to place the code in arch/arm/arm64 then?
> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
> new file mode 100644
> index 0000000000..c53ef58336
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-access.c
> @@ -0,0 +1,101 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +#include <xen/rwlock.h>
> +
> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
> + unsigned int len)
Please align with the opening parenthesis (here and everywhere in the
patch series).
> +{
> + int rc;
> + uint32_t val = GENMASK(0, len * 8);
You can just set val = ~0. The return type of the pci_conf_readXX
helpers will already truncate the value.
> +
> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> +
> + if ( unlikely(!bridge) )
> + {
> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
I had a patch to add a custom modifier to out printf format in
order to handle pci_sbdf_t natively:
https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
It missed maintainers Acks and was never committed. Since you are
doing a bunch of work here, and likely adding a lot of SBDF related
prints, feel free to import the modifier (%pp) and use in your code
(do not attempt to switch existing users, or it's likely to get
stuck again).
> + return val;
> + }
> +
> + if ( unlikely(!bridge->ops->read) )
> + return val;
> +
> + rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
There's no need for the uint32_t cast, the sbdf field is already of
such type.
> + if ( rc )
> + printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> +
> + return val;
> +}
> +
> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
> + unsigned int len, uint32_t val)
> +{
> + int rc;
> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> +
> + if ( unlikely(!bridge) )
> + {
> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> + return;
> + }
> +
> + if ( unlikely(!bridge->ops->write) )
> + return;
> +
> + rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
> + if ( rc )
> + printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> +}
> +
> +/*
> + * Wrappers for all PCI configuration access functions.
> + */
> +
> +#define PCI_OP_WRITE(size, type) \
> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
> +{ \
> + pci_config_write(sbdf, reg, size / 8, val); \
> +}
> +
> +#define PCI_OP_READ(size, type) \
> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
> +{ \
> + return pci_config_read(sbdf, reg, size / 8); \
> +}
> +
> +PCI_OP_READ(8, u8)
> +PCI_OP_READ(16, u16)
> +PCI_OP_READ(32, u32)
> +PCI_OP_WRITE(8, u8)
> +PCI_OP_WRITE(16, u16)
> +PCI_OP_WRITE(32, u32)
Please use uintXX_t.
Also, It's nice to add some kind of signals for cscope and friends so
they can find the autogenerated functions, ie:
#define pci_conf_read8
#undef pci_conf_read8
#define pci_conf_read16
#undef pci_conf_read16
...
It's tedious but helps future users find where the code is generated.
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> new file mode 100644
> index 0000000000..c5f98be698
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -0,0 +1,198 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +#include <xen/rwlock.h>
> +#include <xen/vmap.h>
> +
> +/*
> + * List for all the pci host bridges.
> + */
> +
> +static LIST_HEAD(pci_host_bridges);
> +
> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
> + struct pci_config_window *cfg)
> +{
> + const __be32 *cells;
It's my impression that while based on Linux this is not a verbatim
copy of a Linux file, and tries to adhere with the Xen coding style.
If so please use uint32_t here.
> + uint32_t len;
> +
> + cells = dt_get_property(dev, "bus-range", &len);
> + /* bus-range should at least be 2 cells */
> + if ( !cells || (len < (sizeof(*cells) * 2)) )
> + return false;
> +
> + cfg->busn_start = dt_next_cell(1, &cells);
> + cfg->busn_end = dt_next_cell(1, &cells);
> +
> + return true;
> +}
> +
> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
> +{
> + return ioremap_nocache(start, len);
> +}
> +
> +static void pci_ecam_free(struct pci_config_window *cfg)
> +{
> + if ( cfg->win )
> + iounmap(cfg->win);
> +
> + xfree(cfg);
> +}
> +
> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> + struct pci_ecam_ops *ops)
> +{
> + int err;
> + struct pci_config_window *cfg;
> + paddr_t addr, size;
> +
> + cfg = xzalloc(struct pci_config_window);
> + if ( !cfg )
> + return NULL;
> +
> + err = dt_pci_parse_bus_range(dev, cfg);
> + if ( !err ) {
Braces
> + cfg->busn_start = 0;
> + cfg->busn_end = 0xff;
> + printk(XENLOG_ERR "No bus range found for pci controller\n");
> + } else {
> + if ( cfg->busn_end > cfg->busn_start + 0xff )
> + cfg->busn_end = cfg->busn_start + 0xff;
> + }
> +
> + /* Parse our PCI ecam register address*/
> + err = dt_device_get_address(dev, 0, &addr, &size);
> + if ( err )
> + goto err_exit;
> +
> + cfg->phys_addr = addr;
> + cfg->size = size;
> + cfg->ops = ops;
> +
> + /*
> + * On 64-bit systems, we do a single ioremap for the whole config space
> + * since we have enough virtual address range available. On 32-bit, we
> + * ioremap the config space for each bus individually.
> + *
> + * As of now only 64-bit is supported 32-bit is not supported.
> + */
> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
> + if ( !cfg->win )
> + goto err_exit_remap;
> +
> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
> + cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
> +
> + if ( ops->init ) {
> + err = ops->init(cfg);
> + if (err)
> + goto err_exit;
> + }
> +
> + return cfg;
> +
> +err_exit_remap:
> + printk(XENLOG_ERR "ECAM ioremap failed\n");
> +err_exit:
> + pci_ecam_free(cfg);
> + return NULL;
> +}
> +
> +static struct pci_host_bridge * pci_alloc_host_bridge(void)
^ extra space
> +{
> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
> +
> + if ( !bridge )
> + return NULL;
> +
> + INIT_LIST_HEAD(&bridge->node);
> + return bridge;
> +}
> +
> +int pci_host_common_probe(struct dt_device_node *dev,
> + struct pci_ecam_ops *ops)
> +{
> + struct pci_host_bridge *bridge;
> + struct pci_config_window *cfg;
> + u32 segment;
> +
> + bridge = pci_alloc_host_bridge();
> + if ( !bridge )
> + return -ENOMEM;
> +
> + /* Parse and map our Configuration Space windows */
> + cfg = gen_pci_init(dev, ops);
> + if ( !cfg )
> + return -ENOMEM;
You are leaking the allocation of bridge here ...
> +
> + bridge->dt_node = dev;
> + bridge->sysdata = cfg;
> + bridge->ops = &ops->pci_ops;
> +
> + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
> + {
> + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
> + return -ENODEV;
... and here.
> + }
> +
> + bridge->segment = (u16)segment;
> +
> + list_add_tail(&bridge->node, &pci_host_bridges);
> +
> + return 0;
> +}
> +
> +/*
> + * This function will lookup an hostbridge based on the segment and bus
> + * number.
> + */
> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
> +{
> + struct pci_host_bridge *bridge;
> + bool found = false;
> +
> + list_for_each_entry( bridge, &pci_host_bridges, node )
> + {
> + if ( bridge->segment != segment )
> + continue;
> +
> + found = true;
> + break;
> + }
> +
> + return (found) ? bridge : NULL;
This can be much shorter:
struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
{
struct pci_host_bridge *bridge;
list_for_each_entry( bridge, &pci_host_bridges, node )
if ( bridge->segment == segment )
return bridge;
return NULL;
}
Albeit I'm confused by the fact that you pass a bus number that's
completely unused.
> +}
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
> new file mode 100644
> index 0000000000..cd67b3dec6
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-generic.c
> @@ -0,0 +1,131 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/device.h>
> +#include <asm/io.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +
> +/*
> + * Function to get the config space base.
> + */
> +static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
> + uint32_t sbdf, int where)
You would be better passing a pci_sbdf_t directly here. Also 'where'
should be renamed to offset, or reg, and be made unsigned int. AFAICT
you will never pass a negative value here.
For sanity you should also assert that the offset falls between the
PCI config space used by the device, in order to easily catch with
wrong offsets being used.
> +{
> + struct pci_config_window *cfg = bridge->sysdata;
const
> + unsigned int devfn_shift = cfg->ops->bus_shift - 8;
> +
> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
> +
> + unsigned int busn = sbdf_t.bus;
> + void __iomem *base;
IMO adding newlines between variable definitions is not helpful, but
that's my taste.
> +
> + if ( busn < cfg->busn_start || busn > cfg->busn_end )
> + return NULL;
> +
> + base = cfg->win + (busn << cfg->ops->bus_shift);
> +
> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
> +}
> +
> +int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
> + int where, int size, u32 val)
> +{
> + void __iomem *addr;
> +
> + addr = pci_config_base(bridge, sbdf, where);
You can initialize at definition.
> + if ( !addr )
> + return -ENODEV;
> +
> + if ( size == 1 )
> + writeb(val, addr);
> + else if ( size == 2 )
> + writew(val, addr);
> + else
> + writel(val, addr);
Please use a switch, and check against specific values. The default
case should be a BUG();. See pci_conf_read from x86 for an example.
> +
> + return 0;
> +}
> +
> +int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
> + int where, int size, u32 *val)
> +{
> + void __iomem *addr;
> +
> + addr = pci_config_base(bridge, sbdf, where);
> + if ( !addr ) {
> + *val = ~0;
> + return -ENODEV;
> + }
> +
> + if ( size == 1 )
> + *val = readb(addr);
> + else if ( size == 2 )
> + *val = readw(addr);
> + else
> + *val = readl(addr);
> +
> + return 0;
> +}
> +
> +/* ECAM ops */
> +struct pci_ecam_ops pci_generic_ecam_ops = {
> + .bus_shift = 20,
> + .pci_ops = {
> + .read = pci_ecam_config_read,
> + .write = pci_ecam_config_write,
> + }
> +};
> +
> +static const struct dt_device_match gen_pci_dt_match[] = {
> + { .compatible = "pci-host-ecam-generic",
> + .data = &pci_generic_ecam_ops },
> +
> + { },
> +};
> +
> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
> +{
> + const struct dt_device_match *of_id;
> + struct pci_ecam_ops *ops;
> +
> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> + ops = (struct pci_ecam_ops *) of_id->data;
> +
> + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
> + dt_node_full_name(dev), of_id->compatible);
> +
> + return pci_host_common_probe(dev, ops);
> +}
> +
> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
> +.dt_match = gen_pci_dt_match,
> +.init = gen_pci_dt_init,
> +DT_DEVICE_END
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
> new file mode 100644
> index 0000000000..f8cbb99591
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci.c
> @@ -0,0 +1,112 @@
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/acpi.h>
> +#include <xen/device_tree.h>
> +#include <xen/errno.h>
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <xen/param.h>
> +
> +static int __init dt_pci_init(void)
> +{
> + struct dt_device_node *np;
> + int rc;
> +
> + dt_for_each_device_node(dt_host, np)
> + {
> + rc = device_init(np, DEVICE_PCI, NULL);
> + if( !rc )
> + continue;
> + /*
> + * Ignore the following error codes:
> + * - EBADF: Indicate the current is not an pci
> + * - ENODEV: The pci device is not present or cannot be used by
> + * Xen.
> + */
> + else if ( rc != -EBADF && rc != -ENODEV )
> + {
> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
> + return rc;
> + }
> + }
> +
> + return 0;
> +}
> +
> +#ifdef CONFIG_ACPI
> +static void __init acpi_pci_init(void)
> +{
> + printk(XENLOG_ERR "ACPI pci init not supported \n");
> + return;
> +}
> +#else
> +static inline void __init acpi_pci_init(void) { }
> +#endif
> +
> +static bool __initdata param_pci_enable;
> +static int __init parse_pci_param(const char *arg)
> +{
> + if ( !arg )
> + {
> + param_pci_enable = false;
> + return 0;
> + }
> +
> + switch ( parse_bool(arg, NULL) )
> + {
> + case 0:
> + param_pci_enable = false;
> + return 0;
> + case 1:
> + param_pci_enable = true;
> + return 0;
> + }
> +
> + return -EINVAL;
> +}
> +custom_param("pci", parse_pci_param);
You need to introduce the documentation for the parameter at
docs/misc/xen-command-line.pandoc
Albeit I'm not sure I like it, why do you need to enable PCI
explicitly?
Shouldn't it be discovered automatically and enabled by default?
> +void __init pci_init(void)
> +{
> + /*
> + * Enable PCI when has been enabled explicitly (pci=on)
> + */
> + if ( !param_pci_enable)
> + goto disable;
Just return here, there's not point in having a label to perform a
return.
> +
> + if ( acpi_disabled )
> + dt_pci_init();
> + else
> + acpi_pci_init();
Isn't there an enum or something that tells you whether the system
description is coming from ACPI or from DT?
This if .. else seems fragile.
Also for ACPI you will get called by acpi_boot_init, and likely need
to implement a acpi_mmcfg_init or pci_mmcfg_arch_{init,enable}. I'm
not sure whether the code in acpi_mmcfg_init could be made shared
between both x86 and Arm.
> +
> +#ifdef CONFIG_HAS_PCI
> + pci_segments_init();
> +#endif
> +
> +disable:
> + return;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index 7968cee47d..2d7f1db44f 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>
> setup_virt_paging();
>
> + pci_init();
> +
> do_initcalls();
>
> /*
> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
> index ee7cff2d44..28f8049cfd 100644
> --- a/xen/include/asm-arm/device.h
> +++ b/xen/include/asm-arm/device.h
> @@ -4,6 +4,7 @@
> enum device_type
> {
> DEV_DT,
> + DEV_PCI,
> };
>
> struct dev_archdata {
> @@ -25,15 +26,15 @@ typedef struct device device_t;
>
> #include <xen/device_tree.h>
>
> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
> -#define dev_is_pci(dev) ((void)(dev), 0)
> -#define dev_is_dt(dev) ((dev->type == DEV_DT)
> +#define dev_is_pci(dev) (dev->type == DEV_PCI)
> +#define dev_is_dt(dev) (dev->type == DEV_DT)
>
> enum device_class
> {
> DEVICE_SERIAL,
> DEVICE_IOMMU,
> DEVICE_GIC,
> + DEVICE_PCI,
It seems to be like this wants to be DEVICE_PCI_HOST_BRIDGE or some
such, since this is not used to identify all PCI devices, but just
bridges?
> /* Use for error */
> DEVICE_UNKNOWN,
> };
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index de13359f65..94fd00360a 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -1,7 +1,98 @@
> -#ifndef __X86_PCI_H__
> -#define __X86_PCI_H__
> +/*
> + * Copyright (C) 2020 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
>
> +#ifndef __ARM_PCI_H__
> +#define __ARM_PCI_H__
> +
> +#include <xen/pci.h>
> +#include <xen/device_tree.h>
> +#include <asm/device.h>
> +
> +#ifdef CONFIG_ARM_PCI
> +
> +/* Arch pci dev struct */
> struct arch_pci_dev {
> + struct device dev;
> +};
This seems to be completely unused?
> +
> +#define PRI_pci "%04x:%02x:%02x.%u"
> +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
> +
> +/*
> + * struct to hold the mappings of a config space window. This
> + * is expected to be used as sysdata for PCI controllers that
> + * use ECAM.
> + */
> +struct pci_config_window {
> + paddr_t phys_addr;
> + paddr_t size;
> + uint8_t busn_start;
> + uint8_t busn_end;
> + struct pci_ecam_ops *ops;
const?
> + void __iomem *win;
> +};
> +
> +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
> +struct pci_host_bridge;
> +
> +struct pci_ops {
> + int (*read)(struct pci_host_bridge *bridge,
> + uint32_t sbdf, int where, int size, u32 *val);
> + int (*write)(struct pci_host_bridge *bridge,
> + uint32_t sbdf, int where, int size, u32 val);
> +};
> +
> +/*
> + * struct to hold pci ops and bus shift of the config window
> + * for a PCI controller.
> + */
> +struct pci_ecam_ops {
> + unsigned int bus_shift;
> + struct pci_ops pci_ops;
> + int (*init)(struct pci_config_window *);
> +};
> +
> +/*
> + * struct to hold pci host bridge information
> + * for a PCI controller.
> + */
> +struct pci_host_bridge {
> + struct dt_device_node *dt_node; /* Pointer to the associated DT node */
> + struct list_head node; /* Node in list of host bridges */
> + uint16_t segment; /* Segment number */
> + void *sysdata; /* Pointer to the config space window*/
> + const struct pci_ops *ops;
You seem to introduce a lot of ops structs, yet there's only one
implementation the generic ECAM one, and adding such complexity should
IMO be done when further implementations are added. Also given this is
a fully ECAM compliant bridge you could just use most of the existing
logic for x86?
I understand the discovery needs to be different, but x86 MCFG logic
should already be capable of handling multiple ECAM regions.
I also agree with Julien that splitting this into separate patch would
make it easier to review. For example you can start with the discovery
logic, followed by the initialization and the add the accessors to the
config space lastly.
Thanks, Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
2020-07-23 20:44 ` Stefano Stabellini
2020-07-24 7:14 ` Oleksandr Andrushchenko
@ 2020-07-24 14:49 ` Roger Pau Monné
2020-07-27 8:40 ` Rahul Singh
2 siblings, 0 replies; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-24 14:49 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Rahul Singh, Julien Grall, andrew.cooper3, Bertrand.Marquis,
jbeulich, xen-devel, nd, Volodymyr Babchuk
On Thu, Jul 23, 2020 at 01:44:03PM -0700, Stefano Stabellini wrote:
> On Thu, 23 Jul 2020, Rahul Singh wrote:
> > Hardware domain is in charge of doing the PCI enumeration and will
> > discover the PCI devices and then will communicate to XEN via hyper
> > call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
> >
> > Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
>
> Same question about Change-Id
>
>
> > Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> > ---
> > xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
> > 1 file changed, 39 insertions(+), 3 deletions(-)
> >
> > diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
> > index e91355fe22..274720f98a 100644
> > --- a/xen/arch/arm/physdev.c
> > +++ b/xen/arch/arm/physdev.c
> > @@ -9,12 +9,48 @@
> > #include <xen/errno.h>
> > #include <xen/sched.h>
> > #include <asm/hypercall.h>
> > -
> > +#include <xen/guest_access.h>
> > +#include <xsm/xsm.h>
> >
> > int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
> > {
> > - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> > - return -ENOSYS;
> > + int ret = 0;
> > +
> > + switch ( cmd )
> > + {
> > +#ifdef CONFIG_HAS_PCI
> > + case PHYSDEVOP_pci_device_add:
> > + {
> > + struct physdev_pci_device_add add;
> > + struct pci_dev_info pdev_info;
> > + nodeid_t node = NUMA_NO_NODE;
> > +
> > + ret = -EFAULT;
> > + if ( copy_from_guest(&add, arg, 1) != 0 )
> > + break;
> > +
> > + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
> > + if ( add.flags & XEN_PCI_DEV_VIRTFN )
> > + {
> > + pdev_info.is_virtfn = 1;
> > + pdev_info.physfn.bus = add.physfn.bus;
> > + pdev_info.physfn.devfn = add.physfn.devfn;
> > + }
> > + else
> > + pdev_info.is_virtfn = 0;
> > +
> > + ret = pci_add_device(add.seg, add.bus, add.devfn,
> > + &pdev_info, node);
> > +
> > + break;
> > + }
> > +#endif
> > + default:
> > + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> > + ret = -ENOSYS;
> > + }
>
> I think we should make the implementation common between arm and x86 by
> creating xen/common/physdev.c:do_physdev_op as a shared entry point for
> PHYSDEVOP hypercalls implementations. See for instance:
>
> xen/common/sysctl.c:do_sysctl
>
> and
>
> xen/arch/arm/sysctl.c:arch_do_sysctl
> xen/arch/x86/sysctl.c:arch_do_sysctl
>
>
> Jan, Andrew, Roger, any opinions?
Oh, physdev ops don't have a common entry point, it's all per-arch.
Since Arm has no physdev ops at all, I think we should start by adding
a common do_physdev_op and move PHYSDEVOP_pci_device_add into it,
leaving the rest of x86 operations as arch_do_physdev_op.
Thanks, Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM.
2020-07-23 15:40 ` [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
2020-07-23 23:39 ` Stefano Stabellini
@ 2020-07-24 15:08 ` Roger Pau Monné
1 sibling, 0 replies; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-24 15:08 UTC (permalink / raw)
To: Rahul Singh
Cc: Stefano Stabellini, Julien Grall, Paul Durrant, Bertrand.Marquis,
Jan Beulich, xen-devel, nd, Volodymyr Babchuk
On Thu, Jul 23, 2020 at 04:40:23PM +0100, Rahul Singh wrote:
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the PCI device to emulate the PCI devices.
>
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
>
> VPCI MSI support is disable for ARM as it is not tested on ARM.
I'm not seeing anything on this patch that would disable vPCI MSI
support?
>
> Change-Id: I5501db2781f8064640403fecce53713091cd9ab4
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/domain.c | 4 ++
> xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++
> xen/arch/arm/vpci.h | 37 ++++++++++++
> xen/drivers/passthrough/pci.c | 7 +++
> xen/include/asm-arm/domain.h | 5 ++
> xen/include/public/arch-arm.h | 4 ++
> 7 files changed, 160 insertions(+)
> create mode 100644 xen/arch/arm/vpci.c
> create mode 100644 xen/arch/arm/vpci.h
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 345cb83eed..5a23ec5cc0 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> obj-$(CONFIG_ARM_PCI) += pci/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 31169326b2..23098ffd02 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -39,6 +39,7 @@
> #include <asm/vtimer.h>
>
> #include "vuart.h"
> +#include "vpci.h"
>
> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>
> @@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,
> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
> goto fail;
>
> + if ( (rc = domain_vpci_init(d)) != 0 )
> + goto fail;
> +
> return 0;
>
> fail:
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..49e473ab0d
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,102 @@
> +/*
> + * xen/arch/arm/vpci.c
> + * Copyright (c) 2020 Arm Ltd.
> + *
> + * Based on arch/x86/hvm/io.c
> + * Copyright (c) 2004, Intel Corporation.
> + * Copyright (c) 2005, International Business Machines Corporation.
> + * Copyright (c) 2008, Citrix Systems, Inc.
> + *
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +#include <asm/mmio.h>
> +
> +/* Do some sanity checks. */
> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
This is just a copy of vpci_access_allowed from x86, I think you
should consider moving the function to common vpci code and just share
it between x86 and Arm?
> +{
> + /* Check access size. */
> + if ( len != 1 && len != 2 && len != 4 && len != 8 )
> + return false;
> +
> + /* Check that access is size aligned. */
> + if ( (reg & (len - 1)) )
> + return false;
> +
> + return true;
> +}
> +
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> + register_t *r, void *priv)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + uint32_t data = 0;
> + unsigned int size = 1U << info->dabt.size;
> +
> + sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> +
> + if ( !vpci_mmio_access_allowed(reg, size) )
> + return 1;
> +
> + data = vpci_read(sbdf, reg, size);
> +
> + memcpy(r, &data, size);
> +
> + return 1;
> +}
> +
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> + register_t r, void *priv)
> +{
> + unsigned int reg;
> + pci_sbdf_t sbdf;
> + uint32_t data = r;
> + unsigned int size = 1U << info->dabt.size;
> +
> + sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> +
> + if ( !vpci_mmio_access_allowed(reg, size) )
> + return 1;
> +
> + vpci_write(sbdf, reg, size, data);
> +
> + return 1;
Both functions will only return 1 always, so can likely drop the
return value completely?
> +}
> +
> +static const struct mmio_handler_ops vpci_mmio_handler = {
> + .read = vpci_mmio_read,
> + .write = vpci_mmio_write,
> +};
> +
> +int domain_vpci_init(struct domain *d)
FWIW, I think you can drop the domain_ prefix, vPCI is always tied to
a domain.
> +{
> + if ( !has_vpci(d) || is_hardware_domain(d) )
I wouldn't add a hardware domain exception here, and just make sure
the VPCI flag is not set for the hardware domain on Arm if you don't
want to use it there.
> + return 0;
> +
> + register_mmio_handler(d, &vpci_mmio_handler,
> + GUEST_VPCI_ECAM_BASE,GUEST_VPCI_ECAM_SIZE,NULL);
Missing spaces, and proper indentation.
> +
> + return 0;
Doesn't seem like domain_vpci_init can fail, so you can likely skip
the return value?
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> +
> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
> new file mode 100644
> index 0000000000..20dce1f4c4
> --- /dev/null
> +++ b/xen/arch/arm/vpci.h
> @@ -0,0 +1,37 @@
> +/*
> + * xen/arch/arm/vpci.h
> + * Copyright (c) 2020 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef __ARCH_ARM_VPCI_H__
> +#define __ARCH_ARM_VPCI_H__
> +
> +#ifdef CONFIG_HAS_VPCI
> +int domain_vpci_init(struct domain *d);
> +#else
> +static inline int domain_vpci_init(struct domain *d)
> +{
> + return 0;
> +}
> +#endif
> +
> +#endif /* __ARCH_ARM_VPCI_H__ */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> index 5846978890..28511eb641 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
> else
> iommu_enable_device(pdev);
>
> +#ifdef CONFIG_ARM
> + ret = vpci_add_handlers(pdev);
Don't you need to drop the __hwdom_init from that function? Or else it
might be freed by the time dom0 calls pci_add_device?
> + if ( ret ) {
> + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
> + goto out;
> + }
> +#endif
> pci_enable_acs(pdev);
>
> out:
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 4e2f582006..ad70610226 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -34,6 +34,11 @@ enum domain_type {
> /* The hardware domain has always its memory direct mapped. */
> #define is_domain_direct_mapped(d) ((d) == hardware_domain)
>
> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
> + * for ARM we enable support VPCI for guest domain also.
> + */
> +#define has_vpci(d) (true)
Urg, that's kind of unconditional, couldn't you pass a flag for
user-space in order to signal whether vPCI should be enabled? There's
no point on enabling it if the domain doesn't support PCI passthrough,
or if there are no PCI devices on the system.
Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 14:44 ` Roger Pau Monné
@ 2020-07-24 15:15 ` Julien Grall
2020-07-24 15:29 ` Roger Pau Monné
2020-07-24 16:01 ` Jan Beulich
2020-07-28 8:06 ` Rahul Singh
1 sibling, 2 replies; 48+ messages in thread
From: Julien Grall @ 2020-07-24 15:15 UTC (permalink / raw)
To: Roger Pau Monné, Rahul Singh
Cc: xen-devel, nd, Stefano Stabellini, Volodymyr Babchuk, Bertrand.Marquis
On 24/07/2020 15:44, Roger Pau Monné wrote:
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> new file mode 100644
>> index 0000000000..358508b787
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -0,0 +1,4 @@
>> +obj-y += pci.o
>> +obj-y += pci-host-generic.o
>> +obj-y += pci-host-common.o
>> +obj-y += pci-access.o
>
> The Kconfig option mentions the support being explicitly for ARM64,
> would it be better to place the code in arch/arm/arm64 then?
I don't believe any of the code in this series is very arm64 specific. I
guess it was just only tested on arm64. So I would rather keep that
under arm/pci.
>> +
>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> + if ( unlikely(!bridge) )
>> + {
>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>
> I had a patch to add a custom modifier to out printf format in
> order to handle pci_sbdf_t natively:
>
> https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
>
> It missed maintainers Acks and was never committed. Since you are
> doing a bunch of work here, and likely adding a lot of SBDF related
> prints, feel free to import the modifier (%pp) and use in your code
> (do not attempt to switch existing users, or it's likely to get
> stuck again).
I forgot about this patch :/. It would be good to revive it. Which acks
are you missing?
[...]
>> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
>> + struct pci_config_window *cfg)
>> +{
>> + const __be32 *cells;
>
> It's my impression that while based on Linux this is not a verbatim
> copy of a Linux file, and tries to adhere with the Xen coding style.
> If so please use uint32_t here.
uint32_t would be incorrect because this is a 32-bit value always in big
endian. I don't think we have other typedef to show it is a 32-bit BE
value, so __be32 is the best choice.
[...]
>> +
>> + if ( acpi_disabled )
>> + dt_pci_init();
>> + else
>> + acpi_pci_init();
>
> Isn't there an enum or something that tells you whether the system
> description is coming from ACPI or from DT?
>
> This if .. else seems fragile.
>
This is the common way we do it on Arm.... I would welcome any
improvement, but I don't think this should be part of this work.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 15:15 ` Julien Grall
@ 2020-07-24 15:29 ` Roger Pau Monné
2020-07-24 15:42 ` Roger Pau Monné
2020-07-24 15:46 ` Julien Grall
2020-07-24 16:01 ` Jan Beulich
1 sibling, 2 replies; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-24 15:29 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, Bertrand.Marquis, Stefano Stabellini, xen-devel, nd,
Volodymyr Babchuk
On Fri, Jul 24, 2020 at 04:15:47PM +0100, Julien Grall wrote:
>
>
> On 24/07/2020 15:44, Roger Pau Monné wrote:
> > > diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> > > new file mode 100644
> > > index 0000000000..358508b787
> > > --- /dev/null
> > > +++ b/xen/arch/arm/pci/Makefile
> > > @@ -0,0 +1,4 @@
> > > +obj-y += pci.o
> > > +obj-y += pci-host-generic.o
> > > +obj-y += pci-host-common.o
> > > +obj-y += pci-access.o
> >
> > The Kconfig option mentions the support being explicitly for ARM64,
> > would it be better to place the code in arch/arm/arm64 then?
> I don't believe any of the code in this series is very arm64 specific. I
> guess it was just only tested on arm64. So I would rather keep that under
> arm/pci.
Ack. Could the Kconfg be adjusted to not depend on ARM_64? Just
stating it's only been tested on Arm64 would be enough IMO.
> > > +
> > > + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> > > +
> > > + if ( unlikely(!bridge) )
> > > + {
> > > + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> > > + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> >
> > I had a patch to add a custom modifier to out printf format in
> > order to handle pci_sbdf_t natively:
> >
> > https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
> >
> > It missed maintainers Acks and was never committed. Since you are
> > doing a bunch of work here, and likely adding a lot of SBDF related
> > prints, feel free to import the modifier (%pp) and use in your code
> > (do not attempt to switch existing users, or it's likely to get
> > stuck again).
>
> I forgot about this patch :/. It would be good to revive it. Which acks are
> you missing?
I only had an Ack from Jan, so I was missing Intel and AMD Acks, which
would now only be Intel since AMD has been absorbed by the x86
maintainers.
> [...]
>
> > > +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
> > > + struct pci_config_window *cfg)
> > > +{
> > > + const __be32 *cells;
> >
> > It's my impression that while based on Linux this is not a verbatim
> > copy of a Linux file, and tries to adhere with the Xen coding style.
> > If so please use uint32_t here.
>
> uint32_t would be incorrect because this is a 32-bit value always in big
> endian. I don't think we have other typedef to show it is a 32-bit BE value,
> so __be32 is the best choice.
Oh, OK, so this is done to explicitly denote the endianness of a value
on the type itself.
> [...]
>
> > > +
> > > + if ( acpi_disabled )
> > > + dt_pci_init();
> > > + else
> > > + acpi_pci_init();
> >
> > Isn't there an enum or something that tells you whether the system
> > description is coming from ACPI or from DT?
> >
> > This if .. else seems fragile.
> >
>
> This is the common way we do it on Arm.... I would welcome any improvement,
> but I don't think this should be part of this work.
Ack. In any case I think for ACPI PCI init will get called by
acpi_mmcfg_init as part of acpi_boot_init, so I'm not sure there's
much point in having something about ACPI added here, as it seems this
will be DT only?
Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 15:29 ` Roger Pau Monné
@ 2020-07-24 15:42 ` Roger Pau Monné
2020-07-24 15:46 ` Julien Grall
1 sibling, 0 replies; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-24 15:42 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, Bertrand.Marquis, Stefano Stabellini, xen-devel, nd,
Volodymyr Babchuk
On Fri, Jul 24, 2020 at 05:29:05PM +0200, Roger Pau Monné wrote:
> On Fri, Jul 24, 2020 at 04:15:47PM +0100, Julien Grall wrote:
> >
> >
> > On 24/07/2020 15:44, Roger Pau Monné wrote:
> > > > +
> > > > + if ( acpi_disabled )
> > > > + dt_pci_init();
> > > > + else
> > > > + acpi_pci_init();
> > >
> > > Isn't there an enum or something that tells you whether the system
> > > description is coming from ACPI or from DT?
> > >
> > > This if .. else seems fragile.
> > >
> >
> > This is the common way we do it on Arm.... I would welcome any improvement,
> > but I don't think this should be part of this work.
>
> Ack. In any case I think for ACPI PCI init will get called by
> acpi_mmcfg_init as part of acpi_boot_init, so I'm not sure there's
> much point in having something about ACPI added here, as it seems this
> will be DT only?
Sorry I got confused and acpi_boot_init is a x86 specific function,
wrong context. If Arm is not using that path then maybe it makes sense
to init PCI here for ACPI also.
Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 15:29 ` Roger Pau Monné
2020-07-24 15:42 ` Roger Pau Monné
@ 2020-07-24 15:46 ` Julien Grall
1 sibling, 0 replies; 48+ messages in thread
From: Julien Grall @ 2020-07-24 15:46 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Rahul Singh, Bertrand.Marquis, Stefano Stabellini, xen-devel, nd,
Volodymyr Babchuk
On 24/07/2020 16:29, Roger Pau Monné wrote:
> On Fri, Jul 24, 2020 at 04:15:47PM +0100, Julien Grall wrote:
>>
>>
>> On 24/07/2020 15:44, Roger Pau Monné wrote:
>>>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>>>> new file mode 100644
>>>> index 0000000000..358508b787
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/pci/Makefile
>>>> @@ -0,0 +1,4 @@
>>>> +obj-y += pci.o
>>>> +obj-y += pci-host-generic.o
>>>> +obj-y += pci-host-common.o
>>>> +obj-y += pci-access.o
>>>
>>> The Kconfig option mentions the support being explicitly for ARM64,
>>> would it be better to place the code in arch/arm/arm64 then?
>> I don't believe any of the code in this series is very arm64 specific. I
>> guess it was just only tested on arm64. So I would rather keep that under
>> arm/pci.
>
> Ack. Could the Kconfg be adjusted to not depend on ARM_64? Just
> stating it's only been tested on Arm64 would be enough IMO.
We already have an option to select PCI (see CONFIG_HAS_PCI). So I would
prefer if we reuse it (possible renamed to CONFIG_PCI) rather than
inventing one for Arm specific.
Regarding the dependency, it will depend if it is possible to make it
build easily on Arm32. If not, then we will need to keep the ARM_64.
>
>>>> +
>>>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>>> +
>>>> + if ( unlikely(!bridge) )
>>>> + {
>>>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>>
>>> I had a patch to add a custom modifier to out printf format in
>>> order to handle pci_sbdf_t natively:
>>>
>>> https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
>>>
>>> It missed maintainers Acks and was never committed. Since you are
>>> doing a bunch of work here, and likely adding a lot of SBDF related
>>> prints, feel free to import the modifier (%pp) and use in your code
>>> (do not attempt to switch existing users, or it's likely to get
>>> stuck again).
>>
>> I forgot about this patch :/. It would be good to revive it. Which acks are
>> you missing?
>
> I only had an Ack from Jan, so I was missing Intel and AMD Acks, which
> would now only be Intel since AMD has been absorbed by the x86
> maintainers.
Ok. So, it should be easier to get it acked now :).
>
>> [...]
>>
>>>> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
>>>> + struct pci_config_window *cfg)
>>>> +{
>>>> + const __be32 *cells;
>>>
>>> It's my impression that while based on Linux this is not a verbatim
>>> copy of a Linux file, and tries to adhere with the Xen coding style.
>>> If so please use uint32_t here.
>>
>> uint32_t would be incorrect because this is a 32-bit value always in big
>> endian. I don't think we have other typedef to show it is a 32-bit BE value,
>> so __be32 is the best choice.
>
> Oh, OK, so this is done to explicitly denote the endianness of a value
> on the type itself.
That's correct. On Linux, they use sparse to then check the BE and LE
fields are not mixed together. We don't have that on Xen, but at least
this makes more obvious what we are using.
>
>> [...]
>>
>>>> +
>>>> + if ( acpi_disabled )
>>>> + dt_pci_init();
>>>> + else
>>>> + acpi_pci_init();
>>>
>>> Isn't there an enum or something that tells you whether the system
>>> description is coming from ACPI or from DT?
>>>
>>> This if .. else seems fragile.
>>>
>>
>> This is the common way we do it on Arm.... I would welcome any improvement,
>> but I don't think this should be part of this work.
>
> Ack. In any case I think for ACPI PCI init will get called by
> acpi_mmcfg_init as part of acpi_boot_init, so I'm not sure there's
> much point in having something about ACPI added here, as it seems this
> will be DT only?
acpi_boot_init() does not exist on Arm. Looking at x86,
acpi_mmcfg_init() is not event called from that function.
In general, I would prefer if each subsystem takes care of
initialization itself. This makes easier to figure out the difference
between ACPI and DT. FWIW, this is inline with majority of the Arm code.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 15:15 ` Julien Grall
2020-07-24 15:29 ` Roger Pau Monné
@ 2020-07-24 16:01 ` Jan Beulich
2020-07-24 16:54 ` Julien Grall
1 sibling, 1 reply; 48+ messages in thread
From: Jan Beulich @ 2020-07-24 16:01 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, Bertrand.Marquis, Stefano Stabellini, xen-devel, nd,
Volodymyr Babchuk, Roger Pau Monné
On 24.07.2020 17:15, Julien Grall wrote:
> On 24/07/2020 15:44, Roger Pau Monné wrote:
>>> +
>>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>> +
>>> + if ( unlikely(!bridge) )
>>> + {
>>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>
>> I had a patch to add a custom modifier to out printf format in
>> order to handle pci_sbdf_t natively:
>>
>> https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
>>
>> It missed maintainers Acks and was never committed. Since you are
>> doing a bunch of work here, and likely adding a lot of SBDF related
>> prints, feel free to import the modifier (%pp) and use in your code
>> (do not attempt to switch existing users, or it's likely to get
>> stuck again).
>
> I forgot about this patch :/. It would be good to revive it. Which acks
> are you missing?
It wasn't so much missing acks, but a controversy. And that not so much
about switching existing users, but whether to indeed derive this from
%p (which I continue to consider inefficient).
Jan
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 16:01 ` Jan Beulich
@ 2020-07-24 16:54 ` Julien Grall
2020-07-27 10:34 ` Roger Pau Monné
0 siblings, 1 reply; 48+ messages in thread
From: Julien Grall @ 2020-07-24 16:54 UTC (permalink / raw)
To: Jan Beulich
Cc: Rahul Singh, Bertrand.Marquis, Stefano Stabellini, xen-devel, nd,
Volodymyr Babchuk, Roger Pau Monné
Hi Jan,
On 24/07/2020 17:01, Jan Beulich wrote:
> On 24.07.2020 17:15, Julien Grall wrote:
>> On 24/07/2020 15:44, Roger Pau Monné wrote:
>>>> +
>>>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>>> +
>>>> + if ( unlikely(!bridge) )
>>>> + {
>>>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>>
>>> I had a patch to add a custom modifier to out printf format in
>>> order to handle pci_sbdf_t natively:
>>>
>>> https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
>>>
>>> It missed maintainers Acks and was never committed. Since you are
>>> doing a bunch of work here, and likely adding a lot of SBDF related
>>> prints, feel free to import the modifier (%pp) and use in your code
>>> (do not attempt to switch existing users, or it's likely to get
>>> stuck again).
>>
>> I forgot about this patch :/. It would be good to revive it. Which acks
>> are you missing?
>
> It wasn't so much missing acks, but a controversy. And that not so much
> about switching existing users, but whether to indeed derive this from
> %p (which I continue to consider inefficient).
Looking at the thread, I can see you (relunctantly) acked any components
that you are the sole maintainers. Kevin gave his acked for the vtd code
and I gave it mine for the common code.
I would suggest to not rehash the argument unless another maintainer
agree with your position. It loosk like to me the next step is for Roger
(or someone else) to resend the patch so we could collect the missing
ack (I think there is only one missing from Andrew).
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 8:44 ` Julien Grall
@ 2020-07-24 17:41 ` Stefano Stabellini
2020-07-24 18:21 ` Julien Grall
0 siblings, 1 reply; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-24 17:41 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, andrew.cooper3, Bertrand.Marquis,
Rahul Singh, jbeulich, xen-devel, nd, Volodymyr Babchuk,
roger.pau
On Fri, 24 Jul 2020, Julien Grall wrote:
> On 24/07/2020 00:38, Stefano Stabellini wrote:
> > > + bridge->dt_node = dev;
> > > + bridge->sysdata = cfg;
> > > + bridge->ops = &ops->pci_ops;
> > > +
> > > + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
> > > + {
> > > + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available
> > > in DT\n");
> > > + return -ENODEV;
> > > + }
> > > +
> > > + bridge->segment = (u16)segment;
> >
> > My understanding is that a Linux pci-domain doesn't correspond exactly
> > to a PCI segment. See for instance:
> >
> > https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
> >
> > Do we need to care about the difference? If we mean pci-domain here,
> > should we just call them as such instead of calling them "segments" in
> > Xen (if they are not segments)?
>
> So we definitely need a segment number in hand because this is what the admin
> will use to assign a PCI device to a guest.
Yeah
> The segment number is just a value defined by the software. So as long as
> Linux and Xen agrees with the number, then we should be ok.
As far as I understand a Linux "domain" (linux,pci-domain in device
tree) is a value defined by the software. The PCI segment has a
definition in the PCI spec and it is returned by _SEG on ACPI systems.
The link above suggests that a Linux domain corresponds to (_SEG,
_BBN) where _SEG is the segment and _BBN is the "Bus Number".
I just would like to be precise with the terminology: if we are talking
about domains in the linux sense of the word, as it looks like we are
doing, then let's call them domain instead of segments which seem to
have a different definition.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 8:05 ` Julien Grall
@ 2020-07-24 17:47 ` Stefano Stabellini
2020-07-27 15:27 ` Rahul Singh
1 sibling, 0 replies; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-24 17:47 UTC (permalink / raw)
To: Julien Grall
Cc: nd, Stefano Stabellini, Oleksandr Andrushchenko, andrew.cooper3,
Bertrand.Marquis, jbeulich, xen-devel, Rahul Singh,
Volodymyr Babchuk, roger.pau
On Fri, 24 Jul 2020, Julien Grall wrote:
> > > > + list_add_tail(&bridge->node, &pci_host_bridges);
> > > It looks like &pci_host_bridges should be an ordered list, ordered by
> > > segment number?
> >
> > Why? Do you expect bridge access in some specific order so ordered
> >
> > list will make it faster?
>
> Access to the configure space will be pretty random. So I don't think ordering
> the list will make anything better.
>
> However, looking up for the bridge for every config spec access is pretty
> slow. When I was working on the PCI passthrough, I wanted to look whether it
> would be possible to have a pointer to the PCI host bridge passed in argument
> to the helpers (rather than the segment).
I was suggesting ordering the list because it is a rather cheap
optimization: it is easy to implement and it typically leads to decent
improvements (although it varies on a case by case basis). Something
more sophisticated as you mention here would be even better.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 17:41 ` Stefano Stabellini
@ 2020-07-24 18:21 ` Julien Grall
2020-07-24 18:32 ` Stefano Stabellini
0 siblings, 1 reply; 48+ messages in thread
From: Julien Grall @ 2020-07-24 18:21 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Rahul Singh, andrew.cooper3, Bertrand.Marquis, jbeulich,
xen-devel, nd, Volodymyr Babchuk, roger.pau
On 24/07/2020 18:41, Stefano Stabellini wrote:
> On Fri, 24 Jul 2020, Julien Grall wrote:
>> On 24/07/2020 00:38, Stefano Stabellini wrote:
>> The segment number is just a value defined by the software. So as long as
>> Linux and Xen agrees with the number, then we should be ok.
>
> As far as I understand a Linux "domain" (linux,pci-domain in device
> tree) is a value defined by the software. The PCI segment has a
> definition in the PCI spec and it is returned by _SEG on ACPI systems.
>
> The link above suggests that a Linux domain corresponds to (_SEG,
> _BBN) where _SEG is the segment and _BBN is the "Bus Number".
>
> I just would like to be precise with the terminology: if we are talking
> about domains in the linux sense of the word, as it looks like we are
> doing, then let's call them domain instead of segments which seem to
> have a different definition.
You seem to argue on the name but this doesn't resolve the underlying
problem. Indeed, all our external interfaces are expecting a segment number.
If they are not equal, then I fail to see why it would be useful to have
this value in Xen. In which case, we need to use
PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the
segment number.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 18:21 ` Julien Grall
@ 2020-07-24 18:32 ` Stefano Stabellini
2020-07-24 19:24 ` Julien Grall
0 siblings, 1 reply; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-24 18:32 UTC (permalink / raw)
To: Julien Grall
Cc: nd, Stefano Stabellini, andrew.cooper3, Bertrand.Marquis,
jbeulich, xen-devel, Rahul Singh, Volodymyr Babchuk, roger.pau
On Fri, 24 Jul 2020, Julien Grall wrote:
> On 24/07/2020 18:41, Stefano Stabellini wrote:
> > On Fri, 24 Jul 2020, Julien Grall wrote:
> > > On 24/07/2020 00:38, Stefano Stabellini wrote:
> > > The segment number is just a value defined by the software. So as long as
> > > Linux and Xen agrees with the number, then we should be ok.
> >
> > As far as I understand a Linux "domain" (linux,pci-domain in device
> > tree) is a value defined by the software. The PCI segment has a
> > definition in the PCI spec and it is returned by _SEG on ACPI systems.
> >
> > The link above suggests that a Linux domain corresponds to (_SEG,
> > _BBN) where _SEG is the segment and _BBN is the "Bus Number".
> >
> > I just would like to be precise with the terminology: if we are talking
> > about domains in the linux sense of the word, as it looks like we are
> > doing, then let's call them domain instead of segments which seem to
> > have a different definition.
>
> You seem to argue on the name but this doesn't resolve the underlying problem.
> Indeed, all our external interfaces are expecting a segment number.
Yes, you are right, that is a bigger problem.
> If they are not equal, then I fail to see why it would be useful to have this
> value in Xen.
I think that's because the domain is actually more convenient to use
because a segment can span multiple PCI host bridges. So my
understanding is that a segment alone is not sufficient to identify a
host bridge. From a software implementation point of view it would be
better to use domains.
> In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> Dom0 and Xen can synchronize on the segment number.
I was hoping we could write down the assumption somewhere that for the
cases we care about domain == segment, and error out if it is not the
case.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 18:32 ` Stefano Stabellini
@ 2020-07-24 19:24 ` Julien Grall
2020-07-24 23:46 ` Stefano Stabellini
0 siblings, 1 reply; 48+ messages in thread
From: Julien Grall @ 2020-07-24 19:24 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Rahul Singh, Andrew Cooper, Bertrand Marquis, Jan Beulich,
xen-devel, nd, Volodymyr Babchuk, Roger Pau Monné
On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > If they are not equal, then I fail to see why it would be useful to have this
> > value in Xen.
>
> I think that's because the domain is actually more convenient to use
> because a segment can span multiple PCI host bridges. So my
> understanding is that a segment alone is not sufficient to identify a
> host bridge. From a software implementation point of view it would be
> better to use domains.
AFAICT, this would be a matter of one check vs two checks in Xen :).
But... looking at Linux, they will also use domain == segment for ACPI
(see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
>
> > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > Dom0 and Xen can synchronize on the segment number.
>
> I was hoping we could write down the assumption somewhere that for the
> cases we care about domain == segment, and error out if it is not the
> case.
Given that we have only the domain in hand, how would you enforce that?
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit
concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
really represent the segment?
Cheers,
[1] https://elixir.bootlin.com/linux/latest/source/arch/arm64/kernel/pci.c#L74
Cheers,
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 19:24 ` Julien Grall
@ 2020-07-24 23:46 ` Stefano Stabellini
2020-07-25 9:59 ` Julien Grall
2020-07-26 7:01 ` Jan Beulich
0 siblings, 2 replies; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-24 23:46 UTC (permalink / raw)
To: Julien Grall
Cc: nd, Stefano Stabellini, Andrew Cooper, Bertrand Marquis,
Jan Beulich, xen-devel, Rahul Singh, Volodymyr Babchuk,
Roger Pau Monné
On Fri, 24 Jul 2020, Julien Grall wrote:
> On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > If they are not equal, then I fail to see why it would be useful to have this
> > > value in Xen.
> >
> > I think that's because the domain is actually more convenient to use
> > because a segment can span multiple PCI host bridges. So my
> > understanding is that a segment alone is not sufficient to identify a
> > host bridge. From a software implementation point of view it would be
> > better to use domains.
>
> AFAICT, this would be a matter of one check vs two checks in Xen :).
> But... looking at Linux, they will also use domain == segment for ACPI
> (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
>
> > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > > Dom0 and Xen can synchronize on the segment number.
> >
> > I was hoping we could write down the assumption somewhere that for the
> > cases we care about domain == segment, and error out if it is not the
> > case.
>
> Given that we have only the domain in hand, how would you enforce that?
>
> >From this discussion, it also looks like there is a mismatch between the
> implementation and the understanding on QEMU devel. So I am a bit
> concerned that this is not stable and may change in future Linux version.
>
> IOW, we are know tying Xen to Linux. So could we implement
> PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
> really represent the segment?
I don't think we are tying Xen to Linux. Rob has already said that
linux,pci-domain is basically a generic device tree property. And if we
look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
"PCI domain" is described and seems to match the Linux definition.
I do think we need to understand the definitions and the differences.
Reading online [1][2] it looks like a Linux PCI domain matches a "PCI
Segment Group Number" in PCI Express which is probably why Linux is
making the assumption that it is making.
So maybe it is OK to use domains == segments, but we need to verify this
in the specs and also clarify the terminology we use in a doc for our
own sanity -- I am hoping that Rahul can come up with a good
explanation on the topic :-)
[1] https://stackoverflow.com/questions/49050847/how-is-pci-segmentdomain-related-to-multiple-host-bridgesor-root-bridges
[2] https://wiki.osdev.org/PCI_Express
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 23:46 ` Stefano Stabellini
@ 2020-07-25 9:59 ` Julien Grall
2020-07-27 11:06 ` Roger Pau Monné
2020-07-26 7:01 ` Jan Beulich
1 sibling, 1 reply; 48+ messages in thread
From: Julien Grall @ 2020-07-25 9:59 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Rahul Singh, Andrew Cooper, Bertrand Marquis, Jan Beulich,
xen-devel, nd, Volodymyr Babchuk, Roger Pau Monné
On Sat, 25 Jul 2020 at 00:46, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Fri, 24 Jul 2020, Julien Grall wrote:
> > On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > > If they are not equal, then I fail to see why it would be useful to have this
> > > > value in Xen.
> > >
> > > I think that's because the domain is actually more convenient to use
> > > because a segment can span multiple PCI host bridges. So my
> > > understanding is that a segment alone is not sufficient to identify a
> > > host bridge. From a software implementation point of view it would be
> > > better to use domains.
> >
> > AFAICT, this would be a matter of one check vs two checks in Xen :).
> > But... looking at Linux, they will also use domain == segment for ACPI
> > (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
> >
> > > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > > > Dom0 and Xen can synchronize on the segment number.
> > >
> > > I was hoping we could write down the assumption somewhere that for the
> > > cases we care about domain == segment, and error out if it is not the
> > > case.
> >
> > Given that we have only the domain in hand, how would you enforce that?
> >
> > >From this discussion, it also looks like there is a mismatch between the
> > implementation and the understanding on QEMU devel. So I am a bit
> > concerned that this is not stable and may change in future Linux version.
> >
> > IOW, we are know tying Xen to Linux. So could we implement
> > PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
> > really represent the segment?
>
> I don't think we are tying Xen to Linux. Rob has already said that
> linux,pci-domain is basically a generic device tree property.
My concern is not so much the name of the property, but the definition of it.
AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)
> And if we
> look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
> "PCI domain" is described and seems to match the Linux definition.
>
> I do think we need to understand the definitions and the differences.
+1
> Reading online [1][2] it looks like a Linux PCI domain matches a "PCI
> Segment Group Number" in PCI Express which is probably why Linux is
> making the assumption that it is making.
>
> So maybe it is OK to use domains == segments, but we need to verify this
> in the specs and also clarify the terminology we use in a doc for our
> own sanity -- I am hoping that Rahul can come up with a good
> explanation on the topic :-)
I am a bit confused.... You were the one arguing that domain ==
(segment, bus) in this thread. So may I ask why the interpretation
wouldn't be valid anymore?
Cheers,
> [1] https://stackoverflow.com/questions/49050847/how-is-pci-segmentdomain-related-to-multiple-host-bridgesor-root-bridges
> [2] https://wiki.osdev.org/PCI_Express
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 23:46 ` Stefano Stabellini
2020-07-25 9:59 ` Julien Grall
@ 2020-07-26 7:01 ` Jan Beulich
1 sibling, 0 replies; 48+ messages in thread
From: Jan Beulich @ 2020-07-26 7:01 UTC (permalink / raw)
To: Stefano Stabellini, Julien Grall
Cc: Rahul Singh, Andrew Cooper, Bertrand Marquis, xen-devel, nd,
Volodymyr Babchuk, Roger Pau Monné
On 25.07.2020 01:46, Stefano Stabellini wrote:
> On Fri, 24 Jul 2020, Julien Grall wrote:
>> On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>> If they are not equal, then I fail to see why it would be useful to have this
>>>> value in Xen.
>>>
>>> I think that's because the domain is actually more convenient to use
>>> because a segment can span multiple PCI host bridges. So my
>>> understanding is that a segment alone is not sufficient to identify a
>>> host bridge. From a software implementation point of view it would be
>>> better to use domains.
>>
>> AFAICT, this would be a matter of one check vs two checks in Xen :).
>> But... looking at Linux, they will also use domain == segment for ACPI
>> (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
>>
>>>> In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
>>>> Dom0 and Xen can synchronize on the segment number.
>>>
>>> I was hoping we could write down the assumption somewhere that for the
>>> cases we care about domain == segment, and error out if it is not the
>>> case.
>>
>> Given that we have only the domain in hand, how would you enforce that?
>>
>> >From this discussion, it also looks like there is a mismatch between the
>> implementation and the understanding on QEMU devel. So I am a bit
>> concerned that this is not stable and may change in future Linux version.
>>
>> IOW, we are know tying Xen to Linux. So could we implement
>> PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
>> really represent the segment?
>
> I don't think we are tying Xen to Linux. Rob has already said that
> linux,pci-domain is basically a generic device tree property. And if we
> look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
> "PCI domain" is described and seems to match the Linux definition.
>
> I do think we need to understand the definitions and the differences.
> Reading online [1][2] it looks like a Linux PCI domain matches a "PCI
> Segment Group Number" in PCI Express which is probably why Linux is
> making the assumption that it is making.
If I may, I'd like to put the question a little differently, in the hope
for me to understand the actual issue here: On the x86 side, by way of
using ACPI, Linux and Xen "naturally" agree on segment numbering (as far
as normal devices go; Intel's Volume Management Device concept still
needs accommodating so that it would work with Xen). This includes the
multiple host bridges case then naturally. How is the Device Tree model
different from ACPI?
Jan
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
2020-07-23 20:44 ` Stefano Stabellini
2020-07-24 7:14 ` Oleksandr Andrushchenko
2020-07-24 14:49 ` Roger Pau Monné
@ 2020-07-27 8:40 ` Rahul Singh
2 siblings, 0 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-27 8:40 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, andrew.cooper3, Bertrand Marquis, jbeulich,
xen-devel, nd, Volodymyr Babchuk, roger.pau
Sorry for the late reply.
> On 23 Jul 2020, at 9:44 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 23 Jul 2020, Rahul Singh wrote:
>> Hardware domain is in charge of doing the PCI enumeration and will
>> discover the PCI devices and then will communicate to XEN via hyper
>> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
>>
>> Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
>
> Same question about Change-Id
I think by-mistake Gerrit Change-id is added in the patch series. I will remove the Change-Id in next version of the patch.
>
>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
>> 1 file changed, 39 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
>> index e91355fe22..274720f98a 100644
>> --- a/xen/arch/arm/physdev.c
>> +++ b/xen/arch/arm/physdev.c
>> @@ -9,12 +9,48 @@
>> #include <xen/errno.h>
>> #include <xen/sched.h>
>> #include <asm/hypercall.h>
>> -
>> +#include <xen/guest_access.h>
>> +#include <xsm/xsm.h>
>>
>> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>> {
>> - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>> - return -ENOSYS;
>> + int ret = 0;
>> +
>> + switch ( cmd )
>> + {
>> +#ifdef CONFIG_HAS_PCI
>> + case PHYSDEVOP_pci_device_add:
>> + {
>> + struct physdev_pci_device_add add;
>> + struct pci_dev_info pdev_info;
>> + nodeid_t node = NUMA_NO_NODE;
>> +
>> + ret = -EFAULT;
>> + if ( copy_from_guest(&add, arg, 1) != 0 )
>> + break;
>> +
>> + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
>> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
>> + {
>> + pdev_info.is_virtfn = 1;
>> + pdev_info.physfn.bus = add.physfn.bus;
>> + pdev_info.physfn.devfn = add.physfn.devfn;
>> + }
>> + else
>> + pdev_info.is_virtfn = 0;
>> +
>> + ret = pci_add_device(add.seg, add.bus, add.devfn,
>> + &pdev_info, node);
>> +
>> + break;
>> + }
>> +#endif
>> + default:
>> + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>> + ret = -ENOSYS;
>> + }
>
> I think we should make the implementation common between arm and x86 by
> creating xen/common/physdev.c:do_physdev_op as a shared entry point for
> PHYSDEVOP hypercalls implementations. See for instance:
>
> xen/common/sysctl.c:do_sysctl
>
> and
>
> xen/arch/arm/sysctl.c:arch_do_sysctl
> xen/arch/x86/sysctl.c:arch_do_sysctl
>
Ok sure I will check if we can create a common entry for ARM and x86 for do_physdev_op().
> Jan, Andrew, Roger, any opinions?
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 16:54 ` Julien Grall
@ 2020-07-27 10:34 ` Roger Pau Monné
0 siblings, 0 replies; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-27 10:34 UTC (permalink / raw)
To: Julien Grall
Cc: Rahul Singh, Bertrand.Marquis, Stefano Stabellini, Jan Beulich,
xen-devel, nd, Volodymyr Babchuk
On Fri, Jul 24, 2020 at 05:54:20PM +0100, Julien Grall wrote:
> Hi Jan,
>
> On 24/07/2020 17:01, Jan Beulich wrote:
> > On 24.07.2020 17:15, Julien Grall wrote:
> > > On 24/07/2020 15:44, Roger Pau Monné wrote:
> > > > > +
> > > > > + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> > > > > +
> > > > > + if ( unlikely(!bridge) )
> > > > > + {
> > > > > + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> > > > > + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> > > >
> > > > I had a patch to add a custom modifier to out printf format in
> > > > order to handle pci_sbdf_t natively:
> > > >
> > > > https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
> > > >
> > > > It missed maintainers Acks and was never committed. Since you are
> > > > doing a bunch of work here, and likely adding a lot of SBDF related
> > > > prints, feel free to import the modifier (%pp) and use in your code
> > > > (do not attempt to switch existing users, or it's likely to get
> > > > stuck again).
> > >
> > > I forgot about this patch :/. It would be good to revive it. Which acks
> > > are you missing?
> >
> > It wasn't so much missing acks, but a controversy. And that not so much
> > about switching existing users, but whether to indeed derive this from
> > %p (which I continue to consider inefficient).
>
> Looking at the thread, I can see you (relunctantly) acked any components
> that you are the sole maintainers. Kevin gave his acked for the vtd code and
> I gave it mine for the common code.
>
> I would suggest to not rehash the argument unless another maintainer agree
> with your position. It loosk like to me the next step is for Roger (or
> someone else) to resend the patch so we could collect the missing ack (I
> think there is only one missing from Andrew).
I've rebased and sent the updated patch with the collected Acks.
Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-25 9:59 ` Julien Grall
@ 2020-07-27 11:06 ` Roger Pau Monné
2020-07-28 0:06 ` Stefano Stabellini
0 siblings, 1 reply; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-27 11:06 UTC (permalink / raw)
To: Julien Grall
Cc: nd, Stefano Stabellini, Andrew Cooper, Bertrand Marquis,
Jan Beulich, xen-devel, Rahul Singh, Volodymyr Babchuk
On Sat, Jul 25, 2020 at 10:59:50AM +0100, Julien Grall wrote:
> On Sat, 25 Jul 2020 at 00:46, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >
> > On Fri, 24 Jul 2020, Julien Grall wrote:
> > > On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > > > If they are not equal, then I fail to see why it would be useful to have this
> > > > > value in Xen.
> > > >
> > > > I think that's because the domain is actually more convenient to use
> > > > because a segment can span multiple PCI host bridges. So my
> > > > understanding is that a segment alone is not sufficient to identify a
> > > > host bridge. From a software implementation point of view it would be
> > > > better to use domains.
> > >
> > > AFAICT, this would be a matter of one check vs two checks in Xen :).
> > > But... looking at Linux, they will also use domain == segment for ACPI
> > > (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
You have to use the (segment, bus) tuple when doing a lookup because
MMCFG regions on ACPI are defined for a segment and a bus range, you
can have a MMCFG region that covers segment 0 bus [0, 20) and another
MMCFG region that covers segment 0 bus [20, 255], and those will use
different addresses in the MMIO space.
> > > > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > > > > Dom0 and Xen can synchronize on the segment number.
> > > >
> > > > I was hoping we could write down the assumption somewhere that for the
> > > > cases we care about domain == segment, and error out if it is not the
> > > > case.
> > >
> > > Given that we have only the domain in hand, how would you enforce that?
> > >
> > > >From this discussion, it also looks like there is a mismatch between the
> > > implementation and the understanding on QEMU devel. So I am a bit
> > > concerned that this is not stable and may change in future Linux version.
> > >
> > > IOW, we are know tying Xen to Linux. So could we implement
> > > PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
> > > really represent the segment?
> >
> > I don't think we are tying Xen to Linux. Rob has already said that
> > linux,pci-domain is basically a generic device tree property.
>
> My concern is not so much the name of the property, but the definition of it.
>
> AFAICT, from this thread there can be two interpretation:
> - domain == segment
> - domain == (segment, bus)
I think domain is just an alias for segment, the difference seems to
be that when using DT all bridges get a different segment (or domain)
number, and thus you will always end up starting numbering at bus 0
for each bridge?
Ideally you would need a way to specify the segment and start/end bus
numbers of each bridge, if not you cannot match what ACPI does. Albeit
it might be fine as long as the OS and Xen agree on the segments and
bus numbers that belong to each bridge (and thus each ECAM region).
Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-23 23:38 ` Stefano Stabellini
2020-07-24 7:03 ` Oleksandr Andrushchenko
2020-07-24 8:44 ` Julien Grall
@ 2020-07-27 13:27 ` Rahul Singh
2 siblings, 0 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-27 13:27 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, andrew.cooper3, Bertrand Marquis, jbeulich,
xen-devel, nd, Volodymyr Babchuk, roger.pau
> On 24 Jul 2020, at 12:38 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> + Jan, Andrew, Roger
>
> Please have a look at my comment on whether we should share the MMCFG
> code below, feel free to ignore the rest :-)
>
>
> On Thu, 23 Jul 2020, Rahul Singh wrote:
>> XEN during boot will read the PCI device tree node “reg” property
>> and will map the PCI config space to the XEN memory.
>>
>> XEN will read the “linux, pci-domain” property from the device tree
>> node and configure the host bridge segment number accordingly.
>>
>> As of now "pci-host-ecam-generic" compatible board is supported.
>>
>> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
>
> What is this Change-Id property?
Will remove this in next patch series.
>
>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/Kconfig | 7 +
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/pci/Makefile | 4 +
>> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
>> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
>> xen/arch/arm/setup.c | 2 +
>> xen/include/asm-arm/device.h | 7 +-
>> xen/include/asm-arm/pci.h | 97 +++++++++++++-
>> 10 files changed, 654 insertions(+), 6 deletions(-)
>> create mode 100644 xen/arch/arm/pci/Makefile
>> create mode 100644 xen/arch/arm/pci/pci-access.c
>> create mode 100644 xen/arch/arm/pci/pci-host-common.c
>> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
>> create mode 100644 xen/arch/arm/pci/pci.c
>>
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 2777388265..ee13339ae9 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -31,6 +31,13 @@ menu "Architecture Features"
>>
>> source "arch/Kconfig"
>>
>> +config ARM_PCI
>> + bool "PCI Passthrough Support"
>> + depends on ARM_64
>> + ---help---
>> +
>> + PCI passthrough support for Xen on ARM64.
>> +
>> config ACPI
>> bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
>> depends on ARM_64
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 7e82b2178c..345cb83eed 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_ARM_PCI) += pci/
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> new file mode 100644
>> index 0000000000..358508b787
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -0,0 +1,4 @@
>> +obj-y += pci.o
>> +obj-y += pci-host-generic.o
>> +obj-y += pci-host-common.o
>> +obj-y += pci-access.o
>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>> new file mode 100644
>> index 0000000000..c53ef58336
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -0,0 +1,101 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +#include <xen/rwlock.h>
>> +
>> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>> + unsigned int len)
>> +{
>> + int rc;
>> + uint32_t val = GENMASK(0, len * 8);
>> +
>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> + if ( unlikely(!bridge) )
>> + {
>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> + return val;
>> + }
>> +
>> + if ( unlikely(!bridge->ops->read) )
>> + return val;
>> +
>> + rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
>> + if ( rc )
>> + printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> +
>> + return val;
>> +}
>> +
>> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>> + unsigned int len, uint32_t val)
>> +{
>> + int rc;
>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> + if ( unlikely(!bridge) )
>> + {
>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> + return;
>> + }
>> +
>> + if ( unlikely(!bridge->ops->write) )
>> + return;
>> +
>> + rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
>> + if ( rc )
>> + printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> +}
>> +
>> +/*
>> + * Wrappers for all PCI configuration access functions.
>> + */
>> +
>> +#define PCI_OP_WRITE(size, type) \
>> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
>> +{ \
>> + pci_config_write(sbdf, reg, size / 8, val); \
>> +}
>> +
>> +#define PCI_OP_READ(size, type) \
>> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
>> +{ \
>> + return pci_config_read(sbdf, reg, size / 8); \
>> +}
>> +
>> +PCI_OP_READ(8, u8)
>> +PCI_OP_READ(16, u16)
>> +PCI_OP_READ(32, u32)
>> +PCI_OP_WRITE(8, u8)
>> +PCI_OP_WRITE(16, u16)
>> +PCI_OP_WRITE(32, u32)
>
> This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?
>
> MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not
> mistaken. Is there any value in sharing the code with x86? It is OK if
> we don't, but I would like to understand the reasoning.
We have just started to investigate the ACPI support for PCI within XEN on ARM. Once we have understanding how we can support the ACPI we will have a look x86 MMCFG access and will check if we can have a common interface to access the configuration space.
>
>
>
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
>> new file mode 100644
>> index 0000000000..c5f98be698
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-common.c
>> @@ -0,0 +1,198 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +#include <xen/rwlock.h>
>> +#include <xen/vmap.h>
>> +
>> +/*
>> + * List for all the pci host bridges.
>> + */
>> +
>> +static LIST_HEAD(pci_host_bridges);
>> +
>> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
>> + struct pci_config_window *cfg)
>> +{
>> + const __be32 *cells;
>> + uint32_t len;
>> +
>> + cells = dt_get_property(dev, "bus-range", &len);
>> + /* bus-range should at least be 2 cells */
>> + if ( !cells || (len < (sizeof(*cells) * 2)) )
>> + return false;
>> +
>> + cfg->busn_start = dt_next_cell(1, &cells);
>> + cfg->busn_end = dt_next_cell(1, &cells);
>> +
>> + return true;
>> +}
>> +
>> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
>> +{
>> + return ioremap_nocache(start, len);
>> +}
>> +
>> +static void pci_ecam_free(struct pci_config_window *cfg)
>> +{
>> + if ( cfg->win )
>> + iounmap(cfg->win);
>> +
>> + xfree(cfg);
>> +}
>> +
>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops)
>> +{
>> + int err;
>> + struct pci_config_window *cfg;
>> + paddr_t addr, size;
>> +
>> + cfg = xzalloc(struct pci_config_window);
>> + if ( !cfg )
>> + return NULL;
>> +
>> + err = dt_pci_parse_bus_range(dev, cfg);
>> + if ( !err ) {
>> + cfg->busn_start = 0;
>> + cfg->busn_end = 0xff;
>> + printk(XENLOG_ERR "No bus range found for pci controller\n");
>> + } else {
>> + if ( cfg->busn_end > cfg->busn_start + 0xff )
>> + cfg->busn_end = cfg->busn_start + 0xff;
>> + }
>> +
>> + /* Parse our PCI ecam register address*/
>> + err = dt_device_get_address(dev, 0, &addr, &size);
>> + if ( err )
>> + goto err_exit;
>
> Shouldn't we handle the possibility of multiple addresses? Is it
> possible to have more than one range for an ECAM compliant host bridge?
As of now we are supporting the PCI controllers that have "pci-host-ecam-generic” compatible node in device tree binding.As per my understanding controller that have "pci-host-ecam-generic” compatible property have only one reg property value at index 0.
Yes there are PCI controllers that support ECAM and have more than one "reg" property values.As per my understanding once we support other controller in XEN at that time we have to handle the possibility of multiple addresses and that code is specific to that controller .
>
>
>> + cfg->phys_addr = addr;
>> + cfg->size = size;
>> + cfg->ops = ops;
>> +
>> + /*
>> + * On 64-bit systems, we do a single ioremap for the whole config space
>> + * since we have enough virtual address range available. On 32-bit, we
>> + * ioremap the config space for each bus individually.
>> + *
>> + * As of now only 64-bit is supported 32-bit is not supported.
>> + */
>> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
>> + if ( !cfg->win )
>> + goto err_exit_remap;
>> +
>> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>> + cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
>> +
>> + if ( ops->init ) {
>> + err = ops->init(cfg);
>> + if (err)
>> + goto err_exit;
>> + }
>> +
>> + return cfg;
>> +
>> +err_exit_remap:
>> + printk(XENLOG_ERR "ECAM ioremap failed\n");
>> +err_exit:
>> + pci_ecam_free(cfg);
>> + return NULL;
>> +}
>> +
>> +static struct pci_host_bridge * pci_alloc_host_bridge(void)
>> +{
>> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
>> +
>> + if ( !bridge )
>> + return NULL;
>> +
>> + INIT_LIST_HEAD(&bridge->node);
>> + return bridge;
>> +}
>> +
>> +int pci_host_common_probe(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops)
>> +{
>> + struct pci_host_bridge *bridge;
>> + struct pci_config_window *cfg;
>> + u32 segment;
>> +
>> + bridge = pci_alloc_host_bridge();
>> + if ( !bridge )
>> + return -ENOMEM;
>> +
>> + /* Parse and map our Configuration Space windows */
>> + cfg = gen_pci_init(dev, ops);
>> + if ( !cfg )
>> + return -ENOMEM;
>
> In case of errors the allocated bridge is not freed.
>
Will fix this in next version.
>
>> + bridge->dt_node = dev;
>> + bridge->sysdata = cfg;
>> + bridge->ops = &ops->pci_ops;
>> +
>> + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
>> + {
>> + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
>> + return -ENODEV;
>> + }
>> +
>> + bridge->segment = (u16)segment;
>
> My understanding is that a Linux pci-domain doesn't correspond exactly
> to a PCI segment. See for instance:
>
> https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
>
> Do we need to care about the difference? If we mean pci-domain here,
> should we just call them as such instead of calling them "segments" in
> Xen (if they are not segments)?
>
As per my understanding linux domain number is an alias to segment number.Device tree binding uses the property domain number and ACPI uses the property segment number.
We have just started the investigation on ACPI, once we have more information about MMCFG ACPI table how it works will share the information and will update the patches and design doc accordingly.
>
>> + list_add_tail(&bridge->node, &pci_host_bridges);
>
> It looks like &pci_host_bridges should be an ordered list, ordered by
> segment number?
>
As Julien also mentioned access to the PCI config space is random there is no need to have an ordered list.
>
>> + return 0;
>> +}
>> +
>> +/*
>> + * This function will lookup an hostbridge based on the segment and bus
>> + * number.
>> + */
>> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
>> +{
>> + struct pci_host_bridge *bridge;
>> + bool found = false;
>> +
>> + list_for_each_entry( bridge, &pci_host_bridges, node )
>> + {
>> + if ( bridge->segment != segment )
>> + continue;
>> +
>> + found = true;
>> + break;
>> + }
>> +
>> + return (found) ? bridge : NULL;
>> +}
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
>> new file mode 100644
>> index 0000000000..cd67b3dec6
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-generic.c
>> @@ -0,0 +1,131 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/device.h>
>> +#include <asm/io.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +
>> +/*
>> + * Function to get the config space base.
>> + */
>> +static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where)
>
> I think the function is misnamed because reading the code below it looks
> like it is not just returning the base config space address but also the
> specific address we need to read/write (adding the device offset,
> "where", and everything).
>
> Maybe pci_config_get_address or something like that?
OK yes will rename the function name.
>
>
>> +{
>> + struct pci_config_window *cfg = bridge->sysdata;
>> + unsigned int devfn_shift = cfg->ops->bus_shift - 8;
>> +
>> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
>> +
>> + unsigned int busn = sbdf_t.bus;
>> + void __iomem *base;
>> +
>> + if ( busn < cfg->busn_start || busn > cfg->busn_end )
>> + return NULL;
>> +
>> + base = cfg->win + (busn << cfg->ops->bus_shift);
>> +
>> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
>> +}
>> +
>> +int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>> + int where, int size, u32 val)
>> +{
>> + void __iomem *addr;
>> +
>> + addr = pci_config_base(bridge, sbdf, where);
>> + if ( !addr )
>> + return -ENODEV;
>> +
>> + if ( size == 1 )
>> + writeb(val, addr);
>> + else if ( size == 2 )
>> + writew(val, addr);
>> + else
>> + writel(val, addr);
>
> please use a switch
>
>
>> + return 0;
>> +}
>> +
>> +int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
>> + int where, int size, u32 *val)
>> +{
>> + void __iomem *addr;
>> +
>> + addr = pci_config_base(bridge, sbdf, where);
>> + if ( !addr ) {
>> + *val = ~0;
>> + return -ENODEV;
>> + }
>> +
>> + if ( size == 1 )
>> + *val = readb(addr);
>> + else if ( size == 2 )
>> + *val = readw(addr);
>> + else
>> + *val = readl(addr);
>
> please use a switch
Ok.
>
>
>> + return 0;
>> +}
>> +
>> +/* ECAM ops */
>> +struct pci_ecam_ops pci_generic_ecam_ops = {
>> + .bus_shift = 20,
>> + .pci_ops = {
>> + .read = pci_ecam_config_read,
>> + .write = pci_ecam_config_write,
>> + }
>> +};
>> +
>> +static const struct dt_device_match gen_pci_dt_match[] = {
>> + { .compatible = "pci-host-ecam-generic",
>> + .data = &pci_generic_ecam_ops },
>
> spurious blank line
Ok will fix this.
>
>
>> + { },
>> +};
>> +
>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>> +{
>> + const struct dt_device_match *of_id;
>> + struct pci_ecam_ops *ops;
>> +
>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>> + ops = (struct pci_ecam_ops *) of_id->data;
>> +
>> + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>> + dt_node_full_name(dev), of_id->compatible);
>> +
>> + return pci_host_common_probe(dev, ops);
>> +}
>> +
>> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
>> +.dt_match = gen_pci_dt_match,
>> +.init = gen_pci_dt_init,
>> +DT_DEVICE_END
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>> new file mode 100644
>> index 0000000000..f8cbb99591
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci.c
>> @@ -0,0 +1,112 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/acpi.h>
>> +#include <xen/device_tree.h>
>> +#include <xen/errno.h>
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <xen/param.h>
>> +
>> +static int __init dt_pci_init(void)
>> +{
>> + struct dt_device_node *np;
>> + int rc;
>> +
>> + dt_for_each_device_node(dt_host, np)
>> + {
>> + rc = device_init(np, DEVICE_PCI, NULL);
>> + if( !rc )
>> + continue;
>> + /*
>> + * Ignore the following error codes:
>> + * - EBADF: Indicate the current is not an pci
>> + * - ENODEV: The pci device is not present or cannot be used by
>> + * Xen.
>> + */
>> + else if ( rc != -EBADF && rc != -ENODEV )
>> + {
>> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
>> + return rc;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +#ifdef CONFIG_ACPI
>> +static void __init acpi_pci_init(void)
>> +{
>> + printk(XENLOG_ERR "ACPI pci init not supported \n");
>> + return;
>> +}
>> +#else
>> +static inline void __init acpi_pci_init(void) { }
>> +#endif
>> +
>> +static bool __initdata param_pci_enable;
>> +static int __init parse_pci_param(const char *arg)
>> +{
>> + if ( !arg )
>> + {
>> + param_pci_enable = false;
>> + return 0;
>> + }
>> +
>> + switch ( parse_bool(arg, NULL) )
>> + {
>> + case 0:
>> + param_pci_enable = false;
>> + return 0;
>> + case 1:
>> + param_pci_enable = true;
>> + return 0;
>> + }
>> +
>> + return -EINVAL;
>> +}
>> +custom_param("pci", parse_pci_param);
>
> When adding new command line parameters please also add its
> documentation (docs/misc/xen-command-line.pandoc) in the same patch,
> unless this is meant to be just transient and we'll get removed before
> the final commit of the series.
>
Ok yes will add it to the documentation.
>
>> +void __init pci_init(void)
>> +{
>> + /*
>> + * Enable PCI when has been enabled explicitly (pci=on)
>> + */
>> + if ( !param_pci_enable)
>> + goto disable;
>> +
>> + if ( acpi_disabled )
>> + dt_pci_init();
>> + else
>> + acpi_pci_init();
>> +
>> +#ifdef CONFIG_HAS_PCI
>> + pci_segments_init();
>> +#endif
>> +
>> +disable:
>> + return;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>> index 7968cee47d..2d7f1db44f 100644
>> --- a/xen/arch/arm/setup.c
>> +++ b/xen/arch/arm/setup.c
>> @@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>>
>> setup_virt_paging();
>>
>> + pci_init();
>
> pci_init should probably be an initcall.
OK.
>
>
>> do_initcalls();
>>
>> /*
>> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
>> index ee7cff2d44..28f8049cfd 100644
>> --- a/xen/include/asm-arm/device.h
>> +++ b/xen/include/asm-arm/device.h
>> @@ -4,6 +4,7 @@
>> enum device_type
>> {
>> DEV_DT,
>> + DEV_PCI,
>> };
>>
>> struct dev_archdata {
>> @@ -25,15 +26,15 @@ typedef struct device device_t;
>>
>> #include <xen/device_tree.h>
>>
>> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
>> -#define dev_is_pci(dev) ((void)(dev), 0)
>> -#define dev_is_dt(dev) ((dev->type == DEV_DT)
>> +#define dev_is_pci(dev) (dev->type == DEV_PCI)
>> +#define dev_is_dt(dev) (dev->type == DEV_DT)
>>
>> enum device_class
>> {
>> DEVICE_SERIAL,
>> DEVICE_IOMMU,
>> DEVICE_GIC,
>> + DEVICE_PCI,
>> /* Use for error */
>> DEVICE_UNKNOWN,
>> };
>> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>> index de13359f65..94fd00360a 100644
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -1,7 +1,98 @@
>> -#ifndef __X86_PCI_H__
>> -#define __X86_PCI_H__
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>>
>> +#ifndef __ARM_PCI_H__
>> +#define __ARM_PCI_H__
>> +
>> +#include <xen/pci.h>
>> +#include <xen/device_tree.h>
>> +#include <asm/device.h>
>> +
>> +#ifdef CONFIG_ARM_PCI
>> +
>> +/* Arch pci dev struct */
>> struct arch_pci_dev {
>> + struct device dev;
>> +};
>
> Are you actually using struct device in struct arch_pci_dev?
> struct device is already part of struct dt_device_node and a pointer to
> it is stored in bridge->dt_node.
Will be using this going forward once we have full PCI passthrough support.
>
>
>> +#define PRI_pci "%04x:%02x:%02x.%u"
>> +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
>> +
>> +/*
>> + * struct to hold the mappings of a config space window. This
>> + * is expected to be used as sysdata for PCI controllers that
>> + * use ECAM.
>> + */
>> +struct pci_config_window {
>> + paddr_t phys_addr;
>> + paddr_t size;
>> + uint8_t busn_start;
>> + uint8_t busn_end;
>> + struct pci_ecam_ops *ops;
>> + void __iomem *win;
>> +};
>> +
>> +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
>> +struct pci_host_bridge;
>> +
>> +struct pci_ops {
>> + int (*read)(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where, int size, u32 *val);
>> + int (*write)(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where, int size, u32 val);
>
> I'd prefer if we could use explicitly-sized integers for "where" and
> "size" too. Also, should they be unsigned rather than signed?
>
> Can we use pci_sbdf_t for the sbdf parameter?
Ok will fix in next version.
>
>
>> +};
>> +
>> +/*
>> + * struct to hold pci ops and bus shift of the config window
>> + * for a PCI controller.
>> + */
>> +struct pci_ecam_ops {
>> + unsigned int bus_shift;
>> + struct pci_ops pci_ops;
>> + int (*init)(struct pci_config_window *);
>> +};
>
> Although I realize that we are only targeting ECAM now, and the
> implementation is based on ECAM, the interface doesn't seem to have
> anything ECAM-specific in it. I would rename pci_ecam_ops to something
> else, maybe simply "pci_ops".
Ok will have a look and modify accordingly.
>
>
>> +/*
>> + * struct to hold pci host bridge information
>> + * for a PCI controller.
>> + */
>> +struct pci_host_bridge {
>> + struct dt_device_node *dt_node; /* Pointer to the associated DT node */
>> + struct list_head node; /* Node in list of host bridges */
>> + uint16_t segment; /* Segment number */
>> + void *sysdata; /* Pointer to the config space window*/
>> + const struct pci_ops *ops;
>> };
>>
>> -#endif /* __X86_PCI_H__ */
>> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
>> +
>> +int pci_host_common_probe(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops);
>> +
>> +void pci_init(void);
>> +
>> +#else /*!CONFIG_ARM_PCI*/
>> +struct arch_pci_dev { };
>> +static inline void pci_init(void) { }
>> +#endif /*!CONFIG_ARM_PCI*/
>> +#endif /* __ARM_PCI_H__ */
>> --
>> 2.17.1
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
2020-07-23 23:39 ` Stefano Stabellini
2020-07-24 7:55 ` Oleksandr Andrushchenko
2020-07-24 9:11 ` Julien Grall
@ 2020-07-27 13:40 ` Rahul Singh
2 siblings, 0 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-27 13:40 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, Wei Liu, Ian Jackson, Bertrand Marquis,
Anthony PERARD, xen-devel, nd, Volodymyr Babchuk
> On 24 Jul 2020, at 12:39 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 23 Jul 2020, Rahul Singh wrote:
>> libxl will create an emulated PCI device tree node in the
>> device tree to enable the guest OS to discover the virtual
>> PCI during guest boot.
>>
>> We introduced the new config option [vpci="ecam"] for guests.
>> When this config option is enabled in a guest configuration,
>> a PCI device tree node will be created in the guest device tree.
>>
>> A new area has been reserved in the arm guest physical map at
>> which the VPCI bus is declared in the device tree (reg and ranges
>> parameters of the node).
>>
>> Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd
>
> Same question
I will remove the change-id in the next version.
>
>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++
>> tools/libxl/libxl_types.idl | 6 +
>> tools/xl/xl_parse.c | 7 ++
>> xen/include/public/arch-arm.h | 28 +++++
>> 4 files changed, 241 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
>> index 34f8a29056..84568e9dc9 100644
>> --- a/tools/libxl/libxl_arm.c
>> +++ b/tools/libxl/libxl_arm.c
>> @@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>> return fdt_property(fdt, "reg", regs, sizeof(regs));
>> }
>>
>> +static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
>> + unsigned num_cells, ...)
>> +{
>> + uint32_t bus_range[num_cells];
>> + be32 *cells = &bus_range[0];
>> + int i;
>> + va_list ap;
>> + uint32_t arg;
>> +
>> + va_start(ap, num_cells);
>> + for (i = 0 ; i < num_cells; i++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1, arg);
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
>> +}
>> +
>> +static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
>> + unsigned num_cells, ...)
>> +{
>> + uint32_t interrupt_map_mask[num_cells];
>> + be32 *cells = &interrupt_map_mask[0];
>> + int i;
>> + va_list ap;
>> + uint32_t arg;
>> +
>> + va_start(ap, num_cells);
>> + for (i = 0 ; i < num_cells; i++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1, arg);
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
>> + sizeof(interrupt_map_mask));
>> +}
>> +
>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
>> + unsigned vpci_addr_cells,
>> + unsigned cpu_addr_cells,
>> + unsigned vpci_size_cells,
>> + unsigned num_regs, ...)
>> +{
>> + uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
>> + be32 *cells = ®s[0];
>> + int i;
>> + va_list ap;
>> + uint64_t arg;
>> +
>> + va_start(ap, num_regs);
>> + for (i = 0 ; i < num_regs; i++) {
>> + /* Set the memory bit field */
>> + arg = va_arg(ap, uint64_t);
>> + set_cell(&cells, 1, arg);
>> +
>> + /* Set the vpci bus address */
>> + arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
>> + set_cell(&cells, 2 , arg);
>> +
>> + /* Set the cpu bus address where vpci address is mapped */
>> + arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
>> + set_cell(&cells, cpu_addr_cells, arg);
>> +
>> + /* Set the vpci size requested */
>> + arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
>> + set_cell(&cells, vpci_size_cells,arg);
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "ranges", regs, sizeof(regs));
>> +}
>> +
>> +static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
>> + unsigned child_unit_addr_cells,
>> + unsigned child_interrupt_specifier_cells,
>> + unsigned parent_unit_addr_cells,
>> + unsigned parent_interrupt_specifier_cells,
>> + unsigned num_regs, ...)
>> +{
>> + uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
>> + child_interrupt_specifier_cells + parent_unit_addr_cells
>> + + parent_interrupt_specifier_cells + 1)];
>> + be32 *cells = &interrupt_map[0];
>> + int i,j;
>> + va_list ap;
>> + uint64_t arg;
>> +
>> + va_start(ap, num_regs);
>> + for (i = 0 ; i < num_regs; i++) {
>> + /* Set the child unit address*/
>> + for (j = 0 ; j < child_unit_addr_cells; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> +
>> + /* Set the child interrupt specifier*/
>> + for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> +
>> + /* Set the interrupt-parent*/
>> + set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
>> +
>> + /* Set the parent unit address*/
>> + for (j = 0 ; j < parent_unit_addr_cells; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> +
>> + /* Set the parent interrupt specifier*/
>> + for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
>> + arg = va_arg(ap, uint32_t);
>> + set_cell(&cells, 1 , arg);
>> + }
>> + }
>> + va_end(ap);
>> +
>> + return fdt_property(fdt, "interrupt-map", interrupt_map,
>> + sizeof(interrupt_map));
>> +}
>> +
>> static int make_root_properties(libxl__gc *gc,
>> const libxl_version_info *vers,
>> void *fdt)
>> @@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>> return 0;
>> }
>>
>> +static int make_vpci_node(libxl__gc *gc, void *fdt,
>> + const struct arch_info *ainfo,
>> + struct xc_dom_image *dom)
>> +{
>> + int res;
>> + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
>> + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
>> + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
>> +
>> + res = fdt_begin_node(fdt, name);
>> + if (res) return res;
>> +
>> + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
>> + if (res) return res;
>> +
>> + res = fdt_property_string(fdt, "device_type", "pci");
>> + if (res) return res;
>> +
>> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>> + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
>> + if (res) return res;
>> +
>> + res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "linux,pci-domain", 0);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "#address-cells", 3);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "#size-cells", 2);
>> + if (res) return res;
>> +
>> + res = fdt_property_cell(fdt, "#interrupt-cells", 1);
>> + if (res) return res;
>> +
>> + res = fdt_property_string(fdt, "status", "okay");
>> + if (res) return res;
>> +
>> + res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
>> + GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,
>> + 3,
>> + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
>> + GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
>> + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
>> + GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
>> + GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
>> + GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
>> + if (res) return res;
>> +
>> + res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);
>
> it would make sense to separate out child_unit_addr_cells and
> child_interrupt_specifier_cells here like we do below with
> fdt_property_vpci_interrupt_map
Ok will fix.
>
>
>> + if (res) return res;
>> +
>> + /*
>> + * Legacy interrupt is forced and assigned to the guest.
>> + * This will be removed once we have implementation for MSI support.
>> + *
>> + */
>> + res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
>> + 4,
>> + 0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
>> + 0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
>
> The 4 interrupt allocated for this need to be defined in
> xen/include/public/arch-arm.h as well. Also, why would we want to get
> rid of the legacy interrupts completely? It would be possible to still
> find device or software that rely on them.
>
Ok will fix that.
Regarding legacy interrupt we have just tested on one of the board don’t know how it will work on other boards.
We will mostly support MSI and will see if we have to support the legacy interrupt going forward.
>
>> + if (res) return res;
>> +
>> + res = fdt_end_node(fdt);
>> + if (res) return res;
>> +
>> + return 0;
>> +}
>> +
>> static const struct arch_info *get_arch_info(libxl__gc *gc,
>> const struct xc_dom_image *dom)
>> {
>
> [...]
>
>
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index 7364a07362..4e19c62948 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)
>> #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
>>
>> +#define GUEST_PCI_ADDRESS_CELLS 3
>> +#define GUEST_PCI_SIZE_CELLS 2
>> +
>> +/* PCI-PCIe memory space types */
>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
>> +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000)
>> +#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000)
>> +
>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
>> +#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000)
>
> It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?
Ok yes will fix that and will define the address ranges for guest once we finalised the VPCI topology for the guest. We are currently investigating if we want to follow the hardware topology for the guest or we will create a different virtual topology for the guest independent of the hw topology.
>
>
>> +#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000)
>> +#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800)
>
> 0xC0200800 looks like it could conflict with
> GUEST_RAM0_BASE+GUEST_RAM0_SIZE?
>
Same comment above.
>
>> +/*
>> + * This is hardcoded values for the real PCI physical addresses.
>> + * This will be removed once we read the real PCI-PCIe physical
>> + * addresses form the config space and map to the guest memory map
>> + * when assigning the device to guest via VPCI.
>> + *
>> + */
>> +#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000)
>> +#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000)
>> +#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000)
>> +
>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000)
>> +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000)
>
> How did you chose these sizes? GUEST_VPCI_MEM_SIZE and/or
> GUEST_VPCI_PREFETCH_MEM_SIZE are supposed to potentially cover all the
> PCI BARs, including potential future hotplug devices, right?
>
> If so, maybe we need to increase GUEST_VPCI_MEM_SIZE to a couple of GB
> and GUEST_VPCI_PREFETCH_MEM_SIZE to even more?
Same comments above.
>
>
>
>
>> +#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000)
>> +
>> /*
>> * 16MB == 4096 pages reserved for guest to use as a region to map its
>> * grant table in.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 7:03 ` Oleksandr Andrushchenko
2020-07-24 8:05 ` Julien Grall
@ 2020-07-27 15:20 ` Rahul Singh
1 sibling, 0 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-27 15:20 UTC (permalink / raw)
To: Oleksandr Andrushchenko
Cc: Stefano Stabellini, Julien Grall, andrew.cooper3,
Bertrand Marquis, jbeulich, xen-devel, nd, Volodymyr Babchuk,
roger.pau
> On 24 Jul 2020, at 8:03 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
>
> On 7/24/20 2:38 AM, Stefano Stabellini wrote:
>> + Jan, Andrew, Roger
>>
>> Please have a look at my comment on whether we should share the MMCFG
>> code below, feel free to ignore the rest :-)
>>
>>
>> On Thu, 23 Jul 2020, Rahul Singh wrote:
>>> XEN during boot will read the PCI device tree node “reg” property
>>> and will map the PCI config space to the XEN memory.
>>>
>>> XEN will read the “linux, pci-domain” property from the device tree
>>> node and configure the host bridge segment number accordingly.
>>>
>>> As of now "pci-host-ecam-generic" compatible board is supported.
>>>
>>> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
>> What is this Change-Id property?
> Gerrit ;)
>>
>>
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> ---
>>> xen/arch/arm/Kconfig | 7 +
>>> xen/arch/arm/Makefile | 1 +
>>> xen/arch/arm/pci/Makefile | 4 +
>>> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
>>> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
>>> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
>>> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
>>> xen/arch/arm/setup.c | 2 +
>>> xen/include/asm-arm/device.h | 7 +-
>>> xen/include/asm-arm/pci.h | 97 +++++++++++++-
>>> 10 files changed, 654 insertions(+), 6 deletions(-)
>>> create mode 100644 xen/arch/arm/pci/Makefile
>>> create mode 100644 xen/arch/arm/pci/pci-access.c
>>> create mode 100644 xen/arch/arm/pci/pci-host-common.c
>>> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
>>> create mode 100644 xen/arch/arm/pci/pci.c
>>>
>>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>>> index 2777388265..ee13339ae9 100644
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -31,6 +31,13 @@ menu "Architecture Features"
>>>
>>> source "arch/Kconfig"
>>>
>>> +config ARM_PCI
>>> + bool "PCI Passthrough Support"
>>> + depends on ARM_64
>>> + ---help---
>>> +
>>> + PCI passthrough support for Xen on ARM64.
>>> +
>>> config ACPI
>>> bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
>>> depends on ARM_64
>>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>>> index 7e82b2178c..345cb83eed 100644
>>> --- a/xen/arch/arm/Makefile
>>> +++ b/xen/arch/arm/Makefile
>>> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>> obj-y += platforms/
>>> endif
>>> obj-$(CONFIG_TEE) += tee/
>>> +obj-$(CONFIG_ARM_PCI) += pci/
>>>
>>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>> obj-y += bootfdt.init.o
>>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>>> new file mode 100644
>>> index 0000000000..358508b787
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/Makefile
>>> @@ -0,0 +1,4 @@
>>> +obj-y += pci.o
>>> +obj-y += pci-host-generic.o
>>> +obj-y += pci-host-common.o
>>> +obj-y += pci-access.o
>>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>>> new file mode 100644
>>> index 0000000000..c53ef58336
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/pci-access.c
>>> @@ -0,0 +1,101 @@
>>> +/*
>>> + * Copyright (C) 2020 Arm Ltd.
> I think SPDX will fit better in any new code.
It will be good if community helps us to decide which license is best for new files.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <xen/init.h>
>>> +#include <xen/pci.h>
>>> +#include <asm/pci.h>
>>> +#include <xen/rwlock.h>
>>> +
>>> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>>> + unsigned int len)
>>> +{
>>> + int rc;
>>> + uint32_t val = GENMASK(0, len * 8);
>>> +
>>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>> +
>>> + if ( unlikely(!bridge) )
>>> + {
>>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>> + return val;
>>> + }
>>> +
>>> + if ( unlikely(!bridge->ops->read) )
>>> + return val;
>>> +
>>> + rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
>>> + if ( rc )
>>> + printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
>>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>> +
>>> + return val;
>>> +}
>>> +
>>> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>>> + unsigned int len, uint32_t val)
>>> +{
>>> + int rc;
>>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>> +
>>> + if ( unlikely(!bridge) )
>>> + {
>>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>> + return;
>>> + }
>>> +
>>> + if ( unlikely(!bridge->ops->write) )
>>> + return;
>>> +
>>> + rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
>>> + if ( rc )
>>> + printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
>>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>> +}
>>> +
>>> +/*
>>> + * Wrappers for all PCI configuration access functions.
>>> + */
>>> +
>>> +#define PCI_OP_WRITE(size, type) \
>>> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
>>> +{ \
>>> + pci_config_write(sbdf, reg, size / 8, val); \
>>> +}
>>> +
>>> +#define PCI_OP_READ(size, type) \
>>> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
>>> +{ \
>>> + return pci_config_read(sbdf, reg, size / 8); \
>>> +}
>>> +
>>> +PCI_OP_READ(8, u8)
>>> +PCI_OP_READ(16, u16)
>>> +PCI_OP_READ(32, u32)
>>> +PCI_OP_WRITE(8, u8)
>>> +PCI_OP_WRITE(16, u16)
>>> +PCI_OP_WRITE(32, u32)
>> This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?
>>
>> MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not
>> mistaken. Is there any value in sharing the code with x86? It is OK if
>> we don't, but I would like to understand the reasoning.
>>
>>
>>
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * tab-width: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
>>> new file mode 100644
>>> index 0000000000..c5f98be698
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/pci-host-common.c
>>> @@ -0,0 +1,198 @@
>>> +/*
>>> + * Copyright (C) 2020 Arm Ltd.
>>> + *
>>> + * Based on Linux drivers/pci/ecam.c
>>> + * Copyright 2016 Broadcom.
>>> + *
>>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>>> + *
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <xen/init.h>
>>> +#include <xen/pci.h>
>>> +#include <asm/pci.h>
>>> +#include <xen/rwlock.h>
>>> +#include <xen/vmap.h>
>>> +
>>> +/*
>>> + * List for all the pci host bridges.
>>> + */
>>> +
>>> +static LIST_HEAD(pci_host_bridges);
>>> +
>>> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
>>> + struct pci_config_window *cfg)
>>> +{
>>> + const __be32 *cells;
>>> + uint32_t len;
>>> +
>>> + cells = dt_get_property(dev, "bus-range", &len);
>>> + /* bus-range should at least be 2 cells */
>>> + if ( !cells || (len < (sizeof(*cells) * 2)) )
>>> + return false;
>>> +
>>> + cfg->busn_start = dt_next_cell(1, &cells);
>>> + cfg->busn_end = dt_next_cell(1, &cells);
>>> +
>>> + return true;
>>> +}
>>> +
>>> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
>>> +{
>>> + return ioremap_nocache(start, len);
>>> +}
>>> +
>>> +static void pci_ecam_free(struct pci_config_window *cfg)
>>> +{
>>> + if ( cfg->win )
>>> + iounmap(cfg->win);
>>> +
>>> + xfree(cfg);
>>> +}
>
> The two functions above seem to deal with the same resources, e.g. cfg->win
>
> and map/unmap. Would it make sense to align those, something like
>
> s/pci_remap_cfgspace/pci_ecam_alloc and pci_ecam_alloc handles cfg->win?
>
> Or anything which makes them look init/fini style?
Ok will rename the function name.
>
>>> +
>>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>>> + struct pci_ecam_ops *ops)
>>> +{
>>> + int err;
>>> + struct pci_config_window *cfg;
>>> + paddr_t addr, size;
>>> +
>>> + cfg = xzalloc(struct pci_config_window);
>>> + if ( !cfg )
>>> + return NULL;
>>> +
>>> + err = dt_pci_parse_bus_range(dev, cfg);
>>> + if ( !err ) {
>>> + cfg->busn_start = 0;
>>> + cfg->busn_end = 0xff;
>>> + printk(XENLOG_ERR "No bus range found for pci controller\n");
>>> + } else {
>>> + if ( cfg->busn_end > cfg->busn_start + 0xff )
>>> + cfg->busn_end = cfg->busn_start + 0xff;
>
> So, if bus start is, for example, 0x10 then we'll end up with bus end at (0x10 + 0xff) > 0xff
>
> which doesn't seem to be what we want
This is to fix if in device tree bus-ranges property, bus end corresponds to more than 256 bus.
>
>>> + }
>>> +
>>> + /* Parse our PCI ecam register address*/
>>> + err = dt_device_get_address(dev, 0, &addr, &size);
>>> + if ( err )
>>> + goto err_exit;
>> Shouldn't we handle the possibility of multiple addresses? Is it
>> possible to have more than one range for an ECAM compliant host bridge?
>>
>>
>>> + cfg->phys_addr = addr;
>>> + cfg->size = size;
>>> + cfg->ops = ops;
>>> +
>>> + /*
>>> + * On 64-bit systems, we do a single ioremap for the whole config space
>>> + * since we have enough virtual address range available. On 32-bit, we
>>> + * ioremap the config space for each bus individually.
>>> + *
>>> + * As of now only 64-bit is supported 32-bit is not supported.
>>> + */
>>> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
>
> I am fine with "win", but can we think of something that tells us that
>
> "win" is actually ECAM base address, so one doesn't need to map "win" to "ECAM"
>
> while reading?
Ack. Will fix.
>
>>> + if ( !cfg->win )
>>> + goto err_exit_remap;
>>> +
>>> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>>> + cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
>>> +
>>> + if ( ops->init ) {
>>> + err = ops->init(cfg);
>>> + if (err)
>>> + goto err_exit;
>>> + }
>>> +
>>> + return cfg;
>>> +
>>> +err_exit_remap:
>>> + printk(XENLOG_ERR "ECAM ioremap failed\n");
>>> +err_exit:
>>> + pci_ecam_free(cfg);
>>> + return NULL;
>>> +}
>>> +
>>> +static struct pci_host_bridge * pci_alloc_host_bridge(void)
>>> +{
>>> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
>>> +
>>> + if ( !bridge )
>>> + return NULL;
>>> +
>>> + INIT_LIST_HEAD(&bridge->node);
>>> + return bridge;
>>> +}
>>> +
>>> +int pci_host_common_probe(struct dt_device_node *dev,
>>> + struct pci_ecam_ops *ops)
>>> +{
>>> + struct pci_host_bridge *bridge;
>>> + struct pci_config_window *cfg;
>>> + u32 segment;
>>> +
>>> + bridge = pci_alloc_host_bridge();
>>> + if ( !bridge )
>>> + return -ENOMEM;
>>> +
>>> + /* Parse and map our Configuration Space windows */
> Do you expect multiple windows here as the comment says?
If going forward we want to support 32-bit we have to ioremap the config space for each bus individually, but as of only 64 bit is supported.
>>> + cfg = gen_pci_init(dev, ops)
>>> + if ( !cfg )
>>> + return -ENOMEM;
>> In case of errors the allocated bridge is not freed.
>>
>>
>>> + bridge->dt_node = dev;
>>> + bridge->sysdata = cfg;
>>> + bridge->ops = &ops->pci_ops;
>
> Can we have some sort of dummy ops so we don't have to check for ops != NULL every time
>
> we read/write config? Is it really possible that we have ops set to NULL after we have
>
> the development finished?
We don’t want to support host-bridges with dump ops. Proper ops has to be setup for the host-bridge to access the read/write.
Once we access the config space we are sure that ops is already setup so no need to check for NULL each time.
>
>>> +
>>> + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
>>> + {
>>> + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
>>> + return -ENODEV;
>>> + }
>>> +
>>> + bridge->segment = (u16)segment;
>> My understanding is that a Linux pci-domain doesn't correspond exactly
>> to a PCI segment. See for instance:
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
>>
>> Do we need to care about the difference? If we mean pci-domain here,
>> should we just call them as such instead of calling them "segments" in
>> Xen (if they are not segments)?
>
>>
>>
>>> + list_add_tail(&bridge->node, &pci_host_bridges);
>> It looks like &pci_host_bridges should be an ordered list, ordered by
>> segment number?
>
> Why? Do you expect bridge access in some specific order so ordered
>
> list will make it faster?
As I mentioned in earlier no need to have ordered list as PCI config space access is random.
>
>>
>>
>>> + return 0;
>>> +}
>>> +
>>> +/*
>>> + * This function will lookup an hostbridge based on the segment and bus
>>> + * number.
>>> + */
>>> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
>>> +{
>>> + struct pci_host_bridge *bridge;
>>> + bool found = false;
>>> +
>>> + list_for_each_entry( bridge, &pci_host_bridges, node )
>>> + {
>>> + if ( bridge->segment != segment )
>>> + continue;
>>> +
>>> + found = true;
>>> + break;
>>> + }
>>> +
>>> + return (found) ? bridge : NULL;
>>> +}
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * tab-width: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
>>> new file mode 100644
>>> index 0000000000..cd67b3dec6
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/pci-host-generic.c
>>> @@ -0,0 +1,131 @@
>>> +/*
>>> + * Copyright (C) 2020 Arm Ltd.
>>> + *
>>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <asm/device.h>
>>> +#include <asm/io.h>
>>> +#include <xen/pci.h>
>>> +#include <asm/pci.h>
>>> +
>>> +/*
>>> + * Function to get the config space base.
>>> + */
>>> +static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
>>> + uint32_t sbdf, int where)
>> I think the function is misnamed because reading the code below it looks
>> like it is not just returning the base config space address but also the
>> specific address we need to read/write (adding the device offset,
>> "where", and everything).
>>
>> Maybe pci_config_get_address or something like that?
>>
>>
>>> +{
>>> + struct pci_config_window *cfg = bridge->sysdata;
>
> I am a bit confused of the naming ;)
>
> We already have 2 maps: win -> ECAM base and now sysdata -> cfg.
>
> Can we please have that aligned somehow so it is easier to follow?
Ack.
>
>>> + unsigned int devfn_shift = cfg->ops->bus_shift - 8;
>
> We are not checking cfg->ops for NULL, so probably we do not want bridges
>
> with NULL ops as well?
Yes correct we don’t want host-bridge with NULL ops. Do you see any use case to have host-bridge with NULL ops?
>
>>> +
>>> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
>>> +
>>> + unsigned int busn = sbdf_t.bus;
>>> + void __iomem *base;
>>> +
>>> + if ( busn < cfg->busn_start || busn > cfg->busn_end )
>>> + return NULL;
>>> +
>>> + base = cfg->win + (busn << cfg->ops->bus_shift);
>>> +
>>> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
>>> +}
>>> +
>>> +int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>>> + int where, int size, u32 val)
>>> +{
>>> + void __iomem *addr;
>>> +
>>> + addr = pci_config_base(bridge, sbdf, where);
>>> + if ( !addr )
>>> + return -ENODEV;
>>> +
>>> + if ( size == 1 )
>>> + writeb(val, addr);
>>> + else if ( size == 2 )
>>> + writew(val, addr);
>>> + else
>>> + writel(val, addr);
>> please use a switch
>>
>>
>>> + return 0;
>>> +}
>>> +
>>> +int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
>>> + int where, int size, u32 *val)
>>> +{
>>> + void __iomem *addr;
>>> +
>>> + addr = pci_config_base(bridge, sbdf, where);
>>> + if ( !addr ) {
>>> + *val = ~0;
>>> + return -ENODEV;
>>> + }
>>> +
>>> + if ( size == 1 )
>>> + *val = readb(addr);
>>> + else if ( size == 2 )
>>> + *val = readw(addr);
>>> + else
>>> + *val = readl(addr);
>> please use a switch
>>
>>
>>> + return 0;
>>> +}
>>> +
>>> +/* ECAM ops */
>>> +struct pci_ecam_ops pci_generic_ecam_ops = {
>>> + .bus_shift = 20,
>>> + .pci_ops = {
>>> + .read = pci_ecam_config_read,
>>> + .write = pci_ecam_config_write,
>>> + }
>>> +};
>>> +
>>> +static const struct dt_device_match gen_pci_dt_match[] = {
>>> + { .compatible = "pci-host-ecam-generic",
>>> + .data = &pci_generic_ecam_ops },
>> spurious blank line
>>
>>
>>> + { },
>>> +};
>>> +
>>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>>> +{
>>> + const struct dt_device_match *of_id;
>>> + struct pci_ecam_ops *ops;
>>> +
>>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>>> + ops = (struct pci_ecam_ops *) of_id->data;
>>> +
>>> + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>>> + dt_node_full_name(dev), of_id->compatible);
>>> +
>>> + return pci_host_common_probe(dev, ops);
>>> +}
>>> +
>>> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
>>> +.dt_match = gen_pci_dt_match,
>>> +.init = gen_pci_dt_init,
>>> +DT_DEVICE_END
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * tab-width: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>>> new file mode 100644
>>> index 0000000000..f8cbb99591
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/pci.c
>>> @@ -0,0 +1,112 @@
>>> +/*
>>> + * Copyright (C) 2020 Arm Ltd.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <xen/acpi.h>
>>> +#include <xen/device_tree.h>
>>> +#include <xen/errno.h>
>>> +#include <xen/init.h>
>>> +#include <xen/pci.h>
>>> +#include <xen/param.h>
>>> +
>>> +static int __init dt_pci_init(void)
>>> +{
>>> + struct dt_device_node *np;
>>> + int rc;
>>> +
>>> + dt_for_each_device_node(dt_host, np)
>>> + {
>>> + rc = device_init(np, DEVICE_PCI, NULL);
>>> + if( !rc )
>>> + continue;
>>> + /*
>>> + * Ignore the following error codes:
>>> + * - EBADF: Indicate the current is not an pci
>>> + * - ENODEV: The pci device is not present or cannot be used by
>>> + * Xen.
>>> + */
>>> + else if ( rc != -EBADF && rc != -ENODEV )
>>> + {
>>> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
>>> + return rc;
>>> + }
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +#ifdef CONFIG_ACPI
>>> +static void __init acpi_pci_init(void)
>>> +{
>>> + printk(XENLOG_ERR "ACPI pci init not supported \n");
>>> + return;
>>> +}
>>> +#else
>>> +static inline void __init acpi_pci_init(void) { }
>>> +#endif
>>> +
>>> +static bool __initdata param_pci_enable;
>>> +static int __init parse_pci_param(const char *arg)
>>> +{
>>> + if ( !arg )
>>> + {
>>> + param_pci_enable = false;
>>> + return 0;
>>> + }
>>> +
>>> + switch ( parse_bool(arg, NULL) )
>>> + {
>>> + case 0:
>>> + param_pci_enable = false;
>>> + return 0;
>>> + case 1:
>>> + param_pci_enable = true;
>>> + return 0;
>>> + }
>>> +
>>> + return -EINVAL;
>>> +}
>>> +custom_param("pci", parse_pci_param);
>> When adding new command line parameters please also add its
>> documentation (docs/misc/xen-command-line.pandoc) in the same patch,
>> unless this is meant to be just transient and we'll get removed before
>> the final commit of the series.
>>
>>
>>> +void __init pci_init(void)
>>> +{
>>> + /*
>>> + * Enable PCI when has been enabled explicitly (pci=on)
>>> + */
>>> + if ( !param_pci_enable)
>>> + goto disable;
>>> +
>>> + if ( acpi_disabled )
>>> + dt_pci_init();
>>> + else
>>> + acpi_pci_init();
>>> +
>>> +#ifdef CONFIG_HAS_PCI
>>> + pci_segments_init();
>>> +#endif
>>> +
>>> +disable:
>>> + return;
>>> +}
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * tab-width: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>>> index 7968cee47d..2d7f1db44f 100644
>>> --- a/xen/arch/arm/setup.c
>>> +++ b/xen/arch/arm/setup.c
>>> @@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>>>
>>> setup_virt_paging();
>>>
>>> + pci_init();
>> pci_init should probably be an initcall
>>
>>
>>> do_initcalls();
>>>
>>> /*
>>> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
>>> index ee7cff2d44..28f8049cfd 100644
>>> --- a/xen/include/asm-arm/device.h
>>> +++ b/xen/include/asm-arm/device.h
>>> @@ -4,6 +4,7 @@
>>> enum device_type
>>> {
>>> DEV_DT,
>>> + DEV_PCI,
>>> };
>>>
>>> struct dev_archdata {
>>> @@ -25,15 +26,15 @@ typedef struct device device_t;
>>>
>>> #include <xen/device_tree.h>
>>>
>>> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
>>> -#define dev_is_pci(dev) ((void)(dev), 0)
>>> -#define dev_is_dt(dev) ((dev->type == DEV_DT)
> Didn't we have a patch for that recently or talked about?
Not sure need to check.
>>> +#define dev_is_pci(dev) (dev->type == DEV_PCI)
>>> +#define dev_is_dt(dev) (dev->type == DEV_DT)
>>>
>>> enum device_class
>>> {
>>> DEVICE_SERIAL,
>>> DEVICE_IOMMU,
>>> DEVICE_GIC,
>>> + DEVICE_PCI,
>>> /* Use for error */
>>> DEVICE_UNKNOWN,
>>> };
>>> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>>> index de13359f65..94fd00360a 100644
>>> --- a/xen/include/asm-arm/pci.h
>>> +++ b/xen/include/asm-arm/pci.h
>>> @@ -1,7 +1,98 @@
>>> -#ifndef __X86_PCI_H__
>>> -#define __X86_PCI_H__
>>> +/*
>>> + * Copyright (C) 2020 Arm Ltd.
>>> + *
>>> + * Based on Linux drivers/pci/ecam.c
>>> + * Copyright 2016 Broadcom.
>>> + *
>>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>>
>>> +#ifndef __ARM_PCI_H__
>>> +#define __ARM_PCI_H__
>>> +
>>> +#include <xen/pci.h>
>>> +#include <xen/device_tree.h>
>>> +#include <asm/device.h>
>>> +
>>> +#ifdef CONFIG_ARM_PCI
>>> +
>>> +/* Arch pci dev struct */
>>> struct arch_pci_dev {
>>> + struct device dev;
>>> +};
>> Are you actually using struct device in struct arch_pci_dev?
>> struct device is already part of struct dt_device_node and a pointer to
>> it is stored in bridge->dt_node.
>>
>>
>>> +#define PRI_pci "%04x:%02x:%02x.%u"
>>> +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
>>> +
>>> +/*
>>> + * struct to hold the mappings of a config space window. This
>>> + * is expected to be used as sysdata for PCI controllers that
>>> + * use ECAM.
>>> + */
>>> +struct pci_config_window {
>>> + paddr_t phys_addr;
>>> + paddr_t size;
>>> + uint8_t busn_start;
>>> + uint8_t busn_end;
>>> + struct pci_ecam_ops *ops;
>>> + void __iomem *win;
>>> +};
>>> +
>>> +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
>>> +struct pci_host_bridge;
>>> +
>>> +struct pci_ops {
>>> + int (*read)(struct pci_host_bridge *bridge,
>>> + uint32_t sbdf, int where, int size, u32 *val);
>>> + int (*write)(struct pci_host_bridge *bridge,
>>> + uint32_t sbdf, int where, int size, u32 val);
>> I'd prefer if we could use explicitly-sized integers for "where" and
>> "size" too. Also, should they be unsigned rather than signed?
>>
>> Can we use pci_sbdf_t for the sbdf parameter?
>>
>>
>>> +};
>>> +
>>> +/*
>>> + * struct to hold pci ops and bus shift of the config window
>>> + * for a PCI controller.
>>> + */
>>> +struct pci_ecam_ops {
>>> + unsigned int bus_shift;
>>> + struct pci_ops pci_ops;
>>> + int (*init)(struct pci_config_window *);
>>> +};
>> Although I realize that we are only targeting ECAM now, and the
>> implementation is based on ECAM, the interface doesn't seem to have
>> anything ECAM-specific in it. I would rename pci_ecam_ops to something
>> else, maybe simply "pci_ops".
>
> Yes, please, bear in mind that we are about to work with this code on
>
> non-ECAM HW from the very beginning, so this is something that we would
>
> like to see from the ground up
Ok. Will take care of that in next version of the patch.
>
>>
>>
>>> +/*
>>> + * struct to hold pci host bridge information
>>> + * for a PCI controller.
>>> + */
>>> +struct pci_host_bridge {
>>> + struct dt_device_node *dt_node; /* Pointer to the associated DT node */
>>> + struct list_head node; /* Node in list of host bridges */
>>> + uint16_t segment; /* Segment number */
>>> + void *sysdata; /* Pointer to the config space window*/
>>> + const struct pci_ops *ops;
>>> };
>>>
>>> -#endif /* __X86_PCI_H__ */
>>> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
>>> +
>>> +int pci_host_common_probe(struct dt_device_node *dev,
>>> + struct pci_ecam_ops *ops);
>>> +
>>> +void pci_init(void);
>>> +
>>> +#else /*!CONFIG_ARM_PCI*/
>>> +struct arch_pci_dev { };
>>> +static inline void pci_init(void) { }
>>> +#endif /*!CONFIG_ARM_PCI*/
>>> +#endif /* __ARM_PCI_H__ */
>>> --
>>> 2.17.1
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 8:05 ` Julien Grall
2020-07-24 17:47 ` Stefano Stabellini
@ 2020-07-27 15:27 ` Rahul Singh
1 sibling, 0 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-27 15:27 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Oleksandr Andrushchenko, andrew.cooper3,
Bertrand Marquis, jbeulich, xen-devel, nd, Volodymyr Babchuk,
roger.pau
> On 24 Jul 2020, at 9:05 am, Julien Grall <julien@xen.org> wrote:
>
> Hi,
>
> On 24/07/2020 08:03, Oleksandr Andrushchenko wrote:
>>>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>>>> new file mode 100644
>>>> index 0000000000..c53ef58336
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/pci/pci-access.c
>>>> @@ -0,0 +1,101 @@
>>>> +/*
>>>> + * Copyright (C) 2020 Arm Ltd.
>> I think SPDX will fit better in any new code.
>
> While I would love to use SPDX in Xen, there was some push back in the past to use it. So the new code should use the full-blown copyright until there is an agreement to use it.
Ack. We will use the copyright until we will get the confirmation from the community to use the SPDX in XEN.
>
>>>
>>>> + list_add_tail(&bridge->node, &pci_host_bridges);
>>> It looks like &pci_host_bridges should be an ordered list, ordered by
>>> segment number?
>> Why? Do you expect bridge access in some specific order so ordered
>> list will make it faster?
>
> Access to the configure space will be pretty random. So I don't think ordering the list will make anything better.
>
> However, looking up for the bridge for every config spec access is pretty slow. When I was working on the PCI passthrough, I wanted to look whether it would be possible to have a pointer to the PCI host bridge passed in argument to the helpers (rather than the segment).
Yes true every config space access has to scan the host-bridges list to find the corresponding host-bridge. I will try to find better solution so that no need to scan the host-bridge every time for read/write access.
> Cheers,
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 8:23 ` Julien Grall
@ 2020-07-27 15:29 ` Rahul Singh
0 siblings, 0 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-27 15:29 UTC (permalink / raw)
To: Julien Grall
Cc: xen-devel, nd, Volodymyr Babchuk, Stefano Stabellini, Bertrand Marquis
> On 24 Jul 2020, at 9:23 am, Julien Grall <julien@xen.org> wrote:
>
> Hi Rahul,
>
> On 23/07/2020 16:40, Rahul Singh wrote:
>> XEN during boot will read the PCI device tree node “reg” property
>> and will map the PCI config space to the XEN memory.
>> XEN will read the “linux, pci-domain” property from the device tree
>> node and configure the host bridge segment number accordingly.
>> As of now "pci-host-ecam-generic" compatible board is supported.
>> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/Kconfig | 7 +
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/pci/Makefile | 4 +
>> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
>> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
>> xen/arch/arm/setup.c | 2 +
>> xen/include/asm-arm/device.h | 7 +-
>> xen/include/asm-arm/pci.h | 97 +++++++++++++-
>> 10 files changed, 654 insertions(+), 6 deletions(-)
>
> As a general comment, I would suggest to split the patch in smaller chunk. This would help the review and also allow to provide more explanation on what is done.
Ok I will split the patches in next version of the patch series.
>
> For instance, I think it is possible to a split looking like:
> - Add framework to access an hostbridge
> - Add support for ECAM
> - Add code to initialize the PCI subsystem
>
> There is also some small fixes in this code that probably can move in there own patches.
Ack.
>
> Cheers,
>
> --
> Julien Grall
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
2020-07-24 7:14 ` Oleksandr Andrushchenko
2020-07-24 8:19 ` Julien Grall
@ 2020-07-27 16:10 ` Rahul Singh
1 sibling, 0 replies; 48+ messages in thread
From: Rahul Singh @ 2020-07-27 16:10 UTC (permalink / raw)
To: Oleksandr Andrushchenko
Cc: Stefano Stabellini, Julien Grall, andrew.cooper3,
Bertrand Marquis, jbeulich, xen-devel, nd, Volodymyr Babchuk,
roger.pau
> On 24 Jul 2020, at 8:14 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
>
> On 7/23/20 11:44 PM, Stefano Stabellini wrote:
>> On Thu, 23 Jul 2020, Rahul Singh wrote:
>>> Hardware domain is in charge of doing the PCI enumeration and will
>>> discover the PCI devices and then will communicate to XEN via hyper
>>> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
>>>
>>> Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
>> Same question about Change-Id
>>
>>
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> ---
>>> xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
>>> 1 file changed, 39 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
>>> index e91355fe22..274720f98a 100644
>>> --- a/xen/arch/arm/physdev.c
>>> +++ b/xen/arch/arm/physdev.c
>>> @@ -9,12 +9,48 @@
>>> #include <xen/errno.h>
>>> #include <xen/sched.h>
>>> #include <asm/hypercall.h>
>>> -
>>> +#include <xen/guest_access.h>
>>> +#include <xsm/xsm.h>
>>>
>>> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>>> {
>>> - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>>> - return -ENOSYS;
>>> + int ret = 0;
>>> +
>>> + switch ( cmd )
>>> + {
>>> +#ifdef CONFIG_HAS_PCI
>
> In the cover letter you were saying "we are not enabling the HAS_PCI and HAS_VPCI flags for ARM".
>
> Is this still valid?
Yes right we are not enabling it because full support of PCI passthrough is not implemented and tested.
>
>>> + case PHYSDEVOP_pci_device_add:
>>> + {
>>> + struct physdev_pci_device_add add;
>>> + struct pci_dev_info pdev_info;
>>> + nodeid_t node = NUMA_NO_NODE;
>>> +
>>> + ret = -EFAULT;
>>> + if ( copy_from_guest(&add, arg, 1) != 0 )
>>> + break;
>>> +
>>> + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
>>> + if ( add.flags & XEN_PCI_DEV_VIRTFN )
>>> + {
>>> + pdev_info.is_virtfn = 1;
>>> + pdev_info.physfn.bus = add.physfn.bus;
>>> + pdev_info.physfn.devfn = add.physfn.devfn;
>>> + }
>>> + else
>>> + pdev_info.is_virtfn = 0;
>>> +
>>> + ret = pci_add_device(add.seg, add.bus, add.devfn,
>>> + &pdev_info, node);
>>> +
>>> + break;
>>> + }
>>> +#endif
>>> + default:
>>> + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>>> + ret = -ENOSYS;
>>> + }
>> I think we should make the implementation common between arm and x86 by
>> creating xen/common/physdev.c:do_physdev_op as a shared entry point for
>> PHYSDEVOP hypercalls implementations. See for instance:
>>
>> xen/common/sysctl.c:do_sysctl
>>
>> and
>>
>> xen/arch/arm/sysctl.c:arch_do_sysctl
>> xen/arch/x86/sysctl.c:arch_do_sysctl
>>
>>
>> Jan, Andrew, Roger, any opinions?
>>
>>
> I think we can also have a look at [1] by Julien. That implementation,
>
> IMO, had some thoughts on making Arm/x86 code common where possible
Ok. Thanks for the pointer. We will have a look.
>
>
> [1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-27 11:06 ` Roger Pau Monné
@ 2020-07-28 0:06 ` Stefano Stabellini
2020-07-28 8:33 ` Roger Pau Monné
0 siblings, 1 reply; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-28 0:06 UTC (permalink / raw)
To: Roger Pau Monné
Cc: nd, Stefano Stabellini, Andrew Cooper, Bertrand Marquis,
Jan Beulich, xen-devel, Rahul Singh, Volodymyr Babchuk,
Julien Grall
[-- Attachment #1: Type: text/plain, Size: 3817 bytes --]
On Mon, 27 Jul 2020, Roger Pau Monné wrote:
> On Sat, Jul 25, 2020 at 10:59:50AM +0100, Julien Grall wrote:
> > On Sat, 25 Jul 2020 at 00:46, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > >
> > > On Fri, 24 Jul 2020, Julien Grall wrote:
> > > > On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > > > > If they are not equal, then I fail to see why it would be useful to have this
> > > > > > value in Xen.
> > > > >
> > > > > I think that's because the domain is actually more convenient to use
> > > > > because a segment can span multiple PCI host bridges. So my
> > > > > understanding is that a segment alone is not sufficient to identify a
> > > > > host bridge. From a software implementation point of view it would be
> > > > > better to use domains.
> > > >
> > > > AFAICT, this would be a matter of one check vs two checks in Xen :).
> > > > But... looking at Linux, they will also use domain == segment for ACPI
> > > > (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
>
> You have to use the (segment, bus) tuple when doing a lookup because
> MMCFG regions on ACPI are defined for a segment and a bus range, you
> can have a MMCFG region that covers segment 0 bus [0, 20) and another
> MMCFG region that covers segment 0 bus [20, 255], and those will use
> different addresses in the MMIO space.
Thanks for the clarification!
> > > > > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > > > > > Dom0 and Xen can synchronize on the segment number.
> > > > >
> > > > > I was hoping we could write down the assumption somewhere that for the
> > > > > cases we care about domain == segment, and error out if it is not the
> > > > > case.
> > > >
> > > > Given that we have only the domain in hand, how would you enforce that?
> > > >
> > > > >From this discussion, it also looks like there is a mismatch between the
> > > > implementation and the understanding on QEMU devel. So I am a bit
> > > > concerned that this is not stable and may change in future Linux version.
> > > >
> > > > IOW, we are know tying Xen to Linux. So could we implement
> > > > PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
> > > > really represent the segment?
> > >
> > > I don't think we are tying Xen to Linux. Rob has already said that
> > > linux,pci-domain is basically a generic device tree property.
> >
> > My concern is not so much the name of the property, but the definition of it.
> >
> > AFAICT, from this thread there can be two interpretation:
> > - domain == segment
> > - domain == (segment, bus)
>
> I think domain is just an alias for segment, the difference seems to
> be that when using DT all bridges get a different segment (or domain)
> number, and thus you will always end up starting numbering at bus 0
> for each bridge?
>
> Ideally you would need a way to specify the segment and start/end bus
> numbers of each bridge, if not you cannot match what ACPI does. Albeit
> it might be fine as long as the OS and Xen agree on the segments and
> bus numbers that belong to each bridge (and thus each ECAM region).
That is what I thought and it is why I was asking to clarify the naming
and/or writing a document to explain the assumptions, if any.
Then after Julien's email I followed up in the Linux codebase and
clearly there is a different assumption baked in the Linux kernel for
architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).
The assumption is that segment == domain == unique host bridge. It
looks like it is coming from IEEE Std 1275-1994 but I am not certain.
In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly
match. So I am starting to think that domain == segment for IEEE Std
1275-1994 compliant device tree based systems.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-24 14:44 ` Roger Pau Monné
2020-07-24 15:15 ` Julien Grall
@ 2020-07-28 8:06 ` Rahul Singh
2020-07-28 8:21 ` Roger Pau Monné
1 sibling, 1 reply; 48+ messages in thread
From: Rahul Singh @ 2020-07-28 8:06 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, xen-devel,
nd, Volodymyr Babchuk
> On 24 Jul 2020, at 3:44 pm, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Thu, Jul 23, 2020 at 04:40:21PM +0100, Rahul Singh wrote:
>> XEN during boot will read the PCI device tree node “reg” property
>> and will map the PCI config space to the XEN memory.
>>
>> XEN will read the “linux, pci-domain” property from the device tree
>> node and configure the host bridge segment number accordingly.
>>
>> As of now "pci-host-ecam-generic" compatible board is supported.
>>
>> Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/Kconfig | 7 +
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/pci/Makefile | 4 +
>> xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
>> xen/arch/arm/pci/pci.c | 112 ++++++++++++++++
>> xen/arch/arm/setup.c | 2 +
>> xen/include/asm-arm/device.h | 7 +-
>> xen/include/asm-arm/pci.h | 97 +++++++++++++-
>> 10 files changed, 654 insertions(+), 6 deletions(-)
>> create mode 100644 xen/arch/arm/pci/Makefile
>> create mode 100644 xen/arch/arm/pci/pci-access.c
>> create mode 100644 xen/arch/arm/pci/pci-host-common.c
>> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
>> create mode 100644 xen/arch/arm/pci/pci.c
>>
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 2777388265..ee13339ae9 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -31,6 +31,13 @@ menu "Architecture Features"
>>
>> source "arch/Kconfig"
>>
>> +config ARM_PCI
>> + bool "PCI Passthrough Support"
>> + depends on ARM_64
>> + ---help---
>> +
>> + PCI passthrough support for Xen on ARM64.
>> +
>> config ACPI
>> bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
>> depends on ARM_64
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 7e82b2178c..345cb83eed 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>> obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_ARM_PCI) += pci/
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> new file mode 100644
>> index 0000000000..358508b787
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -0,0 +1,4 @@
>> +obj-y += pci.o
>> +obj-y += pci-host-generic.o
>> +obj-y += pci-host-common.o
>> +obj-y += pci-access.o
>
> The Kconfig option mentions the support being explicitly for ARM64,
> would it be better to place the code in arch/arm/arm64 then?
Ok.As julien also mentioned we tested on ARM64 , We have to test on ARM32 platforms.
>
>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>> new file mode 100644
>> index 0000000000..c53ef58336
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -0,0 +1,101 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +#include <xen/rwlock.h>
>> +
>> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>> + unsigned int len)
>
> Please align with the opening parenthesis (here and everywhere in the
> patch series).
Ack.
>
>> +{
>> + int rc;
>> + uint32_t val = GENMASK(0, len * 8);
>
> You can just set val = ~0. The return type of the pci_conf_readXX
> helpers will already truncate the value.
>
Ack.
>> +
>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> + if ( unlikely(!bridge) )
>> + {
>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>
> I had a patch to add a custom modifier to out printf format in
> order to handle pci_sbdf_t natively:
>
> https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
>
> It missed maintainers Acks and was never committed. Since you are
> doing a bunch of work here, and likely adding a lot of SBDF related
> prints, feel free to import the modifier (%pp) and use in your code
> (do not attempt to switch existing users, or it's likely to get
> stuck again).
Ok Will integrate that patch once submitted.
>
>> + return val;
>> + }
>> +
>> + if ( unlikely(!bridge->ops->read) )
>> + return val;
>> +
>> + rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
>
> There's no need for the uint32_t cast, the sbdf field is already of
> such type
Ack.
>> + if ( rc )
>> + printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> +
>> + return val;
>> +}
>> +
>> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>> + unsigned int len, uint32_t val)
>> +{
>> + int rc;
>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> + if ( unlikely(!bridge) )
>> + {
>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> + return;
>> + }
>> +
>> + if ( unlikely(!bridge->ops->write) )
>> + return;
>> +
>> + rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
>> + if ( rc )
>> + printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
>> + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> +}
>> +
>> +/*
>> + * Wrappers for all PCI configuration access functions.
>> + */
>> +
>> +#define PCI_OP_WRITE(size, type) \
>> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
>> +{ \
>> + pci_config_write(sbdf, reg, size / 8, val); \
>> +}
>> +
>> +#define PCI_OP_READ(size, type) \
>> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
>> +{ \
>> + return pci_config_read(sbdf, reg, size / 8); \
>> +}
>> +
>> +PCI_OP_READ(8, u8)
>> +PCI_OP_READ(16, u16)
>> +PCI_OP_READ(32, u32)
>> +PCI_OP_WRITE(8, u8)
>> +PCI_OP_WRITE(16, u16)
>> +PCI_OP_WRITE(32, u32)
>
> Please use uintXX_t.
>
> Also, It's nice to add some kind of signals for cscope and friends so
> they can find the autogenerated functions, ie:
>
> #define pci_conf_read8
> #undef pci_conf_read8
> #define pci_conf_read16
> #undef pci_conf_read16
> ...
>
> It's tedious but helps future users find where the code is generated.
Ack.
>
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
>> new file mode 100644
>> index 0000000000..c5f98be698
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-common.c
>> @@ -0,0 +1,198 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +#include <xen/rwlock.h>
>> +#include <xen/vmap.h>
>> +
>> +/*
>> + * List for all the pci host bridges.
>> + */
>> +
>> +static LIST_HEAD(pci_host_bridges);
>> +
>> +static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
>> + struct pci_config_window *cfg)
>> +{
>> + const __be32 *cells;
>
> It's my impression that while based on Linux this is not a verbatim
> copy of a Linux file, and tries to adhere with the Xen coding style.
> If so please use uint32_t here.
>
>> + uint32_t len;
>> +
>> + cells = dt_get_property(dev, "bus-range", &len);
>> + /* bus-range should at least be 2 cells */
>> + if ( !cells || (len < (sizeof(*cells) * 2)) )
>> + return false;
>> +
>> + cfg->busn_start = dt_next_cell(1, &cells);
>> + cfg->busn_end = dt_next_cell(1, &cells);
>> +
>> + return true;
>> +}
>> +
>> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
>> +{
>> + return ioremap_nocache(start, len);
>> +}
>> +
>> +static void pci_ecam_free(struct pci_config_window *cfg)
>> +{
>> + if ( cfg->win )
>> + iounmap(cfg->win);
>> +
>> + xfree(cfg);
>> +}
>> +
>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops)
>> +{
>> + int err;
>> + struct pci_config_window *cfg;
>> + paddr_t addr, size;
>> +
>> + cfg = xzalloc(struct pci_config_window);
>> + if ( !cfg )
>> + return NULL;
>> +
>> + err = dt_pci_parse_bus_range(dev, cfg);
>> + if ( !err ) {
>
> Braces
Ack
>
>> + cfg->busn_start = 0;
>> + cfg->busn_end = 0xff;
>> + printk(XENLOG_ERR "No bus range found for pci controller\n");
>> + } else {
>> + if ( cfg->busn_end > cfg->busn_start + 0xff )
>> + cfg->busn_end = cfg->busn_start + 0xff;
>> + }
>> +
>> + /* Parse our PCI ecam register address*/
>> + err = dt_device_get_address(dev, 0, &addr, &size);
>> + if ( err )
>> + goto err_exit;
>> +
>> + cfg->phys_addr = addr;
>> + cfg->size = size;
>> + cfg->ops = ops;
>> +
>> + /*
>> + * On 64-bit systems, we do a single ioremap for the whole config space
>> + * since we have enough virtual address range available. On 32-bit, we
>> + * ioremap the config space for each bus individually.
>> + *
>> + * As of now only 64-bit is supported 32-bit is not supported.
>> + */
>> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
>> + if ( !cfg->win )
>> + goto err_exit_remap;
>> +
>> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>> + cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
>> +
>> + if ( ops->init ) {
>> + err = ops->init(cfg);
>> + if (err)
>> + goto err_exit;
>> + }
>> +
>> + return cfg;
>> +
>> +err_exit_remap:
>> + printk(XENLOG_ERR "ECAM ioremap failed\n");
>> +err_exit:
>> + pci_ecam_free(cfg);
>> + return NULL;
>> +}
>> +
>> +static struct pci_host_bridge * pci_alloc_host_bridge(void)
> ^ extra space
Ack.
>> +{
>> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
>> +
>> + if ( !bridge )
>> + return NULL;
>> +
>> + INIT_LIST_HEAD(&bridge->node);
>> + return bridge;
>> +}
>> +
>> +int pci_host_common_probe(struct dt_device_node *dev,
>> + struct pci_ecam_ops *ops)
>> +{
>> + struct pci_host_bridge *bridge;
>> + struct pci_config_window *cfg;
>> + u32 segment;
>> +
>> + bridge = pci_alloc_host_bridge();
>> + if ( !bridge )
>> + return -ENOMEM;
>> +
>> + /* Parse and map our Configuration Space windows */
>> + cfg = gen_pci_init(dev, ops);
>> + if ( !cfg )
>> + return -ENOMEM;
>
> You are leaking the allocation of bridge here ...
>
>> +
>> + bridge->dt_node = dev;
>> + bridge->sysdata = cfg;
>> + bridge->ops = &ops->pci_ops;
>> +
>> + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
>> + {
>> + printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
>> + return -ENODEV;
>
> ... and here.
>
>> + }
>> +
>> + bridge->segment = (u16)segment;
>> +
>> + list_add_tail(&bridge->node, &pci_host_bridges);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * This function will lookup an hostbridge based on the segment and bus
>> + * number.
>> + */
>> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
>> +{
>> + struct pci_host_bridge *bridge;
>> + bool found = false;
>> +
>> + list_for_each_entry( bridge, &pci_host_bridges, node )
>> + {
>> + if ( bridge->segment != segment )
>> + continue;
>> +
>> + found = true;
>> + break;
>> + }
>> +
>> + return (found) ? bridge : NULL;
>
> This can be much shorter:
>
> struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
> {
> struct pci_host_bridge *bridge;
>
> list_for_each_entry( bridge, &pci_host_bridges, node )
> if ( bridge->segment == segment )
> return bridge;
>
> return NULL;
> }
>
> Albeit I'm confused by the fact that you pass a bus number that's
> completely unused.
Ok Will fix.
>
>> +}
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
>> new file mode 100644
>> index 0000000000..cd67b3dec6
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-generic.c
>> @@ -0,0 +1,131 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/device.h>
>> +#include <asm/io.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +
>> +/*
>> + * Function to get the config space base.
>> + */
>> +static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where)
>
> You would be better passing a pci_sbdf_t directly here. Also 'where'
> should be renamed to offset, or reg, and be made unsigned int. AFAICT
> you will never pass a negative value here.
>
> For sanity you should also assert that the offset falls between the
> PCI config space used by the device, in order to easily catch with
> wrong offsets being used.
Ack.
>
>> +{
>> + struct pci_config_window *cfg = bridge->sysdata;
>
> const
Ack.
>
>> + unsigned int devfn_shift = cfg->ops->bus_shift - 8;
>> +
>> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
>> +
>> + unsigned int busn = sbdf_t.bus;
>> + void __iomem *base;
>
> IMO adding newlines between variable definitions is not helpful, but
> that's my taste.
Ack.
>
>> +
>> + if ( busn < cfg->busn_start || busn > cfg->busn_end )
>> + return NULL;
>> +
>> + base = cfg->win + (busn << cfg->ops->bus_shift);
>> +
>> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
>> +}
>> +
>> +int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>> + int where, int size, u32 val)
>> +{
>> + void __iomem *addr;
>> +
>> + addr = pci_config_base(bridge, sbdf, where);
>
> You can initialize at definition.
Ack.
>
>> + if ( !addr )
>> + return -ENODEV;
>> +
>> + if ( size == 1 )
>> + writeb(val, addr);
>> + else if ( size == 2 )
>> + writew(val, addr);
>> + else
>> + writel(val, addr);
>
> Please use a switch, and check against specific values. The default
> case should be a BUG();. See pci_conf_read from x86 for an example.
Ack.
>
>> +
>> + return 0;
>> +}
>> +
>> +int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
>> + int where, int size, u32 *val)
>> +{
>> + void __iomem *addr;
>> +
>> + addr = pci_config_base(bridge, sbdf, where);
>> + if ( !addr ) {
>> + *val = ~0;
>> + return -ENODEV;
>> + }
>> +
>> + if ( size == 1 )
>> + *val = readb(addr);
>> + else if ( size == 2 )
>> + *val = readw(addr);
>> + else
>> + *val = readl(addr);
>> +
>> + return 0;
>> +}
>> +
>> +/* ECAM ops */
>> +struct pci_ecam_ops pci_generic_ecam_ops = {
>> + .bus_shift = 20,
>> + .pci_ops = {
>> + .read = pci_ecam_config_read,
>> + .write = pci_ecam_config_write,
>> + }
>> +};
>> +
>> +static const struct dt_device_match gen_pci_dt_match[] = {
>> + { .compatible = "pci-host-ecam-generic",
>> + .data = &pci_generic_ecam_ops },
>> +
>> + { },
>> +};
>> +
>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>> +{
>> + const struct dt_device_match *of_id;
>> + struct pci_ecam_ops *ops;
>> +
>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>> + ops = (struct pci_ecam_ops *) of_id->data;
>> +
>> + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>> + dt_node_full_name(dev), of_id->compatible);
>> +
>> + return pci_host_common_probe(dev, ops);
>> +}
>> +
>> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
>> +.dt_match = gen_pci_dt_match,
>> +.init = gen_pci_dt_init,
>> +DT_DEVICE_END
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>> new file mode 100644
>> index 0000000000..f8cbb99591
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci.c
>> @@ -0,0 +1,112 @@
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/acpi.h>
>> +#include <xen/device_tree.h>
>> +#include <xen/errno.h>
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <xen/param.h>
>> +
>> +static int __init dt_pci_init(void)
>> +{
>> + struct dt_device_node *np;
>> + int rc;
>> +
>> + dt_for_each_device_node(dt_host, np)
>> + {
>> + rc = device_init(np, DEVICE_PCI, NULL);
>> + if( !rc )
>> + continue;
>> + /*
>> + * Ignore the following error codes:
>> + * - EBADF: Indicate the current is not an pci
>> + * - ENODEV: The pci device is not present or cannot be used by
>> + * Xen.
>> + */
>> + else if ( rc != -EBADF && rc != -ENODEV )
>> + {
>> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
>> + return rc;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +#ifdef CONFIG_ACPI
>> +static void __init acpi_pci_init(void)
>> +{
>> + printk(XENLOG_ERR "ACPI pci init not supported \n");
>> + return;
>> +}
>> +#else
>> +static inline void __init acpi_pci_init(void) { }
>> +#endif
>> +
>> +static bool __initdata param_pci_enable;
>> +static int __init parse_pci_param(const char *arg)
>> +{
>> + if ( !arg )
>> + {
>> + param_pci_enable = false;
>> + return 0;
>> + }
>> +
>> + switch ( parse_bool(arg, NULL) )
>> + {
>> + case 0:
>> + param_pci_enable = false;
>> + return 0;
>> + case 1:
>> + param_pci_enable = true;
>> + return 0;
>> + }
>> +
>> + return -EINVAL;
>> +}
>> +custom_param("pci", parse_pci_param);
>
> You need to introduce the documentation for the parameter at
> docs/misc/xen-command-line.pandoc
>
> Albeit I'm not sure I like it, why do you need to enable PCI
> explicitly?
>
> Shouldn't it be discovered automatically and enabled by default?
Ack.
>
>> +void __init pci_init(void)
>> +{
>> + /*
>> + * Enable PCI when has been enabled explicitly (pci=on)
>> + */
>> + if ( !param_pci_enable)
>> + goto disable;
>
> Just return here, there's not point in having a label to perform a
> return.
Ack.
>
>> +
>> + if ( acpi_disabled )
>> + dt_pci_init();
>> + else
>> + acpi_pci_init();
>
> Isn't there an enum or something that tells you whether the system
> description is coming from ACPI or from DT?
>
> This if .. else seems fragile.
>
> Also for ACPI you will get called by acpi_boot_init, and likely need
> to implement a acpi_mmcfg_init or pci_mmcfg_arch_{init,enable}. I'm
> not sure whether the code in acpi_mmcfg_init could be made shared
> between both x86 and Arm.
>
>> +
>> +#ifdef CONFIG_HAS_PCI
>> + pci_segments_init();
>> +#endif
>> +
>> +disable:
>> + return;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>> index 7968cee47d..2d7f1db44f 100644
>> --- a/xen/arch/arm/setup.c
>> +++ b/xen/arch/arm/setup.c
>> @@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>>
>> setup_virt_paging();
>>
>> + pci_init();
>> +
>> do_initcalls();
>>
>> /*
>> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
>> index ee7cff2d44..28f8049cfd 100644
>> --- a/xen/include/asm-arm/device.h
>> +++ b/xen/include/asm-arm/device.h
>> @@ -4,6 +4,7 @@
>> enum device_type
>> {
>> DEV_DT,
>> + DEV_PCI,
>> };
>>
>> struct dev_archdata {
>> @@ -25,15 +26,15 @@ typedef struct device device_t;
>>
>> #include <xen/device_tree.h>
>>
>> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
>> -#define dev_is_pci(dev) ((void)(dev), 0)
>> -#define dev_is_dt(dev) ((dev->type == DEV_DT)
>> +#define dev_is_pci(dev) (dev->type == DEV_PCI)
>> +#define dev_is_dt(dev) (dev->type == DEV_DT)
>>
>> enum device_class
>> {
>> DEVICE_SERIAL,
>> DEVICE_IOMMU,
>> DEVICE_GIC,
>> + DEVICE_PCI,
>
> It seems to be like this wants to be DEVICE_PCI_HOST_BRIDGE or some
> such, since this is not used to identify all PCI devices, but just
> bridges?
>
>> /* Use for error */
>> DEVICE_UNKNOWN,
>> };
>> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>> index de13359f65..94fd00360a 100644
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -1,7 +1,98 @@
>> -#ifndef __X86_PCI_H__
>> -#define __X86_PCI_H__
>> +/*
>> + * Copyright (C) 2020 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>>
>> +#ifndef __ARM_PCI_H__
>> +#define __ARM_PCI_H__
>> +
>> +#include <xen/pci.h>
>> +#include <xen/device_tree.h>
>> +#include <asm/device.h>
>> +
>> +#ifdef CONFIG_ARM_PCI
>> +
>> +/* Arch pci dev struct */
>> struct arch_pci_dev {
>> + struct device dev;
>> +};
>
> This seems to be completely unused?
Will be using going forward.
>
>> +
>> +#define PRI_pci "%04x:%02x:%02x.%u"
>> +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
>> +
>> +/*
>> + * struct to hold the mappings of a config space window. This
>> + * is expected to be used as sysdata for PCI controllers that
>> + * use ECAM.
>> + */
>> +struct pci_config_window {
>> + paddr_t phys_addr;
>> + paddr_t size;
>> + uint8_t busn_start;
>> + uint8_t busn_end;
>> + struct pci_ecam_ops *ops;
>
> const?
>
>> + void __iomem *win;
>> +};
>> +
>> +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
>> +struct pci_host_bridge;
>> +
>> +struct pci_ops {
>> + int (*read)(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where, int size, u32 *val);
>> + int (*write)(struct pci_host_bridge *bridge,
>> + uint32_t sbdf, int where, int size, u32 val);
>> +};
>> +
>> +/*
>> + * struct to hold pci ops and bus shift of the config window
>> + * for a PCI controller.
>> + */
>> +struct pci_ecam_ops {
>> + unsigned int bus_shift;
>> + struct pci_ops pci_ops;
>> + int (*init)(struct pci_config_window *);
>> +};
>> +
>> +/*
>> + * struct to hold pci host bridge information
>> + * for a PCI controller.
>> + */
>> +struct pci_host_bridge {
>> + struct dt_device_node *dt_node; /* Pointer to the associated DT node */
>> + struct list_head node; /* Node in list of host bridges */
>> + uint16_t segment; /* Segment number */
>> + void *sysdata; /* Pointer to the config space window*/
>> + const struct pci_ops *ops;
>
> You seem to introduce a lot of ops structs, yet there's only one
> implementation the generic ECAM one, and adding such complexity should
> IMO be done when further implementations are added. Also given this is
> a fully ECAM compliant bridge you could just use most of the existing
> logic for x86?
>
> I understand the discovery needs to be different, but x86 MCFG logic
> should already be capable of handling multiple ECAM regions.
>
> I also agree with Julien that splitting this into separate patch would
> make it easier to review. For example you can start with the discovery
> logic, followed by the initialization and the add the accessors to the
> config space lastly.
Ok . Will spilt the patches in the next version of the patch series.
>
> Thanks, Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-28 8:06 ` Rahul Singh
@ 2020-07-28 8:21 ` Roger Pau Monné
0 siblings, 0 replies; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-28 8:21 UTC (permalink / raw)
To: Rahul Singh
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, xen-devel,
nd, Volodymyr Babchuk
On Tue, Jul 28, 2020 at 08:06:17AM +0000, Rahul Singh wrote:
>
>
> > On 24 Jul 2020, at 3:44 pm, Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Thu, Jul 23, 2020 at 04:40:21PM +0100, Rahul Singh wrote:
> >> +
> >> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> >> +
> >> + if ( unlikely(!bridge) )
> >> + {
> >> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> >> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> >
> > I had a patch to add a custom modifier to out printf format in
> > order to handle pci_sbdf_t natively:
> >
> > https://patchew.org/Xen/20190822065132.48200-1-roger.pau@citrix.com/
> >
> > It missed maintainers Acks and was never committed. Since you are
> > doing a bunch of work here, and likely adding a lot of SBDF related
> > prints, feel free to import the modifier (%pp) and use in your code
> > (do not attempt to switch existing users, or it's likely to get
> > stuck again).
>
> Ok Will integrate that patch once submitted.
I've posted an updated version to the list yesterday:
https://lore.kernel.org/xen-devel/20200727103136.53343-1-roger.pau@citrix.com/
> >> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> >> index de13359f65..94fd00360a 100644
> >> --- a/xen/include/asm-arm/pci.h
> >> +++ b/xen/include/asm-arm/pci.h
> >> @@ -1,7 +1,98 @@
> >> -#ifndef __X86_PCI_H__
> >> -#define __X86_PCI_H__
> >> +/*
> >> + * Copyright (C) 2020 Arm Ltd.
> >> + *
> >> + * Based on Linux drivers/pci/ecam.c
> >> + * Copyright 2016 Broadcom.
> >> + *
> >> + * Based on Linux drivers/pci/controller/pci-host-common.c
> >> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> >> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License version 2 as
> >> + * published by the Free Software Foundation.
> >> + *
> >> + * This program is distributed in the hope that it will be useful,
> >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >> + * GNU General Public License for more details.
> >> + *
> >> + * You should have received a copy of the GNU General Public License
> >> + * along with this program. If not, see <http://secure-web.cisco.com/1YIShdE6aU0WIMsRjhIYYGdRo_H_wyOVeLqk0ITHI7i_FMIXuMEmV9Y6lR76qCBB4XHss0ba_WMfQa0mLGD37bM1S3AFMBVC1WUEF0LmbURPWld2kU2zuyOamUxqnYwy4ZAv4N2EUJy7HCgDYn2s183YCQ-5YBc98vH8TO-tfBgEShkKVzAm23AgzIqikuKN_BIhcQhAbsjypr9ffeNZDsSX2jC8ClqegEPXFsCWJldrASGgWoR16rMjI-INfPNy55m9nGP5UmHIYWBmTpnHLjXtqCkruENixh20vPWIUROhcjjtzVBrR8d-Q5HnJy0hSR97WdlUSAryfxeH-8VxcPw/http%3A%2F%2Fwww.gnu.org%2Flicenses%2F>.
> >> + */
> >>
> >> +#ifndef __ARM_PCI_H__
> >> +#define __ARM_PCI_H__
> >> +
> >> +#include <xen/pci.h>
> >> +#include <xen/device_tree.h>
> >> +#include <asm/device.h>
> >> +
> >> +#ifdef CONFIG_ARM_PCI
> >> +
> >> +/* Arch pci dev struct */
> >> struct arch_pci_dev {
> >> + struct device dev;
> >> +};
> >
> > This seems to be completely unused?
>
> Will be using going forward.
Please introduce it when it's required then.
Thanks, Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-28 0:06 ` Stefano Stabellini
@ 2020-07-28 8:33 ` Roger Pau Monné
2020-07-28 18:33 ` Stefano Stabellini
0 siblings, 1 reply; 48+ messages in thread
From: Roger Pau Monné @ 2020-07-28 8:33 UTC (permalink / raw)
To: Stefano Stabellini
Cc: nd, Andrew Cooper, Bertrand Marquis, Jan Beulich, xen-devel,
Rahul Singh, Volodymyr Babchuk, Julien Grall
On Mon, Jul 27, 2020 at 05:06:25PM -0700, Stefano Stabellini wrote:
> On Mon, 27 Jul 2020, Roger Pau Monné wrote:
> > On Sat, Jul 25, 2020 at 10:59:50AM +0100, Julien Grall wrote:
> > > On Sat, 25 Jul 2020 at 00:46, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > >
> > > > On Fri, 24 Jul 2020, Julien Grall wrote:
> > > > > On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > > > > > If they are not equal, then I fail to see why it would be useful to have this
> > > > > > > value in Xen.
> > > > > >
> > > > > > I think that's because the domain is actually more convenient to use
> > > > > > because a segment can span multiple PCI host bridges. So my
> > > > > > understanding is that a segment alone is not sufficient to identify a
> > > > > > host bridge. From a software implementation point of view it would be
> > > > > > better to use domains.
> > > > >
> > > > > AFAICT, this would be a matter of one check vs two checks in Xen :).
> > > > > But... looking at Linux, they will also use domain == segment for ACPI
> > > > > (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
> >
> > You have to use the (segment, bus) tuple when doing a lookup because
> > MMCFG regions on ACPI are defined for a segment and a bus range, you
> > can have a MMCFG region that covers segment 0 bus [0, 20) and another
> > MMCFG region that covers segment 0 bus [20, 255], and those will use
> > different addresses in the MMIO space.
>
> Thanks for the clarification!
>
>
> > > > > > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > > > > > > Dom0 and Xen can synchronize on the segment number.
> > > > > >
> > > > > > I was hoping we could write down the assumption somewhere that for the
> > > > > > cases we care about domain == segment, and error out if it is not the
> > > > > > case.
> > > > >
> > > > > Given that we have only the domain in hand, how would you enforce that?
> > > > >
> > > > > >From this discussion, it also looks like there is a mismatch between the
> > > > > implementation and the understanding on QEMU devel. So I am a bit
> > > > > concerned that this is not stable and may change in future Linux version.
> > > > >
> > > > > IOW, we are know tying Xen to Linux. So could we implement
> > > > > PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
> > > > > really represent the segment?
> > > >
> > > > I don't think we are tying Xen to Linux. Rob has already said that
> > > > linux,pci-domain is basically a generic device tree property.
> > >
> > > My concern is not so much the name of the property, but the definition of it.
> > >
> > > AFAICT, from this thread there can be two interpretation:
> > > - domain == segment
> > > - domain == (segment, bus)
> >
> > I think domain is just an alias for segment, the difference seems to
> > be that when using DT all bridges get a different segment (or domain)
> > number, and thus you will always end up starting numbering at bus 0
> > for each bridge?
> >
> > Ideally you would need a way to specify the segment and start/end bus
> > numbers of each bridge, if not you cannot match what ACPI does. Albeit
> > it might be fine as long as the OS and Xen agree on the segments and
> > bus numbers that belong to each bridge (and thus each ECAM region).
>
> That is what I thought and it is why I was asking to clarify the naming
> and/or writing a document to explain the assumptions, if any.
>
> Then after Julien's email I followed up in the Linux codebase and
> clearly there is a different assumption baked in the Linux kernel for
> architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).
>
> The assumption is that segment == domain == unique host bridge. It
> looks like it is coming from IEEE Std 1275-1994 but I am not certain.
> In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly
> match. So I am starting to think that domain == segment for IEEE Std
> 1275-1994 compliant device tree based systems.
I don't think the ACPI MCFG spec contains the notion of bridges, it
just describes ECAM (or MMCFG) regions, but those could be made up by
concatenating different bridge ECAM regions by the firmware itself, so
you could AFAICT end up with multiple bridges being aggregated into a
single ECAM region, and thus using the same segment number, which
seems not possible with the DT spec, where each bridge must get a
different segment number?
If you could assign both a segment number and a bus start and end
values to a bridge then I think it would be kind of equivalent to ACPI
MCFG.
I assume we would never support a system where Xen is getting the
hardware description from a DT and the hardware domain is using ACPI
(or the other way around)?
If so, I don't think we care that enumeration when using DT is
different than when using ACPI, as we can only guarantee consistency
when both Xen and the hardware domain use the same source for the
hardware description.
If when using DT each bridge has a unique segment number that's fine
as long as Xen and the OS agree to not change such values.
Roger.
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
2020-07-28 8:33 ` Roger Pau Monné
@ 2020-07-28 18:33 ` Stefano Stabellini
0 siblings, 0 replies; 48+ messages in thread
From: Stefano Stabellini @ 2020-07-28 18:33 UTC (permalink / raw)
To: Roger Pau Monné
Cc: nd, Stefano Stabellini, Andrew Cooper, Bertrand Marquis,
Jan Beulich, xen-devel, Rahul Singh, Volodymyr Babchuk,
Julien Grall
[-- Attachment #1: Type: text/plain, Size: 5459 bytes --]
On Tue, 28 Jul 2020, Roger Pau Monné wrote:
> On Mon, Jul 27, 2020 at 05:06:25PM -0700, Stefano Stabellini wrote:
> > On Mon, 27 Jul 2020, Roger Pau Monné wrote:
> > > On Sat, Jul 25, 2020 at 10:59:50AM +0100, Julien Grall wrote:
> > > > On Sat, 25 Jul 2020 at 00:46, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > > >
> > > > > On Fri, 24 Jul 2020, Julien Grall wrote:
> > > > > > On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > > > > > > If they are not equal, then I fail to see why it would be useful to have this
> > > > > > > > value in Xen.
> > > > > > >
> > > > > > > I think that's because the domain is actually more convenient to use
> > > > > > > because a segment can span multiple PCI host bridges. So my
> > > > > > > understanding is that a segment alone is not sufficient to identify a
> > > > > > > host bridge. From a software implementation point of view it would be
> > > > > > > better to use domains.
> > > > > >
> > > > > > AFAICT, this would be a matter of one check vs two checks in Xen :).
> > > > > > But... looking at Linux, they will also use domain == segment for ACPI
> > > > > > (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
> > >
> > > You have to use the (segment, bus) tuple when doing a lookup because
> > > MMCFG regions on ACPI are defined for a segment and a bus range, you
> > > can have a MMCFG region that covers segment 0 bus [0, 20) and another
> > > MMCFG region that covers segment 0 bus [20, 255], and those will use
> > > different addresses in the MMIO space.
> >
> > Thanks for the clarification!
> >
> >
> > > > > > > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > > > > > > > Dom0 and Xen can synchronize on the segment number.
> > > > > > >
> > > > > > > I was hoping we could write down the assumption somewhere that for the
> > > > > > > cases we care about domain == segment, and error out if it is not the
> > > > > > > case.
> > > > > >
> > > > > > Given that we have only the domain in hand, how would you enforce that?
> > > > > >
> > > > > > >From this discussion, it also looks like there is a mismatch between the
> > > > > > implementation and the understanding on QEMU devel. So I am a bit
> > > > > > concerned that this is not stable and may change in future Linux version.
> > > > > >
> > > > > > IOW, we are know tying Xen to Linux. So could we implement
> > > > > > PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
> > > > > > really represent the segment?
> > > > >
> > > > > I don't think we are tying Xen to Linux. Rob has already said that
> > > > > linux,pci-domain is basically a generic device tree property.
> > > >
> > > > My concern is not so much the name of the property, but the definition of it.
> > > >
> > > > AFAICT, from this thread there can be two interpretation:
> > > > - domain == segment
> > > > - domain == (segment, bus)
> > >
> > > I think domain is just an alias for segment, the difference seems to
> > > be that when using DT all bridges get a different segment (or domain)
> > > number, and thus you will always end up starting numbering at bus 0
> > > for each bridge?
> > >
> > > Ideally you would need a way to specify the segment and start/end bus
> > > numbers of each bridge, if not you cannot match what ACPI does. Albeit
> > > it might be fine as long as the OS and Xen agree on the segments and
> > > bus numbers that belong to each bridge (and thus each ECAM region).
> >
> > That is what I thought and it is why I was asking to clarify the naming
> > and/or writing a document to explain the assumptions, if any.
> >
> > Then after Julien's email I followed up in the Linux codebase and
> > clearly there is a different assumption baked in the Linux kernel for
> > architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).
> >
> > The assumption is that segment == domain == unique host bridge. It
> > looks like it is coming from IEEE Std 1275-1994 but I am not certain.
> > In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly
> > match. So I am starting to think that domain == segment for IEEE Std
> > 1275-1994 compliant device tree based systems.
>
> I don't think the ACPI MCFG spec contains the notion of bridges, it
> just describes ECAM (or MMCFG) regions, but those could be made up by
> concatenating different bridge ECAM regions by the firmware itself, so
> you could AFAICT end up with multiple bridges being aggregated into a
> single ECAM region, and thus using the same segment number, which
> seems not possible with the DT spec, where each bridge must get a
> different segment number?
Yes, that's my understanding too
> If you could assign both a segment number and a bus start and end
> values to a bridge then I think it would be kind of equivalent to ACPI
> MCFG.
>
> I assume we would never support a system where Xen is getting the
> hardware description from a DT and the hardware domain is using ACPI
> (or the other way around)?
Yeah, I think it is a good assumption
> If so, I don't think we care that enumeration when using DT is
> different than when using ACPI, as we can only guarantee consistency
> when both Xen and the hardware domain use the same source for the
> hardware description.
>
> If when using DT each bridge has a unique segment number that's fine
> as long as Xen and the OS agree to not change such values.
I agree
^ permalink raw reply [flat|nested] 48+ messages in thread
end of thread, other threads:[~2020-07-28 18:34 UTC | newest]
Thread overview: 48+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-23 15:40 [RFC PATCH v1 0/4] PCI devices passthrough on Arm Rahul Singh
2020-07-23 15:40 ` [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM Rahul Singh
2020-07-23 23:38 ` Stefano Stabellini
2020-07-24 7:03 ` Oleksandr Andrushchenko
2020-07-24 8:05 ` Julien Grall
2020-07-24 17:47 ` Stefano Stabellini
2020-07-27 15:27 ` Rahul Singh
2020-07-27 15:20 ` Rahul Singh
2020-07-24 8:44 ` Julien Grall
2020-07-24 17:41 ` Stefano Stabellini
2020-07-24 18:21 ` Julien Grall
2020-07-24 18:32 ` Stefano Stabellini
2020-07-24 19:24 ` Julien Grall
2020-07-24 23:46 ` Stefano Stabellini
2020-07-25 9:59 ` Julien Grall
2020-07-27 11:06 ` Roger Pau Monné
2020-07-28 0:06 ` Stefano Stabellini
2020-07-28 8:33 ` Roger Pau Monné
2020-07-28 18:33 ` Stefano Stabellini
2020-07-26 7:01 ` Jan Beulich
2020-07-27 13:27 ` Rahul Singh
2020-07-24 8:23 ` Julien Grall
2020-07-27 15:29 ` Rahul Singh
2020-07-24 14:44 ` Roger Pau Monné
2020-07-24 15:15 ` Julien Grall
2020-07-24 15:29 ` Roger Pau Monné
2020-07-24 15:42 ` Roger Pau Monné
2020-07-24 15:46 ` Julien Grall
2020-07-24 16:01 ` Jan Beulich
2020-07-24 16:54 ` Julien Grall
2020-07-27 10:34 ` Roger Pau Monné
2020-07-28 8:06 ` Rahul Singh
2020-07-28 8:21 ` Roger Pau Monné
2020-07-23 15:40 ` [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
2020-07-23 20:44 ` Stefano Stabellini
2020-07-24 7:14 ` Oleksandr Andrushchenko
2020-07-24 8:19 ` Julien Grall
2020-07-27 16:10 ` Rahul Singh
2020-07-24 14:49 ` Roger Pau Monné
2020-07-27 8:40 ` Rahul Singh
2020-07-23 15:40 ` [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
2020-07-23 23:39 ` Stefano Stabellini
2020-07-24 15:08 ` Roger Pau Monné
2020-07-23 15:40 ` [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
2020-07-23 23:39 ` Stefano Stabellini
2020-07-24 7:55 ` Oleksandr Andrushchenko
2020-07-24 9:11 ` Julien Grall
2020-07-27 13:40 ` Rahul Singh
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.