Hello All, The purpose of this patch series is to add PCI passthrough support to Xen on Arm. PCI passthrough support on ARM is the collaboration work between EPAM and ARM. ARM submitted the partial RFC [1][2] last year to get early feedback. We tried to fix all the comments and added more features to this patch series. Working POC with all the features can be found at [3]. Working POC is tested on x86 so that there will be no regression on x86. Design presentation can be found at [4] PCI passthrough support is divided into different patches. This patch series includes following features: Preparatory work to implement the PCI passthrough support for the ARM: - Refactor MSI code. - Fixed compilation error when HAS_PCI enabled for ARM. Discovering PCI Host Bridge in XEN: - PCI init to initialize the PCI driver. - PCI host bridge discovery in XEN and map the PCI ECAM configuration space to the XEN memory. - PCI access functions. Discovering PCI devices: - To support the PCI passthrough, XEN should be aware of the PCI devices. - Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then communicate to the XEN via a hypercall to add the PCI devices in XEN. Enable the existing x86 virtual PCI support for ARM: - Add VPCI trap handler for each of the PCI device added for config space access. - Register the trap handler in XEN for each of the host bridge PCI ECAM config space access. Emulated PCI device tree node in libxl: - Create a virtual PCI device tree node in libxl to enable the guest OS to discover the virtual PCI during guest boot. This patch series does not inlcude the following features. Following features will be send for review in the next version of the patch series once initial patch series merged. - VPCI support for DOMU guests (Non-identity mappings guest view of the BARs) - Virtual bus topology implementation - IOMMU related changes (generic, SMMUv2, SMMUv3) - MSI support for DOMU guests. - Virual ITS support for DOMU guests [1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg01184.html [2] https://lists.xenproject.org/archives/html/xen-devel/2020-07/threads.html#01184 [3] https://gitlab.com/rahsingh/xen-integration/-/commits/pci-passthrough-upstream-all [4] https://static.sched.com/hosted_files/xen2021/e4/PCI_Device_Passthrough_On_Arm.pdf Oleksandr Andrushchenko (1): xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh (13): xen/pci: Refactor MSI code that implements MSI functionality within XEN xen/pci: solve compilation error on ARM with HAS_PCI enabled xen/pci: solve compilation error on ARM with ACPI && HAS_PCI xen/arm: Add support for PCI init to initialize the PCI driver. xen/arm: PCI host bridge discovery within XEN on ARM xen/arm: Add support for PCI ecam operations xen:arm: Implement pci access functions xen/arm: Add cmdline boot option "pci=on" xen/arm: Discovering PCI devices and add the PCI devices in XEN. xen/arm: Enable the existing x86 virtual PCI support for ARM. arm/libxl: Emulated PCI device tree node in libxl xen/arm: Fixed error when PCI device is assigned to guest xen/arm: Add linux,pci-domain property for hwdom if not available. tools/libs/light/libxl_arm.c | 109 +++++++++ tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 2 + xen/arch/arm/Makefile | 2 + xen/arch/arm/domain.c | 4 + xen/arch/arm/domain_build.c | 18 ++ xen/arch/arm/domctl.c | 2 + xen/arch/arm/pci/Makefile | 6 + xen/arch/arm/pci/ecam.c | 63 ++++++ xen/arch/arm/pci/pci-access.c | 143 ++++++++++++ xen/arch/arm/pci/pci-host-common.c | 312 ++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 59 +++++ xen/arch/arm/pci/pci-host-zynqmp.c | 59 +++++ xen/arch/arm/pci/pci.c | 116 ++++++++++ xen/arch/arm/physdev.c | 42 +++- xen/arch/arm/vpci.c | 96 ++++++++ xen/arch/arm/vpci.h | 37 +++ xen/arch/x86/Kconfig | 1 + xen/drivers/passthrough/Makefile | 1 + xen/drivers/passthrough/arm/iommu.c | 5 + xen/drivers/passthrough/msi.c | 96 ++++++++ xen/drivers/passthrough/pci.c | 63 ++---- xen/drivers/pci/Kconfig | 4 + xen/drivers/vpci/Makefile | 3 +- xen/drivers/vpci/header.c | 2 + xen/include/asm-arm/device.h | 1 + xen/include/asm-arm/domain.h | 5 +- xen/include/asm-arm/pci.h | 106 ++++++++- xen/include/public/arch-arm.h | 15 ++ xen/include/public/device_tree_defs.h | 1 + xen/include/xen/msi.h | 56 +++++ xen/xsm/flask/hooks.c | 8 +- 32 files changed, 1384 insertions(+), 54 deletions(-) create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/ecam.c create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c create mode 100644 xen/arch/arm/pci/pci.c create mode 100644 xen/arch/arm/vpci.c create mode 100644 xen/arch/arm/vpci.h create mode 100644 xen/drivers/passthrough/msi.c create mode 100644 xen/include/xen/msi.h -- 2.17.1
MSI code that implements MSI functionality to support MSI within XEN is not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate the code for ARM. No functional change intended. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/x86/Kconfig | 1 + xen/drivers/passthrough/Makefile | 1 + xen/drivers/passthrough/msi.c | 96 ++++++++++++++++++++++++++++++++ xen/drivers/passthrough/pci.c | 54 +++++------------- xen/drivers/pci/Kconfig | 4 ++ xen/include/xen/msi.h | 56 +++++++++++++++++++ xen/xsm/flask/hooks.c | 8 +-- 7 files changed, 175 insertions(+), 45 deletions(-) create mode 100644 xen/drivers/passthrough/msi.c create mode 100644 xen/include/xen/msi.h diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig index 9b164db641..7b46ee98c5 100644 --- a/xen/arch/x86/Kconfig +++ b/xen/arch/x86/Kconfig @@ -19,6 +19,7 @@ config X86 select HAS_NS16550 select HAS_PASSTHROUGH select HAS_PCI + select HAS_PCI_MSI select HAS_PDX select HAS_SCHED_GRANULARITY select HAS_UBSAN diff --git a/xen/drivers/passthrough/Makefile b/xen/drivers/passthrough/Makefile index 445690e3e5..a5efa22714 100644 --- a/xen/drivers/passthrough/Makefile +++ b/xen/drivers/passthrough/Makefile @@ -7,3 +7,4 @@ obj-y += iommu.o obj-$(CONFIG_HAS_PCI) += pci.o obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o obj-$(CONFIG_HAS_PCI) += ats.o +obj-$(CONFIG_HAS_PCI_MSI) += msi.o diff --git a/xen/drivers/passthrough/msi.c b/xen/drivers/passthrough/msi.c new file mode 100644 index 0000000000..15ad0f8160 --- /dev/null +++ b/xen/drivers/passthrough/msi.c @@ -0,0 +1,96 @@ +/* + * Copyright (C) 2008, Netronome Systems, Inc. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * this program; If not, see <http://www.gnu.org/licenses/>. + */ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/msi.h> +#include <asm/hvm/io.h> + +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) +{ + int rc; + + if ( pdev->msix ) + { + rc = pci_reset_msix_state(pdev); + if ( rc ) + return rc; + msixtbl_init(d); + } + + return 0; +} + +int pdev_msi_init(struct pci_dev *pdev) +{ + unsigned int pos; + + INIT_LIST_HEAD(&pdev->msi_list); + + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI); + if ( pos ) + { + uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos)); + + pdev->msi_maxvec = multi_msi_capable(ctrl); + } + + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX); + if ( pos ) + { + struct arch_msix *msix = xzalloc(struct arch_msix); + uint16_t ctrl; + + if ( !msix ) + return -ENOMEM; + + spin_lock_init(&msix->table_lock); + + ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); + msix->nr_entries = msix_table_size(ctrl); + + pdev->msix = msix; + } + + return 0; +} + +void pdev_msi_deinit(struct pci_dev *pdev) +{ + XFREE(pdev->msix); +} + +void pdev_dump_msi(const struct pci_dev *pdev) +{ + const struct msi_desc *msi; + + printk("- MSIs < "); + list_for_each_entry ( msi, &pdev->msi_list, list ) + printk("%d ", msi->irq); + printk(">"); +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index fc4fa2e5c3..67f5686ab6 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -32,8 +32,8 @@ #include <xen/softirq.h> #include <xen/tasklet.h> #include <xen/vpci.h> +#include <xen/msi.h> #include <xsm/xsm.h> -#include <asm/msi.h> #include "ats.h" struct pci_seg { @@ -314,6 +314,7 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) { struct pci_dev *pdev; unsigned int pos; + int rc; list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) if ( pdev->bus == bus && pdev->devfn == devfn ) @@ -327,35 +328,12 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) *((u8*) &pdev->bus) = bus; *((u8*) &pdev->devfn) = devfn; pdev->domain = NULL; - INIT_LIST_HEAD(&pdev->msi_list); - - pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), - PCI_CAP_ID_MSI); - if ( pos ) - { - uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos)); - - pdev->msi_maxvec = multi_msi_capable(ctrl); - } - pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), - PCI_CAP_ID_MSIX); - if ( pos ) + rc = pdev_msi_init(pdev); + if ( rc ) { - struct arch_msix *msix = xzalloc(struct arch_msix); - uint16_t ctrl; - - if ( !msix ) - { - xfree(pdev); - return NULL; - } - spin_lock_init(&msix->table_lock); - - ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); - msix->nr_entries = msix_table_size(ctrl); - - pdev->msix = msix; + xfree(pdev); + return NULL; } list_add(&pdev->alldevs_list, &pseg->alldevs_list); @@ -449,7 +427,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev *pdev) } list_del(&pdev->alldevs_list); - xfree(pdev->msix); + pdev_msi_deinit(pdev); xfree(pdev); } @@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) static int _dump_pci_devices(struct pci_seg *pseg, void *arg) { struct pci_dev *pdev; - struct msi_desc *msi; printk("==== segment %04x ====\n", pseg->nr); list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) { - printk("%pp - %pd - node %-3d - MSIs < ", + printk("%pp - %pd - node %-3d ", &pdev->sbdf, pdev->domain, (pdev->node != NUMA_NO_NODE) ? pdev->node : -1); - list_for_each_entry ( msi, &pdev->msi_list, list ) - printk("%d ", msi->irq); - printk(">\n"); + pdev_dump_msi(pdev); + printk("\n"); } return 0; @@ -1422,13 +1398,9 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) ASSERT(pdev && (pdev->domain == hardware_domain || pdev->domain == dom_io)); - if ( pdev->msix ) - { - rc = pci_reset_msix_state(pdev); - if ( rc ) - goto done; - msixtbl_init(d); - } + rc = pdev_msix_assign(d, pdev); + if ( rc ) + goto done; pdev->fault.count = 0; diff --git a/xen/drivers/pci/Kconfig b/xen/drivers/pci/Kconfig index 7da03fa13b..c6a7bc8007 100644 --- a/xen/drivers/pci/Kconfig +++ b/xen/drivers/pci/Kconfig @@ -1,3 +1,7 @@ config HAS_PCI bool + +config HAS_PCI_MSI + bool + depends on HAS_PCI diff --git a/xen/include/xen/msi.h b/xen/include/xen/msi.h new file mode 100644 index 0000000000..b2d5bc6f9d --- /dev/null +++ b/xen/include/xen/msi.h @@ -0,0 +1,56 @@ +/* + * Copyright (C) 2008, Netronome Systems, Inc. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * this program; If not, see <http://www.gnu.org/licenses/>. + */ + +#ifndef __XEN_MSI_H_ +#define __XEN_MSI_H_ + +#ifdef CONFIG_HAS_PCI_MSI + +#include <asm/msi.h> + +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev); +int pdev_msi_init(struct pci_dev *pdev); +void pdev_msi_deinit(struct pci_dev *pdev); +void pdev_dump_msi(const struct pci_dev *pdev); + +#else /* !CONFIG_HAS_PCI_MSI */ +static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) +{ + return 0; +} + +static inline int pdev_msi_init(struct pci_dev *pdev) +{ + return 0; +} + +static inline void pdev_msi_deinit(struct pci_dev *pdev) {} +static inline void pci_cleanup_msi(struct pci_dev *pdev) {} +static inline void pdev_dump_msi(const struct pci_dev *pdev) {} + +#endif /* CONFIG_HAS_PCI_MSI */ + +#endif /* __XEN_MSI_H */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c index f1a1217c98..fdcfeb984c 100644 --- a/xen/xsm/flask/hooks.c +++ b/xen/xsm/flask/hooks.c @@ -21,7 +21,7 @@ #include <xen/guest_access.h> #include <xen/xenoprof.h> #include <xen/iommu.h> -#ifdef CONFIG_HAS_PCI +#ifdef CONFIG_HAS_PCI_MSI #include <asm/msi.h> #endif #include <public/xen.h> @@ -114,7 +114,7 @@ static int get_irq_sid(int irq, u32 *sid, struct avc_audit_data *ad) } return security_irq_sid(irq, sid); } -#ifdef CONFIG_HAS_PCI +#ifdef CONFIG_HAS_PCI_MSI { struct irq_desc *desc = irq_to_desc(irq); if ( desc->msi_desc && desc->msi_desc->dev ) { @@ -874,7 +874,7 @@ static int flask_map_domain_pirq (struct domain *d) static int flask_map_domain_msi (struct domain *d, int irq, const void *data, u32 *sid, struct avc_audit_data *ad) { -#ifdef CONFIG_HAS_PCI +#ifdef CONFIG_HAS_PCI_MSI const struct msi_info *msi = data; u32 machine_bdf = (msi->seg << 16) | (msi->bus << 8) | msi->devfn; @@ -940,7 +940,7 @@ static int flask_unmap_domain_pirq (struct domain *d) static int flask_unmap_domain_msi (struct domain *d, int irq, const void *data, u32 *sid, struct avc_audit_data *ad) { -#ifdef CONFIG_HAS_PCI +#ifdef CONFIG_HAS_PCI_MSI const struct pci_dev *pdev = data; u32 machine_bdf = (pdev->seg << 16) | (pdev->bus << 8) | pdev->devfn; -- 2.17.1
Compilation error is observed when HAS_PCI is enabled for ARM architecture. Add definition for arch_iommu_use_permitted() and arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to fix compilation error. pci.c: In function ‘deassign_device’: pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’; did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration] pci_to_dev(pdev)); pci.c:18: undefined reference to `pci_conf_read16’ pci.c:880: undefined reference to `arch_pci_clean_pirqs’ pci.c:1392: undefined reference to `arch_iommu_use_permitted' Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 2 + xen/arch/arm/pci/pci-access.c | 61 +++++++++++++++++++++++++++++ xen/arch/arm/pci/pci.c | 32 +++++++++++++++ xen/drivers/passthrough/arm/iommu.c | 5 +++ xen/include/asm-arm/pci.h | 33 ++++++++++++++-- 6 files changed, 131 insertions(+), 3 deletions(-) create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci.c diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile index 3d3b97b5b4..0e14a5e5c8 100644 --- a/xen/arch/arm/Makefile +++ b/xen/arch/arm/Makefile @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ +obj-$(CONFIG_HAS_PCI) += pci/ obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.o diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile new file mode 100644 index 0000000000..a9ee0b9b44 --- /dev/null +++ b/xen/arch/arm/pci/Makefile @@ -0,0 +1,2 @@ +obj-y += pci.o +obj-y += pci-access.o diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c new file mode 100644 index 0000000000..b938047c03 --- /dev/null +++ b/xen/arch/arm/pci/pci-access.c @@ -0,0 +1,61 @@ +/* + * Copyright (C) 2021 Arm Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <xen/pci.h> + +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, + unsigned int len) +{ + return ~0U; +} + +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, + unsigned int len, uint32_t val) +{ +} + +/* + * Wrappers for all PCI configuration access functions. + */ + +#define PCI_OP_WRITE(size, type) \ + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ +{ \ + pci_config_write(sbdf, reg, size / 8, val); \ +} + +#define PCI_OP_READ(size, type) \ + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \ +{ \ + return pci_config_read(sbdf, reg, size / 8); \ +} + +PCI_OP_READ(8, u8) +PCI_OP_READ(16, u16) +PCI_OP_READ(32, u32) +PCI_OP_WRITE(8, u8) +PCI_OP_WRITE(16, u16) +PCI_OP_WRITE(32, u32) + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c new file mode 100644 index 0000000000..dc55d23778 --- /dev/null +++ b/xen/arch/arm/pci/pci.c @@ -0,0 +1,32 @@ +/* + * Copyright (C) 2021 Arm Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <xen/pci.h> + +int arch_pci_clean_pirqs(struct domain *d) +{ + return 0; +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c index db3b07a571..fdec1c5547 100644 --- a/xen/drivers/passthrough/arm/iommu.c +++ b/xen/drivers/passthrough/arm/iommu.c @@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d) void __hwdom_init arch_iommu_hwdom_init(struct domain *d) { } + +bool arch_iommu_use_permitted(const struct domain *d) +{ + return true; +} diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index de13359f65..61e43da088 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -1,7 +1,34 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* + * Copyright (C) 2021 Arm Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ +#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ + +#ifdef CONFIG_HAS_PCI + +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev) + +/* Arch pci dev struct */ struct arch_pci_dev { + struct device dev; }; -#endif /* __X86_PCI_H__ */ +#else /*!CONFIG_HAS_PCI*/ + +struct arch_pci_dev { }; + +#endif /*!CONFIG_HAS_PCI*/ +#endif /* __ARM_PCI_H__ */ -- 2.17.1
Compilation error is observed when ACPI and HAS_PCI is enabled for ARM architecture. Move the code under CONFIG_X86 flag to gate the code for ARM. No functional change intended. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/drivers/passthrough/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 67f5686ab6..c23c8cb06b 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -1150,7 +1150,7 @@ void __hwdom_init setup_hwdom_pci_devices( pcidevs_unlock(); } -#ifdef CONFIG_ACPI +#if defined(CONFIG_ACPI) && defined(CONFIG_X86) #include <acpi/acpi.h> #include <acpi/apei.h> -- 2.17.1
pci_init(..) will be called during xen startup to initialize and probe the PCI host-bridge driver. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/pci/pci.c | 54 ++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/device.h | 1 + 2 files changed, 55 insertions(+) diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c index dc55d23778..d1c9cf997d 100644 --- a/xen/arch/arm/pci/pci.c +++ b/xen/arch/arm/pci/pci.c @@ -14,13 +14,67 @@ * along with this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <xen/acpi.h> +#include <xen/device_tree.h> +#include <xen/errno.h> +#include <xen/init.h> #include <xen/pci.h> +#include <xen/param.h> int arch_pci_clean_pirqs(struct domain *d) { return 0; } +static int __init dt_pci_init(void) +{ + struct dt_device_node *np; + int rc; + + dt_for_each_device_node(dt_host, np) + { + rc = device_init(np, DEVICE_PCI, NULL); + if( !rc ) + continue; + /* + * Ignore the following error codes: + * - EBADF: Indicate the current is not an pci + * - ENODEV: The pci device is not present or cannot be used by + * Xen. + */ + else if ( rc != -EBADF && rc != -ENODEV ) + { + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n"); + return rc; + } + } + + return 0; +} + +#ifdef CONFIG_ACPI +static void __init acpi_pci_init(void) +{ + printk(XENLOG_ERR "ACPI pci init not supported \n"); + return; +} +#else +static inline void __init acpi_pci_init(void) { } +#endif + +static int __init pci_init(void) +{ + if ( acpi_disabled ) + dt_pci_init(); + else + acpi_pci_init(); + + pci_segments_init(); + + return 0; +} +__initcall(pci_init); + /* * Local variables: * mode: C diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h index ee7cff2d44..5ecd5e7bd1 100644 --- a/xen/include/asm-arm/device.h +++ b/xen/include/asm-arm/device.h @@ -34,6 +34,7 @@ enum device_class DEVICE_SERIAL, DEVICE_IOMMU, DEVICE_GIC, + DEVICE_PCI, /* Use for error */ DEVICE_UNKNOWN, }; -- 2.17.1
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory. As of now "pci-host-ecam-generic" compatible board is supported. "linux,pci-domain" device tree property assigns a fixed PCI domain number to a host bridge, otherwise an unstable (across boots) unique number will be assigned by Linux.This property has to be in sync with XEN to access the PCI devices. XEN will read the “linux,pci-domain” property from the device tree node and configure the host bridge segment number accordingly. If this property is not available XEN will allocate the unique segment number to the host bridge. dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly imported from the Linux source tree. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/pci/Makefile | 2 + xen/arch/arm/pci/pci-host-common.c | 261 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 55 ++++++ xen/include/asm-arm/pci.h | 28 +++ 4 files changed, 346 insertions(+) create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile index a9ee0b9b44..f3d97f859e 100644 --- a/xen/arch/arm/pci/Makefile +++ b/xen/arch/arm/pci/Makefile @@ -1,2 +1,4 @@ obj-y += pci.o obj-y += pci-access.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c new file mode 100644 index 0000000000..9dd9b02271 --- /dev/null +++ b/xen/arch/arm/pci/pci-host-common.c @@ -0,0 +1,261 @@ +/* + * Copyright (C) 2021 Arm Ltd. + * + * Based on Linux drivers/pci/ecam.c + * Copyright 2016 Broadcom. + * + * Based on Linux drivers/pci/controller/pci-host-common.c + * Based on Linux drivers/pci/controller/pci-host-generic.c + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> + * + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> +#include <xen/sched.h> +#include <xen/vmap.h> + +/* + * List for all the pci host bridges. + */ + +static LIST_HEAD(pci_host_bridges); + +static atomic_t domain_nr = ATOMIC_INIT(-1); + +bool dt_pci_parse_bus_range(struct dt_device_node *dev, + struct pci_config_window *cfg) +{ + const __be32 *cells; + uint32_t len; + + cells = dt_get_property(dev, "bus-range", &len); + /* bus-range should at least be 2 cells */ + if ( !cells || (len < (sizeof(*cells) * 2)) ) + return false; + + cfg->busn_start = dt_next_cell(1, &cells); + cfg->busn_end = dt_next_cell(1, &cells); + + return true; +} + +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len) +{ + return ioremap_nocache(start, len); +} + +static void pci_ecam_free(struct pci_config_window *cfg) +{ + if ( cfg->win ) + iounmap(cfg->win); + + xfree(cfg); +} + +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, + int ecam_reg_idx) +{ + int err; + struct pci_config_window *cfg; + paddr_t addr, size; + + cfg = xzalloc(struct pci_config_window); + if ( !cfg ) + return NULL; + + err = dt_pci_parse_bus_range(dev, cfg); + if ( !err ) { + cfg->busn_start = 0; + cfg->busn_end = 0xff; + printk(XENLOG_ERR "%s:No bus range found for pci controller\n", + dt_node_full_name(dev)); + } else { + if ( cfg->busn_end > cfg->busn_start + 0xff ) + cfg->busn_end = cfg->busn_start + 0xff; + } + + /* Parse our PCI ecam register address*/ + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size); + if ( err ) + goto err_exit; + + cfg->phys_addr = addr; + cfg->size = size; + + /* + * On 64-bit systems, we do a single ioremap for the whole config space + * since we have enough virtual address range available. On 32-bit, we + * ioremap the config space for each bus individually. + * + * As of now only 64-bit is supported 32-bit is not supported. + */ + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size); + if ( !cfg->win ) + goto err_exit_remap; + + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, + cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); + + return cfg; + +err_exit_remap: + printk(XENLOG_ERR "ECAM ioremap failed\n"); +err_exit: + pci_ecam_free(cfg); + return NULL; +} + +struct pci_host_bridge *pci_alloc_host_bridge(void) +{ + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); + + if ( !bridge ) + return NULL; + + INIT_LIST_HEAD(&bridge->node); + bridge->bus_start = ~0; + bridge->bus_end = ~0; + return bridge; +} + +void pci_add_host_bridge(struct pci_host_bridge *bridge) +{ + list_add_tail(&bridge->node, &pci_host_bridges); +} + +static int pci_get_new_domain_nr(void) +{ + return atomic_inc_return(&domain_nr); +} + +/* + * This function will try to obtain the host bridge domain number by + * finding a property called "linux,pci-domain" of the given device node. + * + * @node: device tree node with the domain information + * + * Returns the associated domain number from DT in the range [0-0xffff], or + * a negative value if the required property is not found. + */ +static int dt_get_pci_domain_nr(struct dt_device_node *node) +{ + u32 domain; + int error; + + error = dt_property_read_u32(node, "linux,pci-domain", &domain); + if ( !error ) + return -EINVAL; + + return (u16)domain; +} + +static int pci_bus_find_domain_nr(struct dt_device_node *dev) +{ + static int use_dt_domains = -1; + int domain; + + domain = dt_get_pci_domain_nr(dev); + + /* + * Check DT domain and use_dt_domains values. + * + * If DT domain property is valid (domain >= 0) and + * use_dt_domains != 0, the DT assignment is valid since this means + * we have not previously allocated a domain number by using + * pci_get_new_domain_nr(); we should also update use_dt_domains to + * 1, to indicate that we have just assigned a domain number from + * DT. + * + * If DT domain property value is not valid (ie domain < 0), and we + * have not previously assigned a domain number from DT + * (use_dt_domains != 1) we should assign a domain number by + * using the: + * + * pci_get_new_domain_nr() + * + * API and update the use_dt_domains value to keep track of method we + * are using to assign domain numbers (use_dt_domains = 0). + * + * All other combinations imply we have a platform that is trying + * to mix domain numbers obtained from DT and pci_get_new_domain_nr(), + * which is a recipe for domain mishandling and it is prevented by + * invalidating the domain value (domain = -1) and printing a + * corresponding error. + */ + if ( domain >= 0 && use_dt_domains ) + { + use_dt_domains = 1; + } + else if ( domain < 0 && use_dt_domains != 1 ) + { + use_dt_domains = 0; + domain = pci_get_new_domain_nr(); + } + else + { + printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n"); + BUG(); + } + + return domain; +} + +int pci_host_common_probe(struct dt_device_node *dev, + int ecam_reg_idx) +{ + struct pci_host_bridge *bridge; + struct pci_config_window *cfg; + int err; + + bridge = pci_alloc_host_bridge(); + if ( !bridge ) + return -ENOMEM; + + /* Parse and map our Configuration Space windows */ + cfg = gen_pci_init(dev, ecam_reg_idx); + if ( !cfg ) + { + err = -ENOMEM; + goto err_exit; + } + + bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->bus_start = cfg->busn_start; + bridge->bus_end = cfg->busn_end; + + bridge->segment = pci_bus_find_domain_nr(dev); + + pci_add_host_bridge(bridge); + + return 0; + +err_exit: + xfree(bridge); + return err; +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c new file mode 100644 index 0000000000..13d0f7f999 --- /dev/null +++ b/xen/arch/arm/pci/pci-host-generic.c @@ -0,0 +1,55 @@ +/* + * Copyright (C) 2021 Arm Ltd. + * + * Based on Linux drivers/pci/controller/pci-host-common.c + * Based on Linux drivers/pci/controller/pci-host-generic.c + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <asm/device.h> +#include <xen/pci.h> +#include <asm/pci.h> + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible = "pci-host-ecam-generic" }, + { }, +}; + +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) +{ + const struct dt_device_match *of_id; + + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); + + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", + dt_node_full_name(dev), of_id->compatible); + + return pci_host_common_probe(dev, 0); +} + +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index 61e43da088..58a51e724e 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -26,6 +26,34 @@ struct arch_pci_dev { struct device dev; }; +/* + * struct to hold the mappings of a config space window. This + * is expected to be used as sysdata for PCI controllers that + * use ECAM. + */ +struct pci_config_window { + paddr_t phys_addr; + paddr_t size; + uint8_t busn_start; + uint8_t busn_end; + void __iomem *win; +}; + +/* + * struct to hold pci host bridge information + * for a PCI controller. + */ +struct pci_host_bridge { + struct dt_device_node *dt_node; /* Pointer to the associated DT node */ + struct list_head node; /* Node in list of host bridges */ + uint16_t segment; /* Segment number */ + u8 bus_start; /* Bus start of this bridge. */ + u8 bus_end; /* Bus end of this bridge. */ + void *sysdata; /* Pointer to the config space window*/ +}; + +int pci_host_common_probe(struct dt_device_node *dev, + int ecam_reg_idx); #else /*!CONFIG_HAS_PCI*/ struct arch_pci_dev { }; -- 2.17.1
Add support for PCI ecam operations to access the PCI configuration space. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/pci/Makefile | 1 + xen/arch/arm/pci/ecam.c | 63 +++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-access.c | 53 ++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 13 +++++- xen/arch/arm/pci/pci-host-generic.c | 8 +++- xen/include/asm-arm/pci.h | 32 +++++++++++++++ 6 files changed, 167 insertions(+), 3 deletions(-) create mode 100644 xen/arch/arm/pci/ecam.c diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile index f3d97f859e..6f32fbbe67 100644 --- a/xen/arch/arm/pci/Makefile +++ b/xen/arch/arm/pci/Makefile @@ -2,3 +2,4 @@ obj-y += pci.o obj-y += pci-access.o obj-y += pci-host-generic.o obj-y += pci-host-common.o +obj-y += ecam.o diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c new file mode 100644 index 0000000000..91c691b41f --- /dev/null +++ b/xen/arch/arm/pci/ecam.c @@ -0,0 +1,63 @@ +/* + * Copyright (C) 2021 Arm Ltd. + * + * Based on Linux drivers/pci/ecam.c + * Copyright 2016 Broadcom + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <xen/pci.h> +#include <xen/sched.h> + +/* + * Function to implement the pci_ops ->map_bus method. + */ +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, + uint32_t sbdf, uint32_t where) +{ + const struct pci_config_window *cfg = bridge->sysdata; + unsigned int devfn_shift = cfg->ops->bus_shift - 8; + void __iomem *base; + + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ; + unsigned int busn = sbdf_t.bus; + + if ( busn < cfg->busn_start || busn > cfg->busn_end ) + return NULL; + + busn -= cfg->busn_start; + base = cfg->win + (busn << cfg->ops->bus_shift); + + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; +} + +/* ECAM ops */ +const struct pci_ecam_ops pci_generic_ecam_ops = { + .bus_shift = 20, + .pci_ops = { + .map_bus = pci_ecam_map_bus, + .read = pci_generic_config_read, + .write = pci_generic_config_write, + } +}; + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c index b938047c03..f39f6a3a38 100644 --- a/xen/arch/arm/pci/pci-access.c +++ b/xen/arch/arm/pci/pci-access.c @@ -15,6 +15,59 @@ */ #include <xen/pci.h> +#include <asm/io.h> + +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, + uint32_t reg, uint32_t len, uint32_t *value) +{ + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); + if (!addr) { + *value = ~0; + return -ENODEV; + } + + switch (len) + { + case 1: + *value = readb(addr); + break; + case 2: + *value = readw(addr); + break; + case 4: + *value = readl(addr); + break; + default: + BUG(); + } + + return 0; +} + +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, + uint32_t reg, uint32_t len, uint32_t value) +{ + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); + if (!addr) + return -ENODEV; + + switch (len) + { + case 1: + writeb(value, addr); + break; + case 2: + writew(value, addr); + break; + case 4: + writel(value, addr); + break; + default: + BUG(); + } + + return 0; +} static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len) diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c index 9dd9b02271..c582527e92 100644 --- a/xen/arch/arm/pci/pci-host-common.c +++ b/xen/arch/arm/pci/pci-host-common.c @@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg) } static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, + const struct pci_ecam_ops *ops, int ecam_reg_idx) { int err; @@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, cfg->phys_addr = addr; cfg->size = size; + cfg->ops = ops; /* * On 64-bit systems, we do a single ioremap for the whole config space @@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); + if ( ops->init ) + { + err = ops->init(cfg); + if (err) + goto err_exit; + } + return cfg; err_exit_remap: @@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev) } int pci_host_common_probe(struct dt_device_node *dev, + const struct pci_ecam_ops *ops, int ecam_reg_idx) { struct pci_host_bridge *bridge; @@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev, return -ENOMEM; /* Parse and map our Configuration Space windows */ - cfg = gen_pci_init(dev, ecam_reg_idx); + cfg = gen_pci_init(dev, ops, ecam_reg_idx); if ( !cfg ) { err = -ENOMEM; @@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev, bridge->dt_node = dev; bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops; bridge->bus_start = cfg->busn_start; bridge->bus_end = cfg->busn_end; diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c index 13d0f7f999..2d652e8910 100644 --- a/xen/arch/arm/pci/pci-host-generic.c +++ b/xen/arch/arm/pci/pci-host-generic.c @@ -23,20 +23,24 @@ #include <asm/pci.h> static const struct dt_device_match gen_pci_dt_match[] = { - { .compatible = "pci-host-ecam-generic" }, + { .compatible = "pci-host-ecam-generic", + .data = &pci_generic_ecam_ops }, + { }, }; static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) { const struct dt_device_match *of_id; + const struct pci_ecam_ops *ops; of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", dt_node_full_name(dev), of_id->compatible); - return pci_host_common_probe(dev, 0); + return pci_host_common_probe(dev, ops, 0); } DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index 58a51e724e..22866244d2 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -37,6 +37,7 @@ struct pci_config_window { uint8_t busn_start; uint8_t busn_end; void __iomem *win; + const struct pci_ecam_ops *ops; }; /* @@ -50,10 +51,41 @@ struct pci_host_bridge { u8 bus_start; /* Bus start of this bridge. */ u8 bus_end; /* Bus end of this bridge. */ void *sysdata; /* Pointer to the config space window*/ + const struct pci_ops *ops; }; +struct pci_ops { + void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf, + uint32_t offset); + int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf, + uint32_t reg, uint32_t len, uint32_t *value); + int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf, + uint32_t reg, uint32_t len, uint32_t value); +}; + +/* + * struct to hold pci ops and bus shift of the config window + * for a PCI controller. + */ +struct pci_ecam_ops { + unsigned int bus_shift; + struct pci_ops pci_ops; + int (*init)(struct pci_config_window *); +}; + +/* Default ECAM ops */ +extern const struct pci_ecam_ops pci_generic_ecam_ops; + int pci_host_common_probe(struct dt_device_node *dev, + const struct pci_ecam_ops *ops, int ecam_reg_idx); +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, + uint32_t reg, uint32_t len, uint32_t *value); +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, + uint32_t reg, uint32_t len, uint32_t value); +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, + uint32_t sbdf, uint32_t where); + #else /*!CONFIG_HAS_PCI*/ struct arch_pci_dev { }; -- 2.17.1
From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> Add support for Xilinx ZynqMP PCI host controller to map the PCI config space to the XEN memory. Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> --- xen/arch/arm/pci/Makefile | 1 + xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile index 6f32fbbe67..1d045ade01 100644 --- a/xen/arch/arm/pci/Makefile +++ b/xen/arch/arm/pci/Makefile @@ -3,3 +3,4 @@ obj-y += pci-access.o obj-y += pci-host-generic.o obj-y += pci-host-common.o obj-y += ecam.o +obj-y += pci-host-zynqmp.o diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c new file mode 100644 index 0000000000..fe103e3855 --- /dev/null +++ b/xen/arch/arm/pci/pci-host-zynqmp.c @@ -0,0 +1,59 @@ +/* + * Copyright (C) 2020-2021 EPAM Systems + * + * Based on Linux drivers/pci/controller/pci-host-common.c + * Based on Linux drivers/pci/controller/pci-host-generic.c + * Based on xen/arch/arm/pci/pci-host-generic.c + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <asm/device.h> +#include <xen/pci.h> +#include <asm/pci.h> + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible = "xlnx,nwl-pcie-2.11", + .data = &pci_generic_ecam_ops }, + { }, +}; + +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) +{ + const struct dt_device_match *of_id; + const struct pci_ecam_ops *ops; + + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; + + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", + dt_node_full_name(dev), of_id->compatible); + + return pci_host_common_probe(dev, ops, 2); +} + +DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ -- 2.17.1
Implement generic pci access functions to read/write the configuration space. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/pci/pci-access.c | 31 +++++++++++++++++++++++++++++- xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++ xen/include/asm-arm/pci.h | 2 ++ 3 files changed, 51 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c index f39f6a3a38..b94de3c3ac 100644 --- a/xen/arch/arm/pci/pci-access.c +++ b/xen/arch/arm/pci/pci-access.c @@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int len) { - return ~0U; + uint32_t val = GENMASK(0, len * 8); + + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); + + if ( unlikely(!bridge) ) + { + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + return val; + } + + if ( unlikely(!bridge->ops->read) ) + return val; + + bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val); + + return val; } static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int len, uint32_t val) { + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); + + if ( unlikely(!bridge) ) + { + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + return; + } + + if ( unlikely(!bridge->ops->write) ) + return; + + bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val); } /* diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c index c582527e92..62715b4676 100644 --- a/xen/arch/arm/pci/pci-host-common.c +++ b/xen/arch/arm/pci/pci-host-common.c @@ -261,6 +261,25 @@ err_exit: return err; } +/* + * This function will lookup an hostbridge based on the segment and bus + * number. + */ +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus) +{ + struct pci_host_bridge *bridge; + + list_for_each_entry( bridge, &pci_host_bridges, node ) + { + if ( bridge->segment != segment ) + continue; + if ( (bus < bridge->bus_start) || (bus > bridge->bus_end) ) + continue; + return bridge; + } + + return NULL; +} /* * Local variables: * mode: C diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index 22866244d2..756f8637ab 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -20,6 +20,7 @@ #ifdef CONFIG_HAS_PCI #define pci_to_dev(pcidev) (&(pcidev)->arch.dev) +#define PRI_pci "%04x:%02x:%02x.%u" /* Arch pci dev struct */ struct arch_pci_dev { @@ -86,6 +87,7 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, uint32_t sbdf, uint32_t where); +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus); #else /*!CONFIG_HAS_PCI*/ struct arch_pci_dev { }; -- 2.17.1
Add cmdline boot option "pci=on" to enable/disable the PCI init during boot. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/pci/pci.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c index d1c9cf997d..dc63bbc2a2 100644 --- a/xen/arch/arm/pci/pci.c +++ b/xen/arch/arm/pci/pci.c @@ -62,8 +62,38 @@ static void __init acpi_pci_init(void) static inline void __init acpi_pci_init(void) { } #endif +static bool __initdata param_pci_enable; + +static int __init parse_pci_param(const char *arg) +{ + if ( !arg ) + { + param_pci_enable = false; + return 0; + } + + switch ( parse_bool(arg, NULL) ) + { + case 0: + param_pci_enable = false; + return 0; + case 1: + param_pci_enable = true; + return 0; + } + + return -EINVAL; +} +custom_param("pci", parse_pci_param); + static int __init pci_init(void) { + /* + * Enable PCI when has been enabled explicitly (pci=on) + */ + if ( !param_pci_enable) + return 0; + if ( acpi_disabled ) dt_pci_init(); else -- 2.17.1
Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/physdev.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c index e91355fe22..ccce8f0eba 100644 --- a/xen/arch/arm/physdev.c +++ b/xen/arch/arm/physdev.c @@ -9,12 +9,45 @@ #include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) { - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd); - return -ENOSYS; + int ret = 0; + + switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI + case PHYSDEVOP_pci_device_add: { + struct physdev_pci_device_add add; + struct pci_dev_info pdev_info; + nodeid_t node = NUMA_NO_NODE; + + ret = -EFAULT; + if ( copy_from_guest(&add, arg, 1) != 0 ) + break; + + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN); + if ( add.flags & XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn = 1; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } + else + pdev_info.is_virtfn = 0; + + ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node); + break; + } +#endif + default: + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd); + ret = -ENOSYS; + } + + return ret; } /* -- 2.17.1
The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the Xen to emulate the PCI devices config space. A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space,XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/Makefile | 1 + xen/arch/arm/domain.c | 4 ++ xen/arch/arm/vpci.c | 96 +++++++++++++++++++++++++++++++++++ xen/arch/arm/vpci.h | 37 ++++++++++++++ xen/drivers/passthrough/pci.c | 7 +++ xen/drivers/vpci/Makefile | 3 +- xen/drivers/vpci/header.c | 2 + xen/include/asm-arm/domain.h | 5 +- xen/include/asm-arm/pci.h | 8 +++ xen/include/public/arch-arm.h | 4 ++ 10 files changed, 165 insertions(+), 2 deletions(-) create mode 100644 xen/arch/arm/vpci.c create mode 100644 xen/arch/arm/vpci.h diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile index 0e14a5e5c8..7cdce684a4 100644 --- a/xen/arch/arm/Makefile +++ b/xen/arch/arm/Makefile @@ -7,6 +7,7 @@ obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ obj-$(CONFIG_HAS_PCI) += pci/ +obj-$(CONFIG_HAS_VPCI) += vpci.o obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.o diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index 19c756ac3d..d99c653626 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -40,6 +40,7 @@ #include <asm/vtimer.h> #include "vuart.h" +#include "vpci.h" DEFINE_PER_CPU(struct vcpu *, curr_vcpu); @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d, if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) ) goto fail; + if ( (rc = domain_vpci_init(d)) != 0 ) + goto fail; + return 0; fail: diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c new file mode 100644 index 0000000000..da8b1ca13c --- /dev/null +++ b/xen/arch/arm/vpci.c @@ -0,0 +1,96 @@ +/* + * xen/arch/arm/vpci.c + * Copyright (c) 2021 Arm Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ +#include <xen/sched.h> +#include <asm/mmio.h> + +/* Do some sanity checks. */ +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len) +{ + /* Check access size. */ + if ( len != 1 && len != 2 && len != 4 && len != 8 ) + return false; + + /* Check that access is size aligned. */ + if ( (reg & (len - 1)) ) + return false; + + return true; +} + +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, + register_t *r, void *p) +{ + unsigned int reg; + pci_sbdf_t sbdf; + uint32_t data = 0; + unsigned int size = 1U << info->dabt.size; + + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); + + if ( !vpci_mmio_access_allowed(reg, size) ) + return 1; + + data = vpci_read(sbdf, reg, size); + + memcpy(r, &data, size); + + return 1; +} + +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, + register_t r, void *p) +{ + unsigned int reg; + pci_sbdf_t sbdf; + uint32_t data = r; + unsigned int size = 1U << info->dabt.size; + + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); + + if ( !vpci_mmio_access_allowed(reg, size) ) + return 1; + + vpci_write(sbdf, reg, size, data); + + return 1; +} + +static const struct mmio_handler_ops vpci_mmio_handler = { + .read = vpci_mmio_read, + .write = vpci_mmio_write, +}; + +int domain_vpci_init(struct domain *d) +{ + if ( !has_vpci(d) ) + return 0; + + register_mmio_handler(d, &vpci_mmio_handler, + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); + + return 0; +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * indent-tabs-mode: nil + * End: + */ + diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h new file mode 100644 index 0000000000..8a093bb705 --- /dev/null +++ b/xen/arch/arm/vpci.h @@ -0,0 +1,37 @@ +/* + * xen/arch/arm/vpci.h + * Copyright (c) 2021 Arm Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ + +#ifndef __ARCH_ARM_VPCI_H__ +#define __ARCH_ARM_VPCI_H__ + +#ifdef CONFIG_HAS_VPCI +int domain_vpci_init(struct domain *d); +#else +static inline int domain_vpci_init(struct domain *d) +{ + return 0; +} +#endif + +#endif /* __ARCH_ARM_VPCI_H__ */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index c23c8cb06b..56e261e9bd 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, else iommu_enable_device(pdev); +#ifdef CONFIG_ARM + ret = vpci_add_handlers(pdev); + if ( ret ) { + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret); + goto out; + } +#endif pci_enable_acs(pdev); out: diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile index 55d1bdfda0..1a1413b93e 100644 --- a/xen/drivers/vpci/Makefile +++ b/xen/drivers/vpci/Makefile @@ -1 +1,2 @@ -obj-y += vpci.o header.o msi.o msix.o +obj-y += vpci.o header.o +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index ba9a036202..f8cd55e7c0 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, * FIXME: punching holes after the p2m has been set up might be racy for * DomU usage, needs to be revisited. */ +#ifdef CONFIG_HAS_PCI_MSI if ( map && !rom_only && vpci_make_msix_hole(pdev) ) return; +#endif for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h index c9277b5c6d..d742b94bd6 100644 --- a/xen/include/asm-arm/domain.h +++ b/xen/include/asm-arm/domain.h @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {} #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag) -#define has_vpci(d) ({ (void)(d); false; }) +/* For X86 VPCI is enabled and tested for PVH DOM0 only but + * for ARM we enable support VPCI for guest domain also. + */ +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); }) #endif /* __ASM_DOMAIN_H__ */ diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index 756f8637ab..c58152de80 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -27,6 +27,14 @@ struct arch_pci_dev { struct device dev; }; +/* Arch-specific MSI data for vPCI. */ +struct vpci_arch_msi { +}; + +/* Arch-specific MSI-X entry data for vPCI. */ +struct vpci_arch_msix_entry { +}; + /* * struct to hold the mappings of a config space window. This * is expected to be used as sysdata for PCI controllers that diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h index 64a2ca30da..0a9749e768 100644 --- a/xen/include/public/arch-arm.h +++ b/xen/include/public/arch-arm.h @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t; #define GUEST_PL011_BASE xen_mk_ullong(0x22000000) #define GUEST_PL011_SIZE xen_mk_ullong(0x00001000) +/* VPCI ECAM mappings */ +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) + /* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in. -- 2.17.1
libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot. Emulated PCI device tree node will only be created when there is any device assigned to guest. A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node). Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- tools/libs/light/libxl_arm.c | 109 ++++++++++++++++++++++++++ tools/libs/light/libxl_types.idl | 1 + tools/xl/xl_parse.c | 2 + xen/include/public/arch-arm.h | 11 +++ xen/include/public/device_tree_defs.h | 1 + 5 files changed, 124 insertions(+) diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c index e3140a6e00..a091e97e76 100644 --- a/tools/libs/light/libxl_arm.c +++ b/tools/libs/light/libxl_arm.c @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt, return fdt_property(fdt, "reg", regs, sizeof(regs)); } +static int fdt_property_values(libxl__gc *gc, void *fdt, + const char *name, unsigned num_cells, ...) +{ + uint32_t prop[num_cells]; + be32 *cells = &prop[0]; + int i; + va_list ap; + uint32_t arg; + + va_start(ap, num_cells); + for (i = 0 ; i < num_cells; i++) { + arg = va_arg(ap, uint32_t); + set_cell(&cells, 1, arg); + } + va_end(ap); + + return fdt_property(fdt, name, prop, sizeof(prop)); +} + +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt, + unsigned addr_cells, + unsigned size_cells, + unsigned num_regs, ...) +{ + uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)]; + be32 *cells = ®s[0]; + int i; + va_list ap; + uint64_t arg; + + va_start(ap, num_regs); + for (i = 0 ; i < num_regs; i++) { + /* Set the memory bit field */ + arg = va_arg(ap, uint64_t); + set_cell(&cells, 1, arg); + + /* Set the vpci bus address */ + arg = addr_cells ? va_arg(ap, uint64_t) : 0; + set_cell(&cells, addr_cells , arg); + + /* Set the cpu bus address where vpci address is mapped */ + set_cell(&cells, addr_cells, arg); + + /* Set the vpci size requested */ + arg = size_cells ? va_arg(ap, uint64_t) : 0; + set_cell(&cells, size_cells,arg); + } + va_end(ap); + + return fdt_property(fdt, "ranges", regs, sizeof(regs)); +} + static int make_root_properties(libxl__gc *gc, const libxl_version_info *vers, void *fdt) @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, return 0; } +static int make_vpci_node(libxl__gc *gc, void *fdt, + const struct arch_info *ainfo, + struct xc_dom_image *dom) +{ + int res; + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE; + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE; + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base); + + res = fdt_begin_node(fdt, name); + if (res) return res; + + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic"); + if (res) return res; + + res = fdt_property_string(fdt, "device_type", "pci"); + if (res) return res; + + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size); + if (res) return res; + + res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17); + if (res) return res; + + res = fdt_property_cell(fdt, "#address-cells", 3); + if (res) return res; + + res = fdt_property_cell(fdt, "#size-cells", 2); + if (res) return res; + + res = fdt_property_string(fdt, "status", "okay"); + if (res) return res; + + res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, + GUEST_ROOT_SIZE_CELLS, 2, + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE, + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR, + GUEST_VPCI_PREFETCH_MEM_SIZE); + if (res) return res; + + res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS, + 0, 0x10000); + if (res) return res; + + res = fdt_end_node(fdt); + if (res) return res; + + return 0; +} + static const struct arch_info *get_arch_info(libxl__gc *gc, const struct xc_dom_image *dom) { @@ -971,6 +1074,9 @@ next_resize: if (info->tee == LIBXL_TEE_TYPE_OPTEE) FDT( make_optee_node(gc, fdt) ); + if (libxl_defbool_val(info->arch_arm.vpci)) + FDT( make_vpci_node(gc, fdt, ainfo, dom) ); + if (pfdt) FDT( copy_partial_fdt(gc, fdt, pfdt) ); @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, /* ACPI is disabled by default */ libxl_defbool_setdefault(&b_info->acpi, false); + /* VPCI is disabled by default */ + libxl_defbool_setdefault(&b_info->arch_arm.vpci, false); + if (b_info->type != LIBXL_DOMAIN_TYPE_PV) return; diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl index 3f9fff653a..78b1ddf0b8 100644 --- a/tools/libs/light/libxl_types.idl +++ b/tools/libs/light/libxl_types.idl @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), ("vuart", libxl_vuart_type), + ("vpci", libxl_defbool), ])), ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), ])), diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c index 17dddb4cd5..ffafbeffb4 100644 --- a/tools/xl/xl_parse.c +++ b/tools/xl/xl_parse.c @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source, } if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV) libxl_defbool_set(&b_info->u.pv.e820_host, true); + if (d_config->num_pcidevs) + libxl_defbool_set(&b_info->arch_arm.vpci, true); } if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) { diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h index 0a9749e768..01d13e669e 100644 --- a/xen/include/public/arch-arm.h +++ b/xen/include/public/arch-arm.h @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t; #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) +/* PCI-PCIe memory space types */ +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) + +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ +#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x900000000) +#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x20000000) + +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x2000000000) +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x04000000) + /* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in. diff --git a/xen/include/public/device_tree_defs.h b/xen/include/public/device_tree_defs.h index 209d43de3f..d24d86b1fc 100644 --- a/xen/include/public/device_tree_defs.h +++ b/xen/include/public/device_tree_defs.h @@ -7,6 +7,7 @@ * onwards. Reserve a high value for the GIC phandle. */ #define GUEST_PHANDLE_GIC (65000) +#define GUEST_PHANDLE_ITS (64999) #define GUEST_ROOT_ADDRESS_CELLS 2 #define GUEST_ROOT_SIZE_CELLS 2 -- 2.17.1
XEN_DOMCTL_ioport_permission, PHYSDEVOP_unmap_pirq, PHYSDEVOP_unmap_pirq are unimplemented for ARM. When libxl assigning a PCI device to the guest error is observed related to above functions. Implement dummy functions to fix the error. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/domctl.c | 2 ++ xen/arch/arm/physdev.c | 3 +++ 2 files changed, 5 insertions(+) diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c index b7d27f37df..38813be893 100644 --- a/xen/arch/arm/domctl.c +++ b/xen/arch/arm/domctl.c @@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d, return rc; } + case XEN_DOMCTL_ioport_permission: + return 0; default: { int rc; diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c index ccce8f0eba..4a0affeada 100644 --- a/xen/arch/arm/physdev.c +++ b/xen/arch/arm/physdev.c @@ -42,6 +42,9 @@ int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) break; } #endif + case PHYSDEVOP_unmap_pirq: + case PHYSDEVOP_map_pirq: + break; default: gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd); ret = -ENOSYS; -- 2.17.1
If the property is not present in the device tree node for host bridge, XEN while creating the dtb for hwdom will create this property and assigns the already allocated segment to the host bridge so that XEN and linux will have the same segment for the host bridges. Signed-off-by: Rahul Singh <rahul.singh@arm.com> --- xen/arch/arm/domain_build.c | 18 ++++++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++ xen/include/asm-arm/pci.h | 3 +++ 3 files changed, 42 insertions(+) diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index 6c86d52781..e0cf2ff19d 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -581,6 +581,24 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo, return res; } +#ifdef CONFIG_HAS_PCI + if ( dt_device_type_is_equal(node, "pci") ) + { + if ( !dt_find_property(node, "linux,pci-domain", NULL) ) + { + uint16_t segment; + + res = pci_get_host_bridge_segment(node, &segment); + if ( res < 0 ) + return res; + + res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment); + if ( res ) + return res; + } + } +#endif + /* * Override the property "status" to disable the device when it's * marked for passthrough. diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c index 62715b4676..5e34252deb 100644 --- a/xen/arch/arm/pci/pci-host-common.c +++ b/xen/arch/arm/pci/pci-host-common.c @@ -280,6 +280,27 @@ struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus) return NULL; } + +/* + * This function will lookup an hostbridge based on config space address. + */ +int pci_get_host_bridge_segment(const struct dt_device_node *node, + uint16_t *segment) +{ + struct pci_host_bridge *bridge; + + list_for_each_entry( bridge, &pci_host_bridges, node ) + { + if ( bridge->dt_node != node ) + continue; + + *segment = bridge->segment; + return 0; + } + + return -EINVAL; +} + /* * Local variables: * mode: C diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index c58152de80..2d4610a23a 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -96,6 +96,9 @@ void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, uint32_t sbdf, uint32_t where); struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus); +int pci_get_host_bridge_segment(const struct dt_device_node *node, + uint16_t *segment); + #else /*!CONFIG_HAS_PCI*/ struct arch_pci_dev { }; -- 2.17.1
On 19.08.2021 14:02, Rahul Singh wrote:
> Add cmdline boot option "pci=on" to enable/disable the PCI init during
> boot.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/pci/pci.c | 30 ++++++++++++++++++++++++++++++
> 1 file changed, 30 insertions(+)
Any addition or change of a command line option should be accompanied
by an adjustment to docs/misc/xen-command-line.pandoc, please.
Jan
On 19.08.2021 14:02, Rahul Singh wrote: > --- a/xen/arch/arm/domctl.c > +++ b/xen/arch/arm/domctl.c > @@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d, > > return rc; > } > + case XEN_DOMCTL_ioport_permission: > + return 0; I don't think returning success for something that doesn't make much sense in the first place (there aren't truly "I/O ports" on Arm afaik) is a good idea. Instead I think the tool stack should avoid making arch-specific calls in an arch-independent way. > --- a/xen/arch/arm/physdev.c > +++ b/xen/arch/arm/physdev.c > @@ -42,6 +42,9 @@ int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) > break; > } > #endif > + case PHYSDEVOP_unmap_pirq: > + case PHYSDEVOP_map_pirq: > + break; Less sure here, but I'm not convinced either. Jan
Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > MSI code that implements MSI functionality to support MSI within XEN is > not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate Can you clarify what you mean by not usable? Is it because we lack of support or we have no plan to use it? Cheers, > the code for ARM. > > No functional change intended. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/x86/Kconfig | 1 + > xen/drivers/passthrough/Makefile | 1 + > xen/drivers/passthrough/msi.c | 96 ++++++++++++++++++++++++++++++++ > xen/drivers/passthrough/pci.c | 54 +++++------------- > xen/drivers/pci/Kconfig | 4 ++ > xen/include/xen/msi.h | 56 +++++++++++++++++++ > xen/xsm/flask/hooks.c | 8 +-- > 7 files changed, 175 insertions(+), 45 deletions(-) > create mode 100644 xen/drivers/passthrough/msi.c > create mode 100644 xen/include/xen/msi.h > > diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig > index 9b164db641..7b46ee98c5 100644 > --- a/xen/arch/x86/Kconfig > +++ b/xen/arch/x86/Kconfig > @@ -19,6 +19,7 @@ config X86 > select HAS_NS16550 > select HAS_PASSTHROUGH > select HAS_PCI > + select HAS_PCI_MSI > select HAS_PDX > select HAS_SCHED_GRANULARITY > select HAS_UBSAN > diff --git a/xen/drivers/passthrough/Makefile b/xen/drivers/passthrough/Makefile > index 445690e3e5..a5efa22714 100644 > --- a/xen/drivers/passthrough/Makefile > +++ b/xen/drivers/passthrough/Makefile > @@ -7,3 +7,4 @@ obj-y += iommu.o > obj-$(CONFIG_HAS_PCI) += pci.o > obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o > obj-$(CONFIG_HAS_PCI) += ats.o > +obj-$(CONFIG_HAS_PCI_MSI) += msi.o > diff --git a/xen/drivers/passthrough/msi.c b/xen/drivers/passthrough/msi.c > new file mode 100644 > index 0000000000..15ad0f8160 > --- /dev/null > +++ b/xen/drivers/passthrough/msi.c > @@ -0,0 +1,96 @@ > +/* > + * Copyright (C) 2008, Netronome Systems, Inc. > + * > + * This program is free software; you can redistribute it and/or modify it > + * under the terms and conditions of the GNU General Public License, > + * version 2, as published by the Free Software Foundation. > + * > + * This program is distributed in the hope it will be useful, but WITHOUT > + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or > + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for > + * more details. > + * > + * You should have received a copy of the GNU General Public License along with > + * this program; If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/init.h> > +#include <xen/pci.h> > +#include <asm/msi.h> > +#include <asm/hvm/io.h> > + > +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) > +{ > + int rc; > + > + if ( pdev->msix ) > + { > + rc = pci_reset_msix_state(pdev); > + if ( rc ) > + return rc; > + msixtbl_init(d); > + } > + > + return 0; > +} > + > +int pdev_msi_init(struct pci_dev *pdev) > +{ > + unsigned int pos; > + > + INIT_LIST_HEAD(&pdev->msi_list); > + > + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), > + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI); > + if ( pos ) > + { > + uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos)); > + > + pdev->msi_maxvec = multi_msi_capable(ctrl); > + } > + > + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), > + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX); > + if ( pos ) > + { > + struct arch_msix *msix = xzalloc(struct arch_msix); > + uint16_t ctrl; > + > + if ( !msix ) > + return -ENOMEM; > + > + spin_lock_init(&msix->table_lock); > + > + ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); > + msix->nr_entries = msix_table_size(ctrl); > + > + pdev->msix = msix; > + } > + > + return 0; > +} > + > +void pdev_msi_deinit(struct pci_dev *pdev) > +{ > + XFREE(pdev->msix); > +} > + > +void pdev_dump_msi(const struct pci_dev *pdev) > +{ > + const struct msi_desc *msi; > + > + printk("- MSIs < "); > + list_for_each_entry ( msi, &pdev->msi_list, list ) > + printk("%d ", msi->irq); > + printk(">"); > +} > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c > index fc4fa2e5c3..67f5686ab6 100644 > --- a/xen/drivers/passthrough/pci.c > +++ b/xen/drivers/passthrough/pci.c > @@ -32,8 +32,8 @@ > #include <xen/softirq.h> > #include <xen/tasklet.h> > #include <xen/vpci.h> > +#include <xen/msi.h> > #include <xsm/xsm.h> > -#include <asm/msi.h> > #include "ats.h" > > struct pci_seg { > @@ -314,6 +314,7 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) > { > struct pci_dev *pdev; > unsigned int pos; > + int rc; > > list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) > if ( pdev->bus == bus && pdev->devfn == devfn ) > @@ -327,35 +328,12 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) > *((u8*) &pdev->bus) = bus; > *((u8*) &pdev->devfn) = devfn; > pdev->domain = NULL; > - INIT_LIST_HEAD(&pdev->msi_list); > - > - pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), > - PCI_CAP_ID_MSI); > - if ( pos ) > - { > - uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos)); > - > - pdev->msi_maxvec = multi_msi_capable(ctrl); > - } > > - pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), > - PCI_CAP_ID_MSIX); > - if ( pos ) > + rc = pdev_msi_init(pdev); > + if ( rc ) > { > - struct arch_msix *msix = xzalloc(struct arch_msix); > - uint16_t ctrl; > - > - if ( !msix ) > - { > - xfree(pdev); > - return NULL; > - } > - spin_lock_init(&msix->table_lock); > - > - ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); > - msix->nr_entries = msix_table_size(ctrl); > - > - pdev->msix = msix; > + xfree(pdev); > + return NULL; > } > > list_add(&pdev->alldevs_list, &pseg->alldevs_list); > @@ -449,7 +427,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev *pdev) > } > > list_del(&pdev->alldevs_list); > - xfree(pdev->msix); > + pdev_msi_deinit(pdev); > xfree(pdev); > } > > @@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) > static int _dump_pci_devices(struct pci_seg *pseg, void *arg) > { > struct pci_dev *pdev; > - struct msi_desc *msi; > > printk("==== segment %04x ====\n", pseg->nr); > > list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) > { > - printk("%pp - %pd - node %-3d - MSIs < ", > + printk("%pp - %pd - node %-3d ", > &pdev->sbdf, pdev->domain, > (pdev->node != NUMA_NO_NODE) ? pdev->node : -1); > - list_for_each_entry ( msi, &pdev->msi_list, list ) > - printk("%d ", msi->irq); > - printk(">\n"); > + pdev_dump_msi(pdev); > + printk("\n"); > } > > return 0; > @@ -1422,13 +1398,9 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) > ASSERT(pdev && (pdev->domain == hardware_domain || > pdev->domain == dom_io)); > > - if ( pdev->msix ) > - { > - rc = pci_reset_msix_state(pdev); > - if ( rc ) > - goto done; > - msixtbl_init(d); > - } > + rc = pdev_msix_assign(d, pdev); > + if ( rc ) > + goto done; > > pdev->fault.count = 0; > > diff --git a/xen/drivers/pci/Kconfig b/xen/drivers/pci/Kconfig > index 7da03fa13b..c6a7bc8007 100644 > --- a/xen/drivers/pci/Kconfig > +++ b/xen/drivers/pci/Kconfig > @@ -1,3 +1,7 @@ > > config HAS_PCI > bool > + > +config HAS_PCI_MSI > + bool > + depends on HAS_PCI > diff --git a/xen/include/xen/msi.h b/xen/include/xen/msi.h > new file mode 100644 > index 0000000000..b2d5bc6f9d > --- /dev/null > +++ b/xen/include/xen/msi.h > @@ -0,0 +1,56 @@ > +/* > + * Copyright (C) 2008, Netronome Systems, Inc. > + * > + * This program is free software; you can redistribute it and/or modify it > + * under the terms and conditions of the GNU General Public License, > + * version 2, as published by the Free Software Foundation. > + * > + * This program is distributed in the hope it will be useful, but WITHOUT > + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or > + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for > + * more details. > + * > + * You should have received a copy of the GNU General Public License along with > + * this program; If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#ifndef __XEN_MSI_H_ > +#define __XEN_MSI_H_ > + > +#ifdef CONFIG_HAS_PCI_MSI > + > +#include <asm/msi.h> > + > +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev); > +int pdev_msi_init(struct pci_dev *pdev); > +void pdev_msi_deinit(struct pci_dev *pdev); > +void pdev_dump_msi(const struct pci_dev *pdev); > + > +#else /* !CONFIG_HAS_PCI_MSI */ > +static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) > +{ > + return 0; > +} > + > +static inline int pdev_msi_init(struct pci_dev *pdev) > +{ > + return 0; > +} > + > +static inline void pdev_msi_deinit(struct pci_dev *pdev) {} > +static inline void pci_cleanup_msi(struct pci_dev *pdev) {} > +static inline void pdev_dump_msi(const struct pci_dev *pdev) {} > + > +#endif /* CONFIG_HAS_PCI_MSI */ > + > +#endif /* __XEN_MSI_H */ > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c > index f1a1217c98..fdcfeb984c 100644 > --- a/xen/xsm/flask/hooks.c > +++ b/xen/xsm/flask/hooks.c > @@ -21,7 +21,7 @@ > #include <xen/guest_access.h> > #include <xen/xenoprof.h> > #include <xen/iommu.h> > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > #include <asm/msi.h> > #endif > #include <public/xen.h> > @@ -114,7 +114,7 @@ static int get_irq_sid(int irq, u32 *sid, struct avc_audit_data *ad) > } > return security_irq_sid(irq, sid); > } > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > { > struct irq_desc *desc = irq_to_desc(irq); > if ( desc->msi_desc && desc->msi_desc->dev ) { > @@ -874,7 +874,7 @@ static int flask_map_domain_pirq (struct domain *d) > static int flask_map_domain_msi (struct domain *d, int irq, const void *data, > u32 *sid, struct avc_audit_data *ad) > { > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > const struct msi_info *msi = data; > u32 machine_bdf = (msi->seg << 16) | (msi->bus << 8) | msi->devfn; > > @@ -940,7 +940,7 @@ static int flask_unmap_domain_pirq (struct domain *d) > static int flask_unmap_domain_msi (struct domain *d, int irq, const void *data, > u32 *sid, struct avc_audit_data *ad) > { > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > const struct pci_dev *pdev = data; > u32 machine_bdf = (pdev->seg << 16) | (pdev->bus << 8) | pdev->devfn; > > -- Julien Grall
Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > Compilation error is observed when HAS_PCI is enabled for ARM > architecture. To be pedantic, what you are trying to solve is not a compilation error but the fact that the PCI mandates helpers that doesn't yet exist on Arm. So... > Add definition for arch_iommu_use_permitted() and > arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to > fix compilation error. ... I am not really in favor of adding dummy implementation here. Instead, the series should be re-ordered so we add the pci-access helpers first and then enable HAS_PCI towards the end of the series. > pci.c: In function ‘deassign_device’: > pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’; > did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration] > pci_to_dev(pdev)); > > pci.c:18: undefined reference to `pci_conf_read16’ > pci.c:880: undefined reference to `arch_pci_clean_pirqs’ > pci.c:1392: undefined reference to `arch_iommu_use_permitted' > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/Makefile | 1 + > xen/arch/arm/pci/Makefile | 2 + > xen/arch/arm/pci/pci-access.c | 61 +++++++++++++++++++++++++++++ > xen/arch/arm/pci/pci.c | 32 +++++++++++++++ > xen/drivers/passthrough/arm/iommu.c | 5 +++ > xen/include/asm-arm/pci.h | 33 ++++++++++++++-- > 6 files changed, 131 insertions(+), 3 deletions(-) > create mode 100644 xen/arch/arm/pci/Makefile > create mode 100644 xen/arch/arm/pci/pci-access.c > create mode 100644 xen/arch/arm/pci/pci.c > > diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile > index 3d3b97b5b4..0e14a5e5c8 100644 > --- a/xen/arch/arm/Makefile > +++ b/xen/arch/arm/Makefile > @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) > obj-y += platforms/ > endif > obj-$(CONFIG_TEE) += tee/ > +obj-$(CONFIG_HAS_PCI) += pci/ > > obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o > obj-y += bootfdt.init.o > diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile > new file mode 100644 > index 0000000000..a9ee0b9b44 > --- /dev/null > +++ b/xen/arch/arm/pci/Makefile > @@ -0,0 +1,2 @@ > +obj-y += pci.o > +obj-y += pci-access.o > diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c > new file mode 100644 > index 0000000000..b938047c03 > --- /dev/null > +++ b/xen/arch/arm/pci/pci-access.c > @@ -0,0 +1,61 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/pci.h> > + > +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, > + unsigned int len) > +{ > + return ~0U; > +} > + > +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, > + unsigned int len, uint32_t val) > +{ > +} > + > +/* > + * Wrappers for all PCI configuration access functions. > + */ > + > +#define PCI_OP_WRITE(size, type) \ > + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ > +{ \ > + pci_config_write(sbdf, reg, size / 8, val); \ > +} > + > +#define PCI_OP_READ(size, type) \ > + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \ > +{ \ > + return pci_config_read(sbdf, reg, size / 8); \ > +} > + > +PCI_OP_READ(8, u8) > +PCI_OP_READ(16, u16) > +PCI_OP_READ(32, u32) > +PCI_OP_WRITE(8, u8) > +PCI_OP_WRITE(16, u16) > +PCI_OP_WRITE(32, u32) > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c > new file mode 100644 > index 0000000000..dc55d23778 > --- /dev/null > +++ b/xen/arch/arm/pci/pci.c > @@ -0,0 +1,32 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/pci.h> > + > +int arch_pci_clean_pirqs(struct domain *d) > +{ > + return 0; > +} Please add a comment explaining why this just returns 0. > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c > index db3b07a571..fdec1c5547 100644 > --- a/xen/drivers/passthrough/arm/iommu.c > +++ b/xen/drivers/passthrough/arm/iommu.c > @@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d) > void __hwdom_init arch_iommu_hwdom_init(struct domain *d) > { > } > + > +bool arch_iommu_use_permitted(const struct domain *d) > +{ > + return true; > +} Please add a comment explaining why returning true is always fine. > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index de13359f65..61e43da088 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -1,7 +1,34 @@ > -#ifndef __X86_PCI_H__ > -#define __X86_PCI_H__ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > > +#ifndef __ARM_PCI_H__ > +#define __ARM_PCI_H__ > + > +#ifdef CONFIG_HAS_PCI > + > +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev) > + > +/* Arch pci dev struct */ > struct arch_pci_dev { > + struct device dev; > }; > > -#endif /* __X86_PCI_H__ */ > +#else /*!CONFIG_HAS_PCI*/ > + > +struct arch_pci_dev { }; > + > +#endif /*!CONFIG_HAS_PCI*/ > +#endif /* __ARM_PCI_H__ */ > -- Julien Grall
Hi Rahul,
On 19/08/2021 13:02, Rahul Singh wrote:
> Add cmdline boot option "pci=on" to enable/disable the PCI init during
> boot.
I read this as "PCI" will be either disabled/enabled for the platform.
Whereas, I think it will be used to decide whether Xen discover PCI and
PCI passthrough is supported or not.
Can you also clarify why a user would want to select "pci=off"?
Cheers,
--
Julien Grall
(+ Jan) Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > Hardware domain is in charge of doing the PCI enumeration and will > discover the PCI devices and then will communicate to XEN via hyper > call PHYSDEVOP_pci_device_add to add the PCI devices in XEN. There are other PHYSDEVOP operations to add PCI devices. I think it is fine to only implement the latest (CC Jan for some opinion and confirm this is the latest). However, this ought to be explained in the commit message. Also, public/arch-arm.h will need to be updated as we now support the PHYSDEVOP hypercall. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/physdev.c | 39 ++++++++++++++++++++++++++++++++++++--- > 1 file changed, 36 insertions(+), 3 deletions(-) > > diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c > index e91355fe22..ccce8f0eba 100644 > --- a/xen/arch/arm/physdev.c > +++ b/xen/arch/arm/physdev.c > @@ -9,12 +9,45 @@ > #include <xen/errno.h> > #include <xen/sched.h> > #include <asm/hypercall.h> > - > +#include <xen/guest_access.h> > +#include <xsm/xsm.h> > > int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) > { > - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd); > - return -ENOSYS; > + int ret = 0; > + > + switch ( cmd ) > + { > +#ifdef CONFIG_HAS_PCI > + case PHYSDEVOP_pci_device_add: { > + struct physdev_pci_device_add add; > + struct pci_dev_info pdev_info; > + nodeid_t node = NUMA_NO_NODE; > + > + ret = -EFAULT; > + if ( copy_from_guest(&add, arg, 1) != 0 ) > + break; > + > + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN); > + if ( add.flags & XEN_PCI_DEV_VIRTFN ) > + { > + pdev_info.is_virtfn = 1; > + pdev_info.physfn.bus = add.physfn.bus; > + pdev_info.physfn.devfn = add.physfn.devfn; > + } > + else > + pdev_info.is_virtfn = 0; > + > + ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node); > + break; > + } This is pretty much a copy of the x86 version without the NUMA bit. So I think we want to move the implementation in common code. > +#endif > + default: > + gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd); > + ret = -ENOSYS; > + } > + > + return ret; > } > > /* > Cheers, -- Julien Grall
Hi, On 19/08/2021 13:12, Jan Beulich wrote: > On 19.08.2021 14:02, Rahul Singh wrote: >> --- a/xen/arch/arm/domctl.c >> +++ b/xen/arch/arm/domctl.c >> @@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d, >> >> return rc; >> } >> + case XEN_DOMCTL_ioport_permission: >> + return 0; > > I don't think returning success for something that doesn't make > much sense in the first place (there aren't truly "I/O ports" on > Arm afaik) is a good idea. They are memory mapped. IIRC, there is a region that is reserved to fake them (for PCI cards that still using them). > Instead I think the tool stack should > avoid making arch-specific calls in an arch-independent way. +1. The current approach means that it will be impossible to implement those sub operations without breaking Xen if we ever needs it. Cheers, -- Julien Grall
Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > libxl will create an emulated PCI device tree node in the device tree to > enable the guest OS to discover the virtual PCI during guest boot. > Emulated PCI device tree node will only be created when there is any > device assigned to guest. > > A new area has been reserved in the arm guest physical map at > which the VPCI bus is declared in the device tree (reg and ranges > parameters of the node). > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > tools/libs/light/libxl_arm.c | 109 ++++++++++++++++++++++++++ > tools/libs/light/libxl_types.idl | 1 + > tools/xl/xl_parse.c | 2 + > xen/include/public/arch-arm.h | 11 +++ > xen/include/public/device_tree_defs.h | 1 + > 5 files changed, 124 insertions(+) > > diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c > index e3140a6e00..a091e97e76 100644 > --- a/tools/libs/light/libxl_arm.c > +++ b/tools/libs/light/libxl_arm.c > @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt, > return fdt_property(fdt, "reg", regs, sizeof(regs)); > } > > +static int fdt_property_values(libxl__gc *gc, void *fdt, > + const char *name, unsigned num_cells, ...) > +{ > + uint32_t prop[num_cells]; > + be32 *cells = &prop[0]; > + int i; > + va_list ap; > + uint32_t arg; > + > + va_start(ap, num_cells); > + for (i = 0 ; i < num_cells; i++) { > + arg = va_arg(ap, uint32_t); > + set_cell(&cells, 1, arg); > + } > + va_end(ap); > + > + return fdt_property(fdt, name, prop, sizeof(prop)); > +} > + > +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt, > + unsigned addr_cells, > + unsigned size_cells, > + unsigned num_regs, ...) > +{ > + uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)]; > + be32 *cells = ®s[0]; > + int i; > + va_list ap; > + uint64_t arg; > + > + va_start(ap, num_regs); > + for (i = 0 ; i < num_regs; i++) { > + /* Set the memory bit field */ > + arg = va_arg(ap, uint64_t); > + set_cell(&cells, 1, arg); > + > + /* Set the vpci bus address */ > + arg = addr_cells ? va_arg(ap, uint64_t) : 0; > + set_cell(&cells, addr_cells , arg); > + > + /* Set the cpu bus address where vpci address is mapped */ > + set_cell(&cells, addr_cells, arg); > + > + /* Set the vpci size requested */ > + arg = size_cells ? va_arg(ap, uint64_t) : 0; > + set_cell(&cells, size_cells,arg); > + } > + va_end(ap); > + > + return fdt_property(fdt, "ranges", regs, sizeof(regs)); > +} > + > static int make_root_properties(libxl__gc *gc, > const libxl_version_info *vers, > void *fdt) > @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, > return 0; > } > > +static int make_vpci_node(libxl__gc *gc, void *fdt, > + const struct arch_info *ainfo, > + struct xc_dom_image *dom) > +{ > + int res; > + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE; > + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE; > + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base); > + > + res = fdt_begin_node(fdt, name); > + if (res) return res; > + > + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic"); > + if (res) return res; > + > + res = fdt_property_string(fdt, "device_type", "pci"); > + if (res) return res; > + > + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, > + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size); > + if (res) return res; > + > + res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17); AFAICT, the "bus-range" is optional. Can you explain why we need it? > + if (res) return res; > + > + res = fdt_property_cell(fdt, "#address-cells", 3); > + if (res) return res; > + > + res = fdt_property_cell(fdt, "#size-cells", 2); > + if (res) return res; > + > + res = fdt_property_string(fdt, "status", "okay"); > + if (res) return res; > + > + res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, > + GUEST_ROOT_SIZE_CELLS, 2, > + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE, > + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR, > + GUEST_VPCI_PREFETCH_MEM_SIZE); > + if (res) return res; > + > + res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS, > + 0, 0x10000); The ITS is not exposed to the guest so far. So I think this wants to be moved to the future series that will add ITS support. > + if (res) return res; > + > + res = fdt_end_node(fdt); > + if (res) return res; > + > + return 0; > +} > + > static const struct arch_info *get_arch_info(libxl__gc *gc, > const struct xc_dom_image *dom) > { > @@ -971,6 +1074,9 @@ next_resize: > if (info->tee == LIBXL_TEE_TYPE_OPTEE) > FDT( make_optee_node(gc, fdt) ); > > + if (libxl_defbool_val(info->arch_arm.vpci)) > + FDT( make_vpci_node(gc, fdt, ainfo, dom) ); > + > if (pfdt) > FDT( copy_partial_fdt(gc, fdt, pfdt) ); > > @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, > /* ACPI is disabled by default */ > libxl_defbool_setdefault(&b_info->acpi, false); > > + /* VPCI is disabled by default */ > + libxl_defbool_setdefault(&b_info->arch_arm.vpci, false); Here you say that vPCI is disabled by default. However, AFAICT, in the previous patch you will unconditionally enable vPCI if Xen has been built with VPCI. Rather than unconditionally enable VPCI for a domain, I think we should introduce a new flag that is passed to Xen at the domain creation. The flag will be set based on arch_arm.vpci. > + > if (b_info->type != LIBXL_DOMAIN_TYPE_PV) > return; > > diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl > index 3f9fff653a..78b1ddf0b8 100644 > --- a/tools/libs/light/libxl_types.idl > +++ b/tools/libs/light/libxl_types.idl > @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ > > ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), > ("vuart", libxl_vuart_type), > + ("vpci", libxl_defbool), Any new addition in the structure should be accompanied with a LIBXL_HAVE_* in the libxl.h header. > ])), > ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), > ])), > diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c > index 17dddb4cd5..ffafbeffb4 100644 > --- a/tools/xl/xl_parse.c > +++ b/tools/xl/xl_parse.c > @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source, > } > if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV) > libxl_defbool_set(&b_info->u.pv.e820_host, true); > + if (d_config->num_pcidevs) > + libxl_defbool_set(&b_info->arch_arm.vpci, true); > } > > if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) { > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h > index 0a9749e768..01d13e669e 100644 > --- a/xen/include/public/arch-arm.h > +++ b/xen/include/public/arch-arm.h > @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t; > #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) > #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) > > +/* PCI-PCIe memory space types */ > +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) > +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) What the size of those regions? > + > +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ > +#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x900000000) > +#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x20000000) So far the memory layout defines the address in ascending order. So please add that after GUEST_RAM_BANK_BASES_*. However, if I am not mistaken that base address you provide will clash with RAM bank 1. It also seem to be pretty high which means that this will not work for 32-bit domain or on CPUs that don't allow offer large IPA bits. I think we need to start making the guest layout more dynamic. The VPCI memory space would have to go right after the end of the RAM allocated for a given guest. > + > +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x2000000000) > +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x04000000) It would be better if the size for each region is defined right after each base. Also, how did you decide the size of each region? > + > /* > * 16MB == 4096 pages reserved for guest to use as a region to map its > * grant table in. > diff --git a/xen/include/public/device_tree_defs.h b/xen/include/public/device_tree_defs.h > index 209d43de3f..d24d86b1fc 100644 > --- a/xen/include/public/device_tree_defs.h > +++ b/xen/include/public/device_tree_defs.h > @@ -7,6 +7,7 @@ > * onwards. Reserve a high value for the GIC phandle. > */ > #define GUEST_PHANDLE_GIC (65000) > +#define GUEST_PHANDLE_ITS (64999) > > #define GUEST_ROOT_ADDRESS_CELLS 2 > #define GUEST_ROOT_SIZE_CELLS 2 > Cheers, -- Julien Grall
On 19.08.2021 14:35, Julien Grall wrote: > On 19/08/2021 13:02, Rahul Singh wrote: >> Hardware domain is in charge of doing the PCI enumeration and will >> discover the PCI devices and then will communicate to XEN via hyper >> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN. > > There are other PHYSDEVOP operations to add PCI devices. I think it is > fine to only implement the latest (CC Jan for some opinion and confirm > this is the latest). However, this ought to be explained in the commit > message. I don't think "latest" matters much here. Considering there was no physdevop support at all on Arm, enabling whichever set seems like a good fit would be okay. Having written this I realize that by "latest" you may mean whether the used sub-ops have not been obsoleted by newer ones (rather than the last ones that were added to the physdevops set). While indeed PHYSDEVOP_pci_device_add hasn't been superseded so far, I have a vague recollection of there being some missing part. Without me remembering details I'm afraid using what is there is the best we can do for for the moment. However, ... >> --- a/xen/arch/arm/physdev.c >> +++ b/xen/arch/arm/physdev.c >> @@ -9,12 +9,45 @@ >> #include <xen/errno.h> >> #include <xen/sched.h> >> #include <asm/hypercall.h> >> - >> +#include <xen/guest_access.h> >> +#include <xsm/xsm.h> >> >> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) >> { >> - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd); >> - return -ENOSYS; >> + int ret = 0; >> + >> + switch ( cmd ) >> + { >> +#ifdef CONFIG_HAS_PCI >> + case PHYSDEVOP_pci_device_add: { >> + struct physdev_pci_device_add add; >> + struct pci_dev_info pdev_info; >> + nodeid_t node = NUMA_NO_NODE; >> + >> + ret = -EFAULT; >> + if ( copy_from_guest(&add, arg, 1) != 0 ) >> + break; >> + >> + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN); >> + if ( add.flags & XEN_PCI_DEV_VIRTFN ) >> + { >> + pdev_info.is_virtfn = 1; >> + pdev_info.physfn.bus = add.physfn.bus; >> + pdev_info.physfn.devfn = add.physfn.devfn; >> + } >> + else >> + pdev_info.is_virtfn = 0; >> + >> + ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node); >> + break; >> + } ... I don't think it should be only "add" which gets supported. "remove" exists not just for the purpose of hot-unplug, but also for Dom0 to remove (and then re-add) devices after e.g. bus re-numbering. (There are some gaps there iirc, but still ...) > This is pretty much a copy of the x86 version without the NUMA bit. So I > think we want to move the implementation in common code. +1 (if sensibly possible) Jan
Hi Julien,
> On 19 Aug 2021, at 1:18 pm, Julien Grall <julien@xen.org> wrote:
>
> Hi Rahul,
>
> On 19/08/2021 13:02, Rahul Singh wrote:
>> MSI code that implements MSI functionality to support MSI within XEN is
>> not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate
>
> Can you clarify what you mean by not usable? Is it because we lack of support or we have no plan to use it?
We have no plan to use it. Code moved to CONFIG_HAS_PCI_MSI will only be required for ARM if we
decide to support PCI device access (PCI MSI interrupt support) within XEN. As of now, we are planning
to add support for PCI device access for DOM0/DOMU guests not for XEN.
Regards,
Rahul
On 19.08.2021 14:02, Rahul Singh wrote: > Compilation error is observed when HAS_PCI is enabled for ARM > architecture. > > Add definition for arch_iommu_use_permitted() and > arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to > fix compilation error. > > pci.c: In function ‘deassign_device’: > pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’; > did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration] > pci_to_dev(pdev)); > > pci.c:18: undefined reference to `pci_conf_read16’ > pci.c:880: undefined reference to `arch_pci_clean_pirqs’ > pci.c:1392: undefined reference to `arch_iommu_use_permitted' > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> A couple of nits, notwithstanding Julien's more general concern: > --- /dev/null > +++ b/xen/arch/arm/pci/pci-access.c > @@ -0,0 +1,61 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/pci.h> > + > +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, > + unsigned int len) > +{ > + return ~0U; > +} > + > +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, > + unsigned int len, uint32_t val) > +{ > +} > + > +/* > + * Wrappers for all PCI configuration access functions. > + */ > + > +#define PCI_OP_WRITE(size, type) \ > + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ One of the two blanks preceding the backslash wants to move after the first comma. And the blank preceding the opening parenthesis wants to be dropped. > +{ \ > + pci_config_write(sbdf, reg, size / 8, val); \ > +} > + > +#define PCI_OP_READ(size, type) \ > + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \ The latter of the two applies here as well. > +{ \ > + return pci_config_read(sbdf, reg, size / 8); \ > +} > + > +PCI_OP_READ(8, u8) > +PCI_OP_READ(16, u16) > +PCI_OP_READ(32, u32) > +PCI_OP_WRITE(8, u8) > +PCI_OP_WRITE(16, u16) > +PCI_OP_WRITE(32, u32) We aim at eliminating u<N> uses in favor of uint<N>_t - please don't introduce new uses. Jan
On 19.08.2021 14:02, Rahul Singh wrote:
> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
> architecture. Move the code under CONFIG_X86 flag to gate the code for
> ARM.
Please give at least one example of what it is that fails to compile.
What an appropriate action is depends on the nature of the error(s),
and from looking at the enclosed code I cannot easily see what it
might be that breaks for Arm.
And as suggested by Julien for the earlier patch - you will want to
re-order things such that compilation doesn't "break" in the first
place. Title and description would then want adjusting accordingly.
Jan
Hi Julien, > On 19 Aug 2021, at 1:28 pm, Julien Grall <julien@xen.org> wrote: > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> Compilation error is observed when HAS_PCI is enabled for ARM >> architecture. > > To be pedantic, what you are trying to solve is not a compilation error but the fact that the PCI mandates helpers that doesn't yet exist on Arm. So... > >> Add definition for arch_iommu_use_permitted() and >> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to >> fix compilation error. > > ... I am not really in favor of adding dummy implementation here. Instead, the series should be re-ordered so we add the pci-access helpers first and then enable HAS_PCI towards the end of the series. Ok. I will remove the dummy function implementation from this patch and will enable the HAS_PCI after adding the PCI access functions patch and other bits. > >> pci.c: In function ‘deassign_device’: >> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’; >> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration] >> pci_to_dev(pdev)); >> pci.c:18: undefined reference to `pci_conf_read16’ >> pci.c:880: undefined reference to `arch_pci_clean_pirqs’ >> pci.c:1392: undefined reference to `arch_iommu_use_permitted' >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/Makefile | 1 + >> xen/arch/arm/pci/Makefile | 2 + >> xen/arch/arm/pci/pci-access.c | 61 +++++++++++++++++++++++++++++ >> xen/arch/arm/pci/pci.c | 32 +++++++++++++++ >> xen/drivers/passthrough/arm/iommu.c | 5 +++ >> xen/include/asm-arm/pci.h | 33 ++++++++++++++-- >> 6 files changed, 131 insertions(+), 3 deletions(-) >> create mode 100644 xen/arch/arm/pci/Makefile >> create mode 100644 xen/arch/arm/pci/pci-access.c >> create mode 100644 xen/arch/arm/pci/pci.c >> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile >> index 3d3b97b5b4..0e14a5e5c8 100644 >> --- a/xen/arch/arm/Makefile >> +++ b/xen/arch/arm/Makefile >> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) >> obj-y += platforms/ >> endif >> obj-$(CONFIG_TEE) += tee/ >> +obj-$(CONFIG_HAS_PCI) += pci/ >> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o >> obj-y += bootfdt.init.o >> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >> new file mode 100644 >> index 0000000000..a9ee0b9b44 >> --- /dev/null >> +++ b/xen/arch/arm/pci/Makefile >> @@ -0,0 +1,2 @@ >> +obj-y += pci.o >> +obj-y += pci-access.o >> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c >> new file mode 100644 >> index 0000000000..b938047c03 >> --- /dev/null >> +++ b/xen/arch/arm/pci/pci-access.c >> @@ -0,0 +1,61 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/pci.h> >> + >> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, >> + unsigned int len) >> +{ >> + return ~0U; >> +} >> + >> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, >> + unsigned int len, uint32_t val) >> +{ >> +} >> + >> +/* >> + * Wrappers for all PCI configuration access functions. >> + */ >> + >> +#define PCI_OP_WRITE(size, type) \ >> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ >> +{ \ >> + pci_config_write(sbdf, reg, size / 8, val); \ >> +} >> + >> +#define PCI_OP_READ(size, type) \ >> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \ >> +{ \ >> + return pci_config_read(sbdf, reg, size / 8); \ >> +} >> + >> +PCI_OP_READ(8, u8) >> +PCI_OP_READ(16, u16) >> +PCI_OP_READ(32, u32) >> +PCI_OP_WRITE(8, u8) >> +PCI_OP_WRITE(16, u16) >> +PCI_OP_WRITE(32, u32) >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * tab-width: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c >> new file mode 100644 >> index 0000000000..dc55d23778 >> --- /dev/null >> +++ b/xen/arch/arm/pci/pci.c >> @@ -0,0 +1,32 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/pci.h> >> + >> +int arch_pci_clean_pirqs(struct domain *d) >> +{ >> + return 0; >> +} > > Please add a comment explaining why this just returns 0. Here is the comment that I will add in next version. /* * Return 0 as on ARM there is no pci physical irqs that required cleanup. */ > >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * tab-width: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c >> index db3b07a571..fdec1c5547 100644 >> --- a/xen/drivers/passthrough/arm/iommu.c >> +++ b/xen/drivers/passthrough/arm/iommu.c >> @@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d) >> void __hwdom_init arch_iommu_hwdom_init(struct domain *d) >> { >> } >> + >> +bool arch_iommu_use_permitted(const struct domain *d) >> +{ >> + return true; >> +} > > Please add a comment explaining why returning true is always fine. Here is the comment that I will add in next version: /* * Return true as iommu use is always permitted if mem-sharing, * mem-paging, or log-dirty functionality is not enabled. */ Regards, Rahul
Hi Jan > On 20 Aug 2021, at 8:01 am, Jan Beulich <jbeulich@suse.com> wrote: > > On 19.08.2021 14:02, Rahul Singh wrote: >> Compilation error is observed when HAS_PCI is enabled for ARM >> architecture. >> >> Add definition for arch_iommu_use_permitted() and >> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to >> fix compilation error. >> >> pci.c: In function ‘deassign_device’: >> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’; >> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration] >> pci_to_dev(pdev)); >> >> pci.c:18: undefined reference to `pci_conf_read16’ >> pci.c:880: undefined reference to `arch_pci_clean_pirqs’ >> pci.c:1392: undefined reference to `arch_iommu_use_permitted' >> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> > > A couple of nits, notwithstanding Julien's more general concern: > >> --- /dev/null >> +++ b/xen/arch/arm/pci/pci-access.c >> @@ -0,0 +1,61 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/pci.h> >> + >> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, >> + unsigned int len) >> +{ >> + return ~0U; >> +} >> + >> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, >> + unsigned int len, uint32_t val) >> +{ >> +} >> + >> +/* >> + * Wrappers for all PCI configuration access functions. >> + */ >> + >> +#define PCI_OP_WRITE(size, type) \ >> + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ > > One of the two blanks preceding the backslash wants to move after the first > comma. And the blank preceding the opening parenthesis wants to be dropped. I will modify as below . Please correct me if I misunderstood wrongly. #define PCI_OP_WRITE(size, type) \ void pci_conf_write##size(pci_sbdf_t sbdf, \ unsigned int reg, type val) \ { \ pci_config_write(sbdf, reg, size / 8, val); \ } > >> +{ \ >> + pci_config_write(sbdf, reg, size / 8, val); \ >> +} >> + >> +#define PCI_OP_READ(size, type) \ >> + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \ > > The latter of the two applies here as well. > >> +{ \ >> + return pci_config_read(sbdf, reg, size / 8); \ >> +} >> + >> +PCI_OP_READ(8, u8) >> +PCI_OP_READ(16, u16) >> +PCI_OP_READ(32, u32) >> +PCI_OP_WRITE(8, u8) >> +PCI_OP_WRITE(16, u16) >> +PCI_OP_WRITE(32, u32) > > We aim at eliminating u<N> uses in favor of uint<N>_t - please don't > introduce new uses. Ok . I will modify in next version. Regards, Rahul > > Jan
Hi Rahul, On 20/08/2021 11:30, Rahul Singh wrote: >> Please add a comment explaining why this just returns 0. > > Here is the comment that I will add in next version. > /* > * Return 0 as on ARM there is no pci physical irqs that required cleanup. > */ In this context, PIRQ means an interrupts that was routed to the guest and could be mapped to an event channel. We have no such concept on Arm (see allocate_pirq_struct()). So I would write "PIRQ event channels are not supported on Arm, so nothing to do". > >> >>> + >>> +/* >>> + * Local variables: >>> + * mode: C >>> + * c-file-style: "BSD" >>> + * c-basic-offset: 4 >>> + * tab-width: 4 >>> + * indent-tabs-mode: nil >>> + * End: >>> + */ >>> diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c >>> index db3b07a571..fdec1c5547 100644 >>> --- a/xen/drivers/passthrough/arm/iommu.c >>> +++ b/xen/drivers/passthrough/arm/iommu.c >>> @@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d) >>> void __hwdom_init arch_iommu_hwdom_init(struct domain *d) >>> { >>> } >>> + >>> +bool arch_iommu_use_permitted(const struct domain *d) >>> +{ >>> + return true; >>> +} >> >> Please add a comment explaining why returning true is always fine. > > Here is the comment that I will add in next version: > > /* > * Return true as iommu use is always permitted if mem-sharing, > * mem-paging, or log-dirty functionality is not enabled. > */ How about writing "Unlike x86, Arm doesn't support mem-sharing, mem-paging and log-dirty (yet). So there is no restriction to use the IOMMU". This would make clear why you are mentioning mem-sharing, mem-paging & co. Cheers, -- Julien Grall
Hi Jan > On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote: > > On 19.08.2021 14:02, Rahul Singh wrote: >> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM >> architecture. Move the code under CONFIG_X86 flag to gate the code for >> ARM. > > Please give at least one example of what it is that fails to compile. > What an appropriate action is depends on the nature of the error(s), > and from looking at the enclosed code I cannot easily see what it > might be that breaks for Arm. I am observing below error when enabled ACPI && HAS_PCI for ARM. prelink.o: In function `pcie_aer_get_firmware_first’: /xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse' aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code. obj-$(CONFIG_X86) += apei/ I am not sure whether we need this code for ARM architecture that is why I gate the code for ARM via CONFIG_X86 > > And as suggested by Julien for the earlier patch - you will want to > re-order things such that compilation doesn't "break" in the first > place. Title and description would then want adjusting accordingly. Let me reorder the patch series in next version. Regards, Rahul > > Jan >
On 20.08.2021 13:41, Rahul Singh wrote:
> Hi Jan
>
>> On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 19.08.2021 14:02, Rahul Singh wrote:
>>> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
>>> architecture. Move the code under CONFIG_X86 flag to gate the code for
>>> ARM.
>>
>> Please give at least one example of what it is that fails to compile.
>> What an appropriate action is depends on the nature of the error(s),
>> and from looking at the enclosed code I cannot easily see what it
>> might be that breaks for Arm.
>
> I am observing below error when enabled ACPI && HAS_PCI for ARM.
>
> prelink.o: In function `pcie_aer_get_firmware_first’:
> /xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse'
> aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined
>
> I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code.
> obj-$(CONFIG_X86) += apei/
>
> I am not sure whether we need this code for ARM architecture
> that is why I gate the code for ARM via CONFIG_X86
So you Arm folks will probably want to settle on that aspect first. What
is wanted to keep things building depends on that.
Jan
On 20.08.2021 13:37, Julien Grall wrote:
> On 20/08/2021 11:30, Rahul Singh wrote:
>>> Please add a comment explaining why this just returns 0.
>>
>> Here is the comment that I will add in next version.
>> /*
>> * Return 0 as on ARM there is no pci physical irqs that required cleanup.
>> */
>
> In this context, PIRQ means an interrupts that was routed to the guest
> and could be mapped to an event channel. We have no such concept on Arm
> (see allocate_pirq_struct()).
>
> So I would write "PIRQ event channels are not supported on Arm, so
> nothing to do".
Does this mean ASSERT_UNREACHABLE() might be appropriate here?
Jan
Hi Jan,
On 20/08/2021 12:55, Jan Beulich wrote:
> On 20.08.2021 13:37, Julien Grall wrote:
>> On 20/08/2021 11:30, Rahul Singh wrote:
>>>> Please add a comment explaining why this just returns 0.
>>>
>>> Here is the comment that I will add in next version.
>>> /*
>>> * Return 0 as on ARM there is no pci physical irqs that required cleanup.
>>> */
>>
>> In this context, PIRQ means an interrupts that was routed to the guest
>> and could be mapped to an event channel. We have no such concept on Arm
>> (see allocate_pirq_struct()).
>>
>> So I would write "PIRQ event channels are not supported on Arm, so
>> nothing to do".
>
> Does this mean ASSERT_UNREACHABLE() might be appropriate here?
Unfortunately no. This is call unconditionally from pci_release_devices().
Cheers,
--
Julien Grall
Hi Julien, > On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote: > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> Add cmdline boot option "pci=on" to enable/disable the PCI init during >> boot. > > I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not. Yes. I will modify the option to "pci-passthrough== <boolean>" > > Can you also clarify why a user would want to select "pci=off"? As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device. I am ok to drop this patch if you feel adding the option is not required at all. Regards, Rahul
Hi Julien, > On 19 Aug 2021, at 1:35 pm, Julien Grall <julien@xen.org> wrote: > > (+ Jan) > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> Hardware domain is in charge of doing the PCI enumeration and will >> discover the PCI devices and then will communicate to XEN via hyper >> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN. > > There are other PHYSDEVOP operations to add PCI devices. I think it is fine to only implement the latest (CC Jan for some opinion and confirm this is the latest). However, this ought to be explained in the commit message. As per Jan comments I will add the PHYSDEVOP_pci_device_remove() in the next version. > > Also, public/arch-arm.h will need to be updated as we now support the PHYSDEVOP hypercall. Ok. > >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/physdev.c | 39 ++++++++++++++++++++++++++++++++++++--- >> 1 file changed, 36 insertions(+), 3 deletions(-) >> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c >> index e91355fe22..ccce8f0eba 100644 >> --- a/xen/arch/arm/physdev.c >> +++ b/xen/arch/arm/physdev.c >> @@ -9,12 +9,45 @@ >> #include <xen/errno.h> >> #include <xen/sched.h> >> #include <asm/hypercall.h> >> - >> +#include <xen/guest_access.h> >> +#include <xsm/xsm.h> >> int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) >> { >> - gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd); >> - return -ENOSYS; >> + int ret = 0; >> + >> + switch ( cmd ) >> + { >> +#ifdef CONFIG_HAS_PCI >> + case PHYSDEVOP_pci_device_add: { >> + struct physdev_pci_device_add add; >> + struct pci_dev_info pdev_info; >> + nodeid_t node = NUMA_NO_NODE; >> + >> + ret = -EFAULT; >> + if ( copy_from_guest(&add, arg, 1) != 0 ) >> + break; >> + >> + pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN); >> + if ( add.flags & XEN_PCI_DEV_VIRTFN ) >> + { >> + pdev_info.is_virtfn = 1; >> + pdev_info.physfn.bus = add.physfn.bus; >> + pdev_info.physfn.devfn = add.physfn.devfn; >> + } >> + else >> + pdev_info.is_virtfn = 0; >> + >> + ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node); >> + break; >> + } > > This is pretty much a copy of the x86 version without the NUMA bit. So I think we want to move the implementation in common code. Ok. Let me move the PHYSDEVOP_pci_device_* to common code. Regards, Rahul
On 20/08/2021 13:19, Rahul Singh wrote: > Hi Julien, Hi Rahul, > >> On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote: >> >> Hi Rahul, >> >> On 19/08/2021 13:02, Rahul Singh wrote: >>> Add cmdline boot option "pci=on" to enable/disable the PCI init during >>> boot. >> >> I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not. > > Yes. I will modify the option to "pci-passthrough== <boolean>" >> >> Can you also clarify why a user would want to select "pci=off"? > > As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to > boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device. Dom0 will always have direct access to the PCI device. The only difference is whether the access to the hostbridge and config space will be trapped by Xen. I expect the both to mainly happen during boot and therefore the overhead will be limited. > > I am ok to drop this patch if you feel adding the option is not required at all. One of the reason I could see this option to be useful is to figure out if an issue occurs because of the hostbridge emulation. Yet, I am still not fully convinced adding an option is worth it. Jan and others, any opinions? Cheers, -- Julien Grall
On 20.08.2021 16:34, Julien Grall wrote:
> On 20/08/2021 13:19, Rahul Singh wrote:
>>> On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote:
>>> On 19/08/2021 13:02, Rahul Singh wrote:
>>>> Add cmdline boot option "pci=on" to enable/disable the PCI init during
>>>> boot.
>>>
>>> I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not.
>>
>> Yes. I will modify the option to "pci-passthrough== <boolean>"
>>>
>>> Can you also clarify why a user would want to select "pci=off"?
>>
>> As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to
>> boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device.
>
> Dom0 will always have direct access to the PCI device. The only
> difference is whether the access to the hostbridge and config space will
> be trapped by Xen. I expect the both to mainly happen during boot and
> therefore the overhead will be limited.
>
>>
>> I am ok to drop this patch if you feel adding the option is not required at all.
> One of the reason I could see this option to be useful is to figure out
> if an issue occurs because of the hostbridge emulation. Yet, I am still
> not fully convinced adding an option is worth it.
>
> Jan and others, any opinions?
Well, if there's a proper fallback, then why not allow using it in
case of problems?
Jan
Hi Julien, > On 19 Aug 2021, at 2:00 pm, Julien Grall <julien@xen.org> wrote: > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> libxl will create an emulated PCI device tree node in the device tree to >> enable the guest OS to discover the virtual PCI during guest boot. >> Emulated PCI device tree node will only be created when there is any >> device assigned to guest. >> A new area has been reserved in the arm guest physical map at >> which the VPCI bus is declared in the device tree (reg and ranges >> parameters of the node). >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> tools/libs/light/libxl_arm.c | 109 ++++++++++++++++++++++++++ >> tools/libs/light/libxl_types.idl | 1 + >> tools/xl/xl_parse.c | 2 + >> xen/include/public/arch-arm.h | 11 +++ >> xen/include/public/device_tree_defs.h | 1 + >> 5 files changed, 124 insertions(+) >> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c >> index e3140a6e00..a091e97e76 100644 >> --- a/tools/libs/light/libxl_arm.c >> +++ b/tools/libs/light/libxl_arm.c >> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt, >> return fdt_property(fdt, "reg", regs, sizeof(regs)); >> } >> +static int fdt_property_values(libxl__gc *gc, void *fdt, >> + const char *name, unsigned num_cells, ...) >> +{ >> + uint32_t prop[num_cells]; >> + be32 *cells = &prop[0]; >> + int i; >> + va_list ap; >> + uint32_t arg; >> + >> + va_start(ap, num_cells); >> + for (i = 0 ; i < num_cells; i++) { >> + arg = va_arg(ap, uint32_t); >> + set_cell(&cells, 1, arg); >> + } >> + va_end(ap); >> + >> + return fdt_property(fdt, name, prop, sizeof(prop)); >> +} >> + >> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt, >> + unsigned addr_cells, >> + unsigned size_cells, >> + unsigned num_regs, ...) >> +{ >> + uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)]; >> + be32 *cells = ®s[0]; >> + int i; >> + va_list ap; >> + uint64_t arg; >> + >> + va_start(ap, num_regs); >> + for (i = 0 ; i < num_regs; i++) { >> + /* Set the memory bit field */ >> + arg = va_arg(ap, uint64_t); >> + set_cell(&cells, 1, arg); >> + >> + /* Set the vpci bus address */ >> + arg = addr_cells ? va_arg(ap, uint64_t) : 0; >> + set_cell(&cells, addr_cells , arg); >> + >> + /* Set the cpu bus address where vpci address is mapped */ >> + set_cell(&cells, addr_cells, arg); >> + >> + /* Set the vpci size requested */ >> + arg = size_cells ? va_arg(ap, uint64_t) : 0; >> + set_cell(&cells, size_cells,arg); >> + } >> + va_end(ap); >> + >> + return fdt_property(fdt, "ranges", regs, sizeof(regs)); >> +} >> + >> static int make_root_properties(libxl__gc *gc, >> const libxl_version_info *vers, >> void *fdt) >> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, >> return 0; >> } >> +static int make_vpci_node(libxl__gc *gc, void *fdt, >> + const struct arch_info *ainfo, >> + struct xc_dom_image *dom) >> +{ >> + int res; >> + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE; >> + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE; >> + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base); >> + >> + res = fdt_begin_node(fdt, name); >> + if (res) return res; >> + >> + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic"); >> + if (res) return res; >> + >> + res = fdt_property_string(fdt, "device_type", "pci"); >> + if (res) return res; >> + >> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, >> + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size); >> + if (res) return res; >> + >> + res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17); > > AFAICT, the "bus-range" is optional. Can you explain why we need it? We need it to implement the function pci_ecam_map_bus(). > >> + if (res) return res; >> + >> + res = fdt_property_cell(fdt, "#address-cells", 3); >> + if (res) return res; >> + >> + res = fdt_property_cell(fdt, "#size-cells", 2); >> + if (res) return res; >> + >> + res = fdt_property_string(fdt, "status", "okay"); >> + if (res) return res; >> + >> + res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, >> + GUEST_ROOT_SIZE_CELLS, 2, >> + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE, >> + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR, >> + GUEST_VPCI_PREFETCH_MEM_SIZE); >> + if (res) return res; >> + >> + res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS, >> + 0, 0x10000); > > The ITS is not exposed to the guest so far. So I think this wants to be moved to the future series that will add ITS support. Ok. I will remove this in next version. >> + if (res) return res; >> + >> + res = fdt_end_node(fdt); >> + if (res) return res; >> + >> + return 0; >> +} >> + >> static const struct arch_info *get_arch_info(libxl__gc *gc, >> const struct xc_dom_image *dom) >> { >> @@ -971,6 +1074,9 @@ next_resize: >> if (info->tee == LIBXL_TEE_TYPE_OPTEE) >> FDT( make_optee_node(gc, fdt) ); >> + if (libxl_defbool_val(info->arch_arm.vpci)) >> + FDT( make_vpci_node(gc, fdt, ainfo, dom) ); >> + >> if (pfdt) >> FDT( copy_partial_fdt(gc, fdt, pfdt) ); >> @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, >> /* ACPI is disabled by default */ >> libxl_defbool_setdefault(&b_info->acpi, false); >> + /* VPCI is disabled by default */ >> + libxl_defbool_setdefault(&b_info->arch_arm.vpci, false); > > Here you say that vPCI is disabled by default. However, AFAICT, in the previous patch you will unconditionally enable vPCI if Xen has been built with VPCI. > > Rather than unconditionally enable VPCI for a domain, I think we should introduce a new flag that is passed to Xen at the domain creation. The flag will be set based on arch_arm.vpci. OK. I will introduce the flag in next version. > >> + >> if (b_info->type != LIBXL_DOMAIN_TYPE_PV) >> return; >> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl >> index 3f9fff653a..78b1ddf0b8 100644 >> --- a/tools/libs/light/libxl_types.idl >> +++ b/tools/libs/light/libxl_types.idl >> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ >> ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), >> ("vuart", libxl_vuart_type), >> + ("vpci", libxl_defbool), > > Any new addition in the structure should be accompanied with a LIBXL_HAVE_* in the libxl.h header. OK. > >> ])), >> ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), >> ])), >> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c >> index 17dddb4cd5..ffafbeffb4 100644 >> --- a/tools/xl/xl_parse.c >> +++ b/tools/xl/xl_parse.c >> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source, >> } >> if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV) >> libxl_defbool_set(&b_info->u.pv.e820_host, true); >> + if (d_config->num_pcidevs) >> + libxl_defbool_set(&b_info->arch_arm.vpci, true); >> } >> if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) { >> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h >> index 0a9749e768..01d13e669e 100644 >> --- a/xen/include/public/arch-arm.h >> +++ b/xen/include/public/arch-arm.h >> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t; >> #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) >> #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) >> +/* PCI-PCIe memory space types */ >> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) >> +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) > > What the size of those regions? Non Prefetch Memory: Size 64 MB start at 512 MB Prefetch Memory: Size 128 GB start at 36 GB > >> + >> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ >> +#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x900000000) >> +#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x20000000) > So far the memory layout defines the address in ascending order. So please add that after GUEST_RAM_BANK_BASES_*. Ok. > > However, if I am not mistaken that base address you provide will clash with RAM bank 1. It also seem to be pretty high which means that this will not work for 32-bit domain or on CPUs that don't allow offer large IPA bits. Yes I also checked that now that it is having clash with RAM bank 1. There is unused space is guest memory that we can use for Non Prefetch Memory as per below guest memory map. https://gitlab.com/xen-project/fusa/fusa-docs/-/blob/master/high-level/guest-memory-layout-arm.rst Proposed value: Non Prefetch Memory: Size 64 MB start at 0x22001000 Prefetch Memory: Size 4 GB start at 4 GB. > > I think we need to start making the guest layout more dynamic. The VPCI memory space would have to go right after the end of the RAM allocated for a given guest. > >> + >> +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x2000000000) >> +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x04000000) > > It would be better if the size for each region is defined right after each base. OK. > > Also, how did you decide the size of each region? I thought 64 MB will be sufficient. I think it should be based on number of devices we can assign to the guest. Regards, Rahul > >> + >> /* >> * 16MB == 4096 pages reserved for guest to use as a region to map its >> * grant table in. >> diff --git a/xen/include/public/device_tree_defs.h b/xen/include/public/device_tree_defs.h >> index 209d43de3f..d24d86b1fc 100644 >> --- a/xen/include/public/device_tree_defs.h >> +++ b/xen/include/public/device_tree_defs.h >> @@ -7,6 +7,7 @@ >> * onwards. Reserve a high value for the GIC phandle. >> */ >> #define GUEST_PHANDLE_GIC (65000) >> +#define GUEST_PHANDLE_ITS (64999) >> #define GUEST_ROOT_ADDRESS_CELLS 2 >> #define GUEST_ROOT_SIZE_CELLS 2 > > Cheers, > > -- > Julien Grall
Hi Jan, > On 19 Aug 2021, at 1:12 pm, Jan Beulich <jbeulich@suse.com> wrote: > > On 19.08.2021 14:02, Rahul Singh wrote: >> --- a/xen/arch/arm/domctl.c >> +++ b/xen/arch/arm/domctl.c >> @@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d, >> >> return rc; >> } >> + case XEN_DOMCTL_ioport_permission: >> + return 0; > > I don't think returning success for something that doesn't make > much sense in the first place (there aren't truly "I/O ports" on > Arm afaik) is a good idea. > Instead I think the tool stack should > avoid making arch-specific calls in an arch-independent way. I agree with you let me try to modify the toolstack not to call the arch-specific call. Regards, Rahul > >> --- a/xen/arch/arm/physdev.c >> +++ b/xen/arch/arm/physdev.c >> @@ -42,6 +42,9 @@ int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) >> break; >> } >> #endif >> + case PHYSDEVOP_unmap_pirq: >> + case PHYSDEVOP_map_pirq: >> + break; > > Less sure here, but I'm not convinced either. > > Jan >
On 19.08.2021 14:02, Rahul Singh wrote: > --- /dev/null > +++ b/xen/drivers/passthrough/msi.c > @@ -0,0 +1,96 @@ > +/* > + * Copyright (C) 2008, Netronome Systems, Inc. While generally copying copyright statements when splitting source files is probably wanted (or even necessary) I doubt this is suitable here: None of the MSI code that you move was contributed by them afaict. > + * This program is free software; you can redistribute it and/or modify it > + * under the terms and conditions of the GNU General Public License, > + * version 2, as published by the Free Software Foundation. > + * > + * This program is distributed in the hope it will be useful, but WITHOUT > + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or > + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for > + * more details. > + * > + * You should have received a copy of the GNU General Public License along with > + * this program; If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/init.h> > +#include <xen/pci.h> > +#include <asm/msi.h> You surely mean xen/msi.h here: Headers like this one should always be included by the producer, no matter that it builds fine without. Else you risk declarations and definitions to go out of sync. > +#include <asm/hvm/io.h> > + > +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) > +{ > + int rc; > + > + if ( pdev->msix ) > + { > + rc = pci_reset_msix_state(pdev); > + if ( rc ) > + return rc; > + msixtbl_init(d); > + } > + > + return 0; > +} > + > +int pdev_msi_init(struct pci_dev *pdev) > +{ > + unsigned int pos; > + > + INIT_LIST_HEAD(&pdev->msi_list); > + > + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), > + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI); > + if ( pos ) > + { > + uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos)); > + > + pdev->msi_maxvec = multi_msi_capable(ctrl); > + } > + > + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), > + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX); > + if ( pos ) > + { > + struct arch_msix *msix = xzalloc(struct arch_msix); > + uint16_t ctrl; > + > + if ( !msix ) > + return -ENOMEM; > + > + spin_lock_init(&msix->table_lock); > + > + ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); > + msix->nr_entries = msix_table_size(ctrl); > + > + pdev->msix = msix; > + } > + > + return 0; > +} > + > +void pdev_msi_deinit(struct pci_dev *pdev) > +{ > + XFREE(pdev->msix); > +} > + > +void pdev_dump_msi(const struct pci_dev *pdev) > +{ > + const struct msi_desc *msi; > + > + printk("- MSIs < "); > + list_for_each_entry ( msi, &pdev->msi_list, list ) > + printk("%d ", msi->irq); > + printk(">"); While not an exact equivalent of the original code then, could I talk you into adding an early list_empty() check, suppressing any output from this function if that one returned "true"? > @@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) > static int _dump_pci_devices(struct pci_seg *pseg, void *arg) > { > struct pci_dev *pdev; > - struct msi_desc *msi; > > printk("==== segment %04x ====\n", pseg->nr); > > list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) > { > - printk("%pp - %pd - node %-3d - MSIs < ", > + printk("%pp - %pd - node %-3d ", Together with the request above the trailin blank here also wants to become a leading blank in pdev_dump_msi(). > --- /dev/null > +++ b/xen/include/xen/msi.h > @@ -0,0 +1,56 @@ > +/* > + * Copyright (C) 2008, Netronome Systems, Inc. > + * > + * This program is free software; you can redistribute it and/or modify it > + * under the terms and conditions of the GNU General Public License, > + * version 2, as published by the Free Software Foundation. > + * > + * This program is distributed in the hope it will be useful, but WITHOUT > + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or > + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for > + * more details. > + * > + * You should have received a copy of the GNU General Public License along with > + * this program; If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#ifndef __XEN_MSI_H_ > +#define __XEN_MSI_H_ > + > +#ifdef CONFIG_HAS_PCI_MSI > + > +#include <asm/msi.h> > + > +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev); > +int pdev_msi_init(struct pci_dev *pdev); > +void pdev_msi_deinit(struct pci_dev *pdev); > +void pdev_dump_msi(const struct pci_dev *pdev); > + > +#else /* !CONFIG_HAS_PCI_MSI */ > +static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) Please be consistent with blank lines you add; here you also want one after the #else. > +{ > + return 0; > +} > + > +static inline int pdev_msi_init(struct pci_dev *pdev) > +{ > + return 0; > +} > + > +static inline void pdev_msi_deinit(struct pci_dev *pdev) {} > +static inline void pci_cleanup_msi(struct pci_dev *pdev) {} > +static inline void pdev_dump_msi(const struct pci_dev *pdev) {} Especially for (but perhaps not limited to) this !HAS_PCI_MSI case (where you don't include asm/msi.h and its possible dependents) please forward-declare struct-s you use in prototypes or inline stubs (outside the #ifdef, that is). This will allow including this header without having to care about prereq headers. If you agree with and make all the suggested or requested changes, feel free to add Reviewed-by: Jan Beulich <jbeulich@suse.com> Jan
On 19.08.2021 14:02, Rahul Singh wrote: > --- a/xen/drivers/passthrough/pci.c > +++ b/xen/drivers/passthrough/pci.c > @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, > else > iommu_enable_device(pdev); > > +#ifdef CONFIG_ARM > + ret = vpci_add_handlers(pdev); > + if ( ret ) { > + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret); > + goto out; > + } > +#endif > pci_enable_acs(pdev); I'm afraid I can't see why this is to be Arm-specific. The present placement of the existing call to vpci_add_handlers() looks to be sub-optimal anyway - did you look into whether it could be moved to a place (potentially right here) fitting everyone? If you did, would mind justifying the Arm-specific code in a non-Arm-specific file in at least a post-commit-message remark? If it were to remain like this, please add a blank line after the #endif. > --- a/xen/drivers/vpci/Makefile > +++ b/xen/drivers/vpci/Makefile > @@ -1 +1,2 @@ > -obj-y += vpci.o header.o msi.o msix.o > +obj-y += vpci.o header.o > +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o I continue to consider this a wrong connection - HAS_PCI_MSI expresses (quoting text from patch 1 of this series) "code that implements MSI functionality to support MSI within XEN", while here we're talking of vPCI (i.e. guest support). I can accept that this is transiently the best you can do, but then please add a comment to this effect (if need be in multiple places), such that future readers or people wanting to further adjust this understand why it is the way it is. But perhaps you instead want to introduce a HAS_VPCI_MSI (or, less desirable because of possible ambiguity, HAS_VMSI) Kconfig option? > --- a/xen/drivers/vpci/header.c > +++ b/xen/drivers/vpci/header.c > @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, > * FIXME: punching holes after the p2m has been set up might be racy for > * DomU usage, needs to be revisited. > */ > +#ifdef CONFIG_HAS_PCI_MSI > if ( map && !rom_only && vpci_make_msix_hole(pdev) ) > return; > +#endif (This would be another such place.) > --- a/xen/include/asm-arm/domain.h > +++ b/xen/include/asm-arm/domain.h > @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {} > > #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag) > > -#define has_vpci(d) ({ (void)(d); false; }) > +/* For X86 VPCI is enabled and tested for PVH DOM0 only but > + * for ARM we enable support VPCI for guest domain also. > + */ Comment style (/* goes on its own line). > +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); }) Personally I'd recommend to get away without using extensions whenever possible, i.e. use #define has_vpci(d) ((void)(d), IS_ENABLED(CONFIG_HAS_VPCI)) here. > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -27,6 +27,14 @@ struct arch_pci_dev { > struct device dev; > }; > > +/* Arch-specific MSI data for vPCI. */ > +struct vpci_arch_msi { > +}; > + > +/* Arch-specific MSI-X entry data for vPCI. */ > +struct vpci_arch_msix_entry { > +}; But isn't it that you don't support vPCI's MSI in the first place? Perhaps the need for these would go away with CONFIG_HAS_VCPI_MSI? Jan
Hi, Jan! On 24.08.21 19:09, Jan Beulich wrote: > On 19.08.2021 14:02, Rahul Singh wrote: >> --- a/xen/drivers/passthrough/pci.c >> +++ b/xen/drivers/passthrough/pci.c >> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, >> else >> iommu_enable_device(pdev); >> >> +#ifdef CONFIG_ARM >> + ret = vpci_add_handlers(pdev); >> + if ( ret ) { >> + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret); >> + goto out; >> + } >> +#endif >> pci_enable_acs(pdev); > I'm afraid I can't see why this is to be Arm-specific. The present > placement of the existing call to vpci_add_handlers() looks to be > sub-optimal anyway - did you look into whether it could be moved > to a place (potentially right here) fitting everyone? If you did, > would mind justifying the Arm-specific code in a non-Arm-specific > file in at least a post-commit-message remark? > > If it were to remain like this, please add a blank line after the #endif. > >> --- a/xen/drivers/vpci/Makefile >> +++ b/xen/drivers/vpci/Makefile >> @@ -1 +1,2 @@ >> -obj-y += vpci.o header.o msi.o msix.o >> +obj-y += vpci.o header.o >> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o > I continue to consider this a wrong connection - HAS_PCI_MSI expresses > (quoting text from patch 1 of this series) "code that implements MSI > functionality to support MSI within XEN", while here we're talking of > vPCI (i.e. guest support). I can accept that this is transiently the > best you can do, but then please add a comment to this effect (if > need be in multiple places), such that future readers or people > wanting to further adjust this understand why it is the way it is. > > But perhaps you instead want to introduce a HAS_VPCI_MSI (or, less > desirable because of possible ambiguity, HAS_VMSI) Kconfig option? Eventually we'll have the code like that: obj-y += vpci.o header.o msi.o msix.o obj-$(CONFIG_X86) += x86/ obj-$(CONFIG_ARM) += arm/ So, this is indeed a transitional change. Will you be ok with a comment about that then? > >> --- a/xen/drivers/vpci/header.c >> +++ b/xen/drivers/vpci/header.c >> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, >> * FIXME: punching holes after the p2m has been set up might be racy for >> * DomU usage, needs to be revisited. >> */ >> +#ifdef CONFIG_HAS_PCI_MSI >> if ( map && !rom_only && vpci_make_msix_hole(pdev) ) >> return; >> +#endif > (This would be another such place.) > >> --- a/xen/include/asm-arm/domain.h >> +++ b/xen/include/asm-arm/domain.h >> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {} >> >> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag) >> >> -#define has_vpci(d) ({ (void)(d); false; }) >> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but >> + * for ARM we enable support VPCI for guest domain also. >> + */ > Comment style (/* goes on its own line). > >> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); }) > Personally I'd recommend to get away without using extensions whenever > possible, i.e. use > > #define has_vpci(d) ((void)(d), IS_ENABLED(CONFIG_HAS_VPCI)) > > here. > >> --- a/xen/include/asm-arm/pci.h >> +++ b/xen/include/asm-arm/pci.h >> @@ -27,6 +27,14 @@ struct arch_pci_dev { >> struct device dev; >> }; >> >> +/* Arch-specific MSI data for vPCI. */ >> +struct vpci_arch_msi { >> +}; >> + >> +/* Arch-specific MSI-X entry data for vPCI. */ >> +struct vpci_arch_msix_entry { >> +}; > But isn't it that you don't support vPCI's MSI in the first place? > Perhaps the need for these would go away with CONFIG_HAS_VCPI_MSI? > > Jan > >
On 25.08.2021 07:44, Oleksandr Andrushchenko wrote:
> Hi, Jan!
>
> On 24.08.21 19:09, Jan Beulich wrote:
>> On 19.08.2021 14:02, Rahul Singh wrote:
>>> --- a/xen/drivers/passthrough/pci.c
>>> +++ b/xen/drivers/passthrough/pci.c
>>> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>> else
>>> iommu_enable_device(pdev);
>>>
>>> +#ifdef CONFIG_ARM
>>> + ret = vpci_add_handlers(pdev);
>>> + if ( ret ) {
>>> + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
>>> + goto out;
>>> + }
>>> +#endif
>>> pci_enable_acs(pdev);
>> I'm afraid I can't see why this is to be Arm-specific. The present
>> placement of the existing call to vpci_add_handlers() looks to be
>> sub-optimal anyway - did you look into whether it could be moved
>> to a place (potentially right here) fitting everyone? If you did,
>> would mind justifying the Arm-specific code in a non-Arm-specific
>> file in at least a post-commit-message remark?
>>
>> If it were to remain like this, please add a blank line after the #endif.
>>
>>> --- a/xen/drivers/vpci/Makefile
>>> +++ b/xen/drivers/vpci/Makefile
>>> @@ -1 +1,2 @@
>>> -obj-y += vpci.o header.o msi.o msix.o
>>> +obj-y += vpci.o header.o
>>> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
>> I continue to consider this a wrong connection - HAS_PCI_MSI expresses
>> (quoting text from patch 1 of this series) "code that implements MSI
>> functionality to support MSI within XEN", while here we're talking of
>> vPCI (i.e. guest support). I can accept that this is transiently the
>> best you can do, but then please add a comment to this effect (if
>> need be in multiple places), such that future readers or people
>> wanting to further adjust this understand why it is the way it is.
>>
>> But perhaps you instead want to introduce a HAS_VPCI_MSI (or, less
>> desirable because of possible ambiguity, HAS_VMSI) Kconfig option?
>
> Eventually we'll have the code like that:
>
> obj-y += vpci.o header.o msi.o msix.o
> obj-$(CONFIG_X86) += x86/
> obj-$(CONFIG_ARM) += arm/
>
> So, this is indeed a transitional change. Will you be ok with a comment
> about that then?
Well, yes, as said - if this is a transitional state, then a comment
will do. I was suggesting the further Kconfig option merely because
all I've been hearing so far was that MSI works entirely differently
on Arm, and hence the MSI code we have is not going to be used there
at all. If that was a correct understanding of mine (including its
extending to vPCI's MSI code), then adding just a comment would
continue to be misleading imo.
Jan
On 8/19/21 8:02 AM, Rahul Singh wrote: > MSI code that implements MSI functionality to support MSI within XEN is > not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate > the code for ARM. > > No functional change intended. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/x86/Kconfig | 1 + > xen/drivers/passthrough/Makefile | 1 + > xen/drivers/passthrough/msi.c | 96 ++++++++++++++++++++++++++++++++ > xen/drivers/passthrough/pci.c | 54 +++++------------- > xen/drivers/pci/Kconfig | 4 ++ > xen/include/xen/msi.h | 56 +++++++++++++++++++ > xen/xsm/flask/hooks.c | 8 +-- > 7 files changed, 175 insertions(+), 45 deletions(-) > create mode 100644 xen/drivers/passthrough/msi.c > create mode 100644 xen/include/xen/msi.h > ... > diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c > index f1a1217c98..fdcfeb984c 100644 > --- a/xen/xsm/flask/hooks.c > +++ b/xen/xsm/flask/hooks.c > @@ -21,7 +21,7 @@ > #include <xen/guest_access.h> > #include <xen/xenoprof.h> > #include <xen/iommu.h> > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > #include <asm/msi.h> > #endif > #include <public/xen.h> > @@ -114,7 +114,7 @@ static int get_irq_sid(int irq, u32 *sid, struct avc_audit_data *ad) > } > return security_irq_sid(irq, sid); > } > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > { > struct irq_desc *desc = irq_to_desc(irq); > if ( desc->msi_desc && desc->msi_desc->dev ) { > @@ -874,7 +874,7 @@ static int flask_map_domain_pirq (struct domain *d) > static int flask_map_domain_msi (struct domain *d, int irq, const void *data, > u32 *sid, struct avc_audit_data *ad) > { > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > const struct msi_info *msi = data; > u32 machine_bdf = (msi->seg << 16) | (msi->bus << 8) | msi->devfn; > > @@ -940,7 +940,7 @@ static int flask_unmap_domain_pirq (struct domain *d) > static int flask_unmap_domain_msi (struct domain *d, int irq, const void *data, > u32 *sid, struct avc_audit_data *ad) > { > -#ifdef CONFIG_HAS_PCI > +#ifdef CONFIG_HAS_PCI_MSI > const struct pci_dev *pdev = data; > u32 machine_bdf = (pdev->seg << 16) | (pdev->bus << 8) | pdev->devfn; > > Straightforward, so I see no issue with the flask related changes. Reviewed-by: Daniel P. Smith <dpsmith@apertussolutions.com> v/r dps
Hi Jan, > On 24 Aug 2021, at 4:53 pm, Jan Beulich <jbeulich@suse.com> wrote: > > On 19.08.2021 14:02, Rahul Singh wrote: >> --- /dev/null >> +++ b/xen/drivers/passthrough/msi.c >> @@ -0,0 +1,96 @@ >> +/* >> + * Copyright (C) 2008, Netronome Systems, Inc. > > While generally copying copyright statements when splitting source > files is probably wanted (or even necessary) I doubt this is > suitable here: None of the MSI code that you move was contributed > by them afaict. Let me remove the "Copyright Copyright (C) 2008, Netronome Systems, Inc.” . Can you please help me what copyright I will add for the next patch ? > >> + * This program is free software; you can redistribute it and/or modify it >> + * under the terms and conditions of the GNU General Public License, >> + * version 2, as published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope it will be useful, but WITHOUT >> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or >> + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for >> + * more details. >> + * >> + * You should have received a copy of the GNU General Public License along with >> + * this program; If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/init.h> >> +#include <xen/pci.h> >> +#include <asm/msi.h> > > You surely mean xen/msi.h here: Headers like this one should always > be included by the producer, no matter that it builds fine without. > Else you risk declarations and definitions to go out of sync. Ok . Let me include here “xen/msi.h” and move other required includes to “xen/msi.h" > >> +#include <asm/hvm/io.h> >> + >> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) >> +{ >> + int rc; >> + >> + if ( pdev->msix ) >> + { >> + rc = pci_reset_msix_state(pdev); >> + if ( rc ) >> + return rc; >> + msixtbl_init(d); >> + } >> + >> + return 0; >> +} >> + >> +int pdev_msi_init(struct pci_dev *pdev) >> +{ >> + unsigned int pos; >> + >> + INIT_LIST_HEAD(&pdev->msi_list); >> + >> + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), >> + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI); >> + if ( pos ) >> + { >> + uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos)); >> + >> + pdev->msi_maxvec = multi_msi_capable(ctrl); >> + } >> + >> + pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), >> + PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX); >> + if ( pos ) >> + { >> + struct arch_msix *msix = xzalloc(struct arch_msix); >> + uint16_t ctrl; >> + >> + if ( !msix ) >> + return -ENOMEM; >> + >> + spin_lock_init(&msix->table_lock); >> + >> + ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); >> + msix->nr_entries = msix_table_size(ctrl); >> + >> + pdev->msix = msix; >> + } >> + >> + return 0; >> +} >> + >> +void pdev_msi_deinit(struct pci_dev *pdev) >> +{ >> + XFREE(pdev->msix); >> +} >> + >> +void pdev_dump_msi(const struct pci_dev *pdev) >> +{ >> + const struct msi_desc *msi; >> + >> + printk("- MSIs < "); >> + list_for_each_entry ( msi, &pdev->msi_list, list ) >> + printk("%d ", msi->irq); >> + printk(">"); > > While not an exact equivalent of the original code then, could I > talk you into adding an early list_empty() check, suppressing any > output from this function if that one returned "true”? Ok. > >> @@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) >> static int _dump_pci_devices(struct pci_seg *pseg, void *arg) >> { >> struct pci_dev *pdev; >> - struct msi_desc *msi; >> >> printk("==== segment %04x ====\n", pseg->nr); >> >> list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) >> { >> - printk("%pp - %pd - node %-3d - MSIs < ", >> + printk("%pp - %pd - node %-3d ", > > Together with the request above the trailin blank here also wants to > become a leading blank in pdev_dump_msi() Ok. >> --- /dev/null >> +++ b/xen/include/xen/msi.h >> @@ -0,0 +1,56 @@ >> +/* >> + * Copyright (C) 2008, Netronome Systems, Inc. >> + * >> + * This program is free software; you can redistribute it and/or modify it >> + * under the terms and conditions of the GNU General Public License, >> + * version 2, as published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope it will be useful, but WITHOUT >> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or >> + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for >> + * more details. >> + * >> + * You should have received a copy of the GNU General Public License along with >> + * this program; If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#ifndef __XEN_MSI_H_ >> +#define __XEN_MSI_H_ >> + >> +#ifdef CONFIG_HAS_PCI_MSI >> + >> +#include <asm/msi.h> >> + >> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev); >> +int pdev_msi_init(struct pci_dev *pdev); >> +void pdev_msi_deinit(struct pci_dev *pdev); >> +void pdev_dump_msi(const struct pci_dev *pdev); >> + >> +#else /* !CONFIG_HAS_PCI_MSI */ >> +static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev) > > Please be consistent with blank lines you add; here you also want one > after the #else. Ok. > >> +{ >> + return 0; >> +} >> + >> +static inline int pdev_msi_init(struct pci_dev *pdev) >> +{ >> + return 0; >> +} >> + >> +static inline void pdev_msi_deinit(struct pci_dev *pdev) {} >> +static inline void pci_cleanup_msi(struct pci_dev *pdev) {} >> +static inline void pdev_dump_msi(const struct pci_dev *pdev) {} > > Especially for (but perhaps not limited to) this !HAS_PCI_MSI case > (where you don't include asm/msi.h and its possible dependents) > please forward-declare struct-s you use in prototypes or inline > stubs (outside the #ifdef, that is). This will allow including > this header without having to care about prereq headers. Ok. Let me do modification in next version. Regards, Rahul > > If you agree with and make all the suggested or requested changes, > feel free to add > Reviewed-by: Jan Beulich <jbeulich@suse.com> > > Jan >
On 31.08.2021 14:31, Rahul Singh wrote: >> On 24 Aug 2021, at 4:53 pm, Jan Beulich <jbeulich@suse.com> wrote: >> On 19.08.2021 14:02, Rahul Singh wrote: >>> --- /dev/null >>> +++ b/xen/drivers/passthrough/msi.c >>> @@ -0,0 +1,96 @@ >>> +/* >>> + * Copyright (C) 2008, Netronome Systems, Inc. >> >> While generally copying copyright statements when splitting source >> files is probably wanted (or even necessary) I doubt this is >> suitable here: None of the MSI code that you move was contributed >> by them afaict. > > Let me remove the "Copyright Copyright (C) 2008, Netronome Systems, Inc.” . > Can you please help me what copyright I will add for the next patch ? None? If you look around, you will find that far from all source files have such a line (or multiple of them). >>> + * This program is free software; you can redistribute it and/or modify it >>> + * under the terms and conditions of the GNU General Public License, >>> + * version 2, as published by the Free Software Foundation. >>> + * >>> + * This program is distributed in the hope it will be useful, but WITHOUT >>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or >>> + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for >>> + * more details. >>> + * >>> + * You should have received a copy of the GNU General Public License along with >>> + * this program; If not, see <http://www.gnu.org/licenses/>. >>> + */ >>> + >>> +#include <xen/init.h> >>> +#include <xen/pci.h> >>> +#include <asm/msi.h> >> >> You surely mean xen/msi.h here: Headers like this one should always >> be included by the producer, no matter that it builds fine without. >> Else you risk declarations and definitions to go out of sync. > Ok . Let me include here “xen/msi.h” and move other required includes to “xen/msi.h" Why move stuff? xen/msi.h is fins to include asm/msi.h. It's just that including asm/msi.h here is not enough. Jan
Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > pci_init(..) will be called during xen startup to initialize and probe > the PCI host-bridge driver. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/pci.c | 54 ++++++++++++++++++++++++++++++++++++ > xen/include/asm-arm/device.h | 1 + > 2 files changed, 55 insertions(+) > > diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c > index dc55d23778..d1c9cf997d 100644 > --- a/xen/arch/arm/pci/pci.c > +++ b/xen/arch/arm/pci/pci.c > @@ -14,13 +14,67 @@ > * along with this program. If not, see <http://www.gnu.org/licenses/>. > */ > > +#include <xen/acpi.h> > +#include <xen/device_tree.h> > +#include <xen/errno.h> > +#include <xen/init.h> > #include <xen/pci.h> > +#include <xen/param.h> This include doesn't look to be necessary (yet?). > > int arch_pci_clean_pirqs(struct domain *d) > { > return 0; > } > > +static int __init dt_pci_init(void) > +{ > + struct dt_device_node *np; > + int rc; > + > + dt_for_each_device_node(dt_host, np) > + { > + rc = device_init(np, DEVICE_PCI, NULL); > + if( !rc ) > + continue; > + /* > + * Ignore the following error codes: > + * - EBADF: Indicate the current is not an pci > + * - ENODEV: The pci device is not present or cannot be used by > + * Xen. > + */ > + else if ( rc != -EBADF && rc != -ENODEV ) > + { > + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n"); > + return rc; > + } > + } > + > + return 0; > +} > + > +#ifdef CONFIG_ACPI > +static void __init acpi_pci_init(void) > +{ > + printk(XENLOG_ERR "ACPI pci init not supported \n"); > + return; > +} > +#else > +static inline void __init acpi_pci_init(void) { } > +#endif > + > +static int __init pci_init(void) > +{ > + if ( acpi_disabled ) > + dt_pci_init(); > + else > + acpi_pci_init(); > + > + pci_segments_init(); Shouldn't this happen before the PCI initialization? > + > + return 0; > +} > +__initcall(pci_init); > + > /* > * Local variables: > * mode: C > diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h > index ee7cff2d44..5ecd5e7bd1 100644 > --- a/xen/include/asm-arm/device.h > +++ b/xen/include/asm-arm/device.h > @@ -34,6 +34,7 @@ enum device_class > DEVICE_SERIAL, > DEVICE_IOMMU, > DEVICE_GIC, > + DEVICE_PCI, > /* Use for error */ > DEVICE_UNKNOWN, > }; > Cheers, -- Julien Grall
Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > XEN during boot will read the PCI device tree node “reg” property > and will map the PCI config space to the XEN memory. > > As of now "pci-host-ecam-generic" compatible board is supported. I think the word "only" is missing. > > "linux,pci-domain" device tree property assigns a fixed PCI domain > number to a host bridge, otherwise an unstable (across boots) unique > number will be assigned by Linux.This property has to be in sync with Typo: missing space after the '.' > XEN to access the PCI devices. I would expand a little bit the last sentence to explain why the need to be sync-ed. > > XEN will read the “linux,pci-domain” property from the device tree node > and configure the host bridge segment number accordingly. If this > property is not available XEN will allocate the unique segment number > to the host bridge. > > dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly > imported from the Linux source tree. What was the Linux commit used? I also read "directly imported" as a verbartim copy but AFAICT the implementation has been slightly reworked. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/Makefile | 2 + > xen/arch/arm/pci/pci-host-common.c | 261 ++++++++++++++++++++++++++++ > xen/arch/arm/pci/pci-host-generic.c | 55 ++++++ > xen/include/asm-arm/pci.h | 28 +++ > 4 files changed, 346 insertions(+) > create mode 100644 xen/arch/arm/pci/pci-host-common.c > create mode 100644 xen/arch/arm/pci/pci-host-generic.c > > diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile > index a9ee0b9b44..f3d97f859e 100644 > --- a/xen/arch/arm/pci/Makefile > +++ b/xen/arch/arm/pci/Makefile > @@ -1,2 +1,4 @@ > obj-y += pci.o > obj-y += pci-access.o > +obj-y += pci-host-generic.o > +obj-y += pci-host-common.o > diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c > new file mode 100644 > index 0000000000..9dd9b02271 > --- /dev/null > +++ b/xen/arch/arm/pci/pci-host-common.c > @@ -0,0 +1,261 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * Based on Linux drivers/pci/ecam.c > + * Copyright 2016 Broadcom. > + * > + * Based on Linux drivers/pci/controller/pci-host-common.c > + * Based on Linux drivers/pci/controller/pci-host-generic.c > + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> > + * > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/init.h> > +#include <xen/pci.h> > +#include <asm/pci.h> AFAICT, <xen/pci.h> already includes <asm/pci.h>. So this looks unneccessary. > +#include <xen/rwlock.h> > +#include <xen/sched.h> > +#include <xen/vmap.h> > + > +/* > + * List for all the pci host bridges. > + */ > + > +static LIST_HEAD(pci_host_bridges); > + > +static atomic_t domain_nr = ATOMIC_INIT(-1); > + > +bool dt_pci_parse_bus_range(struct dt_device_node *dev, > + struct pci_config_window *cfg) Aside, "pci_config_window", the function is not Arm specific. Would it be possible to consider to introduce "struct resource" in Xen so this function can be moved in common/device_tree.c? > +{ > + const __be32 *cells; > + uint32_t len; > + > + cells = dt_get_property(dev, "bus-range", &len); > + /* bus-range should at least be 2 cells */ > + if ( !cells || (len < (sizeof(*cells) * 2)) ) > + return false; How about introducing dt_property_read_u32_array()? > + > + cfg->busn_start = dt_next_cell(1, &cells); > + cfg->busn_end = dt_next_cell(1, &cells); > + > + return true; > +} > + > +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len) > +{ > + return ioremap_nocache(start, len); > +} > + > +static void pci_ecam_free(struct pci_config_window *cfg) > +{ > + if ( cfg->win ) > + iounmap(cfg->win); > + > + xfree(cfg); > +} > + > +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > + int ecam_reg_idx) > +{ > + int err; > + struct pci_config_window *cfg; > + paddr_t addr, size; > + > + cfg = xzalloc(struct pci_config_window); > + if ( !cfg ) > + return NULL; > + > + err = dt_pci_parse_bus_range(dev, cfg); > + if ( !err ) { > + cfg->busn_start = 0; > + cfg->busn_end = 0xff; > + printk(XENLOG_ERR "%s:No bus range found for pci controller\n", Typo: Missing space after ':'. > + dt_node_full_name(dev)); > + } else { > + if ( cfg->busn_end > cfg->busn_start + 0xff ) > + cfg->busn_end = cfg->busn_start + 0xff; > + } > + > + /* Parse our PCI ecam register address*/ > + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size); > + if ( err ) > + goto err_exit; > + > + cfg->phys_addr = addr; > + cfg->size = size; > + > + /* > + * On 64-bit systems, we do a single ioremap for the whole config space > + * since we have enough virtual address range available. On 32-bit, we In Xen on Arm64, the VMAP is actually only 1GB. So it is not that big and this will compete with other mapping like ITS, global domain mapping... So I think the vMAP area will need to increase to cater to increase usage. > + * ioremap the config space for each bus individually. > + * > + * As of now only 64-bit is supported 32-bit is not supported. > + */ > + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size); > + if ( !cfg->win ) > + goto err_exit_remap; > + > + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, The physical address is a paddr_t. So this needs to use PRIpaddr. Also, please use preprent hexadecimal with 0x. This makes a lot easier to differentiate hexa vs decimal in the log. > + cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); > + > + return cfg; > + > +err_exit_remap: > + printk(XENLOG_ERR "ECAM ioremap failed\n"); > +err_exit: > + pci_ecam_free(cfg); Coding style: Please add a new line before return. > + return NULL; > +} > + > +struct pci_host_bridge *pci_alloc_host_bridge(void) > +{ > + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); > + > + if ( !bridge ) > + return NULL; > + > + INIT_LIST_HEAD(&bridge->node); > + bridge->bus_start = ~0; > + bridge->bus_end = ~0; Coding style: Please add a new line before return. > + return bridge; > +} > + > +void pci_add_host_bridge(struct pci_host_bridge *bridge) > +{ > + list_add_tail(&bridge->node, &pci_host_bridges); > +} > + > +static int pci_get_new_domain_nr(void) > +{ > + return atomic_inc_return(&domain_nr); > +} > + > +/* > + * This function will try to obtain the host bridge domain number by > + * finding a property called "linux,pci-domain" of the given device node. > + * > + * @node: device tree node with the domain information > + * > + * Returns the associated domain number from DT in the range [0-0xffff], or > + * a negative value if the required property is not found. > + */ > +static int dt_get_pci_domain_nr(struct dt_device_node *node) Nothing looks Arm specific for this function. Can you move it in device_tree.c? > +{ > + u32 domain; > + int error; > + > + error = dt_property_read_u32(node, "linux,pci-domain", &domain); > + if ( !error ) > + return -EINVAL; > + > + return (u16)domain; > +} > + > +static int pci_bus_find_domain_nr(struct dt_device_node *dev) > +{ > + static int use_dt_domains = -1; > + int domain; > + > + domain = dt_get_pci_domain_nr(dev); > + > + /* > + * Check DT domain and use_dt_domains values. > + * > + * If DT domain property is valid (domain >= 0) and > + * use_dt_domains != 0, the DT assignment is valid since this means > + * we have not previously allocated a domain number by using > + * pci_get_new_domain_nr(); we should also update use_dt_domains to > + * 1, to indicate that we have just assigned a domain number from > + * DT. > + * > + * If DT domain property value is not valid (ie domain < 0), and we > + * have not previously assigned a domain number from DT > + * (use_dt_domains != 1) we should assign a domain number by > + * using the: > + * > + * pci_get_new_domain_nr() > + * > + * API and update the use_dt_domains value to keep track of method we > + * are using to assign domain numbers (use_dt_domains = 0). > + * > + * All other combinations imply we have a platform that is trying > + * to mix domain numbers obtained from DT and pci_get_new_domain_nr(), > + * which is a recipe for domain mishandling and it is prevented by > + * invalidating the domain value (domain = -1) and printing a > + * corresponding error. > + */ > + if ( domain >= 0 && use_dt_domains ) > + { > + use_dt_domains = 1; > + } > + else if ( domain < 0 && use_dt_domains != 1 ) > + { > + use_dt_domains = 0; > + domain = pci_get_new_domain_nr(); > + } > + else > + { > + printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n"); > + BUG(); I don't think pci_bus_find_domain_nr() should be the function that crashes Xen. Instead, this should be propagated to the highest possible callers and let it decide what to do. > + } > + > + return domain; > +} > + > +int pci_host_common_probe(struct dt_device_node *dev, > + int ecam_reg_idx) > +{ > + struct pci_host_bridge *bridge; > + struct pci_config_window *cfg; > + int err; > + > + bridge = pci_alloc_host_bridge(); > + if ( !bridge ) > + return -ENOMEM; > + > + /* Parse and map our Configuration Space windows */ > + cfg = gen_pci_init(dev, ecam_reg_idx); > + if ( !cfg ) > + { > + err = -ENOMEM; > + goto err_exit; > + } > + > + bridge->dt_node = dev; > + bridge->sysdata = cfg; > + bridge->bus_start = cfg->busn_start; > + bridge->bus_end = cfg->busn_end; > + > + bridge->segment = pci_bus_find_domain_nr(dev); > + > + pci_add_host_bridge(bridge); > + > + return 0; > + > +err_exit: > + xfree(bridge); > + return err; > +} > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c > new file mode 100644 > index 0000000000..13d0f7f999 > --- /dev/null > +++ b/xen/arch/arm/pci/pci-host-generic.c > @@ -0,0 +1,55 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * Based on Linux drivers/pci/controller/pci-host-common.c > + * Based on Linux drivers/pci/controller/pci-host-generic.c > + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <asm/device.h> > +#include <xen/pci.h> > +#include <asm/pci.h> > + > +static const struct dt_device_match gen_pci_dt_match[] = { > + { .compatible = "pci-host-ecam-generic" }, > + { }, > +}; > + > +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) > +{ > + const struct dt_device_match *of_id; > + > + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); This seems to be a bit pointless to me as you already know the compatible (there is only one possible...). > + > + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", > + dt_node_full_name(dev), of_id->compatible); > + > + return pci_host_common_probe(dev, 0); > +} > + > +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) > +.dt_match = gen_pci_dt_match, > +.init = gen_pci_dt_init, > +DT_DEVICE_END > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index 61e43da088..58a51e724e 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -26,6 +26,34 @@ struct arch_pci_dev { > struct device dev; > }; > > +/* > + * struct to hold the mappings of a config space window. This > + * is expected to be used as sysdata for PCI controllers that > + * use ECAM. > + */ > +struct pci_config_window { > + paddr_t phys_addr; > + paddr_t size; > + uint8_t busn_start; > + uint8_t busn_end; > + void __iomem *win; > +}; > + > +/* > + * struct to hold pci host bridge information > + * for a PCI controller. > + */ > +struct pci_host_bridge { > + struct dt_device_node *dt_node; /* Pointer to the associated DT node */ > + struct list_head node; /* Node in list of host bridges */ > + uint16_t segment; /* Segment number */ > + u8 bus_start; /* Bus start of this bridge. */ > + u8 bus_end; /* Bus end of this bridge. */ Please use uint8_t rather than u8. > + void *sysdata; /* Pointer to the config space window*/ > +}; > + > +int pci_host_common_probe(struct dt_device_node *dev, > + int ecam_reg_idx); > #else /*!CONFIG_HAS_PCI*/ > > struct arch_pci_dev { }; > Cheers, -- Julien Grall
On 19/08/2021 15:16, Rahul Singh wrote: > Hi Julien, Hi Rahul, Sorry for the late reply. >> On 19 Aug 2021, at 1:18 pm, Julien Grall <julien@xen.org> wrote: >> >> Hi Rahul, >> >> On 19/08/2021 13:02, Rahul Singh wrote: >>> MSI code that implements MSI functionality to support MSI within XEN is >>> not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate >> >> Can you clarify what you mean by not usable? Is it because we lack of support or we have no plan to use it? > > We have no plan to use it. Code moved to CONFIG_HAS_PCI_MSI will only be required for ARM if we I read "not usable", as the code doesn't yet work on Arm. This is quite possible, but from you wrote the main reason is that the code is not necessary yet (see below) on Arm. > decide to support PCI device access (PCI MSI interrupt support) within XEN. As of now, we are planning > to add support for PCI device access for DOM0/DOMU guests not for XEN. This is probably going to happen sooner rather than later as, AFAICT, there are SMMUs out which signals using MSIs. But, AFAICT, this code would also be used if we need to manage the MSIs in Xen on behalf of the guest (such as if the platform is using GICv2m). That said, I would be fine with gating the MSI code behind a new config. However, I think the commit message wants to be clarified into why we don't need the option. Maybe something like: "On Arm, the inital plan is to only support GICv3 ITS which doesn't require us to manage the MSIs because the HW will protect against spoofing. Introduce a new option XXXX to protect the MSI code. " Cheers, -- Julien Grall
[-- Attachment #1: Type: text/plain, Size: 2022 bytes --] On Fri, 20 Aug 2021, Jan Beulich wrote: > On 20.08.2021 13:41, Rahul Singh wrote: > > Hi Jan > > > >> On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote: > >> > >> On 19.08.2021 14:02, Rahul Singh wrote: > >>> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM > >>> architecture. Move the code under CONFIG_X86 flag to gate the code for > >>> ARM. > >> > >> Please give at least one example of what it is that fails to compile. > >> What an appropriate action is depends on the nature of the error(s), > >> and from looking at the enclosed code I cannot easily see what it > >> might be that breaks for Arm. > > > > I am observing below error when enabled ACPI && HAS_PCI for ARM. > > > > prelink.o: In function `pcie_aer_get_firmware_first’: > > /xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse' > > aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined > > > > I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code. > > obj-$(CONFIG_X86) += apei/ > > > > I am not sure whether we need this code for ARM architecture > > that is why I gate the code for ARM via CONFIG_X86 > > So you Arm folks will probably want to settle on that aspect first. What > is wanted to keep things building depends on that. Reading the APEI description, it looks like there might be some use for it on ARM but it would work a bit differently from x86 as there are no NMIs on ARM. So enabling APEI on ARM is not just a matter of enabling the build of apei/, it is not going to be straightforward. For the scope of this series (which is actually about PCI), I would leave it alone, and keep apei/ x86 only, which means #ifdefing pcie_aer_get_firmware_first. I would just add an in-code comment saying "APEI not supported on ARM yet". Another option would be to introduce a symbol like HAS_ACPI_APEI but it is a bit overkill for this.
On Thu, 19 Aug 2021, Rahul Singh wrote: > pci_init(..) will be called during xen startup to initialize and probe > the PCI host-bridge driver. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/pci.c | 54 ++++++++++++++++++++++++++++++++++++ > xen/include/asm-arm/device.h | 1 + > 2 files changed, 55 insertions(+) > > diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c > index dc55d23778..d1c9cf997d 100644 > --- a/xen/arch/arm/pci/pci.c > +++ b/xen/arch/arm/pci/pci.c > @@ -14,13 +14,67 @@ > * along with this program. If not, see <http://www.gnu.org/licenses/>. > */ > > +#include <xen/acpi.h> > +#include <xen/device_tree.h> > +#include <xen/errno.h> > +#include <xen/init.h> > #include <xen/pci.h> > +#include <xen/param.h> > > int arch_pci_clean_pirqs(struct domain *d) > { > return 0; > } > > +static int __init dt_pci_init(void) > +{ > + struct dt_device_node *np; > + int rc; > + > + dt_for_each_device_node(dt_host, np) > + { > + rc = device_init(np, DEVICE_PCI, NULL); > + if( !rc ) > + continue; > + /* > + * Ignore the following error codes: > + * - EBADF: Indicate the current is not an pci ^ a > + * - ENODEV: The pci device is not present or cannot be used by > + * Xen. > + */ > + else if ( rc != -EBADF && rc != -ENODEV ) > + { > + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n"); > + return rc; > + } > + } > + > + return 0; > +} > + > +#ifdef CONFIG_ACPI > +static void __init acpi_pci_init(void) If the DT init function returns int, then it would make sense for the ACPI init function to return int as well? > +{ > + printk(XENLOG_ERR "ACPI pci init not supported \n"); > + return; > +} > +#else > +static inline void __init acpi_pci_init(void) { } > +#endif > + > +static int __init pci_init(void) > +{ > + if ( acpi_disabled ) > + dt_pci_init(); > + else > + acpi_pci_init(); > + > + pci_segments_init(); > + > + return 0; > +} > +__initcall(pci_init); > + > /* > * Local variables: > * mode: C > diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h > index ee7cff2d44..5ecd5e7bd1 100644 > --- a/xen/include/asm-arm/device.h > +++ b/xen/include/asm-arm/device.h > @@ -34,6 +34,7 @@ enum device_class > DEVICE_SERIAL, > DEVICE_IOMMU, > DEVICE_GIC, > + DEVICE_PCI, > /* Use for error */ > DEVICE_UNKNOWN, > }; > -- > 2.17.1 >
Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > Add support for PCI ecam operations to access the PCI > configuration space. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/Makefile | 1 + > xen/arch/arm/pci/ecam.c | 63 +++++++++++++++++++++++++++++ > xen/arch/arm/pci/pci-access.c | 53 ++++++++++++++++++++++++ > xen/arch/arm/pci/pci-host-common.c | 13 +++++- > xen/arch/arm/pci/pci-host-generic.c | 8 +++- > xen/include/asm-arm/pci.h | 32 +++++++++++++++ > 6 files changed, 167 insertions(+), 3 deletions(-) > create mode 100644 xen/arch/arm/pci/ecam.c > > diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile > index f3d97f859e..6f32fbbe67 100644 > --- a/xen/arch/arm/pci/Makefile > +++ b/xen/arch/arm/pci/Makefile > @@ -2,3 +2,4 @@ obj-y += pci.o > obj-y += pci-access.o > obj-y += pci-host-generic.o > obj-y += pci-host-common.o > +obj-y += ecam.o > diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c > new file mode 100644 > index 0000000000..91c691b41f > --- /dev/null > +++ b/xen/arch/arm/pci/ecam.c > @@ -0,0 +1,63 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * Based on Linux drivers/pci/ecam.c > + * Copyright 2016 Broadcom > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/pci.h> > +#include <xen/sched.h> > + > +/* > + * Function to implement the pci_ops ->map_bus method. > + */ > +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, > + uint32_t sbdf, uint32_t where) > +{ > + const struct pci_config_window *cfg = bridge->sysdata; > + unsigned int devfn_shift = cfg->ops->bus_shift - 8; > + void __iomem *base; > + > + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ; AFAICT, pci_sbdf is an union between a 32-bit and a structure. So please don't use the cast and use the 32-bit field to assign the value. Also, there is an extra space before ';'. > + unsigned int busn = sbdf_t.bus; > + > + if ( busn < cfg->busn_start || busn > cfg->busn_end ) > + return NULL; > + > + busn -= cfg->busn_start; > + base = cfg->win + (busn << cfg->ops->bus_shift); > + > + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; How about using PCI_DEVFN2(sbdf)? This would allow you to drop the use of sbdf_t completely (sbdf_t.bus could be replaced with PCI_BUS(sdbf)). > +} > + > +/* ECAM ops */ > +const struct pci_ecam_ops pci_generic_ecam_ops = { > + .bus_shift = 20, > + .pci_ops = { > + .map_bus = pci_ecam_map_bus, > + .read = pci_generic_config_read, > + .write = pci_generic_config_write, > + } > +}; > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c > index b938047c03..f39f6a3a38 100644 > --- a/xen/arch/arm/pci/pci-access.c > +++ b/xen/arch/arm/pci/pci-access.c > @@ -15,6 +15,59 @@ > */ > > #include <xen/pci.h> > +#include <asm/io.h> > + > +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t *value) > +{ > + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); Please add a newline here. > + if (!addr) { You seem to use a mix of Xen and Linux coding style. If the file contains mostly Xen code, then we should use the former. > + *value = ~0; > + return -ENODEV; > + } > + > + switch (len) > + { > + case 1: > + *value = readb(addr); > + break; > + case 2: > + *value = readw(addr); > + break; > + case 4: > + *value = readl(addr); > + break; > + default: > + BUG(); > + } > + > + return 0; > +} > + > +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t value) > +{ > + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); > + if (!addr) > + return -ENODEV; > + > + switch (len) > + { > + case 1: > + writeb(value, addr); > + break; > + case 2: > + writew(value, addr); > + break; > + case 4: > + writel(value, addr); > + break; > + default: > + BUG(); > + } > + > + return 0; > +} > > static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, > unsigned int len) > diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c > index 9dd9b02271..c582527e92 100644 > --- a/xen/arch/arm/pci/pci-host-common.c > +++ b/xen/arch/arm/pci/pci-host-common.c > @@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg) > } > > static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > + const struct pci_ecam_ops *ops, > int ecam_reg_idx) > { > int err; > @@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > > cfg->phys_addr = addr; > cfg->size = size; > + cfg->ops = ops; > > /* > * On 64-bit systems, we do a single ioremap for the whole config space > @@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, > cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); > > + if ( ops->init ) > + { > + err = ops->init(cfg); > + if (err) > + goto err_exit; > + } > + > return cfg; > > err_exit_remap: > @@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev) > } > > int pci_host_common_probe(struct dt_device_node *dev, > + const struct pci_ecam_ops *ops, > int ecam_reg_idx) > { > struct pci_host_bridge *bridge; > @@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev, > return -ENOMEM; > > /* Parse and map our Configuration Space windows */ > - cfg = gen_pci_init(dev, ecam_reg_idx); > + cfg = gen_pci_init(dev, ops, ecam_reg_idx); > if ( !cfg ) > { > err = -ENOMEM; > @@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev, > > bridge->dt_node = dev; > bridge->sysdata = cfg; > + bridge->ops = &ops->pci_ops; > bridge->bus_start = cfg->busn_start; > bridge->bus_end = cfg->busn_end; > > diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c > index 13d0f7f999..2d652e8910 100644 > --- a/xen/arch/arm/pci/pci-host-generic.c > +++ b/xen/arch/arm/pci/pci-host-generic.c > @@ -23,20 +23,24 @@ > #include <asm/pci.h> > > static const struct dt_device_match gen_pci_dt_match[] = { > - { .compatible = "pci-host-ecam-generic" }, > + { .compatible = "pci-host-ecam-generic", > + .data = &pci_generic_ecam_ops }, > + > { }, > }; > > static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) > { > const struct dt_device_match *of_id; > + const struct pci_ecam_ops *ops; > > of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); > + ops = (struct pci_ecam_ops *) of_id->data; > > printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", > dt_node_full_name(dev), of_id->compatible); > > - return pci_host_common_probe(dev, 0); > + return pci_host_common_probe(dev, ops, 0); > } > > DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index 58a51e724e..22866244d2 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -37,6 +37,7 @@ struct pci_config_window { > uint8_t busn_start; > uint8_t busn_end; > void __iomem *win; > + const struct pci_ecam_ops *ops; > }; > > /* > @@ -50,10 +51,41 @@ struct pci_host_bridge { > u8 bus_start; /* Bus start of this bridge. */ > u8 bus_end; /* Bus end of this bridge. */ > void *sysdata; /* Pointer to the config space window*/ > + const struct pci_ops *ops; > }; > > +struct pci_ops { > + void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t offset); > + int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t *value); > + int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t value); > +}; > + > +/* > + * struct to hold pci ops and bus shift of the config window > + * for a PCI controller. > + */ > +struct pci_ecam_ops { > + unsigned int bus_shift; > + struct pci_ops pci_ops; > + int (*init)(struct pci_config_window *); > +}; > + > +/* Default ECAM ops */ > +extern const struct pci_ecam_ops pci_generic_ecam_ops; > + > int pci_host_common_probe(struct dt_device_node *dev, > + const struct pci_ecam_ops *ops, > int ecam_reg_idx); > +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t *value); > +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t value); > +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, > + uint32_t sbdf, uint32_t where); > + > #else /*!CONFIG_HAS_PCI*/ > > struct arch_pci_dev { }; > Cheers, -- Julien Grall
Hi,
On 19/08/2021 13:02, Rahul Singh wrote:
> Compilation error is observed when HAS_PCI is enabled for ARM
> architecture.
>
> Add definition for arch_iommu_use_permitted() and
> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to
> fix compilation error.
>
> pci.c: In function ‘deassign_device’:
> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’;
> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration]
> pci_to_dev(pdev));
>
> pci.c:18: undefined reference to `pci_conf_read16’
> pci.c:880: undefined reference to `arch_pci_clean_pirqs’
> pci.c:1392: undefined reference to `arch_iommu_use_permitted'
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/pci/Makefile | 2 +
> xen/arch/arm/pci/pci-access.c | 61 +++++++++++++++++++++++++++++
> xen/arch/arm/pci/pci.c | 32 +++++++++++++++
> xen/drivers/passthrough/arm/iommu.c | 5 +++
> xen/include/asm-arm/pci.h | 33 ++++++++++++++--
> 6 files changed, 131 insertions(+), 3 deletions(-)
> create mode 100644 xen/arch/arm/pci/Makefile
> create mode 100644 xen/arch/arm/pci/pci-access.c
> create mode 100644 xen/arch/arm/pci/pci.c
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 3d3b97b5b4..0e14a5e5c8 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
> obj-y += platforms/
> endif
> obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_HAS_PCI) += pci/
This should go before platforms/ to keep the order alphabetically.
Cheers,
--
Julien Grall
Hi Rahul, On 19/08/2021 13:02, Rahul Singh wrote: > The existing VPCI support available for X86 is adapted for Arm. > When the device is added to XEN via the hyper call > “PHYSDEVOP_pci_device_add”, VPCI handler for the config space > access is added to the Xen to emulate the PCI devices config space. > > A MMIO trap handler for the PCI ECAM space is registered in XEN > so that when guest is trying to access the PCI config space,XEN > will trap the access and emulate read/write using the VPCI and > not the real PCI hardware. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/Makefile | 1 + > xen/arch/arm/domain.c | 4 ++ > xen/arch/arm/vpci.c | 96 +++++++++++++++++++++++++++++++++++ > xen/arch/arm/vpci.h | 37 ++++++++++++++ > xen/drivers/passthrough/pci.c | 7 +++ > xen/drivers/vpci/Makefile | 3 +- > xen/drivers/vpci/header.c | 2 + > xen/include/asm-arm/domain.h | 5 +- > xen/include/asm-arm/pci.h | 8 +++ > xen/include/public/arch-arm.h | 4 ++ > 10 files changed, 165 insertions(+), 2 deletions(-) > create mode 100644 xen/arch/arm/vpci.c > create mode 100644 xen/arch/arm/vpci.h > > diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile > index 0e14a5e5c8..7cdce684a4 100644 > --- a/xen/arch/arm/Makefile > +++ b/xen/arch/arm/Makefile > @@ -7,6 +7,7 @@ obj-y += platforms/ > endif > obj-$(CONFIG_TEE) += tee/ > obj-$(CONFIG_HAS_PCI) += pci/ > +obj-$(CONFIG_HAS_VPCI) += vpci.o > > obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o > obj-y += bootfdt.init.o > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c > index 19c756ac3d..d99c653626 100644 > --- a/xen/arch/arm/domain.c > +++ b/xen/arch/arm/domain.c > @@ -40,6 +40,7 @@ > #include <asm/vtimer.h> > > #include "vuart.h" > +#include "vpci.h" Please order the includes alphabetically. So this one should go before "vuart.h". > > DEFINE_PER_CPU(struct vcpu *, curr_vcpu); > > @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d, > if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) ) > goto fail; > > + if ( (rc = domain_vpci_init(d)) != 0 ) > + goto fail; > + > return 0; > > fail: > diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c > new file mode 100644 > index 0000000000..da8b1ca13c > --- /dev/null > +++ b/xen/arch/arm/vpci.c > @@ -0,0 +1,96 @@ > +/* > + * xen/arch/arm/vpci.c > + * Copyright (c) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + */ > +#include <xen/sched.h> NIT: Please add a newline between generic and arch specific includes. > +#include <asm/mmio.h> > + > +/* Do some sanity checks. */ > +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len) > +{ > + /* Check access size. */ > + if ( len != 1 && len != 2 && len != 4 && len != 8 ) > + return false; You will allow all the possible value of len (this is coming from the HW). So I feels this is a bit too much to check for every I/O to the vPCI. If you really want to keep the check then you can simply check that len is < 8 because the two callers compute it with (1 << S). So there is no way to set it 3, 5, 6 and 7. > + > + /* Check that access is size aligned. */ > + if ( (reg & (len - 1)) ) > + return false; > + > + return true; > +} > + > +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, > + register_t *r, void *p) > +{ > + unsigned int reg; > + pci_sbdf_t sbdf; > + uint32_t data = 0; > + unsigned int size = 1U << info->dabt.size; > + > + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); > + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); This logic is the same as below in vpci_mmio_write(). So I think you want to provide an helper because this is not trivial to read. Also, for the first line, I think you can re-use MMCFG_BDF() from the x86 code. For the second line, I would define the value so it is clearer to understand that they mean (although & 3 is fine to me) . > + > + if ( !vpci_mmio_access_allowed(reg, size) ) > + return 1; So, you will a guest will read 0 if the access is unaligned. This seems an odd behavior given this is not an allowed access. AFAIU, the HW would likely trow a data abort because you can't do unalign access on uncachable memory. So I think we should return 0 here to let the MMIO handler inject a data abort. > + > + data = vpci_read(sbdf, reg, size); So in vpci_mmio_access_allowed(), you will allow a guest to read a 64-bit value. But... vpci_read() will return a 32-bit value. Looking at the x86 code, they have a second call to vpci_read() to handle the top 32-bit. Any reason why this was not implemented on Arm? If we need to implement it then I think this should be implement in vpci_read() to avoid duplication between x86 and arm. > + > + memcpy(r, &data, size); From my understanding, any unused bit should be 0. So this should be: *r = data; > + > + return 1; > +} > + > +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, > + register_t r, void *p) My remarks on vpci_mmio_read() applies here too. > +{ > + unsigned int reg; > + pci_sbdf_t sbdf; > + uint32_t data = r; > + unsigned int size = 1U << info->dabt.size; > + > + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); > + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); > + > + if ( !vpci_mmio_access_allowed(reg, size) ) > + return 1; > + > + vpci_write(sbdf, reg, size, data); > + > + return 1; > +} > + > +static const struct mmio_handler_ops vpci_mmio_handler = { > + .read = vpci_mmio_read, > + .write = vpci_mmio_write, > +}; > + > +int domain_vpci_init(struct domain *d) > +{ > + if ( !has_vpci(d) ) > + return 0; > + > + register_mmio_handler(d, &vpci_mmio_handler, > + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); > + > + return 0; > +} > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * indent-tabs-mode: nil > + * End: > + */ > + > diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h > new file mode 100644 > index 0000000000..8a093bb705 > --- /dev/null > +++ b/xen/arch/arm/vpci.h > @@ -0,0 +1,37 @@ > +/* > + * xen/arch/arm/vpci.h > + * Copyright (c) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + */ > + > +#ifndef __ARCH_ARM_VPCI_H__ > +#define __ARCH_ARM_VPCI_H__ > + > +#ifdef CONFIG_HAS_VPCI > +int domain_vpci_init(struct domain *d); > +#else > +static inline int domain_vpci_init(struct domain *d) > +{ > + return 0; > +} > +#endif > + > +#endif /* __ARCH_ARM_VPCI_H__ */ > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c > index c23c8cb06b..56e261e9bd 100644 > --- a/xen/drivers/passthrough/pci.c > +++ b/xen/drivers/passthrough/pci.c > @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, > else > iommu_enable_device(pdev); > > +#ifdef CONFIG_ARM > + ret = vpci_add_handlers(pdev); > + if ( ret ) { > + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret); > + goto out; > + } > +#endif > pci_enable_acs(pdev); > > out: > diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile > index 55d1bdfda0..1a1413b93e 100644 > --- a/xen/drivers/vpci/Makefile > +++ b/xen/drivers/vpci/Makefile > @@ -1 +1,2 @@ > -obj-y += vpci.o header.o msi.o msix.o > +obj-y += vpci.o header.o > +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o > diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c > index ba9a036202..f8cd55e7c0 100644 > --- a/xen/drivers/vpci/header.c > +++ b/xen/drivers/vpci/header.c > @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, > * FIXME: punching holes after the p2m has been set up might be racy for > * DomU usage, needs to be revisited. > */ > +#ifdef CONFIG_HAS_PCI_MSI > if ( map && !rom_only && vpci_make_msix_hole(pdev) ) > return; > +#endif > > for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) > { > diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h > index c9277b5c6d..d742b94bd6 100644 > --- a/xen/include/asm-arm/domain.h > +++ b/xen/include/asm-arm/domain.h > @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {} > > #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag) > > -#define has_vpci(d) ({ (void)(d); false; }) > +/* For X86 VPCI is enabled and tested for PVH DOM0 only but > + * for ARM we enable support VPCI for guest domain also. > + */ > +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); }) > > #endif /* __ASM_DOMAIN_H__ */ > > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index 756f8637ab..c58152de80 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -27,6 +27,14 @@ struct arch_pci_dev { > struct device dev; > }; > > +/* Arch-specific MSI data for vPCI. */ > +struct vpci_arch_msi { > +}; > + > +/* Arch-specific MSI-X entry data for vPCI. */ > +struct vpci_arch_msix_entry { > +}; > + > /* > * struct to hold the mappings of a config space window. This > * is expected to be used as sysdata for PCI controllers that > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h > index 64a2ca30da..0a9749e768 100644 > --- a/xen/include/public/arch-arm.h > +++ b/xen/include/public/arch-arm.h > @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t; > #define GUEST_PL011_BASE xen_mk_ullong(0x22000000) > #define GUEST_PL011_SIZE xen_mk_ullong(0x00001000) > > +/* VPCI ECAM mappings */ > +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) All the values for the memory layout has been defined in ascending order. So please add the vCPI at the correct place. If I am not mistaken, dhis should be before the GUEST_ACPI_*. > +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) Please document how to decide the size. This is important for the future if we need to change the size. Cheers, -- Julien Grall
On 20/08/2021 17:03, Rahul Singh wrote: > Hi Julien, Hi Rahul, >> On 19 Aug 2021, at 2:00 pm, Julien Grall <julien@xen.org> wrote: >> >> Hi Rahul, >> >> On 19/08/2021 13:02, Rahul Singh wrote: >>> libxl will create an emulated PCI device tree node in the device tree to >>> enable the guest OS to discover the virtual PCI during guest boot. >>> Emulated PCI device tree node will only be created when there is any >>> device assigned to guest. >>> A new area has been reserved in the arm guest physical map at >>> which the VPCI bus is declared in the device tree (reg and ranges >>> parameters of the node). >>> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >>> --- >>> tools/libs/light/libxl_arm.c | 109 ++++++++++++++++++++++++++ >>> tools/libs/light/libxl_types.idl | 1 + >>> tools/xl/xl_parse.c | 2 + >>> xen/include/public/arch-arm.h | 11 +++ >>> xen/include/public/device_tree_defs.h | 1 + >>> 5 files changed, 124 insertions(+) >>> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c >>> index e3140a6e00..a091e97e76 100644 >>> --- a/tools/libs/light/libxl_arm.c >>> +++ b/tools/libs/light/libxl_arm.c >>> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt, >>> return fdt_property(fdt, "reg", regs, sizeof(regs)); >>> } >>> +static int fdt_property_values(libxl__gc *gc, void *fdt, >>> + const char *name, unsigned num_cells, ...) >>> +{ >>> + uint32_t prop[num_cells]; >>> + be32 *cells = &prop[0]; >>> + int i; >>> + va_list ap; >>> + uint32_t arg; >>> + >>> + va_start(ap, num_cells); >>> + for (i = 0 ; i < num_cells; i++) { >>> + arg = va_arg(ap, uint32_t); >>> + set_cell(&cells, 1, arg); >>> + } >>> + va_end(ap); >>> + >>> + return fdt_property(fdt, name, prop, sizeof(prop)); >>> +} >>> + >>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt, >>> + unsigned addr_cells, >>> + unsigned size_cells, >>> + unsigned num_regs, ...) >>> +{ >>> + uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)]; >>> + be32 *cells = ®s[0]; >>> + int i; >>> + va_list ap; >>> + uint64_t arg; >>> + >>> + va_start(ap, num_regs); >>> + for (i = 0 ; i < num_regs; i++) { >>> + /* Set the memory bit field */ >>> + arg = va_arg(ap, uint64_t); >>> + set_cell(&cells, 1, arg); >>> + >>> + /* Set the vpci bus address */ >>> + arg = addr_cells ? va_arg(ap, uint64_t) : 0; >>> + set_cell(&cells, addr_cells , arg); >>> + >>> + /* Set the cpu bus address where vpci address is mapped */ >>> + set_cell(&cells, addr_cells, arg); >>> + >>> + /* Set the vpci size requested */ >>> + arg = size_cells ? va_arg(ap, uint64_t) : 0; >>> + set_cell(&cells, size_cells,arg); >>> + } >>> + va_end(ap); >>> + >>> + return fdt_property(fdt, "ranges", regs, sizeof(regs)); >>> +} >>> + >>> static int make_root_properties(libxl__gc *gc, >>> const libxl_version_info *vers, >>> void *fdt) >>> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, >>> return 0; >>> } >>> +static int make_vpci_node(libxl__gc *gc, void *fdt, >>> + const struct arch_info *ainfo, >>> + struct xc_dom_image *dom) >>> +{ >>> + int res; >>> + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE; >>> + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE; >>> + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base); >>> + >>> + res = fdt_begin_node(fdt, name); >>> + if (res) return res; >>> + >>> + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic"); >>> + if (res) return res; >>> + >>> + res = fdt_property_string(fdt, "device_type", "pci"); >>> + if (res) return res; >>> + >>> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, >>> + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size); >>> + if (res) return res; >>> + >>> + res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17); >> >> AFAICT, the "bus-range" is optional. Can you explain why we need it? > > We need it to implement the function pci_ecam_map_bus(). Ok. Then why next question is what does the 17 mean? Is it tie to how we implement the vPCI in Xen or the region we reserved? [...] >> >>> + >>> if (b_info->type != LIBXL_DOMAIN_TYPE_PV) >>> return; >>> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl >>> index 3f9fff653a..78b1ddf0b8 100644 >>> --- a/tools/libs/light/libxl_types.idl >>> +++ b/tools/libs/light/libxl_types.idl >>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ >>> ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), >>> ("vuart", libxl_vuart_type), >>> + ("vpci", libxl_defbool), >> >> Any new addition in the structure should be accompanied with a LIBXL_HAVE_* in the libxl.h header. > > OK. >> >>> ])), >>> ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), >>> ])), >>> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c >>> index 17dddb4cd5..ffafbeffb4 100644 >>> --- a/tools/xl/xl_parse.c >>> +++ b/tools/xl/xl_parse.c >>> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source, >>> } >>> if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV) >>> libxl_defbool_set(&b_info->u.pv.e820_host, true); >>> + if (d_config->num_pcidevs) >>> + libxl_defbool_set(&b_info->arch_arm.vpci, true); >>> } >>> if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) { >>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h >>> index 0a9749e768..01d13e669e 100644 >>> --- a/xen/include/public/arch-arm.h >>> +++ b/xen/include/public/arch-arm.h >>> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t; >>> #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) >>> #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) >>> +/* PCI-PCIe memory space types */ >>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) >>> +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) >> >> What the size of those regions? > > Non Prefetch Memory: Size 64 MB start at 512 MB > Prefetch Memory: Size 128 GB start at 36 GB >> >>> + >>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ >>> +#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x900000000) >>> +#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x20000000) >> So far the memory layout defines the address in ascending order. So please add that after GUEST_RAM_BANK_BASES_*. > > Ok. >> >> However, if I am not mistaken that base address you provide will clash with RAM bank 1. It also seem to be pretty high which means that this will not work for 32-bit domain or on CPUs that don't allow offer large IPA bits. > > Yes I also checked that now that it is having clash with RAM bank 1. > There is unused space is guest memory that we can use for Non Prefetch Memory as per below guest memory map. > https://gitlab.com/xen-project/fusa/fusa-docs/-/blob/master/high-level/guest-memory-layout-arm.rst > > Proposed value: > Non Prefetch Memory: Size 64 MB start at 0x22001000 > Prefetch Memory: Size 4 GB start at 4 GB. The base address looks fine to me. However, the sizes are much smaller to what you initially suggested. Would you be able to clarify why the smaller sizes are fine? > >> >> I think we need to start making the guest layout more dynamic. The VPCI memory space would have to go right after the end of the RAM allocated for a given guest. >> >>> + >>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x2000000000) >>> +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x04000000) >> >> It would be better if the size for each region is defined right after each base. > OK. > >> >> Also, how did you decide the size of each region? > > I thought 64 MB will be sufficient. I think it should be based on number of devices we can assign to the guest. We don't have to get the size right now. What I am more interested is to have a trace about how those values were decided (even if it just saying random). This will help to make any decision if in the future we need to resize (in particular downsize) the regions. Cheers, -- Julien Grall
[-- Attachment #1: Type: text/plain, Size: 13192 bytes --] On Thu, 19 Aug 2021, Rahul Singh wrote: > XEN during boot will read the PCI device tree node “reg” property > and will map the PCI config space to the XEN memory. > > As of now "pci-host-ecam-generic" compatible board is supported. > > "linux,pci-domain" device tree property assigns a fixed PCI domain > number to a host bridge, otherwise an unstable (across boots) unique > number will be assigned by Linux.This property has to be in sync with > XEN to access the PCI devices. > > XEN will read the “linux,pci-domain” property from the device tree node > and configure the host bridge segment number accordingly. If this > property is not available XEN will allocate the unique segment number > to the host bridge. > > dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly > imported from the Linux source tree. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/Makefile | 2 + > xen/arch/arm/pci/pci-host-common.c | 261 ++++++++++++++++++++++++++++ > xen/arch/arm/pci/pci-host-generic.c | 55 ++++++ > xen/include/asm-arm/pci.h | 28 +++ > 4 files changed, 346 insertions(+) > create mode 100644 xen/arch/arm/pci/pci-host-common.c > create mode 100644 xen/arch/arm/pci/pci-host-generic.c > > diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile > index a9ee0b9b44..f3d97f859e 100644 > --- a/xen/arch/arm/pci/Makefile > +++ b/xen/arch/arm/pci/Makefile > @@ -1,2 +1,4 @@ > obj-y += pci.o > obj-y += pci-access.o > +obj-y += pci-host-generic.o > +obj-y += pci-host-common.o > diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c > new file mode 100644 > index 0000000000..9dd9b02271 > --- /dev/null > +++ b/xen/arch/arm/pci/pci-host-common.c > @@ -0,0 +1,261 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * Based on Linux drivers/pci/ecam.c > + * Copyright 2016 Broadcom. > + * > + * Based on Linux drivers/pci/controller/pci-host-common.c > + * Based on Linux drivers/pci/controller/pci-host-generic.c > + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> > + * > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/init.h> > +#include <xen/pci.h> > +#include <asm/pci.h> > +#include <xen/rwlock.h> > +#include <xen/sched.h> > +#include <xen/vmap.h> > + > +/* > + * List for all the pci host bridges. > + */ > + > +static LIST_HEAD(pci_host_bridges); > + > +static atomic_t domain_nr = ATOMIC_INIT(-1); > + > +bool dt_pci_parse_bus_range(struct dt_device_node *dev, > + struct pci_config_window *cfg) > +{ > + const __be32 *cells; > + uint32_t len; > + > + cells = dt_get_property(dev, "bus-range", &len); > + /* bus-range should at least be 2 cells */ > + if ( !cells || (len < (sizeof(*cells) * 2)) ) > + return false; > + > + cfg->busn_start = dt_next_cell(1, &cells); > + cfg->busn_end = dt_next_cell(1, &cells); > + > + return true; > +} > + > +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len) > +{ > + return ioremap_nocache(start, len); > +} > + > +static void pci_ecam_free(struct pci_config_window *cfg) > +{ > + if ( cfg->win ) > + iounmap(cfg->win); > + > + xfree(cfg); > +} > + > +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > + int ecam_reg_idx) If it is only called at init time, then the function should be __init > +{ > + int err; > + struct pci_config_window *cfg; > + paddr_t addr, size; > + > + cfg = xzalloc(struct pci_config_window); > + if ( !cfg ) > + return NULL; > + > + err = dt_pci_parse_bus_range(dev, cfg); > + if ( !err ) { > + cfg->busn_start = 0; > + cfg->busn_end = 0xff; > + printk(XENLOG_ERR "%s:No bus range found for pci controller\n", > + dt_node_full_name(dev)); > + } else { > + if ( cfg->busn_end > cfg->busn_start + 0xff ) > + cfg->busn_end = cfg->busn_start + 0xff; Is this a hard limit in the specification? Or is it a limit in the Xen implementation? > + } > + > + /* Parse our PCI ecam register address*/ ^ space > + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size); > + if ( err ) > + goto err_exit; > + > + cfg->phys_addr = addr; > + cfg->size = size; > + > + /* > + * On 64-bit systems, we do a single ioremap for the whole config space > + * since we have enough virtual address range available. On 32-bit, we > + * ioremap the config space for each bus individually. > + * > + * As of now only 64-bit is supported 32-bit is not supported. > + */ > + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size); > + if ( !cfg->win ) > + goto err_exit_remap; > + > + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, > + cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); > + > + return cfg; > + > +err_exit_remap: > + printk(XENLOG_ERR "ECAM ioremap failed\n"); > +err_exit: > + pci_ecam_free(cfg); > + return NULL; > +} > + > +struct pci_host_bridge *pci_alloc_host_bridge(void) > +{ > + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); > + > + if ( !bridge ) > + return NULL; > + > + INIT_LIST_HEAD(&bridge->node); > + bridge->bus_start = ~0; > + bridge->bus_end = ~0; Please use INVALID_PADDR instead of ~0 > + return bridge; > +} > + > +void pci_add_host_bridge(struct pci_host_bridge *bridge) > +{ > + list_add_tail(&bridge->node, &pci_host_bridges); > +} > + > +static int pci_get_new_domain_nr(void) > +{ > + return atomic_inc_return(&domain_nr); > +} > + > +/* > + * This function will try to obtain the host bridge domain number by > + * finding a property called "linux,pci-domain" of the given device node. > + * > + * @node: device tree node with the domain information > + * > + * Returns the associated domain number from DT in the range [0-0xffff], or > + * a negative value if the required property is not found. > + */ > +static int dt_get_pci_domain_nr(struct dt_device_node *node) > +{ > + u32 domain; > + int error; > + > + error = dt_property_read_u32(node, "linux,pci-domain", &domain); > + if ( !error ) > + return -EINVAL; > + > + return (u16)domain; Let's check that domain <= UINT16_MAX > +} > + > +static int pci_bus_find_domain_nr(struct dt_device_node *dev) > +{ > + static int use_dt_domains = -1; > + int domain; > + > + domain = dt_get_pci_domain_nr(dev); > + > + /* > + * Check DT domain and use_dt_domains values. > + * > + * If DT domain property is valid (domain >= 0) and > + * use_dt_domains != 0, the DT assignment is valid since this means > + * we have not previously allocated a domain number by using > + * pci_get_new_domain_nr(); we should also update use_dt_domains to > + * 1, to indicate that we have just assigned a domain number from > + * DT. > + * > + * If DT domain property value is not valid (ie domain < 0), and we > + * have not previously assigned a domain number from DT > + * (use_dt_domains != 1) we should assign a domain number by > + * using the: > + * > + * pci_get_new_domain_nr() > + * > + * API and update the use_dt_domains value to keep track of method we > + * are using to assign domain numbers (use_dt_domains = 0). > + * > + * All other combinations imply we have a platform that is trying > + * to mix domain numbers obtained from DT and pci_get_new_domain_nr(), > + * which is a recipe for domain mishandling and it is prevented by > + * invalidating the domain value (domain = -1) and printing a > + * corresponding error. > + */ > + if ( domain >= 0 && use_dt_domains ) > + { > + use_dt_domains = 1; > + } > + else if ( domain < 0 && use_dt_domains != 1 ) > + { > + use_dt_domains = 0; > + domain = pci_get_new_domain_nr(); > + } > + else > + { > + printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n"); > + BUG(); > + } > + > + return domain; > +} > + > +int pci_host_common_probe(struct dt_device_node *dev, > + int ecam_reg_idx) > +{ > + struct pci_host_bridge *bridge; > + struct pci_config_window *cfg; > + int err; > + > + bridge = pci_alloc_host_bridge(); > + if ( !bridge ) > + return -ENOMEM; > + > + /* Parse and map our Configuration Space windows */ > + cfg = gen_pci_init(dev, ecam_reg_idx); > + if ( !cfg ) > + { > + err = -ENOMEM; > + goto err_exit; > + } > + > + bridge->dt_node = dev; > + bridge->sysdata = cfg; > + bridge->bus_start = cfg->busn_start; > + bridge->bus_end = cfg->busn_end; > + > + bridge->segment = pci_bus_find_domain_nr(dev); > + > + pci_add_host_bridge(bridge); > + > + return 0; > + > +err_exit: > + xfree(bridge); > + return err; > +} > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c > new file mode 100644 > index 0000000000..13d0f7f999 > --- /dev/null > +++ b/xen/arch/arm/pci/pci-host-generic.c > @@ -0,0 +1,55 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * Based on Linux drivers/pci/controller/pci-host-common.c > + * Based on Linux drivers/pci/controller/pci-host-generic.c > + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <asm/device.h> > +#include <xen/pci.h> > +#include <asm/pci.h> > + > +static const struct dt_device_match gen_pci_dt_match[] = { > + { .compatible = "pci-host-ecam-generic" }, > + { }, > +}; > + > +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) > +{ > + const struct dt_device_match *of_id; > + > + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); > + > + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", > + dt_node_full_name(dev), of_id->compatible); > + > + return pci_host_common_probe(dev, 0); > +} > + > +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) > +.dt_match = gen_pci_dt_match, > +.init = gen_pci_dt_init, > +DT_DEVICE_END > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index 61e43da088..58a51e724e 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -26,6 +26,34 @@ struct arch_pci_dev { > struct device dev; > }; > > +/* > + * struct to hold the mappings of a config space window. This > + * is expected to be used as sysdata for PCI controllers that > + * use ECAM. > + */ > +struct pci_config_window { > + paddr_t phys_addr; > + paddr_t size; > + uint8_t busn_start; > + uint8_t busn_end; > + void __iomem *win; > +}; > + > +/* > + * struct to hold pci host bridge information > + * for a PCI controller. > + */ > +struct pci_host_bridge { > + struct dt_device_node *dt_node; /* Pointer to the associated DT node */ > + struct list_head node; /* Node in list of host bridges */ > + uint16_t segment; /* Segment number */ > + u8 bus_start; /* Bus start of this bridge. */ > + u8 bus_end; /* Bus end of this bridge. */ > + void *sysdata; /* Pointer to the config space window*/ > +}; > + > +int pci_host_common_probe(struct dt_device_node *dev, > + int ecam_reg_idx); > #else /*!CONFIG_HAS_PCI*/ > > struct arch_pci_dev { }; > -- > 2.17.1 >
On Thu, 19 Aug 2021, Rahul Singh wrote: > Add support for PCI ecam operations to access the PCI > configuration space. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/Makefile | 1 + > xen/arch/arm/pci/ecam.c | 63 +++++++++++++++++++++++++++++ > xen/arch/arm/pci/pci-access.c | 53 ++++++++++++++++++++++++ > xen/arch/arm/pci/pci-host-common.c | 13 +++++- > xen/arch/arm/pci/pci-host-generic.c | 8 +++- > xen/include/asm-arm/pci.h | 32 +++++++++++++++ > 6 files changed, 167 insertions(+), 3 deletions(-) > create mode 100644 xen/arch/arm/pci/ecam.c > > diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile > index f3d97f859e..6f32fbbe67 100644 > --- a/xen/arch/arm/pci/Makefile > +++ b/xen/arch/arm/pci/Makefile > @@ -2,3 +2,4 @@ obj-y += pci.o > obj-y += pci-access.o > obj-y += pci-host-generic.o > obj-y += pci-host-common.o > +obj-y += ecam.o > diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c > new file mode 100644 > index 0000000000..91c691b41f > --- /dev/null > +++ b/xen/arch/arm/pci/ecam.c > @@ -0,0 +1,63 @@ > +/* > + * Copyright (C) 2021 Arm Ltd. > + * > + * Based on Linux drivers/pci/ecam.c > + * Copyright 2016 Broadcom > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/pci.h> > +#include <xen/sched.h> > + > +/* > + * Function to implement the pci_ops ->map_bus method. > + */ > +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, > + uint32_t sbdf, uint32_t where) Code style: alignment > +{ > + const struct pci_config_window *cfg = bridge->sysdata; > + unsigned int devfn_shift = cfg->ops->bus_shift - 8; Is it a guarantee that devfn_shift == bus_shift - 8, or is it just so for ECAM? > + void __iomem *base; > + > + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ; > + unsigned int busn = sbdf_t.bus; > + > + if ( busn < cfg->busn_start || busn > cfg->busn_end ) Genuine question: should it be busn >= cfg->busn_end ? I don't know if the range includes busn_end or not. > + return NULL; > + > + busn -= cfg->busn_start; > + base = cfg->win + (busn << cfg->ops->bus_shift); > + > + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; > +} I understand that the arm32 part is not implemented and not part of this series, that's fine. However if the plan is that arm32 will dynamically map each bus individually, then I imagine this function will have an ioremap in the arm32 version. Which means that we also need an unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus would be a NOP today for arm64, but I think it makes sense to have it if we want the API to be generic. > +/* ECAM ops */ > +const struct pci_ecam_ops pci_generic_ecam_ops = { > + .bus_shift = 20, > + .pci_ops = { > + .map_bus = pci_ecam_map_bus, > + .read = pci_generic_config_read, > + .write = pci_generic_config_write, > + } > +}; > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c > index b938047c03..f39f6a3a38 100644 > --- a/xen/arch/arm/pci/pci-access.c > +++ b/xen/arch/arm/pci/pci-access.c > @@ -15,6 +15,59 @@ > */ > > #include <xen/pci.h> > +#include <asm/io.h> > + > +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t *value) > +{ > + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); > + if (!addr) { > + *value = ~0; Is this a standard error? If so, I think we should define it with a macro (e.g. INVALID_PADDR). > + return -ENODEV; > + } > + > + switch (len) > + { > + case 1: > + *value = readb(addr); > + break; > + case 2: > + *value = readw(addr); > + break; > + case 4: > + *value = readl(addr); > + break; > + default: > + BUG(); A BUG here is harsh because it could be potentially guest-triggered. An ASSERT would be better. > + } > + > + return 0; > +} > + > +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t value) > +{ > + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); > + if (!addr) > + return -ENODEV; > + > + switch (len) > + { > + case 1: > + writeb(value, addr); > + break; > + case 2: > + writew(value, addr); > + break; > + case 4: > + writel(value, addr); > + break; > + default: > + BUG(); Same here > + } > + > + return 0; > +} > > static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, > unsigned int len) > diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c > index 9dd9b02271..c582527e92 100644 > --- a/xen/arch/arm/pci/pci-host-common.c > +++ b/xen/arch/arm/pci/pci-host-common.c > @@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg) > } > > static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > + const struct pci_ecam_ops *ops, > int ecam_reg_idx) > { > int err; > @@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > > cfg->phys_addr = addr; > cfg->size = size; > + cfg->ops = ops; > > /* > * On 64-bit systems, we do a single ioremap for the whole config space > @@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, > cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); > > + if ( ops->init ) > + { > + err = ops->init(cfg); > + if (err) > + goto err_exit; > + } > + > return cfg; > > err_exit_remap: > @@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev) > } > > int pci_host_common_probe(struct dt_device_node *dev, > + const struct pci_ecam_ops *ops, > int ecam_reg_idx) > { > struct pci_host_bridge *bridge; > @@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev, > return -ENOMEM; > > /* Parse and map our Configuration Space windows */ > - cfg = gen_pci_init(dev, ecam_reg_idx); > + cfg = gen_pci_init(dev, ops, ecam_reg_idx); > if ( !cfg ) > { > err = -ENOMEM; > @@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev, > > bridge->dt_node = dev; > bridge->sysdata = cfg; > + bridge->ops = &ops->pci_ops; > bridge->bus_start = cfg->busn_start; > bridge->bus_end = cfg->busn_end; > > diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c > index 13d0f7f999..2d652e8910 100644 > --- a/xen/arch/arm/pci/pci-host-generic.c > +++ b/xen/arch/arm/pci/pci-host-generic.c > @@ -23,20 +23,24 @@ > #include <asm/pci.h> > > static const struct dt_device_match gen_pci_dt_match[] = { > - { .compatible = "pci-host-ecam-generic" }, > + { .compatible = "pci-host-ecam-generic", > + .data = &pci_generic_ecam_ops }, > + > { }, > }; > > static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) > { > const struct dt_device_match *of_id; > + const struct pci_ecam_ops *ops; > > of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); > + ops = (struct pci_ecam_ops *) of_id->data; > > printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", > dt_node_full_name(dev), of_id->compatible); > > - return pci_host_common_probe(dev, 0); > + return pci_host_common_probe(dev, ops, 0); > } > > DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index 58a51e724e..22866244d2 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -37,6 +37,7 @@ struct pci_config_window { > uint8_t busn_start; > uint8_t busn_end; > void __iomem *win; > + const struct pci_ecam_ops *ops; > }; > > /* > @@ -50,10 +51,41 @@ struct pci_host_bridge { > u8 bus_start; /* Bus start of this bridge. */ > u8 bus_end; /* Bus end of this bridge. */ > void *sysdata; /* Pointer to the config space window*/ > + const struct pci_ops *ops; > }; > > +struct pci_ops { > + void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t offset); > + int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t *value); > + int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t value); > +}; > + > +/* > + * struct to hold pci ops and bus shift of the config window > + * for a PCI controller. > + */ > +struct pci_ecam_ops { > + unsigned int bus_shift; > + struct pci_ops pci_ops; > + int (*init)(struct pci_config_window *); Is this last member of the struct needed/used? > +}; > + > +/* Default ECAM ops */ > +extern const struct pci_ecam_ops pci_generic_ecam_ops; > + > int pci_host_common_probe(struct dt_device_node *dev, > + const struct pci_ecam_ops *ops, > int ecam_reg_idx); > +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t *value); > +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, > + uint32_t reg, uint32_t len, uint32_t value); > +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, > + uint32_t sbdf, uint32_t where); > + > #else /*!CONFIG_HAS_PCI*/ > > struct arch_pci_dev { }; > -- > 2.17.1 >
On Thu, 19 Aug 2021, Rahul Singh wrote: > From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > > Add support for Xilinx ZynqMP PCI host controller to map the PCI config > space to the XEN memory. > > Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > --- > xen/arch/arm/pci/Makefile | 1 + > xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++ > 2 files changed, 60 insertions(+) > create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c > > diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile > index 6f32fbbe67..1d045ade01 100644 > --- a/xen/arch/arm/pci/Makefile > +++ b/xen/arch/arm/pci/Makefile > @@ -3,3 +3,4 @@ obj-y += pci-access.o > obj-y += pci-host-generic.o > obj-y += pci-host-common.o > obj-y += ecam.o > +obj-y += pci-host-zynqmp.o > diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c > new file mode 100644 > index 0000000000..fe103e3855 > --- /dev/null > +++ b/xen/arch/arm/pci/pci-host-zynqmp.c > @@ -0,0 +1,59 @@ > +/* > + * Copyright (C) 2020-2021 EPAM Systems > + * > + * Based on Linux drivers/pci/controller/pci-host-common.c > + * Based on Linux drivers/pci/controller/pci-host-generic.c > + * Based on xen/arch/arm/pci/pci-host-generic.c > + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> Only one Copyright line per file is enough :-) But actually all the Copyright lines with a name or a company name are not really required or useful, as the copyright is noted in full details in the commit messages (author and signed-off-by lines). I would remove them all from the new files added by this series. > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <asm/device.h> > +#include <xen/pci.h> > +#include <asm/pci.h> > + > +static const struct dt_device_match gen_pci_dt_match[] = { > + { .compatible = "xlnx,nwl-pcie-2.11", > + .data = &pci_generic_ecam_ops }, > + { }, > +}; > + > +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) > +{ > + const struct dt_device_match *of_id; > + const struct pci_ecam_ops *ops; > + > + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); This should be superfluous > + ops = (struct pci_ecam_ops *) of_id->data; > + > + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", > + dt_node_full_name(dev), of_id->compatible); > + > + return pci_host_common_probe(dev, ops, 2); > +} > + > +DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI) > +.dt_match = gen_pci_dt_match, > +.init = gen_pci_dt_init, > +DT_DEVICE_END > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * tab-width: 4 > + * indent-tabs-mode: nil > + * End: > + */
On Thu, 19 Aug 2021, Rahul Singh wrote: > Implement generic pci access functions to read/write the configuration > space. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/pci-access.c | 31 +++++++++++++++++++++++++++++- > xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++ > xen/include/asm-arm/pci.h | 2 ++ > 3 files changed, 51 insertions(+), 1 deletion(-) > > diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c > index f39f6a3a38..b94de3c3ac 100644 > --- a/xen/arch/arm/pci/pci-access.c > +++ b/xen/arch/arm/pci/pci-access.c > @@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, > static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, > unsigned int len) > { > - return ~0U; > + uint32_t val = GENMASK(0, len * 8); This seems to be another default error value that it would be better to define with its own macro > + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); > + > + if ( unlikely(!bridge) ) > + { > + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", > + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); You are not actually printing sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn ? > + return val; > + } > + > + if ( unlikely(!bridge->ops->read) ) > + return val; > + > + bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val); Would it make sense to make the interface take a pci_sbdf_t directly instead of casting to uint32_t and back? > + return val; > } > > static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, > unsigned int len, uint32_t val) > { > + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); > + > + if ( unlikely(!bridge) ) > + { > + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", > + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); same here > + return; > + } > + > + if ( unlikely(!bridge->ops->write) ) > + return; > + > + bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val); same here > } > > /* > diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c > index c582527e92..62715b4676 100644 > --- a/xen/arch/arm/pci/pci-host-common.c > +++ b/xen/arch/arm/pci/pci-host-common.c > @@ -261,6 +261,25 @@ err_exit: > return err; > } > > +/* > + * This function will lookup an hostbridge based on the segment and bus > + * number. > + */ > +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus) > +{ > + struct pci_host_bridge *bridge; > + > + list_for_each_entry( bridge, &pci_host_bridges, node ) > + { > + if ( bridge->segment != segment ) > + continue; > + if ( (bus < bridge->bus_start) || (bus > bridge->bus_end) ) > + continue; > + return bridge; > + } > + > + return NULL; > +} > /* > * Local variables: > * mode: C > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index 22866244d2..756f8637ab 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -20,6 +20,7 @@ > #ifdef CONFIG_HAS_PCI > > #define pci_to_dev(pcidev) (&(pcidev)->arch.dev) > +#define PRI_pci "%04x:%02x:%02x.%u" > > /* Arch pci dev struct */ > struct arch_pci_dev { > @@ -86,6 +87,7 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, > void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, > uint32_t sbdf, uint32_t where); > > +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus); > #else /*!CONFIG_HAS_PCI*/ > > struct arch_pci_dev { }; > -- > 2.17.1 >
On Fri, 20 Aug 2021, Jan Beulich wrote:
> On 20.08.2021 16:34, Julien Grall wrote:
> > On 20/08/2021 13:19, Rahul Singh wrote:
> >>> On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote:
> >>> On 19/08/2021 13:02, Rahul Singh wrote:
> >>>> Add cmdline boot option "pci=on" to enable/disable the PCI init during
> >>>> boot.
> >>>
> >>> I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not.
> >>
> >> Yes. I will modify the option to "pci-passthrough== <boolean>"
> >>>
> >>> Can you also clarify why a user would want to select "pci=off"?
> >>
> >> As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to
> >> boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device.
> >
> > Dom0 will always have direct access to the PCI device. The only
> > difference is whether the access to the hostbridge and config space will
> > be trapped by Xen. I expect the both to mainly happen during boot and
> > therefore the overhead will be limited.
> >
> >>
> >> I am ok to drop this patch if you feel adding the option is not required at all.
> > One of the reason I could see this option to be useful is to figure out
> > if an issue occurs because of the hostbridge emulation. Yet, I am still
> > not fully convinced adding an option is worth it.
> >
> > Jan and others, any opinions?
>
> Well, if there's a proper fallback, then why not allow using it in
> case of problems?
I think it would be good to have the option, if nothing else for
debugging.
On Thu, 19 Aug 2021, Rahul Singh wrote: > Add cmdline boot option "pci=on" to enable/disable the PCI init during > boot. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/pci/pci.c | 30 ++++++++++++++++++++++++++++++ > 1 file changed, 30 insertions(+) > > diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c > index d1c9cf997d..dc63bbc2a2 100644 > --- a/xen/arch/arm/pci/pci.c > +++ b/xen/arch/arm/pci/pci.c > @@ -62,8 +62,38 @@ static void __init acpi_pci_init(void) > static inline void __init acpi_pci_init(void) { } > #endif > > +static bool __initdata param_pci_enable; > + > +static int __init parse_pci_param(const char *arg) > +{ > + if ( !arg ) > + { > + param_pci_enable = false; > + return 0; > + } > + > + switch ( parse_bool(arg, NULL) ) > + { > + case 0: > + param_pci_enable = false; > + return 0; > + case 1: > + param_pci_enable = true; > + return 0; > + } > + > + return -EINVAL; > +} > +custom_param("pci", parse_pci_param); Consider using boolean_param instead. It supports "on". > static int __init pci_init(void) > { > + /* > + * Enable PCI when has been enabled explicitly (pci=on) > + */ > + if ( !param_pci_enable) > + return 0; > + > if ( acpi_disabled ) > dt_pci_init(); > else > -- > 2.17.1 >
[-- Attachment #1: Type: text/plain, Size: 9937 bytes --] On Thu, 19 Aug 2021, Rahul Singh wrote: > The existing VPCI support available for X86 is adapted for Arm. > When the device is added to XEN via the hyper call > “PHYSDEVOP_pci_device_add”, VPCI handler for the config space > access is added to the Xen to emulate the PCI devices config space. This is done just for device discovery, right? Although it is currently not implemented (and I am not asking to implement it now, I am only trying to understand the architecture), it would be possible to discover all PCI devices just by walking down the PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we have an ECAM driver. I take that would be the way to implement PCI support for Dom0less? > A MMIO trap handler for the PCI ECAM space is registered in XEN > so that when guest is trying to access the PCI config space,XEN > will trap the access and emulate read/write using the VPCI and > not the real PCI hardware. > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > xen/arch/arm/Makefile | 1 + > xen/arch/arm/domain.c | 4 ++ > xen/arch/arm/vpci.c | 96 +++++++++++++++++++++++++++++++++++ > xen/arch/arm/vpci.h | 37 ++++++++++++++ > xen/drivers/passthrough/pci.c | 7 +++ > xen/drivers/vpci/Makefile | 3 +- > xen/drivers/vpci/header.c | 2 + > xen/include/asm-arm/domain.h | 5 +- > xen/include/asm-arm/pci.h | 8 +++ > xen/include/public/arch-arm.h | 4 ++ > 10 files changed, 165 insertions(+), 2 deletions(-) > create mode 100644 xen/arch/arm/vpci.c > create mode 100644 xen/arch/arm/vpci.h > > diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile > index 0e14a5e5c8..7cdce684a4 100644 > --- a/xen/arch/arm/Makefile > +++ b/xen/arch/arm/Makefile > @@ -7,6 +7,7 @@ obj-y += platforms/ > endif > obj-$(CONFIG_TEE) += tee/ > obj-$(CONFIG_HAS_PCI) += pci/ > +obj-$(CONFIG_HAS_VPCI) += vpci.o > > obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o > obj-y += bootfdt.init.o > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c > index 19c756ac3d..d99c653626 100644 > --- a/xen/arch/arm/domain.c > +++ b/xen/arch/arm/domain.c > @@ -40,6 +40,7 @@ > #include <asm/vtimer.h> > > #include "vuart.h" > +#include "vpci.h" > > DEFINE_PER_CPU(struct vcpu *, curr_vcpu); > > @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d, > if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) ) > goto fail; > > + if ( (rc = domain_vpci_init(d)) != 0 ) > + goto fail; > + > return 0; > > fail: > diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c > new file mode 100644 > index 0000000000..da8b1ca13c > --- /dev/null > +++ b/xen/arch/arm/vpci.c > @@ -0,0 +1,96 @@ > +/* > + * xen/arch/arm/vpci.c > + * Copyright (c) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + */ > +#include <xen/sched.h> > +#include <asm/mmio.h> > + > +/* Do some sanity checks. */ > +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len) > +{ > + /* Check access size. */ > + if ( len != 1 && len != 2 && len != 4 && len != 8 ) > + return false; > + > + /* Check that access is size aligned. */ > + if ( (reg & (len - 1)) ) > + return false; > + > + return true; > +} > + > +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, > + register_t *r, void *p) > +{ > + unsigned int reg; > + pci_sbdf_t sbdf; > + uint32_t data = 0; > + unsigned int size = 1U << info->dabt.size; > + > + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); > + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); > + > + if ( !vpci_mmio_access_allowed(reg, size) ) > + return 1; > + > + data = vpci_read(sbdf, reg, size); > + > + memcpy(r, &data, size); > + > + return 1; > +} > + > +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, > + register_t r, void *p) > +{ > + unsigned int reg; > + pci_sbdf_t sbdf; > + uint32_t data = r; > + unsigned int size = 1U << info->dabt.size; > + > + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); > + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); > + > + if ( !vpci_mmio_access_allowed(reg, size) ) > + return 1; > + > + vpci_write(sbdf, reg, size, data); > + > + return 1; > +} > + > +static const struct mmio_handler_ops vpci_mmio_handler = { > + .read = vpci_mmio_read, > + .write = vpci_mmio_write, > +}; > + > +int domain_vpci_init(struct domain *d) > +{ > + if ( !has_vpci(d) ) > + return 0; > + > + register_mmio_handler(d, &vpci_mmio_handler, > + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); > + > + return 0; > +} > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * indent-tabs-mode: nil > + * End: > + */ > + > diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h > new file mode 100644 > index 0000000000..8a093bb705 > --- /dev/null > +++ b/xen/arch/arm/vpci.h > @@ -0,0 +1,37 @@ > +/* > + * xen/arch/arm/vpci.h > + * Copyright (c) 2021 Arm Ltd. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; either version 2 of the License, or > + * (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + */ > + > +#ifndef __ARCH_ARM_VPCI_H__ > +#define __ARCH_ARM_VPCI_H__ > + > +#ifdef CONFIG_HAS_VPCI > +int domain_vpci_init(struct domain *d); > +#else > +static inline int domain_vpci_init(struct domain *d) > +{ > + return 0; > +} > +#endif > + > +#endif /* __ARCH_ARM_VPCI_H__ */ > + > +/* > + * Local variables: > + * mode: C > + * c-file-style: "BSD" > + * c-basic-offset: 4 > + * indent-tabs-mode: nil > + * End: > + */ > diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c > index c23c8cb06b..56e261e9bd 100644 > --- a/xen/drivers/passthrough/pci.c > +++ b/xen/drivers/passthrough/pci.c > @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, > else > iommu_enable_device(pdev); > > +#ifdef CONFIG_ARM > + ret = vpci_add_handlers(pdev); > + if ( ret ) { > + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret); > + goto out; > + } > +#endif > pci_enable_acs(pdev); > > out: > diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile > index 55d1bdfda0..1a1413b93e 100644 > --- a/xen/drivers/vpci/Makefile > +++ b/xen/drivers/vpci/Makefile > @@ -1 +1,2 @@ > -obj-y += vpci.o header.o msi.o msix.o > +obj-y += vpci.o header.o > +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o > diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c > index ba9a036202..f8cd55e7c0 100644 > --- a/xen/drivers/vpci/header.c > +++ b/xen/drivers/vpci/header.c > @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, > * FIXME: punching holes after the p2m has been set up might be racy for > * DomU usage, needs to be revisited. > */ > +#ifdef CONFIG_HAS_PCI_MSI > if ( map && !rom_only && vpci_make_msix_hole(pdev) ) > return; > +#endif > > for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) > { > diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h > index c9277b5c6d..d742b94bd6 100644 > --- a/xen/include/asm-arm/domain.h > +++ b/xen/include/asm-arm/domain.h > @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {} > > #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag) > > -#define has_vpci(d) ({ (void)(d); false; }) > +/* For X86 VPCI is enabled and tested for PVH DOM0 only but > + * for ARM we enable support VPCI for guest domain also. > + */ > +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); }) > > #endif /* __ASM_DOMAIN_H__ */ > > diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h > index 756f8637ab..c58152de80 100644 > --- a/xen/include/asm-arm/pci.h > +++ b/xen/include/asm-arm/pci.h > @@ -27,6 +27,14 @@ struct arch_pci_dev { > struct device dev; > }; > > +/* Arch-specific MSI data for vPCI. */ > +struct vpci_arch_msi { > +}; > + > +/* Arch-specific MSI-X entry data for vPCI. */ > +struct vpci_arch_msix_entry { > +}; > + > /* > * struct to hold the mappings of a config space window. This > * is expected to be used as sysdata for PCI controllers that > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h > index 64a2ca30da..0a9749e768 100644 > --- a/xen/include/public/arch-arm.h > +++ b/xen/include/public/arch-arm.h > @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t; > #define GUEST_PL011_BASE xen_mk_ullong(0x22000000) > #define GUEST_PL011_SIZE xen_mk_ullong(0x00001000) > > +/* VPCI ECAM mappings */ > +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) > +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) Isn't 256MB a bit too small? Usually PCI comes with two ranges: - a small range below 4GB like this one - an additional large range above 4GB, around 1GB or larger We are missing the second larger range?
On Thu, 19 Aug 2021, Rahul Singh wrote: > libxl will create an emulated PCI device tree node in the device tree to > enable the guest OS to discover the virtual PCI during guest boot. > Emulated PCI device tree node will only be created when there is any > device assigned to guest. > > A new area has been reserved in the arm guest physical map at > which the VPCI bus is declared in the device tree (reg and ranges > parameters of the node). > > Signed-off-by: Rahul Singh <rahul.singh@arm.com> > --- > tools/libs/light/libxl_arm.c | 109 ++++++++++++++++++++++++++ > tools/libs/light/libxl_types.idl | 1 + > tools/xl/xl_parse.c | 2 + > xen/include/public/arch-arm.h | 11 +++ > xen/include/public/device_tree_defs.h | 1 + > 5 files changed, 124 insertions(+) > > diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c > index e3140a6e00..a091e97e76 100644 > --- a/tools/libs/light/libxl_arm.c > +++ b/tools/libs/light/libxl_arm.c > @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt, > return fdt_property(fdt, "reg", regs, sizeof(regs)); > } > > +static int fdt_property_values(libxl__gc *gc, void *fdt, > + const char *name, unsigned num_cells, ...) > +{ > + uint32_t prop[num_cells]; > + be32 *cells = &prop[0]; > + int i; > + va_list ap; > + uint32_t arg; > + > + va_start(ap, num_cells); > + for (i = 0 ; i < num_cells; i++) { > + arg = va_arg(ap, uint32_t); > + set_cell(&cells, 1, arg); > + } > + va_end(ap); > + > + return fdt_property(fdt, name, prop, sizeof(prop)); > +} > + > +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt, > + unsigned addr_cells, > + unsigned size_cells, > + unsigned num_regs, ...) > +{ > + uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)]; > + be32 *cells = ®s[0]; > + int i; > + va_list ap; > + uint64_t arg; > + > + va_start(ap, num_regs); > + for (i = 0 ; i < num_regs; i++) { > + /* Set the memory bit field */ > + arg = va_arg(ap, uint64_t); > + set_cell(&cells, 1, arg); Shouldn't this be uint32_t given that it is 1 cell exactly? > + /* Set the vpci bus address */ > + arg = addr_cells ? va_arg(ap, uint64_t) : 0; > + set_cell(&cells, addr_cells , arg); > + > + /* Set the cpu bus address where vpci address is mapped */ > + set_cell(&cells, addr_cells, arg); > + > + /* Set the vpci size requested */ > + arg = size_cells ? va_arg(ap, uint64_t) : 0; > + set_cell(&cells, size_cells,arg); ^ space > + } > + va_end(ap); > + > + return fdt_property(fdt, "ranges", regs, sizeof(regs)); > +} > + > static int make_root_properties(libxl__gc *gc, > const libxl_version_info *vers, > void *fdt) > @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, > return 0; > } > > +static int make_vpci_node(libxl__gc *gc, void *fdt, > + const struct arch_info *ainfo, > + struct xc_dom_image *dom) > +{ > + int res; > + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE; > + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE; > + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base); > + > + res = fdt_begin_node(fdt, name); > + if (res) return res; > + > + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic"); > + if (res) return res; > + > + res = fdt_property_string(fdt, "device_type", "pci"); > + if (res) return res; > + > + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, > + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size); > + if (res) return res; > + > + res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17); ^ space > + if (res) return res; > + > + res = fdt_property_cell(fdt, "#address-cells", 3); > + if (res) return res; > + > + res = fdt_property_cell(fdt, "#size-cells", 2); > + if (res) return res; > + > + res = fdt_property_string(fdt, "status", "okay"); > + if (res) return res; > + > + res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, > + GUEST_ROOT_SIZE_CELLS, 2, > + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE, > + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR, > + GUEST_VPCI_PREFETCH_MEM_SIZE); > + if (res) return res; > + > + res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS, > + 0, 0x10000); > + if (res) return res; I agree with Julien that we shouldn't add it now if it is not working. One question: what about legacy interrupts? If they are supported, shouldn't we have interrupts and interrupt-parent properties? > + res = fdt_end_node(fdt); > + if (res) return res; > + > + return 0; > +} > + > static const struct arch_info *get_arch_info(libxl__gc *gc, > const struct xc_dom_image *dom) > { > @@ -971,6 +1074,9 @@ next_resize: > if (info->tee == LIBXL_TEE_TYPE_OPTEE) > FDT( make_optee_node(gc, fdt) ); > > + if (libxl_defbool_val(info->arch_arm.vpci)) > + FDT( make_vpci_node(gc, fdt, ainfo, dom) ); > + > if (pfdt) > FDT( copy_partial_fdt(gc, fdt, pfdt) ); > > @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, > /* ACPI is disabled by default */ > libxl_defbool_setdefault(&b_info->acpi, false); > > + /* VPCI is disabled by default */ > + libxl_defbool_setdefault(&b_info->arch_arm.vpci, false); > + > if (b_info->type != LIBXL_DOMAIN_TYPE_PV) > return; > > diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl > index 3f9fff653a..78b1ddf0b8 100644 > --- a/tools/libs/light/libxl_types.idl > +++ b/tools/libs/light/libxl_types.idl > @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ > > ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), > ("vuart", libxl_vuart_type), > + ("vpci", libxl_defbool), > ])), > ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), > ])), > diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c > index 17dddb4cd5..ffafbeffb4 100644 > --- a/tools/xl/xl_parse.c > +++ b/tools/xl/xl_parse.c > @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source, > } > if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV) > libxl_defbool_set(&b_info->u.pv.e820_host, true); > + if (d_config->num_pcidevs) > + libxl_defbool_set(&b_info->arch_arm.vpci, true); > } > > if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) { > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h > index 0a9749e768..01d13e669e 100644 > --- a/xen/include/public/arch-arm.h > +++ b/xen/include/public/arch-arm.h > @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t; > #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) > #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) > > +/* PCI-PCIe memory space types */ > +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) > +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) > + > +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ > +#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x900000000) > +#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x20000000) > + > +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x2000000000) > +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x04000000) Now I get it: GUEST_VPCI_ECAM_BASE was just for the config space, not the aperture. The apertures are starting at 0x20000000 and 0x900000000. Speaking of which, isn't GUEST_VPCI_MEM_SIZE a bit too small? There could be PCI devices with a <4GB requirement. Maybe 256MB?
On Thu, 19 Aug 2021, Rahul Singh wrote:
> If the property is not present in the device tree node for host bridge,
> XEN while creating the dtb for hwdom will create this property and
> assigns the already allocated segment to the host bridge
> so that XEN and linux will have the same segment for the host bridges.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
> xen/arch/arm/domain_build.c | 18 ++++++++++++++++++
> xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++
> xen/include/asm-arm/pci.h | 3 +++
> 3 files changed, 42 insertions(+)
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 6c86d52781..e0cf2ff19d 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -581,6 +581,24 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
> return res;
> }
>
> +#ifdef CONFIG_HAS_PCI
> + if ( dt_device_type_is_equal(node, "pci") )
> + {
> + if ( !dt_find_property(node, "linux,pci-domain", NULL) )
> + {
> + uint16_t segment;
> +
> + res = pci_get_host_bridge_segment(node, &segment);
> + if ( res < 0 )
> + return res;
> +
> + res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment);
> + if ( res )
> + return res;
> + }
> + }
> +#endif
If param_pci_enable is false it might be possible that Xen didn't
allocate a segment. In that case, we should just let Linux do whatever
it wants in terms of segment allocation. So I think the code here should
not return error if param_pci_enable is false.
returning an error instead.
Hi Stefano, > On 9 Sep 2021, at 2:11 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Fri, 20 Aug 2021, Jan Beulich wrote: >> On 20.08.2021 13:41, Rahul Singh wrote: >>> Hi Jan >>> >>>> On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote: >>>> >>>> On 19.08.2021 14:02, Rahul Singh wrote: >>>>> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM >>>>> architecture. Move the code under CONFIG_X86 flag to gate the code for >>>>> ARM. >>>> >>>> Please give at least one example of what it is that fails to compile. >>>> What an appropriate action is depends on the nature of the error(s), >>>> and from looking at the enclosed code I cannot easily see what it >>>> might be that breaks for Arm. >>> >>> I am observing below error when enabled ACPI && HAS_PCI for ARM. >>> >>> prelink.o: In function `pcie_aer_get_firmware_first’: >>> /xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse' >>> aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined >>> >>> I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code. >>> obj-$(CONFIG_X86) += apei/ >>> >>> I am not sure whether we need this code for ARM architecture >>> that is why I gate the code for ARM via CONFIG_X86 >> >> So you Arm folks will probably want to settle on that aspect first. What >> is wanted to keep things building depends on that. > > Reading the APEI description, it looks like there might be some use for > it on ARM but it would work a bit differently from x86 as there are no > NMIs on ARM. So enabling APEI on ARM is not just a matter of enabling > the build of apei/, it is not going to be straightforward. > > For the scope of this series (which is actually about PCI), I would > leave it alone, and keep apei/ x86 only, which means #ifdefing > pcie_aer_get_firmware_first. > > I would just add an in-code comment saying "APEI not supported on ARM > yet”. Ok. I will add the comment in next version. > Another option would be to introduce a symbol like HAS_ACPI_APEI > but it is a bit overkill for this. Regards, Rahul
Hi Stefano,
> On 9 Sep 2021, at 2:16 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> pci_init(..) will be called during xen startup to initialize and probe
>> the PCI host-bridge driver.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/pci/pci.c | 54 ++++++++++++++++++++++++++++++++++++
>> xen/include/asm-arm/device.h | 1 +
>> 2 files changed, 55 insertions(+)
>>
>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>> index dc55d23778..d1c9cf997d 100644
>> --- a/xen/arch/arm/pci/pci.c
>> +++ b/xen/arch/arm/pci/pci.c
>> @@ -14,13 +14,67 @@
>> * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> */
>>
>> +#include <xen/acpi.h>
>> +#include <xen/device_tree.h>
>> +#include <xen/errno.h>
>> +#include <xen/init.h>
>> #include <xen/pci.h>
>> +#include <xen/param.h>
>>
>> int arch_pci_clean_pirqs(struct domain *d)
>> {
>> return 0;
>> }
>>
>> +static int __init dt_pci_init(void)
>> +{
>> + struct dt_device_node *np;
>> + int rc;
>> +
>> + dt_for_each_device_node(dt_host, np)
>> + {
>> + rc = device_init(np, DEVICE_PCI, NULL);
>> + if( !rc )
>> + continue;
>> + /*
>> + * Ignore the following error codes:
>> + * - EBADF: Indicate the current is not an pci
> ^ a
>
>> + * - ENODEV: The pci device is not present or cannot be used by
>> + * Xen.
>> + */
>> + else if ( rc != -EBADF && rc != -ENODEV )
>> + {
>> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
>> + return rc;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +#ifdef CONFIG_ACPI
>> +static void __init acpi_pci_init(void)
>
> If the DT init function returns int, then it would make sense for the
> ACPI init function to return int as well?
Ok. I will modify acpi_pci_init(..) function to return int.
Regards,
Rahul
Hi Julien, > On 7 Sep 2021, at 9:20 am, Julien Grall <julien@xen.org> wrote: > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> pci_init(..) will be called during xen startup to initialize and probe >> the PCI host-bridge driver. >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/pci/pci.c | 54 ++++++++++++++++++++++++++++++++++++ >> xen/include/asm-arm/device.h | 1 + >> 2 files changed, 55 insertions(+) >> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c >> index dc55d23778..d1c9cf997d 100644 >> --- a/xen/arch/arm/pci/pci.c >> +++ b/xen/arch/arm/pci/pci.c >> @@ -14,13 +14,67 @@ >> * along with this program. If not, see <http://www.gnu.org/licenses/>. >> */ >> +#include <xen/acpi.h> >> +#include <xen/device_tree.h> >> +#include <xen/errno.h> >> +#include <xen/init.h> >> #include <xen/pci.h> >> +#include <xen/param.h> > > This include doesn't look to be necessary (yet?). Yes you are right this is not necessary .I will remove "param.h" > >> int arch_pci_clean_pirqs(struct domain *d) >> { >> return 0; >> } >> +static int __init dt_pci_init(void) >> +{ >> + struct dt_device_node *np; >> + int rc; >> + >> + dt_for_each_device_node(dt_host, np) >> + { >> + rc = device_init(np, DEVICE_PCI, NULL); >> + if( !rc ) >> + continue; >> + /* >> + * Ignore the following error codes: >> + * - EBADF: Indicate the current is not an pci >> + * - ENODEV: The pci device is not present or cannot be used by >> + * Xen. >> + */ >> + else if ( rc != -EBADF && rc != -ENODEV ) >> + { >> + printk(XENLOG_ERR "No driver found in XEN or driver init error.\n"); >> + return rc; >> + } >> + } >> + >> + return 0; >> +} >> + >> +#ifdef CONFIG_ACPI >> +static void __init acpi_pci_init(void) >> +{ >> + printk(XENLOG_ERR "ACPI pci init not supported \n"); >> + return; >> +} >> +#else >> +static inline void __init acpi_pci_init(void) { } >> +#endif >> + >> +static int __init pci_init(void) >> +{ >> + if ( acpi_disabled ) >> + dt_pci_init(); >> + else >> + acpi_pci_init(); >> + >> + pci_segments_init(); > > Shouldn't this happen before the PCI initialization? Calling pci_segment_init(..) before or after PCI initialization will not make any difference as this is independent call. Anyway I will move the pci_segment_init(..) before PCI initialization. Regards, Rahul
Hi Julien > On 7 Sep 2021, at 10:05 am, Julien Grall <julien@xen.org> wrote: > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> XEN during boot will read the PCI device tree node “reg” property >> and will map the PCI config space to the XEN memory. >> As of now "pci-host-ecam-generic" compatible board is supported. > > I think the word "only" is missing. Ok. > >> "linux,pci-domain" device tree property assigns a fixed PCI domain >> number to a host bridge, otherwise an unstable (across boots) unique >> number will be assigned by Linux.This property has to be in sync with > > Typo: missing space after the ‘.’ Ack. >> XEN to access the PCI devices. > > I would expand a little bit the last sentence to explain why the need to be sync-ed. I will explain why segment and domain need to be in sync in next version. > >> > XEN will read the “linux,pci-domain” property from the device tree node >> and configure the host bridge segment number accordingly. If this >> property is not available XEN will allocate the unique segment number >> to the host bridge. >> dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly >> imported from the Linux source tree. > > What was the Linux commit used? I also read "directly imported" as a verbartim copy but AFAICT the implementation has been slightly reworked. I will add the Linux commit used in commit msg and also will add that code is slightly modified. > >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/pci/Makefile | 2 + >> xen/arch/arm/pci/pci-host-common.c | 261 ++++++++++++++++++++++++++++ >> xen/arch/arm/pci/pci-host-generic.c | 55 ++++++ >> xen/include/asm-arm/pci.h | 28 +++ >> 4 files changed, 346 insertions(+) >> create mode 100644 xen/arch/arm/pci/pci-host-common.c >> create mode 100644 xen/arch/arm/pci/pci-host-generic.c >> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >> index a9ee0b9b44..f3d97f859e 100644 >> --- a/xen/arch/arm/pci/Makefile >> +++ b/xen/arch/arm/pci/Makefile >> @@ -1,2 +1,4 @@ >> obj-y += pci.o >> obj-y += pci-access.o >> +obj-y += pci-host-generic.o >> +obj-y += pci-host-common.o >> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c >> new file mode 100644 >> index 0000000000..9dd9b02271 >> --- /dev/null >> +++ b/xen/arch/arm/pci/pci-host-common.c >> @@ -0,0 +1,261 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * Based on Linux drivers/pci/ecam.c >> + * Copyright 2016 Broadcom. >> + * >> + * Based on Linux drivers/pci/controller/pci-host-common.c >> + * Based on Linux drivers/pci/controller/pci-host-generic.c >> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> >> + * >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/init.h> >> +#include <xen/pci.h> >> +#include <asm/pci.h> > > AFAICT, <xen/pci.h> already includes <asm/pci.h>. So this looks unneccessary. Ack. > >> +#include <xen/rwlock.h> >> +#include <xen/sched.h> >> +#include <xen/vmap.h> >> + >> +/* >> + * List for all the pci host bridges. >> + */ >> + >> +static LIST_HEAD(pci_host_bridges); >> + >> +static atomic_t domain_nr = ATOMIC_INIT(-1); >> + >> +bool dt_pci_parse_bus_range(struct dt_device_node *dev, >> + struct pci_config_window *cfg) > > Aside, "pci_config_window", the function is not Arm specific. Would it be possible to consider to introduce "struct resource" in Xen so this function can be moved in common/device_tree.c? I can introduce the "struct resource” but I am not sure whether "struct resource” will be useful later point in time. What I prefer as of now, we can have this function and we can move this to common/device_tree.c once we have the requirement for "struct resource”. > >> +{ >> + const __be32 *cells; >> + uint32_t len; >> + >> + cells = dt_get_property(dev, "bus-range", &len); >> + /* bus-range should at least be 2 cells */ >> + if ( !cells || (len < (sizeof(*cells) * 2)) ) >> + return false; > > How about introducing dt_property_read_u32_array()? Ok. I will introduce dt_property_read_u32_array(). > >> + >> + cfg->busn_start = dt_next_cell(1, &cells); >> + cfg->busn_end = dt_next_cell(1, &cells); >> + >> + return true; >> +} >> + >> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len) >> +{ >> + return ioremap_nocache(start, len); >> +} >> + >> +static void pci_ecam_free(struct pci_config_window *cfg) >> +{ >> + if ( cfg->win ) >> + iounmap(cfg->win); >> + >> + xfree(cfg); >> +} >> + >> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, >> + int ecam_reg_idx) >> +{ >> + int err; >> + struct pci_config_window *cfg; >> + paddr_t addr, size; >> + >> + cfg = xzalloc(struct pci_config_window); >> + if ( !cfg ) >> + return NULL; >> + >> + err = dt_pci_parse_bus_range(dev, cfg); >> + if ( !err ) { >> + cfg->busn_start = 0; >> + cfg->busn_end = 0xff; >> + printk(XENLOG_ERR "%s:No bus range found for pci controller\n", > > Typo: Missing space after ':’. Ack. > >> + dt_node_full_name(dev)); >> + } else { >> + if ( cfg->busn_end > cfg->busn_start + 0xff ) >> + cfg->busn_end = cfg->busn_start + 0xff; >> + } >> + >> + /* Parse our PCI ecam register address*/ >> + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size); >> + if ( err ) >> + goto err_exit; >> + >> + cfg->phys_addr = addr; >> + cfg->size = size; >> + >> + /* >> + * On 64-bit systems, we do a single ioremap for the whole config space >> + * since we have enough virtual address range available. On 32-bit, we > > In Xen on Arm64, the VMAP is actually only 1GB. So it is not that big and this will compete with other mapping like ITS, global domain mapping... > > So I think the vMAP area will need to increase to cater to increase usage. Let me check on this and come back to you. > >> + * ioremap the config space for each bus individually. >> + * >> + * As of now only 64-bit is supported 32-bit is not supported. >> + */ >> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size); >> + if ( !cfg->win ) >> + goto err_exit_remap; >> + >> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, > > The physical address is a paddr_t. So this needs to use PRIpaddr. Also, please use preprent hexadecimal with 0x. This makes a lot easier to differentiate hexa vs decimal in the log. Ack. > >> + cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); >> + >> + return cfg; >> + >> +err_exit_remap: >> + printk(XENLOG_ERR "ECAM ioremap failed\n"); >> +err_exit: >> + pci_ecam_free(cfg); > > Coding style: Please add a new line before return. Ack. > >> + return NULL; >> +} >> + >> +struct pci_host_bridge *pci_alloc_host_bridge(void) >> +{ >> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); >> + >> + if ( !bridge ) >> + return NULL; >> + >> + INIT_LIST_HEAD(&bridge->node); >> + bridge->bus_start = ~0; >> + bridge->bus_end = ~0; > > Coding style: Please add a new line before return. Ack. > > >> + return bridge; >> +} >> + >> +void pci_add_host_bridge(struct pci_host_bridge *bridge) >> +{ >> + list_add_tail(&bridge->node, &pci_host_bridges); >> +} >> + >> +static int pci_get_new_domain_nr(void) >> +{ >> + return atomic_inc_return(&domain_nr); >> +} >> + >> +/* >> + * This function will try to obtain the host bridge domain number by >> + * finding a property called "linux,pci-domain" of the given device node. >> + * >> + * @node: device tree node with the domain information >> + * >> + * Returns the associated domain number from DT in the range [0-0xffff], or >> + * a negative value if the required property is not found. >> + */ >> +static int dt_get_pci_domain_nr(struct dt_device_node *node) > > Nothing looks Arm specific for this function. Can you move it in device_tree.c? Ack. > >> +{ >> + u32 domain; >> + int error; >> + >> + error = dt_property_read_u32(node, "linux,pci-domain", &domain); >> + if ( !error ) >> + return -EINVAL; >> + >> + return (u16)domain; >> +} >> + >> +static int pci_bus_find_domain_nr(struct dt_device_node *dev) >> +{ >> + static int use_dt_domains = -1; >> + int domain; >> + >> + domain = dt_get_pci_domain_nr(dev); >> + >> + /* >> + * Check DT domain and use_dt_domains values. >> + * >> + * If DT domain property is valid (domain >= 0) and >> + * use_dt_domains != 0, the DT assignment is valid since this means >> + * we have not previously allocated a domain number by using >> + * pci_get_new_domain_nr(); we should also update use_dt_domains to >> + * 1, to indicate that we have just assigned a domain number from >> + * DT. >> + * >> + * If DT domain property value is not valid (ie domain < 0), and we >> + * have not previously assigned a domain number from DT >> + * (use_dt_domains != 1) we should assign a domain number by >> + * using the: >> + * >> + * pci_get_new_domain_nr() >> + * >> + * API and update the use_dt_domains value to keep track of method we >> + * are using to assign domain numbers (use_dt_domains = 0). >> + * >> + * All other combinations imply we have a platform that is trying >> + * to mix domain numbers obtained from DT and pci_get_new_domain_nr(), >> + * which is a recipe for domain mishandling and it is prevented by >> + * invalidating the domain value (domain = -1) and printing a >> + * corresponding error. >> + */ >> + if ( domain >= 0 && use_dt_domains ) >> + { >> + use_dt_domains = 1; >> + } >> + else if ( domain < 0 && use_dt_domains != 1 ) >> + { >> + use_dt_domains = 0; >> + domain = pci_get_new_domain_nr(); >> + } >> + else >> + { >> + printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n"); >> + BUG(); > > I don't think pci_bus_find_domain_nr() should be the function that crashes Xen. Instead, this should be propagated to the highest possible callers and let it decide what to do. Ack. > >> + } >> + >> + return domain; >> +} >> + >> +int pci_host_common_probe(struct dt_device_node *dev, >> + int ecam_reg_idx) >> +{ >> + struct pci_host_bridge *bridge; >> + struct pci_config_window *cfg; >> + int err; >> + >> + bridge = pci_alloc_host_bridge(); >> + if ( !bridge ) >> + return -ENOMEM; >> + >> + /* Parse and map our Configuration Space windows */ >> + cfg = gen_pci_init(dev, ecam_reg_idx); >> + if ( !cfg ) >> + { >> + err = -ENOMEM; >> + goto err_exit; >> + } >> + >> + bridge->dt_node = dev; >> + bridge->sysdata = cfg; >> + bridge->bus_start = cfg->busn_start; >> + bridge->bus_end = cfg->busn_end; >> + >> + bridge->segment = pci_bus_find_domain_nr(dev); >> + >> + pci_add_host_bridge(bridge); >> + >> + return 0; >> + >> +err_exit: >> + xfree(bridge); >> + return err; >> +} >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * tab-width: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c >> new file mode 100644 >> index 0000000000..13d0f7f999 >> --- /dev/null >> +++ b/xen/arch/arm/pci/pci-host-generic.c >> @@ -0,0 +1,55 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * Based on Linux drivers/pci/controller/pci-host-common.c >> + * Based on Linux drivers/pci/controller/pci-host-generic.c >> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <asm/device.h> >> +#include <xen/pci.h> >> +#include <asm/pci.h> >> + >> +static const struct dt_device_match gen_pci_dt_match[] = { >> + { .compatible = "pci-host-ecam-generic" }, >> + { }, >> +}; >> + >> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) >> +{ >> + const struct dt_device_match *of_id; >> + >> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); > > This seems to be a bit pointless to me as you already know the compatible (there is only one possible...). As of now we are only implementing the "pci-host-ecam-generic” compatible PCI, but in future we might need to implement the other compatible like Linux see as below. https://elixir.bootlin.com/linux/latest/source/drivers/pci/controller/pci-host-generic.c#L59 > >> + >> + printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", >> + dt_node_full_name(dev), of_id->compatible); >> + >> + return pci_host_common_probe(dev, 0); >> +} >> + >> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) >> +.dt_match = gen_pci_dt_match, >> +.init = gen_pci_dt_init, >> +DT_DEVICE_END >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * tab-width: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h >> index 61e43da088..58a51e724e 100644 >> --- a/xen/include/asm-arm/pci.h >> +++ b/xen/include/asm-arm/pci.h >> @@ -26,6 +26,34 @@ struct arch_pci_dev { >> struct device dev; >> }; >> +/* >> + * struct to hold the mappings of a config space window. This >> + * is expected to be used as sysdata for PCI controllers that >> + * use ECAM. >> + */ >> +struct pci_config_window { >> + paddr_t phys_addr; >> + paddr_t size; >> + uint8_t busn_start; >> + uint8_t busn_end; >> + void __iomem *win; >> +}; >> + >> +/* >> + * struct to hold pci host bridge information >> + * for a PCI controller. >> + */ >> +struct pci_host_bridge { >> + struct dt_device_node *dt_node; /* Pointer to the associated DT node */ >> + struct list_head node; /* Node in list of host bridges */ >> + uint16_t segment; /* Segment number */ >> + u8 bus_start; /* Bus start of this bridge. */ >> + u8 bus_end; /* Bus end of this bridge. */ > > Please use uint8_t rather than u8. Ack. Regards, Rahul
Hi Stefano, > On 9 Sep 2021, at 11:54 pm, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Thu, 19 Aug 2021, Rahul Singh wrote: >> XEN during boot will read the PCI device tree node “reg” property >> and will map the PCI config space to the XEN memory. >> >> As of now "pci-host-ecam-generic" compatible board is supported. >> >> "linux,pci-domain" device tree property assigns a fixed PCI domain >> number to a host bridge, otherwise an unstable (across boots) unique >> number will be assigned by Linux.This property has to be in sync with >> XEN to access the PCI devices. >> >> XEN will read the “linux,pci-domain” property from the device tree node >> and configure the host bridge segment number accordingly. If this >> property is not available XEN will allocate the unique segment number >> to the host bridge. >> >> dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly >> imported from the Linux source tree. >> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/pci/Makefile | 2 + >> xen/arch/arm/pci/pci-host-common.c | 261 ++++++++++++++++++++++++++++ >> xen/arch/arm/pci/pci-host-generic.c | 55 ++++++ >> xen/include/asm-arm/pci.h | 28 +++ >> 4 files changed, 346 insertions(+) >> create mode 100644 xen/arch/arm/pci/pci-host-common.c >> create mode 100644 xen/arch/arm/pci/pci-host-generic.c >> >> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >> index a9ee0b9b44..f3d97f859e 100644 >> --- a/xen/arch/arm/pci/Makefile >> +++ b/xen/arch/arm/pci/Makefile >> @@ -1,2 +1,4 @@ >> obj-y += pci.o >> obj-y += pci-access.o >> +obj-y += pci-host-generic.o >> +obj-y += pci-host-common.o >> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c >> new file mode 100644 >> index 0000000000..9dd9b02271 >> --- /dev/null >> +++ b/xen/arch/arm/pci/pci-host-common.c >> @@ -0,0 +1,261 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * Based on Linux drivers/pci/ecam.c >> + * Copyright 2016 Broadcom. >> + * >> + * Based on Linux drivers/pci/controller/pci-host-common.c >> + * Based on Linux drivers/pci/controller/pci-host-generic.c >> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> >> + * >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/init.h> >> +#include <xen/pci.h> >> +#include <asm/pci.h> >> +#include <xen/rwlock.h> >> +#include <xen/sched.h> >> +#include <xen/vmap.h> >> + >> +/* >> + * List for all the pci host bridges. >> + */ >> + >> +static LIST_HEAD(pci_host_bridges); >> + >> +static atomic_t domain_nr = ATOMIC_INIT(-1); >> + >> +bool dt_pci_parse_bus_range(struct dt_device_node *dev, >> + struct pci_config_window *cfg) >> +{ >> + const __be32 *cells; >> + uint32_t len; >> + >> + cells = dt_get_property(dev, "bus-range", &len); >> + /* bus-range should at least be 2 cells */ >> + if ( !cells || (len < (sizeof(*cells) * 2)) ) >> + return false; >> + >> + cfg->busn_start = dt_next_cell(1, &cells); >> + cfg->busn_end = dt_next_cell(1, &cells); >> + >> + return true; >> +} >> + >> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len) >> +{ >> + return ioremap_nocache(start, len); >> +} >> + >> +static void pci_ecam_free(struct pci_config_window *cfg) >> +{ >> + if ( cfg->win ) >> + iounmap(cfg->win); >> + >> + xfree(cfg); >> +} >> + >> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, >> + int ecam_reg_idx) > > If it is only called at init time, then the function should be __init Ack. > > >> +{ >> + int err; >> + struct pci_config_window *cfg; >> + paddr_t addr, size; >> + >> + cfg = xzalloc(struct pci_config_window); >> + if ( !cfg ) >> + return NULL; >> + >> + err = dt_pci_parse_bus_range(dev, cfg); >> + if ( !err ) { >> + cfg->busn_start = 0; >> + cfg->busn_end = 0xff; >> + printk(XENLOG_ERR "%s:No bus range found for pci controller\n", >> + dt_node_full_name(dev)); >> + } else { >> + if ( cfg->busn_end > cfg->busn_start + 0xff ) >> + cfg->busn_end = cfg->busn_start + 0xff; > > Is this a hard limit in the specification? Or is it a limit in the Xen > implementation? I just take the reference from Linux code. https://elixir.bootlin.com/linux/latest/source/drivers/pci/of.c#L306 > > >> + } >> + >> + /* Parse our PCI ecam register address*/ > ^ space > Ack. >> + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size); >> + if ( err ) >> + goto err_exit; >> + >> + cfg->phys_addr = addr; >> + cfg->size = size; >> + >> + /* >> + * On 64-bit systems, we do a single ioremap for the whole config space >> + * since we have enough virtual address range available. On 32-bit, we >> + * ioremap the config space for each bus individually. >> + * >> + * As of now only 64-bit is supported 32-bit is not supported. >> + */ >> + cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size); >> + if ( !cfg->win ) >> + goto err_exit_remap; >> + >> + printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, >> + cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); >> + >> + return cfg; >> + >> +err_exit_remap: >> + printk(XENLOG_ERR "ECAM ioremap failed\n"); >> +err_exit: >> + pci_ecam_free(cfg); >> + return NULL; >> +} >> + >> +struct pci_host_bridge *pci_alloc_host_bridge(void) >> +{ >> + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); >> + >> + if ( !bridge ) >> + return NULL; >> + >> + INIT_LIST_HEAD(&bridge->node); >> + bridge->bus_start = ~0; >> + bridge->bus_end = ~0; > > Please use INVALID_PADDR instead of ~0 bridge->bus_start is if type uint8_t if I use INVALID_PADDR I will get overflow error. I will use U8_MAX instead of INVALID_PADDR. > > >> + return bridge; >> +} >> + >> +void pci_add_host_bridge(struct pci_host_bridge *bridge) >> +{ >> + list_add_tail(&bridge->node, &pci_host_bridges); >> +} >> + >> +static int pci_get_new_domain_nr(void) >> +{ >> + return atomic_inc_return(&domain_nr); >> +} >> + >> +/* >> + * This function will try to obtain the host bridge domain number by >> + * finding a property called "linux,pci-domain" of the given device node. >> + * >> + * @node: device tree node with the domain information >> + * >> + * Returns the associated domain number from DT in the range [0-0xffff], or >> + * a negative value if the required property is not found. >> + */ >> +static int dt_get_pci_domain_nr(struct dt_device_node *node) >> +{ >> + u32 domain; >> + int error; >> + >> + error = dt_property_read_u32(node, "linux,pci-domain", &domain); >> + if ( !error ) >> + return -EINVAL; >> + >> + return (u16)domain; > > Let's check that domain <= UINT16_MAX > Ack Regards, Rahul
On 10/09/2021 12:22, Rahul Singh wrote: > Hi Julien Hi Rahul, >> On 7 Sep 2021, at 10:05 am, Julien Grall <julien@xen.org> wrote: >> On 19/08/2021 13:02, Rahul Singh wrote: >>> +#include <xen/rwlock.h> >>> +#include <xen/sched.h> >>> +#include <xen/vmap.h> >>> + >>> +/* >>> + * List for all the pci host bridges. >>> + */ >>> + >>> +static LIST_HEAD(pci_host_bridges); >>> + >>> +static atomic_t domain_nr = ATOMIC_INIT(-1); >>> + >>> +bool dt_pci_parse_bus_range(struct dt_device_node *dev, >>> + struct pci_config_window *cfg) >> >> Aside, "pci_config_window", the function is not Arm specific. Would it be possible to consider to introduce "struct resource" in Xen so this function can be moved in common/device_tree.c? > > I can introduce the "struct resource” but I am not sure whether "struct resource” will be > useful later point in time. What I prefer as of now, we can have this function and we can > move this to common/device_tree.c once we have the requirement for "struct resource”. TBH, I wasn't asking about using "struct resource". This was simply a suggestion how to make the code common. What I am more interested is moving this function in common code so it can be re-used by RISC-v in the future. I don't particularly mind whether this is using "struct resource" or a different structure. >>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c >>> new file mode 100644 >>> index 0000000000..13d0f7f999 >>> --- /dev/null >>> +++ b/xen/arch/arm/pci/pci-host-generic.c >>> @@ -0,0 +1,55 @@ >>> +/* >>> + * Copyright (C) 2021 Arm Ltd. >>> + * >>> + * Based on Linux drivers/pci/controller/pci-host-common.c >>> + * Based on Linux drivers/pci/controller/pci-host-generic.c >>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> >>> + * >>> + * This program is free software; you can redistribute it and/or modify >>> + * it under the terms of the GNU General Public License version 2 as >>> + * published by the Free Software Foundation. >>> + * >>> + * This program is distributed in the hope that it will be useful, >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >>> + * GNU General Public License for more details. >>> + * >>> + * You should have received a copy of the GNU General Public License >>> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >>> + */ >>> + >>> +#include <asm/device.h> >>> +#include <xen/pci.h> >>> +#include <asm/pci.h> >>> + >>> +static const struct dt_device_match gen_pci_dt_match[] = { >>> + { .compatible = "pci-host-ecam-generic" }, >>> + { }, >>> +}; >>> + >>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) >>> +{ >>> + const struct dt_device_match *of_id; >>> + >>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); >> >> This seems to be a bit pointless to me as you already know the compatible (there is only one possible...). > As of now we are only implementing the "pci-host-ecam-generic” compatible PCI, but in future we might > need to implement the other compatible like Linux see as below. > > https://elixir.bootlin.com/linux/latest/source/drivers/pci/controller/pci-host-generic.c#L59 Right... You know the node matched and you don't need any data. So this is a bit pointless to have to walk again the structure for just printing the compatible that matched. Cheers, -- Julien Grall
Hi Stefano, > On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Thu, 19 Aug 2021, Rahul Singh wrote: >> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >> >> Add support for Xilinx ZynqMP PCI host controller to map the PCI config >> space to the XEN memory. >> >> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >> --- >> xen/arch/arm/pci/Makefile | 1 + >> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++ >> 2 files changed, 60 insertions(+) >> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c >> >> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >> index 6f32fbbe67..1d045ade01 100644 >> --- a/xen/arch/arm/pci/Makefile >> +++ b/xen/arch/arm/pci/Makefile >> @@ -3,3 +3,4 @@ obj-y += pci-access.o >> obj-y += pci-host-generic.o >> obj-y += pci-host-common.o >> obj-y += ecam.o >> +obj-y += pci-host-zynqmp.o >> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c >> new file mode 100644 >> index 0000000000..fe103e3855 >> --- /dev/null >> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c >> @@ -0,0 +1,59 @@ >> +/* >> + * Copyright (C) 2020-2021 EPAM Systems >> + * >> + * Based on Linux drivers/pci/controller/pci-host-common.c >> + * Based on Linux drivers/pci/controller/pci-host-generic.c >> + * Based on xen/arch/arm/pci/pci-host-generic.c >> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> > > Only one Copyright line per file is enough :-) > > But actually all the Copyright lines with a name or a company name are > not really required or useful, as the copyright is noted in full details > in the commit messages (author and signed-off-by lines). I would remove > them all from the new files added by this series. Ok. Let me remove in next version. > > >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <asm/device.h> >> +#include <xen/pci.h> >> +#include <asm/pci.h> >> + >> +static const struct dt_device_match gen_pci_dt_match[] = { >> + { .compatible = "xlnx,nwl-pcie-2.11", >> + .data = &pci_generic_ecam_ops }, >> + { }, >> +}; >> + >> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) >> +{ >> + const struct dt_device_match *of_id; >> + const struct pci_ecam_ops *ops; >> + >> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); > > This should be superfluous Ack. I will remove the dt_match_node(..) in next version. Regards, Rahul
On 10.09.21 15:01, Rahul Singh wrote: > Hi Stefano, > >> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote: >> >> On Thu, 19 Aug 2021, Rahul Singh wrote: >>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>> >>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config >>> space to the XEN memory. >>> >>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>> --- >>> xen/arch/arm/pci/Makefile | 1 + >>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++ >>> 2 files changed, 60 insertions(+) >>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c >>> >>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >>> index 6f32fbbe67..1d045ade01 100644 >>> --- a/xen/arch/arm/pci/Makefile >>> +++ b/xen/arch/arm/pci/Makefile >>> @@ -3,3 +3,4 @@ obj-y += pci-access.o >>> obj-y += pci-host-generic.o >>> obj-y += pci-host-common.o >>> obj-y += ecam.o >>> +obj-y += pci-host-zynqmp.o >>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c >>> new file mode 100644 >>> index 0000000000..fe103e3855 >>> --- /dev/null >>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c >>> @@ -0,0 +1,59 @@ >>> +/* >>> + * Copyright (C) 2020-2021 EPAM Systems >>> + * >>> + * Based on Linux drivers/pci/controller/pci-host-common.c >>> + * Based on Linux drivers/pci/controller/pci-host-generic.c >>> + * Based on xen/arch/arm/pci/pci-host-generic.c >>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> >> Only one Copyright line per file is enough :-) >> >> But actually all the Copyright lines with a name or a company name are >> not really required or useful, as the copyright is noted in full details >> in the commit messages (author and signed-off-by lines). I would remove >> them all from the new files added by this series. > Ok. Let me remove in next version. >> >>> + * This program is free software; you can redistribute it and/or modify >>> + * it under the terms of the GNU General Public License version 2 as >>> + * published by the Free Software Foundation. >>> + * >>> + * This program is distributed in the hope that it will be useful, >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >>> + * GNU General Public License for more details. >>> + * >>> + * You should have received a copy of the GNU General Public License >>> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >>> + */ >>> + >>> +#include <asm/device.h> >>> +#include <xen/pci.h> >>> +#include <asm/pci.h> >>> + >>> +static const struct dt_device_match gen_pci_dt_match[] = { >>> + { .compatible = "xlnx,nwl-pcie-2.11", >>> + .data = &pci_generic_ecam_ops }, >>> + { }, >>> +}; >>> + >>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) >>> +{ >>> + const struct dt_device_match *of_id; >>> + const struct pci_ecam_ops *ops; >>> + >>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); >> This should be superfluous > Ack. I will remove the dt_match_node(..) in next version. I am not entirely sure we need this patch at all as the main reason for its existence was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP. So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one won't have it working... Please consider If we decide to remove it then int pci_host_common_probe(struct dt_device_node *dev, const struct pci_ecam_ops *ops, int ecam_reg_idx) doesn't need the last parameter. > > Regards, > Rahul > > Thanks, Oleksandr
Hi, Rahul! On 19.08.21 15:02, Rahul Singh wrote: > XEN during boot will read the PCI device tree node “reg” property > and will map the PCI config space to the XEN memory. [snip] > +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > + int ecam_reg_idx) > +{ > + int err; > + struct pci_config_window *cfg; > + paddr_t addr, size; > + > + cfg = xzalloc(struct pci_config_window); > + if ( !cfg ) > + return NULL; > + > + err = dt_pci_parse_bus_range(dev, cfg); > + if ( !err ) { > + cfg->busn_start = 0; > + cfg->busn_end = 0xff; > + printk(XENLOG_ERR "%s:No bus range found for pci controller\n", > + dt_node_full_name(dev)); > + } else { > + if ( cfg->busn_end > cfg->busn_start + 0xff ) > + cfg->busn_end = cfg->busn_start + 0xff; > + } > + > + /* Parse our PCI ecam register address*/ > + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size); I am a bit worried here that we don't get the reg index from the device tree, but for generic ECAM we use reg[0] and for Xilinx we use reg[2]. For example, for Xilinx we have reg = <0x00 0xfd0e0000 0x00 0x1000 0x00 0xfd480000 0x00 0x1000 0x80 0x00 0x00 0x1000000>; reg-names = "breg\0pcireg\0cfg"; so, we can parse the reg-names and understand that the configuration space is the last in the reg property. The same I think can be done for other device trees probably. Rahul, do you know if reg-names "cfg" is vendor specific of used widely? Thank you, Oleksandr
[-- Attachment #1: Type: text/plain, Size: 1874 bytes --] On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote: > Hi, Rahul! > > On 19.08.21 15:02, Rahul Singh wrote: > > XEN during boot will read the PCI device tree node “reg” property > > and will map the PCI config space to the XEN memory. > [snip] > > +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, > > + int ecam_reg_idx) > > +{ > > + int err; > > + struct pci_config_window *cfg; > > + paddr_t addr, size; > > + > > + cfg = xzalloc(struct pci_config_window); > > + if ( !cfg ) > > + return NULL; > > + > > + err = dt_pci_parse_bus_range(dev, cfg); > > + if ( !err ) { > > + cfg->busn_start = 0; > > + cfg->busn_end = 0xff; > > + printk(XENLOG_ERR "%s:No bus range found for pci controller\n", > > + dt_node_full_name(dev)); > > + } else { > > + if ( cfg->busn_end > cfg->busn_start + 0xff ) > > + cfg->busn_end = cfg->busn_start + 0xff; > > + } > > + > > + /* Parse our PCI ecam register address*/ > > + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size); > > I am a bit worried here that we don't get the reg index from the device tree, > > but for generic ECAM we use reg[0] and for Xilinx we use reg[2]. > > For example, for Xilinx we have > > reg = <0x00 0xfd0e0000 0x00 0x1000 0x00 0xfd480000 0x00 0x1000 0x80 0x00 0x00 0x1000000>; > reg-names = "breg\0pcireg\0cfg"; > > so, we can parse the reg-names and understand that the configuration space is the last in the reg property. > > The same I think can be done for other device trees probably. Well spotted! > Rahul, do you know if reg-names "cfg" is vendor specific of used widely? Unfortunately it seems to be vendor specific :-( Which means that "dt_device_get_address" or similar should be moved to a vendor specific function.
[-- Attachment #1: Type: text/plain, Size: 4694 bytes --] On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote: > On 10.09.21 15:01, Rahul Singh wrote: > > Hi Stefano, > > > >> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > >> > >> On Thu, 19 Aug 2021, Rahul Singh wrote: > >>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > >>> > >>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config > >>> space to the XEN memory. > >>> > >>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > >>> --- > >>> xen/arch/arm/pci/Makefile | 1 + > >>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++ > >>> 2 files changed, 60 insertions(+) > >>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c > >>> > >>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile > >>> index 6f32fbbe67..1d045ade01 100644 > >>> --- a/xen/arch/arm/pci/Makefile > >>> +++ b/xen/arch/arm/pci/Makefile > >>> @@ -3,3 +3,4 @@ obj-y += pci-access.o > >>> obj-y += pci-host-generic.o > >>> obj-y += pci-host-common.o > >>> obj-y += ecam.o > >>> +obj-y += pci-host-zynqmp.o > >>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c > >>> new file mode 100644 > >>> index 0000000000..fe103e3855 > >>> --- /dev/null > >>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c > >>> @@ -0,0 +1,59 @@ > >>> +/* > >>> + * Copyright (C) 2020-2021 EPAM Systems > >>> + * > >>> + * Based on Linux drivers/pci/controller/pci-host-common.c > >>> + * Based on Linux drivers/pci/controller/pci-host-generic.c > >>> + * Based on xen/arch/arm/pci/pci-host-generic.c > >>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> > >> Only one Copyright line per file is enough :-) > >> > >> But actually all the Copyright lines with a name or a company name are > >> not really required or useful, as the copyright is noted in full details > >> in the commit messages (author and signed-off-by lines). I would remove > >> them all from the new files added by this series. > > Ok. Let me remove in next version. > >> > >>> + * This program is free software; you can redistribute it and/or modify > >>> + * it under the terms of the GNU General Public License version 2 as > >>> + * published by the Free Software Foundation. > >>> + * > >>> + * This program is distributed in the hope that it will be useful, > >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of > >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > >>> + * GNU General Public License for more details. > >>> + * > >>> + * You should have received a copy of the GNU General Public License > >>> + * along with this program. If not, see <http://www.gnu.org/licenses/>. > >>> + */ > >>> + > >>> +#include <asm/device.h> > >>> +#include <xen/pci.h> > >>> +#include <asm/pci.h> > >>> + > >>> +static const struct dt_device_match gen_pci_dt_match[] = { > >>> + { .compatible = "xlnx,nwl-pcie-2.11", > >>> + .data = &pci_generic_ecam_ops }, > >>> + { }, > >>> +}; > >>> + > >>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) > >>> +{ > >>> + const struct dt_device_match *of_id; > >>> + const struct pci_ecam_ops *ops; > >>> + > >>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); > >> This should be superfluous > > Ack. I will remove the dt_match_node(..) in next version. > > I am not entirely sure we need this patch at all as the main reason for its existence > > was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going > > to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP. > > So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one > > won't have it working... > > Please consider > > If we decide to remove it then > > int pci_host_common_probe(struct dt_device_node *dev, > const struct pci_ecam_ops *ops, > int ecam_reg_idx) > > doesn't need the last parameter. With my open source maintainer hat on, I don't see this patch as very important; from that point of view I'd be happy for it to be dropped. However, it would be good to have at least one non-default host bridge (doesn't have to be the Xilinx bridge), otherwise it becomes difficult to understand how the generic infrastructure introduced by this series could be useful. Moreover, your recent comment [1] made it even more evident that it would be good to have at least two different drivers to spot compatibility issues between them more easily. [1] https://marc.info/?l=xen-devel&m=163154474008598
On 14.09.21 00:02, Stefano Stabellini wrote: > On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote: >> On 10.09.21 15:01, Rahul Singh wrote: >>> Hi Stefano, >>> >>>> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote: >>>> >>>> On Thu, 19 Aug 2021, Rahul Singh wrote: >>>>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>>>> >>>>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config >>>>> space to the XEN memory. >>>>> >>>>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>>>> --- >>>>> xen/arch/arm/pci/Makefile | 1 + >>>>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++ >>>>> 2 files changed, 60 insertions(+) >>>>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c >>>>> >>>>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >>>>> index 6f32fbbe67..1d045ade01 100644 >>>>> --- a/xen/arch/arm/pci/Makefile >>>>> +++ b/xen/arch/arm/pci/Makefile >>>>> @@ -3,3 +3,4 @@ obj-y += pci-access.o >>>>> obj-y += pci-host-generic.o >>>>> obj-y += pci-host-common.o >>>>> obj-y += ecam.o >>>>> +obj-y += pci-host-zynqmp.o >>>>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c >>>>> new file mode 100644 >>>>> index 0000000000..fe103e3855 >>>>> --- /dev/null >>>>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c >>>>> @@ -0,0 +1,59 @@ >>>>> +/* >>>>> + * Copyright (C) 2020-2021 EPAM Systems >>>>> + * >>>>> + * Based on Linux drivers/pci/controller/pci-host-common.c >>>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c >>>>> + * Based on xen/arch/arm/pci/pci-host-generic.c >>>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> >>>> Only one Copyright line per file is enough :-) >>>> >>>> But actually all the Copyright lines with a name or a company name are >>>> not really required or useful, as the copyright is noted in full details >>>> in the commit messages (author and signed-off-by lines). I would remove >>>> them all from the new files added by this series. >>> Ok. Let me remove in next version. >>>>> + * This program is free software; you can redistribute it and/or modify >>>>> + * it under the terms of the GNU General Public License version 2 as >>>>> + * published by the Free Software Foundation. >>>>> + * >>>>> + * This program is distributed in the hope that it will be useful, >>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >>>>> + * GNU General Public License for more details. >>>>> + * >>>>> + * You should have received a copy of the GNU General Public License >>>>> + * along with this program. If not, see <https://urldefense.com/v3/__http://www.gnu.org/licenses/__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_3j2CfflQ$ [gnu[.]org]>. >>>>> + */ >>>>> + >>>>> +#include <asm/device.h> >>>>> +#include <xen/pci.h> >>>>> +#include <asm/pci.h> >>>>> + >>>>> +static const struct dt_device_match gen_pci_dt_match[] = { >>>>> + { .compatible = "xlnx,nwl-pcie-2.11", >>>>> + .data = &pci_generic_ecam_ops }, >>>>> + { }, >>>>> +}; >>>>> + >>>>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) >>>>> +{ >>>>> + const struct dt_device_match *of_id; >>>>> + const struct pci_ecam_ops *ops; >>>>> + >>>>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); >>>> This should be superfluous >>> Ack. I will remove the dt_match_node(..) in next version. >> I am not entirely sure we need this patch at all as the main reason for its existence >> >> was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going >> >> to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP. >> >> So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one >> >> won't have it working... >> >> Please consider >> >> If we decide to remove it then >> >> int pci_host_common_probe(struct dt_device_node *dev, >> const struct pci_ecam_ops *ops, >> int ecam_reg_idx) >> >> doesn't need the last parameter. > With my open source maintainer hat on, I don't see this patch as very > important; from that point of view I'd be happy for it to be dropped. > However, it would be good to have at least one non-default host bridge > (doesn't have to be the Xilinx bridge), otherwise it becomes difficult > to understand how the generic infrastructure introduced by this series > could be useful. > > Moreover, your recent comment [1] made it even more evident that it > would be good to have at least two different drivers to spot > compatibility issues between them more easily. Don't take me wrong here ;) I still do use Xilinx QEMU for most of the tests, so it is good for me to have this patch in the tree. But, to be fair, Xilinx QEMU won't give you a possibility to see the fully functional system. This is why I say the patch can be dropped. If we add some words in the commit message about this and have the patch in the tree I'll be more than happy > > [1] https://urldefense.com/v3/__https://marc.info/?l=xen-devel&m=163154474008598__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_0bKs6zpA$ [marc[.]info]
On 13.09.21 23:23, Stefano Stabellini wrote:
> On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote:
>> Hi, Rahul!
>>
>> On 19.08.21 15:02, Rahul Singh wrote:
>>> XEN during boot will read the PCI device tree node “reg” property
>>> and will map the PCI config space to the XEN memory.
>> [snip]
>>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>>> + int ecam_reg_idx)
>>> +{
>>> + int err;
>>> + struct pci_config_window *cfg;
>>> + paddr_t addr, size;
>>> +
>>> + cfg = xzalloc(struct pci_config_window);
>>> + if ( !cfg )
>>> + return NULL;
>>> +
>>> + err = dt_pci_parse_bus_range(dev, cfg);
>>> + if ( !err ) {
>>> + cfg->busn_start = 0;
>>> + cfg->busn_end = 0xff;
>>> + printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
>>> + dt_node_full_name(dev));
>>> + } else {
>>> + if ( cfg->busn_end > cfg->busn_start + 0xff )
>>> + cfg->busn_end = cfg->busn_start + 0xff;
>>> + }
>>> +
>>> + /* Parse our PCI ecam register address*/
>>> + err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
>> I am a bit worried here that we don't get the reg index from the device tree,
>>
>> but for generic ECAM we use reg[0] and for Xilinx we use reg[2].
>>
>> For example, for Xilinx we have
>>
>> reg = <0x00 0xfd0e0000 0x00 0x1000 0x00 0xfd480000 0x00 0x1000 0x80 0x00 0x00 0x1000000>;
>> reg-names = "breg\0pcireg\0cfg";
>>
>> so, we can parse the reg-names and understand that the configuration space is the last in the reg property.
>>
>> The same I think can be done for other device trees probably.
> Well spotted!
>
>
>> Rahul, do you know if reg-names "cfg" is vendor specific of used widely?
> Unfortunately it seems to be vendor specific :-(
> Which means that "dt_device_get_address" or similar should be moved to a
> vendor specific function.
I just stepped on this while working on the patch for not mapping
MMIOs and config to Dom0 and need to find a way to tell them from
device registers. So, yes, it seems we will need something more generic
than just an index. We can still have the callback for this optional,
so the bridges with the same reg names can use the default callback
and don't need to implement the same
Hi, Rahul! On 19.08.21 15:02, Rahul Singh wrote: > XEN during boot will read the PCI device tree node “reg” property > and will map the PCI config space to the XEN memory. [snip] > +int pci_host_common_probe(struct dt_device_node *dev, > + int ecam_reg_idx) > +{ > + struct pci_host_bridge *bridge; > + struct pci_config_window *cfg; > + int err; > + Should we check if the bridge is to be passed through? E.g. if ( dt_device_for_passthrough(dev) ) return 0; Thanks, Oleksandr
Hi Julien, > On 9 Sep 2021, at 12:32 pm, Julien Grall <julien@xen.org> wrote: > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> Add support for PCI ecam operations to access the PCI >> configuration space. >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/pci/Makefile | 1 + >> xen/arch/arm/pci/ecam.c | 63 +++++++++++++++++++++++++++++ >> xen/arch/arm/pci/pci-access.c | 53 ++++++++++++++++++++++++ >> xen/arch/arm/pci/pci-host-common.c | 13 +++++- >> xen/arch/arm/pci/pci-host-generic.c | 8 +++- >> xen/include/asm-arm/pci.h | 32 +++++++++++++++ >> 6 files changed, 167 insertions(+), 3 deletions(-) >> create mode 100644 xen/arch/arm/pci/ecam.c >> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >> index f3d97f859e..6f32fbbe67 100644 >> --- a/xen/arch/arm/pci/Makefile >> +++ b/xen/arch/arm/pci/Makefile >> @@ -2,3 +2,4 @@ obj-y += pci.o >> obj-y += pci-access.o >> obj-y += pci-host-generic.o >> obj-y += pci-host-common.o >> +obj-y += ecam.o >> diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c >> new file mode 100644 >> index 0000000000..91c691b41f >> --- /dev/null >> +++ b/xen/arch/arm/pci/ecam.c >> @@ -0,0 +1,63 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * Based on Linux drivers/pci/ecam.c >> + * Copyright 2016 Broadcom >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/pci.h> >> +#include <xen/sched.h> >> + >> +/* >> + * Function to implement the pci_ops ->map_bus method. >> + */ >> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, >> + uint32_t sbdf, uint32_t where) >> +{ >> + const struct pci_config_window *cfg = bridge->sysdata; >> + unsigned int devfn_shift = cfg->ops->bus_shift - 8; >> + void __iomem *base; >> + >> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ; > > AFAICT, pci_sbdf is an union between a 32-bit and a structure. So please don't use the cast and use the 32-bit field to assign the value. > > Also, there is an extra space before ';'. Ok. As per your below comment I will remove the sbdf_t completely. > > >> + unsigned int busn = sbdf_t.bus; >> + >> + if ( busn < cfg->busn_start || busn > cfg->busn_end ) >> + return NULL; >> + >> + busn -= cfg->busn_start; >> + base = cfg->win + (busn << cfg->ops->bus_shift); >> + >> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; > > How about using PCI_DEVFN2(sbdf)? This would allow you to drop the use of sbdf_t completely (sbdf_t.bus could be replaced with PCI_BUS(sdbf)). Make sense. I will modify as per your request. > >> +} >> + >> +/* ECAM ops */ >> +const struct pci_ecam_ops pci_generic_ecam_ops = { >> + .bus_shift = 20, >> + .pci_ops = { >> + .map_bus = pci_ecam_map_bus, >> + .read = pci_generic_config_read, >> + .write = pci_generic_config_write, >> + } >> +}; >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * tab-width: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c >> index b938047c03..f39f6a3a38 100644 >> --- a/xen/arch/arm/pci/pci-access.c >> +++ b/xen/arch/arm/pci/pci-access.c >> @@ -15,6 +15,59 @@ >> */ >> #include <xen/pci.h> >> +#include <asm/io.h> >> + >> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, >> + uint32_t reg, uint32_t len, uint32_t *value) >> +{ >> + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); > > Please add a newline here. Ack. > >> + if (!addr) { > > You seem to use a mix of Xen and Linux coding style. If the file contains mostly Xen code, then we should use the former. Ok. I will fix the coding style in next version. Regards, Rahul
Hi Stefano, > On 10 Sep 2021, at 12:21 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Thu, 19 Aug 2021, Rahul Singh wrote: >> Add support for PCI ecam operations to access the PCI >> configuration space. >> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/pci/Makefile | 1 + >> xen/arch/arm/pci/ecam.c | 63 +++++++++++++++++++++++++++++ >> xen/arch/arm/pci/pci-access.c | 53 ++++++++++++++++++++++++ >> xen/arch/arm/pci/pci-host-common.c | 13 +++++- >> xen/arch/arm/pci/pci-host-generic.c | 8 +++- >> xen/include/asm-arm/pci.h | 32 +++++++++++++++ >> 6 files changed, 167 insertions(+), 3 deletions(-) >> create mode 100644 xen/arch/arm/pci/ecam.c >> >> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >> index f3d97f859e..6f32fbbe67 100644 >> --- a/xen/arch/arm/pci/Makefile >> +++ b/xen/arch/arm/pci/Makefile >> @@ -2,3 +2,4 @@ obj-y += pci.o >> obj-y += pci-access.o >> obj-y += pci-host-generic.o >> obj-y += pci-host-common.o >> +obj-y += ecam.o >> diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c >> new file mode 100644 >> index 0000000000..91c691b41f >> --- /dev/null >> +++ b/xen/arch/arm/pci/ecam.c >> @@ -0,0 +1,63 @@ >> +/* >> + * Copyright (C) 2021 Arm Ltd. >> + * >> + * Based on Linux drivers/pci/ecam.c >> + * Copyright 2016 Broadcom >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License version 2 as >> + * published by the Free Software Foundation. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + * >> + * You should have received a copy of the GNU General Public License >> + * along with this program. If not, see <http://www.gnu.org/licenses/>. >> + */ >> + >> +#include <xen/pci.h> >> +#include <xen/sched.h> >> + >> +/* >> + * Function to implement the pci_ops ->map_bus method. >> + */ >> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge, >> + uint32_t sbdf, uint32_t where) > > Code style: alignment Ack. > > >> +{ >> + const struct pci_config_window *cfg = bridge->sysdata; >> + unsigned int devfn_shift = cfg->ops->bus_shift - 8; > > Is it a guarantee that devfn_shift == bus_shift - 8, or is it just so > for ECAM? Yes it is guarantee that "devfn_shift == bus_shift - 8” as bus number will be 8 bit always. bus_shift can be different based on vendor. https://elixir.bootlin.com/linux/latest/source/drivers/pci/ecam.c#L172 https://elixir.bootlin.com/linux/latest/source/drivers/pci/controller/pci-thunder-pem.c#L30 > > >> + void __iomem *base; >> + >> + pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ; >> + unsigned int busn = sbdf_t.bus; >> + >> + if ( busn < cfg->busn_start || busn > cfg->busn_end ) > > Genuine question: should it be busn >= cfg->busn_end ? I don't know if > the range includes busn_end or not. cfg->bus_end includes the valid last bus number. "busn > cfg->busn_end" is valid check to confirm if we are not accessing pci device outside the valid bus number. > > >> + return NULL; >> + >> + busn -= cfg->busn_start; >> + base = cfg->win + (busn << cfg->ops->bus_shift); >> + >> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; >> +} > > I understand that the arm32 part is not implemented and not part of this > series, that's fine. However if the plan is that arm32 will dynamically > map each bus individually, then I imagine this function will have an > ioremap in the arm32 version. Which means that we also need an > unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus > would be a NOP today for arm64, but I think it makes sense to have it if > we want the API to be generic. As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part. Let me know your view on this. > > >> +/* ECAM ops */ >> +const struct pci_ecam_ops pci_generic_ecam_ops = { >> + .bus_shift = 20, >> + .pci_ops = { >> + .map_bus = pci_ecam_map_bus, >> + .read = pci_generic_config_read, >> + .write = pci_generic_config_write, >> + } >> +}; >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * tab-width: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c >> index b938047c03..f39f6a3a38 100644 >> --- a/xen/arch/arm/pci/pci-access.c >> +++ b/xen/arch/arm/pci/pci-access.c >> @@ -15,6 +15,59 @@ >> */ >> >> #include <xen/pci.h> >> +#include <asm/io.h> >> + >> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf, >> + uint32_t reg, uint32_t len, uint32_t *value) >> +{ >> + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); >> + if (!addr) { >> + *value = ~0; > > Is this a standard error? If so, I think we should define it with a > macro (e.g. INVALID_PADDR). Ok. > > >> + return -ENODEV; >> + } >> + >> + switch (len) >> + { >> + case 1: >> + *value = readb(addr); >> + break; >> + case 2: >> + *value = readw(addr); >> + break; >> + case 4: >> + *value = readl(addr); >> + break; >> + default: >> + BUG(); > > A BUG here is harsh because it could be potentially guest-triggered. An > ASSERT would be better. Ok. > > >> + } >> + >> + return 0; >> +} >> + >> +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, >> + uint32_t reg, uint32_t len, uint32_t value) >> +{ >> + void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg); >> + if (!addr) >> + return -ENODEV; >> + >> + switch (len) >> + { >> + case 1: >> + writeb(value, addr); >> + break; >> + case 2: >> + writew(value, addr); >> + break; >> + case 4: >> + writel(value, addr); >> + break; >> + default: >> + BUG(); > > Same here Ok. > > >> + } >> + >> + return 0; >> +} >> >> static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, >> unsigned int len) >> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c >> index 9dd9b02271..c582527e92 100644 >> --- a/xen/arch/arm/pci/pci-host-common.c >> +++ b/xen/arch/arm/pci/pci-host-common.c >> @@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg) >> } >> >> static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, >> + const struct pci_ecam_ops *ops, >> int ecam_reg_idx) >> { >> int err; >> @@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, >> >> cfg->phys_addr = addr; >> cfg->size = size; >> + cfg->ops = ops; >> >> /* >> * On 64-bit systems, we do a single ioremap for the whole config space >> @@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev, >> printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr, >> cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end); >> >> + if ( ops->init ) >> + { >> + err = ops->init(cfg); >> + if (err) >> + goto err_exit; >> + } >> + >> return cfg; >> >> err_exit_remap: >> @@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev) >> } >> >> int pci_host_common_probe(struct dt_device_node *dev, >> + const struct pci_ecam_ops *ops, >> int ecam_reg_idx) >> { >> struct pci_host_bridge *bridge; >> @@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev, >> return -ENOMEM; >> >> /* Parse and map our Configuration Space windows */ >> - cfg = gen_pci_init(dev, ecam_reg_idx); >> + cfg = gen_pci_init(dev, ops, ecam_reg_idx); >> if ( !cfg ) >> { >> err = -ENOMEM; >> @@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev, >> >> bridge->dt_node = dev; >> bridge->sysdata = cfg; >> + bridge->ops = &ops->pci_ops; >> bridge->bus_start = cfg->busn_start; >> bridge->bus_end = cfg->busn_end; >> >> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c >> index 13d0f7f999..2d652e8910 100644 >> --- a/xen/arch/arm/pci/pci-host-generic.c >> +++ b/xen/arch/arm/pci/pci-host-generic.c >> @@ -23,20 +23,24 @@ >> #include <asm/pci.h> >> >> static const struct dt_device_match gen_pci_dt_match[] = { >> - { .compatible = "pci-host-ecam-generic" }, >> + { .compatible = "pci-host-ecam-generic", >> + .data = &pci_generic_ecam_ops }, >> + >> { }, >> }; >> >> static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) >> { >> const struct dt_device_match *of_id; >> + const struct pci_ecam_ops *ops; >> >> of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); >> + ops = (struct pci_ecam_ops *) of_id->data; >> >> printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n", >> dt_node_full_name(dev), of_id->compatible); >> >> - return pci_host_common_probe(dev, 0); >> + return pci_host_common_probe(dev, ops, 0); >> } >> >> DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI) >> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h >> index 58a51e724e..22866244d2 100644 >> --- a/xen/include/asm-arm/pci.h >> +++ b/xen/include/asm-arm/pci.h >> @@ -37,6 +37,7 @@ struct pci_config_window { >> uint8_t busn_start; >> uint8_t busn_end; >> void __iomem *win; >> + const struct pci_ecam_ops *ops; >> }; >> >> /* >> @@ -50,10 +51,41 @@ struct pci_host_bridge { >> u8 bus_start; /* Bus start of this bridge. */ >> u8 bus_end; /* Bus end of this bridge. */ >> void *sysdata; /* Pointer to the config space window*/ >> + const struct pci_ops *ops; >> }; >> >> +struct pci_ops { >> + void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf, >> + uint32_t offset); >> + int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf, >> + uint32_t reg, uint32_t len, uint32_t *value); >> + int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf, >> + uint32_t reg, uint32_t len, uint32_t value); >> +}; >> + >> +/* >> + * struct to hold pci ops and bus shift of the config window >> + * for a PCI controller. >> + */ >> +struct pci_ecam_ops { >> + unsigned int bus_shift; >> + struct pci_ops pci_ops; >> + int (*init)(struct pci_config_window *); > > Is this last member of the struct needed/used? Yes It will be used by some vendor specific board( N1SDP ). Please check below. https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/commit/?id=04b7e76d0fe6481a803f58e54e008a1489d713a5 Regards, Rahul
Hi Stefano, > On 10 Sep 2021, at 12:41 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Thu, 19 Aug 2021, Rahul Singh wrote: >> Implement generic pci access functions to read/write the configuration >> space. >> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/pci/pci-access.c | 31 +++++++++++++++++++++++++++++- >> xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++ >> xen/include/asm-arm/pci.h | 2 ++ >> 3 files changed, 51 insertions(+), 1 deletion(-) >> >> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c >> index f39f6a3a38..b94de3c3ac 100644 >> --- a/xen/arch/arm/pci/pci-access.c >> +++ b/xen/arch/arm/pci/pci-access.c >> @@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, >> static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, >> unsigned int len) >> { >> - return ~0U; >> + uint32_t val = GENMASK(0, len * 8); > > This seems to be another default error value that it would be better to > define with its own macro This default error is used once do you want to me define as macro. > > >> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); >> + >> + if ( unlikely(!bridge) ) >> + { >> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", >> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); > > You are not actually printing sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn ? Yes I am printing with “PRI_pci". > > >> + return val; >> + } >> + >> + if ( unlikely(!bridge->ops->read) ) >> + return val; >> + >> + bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val); > > Would it make sense to make the interface take a pci_sbdf_t directly > instead of casting to uint32_t and back? pci_sbdf_t is defined in "xen/pci.h” and "xen/pci.h” includes "asm-arm/pci.h”. If I modify the function argument in "asm-arm/pci.h” to use pci_sbdf_t I will get compilation error. > > >> + return val; >> } >> >> static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg, >> unsigned int len, uint32_t val) >> { >> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); >> + >> + if ( unlikely(!bridge) ) >> + { >> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", >> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); > > same here Yes I am printing with “PRI_pci". > Regards Rahul
[-- Attachment #1: Type: text/plain, Size: 3316 bytes --] +Jan for the header include question at the bottom On Tue, 14 Sep 2021, Rahul Singh wrote: > Hi Stefano, > > > On 10 Sep 2021, at 12:41 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > > > On Thu, 19 Aug 2021, Rahul Singh wrote: > >> Implement generic pci access functions to read/write the configuration > >> space. > >> > >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> > >> --- > >> xen/arch/arm/pci/pci-access.c | 31 +++++++++++++++++++++++++++++- > >> xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++ > >> xen/include/asm-arm/pci.h | 2 ++ > >> 3 files changed, 51 insertions(+), 1 deletion(-) > >> > >> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c > >> index f39f6a3a38..b94de3c3ac 100644 > >> --- a/xen/arch/arm/pci/pci-access.c > >> +++ b/xen/arch/arm/pci/pci-access.c > >> @@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf, > >> static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg, > >> unsigned int len) > >> { > >> - return ~0U; > >> + uint32_t val = GENMASK(0, len * 8); > > > > This seems to be another default error value that it would be better to > > define with its own macro > > This default error is used once do you want to me define as macro. Yes. A macro is good even if you are going to use it once because it also serves as documentation for the error. For instance: /* PCI host bridge not found */ #define PCI_ERR_NOTFOUND(len) GENMASK(0, len * 8) really helps with the explanation of what the error is about. > >> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); > >> + > >> + if ( unlikely(!bridge) ) > >> + { > >> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", > >> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); > > > > You are not actually printing sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn ? > > Yes I am printing with “PRI_pci". Sorry I missed it! > >> + return val; > >> + } > >> + > >> + if ( unlikely(!bridge->ops->read) ) > >> + return val; > >> + > >> + bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val); > > > > Would it make sense to make the interface take a pci_sbdf_t directly > > instead of casting to uint32_t and back? > > pci_sbdf_t is defined in "xen/pci.h” and "xen/pci.h” includes "asm-arm/pci.h”. > If I modify the function argument in "asm-arm/pci.h” to use pci_sbdf_t I will get compilation error. This is unfortunate. One way around it is to make the appended change to xen/include/xen/pci.h and then simply add: typedef union pci_sbdf pci_sbdf_t; to xen/include/asm-arm/pci.h. Jan do you have any better suggestions? diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 8e3d4d9454..ae8d48135b 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -41,7 +41,7 @@ #define PCI_SBDF3(s,b,df) \ ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) }) -typedef union { +union pci_sbdf { uint32_t sbdf; struct { union { @@ -60,7 +60,9 @@ typedef union { }; uint16_t seg; }; -} pci_sbdf_t; +}; + +typedef union pci_sbdf pci_sbdf_t; struct pci_dev_info { /*
[-- Attachment #1: Type: text/plain, Size: 4482 bytes --] On Tue, 14 Sep 2021, Rahul Singh wrote: > >> + return NULL; > >> + > >> + busn -= cfg->busn_start; > >> + base = cfg->win + (busn << cfg->ops->bus_shift); > >> + > >> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; > >> +} > > > > I understand that the arm32 part is not implemented and not part of this > > series, that's fine. However if the plan is that arm32 will dynamically > > map each bus individually, then I imagine this function will have an > > ioremap in the arm32 version. Which means that we also need an > > unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus > > would be a NOP today for arm64, but I think it makes sense to have it if > > we want the API to be generic. > > As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the > bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part. > Let me know your view on this. In the patch titled "xen/arm: PCI host bridge discovery within XEN on ARM" there is the following in-code comment: * On 64-bit systems, we do a single ioremap for the whole config space * since we have enough virtual address range available. On 32-bit, we * ioremap the config space for each bus individually. * * As of now only 64-bit is supported 32-bit is not supported. So I take it that on arm32 we don't have enough virtual address range available, therefore we cannot ioremap the whole range. Instead, we'll have to ioremap the config space of each bus individually. I assumed that the idea was to call ioremap and iounmap dynamically, otherwise the total amount of virtual address range required would still be the same. I also thought that the dynamic mapping function, the one which would end up calling ioremap on arm32, would be pci_ecam_map_bus. If so, then we are missing the corresponding unmapping function, the one that would call iounmap on arm32 and do nothing on arm64, called before returning from pci_generic_config_read and pci_generic_config_write. As always, I am not asking for the arm32 implementation, but if we are introducing internal APIs, it would be good that they are consistent. > >> @@ -50,10 +51,41 @@ struct pci_host_bridge { > >> u8 bus_start; /* Bus start of this bridge. */ > >> u8 bus_end; /* Bus end of this bridge. */ > >> void *sysdata; /* Pointer to the config space window*/ > >> + const struct pci_ops *ops; > >> }; > >> > >> +struct pci_ops { > >> + void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf, > >> + uint32_t offset); > >> + int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf, > >> + uint32_t reg, uint32_t len, uint32_t *value); > >> + int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf, > >> + uint32_t reg, uint32_t len, uint32_t value); > >> +}; > >> + > >> +/* > >> + * struct to hold pci ops and bus shift of the config window > >> + * for a PCI controller. > >> + */ > >> +struct pci_ecam_ops { > >> + unsigned int bus_shift; > >> + struct pci_ops pci_ops; > >> + int (*init)(struct pci_config_window *); > > > > Is this last member of the struct needed/used? > > Yes It will be used by some vendor specific board( N1SDP ). Please check below. > https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/commit/?id=04b7e76d0fe6481a803f58e54e008a1489d713a5 OK. I don't doubt that there might be bridge-specific initializations, but we already have things like: DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI) .dt_match = gen_pci_dt_match, .init = gen_pci_dt_init, DT_DEVICE_END The question is do we actually need two different vendor-specific init functions? One is driven by device tree, e.g. ZynqMP is calling gen_pci_dt_init. The other one is pci_ecam_ops->init, which is called from the following chain: DT_DEVICE_START -> gen_pci_dt_init -> pci_host_common_probe -> gen_pci_init -> pci_generic_ecam_ops.init What's the difference between gen_pci_dt_init and pci_ecam_ops.init in terms of purpose? I am happy to have pci_ecam_ops.init if it serves a different purpose from DT_DEVICE_START.init. In that case we might want to add an in-code comment so that an engineer would know where to add vendor-specific code (whether to DT_DEVICE_START.init or to pci_ecam_ops.init).
Hi, Rahul!
>>> static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>>> unsigned int len, uint32_t val)
>>> {
>>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>> +
>>> + if ( unlikely(!bridge) )
>>> + {
>>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> same here
> Yes I am printing with “PRI_pci".
vPCI and the rest are widely using
printk("%pp\n", &sbdf);
So, I think if we have SBDF then it is better to use %pp instead of trying to unfold it manually.
Hi Oleksandr,
> On 15 Sep 2021, at 8:54 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
> Hi, Rahul!
>>>> static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>>>> unsigned int len, uint32_t val)
>>>> {
>>>> + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>>> +
>>>> + if ( unlikely(!bridge) )
>>>> + {
>>>> + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>>> + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>> same here
>> Yes I am printing with “PRI_pci".
>
> vPCI and the rest are widely using
>
> printk("%pp\n", &sbdf);
> So, I think if we have SBDF then it is better to use %pp instead of trying to unfold it manually.
Ok. I will use the %pp for printing the SBDF.
Regards,
Rahul
Hi Stefano, > On 15 Sep 2021, at 12:06 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Tue, 14 Sep 2021, Rahul Singh wrote: >>>> + return NULL; >>>> + >>>> + busn -= cfg->busn_start; >>>> + base = cfg->win + (busn << cfg->ops->bus_shift); >>>> + >>>> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; >>>> +} >>> >>> I understand that the arm32 part is not implemented and not part of this >>> series, that's fine. However if the plan is that arm32 will dynamically >>> map each bus individually, then I imagine this function will have an >>> ioremap in the arm32 version. Which means that we also need an >>> unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus >>> would be a NOP today for arm64, but I think it makes sense to have it if >>> we want the API to be generic. >> >> As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the >> bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part. >> Let me know your view on this. > > In the patch titled "xen/arm: PCI host bridge discovery within XEN on > ARM" there is the following in-code comment: > > * On 64-bit systems, we do a single ioremap for the whole config space > * since we have enough virtual address range available. On 32-bit, we > * ioremap the config space for each bus individually. > * > * As of now only 64-bit is supported 32-bit is not supported. > > > So I take it that on arm32 we don't have enough virtual address range > available, therefore we cannot ioremap the whole range. Instead, we'll > have to ioremap the config space of each bus individually. Yes you are right my understand is also same. > > I assumed that the idea was to call ioremap and iounmap dynamically, > otherwise the total amount of virtual address range required would still > be the same. As per my understanding for 32-bit we need per_bus mapping as we don’t have enough virtual address space in one chunk but we can have virtual address space in different chunk. I am not sure if we need to map/unmap the virtual address space for each read/write call. I just checked the Linux code[1] and there also mapping is done once not for each read/write call. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/ecam.c?id=ea4aae05974334e9837d86ff1cb716bad36b3ca8#n76 > I also thought that the dynamic mapping function, the one > which would end up calling ioremap on arm32, would be pci_ecam_map_bus. > If so, then we are missing the corresponding unmapping function, the one > that would call iounmap on arm32 and do nothing on arm64, called before > returning from pci_generic_config_read and pci_generic_config_write. > > As always, I am not asking for the arm32 implementation, but if we are > introducing internal APIs, it would be good that they are consistent. > > > >>>> @@ -50,10 +51,41 @@ struct pci_host_bridge { >>>> u8 bus_start; /* Bus start of this bridge. */ >>>> u8 bus_end; /* Bus end of this bridge. */ >>>> void *sysdata; /* Pointer to the config space window*/ >>>> + const struct pci_ops *ops; >>>> }; >>>> >>>> +struct pci_ops { >>>> + void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf, >>>> + uint32_t offset); >>>> + int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf, >>>> + uint32_t reg, uint32_t len, uint32_t *value); >>>> + int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf, >>>> + uint32_t reg, uint32_t len, uint32_t value); >>>> +}; >>>> + >>>> +/* >>>> + * struct to hold pci ops and bus shift of the config window >>>> + * for a PCI controller. >>>> + */ >>>> +struct pci_ecam_ops { >>>> + unsigned int bus_shift; >>>> + struct pci_ops pci_ops; >>>> + int (*init)(struct pci_config_window *); >>> >>> Is this last member of the struct needed/used? >> >> Yes It will be used by some vendor specific board( N1SDP ). Please check below. >> https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/commit/?id=04b7e76d0fe6481a803f58e54e008a1489d713a5 > > OK. I don't doubt that there might be bridge-specific initializations, > but we already have things like: > > DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI) > .dt_match = gen_pci_dt_match, > .init = gen_pci_dt_init, > DT_DEVICE_END > > The question is do we actually need two different vendor-specific init > functions? One is driven by device tree, e.g. ZynqMP is calling > gen_pci_dt_init. The other one is pci_ecam_ops->init, which is called > from the following chain: > > DT_DEVICE_START -> gen_pci_dt_init -> pci_host_common_probe -> > gen_pci_init -> pci_generic_ecam_ops.init > > What's the difference between gen_pci_dt_init and pci_ecam_ops.init in > terms of purpose? Yes I also agree with you we don’t need two init function. I will modify the code like below. DT_DEVICE_START ->pci_host_common_probe -> gen_pci_init -> pci_generic_ecam_ops.init Regards, Rahul > > I am happy to have pci_ecam_ops.init if it serves a different purpose > from DT_DEVICE_START.init. In that case we might want to add an in-code > comment so that an engineer would know where to add vendor-specific code > (whether to DT_DEVICE_START.init or to pci_ecam_ops.init).
[-- Attachment #1: Type: text/plain, Size: 3292 bytes --] On Wed, 15 Sep 2021, Rahul Singh wrote: > > On 15 Sep 2021, at 12:06 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Tue, 14 Sep 2021, Rahul Singh wrote: > >>>> + return NULL; > >>>> + > >>>> + busn -= cfg->busn_start; > >>>> + base = cfg->win + (busn << cfg->ops->bus_shift); > >>>> + > >>>> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; > >>>> +} > >>> > >>> I understand that the arm32 part is not implemented and not part of this > >>> series, that's fine. However if the plan is that arm32 will dynamically > >>> map each bus individually, then I imagine this function will have an > >>> ioremap in the arm32 version. Which means that we also need an > >>> unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus > >>> would be a NOP today for arm64, but I think it makes sense to have it if > >>> we want the API to be generic. > >> > >> As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the > >> bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part. > >> Let me know your view on this. > > > > In the patch titled "xen/arm: PCI host bridge discovery within XEN on > > ARM" there is the following in-code comment: > > > > * On 64-bit systems, we do a single ioremap for the whole config space > > * since we have enough virtual address range available. On 32-bit, we > > * ioremap the config space for each bus individually. > > * > > * As of now only 64-bit is supported 32-bit is not supported. > > > > > > So I take it that on arm32 we don't have enough virtual address range > > available, therefore we cannot ioremap the whole range. Instead, we'll > > have to ioremap the config space of each bus individually. > > Yes you are right my understand is also same. > > > > I assumed that the idea was to call ioremap and iounmap dynamically, > > otherwise the total amount of virtual address range required would still > > be the same. > > As per my understanding for 32-bit we need per_bus mapping as we don’t have enough virtual address space in one chunk > but we can have virtual address space in different chunk. Interesting. I would have assumed that the sum of all the individual smaller ioremaps would still be equal to one big ioremap. Maybe for Linux is different, but I don't think that many smaller ioremaps would buy us very much in Xen because it is the total ioremap virtual space that is too small. Or am I missing something? > I am not sure if we need to map/unmap the virtual address space for each read/write call. > I just checked the Linux code[1] and there also mapping is done once not for each read/write call. So my guess is that for arm32 we would have to resort to dynamic map/unmap for each read/write call, unless there is a trick with the individual smaller ioremaps that I haven't spotted (e.g. maybe something doesn't get mapped that way?) That said, given that we are uncertain about this and the arm32 implementation is nowhere close, I think that we are OK to continue like this for this series. Maybe you could add a couple of sentences to the in-code comment so that if somebody wants to jump in and implement arm32 support they would know where to start.
Hi Julien, > On 9 Sep 2021, at 2:50 pm, Julien Grall <julien@xen.org> wrote: > > Hi Rahul, > > On 19/08/2021 13:02, Rahul Singh wrote: >> The existing VPCI support available for X86 is adapted for Arm. >> When the device is added to XEN via the hyper call >> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space >> access is added to the Xen to emulate the PCI devices config space. >> A MMIO trap handler for the PCI ECAM space is registered in XEN >> so that when guest is trying to access the PCI config space,XEN >> will trap the access and emulate read/write using the VPCI and >> not the real PCI hardware. >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/Makefile | 1 + >> xen/arch/arm/domain.c | 4 ++ >> xen/arch/arm/vpci.c | 96 +++++++++++++++++++++++++++++++++++ >> xen/arch/arm/vpci.h | 37 ++++++++++++++ >> xen/drivers/passthrough/pci.c | 7 +++ >> xen/drivers/vpci/Makefile | 3 +- >> xen/drivers/vpci/header.c | 2 + >> xen/include/asm-arm/domain.h | 5 +- >> xen/include/asm-arm/pci.h | 8 +++ >> xen/include/public/arch-arm.h | 4 ++ >> 10 files changed, 165 insertions(+), 2 deletions(-) >> create mode 100644 xen/arch/arm/vpci.c >> create mode 100644 xen/arch/arm/vpci.h >> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile >> index 0e14a5e5c8..7cdce684a4 100644 >> --- a/xen/arch/arm/Makefile >> +++ b/xen/arch/arm/Makefile >> @@ -7,6 +7,7 @@ obj-y += platforms/ >> endif >> obj-$(CONFIG_TEE) += tee/ >> obj-$(CONFIG_HAS_PCI) += pci/ >> +obj-$(CONFIG_HAS_VPCI) += vpci.o >> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o >> obj-y += bootfdt.init.o >> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c >> index 19c756ac3d..d99c653626 100644 >> --- a/xen/arch/arm/domain.c >> +++ b/xen/arch/arm/domain.c >> @@ -40,6 +40,7 @@ >> #include <asm/vtimer.h> >> #include "vuart.h" >> +#include "vpci.h" > > Please order the includes alphabetically. So this one should go before "vuart.h". > >> DEFINE_PER_CPU(struct vcpu *, curr_vcpu); >> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d, >> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) ) >> goto fail; >> + if ( (rc = domain_vpci_init(d)) != 0 ) >> + goto fail; >> + >> return 0; >> fail: >> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c >> new file mode 100644 >> index 0000000000..da8b1ca13c >> --- /dev/null >> +++ b/xen/arch/arm/vpci.c >> @@ -0,0 +1,96 @@ >> +/* >> + * xen/arch/arm/vpci.c >> + * Copyright (c) 2021 Arm Ltd. >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License as published by >> + * the Free Software Foundation; either version 2 of the License, or >> + * (at your option) any later version. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + */ >> +#include <xen/sched.h> > > NIT: Please add a newline between generic and arch specific includes. Ack. > >> +#include <asm/mmio.h> >> + >> +/* Do some sanity checks. */ >> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len) >> +{ >> + /* Check access size. */ >> + if ( len != 1 && len != 2 && len != 4 && len != 8 ) >> + return false; > You will allow all the possible value of len (this is coming from the HW). So I feels this is a bit too much to check for every I/O to the vPCI. > > If you really want to keep the check then you can simply check that len is < 8 because the two callers compute it with (1 << S). So there is no way to set it 3, 5, 6 and 7. I will remove the check for 1,2,4,8 and will have check ( len < 8). > >> + >> + /* Check that access is size aligned. */ >> + if ( (reg & (len - 1)) ) >> + return false; >> + >> + return true; >> +} >> + >> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, >> + register_t *r, void *p) >> +{ >> + unsigned int reg; >> + pci_sbdf_t sbdf; >> + uint32_t data = 0; >> + unsigned int size = 1U << info->dabt.size; >> + >> + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); >> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); > > This logic is the same as below in vpci_mmio_write(). So I think you want to provide an helper because this is not trivial to read. Ok. > > Also, for the first line, I think you can re-use MMCFG_BDF() from the x86 code. For the second line, I would define the value so it is clearer to understand that they mean (although & 3 is fine to me) . Ok. > >> + >> + if ( !vpci_mmio_access_allowed(reg, size) ) >> + return 1; > So, you will a guest will read 0 if the access is unaligned. This seems an odd behavior given this is not an allowed access. AFAIU, the HW would likely trow a data abort because you can't do unalign access on uncachable memory. So I think we should return 0 here to let the MMIO handler inject a data abort. Ack. > >> + >> + data = vpci_read(sbdf, reg, size); > > So in vpci_mmio_access_allowed(), you will allow a guest to read a 64-bit value. But... vpci_read() will return a 32-bit value. > > Looking at the x86 code, they have a second call to vpci_read() to handle the top 32-bit. Any reason why this was not implemented on Arm? I missed that I will fix this in next version. > > If we need to implement it then I think this should be implement in vpci_read() to avoid duplication between x86 and arm. I think it will be good to make vpci_read() & vpci_write() simple not to add extra logic based on 64 bit value. I will add second call for vpci_read() and vpci_write() for ARM. > >> + >> + memcpy(r, &data, size); > > From my understanding, any unused bit should be 0. So this should be: > > *r = data; Ack. > >> + >> + return 1; >> +} >> + >> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, >> + register_t r, void *p) > > My remarks on vpci_mmio_read() applies here too. > >> +{ >> + unsigned int reg; >> + pci_sbdf_t sbdf; >> + uint32_t data = r; >> + unsigned int size = 1U << info->dabt.size; >> + >> + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); >> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); >> + >> + if ( !vpci_mmio_access_allowed(reg, size) ) >> + return 1; >> + >> + vpci_write(sbdf, reg, size, data); >> + >> + return 1; >> +} >> + >> +static const struct mmio_handler_ops vpci_mmio_handler = { >> + .read = vpci_mmio_read, >> + .write = vpci_mmio_write, >> +}; >> + >> +int domain_vpci_init(struct domain *d) >> +{ >> + if ( !has_vpci(d) ) >> + return 0; >> + >> + register_mmio_handler(d, &vpci_mmio_handler, >> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); >> + >> + return 0; >> +} >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> + >> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h >> new file mode 100644 >> index 0000000000..8a093bb705 >> --- /dev/null >> +++ b/xen/arch/arm/vpci.h >> @@ -0,0 +1,37 @@ >> +/* >> + * xen/arch/arm/vpci.h >> + * Copyright (c) 2021 Arm Ltd. >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License as published by >> + * the Free Software Foundation; either version 2 of the License, or >> + * (at your option) any later version. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + */ >> + >> +#ifndef __ARCH_ARM_VPCI_H__ >> +#define __ARCH_ARM_VPCI_H__ >> + >> +#ifdef CONFIG_HAS_VPCI >> +int domain_vpci_init(struct domain *d); >> +#else >> +static inline int domain_vpci_init(struct domain *d) >> +{ >> + return 0; >> +} >> +#endif >> + >> +#endif /* __ARCH_ARM_VPCI_H__ */ >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c >> index c23c8cb06b..56e261e9bd 100644 >> --- a/xen/drivers/passthrough/pci.c >> +++ b/xen/drivers/passthrough/pci.c >> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, >> else >> iommu_enable_device(pdev); >> +#ifdef CONFIG_ARM >> + ret = vpci_add_handlers(pdev); >> + if ( ret ) { >> + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret); >> + goto out; >> + } >> +#endif >> pci_enable_acs(pdev); >> out: >> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile >> index 55d1bdfda0..1a1413b93e 100644 >> --- a/xen/drivers/vpci/Makefile >> +++ b/xen/drivers/vpci/Makefile >> @@ -1 +1,2 @@ >> -obj-y += vpci.o header.o msi.o msix.o >> +obj-y += vpci.o header.o >> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o >> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c >> index ba9a036202..f8cd55e7c0 100644 >> --- a/xen/drivers/vpci/header.c >> +++ b/xen/drivers/vpci/header.c >> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, >> * FIXME: punching holes after the p2m has been set up might be racy for >> * DomU usage, needs to be revisited. >> */ >> +#ifdef CONFIG_HAS_PCI_MSI >> if ( map && !rom_only && vpci_make_msix_hole(pdev) ) >> return; >> +#endif >> for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) >> { >> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h >> index c9277b5c6d..d742b94bd6 100644 >> --- a/xen/include/asm-arm/domain.h >> +++ b/xen/include/asm-arm/domain.h >> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {} >> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag) >> -#define has_vpci(d) ({ (void)(d); false; }) >> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but >> + * for ARM we enable support VPCI for guest domain also. >> + */ >> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); }) >> #endif /* __ASM_DOMAIN_H__ */ >> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h >> index 756f8637ab..c58152de80 100644 >> --- a/xen/include/asm-arm/pci.h >> +++ b/xen/include/asm-arm/pci.h >> @@ -27,6 +27,14 @@ struct arch_pci_dev { >> struct device dev; >> }; >> +/* Arch-specific MSI data for vPCI. */ >> +struct vpci_arch_msi { >> +}; >> + >> +/* Arch-specific MSI-X entry data for vPCI. */ >> +struct vpci_arch_msix_entry { >> +}; >> + >> /* >> * struct to hold the mappings of a config space window. This >> * is expected to be used as sysdata for PCI controllers that >> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h >> index 64a2ca30da..0a9749e768 100644 >> --- a/xen/include/public/arch-arm.h >> +++ b/xen/include/public/arch-arm.h >> @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t; >> #define GUEST_PL011_BASE xen_mk_ullong(0x22000000) >> #define GUEST_PL011_SIZE xen_mk_ullong(0x00001000) >> +/* VPCI ECAM mappings */ >> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) > > All the values for the memory layout has been defined in ascending order. So please add the vCPI at the correct place. If I am not mistaken, this should be before the GUEST_ACPI_*. Ack. > >> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) > > Please document how to decide the size. This is important for the future if we need to change the size. Ok let me document in next version. /* * 256 MB is reserved for VPCI configuration space based on calculation * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB */ #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) Regrads, Rahul > > Cheers, > > -- > Julien Grall
Hi Stefano, > On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Thu, 19 Aug 2021, Rahul Singh wrote: >> The existing VPCI support available for X86 is adapted for Arm. >> When the device is added to XEN via the hyper call >> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space >> access is added to the Xen to emulate the PCI devices config space. > > This is done just for device discovery, right? > > Although it is currently not implemented (and I am not asking to > implement it now, I am only trying to understand the architecture), it > would be possible to discover all PCI devices just by walking down the > PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we > have an ECAM driver. > > I take that would be the way to implement PCI support for Dom0less? It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot. If boot firmware did the enumeration, XEN will discover the PCI device. > > >> A MMIO trap handler for the PCI ECAM space is registered in XEN >> so that when guest is trying to access the PCI config space,XEN >> will trap the access and emulate read/write using the VPCI and >> not the real PCI hardware. >> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> xen/arch/arm/Makefile | 1 + >> xen/arch/arm/domain.c | 4 ++ >> xen/arch/arm/vpci.c | 96 +++++++++++++++++++++++++++++++++++ >> xen/arch/arm/vpci.h | 37 ++++++++++++++ >> xen/drivers/passthrough/pci.c | 7 +++ >> xen/drivers/vpci/Makefile | 3 +- >> xen/drivers/vpci/header.c | 2 + >> xen/include/asm-arm/domain.h | 5 +- >> xen/include/asm-arm/pci.h | 8 +++ >> xen/include/public/arch-arm.h | 4 ++ >> 10 files changed, 165 insertions(+), 2 deletions(-) >> create mode 100644 xen/arch/arm/vpci.c >> create mode 100644 xen/arch/arm/vpci.h >> >> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile >> index 0e14a5e5c8..7cdce684a4 100644 >> --- a/xen/arch/arm/Makefile >> +++ b/xen/arch/arm/Makefile >> @@ -7,6 +7,7 @@ obj-y += platforms/ >> endif >> obj-$(CONFIG_TEE) += tee/ >> obj-$(CONFIG_HAS_PCI) += pci/ >> +obj-$(CONFIG_HAS_VPCI) += vpci.o >> >> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o >> obj-y += bootfdt.init.o >> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c >> index 19c756ac3d..d99c653626 100644 >> --- a/xen/arch/arm/domain.c >> +++ b/xen/arch/arm/domain.c >> @@ -40,6 +40,7 @@ >> #include <asm/vtimer.h> >> >> #include "vuart.h" >> +#include "vpci.h" >> >> DEFINE_PER_CPU(struct vcpu *, curr_vcpu); >> >> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d, >> if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) ) >> goto fail; >> >> + if ( (rc = domain_vpci_init(d)) != 0 ) >> + goto fail; >> + >> return 0; >> >> fail: >> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c >> new file mode 100644 >> index 0000000000..da8b1ca13c >> --- /dev/null >> +++ b/xen/arch/arm/vpci.c >> @@ -0,0 +1,96 @@ >> +/* >> + * xen/arch/arm/vpci.c >> + * Copyright (c) 2021 Arm Ltd. >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License as published by >> + * the Free Software Foundation; either version 2 of the License, or >> + * (at your option) any later version. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + */ >> +#include <xen/sched.h> >> +#include <asm/mmio.h> >> + >> +/* Do some sanity checks. */ >> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len) >> +{ >> + /* Check access size. */ >> + if ( len != 1 && len != 2 && len != 4 && len != 8 ) >> + return false; >> + >> + /* Check that access is size aligned. */ >> + if ( (reg & (len - 1)) ) >> + return false; >> + >> + return true; >> +} >> + >> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, >> + register_t *r, void *p) >> +{ >> + unsigned int reg; >> + pci_sbdf_t sbdf; >> + uint32_t data = 0; >> + unsigned int size = 1U << info->dabt.size; >> + >> + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); >> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); >> + >> + if ( !vpci_mmio_access_allowed(reg, size) ) >> + return 1; >> + >> + data = vpci_read(sbdf, reg, size); >> + >> + memcpy(r, &data, size); >> + >> + return 1; >> +} >> + >> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, >> + register_t r, void *p) >> +{ >> + unsigned int reg; >> + pci_sbdf_t sbdf; >> + uint32_t data = r; >> + unsigned int size = 1U << info->dabt.size; >> + >> + sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12); >> + reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3)); >> + >> + if ( !vpci_mmio_access_allowed(reg, size) ) >> + return 1; >> + >> + vpci_write(sbdf, reg, size, data); >> + >> + return 1; >> +} >> + >> +static const struct mmio_handler_ops vpci_mmio_handler = { >> + .read = vpci_mmio_read, >> + .write = vpci_mmio_write, >> +}; >> + >> +int domain_vpci_init(struct domain *d) >> +{ >> + if ( !has_vpci(d) ) >> + return 0; >> + >> + register_mmio_handler(d, &vpci_mmio_handler, >> + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); >> + >> + return 0; >> +} >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> + >> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h >> new file mode 100644 >> index 0000000000..8a093bb705 >> --- /dev/null >> +++ b/xen/arch/arm/vpci.h >> @@ -0,0 +1,37 @@ >> +/* >> + * xen/arch/arm/vpci.h >> + * Copyright (c) 2021 Arm Ltd. >> + * >> + * This program is free software; you can redistribute it and/or modify >> + * it under the terms of the GNU General Public License as published by >> + * the Free Software Foundation; either version 2 of the License, or >> + * (at your option) any later version. >> + * >> + * This program is distributed in the hope that it will be useful, >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> + * GNU General Public License for more details. >> + */ >> + >> +#ifndef __ARCH_ARM_VPCI_H__ >> +#define __ARCH_ARM_VPCI_H__ >> + >> +#ifdef CONFIG_HAS_VPCI >> +int domain_vpci_init(struct domain *d); >> +#else >> +static inline int domain_vpci_init(struct domain *d) >> +{ >> + return 0; >> +} >> +#endif >> + >> +#endif /* __ARCH_ARM_VPCI_H__ */ >> + >> +/* >> + * Local variables: >> + * mode: C >> + * c-file-style: "BSD" >> + * c-basic-offset: 4 >> + * indent-tabs-mode: nil >> + * End: >> + */ >> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c >> index c23c8cb06b..56e261e9bd 100644 >> --- a/xen/drivers/passthrough/pci.c >> +++ b/xen/drivers/passthrough/pci.c >> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, >> else >> iommu_enable_device(pdev); >> >> +#ifdef CONFIG_ARM >> + ret = vpci_add_handlers(pdev); >> + if ( ret ) { >> + printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret); >> + goto out; >> + } >> +#endif >> pci_enable_acs(pdev); >> >> out: >> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile >> index 55d1bdfda0..1a1413b93e 100644 >> --- a/xen/drivers/vpci/Makefile >> +++ b/xen/drivers/vpci/Makefile >> @@ -1 +1,2 @@ >> -obj-y += vpci.o header.o msi.o msix.o >> +obj-y += vpci.o header.o >> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o >> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c >> index ba9a036202..f8cd55e7c0 100644 >> --- a/xen/drivers/vpci/header.c >> +++ b/xen/drivers/vpci/header.c >> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, >> * FIXME: punching holes after the p2m has been set up might be racy for >> * DomU usage, needs to be revisited. >> */ >> +#ifdef CONFIG_HAS_PCI_MSI >> if ( map && !rom_only && vpci_make_msix_hole(pdev) ) >> return; >> +#endif >> >> for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) >> { >> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h >> index c9277b5c6d..d742b94bd6 100644 >> --- a/xen/include/asm-arm/domain.h >> +++ b/xen/include/asm-arm/domain.h >> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {} >> >> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag) >> >> -#define has_vpci(d) ({ (void)(d); false; }) >> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but >> + * for ARM we enable support VPCI for guest domain also. >> + */ >> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); }) >> >> #endif /* __ASM_DOMAIN_H__ */ >> >> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h >> index 756f8637ab..c58152de80 100644 >> --- a/xen/include/asm-arm/pci.h >> +++ b/xen/include/asm-arm/pci.h >> @@ -27,6 +27,14 @@ struct arch_pci_dev { >> struct device dev; >> }; >> >> +/* Arch-specific MSI data for vPCI. */ >> +struct vpci_arch_msi { >> +}; >> + >> +/* Arch-specific MSI-X entry data for vPCI. */ >> +struct vpci_arch_msix_entry { >> +}; >> + >> /* >> * struct to hold the mappings of a config space window. This >> * is expected to be used as sysdata for PCI controllers that >> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h >> index 64a2ca30da..0a9749e768 100644 >> --- a/xen/include/public/arch-arm.h >> +++ b/xen/include/public/arch-arm.h >> @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t; >> #define GUEST_PL011_BASE xen_mk_ullong(0x22000000) >> #define GUEST_PL011_SIZE xen_mk_ullong(0x00001000) >> >> +/* VPCI ECAM mappings */ >> +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) >> +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) > > Isn't 256MB a bit too small? Usually PCI comes with two ranges: > > - a small range below 4GB like this one > - an additional large range above 4GB, around 1GB or larger > > We are missing the second larger range? As per PCI specification PCI Express extends the Configuration Space to 4096 bytes( 4KB) per Function and based on calculation "256 buses × 32 devices × 8 functions × 4 KB = 256 MB”, 256 MB is sufficient as per my understaning. Regards, Rahul
Hi Julien, > On 9 Sep 2021, at 2:59 pm, Julien Grall <julien@xen.org> wrote: > > > > On 20/08/2021 17:03, Rahul Singh wrote: >> Hi Julien, > > Hi Rahul, > >>> On 19 Aug 2021, at 2:00 pm, Julien Grall <julien@xen.org> wrote: >>> >>> Hi Rahul, >>> >>> On 19/08/2021 13:02, Rahul Singh wrote: >>>> libxl will create an emulated PCI device tree node in the device tree to >>>> enable the guest OS to discover the virtual PCI during guest boot. >>>> Emulated PCI device tree node will only be created when there is any >>>> device assigned to guest. >>>> A new area has been reserved in the arm guest physical map at >>>> which the VPCI bus is declared in the device tree (reg and ranges >>>> parameters of the node). >>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >>>> --- >>>> tools/libs/light/libxl_arm.c | 109 ++++++++++++++++++++++++++ >>>> tools/libs/light/libxl_types.idl | 1 + >>>> tools/xl/xl_parse.c | 2 + >>>> xen/include/public/arch-arm.h | 11 +++ >>>> xen/include/public/device_tree_defs.h | 1 + >>>> 5 files changed, 124 insertions(+) >>>> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c >>>> index e3140a6e00..a091e97e76 100644 >>>> --- a/tools/libs/light/libxl_arm.c >>>> +++ b/tools/libs/light/libxl_arm.c >>>> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt, >>>> return fdt_property(fdt, "reg", regs, sizeof(regs)); >>>> } >>>> +static int fdt_property_values(libxl__gc *gc, void *fdt, >>>> + const char *name, unsigned num_cells, ...) >>>> +{ >>>> + uint32_t prop[num_cells]; >>>> + be32 *cells = &prop[0]; >>>> + int i; >>>> + va_list ap; >>>> + uint32_t arg; >>>> + >>>> + va_start(ap, num_cells); >>>> + for (i = 0 ; i < num_cells; i++) { >>>> + arg = va_arg(ap, uint32_t); >>>> + set_cell(&cells, 1, arg); >>>> + } >>>> + va_end(ap); >>>> + >>>> + return fdt_property(fdt, name, prop, sizeof(prop)); >>>> +} >>>> + >>>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt, >>>> + unsigned addr_cells, >>>> + unsigned size_cells, >>>> + unsigned num_regs, ...) >>>> +{ >>>> + uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)]; >>>> + be32 *cells = ®s[0]; >>>> + int i; >>>> + va_list ap; >>>> + uint64_t arg; >>>> + >>>> + va_start(ap, num_regs); >>>> + for (i = 0 ; i < num_regs; i++) { >>>> + /* Set the memory bit field */ >>>> + arg = va_arg(ap, uint64_t); >>>> + set_cell(&cells, 1, arg); >>>> + >>>> + /* Set the vpci bus address */ >>>> + arg = addr_cells ? va_arg(ap, uint64_t) : 0; >>>> + set_cell(&cells, addr_cells , arg); >>>> + >>>> + /* Set the cpu bus address where vpci address is mapped */ >>>> + set_cell(&cells, addr_cells, arg); >>>> + >>>> + /* Set the vpci size requested */ >>>> + arg = size_cells ? va_arg(ap, uint64_t) : 0; >>>> + set_cell(&cells, size_cells,arg); >>>> + } >>>> + va_end(ap); >>>> + >>>> + return fdt_property(fdt, "ranges", regs, sizeof(regs)); >>>> +} >>>> + >>>> static int make_root_properties(libxl__gc *gc, >>>> const libxl_version_info *vers, >>>> void *fdt) >>>> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, >>>> return 0; >>>> } >>>> +static int make_vpci_node(libxl__gc *gc, void *fdt, >>>> + const struct arch_info *ainfo, >>>> + struct xc_dom_image *dom) >>>> +{ >>>> + int res; >>>> + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE; >>>> + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE; >>>> + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base); >>>> + >>>> + res = fdt_begin_node(fdt, name); >>>> + if (res) return res; >>>> + >>>> + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic"); >>>> + if (res) return res; >>>> + >>>> + res = fdt_property_string(fdt, "device_type", "pci"); >>>> + if (res) return res; >>>> + >>>> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, >>>> + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size); >>>> + if (res) return res; >>>> + >>>> + res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17); >>> >>> AFAICT, the "bus-range" is optional. Can you explain why we need it? >> We need it to implement the function pci_ecam_map_bus(). > > Ok. Then why next question is what does the 17 mean? Is it tie to how we > implement the vPCI in Xen or the region we reserved? > Ah. I forgot to change it to 0xff. It should be 0xff. > [...] > >>> >>>> + >>>> if (b_info->type != LIBXL_DOMAIN_TYPE_PV) >>>> return; >>>> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl >>>> index 3f9fff653a..78b1ddf0b8 100644 >>>> --- a/tools/libs/light/libxl_types.idl >>>> +++ b/tools/libs/light/libxl_types.idl >>>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ >>>> ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), >>>> ("vuart", libxl_vuart_type), >>>> + ("vpci", libxl_defbool), >>> >>> Any new addition in the structure should be accompanied with a LIBXL_HAVE_* in the libxl.h header. >> OK. >>> >>>> ])), >>>> ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), >>>> ])), >>>> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c >>>> index 17dddb4cd5..ffafbeffb4 100644 >>>> --- a/tools/xl/xl_parse.c >>>> +++ b/tools/xl/xl_parse.c >>>> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source, >>>> } >>>> if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV) >>>> libxl_defbool_set(&b_info->u.pv.e820_host, true); >>>> + if (d_config->num_pcidevs) >>>> + libxl_defbool_set(&b_info->arch_arm.vpci, true); >>>> } >>>> if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) { >>>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h >>>> index 0a9749e768..01d13e669e 100644 >>>> --- a/xen/include/public/arch-arm.h >>>> +++ b/xen/include/public/arch-arm.h >>>> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t; >>>> #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) >>>> #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) >>>> +/* PCI-PCIe memory space types */ >>>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) >>>> +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) >>> >>> What the size of those regions? >> Non Prefetch Memory: Size 64 MB start at 512 MB >> Prefetch Memory: Size 128 GB start at 36 GB >>> >>>> + >>>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ >>>> +#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x900000000) >>>> +#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x20000000) >>> So far the memory layout defines the address in ascending order. So please add that after GUEST_RAM_BANK_BASES_*. >> Ok. >>> >>> However, if I am not mistaken that base address you provide will clash with RAM bank 1. It also seem to be pretty high which means that this will not work for 32-bit domain or on CPUs that don't allow offer large IPA bits. >> Yes I also checked that now that it is having clash with RAM bank 1. >> There is unused space is guest memory that we can use for Non Prefetch Memory as per below guest memory map. >> https://gitlab.com/xen-project/fusa/fusa-docs/-/blob/master/high-level/guest-memory-layout-arm.rst >> Proposed value: >> Non Prefetch Memory: Size 64 MB start at 0x22001000 >> Prefetch Memory: Size 4 GB start at 4 GB. > > The base address looks fine to me. However, the sizes are much smaller to what you initially suggested. Would you be able to clarify why the smaller sizes are fine? As per current design we can have 32 device in guest therefore I thought 64 MB is sufficient ( 2 MB per device ). > >>> >>> I think we need to start making the guest layout more dynamic. The VPCI memory space would have to go right after the end of the RAM allocated for a given guest. >>> >>>> + >>>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x2000000000) >>>> +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x04000000) >>> >>> It would be better if the size for each region is defined right after each base. >> OK. >>> >>> Also, how did you decide the size of each region? >> I thought 64 MB will be sufficient. I think it should be based on number of devices we can assign to the guest. > > We don't have to get the size right now. What I am more interested is to have a trace about how those values were decided (even if it just saying random). This will help to make any decision if in the future we need to resize (in particular downsize) the regions. As per Stefano suggestion in another mail he suggested 256 MB. I also think 256 MB will be sufficient ( 8 MB per device) . I will comment this in code. Regards, Rahul > > Cheers, > > -- > Julien Grall
Hi Stefano, > On 10 Sep 2021, at 1:51 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Thu, 19 Aug 2021, Rahul Singh wrote: >> libxl will create an emulated PCI device tree node in the device tree to >> enable the guest OS to discover the virtual PCI during guest boot. >> Emulated PCI device tree node will only be created when there is any >> device assigned to guest. >> >> A new area has been reserved in the arm guest physical map at >> which the VPCI bus is declared in the device tree (reg and ranges >> parameters of the node). >> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com> >> --- >> tools/libs/light/libxl_arm.c | 109 ++++++++++++++++++++++++++ >> tools/libs/light/libxl_types.idl | 1 + >> tools/xl/xl_parse.c | 2 + >> xen/include/public/arch-arm.h | 11 +++ >> xen/include/public/device_tree_defs.h | 1 + >> 5 files changed, 124 insertions(+) >> >> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c >> index e3140a6e00..a091e97e76 100644 >> --- a/tools/libs/light/libxl_arm.c >> +++ b/tools/libs/light/libxl_arm.c >> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt, >> return fdt_property(fdt, "reg", regs, sizeof(regs)); >> } >> >> +static int fdt_property_values(libxl__gc *gc, void *fdt, >> + const char *name, unsigned num_cells, ...) >> +{ >> + uint32_t prop[num_cells]; >> + be32 *cells = &prop[0]; >> + int i; >> + va_list ap; >> + uint32_t arg; >> + >> + va_start(ap, num_cells); >> + for (i = 0 ; i < num_cells; i++) { >> + arg = va_arg(ap, uint32_t); >> + set_cell(&cells, 1, arg); >> + } >> + va_end(ap); >> + >> + return fdt_property(fdt, name, prop, sizeof(prop)); >> +} >> + >> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt, >> + unsigned addr_cells, >> + unsigned size_cells, >> + unsigned num_regs, ...) >> +{ >> + uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)]; >> + be32 *cells = ®s[0]; >> + int i; >> + va_list ap; >> + uint64_t arg; >> + >> + va_start(ap, num_regs); >> + for (i = 0 ; i < num_regs; i++) { >> + /* Set the memory bit field */ >> + arg = va_arg(ap, uint64_t); >> + set_cell(&cells, 1, arg); > > Shouldn't this be uint32_t given that it is 1 cell exactly? Yes I will change it to uint32_t. > > >> + /* Set the vpci bus address */ >> + arg = addr_cells ? va_arg(ap, uint64_t) : 0; >> + set_cell(&cells, addr_cells , arg); >> + >> + /* Set the cpu bus address where vpci address is mapped */ >> + set_cell(&cells, addr_cells, arg); >> + >> + /* Set the vpci size requested */ >> + arg = size_cells ? va_arg(ap, uint64_t) : 0; >> + set_cell(&cells, size_cells,arg); > ^ space Ack. > > >> + } >> + va_end(ap); >> + >> + return fdt_property(fdt, "ranges", regs, sizeof(regs)); >> +} >> + >> static int make_root_properties(libxl__gc *gc, >> const libxl_version_info *vers, >> void *fdt) >> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, >> return 0; >> } >> >> +static int make_vpci_node(libxl__gc *gc, void *fdt, >> + const struct arch_info *ainfo, >> + struct xc_dom_image *dom) >> +{ >> + int res; >> + const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE; >> + const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE; >> + const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base); >> + >> + res = fdt_begin_node(fdt, name); >> + if (res) return res; >> + >> + res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic"); >> + if (res) return res; >> + >> + res = fdt_property_string(fdt, "device_type", "pci"); >> + if (res) return res; >> + >> + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, >> + GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size); >> + if (res) return res; >> + >> + res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17); > ^ space > Ack. > >> + if (res) return res; >> + >> + res = fdt_property_cell(fdt, "#address-cells", 3); >> + if (res) return res; >> + >> + res = fdt_property_cell(fdt, "#size-cells", 2); >> + if (res) return res; >> + >> + res = fdt_property_string(fdt, "status", "okay"); >> + if (res) return res; >> + >> + res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, >> + GUEST_ROOT_SIZE_CELLS, 2, >> + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE, >> + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR, >> + GUEST_VPCI_PREFETCH_MEM_SIZE); >> + if (res) return res; >> + >> + res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS, >> + 0, 0x10000); >> + if (res) return res; > > I agree with Julien that we shouldn't add it now if it is not working. Ack. > > One question: what about legacy interrupts? If they are supported, > shouldn't we have interrupts and interrupt-parent properties? As per current design legacy interrupt will not be supported. > > >> + res = fdt_end_node(fdt); >> + if (res) return res; >> + >> + return 0; >> +} >> + >> static const struct arch_info *get_arch_info(libxl__gc *gc, >> const struct xc_dom_image *dom) >> { >> @@ -971,6 +1074,9 @@ next_resize: >> if (info->tee == LIBXL_TEE_TYPE_OPTEE) >> FDT( make_optee_node(gc, fdt) ); >> >> + if (libxl_defbool_val(info->arch_arm.vpci)) >> + FDT( make_vpci_node(gc, fdt, ainfo, dom) ); >> + >> if (pfdt) >> FDT( copy_partial_fdt(gc, fdt, pfdt) ); >> >> @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc, >> /* ACPI is disabled by default */ >> libxl_defbool_setdefault(&b_info->acpi, false); >> >> + /* VPCI is disabled by default */ >> + libxl_defbool_setdefault(&b_info->arch_arm.vpci, false); >> + >> if (b_info->type != LIBXL_DOMAIN_TYPE_PV) >> return; >> >> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl >> index 3f9fff653a..78b1ddf0b8 100644 >> --- a/tools/libs/light/libxl_types.idl >> +++ b/tools/libs/light/libxl_types.idl >> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ >> >> ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), >> ("vuart", libxl_vuart_type), >> + ("vpci", libxl_defbool), >> ])), >> ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), >> ])), >> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c >> index 17dddb4cd5..ffafbeffb4 100644 >> --- a/tools/xl/xl_parse.c >> +++ b/tools/xl/xl_parse.c >> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source, >> } >> if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV) >> libxl_defbool_set(&b_info->u.pv.e820_host, true); >> + if (d_config->num_pcidevs) >> + libxl_defbool_set(&b_info->arch_arm.vpci, true); >> } >> >> if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) { >> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h >> index 0a9749e768..01d13e669e 100644 >> --- a/xen/include/public/arch-arm.h >> +++ b/xen/include/public/arch-arm.h >> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t; >> #define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) >> #define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) >> >> +/* PCI-PCIe memory space types */ >> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) >> +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) >> + >> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ >> +#define GUEST_VPCI_PREFETCH_MEM_ADDR xen_mk_ullong(0x900000000) >> +#define GUEST_VPCI_MEM_ADDR xen_mk_ullong(0x20000000) >> + >> +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x2000000000) >> +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x04000000) > > Now I get it: GUEST_VPCI_ECAM_BASE was just for the config space, not > the aperture. The apertures are starting at 0x20000000 and 0x900000000. Yes right. > > Speaking of which, isn't GUEST_VPCI_MEM_SIZE a bit too small? There > could be PCI devices with a <4GB requirement. Maybe 256MB? Yes I will allocate the 256 MB. Regards, Rahul
Hi Stefano,
> On 10 Sep 2021, at 2:00 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> If the property is not present in the device tree node for host bridge,
>> XEN while creating the dtb for hwdom will create this property and
>> assigns the already allocated segment to the host bridge
>> so that XEN and linux will have the same segment for the host bridges.
>>
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/domain_build.c | 18 ++++++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++
>> xen/include/asm-arm/pci.h | 3 +++
>> 3 files changed, 42 insertions(+)
>>
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index 6c86d52781..e0cf2ff19d 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -581,6 +581,24 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
>> return res;
>> }
>>
>> +#ifdef CONFIG_HAS_PCI
>> + if ( dt_device_type_is_equal(node, "pci") )
>> + {
>> + if ( !dt_find_property(node, "linux,pci-domain", NULL) )
>> + {
>> + uint16_t segment;
>> +
>> + res = pci_get_host_bridge_segment(node, &segment);
>> + if ( res < 0 )
>> + return res;
>> +
>> + res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment);
>> + if ( res )
>> + return res;
>> + }
>> + }
>> +#endif
>
> If param_pci_enable is false it might be possible that Xen didn't
> allocate a segment. In that case, we should just let Linux do whatever
> it wants in terms of segment allocation. So I think the code here should
> not return error if param_pci_enable is false.
> returning an error instead.
Ack. I will modify the code as per your request in next version.
Regards,
Rahul
Hi Stefano, > On 15 Sep 2021, at 9:45 pm, Stefano Stabellini <sstabellini@kernel.org> wrote: > > On Wed, 15 Sep 2021, Rahul Singh wrote: >>> On 15 Sep 2021, at 12:06 am, Stefano Stabellini <sstabellini@kernel.org> wrote: >>> On Tue, 14 Sep 2021, Rahul Singh wrote: >>>>>> + return NULL; >>>>>> + >>>>>> + busn -= cfg->busn_start; >>>>>> + base = cfg->win + (busn << cfg->ops->bus_shift); >>>>>> + >>>>>> + return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; >>>>>> +} >>>>> >>>>> I understand that the arm32 part is not implemented and not part of this >>>>> series, that's fine. However if the plan is that arm32 will dynamically >>>>> map each bus individually, then I imagine this function will have an >>>>> ioremap in the arm32 version. Which means that we also need an >>>>> unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus >>>>> would be a NOP today for arm64, but I think it makes sense to have it if >>>>> we want the API to be generic. >>>> >>>> As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the >>>> bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part. >>>> Let me know your view on this. >>> >>> In the patch titled "xen/arm: PCI host bridge discovery within XEN on >>> ARM" there is the following in-code comment: >>> >>> * On 64-bit systems, we do a single ioremap for the whole config space >>> * since we have enough virtual address range available. On 32-bit, we >>> * ioremap the config space for each bus individually. >>> * >>> * As of now only 64-bit is supported 32-bit is not supported. >>> >>> >>> So I take it that on arm32 we don't have enough virtual address range >>> available, therefore we cannot ioremap the whole range. Instead, we'll >>> have to ioremap the config space of each bus individually. >> >> Yes you are right my understand is also same. >>> >>> I assumed that the idea was to call ioremap and iounmap dynamically, >>> otherwise the total amount of virtual address range required would still >>> be the same. >> >> As per my understanding for 32-bit we need per_bus mapping as we don’t have enough virtual address space in one chunk >> but we can have virtual address space in different chunk. > > Interesting. I would have assumed that the sum of all the individual > smaller ioremaps would still be equal to one big ioremap. Maybe for > Linux is different, but I don't think that many smaller ioremaps would > buy us very much in Xen because it is the total ioremap virtual space > that is too small. Or am I missing something? > > >> I am not sure if we need to map/unmap the virtual address space for each read/write call. >> I just checked the Linux code[1] and there also mapping is done once not for each read/write call. > > So my guess is that for arm32 we would have to resort to dynamic > map/unmap for each read/write call, unless there is a trick with the > individual smaller ioremaps that I haven't spotted (e.g. maybe something > doesn't get mapped that way?) > > That said, given that we are uncertain about this and the arm32 > implementation is nowhere close, I think that we are OK to continue like > this for this series. Maybe you could add a couple of sentences to the > in-code comment so that if somebody wants to jump in and implement > arm32 support they would know where to start. I am ok with both ways adding comment in code to explain or implement the pci_ecam_add_bus(..) and pci_ecam_remove_bus() like Linux [1] and we can call those function in pci_read()/pci_write. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/ecam.c#n126 Regards, Rahul
[-- Attachment #1: Type: text/plain, Size: 1358 bytes --] On Thu, 16 Sep 2021, Rahul Singh wrote: > Hi Stefano, > > > On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > > > > On Thu, 19 Aug 2021, Rahul Singh wrote: > >> The existing VPCI support available for X86 is adapted for Arm. > >> When the device is added to XEN via the hyper call > >> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space > >> access is added to the Xen to emulate the PCI devices config space. > > > > This is done just for device discovery, right? > > > > Although it is currently not implemented (and I am not asking to > > implement it now, I am only trying to understand the architecture), it > > would be possible to discover all PCI devices just by walking down the > > PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we > > have an ECAM driver. > > > > I take that would be the way to implement PCI support for Dom0less? > > It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot. > If boot firmware did the enumeration, XEN will discover the PCI device. OK, but if the boot firmware does the enumeration, how will Xen discover the PCI device exactly? Will Xen discover it because the PCI device will be present in device tree explicitly (will have its own device tree node)? Or will Xen discover it by walking the PCI bus?
Hi Oleksandr, Stefano, > On 14 Sep 2021, at 5:31 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote: > > > On 14.09.21 00:02, Stefano Stabellini wrote: >> On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote: >>> On 10.09.21 15:01, Rahul Singh wrote: >>>> Hi Stefano, >>>> >>>>> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote: >>>>> >>>>> On Thu, 19 Aug 2021, Rahul Singh wrote: >>>>>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>>>>> >>>>>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config >>>>>> space to the XEN memory. >>>>>> >>>>>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>>>>> --- >>>>>> xen/arch/arm/pci/Makefile | 1 + >>>>>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++ >>>>>> 2 files changed, 60 insertions(+) >>>>>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c >>>>>> >>>>>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile >>>>>> index 6f32fbbe67..1d045ade01 100644 >>>>>> --- a/xen/arch/arm/pci/Makefile >>>>>> +++ b/xen/arch/arm/pci/Makefile >>>>>> @@ -3,3 +3,4 @@ obj-y += pci-access.o >>>>>> obj-y += pci-host-generic.o >>>>>> obj-y += pci-host-common.o >>>>>> obj-y += ecam.o >>>>>> +obj-y += pci-host-zynqmp.o >>>>>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c >>>>>> new file mode 100644 >>>>>> index 0000000000..fe103e3855 >>>>>> --- /dev/null >>>>>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c >>>>>> @@ -0,0 +1,59 @@ >>>>>> +/* >>>>>> + * Copyright (C) 2020-2021 EPAM Systems >>>>>> + * >>>>>> + * Based on Linux drivers/pci/controller/pci-host-common.c >>>>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c >>>>>> + * Based on xen/arch/arm/pci/pci-host-generic.c >>>>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com> >>>>> Only one Copyright line per file is enough :-) >>>>> >>>>> But actually all the Copyright lines with a name or a company name are >>>>> not really required or useful, as the copyright is noted in full details >>>>> in the commit messages (author and signed-off-by lines). I would remove >>>>> them all from the new files added by this series. >>>> Ok. Let me remove in next version. >>>>>> + * This program is free software; you can redistribute it and/or modify >>>>>> + * it under the terms of the GNU General Public License version 2 as >>>>>> + * published by the Free Software Foundation. >>>>>> + * >>>>>> + * This program is distributed in the hope that it will be useful, >>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >>>>>> + * GNU General Public License for more details. >>>>>> + * >>>>>> + * You should have received a copy of the GNU General Public License >>>>>> + * along with this program. If not, see <https://urldefense.com/v3/__http://www.gnu.org/licenses/__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_3j2CfflQ$ [gnu[.]org]>. >>>>>> + */ >>>>>> + >>>>>> +#include <asm/device.h> >>>>>> +#include <xen/pci.h> >>>>>> +#include <asm/pci.h> >>>>>> + >>>>>> +static const struct dt_device_match gen_pci_dt_match[] = { >>>>>> + { .compatible = "xlnx,nwl-pcie-2.11", >>>>>> + .data = &pci_generic_ecam_ops }, >>>>>> + { }, >>>>>> +}; >>>>>> + >>>>>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data) >>>>>> +{ >>>>>> + const struct dt_device_match *of_id; >>>>>> + const struct pci_ecam_ops *ops; >>>>>> + >>>>>> + of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node); >>>>> This should be superfluous >>>> Ack. I will remove the dt_match_node(..) in next version. >>> I am not entirely sure we need this patch at all as the main reason for its existence >>> >>> was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going >>> >>> to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP. >>> >>> So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one >>> >>> won't have it working... >>> >>> Please consider >>> >>> If we decide to remove it then >>> >>> int pci_host_common_probe(struct dt_device_node *dev, >>> const struct pci_ecam_ops *ops, >>> int ecam_reg_idx) >>> >>> doesn't need the last parameter. >> With my open source maintainer hat on, I don't see this patch as very >> important; from that point of view I'd be happy for it to be dropped. >> However, it would be good to have at least one non-default host bridge >> (doesn't have to be the Xilinx bridge), otherwise it becomes difficult >> to understand how the generic infrastructure introduced by this series >> could be useful. >> >> Moreover, your recent comment [1] made it even more evident that it >> would be good to have at least two different drivers to spot >> compatibility issues between them more easily. > > Don't take me wrong here ;) I still do use Xilinx QEMU for most of > > the tests, so it is good for me to have this patch in the tree. But, > > to be fair, Xilinx QEMU won't give you a possibility to see the fully > > functional system. This is why I say the patch can be dropped. > > If we add some words in the commit message about this and have the > > patch in the tree I'll be more than happy I will have this patch in my series and I will add more comments about the patch. Regards, Rahul > >> >> [1] https://urldefense.com/v3/__https://marc.info/?l=xen-devel&m=163154474008598__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_0bKs6zpA$ [marc[.]info]
Hi Stefano,
> On 16 Sep 2021, at 9:26 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> On Thu, 16 Sep 2021, Rahul Singh wrote:
>> Hi Stefano,
>>
>>> On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>
>>> On Thu, 19 Aug 2021, Rahul Singh wrote:
>>>> The existing VPCI support available for X86 is adapted for Arm.
>>>> When the device is added to XEN via the hyper call
>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>>> access is added to the Xen to emulate the PCI devices config space.
>>>
>>> This is done just for device discovery, right?
>>>
>>> Although it is currently not implemented (and I am not asking to
>>> implement it now, I am only trying to understand the architecture), it
>>> would be possible to discover all PCI devices just by walking down the
>>> PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we
>>> have an ECAM driver.
>>>
>>> I take that would be the way to implement PCI support for Dom0less?
>>
>> It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot.
>> If boot firmware did the enumeration, XEN will discover the PCI device.
>
> OK, but if the boot firmware does the enumeration, how will Xen discover
> the PCI device exactly? Will Xen discover it because the PCI device will
> be present in device tree explicitly (will have its own device tree
> node)? Or will Xen discover it by walking the PCI bus?
Yes Xen discover it by walking the PCI bus. There is already a function in XEN scan_pci_devices() that we can use
to discover the PCI device for dom0less system.
Regards,
Rahul
[-- Attachment #1: Type: text/plain, Size: 1948 bytes --] On Tue, 21 Sep 2021, Rahul Singh wrote: > Hi Stefano, > > > On 16 Sep 2021, at 9:26 pm, Stefano Stabellini <sstabellini@kernel.org> wrote: > > > > On Thu, 16 Sep 2021, Rahul Singh wrote: > >> Hi Stefano, > >> > >>> On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote: > >>> > >>> On Thu, 19 Aug 2021, Rahul Singh wrote: > >>>> The existing VPCI support available for X86 is adapted for Arm. > >>>> When the device is added to XEN via the hyper call > >>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space > >>>> access is added to the Xen to emulate the PCI devices config space. > >>> > >>> This is done just for device discovery, right? > >>> > >>> Although it is currently not implemented (and I am not asking to > >>> implement it now, I am only trying to understand the architecture), it > >>> would be possible to discover all PCI devices just by walking down the > >>> PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we > >>> have an ECAM driver. > >>> > >>> I take that would be the way to implement PCI support for Dom0less? > >> > >> It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot. > >> If boot firmware did the enumeration, XEN will discover the PCI device. > > > > OK, but if the boot firmware does the enumeration, how will Xen discover > > the PCI device exactly? Will Xen discover it because the PCI device will > > be present in device tree explicitly (will have its own device tree > > node)? Or will Xen discover it by walking the PCI bus? > > Yes Xen discover it by walking the PCI bus. There is already a function in XEN scan_pci_devices() that we can use > to discover the PCI device for dom0less system. OK. Please add a statement to the commit message on how dom0less support could be implemented. It doesn't have to be implemented now of course but it would be good to keep a note for future reference.