All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v1 00/14] PCI devices passthrough on Arm
@ 2021-08-19 12:02 Rahul Singh
  2021-08-19 12:02 ` [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN Rahul Singh
                   ` (13 more replies)
  0 siblings, 14 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Jan Beulich, Andrew Cooper,
	Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Julien Grall,
	Stefano Stabellini, Paul Durrant, Daniel De Graaf,
	Daniel P. Smith, Volodymyr Babchuk, Anthony PERARD,
	Juergen Gross

Hello All,

The purpose of this patch series is to add PCI passthrough support to Xen on
Arm. PCI passthrough support on ARM is the collaboration work between EPAM and
ARM. ARM submitted the partial RFC [1][2] last year to get early feedback. We
tried to fix all the comments and added more features to this patch series.

Working POC with all the features can be found at [3]. Working POC is tested
on x86 so that there will be no regression on x86. Design presentation can be
found at [4]

PCI passthrough support is divided into different patches. This patch series
includes following features: 

Preparatory work to implement the PCI passthrough support for the ARM:
- Refactor MSI code.
- Fixed compilation error when HAS_PCI enabled for ARM.

Discovering PCI Host Bridge in XEN:
- PCI init to initialize the PCI driver.
- PCI host bridge discovery in XEN and map the PCI ECAM configuration space to
  the XEN memory.
- PCI access functions.

Discovering PCI devices:
- To support the PCI passthrough, XEN should be aware of the PCI
  devices.
- Hardware domain is in charge of doing the PCI enumeration and will discover
  the PCI devices and then communicate to the XEN via a hypercall to add the
  PCI devices in XEN.

Enable the existing x86 virtual PCI support for ARM:
- Add VPCI trap handler for each of the PCI device added for config space
  access.
- Register the trap handler in XEN for each of the host bridge PCI ECAM config
  space access.

Emulated PCI device tree node in libxl:
- Create a virtual PCI device tree node in libxl to enable the guest OS to
  discover the virtual PCI during guest boot.

This patch series does not inlcude the following features. Following features
will be send for review in the next version of the patch series once initial
patch series merged.

- VPCI support for DOMU guests (Non-identity mappings guest view of the BARs) 
- Virtual bus topology implementation
- IOMMU related changes (generic, SMMUv2, SMMUv3)
- MSI support for DOMU guests.
- Virual ITS support for DOMU guests

[1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg01184.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2020-07/threads.html#01184
[3] https://gitlab.com/rahsingh/xen-integration/-/commits/pci-passthrough-upstream-all
[4] https://static.sched.com/hosted_files/xen2021/e4/PCI_Device_Passthrough_On_Arm.pdf

Oleksandr Andrushchenko (1):
  xen/arm: Add support for Xilinx ZynqMP PCI host controller

Rahul Singh (13):
  xen/pci: Refactor MSI code that implements MSI functionality within
    XEN
  xen/pci: solve compilation error on ARM with HAS_PCI enabled
  xen/pci: solve compilation error on ARM with ACPI && HAS_PCI
  xen/arm: Add support for PCI init to initialize the PCI driver.
  xen/arm: PCI host bridge discovery within XEN on ARM
  xen/arm: Add support for PCI ecam operations
  xen:arm: Implement pci access functions
  xen/arm: Add cmdline boot option "pci=on"
  xen/arm: Discovering PCI devices and add the PCI devices in XEN.
  xen/arm: Enable the existing x86 virtual PCI support for ARM.
  arm/libxl: Emulated PCI device tree node in libxl
  xen/arm: Fixed error when PCI device is assigned to guest
  xen/arm: Add linux,pci-domain property for hwdom if not available.

 tools/libs/light/libxl_arm.c          | 109 +++++++++
 tools/libs/light/libxl_types.idl      |   1 +
 tools/xl/xl_parse.c                   |   2 +
 xen/arch/arm/Makefile                 |   2 +
 xen/arch/arm/domain.c                 |   4 +
 xen/arch/arm/domain_build.c           |  18 ++
 xen/arch/arm/domctl.c                 |   2 +
 xen/arch/arm/pci/Makefile             |   6 +
 xen/arch/arm/pci/ecam.c               |  63 ++++++
 xen/arch/arm/pci/pci-access.c         | 143 ++++++++++++
 xen/arch/arm/pci/pci-host-common.c    | 312 ++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c   |  59 +++++
 xen/arch/arm/pci/pci-host-zynqmp.c    |  59 +++++
 xen/arch/arm/pci/pci.c                | 116 ++++++++++
 xen/arch/arm/physdev.c                |  42 +++-
 xen/arch/arm/vpci.c                   |  96 ++++++++
 xen/arch/arm/vpci.h                   |  37 +++
 xen/arch/x86/Kconfig                  |   1 +
 xen/drivers/passthrough/Makefile      |   1 +
 xen/drivers/passthrough/arm/iommu.c   |   5 +
 xen/drivers/passthrough/msi.c         |  96 ++++++++
 xen/drivers/passthrough/pci.c         |  63 ++----
 xen/drivers/pci/Kconfig               |   4 +
 xen/drivers/vpci/Makefile             |   3 +-
 xen/drivers/vpci/header.c             |   2 +
 xen/include/asm-arm/device.h          |   1 +
 xen/include/asm-arm/domain.h          |   5 +-
 xen/include/asm-arm/pci.h             | 106 ++++++++-
 xen/include/public/arch-arm.h         |  15 ++
 xen/include/public/device_tree_defs.h |   1 +
 xen/include/xen/msi.h                 |  56 +++++
 xen/xsm/flask/hooks.c                 |   8 +-
 32 files changed, 1384 insertions(+), 54 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/ecam.c
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c
 create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
 create mode 100644 xen/arch/arm/pci/pci.c
 create mode 100644 xen/arch/arm/vpci.c
 create mode 100644 xen/arch/arm/vpci.h
 create mode 100644 xen/drivers/passthrough/msi.c
 create mode 100644 xen/include/xen/msi.h

-- 
2.17.1



^ permalink raw reply	[flat|nested] 98+ messages in thread

* [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-19 12:18   ` Julien Grall
                     ` (2 more replies)
  2021-08-19 12:02 ` [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled Rahul Singh
                   ` (12 subsequent siblings)
  13 siblings, 3 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Jan Beulich, Andrew Cooper,
	Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Julien Grall,
	Stefano Stabellini, Paul Durrant, Daniel De Graaf,
	Daniel P. Smith

MSI code that implements MSI functionality to support MSI within XEN is
not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate
the code for ARM.

No functional change intended.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/x86/Kconfig             |  1 +
 xen/drivers/passthrough/Makefile |  1 +
 xen/drivers/passthrough/msi.c    | 96 ++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/pci.c    | 54 +++++-------------
 xen/drivers/pci/Kconfig          |  4 ++
 xen/include/xen/msi.h            | 56 +++++++++++++++++++
 xen/xsm/flask/hooks.c            |  8 +--
 7 files changed, 175 insertions(+), 45 deletions(-)
 create mode 100644 xen/drivers/passthrough/msi.c
 create mode 100644 xen/include/xen/msi.h

diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
index 9b164db641..7b46ee98c5 100644
--- a/xen/arch/x86/Kconfig
+++ b/xen/arch/x86/Kconfig
@@ -19,6 +19,7 @@ config X86
 	select HAS_NS16550
 	select HAS_PASSTHROUGH
 	select HAS_PCI
+	select HAS_PCI_MSI
 	select HAS_PDX
 	select HAS_SCHED_GRANULARITY
 	select HAS_UBSAN
diff --git a/xen/drivers/passthrough/Makefile b/xen/drivers/passthrough/Makefile
index 445690e3e5..a5efa22714 100644
--- a/xen/drivers/passthrough/Makefile
+++ b/xen/drivers/passthrough/Makefile
@@ -7,3 +7,4 @@ obj-y += iommu.o
 obj-$(CONFIG_HAS_PCI) += pci.o
 obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
 obj-$(CONFIG_HAS_PCI) += ats.o
+obj-$(CONFIG_HAS_PCI_MSI) += msi.o
diff --git a/xen/drivers/passthrough/msi.c b/xen/drivers/passthrough/msi.c
new file mode 100644
index 0000000000..15ad0f8160
--- /dev/null
+++ b/xen/drivers/passthrough/msi.c
@@ -0,0 +1,96 @@
+/*
+ * Copyright (C) 2008,  Netronome Systems, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/msi.h>
+#include <asm/hvm/io.h>
+
+int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)
+{
+    int rc;
+
+    if ( pdev->msix )
+    {
+        rc = pci_reset_msix_state(pdev);
+        if ( rc )
+            return rc;
+        msixtbl_init(d);
+    }
+
+    return 0;
+}
+
+int pdev_msi_init(struct pci_dev *pdev)
+{
+    unsigned int pos;
+
+    INIT_LIST_HEAD(&pdev->msi_list);
+
+    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
+                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI);
+    if ( pos )
+    {
+        uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos));
+
+        pdev->msi_maxvec = multi_msi_capable(ctrl);
+    }
+
+    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
+                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX);
+    if ( pos )
+    {
+        struct arch_msix *msix = xzalloc(struct arch_msix);
+        uint16_t ctrl;
+
+        if ( !msix )
+            return -ENOMEM;
+
+        spin_lock_init(&msix->table_lock);
+
+        ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos));
+        msix->nr_entries = msix_table_size(ctrl);
+
+        pdev->msix = msix;
+    }
+
+    return 0;
+}
+
+void pdev_msi_deinit(struct pci_dev *pdev)
+{
+    XFREE(pdev->msix);
+}
+
+void pdev_dump_msi(const struct pci_dev *pdev)
+{
+    const struct msi_desc *msi;
+
+    printk("- MSIs < ");
+    list_for_each_entry ( msi, &pdev->msi_list, list )
+        printk("%d ", msi->irq);
+    printk(">");
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index fc4fa2e5c3..67f5686ab6 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -32,8 +32,8 @@
 #include <xen/softirq.h>
 #include <xen/tasklet.h>
 #include <xen/vpci.h>
+#include <xen/msi.h>
 #include <xsm/xsm.h>
-#include <asm/msi.h>
 #include "ats.h"
 
 struct pci_seg {
@@ -314,6 +314,7 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
 {
     struct pci_dev *pdev;
     unsigned int pos;
+    int rc;
 
     list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
         if ( pdev->bus == bus && pdev->devfn == devfn )
@@ -327,35 +328,12 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
     *((u8*) &pdev->bus) = bus;
     *((u8*) &pdev->devfn) = devfn;
     pdev->domain = NULL;
-    INIT_LIST_HEAD(&pdev->msi_list);
-
-    pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
-                              PCI_CAP_ID_MSI);
-    if ( pos )
-    {
-        uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos));
-
-        pdev->msi_maxvec = multi_msi_capable(ctrl);
-    }
 
-    pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
-                              PCI_CAP_ID_MSIX);
-    if ( pos )
+    rc = pdev_msi_init(pdev);
+    if ( rc )
     {
-        struct arch_msix *msix = xzalloc(struct arch_msix);
-        uint16_t ctrl;
-
-        if ( !msix )
-        {
-            xfree(pdev);
-            return NULL;
-        }
-        spin_lock_init(&msix->table_lock);
-
-        ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos));
-        msix->nr_entries = msix_table_size(ctrl);
-
-        pdev->msix = msix;
+        xfree(pdev);
+        return NULL;
     }
 
     list_add(&pdev->alldevs_list, &pseg->alldevs_list);
@@ -449,7 +427,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev *pdev)
     }
 
     list_del(&pdev->alldevs_list);
-    xfree(pdev->msix);
+    pdev_msi_deinit(pdev);
     xfree(pdev);
 }
 
@@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev)
 static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
 {
     struct pci_dev *pdev;
-    struct msi_desc *msi;
 
     printk("==== segment %04x ====\n", pseg->nr);
 
     list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
     {
-        printk("%pp - %pd - node %-3d - MSIs < ",
+        printk("%pp - %pd - node %-3d ",
                &pdev->sbdf, pdev->domain,
                (pdev->node != NUMA_NO_NODE) ? pdev->node : -1);
-        list_for_each_entry ( msi, &pdev->msi_list, list )
-               printk("%d ", msi->irq);
-        printk(">\n");
+        pdev_dump_msi(pdev);
+        printk("\n");
     }
 
     return 0;
@@ -1422,13 +1398,9 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)
     ASSERT(pdev && (pdev->domain == hardware_domain ||
                     pdev->domain == dom_io));
 
-    if ( pdev->msix )
-    {
-        rc = pci_reset_msix_state(pdev);
-        if ( rc )
-            goto done;
-        msixtbl_init(d);
-    }
+    rc = pdev_msix_assign(d, pdev);
+    if ( rc )
+        goto done;
 
     pdev->fault.count = 0;
 
diff --git a/xen/drivers/pci/Kconfig b/xen/drivers/pci/Kconfig
index 7da03fa13b..c6a7bc8007 100644
--- a/xen/drivers/pci/Kconfig
+++ b/xen/drivers/pci/Kconfig
@@ -1,3 +1,7 @@
 
 config HAS_PCI
 	bool
+
+config HAS_PCI_MSI
+	bool
+	depends on HAS_PCI
diff --git a/xen/include/xen/msi.h b/xen/include/xen/msi.h
new file mode 100644
index 0000000000..b2d5bc6f9d
--- /dev/null
+++ b/xen/include/xen/msi.h
@@ -0,0 +1,56 @@
+/*
+ * Copyright (C) 2008,  Netronome Systems, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XEN_MSI_H_
+#define __XEN_MSI_H_
+
+#ifdef CONFIG_HAS_PCI_MSI
+
+#include <asm/msi.h>
+
+int pdev_msix_assign(struct domain *d, struct pci_dev *pdev);
+int pdev_msi_init(struct pci_dev *pdev);
+void pdev_msi_deinit(struct pci_dev *pdev);
+void pdev_dump_msi(const struct pci_dev *pdev);
+
+#else /* !CONFIG_HAS_PCI_MSI */
+static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)
+{
+    return 0;
+}
+
+static inline int pdev_msi_init(struct pci_dev *pdev)
+{
+    return 0;
+}
+
+static inline void pdev_msi_deinit(struct pci_dev *pdev) {}
+static inline void pci_cleanup_msi(struct pci_dev *pdev) {}
+static inline void pdev_dump_msi(const struct pci_dev *pdev) {}
+
+#endif /* CONFIG_HAS_PCI_MSI */
+
+#endif /* __XEN_MSI_H */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index f1a1217c98..fdcfeb984c 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -21,7 +21,7 @@
 #include <xen/guest_access.h>
 #include <xen/xenoprof.h>
 #include <xen/iommu.h>
-#ifdef CONFIG_HAS_PCI
+#ifdef CONFIG_HAS_PCI_MSI
 #include <asm/msi.h>
 #endif
 #include <public/xen.h>
@@ -114,7 +114,7 @@ static int get_irq_sid(int irq, u32 *sid, struct avc_audit_data *ad)
         }
         return security_irq_sid(irq, sid);
     }
-#ifdef CONFIG_HAS_PCI
+#ifdef CONFIG_HAS_PCI_MSI
     {
         struct irq_desc *desc = irq_to_desc(irq);
         if ( desc->msi_desc && desc->msi_desc->dev ) {
@@ -874,7 +874,7 @@ static int flask_map_domain_pirq (struct domain *d)
 static int flask_map_domain_msi (struct domain *d, int irq, const void *data,
                                  u32 *sid, struct avc_audit_data *ad)
 {
-#ifdef CONFIG_HAS_PCI
+#ifdef CONFIG_HAS_PCI_MSI
     const struct msi_info *msi = data;
     u32 machine_bdf = (msi->seg << 16) | (msi->bus << 8) | msi->devfn;
 
@@ -940,7 +940,7 @@ static int flask_unmap_domain_pirq (struct domain *d)
 static int flask_unmap_domain_msi (struct domain *d, int irq, const void *data,
                                    u32 *sid, struct avc_audit_data *ad)
 {
-#ifdef CONFIG_HAS_PCI
+#ifdef CONFIG_HAS_PCI_MSI
     const struct pci_dev *pdev = data;
     u32 machine_bdf = (pdev->seg << 16) | (pdev->bus << 8) | pdev->devfn;
 
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
  2021-08-19 12:02 ` [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-19 12:28   ` Julien Grall
                     ` (2 more replies)
  2021-08-19 12:02 ` [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI Rahul Singh
                   ` (11 subsequent siblings)
  13 siblings, 3 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

Compilation error is observed when HAS_PCI is enabled for ARM
architecture.

Add definition for arch_iommu_use_permitted() and
arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to
fix compilation error.

pci.c: In function ‘deassign_device’:
pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’;
did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration]
            pci_to_dev(pdev));

pci.c:18: undefined reference to `pci_conf_read16’
pci.c:880: undefined reference to `arch_pci_clean_pirqs’
pci.c:1392: undefined reference to `arch_iommu_use_permitted'

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/Makefile               |  1 +
 xen/arch/arm/pci/Makefile           |  2 +
 xen/arch/arm/pci/pci-access.c       | 61 +++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci.c              | 32 +++++++++++++++
 xen/drivers/passthrough/arm/iommu.c |  5 +++
 xen/include/asm-arm/pci.h           | 33 ++++++++++++++--
 6 files changed, 131 insertions(+), 3 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci.c

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 3d3b97b5b4..0e14a5e5c8 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_HAS_PCI) += pci/
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
new file mode 100644
index 0000000000..a9ee0b9b44
--- /dev/null
+++ b/xen/arch/arm/pci/Makefile
@@ -0,0 +1,2 @@
+obj-y += pci.o
+obj-y += pci-access.o
diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
new file mode 100644
index 0000000000..b938047c03
--- /dev/null
+++ b/xen/arch/arm/pci/pci-access.c
@@ -0,0 +1,61 @@
+/*
+ * Copyright (C) 2021 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/pci.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+                                unsigned int len)
+{
+    return ~0U;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+                             unsigned int len, uint32_t val)
+{
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type)                                            \
+    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val)  \
+{                                                                           \
+    pci_config_write(sbdf, reg, size / 8, val);                             \
+}
+
+#define PCI_OP_READ(size, type)                                             \
+    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)            \
+{                                                                           \
+    return pci_config_read(sbdf, reg, size / 8);                            \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
new file mode 100644
index 0000000000..dc55d23778
--- /dev/null
+++ b/xen/arch/arm/pci/pci.c
@@ -0,0 +1,32 @@
+/*
+ * Copyright (C) 2021 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/pci.h>
+
+int arch_pci_clean_pirqs(struct domain *d)
+{
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c
index db3b07a571..fdec1c5547 100644
--- a/xen/drivers/passthrough/arm/iommu.c
+++ b/xen/drivers/passthrough/arm/iommu.c
@@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d)
 void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
 {
 }
+
+bool arch_iommu_use_permitted(const struct domain *d)
+{
+    return true;
+}
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index de13359f65..61e43da088 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -1,7 +1,34 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2021 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+
+#ifdef CONFIG_HAS_PCI
+
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/* Arch pci dev struct */
 struct arch_pci_dev {
+    struct device dev;
 };
 
-#endif /* __X86_PCI_H__ */
+#else   /*!CONFIG_HAS_PCI*/
+
+struct arch_pci_dev { };
+
+#endif  /*!CONFIG_HAS_PCI*/
+#endif /* __ARM_PCI_H__ */
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
  2021-08-19 12:02 ` [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN Rahul Singh
  2021-08-19 12:02 ` [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-20  7:06   ` Jan Beulich
  2021-08-19 12:02 ` [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver Rahul Singh
                   ` (10 subsequent siblings)
  13 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel; +Cc: bertrand.marquis, rahul.singh, Jan Beulich, Paul Durrant

Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
architecture. Move the code under CONFIG_X86 flag to gate the code for
ARM.

No functional change intended.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/drivers/passthrough/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 67f5686ab6..c23c8cb06b 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -1150,7 +1150,7 @@ void __hwdom_init setup_hwdom_pci_devices(
     pcidevs_unlock();
 }
 
-#ifdef CONFIG_ACPI
+#if defined(CONFIG_ACPI) && defined(CONFIG_X86)
 #include <acpi/acpi.h>
 #include <acpi/apei.h>
 
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver.
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (2 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-09-07  8:20   ` Julien Grall
  2021-09-09  1:16   ` Stefano Stabellini
  2021-08-19 12:02 ` [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
                   ` (9 subsequent siblings)
  13 siblings, 2 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

pci_init(..) will be called during xen startup to initialize and probe
the PCI host-bridge driver.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/pci/pci.c       | 54 ++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/device.h |  1 +
 2 files changed, 55 insertions(+)

diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index dc55d23778..d1c9cf997d 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -14,13 +14,67 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
 #include <xen/pci.h>
+#include <xen/param.h>
 
 int arch_pci_clean_pirqs(struct domain *d)
 {
     return 0;
 }
 
+static int __init dt_pci_init(void)
+{
+    struct dt_device_node *np;
+    int rc;
+
+    dt_for_each_device_node(dt_host, np)
+    {
+        rc = device_init(np, DEVICE_PCI, NULL);
+        if( !rc )
+            continue;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current is not an pci
+         *   - ENODEV: The pci device is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+        {
+            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+    printk(XENLOG_ERR "ACPI pci init not supported \n");
+    return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static int __init pci_init(void)
+{
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();
+
+    pci_segments_init();
+
+    return 0;
+}
+__initcall(pci_init);
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
index ee7cff2d44..5ecd5e7bd1 100644
--- a/xen/include/asm-arm/device.h
+++ b/xen/include/asm-arm/device.h
@@ -34,6 +34,7 @@ enum device_class
     DEVICE_SERIAL,
     DEVICE_IOMMU,
     DEVICE_GIC,
+    DEVICE_PCI,
     /* Use for error */
     DEVICE_UNKNOWN,
 };
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (3 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-09-07  9:05   ` Julien Grall
                     ` (3 more replies)
  2021-08-19 12:02 ` [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations Rahul Singh
                   ` (8 subsequent siblings)
  13 siblings, 4 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

XEN during boot will read the PCI device tree node “reg” property
and will map the PCI config space to the XEN memory.

As of now "pci-host-ecam-generic" compatible board is supported.

"linux,pci-domain" device tree property assigns a fixed PCI domain
number to a host bridge, otherwise an unstable (across boots) unique
number will be assigned by Linux.This property has to be in sync with
XEN to access the PCI devices.

XEN will read the “linux,pci-domain” property from the device tree node
and configure the host bridge segment number accordingly. If this
property is not available XEN will allocate the unique segment number
to the host bridge.

dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly
imported from the Linux source tree.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/pci/Makefile           |   2 +
 xen/arch/arm/pci/pci-host-common.c  | 261 ++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c |  55 ++++++
 xen/include/asm-arm/pci.h           |  28 +++
 4 files changed, 346 insertions(+)
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c

diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
index a9ee0b9b44..f3d97f859e 100644
--- a/xen/arch/arm/pci/Makefile
+++ b/xen/arch/arm/pci/Makefile
@@ -1,2 +1,4 @@
 obj-y += pci.o
 obj-y += pci-access.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
new file mode 100644
index 0000000000..9dd9b02271
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -0,0 +1,261 @@
+/*
+ * Copyright (C) 2021 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/sched.h>
+#include <xen/vmap.h>
+
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static atomic_t domain_nr = ATOMIC_INIT(-1);
+
+bool dt_pci_parse_bus_range(struct dt_device_node *dev,
+                            struct pci_config_window *cfg)
+{
+    const __be32 *cells;
+    uint32_t len;
+
+    cells = dt_get_property(dev, "bus-range", &len);
+    /* bus-range should at least be 2 cells */
+    if ( !cells || (len < (sizeof(*cells) * 2)) )
+        return false;
+
+    cfg->busn_start = dt_next_cell(1, &cells);
+    cfg->busn_end = dt_next_cell(1, &cells);
+
+    return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+    return ioremap_nocache(start, len);
+}
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+    if ( cfg->win )
+        iounmap(cfg->win);
+
+    xfree(cfg);
+}
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+                                              int ecam_reg_idx)
+{
+    int err;
+    struct pci_config_window *cfg;
+    paddr_t addr, size;
+
+    cfg = xzalloc(struct pci_config_window);
+    if ( !cfg )
+        return NULL;
+
+    err = dt_pci_parse_bus_range(dev, cfg);
+    if ( !err ) {
+        cfg->busn_start = 0;
+        cfg->busn_end = 0xff;
+        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
+               dt_node_full_name(dev));
+    } else {
+        if ( cfg->busn_end > cfg->busn_start + 0xff )
+            cfg->busn_end = cfg->busn_start + 0xff;
+    }
+
+    /* Parse our PCI ecam register address*/
+    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
+    if ( err )
+        goto err_exit;
+
+    cfg->phys_addr = addr;
+    cfg->size = size;
+
+    /*
+     * On 64-bit systems, we do a single ioremap for the whole config space
+     * since we have enough virtual address range available.  On 32-bit, we
+     * ioremap the config space for each bus individually.
+     *
+     * As of now only 64-bit is supported 32-bit is not supported.
+     */
+    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
+    if ( !cfg->win )
+        goto err_exit_remap;
+
+    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+            cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
+
+    return cfg;
+
+err_exit_remap:
+    printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+    pci_ecam_free(cfg);
+    return NULL;
+}
+
+struct pci_host_bridge *pci_alloc_host_bridge(void)
+{
+    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+    if ( !bridge )
+        return NULL;
+
+    INIT_LIST_HEAD(&bridge->node);
+    bridge->bus_start = ~0;
+    bridge->bus_end = ~0;
+    return bridge;
+}
+
+void pci_add_host_bridge(struct pci_host_bridge *bridge)
+{
+    list_add_tail(&bridge->node, &pci_host_bridges);
+}
+
+static int pci_get_new_domain_nr(void)
+{
+    return atomic_inc_return(&domain_nr);
+}
+
+/*
+ * This function will try to obtain the host bridge domain number by
+ * finding a property called "linux,pci-domain" of the given device node.
+ *
+ * @node: device tree node with the domain information
+ *
+ * Returns the associated domain number from DT in the range [0-0xffff], or
+ * a negative value if the required property is not found.
+ */
+static int dt_get_pci_domain_nr(struct dt_device_node *node)
+{
+    u32 domain;
+    int error;
+
+    error = dt_property_read_u32(node, "linux,pci-domain", &domain);
+    if ( !error )
+        return -EINVAL;
+
+    return (u16)domain;
+}
+
+static int pci_bus_find_domain_nr(struct dt_device_node *dev)
+{
+    static int use_dt_domains = -1;
+    int domain;
+
+    domain = dt_get_pci_domain_nr(dev);
+
+    /*
+     * Check DT domain and use_dt_domains values.
+     *
+     * If DT domain property is valid (domain >= 0) and
+     * use_dt_domains != 0, the DT assignment is valid since this means
+     * we have not previously allocated a domain number by using
+     * pci_get_new_domain_nr(); we should also update use_dt_domains to
+     * 1, to indicate that we have just assigned a domain number from
+     * DT.
+     *
+     * If DT domain property value is not valid (ie domain < 0), and we
+     * have not previously assigned a domain number from DT
+     * (use_dt_domains != 1) we should assign a domain number by
+     * using the:
+     *
+     * pci_get_new_domain_nr()
+     *
+     * API and update the use_dt_domains value to keep track of method we
+     * are using to assign domain numbers (use_dt_domains = 0).
+     *
+     * All other combinations imply we have a platform that is trying
+     * to mix domain numbers obtained from DT and pci_get_new_domain_nr(),
+     * which is a recipe for domain mishandling and it is prevented by
+     * invalidating the domain value (domain = -1) and printing a
+     * corresponding error.
+     */
+    if ( domain >= 0 && use_dt_domains )
+    {
+        use_dt_domains = 1;
+    }
+    else if ( domain < 0 && use_dt_domains != 1 )
+    {
+        use_dt_domains = 0;
+        domain = pci_get_new_domain_nr();
+    }
+    else
+    {
+        printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n");
+        BUG();
+    }
+
+    return domain;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+                          int ecam_reg_idx)
+{
+    struct pci_host_bridge *bridge;
+    struct pci_config_window *cfg;
+    int err;
+
+    bridge = pci_alloc_host_bridge();
+    if ( !bridge )
+        return -ENOMEM;
+
+    /* Parse and map our Configuration Space windows */
+    cfg = gen_pci_init(dev, ecam_reg_idx);
+    if ( !cfg )
+    {
+        err = -ENOMEM;
+        goto err_exit;
+    }
+
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->bus_start = cfg->busn_start;
+    bridge->bus_end = cfg->busn_end;
+
+    bridge->segment = pci_bus_find_domain_nr(dev);
+
+    pci_add_host_bridge(bridge);
+
+    return 0;
+
+err_exit:
+    xfree(bridge);
+    return err;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
new file mode 100644
index 0000000000..13d0f7f999
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-generic.c
@@ -0,0 +1,55 @@
+/*
+ * Copyright (C) 2021 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <asm/device.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "pci-host-ecam-generic" },
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+           dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, 0);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 61e43da088..58a51e724e 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -26,6 +26,34 @@ struct arch_pci_dev {
     struct device dev;
 };
 
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+    paddr_t         phys_addr;
+    paddr_t         size;
+    uint8_t         busn_start;
+    uint8_t         busn_end;
+    void __iomem    *win;
+};
+
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
+    struct list_head node;           /* Node in list of host bridges */
+    uint16_t segment;                /* Segment number */
+    u8 bus_start;                    /* Bus start of this bridge. */
+    u8 bus_end;                      /* Bus end of this bridge. */
+    void *sysdata;                   /* Pointer to the config space window*/
+};
+
+int pci_host_common_probe(struct dt_device_node *dev,
+                          int ecam_reg_idx);
 #else   /*!CONFIG_HAS_PCI*/
 
 struct arch_pci_dev { };
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (4 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-09-09 11:32   ` Julien Grall
  2021-09-09 23:21   ` Stefano Stabellini
  2021-08-19 12:02 ` [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh
                   ` (7 subsequent siblings)
  13 siblings, 2 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

Add support for PCI ecam operations to access the PCI
configuration space.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/pci/Makefile           |  1 +
 xen/arch/arm/pci/ecam.c             | 63 +++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-access.c       | 53 ++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 13 +++++-
 xen/arch/arm/pci/pci-host-generic.c |  8 +++-
 xen/include/asm-arm/pci.h           | 32 +++++++++++++++
 6 files changed, 167 insertions(+), 3 deletions(-)
 create mode 100644 xen/arch/arm/pci/ecam.c

diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
index f3d97f859e..6f32fbbe67 100644
--- a/xen/arch/arm/pci/Makefile
+++ b/xen/arch/arm/pci/Makefile
@@ -2,3 +2,4 @@ obj-y += pci.o
 obj-y += pci-access.o
 obj-y += pci-host-generic.o
 obj-y += pci-host-common.o
+obj-y += ecam.o
diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c
new file mode 100644
index 0000000000..91c691b41f
--- /dev/null
+++ b/xen/arch/arm/pci/ecam.c
@@ -0,0 +1,63 @@
+/*
+ * Copyright (C) 2021 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/pci.h>
+#include <xen/sched.h>
+
+/*
+ * Function to implement the pci_ops ->map_bus method.
+ */
+void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
+                                      uint32_t sbdf, uint32_t where)
+{
+    const struct pci_config_window *cfg = bridge->sysdata;
+    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+    void __iomem *base;
+
+    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+    unsigned int busn = sbdf_t.bus;
+
+    if ( busn < cfg->busn_start || busn > cfg->busn_end )
+        return NULL;
+
+    busn -= cfg->busn_start;
+    base = cfg->win + (busn << cfg->ops->bus_shift);
+
+    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
+/* ECAM ops */
+const struct pci_ecam_ops pci_generic_ecam_ops = {
+    .bus_shift  = 20,
+    .pci_ops    = {
+        .map_bus                = pci_ecam_map_bus,
+        .read                   = pci_generic_config_read,
+        .write                  = pci_generic_config_write,
+    }
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
index b938047c03..f39f6a3a38 100644
--- a/xen/arch/arm/pci/pci-access.c
+++ b/xen/arch/arm/pci/pci-access.c
@@ -15,6 +15,59 @@
  */
 
 #include <xen/pci.h>
+#include <asm/io.h>
+
+int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+                            uint32_t reg, uint32_t len, uint32_t *value)
+{
+    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
+    if (!addr) {
+        *value = ~0;
+        return -ENODEV;
+    }
+
+    switch (len)
+    {
+    case 1:
+        *value = readb(addr);
+        break;
+    case 2:
+        *value = readw(addr);
+        break;
+    case 4:
+        *value = readl(addr);
+        break;
+    default:
+        BUG();
+    }
+
+    return 0;
+}
+
+int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+                            uint32_t reg, uint32_t len, uint32_t value)
+{
+    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
+    if (!addr)
+        return -ENODEV;
+
+    switch (len)
+    {
+    case 1:
+        writeb(value, addr);
+        break;
+    case 2:
+        writew(value, addr);
+        break;
+    case 4:
+        writel(value, addr);
+        break;
+    default:
+        BUG();
+    }
+
+    return 0;
+}
 
 static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
                                 unsigned int len)
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
index 9dd9b02271..c582527e92 100644
--- a/xen/arch/arm/pci/pci-host-common.c
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg)
 }
 
 static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+                                              const struct pci_ecam_ops *ops,
                                               int ecam_reg_idx)
 {
     int err;
@@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
 
     cfg->phys_addr = addr;
     cfg->size = size;
+    cfg->ops = ops;
 
     /*
      * On 64-bit systems, we do a single ioremap for the whole config space
@@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
     printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
             cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
 
+    if ( ops->init )
+    {
+        err = ops->init(cfg);
+        if (err)
+            goto err_exit;
+    }
+
     return cfg;
 
 err_exit_remap:
@@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev)
 }
 
 int pci_host_common_probe(struct dt_device_node *dev,
+                          const struct pci_ecam_ops *ops,
                           int ecam_reg_idx)
 {
     struct pci_host_bridge *bridge;
@@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
         return -ENOMEM;
 
     /* Parse and map our Configuration Space windows */
-    cfg = gen_pci_init(dev, ecam_reg_idx);
+    cfg = gen_pci_init(dev, ops, ecam_reg_idx);
     if ( !cfg )
     {
         err = -ENOMEM;
@@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
 
     bridge->dt_node = dev;
     bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
     bridge->bus_start = cfg->busn_start;
     bridge->bus_end = cfg->busn_end;
 
diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
index 13d0f7f999..2d652e8910 100644
--- a/xen/arch/arm/pci/pci-host-generic.c
+++ b/xen/arch/arm/pci/pci-host-generic.c
@@ -23,20 +23,24 @@
 #include <asm/pci.h>
 
 static const struct dt_device_match gen_pci_dt_match[] = {
-    { .compatible = "pci-host-ecam-generic" },
+    { .compatible = "pci-host-ecam-generic",
+      .data =       &pci_generic_ecam_ops },
+
     { },
 };
 
 static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
 {
     const struct dt_device_match *of_id;
+    const struct pci_ecam_ops *ops;
 
     of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
 
     printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
            dt_node_full_name(dev), of_id->compatible);
 
-    return pci_host_common_probe(dev, 0);
+    return pci_host_common_probe(dev, ops, 0);
 }
 
 DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 58a51e724e..22866244d2 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -37,6 +37,7 @@ struct pci_config_window {
     uint8_t         busn_start;
     uint8_t         busn_end;
     void __iomem    *win;
+    const struct    pci_ecam_ops *ops;
 };
 
 /*
@@ -50,10 +51,41 @@ struct pci_host_bridge {
     u8 bus_start;                    /* Bus start of this bridge. */
     u8 bus_end;                      /* Bus end of this bridge. */
     void *sysdata;                   /* Pointer to the config space window*/
+    const struct pci_ops *ops;
 };
 
+struct pci_ops {
+    void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf,
+                             uint32_t offset);
+    int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf,
+                uint32_t reg, uint32_t len, uint32_t *value);
+    int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf,
+                 uint32_t reg, uint32_t len, uint32_t value);
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+    unsigned int            bus_shift;
+    struct pci_ops          pci_ops;
+    int (*init)(struct pci_config_window *);
+};
+
+/* Default ECAM ops */
+extern const struct pci_ecam_ops pci_generic_ecam_ops;
+
 int pci_host_common_probe(struct dt_device_node *dev,
+                          const struct pci_ecam_ops *ops,
                           int ecam_reg_idx);
+int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+                            uint32_t reg, uint32_t len, uint32_t *value);
+int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+                            uint32_t reg, uint32_t len, uint32_t value);
+void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
+                               uint32_t sbdf, uint32_t where);
+
 #else   /*!CONFIG_HAS_PCI*/
 
 struct arch_pci_dev { };
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (5 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-09-09 23:34   ` Stefano Stabellini
  2021-08-19 12:02 ` [PATCH v1 08/14] xen:arm: Implement pci access functions Rahul Singh
                   ` (6 subsequent siblings)
  13 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Oleksandr Andrushchenko,
	Stefano Stabellini, Julien Grall, Volodymyr Babchuk

From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>

Add support for Xilinx ZynqMP PCI host controller to map the PCI config
space to the XEN memory.

Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
---
 xen/arch/arm/pci/Makefile          |  1 +
 xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)
 create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c

diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
index 6f32fbbe67..1d045ade01 100644
--- a/xen/arch/arm/pci/Makefile
+++ b/xen/arch/arm/pci/Makefile
@@ -3,3 +3,4 @@ obj-y += pci-access.o
 obj-y += pci-host-generic.o
 obj-y += pci-host-common.o
 obj-y += ecam.o
+obj-y += pci-host-zynqmp.o
diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
new file mode 100644
index 0000000000..fe103e3855
--- /dev/null
+++ b/xen/arch/arm/pci/pci-host-zynqmp.c
@@ -0,0 +1,59 @@
+/*
+ * Copyright (C) 2020-2021 EPAM Systems
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Based on xen/arch/arm/pci/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <asm/device.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "xlnx,nwl-pcie-2.11",
+      .data =       &pci_generic_ecam_ops },
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+    const struct pci_ecam_ops *ops;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+            dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, ops, 2);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 08/14] xen:arm: Implement pci access functions
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (6 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-09-09 23:41   ` Stefano Stabellini
  2021-08-19 12:02 ` [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on" Rahul Singh
                   ` (5 subsequent siblings)
  13 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

Implement generic pci access functions to read/write the configuration
space.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/pci/pci-access.c      | 31 +++++++++++++++++++++++++++++-
 xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++
 xen/include/asm-arm/pci.h          |  2 ++
 3 files changed, 51 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
index f39f6a3a38..b94de3c3ac 100644
--- a/xen/arch/arm/pci/pci-access.c
+++ b/xen/arch/arm/pci/pci-access.c
@@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
 static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
                                 unsigned int len)
 {
-    return ~0U;
+    uint32_t val = GENMASK(0, len * 8);
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return val;
+    }
+
+    if ( unlikely(!bridge->ops->read) )
+        return val;
+
+    bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
+
+    return val;
 }
 
 static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
                              unsigned int len, uint32_t val)
 {
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return;
+    }
+
+    if ( unlikely(!bridge->ops->write) )
+        return;
+
+    bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
 }
 
 /*
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
index c582527e92..62715b4676 100644
--- a/xen/arch/arm/pci/pci-host-common.c
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -261,6 +261,25 @@ err_exit:
     return err;
 }
 
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+    struct pci_host_bridge *bridge;
+
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->segment != segment )
+            continue;
+        if ( (bus < bridge->bus_start) || (bus > bridge->bus_end) )
+            continue;
+        return bridge;
+    }
+
+    return NULL;
+}
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 22866244d2..756f8637ab 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -20,6 +20,7 @@
 #ifdef CONFIG_HAS_PCI
 
 #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+#define PRI_pci "%04x:%02x:%02x.%u"
 
 /* Arch pci dev struct */
 struct arch_pci_dev {
@@ -86,6 +87,7 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
 void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
                                uint32_t sbdf, uint32_t where);
 
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
 #else   /*!CONFIG_HAS_PCI*/
 
 struct arch_pci_dev { };
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (7 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 08/14] xen:arm: Implement pci access functions Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-19 12:09   ` Jan Beulich
                     ` (2 more replies)
  2021-08-19 12:02 ` [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
                   ` (4 subsequent siblings)
  13 siblings, 3 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

Add cmdline boot option "pci=on" to enable/disable the PCI init during
boot.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/pci/pci.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index d1c9cf997d..dc63bbc2a2 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -62,8 +62,38 @@ static void __init acpi_pci_init(void)
 static inline void __init acpi_pci_init(void) { }
 #endif
 
+static bool __initdata param_pci_enable;
+
+static int __init parse_pci_param(const char *arg)
+{
+    if ( !arg )
+    {
+        param_pci_enable = false;
+        return 0;
+    }
+
+    switch ( parse_bool(arg, NULL) )
+    {
+    case 0:
+        param_pci_enable = false;
+        return 0;
+    case 1:
+        param_pci_enable = true;
+        return 0;
+    }
+
+    return -EINVAL;
+}
+custom_param("pci", parse_pci_param);
+
 static int __init pci_init(void)
 {
+    /*
+     * Enable PCI when has been enabled explicitly (pci=on)
+     */
+    if ( !param_pci_enable)
+        return 0;
+
     if ( acpi_disabled )
         dt_pci_init();
     else
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (8 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on" Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-19 12:35   ` Julien Grall
  2021-08-19 12:02 ` [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
                   ` (3 subsequent siblings)
  13 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

Hardware domain is in charge of doing the PCI enumeration and will
discover the PCI devices and then will communicate to XEN via hyper
call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/physdev.c | 39 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
index e91355fe22..ccce8f0eba 100644
--- a/xen/arch/arm/physdev.c
+++ b/xen/arch/arm/physdev.c
@@ -9,12 +9,45 @@
 #include <xen/errno.h>
 #include <xen/sched.h>
 #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
 int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI
+    case PHYSDEVOP_pci_device_add: {
+        struct physdev_pci_device_add add;
+        struct pci_dev_info pdev_info;
+        nodeid_t node = NUMA_NO_NODE;
+
+        ret = -EFAULT;
+        if ( copy_from_guest(&add, arg, 1) != 0 )
+            break;
+
+        pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+        if ( add.flags & XEN_PCI_DEV_VIRTFN )
+        {
+            pdev_info.is_virtfn = 1;
+            pdev_info.physfn.bus = add.physfn.bus;
+            pdev_info.physfn.devfn = add.physfn.devfn;
+        }
+        else
+            pdev_info.is_virtfn = 0;
+
+        ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
+        break;
+    }
+#endif
+    default:
+        gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+        ret = -ENOSYS;
+    }
+
+    return ret;
 }
 
 /*
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (9 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-24 16:09   ` Jan Beulich
                     ` (2 more replies)
  2021-08-19 12:02 ` [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
                   ` (2 subsequent siblings)
  13 siblings, 3 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Jan Beulich, Paul Durrant,
	Roger Pau Monné

The existing VPCI support available for X86 is adapted for Arm.
When the device is added to XEN via the hyper call
“PHYSDEVOP_pci_device_add”, VPCI handler for the config space
access is added to the Xen to emulate the PCI devices config space.

A MMIO trap handler for the PCI ECAM space is registered in XEN
so that when guest is trying to access the PCI config space,XEN
will trap the access and emulate read/write using the VPCI and
not the real PCI hardware.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/Makefile         |  1 +
 xen/arch/arm/domain.c         |  4 ++
 xen/arch/arm/vpci.c           | 96 +++++++++++++++++++++++++++++++++++
 xen/arch/arm/vpci.h           | 37 ++++++++++++++
 xen/drivers/passthrough/pci.c |  7 +++
 xen/drivers/vpci/Makefile     |  3 +-
 xen/drivers/vpci/header.c     |  2 +
 xen/include/asm-arm/domain.h  |  5 +-
 xen/include/asm-arm/pci.h     |  8 +++
 xen/include/public/arch-arm.h |  4 ++
 10 files changed, 165 insertions(+), 2 deletions(-)
 create mode 100644 xen/arch/arm/vpci.c
 create mode 100644 xen/arch/arm/vpci.h

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 0e14a5e5c8..7cdce684a4 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -7,6 +7,7 @@ obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
 obj-$(CONFIG_HAS_PCI) += pci/
+obj-$(CONFIG_HAS_VPCI) += vpci.o
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 19c756ac3d..d99c653626 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -40,6 +40,7 @@
 #include <asm/vtimer.h>
 
 #include "vuart.h"
+#include "vpci.h"
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
@@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
     if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
         goto fail;
 
+    if ( (rc = domain_vpci_init(d)) != 0 )
+        goto fail;
+
     return 0;
 
 fail:
diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
new file mode 100644
index 0000000000..da8b1ca13c
--- /dev/null
+++ b/xen/arch/arm/vpci.c
@@ -0,0 +1,96 @@
+/*
+ * xen/arch/arm/vpci.c
+ * Copyright (c) 2021 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+#include <xen/sched.h>
+#include <asm/mmio.h>
+
+/* Do some sanity checks. */
+static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
+{
+    /* Check access size. */
+    if ( len != 1 && len != 2 && len != 4 && len != 8 )
+        return false;
+
+    /* Check that access is size aligned. */
+    if ( (reg & (len - 1)) )
+        return false;
+
+    return true;
+}
+
+static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
+                          register_t *r, void *p)
+{
+    unsigned int reg;
+    pci_sbdf_t sbdf;
+    uint32_t data = 0;
+    unsigned int size = 1U << info->dabt.size;
+
+    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
+    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+    if ( !vpci_mmio_access_allowed(reg, size) )
+        return 1;
+
+    data = vpci_read(sbdf, reg, size);
+
+    memcpy(r, &data, size);
+
+    return 1;
+}
+
+static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
+                           register_t r, void *p)
+{
+    unsigned int reg;
+    pci_sbdf_t sbdf;
+    uint32_t data = r;
+    unsigned int size = 1U << info->dabt.size;
+
+    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
+    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+    if ( !vpci_mmio_access_allowed(reg, size) )
+        return 1;
+
+    vpci_write(sbdf, reg, size, data);
+
+    return 1;
+}
+
+static const struct mmio_handler_ops vpci_mmio_handler = {
+    .read  = vpci_mmio_read,
+    .write = vpci_mmio_write,
+};
+
+int domain_vpci_init(struct domain *d)
+{
+    if ( !has_vpci(d) )
+        return 0;
+
+    register_mmio_handler(d, &vpci_mmio_handler,
+                          GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
+
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
new file mode 100644
index 0000000000..8a093bb705
--- /dev/null
+++ b/xen/arch/arm/vpci.h
@@ -0,0 +1,37 @@
+/*
+ * xen/arch/arm/vpci.h
+ * Copyright (c) 2021 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPCI_H__
+#define __ARCH_ARM_VPCI_H__
+
+#ifdef CONFIG_HAS_VPCI
+int domain_vpci_init(struct domain *d);
+#else
+static inline int domain_vpci_init(struct domain *d)
+{
+    return 0;
+}
+#endif
+
+#endif /* __ARCH_ARM_VPCI_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index c23c8cb06b..56e261e9bd 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     else
         iommu_enable_device(pdev);
 
+#ifdef CONFIG_ARM
+    ret = vpci_add_handlers(pdev);
+    if ( ret ) {
+        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
+        goto out;
+    }
+#endif
     pci_enable_acs(pdev);
 
 out:
diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
index 55d1bdfda0..1a1413b93e 100644
--- a/xen/drivers/vpci/Makefile
+++ b/xen/drivers/vpci/Makefile
@@ -1 +1,2 @@
-obj-y += vpci.o header.o msi.o msix.o
+obj-y += vpci.o header.o
+obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
index ba9a036202..f8cd55e7c0 100644
--- a/xen/drivers/vpci/header.c
+++ b/xen/drivers/vpci/header.c
@@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
      * FIXME: punching holes after the p2m has been set up might be racy for
      * DomU usage, needs to be revisited.
      */
+#ifdef CONFIG_HAS_PCI_MSI
     if ( map && !rom_only && vpci_make_msix_hole(pdev) )
         return;
+#endif
 
     for ( i = 0; i < ARRAY_SIZE(header->bars); i++ )
     {
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index c9277b5c6d..d742b94bd6 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
 
 #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
 
-#define has_vpci(d)    ({ (void)(d); false; })
+/* For X86 VPCI is enabled and tested for PVH DOM0 only but
+ * for ARM we enable support VPCI for guest domain also.
+ */
+#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); })
 
 #endif /* __ASM_DOMAIN_H__ */
 
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index 756f8637ab..c58152de80 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -27,6 +27,14 @@ struct arch_pci_dev {
     struct device dev;
 };
 
+/* Arch-specific MSI data for vPCI. */
+struct vpci_arch_msi {
+};
+
+/* Arch-specific MSI-X entry data for vPCI. */
+struct vpci_arch_msix_entry {
+};
+
 /*
  * struct to hold the mappings of a config space window. This
  * is expected to be used as sysdata for PCI controllers that
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 64a2ca30da..0a9749e768 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
 #define GUEST_PL011_BASE    xen_mk_ullong(0x22000000)
 #define GUEST_PL011_SIZE    xen_mk_ullong(0x00001000)
 
+/* VPCI ECAM mappings */
+#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
+#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
+
 /*
  * 16MB == 4096 pages reserved for guest to use as a region to map its
  * grant table in.
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (10 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-19 13:00   ` Julien Grall
  2021-09-10  0:51   ` Stefano Stabellini
  2021-08-19 12:02 ` [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest Rahul Singh
  2021-08-19 12:02 ` [PATCH v1 14/14] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
  13 siblings, 2 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Ian Jackson, Wei Liu,
	Anthony PERARD, Juergen Gross, Andrew Cooper, George Dunlap,
	Jan Beulich, Julien Grall, Stefano Stabellini, Volodymyr Babchuk

libxl will create an emulated PCI device tree node in the device tree to
enable the guest OS to discover the virtual PCI during guest boot.
Emulated PCI device tree node will only be created when there is any
device assigned to guest.

A new area has been reserved in the arm guest physical map at
which the VPCI bus is declared in the device tree (reg and ranges
parameters of the node).

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 tools/libs/light/libxl_arm.c          | 109 ++++++++++++++++++++++++++
 tools/libs/light/libxl_types.idl      |   1 +
 tools/xl/xl_parse.c                   |   2 +
 xen/include/public/arch-arm.h         |  11 +++
 xen/include/public/device_tree_defs.h |   1 +
 5 files changed, 124 insertions(+)

diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index e3140a6e00..a091e97e76 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
     return fdt_property(fdt, "reg", regs, sizeof(regs));
 }
 
+static int fdt_property_values(libxl__gc *gc, void *fdt,
+        const char *name, unsigned num_cells, ...)
+{
+    uint32_t prop[num_cells];
+    be32 *cells = &prop[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, name, prop, sizeof(prop));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+                                    unsigned addr_cells,
+                                    unsigned size_cells,
+                                    unsigned num_regs, ...)
+{
+    uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
+    be32 *cells = &regs[0];
+    int i;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the memory bit field */
+        arg = va_arg(ap, uint64_t);
+        set_cell(&cells, 1, arg);
+
+        /* Set the vpci bus address */
+        arg = addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, addr_cells , arg);
+
+        /* Set the cpu bus address where vpci address is mapped */
+        set_cell(&cells, addr_cells, arg);
+
+        /* Set the vpci size requested */
+        arg = size_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, size_cells,arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
 static int make_root_properties(libxl__gc *gc,
                                 const libxl_version_info *vers,
                                 void *fdt)
@@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
     return 0;
 }
 
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+        const struct arch_info *ainfo,
+        struct xc_dom_image *dom)
+{
+    int res;
+    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+    res = fdt_begin_node(fdt, name);
+    if (res) return res;
+
+    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "device_type", "pci");
+    if (res) return res;
+
+    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+    if (res) return res;
+
+    res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#address-cells", 3);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#size-cells", 2);
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "status", "okay");
+    if (res) return res;
+
+    res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+        GUEST_ROOT_SIZE_CELLS, 2,
+        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR,
+        GUEST_VPCI_PREFETCH_MEM_SIZE);
+    if (res) return res;
+
+    res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS,
+                              0, 0x10000);
+    if (res) return res;
+
+    res = fdt_end_node(fdt);
+    if (res) return res;
+
+    return 0;
+}
+
 static const struct arch_info *get_arch_info(libxl__gc *gc,
                                              const struct xc_dom_image *dom)
 {
@@ -971,6 +1074,9 @@ next_resize:
         if (info->tee == LIBXL_TEE_TYPE_OPTEE)
             FDT( make_optee_node(gc, fdt) );
 
+        if (libxl_defbool_val(info->arch_arm.vpci))
+            FDT( make_vpci_node(gc, fdt, ainfo, dom) );
+
         if (pfdt)
             FDT( copy_partial_fdt(gc, fdt, pfdt) );
 
@@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
     /* ACPI is disabled by default */
     libxl_defbool_setdefault(&b_info->acpi, false);
 
+    /* VPCI is disabled by default */
+    libxl_defbool_setdefault(&b_info->arch_arm.vpci, false);
+
     if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
         return;
 
diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index 3f9fff653a..78b1ddf0b8 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
 
     ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
                                ("vuart", libxl_vuart_type),
+                               ("vpci", libxl_defbool),
                               ])),
     ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
                               ])),
diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
index 17dddb4cd5..ffafbeffb4 100644
--- a/tools/xl/xl_parse.c
+++ b/tools/xl/xl_parse.c
@@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source,
         }
         if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
             libxl_defbool_set(&b_info->u.pv.e820_host, true);
+        if (d_config->num_pcidevs)
+            libxl_defbool_set(&b_info->arch_arm.vpci, true);
     }
 
     if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) {
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 0a9749e768..01d13e669e 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t;
 #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
 #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
 
+/* PCI-PCIe memory space types */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
+
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_PREFETCH_MEM_ADDR  xen_mk_ullong(0x900000000)
+#define GUEST_VPCI_MEM_ADDR           xen_mk_ullong(0x20000000)
+
+#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x2000000000)
+#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x04000000)
+
 /*
  * 16MB == 4096 pages reserved for guest to use as a region to map its
  * grant table in.
diff --git a/xen/include/public/device_tree_defs.h b/xen/include/public/device_tree_defs.h
index 209d43de3f..d24d86b1fc 100644
--- a/xen/include/public/device_tree_defs.h
+++ b/xen/include/public/device_tree_defs.h
@@ -7,6 +7,7 @@
  * onwards. Reserve a high value for the GIC phandle.
  */
 #define GUEST_PHANDLE_GIC (65000)
+#define GUEST_PHANDLE_ITS (64999)
 
 #define GUEST_ROOT_ADDRESS_CELLS 2
 #define GUEST_ROOT_SIZE_CELLS 2
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (11 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-08-19 12:12   ` Jan Beulich
  2021-08-19 12:02 ` [PATCH v1 14/14] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
  13 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

XEN_DOMCTL_ioport_permission, PHYSDEVOP_unmap_pirq, PHYSDEVOP_unmap_pirq
are unimplemented for ARM. When libxl assigning a PCI device to the
guest error is observed related to above functions.

Implement dummy functions to fix the error.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/domctl.c  | 2 ++
 xen/arch/arm/physdev.c | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
index b7d27f37df..38813be893 100644
--- a/xen/arch/arm/domctl.c
+++ b/xen/arch/arm/domctl.c
@@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
 
         return rc;
     }
+    case XEN_DOMCTL_ioport_permission:
+        return 0;
     default:
     {
         int rc;
diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
index ccce8f0eba..4a0affeada 100644
--- a/xen/arch/arm/physdev.c
+++ b/xen/arch/arm/physdev.c
@@ -42,6 +42,9 @@ int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         break;
     }
 #endif
+    case PHYSDEVOP_unmap_pirq:
+    case PHYSDEVOP_map_pirq:
+        break;
     default:
         gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
         ret = -ENOSYS;
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v1 14/14] xen/arm: Add linux,pci-domain property for hwdom if not available.
  2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
                   ` (12 preceding siblings ...)
  2021-08-19 12:02 ` [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest Rahul Singh
@ 2021-08-19 12:02 ` Rahul Singh
  2021-09-10  1:00   ` Stefano Stabellini
  13 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 12:02 UTC (permalink / raw)
  To: xen-devel
  Cc: bertrand.marquis, rahul.singh, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

If the property is not present in the device tree node for host bridge,
XEN while creating the dtb for hwdom will create this property and
assigns the already allocated segment to the host bridge
so that XEN and linux will have the same segment for the host bridges.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
---
 xen/arch/arm/domain_build.c        | 18 ++++++++++++++++++
 xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++
 xen/include/asm-arm/pci.h          |  3 +++
 3 files changed, 42 insertions(+)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6c86d52781..e0cf2ff19d 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -581,6 +581,24 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
             return res;
     }
 
+#ifdef CONFIG_HAS_PCI
+    if ( dt_device_type_is_equal(node, "pci") )
+    {
+        if ( !dt_find_property(node, "linux,pci-domain", NULL) )
+        {
+            uint16_t segment;
+
+            res = pci_get_host_bridge_segment(node, &segment);
+            if ( res < 0 )
+                return res;
+
+            res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment);
+            if ( res )
+                return res;
+        }
+    }
+#endif
+
     /*
      * Override the property "status" to disable the device when it's
      * marked for passthrough.
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
index 62715b4676..5e34252deb 100644
--- a/xen/arch/arm/pci/pci-host-common.c
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -280,6 +280,27 @@ struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
 
     return NULL;
 }
+
+/*
+ * This function will lookup an hostbridge based on config space address.
+ */
+int pci_get_host_bridge_segment(const struct dt_device_node *node,
+                                uint16_t *segment)
+{
+    struct pci_host_bridge *bridge;
+
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->dt_node != node )
+            continue;
+
+        *segment = bridge->segment;
+        return 0;
+    }
+
+    return -EINVAL;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
index c58152de80..2d4610a23a 100644
--- a/xen/include/asm-arm/pci.h
+++ b/xen/include/asm-arm/pci.h
@@ -96,6 +96,9 @@ void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
                                uint32_t sbdf, uint32_t where);
 
 struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
+int pci_get_host_bridge_segment(const struct dt_device_node *node,
+                                uint16_t *segment);
+
 #else   /*!CONFIG_HAS_PCI*/
 
 struct arch_pci_dev { };
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-19 12:02 ` [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on" Rahul Singh
@ 2021-08-19 12:09   ` Jan Beulich
  2021-08-19 12:31   ` Julien Grall
  2021-09-09 23:48   ` Stefano Stabellini
  2 siblings, 0 replies; 98+ messages in thread
From: Jan Beulich @ 2021-08-19 12:09 UTC (permalink / raw)
  To: Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, xen-devel

On 19.08.2021 14:02, Rahul Singh wrote:
> Add cmdline boot option "pci=on" to enable/disable the PCI init during
> boot.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/pci/pci.c | 30 ++++++++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)

Any addition or change of a command line option should be accompanied
by an adjustment to docs/misc/xen-command-line.pandoc, please.

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest
  2021-08-19 12:02 ` [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest Rahul Singh
@ 2021-08-19 12:12   ` Jan Beulich
  2021-08-19 12:40     ` Julien Grall
  2021-08-20 17:01     ` Rahul Singh
  0 siblings, 2 replies; 98+ messages in thread
From: Jan Beulich @ 2021-08-19 12:12 UTC (permalink / raw)
  To: Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, xen-devel

On 19.08.2021 14:02, Rahul Singh wrote:
> --- a/xen/arch/arm/domctl.c
> +++ b/xen/arch/arm/domctl.c
> @@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
>  
>          return rc;
>      }
> +    case XEN_DOMCTL_ioport_permission:
> +        return 0;

I don't think returning success for something that doesn't make
much sense in the first place (there aren't truly "I/O ports" on
Arm afaik) is a good idea. Instead I think the tool stack should
avoid making arch-specific calls in an arch-independent way.

> --- a/xen/arch/arm/physdev.c
> +++ b/xen/arch/arm/physdev.c
> @@ -42,6 +42,9 @@ int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>          break;
>      }
>  #endif
> +    case PHYSDEVOP_unmap_pirq:
> +    case PHYSDEVOP_map_pirq:
> +        break;

Less sure here, but I'm not convinced either.

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-19 12:02 ` [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN Rahul Singh
@ 2021-08-19 12:18   ` Julien Grall
  2021-08-19 14:16     ` Rahul Singh
  2021-08-24 15:53   ` Jan Beulich
  2021-08-26 13:23   ` Daniel P. Smith
  2 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-08-19 12:18 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Jan Beulich, Andrew Cooper,
	Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Stefano Stabellini,
	Paul Durrant, Daniel De Graaf, Daniel P. Smith

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> MSI code that implements MSI functionality to support MSI within XEN is
> not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate

Can you clarify what you mean by not usable? Is it because we lack of 
support or we have no plan to use it?

Cheers,

> the code for ARM.
> 
> No functional change intended.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/x86/Kconfig             |  1 +
>   xen/drivers/passthrough/Makefile |  1 +
>   xen/drivers/passthrough/msi.c    | 96 ++++++++++++++++++++++++++++++++
>   xen/drivers/passthrough/pci.c    | 54 +++++-------------
>   xen/drivers/pci/Kconfig          |  4 ++
>   xen/include/xen/msi.h            | 56 +++++++++++++++++++
>   xen/xsm/flask/hooks.c            |  8 +--
>   7 files changed, 175 insertions(+), 45 deletions(-)
>   create mode 100644 xen/drivers/passthrough/msi.c
>   create mode 100644 xen/include/xen/msi.h
> 
> diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
> index 9b164db641..7b46ee98c5 100644
> --- a/xen/arch/x86/Kconfig
> +++ b/xen/arch/x86/Kconfig
> @@ -19,6 +19,7 @@ config X86
>   	select HAS_NS16550
>   	select HAS_PASSTHROUGH
>   	select HAS_PCI
> +	select HAS_PCI_MSI
>   	select HAS_PDX
>   	select HAS_SCHED_GRANULARITY
>   	select HAS_UBSAN
> diff --git a/xen/drivers/passthrough/Makefile b/xen/drivers/passthrough/Makefile
> index 445690e3e5..a5efa22714 100644
> --- a/xen/drivers/passthrough/Makefile
> +++ b/xen/drivers/passthrough/Makefile
> @@ -7,3 +7,4 @@ obj-y += iommu.o
>   obj-$(CONFIG_HAS_PCI) += pci.o
>   obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o
>   obj-$(CONFIG_HAS_PCI) += ats.o
> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o
> diff --git a/xen/drivers/passthrough/msi.c b/xen/drivers/passthrough/msi.c
> new file mode 100644
> index 0000000000..15ad0f8160
> --- /dev/null
> +++ b/xen/drivers/passthrough/msi.c
> @@ -0,0 +1,96 @@
> +/*
> + * Copyright (C) 2008,  Netronome Systems, Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/msi.h>
> +#include <asm/hvm/io.h>
> +
> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)
> +{
> +    int rc;
> +
> +    if ( pdev->msix )
> +    {
> +        rc = pci_reset_msix_state(pdev);
> +        if ( rc )
> +            return rc;
> +        msixtbl_init(d);
> +    }
> +
> +    return 0;
> +}
> +
> +int pdev_msi_init(struct pci_dev *pdev)
> +{
> +    unsigned int pos;
> +
> +    INIT_LIST_HEAD(&pdev->msi_list);
> +
> +    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> +                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI);
> +    if ( pos )
> +    {
> +        uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos));
> +
> +        pdev->msi_maxvec = multi_msi_capable(ctrl);
> +    }
> +
> +    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> +                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX);
> +    if ( pos )
> +    {
> +        struct arch_msix *msix = xzalloc(struct arch_msix);
> +        uint16_t ctrl;
> +
> +        if ( !msix )
> +            return -ENOMEM;
> +
> +        spin_lock_init(&msix->table_lock);
> +
> +        ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos));
> +        msix->nr_entries = msix_table_size(ctrl);
> +
> +        pdev->msix = msix;
> +    }
> +
> +    return 0;
> +}
> +
> +void pdev_msi_deinit(struct pci_dev *pdev)
> +{
> +    XFREE(pdev->msix);
> +}
> +
> +void pdev_dump_msi(const struct pci_dev *pdev)
> +{
> +    const struct msi_desc *msi;
> +
> +    printk("- MSIs < ");
> +    list_for_each_entry ( msi, &pdev->msi_list, list )
> +        printk("%d ", msi->irq);
> +    printk(">");
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> index fc4fa2e5c3..67f5686ab6 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -32,8 +32,8 @@
>   #include <xen/softirq.h>
>   #include <xen/tasklet.h>
>   #include <xen/vpci.h>
> +#include <xen/msi.h>
>   #include <xsm/xsm.h>
> -#include <asm/msi.h>
>   #include "ats.h"
>   
>   struct pci_seg {
> @@ -314,6 +314,7 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
>   {
>       struct pci_dev *pdev;
>       unsigned int pos;
> +    int rc;
>   
>       list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
>           if ( pdev->bus == bus && pdev->devfn == devfn )
> @@ -327,35 +328,12 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
>       *((u8*) &pdev->bus) = bus;
>       *((u8*) &pdev->devfn) = devfn;
>       pdev->domain = NULL;
> -    INIT_LIST_HEAD(&pdev->msi_list);
> -
> -    pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
> -                              PCI_CAP_ID_MSI);
> -    if ( pos )
> -    {
> -        uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos));
> -
> -        pdev->msi_maxvec = multi_msi_capable(ctrl);
> -    }
>   
> -    pos = pci_find_cap_offset(pseg->nr, bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
> -                              PCI_CAP_ID_MSIX);
> -    if ( pos )
> +    rc = pdev_msi_init(pdev);
> +    if ( rc )
>       {
> -        struct arch_msix *msix = xzalloc(struct arch_msix);
> -        uint16_t ctrl;
> -
> -        if ( !msix )
> -        {
> -            xfree(pdev);
> -            return NULL;
> -        }
> -        spin_lock_init(&msix->table_lock);
> -
> -        ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos));
> -        msix->nr_entries = msix_table_size(ctrl);
> -
> -        pdev->msix = msix;
> +        xfree(pdev);
> +        return NULL;
>       }
>   
>       list_add(&pdev->alldevs_list, &pseg->alldevs_list);
> @@ -449,7 +427,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev *pdev)
>       }
>   
>       list_del(&pdev->alldevs_list);
> -    xfree(pdev->msix);
> +    pdev_msi_deinit(pdev);
>       xfree(pdev);
>   }
>   
> @@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev)
>   static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
>   {
>       struct pci_dev *pdev;
> -    struct msi_desc *msi;
>   
>       printk("==== segment %04x ====\n", pseg->nr);
>   
>       list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
>       {
> -        printk("%pp - %pd - node %-3d - MSIs < ",
> +        printk("%pp - %pd - node %-3d ",
>                  &pdev->sbdf, pdev->domain,
>                  (pdev->node != NUMA_NO_NODE) ? pdev->node : -1);
> -        list_for_each_entry ( msi, &pdev->msi_list, list )
> -               printk("%d ", msi->irq);
> -        printk(">\n");
> +        pdev_dump_msi(pdev);
> +        printk("\n");
>       }
>   
>       return 0;
> @@ -1422,13 +1398,9 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)
>       ASSERT(pdev && (pdev->domain == hardware_domain ||
>                       pdev->domain == dom_io));
>   
> -    if ( pdev->msix )
> -    {
> -        rc = pci_reset_msix_state(pdev);
> -        if ( rc )
> -            goto done;
> -        msixtbl_init(d);
> -    }
> +    rc = pdev_msix_assign(d, pdev);
> +    if ( rc )
> +        goto done;
>   
>       pdev->fault.count = 0;
>   
> diff --git a/xen/drivers/pci/Kconfig b/xen/drivers/pci/Kconfig
> index 7da03fa13b..c6a7bc8007 100644
> --- a/xen/drivers/pci/Kconfig
> +++ b/xen/drivers/pci/Kconfig
> @@ -1,3 +1,7 @@
>   
>   config HAS_PCI
>   	bool
> +
> +config HAS_PCI_MSI
> +	bool
> +	depends on HAS_PCI
> diff --git a/xen/include/xen/msi.h b/xen/include/xen/msi.h
> new file mode 100644
> index 0000000000..b2d5bc6f9d
> --- /dev/null
> +++ b/xen/include/xen/msi.h
> @@ -0,0 +1,56 @@
> +/*
> + * Copyright (C) 2008,  Netronome Systems, Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef __XEN_MSI_H_
> +#define __XEN_MSI_H_
> +
> +#ifdef CONFIG_HAS_PCI_MSI
> +
> +#include <asm/msi.h>
> +
> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev);
> +int pdev_msi_init(struct pci_dev *pdev);
> +void pdev_msi_deinit(struct pci_dev *pdev);
> +void pdev_dump_msi(const struct pci_dev *pdev);
> +
> +#else /* !CONFIG_HAS_PCI_MSI */
> +static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)
> +{
> +    return 0;
> +}
> +
> +static inline int pdev_msi_init(struct pci_dev *pdev)
> +{
> +    return 0;
> +}
> +
> +static inline void pdev_msi_deinit(struct pci_dev *pdev) {}
> +static inline void pci_cleanup_msi(struct pci_dev *pdev) {}
> +static inline void pdev_dump_msi(const struct pci_dev *pdev) {}
> +
> +#endif /* CONFIG_HAS_PCI_MSI */
> +
> +#endif /* __XEN_MSI_H */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
> index f1a1217c98..fdcfeb984c 100644
> --- a/xen/xsm/flask/hooks.c
> +++ b/xen/xsm/flask/hooks.c
> @@ -21,7 +21,7 @@
>   #include <xen/guest_access.h>
>   #include <xen/xenoprof.h>
>   #include <xen/iommu.h>
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>   #include <asm/msi.h>
>   #endif
>   #include <public/xen.h>
> @@ -114,7 +114,7 @@ static int get_irq_sid(int irq, u32 *sid, struct avc_audit_data *ad)
>           }
>           return security_irq_sid(irq, sid);
>       }
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>       {
>           struct irq_desc *desc = irq_to_desc(irq);
>           if ( desc->msi_desc && desc->msi_desc->dev ) {
> @@ -874,7 +874,7 @@ static int flask_map_domain_pirq (struct domain *d)
>   static int flask_map_domain_msi (struct domain *d, int irq, const void *data,
>                                    u32 *sid, struct avc_audit_data *ad)
>   {
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>       const struct msi_info *msi = data;
>       u32 machine_bdf = (msi->seg << 16) | (msi->bus << 8) | msi->devfn;
>   
> @@ -940,7 +940,7 @@ static int flask_unmap_domain_pirq (struct domain *d)
>   static int flask_unmap_domain_msi (struct domain *d, int irq, const void *data,
>                                      u32 *sid, struct avc_audit_data *ad)
>   {
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>       const struct pci_dev *pdev = data;
>       u32 machine_bdf = (pdev->seg << 16) | (pdev->bus << 8) | pdev->devfn;
>   
> 

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-19 12:02 ` [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled Rahul Singh
@ 2021-08-19 12:28   ` Julien Grall
  2021-08-20 10:30     ` Rahul Singh
  2021-08-20  7:01   ` Jan Beulich
  2021-09-09 13:16   ` Julien Grall
  2 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-08-19 12:28 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> Compilation error is observed when HAS_PCI is enabled for ARM
> architecture.

To be pedantic, what you are trying to solve is not a compilation error 
but the fact that the PCI mandates helpers that doesn't yet exist on 
Arm. So...

> Add definition for arch_iommu_use_permitted() and
> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to
> fix compilation error.

... I am not really in favor of adding dummy implementation here. 
Instead, the series should be re-ordered so we add the pci-access 
helpers first and then enable HAS_PCI towards the end of the series.

> pci.c: In function ‘deassign_device’:
> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’;
> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration]
>              pci_to_dev(pdev));
> 
> pci.c:18: undefined reference to `pci_conf_read16’
> pci.c:880: undefined reference to `arch_pci_clean_pirqs’
> pci.c:1392: undefined reference to `arch_iommu_use_permitted'
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/arm/Makefile               |  1 +
>   xen/arch/arm/pci/Makefile           |  2 +
>   xen/arch/arm/pci/pci-access.c       | 61 +++++++++++++++++++++++++++++
>   xen/arch/arm/pci/pci.c              | 32 +++++++++++++++
>   xen/drivers/passthrough/arm/iommu.c |  5 +++
>   xen/include/asm-arm/pci.h           | 33 ++++++++++++++--
>   6 files changed, 131 insertions(+), 3 deletions(-)
>   create mode 100644 xen/arch/arm/pci/Makefile
>   create mode 100644 xen/arch/arm/pci/pci-access.c
>   create mode 100644 xen/arch/arm/pci/pci.c
> 
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 3d3b97b5b4..0e14a5e5c8 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>   obj-y += platforms/
>   endif
>   obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_HAS_PCI) += pci/
>   
>   obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>   obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> new file mode 100644
> index 0000000000..a9ee0b9b44
> --- /dev/null
> +++ b/xen/arch/arm/pci/Makefile
> @@ -0,0 +1,2 @@
> +obj-y += pci.o
> +obj-y += pci-access.o
> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
> new file mode 100644
> index 0000000000..b938047c03
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-access.c
> @@ -0,0 +1,61 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/pci.h>
> +
> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
> +                                unsigned int len)
> +{
> +    return ~0U;
> +}
> +
> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
> +                             unsigned int len, uint32_t val)
> +{
> +}
> +
> +/*
> + * Wrappers for all PCI configuration access functions.
> + */
> +
> +#define PCI_OP_WRITE(size, type)                                            \
> +    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val)  \
> +{                                                                           \
> +    pci_config_write(sbdf, reg, size / 8, val);                             \
> +}
> +
> +#define PCI_OP_READ(size, type)                                             \
> +    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)            \
> +{                                                                           \
> +    return pci_config_read(sbdf, reg, size / 8);                            \
> +}
> +
> +PCI_OP_READ(8, u8)
> +PCI_OP_READ(16, u16)
> +PCI_OP_READ(32, u32)
> +PCI_OP_WRITE(8, u8)
> +PCI_OP_WRITE(16, u16)
> +PCI_OP_WRITE(32, u32)
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
> new file mode 100644
> index 0000000000..dc55d23778
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci.c
> @@ -0,0 +1,32 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/pci.h>
> +
> +int arch_pci_clean_pirqs(struct domain *d)
> +{
> +    return 0;
> +}

Please add a comment explaining why this just returns 0.

> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c
> index db3b07a571..fdec1c5547 100644
> --- a/xen/drivers/passthrough/arm/iommu.c
> +++ b/xen/drivers/passthrough/arm/iommu.c
> @@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d)
>   void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
>   {
>   }
> +
> +bool arch_iommu_use_permitted(const struct domain *d)
> +{
> +    return true;
> +}

Please add a comment explaining why returning true is always fine.

> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index de13359f65..61e43da088 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -1,7 +1,34 @@
> -#ifndef __X86_PCI_H__
> -#define __X86_PCI_H__
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
>   
> +#ifndef __ARM_PCI_H__
> +#define __ARM_PCI_H__
> +
> +#ifdef CONFIG_HAS_PCI
> +
> +#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
> +
> +/* Arch pci dev struct */
>   struct arch_pci_dev {
> +    struct device dev;
>   };
>   
> -#endif /* __X86_PCI_H__ */
> +#else   /*!CONFIG_HAS_PCI*/
> +
> +struct arch_pci_dev { };
> +
> +#endif  /*!CONFIG_HAS_PCI*/
> +#endif /* __ARM_PCI_H__ */
> 

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-19 12:02 ` [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on" Rahul Singh
  2021-08-19 12:09   ` Jan Beulich
@ 2021-08-19 12:31   ` Julien Grall
  2021-08-20 12:19     ` Rahul Singh
  2021-09-09 23:48   ` Stefano Stabellini
  2 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-08-19 12:31 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> Add cmdline boot option "pci=on" to enable/disable the PCI init during
> boot.

I read this as "PCI" will be either disabled/enabled for the platform. 
Whereas, I think it will be used to decide whether Xen discover PCI and 
PCI passthrough is supported or not.

Can you also clarify why a user would want to select "pci=off"?

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
  2021-08-19 12:02 ` [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
@ 2021-08-19 12:35   ` Julien Grall
  2021-08-19 13:40     ` Jan Beulich
  2021-08-20 13:05     ` Rahul Singh
  0 siblings, 2 replies; 98+ messages in thread
From: Julien Grall @ 2021-08-19 12:35 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk, Jan Beulich

(+ Jan)

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> Hardware domain is in charge of doing the PCI enumeration and will
> discover the PCI devices and then will communicate to XEN via hyper
> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

There are other PHYSDEVOP operations to add PCI devices. I think it is 
fine to only implement the latest (CC Jan for some opinion and confirm 
this is the latest). However, this ought to be explained in the commit 
message.

Also, public/arch-arm.h will need to be updated as we now support the 
PHYSDEVOP hypercall.

> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/arm/physdev.c | 39 ++++++++++++++++++++++++++++++++++++---
>   1 file changed, 36 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
> index e91355fe22..ccce8f0eba 100644
> --- a/xen/arch/arm/physdev.c
> +++ b/xen/arch/arm/physdev.c
> @@ -9,12 +9,45 @@
>   #include <xen/errno.h>
>   #include <xen/sched.h>
>   #include <asm/hypercall.h>
> -
> +#include <xen/guest_access.h>
> +#include <xsm/xsm.h>
>   
>   int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>   {
> -    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> -    return -ENOSYS;
> +    int ret = 0;
> +
> +    switch ( cmd )
> +    {
> +#ifdef CONFIG_HAS_PCI
> +    case PHYSDEVOP_pci_device_add: {
> +        struct physdev_pci_device_add add;
> +        struct pci_dev_info pdev_info;
> +        nodeid_t node = NUMA_NO_NODE;
> +
> +        ret = -EFAULT;
> +        if ( copy_from_guest(&add, arg, 1) != 0 )
> +            break;
> +
> +        pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
> +        if ( add.flags & XEN_PCI_DEV_VIRTFN )
> +        {
> +            pdev_info.is_virtfn = 1;
> +            pdev_info.physfn.bus = add.physfn.bus;
> +            pdev_info.physfn.devfn = add.physfn.devfn;
> +        }
> +        else
> +            pdev_info.is_virtfn = 0;
> +
> +        ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
> +        break;
> +    }

This is pretty much a copy of the x86 version without the NUMA bit. So I 
think we want to move the implementation in common code.

> +#endif
> +    default:
> +        gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
> +        ret = -ENOSYS;
> +    }
> +
> +    return ret;
>   }
>   
>   /*
> 

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest
  2021-08-19 12:12   ` Jan Beulich
@ 2021-08-19 12:40     ` Julien Grall
  2021-08-20 17:01     ` Rahul Singh
  1 sibling, 0 replies; 98+ messages in thread
From: Julien Grall @ 2021-08-19 12:40 UTC (permalink / raw)
  To: Jan Beulich, Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk, xen-devel

Hi,

On 19/08/2021 13:12, Jan Beulich wrote:
> On 19.08.2021 14:02, Rahul Singh wrote:
>> --- a/xen/arch/arm/domctl.c
>> +++ b/xen/arch/arm/domctl.c
>> @@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
>>   
>>           return rc;
>>       }
>> +    case XEN_DOMCTL_ioport_permission:
>> +        return 0;
> 
> I don't think returning success for something that doesn't make
> much sense in the first place (there aren't truly "I/O ports" on
> Arm afaik) is a good idea.

They are memory mapped. IIRC, there is a region that is reserved to fake 
them (for PCI cards that still using them).

> Instead I think the tool stack should
> avoid making arch-specific calls in an arch-independent way.

+1. The current approach means that it will be impossible to implement 
those sub operations without breaking Xen if we ever needs it.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl
  2021-08-19 12:02 ` [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
@ 2021-08-19 13:00   ` Julien Grall
  2021-08-20 16:03     ` Rahul Singh
  2021-09-10  0:51   ` Stefano Stabellini
  1 sibling, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-08-19 13:00 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Ian Jackson, Wei Liu, Anthony PERARD,
	Juergen Gross, Andrew Cooper, George Dunlap, Jan Beulich,
	Stefano Stabellini, Volodymyr Babchuk

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> libxl will create an emulated PCI device tree node in the device tree to
> enable the guest OS to discover the virtual PCI during guest boot.
> Emulated PCI device tree node will only be created when there is any
> device assigned to guest.
> 
> A new area has been reserved in the arm guest physical map at
> which the VPCI bus is declared in the device tree (reg and ranges
> parameters of the node).
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   tools/libs/light/libxl_arm.c          | 109 ++++++++++++++++++++++++++
>   tools/libs/light/libxl_types.idl      |   1 +
>   tools/xl/xl_parse.c                   |   2 +
>   xen/include/public/arch-arm.h         |  11 +++
>   xen/include/public/device_tree_defs.h |   1 +
>   5 files changed, 124 insertions(+)
> 
> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> index e3140a6e00..a091e97e76 100644
> --- a/tools/libs/light/libxl_arm.c
> +++ b/tools/libs/light/libxl_arm.c
> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>       return fdt_property(fdt, "reg", regs, sizeof(regs));
>   }
>   
> +static int fdt_property_values(libxl__gc *gc, void *fdt,
> +        const char *name, unsigned num_cells, ...)
> +{
> +    uint32_t prop[num_cells];
> +    be32 *cells = &prop[0];
> +    int i;
> +    va_list ap;
> +    uint32_t arg;
> +
> +    va_start(ap, num_cells);
> +    for (i = 0 ; i < num_cells; i++) {
> +        arg = va_arg(ap, uint32_t);
> +        set_cell(&cells, 1, arg);
> +    }
> +    va_end(ap);
> +
> +    return fdt_property(fdt, name, prop, sizeof(prop));
> +}
> +
> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
> +                                    unsigned addr_cells,
> +                                    unsigned size_cells,
> +                                    unsigned num_regs, ...)
> +{
> +    uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
> +    be32 *cells = &regs[0];
> +    int i;
> +    va_list ap;
> +    uint64_t arg;
> +
> +    va_start(ap, num_regs);
> +    for (i = 0 ; i < num_regs; i++) {
> +        /* Set the memory bit field */
> +        arg = va_arg(ap, uint64_t);
> +        set_cell(&cells, 1, arg);
> +
> +        /* Set the vpci bus address */
> +        arg = addr_cells ? va_arg(ap, uint64_t) : 0;
> +        set_cell(&cells, addr_cells , arg);
> +
> +        /* Set the cpu bus address where vpci address is mapped */
> +        set_cell(&cells, addr_cells, arg);
> +
> +        /* Set the vpci size requested */
> +        arg = size_cells ? va_arg(ap, uint64_t) : 0;
> +        set_cell(&cells, size_cells,arg);
> +    }
> +    va_end(ap);
> +
> +    return fdt_property(fdt, "ranges", regs, sizeof(regs));
> +}
> +
>   static int make_root_properties(libxl__gc *gc,
>                                   const libxl_version_info *vers,
>                                   void *fdt)
> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>       return 0;
>   }
>   
> +static int make_vpci_node(libxl__gc *gc, void *fdt,
> +        const struct arch_info *ainfo,
> +        struct xc_dom_image *dom)
> +{
> +    int res;
> +    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
> +    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
> +    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
> +
> +    res = fdt_begin_node(fdt, name);
> +    if (res) return res;
> +
> +    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
> +    if (res) return res;
> +
> +    res = fdt_property_string(fdt, "device_type", "pci");
> +    if (res) return res;
> +
> +    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> +            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
> +    if (res) return res;
> +
> +    res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17);

AFAICT, the "bus-range" is optional. Can you explain why we need it?

> +    if (res) return res;
> +
> +    res = fdt_property_cell(fdt, "#address-cells", 3);
> +    if (res) return res;
> +
> +    res = fdt_property_cell(fdt, "#size-cells", 2);
> +    if (res) return res;
> +
> +    res = fdt_property_string(fdt, "status", "okay");
> +    if (res) return res;
> +
> +    res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> +        GUEST_ROOT_SIZE_CELLS, 2,
> +        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE,
> +        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR,
> +        GUEST_VPCI_PREFETCH_MEM_SIZE);
> +    if (res) return res;
> +
> +    res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS,
> +                              0, 0x10000);

The ITS is not exposed to the guest so far. So I think this wants to be 
moved to the future series that will add ITS support.

> +    if (res) return res;
> +
> +    res = fdt_end_node(fdt);
> +    if (res) return res;
> +
> +    return 0;
> +}
> +
>   static const struct arch_info *get_arch_info(libxl__gc *gc,
>                                                const struct xc_dom_image *dom)
>   {
> @@ -971,6 +1074,9 @@ next_resize:
>           if (info->tee == LIBXL_TEE_TYPE_OPTEE)
>               FDT( make_optee_node(gc, fdt) );
>   
> +        if (libxl_defbool_val(info->arch_arm.vpci))
> +            FDT( make_vpci_node(gc, fdt, ainfo, dom) );
> +
>           if (pfdt)
>               FDT( copy_partial_fdt(gc, fdt, pfdt) );
>   
> @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>       /* ACPI is disabled by default */
>       libxl_defbool_setdefault(&b_info->acpi, false);
>   
> +    /* VPCI is disabled by default */
> +    libxl_defbool_setdefault(&b_info->arch_arm.vpci, false);

Here you say that vPCI is disabled by default. However, AFAICT, in the 
previous patch you will unconditionally enable vPCI if Xen has been 
built with VPCI.

Rather than unconditionally enable VPCI for a domain, I think we should 
introduce a new flag that is passed to Xen at the domain creation. The 
flag will be set based on arch_arm.vpci.

> +
>       if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
>           return;
>   
> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> index 3f9fff653a..78b1ddf0b8 100644
> --- a/tools/libs/light/libxl_types.idl
> +++ b/tools/libs/light/libxl_types.idl
> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>   
>       ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>                                  ("vuart", libxl_vuart_type),
> +                               ("vpci", libxl_defbool),

Any new addition in the structure should be accompanied with a 
LIBXL_HAVE_* in the libxl.h header.

>                                 ])),
>       ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
>                                 ])),
> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
> index 17dddb4cd5..ffafbeffb4 100644
> --- a/tools/xl/xl_parse.c
> +++ b/tools/xl/xl_parse.c
> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source,
>           }
>           if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
>               libxl_defbool_set(&b_info->u.pv.e820_host, true);
> +        if (d_config->num_pcidevs)
> +            libxl_defbool_set(&b_info->arch_arm.vpci, true);
>       }
>   
>       if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) {
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 0a9749e768..01d13e669e 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t;
>   #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
>   #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
>   
> +/* PCI-PCIe memory space types */
> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
> +#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)

What the size of those regions?

> +
> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
> +#define GUEST_VPCI_PREFETCH_MEM_ADDR  xen_mk_ullong(0x900000000)
> +#define GUEST_VPCI_MEM_ADDR           xen_mk_ullong(0x20000000)
So far the memory layout defines the address in ascending order. So 
please add that after GUEST_RAM_BANK_BASES_*.

However, if I am not mistaken that base address you provide will clash 
with RAM bank 1. It also seem to be pretty high which means that this 
will not work for 32-bit domain or on CPUs that don't allow offer large 
IPA bits.

I think we need to start making the guest layout more dynamic. The VPCI 
memory space would have to go right after the end of the RAM allocated 
for a given guest.

> +
> +#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x2000000000)
> +#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x04000000)

It would be better if the size for each region is defined right after 
each base.

Also, how did you decide the size of each region?

> +
>   /*
>    * 16MB == 4096 pages reserved for guest to use as a region to map its
>    * grant table in.
> diff --git a/xen/include/public/device_tree_defs.h b/xen/include/public/device_tree_defs.h
> index 209d43de3f..d24d86b1fc 100644
> --- a/xen/include/public/device_tree_defs.h
> +++ b/xen/include/public/device_tree_defs.h
> @@ -7,6 +7,7 @@
>    * onwards. Reserve a high value for the GIC phandle.
>    */
>   #define GUEST_PHANDLE_GIC (65000)
> +#define GUEST_PHANDLE_ITS (64999)
>   
>   #define GUEST_ROOT_ADDRESS_CELLS 2
>   #define GUEST_ROOT_SIZE_CELLS 2
> 

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
  2021-08-19 12:35   ` Julien Grall
@ 2021-08-19 13:40     ` Jan Beulich
  2021-08-20 13:05     ` Rahul Singh
  1 sibling, 0 replies; 98+ messages in thread
From: Jan Beulich @ 2021-08-19 13:40 UTC (permalink / raw)
  To: Julien Grall, Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk, xen-devel

On 19.08.2021 14:35, Julien Grall wrote:
> On 19/08/2021 13:02, Rahul Singh wrote:
>> Hardware domain is in charge of doing the PCI enumeration and will
>> discover the PCI devices and then will communicate to XEN via hyper
>> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
> 
> There are other PHYSDEVOP operations to add PCI devices. I think it is 
> fine to only implement the latest (CC Jan for some opinion and confirm 
> this is the latest). However, this ought to be explained in the commit 
> message.

I don't think "latest" matters much here. Considering there was no
physdevop support at all on Arm, enabling whichever set seems like
a good fit would be okay.

Having written this I realize that by "latest" you may mean whether
the used sub-ops have not been obsoleted by newer ones (rather than
the last ones that were added to the physdevops set). While indeed
PHYSDEVOP_pci_device_add hasn't been superseded so far, I have a
vague recollection of there being some missing part. Without me
remembering details I'm afraid using what is there is the best we
can do for for the moment. However, ...

>> --- a/xen/arch/arm/physdev.c
>> +++ b/xen/arch/arm/physdev.c
>> @@ -9,12 +9,45 @@
>>   #include <xen/errno.h>
>>   #include <xen/sched.h>
>>   #include <asm/hypercall.h>
>> -
>> +#include <xen/guest_access.h>
>> +#include <xsm/xsm.h>
>>   
>>   int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>>   {
>> -    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>> -    return -ENOSYS;
>> +    int ret = 0;
>> +
>> +    switch ( cmd )
>> +    {
>> +#ifdef CONFIG_HAS_PCI
>> +    case PHYSDEVOP_pci_device_add: {
>> +        struct physdev_pci_device_add add;
>> +        struct pci_dev_info pdev_info;
>> +        nodeid_t node = NUMA_NO_NODE;
>> +
>> +        ret = -EFAULT;
>> +        if ( copy_from_guest(&add, arg, 1) != 0 )
>> +            break;
>> +
>> +        pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
>> +        if ( add.flags & XEN_PCI_DEV_VIRTFN )
>> +        {
>> +            pdev_info.is_virtfn = 1;
>> +            pdev_info.physfn.bus = add.physfn.bus;
>> +            pdev_info.physfn.devfn = add.physfn.devfn;
>> +        }
>> +        else
>> +            pdev_info.is_virtfn = 0;
>> +
>> +        ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
>> +        break;
>> +    }

... I don't think it should be only "add" which gets supported. "remove"
exists not just for the purpose of hot-unplug, but also for Dom0 to
remove (and then re-add) devices after e.g. bus re-numbering. (There are
some gaps there iirc, but still ...)

> This is pretty much a copy of the x86 version without the NUMA bit. So I 
> think we want to move the implementation in common code.

+1 (if sensibly possible)

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-19 12:18   ` Julien Grall
@ 2021-08-19 14:16     ` Rahul Singh
  2021-09-07 10:01       ` Julien Grall
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-19 14:16 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Jan Beulich, Andrew Cooper,
	Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Stefano Stabellini,
	Paul Durrant, Daniel De Graaf, Daniel P. Smith

Hi Julien,

> On 19 Aug 2021, at 1:18 pm, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> MSI code that implements MSI functionality to support MSI within XEN is
>> not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate
> 
> Can you clarify what you mean by not usable? Is it because we lack of support or we have no plan to use it?

We have no plan to use it. Code moved to CONFIG_HAS_PCI_MSI will only be required for ARM if we 
decide to support PCI device access (PCI MSI interrupt support) within XEN. As of now, we are planning 
to add support for PCI device access for DOM0/DOMU guests not for XEN. 

Regards,
Rahul


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-19 12:02 ` [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled Rahul Singh
  2021-08-19 12:28   ` Julien Grall
@ 2021-08-20  7:01   ` Jan Beulich
  2021-08-20 11:21     ` Rahul Singh
  2021-09-09 13:16   ` Julien Grall
  2 siblings, 1 reply; 98+ messages in thread
From: Jan Beulich @ 2021-08-20  7:01 UTC (permalink / raw)
  To: Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, xen-devel

On 19.08.2021 14:02, Rahul Singh wrote:
> Compilation error is observed when HAS_PCI is enabled for ARM
> architecture.
> 
> Add definition for arch_iommu_use_permitted() and
> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to
> fix compilation error.
> 
> pci.c: In function ‘deassign_device’:
> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’;
> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration]
>             pci_to_dev(pdev));
> 
> pci.c:18: undefined reference to `pci_conf_read16’
> pci.c:880: undefined reference to `arch_pci_clean_pirqs’
> pci.c:1392: undefined reference to `arch_iommu_use_permitted'
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>

A couple of nits, notwithstanding Julien's more general concern:

> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-access.c
> @@ -0,0 +1,61 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/pci.h>
> +
> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
> +                                unsigned int len)
> +{
> +    return ~0U;
> +}
> +
> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
> +                             unsigned int len, uint32_t val)
> +{
> +}
> +
> +/*
> + * Wrappers for all PCI configuration access functions.
> + */
> +
> +#define PCI_OP_WRITE(size, type)                                            \
> +    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val)  \

One of the two blanks preceding the backslash wants to move after the first
comma. And the blank preceding the opening parenthesis wants to be dropped.

> +{                                                                           \
> +    pci_config_write(sbdf, reg, size / 8, val);                             \
> +}
> +
> +#define PCI_OP_READ(size, type)                                             \
> +    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)            \

The latter of the two applies here as well.

> +{                                                                           \
> +    return pci_config_read(sbdf, reg, size / 8);                            \
> +}
> +
> +PCI_OP_READ(8, u8)
> +PCI_OP_READ(16, u16)
> +PCI_OP_READ(32, u32)
> +PCI_OP_WRITE(8, u8)
> +PCI_OP_WRITE(16, u16)
> +PCI_OP_WRITE(32, u32)

We aim at eliminating u<N> uses in favor of uint<N>_t - please don't
introduce new uses.

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI
  2021-08-19 12:02 ` [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI Rahul Singh
@ 2021-08-20  7:06   ` Jan Beulich
  2021-08-20 11:41     ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Jan Beulich @ 2021-08-20  7:06 UTC (permalink / raw)
  To: Rahul Singh; +Cc: bertrand.marquis, Paul Durrant, xen-devel

On 19.08.2021 14:02, Rahul Singh wrote:
> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
> architecture. Move the code under CONFIG_X86 flag to gate the code for
> ARM.

Please give at least one example of what it is that fails to compile.
What an appropriate action is depends on the nature of the error(s),
and from looking at the enclosed code I cannot easily see what it
might be that breaks for Arm.

And as suggested by Julien for the earlier patch - you will want to
re-order things such that compilation doesn't "break" in the first
place. Title and description would then want adjusting accordingly.

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-19 12:28   ` Julien Grall
@ 2021-08-20 10:30     ` Rahul Singh
  2021-08-20 11:37       ` Julien Grall
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-20 10:30 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Julien,

> On 19 Aug 2021, at 1:28 pm, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> Compilation error is observed when HAS_PCI is enabled for ARM
>> architecture.
> 
> To be pedantic, what you are trying to solve is not a compilation error but the fact that the PCI mandates helpers that doesn't yet exist on Arm. So...
> 
>> Add definition for arch_iommu_use_permitted() and
>> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to
>> fix compilation error.
> 
> ... I am not really in favor of adding dummy implementation here. Instead, the series should be re-ordered so we add the pci-access helpers first and then enable HAS_PCI towards the end of the series.

Ok. I will remove the dummy function implementation from this patch and will enable the HAS_PCI 
after adding the PCI access functions patch and other bits.

> 
>> pci.c: In function ‘deassign_device’:
>> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’;
>> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration]
>>             pci_to_dev(pdev));
>> pci.c:18: undefined reference to `pci_conf_read16’
>> pci.c:880: undefined reference to `arch_pci_clean_pirqs’
>> pci.c:1392: undefined reference to `arch_iommu_use_permitted'
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>  xen/arch/arm/Makefile               |  1 +
>>  xen/arch/arm/pci/Makefile           |  2 +
>>  xen/arch/arm/pci/pci-access.c       | 61 +++++++++++++++++++++++++++++
>>  xen/arch/arm/pci/pci.c              | 32 +++++++++++++++
>>  xen/drivers/passthrough/arm/iommu.c |  5 +++
>>  xen/include/asm-arm/pci.h           | 33 ++++++++++++++--
>>  6 files changed, 131 insertions(+), 3 deletions(-)
>>  create mode 100644 xen/arch/arm/pci/Makefile
>>  create mode 100644 xen/arch/arm/pci/pci-access.c
>>  create mode 100644 xen/arch/arm/pci/pci.c
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 3d3b97b5b4..0e14a5e5c8 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>>  obj-y += platforms/
>>  endif
>>  obj-$(CONFIG_TEE) += tee/
>> +obj-$(CONFIG_HAS_PCI) += pci/
>>    obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>  obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> new file mode 100644
>> index 0000000000..a9ee0b9b44
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -0,0 +1,2 @@
>> +obj-y += pci.o
>> +obj-y += pci-access.o
>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>> new file mode 100644
>> index 0000000000..b938047c03
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -0,0 +1,61 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/pci.h>
>> +
>> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>> +                                unsigned int len)
>> +{
>> +    return ~0U;
>> +}
>> +
>> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>> +                             unsigned int len, uint32_t val)
>> +{
>> +}
>> +
>> +/*
>> + * Wrappers for all PCI configuration access functions.
>> + */
>> +
>> +#define PCI_OP_WRITE(size, type)                                            \
>> +    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val)  \
>> +{                                                                           \
>> +    pci_config_write(sbdf, reg, size / 8, val);                             \
>> +}
>> +
>> +#define PCI_OP_READ(size, type)                                             \
>> +    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)            \
>> +{                                                                           \
>> +    return pci_config_read(sbdf, reg, size / 8);                            \
>> +}
>> +
>> +PCI_OP_READ(8, u8)
>> +PCI_OP_READ(16, u16)
>> +PCI_OP_READ(32, u32)
>> +PCI_OP_WRITE(8, u8)
>> +PCI_OP_WRITE(16, u16)
>> +PCI_OP_WRITE(32, u32)
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>> new file mode 100644
>> index 0000000000..dc55d23778
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci.c
>> @@ -0,0 +1,32 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/pci.h>
>> +
>> +int arch_pci_clean_pirqs(struct domain *d)
>> +{
>> +    return 0;
>> +}
> 
> Please add a comment explaining why this just returns 0.

Here is the comment that I will add in next version.
/*
 * Return 0 as on ARM there is no pci physical irqs that required cleanup.
 */ 

> 
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c
>> index db3b07a571..fdec1c5547 100644
>> --- a/xen/drivers/passthrough/arm/iommu.c
>> +++ b/xen/drivers/passthrough/arm/iommu.c
>> @@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d)
>>  void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
>>  {
>>  }
>> +
>> +bool arch_iommu_use_permitted(const struct domain *d)
>> +{
>> +    return true;
>> +}
> 
> Please add a comment explaining why returning true is always fine.

Here is the comment that I will add in next version:

/*
 * Return true as iommu use is always permitted if mem-sharing, 
 * mem-paging, or log-dirty functionality is not enabled.
 */ 

Regards,
Rahul

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-20  7:01   ` Jan Beulich
@ 2021-08-20 11:21     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-20 11:21 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Bertrand Marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, xen-devel

Hi Jan

> On 20 Aug 2021, at 8:01 am, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 19.08.2021 14:02, Rahul Singh wrote:
>> Compilation error is observed when HAS_PCI is enabled for ARM
>> architecture.
>> 
>> Add definition for arch_iommu_use_permitted() and
>> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to
>> fix compilation error.
>> 
>> pci.c: In function ‘deassign_device’:
>> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’;
>> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration]
>>            pci_to_dev(pdev));
>> 
>> pci.c:18: undefined reference to `pci_conf_read16’
>> pci.c:880: undefined reference to `arch_pci_clean_pirqs’
>> pci.c:1392: undefined reference to `arch_iommu_use_permitted'
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> 
> A couple of nits, notwithstanding Julien's more general concern:
> 
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -0,0 +1,61 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/pci.h>
>> +
>> +static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>> +                                unsigned int len)
>> +{
>> +    return ~0U;
>> +}
>> +
>> +static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>> +                             unsigned int len, uint32_t val)
>> +{
>> +}
>> +
>> +/*
>> + * Wrappers for all PCI configuration access functions.
>> + */
>> +
>> +#define PCI_OP_WRITE(size, type)                                            \
>> +    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val)  \
> 
> One of the two blanks preceding the backslash wants to move after the first
> comma. And the blank preceding the opening parenthesis wants to be dropped.

I will modify as below . Please correct me if I misunderstood wrongly.

#define PCI_OP_WRITE(size, type)                            \                   
    void pci_conf_write##size(pci_sbdf_t sbdf,              \                   
                              unsigned int reg, type val)  	     \                   
{                                                                                    \                   
    pci_config_write(sbdf, reg, size / 8, val);                   \                   
}

> 
>> +{                                                                           \
>> +    pci_config_write(sbdf, reg, size / 8, val);                             \
>> +}
>> +
>> +#define PCI_OP_READ(size, type)                                             \
>> +    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)            \
> 
> The latter of the two applies here as well.
> 
>> +{                                                                           \
>> +    return pci_config_read(sbdf, reg, size / 8);                            \
>> +}
>> +
>> +PCI_OP_READ(8, u8)
>> +PCI_OP_READ(16, u16)
>> +PCI_OP_READ(32, u32)
>> +PCI_OP_WRITE(8, u8)
>> +PCI_OP_WRITE(16, u16)
>> +PCI_OP_WRITE(32, u32)
> 
> We aim at eliminating u<N> uses in favor of uint<N>_t - please don't
> introduce new uses.

Ok . I will modify in next version.

Regards,
Rahul
> 
> Jan


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-20 10:30     ` Rahul Singh
@ 2021-08-20 11:37       ` Julien Grall
  2021-08-20 11:55         ` Jan Beulich
  0 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-08-20 11:37 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Rahul,

On 20/08/2021 11:30, Rahul Singh wrote:
>> Please add a comment explaining why this just returns 0.
> 
> Here is the comment that I will add in next version.
> /*
>   * Return 0 as on ARM there is no pci physical irqs that required cleanup.
>   */

In this context, PIRQ means an interrupts that was routed to the guest 
and could be mapped to an event channel. We have no such concept on Arm 
(see allocate_pirq_struct()).

So I would write "PIRQ event channels are not supported on Arm, so 
nothing to do".

> 
>>
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * tab-width: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c
>>> index db3b07a571..fdec1c5547 100644
>>> --- a/xen/drivers/passthrough/arm/iommu.c
>>> +++ b/xen/drivers/passthrough/arm/iommu.c
>>> @@ -135,3 +135,8 @@ void arch_iommu_domain_destroy(struct domain *d)
>>>   void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
>>>   {
>>>   }
>>> +
>>> +bool arch_iommu_use_permitted(const struct domain *d)
>>> +{
>>> +    return true;
>>> +}
>>
>> Please add a comment explaining why returning true is always fine.
> 
> Here is the comment that I will add in next version:
> 
> /*
>   * Return true as iommu use is always permitted if mem-sharing,
>   * mem-paging, or log-dirty functionality is not enabled.
>   */
How about writing "Unlike x86, Arm doesn't support mem-sharing, 
mem-paging and log-dirty (yet). So there is no restriction to use the 
IOMMU".

This would make clear why you are mentioning mem-sharing, mem-paging & co.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI
  2021-08-20  7:06   ` Jan Beulich
@ 2021-08-20 11:41     ` Rahul Singh
  2021-08-20 11:54       ` Jan Beulich
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-20 11:41 UTC (permalink / raw)
  To: Jan Beulich; +Cc: Bertrand Marquis, Paul Durrant, xen-devel

Hi Jan

> On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 19.08.2021 14:02, Rahul Singh wrote:
>> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
>> architecture. Move the code under CONFIG_X86 flag to gate the code for
>> ARM.
> 
> Please give at least one example of what it is that fails to compile.
> What an appropriate action is depends on the nature of the error(s),
> and from looking at the enclosed code I cannot easily see what it
> might be that breaks for Arm.

I am observing below error when enabled ACPI && HAS_PCI for ARM.

prelink.o: In function `pcie_aer_get_firmware_first’:
/xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse'
aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined

I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code.
obj-$(CONFIG_X86) += apei/

I am not sure whether we need this code for ARM architecture 
that is why I gate the code for ARM via CONFIG_X86

> 
> And as suggested by Julien for the earlier patch - you will want to
> re-order things such that compilation doesn't "break" in the first
> place. Title and description would then want adjusting accordingly.

Let me reorder  the patch series in next version.
Regards,
Rahul
> 
> Jan
> 





^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI
  2021-08-20 11:41     ` Rahul Singh
@ 2021-08-20 11:54       ` Jan Beulich
  2021-09-09  1:11         ` Stefano Stabellini
  0 siblings, 1 reply; 98+ messages in thread
From: Jan Beulich @ 2021-08-20 11:54 UTC (permalink / raw)
  To: Rahul Singh; +Cc: Bertrand Marquis, Paul Durrant, xen-devel

On 20.08.2021 13:41, Rahul Singh wrote:
> Hi Jan
> 
>> On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 19.08.2021 14:02, Rahul Singh wrote:
>>> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
>>> architecture. Move the code under CONFIG_X86 flag to gate the code for
>>> ARM.
>>
>> Please give at least one example of what it is that fails to compile.
>> What an appropriate action is depends on the nature of the error(s),
>> and from looking at the enclosed code I cannot easily see what it
>> might be that breaks for Arm.
> 
> I am observing below error when enabled ACPI && HAS_PCI for ARM.
> 
> prelink.o: In function `pcie_aer_get_firmware_first’:
> /xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse'
> aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined
> 
> I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code.
> obj-$(CONFIG_X86) += apei/
> 
> I am not sure whether we need this code for ARM architecture 
> that is why I gate the code for ARM via CONFIG_X86

So you Arm folks will probably want to settle on that aspect first. What
is wanted to keep things building depends on that.

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-20 11:37       ` Julien Grall
@ 2021-08-20 11:55         ` Jan Beulich
  2021-08-20 12:10           ` Julien Grall
  0 siblings, 1 reply; 98+ messages in thread
From: Jan Beulich @ 2021-08-20 11:55 UTC (permalink / raw)
  To: Julien Grall, Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

On 20.08.2021 13:37, Julien Grall wrote:
> On 20/08/2021 11:30, Rahul Singh wrote:
>>> Please add a comment explaining why this just returns 0.
>>
>> Here is the comment that I will add in next version.
>> /*
>>   * Return 0 as on ARM there is no pci physical irqs that required cleanup.
>>   */
> 
> In this context, PIRQ means an interrupts that was routed to the guest 
> and could be mapped to an event channel. We have no such concept on Arm 
> (see allocate_pirq_struct()).
> 
> So I would write "PIRQ event channels are not supported on Arm, so 
> nothing to do".

Does this mean ASSERT_UNREACHABLE() might be appropriate here?

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-20 11:55         ` Jan Beulich
@ 2021-08-20 12:10           ` Julien Grall
  0 siblings, 0 replies; 98+ messages in thread
From: Julien Grall @ 2021-08-20 12:10 UTC (permalink / raw)
  To: Jan Beulich, Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Jan,

On 20/08/2021 12:55, Jan Beulich wrote:
> On 20.08.2021 13:37, Julien Grall wrote:
>> On 20/08/2021 11:30, Rahul Singh wrote:
>>>> Please add a comment explaining why this just returns 0.
>>>
>>> Here is the comment that I will add in next version.
>>> /*
>>>    * Return 0 as on ARM there is no pci physical irqs that required cleanup.
>>>    */
>>
>> In this context, PIRQ means an interrupts that was routed to the guest
>> and could be mapped to an event channel. We have no such concept on Arm
>> (see allocate_pirq_struct()).
>>
>> So I would write "PIRQ event channels are not supported on Arm, so
>> nothing to do".
> 
> Does this mean ASSERT_UNREACHABLE() might be appropriate here?

Unfortunately no. This is call unconditionally from pci_release_devices().

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-19 12:31   ` Julien Grall
@ 2021-08-20 12:19     ` Rahul Singh
  2021-08-20 14:34       ` Julien Grall
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-20 12:19 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Julien,

> On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> Add cmdline boot option "pci=on" to enable/disable the PCI init during
>> boot.
> 
> I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not.

Yes. I will modify the option to "pci-passthrough== <boolean>"
> 
> Can you also clarify why a user would want to select "pci=off"?

As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to 
boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device.

I am ok to drop this patch if you feel adding the option is not required at all.
 
Regards,
Rahul

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
  2021-08-19 12:35   ` Julien Grall
  2021-08-19 13:40     ` Jan Beulich
@ 2021-08-20 13:05     ` Rahul Singh
  1 sibling, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-20 13:05 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini,
	Volodymyr Babchuk, Jan Beulich

Hi Julien,

> On 19 Aug 2021, at 1:35 pm, Julien Grall <julien@xen.org> wrote:
> 
> (+ Jan)
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> Hardware domain is in charge of doing the PCI enumeration and will
>> discover the PCI devices and then will communicate to XEN via hyper
>> call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
> 
> There are other PHYSDEVOP operations to add PCI devices. I think it is fine to only implement the latest (CC Jan for some opinion and confirm this is the latest). However, this ought to be explained in the commit message.

As per Jan comments I will add the PHYSDEVOP_pci_device_remove() in the next version.
> 
> Also, public/arch-arm.h will need to be updated as we now support the PHYSDEVOP hypercall.

Ok.
> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>  xen/arch/arm/physdev.c | 39 ++++++++++++++++++++++++++++++++++++---
>>  1 file changed, 36 insertions(+), 3 deletions(-)
>> diff --git a/xen/arch/arm/physdev.c b/xen/arch/arm/physdev.c
>> index e91355fe22..ccce8f0eba 100644
>> --- a/xen/arch/arm/physdev.c
>> +++ b/xen/arch/arm/physdev.c
>> @@ -9,12 +9,45 @@
>>  #include <xen/errno.h>
>>  #include <xen/sched.h>
>>  #include <asm/hypercall.h>
>> -
>> +#include <xen/guest_access.h>
>> +#include <xsm/xsm.h>
>>    int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>>  {
>> -    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
>> -    return -ENOSYS;
>> +    int ret = 0;
>> +
>> +    switch ( cmd )
>> +    {
>> +#ifdef CONFIG_HAS_PCI
>> +    case PHYSDEVOP_pci_device_add: {
>> +        struct physdev_pci_device_add add;
>> +        struct pci_dev_info pdev_info;
>> +        nodeid_t node = NUMA_NO_NODE;
>> +
>> +        ret = -EFAULT;
>> +        if ( copy_from_guest(&add, arg, 1) != 0 )
>> +            break;
>> +
>> +        pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
>> +        if ( add.flags & XEN_PCI_DEV_VIRTFN )
>> +        {
>> +            pdev_info.is_virtfn = 1;
>> +            pdev_info.physfn.bus = add.physfn.bus;
>> +            pdev_info.physfn.devfn = add.physfn.devfn;
>> +        }
>> +        else
>> +            pdev_info.is_virtfn = 0;
>> +
>> +        ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
>> +        break;
>> +    }
> 
> This is pretty much a copy of the x86 version without the NUMA bit. So I think we want to move the implementation in common code.

Ok. Let me move the PHYSDEVOP_pci_device_* to common code.

Regards,
Rahul



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-20 12:19     ` Rahul Singh
@ 2021-08-20 14:34       ` Julien Grall
  2021-08-20 14:37         ` Jan Beulich
  0 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-08-20 14:34 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk



On 20/08/2021 13:19, Rahul Singh wrote:
> Hi Julien,

Hi Rahul,

> 
>> On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote:
>>
>> Hi Rahul,
>>
>> On 19/08/2021 13:02, Rahul Singh wrote:
>>> Add cmdline boot option "pci=on" to enable/disable the PCI init during
>>> boot.
>>
>> I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not.
> 
> Yes. I will modify the option to "pci-passthrough== <boolean>"
>>
>> Can you also clarify why a user would want to select "pci=off"?
> 
> As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to
> boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device.

Dom0 will always have direct access to the PCI device. The only 
difference is whether the access to the hostbridge and config space will 
be trapped by Xen. I expect the both to mainly happen during boot and 
therefore the overhead will be limited.

> 
> I am ok to drop this patch if you feel adding the option is not required at all.
One of the reason I could see this option to be useful is to figure out 
if an issue occurs because of the hostbridge emulation. Yet, I am still 
not fully convinced adding an option is worth it.

Jan and others, any opinions?

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-20 14:34       ` Julien Grall
@ 2021-08-20 14:37         ` Jan Beulich
  2021-09-09 23:46           ` Stefano Stabellini
  0 siblings, 1 reply; 98+ messages in thread
From: Jan Beulich @ 2021-08-20 14:37 UTC (permalink / raw)
  To: Julien Grall, Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

On 20.08.2021 16:34, Julien Grall wrote:
> On 20/08/2021 13:19, Rahul Singh wrote:
>>> On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote:
>>> On 19/08/2021 13:02, Rahul Singh wrote:
>>>> Add cmdline boot option "pci=on" to enable/disable the PCI init during
>>>> boot.
>>>
>>> I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not.
>>
>> Yes. I will modify the option to "pci-passthrough== <boolean>"
>>>
>>> Can you also clarify why a user would want to select "pci=off"?
>>
>> As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to
>> boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device.
> 
> Dom0 will always have direct access to the PCI device. The only 
> difference is whether the access to the hostbridge and config space will 
> be trapped by Xen. I expect the both to mainly happen during boot and 
> therefore the overhead will be limited.
> 
>>
>> I am ok to drop this patch if you feel adding the option is not required at all.
> One of the reason I could see this option to be useful is to figure out 
> if an issue occurs because of the hostbridge emulation. Yet, I am still 
> not fully convinced adding an option is worth it.
> 
> Jan and others, any opinions?

Well, if there's a proper fallback, then why not allow using it in
case of problems?

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl
  2021-08-19 13:00   ` Julien Grall
@ 2021-08-20 16:03     ` Rahul Singh
  2021-09-09 13:59       ` Julien Grall
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-20 16:03 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Ian Jackson, Wei Liu,
	Anthony PERARD, Juergen Gross, Andrew Cooper, George Dunlap,
	Jan Beulich, Stefano Stabellini, Volodymyr Babchuk

Hi Julien,

> On 19 Aug 2021, at 2:00 pm, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> libxl will create an emulated PCI device tree node in the device tree to
>> enable the guest OS to discover the virtual PCI during guest boot.
>> Emulated PCI device tree node will only be created when there is any
>> device assigned to guest.
>> A new area has been reserved in the arm guest physical map at
>> which the VPCI bus is declared in the device tree (reg and ranges
>> parameters of the node).
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>  tools/libs/light/libxl_arm.c          | 109 ++++++++++++++++++++++++++
>>  tools/libs/light/libxl_types.idl      |   1 +
>>  tools/xl/xl_parse.c                   |   2 +
>>  xen/include/public/arch-arm.h         |  11 +++
>>  xen/include/public/device_tree_defs.h |   1 +
>>  5 files changed, 124 insertions(+)
>> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
>> index e3140a6e00..a091e97e76 100644
>> --- a/tools/libs/light/libxl_arm.c
>> +++ b/tools/libs/light/libxl_arm.c
>> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>>      return fdt_property(fdt, "reg", regs, sizeof(regs));
>>  }
>>  +static int fdt_property_values(libxl__gc *gc, void *fdt,
>> +        const char *name, unsigned num_cells, ...)
>> +{
>> +    uint32_t prop[num_cells];
>> +    be32 *cells = &prop[0];
>> +    int i;
>> +    va_list ap;
>> +    uint32_t arg;
>> +
>> +    va_start(ap, num_cells);
>> +    for (i = 0 ; i < num_cells; i++) {
>> +        arg = va_arg(ap, uint32_t);
>> +        set_cell(&cells, 1, arg);
>> +    }
>> +    va_end(ap);
>> +
>> +    return fdt_property(fdt, name, prop, sizeof(prop));
>> +}
>> +
>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
>> +                                    unsigned addr_cells,
>> +                                    unsigned size_cells,
>> +                                    unsigned num_regs, ...)
>> +{
>> +    uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
>> +    be32 *cells = &regs[0];
>> +    int i;
>> +    va_list ap;
>> +    uint64_t arg;
>> +
>> +    va_start(ap, num_regs);
>> +    for (i = 0 ; i < num_regs; i++) {
>> +        /* Set the memory bit field */
>> +        arg = va_arg(ap, uint64_t);
>> +        set_cell(&cells, 1, arg);
>> +
>> +        /* Set the vpci bus address */
>> +        arg = addr_cells ? va_arg(ap, uint64_t) : 0;
>> +        set_cell(&cells, addr_cells , arg);
>> +
>> +        /* Set the cpu bus address where vpci address is mapped */
>> +        set_cell(&cells, addr_cells, arg);
>> +
>> +        /* Set the vpci size requested */
>> +        arg = size_cells ? va_arg(ap, uint64_t) : 0;
>> +        set_cell(&cells, size_cells,arg);
>> +    }
>> +    va_end(ap);
>> +
>> +    return fdt_property(fdt, "ranges", regs, sizeof(regs));
>> +}
>> +
>>  static int make_root_properties(libxl__gc *gc,
>>                                  const libxl_version_info *vers,
>>                                  void *fdt)
>> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>>      return 0;
>>  }
>>  +static int make_vpci_node(libxl__gc *gc, void *fdt,
>> +        const struct arch_info *ainfo,
>> +        struct xc_dom_image *dom)
>> +{
>> +    int res;
>> +    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
>> +    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
>> +    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
>> +
>> +    res = fdt_begin_node(fdt, name);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
>> +    if (res) return res;
>> +
>> +    res = fdt_property_string(fdt, "device_type", "pci");
>> +    if (res) return res;
>> +
>> +    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>> +            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17);
> 
> AFAICT, the "bus-range" is optional. Can you explain why we need it?

We need it to implement the function pci_ecam_map_bus().
> 
>> +    if (res) return res;
>> +
>> +    res = fdt_property_cell(fdt, "#address-cells", 3);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_cell(fdt, "#size-cells", 2);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_string(fdt, "status", "okay");
>> +    if (res) return res;
>> +
>> +    res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>> +        GUEST_ROOT_SIZE_CELLS, 2,
>> +        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE,
>> +        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR,
>> +        GUEST_VPCI_PREFETCH_MEM_SIZE);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS,
>> +                              0, 0x10000);
> 
> The ITS is not exposed to the guest so far. So I think this wants to be moved to the future series that will add ITS support.

Ok. I will remove this in next version.

>> +    if (res) return res;
>> +
>> +    res = fdt_end_node(fdt);
>> +    if (res) return res;
>> +
>> +    return 0;
>> +}
>> +
>>  static const struct arch_info *get_arch_info(libxl__gc *gc,
>>                                               const struct xc_dom_image *dom)
>>  {
>> @@ -971,6 +1074,9 @@ next_resize:
>>          if (info->tee == LIBXL_TEE_TYPE_OPTEE)
>>              FDT( make_optee_node(gc, fdt) );
>>  +        if (libxl_defbool_val(info->arch_arm.vpci))
>> +            FDT( make_vpci_node(gc, fdt, ainfo, dom) );
>> +
>>          if (pfdt)
>>              FDT( copy_partial_fdt(gc, fdt, pfdt) );
>>  @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>>      /* ACPI is disabled by default */
>>      libxl_defbool_setdefault(&b_info->acpi, false);
>>  +    /* VPCI is disabled by default */
>> +    libxl_defbool_setdefault(&b_info->arch_arm.vpci, false);
> 
> Here you say that vPCI is disabled by default. However, AFAICT, in the previous patch you will unconditionally enable vPCI if Xen has been built with VPCI.
> 
> Rather than unconditionally enable VPCI for a domain, I think we should introduce a new flag that is passed to Xen at the domain creation. The flag will be set based on arch_arm.vpci.

OK. I will introduce the flag in next version.

> 
>> +
>>      if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
>>          return;
>>  diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
>> index 3f9fff653a..78b1ddf0b8 100644
>> --- a/tools/libs/light/libxl_types.idl
>> +++ b/tools/libs/light/libxl_types.idl
>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>>        ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>>                                 ("vuart", libxl_vuart_type),
>> +                               ("vpci", libxl_defbool),
> 
> Any new addition in the structure should be accompanied with a LIBXL_HAVE_* in the libxl.h header.

OK.
> 
>>                                ])),
>>      ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
>>                                ])),
>> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
>> index 17dddb4cd5..ffafbeffb4 100644
>> --- a/tools/xl/xl_parse.c
>> +++ b/tools/xl/xl_parse.c
>> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source,
>>          }
>>          if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
>>              libxl_defbool_set(&b_info->u.pv.e820_host, true);
>> +        if (d_config->num_pcidevs)
>> +            libxl_defbool_set(&b_info->arch_arm.vpci, true);
>>      }
>>        if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) {
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index 0a9749e768..01d13e669e 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t;
>>  #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
>>  #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
>>  +/* PCI-PCIe memory space types */
>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
>> +#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
> 
> What the size of those regions?

Non Prefetch Memory: Size 64 MB start at 512 MB
Prefetch Memory: Size 128 GB start at 36 GB
> 
>> +
>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
>> +#define GUEST_VPCI_PREFETCH_MEM_ADDR  xen_mk_ullong(0x900000000)
>> +#define GUEST_VPCI_MEM_ADDR           xen_mk_ullong(0x20000000)
> So far the memory layout defines the address in ascending order. So please add that after GUEST_RAM_BANK_BASES_*.

Ok.
> 
> However, if I am not mistaken that base address you provide will clash with RAM bank 1. It also seem to be pretty high which means that this will not work for 32-bit domain or on CPUs that don't allow offer large IPA bits.

Yes I also checked that now that it is having clash with RAM bank 1.
There is unused space is guest memory that we can use for Non Prefetch Memory as per below guest memory map.
https://gitlab.com/xen-project/fusa/fusa-docs/-/blob/master/high-level/guest-memory-layout-arm.rst

Proposed value:
Non Prefetch Memory: Size 64 MB start at 0x22001000
Prefetch Memory: Size 4 GB start at 4 GB.

> 
> I think we need to start making the guest layout more dynamic. The VPCI memory space would have to go right after the end of the RAM allocated for a given guest.
> 
>> +
>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x2000000000)
>> +#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x04000000)
> 
> It would be better if the size for each region is defined right after each base.
OK. 

> 
> Also, how did you decide the size of each region?

I thought 64 MB will be sufficient. I think it should be based on number of devices we can assign to the guest. 

Regards,
Rahul

> 
>> +
>>  /*
>>   * 16MB == 4096 pages reserved for guest to use as a region to map its
>>   * grant table in.
>> diff --git a/xen/include/public/device_tree_defs.h b/xen/include/public/device_tree_defs.h
>> index 209d43de3f..d24d86b1fc 100644
>> --- a/xen/include/public/device_tree_defs.h
>> +++ b/xen/include/public/device_tree_defs.h
>> @@ -7,6 +7,7 @@
>>   * onwards. Reserve a high value for the GIC phandle.
>>   */
>>  #define GUEST_PHANDLE_GIC (65000)
>> +#define GUEST_PHANDLE_ITS (64999)
>>    #define GUEST_ROOT_ADDRESS_CELLS 2
>>  #define GUEST_ROOT_SIZE_CELLS 2
> 
> Cheers,
> 
> -- 
> Julien Grall



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest
  2021-08-19 12:12   ` Jan Beulich
  2021-08-19 12:40     ` Julien Grall
@ 2021-08-20 17:01     ` Rahul Singh
  1 sibling, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-08-20 17:01 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Bertrand Marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, xen-devel

Hi Jan,

> On 19 Aug 2021, at 1:12 pm, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 19.08.2021 14:02, Rahul Singh wrote:
>> --- a/xen/arch/arm/domctl.c
>> +++ b/xen/arch/arm/domctl.c
>> @@ -173,6 +173,8 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
>> 
>>         return rc;
>>     }
>> +    case XEN_DOMCTL_ioport_permission:
>> +        return 0;
> 
> I don't think returning success for something that doesn't make
> much sense in the first place (there aren't truly "I/O ports" on
> Arm afaik) is a good idea.
> Instead I think the tool stack should
> avoid making arch-specific calls in an arch-independent way.

I agree with you let me try to modify the toolstack not to call the arch-specific call.

Regards,
Rahul
> 
>> --- a/xen/arch/arm/physdev.c
>> +++ b/xen/arch/arm/physdev.c
>> @@ -42,6 +42,9 @@ int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>>         break;
>>     }
>> #endif
>> +    case PHYSDEVOP_unmap_pirq:
>> +    case PHYSDEVOP_map_pirq:
>> +        break;
> 
> Less sure here, but I'm not convinced either.
> 
> Jan
> 



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-19 12:02 ` [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN Rahul Singh
  2021-08-19 12:18   ` Julien Grall
@ 2021-08-24 15:53   ` Jan Beulich
  2021-08-31 12:31     ` Rahul Singh
  2021-08-26 13:23   ` Daniel P. Smith
  2 siblings, 1 reply; 98+ messages in thread
From: Jan Beulich @ 2021-08-24 15:53 UTC (permalink / raw)
  To: Rahul Singh
  Cc: bertrand.marquis, Andrew Cooper, Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Julien Grall,
	Stefano Stabellini, Paul Durrant, Daniel De Graaf,
	Daniel P. Smith, xen-devel

On 19.08.2021 14:02, Rahul Singh wrote:
> --- /dev/null
> +++ b/xen/drivers/passthrough/msi.c
> @@ -0,0 +1,96 @@
> +/*
> + * Copyright (C) 2008,  Netronome Systems, Inc.

While generally copying copyright statements when splitting source
files is probably wanted (or even necessary) I doubt this is
suitable here: None of the MSI code that you move was contributed
by them afaict.

> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/msi.h>

You surely mean xen/msi.h here: Headers like this one should always
be included by the producer, no matter that it builds fine without.
Else you risk declarations and definitions to go out of sync.

> +#include <asm/hvm/io.h>
> +
> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)
> +{
> +    int rc;
> +
> +    if ( pdev->msix )
> +    {
> +        rc = pci_reset_msix_state(pdev);
> +        if ( rc )
> +            return rc;
> +        msixtbl_init(d);
> +    }
> +
> +    return 0;
> +}
> +
> +int pdev_msi_init(struct pci_dev *pdev)
> +{
> +    unsigned int pos;
> +
> +    INIT_LIST_HEAD(&pdev->msi_list);
> +
> +    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> +                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI);
> +    if ( pos )
> +    {
> +        uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos));
> +
> +        pdev->msi_maxvec = multi_msi_capable(ctrl);
> +    }
> +
> +    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> +                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX);
> +    if ( pos )
> +    {
> +        struct arch_msix *msix = xzalloc(struct arch_msix);
> +        uint16_t ctrl;
> +
> +        if ( !msix )
> +            return -ENOMEM;
> +
> +        spin_lock_init(&msix->table_lock);
> +
> +        ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos));
> +        msix->nr_entries = msix_table_size(ctrl);
> +
> +        pdev->msix = msix;
> +    }
> +
> +    return 0;
> +}
> +
> +void pdev_msi_deinit(struct pci_dev *pdev)
> +{
> +    XFREE(pdev->msix);
> +}
> +
> +void pdev_dump_msi(const struct pci_dev *pdev)
> +{
> +    const struct msi_desc *msi;
> +
> +    printk("- MSIs < ");
> +    list_for_each_entry ( msi, &pdev->msi_list, list )
> +        printk("%d ", msi->irq);
> +    printk(">");

While not an exact equivalent of the original code then, could I
talk you into adding an early list_empty() check, suppressing any
output from this function if that one returned "true"?

> @@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev)
>  static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
>  {
>      struct pci_dev *pdev;
> -    struct msi_desc *msi;
>  
>      printk("==== segment %04x ====\n", pseg->nr);
>  
>      list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
>      {
> -        printk("%pp - %pd - node %-3d - MSIs < ",
> +        printk("%pp - %pd - node %-3d ",

Together with the request above the trailin blank here also wants to
become a leading blank in pdev_dump_msi().

> --- /dev/null
> +++ b/xen/include/xen/msi.h
> @@ -0,0 +1,56 @@
> +/*
> + * Copyright (C) 2008,  Netronome Systems, Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef __XEN_MSI_H_
> +#define __XEN_MSI_H_
> +
> +#ifdef CONFIG_HAS_PCI_MSI
> +
> +#include <asm/msi.h>
> +
> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev);
> +int pdev_msi_init(struct pci_dev *pdev);
> +void pdev_msi_deinit(struct pci_dev *pdev);
> +void pdev_dump_msi(const struct pci_dev *pdev);
> +
> +#else /* !CONFIG_HAS_PCI_MSI */
> +static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)

Please be consistent with blank lines you add; here you also want one
after the #else.

> +{
> +    return 0;
> +}
> +
> +static inline int pdev_msi_init(struct pci_dev *pdev)
> +{
> +    return 0;
> +}
> +
> +static inline void pdev_msi_deinit(struct pci_dev *pdev) {}
> +static inline void pci_cleanup_msi(struct pci_dev *pdev) {}
> +static inline void pdev_dump_msi(const struct pci_dev *pdev) {}

Especially for (but perhaps not limited to) this !HAS_PCI_MSI case
(where you don't include asm/msi.h and its possible dependents)
please forward-declare struct-s you use in prototypes or inline
stubs (outside the #ifdef, that is). This will allow including
this header without having to care about prereq headers.

If you agree with and make all the suggested or requested changes,
feel free to add
Reviewed-by: Jan Beulich <jbeulich@suse.com>

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-08-19 12:02 ` [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
@ 2021-08-24 16:09   ` Jan Beulich
  2021-08-25  5:44     ` Oleksandr Andrushchenko
  2021-09-09 13:50   ` Julien Grall
  2021-09-10  0:26   ` Stefano Stabellini
  2 siblings, 1 reply; 98+ messages in thread
From: Jan Beulich @ 2021-08-24 16:09 UTC (permalink / raw)
  To: Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Paul Durrant, Roger Pau Monné,
	xen-devel

On 19.08.2021 14:02, Rahul Singh wrote:
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>      else
>          iommu_enable_device(pdev);
>  
> +#ifdef CONFIG_ARM
> +    ret = vpci_add_handlers(pdev);
> +    if ( ret ) {
> +        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
> +        goto out;
> +    }
> +#endif
>      pci_enable_acs(pdev);

I'm afraid I can't see why this is to be Arm-specific. The present
placement of the existing call to vpci_add_handlers() looks to be
sub-optimal anyway - did you look into whether it could be moved
to a place (potentially right here) fitting everyone? If you did,
would mind justifying the Arm-specific code in a non-Arm-specific
file in at least a post-commit-message remark?

If it were to remain like this, please add a blank line after the #endif.

> --- a/xen/drivers/vpci/Makefile
> +++ b/xen/drivers/vpci/Makefile
> @@ -1 +1,2 @@
> -obj-y += vpci.o header.o msi.o msix.o
> +obj-y += vpci.o header.o
> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o

I continue to consider this a wrong connection - HAS_PCI_MSI expresses
(quoting text from patch 1 of this series) "code that implements MSI
functionality to support MSI within XEN", while here we're talking of
vPCI (i.e. guest support). I can accept that this is transiently the
best you can do, but then please add a comment to this effect (if
need be in multiple places), such that future readers or people
wanting to further adjust this understand why it is the way it is.

But perhaps you instead want to introduce a HAS_VPCI_MSI (or, less
desirable because of possible ambiguity, HAS_VMSI) Kconfig option?

> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>       * FIXME: punching holes after the p2m has been set up might be racy for
>       * DomU usage, needs to be revisited.
>       */
> +#ifdef CONFIG_HAS_PCI_MSI
>      if ( map && !rom_only && vpci_make_msix_hole(pdev) )
>          return;
> +#endif

(This would be another such place.)

> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>  
>  #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>  
> -#define has_vpci(d)    ({ (void)(d); false; })
> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
> + * for ARM we enable support VPCI for guest domain also.
> + */

Comment style (/* goes on its own line).

> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); })

Personally I'd recommend to get away without using extensions whenever
possible, i.e. use

#define has_vpci(d) ((void)(d), IS_ENABLED(CONFIG_HAS_VPCI))

here.

> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -27,6 +27,14 @@ struct arch_pci_dev {
>      struct device dev;
>  };
>  
> +/* Arch-specific MSI data for vPCI. */
> +struct vpci_arch_msi {
> +};
> +
> +/* Arch-specific MSI-X entry data for vPCI. */
> +struct vpci_arch_msix_entry {
> +};

But isn't it that you don't support vPCI's MSI in the first place?
Perhaps the need for these would go away with CONFIG_HAS_VCPI_MSI?

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-08-24 16:09   ` Jan Beulich
@ 2021-08-25  5:44     ` Oleksandr Andrushchenko
  2021-08-25  6:35       ` Jan Beulich
  0 siblings, 1 reply; 98+ messages in thread
From: Oleksandr Andrushchenko @ 2021-08-25  5:44 UTC (permalink / raw)
  To: Jan Beulich, Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Paul Durrant, Roger Pau Monné,
	xen-devel

Hi, Jan!

On 24.08.21 19:09, Jan Beulich wrote:
> On 19.08.2021 14:02, Rahul Singh wrote:
>> --- a/xen/drivers/passthrough/pci.c
>> +++ b/xen/drivers/passthrough/pci.c
>> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>       else
>>           iommu_enable_device(pdev);
>>   
>> +#ifdef CONFIG_ARM
>> +    ret = vpci_add_handlers(pdev);
>> +    if ( ret ) {
>> +        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
>> +        goto out;
>> +    }
>> +#endif
>>       pci_enable_acs(pdev);
> I'm afraid I can't see why this is to be Arm-specific. The present
> placement of the existing call to vpci_add_handlers() looks to be
> sub-optimal anyway - did you look into whether it could be moved
> to a place (potentially right here) fitting everyone? If you did,
> would mind justifying the Arm-specific code in a non-Arm-specific
> file in at least a post-commit-message remark?
>
> If it were to remain like this, please add a blank line after the #endif.
>
>> --- a/xen/drivers/vpci/Makefile
>> +++ b/xen/drivers/vpci/Makefile
>> @@ -1 +1,2 @@
>> -obj-y += vpci.o header.o msi.o msix.o
>> +obj-y += vpci.o header.o
>> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
> I continue to consider this a wrong connection - HAS_PCI_MSI expresses
> (quoting text from patch 1 of this series) "code that implements MSI
> functionality to support MSI within XEN", while here we're talking of
> vPCI (i.e. guest support). I can accept that this is transiently the
> best you can do, but then please add a comment to this effect (if
> need be in multiple places), such that future readers or people
> wanting to further adjust this understand why it is the way it is.
>
> But perhaps you instead want to introduce a HAS_VPCI_MSI (or, less
> desirable because of possible ambiguity, HAS_VMSI) Kconfig option?

Eventually we'll have the code like that:

obj-y += vpci.o header.o msi.o msix.o
obj-$(CONFIG_X86) += x86/
obj-$(CONFIG_ARM) += arm/

So, this is indeed a transitional change. Will you be ok with a comment

about that then?

>
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>>        * FIXME: punching holes after the p2m has been set up might be racy for
>>        * DomU usage, needs to be revisited.
>>        */
>> +#ifdef CONFIG_HAS_PCI_MSI
>>       if ( map && !rom_only && vpci_make_msix_hole(pdev) )
>>           return;
>> +#endif
> (This would be another such place.)
>
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>>   
>>   #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>>   
>> -#define has_vpci(d)    ({ (void)(d); false; })
>> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
>> + * for ARM we enable support VPCI for guest domain also.
>> + */
> Comment style (/* goes on its own line).
>
>> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); })
> Personally I'd recommend to get away without using extensions whenever
> possible, i.e. use
>
> #define has_vpci(d) ((void)(d), IS_ENABLED(CONFIG_HAS_VPCI))
>
> here.
>
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -27,6 +27,14 @@ struct arch_pci_dev {
>>       struct device dev;
>>   };
>>   
>> +/* Arch-specific MSI data for vPCI. */
>> +struct vpci_arch_msi {
>> +};
>> +
>> +/* Arch-specific MSI-X entry data for vPCI. */
>> +struct vpci_arch_msix_entry {
>> +};
> But isn't it that you don't support vPCI's MSI in the first place?
> Perhaps the need for these would go away with CONFIG_HAS_VCPI_MSI?
>
> Jan
>
>


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-08-25  5:44     ` Oleksandr Andrushchenko
@ 2021-08-25  6:35       ` Jan Beulich
  0 siblings, 0 replies; 98+ messages in thread
From: Jan Beulich @ 2021-08-25  6:35 UTC (permalink / raw)
  To: Oleksandr Andrushchenko, Rahul Singh
  Cc: bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Paul Durrant, Roger Pau Monné,
	xen-devel

On 25.08.2021 07:44, Oleksandr Andrushchenko wrote:
> Hi, Jan!
> 
> On 24.08.21 19:09, Jan Beulich wrote:
>> On 19.08.2021 14:02, Rahul Singh wrote:
>>> --- a/xen/drivers/passthrough/pci.c
>>> +++ b/xen/drivers/passthrough/pci.c
>>> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>>       else
>>>           iommu_enable_device(pdev);
>>>   
>>> +#ifdef CONFIG_ARM
>>> +    ret = vpci_add_handlers(pdev);
>>> +    if ( ret ) {
>>> +        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
>>> +        goto out;
>>> +    }
>>> +#endif
>>>       pci_enable_acs(pdev);
>> I'm afraid I can't see why this is to be Arm-specific. The present
>> placement of the existing call to vpci_add_handlers() looks to be
>> sub-optimal anyway - did you look into whether it could be moved
>> to a place (potentially right here) fitting everyone? If you did,
>> would mind justifying the Arm-specific code in a non-Arm-specific
>> file in at least a post-commit-message remark?
>>
>> If it were to remain like this, please add a blank line after the #endif.
>>
>>> --- a/xen/drivers/vpci/Makefile
>>> +++ b/xen/drivers/vpci/Makefile
>>> @@ -1 +1,2 @@
>>> -obj-y += vpci.o header.o msi.o msix.o
>>> +obj-y += vpci.o header.o
>>> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
>> I continue to consider this a wrong connection - HAS_PCI_MSI expresses
>> (quoting text from patch 1 of this series) "code that implements MSI
>> functionality to support MSI within XEN", while here we're talking of
>> vPCI (i.e. guest support). I can accept that this is transiently the
>> best you can do, but then please add a comment to this effect (if
>> need be in multiple places), such that future readers or people
>> wanting to further adjust this understand why it is the way it is.
>>
>> But perhaps you instead want to introduce a HAS_VPCI_MSI (or, less
>> desirable because of possible ambiguity, HAS_VMSI) Kconfig option?
> 
> Eventually we'll have the code like that:
> 
> obj-y += vpci.o header.o msi.o msix.o
> obj-$(CONFIG_X86) += x86/
> obj-$(CONFIG_ARM) += arm/
> 
> So, this is indeed a transitional change. Will you be ok with a comment
> about that then?

Well, yes, as said - if this is a transitional state, then a comment
will do. I was suggesting the further Kconfig option merely because
all I've been hearing so far was that MSI works entirely differently
on Arm, and hence the MSI code we have is not going to be used there
at all. If that was a correct understanding of mine (including its
extending to vPCI's MSI code), then adding just a comment would
continue to be misleading imo.

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-19 12:02 ` [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN Rahul Singh
  2021-08-19 12:18   ` Julien Grall
  2021-08-24 15:53   ` Jan Beulich
@ 2021-08-26 13:23   ` Daniel P. Smith
  2 siblings, 0 replies; 98+ messages in thread
From: Daniel P. Smith @ 2021-08-26 13:23 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Jan Beulich, Andrew Cooper,
	Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Julien Grall,
	Stefano Stabellini, Paul Durrant, Daniel De Graaf

On 8/19/21 8:02 AM, Rahul Singh wrote:
> MSI code that implements MSI functionality to support MSI within XEN is
> not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate
> the code for ARM.
> 
> No functional change intended.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/x86/Kconfig             |  1 +
>  xen/drivers/passthrough/Makefile |  1 +
>  xen/drivers/passthrough/msi.c    | 96 ++++++++++++++++++++++++++++++++
>  xen/drivers/passthrough/pci.c    | 54 +++++-------------
>  xen/drivers/pci/Kconfig          |  4 ++
>  xen/include/xen/msi.h            | 56 +++++++++++++++++++
>  xen/xsm/flask/hooks.c            |  8 +--
>  7 files changed, 175 insertions(+), 45 deletions(-)
>  create mode 100644 xen/drivers/passthrough/msi.c
>  create mode 100644 xen/include/xen/msi.h
> 

...

> diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
> index f1a1217c98..fdcfeb984c 100644
> --- a/xen/xsm/flask/hooks.c
> +++ b/xen/xsm/flask/hooks.c
> @@ -21,7 +21,7 @@
>  #include <xen/guest_access.h>
>  #include <xen/xenoprof.h>
>  #include <xen/iommu.h>
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>  #include <asm/msi.h>
>  #endif
>  #include <public/xen.h>
> @@ -114,7 +114,7 @@ static int get_irq_sid(int irq, u32 *sid, struct avc_audit_data *ad)
>          }
>          return security_irq_sid(irq, sid);
>      }
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>      {
>          struct irq_desc *desc = irq_to_desc(irq);
>          if ( desc->msi_desc && desc->msi_desc->dev ) {
> @@ -874,7 +874,7 @@ static int flask_map_domain_pirq (struct domain *d)
>  static int flask_map_domain_msi (struct domain *d, int irq, const void *data,
>                                   u32 *sid, struct avc_audit_data *ad)
>  {
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>      const struct msi_info *msi = data;
>      u32 machine_bdf = (msi->seg << 16) | (msi->bus << 8) | msi->devfn;
>  
> @@ -940,7 +940,7 @@ static int flask_unmap_domain_pirq (struct domain *d)
>  static int flask_unmap_domain_msi (struct domain *d, int irq, const void *data,
>                                     u32 *sid, struct avc_audit_data *ad)
>  {
> -#ifdef CONFIG_HAS_PCI
> +#ifdef CONFIG_HAS_PCI_MSI
>      const struct pci_dev *pdev = data;
>      u32 machine_bdf = (pdev->seg << 16) | (pdev->bus << 8) | pdev->devfn;
>  
> 

Straightforward, so I see no issue with the flask related changes.

Reviewed-by: Daniel P. Smith <dpsmith@apertussolutions.com>


v/r
dps


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-24 15:53   ` Jan Beulich
@ 2021-08-31 12:31     ` Rahul Singh
  2021-08-31 13:00       ` Jan Beulich
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-08-31 12:31 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Bertrand Marquis, Andrew Cooper, Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Julien Grall,
	Stefano Stabellini, Paul Durrant, Daniel De Graaf,
	Daniel P. Smith, xen-devel

Hi Jan,

> On 24 Aug 2021, at 4:53 pm, Jan Beulich <jbeulich@suse.com> wrote:
> 
> On 19.08.2021 14:02, Rahul Singh wrote:
>> --- /dev/null
>> +++ b/xen/drivers/passthrough/msi.c
>> @@ -0,0 +1,96 @@
>> +/*
>> + * Copyright (C) 2008,  Netronome Systems, Inc.
> 
> While generally copying copyright statements when splitting source
> files is probably wanted (or even necessary) I doubt this is
> suitable here: None of the MSI code that you move was contributed
> by them afaict.

Let me remove the "Copyright Copyright (C) 2008,  Netronome Systems, Inc.” . 
Can you please help me what copyright I will add for the next patch ?
> 
>> + * This program is free software; you can redistribute it and/or modify it
>> + * under the terms and conditions of the GNU General Public License,
>> + * version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope it will be useful, but WITHOUT
>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>> + * more details.
>> + *
>> + * You should have received a copy of the GNU General Public License along with
>> + * this program; If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/msi.h>
> 
> You surely mean xen/msi.h here: Headers like this one should always
> be included by the producer, no matter that it builds fine without.
> Else you risk declarations and definitions to go out of sync.
Ok . Let me include here “xen/msi.h” and move other required includes to “xen/msi.h"
> 
>> +#include <asm/hvm/io.h>
>> +
>> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)
>> +{
>> +    int rc;
>> +
>> +    if ( pdev->msix )
>> +    {
>> +        rc = pci_reset_msix_state(pdev);
>> +        if ( rc )
>> +            return rc;
>> +        msixtbl_init(d);
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +int pdev_msi_init(struct pci_dev *pdev)
>> +{
>> +    unsigned int pos;
>> +
>> +    INIT_LIST_HEAD(&pdev->msi_list);
>> +
>> +    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
>> +                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSI);
>> +    if ( pos )
>> +    {
>> +        uint16_t ctrl = pci_conf_read16(pdev->sbdf, msi_control_reg(pos));
>> +
>> +        pdev->msi_maxvec = multi_msi_capable(ctrl);
>> +    }
>> +
>> +    pos = pci_find_cap_offset(pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
>> +                              PCI_FUNC(pdev->devfn), PCI_CAP_ID_MSIX);
>> +    if ( pos )
>> +    {
>> +        struct arch_msix *msix = xzalloc(struct arch_msix);
>> +        uint16_t ctrl;
>> +
>> +        if ( !msix )
>> +            return -ENOMEM;
>> +
>> +        spin_lock_init(&msix->table_lock);
>> +
>> +        ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos));
>> +        msix->nr_entries = msix_table_size(ctrl);
>> +
>> +        pdev->msix = msix;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +void pdev_msi_deinit(struct pci_dev *pdev)
>> +{
>> +    XFREE(pdev->msix);
>> +}
>> +
>> +void pdev_dump_msi(const struct pci_dev *pdev)
>> +{
>> +    const struct msi_desc *msi;
>> +
>> +    printk("- MSIs < ");
>> +    list_for_each_entry ( msi, &pdev->msi_list, list )
>> +        printk("%d ", msi->irq);
>> +    printk(">");
> 
> While not an exact equivalent of the original code then, could I
> talk you into adding an early list_empty() check, suppressing any
> output from this function if that one returned "true”?
Ok.
> 
>> @@ -1271,18 +1249,16 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev)
>> static int _dump_pci_devices(struct pci_seg *pseg, void *arg)
>> {
>>     struct pci_dev *pdev;
>> -    struct msi_desc *msi;
>> 
>>     printk("==== segment %04x ====\n", pseg->nr);
>> 
>>     list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list )
>>     {
>> -        printk("%pp - %pd - node %-3d - MSIs < ",
>> +        printk("%pp - %pd - node %-3d ",
> 
> Together with the request above the trailin blank here also wants to
> become a leading blank in pdev_dump_msi()
Ok.
>> --- /dev/null
>> +++ b/xen/include/xen/msi.h
>> @@ -0,0 +1,56 @@
>> +/*
>> + * Copyright (C) 2008,  Netronome Systems, Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or modify it
>> + * under the terms and conditions of the GNU General Public License,
>> + * version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope it will be useful, but WITHOUT
>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>> + * more details.
>> + *
>> + * You should have received a copy of the GNU General Public License along with
>> + * this program; If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#ifndef __XEN_MSI_H_
>> +#define __XEN_MSI_H_
>> +
>> +#ifdef CONFIG_HAS_PCI_MSI
>> +
>> +#include <asm/msi.h>
>> +
>> +int pdev_msix_assign(struct domain *d, struct pci_dev *pdev);
>> +int pdev_msi_init(struct pci_dev *pdev);
>> +void pdev_msi_deinit(struct pci_dev *pdev);
>> +void pdev_dump_msi(const struct pci_dev *pdev);
>> +
>> +#else /* !CONFIG_HAS_PCI_MSI */
>> +static inline int pdev_msix_assign(struct domain *d, struct pci_dev *pdev)
> 
> Please be consistent with blank lines you add; here you also want one
> after the #else.

Ok.

> 
>> +{
>> +    return 0;
>> +}
>> +
>> +static inline int pdev_msi_init(struct pci_dev *pdev)
>> +{
>> +    return 0;
>> +}
>> +
>> +static inline void pdev_msi_deinit(struct pci_dev *pdev) {}
>> +static inline void pci_cleanup_msi(struct pci_dev *pdev) {}
>> +static inline void pdev_dump_msi(const struct pci_dev *pdev) {}
> 
> Especially for (but perhaps not limited to) this !HAS_PCI_MSI case
> (where you don't include asm/msi.h and its possible dependents)
> please forward-declare struct-s you use in prototypes or inline
> stubs (outside the #ifdef, that is). This will allow including
> this header without having to care about prereq headers.

Ok. Let me do modification in next version.

Regards,
Rahul
> 
> If you agree with and make all the suggested or requested changes,
> feel free to add
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
> 
> Jan
> 


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-31 12:31     ` Rahul Singh
@ 2021-08-31 13:00       ` Jan Beulich
  0 siblings, 0 replies; 98+ messages in thread
From: Jan Beulich @ 2021-08-31 13:00 UTC (permalink / raw)
  To: Rahul Singh
  Cc: Bertrand Marquis, Andrew Cooper, Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Julien Grall,
	Stefano Stabellini, Paul Durrant, Daniel De Graaf,
	Daniel P. Smith, xen-devel

On 31.08.2021 14:31, Rahul Singh wrote:
>> On 24 Aug 2021, at 4:53 pm, Jan Beulich <jbeulich@suse.com> wrote:
>> On 19.08.2021 14:02, Rahul Singh wrote:
>>> --- /dev/null
>>> +++ b/xen/drivers/passthrough/msi.c
>>> @@ -0,0 +1,96 @@
>>> +/*
>>> + * Copyright (C) 2008,  Netronome Systems, Inc.
>>
>> While generally copying copyright statements when splitting source
>> files is probably wanted (or even necessary) I doubt this is
>> suitable here: None of the MSI code that you move was contributed
>> by them afaict.
> 
> Let me remove the "Copyright Copyright (C) 2008,  Netronome Systems, Inc.” . 
> Can you please help me what copyright I will add for the next patch ?

None? If you look around, you will find that far from all source files
have such a line (or multiple of them).

>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms and conditions of the GNU General Public License,
>>> + * version 2, as published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope it will be useful, but WITHOUT
>>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>>> + * more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License along with
>>> + * this program; If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <xen/init.h>
>>> +#include <xen/pci.h>
>>> +#include <asm/msi.h>
>>
>> You surely mean xen/msi.h here: Headers like this one should always
>> be included by the producer, no matter that it builds fine without.
>> Else you risk declarations and definitions to go out of sync.
> Ok . Let me include here “xen/msi.h” and move other required includes to “xen/msi.h"

Why move stuff? xen/msi.h is fins to include asm/msi.h. It's just that
including asm/msi.h here is not enough.

Jan



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver.
  2021-08-19 12:02 ` [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver Rahul Singh
@ 2021-09-07  8:20   ` Julien Grall
  2021-09-10 10:47     ` Rahul Singh
  2021-09-09  1:16   ` Stefano Stabellini
  1 sibling, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-09-07  8:20 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> pci_init(..) will be called during xen startup to initialize and probe
> the PCI host-bridge driver.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/arm/pci/pci.c       | 54 ++++++++++++++++++++++++++++++++++++
>   xen/include/asm-arm/device.h |  1 +
>   2 files changed, 55 insertions(+)
> 
> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
> index dc55d23778..d1c9cf997d 100644
> --- a/xen/arch/arm/pci/pci.c
> +++ b/xen/arch/arm/pci/pci.c
> @@ -14,13 +14,67 @@
>    * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>    */
>   
> +#include <xen/acpi.h>
> +#include <xen/device_tree.h>
> +#include <xen/errno.h>
> +#include <xen/init.h>
>   #include <xen/pci.h>
> +#include <xen/param.h>

This include doesn't look to be necessary (yet?).

>   
>   int arch_pci_clean_pirqs(struct domain *d)
>   {
>       return 0;
>   }
>   
> +static int __init dt_pci_init(void)
> +{
> +    struct dt_device_node *np;
> +    int rc;
> +
> +    dt_for_each_device_node(dt_host, np)
> +    {
> +        rc = device_init(np, DEVICE_PCI, NULL);
> +        if( !rc )
> +            continue;
> +        /*
> +         * Ignore the following error codes:
> +         *   - EBADF: Indicate the current is not an pci
> +         *   - ENODEV: The pci device is not present or cannot be used by
> +         *     Xen.
> +         */
> +        else if ( rc != -EBADF && rc != -ENODEV )
> +        {
> +            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
> +            return rc;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
> +#ifdef CONFIG_ACPI
> +static void __init acpi_pci_init(void)
> +{
> +    printk(XENLOG_ERR "ACPI pci init not supported \n");
> +    return;
> +}
> +#else
> +static inline void __init acpi_pci_init(void) { }
> +#endif
> +
> +static int __init pci_init(void)
> +{
> +    if ( acpi_disabled )
> +        dt_pci_init();
> +    else
> +        acpi_pci_init();
> +
> +    pci_segments_init();

Shouldn't this happen before the PCI initialization?

> +
> +    return 0;
> +}
> +__initcall(pci_init);
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
> index ee7cff2d44..5ecd5e7bd1 100644
> --- a/xen/include/asm-arm/device.h
> +++ b/xen/include/asm-arm/device.h
> @@ -34,6 +34,7 @@ enum device_class
>       DEVICE_SERIAL,
>       DEVICE_IOMMU,
>       DEVICE_GIC,
> +    DEVICE_PCI,
>       /* Use for error */
>       DEVICE_UNKNOWN,
>   };
> 

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-08-19 12:02 ` [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
@ 2021-09-07  9:05   ` Julien Grall
  2021-09-10 11:22     ` Rahul Singh
  2021-09-09 22:54   ` Stefano Stabellini
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-09-07  9:05 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> XEN during boot will read the PCI device tree node “reg” property
> and will map the PCI config space to the XEN memory.
> 
> As of now "pci-host-ecam-generic" compatible board is supported.

I think the word "only" is missing.

> 
> "linux,pci-domain" device tree property assigns a fixed PCI domain
> number to a host bridge, otherwise an unstable (across boots) unique
> number will be assigned by Linux.This property has to be in sync with

Typo: missing space after the '.'

> XEN to access the PCI devices.

I would expand a little bit the last sentence to explain why the need to 
be sync-ed.

> > XEN will read the “linux,pci-domain” property from the device tree node
> and configure the host bridge segment number accordingly. If this
> property is not available XEN will allocate the unique segment number
> to the host bridge.
> 
> dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly
> imported from the Linux source tree.

What was the Linux commit used? I also read "directly imported" as a 
verbartim copy but AFAICT the implementation has been slightly reworked.

> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/arm/pci/Makefile           |   2 +
>   xen/arch/arm/pci/pci-host-common.c  | 261 ++++++++++++++++++++++++++++
>   xen/arch/arm/pci/pci-host-generic.c |  55 ++++++
>   xen/include/asm-arm/pci.h           |  28 +++
>   4 files changed, 346 insertions(+)
>   create mode 100644 xen/arch/arm/pci/pci-host-common.c
>   create mode 100644 xen/arch/arm/pci/pci-host-generic.c
> 
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> index a9ee0b9b44..f3d97f859e 100644
> --- a/xen/arch/arm/pci/Makefile
> +++ b/xen/arch/arm/pci/Makefile
> @@ -1,2 +1,4 @@
>   obj-y += pci.o
>   obj-y += pci-access.o
> +obj-y += pci-host-generic.o
> +obj-y += pci-host-common.o
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> new file mode 100644
> index 0000000000..9dd9b02271
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -0,0 +1,261 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>

AFAICT, <xen/pci.h> already includes <asm/pci.h>. So this looks 
unneccessary.

> +#include <xen/rwlock.h>
> +#include <xen/sched.h>
> +#include <xen/vmap.h>
> +
> +/*
> + * List for all the pci host bridges.
> + */
> +
> +static LIST_HEAD(pci_host_bridges);
> +
> +static atomic_t domain_nr = ATOMIC_INIT(-1);
> +
> +bool dt_pci_parse_bus_range(struct dt_device_node *dev,
> +                            struct pci_config_window *cfg)

Aside, "pci_config_window", the function is not Arm specific. Would it 
be possible to consider to introduce "struct resource" in Xen so this 
function can be moved in common/device_tree.c?

> +{
> +    const __be32 *cells;
> +    uint32_t len;
> +
> +    cells = dt_get_property(dev, "bus-range", &len);
> +    /* bus-range should at least be 2 cells */
> +    if ( !cells || (len < (sizeof(*cells) * 2)) )
> +        return false;

How about introducing dt_property_read_u32_array()?

> +
> +    cfg->busn_start = dt_next_cell(1, &cells);
> +    cfg->busn_end = dt_next_cell(1, &cells);
> +
> +    return true;
> +}
> +
> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
> +{
> +    return ioremap_nocache(start, len);
> +}
> +
> +static void pci_ecam_free(struct pci_config_window *cfg)
> +{
> +    if ( cfg->win )
> +        iounmap(cfg->win);
> +
> +    xfree(cfg);
> +}
> +
> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> +                                              int ecam_reg_idx)
> +{
> +    int err;
> +    struct pci_config_window *cfg;
> +    paddr_t addr, size;
> +
> +    cfg = xzalloc(struct pci_config_window);
> +    if ( !cfg )
> +        return NULL;
> +
> +    err = dt_pci_parse_bus_range(dev, cfg);
> +    if ( !err ) {
> +        cfg->busn_start = 0;
> +        cfg->busn_end = 0xff;
> +        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",

Typo: Missing space after ':'.

> +               dt_node_full_name(dev));
> +    } else {
> +        if ( cfg->busn_end > cfg->busn_start + 0xff )
> +            cfg->busn_end = cfg->busn_start + 0xff;
> +    }
> +
> +    /* Parse our PCI ecam register address*/
> +    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
> +    if ( err )
> +        goto err_exit;
> +
> +    cfg->phys_addr = addr;
> +    cfg->size = size;
> +
> +    /*
> +     * On 64-bit systems, we do a single ioremap for the whole config space
> +     * since we have enough virtual address range available.  On 32-bit, we

In Xen on Arm64, the VMAP is actually only 1GB. So it is not that big 
and this will compete with other mapping like ITS, global domain mapping...

So I think the vMAP area will need to increase to cater to increase usage.

> +     * ioremap the config space for each bus individually.
> +     *
> +     * As of now only 64-bit is supported 32-bit is not supported.
> +     */
> +    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
> +    if ( !cfg->win )
> +        goto err_exit_remap;
> +
> +    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,

The physical address is a paddr_t. So this needs to use PRIpaddr. Also, 
please use preprent hexadecimal with 0x. This makes a lot easier to 
differentiate hexa vs decimal in the log.

> +            cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
> +
> +    return cfg;
> +
> +err_exit_remap:
> +    printk(XENLOG_ERR "ECAM ioremap failed\n");
> +err_exit:
> +    pci_ecam_free(cfg);

Coding style: Please add a new line before return.

> +    return NULL;
> +}
> +
> +struct pci_host_bridge *pci_alloc_host_bridge(void)
> +{
> +    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
> +
> +    if ( !bridge )
> +        return NULL;
> +
> +    INIT_LIST_HEAD(&bridge->node);
> +    bridge->bus_start = ~0;
> +    bridge->bus_end = ~0;

Coding style: Please add a new line before return.


> +    return bridge;
> +}
> +
> +void pci_add_host_bridge(struct pci_host_bridge *bridge)
> +{
> +    list_add_tail(&bridge->node, &pci_host_bridges);
> +}
> +
> +static int pci_get_new_domain_nr(void)
> +{
> +    return atomic_inc_return(&domain_nr);
> +}
> +
> +/*
> + * This function will try to obtain the host bridge domain number by
> + * finding a property called "linux,pci-domain" of the given device node.
> + *
> + * @node: device tree node with the domain information
> + *
> + * Returns the associated domain number from DT in the range [0-0xffff], or
> + * a negative value if the required property is not found.
> + */
> +static int dt_get_pci_domain_nr(struct dt_device_node *node)

Nothing looks Arm specific for this function. Can you move it in 
device_tree.c?

> +{
> +    u32 domain;
> +    int error;
> +
> +    error = dt_property_read_u32(node, "linux,pci-domain", &domain);
> +    if ( !error )
> +        return -EINVAL;
> +
> +    return (u16)domain;
> +}
> +
> +static int pci_bus_find_domain_nr(struct dt_device_node *dev)
> +{
> +    static int use_dt_domains = -1;
> +    int domain;
> +
> +    domain = dt_get_pci_domain_nr(dev);
> +
> +    /*
> +     * Check DT domain and use_dt_domains values.
> +     *
> +     * If DT domain property is valid (domain >= 0) and
> +     * use_dt_domains != 0, the DT assignment is valid since this means
> +     * we have not previously allocated a domain number by using
> +     * pci_get_new_domain_nr(); we should also update use_dt_domains to
> +     * 1, to indicate that we have just assigned a domain number from
> +     * DT.
> +     *
> +     * If DT domain property value is not valid (ie domain < 0), and we
> +     * have not previously assigned a domain number from DT
> +     * (use_dt_domains != 1) we should assign a domain number by
> +     * using the:
> +     *
> +     * pci_get_new_domain_nr()
> +     *
> +     * API and update the use_dt_domains value to keep track of method we
> +     * are using to assign domain numbers (use_dt_domains = 0).
> +     *
> +     * All other combinations imply we have a platform that is trying
> +     * to mix domain numbers obtained from DT and pci_get_new_domain_nr(),
> +     * which is a recipe for domain mishandling and it is prevented by
> +     * invalidating the domain value (domain = -1) and printing a
> +     * corresponding error.
> +     */
> +    if ( domain >= 0 && use_dt_domains )
> +    {
> +        use_dt_domains = 1;
> +    }
> +    else if ( domain < 0 && use_dt_domains != 1 )
> +    {
> +        use_dt_domains = 0;
> +        domain = pci_get_new_domain_nr();
> +    }
> +    else
> +    {
> +        printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n");
> +        BUG();

I don't think pci_bus_find_domain_nr() should be the function that 
crashes Xen. Instead, this should be propagated to the highest possible 
callers and let it decide what to do.

> +    }
> +
> +    return domain;
> +}
> +
> +int pci_host_common_probe(struct dt_device_node *dev,
> +                          int ecam_reg_idx)
> +{
> +    struct pci_host_bridge *bridge;
> +    struct pci_config_window *cfg;
> +    int err;
> +
> +    bridge = pci_alloc_host_bridge();
> +    if ( !bridge )
> +        return -ENOMEM;
> +
> +    /* Parse and map our Configuration Space windows */
> +    cfg = gen_pci_init(dev, ecam_reg_idx);
> +    if ( !cfg )
> +    {
> +        err = -ENOMEM;
> +        goto err_exit;
> +    }
> +
> +    bridge->dt_node = dev;
> +    bridge->sysdata = cfg;
> +    bridge->bus_start = cfg->busn_start;
> +    bridge->bus_end = cfg->busn_end;
> +
> +    bridge->segment = pci_bus_find_domain_nr(dev);
> +
> +    pci_add_host_bridge(bridge);
> +
> +    return 0;
> +
> +err_exit:
> +    xfree(bridge);
> +    return err;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
> new file mode 100644
> index 0000000000..13d0f7f999
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-generic.c
> @@ -0,0 +1,55 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/device.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +
> +static const struct dt_device_match gen_pci_dt_match[] = {
> +    { .compatible = "pci-host-ecam-generic" },
> +    { },
> +};
> +
> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
> +{
> +    const struct dt_device_match *of_id;
> +
> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);

This seems to be a bit pointless to me as you already know the 
compatible (there is only one possible...).

> +
> +    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
> +           dt_node_full_name(dev), of_id->compatible);
> +
> +    return pci_host_common_probe(dev, 0);
> +}
> +
> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
> +.dt_match = gen_pci_dt_match,
> +.init = gen_pci_dt_init,
> +DT_DEVICE_END
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 61e43da088..58a51e724e 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -26,6 +26,34 @@ struct arch_pci_dev {
>       struct device dev;
>   };
>   
> +/*
> + * struct to hold the mappings of a config space window. This
> + * is expected to be used as sysdata for PCI controllers that
> + * use ECAM.
> + */
> +struct pci_config_window {
> +    paddr_t         phys_addr;
> +    paddr_t         size;
> +    uint8_t         busn_start;
> +    uint8_t         busn_end;
> +    void __iomem    *win;
> +};
> +
> +/*
> + * struct to hold pci host bridge information
> + * for a PCI controller.
> + */
> +struct pci_host_bridge {
> +    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
> +    struct list_head node;           /* Node in list of host bridges */
> +    uint16_t segment;                /* Segment number */
> +    u8 bus_start;                    /* Bus start of this bridge. */
> +    u8 bus_end;                      /* Bus end of this bridge. */

Please use uint8_t rather than u8.

> +    void *sysdata;                   /* Pointer to the config space window*/
> +};
> +
> +int pci_host_common_probe(struct dt_device_node *dev,
> +                          int ecam_reg_idx);
>   #else   /*!CONFIG_HAS_PCI*/
>   
>   struct arch_pci_dev { };
> 

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN
  2021-08-19 14:16     ` Rahul Singh
@ 2021-09-07 10:01       ` Julien Grall
  0 siblings, 0 replies; 98+ messages in thread
From: Julien Grall @ 2021-09-07 10:01 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Jan Beulich, Andrew Cooper,
	Roger Pau Monné,
	Wei Liu, George Dunlap, Ian Jackson, Stefano Stabellini,
	Paul Durrant, Daniel De Graaf, Daniel P. Smith



On 19/08/2021 15:16, Rahul Singh wrote:
> Hi Julien,

Hi Rahul,

Sorry for the late reply.

>> On 19 Aug 2021, at 1:18 pm, Julien Grall <julien@xen.org> wrote:
>>
>> Hi Rahul,
>>
>> On 19/08/2021 13:02, Rahul Singh wrote:
>>> MSI code that implements MSI functionality to support MSI within XEN is
>>> not usable on ARM. Move the code under CONFIG_HAS_PCI_MSI flag to gate
>>
>> Can you clarify what you mean by not usable? Is it because we lack of support or we have no plan to use it?
> 
> We have no plan to use it. Code moved to CONFIG_HAS_PCI_MSI will only be required for ARM if we
I read "not usable", as the code doesn't yet work on Arm. This is quite 
possible, but from you wrote the main reason is that the code is not 
necessary yet (see below) on Arm.

> decide to support PCI device access (PCI MSI interrupt support) within XEN. As of now, we are planning
> to add support for PCI device access for DOM0/DOMU guests not for XEN.

This is probably going to happen sooner rather than later as, AFAICT, 
there are SMMUs out which signals using MSIs.

But, AFAICT, this code would also be used if we need to manage the MSIs 
in Xen on behalf of the guest (such as if the platform is using GICv2m).

That said, I would be fine with gating the MSI code behind a new config. 
However, I think the commit message wants to be clarified into why we 
don't need the option. Maybe something like:

"On Arm, the inital plan is to only support GICv3 ITS which doesn't 
require us to manage the MSIs because the HW will protect against 
spoofing. Introduce a new option XXXX to protect the MSI code.
"

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI
  2021-08-20 11:54       ` Jan Beulich
@ 2021-09-09  1:11         ` Stefano Stabellini
  2021-09-10 10:22           ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09  1:11 UTC (permalink / raw)
  To: Jan Beulich; +Cc: Rahul Singh, Bertrand Marquis, Paul Durrant, xen-devel

[-- Attachment #1: Type: text/plain, Size: 2022 bytes --]

On Fri, 20 Aug 2021, Jan Beulich wrote:
> On 20.08.2021 13:41, Rahul Singh wrote:
> > Hi Jan
> > 
> >> On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote:
> >>
> >> On 19.08.2021 14:02, Rahul Singh wrote:
> >>> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
> >>> architecture. Move the code under CONFIG_X86 flag to gate the code for
> >>> ARM.
> >>
> >> Please give at least one example of what it is that fails to compile.
> >> What an appropriate action is depends on the nature of the error(s),
> >> and from looking at the enclosed code I cannot easily see what it
> >> might be that breaks for Arm.
> > 
> > I am observing below error when enabled ACPI && HAS_PCI for ARM.
> > 
> > prelink.o: In function `pcie_aer_get_firmware_first’:
> > /xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse'
> > aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined
> > 
> > I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code.
> > obj-$(CONFIG_X86) += apei/
> > 
> > I am not sure whether we need this code for ARM architecture 
> > that is why I gate the code for ARM via CONFIG_X86
> 
> So you Arm folks will probably want to settle on that aspect first. What
> is wanted to keep things building depends on that.

Reading the APEI description, it looks like there might be some use for
it on ARM but it would work a bit differently from x86 as there are no
NMIs on ARM. So enabling APEI on ARM is not just a matter of enabling
the build of apei/, it is not going to be straightforward.

For the scope of this series (which is actually about PCI), I would
leave it alone, and keep apei/ x86 only, which means #ifdefing
pcie_aer_get_firmware_first.

I would just add an in-code comment saying "APEI not supported on ARM
yet". Another option would be to introduce a symbol like HAS_ACPI_APEI
but it is a bit overkill for this.

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver.
  2021-08-19 12:02 ` [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver Rahul Singh
  2021-09-07  8:20   ` Julien Grall
@ 2021-09-09  1:16   ` Stefano Stabellini
  2021-09-10 10:32     ` Rahul Singh
  1 sibling, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09  1:16 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

On Thu, 19 Aug 2021, Rahul Singh wrote:
> pci_init(..) will be called during xen startup to initialize and probe
> the PCI host-bridge driver.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/pci/pci.c       | 54 ++++++++++++++++++++++++++++++++++++
>  xen/include/asm-arm/device.h |  1 +
>  2 files changed, 55 insertions(+)
> 
> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
> index dc55d23778..d1c9cf997d 100644
> --- a/xen/arch/arm/pci/pci.c
> +++ b/xen/arch/arm/pci/pci.c
> @@ -14,13 +14,67 @@
>   * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>   */
>  
> +#include <xen/acpi.h>
> +#include <xen/device_tree.h>
> +#include <xen/errno.h>
> +#include <xen/init.h>
>  #include <xen/pci.h>
> +#include <xen/param.h>
>  
>  int arch_pci_clean_pirqs(struct domain *d)
>  {
>      return 0;
>  }
>  
> +static int __init dt_pci_init(void)
> +{
> +    struct dt_device_node *np;
> +    int rc;
> +
> +    dt_for_each_device_node(dt_host, np)
> +    {
> +        rc = device_init(np, DEVICE_PCI, NULL);
> +        if( !rc )
> +            continue;
> +        /*
> +         * Ignore the following error codes:
> +         *   - EBADF: Indicate the current is not an pci
                                                     ^ a

> +         *   - ENODEV: The pci device is not present or cannot be used by
> +         *     Xen.
> +         */
> +        else if ( rc != -EBADF && rc != -ENODEV )
> +        {
> +            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
> +            return rc;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
> +#ifdef CONFIG_ACPI
> +static void __init acpi_pci_init(void)

If the DT init function returns int, then it would make sense for the
ACPI init function to return int as well?


> +{
> +    printk(XENLOG_ERR "ACPI pci init not supported \n");
> +    return;
> +}
> +#else
> +static inline void __init acpi_pci_init(void) { }
> +#endif
> +
> +static int __init pci_init(void)
> +{
> +    if ( acpi_disabled )
> +        dt_pci_init();
> +    else
> +        acpi_pci_init();
> +
> +    pci_segments_init();
> +
> +    return 0;
> +}
> +__initcall(pci_init);
> +
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
> index ee7cff2d44..5ecd5e7bd1 100644
> --- a/xen/include/asm-arm/device.h
> +++ b/xen/include/asm-arm/device.h
> @@ -34,6 +34,7 @@ enum device_class
>      DEVICE_SERIAL,
>      DEVICE_IOMMU,
>      DEVICE_GIC,
> +    DEVICE_PCI,
>      /* Use for error */
>      DEVICE_UNKNOWN,
>  };
> -- 
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-08-19 12:02 ` [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations Rahul Singh
@ 2021-09-09 11:32   ` Julien Grall
  2021-09-14  8:13     ` Rahul Singh
  2021-09-09 23:21   ` Stefano Stabellini
  1 sibling, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-09-09 11:32 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> Add support for PCI ecam operations to access the PCI
> configuration space.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/arm/pci/Makefile           |  1 +
>   xen/arch/arm/pci/ecam.c             | 63 +++++++++++++++++++++++++++++
>   xen/arch/arm/pci/pci-access.c       | 53 ++++++++++++++++++++++++
>   xen/arch/arm/pci/pci-host-common.c  | 13 +++++-
>   xen/arch/arm/pci/pci-host-generic.c |  8 +++-
>   xen/include/asm-arm/pci.h           | 32 +++++++++++++++
>   6 files changed, 167 insertions(+), 3 deletions(-)
>   create mode 100644 xen/arch/arm/pci/ecam.c
> 
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> index f3d97f859e..6f32fbbe67 100644
> --- a/xen/arch/arm/pci/Makefile
> +++ b/xen/arch/arm/pci/Makefile
> @@ -2,3 +2,4 @@ obj-y += pci.o
>   obj-y += pci-access.o
>   obj-y += pci-host-generic.o
>   obj-y += pci-host-common.o
> +obj-y += ecam.o
> diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c
> new file mode 100644
> index 0000000000..91c691b41f
> --- /dev/null
> +++ b/xen/arch/arm/pci/ecam.c
> @@ -0,0 +1,63 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/pci.h>
> +#include <xen/sched.h>
> +
> +/*
> + * Function to implement the pci_ops ->map_bus method.
> + */
> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
> +                                      uint32_t sbdf, uint32_t where)
> +{
> +    const struct pci_config_window *cfg = bridge->sysdata;
> +    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
> +    void __iomem *base;
> +
> +    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;

AFAICT, pci_sbdf is an union between a 32-bit and a structure. So please 
don't use the cast and use the 32-bit field to assign the value.

Also, there is an extra space before ';'.

> +    unsigned int busn = sbdf_t.bus;
> +
> +    if ( busn < cfg->busn_start || busn > cfg->busn_end )
> +        return NULL;
> +
> +    busn -= cfg->busn_start;
> +    base = cfg->win + (busn << cfg->ops->bus_shift);
> +
> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;

How about using PCI_DEVFN2(sbdf)? This would allow you to drop the use 
of sbdf_t completely (sbdf_t.bus could be replaced with PCI_BUS(sdbf)).

> +}
> +
> +/* ECAM ops */
> +const struct pci_ecam_ops pci_generic_ecam_ops = {
> +    .bus_shift  = 20,
> +    .pci_ops    = {
> +        .map_bus                = pci_ecam_map_bus,
> +        .read                   = pci_generic_config_read,
> +        .write                  = pci_generic_config_write,
> +    }
> +};
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
> index b938047c03..f39f6a3a38 100644
> --- a/xen/arch/arm/pci/pci-access.c
> +++ b/xen/arch/arm/pci/pci-access.c
> @@ -15,6 +15,59 @@
>    */
>   
>   #include <xen/pci.h>
> +#include <asm/io.h>
> +
> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t *value)
> +{
> +    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);

Please add a newline here.

> +    if (!addr) {

You seem to use a mix of Xen and Linux coding style. If the file 
contains mostly Xen code, then we should use the former.

> +        *value = ~0;
> +        return -ENODEV;
> +    }
> +
> +    switch (len)
> +    {
> +    case 1:
> +        *value = readb(addr);
> +        break;
> +    case 2:
> +        *value = readw(addr);
> +        break;
> +    case 4:
> +        *value = readl(addr);
> +        break;
> +    default:
> +        BUG();
> +    }
> +
> +    return 0;
> +}
> +
> +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t value)
> +{
> +    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
> +    if (!addr)
> +        return -ENODEV;
> +
> +    switch (len)
> +    {
> +    case 1:
> +        writeb(value, addr);
> +        break;
> +    case 2:
> +        writew(value, addr);
> +        break;
> +    case 4:
> +        writel(value, addr);
> +        break;
> +    default:
> +        BUG();
> +    }
> +
> +    return 0;
> +}
>   
>   static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>                                   unsigned int len)
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> index 9dd9b02271..c582527e92 100644
> --- a/xen/arch/arm/pci/pci-host-common.c
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg)
>   }
>   
>   static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> +                                              const struct pci_ecam_ops *ops,
>                                                 int ecam_reg_idx)
>   {
>       int err;
> @@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>   
>       cfg->phys_addr = addr;
>       cfg->size = size;
> +    cfg->ops = ops;
>   
>       /*
>        * On 64-bit systems, we do a single ioremap for the whole config space
> @@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>       printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>               cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
>   
> +    if ( ops->init )
> +    {
> +        err = ops->init(cfg);
> +        if (err)
> +            goto err_exit;
> +    }
> +
>       return cfg;
>   
>   err_exit_remap:
> @@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev)
>   }
>   
>   int pci_host_common_probe(struct dt_device_node *dev,
> +                          const struct pci_ecam_ops *ops,
>                             int ecam_reg_idx)
>   {
>       struct pci_host_bridge *bridge;
> @@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
>           return -ENOMEM;
>   
>       /* Parse and map our Configuration Space windows */
> -    cfg = gen_pci_init(dev, ecam_reg_idx);
> +    cfg = gen_pci_init(dev, ops, ecam_reg_idx);
>       if ( !cfg )
>       {
>           err = -ENOMEM;
> @@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
>   
>       bridge->dt_node = dev;
>       bridge->sysdata = cfg;
> +    bridge->ops = &ops->pci_ops;
>       bridge->bus_start = cfg->busn_start;
>       bridge->bus_end = cfg->busn_end;
>   
> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
> index 13d0f7f999..2d652e8910 100644
> --- a/xen/arch/arm/pci/pci-host-generic.c
> +++ b/xen/arch/arm/pci/pci-host-generic.c
> @@ -23,20 +23,24 @@
>   #include <asm/pci.h>
>   
>   static const struct dt_device_match gen_pci_dt_match[] = {
> -    { .compatible = "pci-host-ecam-generic" },
> +    { .compatible = "pci-host-ecam-generic",
> +      .data =       &pci_generic_ecam_ops },
> +
>       { },
>   };
>   
>   static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>   {
>       const struct dt_device_match *of_id;
> +    const struct pci_ecam_ops *ops;
>   
>       of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> +    ops = (struct pci_ecam_ops *) of_id->data;
>   
>       printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>              dt_node_full_name(dev), of_id->compatible);
>   
> -    return pci_host_common_probe(dev, 0);
> +    return pci_host_common_probe(dev, ops, 0);
>   }
>   
>   DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 58a51e724e..22866244d2 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -37,6 +37,7 @@ struct pci_config_window {
>       uint8_t         busn_start;
>       uint8_t         busn_end;
>       void __iomem    *win;
> +    const struct    pci_ecam_ops *ops;
>   };
>   
>   /*
> @@ -50,10 +51,41 @@ struct pci_host_bridge {
>       u8 bus_start;                    /* Bus start of this bridge. */
>       u8 bus_end;                      /* Bus end of this bridge. */
>       void *sysdata;                   /* Pointer to the config space window*/
> +    const struct pci_ops *ops;
>   };
>   
> +struct pci_ops {
> +    void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                             uint32_t offset);
> +    int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                uint32_t reg, uint32_t len, uint32_t *value);
> +    int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                 uint32_t reg, uint32_t len, uint32_t value);
> +};
> +
> +/*
> + * struct to hold pci ops and bus shift of the config window
> + * for a PCI controller.
> + */
> +struct pci_ecam_ops {
> +    unsigned int            bus_shift;
> +    struct pci_ops          pci_ops;
> +    int (*init)(struct pci_config_window *);
> +};
> +
> +/* Default ECAM ops */
> +extern const struct pci_ecam_ops pci_generic_ecam_ops;
> +
>   int pci_host_common_probe(struct dt_device_node *dev,
> +                          const struct pci_ecam_ops *ops,
>                             int ecam_reg_idx);
> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t *value);
> +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t value);
> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
> +                               uint32_t sbdf, uint32_t where);
> +
>   #else   /*!CONFIG_HAS_PCI*/
>   
>   struct arch_pci_dev { };
> 

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled
  2021-08-19 12:02 ` [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled Rahul Singh
  2021-08-19 12:28   ` Julien Grall
  2021-08-20  7:01   ` Jan Beulich
@ 2021-09-09 13:16   ` Julien Grall
  2 siblings, 0 replies; 98+ messages in thread
From: Julien Grall @ 2021-09-09 13:16 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk

Hi,

On 19/08/2021 13:02, Rahul Singh wrote:
> Compilation error is observed when HAS_PCI is enabled for ARM
> architecture.
> 
> Add definition for arch_iommu_use_permitted() and
> arch_pci_clean_pirqs().Implement dummy functions for pci_conf_read*() to
> fix compilation error.
> 
> pci.c: In function ‘deassign_device’:
> pci.c:849:49: error: implicit declaration of function ‘pci_to_dev’;
> did you mean ‘dt_to_dev’? [-Werror=implicit-function-declaration]
>              pci_to_dev(pdev));
> 
> pci.c:18: undefined reference to `pci_conf_read16’
> pci.c:880: undefined reference to `arch_pci_clean_pirqs’
> pci.c:1392: undefined reference to `arch_iommu_use_permitted'
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/arm/Makefile               |  1 +
>   xen/arch/arm/pci/Makefile           |  2 +
>   xen/arch/arm/pci/pci-access.c       | 61 +++++++++++++++++++++++++++++
>   xen/arch/arm/pci/pci.c              | 32 +++++++++++++++
>   xen/drivers/passthrough/arm/iommu.c |  5 +++
>   xen/include/asm-arm/pci.h           | 33 ++++++++++++++--
>   6 files changed, 131 insertions(+), 3 deletions(-)
>   create mode 100644 xen/arch/arm/pci/Makefile
>   create mode 100644 xen/arch/arm/pci/pci-access.c
>   create mode 100644 xen/arch/arm/pci/pci.c
> 
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 3d3b97b5b4..0e14a5e5c8 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
>   obj-y += platforms/
>   endif
>   obj-$(CONFIG_TEE) += tee/
> +obj-$(CONFIG_HAS_PCI) += pci/

This should go before platforms/ to keep the order alphabetically.


Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-08-19 12:02 ` [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
  2021-08-24 16:09   ` Jan Beulich
@ 2021-09-09 13:50   ` Julien Grall
  2021-09-16 10:46     ` Rahul Singh
  2021-09-10  0:26   ` Stefano Stabellini
  2 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-09-09 13:50 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: bertrand.marquis, Stefano Stabellini, Volodymyr Babchuk,
	Jan Beulich, Paul Durrant, Roger Pau Monné

Hi Rahul,

On 19/08/2021 13:02, Rahul Singh wrote:
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the Xen to emulate the PCI devices config space.
> 
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>   xen/arch/arm/Makefile         |  1 +
>   xen/arch/arm/domain.c         |  4 ++
>   xen/arch/arm/vpci.c           | 96 +++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vpci.h           | 37 ++++++++++++++
>   xen/drivers/passthrough/pci.c |  7 +++
>   xen/drivers/vpci/Makefile     |  3 +-
>   xen/drivers/vpci/header.c     |  2 +
>   xen/include/asm-arm/domain.h  |  5 +-
>   xen/include/asm-arm/pci.h     |  8 +++
>   xen/include/public/arch-arm.h |  4 ++
>   10 files changed, 165 insertions(+), 2 deletions(-)
>   create mode 100644 xen/arch/arm/vpci.c
>   create mode 100644 xen/arch/arm/vpci.h
> 
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 0e14a5e5c8..7cdce684a4 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ obj-y += platforms/
>   endif
>   obj-$(CONFIG_TEE) += tee/
>   obj-$(CONFIG_HAS_PCI) += pci/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>   
>   obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>   obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 19c756ac3d..d99c653626 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -40,6 +40,7 @@
>   #include <asm/vtimer.h>
>   
>   #include "vuart.h"
> +#include "vpci.h"

Please order the includes alphabetically. So this one should go before 
"vuart.h".

>   
>   DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>   
> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>       if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>           goto fail;
>   
> +    if ( (rc = domain_vpci_init(d)) != 0 )
> +        goto fail;
> +
>       return 0;
>   
>   fail:
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..da8b1ca13c
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,96 @@
> +/*
> + * xen/arch/arm/vpci.c
> + * Copyright (c) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>

NIT: Please add a newline between generic and arch specific includes.

> +#include <asm/mmio.h>
> +
> +/* Do some sanity checks. */
> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> +{
> +    /* Check access size. */
> +    if ( len != 1 && len != 2 && len != 4 && len != 8 )
> +        return false;
You will allow all the possible value of len (this is coming from the 
HW). So I feels this is a bit too much to check for every I/O to the vPCI.

If you really want to keep the check then you can simply check that len 
is < 8 because the two callers compute it with (1 << S). So there is no 
way to set it 3, 5, 6 and 7.

> +
> +    /* Check that access is size aligned. */
> +    if ( (reg & (len - 1)) )
> +        return false;
> +
> +    return true;
> +}
> +
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> +                          register_t *r, void *p)
> +{
> +    unsigned int reg;
> +    pci_sbdf_t sbdf;
> +    uint32_t data = 0;
> +    unsigned int size = 1U << info->dabt.size;
> +
> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));

This logic is the same as below in vpci_mmio_write(). So I think you 
want to provide an helper because this is not trivial to read.

Also, for the first line, I think you can re-use MMCFG_BDF() from the 
x86 code. For the second line, I would define the value so it is clearer 
to understand that they mean (although & 3 is fine to me) .

> +
> +    if ( !vpci_mmio_access_allowed(reg, size) )
> +        return 1;
So, you will a guest will read 0 if the access is unaligned. This seems 
an odd behavior given this is not an allowed access. AFAIU, the HW would 
likely trow a data abort because you can't do unalign access on 
uncachable memory. So I think we should return 0 here to let the MMIO 
handler inject a data abort.

> +
> +    data = vpci_read(sbdf, reg, size);

So in vpci_mmio_access_allowed(), you will allow a guest to read a 
64-bit value. But... vpci_read() will return a 32-bit value.

Looking at the x86 code, they have a second call to vpci_read() to 
handle the top 32-bit. Any reason why this was not implemented on Arm?

If we need to implement it then I think this should be implement in 
vpci_read() to avoid duplication between x86 and arm.

> +
> +    memcpy(r, &data, size);

 From my understanding, any unused bit should be 0. So this should be:

*r = data;

> +
> +    return 1;
> +}
> +
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> +                           register_t r, void *p)

My remarks on vpci_mmio_read() applies here too.

> +{
> +    unsigned int reg;
> +    pci_sbdf_t sbdf;
> +    uint32_t data = r;
> +    unsigned int size = 1U << info->dabt.size;
> +
> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> +
> +    if ( !vpci_mmio_access_allowed(reg, size) )
> +        return 1;
> +
> +    vpci_write(sbdf, reg, size, data);
> +
> +    return 1;
> +}
> +
> +static const struct mmio_handler_ops vpci_mmio_handler = {
> +    .read  = vpci_mmio_read,
> +    .write = vpci_mmio_write,
> +};
> +
> +int domain_vpci_init(struct domain *d)
> +{
> +    if ( !has_vpci(d) )
> +        return 0;
> +
> +    register_mmio_handler(d, &vpci_mmio_handler,
> +                          GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
> +
> +    return 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> +
> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
> new file mode 100644
> index 0000000000..8a093bb705
> --- /dev/null
> +++ b/xen/arch/arm/vpci.h
> @@ -0,0 +1,37 @@
> +/*
> + * xen/arch/arm/vpci.h
> + * Copyright (c) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef __ARCH_ARM_VPCI_H__
> +#define __ARCH_ARM_VPCI_H__
> +
> +#ifdef CONFIG_HAS_VPCI
> +int domain_vpci_init(struct domain *d);
> +#else
> +static inline int domain_vpci_init(struct domain *d)
> +{
> +    return 0;
> +}
> +#endif
> +
> +#endif /* __ARCH_ARM_VPCI_H__ */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> index c23c8cb06b..56e261e9bd 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>       else
>           iommu_enable_device(pdev);
>   
> +#ifdef CONFIG_ARM
> +    ret = vpci_add_handlers(pdev);
> +    if ( ret ) {
> +        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
> +        goto out;
> +    }
> +#endif
>       pci_enable_acs(pdev);
>   
>   out:
> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
> index 55d1bdfda0..1a1413b93e 100644
> --- a/xen/drivers/vpci/Makefile
> +++ b/xen/drivers/vpci/Makefile
> @@ -1 +1,2 @@
> -obj-y += vpci.o header.o msi.o msix.o
> +obj-y += vpci.o header.o
> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
> index ba9a036202..f8cd55e7c0 100644
> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>        * FIXME: punching holes after the p2m has been set up might be racy for
>        * DomU usage, needs to be revisited.
>        */
> +#ifdef CONFIG_HAS_PCI_MSI
>       if ( map && !rom_only && vpci_make_msix_hole(pdev) )
>           return;
> +#endif
>   
>       for ( i = 0; i < ARRAY_SIZE(header->bars); i++ )
>       {
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index c9277b5c6d..d742b94bd6 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>   
>   #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>   
> -#define has_vpci(d)    ({ (void)(d); false; })
> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
> + * for ARM we enable support VPCI for guest domain also.
> + */
> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); })
>   
>   #endif /* __ASM_DOMAIN_H__ */
>   
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 756f8637ab..c58152de80 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -27,6 +27,14 @@ struct arch_pci_dev {
>       struct device dev;
>   };
>   
> +/* Arch-specific MSI data for vPCI. */
> +struct vpci_arch_msi {
> +};
> +
> +/* Arch-specific MSI-X entry data for vPCI. */
> +struct vpci_arch_msix_entry {
> +};
> +
>   /*
>    * struct to hold the mappings of a config space window. This
>    * is expected to be used as sysdata for PCI controllers that
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 64a2ca30da..0a9749e768 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
>   #define GUEST_PL011_BASE    xen_mk_ullong(0x22000000)
>   #define GUEST_PL011_SIZE    xen_mk_ullong(0x00001000)
>   
> +/* VPCI ECAM mappings */
> +#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)

All the values for the memory layout has been defined in ascending 
order. So please add the vCPI at the correct place. If I am not 
mistaken, dhis should be before the GUEST_ACPI_*.

> +#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)

Please document how to decide the size. This is important for the future 
if we need to change the size.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl
  2021-08-20 16:03     ` Rahul Singh
@ 2021-09-09 13:59       ` Julien Grall
  2021-09-16 16:16         ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Julien Grall @ 2021-09-09 13:59 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Ian Jackson, Wei Liu,
	Anthony PERARD, Juergen Gross, Andrew Cooper, George Dunlap,
	Jan Beulich, Stefano Stabellini, Volodymyr Babchuk



On 20/08/2021 17:03, Rahul Singh wrote:
> Hi Julien,

Hi Rahul,

>> On 19 Aug 2021, at 2:00 pm, Julien Grall <julien@xen.org> wrote:
>>
>> Hi Rahul,
>>
>> On 19/08/2021 13:02, Rahul Singh wrote:
>>> libxl will create an emulated PCI device tree node in the device tree to
>>> enable the guest OS to discover the virtual PCI during guest boot.
>>> Emulated PCI device tree node will only be created when there is any
>>> device assigned to guest.
>>> A new area has been reserved in the arm guest physical map at
>>> which the VPCI bus is declared in the device tree (reg and ranges
>>> parameters of the node).
>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>> ---
>>>   tools/libs/light/libxl_arm.c          | 109 ++++++++++++++++++++++++++
>>>   tools/libs/light/libxl_types.idl      |   1 +
>>>   tools/xl/xl_parse.c                   |   2 +
>>>   xen/include/public/arch-arm.h         |  11 +++
>>>   xen/include/public/device_tree_defs.h |   1 +
>>>   5 files changed, 124 insertions(+)
>>> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
>>> index e3140a6e00..a091e97e76 100644
>>> --- a/tools/libs/light/libxl_arm.c
>>> +++ b/tools/libs/light/libxl_arm.c
>>> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>>>       return fdt_property(fdt, "reg", regs, sizeof(regs));
>>>   }
>>>   +static int fdt_property_values(libxl__gc *gc, void *fdt,
>>> +        const char *name, unsigned num_cells, ...)
>>> +{
>>> +    uint32_t prop[num_cells];
>>> +    be32 *cells = &prop[0];
>>> +    int i;
>>> +    va_list ap;
>>> +    uint32_t arg;
>>> +
>>> +    va_start(ap, num_cells);
>>> +    for (i = 0 ; i < num_cells; i++) {
>>> +        arg = va_arg(ap, uint32_t);
>>> +        set_cell(&cells, 1, arg);
>>> +    }
>>> +    va_end(ap);
>>> +
>>> +    return fdt_property(fdt, name, prop, sizeof(prop));
>>> +}
>>> +
>>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
>>> +                                    unsigned addr_cells,
>>> +                                    unsigned size_cells,
>>> +                                    unsigned num_regs, ...)
>>> +{
>>> +    uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
>>> +    be32 *cells = &regs[0];
>>> +    int i;
>>> +    va_list ap;
>>> +    uint64_t arg;
>>> +
>>> +    va_start(ap, num_regs);
>>> +    for (i = 0 ; i < num_regs; i++) {
>>> +        /* Set the memory bit field */
>>> +        arg = va_arg(ap, uint64_t);
>>> +        set_cell(&cells, 1, arg);
>>> +
>>> +        /* Set the vpci bus address */
>>> +        arg = addr_cells ? va_arg(ap, uint64_t) : 0;
>>> +        set_cell(&cells, addr_cells , arg);
>>> +
>>> +        /* Set the cpu bus address where vpci address is mapped */
>>> +        set_cell(&cells, addr_cells, arg);
>>> +
>>> +        /* Set the vpci size requested */
>>> +        arg = size_cells ? va_arg(ap, uint64_t) : 0;
>>> +        set_cell(&cells, size_cells,arg);
>>> +    }
>>> +    va_end(ap);
>>> +
>>> +    return fdt_property(fdt, "ranges", regs, sizeof(regs));
>>> +}
>>> +
>>>   static int make_root_properties(libxl__gc *gc,
>>>                                   const libxl_version_info *vers,
>>>                                   void *fdt)
>>> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>>>       return 0;
>>>   }
>>>   +static int make_vpci_node(libxl__gc *gc, void *fdt,
>>> +        const struct arch_info *ainfo,
>>> +        struct xc_dom_image *dom)
>>> +{
>>> +    int res;
>>> +    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
>>> +    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
>>> +    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
>>> +
>>> +    res = fdt_begin_node(fdt, name);
>>> +    if (res) return res;
>>> +
>>> +    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
>>> +    if (res) return res;
>>> +
>>> +    res = fdt_property_string(fdt, "device_type", "pci");
>>> +    if (res) return res;
>>> +
>>> +    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>>> +            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
>>> +    if (res) return res;
>>> +
>>> +    res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17);
>>
>> AFAICT, the "bus-range" is optional. Can you explain why we need it?
> 
> We need it to implement the function pci_ecam_map_bus().

Ok. Then why next question is what does the 17 mean? Is it tie to how we
implement the vPCI in Xen or the region we reserved?

[...]

>>
>>> +
>>>       if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
>>>           return;
>>>   diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
>>> index 3f9fff653a..78b1ddf0b8 100644
>>> --- a/tools/libs/light/libxl_types.idl
>>> +++ b/tools/libs/light/libxl_types.idl
>>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>>>         ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>>>                                  ("vuart", libxl_vuart_type),
>>> +                               ("vpci", libxl_defbool),
>>
>> Any new addition in the structure should be accompanied with a LIBXL_HAVE_* in the libxl.h header.
> 
> OK.
>>
>>>                                 ])),
>>>       ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
>>>                                 ])),
>>> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
>>> index 17dddb4cd5..ffafbeffb4 100644
>>> --- a/tools/xl/xl_parse.c
>>> +++ b/tools/xl/xl_parse.c
>>> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source,
>>>           }
>>>           if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
>>>               libxl_defbool_set(&b_info->u.pv.e820_host, true);
>>> +        if (d_config->num_pcidevs)
>>> +            libxl_defbool_set(&b_info->arch_arm.vpci, true);
>>>       }
>>>         if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) {
>>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>>> index 0a9749e768..01d13e669e 100644
>>> --- a/xen/include/public/arch-arm.h
>>> +++ b/xen/include/public/arch-arm.h
>>> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t;
>>>   #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
>>>   #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
>>>   +/* PCI-PCIe memory space types */
>>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
>>> +#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
>>
>> What the size of those regions?
> 
> Non Prefetch Memory: Size 64 MB start at 512 MB
> Prefetch Memory: Size 128 GB start at 36 GB
>>
>>> +
>>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
>>> +#define GUEST_VPCI_PREFETCH_MEM_ADDR  xen_mk_ullong(0x900000000)
>>> +#define GUEST_VPCI_MEM_ADDR           xen_mk_ullong(0x20000000)
>> So far the memory layout defines the address in ascending order. So please add that after GUEST_RAM_BANK_BASES_*.
> 
> Ok.
>>
>> However, if I am not mistaken that base address you provide will clash with RAM bank 1. It also seem to be pretty high which means that this will not work for 32-bit domain or on CPUs that don't allow offer large IPA bits.
> 
> Yes I also checked that now that it is having clash with RAM bank 1.
> There is unused space is guest memory that we can use for Non Prefetch Memory as per below guest memory map.
> https://gitlab.com/xen-project/fusa/fusa-docs/-/blob/master/high-level/guest-memory-layout-arm.rst
> 
> Proposed value:
> Non Prefetch Memory: Size 64 MB start at 0x22001000
> Prefetch Memory: Size 4 GB start at 4 GB.

The base address looks fine to me. However, the sizes are much smaller 
to what you initially suggested. Would you be able to clarify why the 
smaller sizes are fine?

> 
>>
>> I think we need to start making the guest layout more dynamic. The VPCI memory space would have to go right after the end of the RAM allocated for a given guest.
>>
>>> +
>>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x2000000000)
>>> +#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x04000000)
>>
>> It would be better if the size for each region is defined right after each base.
> OK.
> 
>>
>> Also, how did you decide the size of each region?
> 
> I thought 64 MB will be sufficient. I think it should be based on number of devices we can assign to the guest.

We don't have to get the size right now. What I am more interested is to 
have a trace about how those values were decided (even if it just saying 
random). This will help to make any decision if in the future we need to 
resize (in particular downsize) the regions.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-08-19 12:02 ` [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
  2021-09-07  9:05   ` Julien Grall
@ 2021-09-09 22:54   ` Stefano Stabellini
  2021-09-10 11:53     ` Rahul Singh
  2021-09-13 14:52   ` Oleksandr Andrushchenko
  2021-09-14  7:53   ` Oleksandr Andrushchenko
  3 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09 22:54 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

[-- Attachment #1: Type: text/plain, Size: 13192 bytes --]

On Thu, 19 Aug 2021, Rahul Singh wrote:
> XEN during boot will read the PCI device tree node “reg” property
> and will map the PCI config space to the XEN memory.
> 
> As of now "pci-host-ecam-generic" compatible board is supported.
> 
> "linux,pci-domain" device tree property assigns a fixed PCI domain
> number to a host bridge, otherwise an unstable (across boots) unique
> number will be assigned by Linux.This property has to be in sync with
> XEN to access the PCI devices.
> 
> XEN will read the “linux,pci-domain” property from the device tree node
> and configure the host bridge segment number accordingly. If this
> property is not available XEN will allocate the unique segment number
> to the host bridge.
> 
> dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly
> imported from the Linux source tree.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/pci/Makefile           |   2 +
>  xen/arch/arm/pci/pci-host-common.c  | 261 ++++++++++++++++++++++++++++
>  xen/arch/arm/pci/pci-host-generic.c |  55 ++++++
>  xen/include/asm-arm/pci.h           |  28 +++
>  4 files changed, 346 insertions(+)
>  create mode 100644 xen/arch/arm/pci/pci-host-common.c
>  create mode 100644 xen/arch/arm/pci/pci-host-generic.c
> 
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> index a9ee0b9b44..f3d97f859e 100644
> --- a/xen/arch/arm/pci/Makefile
> +++ b/xen/arch/arm/pci/Makefile
> @@ -1,2 +1,4 @@
>  obj-y += pci.o
>  obj-y += pci-access.o
> +obj-y += pci-host-generic.o
> +obj-y += pci-host-common.o
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> new file mode 100644
> index 0000000000..9dd9b02271
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -0,0 +1,261 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +#include <xen/rwlock.h>
> +#include <xen/sched.h>
> +#include <xen/vmap.h>
> +
> +/*
> + * List for all the pci host bridges.
> + */
> +
> +static LIST_HEAD(pci_host_bridges);
> +
> +static atomic_t domain_nr = ATOMIC_INIT(-1);
> +
> +bool dt_pci_parse_bus_range(struct dt_device_node *dev,
> +                            struct pci_config_window *cfg)
> +{
> +    const __be32 *cells;
> +    uint32_t len;
> +
> +    cells = dt_get_property(dev, "bus-range", &len);
> +    /* bus-range should at least be 2 cells */
> +    if ( !cells || (len < (sizeof(*cells) * 2)) )
> +        return false;
> +
> +    cfg->busn_start = dt_next_cell(1, &cells);
> +    cfg->busn_end = dt_next_cell(1, &cells);
> +
> +    return true;
> +}
> +
> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
> +{
> +    return ioremap_nocache(start, len);
> +}
> +
> +static void pci_ecam_free(struct pci_config_window *cfg)
> +{
> +    if ( cfg->win )
> +        iounmap(cfg->win);
> +
> +    xfree(cfg);
> +}
> +
> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> +                                              int ecam_reg_idx)

If it is only called at init time, then the function should be __init


> +{
> +    int err;
> +    struct pci_config_window *cfg;
> +    paddr_t addr, size;
> +
> +    cfg = xzalloc(struct pci_config_window);
> +    if ( !cfg )
> +        return NULL;
> +
> +    err = dt_pci_parse_bus_range(dev, cfg);
> +    if ( !err ) {
> +        cfg->busn_start = 0;
> +        cfg->busn_end = 0xff;
> +        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
> +               dt_node_full_name(dev));
> +    } else {
> +        if ( cfg->busn_end > cfg->busn_start + 0xff )
> +            cfg->busn_end = cfg->busn_start + 0xff;

Is this a hard limit in the specification? Or is it a limit in the Xen
implementation?


> +    }
> +
> +    /* Parse our PCI ecam register address*/
                                            ^ space

> +    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
> +    if ( err )
> +        goto err_exit;
> +
> +    cfg->phys_addr = addr;
> +    cfg->size = size;
> +
> +    /*
> +     * On 64-bit systems, we do a single ioremap for the whole config space
> +     * since we have enough virtual address range available.  On 32-bit, we
> +     * ioremap the config space for each bus individually.
> +     *
> +     * As of now only 64-bit is supported 32-bit is not supported.
> +     */
> +    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
> +    if ( !cfg->win )
> +        goto err_exit_remap;
> +
> +    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
> +            cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
> +
> +    return cfg;
> +
> +err_exit_remap:
> +    printk(XENLOG_ERR "ECAM ioremap failed\n");
> +err_exit:
> +    pci_ecam_free(cfg);
> +    return NULL;
> +}
> +
> +struct pci_host_bridge *pci_alloc_host_bridge(void)
> +{
> +    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
> +
> +    if ( !bridge )
> +        return NULL;
> +
> +    INIT_LIST_HEAD(&bridge->node);
> +    bridge->bus_start = ~0;
> +    bridge->bus_end = ~0;

Please use INVALID_PADDR instead of ~0


> +    return bridge;
> +}
> +
> +void pci_add_host_bridge(struct pci_host_bridge *bridge)
> +{
> +    list_add_tail(&bridge->node, &pci_host_bridges);
> +}
> +
> +static int pci_get_new_domain_nr(void)
> +{
> +    return atomic_inc_return(&domain_nr);
> +}
> +
> +/*
> + * This function will try to obtain the host bridge domain number by
> + * finding a property called "linux,pci-domain" of the given device node.
> + *
> + * @node: device tree node with the domain information
> + *
> + * Returns the associated domain number from DT in the range [0-0xffff], or
> + * a negative value if the required property is not found.
> + */
> +static int dt_get_pci_domain_nr(struct dt_device_node *node)
> +{
> +    u32 domain;
> +    int error;
> +
> +    error = dt_property_read_u32(node, "linux,pci-domain", &domain);
> +    if ( !error )
> +        return -EINVAL;
> +
> +    return (u16)domain;

Let's check that domain <= UINT16_MAX


> +}
> +
> +static int pci_bus_find_domain_nr(struct dt_device_node *dev)
> +{
> +    static int use_dt_domains = -1;
> +    int domain;
> +
> +    domain = dt_get_pci_domain_nr(dev);
> +
> +    /*
> +     * Check DT domain and use_dt_domains values.
> +     *
> +     * If DT domain property is valid (domain >= 0) and
> +     * use_dt_domains != 0, the DT assignment is valid since this means
> +     * we have not previously allocated a domain number by using
> +     * pci_get_new_domain_nr(); we should also update use_dt_domains to
> +     * 1, to indicate that we have just assigned a domain number from
> +     * DT.
> +     *
> +     * If DT domain property value is not valid (ie domain < 0), and we
> +     * have not previously assigned a domain number from DT
> +     * (use_dt_domains != 1) we should assign a domain number by
> +     * using the:
> +     *
> +     * pci_get_new_domain_nr()
> +     *
> +     * API and update the use_dt_domains value to keep track of method we
> +     * are using to assign domain numbers (use_dt_domains = 0).
> +     *
> +     * All other combinations imply we have a platform that is trying
> +     * to mix domain numbers obtained from DT and pci_get_new_domain_nr(),
> +     * which is a recipe for domain mishandling and it is prevented by
> +     * invalidating the domain value (domain = -1) and printing a
> +     * corresponding error.
> +     */
> +    if ( domain >= 0 && use_dt_domains )
> +    {
> +        use_dt_domains = 1;
> +    }
> +    else if ( domain < 0 && use_dt_domains != 1 )
> +    {
> +        use_dt_domains = 0;
> +        domain = pci_get_new_domain_nr();
> +    }
> +    else
> +    {
> +        printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n");
> +        BUG();
> +    }
> +
> +    return domain;
> +}
> +
> +int pci_host_common_probe(struct dt_device_node *dev,
> +                          int ecam_reg_idx)
> +{
> +    struct pci_host_bridge *bridge;
> +    struct pci_config_window *cfg;
> +    int err;
> +
> +    bridge = pci_alloc_host_bridge();
> +    if ( !bridge )
> +        return -ENOMEM;
> +
> +    /* Parse and map our Configuration Space windows */
> +    cfg = gen_pci_init(dev, ecam_reg_idx);
> +    if ( !cfg )
> +    {
> +        err = -ENOMEM;
> +        goto err_exit;
> +    }
> +
> +    bridge->dt_node = dev;
> +    bridge->sysdata = cfg;
> +    bridge->bus_start = cfg->busn_start;
> +    bridge->bus_end = cfg->busn_end;
> +
> +    bridge->segment = pci_bus_find_domain_nr(dev);
> +
> +    pci_add_host_bridge(bridge);
> +
> +    return 0;
> +
> +err_exit:
> +    xfree(bridge);
> +    return err;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
> new file mode 100644
> index 0000000000..13d0f7f999
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-generic.c
> @@ -0,0 +1,55 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/device.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +
> +static const struct dt_device_match gen_pci_dt_match[] = {
> +    { .compatible = "pci-host-ecam-generic" },
> +    { },
> +};
> +
> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
> +{
> +    const struct dt_device_match *of_id;
> +
> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> +
> +    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
> +           dt_node_full_name(dev), of_id->compatible);
> +
> +    return pci_host_common_probe(dev, 0);
> +}
> +
> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
> +.dt_match = gen_pci_dt_match,
> +.init = gen_pci_dt_init,
> +DT_DEVICE_END
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 61e43da088..58a51e724e 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -26,6 +26,34 @@ struct arch_pci_dev {
>      struct device dev;
>  };
>  
> +/*
> + * struct to hold the mappings of a config space window. This
> + * is expected to be used as sysdata for PCI controllers that
> + * use ECAM.
> + */
> +struct pci_config_window {
> +    paddr_t         phys_addr;
> +    paddr_t         size;
> +    uint8_t         busn_start;
> +    uint8_t         busn_end;
> +    void __iomem    *win;
> +};
> +
> +/*
> + * struct to hold pci host bridge information
> + * for a PCI controller.
> + */
> +struct pci_host_bridge {
> +    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
> +    struct list_head node;           /* Node in list of host bridges */
> +    uint16_t segment;                /* Segment number */
> +    u8 bus_start;                    /* Bus start of this bridge. */
> +    u8 bus_end;                      /* Bus end of this bridge. */
> +    void *sysdata;                   /* Pointer to the config space window*/
> +};
> +
> +int pci_host_common_probe(struct dt_device_node *dev,
> +                          int ecam_reg_idx);
>  #else   /*!CONFIG_HAS_PCI*/
>  
>  struct arch_pci_dev { };
> -- 
> 2.17.1
> 

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-08-19 12:02 ` [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations Rahul Singh
  2021-09-09 11:32   ` Julien Grall
@ 2021-09-09 23:21   ` Stefano Stabellini
  2021-09-14 11:13     ` Rahul Singh
  1 sibling, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09 23:21 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

On Thu, 19 Aug 2021, Rahul Singh wrote:
> Add support for PCI ecam operations to access the PCI
> configuration space.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/pci/Makefile           |  1 +
>  xen/arch/arm/pci/ecam.c             | 63 +++++++++++++++++++++++++++++
>  xen/arch/arm/pci/pci-access.c       | 53 ++++++++++++++++++++++++
>  xen/arch/arm/pci/pci-host-common.c  | 13 +++++-
>  xen/arch/arm/pci/pci-host-generic.c |  8 +++-
>  xen/include/asm-arm/pci.h           | 32 +++++++++++++++
>  6 files changed, 167 insertions(+), 3 deletions(-)
>  create mode 100644 xen/arch/arm/pci/ecam.c
> 
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> index f3d97f859e..6f32fbbe67 100644
> --- a/xen/arch/arm/pci/Makefile
> +++ b/xen/arch/arm/pci/Makefile
> @@ -2,3 +2,4 @@ obj-y += pci.o
>  obj-y += pci-access.o
>  obj-y += pci-host-generic.o
>  obj-y += pci-host-common.o
> +obj-y += ecam.o
> diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c
> new file mode 100644
> index 0000000000..91c691b41f
> --- /dev/null
> +++ b/xen/arch/arm/pci/ecam.c
> @@ -0,0 +1,63 @@
> +/*
> + * Copyright (C) 2021 Arm Ltd.
> + *
> + * Based on Linux drivers/pci/ecam.c
> + * Copyright 2016 Broadcom
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/pci.h>
> +#include <xen/sched.h>
> +
> +/*
> + * Function to implement the pci_ops ->map_bus method.
> + */
> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
> +                                      uint32_t sbdf, uint32_t where)

Code style: alignment


> +{
> +    const struct pci_config_window *cfg = bridge->sysdata;
> +    unsigned int devfn_shift = cfg->ops->bus_shift - 8;

Is it a guarantee that devfn_shift == bus_shift - 8, or is it just so
for ECAM?


> +    void __iomem *base;
> +
> +    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
> +    unsigned int busn = sbdf_t.bus;
> +
> +    if ( busn < cfg->busn_start || busn > cfg->busn_end )

Genuine question: should it be busn >= cfg->busn_end ?  I don't know if
the range includes busn_end or not.


> +        return NULL;
> +
> +    busn -= cfg->busn_start;
> +    base = cfg->win + (busn << cfg->ops->bus_shift);
> +
> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
> +}

I understand that the arm32 part is not implemented and not part of this
series, that's fine. However if the plan is that arm32 will dynamically
map each bus individually, then I imagine this function will have an
ioremap in the arm32 version. Which means that we also need an
unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus
would be a NOP today for arm64, but I think it makes sense to have it if
we want the API to be generic.


> +/* ECAM ops */
> +const struct pci_ecam_ops pci_generic_ecam_ops = {
> +    .bus_shift  = 20,
> +    .pci_ops    = {
> +        .map_bus                = pci_ecam_map_bus,
> +        .read                   = pci_generic_config_read,
> +        .write                  = pci_generic_config_write,
> +    }
> +};
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
> index b938047c03..f39f6a3a38 100644
> --- a/xen/arch/arm/pci/pci-access.c
> +++ b/xen/arch/arm/pci/pci-access.c
> @@ -15,6 +15,59 @@
>   */
>  
>  #include <xen/pci.h>
> +#include <asm/io.h>
> +
> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t *value)
> +{
> +    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
> +    if (!addr) {
> +        *value = ~0;

Is this a standard error? If so, I think we should define it with a
macro (e.g. INVALID_PADDR).


> +        return -ENODEV;
> +    }
> +
> +    switch (len)
> +    {
> +    case 1:
> +        *value = readb(addr);
> +        break;
> +    case 2:
> +        *value = readw(addr);
> +        break;
> +    case 4:
> +        *value = readl(addr);
> +        break;
> +    default:
> +        BUG();

A BUG here is harsh because it could be potentially guest-triggered. An
ASSERT would be better.


> +    }
> +
> +    return 0;
> +}
> +
> +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t value)
> +{
> +    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
> +    if (!addr)
> +        return -ENODEV;
> +
> +    switch (len)
> +    {
> +    case 1:
> +        writeb(value, addr);
> +        break;
> +    case 2:
> +        writew(value, addr);
> +        break;
> +    case 4:
> +        writel(value, addr);
> +        break;
> +    default:
> +        BUG();

Same here


> +    }
> +
> +    return 0;
> +}
>  
>  static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>                                  unsigned int len)
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> index 9dd9b02271..c582527e92 100644
> --- a/xen/arch/arm/pci/pci-host-common.c
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg)
>  }
>  
>  static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> +                                              const struct pci_ecam_ops *ops,
>                                                int ecam_reg_idx)
>  {
>      int err;
> @@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>  
>      cfg->phys_addr = addr;
>      cfg->size = size;
> +    cfg->ops = ops;
>  
>      /*
>       * On 64-bit systems, we do a single ioremap for the whole config space
> @@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>      printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>              cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
>  
> +    if ( ops->init )
> +    {
> +        err = ops->init(cfg);
> +        if (err)
> +            goto err_exit;
> +    }
> +
>      return cfg;
>  
>  err_exit_remap:
> @@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev)
>  }
>  
>  int pci_host_common_probe(struct dt_device_node *dev,
> +                          const struct pci_ecam_ops *ops,
>                            int ecam_reg_idx)
>  {
>      struct pci_host_bridge *bridge;
> @@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
>          return -ENOMEM;
>  
>      /* Parse and map our Configuration Space windows */
> -    cfg = gen_pci_init(dev, ecam_reg_idx);
> +    cfg = gen_pci_init(dev, ops, ecam_reg_idx);
>      if ( !cfg )
>      {
>          err = -ENOMEM;
> @@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
>  
>      bridge->dt_node = dev;
>      bridge->sysdata = cfg;
> +    bridge->ops = &ops->pci_ops;
>      bridge->bus_start = cfg->busn_start;
>      bridge->bus_end = cfg->busn_end;
>  
> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
> index 13d0f7f999..2d652e8910 100644
> --- a/xen/arch/arm/pci/pci-host-generic.c
> +++ b/xen/arch/arm/pci/pci-host-generic.c
> @@ -23,20 +23,24 @@
>  #include <asm/pci.h>
>  
>  static const struct dt_device_match gen_pci_dt_match[] = {
> -    { .compatible = "pci-host-ecam-generic" },
> +    { .compatible = "pci-host-ecam-generic",
> +      .data =       &pci_generic_ecam_ops },
> +
>      { },
>  };
>  
>  static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>  {
>      const struct dt_device_match *of_id;
> +    const struct pci_ecam_ops *ops;
>  
>      of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> +    ops = (struct pci_ecam_ops *) of_id->data;
>  
>      printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>             dt_node_full_name(dev), of_id->compatible);
>  
> -    return pci_host_common_probe(dev, 0);
> +    return pci_host_common_probe(dev, ops, 0);
>  }
>  
>  DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 58a51e724e..22866244d2 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -37,6 +37,7 @@ struct pci_config_window {
>      uint8_t         busn_start;
>      uint8_t         busn_end;
>      void __iomem    *win;
> +    const struct    pci_ecam_ops *ops;
>  };
>  
>  /*
> @@ -50,10 +51,41 @@ struct pci_host_bridge {
>      u8 bus_start;                    /* Bus start of this bridge. */
>      u8 bus_end;                      /* Bus end of this bridge. */
>      void *sysdata;                   /* Pointer to the config space window*/
> +    const struct pci_ops *ops;
>  };
>  
> +struct pci_ops {
> +    void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                             uint32_t offset);
> +    int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                uint32_t reg, uint32_t len, uint32_t *value);
> +    int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                 uint32_t reg, uint32_t len, uint32_t value);
> +};
> +
> +/*
> + * struct to hold pci ops and bus shift of the config window
> + * for a PCI controller.
> + */
> +struct pci_ecam_ops {
> +    unsigned int            bus_shift;
> +    struct pci_ops          pci_ops;
> +    int (*init)(struct pci_config_window *);

Is this last member of the struct needed/used?


> +};
> +
> +/* Default ECAM ops */
> +extern const struct pci_ecam_ops pci_generic_ecam_ops;
> +
>  int pci_host_common_probe(struct dt_device_node *dev,
> +                          const struct pci_ecam_ops *ops,
>                            int ecam_reg_idx);
> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t *value);
> +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
> +                            uint32_t reg, uint32_t len, uint32_t value);
> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
> +                               uint32_t sbdf, uint32_t where);
> +
>  #else   /*!CONFIG_HAS_PCI*/
>  
>  struct arch_pci_dev { };
> -- 
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller
  2021-08-19 12:02 ` [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh
@ 2021-09-09 23:34   ` Stefano Stabellini
  2021-09-10 12:01     ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09 23:34 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Oleksandr Andrushchenko,
	Stefano Stabellini, Julien Grall, Volodymyr Babchuk

On Thu, 19 Aug 2021, Rahul Singh wrote:
> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> 
> Add support for Xilinx ZynqMP PCI host controller to map the PCI config
> space to the XEN memory.
> 
> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> ---
>  xen/arch/arm/pci/Makefile          |  1 +
>  xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++
>  2 files changed, 60 insertions(+)
>  create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
> 
> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> index 6f32fbbe67..1d045ade01 100644
> --- a/xen/arch/arm/pci/Makefile
> +++ b/xen/arch/arm/pci/Makefile
> @@ -3,3 +3,4 @@ obj-y += pci-access.o
>  obj-y += pci-host-generic.o
>  obj-y += pci-host-common.o
>  obj-y += ecam.o
> +obj-y += pci-host-zynqmp.o
> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
> new file mode 100644
> index 0000000000..fe103e3855
> --- /dev/null
> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c
> @@ -0,0 +1,59 @@
> +/*
> + * Copyright (C) 2020-2021 EPAM Systems
> + *
> + * Based on Linux drivers/pci/controller/pci-host-common.c
> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> + * Based on xen/arch/arm/pci/pci-host-generic.c
> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>

Only one Copyright line per file is enough :-)

But actually all the Copyright lines with a name or a company name are
not really required or useful, as the copyright is noted in full details
in the commit messages (author and signed-off-by lines). I would remove
them all from the new files added by this series.


> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/device.h>
> +#include <xen/pci.h>
> +#include <asm/pci.h>
> +
> +static const struct dt_device_match gen_pci_dt_match[] = {
> +    { .compatible = "xlnx,nwl-pcie-2.11",
> +      .data =       &pci_generic_ecam_ops },
> +    { },
> +};
> +
> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
> +{
> +    const struct dt_device_match *of_id;
> +    const struct pci_ecam_ops *ops;
> +
> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);

This should be superfluous


> +    ops = (struct pci_ecam_ops *) of_id->data;
> +
> +    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
> +            dt_node_full_name(dev), of_id->compatible);
> +
> +    return pci_host_common_probe(dev, ops, 2);
> +}
> +
> +DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI)
> +.dt_match = gen_pci_dt_match,
> +.init = gen_pci_dt_init,
> +DT_DEVICE_END
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 08/14] xen:arm: Implement pci access functions
  2021-08-19 12:02 ` [PATCH v1 08/14] xen:arm: Implement pci access functions Rahul Singh
@ 2021-09-09 23:41   ` Stefano Stabellini
  2021-09-14 16:05     ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09 23:41 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

On Thu, 19 Aug 2021, Rahul Singh wrote:
> Implement generic pci access functions to read/write the configuration
> space.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/pci/pci-access.c      | 31 +++++++++++++++++++++++++++++-
>  xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++
>  xen/include/asm-arm/pci.h          |  2 ++
>  3 files changed, 51 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
> index f39f6a3a38..b94de3c3ac 100644
> --- a/xen/arch/arm/pci/pci-access.c
> +++ b/xen/arch/arm/pci/pci-access.c
> @@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>  static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>                                  unsigned int len)
>  {
> -    return ~0U;
> +    uint32_t val = GENMASK(0, len * 8);

This seems to be another default error value that it would be better to
define with its own macro


> +    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> +
> +    if ( unlikely(!bridge) )
> +    {
> +        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> +                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

You are not actually printing sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn ?


> +        return val;
> +    }
> +
> +    if ( unlikely(!bridge->ops->read) )
> +        return val;
> +
> +    bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);

Would it make sense to make the interface take a pci_sbdf_t directly
instead of casting to uint32_t and back?


> +    return val;
>  }
>  
>  static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>                               unsigned int len, uint32_t val)
>  {
> +    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> +
> +    if ( unlikely(!bridge) )
> +    {
> +        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> +                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

same here


> +        return;
> +    }
> +
> +    if ( unlikely(!bridge->ops->write) )
> +        return;
> +
> +    bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);

same here


>  }
>  
>  /*
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> index c582527e92..62715b4676 100644
> --- a/xen/arch/arm/pci/pci-host-common.c
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -261,6 +261,25 @@ err_exit:
>      return err;
>  }
>  
> +/*
> + * This function will lookup an hostbridge based on the segment and bus
> + * number.
> + */
> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
> +{
> +    struct pci_host_bridge *bridge;
> +
> +    list_for_each_entry( bridge, &pci_host_bridges, node )
> +    {
> +        if ( bridge->segment != segment )
> +            continue;
> +        if ( (bus < bridge->bus_start) || (bus > bridge->bus_end) )
> +            continue;
> +        return bridge;
> +    }
> +
> +    return NULL;
> +}
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 22866244d2..756f8637ab 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -20,6 +20,7 @@
>  #ifdef CONFIG_HAS_PCI
>  
>  #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
> +#define PRI_pci "%04x:%02x:%02x.%u"
>  
>  /* Arch pci dev struct */
>  struct arch_pci_dev {
> @@ -86,6 +87,7 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>  void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
>                                 uint32_t sbdf, uint32_t where);
>  
> +struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
>  #else   /*!CONFIG_HAS_PCI*/
>  
>  struct arch_pci_dev { };
> -- 
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-20 14:37         ` Jan Beulich
@ 2021-09-09 23:46           ` Stefano Stabellini
  0 siblings, 0 replies; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09 23:46 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Julien Grall, Rahul Singh, xen-devel, Bertrand Marquis,
	Stefano Stabellini, Volodymyr Babchuk

On Fri, 20 Aug 2021, Jan Beulich wrote:
> On 20.08.2021 16:34, Julien Grall wrote:
> > On 20/08/2021 13:19, Rahul Singh wrote:
> >>> On 19 Aug 2021, at 1:31 pm, Julien Grall <julien@xen.org> wrote:
> >>> On 19/08/2021 13:02, Rahul Singh wrote:
> >>>> Add cmdline boot option "pci=on" to enable/disable the PCI init during
> >>>> boot.
> >>>
> >>> I read this as "PCI" will be either disabled/enabled for the platform. Whereas, I think it will be used to decide whether Xen discover PCI and PCI passthrough is supported or not.
> >>
> >> Yes. I will modify the option to "pci-passthrough== <boolean>"
> >>>
> >>> Can you also clarify why a user would want to select "pci=off"?
> >>
> >> As pci-passthrough support emulate the PCI devices for DOM0 also, I thought if someone want to
> >> boot the DOM0 without emulating the PCI device in XEN and wants to have direct access to device.
> > 
> > Dom0 will always have direct access to the PCI device. The only 
> > difference is whether the access to the hostbridge and config space will 
> > be trapped by Xen. I expect the both to mainly happen during boot and 
> > therefore the overhead will be limited.
> > 
> >>
> >> I am ok to drop this patch if you feel adding the option is not required at all.
> > One of the reason I could see this option to be useful is to figure out 
> > if an issue occurs because of the hostbridge emulation. Yet, I am still 
> > not fully convinced adding an option is worth it.
> > 
> > Jan and others, any opinions?
> 
> Well, if there's a proper fallback, then why not allow using it in
> case of problems?

I think it would be good to have the option, if nothing else for
debugging.


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on"
  2021-08-19 12:02 ` [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on" Rahul Singh
  2021-08-19 12:09   ` Jan Beulich
  2021-08-19 12:31   ` Julien Grall
@ 2021-09-09 23:48   ` Stefano Stabellini
  2 siblings, 0 replies; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-09 23:48 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

On Thu, 19 Aug 2021, Rahul Singh wrote:
> Add cmdline boot option "pci=on" to enable/disable the PCI init during
> boot.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/pci/pci.c | 30 ++++++++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)
> 
> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
> index d1c9cf997d..dc63bbc2a2 100644
> --- a/xen/arch/arm/pci/pci.c
> +++ b/xen/arch/arm/pci/pci.c
> @@ -62,8 +62,38 @@ static void __init acpi_pci_init(void)
>  static inline void __init acpi_pci_init(void) { }
>  #endif
>  
> +static bool __initdata param_pci_enable;
> +
> +static int __init parse_pci_param(const char *arg)
> +{
> +    if ( !arg )
> +    {
> +        param_pci_enable = false;
> +        return 0;
> +    }
> +
> +    switch ( parse_bool(arg, NULL) )
> +    {
> +    case 0:
> +        param_pci_enable = false;
> +        return 0;
> +    case 1:
> +        param_pci_enable = true;
> +        return 0;
> +    }
> +
> +    return -EINVAL;
> +}
> +custom_param("pci", parse_pci_param);

Consider using boolean_param instead. It supports "on".


>  static int __init pci_init(void)
>  {
> +    /*
> +     * Enable PCI when has been enabled explicitly (pci=on)
> +     */
> +    if ( !param_pci_enable)
> +        return 0;
> +
>      if ( acpi_disabled )
>          dt_pci_init();
>      else
> -- 
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-08-19 12:02 ` [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
  2021-08-24 16:09   ` Jan Beulich
  2021-09-09 13:50   ` Julien Grall
@ 2021-09-10  0:26   ` Stefano Stabellini
  2021-09-16 11:01     ` Rahul Singh
  2 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-10  0:26 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk, Jan Beulich, Paul Durrant,
	Roger Pau Monné

[-- Attachment #1: Type: text/plain, Size: 9937 bytes --]

On Thu, 19 Aug 2021, Rahul Singh wrote:
> The existing VPCI support available for X86 is adapted for Arm.
> When the device is added to XEN via the hyper call
> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> access is added to the Xen to emulate the PCI devices config space.

This is done just for device discovery, right?

Although it is currently not implemented (and I am not asking to
implement it now, I am only trying to understand the architecture), it
would be possible to discover all PCI devices just by walking down the
PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we
have an ECAM driver.

I take that would be the way to implement PCI support for Dom0less?

 
> A MMIO trap handler for the PCI ECAM space is registered in XEN
> so that when guest is trying to access the PCI config space,XEN
> will trap the access and emulate read/write using the VPCI and
> not the real PCI hardware.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/Makefile         |  1 +
>  xen/arch/arm/domain.c         |  4 ++
>  xen/arch/arm/vpci.c           | 96 +++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vpci.h           | 37 ++++++++++++++
>  xen/drivers/passthrough/pci.c |  7 +++
>  xen/drivers/vpci/Makefile     |  3 +-
>  xen/drivers/vpci/header.c     |  2 +
>  xen/include/asm-arm/domain.h  |  5 +-
>  xen/include/asm-arm/pci.h     |  8 +++
>  xen/include/public/arch-arm.h |  4 ++
>  10 files changed, 165 insertions(+), 2 deletions(-)
>  create mode 100644 xen/arch/arm/vpci.c
>  create mode 100644 xen/arch/arm/vpci.h
> 
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 0e14a5e5c8..7cdce684a4 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -7,6 +7,7 @@ obj-y += platforms/
>  endif
>  obj-$(CONFIG_TEE) += tee/
>  obj-$(CONFIG_HAS_PCI) += pci/
> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>  
>  obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>  obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 19c756ac3d..d99c653626 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -40,6 +40,7 @@
>  #include <asm/vtimer.h>
>  
>  #include "vuart.h"
> +#include "vpci.h"
>  
>  DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>  
> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>      if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>          goto fail;
>  
> +    if ( (rc = domain_vpci_init(d)) != 0 )
> +        goto fail;
> +
>      return 0;
>  
>  fail:
> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
> new file mode 100644
> index 0000000000..da8b1ca13c
> --- /dev/null
> +++ b/xen/arch/arm/vpci.c
> @@ -0,0 +1,96 @@
> +/*
> + * xen/arch/arm/vpci.c
> + * Copyright (c) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +#include <xen/sched.h>
> +#include <asm/mmio.h>
> +
> +/* Do some sanity checks. */
> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
> +{
> +    /* Check access size. */
> +    if ( len != 1 && len != 2 && len != 4 && len != 8 )
> +        return false;
> +
> +    /* Check that access is size aligned. */
> +    if ( (reg & (len - 1)) )
> +        return false;
> +
> +    return true;
> +}
> +
> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
> +                          register_t *r, void *p)
> +{
> +    unsigned int reg;
> +    pci_sbdf_t sbdf;
> +    uint32_t data = 0;
> +    unsigned int size = 1U << info->dabt.size;
> +
> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> +
> +    if ( !vpci_mmio_access_allowed(reg, size) )
> +        return 1;
> +
> +    data = vpci_read(sbdf, reg, size);
> +
> +    memcpy(r, &data, size);
> +
> +    return 1;
> +}
> +
> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
> +                           register_t r, void *p)
> +{
> +    unsigned int reg;
> +    pci_sbdf_t sbdf;
> +    uint32_t data = r;
> +    unsigned int size = 1U << info->dabt.size;
> +
> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> +
> +    if ( !vpci_mmio_access_allowed(reg, size) )
> +        return 1;
> +
> +    vpci_write(sbdf, reg, size, data);
> +
> +    return 1;
> +}
> +
> +static const struct mmio_handler_ops vpci_mmio_handler = {
> +    .read  = vpci_mmio_read,
> +    .write = vpci_mmio_write,
> +};
> +
> +int domain_vpci_init(struct domain *d)
> +{
> +    if ( !has_vpci(d) )
> +        return 0;
> +
> +    register_mmio_handler(d, &vpci_mmio_handler,
> +                          GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
> +
> +    return 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> +
> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
> new file mode 100644
> index 0000000000..8a093bb705
> --- /dev/null
> +++ b/xen/arch/arm/vpci.h
> @@ -0,0 +1,37 @@
> +/*
> + * xen/arch/arm/vpci.h
> + * Copyright (c) 2021 Arm Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef __ARCH_ARM_VPCI_H__
> +#define __ARCH_ARM_VPCI_H__
> +
> +#ifdef CONFIG_HAS_VPCI
> +int domain_vpci_init(struct domain *d);
> +#else
> +static inline int domain_vpci_init(struct domain *d)
> +{
> +    return 0;
> +}
> +#endif
> +
> +#endif /* __ARCH_ARM_VPCI_H__ */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
> index c23c8cb06b..56e261e9bd 100644
> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>      else
>          iommu_enable_device(pdev);
>  
> +#ifdef CONFIG_ARM
> +    ret = vpci_add_handlers(pdev);
> +    if ( ret ) {
> +        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
> +        goto out;
> +    }
> +#endif
>      pci_enable_acs(pdev);
>  
>  out:
> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
> index 55d1bdfda0..1a1413b93e 100644
> --- a/xen/drivers/vpci/Makefile
> +++ b/xen/drivers/vpci/Makefile
> @@ -1 +1,2 @@
> -obj-y += vpci.o header.o msi.o msix.o
> +obj-y += vpci.o header.o
> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
> index ba9a036202..f8cd55e7c0 100644
> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>       * FIXME: punching holes after the p2m has been set up might be racy for
>       * DomU usage, needs to be revisited.
>       */
> +#ifdef CONFIG_HAS_PCI_MSI
>      if ( map && !rom_only && vpci_make_msix_hole(pdev) )
>          return;
> +#endif
>  
>      for ( i = 0; i < ARRAY_SIZE(header->bars); i++ )
>      {
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index c9277b5c6d..d742b94bd6 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>  
>  #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>  
> -#define has_vpci(d)    ({ (void)(d); false; })
> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
> + * for ARM we enable support VPCI for guest domain also.
> + */
> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); })
>  
>  #endif /* __ASM_DOMAIN_H__ */
>  
> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
> index 756f8637ab..c58152de80 100644
> --- a/xen/include/asm-arm/pci.h
> +++ b/xen/include/asm-arm/pci.h
> @@ -27,6 +27,14 @@ struct arch_pci_dev {
>      struct device dev;
>  };
>  
> +/* Arch-specific MSI data for vPCI. */
> +struct vpci_arch_msi {
> +};
> +
> +/* Arch-specific MSI-X entry data for vPCI. */
> +struct vpci_arch_msix_entry {
> +};
> +
>  /*
>   * struct to hold the mappings of a config space window. This
>   * is expected to be used as sysdata for PCI controllers that
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 64a2ca30da..0a9749e768 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
>  #define GUEST_PL011_BASE    xen_mk_ullong(0x22000000)
>  #define GUEST_PL011_SIZE    xen_mk_ullong(0x00001000)
>  
> +/* VPCI ECAM mappings */
> +#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
> +#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)

Isn't 256MB a bit too small? Usually PCI comes with two ranges:

- a small range below 4GB like this one
- an additional large range above 4GB, around 1GB or larger

We are missing the second larger range?

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl
  2021-08-19 12:02 ` [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
  2021-08-19 13:00   ` Julien Grall
@ 2021-09-10  0:51   ` Stefano Stabellini
  2021-09-16 16:35     ` Rahul Singh
  1 sibling, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-10  0:51 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Ian Jackson, Wei Liu,
	Anthony PERARD, Juergen Gross, Andrew Cooper, George Dunlap,
	Jan Beulich, Julien Grall, Stefano Stabellini, Volodymyr Babchuk

On Thu, 19 Aug 2021, Rahul Singh wrote:
> libxl will create an emulated PCI device tree node in the device tree to
> enable the guest OS to discover the virtual PCI during guest boot.
> Emulated PCI device tree node will only be created when there is any
> device assigned to guest.
> 
> A new area has been reserved in the arm guest physical map at
> which the VPCI bus is declared in the device tree (reg and ranges
> parameters of the node).
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  tools/libs/light/libxl_arm.c          | 109 ++++++++++++++++++++++++++
>  tools/libs/light/libxl_types.idl      |   1 +
>  tools/xl/xl_parse.c                   |   2 +
>  xen/include/public/arch-arm.h         |  11 +++
>  xen/include/public/device_tree_defs.h |   1 +
>  5 files changed, 124 insertions(+)
> 
> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> index e3140a6e00..a091e97e76 100644
> --- a/tools/libs/light/libxl_arm.c
> +++ b/tools/libs/light/libxl_arm.c
> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>      return fdt_property(fdt, "reg", regs, sizeof(regs));
>  }
>  
> +static int fdt_property_values(libxl__gc *gc, void *fdt,
> +        const char *name, unsigned num_cells, ...)
> +{
> +    uint32_t prop[num_cells];
> +    be32 *cells = &prop[0];
> +    int i;
> +    va_list ap;
> +    uint32_t arg;
> +
> +    va_start(ap, num_cells);
> +    for (i = 0 ; i < num_cells; i++) {
> +        arg = va_arg(ap, uint32_t);
> +        set_cell(&cells, 1, arg);
> +    }
> +    va_end(ap);
> +
> +    return fdt_property(fdt, name, prop, sizeof(prop));
> +}
> +
> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
> +                                    unsigned addr_cells,
> +                                    unsigned size_cells,
> +                                    unsigned num_regs, ...)
> +{
> +    uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
> +    be32 *cells = &regs[0];
> +    int i;
> +    va_list ap;
> +    uint64_t arg;
> +
> +    va_start(ap, num_regs);
> +    for (i = 0 ; i < num_regs; i++) {
> +        /* Set the memory bit field */
> +        arg = va_arg(ap, uint64_t);
> +        set_cell(&cells, 1, arg);

Shouldn't this be uint32_t given that it is 1 cell exactly?


> +        /* Set the vpci bus address */
> +        arg = addr_cells ? va_arg(ap, uint64_t) : 0;
> +        set_cell(&cells, addr_cells , arg);
> +
> +        /* Set the cpu bus address where vpci address is mapped */
> +        set_cell(&cells, addr_cells, arg);
> +
> +        /* Set the vpci size requested */
> +        arg = size_cells ? va_arg(ap, uint64_t) : 0;
> +        set_cell(&cells, size_cells,arg);
                                       ^ space


> +    }
> +    va_end(ap);
> +
> +    return fdt_property(fdt, "ranges", regs, sizeof(regs));
> +}
> +
>  static int make_root_properties(libxl__gc *gc,
>                                  const libxl_version_info *vers,
>                                  void *fdt)
> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>      return 0;
>  }
>  
> +static int make_vpci_node(libxl__gc *gc, void *fdt,
> +        const struct arch_info *ainfo,
> +        struct xc_dom_image *dom)
> +{
> +    int res;
> +    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
> +    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
> +    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
> +
> +    res = fdt_begin_node(fdt, name);
> +    if (res) return res;
> +
> +    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
> +    if (res) return res;
> +
> +    res = fdt_property_string(fdt, "device_type", "pci");
> +    if (res) return res;
> +
> +    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> +            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
> +    if (res) return res;
> +
> +    res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17);
                                                            ^ space


> +    if (res) return res;
> +
> +    res = fdt_property_cell(fdt, "#address-cells", 3);
> +    if (res) return res;
> +
> +    res = fdt_property_cell(fdt, "#size-cells", 2);
> +    if (res) return res;
> +
> +    res = fdt_property_string(fdt, "status", "okay");
> +    if (res) return res;
> +
> +    res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
> +        GUEST_ROOT_SIZE_CELLS, 2,
> +        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE,
> +        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR,
> +        GUEST_VPCI_PREFETCH_MEM_SIZE);
> +    if (res) return res;
> +
> +    res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS,
> +                              0, 0x10000);
> +    if (res) return res;

I agree with Julien that we shouldn't add it now if it is not working.

One question: what about legacy interrupts? If they are supported,
shouldn't we have interrupts and interrupt-parent properties?


> +    res = fdt_end_node(fdt);
> +    if (res) return res;
> +
> +    return 0;
> +}
> +
>  static const struct arch_info *get_arch_info(libxl__gc *gc,
>                                               const struct xc_dom_image *dom)
>  {
> @@ -971,6 +1074,9 @@ next_resize:
>          if (info->tee == LIBXL_TEE_TYPE_OPTEE)
>              FDT( make_optee_node(gc, fdt) );
>  
> +        if (libxl_defbool_val(info->arch_arm.vpci))
> +            FDT( make_vpci_node(gc, fdt, ainfo, dom) );
> +
>          if (pfdt)
>              FDT( copy_partial_fdt(gc, fdt, pfdt) );
>  
> @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>      /* ACPI is disabled by default */
>      libxl_defbool_setdefault(&b_info->acpi, false);
>  
> +    /* VPCI is disabled by default */
> +    libxl_defbool_setdefault(&b_info->arch_arm.vpci, false);
> +
>      if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
>          return;
>  
> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
> index 3f9fff653a..78b1ddf0b8 100644
> --- a/tools/libs/light/libxl_types.idl
> +++ b/tools/libs/light/libxl_types.idl
> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>  
>      ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>                                 ("vuart", libxl_vuart_type),
> +                               ("vpci", libxl_defbool),
>                                ])),
>      ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
>                                ])),
> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
> index 17dddb4cd5..ffafbeffb4 100644
> --- a/tools/xl/xl_parse.c
> +++ b/tools/xl/xl_parse.c
> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source,
>          }
>          if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
>              libxl_defbool_set(&b_info->u.pv.e820_host, true);
> +        if (d_config->num_pcidevs)
> +            libxl_defbool_set(&b_info->arch_arm.vpci, true);
>      }
>  
>      if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) {
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 0a9749e768..01d13e669e 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t;
>  #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
>  #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
>  
> +/* PCI-PCIe memory space types */
> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
> +#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
> +
> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
> +#define GUEST_VPCI_PREFETCH_MEM_ADDR  xen_mk_ullong(0x900000000)
> +#define GUEST_VPCI_MEM_ADDR           xen_mk_ullong(0x20000000)
> +
> +#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x2000000000)
> +#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x04000000)

Now I get it: GUEST_VPCI_ECAM_BASE was just for the config space, not
the aperture. The apertures are starting at 0x20000000 and 0x900000000.

Speaking of which, isn't GUEST_VPCI_MEM_SIZE a bit too small? There
could be PCI devices with a <4GB requirement. Maybe 256MB?


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 14/14] xen/arm: Add linux,pci-domain property for hwdom if not available.
  2021-08-19 12:02 ` [PATCH v1 14/14] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
@ 2021-09-10  1:00   ` Stefano Stabellini
  2021-09-16 16:36     ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-10  1:00 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, bertrand.marquis, Stefano Stabellini, Julien Grall,
	Volodymyr Babchuk

On Thu, 19 Aug 2021, Rahul Singh wrote:
> If the property is not present in the device tree node for host bridge,
> XEN while creating the dtb for hwdom will create this property and
> assigns the already allocated segment to the host bridge
> so that XEN and linux will have the same segment for the host bridges.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> ---
>  xen/arch/arm/domain_build.c        | 18 ++++++++++++++++++
>  xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++
>  xen/include/asm-arm/pci.h          |  3 +++
>  3 files changed, 42 insertions(+)
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 6c86d52781..e0cf2ff19d 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -581,6 +581,24 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
>              return res;
>      }
>  
> +#ifdef CONFIG_HAS_PCI
> +    if ( dt_device_type_is_equal(node, "pci") )
> +    {
> +        if ( !dt_find_property(node, "linux,pci-domain", NULL) )
> +        {
> +            uint16_t segment;
> +
> +            res = pci_get_host_bridge_segment(node, &segment);
> +            if ( res < 0 )
> +                return res;
> +
> +            res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment);
> +            if ( res )
> +                return res;
> +        }
> +    }
> +#endif

If param_pci_enable is false it might be possible that Xen didn't
allocate a segment. In that case, we should just let Linux do whatever
it wants in terms of segment allocation. So I think the code here should
not return error if param_pci_enable is false.
returning an error instead.


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI
  2021-09-09  1:11         ` Stefano Stabellini
@ 2021-09-10 10:22           ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-10 10:22 UTC (permalink / raw)
  To: Stefano Stabellini; +Cc: Jan Beulich, Bertrand Marquis, Paul Durrant, xen-devel

Hi Stefano,

> On 9 Sep 2021, at 2:11 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Fri, 20 Aug 2021, Jan Beulich wrote:
>> On 20.08.2021 13:41, Rahul Singh wrote:
>>> Hi Jan
>>> 
>>>> On 20 Aug 2021, at 8:06 am, Jan Beulich <jbeulich@suse.com> wrote:
>>>> 
>>>> On 19.08.2021 14:02, Rahul Singh wrote:
>>>>> Compilation error is observed when ACPI and HAS_PCI is enabled for ARM
>>>>> architecture. Move the code under CONFIG_X86 flag to gate the code for
>>>>> ARM.
>>>> 
>>>> Please give at least one example of what it is that fails to compile.
>>>> What an appropriate action is depends on the nature of the error(s),
>>>> and from looking at the enclosed code I cannot easily see what it
>>>> might be that breaks for Arm.
>>> 
>>> I am observing below error when enabled ACPI && HAS_PCI for ARM.
>>> 
>>> prelink.o: In function `pcie_aer_get_firmware_first’:
>>> /xen/drivers/passthrough/pci.c:1251: undefined reference to `apei_hest_parse'
>>> aarch64-linux-gnu-ld: /home/rahsin01/work/xen/pci-passthrough-upstream/xen/xen/.xen-syms.0: hidden symbol `apei_hest_parse' isn't defined
>>> 
>>> I found that apei/ is only enabled for x86 and pcie_aer_get_firmware_first() is only called from x86 code.
>>> obj-$(CONFIG_X86) += apei/
>>> 
>>> I am not sure whether we need this code for ARM architecture 
>>> that is why I gate the code for ARM via CONFIG_X86
>> 
>> So you Arm folks will probably want to settle on that aspect first. What
>> is wanted to keep things building depends on that.
> 
> Reading the APEI description, it looks like there might be some use for
> it on ARM but it would work a bit differently from x86 as there are no
> NMIs on ARM. So enabling APEI on ARM is not just a matter of enabling
> the build of apei/, it is not going to be straightforward.
> 
> For the scope of this series (which is actually about PCI), I would
> leave it alone, and keep apei/ x86 only, which means #ifdefing
> pcie_aer_get_firmware_first.
> 
> I would just add an in-code comment saying "APEI not supported on ARM
> yet”.
 
Ok. I will add the comment in next version.

> Another option would be to introduce a symbol like HAS_ACPI_APEI
> but it is a bit overkill for this.

Regards,
Rahul

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver.
  2021-09-09  1:16   ` Stefano Stabellini
@ 2021-09-10 10:32     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-10 10:32 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 9 Sep 2021, at 2:16 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> pci_init(..) will be called during xen startup to initialize and probe
>> the PCI host-bridge driver.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/pci/pci.c       | 54 ++++++++++++++++++++++++++++++++++++
>> xen/include/asm-arm/device.h |  1 +
>> 2 files changed, 55 insertions(+)
>> 
>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>> index dc55d23778..d1c9cf997d 100644
>> --- a/xen/arch/arm/pci/pci.c
>> +++ b/xen/arch/arm/pci/pci.c
>> @@ -14,13 +14,67 @@
>>  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>  */
>> 
>> +#include <xen/acpi.h>
>> +#include <xen/device_tree.h>
>> +#include <xen/errno.h>
>> +#include <xen/init.h>
>> #include <xen/pci.h>
>> +#include <xen/param.h>
>> 
>> int arch_pci_clean_pirqs(struct domain *d)
>> {
>>     return 0;
>> }
>> 
>> +static int __init dt_pci_init(void)
>> +{
>> +    struct dt_device_node *np;
>> +    int rc;
>> +
>> +    dt_for_each_device_node(dt_host, np)
>> +    {
>> +        rc = device_init(np, DEVICE_PCI, NULL);
>> +        if( !rc )
>> +            continue;
>> +        /*
>> +         * Ignore the following error codes:
>> +         *   - EBADF: Indicate the current is not an pci
>                                                     ^ a
> 
>> +         *   - ENODEV: The pci device is not present or cannot be used by
>> +         *     Xen.
>> +         */
>> +        else if ( rc != -EBADF && rc != -ENODEV )
>> +        {
>> +            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
>> +            return rc;
>> +        }
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +#ifdef CONFIG_ACPI
>> +static void __init acpi_pci_init(void)
> 
> If the DT init function returns int, then it would make sense for the
> ACPI init function to return int as well?

Ok. I will modify acpi_pci_init(..)  function to return int.

Regards,
Rahul


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver.
  2021-09-07  8:20   ` Julien Grall
@ 2021-09-10 10:47     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-10 10:47 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Julien,

> On 7 Sep 2021, at 9:20 am, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> pci_init(..) will be called during xen startup to initialize and probe
>> the PCI host-bridge driver.
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>  xen/arch/arm/pci/pci.c       | 54 ++++++++++++++++++++++++++++++++++++
>>  xen/include/asm-arm/device.h |  1 +
>>  2 files changed, 55 insertions(+)
>> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
>> index dc55d23778..d1c9cf997d 100644
>> --- a/xen/arch/arm/pci/pci.c
>> +++ b/xen/arch/arm/pci/pci.c
>> @@ -14,13 +14,67 @@
>>   * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>   */
>>  +#include <xen/acpi.h>
>> +#include <xen/device_tree.h>
>> +#include <xen/errno.h>
>> +#include <xen/init.h>
>>  #include <xen/pci.h>
>> +#include <xen/param.h>
> 
> This include doesn't look to be necessary (yet?).
 
Yes you are right this is not necessary .I will remove "param.h"
> 
>>    int arch_pci_clean_pirqs(struct domain *d)
>>  {
>>      return 0;
>>  }
>>  +static int __init dt_pci_init(void)
>> +{
>> +    struct dt_device_node *np;
>> +    int rc;
>> +
>> +    dt_for_each_device_node(dt_host, np)
>> +    {
>> +        rc = device_init(np, DEVICE_PCI, NULL);
>> +        if( !rc )
>> +            continue;
>> +        /*
>> +         * Ignore the following error codes:
>> +         *   - EBADF: Indicate the current is not an pci
>> +         *   - ENODEV: The pci device is not present or cannot be used by
>> +         *     Xen.
>> +         */
>> +        else if ( rc != -EBADF && rc != -ENODEV )
>> +        {
>> +            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
>> +            return rc;
>> +        }
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +#ifdef CONFIG_ACPI
>> +static void __init acpi_pci_init(void)
>> +{
>> +    printk(XENLOG_ERR "ACPI pci init not supported \n");
>> +    return;
>> +}
>> +#else
>> +static inline void __init acpi_pci_init(void) { }
>> +#endif
>> +
>> +static int __init pci_init(void)
>> +{
>> +    if ( acpi_disabled )
>> +        dt_pci_init();
>> +    else
>> +        acpi_pci_init();
>> +
>> +    pci_segments_init();
> 
> Shouldn't this happen before the PCI initialization?

Calling pci_segment_init(..) before or after PCI initialization will not make any 
difference as this is independent call. Anyway I will move the pci_segment_init(..)  
before PCI initialization.

Regards,
Rahul

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-09-07  9:05   ` Julien Grall
@ 2021-09-10 11:22     ` Rahul Singh
  2021-09-10 11:53       ` Julien Grall
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-09-10 11:22 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Julien

> On 7 Sep 2021, at 10:05 am, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> XEN during boot will read the PCI device tree node “reg” property
>> and will map the PCI config space to the XEN memory.
>> As of now "pci-host-ecam-generic" compatible board is supported.
> 
> I think the word "only" is missing.
Ok. 
> 
>> "linux,pci-domain" device tree property assigns a fixed PCI domain
>> number to a host bridge, otherwise an unstable (across boots) unique
>> number will be assigned by Linux.This property has to be in sync with
> 
> Typo: missing space after the ‘.’
Ack.
>> XEN to access the PCI devices.
> 
> I would expand a little bit the last sentence to explain why the need to be sync-ed.

I will explain why segment and domain need to be in sync in next version.
> 
>> > XEN will read the “linux,pci-domain” property from the device tree node
>> and configure the host bridge segment number accordingly. If this
>> property is not available XEN will allocate the unique segment number
>> to the host bridge.
>> dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly
>> imported from the Linux source tree.
> 
> What was the Linux commit used? I also read "directly imported" as a verbartim copy but AFAICT the implementation has been slightly reworked.

I will add the Linux commit used in commit msg and also will add that code is slightly modified.
> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>  xen/arch/arm/pci/Makefile           |   2 +
>>  xen/arch/arm/pci/pci-host-common.c  | 261 ++++++++++++++++++++++++++++
>>  xen/arch/arm/pci/pci-host-generic.c |  55 ++++++
>>  xen/include/asm-arm/pci.h           |  28 +++
>>  4 files changed, 346 insertions(+)
>>  create mode 100644 xen/arch/arm/pci/pci-host-common.c
>>  create mode 100644 xen/arch/arm/pci/pci-host-generic.c
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> index a9ee0b9b44..f3d97f859e 100644
>> --- a/xen/arch/arm/pci/Makefile
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -1,2 +1,4 @@
>>  obj-y += pci.o
>>  obj-y += pci-access.o
>> +obj-y += pci-host-generic.o
>> +obj-y += pci-host-common.o
>> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
>> new file mode 100644
>> index 0000000000..9dd9b02271
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-common.c
>> @@ -0,0 +1,261 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
> 
> AFAICT, <xen/pci.h> already includes <asm/pci.h>. So this looks unneccessary.
Ack.
> 
>> +#include <xen/rwlock.h>
>> +#include <xen/sched.h>
>> +#include <xen/vmap.h>
>> +
>> +/*
>> + * List for all the pci host bridges.
>> + */
>> +
>> +static LIST_HEAD(pci_host_bridges);
>> +
>> +static atomic_t domain_nr = ATOMIC_INIT(-1);
>> +
>> +bool dt_pci_parse_bus_range(struct dt_device_node *dev,
>> +                            struct pci_config_window *cfg)
> 
> Aside, "pci_config_window", the function is not Arm specific. Would it be possible to consider to introduce "struct resource" in Xen so this function can be moved in common/device_tree.c?

I can introduce the "struct resource” but I am not sure whether "struct resource” will be
useful later point in time. What I prefer as of now, we can have this function and we can
move this to common/device_tree.c once we have the requirement for "struct resource”.
> 
>> +{
>> +    const __be32 *cells;
>> +    uint32_t len;
>> +
>> +    cells = dt_get_property(dev, "bus-range", &len);
>> +    /* bus-range should at least be 2 cells */
>> +    if ( !cells || (len < (sizeof(*cells) * 2)) )
>> +        return false;
> 
> How about introducing dt_property_read_u32_array()?

Ok. I will introduce dt_property_read_u32_array().

> 
>> +
>> +    cfg->busn_start = dt_next_cell(1, &cells);
>> +    cfg->busn_end = dt_next_cell(1, &cells);
>> +
>> +    return true;
>> +}
>> +
>> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
>> +{
>> +    return ioremap_nocache(start, len);
>> +}
>> +
>> +static void pci_ecam_free(struct pci_config_window *cfg)
>> +{
>> +    if ( cfg->win )
>> +        iounmap(cfg->win);
>> +
>> +    xfree(cfg);
>> +}
>> +
>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>> +                                              int ecam_reg_idx)
>> +{
>> +    int err;
>> +    struct pci_config_window *cfg;
>> +    paddr_t addr, size;
>> +
>> +    cfg = xzalloc(struct pci_config_window);
>> +    if ( !cfg )
>> +        return NULL;
>> +
>> +    err = dt_pci_parse_bus_range(dev, cfg);
>> +    if ( !err ) {
>> +        cfg->busn_start = 0;
>> +        cfg->busn_end = 0xff;
>> +        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
> 
> Typo: Missing space after ':’.
Ack.
> 
>> +               dt_node_full_name(dev));
>> +    } else {
>> +        if ( cfg->busn_end > cfg->busn_start + 0xff )
>> +            cfg->busn_end = cfg->busn_start + 0xff;
>> +    }
>> +
>> +    /* Parse our PCI ecam register address*/
>> +    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
>> +    if ( err )
>> +        goto err_exit;
>> +
>> +    cfg->phys_addr = addr;
>> +    cfg->size = size;
>> +
>> +    /*
>> +     * On 64-bit systems, we do a single ioremap for the whole config space
>> +     * since we have enough virtual address range available.  On 32-bit, we
> 
> In Xen on Arm64, the VMAP is actually only 1GB. So it is not that big and this will compete with other mapping like ITS, global domain mapping...
> 
> So I think the vMAP area will need to increase to cater to increase usage.

Let me check on this and come back to you.

> 
>> +     * ioremap the config space for each bus individually.
>> +     *
>> +     * As of now only 64-bit is supported 32-bit is not supported.
>> +     */
>> +    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
>> +    if ( !cfg->win )
>> +        goto err_exit_remap;
>> +
>> +    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
> 
> The physical address is a paddr_t. So this needs to use PRIpaddr. Also, please use preprent hexadecimal with 0x. This makes a lot easier to differentiate hexa vs decimal in the log.
Ack.
> 
>> +            cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
>> +
>> +    return cfg;
>> +
>> +err_exit_remap:
>> +    printk(XENLOG_ERR "ECAM ioremap failed\n");
>> +err_exit:
>> +    pci_ecam_free(cfg);
> 
> Coding style: Please add a new line before return.
Ack.
> 
>> +    return NULL;
>> +}
>> +
>> +struct pci_host_bridge *pci_alloc_host_bridge(void)
>> +{
>> +    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
>> +
>> +    if ( !bridge )
>> +        return NULL;
>> +
>> +    INIT_LIST_HEAD(&bridge->node);
>> +    bridge->bus_start = ~0;
>> +    bridge->bus_end = ~0;
> 
> Coding style: Please add a new line before return.
Ack.
> 
> 
>> +    return bridge;
>> +}
>> +
>> +void pci_add_host_bridge(struct pci_host_bridge *bridge)
>> +{
>> +    list_add_tail(&bridge->node, &pci_host_bridges);
>> +}
>> +
>> +static int pci_get_new_domain_nr(void)
>> +{
>> +    return atomic_inc_return(&domain_nr);
>> +}
>> +
>> +/*
>> + * This function will try to obtain the host bridge domain number by
>> + * finding a property called "linux,pci-domain" of the given device node.
>> + *
>> + * @node: device tree node with the domain information
>> + *
>> + * Returns the associated domain number from DT in the range [0-0xffff], or
>> + * a negative value if the required property is not found.
>> + */
>> +static int dt_get_pci_domain_nr(struct dt_device_node *node)
> 
> Nothing looks Arm specific for this function. Can you move it in device_tree.c?
Ack.
> 
>> +{
>> +    u32 domain;
>> +    int error;
>> +
>> +    error = dt_property_read_u32(node, "linux,pci-domain", &domain);
>> +    if ( !error )
>> +        return -EINVAL;
>> +
>> +    return (u16)domain;
>> +}
>> +
>> +static int pci_bus_find_domain_nr(struct dt_device_node *dev)
>> +{
>> +    static int use_dt_domains = -1;
>> +    int domain;
>> +
>> +    domain = dt_get_pci_domain_nr(dev);
>> +
>> +    /*
>> +     * Check DT domain and use_dt_domains values.
>> +     *
>> +     * If DT domain property is valid (domain >= 0) and
>> +     * use_dt_domains != 0, the DT assignment is valid since this means
>> +     * we have not previously allocated a domain number by using
>> +     * pci_get_new_domain_nr(); we should also update use_dt_domains to
>> +     * 1, to indicate that we have just assigned a domain number from
>> +     * DT.
>> +     *
>> +     * If DT domain property value is not valid (ie domain < 0), and we
>> +     * have not previously assigned a domain number from DT
>> +     * (use_dt_domains != 1) we should assign a domain number by
>> +     * using the:
>> +     *
>> +     * pci_get_new_domain_nr()
>> +     *
>> +     * API and update the use_dt_domains value to keep track of method we
>> +     * are using to assign domain numbers (use_dt_domains = 0).
>> +     *
>> +     * All other combinations imply we have a platform that is trying
>> +     * to mix domain numbers obtained from DT and pci_get_new_domain_nr(),
>> +     * which is a recipe for domain mishandling and it is prevented by
>> +     * invalidating the domain value (domain = -1) and printing a
>> +     * corresponding error.
>> +     */
>> +    if ( domain >= 0 && use_dt_domains )
>> +    {
>> +        use_dt_domains = 1;
>> +    }
>> +    else if ( domain < 0 && use_dt_domains != 1 )
>> +    {
>> +        use_dt_domains = 0;
>> +        domain = pci_get_new_domain_nr();
>> +    }
>> +    else
>> +    {
>> +        printk(XENLOG_ERR "Inconsistent \"linux,pci-domain\" property in DT\n");
>> +        BUG();
> 
> I don't think pci_bus_find_domain_nr() should be the function that crashes Xen. Instead, this should be propagated to the highest possible callers and let it decide what to do.

Ack. 
> 
>> +    }
>> +
>> +    return domain;
>> +}
>> +
>> +int pci_host_common_probe(struct dt_device_node *dev,
>> +                          int ecam_reg_idx)
>> +{
>> +    struct pci_host_bridge *bridge;
>> +    struct pci_config_window *cfg;
>> +    int err;
>> +
>> +    bridge = pci_alloc_host_bridge();
>> +    if ( !bridge )
>> +        return -ENOMEM;
>> +
>> +    /* Parse and map our Configuration Space windows */
>> +    cfg = gen_pci_init(dev, ecam_reg_idx);
>> +    if ( !cfg )
>> +    {
>> +        err = -ENOMEM;
>> +        goto err_exit;
>> +    }
>> +
>> +    bridge->dt_node = dev;
>> +    bridge->sysdata = cfg;
>> +    bridge->bus_start = cfg->busn_start;
>> +    bridge->bus_end = cfg->busn_end;
>> +
>> +    bridge->segment = pci_bus_find_domain_nr(dev);
>> +
>> +    pci_add_host_bridge(bridge);
>> +
>> +    return 0;
>> +
>> +err_exit:
>> +    xfree(bridge);
>> +    return err;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
>> new file mode 100644
>> index 0000000000..13d0f7f999
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-generic.c
>> @@ -0,0 +1,55 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/device.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +
>> +static const struct dt_device_match gen_pci_dt_match[] = {
>> +    { .compatible = "pci-host-ecam-generic" },
>> +    { },
>> +};
>> +
>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>> +{
>> +    const struct dt_device_match *of_id;
>> +
>> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> 
> This seems to be a bit pointless to me as you already know the compatible (there is only one possible...).
 As of now we are only implementing the "pci-host-ecam-generic” compatible PCI, but in future we might 
need to implement the other compatible like Linux see as below.

https://elixir.bootlin.com/linux/latest/source/drivers/pci/controller/pci-host-generic.c#L59

> 
>> +
>> +    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>> +           dt_node_full_name(dev), of_id->compatible);
>> +
>> +    return pci_host_common_probe(dev, 0);
>> +}
>> +
>> +DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
>> +.dt_match = gen_pci_dt_match,
>> +.init = gen_pci_dt_init,
>> +DT_DEVICE_END
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>> index 61e43da088..58a51e724e 100644
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -26,6 +26,34 @@ struct arch_pci_dev {
>>      struct device dev;
>>  };
>>  +/*
>> + * struct to hold the mappings of a config space window. This
>> + * is expected to be used as sysdata for PCI controllers that
>> + * use ECAM.
>> + */
>> +struct pci_config_window {
>> +    paddr_t         phys_addr;
>> +    paddr_t         size;
>> +    uint8_t         busn_start;
>> +    uint8_t         busn_end;
>> +    void __iomem    *win;
>> +};
>> +
>> +/*
>> + * struct to hold pci host bridge information
>> + * for a PCI controller.
>> + */
>> +struct pci_host_bridge {
>> +    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
>> +    struct list_head node;           /* Node in list of host bridges */
>> +    uint16_t segment;                /* Segment number */
>> +    u8 bus_start;                    /* Bus start of this bridge. */
>> +    u8 bus_end;                      /* Bus end of this bridge. */
> 
> Please use uint8_t rather than u8.
Ack. 

Regards,
Rahul


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-09-09 22:54   ` Stefano Stabellini
@ 2021-09-10 11:53     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-10 11:53 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 9 Sep 2021, at 11:54 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> XEN during boot will read the PCI device tree node “reg” property
>> and will map the PCI config space to the XEN memory.
>> 
>> As of now "pci-host-ecam-generic" compatible board is supported.
>> 
>> "linux,pci-domain" device tree property assigns a fixed PCI domain
>> number to a host bridge, otherwise an unstable (across boots) unique
>> number will be assigned by Linux.This property has to be in sync with
>> XEN to access the PCI devices.
>> 
>> XEN will read the “linux,pci-domain” property from the device tree node
>> and configure the host bridge segment number accordingly. If this
>> property is not available XEN will allocate the unique segment number
>> to the host bridge.
>> 
>> dt_get_pci_domain_nr(..) and dt_pci_bus_find_domain_nr(..) are directly
>> imported from the Linux source tree.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/pci/Makefile           |   2 +
>> xen/arch/arm/pci/pci-host-common.c  | 261 ++++++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-host-generic.c |  55 ++++++
>> xen/include/asm-arm/pci.h           |  28 +++
>> 4 files changed, 346 insertions(+)
>> create mode 100644 xen/arch/arm/pci/pci-host-common.c
>> create mode 100644 xen/arch/arm/pci/pci-host-generic.c
>> 
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> index a9ee0b9b44..f3d97f859e 100644
>> --- a/xen/arch/arm/pci/Makefile
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -1,2 +1,4 @@
>> obj-y += pci.o
>> obj-y += pci-access.o
>> +obj-y += pci-host-generic.o
>> +obj-y += pci-host-common.o
>> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
>> new file mode 100644
>> index 0000000000..9dd9b02271
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-common.c
>> @@ -0,0 +1,261 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom.
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> + *
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +#include <xen/rwlock.h>
>> +#include <xen/sched.h>
>> +#include <xen/vmap.h>
>> +
>> +/*
>> + * List for all the pci host bridges.
>> + */
>> +
>> +static LIST_HEAD(pci_host_bridges);
>> +
>> +static atomic_t domain_nr = ATOMIC_INIT(-1);
>> +
>> +bool dt_pci_parse_bus_range(struct dt_device_node *dev,
>> +                            struct pci_config_window *cfg)
>> +{
>> +    const __be32 *cells;
>> +    uint32_t len;
>> +
>> +    cells = dt_get_property(dev, "bus-range", &len);
>> +    /* bus-range should at least be 2 cells */
>> +    if ( !cells || (len < (sizeof(*cells) * 2)) )
>> +        return false;
>> +
>> +    cfg->busn_start = dt_next_cell(1, &cells);
>> +    cfg->busn_end = dt_next_cell(1, &cells);
>> +
>> +    return true;
>> +}
>> +
>> +static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
>> +{
>> +    return ioremap_nocache(start, len);
>> +}
>> +
>> +static void pci_ecam_free(struct pci_config_window *cfg)
>> +{
>> +    if ( cfg->win )
>> +        iounmap(cfg->win);
>> +
>> +    xfree(cfg);
>> +}
>> +
>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>> +                                              int ecam_reg_idx)
> 
> If it is only called at init time, then the function should be __init

Ack.

> 
> 
>> +{
>> +    int err;
>> +    struct pci_config_window *cfg;
>> +    paddr_t addr, size;
>> +
>> +    cfg = xzalloc(struct pci_config_window);
>> +    if ( !cfg )
>> +        return NULL;
>> +
>> +    err = dt_pci_parse_bus_range(dev, cfg);
>> +    if ( !err ) {
>> +        cfg->busn_start = 0;
>> +        cfg->busn_end = 0xff;
>> +        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
>> +               dt_node_full_name(dev));
>> +    } else {
>> +        if ( cfg->busn_end > cfg->busn_start + 0xff )
>> +            cfg->busn_end = cfg->busn_start + 0xff;
> 
> Is this a hard limit in the specification? Or is it a limit in the Xen
> implementation?

I just take the reference from Linux code.
https://elixir.bootlin.com/linux/latest/source/drivers/pci/of.c#L306

> 
> 
>> +    }
>> +
>> +    /* Parse our PCI ecam register address*/
>                                            ^ space
> 
Ack.
>> +    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
>> +    if ( err )
>> +        goto err_exit;
>> +
>> +    cfg->phys_addr = addr;
>> +    cfg->size = size;
>> +
>> +    /*
>> +     * On 64-bit systems, we do a single ioremap for the whole config space
>> +     * since we have enough virtual address range available.  On 32-bit, we
>> +     * ioremap the config space for each bus individually.
>> +     *
>> +     * As of now only 64-bit is supported 32-bit is not supported.
>> +     */
>> +    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
>> +    if ( !cfg->win )
>> +        goto err_exit_remap;
>> +
>> +    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>> +            cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
>> +
>> +    return cfg;
>> +
>> +err_exit_remap:
>> +    printk(XENLOG_ERR "ECAM ioremap failed\n");
>> +err_exit:
>> +    pci_ecam_free(cfg);
>> +    return NULL;
>> +}
>> +
>> +struct pci_host_bridge *pci_alloc_host_bridge(void)
>> +{
>> +    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
>> +
>> +    if ( !bridge )
>> +        return NULL;
>> +
>> +    INIT_LIST_HEAD(&bridge->node);
>> +    bridge->bus_start = ~0;
>> +    bridge->bus_end = ~0;
> 
> Please use INVALID_PADDR instead of ~0

bridge->bus_start is if type uint8_t if I  use INVALID_PADDR I will get overflow error.
I will use U8_MAX instead of INVALID_PADDR.

> 
> 
>> +    return bridge;
>> +}
>> +
>> +void pci_add_host_bridge(struct pci_host_bridge *bridge)
>> +{
>> +    list_add_tail(&bridge->node, &pci_host_bridges);
>> +}
>> +
>> +static int pci_get_new_domain_nr(void)
>> +{
>> +    return atomic_inc_return(&domain_nr);
>> +}
>> +
>> +/*
>> + * This function will try to obtain the host bridge domain number by
>> + * finding a property called "linux,pci-domain" of the given device node.
>> + *
>> + * @node: device tree node with the domain information
>> + *
>> + * Returns the associated domain number from DT in the range [0-0xffff], or
>> + * a negative value if the required property is not found.
>> + */
>> +static int dt_get_pci_domain_nr(struct dt_device_node *node)
>> +{
>> +    u32 domain;
>> +    int error;
>> +
>> +    error = dt_property_read_u32(node, "linux,pci-domain", &domain);
>> +    if ( !error )
>> +        return -EINVAL;
>> +
>> +    return (u16)domain;
> 
> Let's check that domain <= UINT16_MAX
> 
Ack
 
Regards,
Rahul

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-09-10 11:22     ` Rahul Singh
@ 2021-09-10 11:53       ` Julien Grall
  0 siblings, 0 replies; 98+ messages in thread
From: Julien Grall @ 2021-09-10 11:53 UTC (permalink / raw)
  To: Rahul Singh
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk



On 10/09/2021 12:22, Rahul Singh wrote:
> Hi Julien

Hi Rahul,
>> On 7 Sep 2021, at 10:05 am, Julien Grall <julien@xen.org> wrote:
>> On 19/08/2021 13:02, Rahul Singh wrote:
>>> +#include <xen/rwlock.h>
>>> +#include <xen/sched.h>
>>> +#include <xen/vmap.h>
>>> +
>>> +/*
>>> + * List for all the pci host bridges.
>>> + */
>>> +
>>> +static LIST_HEAD(pci_host_bridges);
>>> +
>>> +static atomic_t domain_nr = ATOMIC_INIT(-1);
>>> +
>>> +bool dt_pci_parse_bus_range(struct dt_device_node *dev,
>>> +                            struct pci_config_window *cfg)
>>
>> Aside, "pci_config_window", the function is not Arm specific. Would it be possible to consider to introduce "struct resource" in Xen so this function can be moved in common/device_tree.c?
> 
> I can introduce the "struct resource” but I am not sure whether "struct resource” will be
> useful later point in time. What I prefer as of now, we can have this function and we can
> move this to common/device_tree.c once we have the requirement for "struct resource”.

TBH, I wasn't asking about using "struct resource". This was simply a 
suggestion how to make the code common.

What I am more interested is moving this function in common code so it 
can be re-used by RISC-v in the future. I don't particularly mind 
whether this is using "struct resource" or a different structure.

>>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
>>> new file mode 100644
>>> index 0000000000..13d0f7f999
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/pci-host-generic.c
>>> @@ -0,0 +1,55 @@
>>> +/*
>>> + * Copyright (C) 2021 Arm Ltd.
>>> + *
>>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <asm/device.h>
>>> +#include <xen/pci.h>
>>> +#include <asm/pci.h>
>>> +
>>> +static const struct dt_device_match gen_pci_dt_match[] = {
>>> +    { .compatible = "pci-host-ecam-generic" },
>>> +    { },
>>> +};
>>> +
>>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>>> +{
>>> +    const struct dt_device_match *of_id;
>>> +
>>> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>>
>> This seems to be a bit pointless to me as you already know the compatible (there is only one possible...).
>   As of now we are only implementing the "pci-host-ecam-generic” compatible PCI, but in future we might
> need to implement the other compatible like Linux see as below.
> 
> https://elixir.bootlin.com/linux/latest/source/drivers/pci/controller/pci-host-generic.c#L59

Right... You know the node matched and you don't need any data. So this 
is a bit pointless to have to walk again the structure for just printing 
the compatible that matched.

Cheers,

-- 
Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller
  2021-09-09 23:34   ` Stefano Stabellini
@ 2021-09-10 12:01     ` Rahul Singh
  2021-09-13 14:46       ` Oleksandr Andrushchenko
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-09-10 12:01 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Oleksandr Andrushchenko,
	Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>> 
>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config
>> space to the XEN memory.
>> 
>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>> ---
>> xen/arch/arm/pci/Makefile          |  1 +
>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++
>> 2 files changed, 60 insertions(+)
>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
>> 
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> index 6f32fbbe67..1d045ade01 100644
>> --- a/xen/arch/arm/pci/Makefile
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -3,3 +3,4 @@ obj-y += pci-access.o
>> obj-y += pci-host-generic.o
>> obj-y += pci-host-common.o
>> obj-y += ecam.o
>> +obj-y += pci-host-zynqmp.o
>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
>> new file mode 100644
>> index 0000000000..fe103e3855
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c
>> @@ -0,0 +1,59 @@
>> +/*
>> + * Copyright (C) 2020-2021 EPAM Systems
>> + *
>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>> + * Based on xen/arch/arm/pci/pci-host-generic.c
>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> 
> Only one Copyright line per file is enough :-)
> 
> But actually all the Copyright lines with a name or a company name are
> not really required or useful, as the copyright is noted in full details
> in the commit messages (author and signed-off-by lines). I would remove
> them all from the new files added by this series.

Ok. Let me remove in next version.
> 
> 
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/device.h>
>> +#include <xen/pci.h>
>> +#include <asm/pci.h>
>> +
>> +static const struct dt_device_match gen_pci_dt_match[] = {
>> +    { .compatible = "xlnx,nwl-pcie-2.11",
>> +      .data =       &pci_generic_ecam_ops },
>> +    { },
>> +};
>> +
>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>> +{
>> +    const struct dt_device_match *of_id;
>> +    const struct pci_ecam_ops *ops;
>> +
>> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> 
> This should be superfluous

Ack. I will remove the dt_match_node(..) in next version.

Regards,
Rahul



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller
  2021-09-10 12:01     ` Rahul Singh
@ 2021-09-13 14:46       ` Oleksandr Andrushchenko
  2021-09-13 21:02         ` Stefano Stabellini
  0 siblings, 1 reply; 98+ messages in thread
From: Oleksandr Andrushchenko @ 2021-09-13 14:46 UTC (permalink / raw)
  To: Rahul Singh, Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Oleksandr Andrushchenko,
	Julien Grall, Volodymyr Babchuk


On 10.09.21 15:01, Rahul Singh wrote:
> Hi Stefano,
>
>> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>
>> On Thu, 19 Aug 2021, Rahul Singh wrote:
>>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>>>
>>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config
>>> space to the XEN memory.
>>>
>>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>>> ---
>>> xen/arch/arm/pci/Makefile          |  1 +
>>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++
>>> 2 files changed, 60 insertions(+)
>>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
>>>
>>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>>> index 6f32fbbe67..1d045ade01 100644
>>> --- a/xen/arch/arm/pci/Makefile
>>> +++ b/xen/arch/arm/pci/Makefile
>>> @@ -3,3 +3,4 @@ obj-y += pci-access.o
>>> obj-y += pci-host-generic.o
>>> obj-y += pci-host-common.o
>>> obj-y += ecam.o
>>> +obj-y += pci-host-zynqmp.o
>>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
>>> new file mode 100644
>>> index 0000000000..fe103e3855
>>> --- /dev/null
>>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c
>>> @@ -0,0 +1,59 @@
>>> +/*
>>> + * Copyright (C) 2020-2021 EPAM Systems
>>> + *
>>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>>> + * Based on xen/arch/arm/pci/pci-host-generic.c
>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>> Only one Copyright line per file is enough :-)
>>
>> But actually all the Copyright lines with a name or a company name are
>> not really required or useful, as the copyright is noted in full details
>> in the commit messages (author and signed-off-by lines). I would remove
>> them all from the new files added by this series.
> Ok. Let me remove in next version.
>>
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <asm/device.h>
>>> +#include <xen/pci.h>
>>> +#include <asm/pci.h>
>>> +
>>> +static const struct dt_device_match gen_pci_dt_match[] = {
>>> +    { .compatible = "xlnx,nwl-pcie-2.11",
>>> +      .data =       &pci_generic_ecam_ops },
>>> +    { },
>>> +};
>>> +
>>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>>> +{
>>> +    const struct dt_device_match *of_id;
>>> +    const struct pci_ecam_ops *ops;
>>> +
>>> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>> This should be superfluous
> Ack. I will remove the dt_match_node(..) in next version.

I am not entirely sure we need this patch at all as the main reason for its existence

was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going

to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP.

So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one

won't have it working...

Please consider

If we decide to remove it then

int pci_host_common_probe(struct dt_device_node *dev,
                           const struct pci_ecam_ops *ops,
                           int ecam_reg_idx)

doesn't need the last parameter.

>
> Regards,
> Rahul
>
>
Thanks,

Oleksandr

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-08-19 12:02 ` [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
  2021-09-07  9:05   ` Julien Grall
  2021-09-09 22:54   ` Stefano Stabellini
@ 2021-09-13 14:52   ` Oleksandr Andrushchenko
  2021-09-13 20:23     ` Stefano Stabellini
  2021-09-14  7:53   ` Oleksandr Andrushchenko
  3 siblings, 1 reply; 98+ messages in thread
From: Oleksandr Andrushchenko @ 2021-09-13 14:52 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: Bertrand Marquis, Stefano Stabellini, Julien Grall, Volodymyr Babchuk

Hi, Rahul!

On 19.08.21 15:02, Rahul Singh wrote:
> XEN during boot will read the PCI device tree node “reg” property
> and will map the PCI config space to the XEN memory.
[snip]
> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> +                                              int ecam_reg_idx)
> +{
> +    int err;
> +    struct pci_config_window *cfg;
> +    paddr_t addr, size;
> +
> +    cfg = xzalloc(struct pci_config_window);
> +    if ( !cfg )
> +        return NULL;
> +
> +    err = dt_pci_parse_bus_range(dev, cfg);
> +    if ( !err ) {
> +        cfg->busn_start = 0;
> +        cfg->busn_end = 0xff;
> +        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
> +               dt_node_full_name(dev));
> +    } else {
> +        if ( cfg->busn_end > cfg->busn_start + 0xff )
> +            cfg->busn_end = cfg->busn_start + 0xff;
> +    }
> +
> +    /* Parse our PCI ecam register address*/
> +    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);

I am a bit worried here that we don't get the reg index from the device tree,

but for generic ECAM we use reg[0] and for Xilinx we use reg[2].

For example, for Xilinx we have

reg = <0x00 0xfd0e0000 0x00 0x1000 0x00 0xfd480000 0x00 0x1000 0x80 0x00 0x00 0x1000000>;
reg-names = "breg\0pcireg\0cfg";

so, we can parse the reg-names and understand that the configuration space is the last in the reg property.

The same I think can be done for other device trees probably.

Rahul, do you know if reg-names "cfg" is vendor specific of used widely?

Thank you,

Oleksandr

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-09-13 14:52   ` Oleksandr Andrushchenko
@ 2021-09-13 20:23     ` Stefano Stabellini
  2021-09-14  4:35       ` Oleksandr Andrushchenko
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-13 20:23 UTC (permalink / raw)
  To: Oleksandr Andrushchenko
  Cc: Rahul Singh, xen-devel, Bertrand Marquis, Stefano Stabellini,
	Julien Grall, Volodymyr Babchuk

[-- Attachment #1: Type: text/plain, Size: 1874 bytes --]

On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote:
> Hi, Rahul!
> 
> On 19.08.21 15:02, Rahul Singh wrote:
> > XEN during boot will read the PCI device tree node “reg” property
> > and will map the PCI config space to the XEN memory.
> [snip]
> > +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
> > +                                              int ecam_reg_idx)
> > +{
> > +    int err;
> > +    struct pci_config_window *cfg;
> > +    paddr_t addr, size;
> > +
> > +    cfg = xzalloc(struct pci_config_window);
> > +    if ( !cfg )
> > +        return NULL;
> > +
> > +    err = dt_pci_parse_bus_range(dev, cfg);
> > +    if ( !err ) {
> > +        cfg->busn_start = 0;
> > +        cfg->busn_end = 0xff;
> > +        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
> > +               dt_node_full_name(dev));
> > +    } else {
> > +        if ( cfg->busn_end > cfg->busn_start + 0xff )
> > +            cfg->busn_end = cfg->busn_start + 0xff;
> > +    }
> > +
> > +    /* Parse our PCI ecam register address*/
> > +    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
> 
> I am a bit worried here that we don't get the reg index from the device tree,
> 
> but for generic ECAM we use reg[0] and for Xilinx we use reg[2].
> 
> For example, for Xilinx we have
> 
> reg = <0x00 0xfd0e0000 0x00 0x1000 0x00 0xfd480000 0x00 0x1000 0x80 0x00 0x00 0x1000000>;
> reg-names = "breg\0pcireg\0cfg";
> 
> so, we can parse the reg-names and understand that the configuration space is the last in the reg property.
> 
> The same I think can be done for other device trees probably.

Well spotted!


> Rahul, do you know if reg-names "cfg" is vendor specific of used widely?

Unfortunately it seems to be vendor specific :-(
Which means that "dt_device_get_address" or similar should be moved to a
vendor specific function.

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller
  2021-09-13 14:46       ` Oleksandr Andrushchenko
@ 2021-09-13 21:02         ` Stefano Stabellini
  2021-09-14  4:31           ` Oleksandr Andrushchenko
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-13 21:02 UTC (permalink / raw)
  To: Oleksandr Andrushchenko
  Cc: Rahul Singh, Stefano Stabellini, xen-devel, Bertrand Marquis,
	Julien Grall, Volodymyr Babchuk

[-- Attachment #1: Type: text/plain, Size: 4694 bytes --]

On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote:
> On 10.09.21 15:01, Rahul Singh wrote:
> > Hi Stefano,
> >
> >> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >>
> >> On Thu, 19 Aug 2021, Rahul Singh wrote:
> >>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> >>>
> >>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config
> >>> space to the XEN memory.
> >>>
> >>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> >>> ---
> >>> xen/arch/arm/pci/Makefile          |  1 +
> >>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++
> >>> 2 files changed, 60 insertions(+)
> >>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
> >>>
> >>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
> >>> index 6f32fbbe67..1d045ade01 100644
> >>> --- a/xen/arch/arm/pci/Makefile
> >>> +++ b/xen/arch/arm/pci/Makefile
> >>> @@ -3,3 +3,4 @@ obj-y += pci-access.o
> >>> obj-y += pci-host-generic.o
> >>> obj-y += pci-host-common.o
> >>> obj-y += ecam.o
> >>> +obj-y += pci-host-zynqmp.o
> >>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
> >>> new file mode 100644
> >>> index 0000000000..fe103e3855
> >>> --- /dev/null
> >>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c
> >>> @@ -0,0 +1,59 @@
> >>> +/*
> >>> + * Copyright (C) 2020-2021 EPAM Systems
> >>> + *
> >>> + * Based on Linux drivers/pci/controller/pci-host-common.c
> >>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
> >>> + * Based on xen/arch/arm/pci/pci-host-generic.c
> >>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
> >> Only one Copyright line per file is enough :-)
> >>
> >> But actually all the Copyright lines with a name or a company name are
> >> not really required or useful, as the copyright is noted in full details
> >> in the commit messages (author and signed-off-by lines). I would remove
> >> them all from the new files added by this series.
> > Ok. Let me remove in next version.
> >>
> >>> + * This program is free software; you can redistribute it and/or modify
> >>> + * it under the terms of the GNU General Public License version 2 as
> >>> + * published by the Free Software Foundation.
> >>> + *
> >>> + * This program is distributed in the hope that it will be useful,
> >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >>> + * GNU General Public License for more details.
> >>> + *
> >>> + * You should have received a copy of the GNU General Public License
> >>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> >>> + */
> >>> +
> >>> +#include <asm/device.h>
> >>> +#include <xen/pci.h>
> >>> +#include <asm/pci.h>
> >>> +
> >>> +static const struct dt_device_match gen_pci_dt_match[] = {
> >>> +    { .compatible = "xlnx,nwl-pcie-2.11",
> >>> +      .data =       &pci_generic_ecam_ops },
> >>> +    { },
> >>> +};
> >>> +
> >>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
> >>> +{
> >>> +    const struct dt_device_match *of_id;
> >>> +    const struct pci_ecam_ops *ops;
> >>> +
> >>> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
> >> This should be superfluous
> > Ack. I will remove the dt_match_node(..) in next version.
> 
> I am not entirely sure we need this patch at all as the main reason for its existence
> 
> was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going
> 
> to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP.
> 
> So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one
> 
> won't have it working...
> 
> Please consider
> 
> If we decide to remove it then
> 
> int pci_host_common_probe(struct dt_device_node *dev,
>                            const struct pci_ecam_ops *ops,
>                            int ecam_reg_idx)
> 
> doesn't need the last parameter.

With my open source maintainer hat on, I don't see this patch as very
important; from that point of view I'd be happy for it to be dropped.
However, it would be good to have at least one non-default host bridge
(doesn't have to be the Xilinx bridge), otherwise it becomes difficult
to understand how the generic infrastructure introduced by this series
could be useful.

Moreover, your recent comment [1] made it even more evident that it
would be good to have at least two different drivers to spot
compatibility issues between them more easily.

[1] https://marc.info/?l=xen-devel&m=163154474008598 

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller
  2021-09-13 21:02         ` Stefano Stabellini
@ 2021-09-14  4:31           ` Oleksandr Andrushchenko
  2021-09-17  7:39             ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Oleksandr Andrushchenko @ 2021-09-14  4:31 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Rahul Singh, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk


On 14.09.21 00:02, Stefano Stabellini wrote:
> On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote:
>> On 10.09.21 15:01, Rahul Singh wrote:
>>> Hi Stefano,
>>>
>>>> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>>
>>>> On Thu, 19 Aug 2021, Rahul Singh wrote:
>>>>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>>>>>
>>>>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config
>>>>> space to the XEN memory.
>>>>>
>>>>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>>>>> ---
>>>>> xen/arch/arm/pci/Makefile          |  1 +
>>>>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++
>>>>> 2 files changed, 60 insertions(+)
>>>>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
>>>>>
>>>>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>>>>> index 6f32fbbe67..1d045ade01 100644
>>>>> --- a/xen/arch/arm/pci/Makefile
>>>>> +++ b/xen/arch/arm/pci/Makefile
>>>>> @@ -3,3 +3,4 @@ obj-y += pci-access.o
>>>>> obj-y += pci-host-generic.o
>>>>> obj-y += pci-host-common.o
>>>>> obj-y += ecam.o
>>>>> +obj-y += pci-host-zynqmp.o
>>>>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
>>>>> new file mode 100644
>>>>> index 0000000000..fe103e3855
>>>>> --- /dev/null
>>>>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c
>>>>> @@ -0,0 +1,59 @@
>>>>> +/*
>>>>> + * Copyright (C) 2020-2021 EPAM Systems
>>>>> + *
>>>>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>>>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>>>>> + * Based on xen/arch/arm/pci/pci-host-generic.c
>>>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>>>> Only one Copyright line per file is enough :-)
>>>>
>>>> But actually all the Copyright lines with a name or a company name are
>>>> not really required or useful, as the copyright is noted in full details
>>>> in the commit messages (author and signed-off-by lines). I would remove
>>>> them all from the new files added by this series.
>>> Ok. Let me remove in next version.
>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>> + * it under the terms of the GNU General Public License version 2 as
>>>>> + * published by the Free Software Foundation.
>>>>> + *
>>>>> + * This program is distributed in the hope that it will be useful,
>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>>> + * GNU General Public License for more details.
>>>>> + *
>>>>> + * You should have received a copy of the GNU General Public License
>>>>> + * along with this program.  If not, see <https://urldefense.com/v3/__http://www.gnu.org/licenses/__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_3j2CfflQ$ [gnu[.]org]>.
>>>>> + */
>>>>> +
>>>>> +#include <asm/device.h>
>>>>> +#include <xen/pci.h>
>>>>> +#include <asm/pci.h>
>>>>> +
>>>>> +static const struct dt_device_match gen_pci_dt_match[] = {
>>>>> +    { .compatible = "xlnx,nwl-pcie-2.11",
>>>>> +      .data =       &pci_generic_ecam_ops },
>>>>> +    { },
>>>>> +};
>>>>> +
>>>>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>>>>> +{
>>>>> +    const struct dt_device_match *of_id;
>>>>> +    const struct pci_ecam_ops *ops;
>>>>> +
>>>>> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>>>> This should be superfluous
>>> Ack. I will remove the dt_match_node(..) in next version.
>> I am not entirely sure we need this patch at all as the main reason for its existence
>>
>> was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going
>>
>> to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP.
>>
>> So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one
>>
>> won't have it working...
>>
>> Please consider
>>
>> If we decide to remove it then
>>
>> int pci_host_common_probe(struct dt_device_node *dev,
>>                             const struct pci_ecam_ops *ops,
>>                             int ecam_reg_idx)
>>
>> doesn't need the last parameter.
> With my open source maintainer hat on, I don't see this patch as very
> important; from that point of view I'd be happy for it to be dropped.
> However, it would be good to have at least one non-default host bridge
> (doesn't have to be the Xilinx bridge), otherwise it becomes difficult
> to understand how the generic infrastructure introduced by this series
> could be useful.
>
> Moreover, your recent comment [1] made it even more evident that it
> would be good to have at least two different drivers to spot
> compatibility issues between them more easily.

Don't take me wrong here ;) I still do use Xilinx QEMU for most of

the tests, so it is good for me to have this patch in the tree. But,

to be fair, Xilinx QEMU won't give you a possibility to see the fully

functional system. This is why I say the patch can be dropped.

If we add some words in the commit message about this and have the

patch in the tree I'll be more than happy

>
> [1] https://urldefense.com/v3/__https://marc.info/?l=xen-devel&m=163154474008598__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_0bKs6zpA$ [marc[.]info]

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-09-13 20:23     ` Stefano Stabellini
@ 2021-09-14  4:35       ` Oleksandr Andrushchenko
  0 siblings, 0 replies; 98+ messages in thread
From: Oleksandr Andrushchenko @ 2021-09-14  4:35 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Rahul Singh, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk


On 13.09.21 23:23, Stefano Stabellini wrote:
> On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote:
>> Hi, Rahul!
>>
>> On 19.08.21 15:02, Rahul Singh wrote:
>>> XEN during boot will read the PCI device tree node “reg” property
>>> and will map the PCI config space to the XEN memory.
>> [snip]
>>> +static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>>> +                                              int ecam_reg_idx)
>>> +{
>>> +    int err;
>>> +    struct pci_config_window *cfg;
>>> +    paddr_t addr, size;
>>> +
>>> +    cfg = xzalloc(struct pci_config_window);
>>> +    if ( !cfg )
>>> +        return NULL;
>>> +
>>> +    err = dt_pci_parse_bus_range(dev, cfg);
>>> +    if ( !err ) {
>>> +        cfg->busn_start = 0;
>>> +        cfg->busn_end = 0xff;
>>> +        printk(XENLOG_ERR "%s:No bus range found for pci controller\n",
>>> +               dt_node_full_name(dev));
>>> +    } else {
>>> +        if ( cfg->busn_end > cfg->busn_start + 0xff )
>>> +            cfg->busn_end = cfg->busn_start + 0xff;
>>> +    }
>>> +
>>> +    /* Parse our PCI ecam register address*/
>>> +    err = dt_device_get_address(dev, ecam_reg_idx, &addr, &size);
>> I am a bit worried here that we don't get the reg index from the device tree,
>>
>> but for generic ECAM we use reg[0] and for Xilinx we use reg[2].
>>
>> For example, for Xilinx we have
>>
>> reg = <0x00 0xfd0e0000 0x00 0x1000 0x00 0xfd480000 0x00 0x1000 0x80 0x00 0x00 0x1000000>;
>> reg-names = "breg\0pcireg\0cfg";
>>
>> so, we can parse the reg-names and understand that the configuration space is the last in the reg property.
>>
>> The same I think can be done for other device trees probably.
> Well spotted!
>
>
>> Rahul, do you know if reg-names "cfg" is vendor specific of used widely?
> Unfortunately it seems to be vendor specific :-(
> Which means that "dt_device_get_address" or similar should be moved to a
> vendor specific function.

I just stepped on this while working on the patch for not mapping

MMIOs and config to Dom0 and need to find a way to tell them from

device registers. So, yes, it seems we will need something more generic

than just an index. We can still have the callback for this optional,

so the bridges with the same reg names can use the default callback

and don't need to implement the same


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM
  2021-08-19 12:02 ` [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
                     ` (2 preceding siblings ...)
  2021-09-13 14:52   ` Oleksandr Andrushchenko
@ 2021-09-14  7:53   ` Oleksandr Andrushchenko
  3 siblings, 0 replies; 98+ messages in thread
From: Oleksandr Andrushchenko @ 2021-09-14  7:53 UTC (permalink / raw)
  To: Rahul Singh, xen-devel
  Cc: Bertrand Marquis, Stefano Stabellini, Julien Grall, Volodymyr Babchuk

Hi, Rahul!

On 19.08.21 15:02, Rahul Singh wrote:
> XEN during boot will read the PCI device tree node “reg” property
> and will map the PCI config space to the XEN memory.
[snip]
> +int pci_host_common_probe(struct dt_device_node *dev,
> +                          int ecam_reg_idx)
> +{
> +    struct pci_host_bridge *bridge;
> +    struct pci_config_window *cfg;
> +    int err;
> +

Should we check if the bridge is to be passed through? E.g.

     if ( dt_device_for_passthrough(dev) )

         return 0;

Thanks,

Oleksandr

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-09-09 11:32   ` Julien Grall
@ 2021-09-14  8:13     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-14  8:13 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini, Volodymyr Babchuk

Hi Julien,

> On 9 Sep 2021, at 12:32 pm, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> Add support for PCI ecam operations to access the PCI
>> configuration space.
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>  xen/arch/arm/pci/Makefile           |  1 +
>>  xen/arch/arm/pci/ecam.c             | 63 +++++++++++++++++++++++++++++
>>  xen/arch/arm/pci/pci-access.c       | 53 ++++++++++++++++++++++++
>>  xen/arch/arm/pci/pci-host-common.c  | 13 +++++-
>>  xen/arch/arm/pci/pci-host-generic.c |  8 +++-
>>  xen/include/asm-arm/pci.h           | 32 +++++++++++++++
>>  6 files changed, 167 insertions(+), 3 deletions(-)
>>  create mode 100644 xen/arch/arm/pci/ecam.c
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> index f3d97f859e..6f32fbbe67 100644
>> --- a/xen/arch/arm/pci/Makefile
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -2,3 +2,4 @@ obj-y += pci.o
>>  obj-y += pci-access.o
>>  obj-y += pci-host-generic.o
>>  obj-y += pci-host-common.o
>> +obj-y += ecam.o
>> diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c
>> new file mode 100644
>> index 0000000000..91c691b41f
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/ecam.c
>> @@ -0,0 +1,63 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/pci.h>
>> +#include <xen/sched.h>
>> +
>> +/*
>> + * Function to implement the pci_ops ->map_bus method.
>> + */
>> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
>> +                                      uint32_t sbdf, uint32_t where)
>> +{
>> +    const struct pci_config_window *cfg = bridge->sysdata;
>> +    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
>> +    void __iomem *base;
>> +
>> +    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
> 
> AFAICT, pci_sbdf is an union between a 32-bit and a structure. So please don't use the cast and use the 32-bit field to assign the value.
> 
> Also, there is an extra space before ';'.

Ok. As per your below comment I will remove the sbdf_t completely.
> 
> 
>> +    unsigned int busn = sbdf_t.bus;
>> +
>> +    if ( busn < cfg->busn_start || busn > cfg->busn_end )
>> +        return NULL;
>> +
>> +    busn -= cfg->busn_start;
>> +    base = cfg->win + (busn << cfg->ops->bus_shift);
>> +
>> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
> 
> How about using PCI_DEVFN2(sbdf)? This would allow you to drop the use of sbdf_t completely (sbdf_t.bus could be replaced with PCI_BUS(sdbf)).
Make sense. I will modify as per your request.
> 
>> +}
>> +
>> +/* ECAM ops */
>> +const struct pci_ecam_ops pci_generic_ecam_ops = {
>> +    .bus_shift  = 20,
>> +    .pci_ops    = {
>> +        .map_bus                = pci_ecam_map_bus,
>> +        .read                   = pci_generic_config_read,
>> +        .write                  = pci_generic_config_write,
>> +    }
>> +};
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>> index b938047c03..f39f6a3a38 100644
>> --- a/xen/arch/arm/pci/pci-access.c
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -15,6 +15,59 @@
>>   */
>>    #include <xen/pci.h>
>> +#include <asm/io.h>
>> +
>> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
>> +                            uint32_t reg, uint32_t len, uint32_t *value)
>> +{
>> +    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
> 
> Please add a newline here.

Ack. 
> 
>> +    if (!addr) {
> 
> You seem to use a mix of Xen and Linux coding style. If the file contains mostly Xen code, then we should use the former.

Ok. I will fix the coding style in next version.
 
Regards,
Rahul

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-09-09 23:21   ` Stefano Stabellini
@ 2021-09-14 11:13     ` Rahul Singh
  2021-09-14 23:06       ` Stefano Stabellini
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-09-14 11:13 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 10 Sep 2021, at 12:21 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> Add support for PCI ecam operations to access the PCI
>> configuration space.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/pci/Makefile           |  1 +
>> xen/arch/arm/pci/ecam.c             | 63 +++++++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-access.c       | 53 ++++++++++++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c  | 13 +++++-
>> xen/arch/arm/pci/pci-host-generic.c |  8 +++-
>> xen/include/asm-arm/pci.h           | 32 +++++++++++++++
>> 6 files changed, 167 insertions(+), 3 deletions(-)
>> create mode 100644 xen/arch/arm/pci/ecam.c
>> 
>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>> index f3d97f859e..6f32fbbe67 100644
>> --- a/xen/arch/arm/pci/Makefile
>> +++ b/xen/arch/arm/pci/Makefile
>> @@ -2,3 +2,4 @@ obj-y += pci.o
>> obj-y += pci-access.o
>> obj-y += pci-host-generic.o
>> obj-y += pci-host-common.o
>> +obj-y += ecam.o
>> diff --git a/xen/arch/arm/pci/ecam.c b/xen/arch/arm/pci/ecam.c
>> new file mode 100644
>> index 0000000000..91c691b41f
>> --- /dev/null
>> +++ b/xen/arch/arm/pci/ecam.c
>> @@ -0,0 +1,63 @@
>> +/*
>> + * Copyright (C) 2021 Arm Ltd.
>> + *
>> + * Based on Linux drivers/pci/ecam.c
>> + * Copyright 2016 Broadcom
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/pci.h>
>> +#include <xen/sched.h>
>> +
>> +/*
>> + * Function to implement the pci_ops ->map_bus method.
>> + */
>> +void __iomem *pci_ecam_map_bus(struct pci_host_bridge *bridge,
>> +                                      uint32_t sbdf, uint32_t where)
> 
> Code style: alignment
Ack.
> 
> 
>> +{
>> +    const struct pci_config_window *cfg = bridge->sysdata;
>> +    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
> 
> Is it a guarantee that devfn_shift == bus_shift - 8, or is it just so
> for ECAM?

Yes it is guarantee that "devfn_shift == bus_shift - 8” as bus number will be 8 bit always.
bus_shift can be different based on vendor.

https://elixir.bootlin.com/linux/latest/source/drivers/pci/ecam.c#L172
https://elixir.bootlin.com/linux/latest/source/drivers/pci/controller/pci-thunder-pem.c#L30

> 
> 
>> +    void __iomem *base;
>> +
>> +    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
>> +    unsigned int busn = sbdf_t.bus;
>> +
>> +    if ( busn < cfg->busn_start || busn > cfg->busn_end )
> 
> Genuine question: should it be busn >= cfg->busn_end ?  I don't know if
> the range includes busn_end or not.

cfg->bus_end includes the valid last bus number.  "busn > cfg->busn_end" is valid check 
to confirm if we are not accessing pci device outside the valid bus number.

> 
> 
>> +        return NULL;
>> +
>> +    busn -= cfg->busn_start;
>> +    base = cfg->win + (busn << cfg->ops->bus_shift);
>> +
>> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
>> +}
> 
> I understand that the arm32 part is not implemented and not part of this
> series, that's fine. However if the plan is that arm32 will dynamically
> map each bus individually, then I imagine this function will have an
> ioremap in the arm32 version. Which means that we also need an
> unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus
> would be a NOP today for arm64, but I think it makes sense to have it if
> we want the API to be generic.

As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the 
bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part.
Let me know your view on this.

> 
> 
>> +/* ECAM ops */
>> +const struct pci_ecam_ops pci_generic_ecam_ops = {
>> +    .bus_shift  = 20,
>> +    .pci_ops    = {
>> +        .map_bus                = pci_ecam_map_bus,
>> +        .read                   = pci_generic_config_read,
>> +        .write                  = pci_generic_config_write,
>> +    }
>> +};
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * tab-width: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>> index b938047c03..f39f6a3a38 100644
>> --- a/xen/arch/arm/pci/pci-access.c
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -15,6 +15,59 @@
>>  */
>> 
>> #include <xen/pci.h>
>> +#include <asm/io.h>
>> +
>> +int pci_generic_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
>> +                            uint32_t reg, uint32_t len, uint32_t *value)
>> +{
>> +    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
>> +    if (!addr) {
>> +        *value = ~0;
> 
> Is this a standard error? If so, I think we should define it with a
> macro (e.g. INVALID_PADDR).

Ok.
> 
> 
>> +        return -ENODEV;
>> +    }
>> +
>> +    switch (len)
>> +    {
>> +    case 1:
>> +        *value = readb(addr);
>> +        break;
>> +    case 2:
>> +        *value = readw(addr);
>> +        break;
>> +    case 4:
>> +        *value = readl(addr);
>> +        break;
>> +    default:
>> +        BUG();
> 
> A BUG here is harsh because it could be potentially guest-triggered. An
> ASSERT would be better.

Ok.
> 
> 
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>> +                            uint32_t reg, uint32_t len, uint32_t value)
>> +{
>> +    void __iomem *addr = bridge->ops->map_bus(bridge, sbdf, reg);
>> +    if (!addr)
>> +        return -ENODEV;
>> +
>> +    switch (len)
>> +    {
>> +    case 1:
>> +        writeb(value, addr);
>> +        break;
>> +    case 2:
>> +        writew(value, addr);
>> +        break;
>> +    case 4:
>> +        writel(value, addr);
>> +        break;
>> +    default:
>> +        BUG();
> 
> Same here
Ok.
> 
> 
>> +    }
>> +
>> +    return 0;
>> +}
>> 
>> static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>>                                 unsigned int len)
>> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
>> index 9dd9b02271..c582527e92 100644
>> --- a/xen/arch/arm/pci/pci-host-common.c
>> +++ b/xen/arch/arm/pci/pci-host-common.c
>> @@ -68,6 +68,7 @@ static void pci_ecam_free(struct pci_config_window *cfg)
>> }
>> 
>> static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>> +                                              const struct pci_ecam_ops *ops,
>>                                               int ecam_reg_idx)
>> {
>>     int err;
>> @@ -96,6 +97,7 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>> 
>>     cfg->phys_addr = addr;
>>     cfg->size = size;
>> +    cfg->ops = ops;
>> 
>>     /*
>>      * On 64-bit systems, we do a single ioremap for the whole config space
>> @@ -111,6 +113,13 @@ static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
>>     printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
>>             cfg->phys_addr + cfg->size - 1, cfg->busn_start, cfg->busn_end);
>> 
>> +    if ( ops->init )
>> +    {
>> +        err = ops->init(cfg);
>> +        if (err)
>> +            goto err_exit;
>> +    }
>> +
>>     return cfg;
>> 
>> err_exit_remap:
>> @@ -216,6 +225,7 @@ static int pci_bus_find_domain_nr(struct dt_device_node *dev)
>> }
>> 
>> int pci_host_common_probe(struct dt_device_node *dev,
>> +                          const struct pci_ecam_ops *ops,
>>                           int ecam_reg_idx)
>> {
>>     struct pci_host_bridge *bridge;
>> @@ -227,7 +237,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
>>         return -ENOMEM;
>> 
>>     /* Parse and map our Configuration Space windows */
>> -    cfg = gen_pci_init(dev, ecam_reg_idx);
>> +    cfg = gen_pci_init(dev, ops, ecam_reg_idx);
>>     if ( !cfg )
>>     {
>>         err = -ENOMEM;
>> @@ -236,6 +246,7 @@ int pci_host_common_probe(struct dt_device_node *dev,
>> 
>>     bridge->dt_node = dev;
>>     bridge->sysdata = cfg;
>> +    bridge->ops = &ops->pci_ops;
>>     bridge->bus_start = cfg->busn_start;
>>     bridge->bus_end = cfg->busn_end;
>> 
>> diff --git a/xen/arch/arm/pci/pci-host-generic.c b/xen/arch/arm/pci/pci-host-generic.c
>> index 13d0f7f999..2d652e8910 100644
>> --- a/xen/arch/arm/pci/pci-host-generic.c
>> +++ b/xen/arch/arm/pci/pci-host-generic.c
>> @@ -23,20 +23,24 @@
>> #include <asm/pci.h>
>> 
>> static const struct dt_device_match gen_pci_dt_match[] = {
>> -    { .compatible = "pci-host-ecam-generic" },
>> +    { .compatible = "pci-host-ecam-generic",
>> +      .data =       &pci_generic_ecam_ops },
>> +
>>     { },
>> };
>> 
>> static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>> {
>>     const struct dt_device_match *of_id;
>> +    const struct pci_ecam_ops *ops;
>> 
>>     of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>> +    ops = (struct pci_ecam_ops *) of_id->data;
>> 
>>     printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
>>            dt_node_full_name(dev), of_id->compatible);
>> 
>> -    return pci_host_common_probe(dev, 0);
>> +    return pci_host_common_probe(dev, ops, 0);
>> }
>> 
>> DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
>> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>> index 58a51e724e..22866244d2 100644
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -37,6 +37,7 @@ struct pci_config_window {
>>     uint8_t         busn_start;
>>     uint8_t         busn_end;
>>     void __iomem    *win;
>> +    const struct    pci_ecam_ops *ops;
>> };
>> 
>> /*
>> @@ -50,10 +51,41 @@ struct pci_host_bridge {
>>     u8 bus_start;                    /* Bus start of this bridge. */
>>     u8 bus_end;                      /* Bus end of this bridge. */
>>     void *sysdata;                   /* Pointer to the config space window*/
>> +    const struct pci_ops *ops;
>> };
>> 
>> +struct pci_ops {
>> +    void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf,
>> +                             uint32_t offset);
>> +    int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf,
>> +                uint32_t reg, uint32_t len, uint32_t *value);
>> +    int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf,
>> +                 uint32_t reg, uint32_t len, uint32_t value);
>> +};
>> +
>> +/*
>> + * struct to hold pci ops and bus shift of the config window
>> + * for a PCI controller.
>> + */
>> +struct pci_ecam_ops {
>> +    unsigned int            bus_shift;
>> +    struct pci_ops          pci_ops;
>> +    int (*init)(struct pci_config_window *);
> 
> Is this last member of the struct needed/used?

Yes It will be used by some vendor specific board( N1SDP ). Please check below.
https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/commit/?id=04b7e76d0fe6481a803f58e54e008a1489d713a5
 
Regards,
Rahul

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 08/14] xen:arm: Implement pci access functions
  2021-09-09 23:41   ` Stefano Stabellini
@ 2021-09-14 16:05     ` Rahul Singh
  2021-09-14 22:40       ` Stefano Stabellini
  2021-09-15  7:54       ` Oleksandr Andrushchenko
  0 siblings, 2 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-14 16:05 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 10 Sep 2021, at 12:41 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> Implement generic pci access functions to read/write the configuration
>> space.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/pci/pci-access.c      | 31 +++++++++++++++++++++++++++++-
>> xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++
>> xen/include/asm-arm/pci.h          |  2 ++
>> 3 files changed, 51 insertions(+), 1 deletion(-)
>> 
>> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
>> index f39f6a3a38..b94de3c3ac 100644
>> --- a/xen/arch/arm/pci/pci-access.c
>> +++ b/xen/arch/arm/pci/pci-access.c
>> @@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
>> static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
>>                                 unsigned int len)
>> {
>> -    return ~0U;
>> +    uint32_t val = GENMASK(0, len * 8);
> 
> This seems to be another default error value that it would be better to
> define with its own macro

This default error is used once do you want to me define as macro.  

> 
> 
>> +    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> +    if ( unlikely(!bridge) )
>> +    {
>> +        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> +                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> 
> You are not actually printing sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn ?

Yes I am printing with “PRI_pci".
> 
> 
>> +        return val;
>> +    }
>> +
>> +    if ( unlikely(!bridge->ops->read) )
>> +        return val;
>> +
>> +    bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
> 
> Would it make sense to make the interface take a pci_sbdf_t directly
> instead of casting to uint32_t and back?

pci_sbdf_t is defined in "xen/pci.h” and "xen/pci.h” includes "asm-arm/pci.h”. 
If I modify the function argument in "asm-arm/pci.h” to use pci_sbdf_t  I will get compilation error.

> 
> 
>> +    return val;
>> }
>> 
>> static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>>                              unsigned int len, uint32_t val)
>> {
>> +    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>> +
>> +    if ( unlikely(!bridge) )
>> +    {
>> +        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>> +                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> 
> same here
Yes I am printing with “PRI_pci".
>  

Regards
Rahul


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 08/14] xen:arm: Implement pci access functions
  2021-09-14 16:05     ` Rahul Singh
@ 2021-09-14 22:40       ` Stefano Stabellini
  2021-09-15  7:54       ` Oleksandr Andrushchenko
  1 sibling, 0 replies; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-14 22:40 UTC (permalink / raw)
  To: Rahul Singh
  Cc: Stefano Stabellini, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk, jbeulich

[-- Attachment #1: Type: text/plain, Size: 3316 bytes --]

+Jan for the header include question at the bottom


On Tue, 14 Sep 2021, Rahul Singh wrote:
> Hi Stefano,
> 
> > On 10 Sep 2021, at 12:41 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > 
> > On Thu, 19 Aug 2021, Rahul Singh wrote:
> >> Implement generic pci access functions to read/write the configuration
> >> space.
> >> 
> >> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> >> ---
> >> xen/arch/arm/pci/pci-access.c      | 31 +++++++++++++++++++++++++++++-
> >> xen/arch/arm/pci/pci-host-common.c | 19 ++++++++++++++++++
> >> xen/include/asm-arm/pci.h          |  2 ++
> >> 3 files changed, 51 insertions(+), 1 deletion(-)
> >> 
> >> diff --git a/xen/arch/arm/pci/pci-access.c b/xen/arch/arm/pci/pci-access.c
> >> index f39f6a3a38..b94de3c3ac 100644
> >> --- a/xen/arch/arm/pci/pci-access.c
> >> +++ b/xen/arch/arm/pci/pci-access.c
> >> @@ -72,12 +72,41 @@ int pci_generic_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
> >> static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
> >>                                 unsigned int len)
> >> {
> >> -    return ~0U;
> >> +    uint32_t val = GENMASK(0, len * 8);
> > 
> > This seems to be another default error value that it would be better to
> > define with its own macro
> 
> This default error is used once do you want to me define as macro.  

Yes. A macro is good even if you are going to use it once because it
also serves as documentation for the error. For instance:

/* PCI host bridge not found */
#define PCI_ERR_NOTFOUND(len) GENMASK(0, len * 8)

really helps with the explanation of what the error is about.


> >> +    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
> >> +
> >> +    if ( unlikely(!bridge) )
> >> +    {
> >> +        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
> >> +                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
> > 
> > You are not actually printing sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn ?
> 
> Yes I am printing with “PRI_pci".

Sorry I missed it!


> >> +        return val;
> >> +    }
> >> +
> >> +    if ( unlikely(!bridge->ops->read) )
> >> +        return val;
> >> +
> >> +    bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
> > 
> > Would it make sense to make the interface take a pci_sbdf_t directly
> > instead of casting to uint32_t and back?
> 
> pci_sbdf_t is defined in "xen/pci.h” and "xen/pci.h” includes "asm-arm/pci.h”. 
> If I modify the function argument in "asm-arm/pci.h” to use pci_sbdf_t  I will get compilation error.

This is unfortunate. One way around it is to make the appended change to
xen/include/xen/pci.h and then simply add:

typedef union pci_sbdf pci_sbdf_t;

to xen/include/asm-arm/pci.h.

Jan do you have any better suggestions?


diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index 8e3d4d9454..ae8d48135b 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -41,7 +41,7 @@
 #define PCI_SBDF3(s,b,df) \
     ((pci_sbdf_t){ .sbdf = (((s) & 0xffff) << 16) | PCI_BDF2(b, df) })
 
-typedef union {
+union pci_sbdf {
     uint32_t sbdf;
     struct {
         union {
@@ -60,7 +60,9 @@ typedef union {
         };
         uint16_t                seg;
     };
-} pci_sbdf_t;
+};
+
+typedef union pci_sbdf pci_sbdf_t;
 
 struct pci_dev_info {
     /*

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-09-14 11:13     ` Rahul Singh
@ 2021-09-14 23:06       ` Stefano Stabellini
  2021-09-15 16:38         ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-14 23:06 UTC (permalink / raw)
  To: Rahul Singh
  Cc: Stefano Stabellini, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk

[-- Attachment #1: Type: text/plain, Size: 4482 bytes --]

On Tue, 14 Sep 2021, Rahul Singh wrote:
> >> +        return NULL;
> >> +
> >> +    busn -= cfg->busn_start;
> >> +    base = cfg->win + (busn << cfg->ops->bus_shift);
> >> +
> >> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
> >> +}
> > 
> > I understand that the arm32 part is not implemented and not part of this
> > series, that's fine. However if the plan is that arm32 will dynamically
> > map each bus individually, then I imagine this function will have an
> > ioremap in the arm32 version. Which means that we also need an
> > unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus
> > would be a NOP today for arm64, but I think it makes sense to have it if
> > we want the API to be generic.
> 
> As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the 
> bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part.
> Let me know your view on this.

In the patch titled "xen/arm: PCI host bridge discovery within XEN on
ARM" there is the following in-code comment:

* On 64-bit systems, we do a single ioremap for the whole config space
* since we have enough virtual address range available.  On 32-bit, we
* ioremap the config space for each bus individually.
*
* As of now only 64-bit is supported 32-bit is not supported.


So I take it that on arm32 we don't have enough virtual address range
available, therefore we cannot ioremap the whole range. Instead, we'll
have to ioremap the config space of each bus individually.

I assumed that the idea was to call ioremap and iounmap dynamically,
otherwise the total amount of virtual address range required would still
be the same. I also thought that the dynamic mapping function, the one
which would end up calling ioremap on arm32, would be pci_ecam_map_bus.
If so, then we are missing the corresponding unmapping function, the one
that would call iounmap on arm32 and do nothing on arm64, called before
returning from pci_generic_config_read and pci_generic_config_write.

As always, I am not asking for the arm32 implementation, but if we are
introducing internal APIs, it would be good that they are consistent.



> >> @@ -50,10 +51,41 @@ struct pci_host_bridge {
> >>     u8 bus_start;                    /* Bus start of this bridge. */
> >>     u8 bus_end;                      /* Bus end of this bridge. */
> >>     void *sysdata;                   /* Pointer to the config space window*/
> >> +    const struct pci_ops *ops;
> >> };
> >> 
> >> +struct pci_ops {
> >> +    void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf,
> >> +                             uint32_t offset);
> >> +    int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf,
> >> +                uint32_t reg, uint32_t len, uint32_t *value);
> >> +    int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf,
> >> +                 uint32_t reg, uint32_t len, uint32_t value);
> >> +};
> >> +
> >> +/*
> >> + * struct to hold pci ops and bus shift of the config window
> >> + * for a PCI controller.
> >> + */
> >> +struct pci_ecam_ops {
> >> +    unsigned int            bus_shift;
> >> +    struct pci_ops          pci_ops;
> >> +    int (*init)(struct pci_config_window *);
> > 
> > Is this last member of the struct needed/used?
> 
> Yes It will be used by some vendor specific board( N1SDP ). Please check below.
> https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/commit/?id=04b7e76d0fe6481a803f58e54e008a1489d713a5

OK. I don't doubt that there might be bridge-specific initializations,
but we already have things like:

DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI)
.dt_match = gen_pci_dt_match,
.init = gen_pci_dt_init,
DT_DEVICE_END

The question is do we actually need two different vendor-specific init
functions? One is driven by device tree, e.g. ZynqMP is calling
gen_pci_dt_init. The other one is pci_ecam_ops->init, which is called
from the following chain:

DT_DEVICE_START -> gen_pci_dt_init -> pci_host_common_probe ->
gen_pci_init -> pci_generic_ecam_ops.init

What's the difference between gen_pci_dt_init and pci_ecam_ops.init in
terms of purpose?

I am happy to have pci_ecam_ops.init if it serves a different purpose
from DT_DEVICE_START.init. In that case we might want to add an in-code
comment so that an engineer would know where to add vendor-specific code
(whether to DT_DEVICE_START.init or to pci_ecam_ops.init).

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 08/14] xen:arm: Implement pci access functions
  2021-09-14 16:05     ` Rahul Singh
  2021-09-14 22:40       ` Stefano Stabellini
@ 2021-09-15  7:54       ` Oleksandr Andrushchenko
  2021-09-15 10:47         ` Rahul Singh
  1 sibling, 1 reply; 98+ messages in thread
From: Oleksandr Andrushchenko @ 2021-09-15  7:54 UTC (permalink / raw)
  To: Rahul Singh, Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi, Rahul!
>>> static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>>>                               unsigned int len, uint32_t val)
>>> {
>>> +    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>> +
>>> +    if ( unlikely(!bridge) )
>>> +    {
>>> +        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>> +                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>> same here
> Yes I am printing with “PRI_pci".

vPCI and the rest are widely using

         printk("%pp\n",  &sbdf);
So, I think if we have SBDF then it is better to use %pp instead of trying to unfold it manually.

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 08/14] xen:arm: Implement pci access functions
  2021-09-15  7:54       ` Oleksandr Andrushchenko
@ 2021-09-15 10:47         ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-15 10:47 UTC (permalink / raw)
  To: Oleksandr Andrushchenko
  Cc: Stefano Stabellini, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk

Hi Oleksandr,

> On 15 Sep 2021, at 8:54 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
> 
> Hi, Rahul!
>>>> static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
>>>>                              unsigned int len, uint32_t val)
>>>> {
>>>> +    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
>>>> +
>>>> +    if ( unlikely(!bridge) )
>>>> +    {
>>>> +        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
>>>> +                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
>>> same here
>> Yes I am printing with “PRI_pci".
> 
> vPCI and the rest are widely using
> 
>         printk("%pp\n",  &sbdf);
> So, I think if we have SBDF then it is better to use %pp instead of trying to unfold it manually.

Ok. I will use the %pp for printing the SBDF.

Regards,
Rahul


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-09-14 23:06       ` Stefano Stabellini
@ 2021-09-15 16:38         ` Rahul Singh
  2021-09-15 20:45           ` Stefano Stabellini
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-09-15 16:38 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 15 Sep 2021, at 12:06 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Tue, 14 Sep 2021, Rahul Singh wrote:
>>>> +        return NULL;
>>>> +
>>>> +    busn -= cfg->busn_start;
>>>> +    base = cfg->win + (busn << cfg->ops->bus_shift);
>>>> +
>>>> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
>>>> +}
>>> 
>>> I understand that the arm32 part is not implemented and not part of this
>>> series, that's fine. However if the plan is that arm32 will dynamically
>>> map each bus individually, then I imagine this function will have an
>>> ioremap in the arm32 version. Which means that we also need an
>>> unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus
>>> would be a NOP today for arm64, but I think it makes sense to have it if
>>> we want the API to be generic.
>> 
>> As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the 
>> bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part.
>> Let me know your view on this.
> 
> In the patch titled "xen/arm: PCI host bridge discovery within XEN on
> ARM" there is the following in-code comment:
> 
> * On 64-bit systems, we do a single ioremap for the whole config space
> * since we have enough virtual address range available.  On 32-bit, we
> * ioremap the config space for each bus individually.
> *
> * As of now only 64-bit is supported 32-bit is not supported.
> 
> 
> So I take it that on arm32 we don't have enough virtual address range
> available, therefore we cannot ioremap the whole range. Instead, we'll
> have to ioremap the config space of each bus individually.

Yes you are right my understand is also same.
> 
> I assumed that the idea was to call ioremap and iounmap dynamically,
> otherwise the total amount of virtual address range required would still
> be the same.

As per my understanding for 32-bit we need per_bus mapping as we don’t have enough virtual address space in one chunk
but we can have virtual address space in different chunk.

I am not sure if we need to map/unmap the virtual address space for each read/write call. 
I just checked the Linux code[1]  and there also mapping is done once not for each read/write call.

[1]  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/ecam.c?id=ea4aae05974334e9837d86ff1cb716bad36b3ca8#n76

> I also thought that the dynamic mapping function, the one
> which would end up calling ioremap on arm32, would be pci_ecam_map_bus.
> If so, then we are missing the corresponding unmapping function, the one
> that would call iounmap on arm32 and do nothing on arm64, called before
> returning from pci_generic_config_read and pci_generic_config_write.
> 
> As always, I am not asking for the arm32 implementation, but if we are
> introducing internal APIs, it would be good that they are consistent.
> 
> 
> 
>>>> @@ -50,10 +51,41 @@ struct pci_host_bridge {
>>>>    u8 bus_start;                    /* Bus start of this bridge. */
>>>>    u8 bus_end;                      /* Bus end of this bridge. */
>>>>    void *sysdata;                   /* Pointer to the config space window*/
>>>> +    const struct pci_ops *ops;
>>>> };
>>>> 
>>>> +struct pci_ops {
>>>> +    void __iomem *(*map_bus)(struct pci_host_bridge *bridge, uint32_t sbdf,
>>>> +                             uint32_t offset);
>>>> +    int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf,
>>>> +                uint32_t reg, uint32_t len, uint32_t *value);
>>>> +    int (*write)(struct pci_host_bridge *bridge, uint32_t sbdf,
>>>> +                 uint32_t reg, uint32_t len, uint32_t value);
>>>> +};
>>>> +
>>>> +/*
>>>> + * struct to hold pci ops and bus shift of the config window
>>>> + * for a PCI controller.
>>>> + */
>>>> +struct pci_ecam_ops {
>>>> +    unsigned int            bus_shift;
>>>> +    struct pci_ops          pci_ops;
>>>> +    int (*init)(struct pci_config_window *);
>>> 
>>> Is this last member of the struct needed/used?
>> 
>> Yes It will be used by some vendor specific board( N1SDP ). Please check below.
>> https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/commit/?id=04b7e76d0fe6481a803f58e54e008a1489d713a5
> 
> OK. I don't doubt that there might be bridge-specific initializations,
> but we already have things like:
> 
> DT_DEVICE_START(pci_gen, "PCI HOST ZYNQMP", DEVICE_PCI)
> .dt_match = gen_pci_dt_match,
> .init = gen_pci_dt_init,
> DT_DEVICE_END
> 
> The question is do we actually need two different vendor-specific init
> functions? One is driven by device tree, e.g. ZynqMP is calling
> gen_pci_dt_init. The other one is pci_ecam_ops->init, which is called
> from the following chain:
> 
> DT_DEVICE_START -> gen_pci_dt_init -> pci_host_common_probe ->
> gen_pci_init -> pci_generic_ecam_ops.init
> 
> What's the difference between gen_pci_dt_init and pci_ecam_ops.init in
> terms of purpose?

Yes I also agree with you we don’t need two init function. I will modify the code like below.
DT_DEVICE_START ->pci_host_common_probe -> gen_pci_init -> pci_generic_ecam_ops.init

Regards,
Rahul
> 
> I am happy to have pci_ecam_ops.init if it serves a different purpose
> from DT_DEVICE_START.init. In that case we might want to add an in-code
> comment so that an engineer would know where to add vendor-specific code
> (whether to DT_DEVICE_START.init or to pci_ecam_ops.init).


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-09-15 16:38         ` Rahul Singh
@ 2021-09-15 20:45           ` Stefano Stabellini
  2021-09-16 16:51             ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-15 20:45 UTC (permalink / raw)
  To: Rahul Singh
  Cc: Stefano Stabellini, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk

[-- Attachment #1: Type: text/plain, Size: 3292 bytes --]

On Wed, 15 Sep 2021, Rahul Singh wrote:
> > On 15 Sep 2021, at 12:06 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > On Tue, 14 Sep 2021, Rahul Singh wrote:
> >>>> +        return NULL;
> >>>> +
> >>>> +    busn -= cfg->busn_start;
> >>>> +    base = cfg->win + (busn << cfg->ops->bus_shift);
> >>>> +
> >>>> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
> >>>> +}
> >>> 
> >>> I understand that the arm32 part is not implemented and not part of this
> >>> series, that's fine. However if the plan is that arm32 will dynamically
> >>> map each bus individually, then I imagine this function will have an
> >>> ioremap in the arm32 version. Which means that we also need an
> >>> unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus
> >>> would be a NOP today for arm64, but I think it makes sense to have it if
> >>> we want the API to be generic.
> >> 
> >> As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the 
> >> bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part.
> >> Let me know your view on this.
> > 
> > In the patch titled "xen/arm: PCI host bridge discovery within XEN on
> > ARM" there is the following in-code comment:
> > 
> > * On 64-bit systems, we do a single ioremap for the whole config space
> > * since we have enough virtual address range available.  On 32-bit, we
> > * ioremap the config space for each bus individually.
> > *
> > * As of now only 64-bit is supported 32-bit is not supported.
> > 
> > 
> > So I take it that on arm32 we don't have enough virtual address range
> > available, therefore we cannot ioremap the whole range. Instead, we'll
> > have to ioremap the config space of each bus individually.
> 
> Yes you are right my understand is also same.
> > 
> > I assumed that the idea was to call ioremap and iounmap dynamically,
> > otherwise the total amount of virtual address range required would still
> > be the same.
> 
> As per my understanding for 32-bit we need per_bus mapping as we don’t have enough virtual address space in one chunk
> but we can have virtual address space in different chunk.

Interesting. I would have assumed that the sum of all the individual
smaller ioremaps would still be equal to one big ioremap. Maybe for
Linux is different, but I don't think that many smaller ioremaps would
buy us very much in Xen because it is the total ioremap virtual space
that is too small. Or am I missing something?


> I am not sure if we need to map/unmap the virtual address space for each read/write call. 
> I just checked the Linux code[1]  and there also mapping is done once not for each read/write call.

So my guess is that for arm32 we would have to resort to dynamic
map/unmap for each read/write call, unless there is a trick with the
individual smaller ioremaps that I haven't spotted (e.g. maybe something
doesn't get mapped that way?)

That said, given that we are uncertain about this and the arm32
implementation is nowhere close, I think that we are OK to continue like
this for this series. Maybe you could add a couple of sentences to the
in-code comment so that if somebody wants to jump in and implement
arm32 support they would know where to start.

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-09-09 13:50   ` Julien Grall
@ 2021-09-16 10:46     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-16 10:46 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Stefano Stabellini,
	Volodymyr Babchuk, Jan Beulich, Paul Durrant,
	Roger Pau Monné

Hi Julien,

> On 9 Sep 2021, at 2:50 pm, Julien Grall <julien@xen.org> wrote:
> 
> Hi Rahul,
> 
> On 19/08/2021 13:02, Rahul Singh wrote:
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>>  xen/arch/arm/Makefile         |  1 +
>>  xen/arch/arm/domain.c         |  4 ++
>>  xen/arch/arm/vpci.c           | 96 +++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/vpci.h           | 37 ++++++++++++++
>>  xen/drivers/passthrough/pci.c |  7 +++
>>  xen/drivers/vpci/Makefile     |  3 +-
>>  xen/drivers/vpci/header.c     |  2 +
>>  xen/include/asm-arm/domain.h  |  5 +-
>>  xen/include/asm-arm/pci.h     |  8 +++
>>  xen/include/public/arch-arm.h |  4 ++
>>  10 files changed, 165 insertions(+), 2 deletions(-)
>>  create mode 100644 xen/arch/arm/vpci.c
>>  create mode 100644 xen/arch/arm/vpci.h
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 0e14a5e5c8..7cdce684a4 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ obj-y += platforms/
>>  endif
>>  obj-$(CONFIG_TEE) += tee/
>>  obj-$(CONFIG_HAS_PCI) += pci/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>>    obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>>  obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index 19c756ac3d..d99c653626 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -40,6 +40,7 @@
>>  #include <asm/vtimer.h>
>>    #include "vuart.h"
>> +#include "vpci.h"
> 
> Please order the includes alphabetically. So this one should go before "vuart.h".
> 
>>    DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>>  @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>      if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>          goto fail;
>>  +    if ( (rc = domain_vpci_init(d)) != 0 )
>> +        goto fail;
>> +
>>      return 0;
>>    fail:
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..da8b1ca13c
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,96 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + * Copyright (c) 2021 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
> 
> NIT: Please add a newline between generic and arch specific includes.
Ack.
> 
>> +#include <asm/mmio.h>
>> +
>> +/* Do some sanity checks. */
>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> +    /* Check access size. */
>> +    if ( len != 1 && len != 2 && len != 4 && len != 8 )
>> +        return false;
> You will allow all the possible value of len (this is coming from the HW). So I feels this is a bit too much to check for every I/O to the vPCI.
> 
> If you really want to keep the check then you can simply check that len is < 8 because the two callers compute it with (1 << S). So there is no way to set it 3, 5, 6 and 7.

I will remove the check for 1,2,4,8 and will have check ( len < 8).
> 
>> +
>> +    /* Check that access is size aligned. */
>> +    if ( (reg & (len - 1)) )
>> +        return false;
>> +
>> +    return true;
>> +}
>> +
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> +                          register_t *r, void *p)
>> +{
>> +    unsigned int reg;
>> +    pci_sbdf_t sbdf;
>> +    uint32_t data = 0;
>> +    unsigned int size = 1U << info->dabt.size;
>> +
>> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
>> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
> 
> This logic is the same as below in vpci_mmio_write(). So I think you want to provide an helper because this is not trivial to read.
Ok.
> 
> Also, for the first line, I think you can re-use MMCFG_BDF() from the x86 code. For the second line, I would define the value so it is clearer to understand that they mean (although & 3 is fine to me) .
Ok.
> 
>> +
>> +    if ( !vpci_mmio_access_allowed(reg, size) )
>> +        return 1;
> So, you will a guest will read 0 if the access is unaligned. This seems an odd behavior given this is not an allowed access. AFAIU, the HW would likely trow a data abort because you can't do unalign access on uncachable memory. So I think we should return 0 here to let the MMIO handler inject a data abort.

Ack. 
> 
>> +
>> +    data = vpci_read(sbdf, reg, size);
> 
> So in vpci_mmio_access_allowed(), you will allow a guest to read a 64-bit value. But... vpci_read() will return a 32-bit value.
> 
> Looking at the x86 code, they have a second call to vpci_read() to handle the top 32-bit. Any reason why this was not implemented on Arm?

I missed that I will fix this in next version.

> 
> If we need to implement it then I think this should be implement in vpci_read() to avoid duplication between x86 and arm.
I think it will be good to make vpci_read() & vpci_write() simple not to add extra logic based on 64 bit value. 
I will add second call for vpci_read() and vpci_write() for ARM. 
> 
>> +
>> +    memcpy(r, &data, size);
> 
> From my understanding, any unused bit should be 0. So this should be:
> 
> *r = data;
Ack.
> 
>> +
>> +    return 1;
>> +}
>> +
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> +                           register_t r, void *p)
> 
> My remarks on vpci_mmio_read() applies here too.
> 
>> +{
>> +    unsigned int reg;
>> +    pci_sbdf_t sbdf;
>> +    uint32_t data = r;
>> +    unsigned int size = 1U << info->dabt.size;
>> +
>> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
>> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
>> +
>> +    if ( !vpci_mmio_access_allowed(reg, size) )
>> +        return 1;
>> +
>> +    vpci_write(sbdf, reg, size, data);
>> +
>> +    return 1;
>> +}
>> +
>> +static const struct mmio_handler_ops vpci_mmio_handler = {
>> +    .read  = vpci_mmio_read,
>> +    .write = vpci_mmio_write,
>> +};
>> +
>> +int domain_vpci_init(struct domain *d)
>> +{
>> +    if ( !has_vpci(d) )
>> +        return 0;
>> +
>> +    register_mmio_handler(d, &vpci_mmio_handler,
>> +                          GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
>> +
>> +    return 0;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> +
>> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
>> new file mode 100644
>> index 0000000000..8a093bb705
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.h
>> @@ -0,0 +1,37 @@
>> +/*
>> + * xen/arch/arm/vpci.h
>> + * Copyright (c) 2021 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef __ARCH_ARM_VPCI_H__
>> +#define __ARCH_ARM_VPCI_H__
>> +
>> +#ifdef CONFIG_HAS_VPCI
>> +int domain_vpci_init(struct domain *d);
>> +#else
>> +static inline int domain_vpci_init(struct domain *d)
>> +{
>> +    return 0;
>> +}
>> +#endif
>> +
>> +#endif /* __ARCH_ARM_VPCI_H__ */
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>> index c23c8cb06b..56e261e9bd 100644
>> --- a/xen/drivers/passthrough/pci.c
>> +++ b/xen/drivers/passthrough/pci.c
>> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>      else
>>          iommu_enable_device(pdev);
>>  +#ifdef CONFIG_ARM
>> +    ret = vpci_add_handlers(pdev);
>> +    if ( ret ) {
>> +        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
>> +        goto out;
>> +    }
>> +#endif
>>      pci_enable_acs(pdev);
>>    out:
>> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
>> index 55d1bdfda0..1a1413b93e 100644
>> --- a/xen/drivers/vpci/Makefile
>> +++ b/xen/drivers/vpci/Makefile
>> @@ -1 +1,2 @@
>> -obj-y += vpci.o header.o msi.o msix.o
>> +obj-y += vpci.o header.o
>> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
>> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
>> index ba9a036202..f8cd55e7c0 100644
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>>       * FIXME: punching holes after the p2m has been set up might be racy for
>>       * DomU usage, needs to be revisited.
>>       */
>> +#ifdef CONFIG_HAS_PCI_MSI
>>      if ( map && !rom_only && vpci_make_msix_hole(pdev) )
>>          return;
>> +#endif
>>        for ( i = 0; i < ARRAY_SIZE(header->bars); i++ )
>>      {
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index c9277b5c6d..d742b94bd6 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>>    #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>>  -#define has_vpci(d)    ({ (void)(d); false; })
>> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
>> + * for ARM we enable support VPCI for guest domain also.
>> + */
>> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); })
>>    #endif /* __ASM_DOMAIN_H__ */
>>  diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>> index 756f8637ab..c58152de80 100644
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -27,6 +27,14 @@ struct arch_pci_dev {
>>      struct device dev;
>>  };
>>  +/* Arch-specific MSI data for vPCI. */
>> +struct vpci_arch_msi {
>> +};
>> +
>> +/* Arch-specific MSI-X entry data for vPCI. */
>> +struct vpci_arch_msix_entry {
>> +};
>> +
>>  /*
>>   * struct to hold the mappings of a config space window. This
>>   * is expected to be used as sysdata for PCI controllers that
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index 64a2ca30da..0a9749e768 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
>>  #define GUEST_PL011_BASE    xen_mk_ullong(0x22000000)
>>  #define GUEST_PL011_SIZE    xen_mk_ullong(0x00001000)
>>  +/* VPCI ECAM mappings */
>> +#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
> 
> All the values for the memory layout has been defined in ascending order. So please add the vCPI at the correct place. If I am not mistaken, this should be before the GUEST_ACPI_*.

Ack. 
> 
>> +#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
> 
> Please document how to decide the size. This is important for the future if we need to change the size.

Ok let me document in next version.

/*                                                                              
 * 256 MB is reserved for VPCI configuration space based on calculation         
 * 256 buses × 32 devices × 8 functions × 4 KB = 256 MB                         
 */                                                                             
#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)                       
#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)


Regrads,
Rahul
> 
> Cheers,
> 
> -- 
> Julien Grall


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-09-10  0:26   ` Stefano Stabellini
@ 2021-09-16 11:01     ` Rahul Singh
  2021-09-16 20:26       ` Stefano Stabellini
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-09-16 11:01 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk,
	Jan Beulich, Paul Durrant, Roger Pau Monné

Hi Stefano,

> On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> The existing VPCI support available for X86 is adapted for Arm.
>> When the device is added to XEN via the hyper call
>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>> access is added to the Xen to emulate the PCI devices config space.
> 
> This is done just for device discovery, right?
> 
> Although it is currently not implemented (and I am not asking to
> implement it now, I am only trying to understand the architecture), it
> would be possible to discover all PCI devices just by walking down the
> PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we
> have an ECAM driver.
> 
> I take that would be the way to implement PCI support for Dom0less?

It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot.
If boot firmware did the enumeration,  XEN will discover the PCI device.

> 
> 
>> A MMIO trap handler for the PCI ECAM space is registered in XEN
>> so that when guest is trying to access the PCI config space,XEN
>> will trap the access and emulate read/write using the VPCI and
>> not the real PCI hardware.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/Makefile         |  1 +
>> xen/arch/arm/domain.c         |  4 ++
>> xen/arch/arm/vpci.c           | 96 +++++++++++++++++++++++++++++++++++
>> xen/arch/arm/vpci.h           | 37 ++++++++++++++
>> xen/drivers/passthrough/pci.c |  7 +++
>> xen/drivers/vpci/Makefile     |  3 +-
>> xen/drivers/vpci/header.c     |  2 +
>> xen/include/asm-arm/domain.h  |  5 +-
>> xen/include/asm-arm/pci.h     |  8 +++
>> xen/include/public/arch-arm.h |  4 ++
>> 10 files changed, 165 insertions(+), 2 deletions(-)
>> create mode 100644 xen/arch/arm/vpci.c
>> create mode 100644 xen/arch/arm/vpci.h
>> 
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 0e14a5e5c8..7cdce684a4 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -7,6 +7,7 @@ obj-y += platforms/
>> endif
>> obj-$(CONFIG_TEE) += tee/
>> obj-$(CONFIG_HAS_PCI) += pci/
>> +obj-$(CONFIG_HAS_VPCI) += vpci.o
>> 
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index 19c756ac3d..d99c653626 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -40,6 +40,7 @@
>> #include <asm/vtimer.h>
>> 
>> #include "vuart.h"
>> +#include "vpci.h"
>> 
>> DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
>> 
>> @@ -767,6 +768,9 @@ int arch_domain_create(struct domain *d,
>>     if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
>>         goto fail;
>> 
>> +    if ( (rc = domain_vpci_init(d)) != 0 )
>> +        goto fail;
>> +
>>     return 0;
>> 
>> fail:
>> diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c
>> new file mode 100644
>> index 0000000000..da8b1ca13c
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.c
>> @@ -0,0 +1,96 @@
>> +/*
>> + * xen/arch/arm/vpci.c
>> + * Copyright (c) 2021 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +#include <xen/sched.h>
>> +#include <asm/mmio.h>
>> +
>> +/* Do some sanity checks. */
>> +static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
>> +{
>> +    /* Check access size. */
>> +    if ( len != 1 && len != 2 && len != 4 && len != 8 )
>> +        return false;
>> +
>> +    /* Check that access is size aligned. */
>> +    if ( (reg & (len - 1)) )
>> +        return false;
>> +
>> +    return true;
>> +}
>> +
>> +static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
>> +                          register_t *r, void *p)
>> +{
>> +    unsigned int reg;
>> +    pci_sbdf_t sbdf;
>> +    uint32_t data = 0;
>> +    unsigned int size = 1U << info->dabt.size;
>> +
>> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
>> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
>> +
>> +    if ( !vpci_mmio_access_allowed(reg, size) )
>> +        return 1;
>> +
>> +    data = vpci_read(sbdf, reg, size);
>> +
>> +    memcpy(r, &data, size);
>> +
>> +    return 1;
>> +}
>> +
>> +static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
>> +                           register_t r, void *p)
>> +{
>> +    unsigned int reg;
>> +    pci_sbdf_t sbdf;
>> +    uint32_t data = r;
>> +    unsigned int size = 1U << info->dabt.size;
>> +
>> +    sbdf.sbdf = (((info->gpa) & 0x0ffff000) >> 12);
>> +    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
>> +
>> +    if ( !vpci_mmio_access_allowed(reg, size) )
>> +        return 1;
>> +
>> +    vpci_write(sbdf, reg, size, data);
>> +
>> +    return 1;
>> +}
>> +
>> +static const struct mmio_handler_ops vpci_mmio_handler = {
>> +    .read  = vpci_mmio_read,
>> +    .write = vpci_mmio_write,
>> +};
>> +
>> +int domain_vpci_init(struct domain *d)
>> +{
>> +    if ( !has_vpci(d) )
>> +        return 0;
>> +
>> +    register_mmio_handler(d, &vpci_mmio_handler,
>> +                          GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL);
>> +
>> +    return 0;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> +
>> diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h
>> new file mode 100644
>> index 0000000000..8a093bb705
>> --- /dev/null
>> +++ b/xen/arch/arm/vpci.h
>> @@ -0,0 +1,37 @@
>> +/*
>> + * xen/arch/arm/vpci.h
>> + * Copyright (c) 2021 Arm Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef __ARCH_ARM_VPCI_H__
>> +#define __ARCH_ARM_VPCI_H__
>> +
>> +#ifdef CONFIG_HAS_VPCI
>> +int domain_vpci_init(struct domain *d);
>> +#else
>> +static inline int domain_vpci_init(struct domain *d)
>> +{
>> +    return 0;
>> +}
>> +#endif
>> +
>> +#endif /* __ARCH_ARM_VPCI_H__ */
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
>> index c23c8cb06b..56e261e9bd 100644
>> --- a/xen/drivers/passthrough/pci.c
>> +++ b/xen/drivers/passthrough/pci.c
>> @@ -767,6 +767,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>>     else
>>         iommu_enable_device(pdev);
>> 
>> +#ifdef CONFIG_ARM
>> +    ret = vpci_add_handlers(pdev);
>> +    if ( ret ) {
>> +        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
>> +        goto out;
>> +    }
>> +#endif
>>     pci_enable_acs(pdev);
>> 
>> out:
>> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile
>> index 55d1bdfda0..1a1413b93e 100644
>> --- a/xen/drivers/vpci/Makefile
>> +++ b/xen/drivers/vpci/Makefile
>> @@ -1 +1,2 @@
>> -obj-y += vpci.o header.o msi.o msix.o
>> +obj-y += vpci.o header.o
>> +obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o
>> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
>> index ba9a036202..f8cd55e7c0 100644
>> --- a/xen/drivers/vpci/header.c
>> +++ b/xen/drivers/vpci/header.c
>> @@ -96,8 +96,10 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>>      * FIXME: punching holes after the p2m has been set up might be racy for
>>      * DomU usage, needs to be revisited.
>>      */
>> +#ifdef CONFIG_HAS_PCI_MSI
>>     if ( map && !rom_only && vpci_make_msix_hole(pdev) )
>>         return;
>> +#endif
>> 
>>     for ( i = 0; i < ARRAY_SIZE(header->bars); i++ )
>>     {
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index c9277b5c6d..d742b94bd6 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -262,7 +262,10 @@ static inline void arch_vcpu_block(struct vcpu *v) {}
>> 
>> #define arch_vm_assist_valid_mask(d) (1UL << VMASST_TYPE_runstate_update_flag)
>> 
>> -#define has_vpci(d)    ({ (void)(d); false; })
>> +/* For X86 VPCI is enabled and tested for PVH DOM0 only but
>> + * for ARM we enable support VPCI for guest domain also.
>> + */
>> +#define has_vpci(d) ({ (void)(d); IS_ENABLED(CONFIG_HAS_VPCI); })
>> 
>> #endif /* __ASM_DOMAIN_H__ */
>> 
>> diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h
>> index 756f8637ab..c58152de80 100644
>> --- a/xen/include/asm-arm/pci.h
>> +++ b/xen/include/asm-arm/pci.h
>> @@ -27,6 +27,14 @@ struct arch_pci_dev {
>>     struct device dev;
>> };
>> 
>> +/* Arch-specific MSI data for vPCI. */
>> +struct vpci_arch_msi {
>> +};
>> +
>> +/* Arch-specific MSI-X entry data for vPCI. */
>> +struct vpci_arch_msix_entry {
>> +};
>> +
>> /*
>>  * struct to hold the mappings of a config space window. This
>>  * is expected to be used as sysdata for PCI controllers that
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index 64a2ca30da..0a9749e768 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_PL011_BASE    xen_mk_ullong(0x22000000)
>> #define GUEST_PL011_SIZE    xen_mk_ullong(0x00001000)
>> 
>> +/* VPCI ECAM mappings */
>> +#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
>> +#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
> 
> Isn't 256MB a bit too small? Usually PCI comes with two ranges:
> 
> - a small range below 4GB like this one
> - an additional large range above 4GB, around 1GB or larger
> 
> We are missing the second larger range?

As per PCI specification PCI Express extends the Configuration Space to 4096 bytes( 4KB)  per Function and based on calculation 
"256 buses × 32 devices × 8 functions × 4 KB = 256 MB”,  256 MB is sufficient as per my understaning.

Regards,
Rahul



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl
  2021-09-09 13:59       ` Julien Grall
@ 2021-09-16 16:16         ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-16 16:16 UTC (permalink / raw)
  To: Julien Grall
  Cc: xen-devel, Bertrand Marquis, Ian Jackson, Wei Liu,
	Anthony PERARD, Juergen Gross, Andrew Cooper, George Dunlap,
	Jan Beulich, Stefano Stabellini, Volodymyr Babchuk

Hi Julien,

> On 9 Sep 2021, at 2:59 pm, Julien Grall <julien@xen.org> wrote:
> 
> 
> 
> On 20/08/2021 17:03, Rahul Singh wrote:
>> Hi Julien,
> 
> Hi Rahul,
> 
>>> On 19 Aug 2021, at 2:00 pm, Julien Grall <julien@xen.org> wrote:
>>> 
>>> Hi Rahul,
>>> 
>>> On 19/08/2021 13:02, Rahul Singh wrote:
>>>> libxl will create an emulated PCI device tree node in the device tree to
>>>> enable the guest OS to discover the virtual PCI during guest boot.
>>>> Emulated PCI device tree node will only be created when there is any
>>>> device assigned to guest.
>>>> A new area has been reserved in the arm guest physical map at
>>>> which the VPCI bus is declared in the device tree (reg and ranges
>>>> parameters of the node).
>>>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>>>> ---
>>>>  tools/libs/light/libxl_arm.c          | 109 ++++++++++++++++++++++++++
>>>>  tools/libs/light/libxl_types.idl      |   1 +
>>>>  tools/xl/xl_parse.c                   |   2 +
>>>>  xen/include/public/arch-arm.h         |  11 +++
>>>>  xen/include/public/device_tree_defs.h |   1 +
>>>>  5 files changed, 124 insertions(+)
>>>> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
>>>> index e3140a6e00..a091e97e76 100644
>>>> --- a/tools/libs/light/libxl_arm.c
>>>> +++ b/tools/libs/light/libxl_arm.c
>>>> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>>>>      return fdt_property(fdt, "reg", regs, sizeof(regs));
>>>>  }
>>>>  +static int fdt_property_values(libxl__gc *gc, void *fdt,
>>>> +        const char *name, unsigned num_cells, ...)
>>>> +{
>>>> +    uint32_t prop[num_cells];
>>>> +    be32 *cells = &prop[0];
>>>> +    int i;
>>>> +    va_list ap;
>>>> +    uint32_t arg;
>>>> +
>>>> +    va_start(ap, num_cells);
>>>> +    for (i = 0 ; i < num_cells; i++) {
>>>> +        arg = va_arg(ap, uint32_t);
>>>> +        set_cell(&cells, 1, arg);
>>>> +    }
>>>> +    va_end(ap);
>>>> +
>>>> +    return fdt_property(fdt, name, prop, sizeof(prop));
>>>> +}
>>>> +
>>>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
>>>> +                                    unsigned addr_cells,
>>>> +                                    unsigned size_cells,
>>>> +                                    unsigned num_regs, ...)
>>>> +{
>>>> +    uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
>>>> +    be32 *cells = &regs[0];
>>>> +    int i;
>>>> +    va_list ap;
>>>> +    uint64_t arg;
>>>> +
>>>> +    va_start(ap, num_regs);
>>>> +    for (i = 0 ; i < num_regs; i++) {
>>>> +        /* Set the memory bit field */
>>>> +        arg = va_arg(ap, uint64_t);
>>>> +        set_cell(&cells, 1, arg);
>>>> +
>>>> +        /* Set the vpci bus address */
>>>> +        arg = addr_cells ? va_arg(ap, uint64_t) : 0;
>>>> +        set_cell(&cells, addr_cells , arg);
>>>> +
>>>> +        /* Set the cpu bus address where vpci address is mapped */
>>>> +        set_cell(&cells, addr_cells, arg);
>>>> +
>>>> +        /* Set the vpci size requested */
>>>> +        arg = size_cells ? va_arg(ap, uint64_t) : 0;
>>>> +        set_cell(&cells, size_cells,arg);
>>>> +    }
>>>> +    va_end(ap);
>>>> +
>>>> +    return fdt_property(fdt, "ranges", regs, sizeof(regs));
>>>> +}
>>>> +
>>>>  static int make_root_properties(libxl__gc *gc,
>>>>                                  const libxl_version_info *vers,
>>>>                                  void *fdt)
>>>> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>>>>      return 0;
>>>>  }
>>>>  +static int make_vpci_node(libxl__gc *gc, void *fdt,
>>>> +        const struct arch_info *ainfo,
>>>> +        struct xc_dom_image *dom)
>>>> +{
>>>> +    int res;
>>>> +    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
>>>> +    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
>>>> +    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
>>>> +
>>>> +    res = fdt_begin_node(fdt, name);
>>>> +    if (res) return res;
>>>> +
>>>> +    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
>>>> +    if (res) return res;
>>>> +
>>>> +    res = fdt_property_string(fdt, "device_type", "pci");
>>>> +    if (res) return res;
>>>> +
>>>> +    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>>>> +            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
>>>> +    if (res) return res;
>>>> +
>>>> +    res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17);
>>> 
>>> AFAICT, the "bus-range" is optional. Can you explain why we need it?
>> We need it to implement the function pci_ecam_map_bus().
> 
> Ok. Then why next question is what does the 17 mean? Is it tie to how we
> implement the vPCI in Xen or the region we reserved?
> 

Ah. I forgot to change it to 0xff. It should be 0xff. 

> [...]
> 
>>> 
>>>> +
>>>>      if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
>>>>          return;
>>>>  diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
>>>> index 3f9fff653a..78b1ddf0b8 100644
>>>> --- a/tools/libs/light/libxl_types.idl
>>>> +++ b/tools/libs/light/libxl_types.idl
>>>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>>>>        ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>>>>                                 ("vuart", libxl_vuart_type),
>>>> +                               ("vpci", libxl_defbool),
>>> 
>>> Any new addition in the structure should be accompanied with a LIBXL_HAVE_* in the libxl.h header.
>> OK.
>>> 
>>>>                                ])),
>>>>      ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
>>>>                                ])),
>>>> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
>>>> index 17dddb4cd5..ffafbeffb4 100644
>>>> --- a/tools/xl/xl_parse.c
>>>> +++ b/tools/xl/xl_parse.c
>>>> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source,
>>>>          }
>>>>          if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
>>>>              libxl_defbool_set(&b_info->u.pv.e820_host, true);
>>>> +        if (d_config->num_pcidevs)
>>>> +            libxl_defbool_set(&b_info->arch_arm.vpci, true);
>>>>      }
>>>>        if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) {
>>>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>>>> index 0a9749e768..01d13e669e 100644
>>>> --- a/xen/include/public/arch-arm.h
>>>> +++ b/xen/include/public/arch-arm.h
>>>> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t;
>>>>  #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
>>>>  #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
>>>>  +/* PCI-PCIe memory space types */
>>>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
>>>> +#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
>>> 
>>> What the size of those regions?
>> Non Prefetch Memory: Size 64 MB start at 512 MB
>> Prefetch Memory: Size 128 GB start at 36 GB
>>> 
>>>> +
>>>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
>>>> +#define GUEST_VPCI_PREFETCH_MEM_ADDR  xen_mk_ullong(0x900000000)
>>>> +#define GUEST_VPCI_MEM_ADDR           xen_mk_ullong(0x20000000)
>>> So far the memory layout defines the address in ascending order. So please add that after GUEST_RAM_BANK_BASES_*.
>> Ok.
>>> 
>>> However, if I am not mistaken that base address you provide will clash with RAM bank 1. It also seem to be pretty high which means that this will not work for 32-bit domain or on CPUs that don't allow offer large IPA bits.
>> Yes I also checked that now that it is having clash with RAM bank 1.
>> There is unused space is guest memory that we can use for Non Prefetch Memory as per below guest memory map.
>> https://gitlab.com/xen-project/fusa/fusa-docs/-/blob/master/high-level/guest-memory-layout-arm.rst
>> Proposed value:
>> Non Prefetch Memory: Size 64 MB start at 0x22001000
>> Prefetch Memory: Size 4 GB start at 4 GB.
> 
> The base address looks fine to me. However, the sizes are much smaller to what you initially suggested. Would you be able to clarify why the smaller sizes are fine?

As per current design we can have 32 device in guest therefore I thought 64 MB is sufficient ( 2 MB per device ).
> 
>>> 
>>> I think we need to start making the guest layout more dynamic. The VPCI memory space would have to go right after the end of the RAM allocated for a given guest.
>>> 
>>>> +
>>>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x2000000000)
>>>> +#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x04000000)
>>> 
>>> It would be better if the size for each region is defined right after each base.
>> OK.
>>> 
>>> Also, how did you decide the size of each region?
>> I thought 64 MB will be sufficient. I think it should be based on number of devices we can assign to the guest.
> 
> We don't have to get the size right now. What I am more interested is to have a trace about how those values were decided (even if it just saying random). This will help to make any decision if in the future we need to resize (in particular downsize) the regions.

As per Stefano suggestion in another mail he suggested 256 MB. I also think 256 MB will be sufficient ( 8 MB per device) . I will comment this in code.

Regards,
Rahul
> 
> Cheers,
> 
> -- 
> Julien Grall



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl
  2021-09-10  0:51   ` Stefano Stabellini
@ 2021-09-16 16:35     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-16 16:35 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Ian Jackson, Wei Liu,
	Anthony PERARD, Juergen Gross, Andrew Cooper, George Dunlap,
	Jan Beulich, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 10 Sep 2021, at 1:51 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> libxl will create an emulated PCI device tree node in the device tree to
>> enable the guest OS to discover the virtual PCI during guest boot.
>> Emulated PCI device tree node will only be created when there is any
>> device assigned to guest.
>> 
>> A new area has been reserved in the arm guest physical map at
>> which the VPCI bus is declared in the device tree (reg and ranges
>> parameters of the node).
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> tools/libs/light/libxl_arm.c          | 109 ++++++++++++++++++++++++++
>> tools/libs/light/libxl_types.idl      |   1 +
>> tools/xl/xl_parse.c                   |   2 +
>> xen/include/public/arch-arm.h         |  11 +++
>> xen/include/public/device_tree_defs.h |   1 +
>> 5 files changed, 124 insertions(+)
>> 
>> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
>> index e3140a6e00..a091e97e76 100644
>> --- a/tools/libs/light/libxl_arm.c
>> +++ b/tools/libs/light/libxl_arm.c
>> @@ -269,6 +269,58 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
>>     return fdt_property(fdt, "reg", regs, sizeof(regs));
>> }
>> 
>> +static int fdt_property_values(libxl__gc *gc, void *fdt,
>> +        const char *name, unsigned num_cells, ...)
>> +{
>> +    uint32_t prop[num_cells];
>> +    be32 *cells = &prop[0];
>> +    int i;
>> +    va_list ap;
>> +    uint32_t arg;
>> +
>> +    va_start(ap, num_cells);
>> +    for (i = 0 ; i < num_cells; i++) {
>> +        arg = va_arg(ap, uint32_t);
>> +        set_cell(&cells, 1, arg);
>> +    }
>> +    va_end(ap);
>> +
>> +    return fdt_property(fdt, name, prop, sizeof(prop));
>> +}
>> +
>> +static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
>> +                                    unsigned addr_cells,
>> +                                    unsigned size_cells,
>> +                                    unsigned num_regs, ...)
>> +{
>> +    uint32_t regs[num_regs*((addr_cells*2)+size_cells+1)];
>> +    be32 *cells = &regs[0];
>> +    int i;
>> +    va_list ap;
>> +    uint64_t arg;
>> +
>> +    va_start(ap, num_regs);
>> +    for (i = 0 ; i < num_regs; i++) {
>> +        /* Set the memory bit field */
>> +        arg = va_arg(ap, uint64_t);
>> +        set_cell(&cells, 1, arg);
> 
> Shouldn't this be uint32_t given that it is 1 cell exactly?

Yes I will change it to uint32_t.
> 
> 
>> +        /* Set the vpci bus address */
>> +        arg = addr_cells ? va_arg(ap, uint64_t) : 0;
>> +        set_cell(&cells, addr_cells , arg);
>> +
>> +        /* Set the cpu bus address where vpci address is mapped */
>> +        set_cell(&cells, addr_cells, arg);
>> +
>> +        /* Set the vpci size requested */
>> +        arg = size_cells ? va_arg(ap, uint64_t) : 0;
>> +        set_cell(&cells, size_cells,arg);
>                                       ^ space

Ack.
> 
> 
>> +    }
>> +    va_end(ap);
>> +
>> +    return fdt_property(fdt, "ranges", regs, sizeof(regs));
>> +}
>> +
>> static int make_root_properties(libxl__gc *gc,
>>                                 const libxl_version_info *vers,
>>                                 void *fdt)
>> @@ -668,6 +720,57 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
>>     return 0;
>> }
>> 
>> +static int make_vpci_node(libxl__gc *gc, void *fdt,
>> +        const struct arch_info *ainfo,
>> +        struct xc_dom_image *dom)
>> +{
>> +    int res;
>> +    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
>> +    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
>> +    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
>> +
>> +    res = fdt_begin_node(fdt, name);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
>> +    if (res) return res;
>> +
>> +    res = fdt_property_string(fdt, "device_type", "pci");
>> +    if (res) return res;
>> +
>> +    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>> +            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_values(gc, fdt, "bus-range", 2, 0,17);
>                                                            ^ space
> 
Ack. 
> 
>> +    if (res) return res;
>> +
>> +    res = fdt_property_cell(fdt, "#address-cells", 3);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_cell(fdt, "#size-cells", 2);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_string(fdt, "status", "okay");
>> +    if (res) return res;
>> +
>> +    res = fdt_property_vpci_ranges(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
>> +        GUEST_ROOT_SIZE_CELLS, 2,
>> +        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_ADDR, GUEST_VPCI_MEM_SIZE,
>> +        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_ADDR,
>> +        GUEST_VPCI_PREFETCH_MEM_SIZE);
>> +    if (res) return res;
>> +
>> +    res = fdt_property_values(gc, fdt, "msi-map", 4, 0, GUEST_PHANDLE_ITS,
>> +                              0, 0x10000);
>> +    if (res) return res;
> 
> I agree with Julien that we shouldn't add it now if it is not working.
Ack.
> 
> One question: what about legacy interrupts? If they are supported,
> shouldn't we have interrupts and interrupt-parent properties?

As per current design legacy interrupt will not be supported.
> 
> 
>> +    res = fdt_end_node(fdt);
>> +    if (res) return res;
>> +
>> +    return 0;
>> +}
>> +
>> static const struct arch_info *get_arch_info(libxl__gc *gc,
>>                                              const struct xc_dom_image *dom)
>> {
>> @@ -971,6 +1074,9 @@ next_resize:
>>         if (info->tee == LIBXL_TEE_TYPE_OPTEE)
>>             FDT( make_optee_node(gc, fdt) );
>> 
>> +        if (libxl_defbool_val(info->arch_arm.vpci))
>> +            FDT( make_vpci_node(gc, fdt, ainfo, dom) );
>> +
>>         if (pfdt)
>>             FDT( copy_partial_fdt(gc, fdt, pfdt) );
>> 
>> @@ -1189,6 +1295,9 @@ void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>>     /* ACPI is disabled by default */
>>     libxl_defbool_setdefault(&b_info->acpi, false);
>> 
>> +    /* VPCI is disabled by default */
>> +    libxl_defbool_setdefault(&b_info->arch_arm.vpci, false);
>> +
>>     if (b_info->type != LIBXL_DOMAIN_TYPE_PV)
>>         return;
>> 
>> diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
>> index 3f9fff653a..78b1ddf0b8 100644
>> --- a/tools/libs/light/libxl_types.idl
>> +++ b/tools/libs/light/libxl_types.idl
>> @@ -644,6 +644,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
>> 
>>     ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
>>                                ("vuart", libxl_vuart_type),
>> +                               ("vpci", libxl_defbool),
>>                               ])),
>>     ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
>>                               ])),
>> diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
>> index 17dddb4cd5..ffafbeffb4 100644
>> --- a/tools/xl/xl_parse.c
>> +++ b/tools/xl/xl_parse.c
>> @@ -1497,6 +1497,8 @@ void parse_config_data(const char *config_source,
>>         }
>>         if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
>>             libxl_defbool_set(&b_info->u.pv.e820_host, true);
>> +        if (d_config->num_pcidevs)
>> +            libxl_defbool_set(&b_info->arch_arm.vpci, true);
>>     }
>> 
>>     if (!xlu_cfg_get_list (config, "dtdev", &dtdevs, 0, 0)) {
>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
>> index 0a9749e768..01d13e669e 100644
>> --- a/xen/include/public/arch-arm.h
>> +++ b/xen/include/public/arch-arm.h
>> @@ -426,6 +426,17 @@ typedef uint64_t xen_callback_t;
>> #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
>> #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
>> 
>> +/* PCI-PCIe memory space types */
>> +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
>> +#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
>> +
>> +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
>> +#define GUEST_VPCI_PREFETCH_MEM_ADDR  xen_mk_ullong(0x900000000)
>> +#define GUEST_VPCI_MEM_ADDR           xen_mk_ullong(0x20000000)
>> +
>> +#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x2000000000)
>> +#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x04000000)
> 
> Now I get it: GUEST_VPCI_ECAM_BASE was just for the config space, not
> the aperture. The apertures are starting at 0x20000000 and 0x900000000.
Yes right.
> 
> Speaking of which, isn't GUEST_VPCI_MEM_SIZE a bit too small? There
> could be PCI devices with a <4GB requirement. Maybe 256MB?

Yes I will allocate the 256 MB.

Regards,
Rahul



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 14/14] xen/arm: Add linux,pci-domain property for hwdom if not available.
  2021-09-10  1:00   ` Stefano Stabellini
@ 2021-09-16 16:36     ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-16 16:36 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 10 Sep 2021, at 2:00 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 19 Aug 2021, Rahul Singh wrote:
>> If the property is not present in the device tree node for host bridge,
>> XEN while creating the dtb for hwdom will create this property and
>> assigns the already allocated segment to the host bridge
>> so that XEN and linux will have the same segment for the host bridges.
>> 
>> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
>> ---
>> xen/arch/arm/domain_build.c        | 18 ++++++++++++++++++
>> xen/arch/arm/pci/pci-host-common.c | 21 +++++++++++++++++++++
>> xen/include/asm-arm/pci.h          |  3 +++
>> 3 files changed, 42 insertions(+)
>> 
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index 6c86d52781..e0cf2ff19d 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -581,6 +581,24 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
>>             return res;
>>     }
>> 
>> +#ifdef CONFIG_HAS_PCI
>> +    if ( dt_device_type_is_equal(node, "pci") )
>> +    {
>> +        if ( !dt_find_property(node, "linux,pci-domain", NULL) )
>> +        {
>> +            uint16_t segment;
>> +
>> +            res = pci_get_host_bridge_segment(node, &segment);
>> +            if ( res < 0 )
>> +                return res;
>> +
>> +            res = fdt_property_cell(kinfo->fdt, "linux,pci-domain", segment);
>> +            if ( res )
>> +                return res;
>> +        }
>> +    }
>> +#endif
> 
> If param_pci_enable is false it might be possible that Xen didn't
> allocate a segment. In that case, we should just let Linux do whatever
> it wants in terms of segment allocation. So I think the code here should
> not return error if param_pci_enable is false.
> returning an error instead.

Ack. I will modify the code as per your request in next version.

Regards,
Rahul



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations
  2021-09-15 20:45           ` Stefano Stabellini
@ 2021-09-16 16:51             ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-16 16:51 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk

Hi Stefano,

> On 15 Sep 2021, at 9:45 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Wed, 15 Sep 2021, Rahul Singh wrote:
>>> On 15 Sep 2021, at 12:06 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>> On Tue, 14 Sep 2021, Rahul Singh wrote:
>>>>>> +        return NULL;
>>>>>> +
>>>>>> +    busn -= cfg->busn_start;
>>>>>> +    base = cfg->win + (busn << cfg->ops->bus_shift);
>>>>>> +
>>>>>> +    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
>>>>>> +}
>>>>> 
>>>>> I understand that the arm32 part is not implemented and not part of this
>>>>> series, that's fine. However if the plan is that arm32 will dynamically
>>>>> map each bus individually, then I imagine this function will have an
>>>>> ioremap in the arm32 version. Which means that we also need an
>>>>> unmap_bus call in struct pci_ops. I understand that pci_ecam_unmap_bus
>>>>> would be a NOP today for arm64, but I think it makes sense to have it if
>>>>> we want the API to be generic.
>>>> 
>>>> As per my understanding we don’t need pci_ecam_unmap_bus(..) as I don’t see any use case to unmap the 
>>>> bus dynamically. We can add the support for per_bus_mapping for ARM32 once we implement arm32 part.
>>>> Let me know your view on this.
>>> 
>>> In the patch titled "xen/arm: PCI host bridge discovery within XEN on
>>> ARM" there is the following in-code comment:
>>> 
>>> * On 64-bit systems, we do a single ioremap for the whole config space
>>> * since we have enough virtual address range available.  On 32-bit, we
>>> * ioremap the config space for each bus individually.
>>> *
>>> * As of now only 64-bit is supported 32-bit is not supported.
>>> 
>>> 
>>> So I take it that on arm32 we don't have enough virtual address range
>>> available, therefore we cannot ioremap the whole range. Instead, we'll
>>> have to ioremap the config space of each bus individually.
>> 
>> Yes you are right my understand is also same.
>>> 
>>> I assumed that the idea was to call ioremap and iounmap dynamically,
>>> otherwise the total amount of virtual address range required would still
>>> be the same.
>> 
>> As per my understanding for 32-bit we need per_bus mapping as we don’t have enough virtual address space in one chunk
>> but we can have virtual address space in different chunk.
> 
> Interesting. I would have assumed that the sum of all the individual
> smaller ioremaps would still be equal to one big ioremap. Maybe for
> Linux is different, but I don't think that many smaller ioremaps would
> buy us very much in Xen because it is the total ioremap virtual space
> that is too small. Or am I missing something?
> 
> 
>> I am not sure if we need to map/unmap the virtual address space for each read/write call. 
>> I just checked the Linux code[1]  and there also mapping is done once not for each read/write call.
> 
> So my guess is that for arm32 we would have to resort to dynamic
> map/unmap for each read/write call, unless there is a trick with the
> individual smaller ioremaps that I haven't spotted (e.g. maybe something
> doesn't get mapped that way?)
> 
> That said, given that we are uncertain about this and the arm32
> implementation is nowhere close, I think that we are OK to continue like
> this for this series. Maybe you could add a couple of sentences to the
> in-code comment so that if somebody wants to jump in and implement
> arm32 support they would know where to start.

I am ok with both ways adding comment in code to explain or implement the pci_ecam_add_bus(..) and pci_ecam_remove_bus() like Linux [1] and 
we can call those function in pci_read()/pci_write.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/ecam.c#n126


Regards,
Rahul


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-09-16 11:01     ` Rahul Singh
@ 2021-09-16 20:26       ` Stefano Stabellini
  2021-09-21 13:49         ` Rahul Singh
  0 siblings, 1 reply; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-16 20:26 UTC (permalink / raw)
  To: Rahul Singh
  Cc: Stefano Stabellini, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk, Jan Beulich, Paul Durrant,
	Roger Pau Monné

[-- Attachment #1: Type: text/plain, Size: 1358 bytes --]

On Thu, 16 Sep 2021, Rahul Singh wrote:
> Hi Stefano,
> 
> > On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > 
> > On Thu, 19 Aug 2021, Rahul Singh wrote:
> >> The existing VPCI support available for X86 is adapted for Arm.
> >> When the device is added to XEN via the hyper call
> >> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> >> access is added to the Xen to emulate the PCI devices config space.
> > 
> > This is done just for device discovery, right?
> > 
> > Although it is currently not implemented (and I am not asking to
> > implement it now, I am only trying to understand the architecture), it
> > would be possible to discover all PCI devices just by walking down the
> > PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we
> > have an ECAM driver.
> > 
> > I take that would be the way to implement PCI support for Dom0less?
> 
> It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot.
> If boot firmware did the enumeration,  XEN will discover the PCI device.

OK, but if the boot firmware does the enumeration, how will Xen discover
the PCI device exactly? Will Xen discover it because the PCI device will
be present in device tree explicitly (will have its own device tree
node)? Or will Xen discover it by walking the PCI bus?

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller
  2021-09-14  4:31           ` Oleksandr Andrushchenko
@ 2021-09-17  7:39             ` Rahul Singh
  0 siblings, 0 replies; 98+ messages in thread
From: Rahul Singh @ 2021-09-17  7:39 UTC (permalink / raw)
  To: Oleksandr Andrushchenko
  Cc: Stefano Stabellini, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk

Hi Oleksandr, Stefano,

> On 14 Sep 2021, at 5:31 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
> 
> 
> On 14.09.21 00:02, Stefano Stabellini wrote:
>> On Mon, 13 Sep 2021, Oleksandr Andrushchenko wrote:
>>> On 10.09.21 15:01, Rahul Singh wrote:
>>>> Hi Stefano,
>>>> 
>>>>> On 10 Sep 2021, at 12:34 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>>> 
>>>>> On Thu, 19 Aug 2021, Rahul Singh wrote:
>>>>>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>>>>>> 
>>>>>> Add support for Xilinx ZynqMP PCI host controller to map the PCI config
>>>>>> space to the XEN memory.
>>>>>> 
>>>>>> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>>>>>> ---
>>>>>> xen/arch/arm/pci/Makefile          |  1 +
>>>>>> xen/arch/arm/pci/pci-host-zynqmp.c | 59 ++++++++++++++++++++++++++++++
>>>>>> 2 files changed, 60 insertions(+)
>>>>>> create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
>>>>>> 
>>>>>> diff --git a/xen/arch/arm/pci/Makefile b/xen/arch/arm/pci/Makefile
>>>>>> index 6f32fbbe67..1d045ade01 100644
>>>>>> --- a/xen/arch/arm/pci/Makefile
>>>>>> +++ b/xen/arch/arm/pci/Makefile
>>>>>> @@ -3,3 +3,4 @@ obj-y += pci-access.o
>>>>>> obj-y += pci-host-generic.o
>>>>>> obj-y += pci-host-common.o
>>>>>> obj-y += ecam.o
>>>>>> +obj-y += pci-host-zynqmp.o
>>>>>> diff --git a/xen/arch/arm/pci/pci-host-zynqmp.c b/xen/arch/arm/pci/pci-host-zynqmp.c
>>>>>> new file mode 100644
>>>>>> index 0000000000..fe103e3855
>>>>>> --- /dev/null
>>>>>> +++ b/xen/arch/arm/pci/pci-host-zynqmp.c
>>>>>> @@ -0,0 +1,59 @@
>>>>>> +/*
>>>>>> + * Copyright (C) 2020-2021 EPAM Systems
>>>>>> + *
>>>>>> + * Based on Linux drivers/pci/controller/pci-host-common.c
>>>>>> + * Based on Linux drivers/pci/controller/pci-host-generic.c
>>>>>> + * Based on xen/arch/arm/pci/pci-host-generic.c
>>>>>> + * Copyright (C) 2014 ARM Limited Will Deacon <will.deacon@arm.com>
>>>>> Only one Copyright line per file is enough :-)
>>>>> 
>>>>> But actually all the Copyright lines with a name or a company name are
>>>>> not really required or useful, as the copyright is noted in full details
>>>>> in the commit messages (author and signed-off-by lines). I would remove
>>>>> them all from the new files added by this series.
>>>> Ok. Let me remove in next version.
>>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>>> + * it under the terms of the GNU General Public License version 2 as
>>>>>> + * published by the Free Software Foundation.
>>>>>> + *
>>>>>> + * This program is distributed in the hope that it will be useful,
>>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>>>> + * GNU General Public License for more details.
>>>>>> + *
>>>>>> + * You should have received a copy of the GNU General Public License
>>>>>> + * along with this program.  If not, see <https://urldefense.com/v3/__http://www.gnu.org/licenses/__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_3j2CfflQ$ [gnu[.]org]>.
>>>>>> + */
>>>>>> +
>>>>>> +#include <asm/device.h>
>>>>>> +#include <xen/pci.h>
>>>>>> +#include <asm/pci.h>
>>>>>> +
>>>>>> +static const struct dt_device_match gen_pci_dt_match[] = {
>>>>>> +    { .compatible = "xlnx,nwl-pcie-2.11",
>>>>>> +      .data =       &pci_generic_ecam_ops },
>>>>>> +    { },
>>>>>> +};
>>>>>> +
>>>>>> +static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
>>>>>> +{
>>>>>> +    const struct dt_device_match *of_id;
>>>>>> +    const struct pci_ecam_ops *ops;
>>>>>> +
>>>>>> +    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
>>>>> This should be superfluous
>>>> Ack. I will remove the dt_match_node(..) in next version.
>>> I am not entirely sure we need this patch at all as the main reason for its existence
>>> 
>>> was that we can run Xilinx QEMU for ZCU102. But, the final setup is not going
>>> 
>>> to be functional as legacy IRQs are not supported and ITS is not a part of ZynqMP.
>>> 
>>> So, QEMU allows to do a lot with PCI passthrough, but at the end of the day one
>>> 
>>> won't have it working...
>>> 
>>> Please consider
>>> 
>>> If we decide to remove it then
>>> 
>>> int pci_host_common_probe(struct dt_device_node *dev,
>>>                            const struct pci_ecam_ops *ops,
>>>                            int ecam_reg_idx)
>>> 
>>> doesn't need the last parameter.
>> With my open source maintainer hat on, I don't see this patch as very
>> important; from that point of view I'd be happy for it to be dropped.
>> However, it would be good to have at least one non-default host bridge
>> (doesn't have to be the Xilinx bridge), otherwise it becomes difficult
>> to understand how the generic infrastructure introduced by this series
>> could be useful.
>> 
>> Moreover, your recent comment [1] made it even more evident that it
>> would be good to have at least two different drivers to spot
>> compatibility issues between them more easily.
> 
> Don't take me wrong here ;) I still do use Xilinx QEMU for most of
> 
> the tests, so it is good for me to have this patch in the tree. But,
> 
> to be fair, Xilinx QEMU won't give you a possibility to see the fully
> 
> functional system. This is why I say the patch can be dropped.
> 
> If we add some words in the commit message about this and have the
> 
> patch in the tree I'll be more than happy

I will have this patch in my series and I will add more comments about the patch.

Regards,
Rahul
> 
>> 
>> [1] https://urldefense.com/v3/__https://marc.info/?l=xen-devel&m=163154474008598__;!!GF_29dbcQIUBPA!lAdL_CvsuMuuX9ai6cwzm3NYiT1vwIIlxGU7nezSqq_nqJk40Zz-kT44LOsemcghJ_0bKs6zpA$ [marc[.]info]



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-09-16 20:26       ` Stefano Stabellini
@ 2021-09-21 13:49         ` Rahul Singh
  2021-09-21 21:38           ` Stefano Stabellini
  0 siblings, 1 reply; 98+ messages in thread
From: Rahul Singh @ 2021-09-21 13:49 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Bertrand Marquis, Julien Grall, Volodymyr Babchuk,
	Jan Beulich, Paul Durrant, Roger Pau Monné

Hi Stefano,

> On 16 Sep 2021, at 9:26 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 16 Sep 2021, Rahul Singh wrote:
>> Hi Stefano,
>> 
>>> On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>> 
>>> On Thu, 19 Aug 2021, Rahul Singh wrote:
>>>> The existing VPCI support available for X86 is adapted for Arm.
>>>> When the device is added to XEN via the hyper call
>>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
>>>> access is added to the Xen to emulate the PCI devices config space.
>>> 
>>> This is done just for device discovery, right?
>>> 
>>> Although it is currently not implemented (and I am not asking to
>>> implement it now, I am only trying to understand the architecture), it
>>> would be possible to discover all PCI devices just by walking down the
>>> PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we
>>> have an ECAM driver.
>>> 
>>> I take that would be the way to implement PCI support for Dom0less?
>> 
>> It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot.
>> If boot firmware did the enumeration,  XEN will discover the PCI device.
> 
> OK, but if the boot firmware does the enumeration, how will Xen discover
> the PCI device exactly? Will Xen discover it because the PCI device will
> be present in device tree explicitly (will have its own device tree
> node)? Or will Xen discover it by walking the PCI bus?

Yes Xen discover it by walking the PCI bus. There is already a function in XEN scan_pci_devices() that we can use 
to discover the PCI device for dom0less system.

Regards,
Rahul


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM.
  2021-09-21 13:49         ` Rahul Singh
@ 2021-09-21 21:38           ` Stefano Stabellini
  0 siblings, 0 replies; 98+ messages in thread
From: Stefano Stabellini @ 2021-09-21 21:38 UTC (permalink / raw)
  To: Rahul Singh
  Cc: Stefano Stabellini, xen-devel, Bertrand Marquis, Julien Grall,
	Volodymyr Babchuk, Jan Beulich, Paul Durrant,
	Roger Pau Monné

[-- Attachment #1: Type: text/plain, Size: 1948 bytes --]

On Tue, 21 Sep 2021, Rahul Singh wrote:
> Hi Stefano,
> 
> > On 16 Sep 2021, at 9:26 pm, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > 
> > On Thu, 16 Sep 2021, Rahul Singh wrote:
> >> Hi Stefano,
> >> 
> >>> On 10 Sep 2021, at 1:26 am, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >>> 
> >>> On Thu, 19 Aug 2021, Rahul Singh wrote:
> >>>> The existing VPCI support available for X86 is adapted for Arm.
> >>>> When the device is added to XEN via the hyper call
> >>>> “PHYSDEVOP_pci_device_add”, VPCI handler for the config space
> >>>> access is added to the Xen to emulate the PCI devices config space.
> >>> 
> >>> This is done just for device discovery, right?
> >>> 
> >>> Although it is currently not implemented (and I am not asking to
> >>> implement it now, I am only trying to understand the architecture), it
> >>> would be possible to discover all PCI devices just by walking down the
> >>> PCI hierarchy by ourselves in Xen (no Dom0 interactions) given that we
> >>> have an ECAM driver.
> >>> 
> >>> I take that would be the way to implement PCI support for Dom0less?
> >> 
> >> It is not possible to discover PCI devices in XEN if enumeration is not done before XEN boot.
> >> If boot firmware did the enumeration,  XEN will discover the PCI device.
> > 
> > OK, but if the boot firmware does the enumeration, how will Xen discover
> > the PCI device exactly? Will Xen discover it because the PCI device will
> > be present in device tree explicitly (will have its own device tree
> > node)? Or will Xen discover it by walking the PCI bus?
> 
> Yes Xen discover it by walking the PCI bus. There is already a function in XEN scan_pci_devices() that we can use 
> to discover the PCI device for dom0less system.

OK. Please add a statement to the commit message on how dom0less support
could be implemented. It doesn't have to be implemented now of course
but it would be good to keep a note for future reference.

^ permalink raw reply	[flat|nested] 98+ messages in thread

end of thread, other threads:[~2021-09-21 21:39 UTC | newest]

Thread overview: 98+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-19 12:02 [PATCH v1 00/14] PCI devices passthrough on Arm Rahul Singh
2021-08-19 12:02 ` [PATCH v1 01/14] xen/pci: Refactor MSI code that implements MSI functionality within XEN Rahul Singh
2021-08-19 12:18   ` Julien Grall
2021-08-19 14:16     ` Rahul Singh
2021-09-07 10:01       ` Julien Grall
2021-08-24 15:53   ` Jan Beulich
2021-08-31 12:31     ` Rahul Singh
2021-08-31 13:00       ` Jan Beulich
2021-08-26 13:23   ` Daniel P. Smith
2021-08-19 12:02 ` [PATCH v1 02/14] xen/pci: solve compilation error on ARM with HAS_PCI enabled Rahul Singh
2021-08-19 12:28   ` Julien Grall
2021-08-20 10:30     ` Rahul Singh
2021-08-20 11:37       ` Julien Grall
2021-08-20 11:55         ` Jan Beulich
2021-08-20 12:10           ` Julien Grall
2021-08-20  7:01   ` Jan Beulich
2021-08-20 11:21     ` Rahul Singh
2021-09-09 13:16   ` Julien Grall
2021-08-19 12:02 ` [PATCH v1 03/14] xen/pci: solve compilation error on ARM with ACPI && HAS_PCI Rahul Singh
2021-08-20  7:06   ` Jan Beulich
2021-08-20 11:41     ` Rahul Singh
2021-08-20 11:54       ` Jan Beulich
2021-09-09  1:11         ` Stefano Stabellini
2021-09-10 10:22           ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 04/14] xen/arm: Add support for PCI init to initialize the PCI driver Rahul Singh
2021-09-07  8:20   ` Julien Grall
2021-09-10 10:47     ` Rahul Singh
2021-09-09  1:16   ` Stefano Stabellini
2021-09-10 10:32     ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 05/14] xen/arm: PCI host bridge discovery within XEN on ARM Rahul Singh
2021-09-07  9:05   ` Julien Grall
2021-09-10 11:22     ` Rahul Singh
2021-09-10 11:53       ` Julien Grall
2021-09-09 22:54   ` Stefano Stabellini
2021-09-10 11:53     ` Rahul Singh
2021-09-13 14:52   ` Oleksandr Andrushchenko
2021-09-13 20:23     ` Stefano Stabellini
2021-09-14  4:35       ` Oleksandr Andrushchenko
2021-09-14  7:53   ` Oleksandr Andrushchenko
2021-08-19 12:02 ` [PATCH v1 06/14] xen/arm: Add support for PCI ecam operations Rahul Singh
2021-09-09 11:32   ` Julien Grall
2021-09-14  8:13     ` Rahul Singh
2021-09-09 23:21   ` Stefano Stabellini
2021-09-14 11:13     ` Rahul Singh
2021-09-14 23:06       ` Stefano Stabellini
2021-09-15 16:38         ` Rahul Singh
2021-09-15 20:45           ` Stefano Stabellini
2021-09-16 16:51             ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 07/14] xen/arm: Add support for Xilinx ZynqMP PCI host controller Rahul Singh
2021-09-09 23:34   ` Stefano Stabellini
2021-09-10 12:01     ` Rahul Singh
2021-09-13 14:46       ` Oleksandr Andrushchenko
2021-09-13 21:02         ` Stefano Stabellini
2021-09-14  4:31           ` Oleksandr Andrushchenko
2021-09-17  7:39             ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 08/14] xen:arm: Implement pci access functions Rahul Singh
2021-09-09 23:41   ` Stefano Stabellini
2021-09-14 16:05     ` Rahul Singh
2021-09-14 22:40       ` Stefano Stabellini
2021-09-15  7:54       ` Oleksandr Andrushchenko
2021-09-15 10:47         ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 09/14] xen/arm: Add cmdline boot option "pci=on" Rahul Singh
2021-08-19 12:09   ` Jan Beulich
2021-08-19 12:31   ` Julien Grall
2021-08-20 12:19     ` Rahul Singh
2021-08-20 14:34       ` Julien Grall
2021-08-20 14:37         ` Jan Beulich
2021-09-09 23:46           ` Stefano Stabellini
2021-09-09 23:48   ` Stefano Stabellini
2021-08-19 12:02 ` [PATCH v1 10/14] xen/arm: Discovering PCI devices and add the PCI devices in XEN Rahul Singh
2021-08-19 12:35   ` Julien Grall
2021-08-19 13:40     ` Jan Beulich
2021-08-20 13:05     ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 11/14] xen/arm: Enable the existing x86 virtual PCI support for ARM Rahul Singh
2021-08-24 16:09   ` Jan Beulich
2021-08-25  5:44     ` Oleksandr Andrushchenko
2021-08-25  6:35       ` Jan Beulich
2021-09-09 13:50   ` Julien Grall
2021-09-16 10:46     ` Rahul Singh
2021-09-10  0:26   ` Stefano Stabellini
2021-09-16 11:01     ` Rahul Singh
2021-09-16 20:26       ` Stefano Stabellini
2021-09-21 13:49         ` Rahul Singh
2021-09-21 21:38           ` Stefano Stabellini
2021-08-19 12:02 ` [PATCH v1 12/14] arm/libxl: Emulated PCI device tree node in libxl Rahul Singh
2021-08-19 13:00   ` Julien Grall
2021-08-20 16:03     ` Rahul Singh
2021-09-09 13:59       ` Julien Grall
2021-09-16 16:16         ` Rahul Singh
2021-09-10  0:51   ` Stefano Stabellini
2021-09-16 16:35     ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 13/14] xen/arm: Fixed error when PCI device is assigned to guest Rahul Singh
2021-08-19 12:12   ` Jan Beulich
2021-08-19 12:40     ` Julien Grall
2021-08-20 17:01     ` Rahul Singh
2021-08-19 12:02 ` [PATCH v1 14/14] xen/arm: Add linux,pci-domain property for hwdom if not available Rahul Singh
2021-09-10  1:00   ` Stefano Stabellini
2021-09-16 16:36     ` Rahul Singh

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.