[PATCH v11 0/7] PCI: qcom: Fix higher MSI vectors handling

[PATCH v11 0/7] PCI: qcom: Fix higher MSI vectors handling

[Dmitry Baryshkov]

I have replied with my Tested-by to the patch at [2], which has landed
in the linux-next as the commit 20f1bfb8dd62 ("PCI: qcom:
Add support for handling MSIs from 8 endpoints"). However lately I
noticed that during the tests I still had 'pcie_pme=nomsi', so the
device was not forced to use higher MSI vectors.

After removing this option I noticed that hight MSI vectors are not
delivered on tested platforms. After additional research I stumbled upon
a patch in msm-4.14 ([1]), which describes that each group of MSI
vectors is mapped to the separate interrupt. Implement corresponding
mapping.

The first patch in the series is a revert of  [2] (landed in pci-next).
Either both patches should be applied or both should be dropped.

Patchseries dependecies: [3] (for the schema change).

Changes since v10:
 - Remove has_split_msi_irqs flag. Trust DT and use split MSI IRQs if
   they are described in the DT. This removes the need for the
   pcie-qcom.c changes (everything is handled by the core (suggested by
   Johan).

Changes since v9:
 - Relax requirements and stop validating the DT. If the has_split_msi
   was specified, parse as many msiN irqs as specified in DT. If there
   are none, fallback to the single "msi" IRQ.

Changes since v8:
 - Fix typos noted by Bjorn Helgaas
 - Add missing links to the patch 1 (revert)
 - Fix sm8250 interrupt-names (Johan)
 - Specify num_vectors in qcom configuration data (Johan)
 - Rework parsing of MSI IRQs (Johan)

Changes since v7:
 - Move code back to the dwc core driver (as required by Rob),
 - Change dt schema to require either a single "msi" interrupt or an
   array of "msi0", "msi1", ... "msi7" IRQs. Disallow specifying a
   part of the array (the DT should specify the exact amount of MSI IRQs
   allowing fallback to a single "msi" IRQ),
 - Fix in the DWC init code for the dma_mapping_error() return value.

Changes since v6:
 - Fix indentation of the arguments as requested by Stanimir

Changes since v5:
 - Fixed commit subject and in-comment code according to Bjorn's
   suggestion,
 - Changed variable idx to i to follow dw_handle_msi_irq() style.

Changes since v4:
 - Fix the minItems/maxItems properties in the YAML schema.

Changes since v3:
 - Reimplement MSI handling scheme in the Qualcomm host controller
   driver.

Changes since v2:
 - Fix and rephrase commit message for patch 2.

Changes since v1:
 - Split a huge patch into three patches as suggested by Bjorn Helgaas
 - snps,dw-pcie removal is now part of [3]

[1] https://git.codelinaro.org/clo/la/kernel/msm-4.14/-/commit/671a3d5f129f4bfe477152292ada2194c8440d22
[2] https://lore.kernel.org/linux-arm-msm/20211214101319.25258-1-manivannan.sadhasivam@linaro.org/
[3] https://lore.kernel.org/linux-arm-msm/20220422211002.2012070-1-dmitry.baryshkov@linaro.org/


Dmitry Baryshkov (7):
  PCI: dwc: Convert msi_irq to the array
  PCI: dwc: split MSI IRQ parsing/allocation to a separate function
  PCI: dwc: Handle MSIs routed to multiple GIC interrupts
  PCI: dwc: Implement special ISR handler for split MSI IRQ setup
  dt-bindings: PCI: qcom: Support additional MSI interrupts
  arm64: dts: qcom: sm8250: provide additional MSI interrupts
  dt-bindings: mfd: qcom,qca639x: add binding for QCA639x defvice

 .../devicetree/bindings/mfd/qcom,qca639x.yaml |  84 ++++++
 .../devicetree/bindings/pci/qcom,pcie.yaml    |  53 +++-
 arch/arm64/boot/dts/qcom/sm8250.dtsi          |  12 +-
 drivers/pci/controller/dwc/pci-dra7xx.c       |   2 +-
 drivers/pci/controller/dwc/pci-exynos.c       |   2 +-
 .../pci/controller/dwc/pcie-designware-host.c | 239 +++++++++++++-----
 drivers/pci/controller/dwc/pcie-designware.h  |   2 +-
 drivers/pci/controller/dwc/pcie-keembay.c     |   2 +-
 drivers/pci/controller/dwc/pcie-spear13xx.c   |   2 +-
 drivers/pci/controller/dwc/pcie-tegra194.c    |   2 +-
 10 files changed, 326 insertions(+), 74 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/mfd/qcom,qca639x.yaml

-- 
2.35.1

[PATCH v11 1/7] PCI: dwc: Convert msi_irq to the array

[Dmitry Baryshkov]

Qualcomm version of DWC PCIe controller supports more than 32 MSI
interrupts, but they are routed to separate interrupts in groups of 32
vectors. To support such configuration, change the msi_irq field into an
array. Let the DWC core handle all interrupts that were set in this
array.

Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
---
 drivers/pci/controller/dwc/pci-dra7xx.c       |  2 +-
 drivers/pci/controller/dwc/pci-exynos.c       |  2 +-
 .../pci/controller/dwc/pcie-designware-host.c | 30 +++++++++++--------
 drivers/pci/controller/dwc/pcie-designware.h  |  2 +-
 drivers/pci/controller/dwc/pcie-keembay.c     |  2 +-
 drivers/pci/controller/dwc/pcie-spear13xx.c   |  2 +-
 drivers/pci/controller/dwc/pcie-tegra194.c    |  2 +-
 7 files changed, 24 insertions(+), 18 deletions(-)

diff --git a/drivers/pci/controller/dwc/pci-dra7xx.c b/drivers/pci/controller/dwc/pci-dra7xx.c
index dfcdeb432dc8..0919c96dcdbd 100644
--- a/drivers/pci/controller/dwc/pci-dra7xx.c
+++ b/drivers/pci/controller/dwc/pci-dra7xx.c
@@ -483,7 +483,7 @@ static int dra7xx_add_pcie_port(struct dra7xx_pcie *dra7xx,
 		return pp->irq;
 
 	/* MSI IRQ is muxed */
-	pp->msi_irq = -ENODEV;
+	pp->msi_irq[0] = -ENODEV;
 
 	ret = dra7xx_pcie_init_irq_domain(pp);
 	if (ret < 0)
diff --git a/drivers/pci/controller/dwc/pci-exynos.c b/drivers/pci/controller/dwc/pci-exynos.c
index 467c8d1cd7e4..4f2010bd9cd7 100644
--- a/drivers/pci/controller/dwc/pci-exynos.c
+++ b/drivers/pci/controller/dwc/pci-exynos.c
@@ -292,7 +292,7 @@ static int exynos_add_pcie_port(struct exynos_pcie *ep,
 	}
 
 	pp->ops = &exynos_pcie_host_ops;
-	pp->msi_irq = -ENODEV;
+	pp->msi_irq[0] = -ENODEV;
 
 	ret = dw_pcie_host_init(pp);
 	if (ret) {
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index af91fe69f542..8dd913f69de7 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -257,8 +257,11 @@ int dw_pcie_allocate_domains(struct pcie_port *pp)
 
 static void dw_pcie_free_msi(struct pcie_port *pp)
 {
-	if (pp->msi_irq > 0)
-		irq_set_chained_handler_and_data(pp->msi_irq, NULL, NULL);
+	u32 ctrl;
+
+	for (ctrl = 0; ctrl < MAX_MSI_CTRLS; ctrl++)
+		if (pp->msi_irq[ctrl] > 0)
+			irq_set_chained_handler_and_data(pp->msi_irq[ctrl], NULL, NULL);
 
 	irq_domain_remove(pp->msi_domain);
 	irq_domain_remove(pp->irq_domain);
@@ -368,13 +371,15 @@ int dw_pcie_host_init(struct pcie_port *pp)
 			for (ctrl = 0; ctrl < num_ctrls; ctrl++)
 				pp->irq_mask[ctrl] = ~0;
 
-			if (!pp->msi_irq) {
-				pp->msi_irq = platform_get_irq_byname_optional(pdev, "msi");
-				if (pp->msi_irq < 0) {
-					pp->msi_irq = platform_get_irq(pdev, 0);
-					if (pp->msi_irq < 0)
-						return pp->msi_irq;
+			if (!pp->msi_irq[0]) {
+				int irq = platform_get_irq_byname_optional(pdev, "msi");
+
+				if (irq < 0) {
+					irq = platform_get_irq(pdev, 0);
+					if (irq < 0)
+						return irq;
 				}
+				pp->msi_irq[0] = irq;
 			}
 
 			pp->msi_irq_chip = &dw_pci_msi_bottom_irq_chip;
@@ -383,10 +388,11 @@ int dw_pcie_host_init(struct pcie_port *pp)
 			if (ret)
 				return ret;
 
-			if (pp->msi_irq > 0)
-				irq_set_chained_handler_and_data(pp->msi_irq,
-							    dw_chained_msi_isr,
-							    pp);
+			for (ctrl = 0; ctrl < num_ctrls; ctrl++)
+				if (pp->msi_irq[ctrl] > 0)
+					irq_set_chained_handler_and_data(pp->msi_irq[ctrl],
+									 dw_chained_msi_isr,
+									 pp);
 
 			ret = dma_set_mask(pci->dev, DMA_BIT_MASK(32));
 			if (ret)
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 7d6e9b7576be..9c1a38b0a6b3 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -187,7 +187,7 @@ struct pcie_port {
 	u32			io_size;
 	int			irq;
 	const struct dw_pcie_host_ops *ops;
-	int			msi_irq;
+	int			msi_irq[MAX_MSI_CTRLS];
 	struct irq_domain	*irq_domain;
 	struct irq_domain	*msi_domain;
 	u16			msi_msg;
diff --git a/drivers/pci/controller/dwc/pcie-keembay.c b/drivers/pci/controller/dwc/pcie-keembay.c
index 1ac29a6eef22..297e6e926c00 100644
--- a/drivers/pci/controller/dwc/pcie-keembay.c
+++ b/drivers/pci/controller/dwc/pcie-keembay.c
@@ -338,7 +338,7 @@ static int keembay_pcie_add_pcie_port(struct keembay_pcie *pcie,
 	int ret;
 
 	pp->ops = &keembay_pcie_host_ops;
-	pp->msi_irq = -ENODEV;
+	pp->msi_irq[0] = -ENODEV;
 
 	ret = keembay_pcie_setup_msi_irq(pcie);
 	if (ret)
diff --git a/drivers/pci/controller/dwc/pcie-spear13xx.c b/drivers/pci/controller/dwc/pcie-spear13xx.c
index 1569e82b5568..cc7776833810 100644
--- a/drivers/pci/controller/dwc/pcie-spear13xx.c
+++ b/drivers/pci/controller/dwc/pcie-spear13xx.c
@@ -172,7 +172,7 @@ static int spear13xx_add_pcie_port(struct spear13xx_pcie *spear13xx_pcie,
 	}
 
 	pp->ops = &spear13xx_pcie_host_ops;
-	pp->msi_irq = -ENODEV;
+	pp->msi_irq[0] = -ENODEV;
 
 	ret = dw_pcie_host_init(pp);
 	if (ret) {
diff --git a/drivers/pci/controller/dwc/pcie-tegra194.c b/drivers/pci/controller/dwc/pcie-tegra194.c
index cc2678490162..7056072637ab 100644
--- a/drivers/pci/controller/dwc/pcie-tegra194.c
+++ b/drivers/pci/controller/dwc/pcie-tegra194.c
@@ -2262,7 +2262,7 @@ static void tegra194_pcie_shutdown(struct platform_device *pdev)
 
 	disable_irq(pcie->pci.pp.irq);
 	if (IS_ENABLED(CONFIG_PCI_MSI))
-		disable_irq(pcie->pci.pp.msi_irq);
+		disable_irq(pcie->pci.pp.msi_irq[0]);
 
 	tegra194_pcie_pme_turnoff(pcie);
 	tegra_pcie_unconfig_controller(pcie);
-- 
2.35.1

[PATCH v11 2/7] PCI: dwc: split MSI IRQ parsing/allocation to a separate function

[Dmitry Baryshkov]

Split handling of MSI host IRQs to a separate dw_pcie_msi_host_init()
function. The code is complex enough to warrant a separate function.

Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
---
 .../pci/controller/dwc/pcie-designware-host.c | 98 +++++++++++--------
 1 file changed, 56 insertions(+), 42 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 8dd913f69de7..a076abe6611c 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -288,6 +288,60 @@ static void dw_pcie_msi_init(struct pcie_port *pp)
 	dw_pcie_writel_dbi(pci, PCIE_MSI_ADDR_HI, upper_32_bits(msi_target));
 }
 
+static int dw_pcie_msi_host_init(struct pcie_port *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct device *dev = pci->dev;
+	struct platform_device *pdev = to_platform_device(dev);
+	int ret;
+	u32 ctrl, num_ctrls;
+
+	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
+	for (ctrl = 0; ctrl < num_ctrls; ctrl++)
+		pp->irq_mask[ctrl] = ~0;
+
+	if (!pp->msi_irq[0]) {
+		int irq = platform_get_irq_byname_optional(pdev, "msi");
+
+		if (irq < 0) {
+			irq = platform_get_irq(pdev, 0);
+			if (irq < 0)
+				return irq;
+		}
+		pp->msi_irq[0] = irq;
+	}
+
+	pp->msi_irq_chip = &dw_pci_msi_bottom_irq_chip;
+
+	ret = dw_pcie_allocate_domains(pp);
+	if (ret)
+		return ret;
+
+	for (ctrl = 0; ctrl < num_ctrls; ctrl++)
+		if (pp->msi_irq[ctrl] > 0)
+			irq_set_chained_handler_and_data(pp->msi_irq[ctrl],
+							 dw_chained_msi_isr,
+							 pp);
+
+	ret = dma_set_mask(pci->dev, DMA_BIT_MASK(32));
+	if (ret)
+		dev_warn(pci->dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
+
+	pp->msi_data = dma_map_single_attrs(pci->dev, &pp->msi_msg,
+					    sizeof(pp->msi_msg),
+					    DMA_FROM_DEVICE,
+					    DMA_ATTR_SKIP_CPU_SYNC);
+	ret = dma_mapping_error(pci->dev, pp->msi_data);
+	if (ret) {
+		dev_err(pci->dev, "Failed to map MSI data\n");
+		pp->msi_data = 0;
+		dw_pcie_free_msi(pp);
+		return ret;
+	}
+
+	return 0;
+}
+
 int dw_pcie_host_init(struct pcie_port *pp)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
@@ -365,49 +419,9 @@ int dw_pcie_host_init(struct pcie_port *pp)
 			if (ret < 0)
 				return ret;
 		} else if (pp->has_msi_ctrl) {
-			u32 ctrl, num_ctrls;
-
-			num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
-			for (ctrl = 0; ctrl < num_ctrls; ctrl++)
-				pp->irq_mask[ctrl] = ~0;
-
-			if (!pp->msi_irq[0]) {
-				int irq = platform_get_irq_byname_optional(pdev, "msi");
-
-				if (irq < 0) {
-					irq = platform_get_irq(pdev, 0);
-					if (irq < 0)
-						return irq;
-				}
-				pp->msi_irq[0] = irq;
-			}
-
-			pp->msi_irq_chip = &dw_pci_msi_bottom_irq_chip;
-
-			ret = dw_pcie_allocate_domains(pp);
-			if (ret)
+			ret = dw_pcie_msi_host_init(pp);
+			if (ret < 0)
 				return ret;
-
-			for (ctrl = 0; ctrl < num_ctrls; ctrl++)
-				if (pp->msi_irq[ctrl] > 0)
-					irq_set_chained_handler_and_data(pp->msi_irq[ctrl],
-									 dw_chained_msi_isr,
-									 pp);
-
-			ret = dma_set_mask(pci->dev, DMA_BIT_MASK(32));
-			if (ret)
-				dev_warn(pci->dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");
-
-			pp->msi_data = dma_map_single_attrs(pci->dev, &pp->msi_msg,
-						      sizeof(pp->msi_msg),
-						      DMA_FROM_DEVICE,
-						      DMA_ATTR_SKIP_CPU_SYNC);
-			ret = dma_mapping_error(pci->dev, pp->msi_data);
-			if (ret) {
-				dev_err(pci->dev, "Failed to map MSI data\n");
-				pp->msi_data = 0;
-				goto err_free_msi;
-			}
 		}
 	}
 
-- 
2.35.1

[PATCH v11 3/7] PCI: dwc: Handle MSIs routed to multiple GIC interrupts

[Dmitry Baryshkov]

On some of Qualcomm platforms each group of 32 MSI vectors is routed to the
separate GIC interrupt. Implement support for such configurations by
parsing "msi0" ... "msiN" interrupts and attaching them to the chained
handler.

Note, that if DT doesn't list an array of MSI interrupts and uses single
"msi" IRQ, the driver will limit the amount of supported MSI vectors
accordingly (to 32).

Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
---
 .../pci/controller/dwc/pcie-designware-host.c | 58 +++++++++++++++++--
 1 file changed, 54 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index a076abe6611c..381bc24d5715 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -288,6 +288,43 @@ static void dw_pcie_msi_init(struct pcie_port *pp)
 	dw_pcie_writel_dbi(pci, PCIE_MSI_ADDR_HI, upper_32_bits(msi_target));
 }
 
+static const char * const split_msi_names[] = {
+	"msi0", "msi1", "msi2", "msi3",
+	"msi4", "msi5", "msi6", "msi7",
+};
+
+static int dw_pcie_parse_split_msi_irq(struct pcie_port *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct device *dev = pci->dev;
+	struct platform_device *pdev = to_platform_device(dev);
+	int irq;
+	u32 ctrl;
+
+	irq = platform_get_irq_byname_optional(pdev, split_msi_names[0]);
+	if (irq == -ENXIO)
+		return -ENXIO;
+
+	pp->msi_irq[0] = irq;
+
+	/* Parse as many IRQs as described in the DTS. */
+	for (ctrl = 1; ctrl < MAX_MSI_CTRLS; ctrl++) {
+		irq = platform_get_irq_byname_optional(pdev, split_msi_names[ctrl]);
+		if (irq == -ENXIO)
+			break;
+		if (irq < 0)
+			return dev_err_probe(dev, irq,
+					     "Failed to parse MSI IRQ '%s'\n",
+					     split_msi_names[ctrl]);
+
+		pp->msi_irq[ctrl] = irq;
+	}
+
+	pp->num_vectors = ctrl * MAX_MSI_IRQS_PER_CTRL;
+
+	return 0;
+}
+
 static int dw_pcie_msi_host_init(struct pcie_port *pp)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
@@ -295,22 +332,34 @@ static int dw_pcie_msi_host_init(struct pcie_port *pp)
 	struct platform_device *pdev = to_platform_device(dev);
 	int ret;
 	u32 ctrl, num_ctrls;
+	bool has_split_msi_irq = false;
 
-	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
-	for (ctrl = 0; ctrl < num_ctrls; ctrl++)
+	for (ctrl = 0; ctrl < MAX_MSI_CTRLS; ctrl++)
 		pp->irq_mask[ctrl] = ~0;
 
+	if (!pp->msi_irq[0]) {
+		ret = dw_pcie_parse_split_msi_irq(pp);
+		if (ret < 0 && ret != -ENXIO)
+			return ret;
+	}
+
+	if (!pp->num_vectors)
+		pp->num_vectors = MSI_DEF_NUM_VECTORS;
+	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
+
 	if (!pp->msi_irq[0]) {
 		int irq = platform_get_irq_byname_optional(pdev, "msi");
 
 		if (irq < 0) {
 			irq = platform_get_irq(pdev, 0);
 			if (irq < 0)
-				return irq;
+				return dev_err_probe(dev, irq, "Failed to parse MSI irq\n");
 		}
 		pp->msi_irq[0] = irq;
 	}
 
+	dev_dbg(dev, "Using %d MSI vectors\n", pp->num_vectors);
+
 	pp->msi_irq_chip = &dw_pci_msi_bottom_irq_chip;
 
 	ret = dw_pcie_allocate_domains(pp);
@@ -407,7 +456,8 @@ int dw_pcie_host_init(struct pcie_port *pp)
 				     of_property_read_bool(np, "msi-parent") ||
 				     of_property_read_bool(np, "msi-map"));
 
-		if (!pp->num_vectors) {
+		/* for the has_msi_ctrl the default assignment is handled inside dw_pcie_msi_host_init() */
+		if (!pp->has_msi_ctrl && !pp->num_vectors) {
 			pp->num_vectors = MSI_DEF_NUM_VECTORS;
 		} else if (pp->num_vectors > MAX_MSI_IRQS) {
 			dev_err(dev, "Invalid number of vectors\n");
-- 
2.35.1

[PATCH v11 4/7] PCI: dwc: Implement special ISR handler for split MSI IRQ setup

[Dmitry Baryshkov]

If the PCIe DWC controller uses split MSI IRQs for reporting MSI
vectors, it is possible to detect, which group triggered the interrupt.
Provide an optimized version of MSI ISR handler that will handle just a
single MSI group instead of handling all of them.

Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
---
 .../pci/controller/dwc/pcie-designware-host.c | 85 ++++++++++++++-----
 1 file changed, 64 insertions(+), 21 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 381bc24d5715..20ab8a0c6359 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -52,34 +52,42 @@ static struct msi_domain_info dw_pcie_msi_domain_info = {
 	.chip	= &dw_pcie_msi_irq_chip,
 };
 
+static inline irqreturn_t dw_handle_single_msi_group(struct pcie_port *pp, int i)
+{
+	int pos;
+	unsigned long val;
+	u32 status;
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+
+	status = dw_pcie_readl_dbi(pci, PCIE_MSI_INTR0_STATUS +
+				   (i * MSI_REG_CTRL_BLOCK_SIZE));
+	if (!status)
+		return IRQ_NONE;
+
+	val = status;
+	pos = 0;
+	while ((pos = find_next_bit(&val, MAX_MSI_IRQS_PER_CTRL,
+				    pos)) != MAX_MSI_IRQS_PER_CTRL) {
+		generic_handle_domain_irq(pp->irq_domain,
+					  (i * MAX_MSI_IRQS_PER_CTRL) +
+					  pos);
+		pos++;
+	}
+
+	return IRQ_HANDLED;
+}
+
 /* MSI int handler */
 irqreturn_t dw_handle_msi_irq(struct pcie_port *pp)
 {
-	int i, pos;
-	unsigned long val;
-	u32 status, num_ctrls;
+	int i;
+	u32 num_ctrls;
 	irqreturn_t ret = IRQ_NONE;
-	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
 
 	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
 
-	for (i = 0; i < num_ctrls; i++) {
-		status = dw_pcie_readl_dbi(pci, PCIE_MSI_INTR0_STATUS +
-					   (i * MSI_REG_CTRL_BLOCK_SIZE));
-		if (!status)
-			continue;
-
-		ret = IRQ_HANDLED;
-		val = status;
-		pos = 0;
-		while ((pos = find_next_bit(&val, MAX_MSI_IRQS_PER_CTRL,
-					    pos)) != MAX_MSI_IRQS_PER_CTRL) {
-			generic_handle_domain_irq(pp->irq_domain,
-						  (i * MAX_MSI_IRQS_PER_CTRL) +
-						  pos);
-			pos++;
-		}
-	}
+	for (i = 0; i < num_ctrls; i++)
+		ret |= dw_handle_single_msi_group(pp, i);
 
 	return ret;
 }
@@ -98,6 +106,38 @@ static void dw_chained_msi_isr(struct irq_desc *desc)
 	chained_irq_exit(chip, desc);
 }
 
+static void dw_split_msi_isr(struct irq_desc *desc)
+{
+	struct irq_chip *chip = irq_desc_get_chip(desc);
+	int irq = irq_desc_get_irq(desc);
+	struct pcie_port *pp;
+	int i;
+	u32 num_ctrls;
+	struct dw_pcie *pci;
+
+	chained_irq_enter(chip, desc);
+
+	pp = irq_desc_get_handler_data(desc);
+	pci = to_dw_pcie_from_pp(pp);
+
+	/*
+	 * Unlike generic dw_handle_msi_irq(), we can determine which group of
+	 * MSIs triggered the IRQ, so process just that group.
+	 */
+	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
+
+	for (i = 0; i < num_ctrls; i++) {
+		if (pp->msi_irq[i] == irq) {
+			dw_handle_single_msi_group(pp, i);
+			break;
+		}
+	}
+
+	WARN_ON_ONCE(i == num_ctrls);
+
+	chained_irq_exit(chip, desc);
+}
+
 static void dw_pci_setup_msi_msg(struct irq_data *d, struct msi_msg *msg)
 {
 	struct pcie_port *pp = irq_data_get_irq_chip_data(d);
@@ -341,6 +381,8 @@ static int dw_pcie_msi_host_init(struct pcie_port *pp)
 		ret = dw_pcie_parse_split_msi_irq(pp);
 		if (ret < 0 && ret != -ENXIO)
 			return ret;
+		else if (!ret)
+			has_split_msi_irq = true;
 	}
 
 	if (!pp->num_vectors)
@@ -369,6 +411,7 @@ static int dw_pcie_msi_host_init(struct pcie_port *pp)
 	for (ctrl = 0; ctrl < num_ctrls; ctrl++)
 		if (pp->msi_irq[ctrl] > 0)
 			irq_set_chained_handler_and_data(pp->msi_irq[ctrl],
+							 has_split_msi_irq ? dw_split_msi_isr :
 							 dw_chained_msi_isr,
 							 pp);
 
-- 
2.35.1

[PATCH v11 5/7] dt-bindings: PCI: qcom: Support additional MSI interrupts

[Dmitry Baryshkov]

On Qualcomm platforms each group of 32 MSI vectors is routed to the
separate GIC interrupt. Document mapping of additional interrupts.

Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
---
 .../devicetree/bindings/pci/qcom,pcie.yaml    | 53 +++++++++++++++++--
 1 file changed, 50 insertions(+), 3 deletions(-)

diff --git a/Documentation/devicetree/bindings/pci/qcom,pcie.yaml b/Documentation/devicetree/bindings/pci/qcom,pcie.yaml
index 0b69b12b849e..fe8f9a62a665 100644
--- a/Documentation/devicetree/bindings/pci/qcom,pcie.yaml
+++ b/Documentation/devicetree/bindings/pci/qcom,pcie.yaml
@@ -43,11 +43,12 @@ properties:
     maxItems: 5
 
   interrupts:
-    maxItems: 1
+    minItems: 1
+    maxItems: 8
 
   interrupt-names:
-    items:
-      - const: msi
+    minItems: 1
+    maxItems: 8
 
   # Common definitions for clocks, clock-names and reset.
   # Platform constraints are described later.
@@ -623,6 +624,52 @@ allOf:
         - resets
         - reset-names
 
+    # On newer chipsets support either 1 or 8 msi interrupts
+    # On older chipsets it's always 1 msi interrupt
+  - if:
+      properties:
+        compatibles:
+          contains:
+            enum:
+              - qcom,pcie-msm8996
+              - qcom,pcie-sc7280
+              - qcom,pcie-sc8180x
+              - qcom,pcie-sdm845
+              - qcom,pcie-sm8150
+              - qcom,pcie-sm8250
+              - qcom,pcie-sm8450-pcie0
+              - qcom,pcie-sm8450-pcie1
+    then:
+      oneOf:
+        - properties:
+            interrupts:
+              maxItems: 1
+            interrupt-names:
+              maxItems: 1
+              items:
+                - const: msi
+        - properties:
+            interrupts:
+              minItems: 8
+            interrupt-names:
+              minItems: 8
+              items:
+                - const: msi0
+                - const: msi1
+                - const: msi2
+                - const: msi3
+                - const: msi4
+                - const: msi5
+                - const: msi6
+                - const: msi7
+    else:
+      properties:
+        interrupts:
+          maxItems: 1
+        interrupt-names:
+          items:
+            - const: msi
+
 unevaluatedProperties: false
 
 examples:
-- 
2.35.1

[PATCH v11 6/7] arm64: dts: qcom: sm8250: provide additional MSI interrupts

[Dmitry Baryshkov]

On SM8250 each group of MSI interrupts is mapped to the separate host
interrupt. Describe each of interrupts in the device tree for PCIe0
host.

Tested on Qualcomm RB5 platform with first group of MSI interrupts being
used by the PME and attached ath11k WiFi chip using second group of MSI
interrupts.

Reviewed-by: Johan Hovold <johan+linaro@kernel.org>
Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
---
 arch/arm64/boot/dts/qcom/sm8250.dtsi | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/boot/dts/qcom/sm8250.dtsi b/arch/arm64/boot/dts/qcom/sm8250.dtsi
index 410272a1e19b..523a035ffc5f 100644
--- a/arch/arm64/boot/dts/qcom/sm8250.dtsi
+++ b/arch/arm64/boot/dts/qcom/sm8250.dtsi
@@ -1807,8 +1807,16 @@ pcie0: pci@1c00000 {
 			ranges = <0x01000000 0x0 0x60200000 0 0x60200000 0x0 0x100000>,
 				 <0x02000000 0x0 0x60300000 0 0x60300000 0x0 0x3d00000>;
 
-			interrupts = <GIC_SPI 141 IRQ_TYPE_LEVEL_HIGH>;
-			interrupt-names = "msi";
+			interrupts = <GIC_SPI 141 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 142 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 143 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 144 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 145 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 146 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 147 IRQ_TYPE_LEVEL_HIGH>,
+				     <GIC_SPI 148 IRQ_TYPE_LEVEL_HIGH>;
+			interrupt-names = "msi0", "msi1", "msi2", "msi3",
+					  "msi4", "msi5", "msi6", "msi7";
 			#interrupt-cells = <1>;
 			interrupt-map-mask = <0 0 0 0x7>;
 			interrupt-map = <0 0 0 1 &intc 0 149 IRQ_TYPE_LEVEL_HIGH>, /* int_a */
-- 
2.35.1

[PATCH v11 7/7] dt-bindings: mfd: qcom,qca639x: add binding for QCA639x defvice

[Dmitry Baryshkov]

Qualcomm QCA639x is a family of WiFi + Bluetooth SoCs, with BT part
being controlled through the UART and WiFi being present on PCIe bus.
Both blocks share common power sources. Add binding to describe power
sequencing required to power up this device.

Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
---
 .../devicetree/bindings/mfd/qcom,qca639x.yaml | 84 +++++++++++++++++++
 1 file changed, 84 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/mfd/qcom,qca639x.yaml

diff --git a/Documentation/devicetree/bindings/mfd/qcom,qca639x.yaml b/Documentation/devicetree/bindings/mfd/qcom,qca639x.yaml
new file mode 100644
index 000000000000..d43c75da136f
--- /dev/null
+++ b/Documentation/devicetree/bindings/mfd/qcom,qca639x.yaml
@@ -0,0 +1,84 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/mfd/qcom,qca639x.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: Qualcomm QCA639x WiFi + Bluetoot SoC bindings
+
+maintainers:
+  - Andy Gross <agross@kernel.org>
+  - Bjorn Andersson <bjorn.andersson@linaro.org>
+
+description: |
+  This binding describes thes Qualcomm QCA6390 or QCA6391 power supplies and
+  enablement pins.
+
+properties:
+  compatible:
+    const: qcom,qca639x
+
+  '#power-domain-cells':
+    const: 0
+
+  pinctrl-0: true
+  pinctrl-1: true
+
+  pinctrl-names:
+    items:
+      - const: default
+      - const: active
+
+  vddaon-supply:
+    description:
+      0.95V always-on LDO power input
+
+  vddpmu-supply:
+    description:
+      0.95V LDO power input to PMU
+
+  vddrfa1-supply:
+    description:
+      0.95V LDO power input to RFA
+
+  vddrfa2-supply:
+    description:
+      1.25V LDO power input to RFA
+
+  vddrfa3-supply:
+    description:
+      2V LDO power input to RFA
+
+  vddpcie1-supply:
+    description:
+      1.25V LDO power input to PCIe part
+
+  vddpcie2-supply:
+    description:
+      2V LDO power input to PCIe part
+
+  vddio-supply:
+    description:
+      1.8V VIO input
+
+additionalProperties: false
+
+examples:
+  - |
+    qca639x: qca639x {
+      compatible = "qcom,qca639x";
+      #power-domain-cells = <0>;
+
+      vddaon-supply = <&vreg_s6a_0p95>;
+      vddpmu-supply = <&vreg_s2f_0p95>;
+      vddrfa1-supply = <&vreg_s2f_0p95>;
+      vddrfa2-supply = <&vreg_s8c_1p3>;
+      vddrfa3-supply = <&vreg_s5a_1p9>;
+      vddpcie1-supply = <&vreg_s8c_1p3>;
+      vddpcie2-supply = <&vreg_s5a_1p9>;
+      vddio-supply = <&vreg_s4a_1p8>;
+      pinctrl-names = "default", "active";
+      pinctrl-0 = <&wlan_default_state &bt_default_state>;
+      pinctrl-1 = <&wlan_active_state &bt_active_state>;
+    };
+...
-- 
2.35.1

Re: [PATCH v11 0/7] PCI: qcom: Fix higher MSI vectors handling

[Johan Hovold]

On Fri, May 20, 2022 at 09:31:07PM +0300, Dmitry Baryshkov wrote:
> I have replied with my Tested-by to the patch at [2], which has landed
> in the linux-next as the commit 20f1bfb8dd62 ("PCI: qcom:
> Add support for handling MSIs from 8 endpoints"). However lately I
> noticed that during the tests I still had 'pcie_pme=nomsi', so the
> device was not forced to use higher MSI vectors.
> 
> After removing this option I noticed that hight MSI vectors are not
> delivered on tested platforms. After additional research I stumbled upon
> a patch in msm-4.14 ([1]), which describes that each group of MSI
> vectors is mapped to the separate interrupt. Implement corresponding
> mapping.
> 
> The first patch in the series is a revert of  [2] (landed in pci-next).
> Either both patches should be applied or both should be dropped.
> 
> Patchseries dependecies: [3] (for the schema change).
> 
> Changes since v10:
>  - Remove has_split_msi_irqs flag. Trust DT and use split MSI IRQs if
>    they are described in the DT. This removes the need for the
>    pcie-qcom.c changes (everything is handled by the core (suggested by
>    Johan).

You could also mention the rebase and fixed warnings with less than
eight msi.
 
> [1] https://git.codelinaro.org/clo/la/kernel/msm-4.14/-/commit/671a3d5f129f4bfe477152292ada2194c8440d22
> [2] https://lore.kernel.org/linux-arm-msm/20211214101319.25258-1-manivannan.sadhasivam@linaro.org/
> [3] https://lore.kernel.org/linux-arm-msm/20220422211002.2012070-1-dmitry.baryshkov@linaro.org/
> 
> 
> Dmitry Baryshkov (7):
>   PCI: dwc: Convert msi_irq to the array
>   PCI: dwc: split MSI IRQ parsing/allocation to a separate function
>   PCI: dwc: Handle MSIs routed to multiple GIC interrupts
>   PCI: dwc: Implement special ISR handler for split MSI IRQ setup
>   dt-bindings: PCI: qcom: Support additional MSI interrupts
>   arm64: dts: qcom: sm8250: provide additional MSI interrupts
>   dt-bindings: mfd: qcom,qca639x: add binding for QCA639x defvice

Looks like you used the wrong offsets from HEAD or something when
generating the series as the first two patches ([1] above, which is not
yet in linux-next, and the dw_pcie_free_msi() fix) are now missing and
the last patch is new and unrelated.

Johan

Re: [PATCH v11 3/7] PCI: dwc: Handle MSIs routed to multiple GIC interrupts

[Johan Hovold]

On Fri, May 20, 2022 at 09:31:10PM +0300, Dmitry Baryshkov wrote:
> On some of Qualcomm platforms each group of 32 MSI vectors is routed to the
> separate GIC interrupt. Implement support for such configurations by
> parsing "msi0" ... "msiN" interrupts and attaching them to the chained
> handler.
> 
> Note, that if DT doesn't list an array of MSI interrupts and uses single
> "msi" IRQ, the driver will limit the amount of supported MSI vectors
> accordingly (to 32).
> 
> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
> ---
>  .../pci/controller/dwc/pcie-designware-host.c | 58 +++++++++++++++++--
>  1 file changed, 54 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index a076abe6611c..381bc24d5715 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -288,6 +288,43 @@ static void dw_pcie_msi_init(struct pcie_port *pp)
>  	dw_pcie_writel_dbi(pci, PCIE_MSI_ADDR_HI, upper_32_bits(msi_target));
>  }
>  
> +static const char * const split_msi_names[] = {
> +	"msi0", "msi1", "msi2", "msi3",
> +	"msi4", "msi5", "msi6", "msi7",
> +};
> +
> +static int dw_pcie_parse_split_msi_irq(struct pcie_port *pp)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct device *dev = pci->dev;
> +	struct platform_device *pdev = to_platform_device(dev);
> +	int irq;
> +	u32 ctrl;
> +
> +	irq = platform_get_irq_byname_optional(pdev, split_msi_names[0]);
> +	if (irq == -ENXIO)
> +		return -ENXIO;

You still need to check for other errors and -EPROBE_DEFER here.

> +
> +	pp->msi_irq[0] = irq;
> +
> +	/* Parse as many IRQs as described in the DTS. */

s/DTS/devicetree/

> +	for (ctrl = 1; ctrl < MAX_MSI_CTRLS; ctrl++) {
> +		irq = platform_get_irq_byname_optional(pdev, split_msi_names[ctrl]);
> +		if (irq == -ENXIO)
> +			break;
> +		if (irq < 0)
> +			return dev_err_probe(dev, irq,
> +					     "Failed to parse MSI IRQ '%s'\n",
> +					     split_msi_names[ctrl]);
> +
> +		pp->msi_irq[ctrl] = irq;
> +	}
> +
> +	pp->num_vectors = ctrl * MAX_MSI_IRQS_PER_CTRL;
> +
> +	return 0;
> +}
> +
>  static int dw_pcie_msi_host_init(struct pcie_port *pp)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> @@ -295,22 +332,34 @@ static int dw_pcie_msi_host_init(struct pcie_port *pp)
>  	struct platform_device *pdev = to_platform_device(dev);
>  	int ret;
>  	u32 ctrl, num_ctrls;
> +	bool has_split_msi_irq = false;

This one should go in the follow-on patch that starts using it.

> -	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
> -	for (ctrl = 0; ctrl < num_ctrls; ctrl++)
> +	for (ctrl = 0; ctrl < MAX_MSI_CTRLS; ctrl++)
>  		pp->irq_mask[ctrl] = ~0;
>  
> +	if (!pp->msi_irq[0]) {
> +		ret = dw_pcie_parse_split_msi_irq(pp);
> +		if (ret < 0 && ret != -ENXIO)
> +			return ret;
> +	}
> +
> +	if (!pp->num_vectors)
> +		pp->num_vectors = MSI_DEF_NUM_VECTORS;

This works, but now you override num_vectors unconditionally when using
split msis (and not just when num_vectors is set to zero).

Is it work allowing to use num_vectors as a maximum as in previous
versions (if only for consistency)?

> +	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
> +
>  	if (!pp->msi_irq[0]) {
>  		int irq = platform_get_irq_byname_optional(pdev, "msi");
>  
>  		if (irq < 0) {
>  			irq = platform_get_irq(pdev, 0);
>  			if (irq < 0)
> -				return irq;
> +				return dev_err_probe(dev, irq, "Failed to parse MSI irq\n");
>  		}
>  		pp->msi_irq[0] = irq;
>  	}
>  
> +	dev_dbg(dev, "Using %d MSI vectors\n", pp->num_vectors);
> +
>  	pp->msi_irq_chip = &dw_pci_msi_bottom_irq_chip;
>  
>  	ret = dw_pcie_allocate_domains(pp);
> @@ -407,7 +456,8 @@ int dw_pcie_host_init(struct pcie_port *pp)
>  				     of_property_read_bool(np, "msi-parent") ||
>  				     of_property_read_bool(np, "msi-map"));
>  
> -		if (!pp->num_vectors) {
> +		/* for the has_msi_ctrl the default assignment is handled inside dw_pcie_msi_host_init() */
> +		if (!pp->has_msi_ctrl && !pp->num_vectors) {
>  			pp->num_vectors = MSI_DEF_NUM_VECTORS;
>  		} else if (pp->num_vectors > MAX_MSI_IRQS) {
>  			dev_err(dev, "Invalid number of vectors\n");

Johan

Re: [PATCH v11 0/7] PCI: qcom: Fix higher MSI vectors handling

[Dmitry Baryshkov]

On 23/05/2022 10:42, Johan Hovold wrote:
> On Fri, May 20, 2022 at 09:31:07PM +0300, Dmitry Baryshkov wrote:
>> I have replied with my Tested-by to the patch at [2], which has landed
>> in the linux-next as the commit 20f1bfb8dd62 ("PCI: qcom:
>> Add support for handling MSIs from 8 endpoints"). However lately I
>> noticed that during the tests I still had 'pcie_pme=nomsi', so the
>> device was not forced to use higher MSI vectors.
>>
>> After removing this option I noticed that hight MSI vectors are not
>> delivered on tested platforms. After additional research I stumbled upon
>> a patch in msm-4.14 ([1]), which describes that each group of MSI
>> vectors is mapped to the separate interrupt. Implement corresponding
>> mapping.
>>
>> The first patch in the series is a revert of  [2] (landed in pci-next).
>> Either both patches should be applied or both should be dropped.
>>
>> Patchseries dependecies: [3] (for the schema change).
>>
>> Changes since v10:
>>   - Remove has_split_msi_irqs flag. Trust DT and use split MSI IRQs if
>>     they are described in the DT. This removes the need for the
>>     pcie-qcom.c changes (everything is handled by the core (suggested by
>>     Johan).
> 
> You could also mention the rebase and fixed warnings with less than
> eight msi.
>   
>> [1] https://git.codelinaro.org/clo/la/kernel/msm-4.14/-/commit/671a3d5f129f4bfe477152292ada2194c8440d22
>> [2] https://lore.kernel.org/linux-arm-msm/20211214101319.25258-1-manivannan.sadhasivam@linaro.org/
>> [3] https://lore.kernel.org/linux-arm-msm/20220422211002.2012070-1-dmitry.baryshkov@linaro.org/
>>
>>
>> Dmitry Baryshkov (7):
>>    PCI: dwc: Convert msi_irq to the array
>>    PCI: dwc: split MSI IRQ parsing/allocation to a separate function
>>    PCI: dwc: Handle MSIs routed to multiple GIC interrupts
>>    PCI: dwc: Implement special ISR handler for split MSI IRQ setup
>>    dt-bindings: PCI: qcom: Support additional MSI interrupts
>>    arm64: dts: qcom: sm8250: provide additional MSI interrupts
>>    dt-bindings: mfd: qcom,qca639x: add binding for QCA639x defvice
> 
> Looks like you used the wrong offsets from HEAD or something when
> generating the series as the first two patches ([1] above, which is not
> yet in linux-next, and the dw_pcie_free_msi() fix) are now missing and
> the last patch is new and unrelated.

Ugh. Please excuse me.

> 
> Johan


-- 
With best wishes
Dmitry

Re: [PATCH v11 3/7] PCI: dwc: Handle MSIs routed to multiple GIC interrupts

[Dmitry Baryshkov]

On 23/05/2022 10:53, Johan Hovold wrote:
> On Fri, May 20, 2022 at 09:31:10PM +0300, Dmitry Baryshkov wrote:
>> On some of Qualcomm platforms each group of 32 MSI vectors is routed to the
>> separate GIC interrupt. Implement support for such configurations by
>> parsing "msi0" ... "msiN" interrupts and attaching them to the chained
>> handler.
>>
>> Note, that if DT doesn't list an array of MSI interrupts and uses single
>> "msi" IRQ, the driver will limit the amount of supported MSI vectors
>> accordingly (to 32).
>>
>> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
>> ---
>>   .../pci/controller/dwc/pcie-designware-host.c | 58 +++++++++++++++++--
>>   1 file changed, 54 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
>> index a076abe6611c..381bc24d5715 100644
>> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
>> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
>> @@ -288,6 +288,43 @@ static void dw_pcie_msi_init(struct pcie_port *pp)
>>   	dw_pcie_writel_dbi(pci, PCIE_MSI_ADDR_HI, upper_32_bits(msi_target));
>>   }
>>   
>> +static const char * const split_msi_names[] = {
>> +	"msi0", "msi1", "msi2", "msi3",
>> +	"msi4", "msi5", "msi6", "msi7",
>> +};
>> +
>> +static int dw_pcie_parse_split_msi_irq(struct pcie_port *pp)
>> +{
>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +	struct device *dev = pci->dev;
>> +	struct platform_device *pdev = to_platform_device(dev);
>> +	int irq;
>> +	u32 ctrl;
>> +
>> +	irq = platform_get_irq_byname_optional(pdev, split_msi_names[0]);
>> +	if (irq == -ENXIO)
>> +		return -ENXIO;
> 
> You still need to check for other errors and -EPROBE_DEFER here.

I think even the if (irq < 0) return irq; will work here.

> 
>> +
>> +	pp->msi_irq[0] = irq;
>> +
>> +	/* Parse as many IRQs as described in the DTS. */
> 
> s/DTS/devicetree/
> 
>> +	for (ctrl = 1; ctrl < MAX_MSI_CTRLS; ctrl++) {
>> +		irq = platform_get_irq_byname_optional(pdev, split_msi_names[ctrl]);
>> +		if (irq == -ENXIO)
>> +			break;
>> +		if (irq < 0)
>> +			return dev_err_probe(dev, irq,
>> +					     "Failed to parse MSI IRQ '%s'\n",
>> +					     split_msi_names[ctrl]);
>> +
>> +		pp->msi_irq[ctrl] = irq;
>> +	}
>> +
>> +	pp->num_vectors = ctrl * MAX_MSI_IRQS_PER_CTRL;
>> +
>> +	return 0;
>> +}
>> +
>>   static int dw_pcie_msi_host_init(struct pcie_port *pp)
>>   {
>>   	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> @@ -295,22 +332,34 @@ static int dw_pcie_msi_host_init(struct pcie_port *pp)
>>   	struct platform_device *pdev = to_platform_device(dev);
>>   	int ret;
>>   	u32 ctrl, num_ctrls;
>> +	bool has_split_msi_irq = false;
> 
> This one should go in the follow-on patch that starts using it.
> 
>> -	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
>> -	for (ctrl = 0; ctrl < num_ctrls; ctrl++)
>> +	for (ctrl = 0; ctrl < MAX_MSI_CTRLS; ctrl++)
>>   		pp->irq_mask[ctrl] = ~0;
>>   
>> +	if (!pp->msi_irq[0]) {
>> +		ret = dw_pcie_parse_split_msi_irq(pp);
>> +		if (ret < 0 && ret != -ENXIO)
>> +			return ret;
>> +	}
>> +
>> +	if (!pp->num_vectors)
>> +		pp->num_vectors = MSI_DEF_NUM_VECTORS;
> 
> This works, but now you override num_vectors unconditionally when using
> split msis (and not just when num_vectors is set to zero) >
> Is it work allowing to use num_vectors as a maximum as in previous
> versions (if only for consistency)?

Let me take a look.

> 
>> +	num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
>> +
>>   	if (!pp->msi_irq[0]) {
>>   		int irq = platform_get_irq_byname_optional(pdev, "msi");
>>   
>>   		if (irq < 0) {
>>   			irq = platform_get_irq(pdev, 0);
>>   			if (irq < 0)
>> -				return irq;
>> +				return dev_err_probe(dev, irq, "Failed to parse MSI irq\n");
>>   		}
>>   		pp->msi_irq[0] = irq;
>>   	}
>>   
>> +	dev_dbg(dev, "Using %d MSI vectors\n", pp->num_vectors);
>> +
>>   	pp->msi_irq_chip = &dw_pci_msi_bottom_irq_chip;
>>   
>>   	ret = dw_pcie_allocate_domains(pp);
>> @@ -407,7 +456,8 @@ int dw_pcie_host_init(struct pcie_port *pp)
>>   				     of_property_read_bool(np, "msi-parent") ||
>>   				     of_property_read_bool(np, "msi-map"));
>>   
>> -		if (!pp->num_vectors) {
>> +		/* for the has_msi_ctrl the default assignment is handled inside dw_pcie_msi_host_init() */
>> +		if (!pp->has_msi_ctrl && !pp->num_vectors) {
>>   			pp->num_vectors = MSI_DEF_NUM_VECTORS;
>>   		} else if (pp->num_vectors > MAX_MSI_IRQS) {
>>   			dev_err(dev, "Invalid number of vectors\n");
> 
> Johan


-- 
With best wishes
Dmitry

Re: [PATCH v11 3/7] PCI: dwc: Handle MSIs routed to multiple GIC interrupts

[Johan Hovold]

On Mon, May 23, 2022 at 04:39:56PM +0300, Dmitry Baryshkov wrote:
> On 23/05/2022 10:53, Johan Hovold wrote:
> > On Fri, May 20, 2022 at 09:31:10PM +0300, Dmitry Baryshkov wrote:

> >> +static int dw_pcie_parse_split_msi_irq(struct pcie_port *pp)
> >> +{
> >> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> >> +	struct device *dev = pci->dev;
> >> +	struct platform_device *pdev = to_platform_device(dev);
> >> +	int irq;
> >> +	u32 ctrl;
> >> +
> >> +	irq = platform_get_irq_byname_optional(pdev, split_msi_names[0]);
> >> +	if (irq == -ENXIO)
> >> +		return -ENXIO;
> > 
> > You still need to check for other errors and -EPROBE_DEFER here.
> 
> I think even the if (irq < 0) return irq; will work here.

No need to print errors unless -EPROBEDEFER as you do below?

> >> +
> >> +	pp->msi_irq[0] = irq;
> >> +
> >> +	/* Parse as many IRQs as described in the DTS. */
> > 
> > s/DTS/devicetree/
> > 
> >> +	for (ctrl = 1; ctrl < MAX_MSI_CTRLS; ctrl++) {
> >> +		irq = platform_get_irq_byname_optional(pdev, split_msi_names[ctrl]);
> >> +		if (irq == -ENXIO)
> >> +			break;
> >> +		if (irq < 0)
> >> +			return dev_err_probe(dev, irq,
> >> +					     "Failed to parse MSI IRQ '%s'\n",
> >> +					     split_msi_names[ctrl]);
> >> +
> >> +		pp->msi_irq[ctrl] = irq;
> >> +	}
> >> +
> >> +	pp->num_vectors = ctrl * MAX_MSI_IRQS_PER_CTRL;
> >> +
> >> +	return 0;
> >> +}

Johan

Re: [PATCH v11 3/7] PCI: dwc: Handle MSIs routed to multiple GIC interrupts

[Dmitry Baryshkov]

On 23/05/2022 17:02, Johan Hovold wrote:
> On Mon, May 23, 2022 at 04:39:56PM +0300, Dmitry Baryshkov wrote:
>> On 23/05/2022 10:53, Johan Hovold wrote:
>>> On Fri, May 20, 2022 at 09:31:10PM +0300, Dmitry Baryshkov wrote:
> 
>>>> +static int dw_pcie_parse_split_msi_irq(struct pcie_port *pp)
>>>> +{
>>>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>>>> +	struct device *dev = pci->dev;
>>>> +	struct platform_device *pdev = to_platform_device(dev);
>>>> +	int irq;
>>>> +	u32 ctrl;
>>>> +
>>>> +	irq = platform_get_irq_byname_optional(pdev, split_msi_names[0]);
>>>> +	if (irq == -ENXIO)
>>>> +		return -ENXIO;
>>>
>>> You still need to check for other errors and -EPROBE_DEFER here.
>>
>> I think even the if (irq < 0) return irq; will work here.
> 
> No need to print errors unless -EPROBEDEFER as you do below?

There is no separate print for the dw_pcie_parse_split_msi_irq() errors.

> 
>>>> +
>>>> +	pp->msi_irq[0] = irq;
>>>> +
>>>> +	/* Parse as many IRQs as described in the DTS. */
>>>
>>> s/DTS/devicetree/
>>>
>>>> +	for (ctrl = 1; ctrl < MAX_MSI_CTRLS; ctrl++) {
>>>> +		irq = platform_get_irq_byname_optional(pdev, split_msi_names[ctrl]);
>>>> +		if (irq == -ENXIO)
>>>> +			break;
>>>> +		if (irq < 0)
>>>> +			return dev_err_probe(dev, irq,
>>>> +					     "Failed to parse MSI IRQ '%s'\n",
>>>> +					     split_msi_names[ctrl]);
>>>> +
>>>> +		pp->msi_irq[ctrl] = irq;
>>>> +	}
>>>> +
>>>> +	pp->num_vectors = ctrl * MAX_MSI_IRQS_PER_CTRL;
>>>> +
>>>> +	return 0;
>>>> +}
> 
> Johan


-- 
With best wishes
Dmitry

Re: [PATCH v11 3/7] PCI: dwc: Handle MSIs routed to multiple GIC interrupts

[Johan Hovold]

On Mon, May 23, 2022 at 06:17:19PM +0300, Dmitry Baryshkov wrote:
> On 23/05/2022 17:02, Johan Hovold wrote:
> > On Mon, May 23, 2022 at 04:39:56PM +0300, Dmitry Baryshkov wrote:
> >> On 23/05/2022 10:53, Johan Hovold wrote:
> >>> On Fri, May 20, 2022 at 09:31:10PM +0300, Dmitry Baryshkov wrote:
> > 
> >>>> +static int dw_pcie_parse_split_msi_irq(struct pcie_port *pp)
> >>>> +{
> >>>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> >>>> +	struct device *dev = pci->dev;
> >>>> +	struct platform_device *pdev = to_platform_device(dev);
> >>>> +	int irq;
> >>>> +	u32 ctrl;
> >>>> +
> >>>> +	irq = platform_get_irq_byname_optional(pdev, split_msi_names[0]);
> >>>> +	if (irq == -ENXIO)
> >>>> +		return -ENXIO;
> >>>
> >>> You still need to check for other errors and -EPROBE_DEFER here.
> >>
> >> I think even the if (irq < 0) return irq; will work here.
> > 
> > No need to print errors unless -EPROBEDEFER as you do below?
> 
> There is no separate print for the dw_pcie_parse_split_msi_irq() errors.

I don't understand what you're referring to here.

My question is: Why would you not be printing error messages for msi0 as
you are for msi1..msi7 in the loop below.

> >>>> +
> >>>> +	pp->msi_irq[0] = irq;
> >>>> +
> >>>> +	/* Parse as many IRQs as described in the DTS. */
> >>>
> >>> s/DTS/devicetree/
> >>>
> >>>> +	for (ctrl = 1; ctrl < MAX_MSI_CTRLS; ctrl++) {
> >>>> +		irq = platform_get_irq_byname_optional(pdev, split_msi_names[ctrl]);
> >>>> +		if (irq == -ENXIO)
> >>>> +			break;
> >>>> +		if (irq < 0)
> >>>> +			return dev_err_probe(dev, irq,
> >>>> +					     "Failed to parse MSI IRQ '%s'\n",
> >>>> +					     split_msi_names[ctrl]);
> >>>> +
> >>>> +		pp->msi_irq[ctrl] = irq;
> >>>> +	}
> >>>> +
> >>>> +	pp->num_vectors = ctrl * MAX_MSI_IRQS_PER_CTRL;
> >>>> +
> >>>> +	return 0;
> >>>> +}

Johan

Re: [PATCH v11 3/7] PCI: dwc: Handle MSIs routed to multiple GIC interrupts

[Dmitry Baryshkov]

On 23/05/2022 18:32, Johan Hovold wrote:
> On Mon, May 23, 2022 at 06:17:19PM +0300, Dmitry Baryshkov wrote:
>> On 23/05/2022 17:02, Johan Hovold wrote:
>>> On Mon, May 23, 2022 at 04:39:56PM +0300, Dmitry Baryshkov wrote:
>>>> On 23/05/2022 10:53, Johan Hovold wrote:
>>>>> On Fri, May 20, 2022 at 09:31:10PM +0300, Dmitry Baryshkov wrote:
>>>
>>>>>> +static int dw_pcie_parse_split_msi_irq(struct pcie_port *pp)
>>>>>> +{
>>>>>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>>>>>> +	struct device *dev = pci->dev;
>>>>>> +	struct platform_device *pdev = to_platform_device(dev);
>>>>>> +	int irq;
>>>>>> +	u32 ctrl;
>>>>>> +
>>>>>> +	irq = platform_get_irq_byname_optional(pdev, split_msi_names[0]);
>>>>>> +	if (irq == -ENXIO)
>>>>>> +		return -ENXIO;
>>>>>
>>>>> You still need to check for other errors and -EPROBE_DEFER here.
>>>>
>>>> I think even the if (irq < 0) return irq; will work here.
>>>
>>> No need to print errors unless -EPROBEDEFER as you do below?
>>
>> There is no separate print for the dw_pcie_parse_split_msi_irq() errors.
> 
> I don't understand what you're referring to here.
> 
> My question is: Why would you not be printing error messages for msi0 as
> you are for msi1..msi7 in the loop below.

Yeah, this seems like a correct idea. Thank you!

> 
>>>>>> +
>>>>>> +	pp->msi_irq[0] = irq;
>>>>>> +
>>>>>> +	/* Parse as many IRQs as described in the DTS. */
>>>>>
>>>>> s/DTS/devicetree/
>>>>>
>>>>>> +	for (ctrl = 1; ctrl < MAX_MSI_CTRLS; ctrl++) {
>>>>>> +		irq = platform_get_irq_byname_optional(pdev, split_msi_names[ctrl]);
>>>>>> +		if (irq == -ENXIO)
>>>>>> +			break;
>>>>>> +		if (irq < 0)
>>>>>> +			return dev_err_probe(dev, irq,
>>>>>> +					     "Failed to parse MSI IRQ '%s'\n",
>>>>>> +					     split_msi_names[ctrl]);
>>>>>> +
>>>>>> +		pp->msi_irq[ctrl] = irq;
>>>>>> +	}
>>>>>> +
>>>>>> +	pp->num_vectors = ctrl * MAX_MSI_IRQS_PER_CTRL;
>>>>>> +
>>>>>> +	return 0;
>>>>>> +}
> 
> Johan


-- 
With best wishes
Dmitry