[PATCH V16 0/7] PCI: Loongson pci improvements and quirks

[PATCH V16 0/7] PCI: Loongson pci improvements and quirks

[Huacai Chen]

This patchset improves Loongson PCI controller driver and resolves some
problems: LS2K/LS7A's PCI config space supports 1/2/4-bytes access, so
the first patch use pci_generic_config_read()/pci_generic_config_write()
for them; the second patch add ACPI init support which will be used by
LoongArch; the third patch improves the mrrs quirk for LS7A chipset; The
fourth patch add a new quirk for LS7A chipset to avoid poweroff/reboot
failure, and the fifth patch add a new quirk for LS7A chipset to fix the
multifunction devices' irq pin mappings.

V1 -> V2:
1, Rework the 4th patch;
2, Improve commit messages;
3, Remove the last patch since there is better solutions.

V2 -> V3:
1, Add more affected device ids for the 4th patch;
2, Improve commit messages to describe root causes.

V3 -> V4:
1, Rework the MRRS quirk patch;
2, Improve commit messages to describe root causes, again.

V4 -> V5:
1, Improve the MRRS quirk patch;
2, Change the order of 2nd and 3rd patch;
3, Improve commit messages to describe root causes, again.

V5 -> V6:
1, Rework the 1st patch;
2, Adjust the order of the series.

V6 -> V7:
1, Use correct pci config access operations;
2, Add ACPI init support for LoongArch;
3, Don't move to quirks.c since the driver has ACPI support;
4, Some other minor improvements.

V7 -> V8:
1, Use CFG1 method for LS2K/LS7A pci config.

V8 -> V9:
1, Use pci_controller_data for the first patch. 

V9 -> V10:
1, Add a patch to avoid LS2K/LS7A access unexisting devices.

V10 -> V11:
1, Rebased on 5.16-rc4.

V11 -> V12:
1, Rebased on 5.17-rc5.

V12 -> V13:
1, Rebased on 5.18-rc2;
2, Some minor improvements (adopt Rob Herring's suggestions).

V13 -> V14:
1, Rebased on 5.18;
2, Split ARM64-specific mcfg_quirks to a separate patch;
3, Refactor "PCI: loongson: Don't access non-existant devices";
4, Refactor "PCI: Add quirk for LS7A to avoid reboot failure";
5, Some other minor improvements (adopt Bjorn's suggestions).

V14 -> V15:
1, Rebased on 5.19-rc4;
2, Refactor "PCI: loongson: Don't access non-existant devices".

V15 -> V16:
1, Rebased on 5.19-rc6;
2, Refactor "PCI: loongson: Add ACPI init support".

Huacai Chen, Tiezhu Yang and Jianmin Lv(6):
 PCI/ACPI: Guard ARM64-specific mcfg_quirks
 PCI: loongson: Use generic 8/16/32-bit config ops on LS2K/LS7A.
 PCI: loongson: Add ACPI init support.
 PCI: loongson: Don't access non-existant devices.
 PCI: loongson: Improve the MRRS quirk for LS7A.
 PCI: Add quirk for LS7A to avoid reboot failure.
 PCI: Add quirk for multifunction devices of LS7A.

Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
Signed-off-by: Jianmin Lv <lvjianmin@loongson.cn> 
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 drivers/acpi/pci_mcfg.c               |  13 ++
 drivers/pci/controller/Kconfig        |   2 +-
 drivers/pci/controller/pci-loongson.c | 233 ++++++++++++++++++++++++++--------
 drivers/pci/pci.c                     |   6 +
 drivers/pci/pcie/portdrv_core.c       |   1 -
 drivers/pci/pcie/portdrv_pci.c        |  20 ++-
 include/linux/pci-ecam.h              |   1 +
 include/linux/pci.h                   |   2 +
 8 files changed, 225 insertions(+), 53 deletions(-)
--
2.27.0

[PATCH V16 1/7] PCI/ACPI: Guard ARM64-specific mcfg_quirks

[Huacai Chen]

Guard ARM64-specific quirks with CONFIG_ARM64 to avoid build errors,
since mcfg_quirks will be shared by more than one architectures.

Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
---
 drivers/acpi/pci_mcfg.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
index 53cab975f612..63b98eae5e75 100644
--- a/drivers/acpi/pci_mcfg.c
+++ b/drivers/acpi/pci_mcfg.c
@@ -41,6 +41,8 @@ struct mcfg_fixup {
 static struct mcfg_fixup mcfg_quirks[] = {
 /*	{ OEM_ID, OEM_TABLE_ID, REV, SEGMENT, BUS_RANGE, ops, cfgres }, */
 
+#ifdef CONFIG_ARM64
+
 #define AL_ECAM(table_id, rev, seg, ops) \
 	{ "AMAZON", table_id, rev, seg, MCFG_BUS_ANY, ops }
 
@@ -169,6 +171,7 @@ static struct mcfg_fixup mcfg_quirks[] = {
 	ALTRA_ECAM_QUIRK(1, 13),
 	ALTRA_ECAM_QUIRK(1, 14),
 	ALTRA_ECAM_QUIRK(1, 15),
+#endif /* ARM64 */
 };
 
 static char mcfg_oem_id[ACPI_OEM_ID_SIZE];
-- 
2.31.1

[PATCH V16 2/7] PCI: loongson: Use generic 8/16/32-bit config ops on LS2K/LS7A

[Huacai Chen]

LS2K/LS7A support 8/16/32-bits PCI config access operations via CFG1, so
we can disable CFG0 for them and safely use pci_generic_config_read()/
pci_generic_config_write() instead of pci_generic_config_read32()/pci_
generic_config_write32().

Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
---
 drivers/pci/controller/pci-loongson.c | 65 +++++++++++++++++++--------
 1 file changed, 46 insertions(+), 19 deletions(-)

diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
index 50a8e1d6f70a..565453882ffe 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -25,11 +25,16 @@
 #define FLAG_CFG1	BIT(1)
 #define FLAG_DEV_FIX	BIT(2)
 
+struct loongson_pci_data {
+	u32 flags;
+	struct pci_ops *ops;
+};
+
 struct loongson_pci {
 	void __iomem *cfg0_base;
 	void __iomem *cfg1_base;
 	struct platform_device *pdev;
-	u32 flags;
+	const struct loongson_pci_data *data;
 };
 
 /* Fixup wrong class code in PCIe bridges */
@@ -126,8 +131,8 @@ static void __iomem *pci_loongson_map_bus(struct pci_bus *bus, unsigned int devf
 	 * Do not read more than one device on the bus other than
 	 * the host bus. For our hardware the root bus is always bus 0.
 	 */
-	if (priv->flags & FLAG_DEV_FIX && busnum != 0 &&
-		PCI_SLOT(devfn) > 0)
+	if (priv->data->flags & FLAG_DEV_FIX &&
+			!pci_is_root_bus(bus) && PCI_SLOT(devfn) > 0)
 		return NULL;
 
 	/* CFG0 can only access standard space */
@@ -159,20 +164,42 @@ static int loongson_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
 	return val;
 }
 
-/* H/w only accept 32-bit PCI operations */
+/* LS2K/LS7A accept 8/16/32-bit PCI config operations */
 static struct pci_ops loongson_pci_ops = {
+	.map_bus = pci_loongson_map_bus,
+	.read	= pci_generic_config_read,
+	.write	= pci_generic_config_write,
+};
+
+/* RS780/SR5690 only accept 32-bit PCI config operations */
+static struct pci_ops loongson_pci_ops32 = {
 	.map_bus = pci_loongson_map_bus,
 	.read	= pci_generic_config_read32,
 	.write	= pci_generic_config_write32,
 };
 
+static const struct loongson_pci_data ls2k_pci_data = {
+	.flags = FLAG_CFG1 | FLAG_DEV_FIX,
+	.ops = &loongson_pci_ops,
+};
+
+static const struct loongson_pci_data ls7a_pci_data = {
+	.flags = FLAG_CFG1 | FLAG_DEV_FIX,
+	.ops = &loongson_pci_ops,
+};
+
+static const struct loongson_pci_data rs780e_pci_data = {
+	.flags = FLAG_CFG0,
+	.ops = &loongson_pci_ops32,
+};
+
 static const struct of_device_id loongson_pci_of_match[] = {
 	{ .compatible = "loongson,ls2k-pci",
-		.data = (void *)(FLAG_CFG0 | FLAG_CFG1 | FLAG_DEV_FIX), },
+		.data = &ls2k_pci_data, },
 	{ .compatible = "loongson,ls7a-pci",
-		.data = (void *)(FLAG_CFG0 | FLAG_CFG1 | FLAG_DEV_FIX), },
+		.data = &ls7a_pci_data, },
 	{ .compatible = "loongson,rs780e-pci",
-		.data = (void *)(FLAG_CFG0), },
+		.data = &rs780e_pci_data, },
 	{}
 };
 
@@ -193,20 +220,20 @@ static int loongson_pci_probe(struct platform_device *pdev)
 
 	priv = pci_host_bridge_priv(bridge);
 	priv->pdev = pdev;
-	priv->flags = (unsigned long)of_device_get_match_data(dev);
+	priv->data = of_device_get_match_data(dev);
 
-	regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
-	if (!regs) {
-		dev_err(dev, "missing mem resources for cfg0\n");
-		return -EINVAL;
+	if (priv->data->flags & FLAG_CFG0) {
+		regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+		if (!regs)
+			dev_err(dev, "missing mem resources for cfg0\n");
+		else {
+			priv->cfg0_base = devm_pci_remap_cfg_resource(dev, regs);
+			if (IS_ERR(priv->cfg0_base))
+				return PTR_ERR(priv->cfg0_base);
+		}
 	}
 
-	priv->cfg0_base = devm_pci_remap_cfg_resource(dev, regs);
-	if (IS_ERR(priv->cfg0_base))
-		return PTR_ERR(priv->cfg0_base);
-
-	/* CFG1 is optional */
-	if (priv->flags & FLAG_CFG1) {
+	if (priv->data->flags & FLAG_CFG1) {
 		regs = platform_get_resource(pdev, IORESOURCE_MEM, 1);
 		if (!regs)
 			dev_info(dev, "missing mem resource for cfg1\n");
@@ -218,7 +245,7 @@ static int loongson_pci_probe(struct platform_device *pdev)
 	}
 
 	bridge->sysdata = priv;
-	bridge->ops = &loongson_pci_ops;
+	bridge->ops = priv->data->ops;
 	bridge->map_irq = loongson_map_irq;
 
 	return pci_host_probe(bridge);
-- 
2.31.1

[PATCH V16 3/7] PCI: loongson: Add ACPI init support

[Huacai Chen]

Loongson PCH (LS7A chipset) will be used by both MIPS-based and
LoongArch-based Loongson processors. MIPS-based Loongson uses FDT
while LoongArch-base Loongson uses ACPI, this patch add ACPI init
support for the driver in drivers/pci/controller/pci-loongson.c
because it is currently FDT-only.

LoongArch is a new RISC ISA, mainline support will come soon, and
documentations are here (in translation):

https://github.com/loongson/LoongArch-Documentation

Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
---
 drivers/acpi/pci_mcfg.c               | 10 +++
 drivers/pci/controller/Kconfig        |  2 +-
 drivers/pci/controller/pci-loongson.c | 92 +++++++++++++++++++++------
 include/linux/pci-ecam.h              |  1 +
 4 files changed, 86 insertions(+), 19 deletions(-)

diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
index 63b98eae5e75..860014b89b8e 100644
--- a/drivers/acpi/pci_mcfg.c
+++ b/drivers/acpi/pci_mcfg.c
@@ -172,6 +172,16 @@ static struct mcfg_fixup mcfg_quirks[] = {
 	ALTRA_ECAM_QUIRK(1, 14),
 	ALTRA_ECAM_QUIRK(1, 15),
 #endif /* ARM64 */
+
+#ifdef CONFIG_LOONGARCH
+#define LOONGSON_ECAM_MCFG(table_id, seg) \
+	{ "LOONGS", table_id, 1, seg, MCFG_BUS_ANY, &loongson_pci_ecam_ops }
+
+	LOONGSON_ECAM_MCFG("\0", 0),
+	LOONGSON_ECAM_MCFG("LOONGSON", 0),
+	LOONGSON_ECAM_MCFG("\0", 1),
+	LOONGSON_ECAM_MCFG("LOONGSON", 1),
+#endif /* LOONGARCH */
 };
 
 static char mcfg_oem_id[ACPI_OEM_ID_SIZE];
diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
index b8d96d38064d..9dbd73898b47 100644
--- a/drivers/pci/controller/Kconfig
+++ b/drivers/pci/controller/Kconfig
@@ -293,7 +293,7 @@ config PCI_HYPERV_INTERFACE
 config PCI_LOONGSON
 	bool "LOONGSON PCI Controller"
 	depends on MACH_LOONGSON64 || COMPILE_TEST
-	depends on OF
+	depends on OF || ACPI
 	depends on PCI_QUIRKS
 	default MACH_LOONGSON64
 	help
diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
index 565453882ffe..2db180a9e628 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -9,6 +9,8 @@
 #include <linux/of_pci.h>
 #include <linux/pci.h>
 #include <linux/pci_ids.h>
+#include <linux/pci-acpi.h>
+#include <linux/pci-ecam.h>
 
 #include "../pci.h"
 
@@ -97,39 +99,53 @@ static void loongson_mrrs_quirk(struct pci_dev *dev)
 }
 DECLARE_PCI_FIXUP_ENABLE(PCI_ANY_ID, PCI_ANY_ID, loongson_mrrs_quirk);
 
-static void __iomem *cfg1_map(struct loongson_pci *priv, int bus,
-				unsigned int devfn, int where)
+static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
 {
-	unsigned long addroff = 0x0;
+	struct pci_config_window *cfg;
 
-	if (bus != 0)
-		addroff |= BIT(28); /* Type 1 Access */
-	addroff |= (where & 0xff) | ((where & 0xf00) << 16);
-	addroff |= (bus << 16) | (devfn << 8);
-	return priv->cfg1_base + addroff;
+	if (acpi_disabled)
+		return (struct loongson_pci *)(bus->sysdata);
+
+	cfg = bus->sysdata;
+	return (struct loongson_pci *)(cfg->priv);
 }
 
-static void __iomem *cfg0_map(struct loongson_pci *priv, int bus,
-				unsigned int devfn, int where)
+static void __iomem *cfg0_map(struct loongson_pci *priv,
+			      struct pci_bus *bus, unsigned int devfn, int where)
 {
 	unsigned long addroff = 0x0;
+	unsigned char busnum = bus->number;
 
-	if (bus != 0)
+	if (!pci_is_root_bus(bus)) {
 		addroff |= BIT(24); /* Type 1 Access */
-	addroff |= (bus << 16) | (devfn << 8) | where;
+		addroff |= (busnum << 16);
+	}
+	addroff |= (devfn << 8) | where;
 	return priv->cfg0_base + addroff;
 }
 
+static void __iomem *cfg1_map(struct loongson_pci *priv,
+			      struct pci_bus *bus, unsigned int devfn, int where)
+{
+	unsigned long addroff = 0x0;
+	unsigned char busnum = bus->number;
+
+	if (!pci_is_root_bus(bus)) {
+		addroff |= BIT(28); /* Type 1 Access */
+		addroff |= (busnum << 16);
+	}
+	addroff |= (devfn << 8) | (where & 0xff) | ((where & 0xf00) << 16);
+	return priv->cfg1_base + addroff;
+}
+
 static void __iomem *pci_loongson_map_bus(struct pci_bus *bus, unsigned int devfn,
 			       int where)
 {
-	unsigned char busnum = bus->number;
-	struct pci_host_bridge *bridge = pci_find_host_bridge(bus);
-	struct loongson_pci *priv =  pci_host_bridge_priv(bridge);
+	struct loongson_pci *priv = pci_bus_to_loongson_pci(bus);
 
 	/*
 	 * Do not read more than one device on the bus other than
-	 * the host bus. For our hardware the root bus is always bus 0.
+	 * the host bus.
 	 */
 	if (priv->data->flags & FLAG_DEV_FIX &&
 			!pci_is_root_bus(bus) && PCI_SLOT(devfn) > 0)
@@ -137,15 +153,17 @@ static void __iomem *pci_loongson_map_bus(struct pci_bus *bus, unsigned int devf
 
 	/* CFG0 can only access standard space */
 	if (where < PCI_CFG_SPACE_SIZE && priv->cfg0_base)
-		return cfg0_map(priv, busnum, devfn, where);
+		return cfg0_map(priv, bus, devfn, where);
 
 	/* CFG1 can access extended space */
 	if (where < PCI_CFG_SPACE_EXP_SIZE && priv->cfg1_base)
-		return cfg1_map(priv, busnum, devfn, where);
+		return cfg1_map(priv, bus, devfn, where);
 
 	return NULL;
 }
 
+#ifdef CONFIG_OF
+
 static int loongson_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
 {
 	int irq;
@@ -259,3 +277,41 @@ static struct platform_driver loongson_pci_driver = {
 	.probe = loongson_pci_probe,
 };
 builtin_platform_driver(loongson_pci_driver);
+
+#endif
+
+#ifdef CONFIG_ACPI
+
+static int loongson_pci_ecam_init(struct pci_config_window *cfg)
+{
+	struct device *dev = cfg->parent;
+	struct loongson_pci *priv;
+	struct loongson_pci_data *data;
+
+	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
+	if (!data)
+		return -ENOMEM;
+
+	cfg->priv = priv;
+	data->flags = FLAG_CFG1;
+	priv->data = data;
+	priv->cfg1_base = cfg->win - (cfg->busr.start << 16);
+
+	return 0;
+}
+
+const struct pci_ecam_ops loongson_pci_ecam_ops = {
+	.bus_shift = 16,
+	.init	   = loongson_pci_ecam_init,
+	.pci_ops   = {
+		.map_bus = pci_loongson_map_bus,
+		.read	 = pci_generic_config_read,
+		.write	 = pci_generic_config_write,
+	}
+};
+
+#endif
diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h
index adea5a4771cf..6b1301e2498e 100644
--- a/include/linux/pci-ecam.h
+++ b/include/linux/pci-ecam.h
@@ -87,6 +87,7 @@ extern const struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 *
 extern const struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */
 extern const struct pci_ecam_ops al_pcie_ops;	/* Amazon Annapurna Labs PCIe */
 extern const struct pci_ecam_ops tegra194_pcie_ops; /* Tegra194 PCIe */
+extern const struct pci_ecam_ops loongson_pci_ecam_ops; /* Loongson PCIe */
 #endif
 
 #if IS_ENABLED(CONFIG_PCI_HOST_COMMON)
-- 
2.31.1

[PATCH V16 4/7] PCI: loongson: Don't access non-existant devices

[Huacai Chen]

On LS2K/LS7A, some non-existant devices don't return 0xffffffff when
scanning (they are hidden devices for debug in fact, access the config
space may cause machine hang). This is a hardware flaw but we can only
avoid it by software now.

Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
---
 drivers/pci/controller/pci-loongson.c | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
index 2db180a9e628..594653154deb 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -26,6 +26,7 @@
 #define FLAG_CFG0	BIT(0)
 #define FLAG_CFG1	BIT(1)
 #define FLAG_DEV_FIX	BIT(2)
+#define FLAG_DEV_HIDDEN	BIT(3)
 
 struct loongson_pci_data {
 	u32 flags;
@@ -138,18 +139,32 @@ static void __iomem *cfg1_map(struct loongson_pci *priv,
 	return priv->cfg1_base + addroff;
 }
 
+static bool pdev_may_exist(struct pci_bus *bus, unsigned int device, unsigned int function)
+{
+	return !(pci_is_root_bus(bus) && (device >= 9 && device <= 20) && (function > 0));
+}
+
 static void __iomem *pci_loongson_map_bus(struct pci_bus *bus, unsigned int devfn,
 			       int where)
 {
+	unsigned int device = PCI_SLOT(devfn);
+	unsigned int function = PCI_FUNC(devfn);
 	struct loongson_pci *priv = pci_bus_to_loongson_pci(bus);
 
 	/*
 	 * Do not read more than one device on the bus other than
 	 * the host bus.
 	 */
-	if (priv->data->flags & FLAG_DEV_FIX &&
-			!pci_is_root_bus(bus) && PCI_SLOT(devfn) > 0)
-		return NULL;
+	if ((priv->data->flags & FLAG_DEV_FIX) && bus->self) {
+		if (!pci_is_root_bus(bus) && (device > 0))
+			return NULL;
+	}
+
+	/* Don't access non-existant devices */
+	if (priv->data->flags & FLAG_DEV_HIDDEN) {
+		if (!pdev_may_exist(bus, device, function))
+			return NULL;
+	}
 
 	/* CFG0 can only access standard space */
 	if (where < PCI_CFG_SPACE_SIZE && priv->cfg0_base)
@@ -197,12 +212,12 @@ static struct pci_ops loongson_pci_ops32 = {
 };
 
 static const struct loongson_pci_data ls2k_pci_data = {
-	.flags = FLAG_CFG1 | FLAG_DEV_FIX,
+	.flags = FLAG_CFG1 | FLAG_DEV_FIX | FLAG_DEV_HIDDEN,
 	.ops = &loongson_pci_ops,
 };
 
 static const struct loongson_pci_data ls7a_pci_data = {
-	.flags = FLAG_CFG1 | FLAG_DEV_FIX,
+	.flags = FLAG_CFG1 | FLAG_DEV_FIX | FLAG_DEV_HIDDEN,
 	.ops = &loongson_pci_ops,
 };
 
@@ -297,7 +312,7 @@ static int loongson_pci_ecam_init(struct pci_config_window *cfg)
 		return -ENOMEM;
 
 	cfg->priv = priv;
-	data->flags = FLAG_CFG1;
+	data->flags = FLAG_CFG1 | FLAG_DEV_HIDDEN;
 	priv->data = data;
 	priv->cfg1_base = cfg->win - (cfg->busr.start << 16);
 
-- 
2.31.1

[PATCH V16 5/7] PCI: loongson: Improve the MRRS quirk for LS7A

[Huacai Chen]

In new revision of LS7A, some PCIe ports support larger value than 256,
but their maximum supported MRRS values are not detectable. Moreover,
the current loongson_mrrs_quirk() cannot avoid devices increasing its
MRRS after pci_enable_device(), and some devices (e.g. Realtek 8169)
will actually set a big value in its driver. So the only possible way
is configure MRRS of all devices in BIOS, and add a pci host bridge bit
flag (i.e., no_inc_mrrs) to stop the increasing MRRS operations.

However, according to PCIe Spec, it is legal for an OS to program any
value for MRRS, and it is also legal for an endpoint to generate a Read
Request with any size up to its MRRS. As the hardware engineers say, the
root cause here is LS7A doesn't break up large read requests. In detail,
LS7A PCIe port reports CA (Completer Abort) if it receives a Memory Read
request with a size that's "too big" ("too big" means larger than the
PCIe ports can handle, which means 256 for some ports and 4096 for the
others, and of course this is a problem in the LS7A's hardware design).

Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
---
 drivers/pci/controller/pci-loongson.c | 44 +++++++++------------------
 drivers/pci/pci.c                     |  6 ++++
 include/linux/pci.h                   |  1 +
 3 files changed, 22 insertions(+), 29 deletions(-)

diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
index 594653154deb..af73bb766e48 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -68,37 +68,23 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
 			DEV_LS7A_LPC, system_bus_quirk);
 
-static void loongson_mrrs_quirk(struct pci_dev *dev)
+static void loongson_mrrs_quirk(struct pci_dev *pdev)
 {
-	struct pci_bus *bus = dev->bus;
-	struct pci_dev *bridge;
-	static const struct pci_device_id bridge_devids[] = {
-		{ PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_0) },
-		{ PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_1) },
-		{ PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_2) },
-		{ 0, },
-	};
-
-	/* look for the matching bridge */
-	while (!pci_is_root_bus(bus)) {
-		bridge = bus->self;
-		bus = bus->parent;
-		/*
-		 * Some Loongson PCIe ports have a h/w limitation of
-		 * 256 bytes maximum read request size. They can't handle
-		 * anything larger than this. So force this limit on
-		 * any devices attached under these ports.
-		 */
-		if (pci_match_id(bridge_devids, bridge)) {
-			if (pcie_get_readrq(dev) > 256) {
-				pci_info(dev, "limiting MRRS to 256\n");
-				pcie_set_readrq(dev, 256);
-			}
-			break;
-		}
-	}
+	/*
+	 * Some Loongson PCIe ports have h/w limitations of maximum read
+	 * request size. They can't handle anything larger than this. So
+	 * force this limit on any devices attached under these ports.
+	 */
+	struct pci_host_bridge *bridge = pci_find_host_bridge(pdev->bus);
+
+	bridge->no_inc_mrrs = 1;
 }
-DECLARE_PCI_FIXUP_ENABLE(PCI_ANY_ID, PCI_ANY_ID, loongson_mrrs_quirk);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_0, loongson_mrrs_quirk);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_1, loongson_mrrs_quirk);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_2, loongson_mrrs_quirk);
 
 static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
 {
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index cfaf40a540a8..79157cbad835 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -6052,6 +6052,7 @@ int pcie_set_readrq(struct pci_dev *dev, int rq)
 {
 	u16 v;
 	int ret;
+	struct pci_host_bridge *bridge = pci_find_host_bridge(dev->bus);
 
 	if (rq < 128 || rq > 4096 || !is_power_of_2(rq))
 		return -EINVAL;
@@ -6070,6 +6071,11 @@ int pcie_set_readrq(struct pci_dev *dev, int rq)
 
 	v = (ffs(rq) - 8) << 12;
 
+	if (bridge->no_inc_mrrs) {
+		if (rq > pcie_get_readrq(dev))
+			return -EINVAL;
+	}
+
 	ret = pcie_capability_clear_and_set_word(dev, PCI_EXP_DEVCTL,
 						  PCI_EXP_DEVCTL_READRQ, v);
 
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 81a57b498f22..a9211074add6 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -569,6 +569,7 @@ struct pci_host_bridge {
 	void		*release_data;
 	unsigned int	ignore_reset_delay:1;	/* For entire hierarchy */
 	unsigned int	no_ext_tags:1;		/* No Extended Tags */
+	unsigned int	no_inc_mrrs:1;		/* No Increase MRRS */
 	unsigned int	native_aer:1;		/* OS may use PCIe AER */
 	unsigned int	native_pcie_hotplug:1;	/* OS may use PCIe hotplug */
 	unsigned int	native_shpc_hotplug:1;	/* OS may use SHPC hotplug */
-- 
2.31.1

Re: [PATCH V16 5/7] PCI: loongson: Improve the MRRS quirk for LS7A

[Jianmin Lv]

Hi, Huacai and Bjorn,

Actually, I don't think we have to fix the MRRS issue of 7A in this 
way(change core code of PCI). The reasons are:

- First, we don't know if other pci controlers have simillar issue, at 
least I don't see yet. I think if *only we* have the issue, maybe we can 
fix it in our controller driver rather than changing core code. And in 
future, if the issue is proved a common one, abstract it then by common way.

- Second, even though we limit the MRRS in pcie_set_readrq() according 
to the flag set in quirk function, some drivers(e.g. in 
drivers/rapidio/devices/tsi721.c, maybe other driver do this in future, 
I dont't know.) directly call pcie_capability_clear_and_set_word to set 
MRRS, which will still break the fix.

- Third, on resuming from S3, the MRRS stored in memory should be 
allowed to set to dev ctrl reg(because the reg has been reset during S3).


Fix MMRS in our controller driver by using self-defined config_write(), 
maybe like this:

static u32 handle_mrrs_limit(struct pci_bus *bus, unsigned int devfn, 
void __iomem *addr, u32 val)
{
      u32 tmp;
      bool runtime_flag = 1;
      int pos = pci_bus_find_capability_nolock(bus, devfn, PCI_CAP_ID_EXP);

#ifdef CONFIG_PM_SLEEP
       if (pm_suspend_target_state == PM_SUSPEND_ON)
             runtime_flag = 0;
#endif

       if (resume_flag && pos != 0 && (pos + PCI_EXP_DEVCTL) == reg) {
             tmp = readl(addr);
             if ((val & PCI_EXP_DEVCTL_READRQ) > (tmp & 
PCI_EXP_DEVCTL_READRQ)) {
                     val &= ~PCI_EXP_DEVCTL_READRQ;
                     val |= (tmp & PCI_EXP_DEVCTL_READRQ);
             }
       }
       return val;

}


static int loongson_pci_config_write32(struct pci_bus *bus, unsigned int 
devfn,
                                int where, int size, u32 val)
{
         void __iomem *addr;
         u32 mask, tmp;
         int reg = where & ~3;

         addr = bus->ops->map_bus(bus, devfn, where & ~0x3);
         if (!addr)
                 return PCIBIOS_DEVICE_NOT_FOUND;
         val = handle_mrrs_limit(bus, devfn, addr, reg, val);
         writel(val, addr);

         return PCIBIOS_SUCCESSFUL;
}

And I still have to emphasize on the fix in this way: It's still does 
not work for pciehp. It's only used for addressing MRRS issue of 7A 
revisions which have no pciehp support.

And in future, for new revision, we just need to skip handle_mrrs_limit.

The way described here is just my immature opinion, we can discuss it
if required.

Thanks.

On 2022/7/14 下午8:42, Huacai Chen wrote:
> In new revision of LS7A, some PCIe ports support larger value than 256,
> but their maximum supported MRRS values are not detectable. Moreover,
> the current loongson_mrrs_quirk() cannot avoid devices increasing its
> MRRS after pci_enable_device(), and some devices (e.g. Realtek 8169)
> will actually set a big value in its driver. So the only possible way
> is configure MRRS of all devices in BIOS, and add a pci host bridge bit
> flag (i.e., no_inc_mrrs) to stop the increasing MRRS operations.
> 
> However, according to PCIe Spec, it is legal for an OS to program any
> value for MRRS, and it is also legal for an endpoint to generate a Read
> Request with any size up to its MRRS. As the hardware engineers say, the
> root cause here is LS7A doesn't break up large read requests. In detail,
> LS7A PCIe port reports CA (Completer Abort) if it receives a Memory Read
> request with a size that's "too big" ("too big" means larger than the
> PCIe ports can handle, which means 256 for some ports and 4096 for the
> others, and of course this is a problem in the LS7A's hardware design).
> 
> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
> ---
>   drivers/pci/controller/pci-loongson.c | 44 +++++++++------------------
>   drivers/pci/pci.c                     |  6 ++++
>   include/linux/pci.h                   |  1 +
>   3 files changed, 22 insertions(+), 29 deletions(-)
> 
> diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
> index 594653154deb..af73bb766e48 100644
> --- a/drivers/pci/controller/pci-loongson.c
> +++ b/drivers/pci/controller/pci-loongson.c
> @@ -68,37 +68,23 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
>   DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
>   			DEV_LS7A_LPC, system_bus_quirk);
>   
> -static void loongson_mrrs_quirk(struct pci_dev *dev)
> +static void loongson_mrrs_quirk(struct pci_dev *pdev)
>   {
> -	struct pci_bus *bus = dev->bus;
> -	struct pci_dev *bridge;
> -	static const struct pci_device_id bridge_devids[] = {
> -		{ PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_0) },
> -		{ PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_1) },
> -		{ PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_2) },
> -		{ 0, },
> -	};
> -
> -	/* look for the matching bridge */
> -	while (!pci_is_root_bus(bus)) {
> -		bridge = bus->self;
> -		bus = bus->parent;
> -		/*
> -		 * Some Loongson PCIe ports have a h/w limitation of
> -		 * 256 bytes maximum read request size. They can't handle
> -		 * anything larger than this. So force this limit on
> -		 * any devices attached under these ports.
> -		 */
> -		if (pci_match_id(bridge_devids, bridge)) {
> -			if (pcie_get_readrq(dev) > 256) {
> -				pci_info(dev, "limiting MRRS to 256\n");
> -				pcie_set_readrq(dev, 256);
> -			}
> -			break;
> -		}
> -	}
> +	/*
> +	 * Some Loongson PCIe ports have h/w limitations of maximum read
> +	 * request size. They can't handle anything larger than this. So
> +	 * force this limit on any devices attached under these ports.
> +	 */
> +	struct pci_host_bridge *bridge = pci_find_host_bridge(pdev->bus);
> +
> +	bridge->no_inc_mrrs = 1;
>   }
> -DECLARE_PCI_FIXUP_ENABLE(PCI_ANY_ID, PCI_ANY_ID, loongson_mrrs_quirk);
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> +			DEV_PCIE_PORT_0, loongson_mrrs_quirk);
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> +			DEV_PCIE_PORT_1, loongson_mrrs_quirk);
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> +			DEV_PCIE_PORT_2, loongson_mrrs_quirk);
>   
>   static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
>   {
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index cfaf40a540a8..79157cbad835 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -6052,6 +6052,7 @@ int pcie_set_readrq(struct pci_dev *dev, int rq)
>   {
>   	u16 v;
>   	int ret;
> +	struct pci_host_bridge *bridge = pci_find_host_bridge(dev->bus);
>   
>   	if (rq < 128 || rq > 4096 || !is_power_of_2(rq))
>   		return -EINVAL;
> @@ -6070,6 +6071,11 @@ int pcie_set_readrq(struct pci_dev *dev, int rq)
>   
>   	v = (ffs(rq) - 8) << 12;
>   
> +	if (bridge->no_inc_mrrs) {
> +		if (rq > pcie_get_readrq(dev))
> +			return -EINVAL;
> +	}
> +
>   	ret = pcie_capability_clear_and_set_word(dev, PCI_EXP_DEVCTL,
>   						  PCI_EXP_DEVCTL_READRQ, v);
>   
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 81a57b498f22..a9211074add6 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -569,6 +569,7 @@ struct pci_host_bridge {
>   	void		*release_data;
>   	unsigned int	ignore_reset_delay:1;	/* For entire hierarchy */
>   	unsigned int	no_ext_tags:1;		/* No Extended Tags */
> +	unsigned int	no_inc_mrrs:1;		/* No Increase MRRS */
>   	unsigned int	native_aer:1;		/* OS may use PCIe AER */
>   	unsigned int	native_pcie_hotplug:1;	/* OS may use PCIe hotplug */
>   	unsigned int	native_shpc_hotplug:1;	/* OS may use SHPC hotplug */
> 

Re: [PATCH V16 5/7] PCI: loongson: Improve the MRRS quirk for LS7A

[Huacai Chen]

Hi, Jianmin,

On Sat, Jul 16, 2022 at 3:15 PM Jianmin Lv <lvjianmin@loongson.cn> wrote:
>
> Hi, Huacai and Bjorn,
>
> Actually, I don't think we have to fix the MRRS issue of 7A in this
> way(change core code of PCI). The reasons are:
>
> - First, we don't know if other pci controlers have simillar issue, at
> least I don't see yet. I think if *only we* have the issue, maybe we can
> fix it in our controller driver rather than changing core code. And in
> future, if the issue is proved a common one, abstract it then by common way.
>
Keystone's pci controller has the same problem (maybe DesignWare based
controllers all have the same problem), this is discussed for a long
time.

Huacai
> - Second, even though we limit the MRRS in pcie_set_readrq() according
> to the flag set in quirk function, some drivers(e.g. in
> drivers/rapidio/devices/tsi721.c, maybe other driver do this in future,
> I dont't know.) directly call pcie_capability_clear_and_set_word to set
> MRRS, which will still break the fix.
>
> - Third, on resuming from S3, the MRRS stored in memory should be
> allowed to set to dev ctrl reg(because the reg has been reset during S3).
>
>
> Fix MMRS in our controller driver by using self-defined config_write(),
> maybe like this:
>
> static u32 handle_mrrs_limit(struct pci_bus *bus, unsigned int devfn,
> void __iomem *addr, u32 val)
> {
>       u32 tmp;
>       bool runtime_flag = 1;
>       int pos = pci_bus_find_capability_nolock(bus, devfn, PCI_CAP_ID_EXP);
>
> #ifdef CONFIG_PM_SLEEP
>        if (pm_suspend_target_state == PM_SUSPEND_ON)
>              runtime_flag = 0;
> #endif
>
>        if (resume_flag && pos != 0 && (pos + PCI_EXP_DEVCTL) == reg) {
>              tmp = readl(addr);
>              if ((val & PCI_EXP_DEVCTL_READRQ) > (tmp &
> PCI_EXP_DEVCTL_READRQ)) {
>                      val &= ~PCI_EXP_DEVCTL_READRQ;
>                      val |= (tmp & PCI_EXP_DEVCTL_READRQ);
>              }
>        }
>        return val;
>
> }
>
>
> static int loongson_pci_config_write32(struct pci_bus *bus, unsigned int
> devfn,
>                                 int where, int size, u32 val)
> {
>          void __iomem *addr;
>          u32 mask, tmp;
>          int reg = where & ~3;
>
>          addr = bus->ops->map_bus(bus, devfn, where & ~0x3);
>          if (!addr)
>                  return PCIBIOS_DEVICE_NOT_FOUND;
>          val = handle_mrrs_limit(bus, devfn, addr, reg, val);
>          writel(val, addr);
>
>          return PCIBIOS_SUCCESSFUL;
> }
>
> And I still have to emphasize on the fix in this way: It's still does
> not work for pciehp. It's only used for addressing MRRS issue of 7A
> revisions which have no pciehp support.
>
> And in future, for new revision, we just need to skip handle_mrrs_limit.
>
> The way described here is just my immature opinion, we can discuss it
> if required.
>
> Thanks.
>
> On 2022/7/14 下午8:42, Huacai Chen wrote:
> > In new revision of LS7A, some PCIe ports support larger value than 256,
> > but their maximum supported MRRS values are not detectable. Moreover,
> > the current loongson_mrrs_quirk() cannot avoid devices increasing its
> > MRRS after pci_enable_device(), and some devices (e.g. Realtek 8169)
> > will actually set a big value in its driver. So the only possible way
> > is configure MRRS of all devices in BIOS, and add a pci host bridge bit
> > flag (i.e., no_inc_mrrs) to stop the increasing MRRS operations.
> >
> > However, according to PCIe Spec, it is legal for an OS to program any
> > value for MRRS, and it is also legal for an endpoint to generate a Read
> > Request with any size up to its MRRS. As the hardware engineers say, the
> > root cause here is LS7A doesn't break up large read requests. In detail,
> > LS7A PCIe port reports CA (Completer Abort) if it receives a Memory Read
> > request with a size that's "too big" ("too big" means larger than the
> > PCIe ports can handle, which means 256 for some ports and 4096 for the
> > others, and of course this is a problem in the LS7A's hardware design).
> >
> > Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
> > ---
> >   drivers/pci/controller/pci-loongson.c | 44 +++++++++------------------
> >   drivers/pci/pci.c                     |  6 ++++
> >   include/linux/pci.h                   |  1 +
> >   3 files changed, 22 insertions(+), 29 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
> > index 594653154deb..af73bb766e48 100644
> > --- a/drivers/pci/controller/pci-loongson.c
> > +++ b/drivers/pci/controller/pci-loongson.c
> > @@ -68,37 +68,23 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> >   DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> >                       DEV_LS7A_LPC, system_bus_quirk);
> >
> > -static void loongson_mrrs_quirk(struct pci_dev *dev)
> > +static void loongson_mrrs_quirk(struct pci_dev *pdev)
> >   {
> > -     struct pci_bus *bus = dev->bus;
> > -     struct pci_dev *bridge;
> > -     static const struct pci_device_id bridge_devids[] = {
> > -             { PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_0) },
> > -             { PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_1) },
> > -             { PCI_VDEVICE(LOONGSON, DEV_PCIE_PORT_2) },
> > -             { 0, },
> > -     };
> > -
> > -     /* look for the matching bridge */
> > -     while (!pci_is_root_bus(bus)) {
> > -             bridge = bus->self;
> > -             bus = bus->parent;
> > -             /*
> > -              * Some Loongson PCIe ports have a h/w limitation of
> > -              * 256 bytes maximum read request size. They can't handle
> > -              * anything larger than this. So force this limit on
> > -              * any devices attached under these ports.
> > -              */
> > -             if (pci_match_id(bridge_devids, bridge)) {
> > -                     if (pcie_get_readrq(dev) > 256) {
> > -                             pci_info(dev, "limiting MRRS to 256\n");
> > -                             pcie_set_readrq(dev, 256);
> > -                     }
> > -                     break;
> > -             }
> > -     }
> > +     /*
> > +      * Some Loongson PCIe ports have h/w limitations of maximum read
> > +      * request size. They can't handle anything larger than this. So
> > +      * force this limit on any devices attached under these ports.
> > +      */
> > +     struct pci_host_bridge *bridge = pci_find_host_bridge(pdev->bus);
> > +
> > +     bridge->no_inc_mrrs = 1;
> >   }
> > -DECLARE_PCI_FIXUP_ENABLE(PCI_ANY_ID, PCI_ANY_ID, loongson_mrrs_quirk);
> > +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> > +                     DEV_PCIE_PORT_0, loongson_mrrs_quirk);
> > +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> > +                     DEV_PCIE_PORT_1, loongson_mrrs_quirk);
> > +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
> > +                     DEV_PCIE_PORT_2, loongson_mrrs_quirk);
> >
> >   static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
> >   {
> > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > index cfaf40a540a8..79157cbad835 100644
> > --- a/drivers/pci/pci.c
> > +++ b/drivers/pci/pci.c
> > @@ -6052,6 +6052,7 @@ int pcie_set_readrq(struct pci_dev *dev, int rq)
> >   {
> >       u16 v;
> >       int ret;
> > +     struct pci_host_bridge *bridge = pci_find_host_bridge(dev->bus);
> >
> >       if (rq < 128 || rq > 4096 || !is_power_of_2(rq))
> >               return -EINVAL;
> > @@ -6070,6 +6071,11 @@ int pcie_set_readrq(struct pci_dev *dev, int rq)
> >
> >       v = (ffs(rq) - 8) << 12;
> >
> > +     if (bridge->no_inc_mrrs) {
> > +             if (rq > pcie_get_readrq(dev))
> > +                     return -EINVAL;
> > +     }
> > +
> >       ret = pcie_capability_clear_and_set_word(dev, PCI_EXP_DEVCTL,
> >                                                 PCI_EXP_DEVCTL_READRQ, v);
> >
> > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > index 81a57b498f22..a9211074add6 100644
> > --- a/include/linux/pci.h
> > +++ b/include/linux/pci.h
> > @@ -569,6 +569,7 @@ struct pci_host_bridge {
> >       void            *release_data;
> >       unsigned int    ignore_reset_delay:1;   /* For entire hierarchy */
> >       unsigned int    no_ext_tags:1;          /* No Extended Tags */
> > +     unsigned int    no_inc_mrrs:1;          /* No Increase MRRS */
> >       unsigned int    native_aer:1;           /* OS may use PCIe AER */
> >       unsigned int    native_pcie_hotplug:1;  /* OS may use PCIe hotplug */
> >       unsigned int    native_shpc_hotplug:1;  /* OS may use SHPC hotplug */
> >
>

[PATCH V16 6/7] PCI: Add quirk for LS7A to avoid reboot failure

[Huacai Chen]

cc27b735ad3a7557 ("PCI/portdrv: Turn off PCIe services during shutdown")
causes poweroff/reboot failure on systems with LS7A chipset. We found
that if we remove "pci_command &= ~PCI_COMMAND_MASTER" in do_pci_disable
_device(), it can work well. The hardware engineer says that the root
cause is that CPU is still accessing PCIe devices while poweroff/reboot,
and if we disable the Bus Master Bit at this time, the PCIe controller
doesn't forward requests to downstream devices, and also does not send
TIMEOUT to CPU, which causes CPU wait forever (hardware deadlock). This
behavior is a PCIe protocol violation (Bus Master should not be involved
in CPU MMIO transactions), and it will be fixed in new revisions of
hardware (add timeout mechanism for CPU read request, whether or not Bus
Master bit is cleared).

On some x86 platforms, radeon/amdgpu devices can cause similar problems
[1][2]. Once before I wanted to make a single patch to solve "all of
these problems" together, but it seems unreasonable because maybe they
are not exactly the same problem. So, this patch add a new function
pcie_portdrv_shutdown(), a slight modified copy of pcie_portdrv_remove()
dedicated for the shutdown path, and then add a quirk just for LS7A to
avoid clearing Bus Master bit in pcie_portdrv_shutdown(). Leave other
platforms behave as before.

[1] https://bugs.freedesktop.org/show_bug.cgi?id=97980
[2] https://bugs.freedesktop.org/show_bug.cgi?id=98638

Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
---
 drivers/pci/controller/pci-loongson.c | 17 +++++++++++++++++
 drivers/pci/pcie/portdrv_core.c       |  1 -
 drivers/pci/pcie/portdrv_pci.c        | 20 +++++++++++++++++++-
 include/linux/pci.h                   |  1 +
 4 files changed, 37 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
index af73bb766e48..05997b51c86d 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -86,6 +86,23 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
 			DEV_PCIE_PORT_2, loongson_mrrs_quirk);
 
+static void loongson_bmaster_quirk(struct pci_dev *pdev)
+{
+	/*
+	 * Some Loongson PCIe ports will cause CPU deadlock if disable
+	 * the Bus Master bit during poweroff/reboot.
+	 */
+	struct pci_host_bridge *bridge = pci_find_host_bridge(pdev->bus);
+
+	bridge->no_dis_bmaster = 1;
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_0, loongson_bmaster_quirk);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_1, loongson_bmaster_quirk);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_2, loongson_bmaster_quirk);
+
 static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
 {
 	struct pci_config_window *cfg;
diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index 604feeb84ee4..ee3d654dcbb4 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -493,7 +493,6 @@ void pcie_port_device_remove(struct pci_dev *dev)
 {
 	device_for_each_child(&dev->dev, NULL, remove_iter);
 	pci_free_irq_vectors(dev);
-	pci_disable_device(dev);
 }
 
 /**
diff --git a/drivers/pci/pcie/portdrv_pci.c b/drivers/pci/pcie/portdrv_pci.c
index 7f8788a970ae..f821f916d020 100644
--- a/drivers/pci/pcie/portdrv_pci.c
+++ b/drivers/pci/pcie/portdrv_pci.c
@@ -148,6 +148,24 @@ static void pcie_portdrv_remove(struct pci_dev *dev)
 	}
 
 	pcie_port_device_remove(dev);
+
+	pci_disable_device(dev);
+}
+
+static void pcie_portdrv_shutdown(struct pci_dev *dev)
+{
+	struct pci_host_bridge *bridge = pci_find_host_bridge(dev->bus);
+
+	if (pci_bridge_d3_possible(dev)) {
+		pm_runtime_forbid(&dev->dev);
+		pm_runtime_get_noresume(&dev->dev);
+		pm_runtime_dont_use_autosuspend(&dev->dev);
+	}
+
+	pcie_port_device_remove(dev);
+
+	if (!bridge->no_dis_bmaster)
+		pci_disable_device(dev);
 }
 
 static pci_ers_result_t pcie_portdrv_error_detected(struct pci_dev *dev,
@@ -198,7 +216,7 @@ static struct pci_driver pcie_portdriver = {
 
 	.probe		= pcie_portdrv_probe,
 	.remove		= pcie_portdrv_remove,
-	.shutdown	= pcie_portdrv_remove,
+	.shutdown	= pcie_portdrv_shutdown,
 
 	.err_handler	= &pcie_portdrv_err_handler,
 
diff --git a/include/linux/pci.h b/include/linux/pci.h
index a9211074add6..0f0908679074 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -570,6 +570,7 @@ struct pci_host_bridge {
 	unsigned int	ignore_reset_delay:1;	/* For entire hierarchy */
 	unsigned int	no_ext_tags:1;		/* No Extended Tags */
 	unsigned int	no_inc_mrrs:1;		/* No Increase MRRS */
+	unsigned int	no_dis_bmaster:1;	/* No Disable Bus Master */
 	unsigned int	native_aer:1;		/* OS may use PCIe AER */
 	unsigned int	native_pcie_hotplug:1;	/* OS may use PCIe hotplug */
 	unsigned int	native_shpc_hotplug:1;	/* OS may use SHPC hotplug */
-- 
2.31.1

[PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Huacai Chen]

From: Jianmin Lv <lvjianmin@loongson.cn>

In LS7A, multifunction device use same PCI PIN (because the PIN register
report the same INTx value to each function) but we need different IRQ
for different functions, so add a quirk to fix it for standard PCI PIN
usage.

This patch only affect ACPI based systems (and only needed by ACPI based
systems, too). For DT based systems, the irq mappings is defined in .dts
files and be handled by of_irq_parse_pci().

Signed-off-by: Jianmin Lv <lvjianmin@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
---
 drivers/pci/controller/pci-loongson.c | 32 +++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
index 05997b51c86d..4043b57bcc86 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -22,6 +22,13 @@
 #define DEV_LS2K_APB	0x7a02
 #define DEV_LS7A_CONF	0x7a10
 #define DEV_LS7A_LPC	0x7a0c
+#define DEV_LS7A_GMAC	0x7a03
+#define DEV_LS7A_DC1	0x7a06
+#define DEV_LS7A_DC2	0x7a36
+#define DEV_LS7A_GPU	0x7a15
+#define DEV_LS7A_AHCI	0x7a08
+#define DEV_LS7A_EHCI	0x7a14
+#define DEV_LS7A_OHCI	0x7a24
 
 #define FLAG_CFG0	BIT(0)
 #define FLAG_CFG1	BIT(1)
@@ -103,6 +110,31 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
 			DEV_PCIE_PORT_2, loongson_bmaster_quirk);
 
+static void loongson_pci_pin_quirk(struct pci_dev *pdev)
+{
+	pdev->pin = 1 + (PCI_FUNC(pdev->devfn) & 3);
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_LS7A_DC1, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_LS7A_DC2, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_LS7A_GPU, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_LS7A_GMAC, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_LS7A_AHCI, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_LS7A_EHCI, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_LS7A_OHCI, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_0, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_1, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+			DEV_PCIE_PORT_2, loongson_pci_pin_quirk);
+
 static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
 {
 	struct pci_config_window *cfg;
-- 
2.31.1

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Bjorn Helgaas]

On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> From: Jianmin Lv <lvjianmin@loongson.cn>
> 
> In LS7A, multifunction device use same PCI PIN (because the PIN register
> report the same INTx value to each function) but we need different IRQ
> for different functions, so add a quirk to fix it for standard PCI PIN
> usage.
> 
> This patch only affect ACPI based systems (and only needed by ACPI based
> systems, too). For DT based systems, the irq mappings is defined in .dts
> files and be handled by of_irq_parse_pci().

I'm sorry, I know you've explained this before, but I don't understand
yet, so let's try again.  I *think* you're saying that:

  - These devices integrated into LS7A all report 0 in their Interrupt
    Pin registers.  Per spec, this means they do not use INTx (PCIe
    r6.0, sec 7.5.1.1.13).

  - However, these devices actually *do* use INTx.  Function 0 uses
    INTA, function 1 uses INTB, ..., function 4 uses INTA, ...

  - The quirk overrides the incorrect values read from the Interrupt
    Pin registers.

That much makes sense to me.

And I even see that in of_irq_parse_pci(), if there's a DT node for
the device, of_irq_parse_one() gets the interrupt info from DT and
returns the IRQ all the way back up to (I think) loongson_map_irq().

But I'm still confused about how loongson_map_irq() gets called.  The
only likely path I see is here:

  pci_device_probe                            # pci_bus_type.probe
    pci_assign_irq
      pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
      if (pin)
	bridge->swizzle_irq(dev, &pin)
	irq = bridge->map_irq(dev, slot, pin)

where bridge->map_irq points to loongson_map_irq().  But
pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
wouldn't call bridge->map_irq().  Obviously I'm missing something.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/setup-irq.c?id=v5.18#n37

> Signed-off-by: Jianmin Lv <lvjianmin@loongson.cn>
> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
> ---
>  drivers/pci/controller/pci-loongson.c | 32 +++++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
> 
> diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
> index 05997b51c86d..4043b57bcc86 100644
> --- a/drivers/pci/controller/pci-loongson.c
> +++ b/drivers/pci/controller/pci-loongson.c
> @@ -22,6 +22,13 @@
>  #define DEV_LS2K_APB	0x7a02
>  #define DEV_LS7A_CONF	0x7a10
>  #define DEV_LS7A_LPC	0x7a0c
> +#define DEV_LS7A_GMAC	0x7a03
> +#define DEV_LS7A_DC1	0x7a06
> +#define DEV_LS7A_DC2	0x7a36
> +#define DEV_LS7A_GPU	0x7a15
> +#define DEV_LS7A_AHCI	0x7a08
> +#define DEV_LS7A_EHCI	0x7a14
> +#define DEV_LS7A_OHCI	0x7a24
>  
>  #define FLAG_CFG0	BIT(0)
>  #define FLAG_CFG1	BIT(1)
> @@ -103,6 +110,31 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
>  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
>  			DEV_PCIE_PORT_2, loongson_bmaster_quirk);
>  
> +static void loongson_pci_pin_quirk(struct pci_dev *pdev)
> +{
> +	pdev->pin = 1 + (PCI_FUNC(pdev->devfn) & 3);
> +}
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_LS7A_DC1, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_LS7A_DC2, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_LS7A_GPU, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_LS7A_GMAC, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_LS7A_AHCI, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_LS7A_EHCI, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_LS7A_OHCI, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_PCIE_PORT_0, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_PCIE_PORT_1, loongson_pci_pin_quirk);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
> +			DEV_PCIE_PORT_2, loongson_pci_pin_quirk);
> +
>  static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
>  {
>  	struct pci_config_window *cfg;
> -- 
> 2.31.1
> 

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Jianmin Lv]

On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
>> From: Jianmin Lv <lvjianmin@loongson.cn>
>>
>> In LS7A, multifunction device use same PCI PIN (because the PIN register
>> report the same INTx value to each function) but we need different IRQ
>> for different functions, so add a quirk to fix it for standard PCI PIN
>> usage.
>>
>> This patch only affect ACPI based systems (and only needed by ACPI based
>> systems, too). For DT based systems, the irq mappings is defined in .dts
>> files and be handled by of_irq_parse_pci().
> 
> I'm sorry, I know you've explained this before, but I don't understand
> yet, so let's try again.  I *think* you're saying that:
> 
>    - These devices integrated into LS7A all report 0 in their Interrupt
>      Pin registers.  Per spec, this means they do not use INTx (PCIe
>      r6.0, sec 7.5.1.1.13).
> 
>    - However, these devices actually *do* use INTx.  Function 0 uses
>      INTA, function 1 uses INTB, ..., function 4 uses INTA, ...
> 
>    - The quirk overrides the incorrect values read from the Interrupt
>      Pin registers.
> 

Yes, right.


> That much makes sense to me.
> 
> And I even see that in of_irq_parse_pci(), if there's a DT node for
> the device, of_irq_parse_one() gets the interrupt info from DT and
> returns the IRQ all the way back up to (I think) loongson_map_irq().
> 

Agree, I think so for DT.


> But I'm still confused about how loongson_map_irq() gets called.  The
> only likely path I see is here:
> 
>    pci_device_probe                            # pci_bus_type.probe
>      pci_assign_irq
>        pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
>        if (pin)
> 	bridge->swizzle_irq(dev, &pin)
> 	irq = bridge->map_irq(dev, slot, pin)
> 
> where bridge->map_irq points to loongson_map_irq().  But
> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> wouldn't call bridge->map_irq().  Obviously I'm missing something.
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/setup-irq.c?id=v5.18#n37
> 

For ACPI, bridge->map_irq is NULL, so in above path,
the pci_assign_irq will return because of !(hbrg->map_irq) as following:

         if (!(hbrg->map_irq)) {
                 pci_dbg(dev, "runtime IRQ mapping not provided by arch\n");
                 return;
         }

And again as I explained in previous version patch, dev->irq is set in
acpi_pci_irq_enable() in the following path for ACPI:

pci_device_probe
   ->pcibios_alloc_irq
     ->acpi_pci_irq_enable
       ->acpi_pci_irq_lookup

And the reason that we fixed the pin is to get an correct entry in prt
table when calling acpi_pci_irq_lookup. With out the fix, we can't find
out a entry.

After found an entry, we get gsi, and map irq as following:

         rc = acpi_register_gsi(&dev->dev, gsi, triggering, polarity);
         if (rc < 0) {
                 dev_warn(&dev->dev, "PCI INT %c: failed to register GSI\n",
                          pin_name(pin));
                 kfree(entry);
                 return rc;
         }
         dev->irq = rc;

Here, dev->irq is set like in pci_assign_irq for DT.


>> Signed-off-by: Jianmin Lv <lvjianmin@loongson.cn>
>> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
>> ---
>>   drivers/pci/controller/pci-loongson.c | 32 +++++++++++++++++++++++++++
>>   1 file changed, 32 insertions(+)
>>
>> diff --git a/drivers/pci/controller/pci-loongson.c b/drivers/pci/controller/pci-loongson.c
>> index 05997b51c86d..4043b57bcc86 100644
>> --- a/drivers/pci/controller/pci-loongson.c
>> +++ b/drivers/pci/controller/pci-loongson.c
>> @@ -22,6 +22,13 @@
>>   #define DEV_LS2K_APB	0x7a02
>>   #define DEV_LS7A_CONF	0x7a10
>>   #define DEV_LS7A_LPC	0x7a0c
>> +#define DEV_LS7A_GMAC	0x7a03
>> +#define DEV_LS7A_DC1	0x7a06
>> +#define DEV_LS7A_DC2	0x7a36
>> +#define DEV_LS7A_GPU	0x7a15
>> +#define DEV_LS7A_AHCI	0x7a08
>> +#define DEV_LS7A_EHCI	0x7a14
>> +#define DEV_LS7A_OHCI	0x7a24
>>   
>>   #define FLAG_CFG0	BIT(0)
>>   #define FLAG_CFG1	BIT(1)
>> @@ -103,6 +110,31 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
>>   DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
>>   			DEV_PCIE_PORT_2, loongson_bmaster_quirk);
>>   
>> +static void loongson_pci_pin_quirk(struct pci_dev *pdev)
>> +{
>> +	pdev->pin = 1 + (PCI_FUNC(pdev->devfn) & 3);
>> +}
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_LS7A_DC1, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_LS7A_DC2, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_LS7A_GPU, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_LS7A_GMAC, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_LS7A_AHCI, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_LS7A_EHCI, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_LS7A_OHCI, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_PCIE_PORT_0, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_PCIE_PORT_1, loongson_pci_pin_quirk);
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
>> +			DEV_PCIE_PORT_2, loongson_pci_pin_quirk);
>> +
>>   static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
>>   {
>>   	struct pci_config_window *cfg;
>> -- 
>> 2.31.1
>>

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Bjorn Helgaas]

On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> > On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> > > From: Jianmin Lv <lvjianmin@loongson.cn>
> > > 
> > > In LS7A, multifunction device use same PCI PIN (because the PIN register
> > > report the same INTx value to each function) but we need different IRQ
> > > for different functions, so add a quirk to fix it for standard PCI PIN
> > > usage.
> > > 
> > > This patch only affect ACPI based systems (and only needed by ACPI based
> > > systems, too). For DT based systems, the irq mappings is defined in .dts
> > > files and be handled by of_irq_parse_pci().
> > 
> > I'm sorry, I know you've explained this before, but I don't understand
> > yet, so let's try again.  I *think* you're saying that:
> > 
> >    - These devices integrated into LS7A all report 0 in their Interrupt
> >      Pin registers.  Per spec, this means they do not use INTx (PCIe
> >      r6.0, sec 7.5.1.1.13).
> > 
> >    - However, these devices actually *do* use INTx.  Function 0 uses
> >      INTA, function 1 uses INTB, ..., function 4 uses INTA, ...
> > 
> >    - The quirk overrides the incorrect values read from the Interrupt
> >      Pin registers.
> 
> Yes, right.
> 
> > That much makes sense to me.
> > 
> > And I even see that in of_irq_parse_pci(), if there's a DT node for
> > the device, of_irq_parse_one() gets the interrupt info from DT and
> > returns the IRQ all the way back up to (I think) loongson_map_irq().
> 
> Agree, I think so for DT.
> 
> > But I'm still confused about how loongson_map_irq() gets called.  The
> > only likely path I see is here:
> > 
> >    pci_device_probe                            # pci_bus_type.probe
> >      pci_assign_irq
> >        pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
> >        if (pin)
> > 	bridge->swizzle_irq(dev, &pin)
> > 	irq = bridge->map_irq(dev, slot, pin)
> > 
> > where bridge->map_irq points to loongson_map_irq().  But
> > pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> > wouldn't call bridge->map_irq().  Obviously I'm missing something.
> > 
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/setup-irq.c?id=v5.18#n37
> 
> For ACPI, bridge->map_irq is NULL, so in above path,
> the pci_assign_irq will return because of !(hbrg->map_irq) as following:
> 
>         if (!(hbrg->map_irq)) {
>                 pci_dbg(dev, "runtime IRQ mapping not provided by arch\n");
>                 return;
>         }
> 
> And again as I explained in previous version patch, dev->irq is set in
> acpi_pci_irq_enable() in the following path for ACPI:
> 
> pci_device_probe
>   ->pcibios_alloc_irq
>     ->acpi_pci_irq_enable
>       ->acpi_pci_irq_lookup
> 
> And the reason that we fixed the pin is to get an correct entry in prt
> table when calling acpi_pci_irq_lookup. With out the fix, we can't find
> out a entry.
> 
> After found an entry, we get gsi, and map irq as following:
> 
>         rc = acpi_register_gsi(&dev->dev, gsi, triggering, polarity);
>         if (rc < 0) {
>                 dev_warn(&dev->dev, "PCI INT %c: failed to register GSI\n",
>                          pin_name(pin));
>                 kfree(entry);
>                 return rc;
>         }
>         dev->irq = rc;
> 
> Here, dev->irq is set like in pci_assign_irq for DT.

Yes.  The above explains how things work for ACPI, but I'm not asking
about that.

I'm asking how this works in the *DT* case.  I see that
pci_assign_irq() is called for both ACPI and DT, and I see that it
does nothing in the ACPI path because bridge->map_irq hasn't been set.

What I *don't* see is how pci_assign_irq() works in the DT case
because it reads PCI_INTERRUPT_PIN, which should return 0 for these
broken devices, and if "pin == 0", it never calls ->map_irq().

Is ->map_irq() called via some other path?

Bjorn

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Jianmin Lv]

On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
>>>>
>>>> In LS7A, multifunction device use same PCI PIN (because the PIN register
>>>> report the same INTx value to each function) but we need different IRQ
>>>> for different functions, so add a quirk to fix it for standard PCI PIN
>>>> usage.
>>>>
>>>> This patch only affect ACPI based systems (and only needed by ACPI based
>>>> systems, too). For DT based systems, the irq mappings is defined in .dts
>>>> files and be handled by of_irq_parse_pci().
>>>
>>> I'm sorry, I know you've explained this before, but I don't understand
>>> yet, so let's try again.  I *think* you're saying that:
>>>
>>>     - These devices integrated into LS7A all report 0 in their Interrupt
>>>       Pin registers.  Per spec, this means they do not use INTx (PCIe
>>>       r6.0, sec 7.5.1.1.13).
>>>
>>>     - However, these devices actually *do* use INTx.  Function 0 uses
>>>       INTA, function 1 uses INTB, ..., function 4 uses INTA, ...
>>>
>>>     - The quirk overrides the incorrect values read from the Interrupt
>>>       Pin registers.
>>
>> Yes, right.
>>

Sorry, I didn't see the first item here carefully, so I have to correct 
it: all the integrated devices in 7A report 1 in PIN reg instead of 0.

>>> That much makes sense to me.
>>>
>>> And I even see that in of_irq_parse_pci(), if there's a DT node for
>>> the device, of_irq_parse_one() gets the interrupt info from DT and
>>> returns the IRQ all the way back up to (I think) loongson_map_irq().
>>
>> Agree, I think so for DT.
>>
>>> But I'm still confused about how loongson_map_irq() gets called.  The
>>> only likely path I see is here:
>>>
>>>     pci_device_probe                            # pci_bus_type.probe
>>>       pci_assign_irq
>>>         pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
>>>         if (pin)
>>> 	bridge->swizzle_irq(dev, &pin)
>>> 	irq = bridge->map_irq(dev, slot, pin)
>>>
>>> where bridge->map_irq points to loongson_map_irq().  But
>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
>>>

Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be 
called.

>>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/setup-irq.c?id=v5.18#n37
>>
>> For ACPI, bridge->map_irq is NULL, so in above path,
>> the pci_assign_irq will return because of !(hbrg->map_irq) as following:
>>
>>          if (!(hbrg->map_irq)) {
>>                  pci_dbg(dev, "runtime IRQ mapping not provided by arch\n");
>>                  return;
>>          }
>>
>> And again as I explained in previous version patch, dev->irq is set in
>> acpi_pci_irq_enable() in the following path for ACPI:
>>
>> pci_device_probe
>>    ->pcibios_alloc_irq
>>      ->acpi_pci_irq_enable
>>        ->acpi_pci_irq_lookup
>>
>> And the reason that we fixed the pin is to get an correct entry in prt
>> table when calling acpi_pci_irq_lookup. With out the fix, we can't find
>> out a entry.
>>
>> After found an entry, we get gsi, and map irq as following:
>>
>>          rc = acpi_register_gsi(&dev->dev, gsi, triggering, polarity);
>>          if (rc < 0) {
>>                  dev_warn(&dev->dev, "PCI INT %c: failed to register GSI\n",
>>                           pin_name(pin));
>>                  kfree(entry);
>>                  return rc;
>>          }
>>          dev->irq = rc;
>>
>> Here, dev->irq is set like in pci_assign_irq for DT.
> 
> Yes.  The above explains how things work for ACPI, but I'm not asking
> about that.
> 
> I'm asking how this works in the *DT* case.  I see that
> pci_assign_irq() is called for both ACPI and DT, and I see that it
> does nothing in the ACPI path because bridge->map_irq hasn't been set.
> 
> What I *don't* see is how pci_assign_irq() works in the DT case
> because it reads PCI_INTERRUPT_PIN, which should return 0 for these
> broken devices, and if "pin == 0", it never calls ->map_irq().
> 
> Is ->map_irq() called via some other path?
> 

Same as above.

> Bjorn
> 

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Bjorn Helgaas]

On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
> > On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
> > > On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> > > > On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> > > > > From: Jianmin Lv <lvjianmin@loongson.cn>
> > > > > 
> > > > > In LS7A, multifunction device use same PCI PIN (because the
> > > > > PIN register report the same INTx value to each function)
> > > > > but we need different IRQ for different functions, so add a
> > > > > quirk to fix it for standard PCI PIN usage.
> > > > > 
> > > > > This patch only affect ACPI based systems (and only needed
> > > > > by ACPI based systems, too). For DT based systems, the irq
> > > > > mappings is defined in .dts files and be handled by
> > > > > of_irq_parse_pci().
> > > > 
> > > > I'm sorry, I know you've explained this before, but I don't
> > > > understand yet, so let's try again.  I *think* you're saying
> > > > that:
> > > > 
> > > >     - These devices integrated into LS7A all report 0 in their
> > > >     Interrupt Pin registers.  Per spec, this means they do not
> > > >     use INTx (PCIe r6.0, sec 7.5.1.1.13).
> > > > 
> > > >     - However, these devices actually *do* use INTx.  Function
> > > >     0 uses INTA, function 1 uses INTB, ..., function 4 uses
> > > >     INTA, ...
> > > > 
> > > >     - The quirk overrides the incorrect values read from the
> > > >     Interrupt Pin registers.
> > > 
> > > Yes, right.
> 
> Sorry, I didn't see the first item here carefully, so I have to
> correct it: all the integrated devices in 7A report 1 in PIN reg
> instead of 0.

> > > > But I'm still confused about how loongson_map_irq() gets called.  The
> > > > only likely path I see is here:
> > > > 
> > > >     pci_device_probe                            # pci_bus_type.probe
> > > >       pci_assign_irq
> > > >         pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
> > > >         if (pin)
> > > > 	bridge->swizzle_irq(dev, &pin)
> > > > 	irq = bridge->map_irq(dev, slot, pin)
> > > > 
> > > > where bridge->map_irq points to loongson_map_irq().  But
> > > > pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> > > > wouldn't call bridge->map_irq().  Obviously I'm missing something.
> > > > 
> 
> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
> called.

OK, that makes a lot more sense, thank you!

But it does leave another question: the quirk applies to
DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
DEV_PCIE_PORT_2 (0x7a29).

According to the .dtsi [1], all those root ports are at function 0,
and if they report INTA, the quirk will also compute INTA.  So why do
you need to apply the quirk for them?

The same would apply to any Device ID that only appears at function 0,
which looks like it also includes DEV_LS7A_OHCI (0x7a24), and
DEV_LS7A_GPU (0x7a15).

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/mips/boot/dts/loongson/ls7a-pch.dtsi?id=v5.18#n231

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Jianmin Lv]

On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
> On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
>> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
>>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
>>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
>>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
>>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
>>>>>>
>>>>>> In LS7A, multifunction device use same PCI PIN (because the
>>>>>> PIN register report the same INTx value to each function)
>>>>>> but we need different IRQ for different functions, so add a
>>>>>> quirk to fix it for standard PCI PIN usage.
>>>>>>
>>>>>> This patch only affect ACPI based systems (and only needed
>>>>>> by ACPI based systems, too). For DT based systems, the irq
>>>>>> mappings is defined in .dts files and be handled by
>>>>>> of_irq_parse_pci().
>>>>>
>>>>> I'm sorry, I know you've explained this before, but I don't
>>>>> understand yet, so let's try again.  I *think* you're saying
>>>>> that:
>>>>>
>>>>>      - These devices integrated into LS7A all report 0 in their
>>>>>      Interrupt Pin registers.  Per spec, this means they do not
>>>>>      use INTx (PCIe r6.0, sec 7.5.1.1.13).
>>>>>
>>>>>      - However, these devices actually *do* use INTx.  Function
>>>>>      0 uses INTA, function 1 uses INTB, ..., function 4 uses
>>>>>      INTA, ...
>>>>>
>>>>>      - The quirk overrides the incorrect values read from the
>>>>>      Interrupt Pin registers.
>>>>
>>>> Yes, right.
>>
>> Sorry, I didn't see the first item here carefully, so I have to
>> correct it: all the integrated devices in 7A report 1 in PIN reg
>> instead of 0.
> 
>>>>> But I'm still confused about how loongson_map_irq() gets called.  The
>>>>> only likely path I see is here:
>>>>>
>>>>>      pci_device_probe                            # pci_bus_type.probe
>>>>>        pci_assign_irq
>>>>>          pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
>>>>>          if (pin)
>>>>> 	bridge->swizzle_irq(dev, &pin)
>>>>> 	irq = bridge->map_irq(dev, slot, pin)
>>>>>
>>>>> where bridge->map_irq points to loongson_map_irq().  But
>>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
>>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
>>>>>
>>
>> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
>> called.
> 
> OK, that makes a lot more sense, thank you!
> 
> But it does leave another question: the quirk applies to
> DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
> DEV_PCIE_PORT_2 (0x7a29).
> 
> According to the .dtsi [1], all those root ports are at function 0,
> and if they report INTA, the quirk will also compute INTA.  So why do
> you need to apply the quirk for them?
> 

Oh, yes, I don't think they are required either. The fix is only 
required for multi-func devices of 7A.

Huacai, we should remove PCIE ports from the patch.

> The same would apply to any Device ID that only appears at function 0,
> which looks like it also includes DEV_LS7A_OHCI (0x7a24), and
> DEV_LS7A_GPU (0x7a15).
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/mips/boot/dts/loongson/ls7a-pch.dtsi?id=v5.18#n231
> 

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Jianmin Lv]

On 2022/7/16 下午2:12, Jianmin Lv wrote:
> 
> 
> On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
>> On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
>>> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
>>>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
>>>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
>>>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
>>>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
>>>>>>>
>>>>>>> In LS7A, multifunction device use same PCI PIN (because the
>>>>>>> PIN register report the same INTx value to each function)
>>>>>>> but we need different IRQ for different functions, so add a
>>>>>>> quirk to fix it for standard PCI PIN usage.
>>>>>>>
>>>>>>> This patch only affect ACPI based systems (and only needed
>>>>>>> by ACPI based systems, too). For DT based systems, the irq
>>>>>>> mappings is defined in .dts files and be handled by
>>>>>>> of_irq_parse_pci().
>>>>>>
>>>>>> I'm sorry, I know you've explained this before, but I don't
>>>>>> understand yet, so let's try again.  I *think* you're saying
>>>>>> that:
>>>>>>
>>>>>>      - These devices integrated into LS7A all report 0 in their
>>>>>>      Interrupt Pin registers.  Per spec, this means they do not
>>>>>>      use INTx (PCIe r6.0, sec 7.5.1.1.13).
>>>>>>
>>>>>>      - However, these devices actually *do* use INTx.  Function
>>>>>>      0 uses INTA, function 1 uses INTB, ..., function 4 uses
>>>>>>      INTA, ...
>>>>>>
>>>>>>      - The quirk overrides the incorrect values read from the
>>>>>>      Interrupt Pin registers.
>>>>>
>>>>> Yes, right.
>>>
>>> Sorry, I didn't see the first item here carefully, so I have to
>>> correct it: all the integrated devices in 7A report 1 in PIN reg
>>> instead of 0.
>>
>>>>>> But I'm still confused about how loongson_map_irq() gets called.  The
>>>>>> only likely path I see is here:
>>>>>>
>>>>>>      pci_device_probe                            # pci_bus_type.probe
>>>>>>        pci_assign_irq
>>>>>>          pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
>>>>>>          if (pin)
>>>>>>     bridge->swizzle_irq(dev, &pin)
>>>>>>     irq = bridge->map_irq(dev, slot, pin)
>>>>>>
>>>>>> where bridge->map_irq points to loongson_map_irq().  But
>>>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
>>>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
>>>>>>
>>>
>>> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
>>> called.
>>
>> OK, that makes a lot more sense, thank you!
>>
>> But it does leave another question: the quirk applies to
>> DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
>> DEV_PCIE_PORT_2 (0x7a29).
>>
>> According to the .dtsi [1], all those root ports are at function 0,
>> and if they report INTA, the quirk will also compute INTA.  So why do
>> you need to apply the quirk for them?
>>
> 
> Oh, yes, I don't think they are required either. The fix is only 
> required for multi-func devices of 7A.
> 
> Huacai, we should remove PCIE ports from the patch.
> 
>> The same would apply to any Device ID that only appears at function 0,
>> which looks like it also includes DEV_LS7A_OHCI (0x7a24), and
>> DEV_LS7A_GPU (0x7a15).
>>

Same thing, they are also not required.

>> [1] 
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/mips/boot/dts/loongson/ls7a-pch.dtsi?id=v5.18#n231 
>>
>>
> 
> 

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Huacai Chen]

Hi, Jianmin,

On Sat, Jul 16, 2022 at 2:12 PM Jianmin Lv <lvjianmin@loongson.cn> wrote:
>
>
>
> On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
> > On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
> >> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
> >>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
> >>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> >>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> >>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
> >>>>>>
> >>>>>> In LS7A, multifunction device use same PCI PIN (because the
> >>>>>> PIN register report the same INTx value to each function)
> >>>>>> but we need different IRQ for different functions, so add a
> >>>>>> quirk to fix it for standard PCI PIN usage.
> >>>>>>
> >>>>>> This patch only affect ACPI based systems (and only needed
> >>>>>> by ACPI based systems, too). For DT based systems, the irq
> >>>>>> mappings is defined in .dts files and be handled by
> >>>>>> of_irq_parse_pci().
> >>>>>
> >>>>> I'm sorry, I know you've explained this before, but I don't
> >>>>> understand yet, so let's try again.  I *think* you're saying
> >>>>> that:
> >>>>>
> >>>>>      - These devices integrated into LS7A all report 0 in their
> >>>>>      Interrupt Pin registers.  Per spec, this means they do not
> >>>>>      use INTx (PCIe r6.0, sec 7.5.1.1.13).
> >>>>>
> >>>>>      - However, these devices actually *do* use INTx.  Function
> >>>>>      0 uses INTA, function 1 uses INTB, ..., function 4 uses
> >>>>>      INTA, ...
> >>>>>
> >>>>>      - The quirk overrides the incorrect values read from the
> >>>>>      Interrupt Pin registers.
> >>>>
> >>>> Yes, right.
> >>
> >> Sorry, I didn't see the first item here carefully, so I have to
> >> correct it: all the integrated devices in 7A report 1 in PIN reg
> >> instead of 0.
> >
> >>>>> But I'm still confused about how loongson_map_irq() gets called.  The
> >>>>> only likely path I see is here:
> >>>>>
> >>>>>      pci_device_probe                            # pci_bus_type.probe
> >>>>>        pci_assign_irq
> >>>>>          pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
> >>>>>          if (pin)
> >>>>>   bridge->swizzle_irq(dev, &pin)
> >>>>>   irq = bridge->map_irq(dev, slot, pin)
> >>>>>
> >>>>> where bridge->map_irq points to loongson_map_irq().  But
> >>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> >>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
> >>>>>
> >>
> >> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
> >> called.
> >
> > OK, that makes a lot more sense, thank you!
> >
> > But it does leave another question: the quirk applies to
> > DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
> > DEV_PCIE_PORT_2 (0x7a29).
> >
> > According to the .dtsi [1], all those root ports are at function 0,
> > and if they report INTA, the quirk will also compute INTA.  So why do
> > you need to apply the quirk for them?
> >
>
> Oh, yes, I don't think they are required either. The fix is only
> required for multi-func devices of 7A.
>
> Huacai, we should remove PCIE ports from the patch.
I agree to remove PCIE ports here. But since Bjorn has already merged
this patch, and the redundant devices listed here have no
side-effects, I suggest keeping it as is (but Bjorn is free to modify
if necessary).

Huacai
>
> > The same would apply to any Device ID that only appears at function 0,
> > which looks like it also includes DEV_LS7A_OHCI (0x7a24), and
> > DEV_LS7A_GPU (0x7a15).
> >
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/mips/boot/dts/loongson/ls7a-pch.dtsi?id=v5.18#n231
> >
>
>

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Bjorn Helgaas]

On Sat, Jul 16, 2022 at 04:37:41PM +0800, Huacai Chen wrote:
> On Sat, Jul 16, 2022 at 2:12 PM Jianmin Lv <lvjianmin@loongson.cn> wrote:
> > On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
> > > On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
> > >> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
> > >>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
> > >>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> > >>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> > >>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
> > >>>>>>
> > >>>>>> In LS7A, multifunction device use same PCI PIN (because the
> > >>>>>> PIN register report the same INTx value to each function)
> > >>>>>> but we need different IRQ for different functions, so add a
> > >>>>>> quirk to fix it for standard PCI PIN usage.
> > >>>>>>
> > >>>>>> This patch only affect ACPI based systems (and only needed
> > >>>>>> by ACPI based systems, too). For DT based systems, the irq
> > >>>>>> mappings is defined in .dts files and be handled by
> > >>>>>> of_irq_parse_pci().
> > >>>>>
> > >>>>> I'm sorry, I know you've explained this before, but I don't
> > >>>>> understand yet, so let's try again.  I *think* you're saying
> > >>>>> that:
> > >>>>>
> > >>>>>      - These devices integrated into LS7A all report 0 in their
> > >>>>>      Interrupt Pin registers.  Per spec, this means they do not
> > >>>>>      use INTx (PCIe r6.0, sec 7.5.1.1.13).
> > >>>>>
> > >>>>>      - However, these devices actually *do* use INTx.  Function
> > >>>>>      0 uses INTA, function 1 uses INTB, ..., function 4 uses
> > >>>>>      INTA, ...
> > >>>>>
> > >>>>>      - The quirk overrides the incorrect values read from the
> > >>>>>      Interrupt Pin registers.
> > >>>>
> > >>>> Yes, right.
> > >>
> > >> Sorry, I didn't see the first item here carefully, so I have to
> > >> correct it: all the integrated devices in 7A report 1 in PIN reg
> > >> instead of 0.
> > >
> > >>>>> But I'm still confused about how loongson_map_irq() gets called.  The
> > >>>>> only likely path I see is here:
> > >>>>>
> > >>>>>      pci_device_probe                            # pci_bus_type.probe
> > >>>>>        pci_assign_irq
> > >>>>>          pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
> > >>>>>          if (pin)
> > >>>>>   bridge->swizzle_irq(dev, &pin)
> > >>>>>   irq = bridge->map_irq(dev, slot, pin)
> > >>>>>
> > >>>>> where bridge->map_irq points to loongson_map_irq().  But
> > >>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> > >>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
> > >>
> > >> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
> > >> called.
> > >
> > > OK, that makes a lot more sense, thank you!
> > >
> > > But it does leave another question: the quirk applies to
> > > DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
> > > DEV_PCIE_PORT_2 (0x7a29).
> > >
> > > According to the .dtsi [1], all those root ports are at function 0,
> > > and if they report INTA, the quirk will also compute INTA.  So why do
> > > you need to apply the quirk for them?
> >
> > Oh, yes, I don't think they are required either. The fix is only
> > required for multi-func devices of 7A.
> >
> > Huacai, we should remove PCIE ports from the patch.
>
> I agree to remove PCIE ports here. But since Bjorn has already merged
> this patch, and the redundant devices listed here have no
> side-effects, I suggest keeping it as is (but Bjorn is free to modify
> if necessary).

I'd be happy to update the branch to remove the devices mentioned
(DEV_PCIE_PORT_x, DEV_LS7A_OHCI, DEV_LS7A_GPU).

But the original patch [2] *only* listed DEV_PCIE_PORT_x, so I'm
really confused about what's going on with them.  I assume [2] fixed
*something*, but now we're suggesting that we don't need it.

[2] https://lore.kernel.org/all/20210514080025.1828197-5-chenhuacai@loongson.cn/

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Jianmin Lv]

On 2022/7/17 上午7:32, Bjorn Helgaas wrote:
> On Sat, Jul 16, 2022 at 04:37:41PM +0800, Huacai Chen wrote:
>> On Sat, Jul 16, 2022 at 2:12 PM Jianmin Lv <lvjianmin@loongson.cn> wrote:
>>> On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
>>>> On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
>>>>> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
>>>>>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
>>>>>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
>>>>>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
>>>>>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
>>>>>>>>>
>>>>>>>>> In LS7A, multifunction device use same PCI PIN (because the
>>>>>>>>> PIN register report the same INTx value to each function)
>>>>>>>>> but we need different IRQ for different functions, so add a
>>>>>>>>> quirk to fix it for standard PCI PIN usage.
>>>>>>>>>
>>>>>>>>> This patch only affect ACPI based systems (and only needed
>>>>>>>>> by ACPI based systems, too). For DT based systems, the irq
>>>>>>>>> mappings is defined in .dts files and be handled by
>>>>>>>>> of_irq_parse_pci().
>>>>>>>>
>>>>>>>> I'm sorry, I know you've explained this before, but I don't
>>>>>>>> understand yet, so let's try again.  I *think* you're saying
>>>>>>>> that:
>>>>>>>>
>>>>>>>>       - These devices integrated into LS7A all report 0 in their
>>>>>>>>       Interrupt Pin registers.  Per spec, this means they do not
>>>>>>>>       use INTx (PCIe r6.0, sec 7.5.1.1.13).
>>>>>>>>
>>>>>>>>       - However, these devices actually *do* use INTx.  Function
>>>>>>>>       0 uses INTA, function 1 uses INTB, ..., function 4 uses
>>>>>>>>       INTA, ...
>>>>>>>>
>>>>>>>>       - The quirk overrides the incorrect values read from the
>>>>>>>>       Interrupt Pin registers.
>>>>>>>
>>>>>>> Yes, right.
>>>>>
>>>>> Sorry, I didn't see the first item here carefully, so I have to
>>>>> correct it: all the integrated devices in 7A report 1 in PIN reg
>>>>> instead of 0.
>>>>
>>>>>>>> But I'm still confused about how loongson_map_irq() gets called.  The
>>>>>>>> only likely path I see is here:
>>>>>>>>
>>>>>>>>       pci_device_probe                            # pci_bus_type.probe
>>>>>>>>         pci_assign_irq
>>>>>>>>           pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
>>>>>>>>           if (pin)
>>>>>>>>    bridge->swizzle_irq(dev, &pin)
>>>>>>>>    irq = bridge->map_irq(dev, slot, pin)
>>>>>>>>
>>>>>>>> where bridge->map_irq points to loongson_map_irq().  But
>>>>>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
>>>>>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
>>>>>
>>>>> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
>>>>> called.
>>>>
>>>> OK, that makes a lot more sense, thank you!
>>>>
>>>> But it does leave another question: the quirk applies to
>>>> DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
>>>> DEV_PCIE_PORT_2 (0x7a29).
>>>>
>>>> According to the .dtsi [1], all those root ports are at function 0,
>>>> and if they report INTA, the quirk will also compute INTA.  So why do
>>>> you need to apply the quirk for them?
>>>
>>> Oh, yes, I don't think they are required either. The fix is only
>>> required for multi-func devices of 7A.
>>>
>>> Huacai, we should remove PCIE ports from the patch.
>>
>> I agree to remove PCIE ports here. But since Bjorn has already merged
>> this patch, and the redundant devices listed here have no
>> side-effects, I suggest keeping it as is (but Bjorn is free to modify
>> if necessary).
> 
> I'd be happy to update the branch to remove the devices mentioned
> (DEV_PCIE_PORT_x, DEV_LS7A_OHCI, DEV_LS7A_GPU).
> 
> But the original patch [2] *only* listed DEV_PCIE_PORT_x, so I'm
> really confused about what's going on with them.  I assume [2] fixed
> *something*, but now we're suggesting that we don't need it.
> 
> [2] https://lore.kernel.org/all/20210514080025.1828197-5-chenhuacai@loongson.cn/
> 

Hi, Bjorn

My original patch(obviously just for simple, unwilling to list so much 
devices in the patch) is following;

+static void loongson_pci_pin_quirk(struct pci_dev *dev)
+{
+    u8 fun = dev->devfn & 7;
+
+    dev->pin = 1 + (fun & 3);
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_ANY_ID, 
loongson_pci_pin_quirk);
+

Mybe Huacai think only PCIE ports need the fix, so he adds the PCIE
ports in the patch when submitting it.

The things has been explained above, only multi-func devices are 
required to fix.

Thanks.

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Huacai Chen]

Hi, Bjorn,

On Sun, Jul 17, 2022 at 9:41 AM Jianmin Lv <lvjianmin@loongson.cn> wrote:
>
>
>
> On 2022/7/17 上午7:32, Bjorn Helgaas wrote:
> > On Sat, Jul 16, 2022 at 04:37:41PM +0800, Huacai Chen wrote:
> >> On Sat, Jul 16, 2022 at 2:12 PM Jianmin Lv <lvjianmin@loongson.cn> wrote:
> >>> On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
> >>>> On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
> >>>>> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
> >>>>>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
> >>>>>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> >>>>>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> >>>>>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
> >>>>>>>>>
> >>>>>>>>> In LS7A, multifunction device use same PCI PIN (because the
> >>>>>>>>> PIN register report the same INTx value to each function)
> >>>>>>>>> but we need different IRQ for different functions, so add a
> >>>>>>>>> quirk to fix it for standard PCI PIN usage.
> >>>>>>>>>
> >>>>>>>>> This patch only affect ACPI based systems (and only needed
> >>>>>>>>> by ACPI based systems, too). For DT based systems, the irq
> >>>>>>>>> mappings is defined in .dts files and be handled by
> >>>>>>>>> of_irq_parse_pci().
> >>>>>>>>
> >>>>>>>> I'm sorry, I know you've explained this before, but I don't
> >>>>>>>> understand yet, so let's try again.  I *think* you're saying
> >>>>>>>> that:
> >>>>>>>>
> >>>>>>>>       - These devices integrated into LS7A all report 0 in their
> >>>>>>>>       Interrupt Pin registers.  Per spec, this means they do not
> >>>>>>>>       use INTx (PCIe r6.0, sec 7.5.1.1.13).
> >>>>>>>>
> >>>>>>>>       - However, these devices actually *do* use INTx.  Function
> >>>>>>>>       0 uses INTA, function 1 uses INTB, ..., function 4 uses
> >>>>>>>>       INTA, ...
> >>>>>>>>
> >>>>>>>>       - The quirk overrides the incorrect values read from the
> >>>>>>>>       Interrupt Pin registers.
> >>>>>>>
> >>>>>>> Yes, right.
> >>>>>
> >>>>> Sorry, I didn't see the first item here carefully, so I have to
> >>>>> correct it: all the integrated devices in 7A report 1 in PIN reg
> >>>>> instead of 0.
> >>>>
> >>>>>>>> But I'm still confused about how loongson_map_irq() gets called.  The
> >>>>>>>> only likely path I see is here:
> >>>>>>>>
> >>>>>>>>       pci_device_probe                            # pci_bus_type.probe
> >>>>>>>>         pci_assign_irq
> >>>>>>>>           pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
> >>>>>>>>           if (pin)
> >>>>>>>>    bridge->swizzle_irq(dev, &pin)
> >>>>>>>>    irq = bridge->map_irq(dev, slot, pin)
> >>>>>>>>
> >>>>>>>> where bridge->map_irq points to loongson_map_irq().  But
> >>>>>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> >>>>>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
> >>>>>
> >>>>> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
> >>>>> called.
> >>>>
> >>>> OK, that makes a lot more sense, thank you!
> >>>>
> >>>> But it does leave another question: the quirk applies to
> >>>> DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
> >>>> DEV_PCIE_PORT_2 (0x7a29).
> >>>>
> >>>> According to the .dtsi [1], all those root ports are at function 0,
> >>>> and if they report INTA, the quirk will also compute INTA.  So why do
> >>>> you need to apply the quirk for them?
> >>>
> >>> Oh, yes, I don't think they are required either. The fix is only
> >>> required for multi-func devices of 7A.
> >>>
> >>> Huacai, we should remove PCIE ports from the patch.
> >>
> >> I agree to remove PCIE ports here. But since Bjorn has already merged
> >> this patch, and the redundant devices listed here have no
> >> side-effects, I suggest keeping it as is (but Bjorn is free to modify
> >> if necessary).
> >
> > I'd be happy to update the branch to remove the devices mentioned
> > (DEV_PCIE_PORT_x, DEV_LS7A_OHCI, DEV_LS7A_GPU).
> >
> > But the original patch [2] *only* listed DEV_PCIE_PORT_x, so I'm
> > really confused about what's going on with them.  I assume [2] fixed
> > *something*, but now we're suggesting that we don't need it.
> >
> > [2] https://lore.kernel.org/all/20210514080025.1828197-5-chenhuacai@loongson.cn/
> >
>
> Hi, Bjorn
>
> My original patch(obviously just for simple, unwilling to list so much
> devices in the patch) is following;
>
> +static void loongson_pci_pin_quirk(struct pci_dev *dev)
> +{
> +    u8 fun = dev->devfn & 7;
> +
> +    dev->pin = 1 + (fun & 3);
> +}
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_ANY_ID,
> loongson_pci_pin_quirk);
> +
>
> Mybe Huacai think only PCIE ports need the fix, so he adds the PCIE
> ports in the patch when submitting it.
Yes, what Jianmin said is correct, the first version of my patch is
wrong and the current version is correct.

Huacai
>
> The things has been explained above, only multi-func devices are
> required to fix.
>
> Thanks.
>

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Bjorn Helgaas]

On Sun, Jul 17, 2022 at 10:11:17PM +0800, Huacai Chen wrote:
> On Sun, Jul 17, 2022 at 9:41 AM Jianmin Lv <lvjianmin@loongson.cn> wrote:
> > On 2022/7/17 上午7:32, Bjorn Helgaas wrote:
> > > On Sat, Jul 16, 2022 at 04:37:41PM +0800, Huacai Chen wrote:
> > >> On Sat, Jul 16, 2022 at 2:12 PM Jianmin Lv <lvjianmin@loongson.cn> wrote:
> > >>> On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
> > >>>> On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
> > >>>>> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
> > >>>>>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
> > >>>>>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> > >>>>>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> > >>>>>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
> > >>>>>>>>>
> > >>>>>>>>> In LS7A, multifunction device use same PCI PIN (because the
> > >>>>>>>>> PIN register report the same INTx value to each function)
> > >>>>>>>>> but we need different IRQ for different functions, so add a
> > >>>>>>>>> quirk to fix it for standard PCI PIN usage.
> > >>>>>>>>>
> > >>>>>>>>> This patch only affect ACPI based systems (and only needed
> > >>>>>>>>> by ACPI based systems, too). For DT based systems, the irq
> > >>>>>>>>> mappings is defined in .dts files and be handled by
> > >>>>>>>>> of_irq_parse_pci().
> > >>>>>>>>
> > >>>>>>>> I'm sorry, I know you've explained this before, but I don't
> > >>>>>>>> understand yet, so let's try again.  I *think* you're saying
> > >>>>>>>> that:
> > >>>>>>>>
> > >>>>>>>>       - These devices integrated into LS7A all report 0 in their
> > >>>>>>>>       Interrupt Pin registers.  Per spec, this means they do not
> > >>>>>>>>       use INTx (PCIe r6.0, sec 7.5.1.1.13).
> > >>>>>>>>
> > >>>>>>>>       - However, these devices actually *do* use INTx.  Function
> > >>>>>>>>       0 uses INTA, function 1 uses INTB, ..., function 4 uses
> > >>>>>>>>       INTA, ...
> > >>>>>>>>
> > >>>>>>>>       - The quirk overrides the incorrect values read from the
> > >>>>>>>>       Interrupt Pin registers.
> > >>>>>>>
> > >>>>>>> Yes, right.
> > >>>>>
> > >>>>> Sorry, I didn't see the first item here carefully, so I have to
> > >>>>> correct it: all the integrated devices in 7A report 1 in PIN reg
> > >>>>> instead of 0.
> > >>>>
> > >>>>>>>> But I'm still confused about how loongson_map_irq() gets called.  The
> > >>>>>>>> only likely path I see is here:
> > >>>>>>>>
> > >>>>>>>>       pci_device_probe                            # pci_bus_type.probe
> > >>>>>>>>         pci_assign_irq
> > >>>>>>>>           pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
> > >>>>>>>>           if (pin)
> > >>>>>>>>    bridge->swizzle_irq(dev, &pin)
> > >>>>>>>>    irq = bridge->map_irq(dev, slot, pin)
> > >>>>>>>>
> > >>>>>>>> where bridge->map_irq points to loongson_map_irq().  But
> > >>>>>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> > >>>>>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
> > >>>>>
> > >>>>> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
> > >>>>> called.
> > >>>>
> > >>>> OK, that makes a lot more sense, thank you!
> > >>>>
> > >>>> But it does leave another question: the quirk applies to
> > >>>> DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
> > >>>> DEV_PCIE_PORT_2 (0x7a29).
> > >>>>
> > >>>> According to the .dtsi [1], all those root ports are at function 0,
> > >>>> and if they report INTA, the quirk will also compute INTA.  So why do
> > >>>> you need to apply the quirk for them?
> > >>>
> > >>> Oh, yes, I don't think they are required either. The fix is only
> > >>> required for multi-func devices of 7A.
> > >>>
> > >>> Huacai, we should remove PCIE ports from the patch.
> > >>
> > >> I agree to remove PCIE ports here. But since Bjorn has already merged
> > >> this patch, and the redundant devices listed here have no
> > >> side-effects, I suggest keeping it as is (but Bjorn is free to modify
> > >> if necessary).
> > >
> > > I'd be happy to update the branch to remove the devices mentioned
> > > (DEV_PCIE_PORT_x, DEV_LS7A_OHCI, DEV_LS7A_GPU).
> > >
> > > But the original patch [2] *only* listed DEV_PCIE_PORT_x, so I'm
> > > really confused about what's going on with them.  I assume [2] fixed
> > > *something*, but now we're suggesting that we don't need it.
> > >
> > > [2] https://lore.kernel.org/all/20210514080025.1828197-5-chenhuacai@loongson.cn/
> >
> > My original patch(obviously just for simple, unwilling to list so much
> > devices in the patch) is following;
> >
> > +static void loongson_pci_pin_quirk(struct pci_dev *dev)
> > +{
> > +    u8 fun = dev->devfn & 7;
> > +
> > +    dev->pin = 1 + (fun & 3);
> > +}
> > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_ANY_ID,
> > loongson_pci_pin_quirk);
> > +
> >
> > Mybe Huacai think only PCIE ports need the fix, so he adds the PCIE
> > ports in the patch when submitting it.
>
> Yes, what Jianmin said is correct, the first version of my patch is
> wrong and the current version is correct.

Thanks for clearing that up.  I dropped DEV_PCIE_PORT_x,
DEV_LS7A_OHCI, DEV_LS7A_GPU from the quirk.

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Huacai Chen]

Hi, Bjorn,

On Tue, Jul 19, 2022 at 1:00 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Sun, Jul 17, 2022 at 10:11:17PM +0800, Huacai Chen wrote:
> > On Sun, Jul 17, 2022 at 9:41 AM Jianmin Lv <lvjianmin@loongson.cn> wrote:
> > > On 2022/7/17 上午7:32, Bjorn Helgaas wrote:
> > > > On Sat, Jul 16, 2022 at 04:37:41PM +0800, Huacai Chen wrote:
> > > >> On Sat, Jul 16, 2022 at 2:12 PM Jianmin Lv <lvjianmin@loongson.cn> wrote:
> > > >>> On 2022/7/16 上午11:23, Bjorn Helgaas wrote:
> > > >>>> On Sat, Jul 16, 2022 at 10:27:00AM +0800, Jianmin Lv wrote:
> > > >>>>> On 2022/7/16 上午12:37, Bjorn Helgaas wrote:
> > > >>>>>> On Fri, Jul 15, 2022 at 04:05:12PM +0800, Jianmin Lv wrote:
> > > >>>>>>> On 2022/7/15 上午11:44, Bjorn Helgaas wrote:
> > > >>>>>>>> On Thu, Jul 14, 2022 at 08:42:16PM +0800, Huacai Chen wrote:
> > > >>>>>>>>> From: Jianmin Lv <lvjianmin@loongson.cn>
> > > >>>>>>>>>
> > > >>>>>>>>> In LS7A, multifunction device use same PCI PIN (because the
> > > >>>>>>>>> PIN register report the same INTx value to each function)
> > > >>>>>>>>> but we need different IRQ for different functions, so add a
> > > >>>>>>>>> quirk to fix it for standard PCI PIN usage.
> > > >>>>>>>>>
> > > >>>>>>>>> This patch only affect ACPI based systems (and only needed
> > > >>>>>>>>> by ACPI based systems, too). For DT based systems, the irq
> > > >>>>>>>>> mappings is defined in .dts files and be handled by
> > > >>>>>>>>> of_irq_parse_pci().
> > > >>>>>>>>
> > > >>>>>>>> I'm sorry, I know you've explained this before, but I don't
> > > >>>>>>>> understand yet, so let's try again.  I *think* you're saying
> > > >>>>>>>> that:
> > > >>>>>>>>
> > > >>>>>>>>       - These devices integrated into LS7A all report 0 in their
> > > >>>>>>>>       Interrupt Pin registers.  Per spec, this means they do not
> > > >>>>>>>>       use INTx (PCIe r6.0, sec 7.5.1.1.13).
> > > >>>>>>>>
> > > >>>>>>>>       - However, these devices actually *do* use INTx.  Function
> > > >>>>>>>>       0 uses INTA, function 1 uses INTB, ..., function 4 uses
> > > >>>>>>>>       INTA, ...
> > > >>>>>>>>
> > > >>>>>>>>       - The quirk overrides the incorrect values read from the
> > > >>>>>>>>       Interrupt Pin registers.
> > > >>>>>>>
> > > >>>>>>> Yes, right.
> > > >>>>>
> > > >>>>> Sorry, I didn't see the first item here carefully, so I have to
> > > >>>>> correct it: all the integrated devices in 7A report 1 in PIN reg
> > > >>>>> instead of 0.
> > > >>>>
> > > >>>>>>>> But I'm still confused about how loongson_map_irq() gets called.  The
> > > >>>>>>>> only likely path I see is here:
> > > >>>>>>>>
> > > >>>>>>>>       pci_device_probe                            # pci_bus_type.probe
> > > >>>>>>>>         pci_assign_irq
> > > >>>>>>>>           pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin)
> > > >>>>>>>>           if (pin)
> > > >>>>>>>>    bridge->swizzle_irq(dev, &pin)
> > > >>>>>>>>    irq = bridge->map_irq(dev, slot, pin)
> > > >>>>>>>>
> > > >>>>>>>> where bridge->map_irq points to loongson_map_irq().  But
> > > >>>>>>>> pci_assign_irq() should read 0 from PCI_INTERRUPT_PIN [1], so it
> > > >>>>>>>> wouldn't call bridge->map_irq().  Obviously I'm missing something.
> > > >>>>>
> > > >>>>> Same thing, PCI_INTERRUPT_PIN reports 1, so bridge->map_irq() will be
> > > >>>>> called.
> > > >>>>
> > > >>>> OK, that makes a lot more sense, thank you!
> > > >>>>
> > > >>>> But it does leave another question: the quirk applies to
> > > >>>> DEV_PCIE_PORT_0 (0x7a09), DEV_PCIE_PORT_1 (0x7a19), and
> > > >>>> DEV_PCIE_PORT_2 (0x7a29).
> > > >>>>
> > > >>>> According to the .dtsi [1], all those root ports are at function 0,
> > > >>>> and if they report INTA, the quirk will also compute INTA.  So why do
> > > >>>> you need to apply the quirk for them?
> > > >>>
> > > >>> Oh, yes, I don't think they are required either. The fix is only
> > > >>> required for multi-func devices of 7A.
> > > >>>
> > > >>> Huacai, we should remove PCIE ports from the patch.
> > > >>
> > > >> I agree to remove PCIE ports here. But since Bjorn has already merged
> > > >> this patch, and the redundant devices listed here have no
> > > >> side-effects, I suggest keeping it as is (but Bjorn is free to modify
> > > >> if necessary).
> > > >
> > > > I'd be happy to update the branch to remove the devices mentioned
> > > > (DEV_PCIE_PORT_x, DEV_LS7A_OHCI, DEV_LS7A_GPU).
> > > >
> > > > But the original patch [2] *only* listed DEV_PCIE_PORT_x, so I'm
> > > > really confused about what's going on with them.  I assume [2] fixed
> > > > *something*, but now we're suggesting that we don't need it.
> > > >
> > > > [2] https://lore.kernel.org/all/20210514080025.1828197-5-chenhuacai@loongson.cn/
> > >
> > > My original patch(obviously just for simple, unwilling to list so much
> > > devices in the patch) is following;
> > >
> > > +static void loongson_pci_pin_quirk(struct pci_dev *dev)
> > > +{
> > > +    u8 fun = dev->devfn & 7;
> > > +
> > > +    dev->pin = 1 + (fun & 3);
> > > +}
> > > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON, PCI_ANY_ID,
> > > loongson_pci_pin_quirk);
> > > +
> > >
> > > Mybe Huacai think only PCIE ports need the fix, so he adds the PCIE
> > > ports in the patch when submitting it.
> >
> > Yes, what Jianmin said is correct, the first version of my patch is
> > wrong and the current version is correct.
>
> Thanks for clearing that up.  I dropped DEV_PCIE_PORT_x,
> DEV_LS7A_OHCI, DEV_LS7A_GPU from the quirk.
Unfortunately, this patch only lists devices in LS7A1000, but some of
LS7A2000 (GNET and HDMI) also need to quirk, can they be squashed in
this patch? If not, we will add them in a new patch.

 #define DEV_LS7A_CONF  0x7a10
 #define DEV_LS7A_LPC   0x7a0c
 #define DEV_LS7A_GMAC  0x7a03
+#define DEV_LS7A_GNET  0x7a13
 #define DEV_LS7A_DC1   0x7a06
 #define DEV_LS7A_DC2   0x7a36
 #define DEV_LS7A_GPU   0x7a15
 #define DEV_LS7A_AHCI  0x7a08
 #define DEV_LS7A_EHCI  0x7a14
 #define DEV_LS7A_OHCI  0x7a24
+#define DEV_LS7A_HDMI  0x7a37

Huacai

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Bjorn Helgaas]

On Thu, Jul 21, 2022 at 12:47:18PM +0800, Huacai Chen wrote:

> Unfortunately, this patch only lists devices in LS7A1000, but some of
> LS7A2000 (GNET and HDMI) also need to quirk, can they be squashed in
> this patch? If not, we will add them in a new patch.
> 
>  #define DEV_LS7A_CONF  0x7a10
>  #define DEV_LS7A_LPC   0x7a0c
>  #define DEV_LS7A_GMAC  0x7a03
> +#define DEV_LS7A_GNET  0x7a13
>  #define DEV_LS7A_DC1   0x7a06
>  #define DEV_LS7A_DC2   0x7a36
>  #define DEV_LS7A_GPU   0x7a15
>  #define DEV_LS7A_AHCI  0x7a08
>  #define DEV_LS7A_EHCI  0x7a14
>  #define DEV_LS7A_OHCI  0x7a24
> +#define DEV_LS7A_HDMI  0x7a37

I squashed these in.  Let me know if I did anything wrong:

https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git/commit/?id=930c6074d7dd

Re: [PATCH V16 7/7] PCI: Add quirk for multifunction devices of LS7A

[Huacai Chen]

Hi, Bjorn,

On Fri, Jul 22, 2022 at 1:50 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Thu, Jul 21, 2022 at 12:47:18PM +0800, Huacai Chen wrote:
>
> > Unfortunately, this patch only lists devices in LS7A1000, but some of
> > LS7A2000 (GNET and HDMI) also need to quirk, can they be squashed in
> > this patch? If not, we will add them in a new patch.
> >
> >  #define DEV_LS7A_CONF  0x7a10
> >  #define DEV_LS7A_LPC   0x7a0c
> >  #define DEV_LS7A_GMAC  0x7a03
> > +#define DEV_LS7A_GNET  0x7a13
> >  #define DEV_LS7A_DC1   0x7a06
> >  #define DEV_LS7A_DC2   0x7a36
> >  #define DEV_LS7A_GPU   0x7a15
> >  #define DEV_LS7A_AHCI  0x7a08
> >  #define DEV_LS7A_EHCI  0x7a14
> >  #define DEV_LS7A_OHCI  0x7a24
> > +#define DEV_LS7A_HDMI  0x7a37
>
> I squashed these in.  Let me know if I did anything wrong:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git/commit/?id=930c6074d7dd
The logic is surely correct. But what is the principle of device list
order? If the order is value ascending, then LPC should be after AHCI;
if not, I prefer to group them with functions as below. :)
---
diff --git a/drivers/pci/controller/pci-loongson.c
b/drivers/pci/controller/pci-loongson.c
index 05997b51c86d..a7c3d5db3be8 100644
--- a/drivers/pci/controller/pci-loongson.c
+++ b/drivers/pci/controller/pci-loongson.c
@@ -22,6 +22,13 @@
 #define DEV_LS2K_APB   0x7a02
 #define DEV_LS7A_CONF  0x7a10
 #define DEV_LS7A_LPC   0x7a0c
+#define DEV_LS7A_DC1   0x7a06
+#define DEV_LS7A_DC2   0x7a36
+#define DEV_LS7A_HDMI  0x7a37
+#define DEV_LS7A_AHCI  0x7a08
+#define DEV_LS7A_EHCI  0x7a14
+#define DEV_LS7A_GMAC  0x7a03
+#define DEV_LS7A_GNET  0x7a13

 #define FLAG_CFG0      BIT(0)
 #define FLAG_CFG1      BIT(1)
@@ -103,6 +110,25 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_LOONGSON,
                        DEV_PCIE_PORT_2, loongson_bmaster_quirk);

+static void loongson_pci_pin_quirk(struct pci_dev *pdev)
+{
+       pdev->pin = 1 + (PCI_FUNC(pdev->devfn) & 3);
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+                       DEV_LS7A_DC1, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+                       DEV_LS7A_DC2, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+                       DEV_LS7A_HDMI, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+                       DEV_LS7A_AHCI, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+                       DEV_LS7A_EHCI, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+                       DEV_LS7A_GMAC, loongson_pci_pin_quirk);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_LOONGSON,
+                       DEV_LS7A_GNET, loongson_pci_pin_quirk);
+
 static struct loongson_pci *pci_bus_to_loongson_pci(struct pci_bus *bus)
 {
        struct pci_config_window *cfg;

---
Huacai

Re: [PATCH V16 0/7] PCI: Loongson pci improvements and quirks

[Bjorn Helgaas]

On Thu, Jul 14, 2022 at 08:42:09PM +0800, Huacai Chen wrote:
> This patchset improves Loongson PCI controller driver and resolves some
> problems: LS2K/LS7A's PCI config space supports 1/2/4-bytes access, so
> the first patch use pci_generic_config_read()/pci_generic_config_write()
> for them; the second patch add ACPI init support which will be used by
> LoongArch; the third patch improves the mrrs quirk for LS7A chipset; The
> fourth patch add a new quirk for LS7A chipset to avoid poweroff/reboot
> failure, and the fifth patch add a new quirk for LS7A chipset to fix the
> multifunction devices' irq pin mappings.
> ...

> Huacai Chen, Tiezhu Yang and Jianmin Lv(6):
>  PCI/ACPI: Guard ARM64-specific mcfg_quirks
>  PCI: loongson: Use generic 8/16/32-bit config ops on LS2K/LS7A.
>  PCI: loongson: Add ACPI init support.
>  PCI: loongson: Don't access non-existant devices.
>  PCI: Add quirk for multifunction devices of LS7A.

I applied the above to pci/ctrl/loongson to get them out of the way.

>  PCI: loongson: Improve the MRRS quirk for LS7A.
>  PCI: Add quirk for LS7A to avoid reboot failure.

These touch core code in some sort of ugly ways and I'm still thinking
about them.

Re: [PATCH V16 0/7] PCI: Loongson pci improvements and quirks

[Huacai Chen]

Hi, Bjorn,

On Sat, Jul 16, 2022 at 6:18 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Thu, Jul 14, 2022 at 08:42:09PM +0800, Huacai Chen wrote:
> > This patchset improves Loongson PCI controller driver and resolves some
> > problems: LS2K/LS7A's PCI config space supports 1/2/4-bytes access, so
> > the first patch use pci_generic_config_read()/pci_generic_config_write()
> > for them; the second patch add ACPI init support which will be used by
> > LoongArch; the third patch improves the mrrs quirk for LS7A chipset; The
> > fourth patch add a new quirk for LS7A chipset to avoid poweroff/reboot
> > failure, and the fifth patch add a new quirk for LS7A chipset to fix the
> > multifunction devices' irq pin mappings.
> > ...
>
> > Huacai Chen, Tiezhu Yang and Jianmin Lv(6):
> >  PCI/ACPI: Guard ARM64-specific mcfg_quirks
> >  PCI: loongson: Use generic 8/16/32-bit config ops on LS2K/LS7A.
> >  PCI: loongson: Add ACPI init support.
> >  PCI: loongson: Don't access non-existant devices.
> >  PCI: Add quirk for multifunction devices of LS7A.
>
> I applied the above to pci/ctrl/loongson to get them out of the way.
Thank you very much!

>
> >  PCI: loongson: Improve the MRRS quirk for LS7A.
> >  PCI: Add quirk for LS7A to avoid reboot failure.
>
> These touch core code in some sort of ugly ways and I'm still thinking
> about them.
Yes, it is ugly, but I hope it can be solved in some way, at least the
MRRS problem is not only Loongson-specific and can be shared by
others.

Huacai