[RFC PATCH] PCI/MSI: Only mask all MSI-X entries when MSI-X is used

[Stefan Roese <sr@denx.de>]

This patch moves the masking of the MSI-X entries to a later stage in
msix_capability_init(), which is not reached on platforms not
supporting MSI-X. Without this, MSI interrupts from a NVMe drive are not
received at all on this ZynqMP based platform, only supporting legacy
and MSI interrupts.

Background:
This patch fixes a problem on our ZynqMP based system working with newer
NVMe drives which support MSI & MSI-X. Running v5.4 all is fine and
these drives correctly configure an MSI interrupt and this IRQ is
received just fine in the ZynqMP rootport. But when updating to v5.10
or later (I also tested with v5.15 and v5.16-rc4) the MSI interrupt
gets assigned but no interrupts are received by the NVMe driver at all.

Note: The ZynqMP PCIe rootport driver only supports legacy and MSI
interrupts, not MSI-X (yet).

I've debugged the MSI integration of the ZynqMP PCIe rootport driver
(pcie-xilinx-nwl.c) and found no issues there. Also the MSI framework
in the Kernel did not reveal any problems - at least for me. Looking
a bit deeper into the lspci output, I found an interesting difference
between v5.4 and v5.10 (or later).

v5.4:
04:00.0 Non-Volatile memory controller: Marvell Technology Group Ltd. Device 1321 (rev 02) (prog-if 02 [NVM Express])
        ...
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fd480000  Data: 0004
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
	...
	Capabilities: [b0] MSI-X: Enable- Count=67 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	...

v5.10:
04:00.0 Non-Volatile memory controller: Marvell Technology Group Ltd. Device 1321 (rev 02) (prog-if 02 [NVM Express])
        ...
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fd480000  Data: 0004
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
        ...
        Capabilities: [b0] MSI-X: Enable- Count=67 Masked+
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        ...

So the only difference here being the "Masked+" compared to the
"Masked-" in the working v5.4 Kernel. Testing in this area has shown,
that the root cause for the masked bit being set was the call to
msix_mask_all() in msix_capability_init(). Without this, all works just
fine and the MSI interrupts are received again by the NVMe driver.

BTW: I've also tested this problem with the latest version of Thomas's
PCI/MSI Spring cleaning on top of v5.16-rc4. No change - the masked bit
is still set and the MSI interrupt are note received by the NVMe driver.

I'm open to other ideas to fix this issue. So please review and comment.

Fixes: aa8092c1d1f1 ("PCI/MSI: Mask all unused MSI-X entries")
Signed-off-by: Stefan Roese <sr@denx.de>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Michal Simek <michal.simek@xilinx.com>
Cc: Marek Vasut <marex@denx.de>
---
 drivers/pci/msi.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index a7a1c7411348..25b659dd5e2b 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -825,9 +825,6 @@ static int msix_capability_init(struct pci_dev *dev, struct msix_entry *entries,
 		goto out_disable;
 	}
 
-	/* Ensure that all table entries are masked. */
-	msix_mask_all(base, tsize);
-
 	ret = msix_setup_entries(dev, base, entries, nvec, affd);
 	if (ret)
 		goto out_disable;
@@ -836,6 +833,9 @@ static int msix_capability_init(struct pci_dev *dev, struct msix_entry *entries,
 	if (ret)
 		goto out_avail;
 
+	/* Ensure that all table entries are masked. */
+	msix_mask_all(base, tsize);
+
 	/* Check if all MSI entries honor device restrictions */
 	ret = msi_verify_entries(dev);
 	if (ret)
-- 
2.34.1