linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Marc Zyngier <maz@kernel.org>
To: "Pali Rohár" <pali@kernel.org>
Cc: "Lorenzo Pieralisi" <lorenzo.pieralisi@arm.com>,
	"Thomas Petazzoni" <thomas.petazzoni@bootlin.com>,
	"Rob Herring" <robh@kernel.org>,
	"Bjorn Helgaas" <bhelgaas@google.com>,
	"Russell King" <rmk+kernel@armlinux.org.uk>,
	"Marek Behún" <kabel@kernel.org>,
	"Remi Pommarel" <repk@triplefau.lt>, Xogium <contact@xogium.me>,
	"Tomasz Maciej Nowak" <tmn505@gmail.com>,
	linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH 17/42] PCI: aardvark: Fix support for MSI interrupts
Date: Fri, 04 Jun 2021 17:22:30 +0100	[thread overview]
Message-ID: <87pmx1zjjt.wl-maz@kernel.org> (raw)
In-Reply-To: <20210604160246.vrix6fngictqpmbg@pali>

On Fri, 04 Jun 2021 17:02:46 +0100,
Pali Rohár <pali@kernel.org> wrote:
> 
> On Friday 07 May 2021 17:24:25 Marc Zyngier wrote:
> > On Fri, 07 May 2021 15:44:20 +0100,
> > Pali Rohár <pali@kernel.org> wrote:
> > > 
> > > On Friday 07 May 2021 11:16:58 Marc Zyngier wrote:
> > > > On Thu, 06 May 2021 16:31:28 +0100,
> > > > Pali Rohár <pali@kernel.org> wrote:
> > > > > 
> > > > > MSI domain callback .alloc (implemented by advk_msi_irq_domain_alloc()
> > > > > function) should return zero on success. Returning non-zero value indicates
> > > > > failure. Fix return value of this function as in many cases it now returns
> > > > > failure while allocating IRQs.
> > > > > 
> > > > > Aardvark hardware supports Multi-MSI and MSI_FLAG_MULTI_PCI_MSI is already
> > > > > set. But when allocating MSI interrupt numbers for Multi-MSI, they need to
> > > > > be properly aligned, otherwise endpoint devices send MSI interrupt with
> > > > > incorrect numbers. Fix this issue by using function bitmap_find_free_region()
> > > > > instead of bitmap_find_next_zero_area().
> > > > > 
> > > > > To ensure that aligned MSI interrupt numbers are used by endpoint devices,
> > > > > we cannot use Linux virtual irq numbers (as they are random and not
> > > > > properly aligned). So use hwirq numbers allocated by the function
> > > > > bitmap_find_free_region(), which are aligned. This needs an update in
> > > > > advk_msi_irq_compose_msi_msg() and advk_pcie_handle_msi() functions to do
> > > > > proper mapping between Linux virtual irq numbers and hwirq MSI inner domain
> > > > > numbers.
> > > > > 
> > > > > Also the whole 16-bit MSI number is stored in the PCIE_MSI_PAYLOAD_REG
> > > > > register, not only lower 8 bits. Fix reading content of this register.
> > > > > 
> > > > > This change fixes receiving MSI interrupts on Armada 3720 boards and allows
> > > > > using NVMe disks which use Multi-MSI feature with 3 interrupts.
> > > > > 
> > > > > Without this change, NVMe disks just freeze booting Linux on Armada 3720
> > > > > boards as linux nvme-core.c driver is waiting 60s for an interrupt.
> > > > > 
> > > > > Signed-off-by: Pali Rohár <pali@kernel.org>
> > > > > Reviewed-by: Marek Behún <kabel@kernel.org>
> > > > > Cc: stable@vger.kernel.org # f21a8b1b6837 ("PCI: aardvark: Move to MSI handling using generic MSI support")
> > > > > ---
> > > > >  drivers/pci/controller/pci-aardvark.c | 32 ++++++++++++++++-----------
> > > > >  1 file changed, 19 insertions(+), 13 deletions(-)
> > > > > 
> > > > > diff --git a/drivers/pci/controller/pci-aardvark.c b/drivers/pci/controller/pci-aardvark.c
> > > > > index 366d7480bc1b..498810c00b6d 100644
> > > > > --- a/drivers/pci/controller/pci-aardvark.c
> > > > > +++ b/drivers/pci/controller/pci-aardvark.c
> > > > > @@ -118,6 +118,7 @@
> > > > >  #define PCIE_MSI_STATUS_REG			(CONTROL_BASE_ADDR + 0x58)
> > > > >  #define PCIE_MSI_MASK_REG			(CONTROL_BASE_ADDR + 0x5C)
> > > > >  #define PCIE_MSI_PAYLOAD_REG			(CONTROL_BASE_ADDR + 0x9C)
> > > > > +#define     PCIE_MSI_DATA_MASK			GENMASK(15, 0)
> > > > 
> > > > See my comment below about this addition.
> > > > 
> > > > >  /* LMI registers base address and register offsets */
> > > > >  #define LMI_BASE_ADDR				0x6000
> > > > > @@ -861,7 +862,7 @@ static void advk_msi_irq_compose_msi_msg(struct irq_data *data,
> > > > >  
> > > > >  	msg->address_lo = lower_32_bits(msi_msg);
> > > > >  	msg->address_hi = upper_32_bits(msi_msg);
> > > > > -	msg->data = data->irq;
> > > > > +	msg->data = data->hwirq;
> > > > >  }
> > > > >  
> > > > >  static int advk_msi_set_affinity(struct irq_data *irq_data,
> > > > > @@ -878,15 +879,11 @@ static int advk_msi_irq_domain_alloc(struct irq_domain *domain,
> > > > >  	int hwirq, i;
> > > > >  
> > > > >  	mutex_lock(&pcie->msi_used_lock);
> > > > > -	hwirq = bitmap_find_next_zero_area(pcie->msi_used, MSI_IRQ_NUM,
> > > > > -					   0, nr_irqs, 0);
> > > > > -	if (hwirq >= MSI_IRQ_NUM) {
> > > > > -		mutex_unlock(&pcie->msi_used_lock);
> > > > > -		return -ENOSPC;
> > > > > -	}
> > > > > -
> > > > > -	bitmap_set(pcie->msi_used, hwirq, nr_irqs);
> > > > > +	hwirq = bitmap_find_free_region(pcie->msi_used, MSI_IRQ_NUM,
> > > > > +					order_base_2(nr_irqs));
> > > > >  	mutex_unlock(&pcie->msi_used_lock);
> > > > > +	if (hwirq < 0)
> > > > > +		return -ENOSPC;
> > > > >  
> > > > >  	for (i = 0; i < nr_irqs; i++)
> > > > >  		irq_domain_set_info(domain, virq + i, hwirq + i,
> > > > > @@ -894,7 +891,7 @@ static int advk_msi_irq_domain_alloc(struct irq_domain *domain,
> > > > >  				    domain->host_data, handle_simple_irq,
> > > > >  				    NULL, NULL);
> > > > >  
> > > > > -	return hwirq;
> > > > > +	return 0;
> > > > >  }
> > > > >  
> > > > >  static void advk_msi_irq_domain_free(struct irq_domain *domain,
> > > > > @@ -904,7 +901,7 @@ static void advk_msi_irq_domain_free(struct irq_domain *domain,
> > > > >  	struct advk_pcie *pcie = domain->host_data;
> > > > >  
> > > > >  	mutex_lock(&pcie->msi_used_lock);
> > > > > -	bitmap_clear(pcie->msi_used, d->hwirq, nr_irqs);
> > > > > +	bitmap_release_region(pcie->msi_used, d->hwirq, order_base_2(nr_irqs));
> > > > >  	mutex_unlock(&pcie->msi_used_lock);
> > > > >  }
> > > > >  
> > > > > @@ -1048,6 +1045,7 @@ static void advk_pcie_handle_msi(struct advk_pcie *pcie)
> > > > >  {
> > > > >  	u32 msi_val, msi_mask, msi_status, msi_idx;
> > > > >  	u16 msi_data;
> > > > > +	int virq;
> > > > >  
> > > > >  	msi_mask = advk_readl(pcie, PCIE_MSI_MASK_REG);
> > > > >  	msi_val = advk_readl(pcie, PCIE_MSI_STATUS_REG);
> > > > > @@ -1057,9 +1055,17 @@ static void advk_pcie_handle_msi(struct advk_pcie *pcie)
> > > > >  		if (!(BIT(msi_idx) & msi_status))
> > > > >  			continue;
> > > > >  
> > > > > +		/*
> > > > > +		 * msi_idx contains bits [4:0] of the msi_data and msi_data
> > > > > +		 * contains 16bit MSI interrupt number from MSI inner domain
> > > > > +		 */
> > > > >  		advk_writel(pcie, BIT(msi_idx), PCIE_MSI_STATUS_REG);
> > > > > -		msi_data = advk_readl(pcie, PCIE_MSI_PAYLOAD_REG) & 0xFF;
> > > > > -		generic_handle_irq(msi_data);
> > > > > +		msi_data = advk_readl(pcie, PCIE_MSI_PAYLOAD_REG) & PCIE_MSI_DATA_MASK;
> > > > 
> > > > Can this be moved to a separate patch? It seems like this patch should
> > > > only focus on correctly dealing with the irq/hwirq issues.
> > > 
> > > Well, hwirq is read from PCIE_MSI_PAYLOAD_REG register and it is 16-bit.
> > > That is why I included this change in this patch, to fix also reading
> > > IRQ number, not only setting IRQ number.
> > 
> > But this irq number still is a 5 bit quantity at this stage, and the
> 
> Yes, it should be 5 bit number. And in case wrongly programmed PCIe card
> sends interrupt with "incorrect number" then A3720 PCIe controller

How? This driver is in control of what gets programmed.

> "should not try" to map this 16-bit unknown MSI interrupt number to
> something in 5-bit domain (by setting upper bits to zero) and trying to
> deliver this invalid interrupt via some existing virq.
> 
> Interrupt number of received MSI is stored in low 16 bits in
> PCIE_MSI_PAYLOAD_REG register and you should use / validate whole
> number, not just few bits from it.

Meh. i still maintain that this isn't a logical split for this patch,
but at this stage I don't care (my 3720 is in the recycling pile).

	M.

-- 
Without deviation from the norm, progress is not possible.

  reply	other threads:[~2021-06-04 16:22 UTC|newest]

Thread overview: 73+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-06 15:31 [PATCH 00/42] PCI: aardvark: Various driver fixes Pali Rohár
2021-05-06 15:31 ` [PATCH 01/42] PCI: aardvark: Fix kernel panic during PIO transfer Pali Rohár
2021-05-19  8:06   ` Pali Rohár
2021-05-06 15:31 ` [PATCH 02/42] PCI: aardvark: Fix checking for PIO Non-posted Request Pali Rohár
2021-05-06 15:31 ` [PATCH 03/42] PCI: aardvark: Fix checking for PIO status Pali Rohár
2021-05-06 15:31 ` [PATCH 04/42] PCI: aardvark: Increase polling delay to 1.5s while waiting for PIO response Pali Rohár
2021-05-06 15:31 ` [PATCH 05/42] PCI: pci-bridge-emul: Add PCIe Root Capabilities Register Pali Rohár
2021-05-06 23:10   ` Bjorn Helgaas
2021-05-07 14:40     ` Pali Rohár
2021-05-07 16:41       ` Bjorn Helgaas
2021-05-06 15:31 ` [PATCH 06/42] PCI: aardvark: Fix reporting CRS Software Visibility on emulated bridge Pali Rohár
2021-05-07 13:03   ` Bjorn Helgaas
2021-05-07 15:25     ` Pali Rohár
2021-05-07 15:33     ` Pali Rohár
2021-05-06 15:31 ` [PATCH 07/42] PCI: aardvark: Fix link training Pali Rohár
2021-05-06 15:31 ` [PATCH 08/42] PCI: Add PCI_EXP_DEVCTL_PAYLOAD_* macros Pali Rohár
2021-05-06 15:31 ` [PATCH 09/42] PCI: aardvark: Fix PCIe Max Payload Size setting Pali Rohár
2021-05-06 15:31 ` [PATCH 10/42] PCI: aardvark: Implement workaround for the readback value of VEND_ID Pali Rohár
2021-05-06 15:31 ` [PATCH 11/42] PCI: aardvark: Do not touch status bits of masked interrupts in interrupt handler Pali Rohár
2021-05-06 15:31 ` [PATCH 12/42] PCI: aardvark: Check for virq mapping when processing INTx IRQ Pali Rohár
2021-05-07  9:15   ` Marc Zyngier
2021-06-04 16:24     ` Pali Rohár
2021-06-04 16:29       ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 13/42] PCI: aardvark: Remove irq_mask_ack callback for INTx interrupts Pali Rohár
2021-05-07  9:16   ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 14/42] PCI: aardvark: Don't mask irq when mapping Pali Rohár
2021-05-07  9:20   ` Marc Zyngier
2021-05-07  9:27     ` Pali Rohár
2021-05-06 15:31 ` [PATCH 15/42] PCI: aardvark: Change name of INTx irq_chip to advk-INT Pali Rohár
2021-05-07  9:08   ` Marc Zyngier
2021-05-24 14:36     ` Marek Behún
2021-05-24 15:14       ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 16/42] PCI: aardvark: Remove unneeded goto Pali Rohár
2021-05-06 15:31 ` [PATCH 17/42] PCI: aardvark: Fix support for MSI interrupts Pali Rohár
2021-05-07 10:16   ` Marc Zyngier
2021-05-07 14:44     ` Pali Rohár
2021-05-07 16:24       ` Marc Zyngier
2021-06-04 16:02         ` Pali Rohár
2021-06-04 16:22           ` Marc Zyngier [this message]
2021-05-06 15:31 ` [PATCH 18/42] PCI: aardvark: Correctly clear and unmask all " Pali Rohár
2021-05-07 10:19   ` Marc Zyngier
2021-05-07 10:21     ` Pali Rohár
2021-05-06 15:31 ` [PATCH 19/42] PCI: aardvark: Fix setting MSI address Pali Rohár
2021-05-07 10:25   ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 20/42] PCI: aardvark: Add support for more than 32 MSI interrupts Pali Rohár
2021-07-02 21:35   ` Pali Rohár
2021-05-06 15:31 ` [PATCH 21/42] PCI: aardvark: Add support for masking " Pali Rohár
2021-05-06 15:31 ` [PATCH 22/42] PCI: aardvark: Enable MSI-X support Pali Rohár
2021-05-06 15:31 ` [PATCH 23/42] PCI: aardvark: Fix support for ERR interrupt on emulated bridge Pali Rohár
2021-05-06 15:31 ` [PATCH 24/42] PCI: aardvark: Fix support for PME " Pali Rohár
2021-05-06 15:31 ` [PATCH 25/42] PCI: aardvark: Fix support for PME requester " Pali Rohár
2021-05-06 15:31 ` [PATCH 26/42] PCI: aardvark: Fix support for bus mastering and PCI_COMMAND " Pali Rohár
2021-05-06 15:31 ` [PATCH 27/42] PCI: aardvark: Disable bus mastering and mask all interrupts when unbinding driver Pali Rohár
2021-05-06 15:31 ` [PATCH 28/42] PCI: aardvark: Free config space for emulated root bridge when unbinding driver to fix memory leak Pali Rohár
2021-05-06 15:31 ` [PATCH 29/42] PCI: aardvark: Reset PCIe card and disable PHY when unbinding driver Pali Rohár
2021-05-06 15:31 ` [PATCH 30/42] PCI: aardvark: Rewrite irq code to chained irq handler Pali Rohár
2021-05-06 15:31 ` [PATCH 31/42] PCI: aardvark: Use separate INTA interrupt for emulated root bridge Pali Rohár
2021-05-06 15:31 ` [PATCH 32/42] PCI: pci-bridge-emul: Add description for class_revision field Pali Rohár
2021-05-06 15:31 ` [PATCH 33/42] PCI: pci-bridge-emul: Add definitions for missing capabilities registers Pali Rohár
2021-05-06 15:31 ` [PATCH 34/42] PCI: aardvark: Add support for DEVCAP2, DEVCTL2, LNKCAP2 and LNKCTL2 registers on emulated bridge Pali Rohár
2021-05-06 15:31 ` [PATCH 35/42] PCI: aardvark: Add support for PCI_BRIDGE_CTL_BUS_RESET " Pali Rohár
2021-05-06 15:31 ` [PATCH 36/42] PCI: aardvark: Replace custom PCIE_CORE_ERR_CAPCTL_* macros by linux/pci_regs.h macros Pali Rohár
2021-05-06 15:31 ` [PATCH 37/42] PCI: aardvark: Replace custom PCIE_CORE_INT_* macros by linux PCI_INTERRUPT_* values Pali Rohár
2021-05-06 15:31 ` [PATCH 38/42] PCI: aardvark: Cleanup some register macros Pali Rohár
2021-05-06 15:31 ` [PATCH 39/42] PCI: aardvark: Add comments for OB_WIN_ENABLE and ADDR_WIN_DISABLE Pali Rohár
2021-05-06 15:31 ` [PATCH 40/42] PCI: pci-bridge-emul: re-arrange register tests Pali Rohár
2021-05-06 15:31 ` [PATCH 41/42] PCI: pci-bridge-emul: add support for PCIe extended capabilities Pali Rohár
2021-05-06 15:31 ` [PATCH 42/42] PCI: aardvark: Add support for Advanced Error Reporting registers on emulated bridge Pali Rohár
2021-06-03 15:16 ` [PATCH 00/42] PCI: aardvark: Various driver fixes Lorenzo Pieralisi
2021-06-03 17:02   ` Pali Rohár
2021-06-03 18:02   ` Simon Glass
2021-06-03 18:18     ` Pali Rohár
2021-06-04 14:05     ` Lorenzo Pieralisi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87pmx1zjjt.wl-maz@kernel.org \
    --to=maz@kernel.org \
    --cc=bhelgaas@google.com \
    --cc=contact@xogium.me \
    --cc=kabel@kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=pali@kernel.org \
    --cc=repk@triplefau.lt \
    --cc=rmk+kernel@armlinux.org.uk \
    --cc=robh@kernel.org \
    --cc=thomas.petazzoni@bootlin.com \
    --cc=tmn505@gmail.com \
    --subject='Re: [PATCH 17/42] PCI: aardvark: Fix support for MSI interrupts' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).