linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Pali Rohár" <pali@kernel.org>
To: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
	Rob Herring <robh@kernel.org>,
	Bjorn Helgaas <bhelgaas@google.com>
Cc: "Russell King" <rmk+kernel@armlinux.org.uk>,
	"Marek Behún" <kabel@kernel.org>,
	"Remi Pommarel" <repk@triplefau.lt>, Xogium <contact@xogium.me>,
	"Tomasz Maciej Nowak" <tmn505@gmail.com>,
	"Marc Zyngier" <maz@kernel.org>,
	linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH 20/42] PCI: aardvark: Add support for more than 32 MSI interrupts
Date: Fri, 2 Jul 2021 23:35:24 +0200	[thread overview]
Message-ID: <20210702213524.mhpu24dxlh2fe7zm@pali> (raw)
In-Reply-To: <20210506153153.30454-21-pali@kernel.org>

FYI this one patch does not work. Please drop it.

On Thursday 06 May 2021 17:31:31 Pali Rohár wrote:
> Aardvark HW can handle MSI interrupt with any 16-bit number. Received MSI
> interrupt number is visible in PCIE_MSI_PAYLOAD_REG register after clearing
> corresponding bit in PCIE_MSI_STATUS_REG register.

After doing heavy load testing I figured out that PCIE_MSI_PAYLOAD_REG
register is either buggy or unusable or there is missing some other
configuration as it contains in most cases just content of the last
received memory write operation to MSI doorbell register. And also even
after clearing corresponding bit in PCIE_MSI_STATUS_REG register.

Therefore we should avoid usage of PCIE_MSI_PAYLOAD_REG register
completely. Which implies that it is needed to use only corresponding
bit from PCIE_MSI_STATUS_REG register and so only 32 MSI interrupts are
supported.

I will address removal of PCIE_MSI_STATUS_REG register in other followup
patch.

> The first 32 interrupt numbers are currently stored in linear map in MSI
> inner domain. Store the rest in dynamic radix tree for space efficiency.
> 
> Free interrupt numbers (available for MSI inner domain allocation) for the
> first 32 interrupts are currently stored in a bitmap. For the rest,
> introduce a linked list of allocated regions.
> 
> In the most common scenario there is only one PCIe card connected on boards
> with Armada 3720 SoC. Since in Multi-MSI mode the PCIe device can use at
> most 32 interrupts, all these interrupts are allocated in the linear map of
> MSI inner domain and marked as used in the bitmap.
> 
> For less common scenarios with PCIe devices with multiple functions or with
> a PCIe Bridge with packet switches with more connected PCIe devices more
> than 32 interrupts are requested. In this case, store each interrupt range
> from each interrupt request into the linked list as one node. In the worst
> case every PCIe function will occupy one node in this linked list.
> 
> This change allows to use all 32 Multi-MSI interrupts on every connected
> PCIe card on the Turris Mox router with Mox G module.
> 
> Signed-off-by: Pali Rohár <pali@kernel.org>
> Reviewed-by: Marek Behún <kabel@kernel.org>
> ---
>  drivers/pci/controller/pci-aardvark.c | 71 ++++++++++++++++++++++++---
>  1 file changed, 64 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/pci/controller/pci-aardvark.c b/drivers/pci/controller/pci-aardvark.c
> index 199015215779..d74e84b0e689 100644
> --- a/drivers/pci/controller/pci-aardvark.c
> +++ b/drivers/pci/controller/pci-aardvark.c
> @@ -178,11 +178,18 @@
>  #define RETRAIN_WAIT_MAX_RETRIES	10
>  #define RETRAIN_WAIT_USLEEP_US		2000
>  
> -#define MSI_IRQ_NUM			32
> +#define MSI_IRQ_LINEAR_COUNT		32
> +#define MSI_IRQ_TOTAL_COUNT		65536
>  
>  #define CFG_RD_UR_VAL			0xffffffff
>  #define CFG_RD_CRS_VAL			0xffff0001
>  
> +struct advk_msi_range {
> +	struct list_head list;
> +	u16 first;
> +	u16 count;
> +};
> +
>  struct advk_pcie {
>  	struct platform_device *pdev;
>  	void __iomem *base;
> @@ -193,7 +200,8 @@ struct advk_pcie {
>  	struct irq_chip msi_bottom_irq_chip;
>  	struct irq_chip msi_irq_chip;
>  	struct msi_domain_info msi_domain_info;
> -	DECLARE_BITMAP(msi_used, MSI_IRQ_NUM);
> +	DECLARE_BITMAP(msi_used_linear, MSI_IRQ_LINEAR_COUNT);
> +	struct list_head msi_used_radix;
>  	struct mutex msi_used_lock;
>  	int link_gen;
>  	struct pci_bridge_emul bridge;
> @@ -885,12 +893,44 @@ static int advk_msi_irq_domain_alloc(struct irq_domain *domain,
>  				     unsigned int nr_irqs, void *args)
>  {
>  	struct advk_pcie *pcie = domain->host_data;
> +	struct advk_msi_range *msi_range, *msi_range_prev, *msi_range_next;
> +	unsigned int first, count, last;
>  	int hwirq, i;
>  
>  	mutex_lock(&pcie->msi_used_lock);
> -	hwirq = bitmap_find_free_region(pcie->msi_used, MSI_IRQ_NUM,
> +
> +	/* First few used interrupt numbers are marked in bitmap (the most common) */
> +	hwirq = bitmap_find_free_region(pcie->msi_used_linear, MSI_IRQ_LINEAR_COUNT,
>  					order_base_2(nr_irqs));
> +
> +	/* And rest used interrupt numbers are stored in linked list as ranges */
> +	if (hwirq < 0) {
> +		count = 1 << order_base_2(nr_irqs);
> +		msi_range_prev = list_entry(&pcie->msi_used_radix, typeof(*msi_range), list);
> +		do {
> +			msi_range_next = list_next_entry(msi_range_prev, list);
> +			last = list_entry_is_head(msi_range_next, &pcie->msi_used_radix, list)
> +				? MSI_IRQ_TOTAL_COUNT : msi_range_next->first;
> +			first = list_entry_is_head(msi_range_prev, &pcie->msi_used_radix, list)
> +				? MSI_IRQ_LINEAR_COUNT : round_up(msi_range_prev->first +
> +								  msi_range_prev->count, count);
> +			if (first + count > last) {
> +				msi_range_prev = msi_range_next;
> +				continue;
> +			}
> +			msi_range = kzalloc(sizeof(*msi_range), GFP_KERNEL);
> +			if (msi_range) {
> +				hwirq = first;
> +				msi_range->first = first;
> +				msi_range->count = count;
> +				list_add(&msi_range->list, &msi_range_prev->list);
> +			}
> +			break;
> +		} while (!list_entry_is_head(msi_range_next, &pcie->msi_used_radix, list));
> +	}
> +
>  	mutex_unlock(&pcie->msi_used_lock);
> +
>  	if (hwirq < 0)
>  		return -ENOSPC;
>  
> @@ -908,9 +948,20 @@ static void advk_msi_irq_domain_free(struct irq_domain *domain,
>  {
>  	struct irq_data *d = irq_domain_get_irq_data(domain, virq);
>  	struct advk_pcie *pcie = domain->host_data;
> +	struct advk_msi_range *msi_range;
>  
>  	mutex_lock(&pcie->msi_used_lock);
> -	bitmap_release_region(pcie->msi_used, d->hwirq, order_base_2(nr_irqs));
> +	if (d->hwirq < MSI_IRQ_LINEAR_COUNT) {
> +		bitmap_release_region(pcie->msi_used_linear, d->hwirq, order_base_2(nr_irqs));
> +	} else {
> +		list_for_each_entry(msi_range, &pcie->msi_used_radix, list) {
> +			if (msi_range->first != d->hwirq)
> +				continue;
> +			list_del(&msi_range->list);
> +			kfree(msi_range);
> +			break;
> +		}
> +	}
>  	mutex_unlock(&pcie->msi_used_lock);
>  }
>  
> @@ -967,6 +1018,7 @@ static int advk_pcie_init_msi_irq_domain(struct advk_pcie *pcie)
>  	struct msi_domain_info *msi_di;
>  
>  	mutex_init(&pcie->msi_used_lock);
> +	INIT_LIST_HEAD(&pcie->msi_used_radix);
>  
>  	bottom_ic = &pcie->msi_bottom_irq_chip;
>  
> @@ -982,9 +1034,14 @@ static int advk_pcie_init_msi_irq_domain(struct advk_pcie *pcie)
>  		MSI_FLAG_MULTI_PCI_MSI;
>  	msi_di->chip = msi_ic;
>  
> +	/*
> +	 * Aardvark HW can handle MSI interrupt with any 16bit number.
> +	 * For optimization first few interrupts are allocated in linear map
> +	 * (which is common scenario) and rest are allocated in radix tree.
> +	 */
>  	pcie->msi_inner_domain =
> -		irq_domain_add_linear(NULL, MSI_IRQ_NUM,
> -				      &advk_msi_domain_ops, pcie);
> +		__irq_domain_add(NULL, MSI_IRQ_LINEAR_COUNT, MSI_IRQ_TOTAL_COUNT, 0,
> +				 &advk_msi_domain_ops, pcie);
>  	if (!pcie->msi_inner_domain)
>  		return -ENOMEM;
>  
> @@ -1052,7 +1109,7 @@ static void advk_pcie_handle_msi(struct advk_pcie *pcie)
>  	msi_val = advk_readl(pcie, PCIE_MSI_STATUS_REG);
>  	msi_status = msi_val & ((~msi_mask) & PCIE_MSI_ALL_MASK);
>  
> -	for (msi_idx = 0; msi_idx < MSI_IRQ_NUM; msi_idx++) {
> +	for (msi_idx = 0; msi_idx < BITS_PER_TYPE(msi_status); msi_idx++) {
>  		if (!(BIT(msi_idx) & msi_status))
>  			continue;
>  
> -- 
> 2.20.1
> 

  reply	other threads:[~2021-07-02 21:35 UTC|newest]

Thread overview: 73+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-06 15:31 [PATCH 00/42] PCI: aardvark: Various driver fixes Pali Rohár
2021-05-06 15:31 ` [PATCH 01/42] PCI: aardvark: Fix kernel panic during PIO transfer Pali Rohár
2021-05-19  8:06   ` Pali Rohár
2021-05-06 15:31 ` [PATCH 02/42] PCI: aardvark: Fix checking for PIO Non-posted Request Pali Rohár
2021-05-06 15:31 ` [PATCH 03/42] PCI: aardvark: Fix checking for PIO status Pali Rohár
2021-05-06 15:31 ` [PATCH 04/42] PCI: aardvark: Increase polling delay to 1.5s while waiting for PIO response Pali Rohár
2021-05-06 15:31 ` [PATCH 05/42] PCI: pci-bridge-emul: Add PCIe Root Capabilities Register Pali Rohár
2021-05-06 23:10   ` Bjorn Helgaas
2021-05-07 14:40     ` Pali Rohár
2021-05-07 16:41       ` Bjorn Helgaas
2021-05-06 15:31 ` [PATCH 06/42] PCI: aardvark: Fix reporting CRS Software Visibility on emulated bridge Pali Rohár
2021-05-07 13:03   ` Bjorn Helgaas
2021-05-07 15:25     ` Pali Rohár
2021-05-07 15:33     ` Pali Rohár
2021-05-06 15:31 ` [PATCH 07/42] PCI: aardvark: Fix link training Pali Rohár
2021-05-06 15:31 ` [PATCH 08/42] PCI: Add PCI_EXP_DEVCTL_PAYLOAD_* macros Pali Rohár
2021-05-06 15:31 ` [PATCH 09/42] PCI: aardvark: Fix PCIe Max Payload Size setting Pali Rohár
2021-05-06 15:31 ` [PATCH 10/42] PCI: aardvark: Implement workaround for the readback value of VEND_ID Pali Rohár
2021-05-06 15:31 ` [PATCH 11/42] PCI: aardvark: Do not touch status bits of masked interrupts in interrupt handler Pali Rohár
2021-05-06 15:31 ` [PATCH 12/42] PCI: aardvark: Check for virq mapping when processing INTx IRQ Pali Rohár
2021-05-07  9:15   ` Marc Zyngier
2021-06-04 16:24     ` Pali Rohár
2021-06-04 16:29       ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 13/42] PCI: aardvark: Remove irq_mask_ack callback for INTx interrupts Pali Rohár
2021-05-07  9:16   ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 14/42] PCI: aardvark: Don't mask irq when mapping Pali Rohár
2021-05-07  9:20   ` Marc Zyngier
2021-05-07  9:27     ` Pali Rohár
2021-05-06 15:31 ` [PATCH 15/42] PCI: aardvark: Change name of INTx irq_chip to advk-INT Pali Rohár
2021-05-07  9:08   ` Marc Zyngier
2021-05-24 14:36     ` Marek Behún
2021-05-24 15:14       ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 16/42] PCI: aardvark: Remove unneeded goto Pali Rohár
2021-05-06 15:31 ` [PATCH 17/42] PCI: aardvark: Fix support for MSI interrupts Pali Rohár
2021-05-07 10:16   ` Marc Zyngier
2021-05-07 14:44     ` Pali Rohár
2021-05-07 16:24       ` Marc Zyngier
2021-06-04 16:02         ` Pali Rohár
2021-06-04 16:22           ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 18/42] PCI: aardvark: Correctly clear and unmask all " Pali Rohár
2021-05-07 10:19   ` Marc Zyngier
2021-05-07 10:21     ` Pali Rohár
2021-05-06 15:31 ` [PATCH 19/42] PCI: aardvark: Fix setting MSI address Pali Rohár
2021-05-07 10:25   ` Marc Zyngier
2021-05-06 15:31 ` [PATCH 20/42] PCI: aardvark: Add support for more than 32 MSI interrupts Pali Rohár
2021-07-02 21:35   ` Pali Rohár [this message]
2021-05-06 15:31 ` [PATCH 21/42] PCI: aardvark: Add support for masking " Pali Rohár
2021-05-06 15:31 ` [PATCH 22/42] PCI: aardvark: Enable MSI-X support Pali Rohár
2021-05-06 15:31 ` [PATCH 23/42] PCI: aardvark: Fix support for ERR interrupt on emulated bridge Pali Rohár
2021-05-06 15:31 ` [PATCH 24/42] PCI: aardvark: Fix support for PME " Pali Rohár
2021-05-06 15:31 ` [PATCH 25/42] PCI: aardvark: Fix support for PME requester " Pali Rohár
2021-05-06 15:31 ` [PATCH 26/42] PCI: aardvark: Fix support for bus mastering and PCI_COMMAND " Pali Rohár
2021-05-06 15:31 ` [PATCH 27/42] PCI: aardvark: Disable bus mastering and mask all interrupts when unbinding driver Pali Rohár
2021-05-06 15:31 ` [PATCH 28/42] PCI: aardvark: Free config space for emulated root bridge when unbinding driver to fix memory leak Pali Rohár
2021-05-06 15:31 ` [PATCH 29/42] PCI: aardvark: Reset PCIe card and disable PHY when unbinding driver Pali Rohár
2021-05-06 15:31 ` [PATCH 30/42] PCI: aardvark: Rewrite irq code to chained irq handler Pali Rohár
2021-05-06 15:31 ` [PATCH 31/42] PCI: aardvark: Use separate INTA interrupt for emulated root bridge Pali Rohár
2021-05-06 15:31 ` [PATCH 32/42] PCI: pci-bridge-emul: Add description for class_revision field Pali Rohár
2021-05-06 15:31 ` [PATCH 33/42] PCI: pci-bridge-emul: Add definitions for missing capabilities registers Pali Rohár
2021-05-06 15:31 ` [PATCH 34/42] PCI: aardvark: Add support for DEVCAP2, DEVCTL2, LNKCAP2 and LNKCTL2 registers on emulated bridge Pali Rohár
2021-05-06 15:31 ` [PATCH 35/42] PCI: aardvark: Add support for PCI_BRIDGE_CTL_BUS_RESET " Pali Rohár
2021-05-06 15:31 ` [PATCH 36/42] PCI: aardvark: Replace custom PCIE_CORE_ERR_CAPCTL_* macros by linux/pci_regs.h macros Pali Rohár
2021-05-06 15:31 ` [PATCH 37/42] PCI: aardvark: Replace custom PCIE_CORE_INT_* macros by linux PCI_INTERRUPT_* values Pali Rohár
2021-05-06 15:31 ` [PATCH 38/42] PCI: aardvark: Cleanup some register macros Pali Rohár
2021-05-06 15:31 ` [PATCH 39/42] PCI: aardvark: Add comments for OB_WIN_ENABLE and ADDR_WIN_DISABLE Pali Rohár
2021-05-06 15:31 ` [PATCH 40/42] PCI: pci-bridge-emul: re-arrange register tests Pali Rohár
2021-05-06 15:31 ` [PATCH 41/42] PCI: pci-bridge-emul: add support for PCIe extended capabilities Pali Rohár
2021-05-06 15:31 ` [PATCH 42/42] PCI: aardvark: Add support for Advanced Error Reporting registers on emulated bridge Pali Rohár
2021-06-03 15:16 ` [PATCH 00/42] PCI: aardvark: Various driver fixes Lorenzo Pieralisi
2021-06-03 17:02   ` Pali Rohár
2021-06-03 18:02   ` Simon Glass
2021-06-03 18:18     ` Pali Rohár
2021-06-04 14:05     ` Lorenzo Pieralisi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210702213524.mhpu24dxlh2fe7zm@pali \
    --to=pali@kernel.org \
    --cc=bhelgaas@google.com \
    --cc=contact@xogium.me \
    --cc=kabel@kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=maz@kernel.org \
    --cc=repk@triplefau.lt \
    --cc=rmk+kernel@armlinux.org.uk \
    --cc=robh@kernel.org \
    --cc=thomas.petazzoni@bootlin.com \
    --cc=tmn505@gmail.com \
    --subject='Re: [PATCH 20/42] PCI: aardvark: Add support for more than 32 MSI interrupts' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).