Re-activate task to add MSI-X to pcie-designware

Re-activate task to add MSI-X to pcie-designware

[Joao Pinto]

Hello to all,

I am sending this e-mail to request your opinion about the tasks needed to add
support for MSI-X for pcie-designware based solutions.

A few months ago I submited a patch-set that added support for MSI-X:

[v2,9/9] pci: remove limitation of the number of the available IRQs
https://patchwork.ozlabs.org/patch/771320/
[v2,8/9] pci: removing old irq api from pcie-designware
https://patchwork.ozlabs.org/patch/771360/
[v2,7/9] pci: keystone SoC driver adapted to new irq API
https://patchwork.ozlabs.org/patch/771361/
[v2,6/9] pci: qcom SoC driver adapted to new irq API
https://patchwork.ozlabs.org/patch/771362/
[v2,5/9] pci: generic PCIe DW driver adapted to new irq API
https://patchwork.ozlabs.org/patch/771365/
[v2,4/9] pci: artpec6 SoC driver adapted to new irq API
https://patchwork.ozlabs.org/patch/771364/
[v2,3/9] pci: imx6 SoC driver adapted to new irq API
https://patchwork.ozlabs.org/patch/771363/
[v2,2/9] pci: exynos SoC driver adapted to new irq API
https://patchwork.ozlabs.org/patch/771359/
[v2,1/9] pci: adding new irq api to pci-designware
https://patchwork.ozlabs.org/patch/771366/

The patch-set was globally accepted, but it broke the MSI mechanism for
Keystone, due to its specificity.

We are now going to resume this task and we would like to have your feedback
about our plan:

Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
Task 2: Port the existing patches to the new kernel version (except the keystone
one)

If TI can do the keystone_msi implementation it would be easier. If it's not
possible, we volunteer to do it, but we will need testing & debug assistance.

I appreciate very much your attention and feedback.

Thanks and have a good day,
Joao Pinto

Re: Re-activate task to add MSI-X to pcie-designware

[Lucas Stach]

Hi Joao,

Am Donnerstag, den 07.12.2017, 15:14 +0000 schrieb Joao Pinto:
> Hello to all,
> 
> I am sending this e-mail to request your opinion about the tasks needed to add
> support for MSI-X for pcie-designware based solutions.
> 
> A few months ago I submited a patch-set that added support for MSI-X:
> 
> [v2,9/9] pci: remove limitation of the number of the available IRQs
> https://patchwork.ozlabs.org/patch/771320/
> [v2,8/9] pci: removing old irq api from pcie-designware
> https://patchwork.ozlabs.org/patch/771360/
> [v2,7/9] pci: keystone SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771361/
> [v2,6/9] pci: qcom SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771362/
> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771365/
> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771364/
> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771363/
> [v2,2/9] pci: exynos SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771359/
> [v2,1/9] pci: adding new irq api to pci-designware
> https://patchwork.ozlabs.org/patch/771366/
> 
> The patch-set was globally accepted, but it broke the MSI mechanism for
> Keystone, due to its specificity.
> 
> We are now going to resume this task and we would like to have your feedback
> about our plan:
> 
> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
> Task 2: Port the existing patches to the new kernel version (except the keystone
> one)
> 
> If TI can do the keystone_msi implementation it would be easier. If it's not
> possible, we volunteer to do it, but we will need testing & debug assistance.
> 
> I appreciate very much your attention and feedback.
> 
> Thanks and have a good day,
> Joao Pinto
> 

The trivial implementation I have for MSI-X support is below. This has
been tested on i.MX6, but I guess it also works on Keystone as it
doesn't change the way the IRQs are set up.

-------------------------------->8-------------------------------------

>From 0d722f67e3995cf26795f6a5d66cca8e4997ed64 Mon Sep 17 00:00:00 2001
From: Lucas Stach <l.stach@pengutronix.de>
Date: Thu, 7 Dec 2017 12:54:04 +0100
Subject: [PATCH] PCI: dwc: implement MSI-X support

The DWC MSI controller does not support different MSI-X target addresses
and does not allow to route individual IRQs to different CPUs. Aside from
those shortcomings it is able to support MSI-X just fine.

Some devices like the Intel i210 network controller depend on MSI-X to
be available to enable all hardware features, so even a feature limited
implementation of MSI-X on the host side is useful.

Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
---
 drivers/pci/dwc/pcie-designware-host.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/dwc/pcie-designware-host.c b/drivers/pci/dwc/pcie-designware-host.c
index 81e2157a7cfb..a85101dd15c9 100644
--- a/drivers/pci/dwc/pcie-designware-host.c
+++ b/drivers/pci/dwc/pcie-designware-host.c
@@ -206,9 +206,6 @@ static int dw_msi_setup_irq(struct msi_controller *chip, struct pci_dev *pdev,
 	int irq, pos;
 	struct pcie_port *pp = pdev->bus->sysdata;
 
-	if (desc->msi_attrib.is_msix)
-		return -EINVAL;
-
 	irq = assign_irq(1, desc, &pos);
 	if (irq < 0)
 		return irq;
@@ -226,9 +223,19 @@ static int dw_msi_setup_irqs(struct msi_controller *chip, struct pci_dev *pdev,
 	struct msi_desc *desc;
 	struct pcie_port *pp = pdev->bus->sysdata;
 
-	/* MSI-X interrupts are not supported */
-	if (type == PCI_CAP_ID_MSIX)
-		return -EINVAL;
+	if (type == PCI_CAP_ID_MSIX) {
+		if ((MAX_MSI_IRQS - bitmap_weight(pp->msi_irq_in_use,
+						  MAX_MSI_IRQS)) < nvec)
+			return -ENOSPC;
+
+		for_each_pci_msi_entry(desc, pdev) {
+			int ret = dw_msi_setup_irq(chip, pdev, desc);
+			if (ret)
+				return ret;
+		}
+
+		return 0;
+	}
 
 	WARN_ON(!list_is_singular(&pdev->dev.msi_list));
 	desc = list_entry(pdev->dev.msi_list.next, struct msi_desc, list);
-- 
2.11.0

Re: Re-activate task to add MSI-X to pcie-designware

[Marc Zyngier]

[+ Lorenzo]

On 07/12/17 15:42, Lucas Stach wrote:
> Hi Joao,
> 
> Am Donnerstag, den 07.12.2017, 15:14 +0000 schrieb Joao Pinto:
>> Hello to all,
>>
>> I am sending this e-mail to request your opinion about the tasks needed to add
>> support for MSI-X for pcie-designware based solutions.
>>
>> A few months ago I submited a patch-set that added support for MSI-X:
>>
>> [v2,9/9] pci: remove limitation of the number of the available IRQs
>> https://patchwork.ozlabs.org/patch/771320/
>> [v2,8/9] pci: removing old irq api from pcie-designware
>> https://patchwork.ozlabs.org/patch/771360/
>> [v2,7/9] pci: keystone SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771361/
>> [v2,6/9] pci: qcom SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771362/
>> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771365/
>> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771364/
>> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771363/
>> [v2,2/9] pci: exynos SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771359/
>> [v2,1/9] pci: adding new irq api to pci-designware
>> https://patchwork.ozlabs.org/patch/771366/
>>
>> The patch-set was globally accepted, but it broke the MSI mechanism for
>> Keystone, due to its specificity.
>>
>> We are now going to resume this task and we would like to have your feedback
>> about our plan:
>>
>> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
>> Task 2: Port the existing patches to the new kernel version (except the keystone
>> one)
>>
>> If TI can do the keystone_msi implementation it would be easier. If it's not
>> possible, we volunteer to do it, but we will need testing & debug assistance.
>>
>> I appreciate very much your attention and feedback.
>>
>> Thanks and have a good day,
>> Joao Pinto
>>
> 
> The trivial implementation I have for MSI-X support is below. This has
> been tested on i.MX6, but I guess it also works on Keystone as it
> doesn't change the way the IRQs are set up.
> 
> -------------------------------->8-------------------------------------
> 
> From 0d722f67e3995cf26795f6a5d66cca8e4997ed64 Mon Sep 17 00:00:00 2001
> From: Lucas Stach <l.stach@pengutronix.de>
> Date: Thu, 7 Dec 2017 12:54:04 +0100
> Subject: [PATCH] PCI: dwc: implement MSI-X support
> 
> The DWC MSI controller does not support different MSI-X target addresses
> and does not allow to route individual IRQs to different CPUs. Aside from
> those shortcomings it is able to support MSI-X just fine.
> 
> Some devices like the Intel i210 network controller depend on MSI-X to
> be available to enable all hardware features, so even a feature limited
> implementation of MSI-X on the host side is useful.
> 
> Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
> ---
>  drivers/pci/dwc/pcie-designware-host.c | 19 +++++++++++++------
>  1 file changed, 13 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/pci/dwc/pcie-designware-host.c b/drivers/pci/dwc/pcie-designware-host.c
> index 81e2157a7cfb..a85101dd15c9 100644
> --- a/drivers/pci/dwc/pcie-designware-host.c
> +++ b/drivers/pci/dwc/pcie-designware-host.c
> @@ -206,9 +206,6 @@ static int dw_msi_setup_irq(struct msi_controller *chip, struct pci_dev *pdev,
>  	int irq, pos;
>  	struct pcie_port *pp = pdev->bus->sysdata;
>  
> -	if (desc->msi_attrib.is_msix)
> -		return -EINVAL;
> -
>  	irq = assign_irq(1, desc, &pos);
>  	if (irq < 0)
>  		return irq;
> @@ -226,9 +223,19 @@ static int dw_msi_setup_irqs(struct msi_controller *chip, struct pci_dev *pdev,
>  	struct msi_desc *desc;
>  	struct pcie_port *pp = pdev->bus->sysdata;
>  
> -	/* MSI-X interrupts are not supported */
> -	if (type == PCI_CAP_ID_MSIX)
> -		return -EINVAL;
> +	if (type == PCI_CAP_ID_MSIX) {
> +		if ((MAX_MSI_IRQS - bitmap_weight(pp->msi_irq_in_use,
> +						  MAX_MSI_IRQS)) < nvec)
> +			return -ENOSPC;
> +
> +		for_each_pci_msi_entry(desc, pdev) {
> +			int ret = dw_msi_setup_irq(chip, pdev, desc);
> +			if (ret)
> +				return ret;
> +		}
> +
> +		return 0;
> +	}
>  
>  	WARN_ON(!list_is_singular(&pdev->dev.msi_list));
>  	desc = list_entry(pdev->dev.msi_list.next, struct msi_desc, list);
> -- 
> 2.11.0
> 

Hi Lucas,

This is exactly what we're trying hard to get rid off. The whole
msi_setup_irqs and msi_controller stuff gets in the way of proper
abstraction, and litters the PCI subsystem with legacy stuff.

I'd rather see Joao's patches making it into mainline instead of keeping
this DWC madness on life support.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: Re-activate task to add MSI-X to pcie-designware

[Lorenzo Pieralisi]

On Thu, Dec 07, 2017 at 03:53:39PM +0000, Marc Zyngier wrote:
> [+ Lorenzo]
> 
> On 07/12/17 15:42, Lucas Stach wrote:
> > Hi Joao,
> > 
> > Am Donnerstag, den 07.12.2017, 15:14 +0000 schrieb Joao Pinto:
> >> Hello to all,
> >>
> >> I am sending this e-mail to request your opinion about the tasks needed to add
> >> support for MSI-X for pcie-designware based solutions.
> >>
> >> A few months ago I submited a patch-set that added support for MSI-X:
> >>
> >> [v2,9/9] pci: remove limitation of the number of the available IRQs
> >> https://patchwork.ozlabs.org/patch/771320/
> >> [v2,8/9] pci: removing old irq api from pcie-designware
> >> https://patchwork.ozlabs.org/patch/771360/
> >> [v2,7/9] pci: keystone SoC driver adapted to new irq API
> >> https://patchwork.ozlabs.org/patch/771361/
> >> [v2,6/9] pci: qcom SoC driver adapted to new irq API
> >> https://patchwork.ozlabs.org/patch/771362/
> >> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
> >> https://patchwork.ozlabs.org/patch/771365/
> >> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
> >> https://patchwork.ozlabs.org/patch/771364/
> >> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
> >> https://patchwork.ozlabs.org/patch/771363/
> >> [v2,2/9] pci: exynos SoC driver adapted to new irq API
> >> https://patchwork.ozlabs.org/patch/771359/
> >> [v2,1/9] pci: adding new irq api to pci-designware
> >> https://patchwork.ozlabs.org/patch/771366/
> >>
> >> The patch-set was globally accepted, but it broke the MSI mechanism for
> >> Keystone, due to its specificity.
> >>
> >> We are now going to resume this task and we would like to have your feedback
> >> about our plan:
> >>
> >> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
> >> Task 2: Port the existing patches to the new kernel version (except the keystone
> >> one)
> >>
> >> If TI can do the keystone_msi implementation it would be easier. If it's not
> >> possible, we volunteer to do it, but we will need testing & debug assistance.
> >>
> >> I appreciate very much your attention and feedback.
> >>
> >> Thanks and have a good day,
> >> Joao Pinto
> >>
> > 
> > The trivial implementation I have for MSI-X support is below. This has
> > been tested on i.MX6, but I guess it also works on Keystone as it
> > doesn't change the way the IRQs are set up.
> > 
> > -------------------------------->8-------------------------------------
> > 
> > From 0d722f67e3995cf26795f6a5d66cca8e4997ed64 Mon Sep 17 00:00:00 2001
> > From: Lucas Stach <l.stach@pengutronix.de>
> > Date: Thu, 7 Dec 2017 12:54:04 +0100
> > Subject: [PATCH] PCI: dwc: implement MSI-X support
> > 
> > The DWC MSI controller does not support different MSI-X target addresses
> > and does not allow to route individual IRQs to different CPUs. Aside from
> > those shortcomings it is able to support MSI-X just fine.
> > 
> > Some devices like the Intel i210 network controller depend on MSI-X to
> > be available to enable all hardware features, so even a feature limited
> > implementation of MSI-X on the host side is useful.
> > 
> > Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
> > ---
> >  drivers/pci/dwc/pcie-designware-host.c | 19 +++++++++++++------
> >  1 file changed, 13 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/pci/dwc/pcie-designware-host.c b/drivers/pci/dwc/pcie-designware-host.c
> > index 81e2157a7cfb..a85101dd15c9 100644
> > --- a/drivers/pci/dwc/pcie-designware-host.c
> > +++ b/drivers/pci/dwc/pcie-designware-host.c
> > @@ -206,9 +206,6 @@ static int dw_msi_setup_irq(struct msi_controller *chip, struct pci_dev *pdev,
> >  	int irq, pos;
> >  	struct pcie_port *pp = pdev->bus->sysdata;
> >  
> > -	if (desc->msi_attrib.is_msix)
> > -		return -EINVAL;
> > -
> >  	irq = assign_irq(1, desc, &pos);
> >  	if (irq < 0)
> >  		return irq;
> > @@ -226,9 +223,19 @@ static int dw_msi_setup_irqs(struct msi_controller *chip, struct pci_dev *pdev,
> >  	struct msi_desc *desc;
> >  	struct pcie_port *pp = pdev->bus->sysdata;
> >  
> > -	/* MSI-X interrupts are not supported */
> > -	if (type == PCI_CAP_ID_MSIX)
> > -		return -EINVAL;
> > +	if (type == PCI_CAP_ID_MSIX) {
> > +		if ((MAX_MSI_IRQS - bitmap_weight(pp->msi_irq_in_use,
> > +						  MAX_MSI_IRQS)) < nvec)
> > +			return -ENOSPC;
> > +
> > +		for_each_pci_msi_entry(desc, pdev) {
> > +			int ret = dw_msi_setup_irq(chip, pdev, desc);
> > +			if (ret)
> > +				return ret;
> > +		}
> > +
> > +		return 0;
> > +	}
> >  
> >  	WARN_ON(!list_is_singular(&pdev->dev.msi_list));
> >  	desc = list_entry(pdev->dev.msi_list.next, struct msi_desc, list);
> > -- 
> > 2.11.0
> > 
> 
> Hi Lucas,
> 
> This is exactly what we're trying hard to get rid off. The whole
> msi_setup_irqs and msi_controller stuff gets in the way of proper
> abstraction, and litters the PCI subsystem with legacy stuff.
> 
> I'd rather see Joao's patches making it into mainline instead of keeping
> this DWC madness on life support.

+1

Thanks,
Lorenzo

Re: Re-activate task to add MSI-X to pcie-designware

[Lorenzo Pieralisi]

On Thu, Dec 07, 2017 at 03:14:22PM +0000, Joao Pinto wrote:
> Hello to all,
> 
> I am sending this e-mail to request your opinion about the tasks needed to add
> support for MSI-X for pcie-designware based solutions.
> 
> A few months ago I submited a patch-set that added support for MSI-X:
> 
> [v2,9/9] pci: remove limitation of the number of the available IRQs
> https://patchwork.ozlabs.org/patch/771320/
> [v2,8/9] pci: removing old irq api from pcie-designware
> https://patchwork.ozlabs.org/patch/771360/
> [v2,7/9] pci: keystone SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771361/
> [v2,6/9] pci: qcom SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771362/
> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771365/
> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771364/
> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771363/
> [v2,2/9] pci: exynos SoC driver adapted to new irq API
> https://patchwork.ozlabs.org/patch/771359/
> [v2,1/9] pci: adding new irq api to pci-designware
> https://patchwork.ozlabs.org/patch/771366/
> 
> The patch-set was globally accepted, but it broke the MSI mechanism for
> Keystone, due to its specificity.

Hi Joao,

would you mind pointing me at the discussion where the breakage was
explained please ? It is to understand what are the TI specific bits
we are talking about here.

> We are now going to resume this task and we would like to have your feedback
> about our plan:
> 
> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
> Task 2: Port the existing patches to the new kernel version (except the keystone
> one)
> 
> If TI can do the keystone_msi implementation it would be easier. If it's not
> possible, we volunteer to do it, but we will need testing & debug assistance.

I would ask Murali to chime in please to understand the direction to
take - at least I expect him to be able to test the changes.

Thanks,
Lorenzo

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi,

On Friday 08 December 2017 04:32 PM, Lorenzo Pieralisi wrote:
> On Thu, Dec 07, 2017 at 03:14:22PM +0000, Joao Pinto wrote:
>> Hello to all,
>>
>> I am sending this e-mail to request your opinion about the tasks needed to add
>> support for MSI-X for pcie-designware based solutions.
>>
>> A few months ago I submited a patch-set that added support for MSI-X:
>>
>> [v2,9/9] pci: remove limitation of the number of the available IRQs
>> https://patchwork.ozlabs.org/patch/771320/
>> [v2,8/9] pci: removing old irq api from pcie-designware
>> https://patchwork.ozlabs.org/patch/771360/
>> [v2,7/9] pci: keystone SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771361/
>> [v2,6/9] pci: qcom SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771362/
>> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771365/
>> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771364/
>> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771363/
>> [v2,2/9] pci: exynos SoC driver adapted to new irq API
>> https://patchwork.ozlabs.org/patch/771359/
>> [v2,1/9] pci: adding new irq api to pci-designware
>> https://patchwork.ozlabs.org/patch/771366/
>>
>> The patch-set was globally accepted, but it broke the MSI mechanism for
>> Keystone, due to its specificity.
> 
> Hi Joao,
> 
> would you mind pointing me at the discussion where the breakage was
> explained please ? It is to understand what are the TI specific bits
> we are talking about here.
> 
>> We are now going to resume this task and we would like to have your feedback
>> about our plan:
>>
>> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
>> Task 2: Port the existing patches to the new kernel version (except the keystone
>> one)
>>
>> If TI can do the keystone_msi implementation it would be easier. If it's not
>> possible, we volunteer to do it, but we will need testing & debug assistance.

sure, I should be able to help with at-least testing if not implementing.

Thanks
Kishon

Re: Re-activate task to add MSI-X to pcie-designware

[Gustavo Pimentel]

Hi Kishon,

I would like to collaborate with you on this subject, I have on my side João's
patches updated to the Bjorn's latest kernel version.

Regards,

Gustavo


On 11/12/2017 13:23, Kishon Vijay Abraham I wrote:
> Hi,
>
> On Friday 08 December 2017 04:32 PM, Lorenzo Pieralisi wrote:
>> On Thu, Dec 07, 2017 at 03:14:22PM +0000, Joao Pinto wrote:
>>> Hello to all,
>>>
>>> I am sending this e-mail to request your opinion about the tasks needed to add
>>> support for MSI-X for pcie-designware based solutions.
>>>
>>> A few months ago I submited a patch-set that added support for MSI-X:
>>>
>>> [v2,9/9] pci: remove limitation of the number of the available IRQs
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771320_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=p3N21H26OSEMkCxG6P0d-nO2j8dsNTPY3vyU1QDWjT0&e=
>>> [v2,8/9] pci: removing old irq api from pcie-designware
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771360_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=qQWZngqGOJ1ILhY65f3YL6xrgp_5UXRR0NrDlSaHSTQ&e=
>>> [v2,7/9] pci: keystone SoC driver adapted to new irq API
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771361_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=4J5yDyNZ0Hp9AD75kGoIjfyT-dr7gWoj5B3XO4cx7Q0&e=
>>> [v2,6/9] pci: qcom SoC driver adapted to new irq API
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771362_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=KpC3k1Y_c4guyLRzb-w5H_Kglt6KWYb8Xk7iWsZqkzY&e=
>>> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771365_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=TNoIYF7RaBl7jXnmFrFSIJD1a_vNJL2IKTYPbaJEpww&e=
>>> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771364_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=p9aOuvgKv8e74QNe5XxXiI7gfvkPeCeanhs6i3FBwnc&e=
>>> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771363_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=ibzCdKoVbLHn-0vo9tzAlnirwgZZsDpILRkTEowHwWs&e=
>>> [v2,2/9] pci: exynos SoC driver adapted to new irq API
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771359_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=D0dvpjeXB-BTo69WreBpia6vlml1pBysJU6ta-9cLOM&e=
>>> [v2,1/9] pci: adding new irq api to pci-designware
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771366_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=Wu_iKSHepvoDa3X5gU02AMMa9yzAgrT2jIfiu6jHgPs&e=
>>>
>>> The patch-set was globally accepted, but it broke the MSI mechanism for
>>> Keystone, due to its specificity.
>> Hi Joao,
>>
>> would you mind pointing me at the discussion where the breakage was
>> explained please ? It is to understand what are the TI specific bits
>> we are talking about here.
>>
>>> We are now going to resume this task and we would like to have your feedback
>>> about our plan:
>>>
>>> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
>>> Task 2: Port the existing patches to the new kernel version (except the keystone
>>> one)
>>>
>>> If TI can do the keystone_msi implementation it would be easier. If it's not
>>> possible, we volunteer to do it, but we will need testing & debug assistance.
> sure, I should be able to help with at-least testing if not implementing.
>
> Thanks
> Kishon

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi,

On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
> Hi Kishon,
> 
> I would like to collaborate with you on this subject, I have on my side João's
> patches updated to the Bjorn's latest kernel version.

cool, do you have a branch so that I can check what breaks in keystone and debug?

Thanks
Kishon
> 
> Regards,
> 
> Gustavo
> 
> 
> On 11/12/2017 13:23, Kishon Vijay Abraham I wrote:
>> Hi,
>>
>> On Friday 08 December 2017 04:32 PM, Lorenzo Pieralisi wrote:
>>> On Thu, Dec 07, 2017 at 03:14:22PM +0000, Joao Pinto wrote:
>>>> Hello to all,
>>>>
>>>> I am sending this e-mail to request your opinion about the tasks needed to add
>>>> support for MSI-X for pcie-designware based solutions.
>>>>
>>>> A few months ago I submited a patch-set that added support for MSI-X:
>>>>
>>>> [v2,9/9] pci: remove limitation of the number of the available IRQs
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771320_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=p3N21H26OSEMkCxG6P0d-nO2j8dsNTPY3vyU1QDWjT0&e=
>>>> [v2,8/9] pci: removing old irq api from pcie-designware
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771360_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=qQWZngqGOJ1ILhY65f3YL6xrgp_5UXRR0NrDlSaHSTQ&e=
>>>> [v2,7/9] pci: keystone SoC driver adapted to new irq API
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771361_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=4J5yDyNZ0Hp9AD75kGoIjfyT-dr7gWoj5B3XO4cx7Q0&e=
>>>> [v2,6/9] pci: qcom SoC driver adapted to new irq API
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771362_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=KpC3k1Y_c4guyLRzb-w5H_Kglt6KWYb8Xk7iWsZqkzY&e=
>>>> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771365_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=TNoIYF7RaBl7jXnmFrFSIJD1a_vNJL2IKTYPbaJEpww&e=
>>>> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771364_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=p9aOuvgKv8e74QNe5XxXiI7gfvkPeCeanhs6i3FBwnc&e=
>>>> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771363_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=ibzCdKoVbLHn-0vo9tzAlnirwgZZsDpILRkTEowHwWs&e=
>>>> [v2,2/9] pci: exynos SoC driver adapted to new irq API
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771359_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=D0dvpjeXB-BTo69WreBpia6vlml1pBysJU6ta-9cLOM&e=
>>>> [v2,1/9] pci: adding new irq api to pci-designware
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771366_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=Wu_iKSHepvoDa3X5gU02AMMa9yzAgrT2jIfiu6jHgPs&e=
>>>>
>>>> The patch-set was globally accepted, but it broke the MSI mechanism for
>>>> Keystone, due to its specificity.
>>> Hi Joao,
>>>
>>> would you mind pointing me at the discussion where the breakage was
>>> explained please ? It is to understand what are the TI specific bits
>>> we are talking about here.
>>>
>>>> We are now going to resume this task and we would like to have your feedback
>>>> about our plan:
>>>>
>>>> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
>>>> Task 2: Port the existing patches to the new kernel version (except the keystone
>>>> one)
>>>>
>>>> If TI can do the keystone_msi implementation it would be easier. If it's not
>>>> possible, we volunteer to do it, but we will need testing & debug assistance.
>> sure, I should be able to help with at-least testing if not implementing.
>>
>> Thanks
>> Kishon
> 
> 

Re: Re-activate task to add MSI-X to pcie-designware

[Gustavo Pimentel]

On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
version of the patches that are running on our equipment, please check with yours
> Hi,
> 
> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>> Hi Kishon,
>>
>> I would like to collaborate with you on this subject, I have on my side João's
>> patches updated to the Bjorn's latest kernel version.
> 
> cool, do you have a branch so that I can check what breaks in keystone and debug?
> 
Unfortunately not, however I'll mailed you the patches immediately.
Once again, thanks.

Gustavo

> Thanks
> Kishon
>>
>> Regards,
>>
>> Gustavo
>>
>>
>> On 11/12/2017 13:23, Kishon Vijay Abraham I wrote:
>>> Hi,
>>>
>>> On Friday 08 December 2017 04:32 PM, Lorenzo Pieralisi wrote:
>>>> On Thu, Dec 07, 2017 at 03:14:22PM +0000, Joao Pinto wrote:
>>>>> Hello to all,
>>>>>
>>>>> I am sending this e-mail to request your opinion about the tasks needed to add
>>>>> support for MSI-X for pcie-designware based solutions.
>>>>>
>>>>> A few months ago I submited a patch-set that added support for MSI-X:
>>>>>
>>>>> [v2,9/9] pci: remove limitation of the number of the available IRQs
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771320_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=p3N21H26OSEMkCxG6P0d-nO2j8dsNTPY3vyU1QDWjT0&e=
>>>>> [v2,8/9] pci: removing old irq api from pcie-designware
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771360_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=qQWZngqGOJ1ILhY65f3YL6xrgp_5UXRR0NrDlSaHSTQ&e=
>>>>> [v2,7/9] pci: keystone SoC driver adapted to new irq API
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771361_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=4J5yDyNZ0Hp9AD75kGoIjfyT-dr7gWoj5B3XO4cx7Q0&e=
>>>>> [v2,6/9] pci: qcom SoC driver adapted to new irq API
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771362_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=KpC3k1Y_c4guyLRzb-w5H_Kglt6KWYb8Xk7iWsZqkzY&e=
>>>>> [v2,5/9] pci: generic PCIe DW driver adapted to new irq API
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771365_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=TNoIYF7RaBl7jXnmFrFSIJD1a_vNJL2IKTYPbaJEpww&e=
>>>>> [v2,4/9] pci: artpec6 SoC driver adapted to new irq API
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771364_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=p9aOuvgKv8e74QNe5XxXiI7gfvkPeCeanhs6i3FBwnc&e=
>>>>> [v2,3/9] pci: imx6 SoC driver adapted to new irq API
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771363_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=ibzCdKoVbLHn-0vo9tzAlnirwgZZsDpILRkTEowHwWs&e=
>>>>> [v2,2/9] pci: exynos SoC driver adapted to new irq API
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771359_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=D0dvpjeXB-BTo69WreBpia6vlml1pBysJU6ta-9cLOM&e=
>>>>> [v2,1/9] pci: adding new irq api to pci-designware
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_771366_&d=DwIC-g&c=DPL6_X_6JkXFx7AXWqB0tg&r=bkWxpLoW-f-E3EdiDCCa0_h0PicsViasSlvIpzZvPxs&m=fBQ8uqCEYYa2ogSRPy8cR1jhWPLboPICkxUdpcYB6Yw&s=Wu_iKSHepvoDa3X5gU02AMMa9yzAgrT2jIfiu6jHgPs&e=
>>>>>
>>>>> The patch-set was globally accepted, but it broke the MSI mechanism for
>>>>> Keystone, due to its specificity.
>>>> Hi Joao,
>>>>
>>>> would you mind pointing me at the discussion where the breakage was
>>>> explained please ? It is to understand what are the TI specific bits
>>>> we are talking about here.
>>>>
>>>>> We are now going to resume this task and we would like to have your feedback
>>>>> about our plan:
>>>>>
>>>>> Task 1: Help TI to create a keystone_msi driver to isolate its custom msi mechanism
>>>>> Task 2: Port the existing patches to the new kernel version (except the keystone
>>>>> one)
>>>>>
>>>>> If TI can do the keystone_msi implementation it would be easier. If it's not
>>>>> possible, we volunteer to do it, but we will need testing & debug assistance.
>>> sure, I should be able to help with at-least testing if not implementing.
>>>
>>> Thanks
>>> Kishon
>>
>>

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi Gustavo,

On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
> version of the patches that are running on our equipment, please check with yours
>> Hi,
>>
>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>> Hi Kishon,
>>>
>>> I would like to collaborate with you on this subject, I have on my side João's
>>> patches updated to the Bjorn's latest kernel version.
>>
>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>
> Unfortunately not, however I'll mailed you the patches immediately.
> Once again, thanks.

Thanks.

I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
are some of my observations after applying your patch series.

root@k2g-evm:~# cat /proc/interrupts
           CPU0
 17:          0     GICv2  29 Level     arch_timer
 18:       1816     GICv2  30 Level     arch_timer
 21:          0     GICv2  36 Edge      arm-pmu
 22:        792     GICv2 196 Edge      ttyS0
 23:          5     GICv2 120 Edge      2530000.i2c
 24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
 25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
 41:          0     GICv2 232 Edge      2700000.edma_ccint
 43:          0     GICv2 249 Edge      2700000.edma_ccerrint
 44:       2497     GICv2 240 Edge      2728000.edma_ccint
 46:          0     GICv2 252 Edge      2728000.edma_ccerrint
 47:       7551     GICv2 128 Edge      mmc0
 48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
 49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
 50:          0     GICv2  96 Edge      21805400.spi
 52:          0     GICv2 100 Edge      21805c00.spi
 53:          0     GICv2 102 Edge      21806000.spi
 54:          0     GICv2  92 Edge      pcie-error-irq
211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
IPI0:          0  CPU wakeup interrupts
IPI1:          0  Timer broadcast interrupts
IPI2:          0  Rescheduling interrupts
IPI3:          0  Function call interrupts
IPI4:          0  CPU stop interrupts
IPI5:          0  IRQ work interrupts
IPI6:          0  completion interrupts
Err:          0

root@k2g-evm:~# pcitest -m 1
[   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
[   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
MSI1:           NOT OKAY

Here there is an off by one error. "pcitest -m 1" should ideally have raised an
interrupt in 281 but from the above debug print it looks like it's raising an
interrupt in 280.

So I tried disabling the pcieportbus driver. These are my observations once I
disable pcieportbus driver

root@k2g-evm:~# cat /proc/interrupts
           CPU0
 17:          0     GICv2  29 Level     arch_timer
 18:       2222     GICv2  30 Level     arch_timer
 21:          0     GICv2  36 Edge      arm-pmu
 22:        821     GICv2 196 Edge      ttyS0
 23:          5     GICv2 120 Edge      2530000.i2c
 24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
 25:        909     GICv2 356 Level     2a00000.msgmgr rx_005_002
 41:          0     GICv2 232 Edge      2700000.edma_ccint
 43:          0     GICv2 249 Edge      2700000.edma_ccerrint
 44:       2442     GICv2 240 Edge      2728000.edma_ccint
 46:          0     GICv2 252 Edge      2728000.edma_ccerrint
 47:       7297     GICv2 128 Edge      mmc0
 48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
 49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
 50:          0     GICv2  96 Edge      21805400.spi
 52:          0     GICv2 100 Edge      21805c00.spi
 53:          0     GICv2 102 Edge      21806000.spi
 54:          0     GICv2  92 Edge      pcie-error-irq
211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
280:          0   PCI-MSI 524288 Edge      pci-endpoint-test
281:          0   PCI-MSI 524289 Edge      pci-endpoint-test
282:          0   PCI-MSI 524290 Edge      pci-endpoint-test
283:          0   PCI-MSI 524291 Edge      pci-endpoint-test
284:          0   PCI-MSI 524292 Edge      pci-endpoint-test
285:          0   PCI-MSI 524293 Edge      pci-endpoint-test
286:          0   PCI-MSI 524294 Edge      pci-endpoint-test
287:          0   PCI-MSI 524295 Edge      pci-endpoint-test
288:          0   PCI-MSI 524296 Edge      pci-endpoint-test
289:          0   PCI-MSI 524297 Edge      pci-endpoint-test
290:          0   PCI-MSI 524298 Edge      pci-endpoint-test
291:          0   PCI-MSI 524299 Edge      pci-endpoint-test
292:          0   PCI-MSI 524300 Edge      pci-endpoint-test
293:          0   PCI-MSI 524301 Edge      pci-endpoint-test
294:          0   PCI-MSI 524302 Edge      pci-endpoint-test
295:          0   PCI-MSI 524303 Edge      pci-endpoint-test
IPI0:          0  CPU wakeup interrupts
IPI1:          0  Timer broadcast interrupts
IPI2:          0  Rescheduling interrupts
IPI3:          0  Function call interrupts
IPI4:          0  CPU stop interrupts
IPI5:          0  IRQ work interrupts
IPI6:          0  completion interrupts
Err:          0

root@k2g-evm:~# pcitest -m 1
[   57.791535] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
[   57.798643] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
MSI1:           OKAY
root@k2g-evm:~# pcitest -m 2
[   59.791509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 272
[   59.798614] keystone-pcie 21801000.pcie: irq: bit 0, vector 1, virq 281
MSI2:           OKAY
root@k2g-evm:~# pcitest -m 3
[   61.271509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 273
[   61.278616] keystone-pcie 21801000.pcie: irq: bit 0, vector 2, virq 282
MSI3:           OKAY
root@k2g-evm:~# pcitest -m 4
[   62.791514] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 274
[   62.798619] keystone-pcie 21801000.pcie: irq: bit 0, vector 3, virq 283
MSI4:           OKAY
root@k2g-evm:~# pcitest -m 5
[   64.311513] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 275
[   64.318620] keystone-pcie 21801000.pcie: irq: bit 0, vector 4, virq 284
MSI5:           OKAY
root@k2g-evm:~# pcitest -m 6
[   66.351518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 276
[   66.358624] keystone-pcie 21801000.pcie: irq: bit 0, vector 5, virq 285
MSI6:           OKAY
root@k2g-evm:~# pcitest -m 7
[   67.811516] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 277
[   67.818623] keystone-pcie 21801000.pcie: irq: bit 0, vector 6, virq 286
MSI7:           OKAY
root@k2g-evm:~# pcitest -m 8
[   69.291518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 278
[   69.298625] keystone-pcie 21801000.pcie: irq: bit 0, vector 7, virq 287
MSI8:           OKAY
root@k2g-evm:~# pcitest -m 9
MSI9:           NOT OKAY
root@k2g-evm:~# pcitest -m 10
MSI10:          NOT OKAY

So once I disable pcieportbus, I could get the first 8 interrupts correctly.
But after that there are no interrupts at all.

Let me know if you have some clues for me to try.

Thanks
Kishon

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi,

On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
> Hi Gustavo,
> 
> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>> version of the patches that are running on our equipment, please check with yours
>>> Hi,
>>>
>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>> Hi Kishon,
>>>>
>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>> patches updated to the Bjorn's latest kernel version.
>>>
>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>
>> Unfortunately not, however I'll mailed you the patches immediately.
>> Once again, thanks.
> 
> Thanks.
> 
> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
> are some of my observations after applying your patch series.
> 
> root@k2g-evm:~# cat /proc/interrupts
>            CPU0
>  17:          0     GICv2  29 Level     arch_timer
>  18:       1816     GICv2  30 Level     arch_timer
>  21:          0     GICv2  36 Edge      arm-pmu
>  22:        792     GICv2 196 Edge      ttyS0
>  23:          5     GICv2 120 Edge      2530000.i2c
>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>  47:       7551     GICv2 128 Edge      mmc0
>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>  50:          0     GICv2  96 Edge      21805400.spi
>  52:          0     GICv2 100 Edge      21805c00.spi
>  53:          0     GICv2 102 Edge      21806000.spi
>  54:          0     GICv2  92 Edge      pcie-error-irq
> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
> IPI0:          0  CPU wakeup interrupts
> IPI1:          0  Timer broadcast interrupts
> IPI2:          0  Rescheduling interrupts
> IPI3:          0  Function call interrupts
> IPI4:          0  CPU stop interrupts
> IPI5:          0  IRQ work interrupts
> IPI6:          0  completion interrupts
> Err:          0
> 
> root@k2g-evm:~# pcitest -m 1
> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
> MSI1:           NOT OKAY
> 
> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
> interrupt in 281 but from the above debug print it looks like it's raising an
> interrupt in 280.

The above error was due to an pci_endpoint bug. I'll send a patch to fix this
separately.
> 
> So I tried disabling the pcieportbus driver. These are my observations once I
> disable pcieportbus driver
> 
> root@k2g-evm:~# cat /proc/interrupts
>            CPU0
>  17:          0     GICv2  29 Level     arch_timer
>  18:       2222     GICv2  30 Level     arch_timer
>  21:          0     GICv2  36 Edge      arm-pmu
>  22:        821     GICv2 196 Edge      ttyS0
>  23:          5     GICv2 120 Edge      2530000.i2c
>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>  25:        909     GICv2 356 Level     2a00000.msgmgr rx_005_002
>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>  44:       2442     GICv2 240 Edge      2728000.edma_ccint
>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>  47:       7297     GICv2 128 Edge      mmc0
>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>  50:          0     GICv2  96 Edge      21805400.spi
>  52:          0     GICv2 100 Edge      21805c00.spi
>  53:          0     GICv2 102 Edge      21806000.spi
>  54:          0     GICv2  92 Edge      pcie-error-irq
> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
> 280:          0   PCI-MSI 524288 Edge      pci-endpoint-test
> 281:          0   PCI-MSI 524289 Edge      pci-endpoint-test
> 282:          0   PCI-MSI 524290 Edge      pci-endpoint-test
> 283:          0   PCI-MSI 524291 Edge      pci-endpoint-test
> 284:          0   PCI-MSI 524292 Edge      pci-endpoint-test
> 285:          0   PCI-MSI 524293 Edge      pci-endpoint-test
> 286:          0   PCI-MSI 524294 Edge      pci-endpoint-test
> 287:          0   PCI-MSI 524295 Edge      pci-endpoint-test
> 288:          0   PCI-MSI 524296 Edge      pci-endpoint-test
> 289:          0   PCI-MSI 524297 Edge      pci-endpoint-test
> 290:          0   PCI-MSI 524298 Edge      pci-endpoint-test
> 291:          0   PCI-MSI 524299 Edge      pci-endpoint-test
> 292:          0   PCI-MSI 524300 Edge      pci-endpoint-test
> 293:          0   PCI-MSI 524301 Edge      pci-endpoint-test
> 294:          0   PCI-MSI 524302 Edge      pci-endpoint-test
> 295:          0   PCI-MSI 524303 Edge      pci-endpoint-test
> IPI0:          0  CPU wakeup interrupts
> IPI1:          0  Timer broadcast interrupts
> IPI2:          0  Rescheduling interrupts
> IPI3:          0  Function call interrupts
> IPI4:          0  CPU stop interrupts
> IPI5:          0  IRQ work interrupts
> IPI6:          0  completion interrupts
> Err:          0
> 
> root@k2g-evm:~# pcitest -m 1
> [   57.791535] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
> [   57.798643] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
> MSI1:           OKAY
> root@k2g-evm:~# pcitest -m 2
> [   59.791509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 272
> [   59.798614] keystone-pcie 21801000.pcie: irq: bit 0, vector 1, virq 281
> MSI2:           OKAY
> root@k2g-evm:~# pcitest -m 3
> [   61.271509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 273
> [   61.278616] keystone-pcie 21801000.pcie: irq: bit 0, vector 2, virq 282
> MSI3:           OKAY
> root@k2g-evm:~# pcitest -m 4
> [   62.791514] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 274
> [   62.798619] keystone-pcie 21801000.pcie: irq: bit 0, vector 3, virq 283
> MSI4:           OKAY
> root@k2g-evm:~# pcitest -m 5
> [   64.311513] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 275
> [   64.318620] keystone-pcie 21801000.pcie: irq: bit 0, vector 4, virq 284
> MSI5:           OKAY
> root@k2g-evm:~# pcitest -m 6
> [   66.351518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 276
> [   66.358624] keystone-pcie 21801000.pcie: irq: bit 0, vector 5, virq 285
> MSI6:           OKAY
> root@k2g-evm:~# pcitest -m 7
> [   67.811516] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 277
> [   67.818623] keystone-pcie 21801000.pcie: irq: bit 0, vector 6, virq 286
> MSI7:           OKAY
> root@k2g-evm:~# pcitest -m 8
> [   69.291518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 278
> [   69.298625] keystone-pcie 21801000.pcie: irq: bit 0, vector 7, virq 287
> MSI8:           OKAY
> root@k2g-evm:~# pcitest -m 9
> MSI9:           NOT OKAY
> root@k2g-evm:~# pcitest -m 10
> MSI10:          NOT OKAY
> 
> So once I disable pcieportbus, I could get the first 8 interrupts correctly.
> But after that there are no interrupts at all.

This issues is due to the previous interrupt not acknowledged and below fixes
the issue.

diff --git a/drivers/pci/dwc/pcie-designware-host.c
b/drivers/pci/dwc/pcie-designware-host.c
index 2816bd98f66c..b25e788e9828 100644
--- a/drivers/pci/dwc/pcie-designware-host.c
+++ b/drivers/pci/dwc/pcie-designware-host.c
@@ -58,8 +58,20 @@ static void dw_msi_unmask_irq(struct irq_data *d)
        irq_chip_unmask_parent(d);
 }

+static void dw_pci_ack_irq(struct irq_data *d)
+{
+       struct msi_desc *msi = irq_data_get_msi_desc(d);
+       struct pcie_port *pp;
+
+       pp = (struct pcie_port *) msi_desc_to_pci_sysdata(msi);
+
+       if (pp->ops->msi_irq_ack)
+               pp->ops->msi_irq_ack(d->irq, pp);
+}
+
 static struct irq_chip dw_pcie_msi_irq_chip = {
        .name = "PCI-MSI",
+       .irq_ack = dw_pci_ack_irq,
        .irq_mask = dw_msi_mask_irq,
        .irq_unmask = dw_msi_unmask_irq,
 };
@@ -193,20 +205,8 @@ static void dw_pci_bottom_unmask(struct irq_data *data)
        spin_unlock_irqrestore(&pp->lock, flags);
 }

-static void dw_pci_bottom_ack(struct irq_data *d)
-{
-       struct msi_desc *msi = irq_data_get_msi_desc(d);
-       struct pcie_port *pp;
-
-       pp = (struct pcie_port *) msi_desc_to_pci_sysdata(msi);
-
-       if (pp->ops->msi_irq_ack)
-               pp->ops->msi_irq_ack(d->irq, pp);
-}
-
 static struct irq_chip dw_pci_msi_bottom_irq_chip = {
        .name = "DWPCI-MSI",
-       .irq_ack                = dw_pci_bottom_ack,
        .irq_compose_msi_msg = dw_pci_setup_msi_msg,
        .irq_set_affinity = dw_pci_msi_set_affinity,
        .irq_mask = dw_pci_bottom_mask,
@@ -240,7 +240,7 @@ static int dw_pcie_irq_domain_alloc(struct irq_domain *domain,
        for (i = 0; i < nr_irqs; i++)
                irq_domain_set_info(domain, virq + i, bit + i,
                                    &dw_pci_msi_bottom_irq_chip,
-                                   domain->host_data, handle_simple_irq,
+                                   domain->host_data, handle_level_irq,
                                    NULL, NULL);

        return 0;

Thanks
Kishon

Re: Re-activate task to add MSI-X to pcie-designware

[Gustavo Pimentel]

On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
> Hi,
> 
> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>> Hi Gustavo,
>>
>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>> version of the patches that are running on our equipment, please check with yours
>>>> Hi,
>>>>
>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>> Hi Kishon,
>>>>>
>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>> patches updated to the Bjorn's latest kernel version.
>>>>
>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>
>>> Unfortunately not, however I'll mailed you the patches immediately.
>>> Once again, thanks.
>>
>> Thanks.
>>
>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>> are some of my observations after applying your patch series.
>>
>> root@k2g-evm:~# cat /proc/interrupts
>>            CPU0
>>  17:          0     GICv2  29 Level     arch_timer
>>  18:       1816     GICv2  30 Level     arch_timer
>>  21:          0     GICv2  36 Edge      arm-pmu
>>  22:        792     GICv2 196 Edge      ttyS0
>>  23:          5     GICv2 120 Edge      2530000.i2c
>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>  47:       7551     GICv2 128 Edge      mmc0
>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>  50:          0     GICv2  96 Edge      21805400.spi
>>  52:          0     GICv2 100 Edge      21805c00.spi
>>  53:          0     GICv2 102 Edge      21806000.spi
>>  54:          0     GICv2  92 Edge      pcie-error-irq
>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>> IPI0:          0  CPU wakeup interrupts
>> IPI1:          0  Timer broadcast interrupts
>> IPI2:          0  Rescheduling interrupts
>> IPI3:          0  Function call interrupts
>> IPI4:          0  CPU stop interrupts
>> IPI5:          0  IRQ work interrupts
>> IPI6:          0  completion interrupts
>> Err:          0
>>
>> root@k2g-evm:~# pcitest -m 1
>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>> MSI1:           NOT OKAY
>>
>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>> interrupt in 281 but from the above debug print it looks like it's raising an
>> interrupt in 280.
> 
> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
> separately.

Ok great!

>>
>> So I tried disabling the pcieportbus driver. These are my observations once I
>> disable pcieportbus driver
>>
>> root@k2g-evm:~# cat /proc/interrupts
>>            CPU0
>>  17:          0     GICv2  29 Level     arch_timer
>>  18:       2222     GICv2  30 Level     arch_timer
>>  21:          0     GICv2  36 Edge      arm-pmu
>>  22:        821     GICv2 196 Edge      ttyS0
>>  23:          5     GICv2 120 Edge      2530000.i2c
>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>  25:        909     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>  44:       2442     GICv2 240 Edge      2728000.edma_ccint
>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>  47:       7297     GICv2 128 Edge      mmc0
>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>  50:          0     GICv2  96 Edge      21805400.spi
>>  52:          0     GICv2 100 Edge      21805c00.spi
>>  53:          0     GICv2 102 Edge      21806000.spi
>>  54:          0     GICv2  92 Edge      pcie-error-irq
>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>> 280:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>> 281:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>> 282:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>> 283:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>> 284:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>> 285:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>> 286:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>> 287:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>> 288:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>> 289:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>> 290:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>> 291:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>> 292:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>> 293:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>> 294:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>> 295:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>> IPI0:          0  CPU wakeup interrupts
>> IPI1:          0  Timer broadcast interrupts
>> IPI2:          0  Rescheduling interrupts
>> IPI3:          0  Function call interrupts
>> IPI4:          0  CPU stop interrupts
>> IPI5:          0  IRQ work interrupts
>> IPI6:          0  completion interrupts
>> Err:          0
>>
>> root@k2g-evm:~# pcitest -m 1
>> [   57.791535] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>> [   57.798643] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>> MSI1:           OKAY
>> root@k2g-evm:~# pcitest -m 2
>> [   59.791509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 272
>> [   59.798614] keystone-pcie 21801000.pcie: irq: bit 0, vector 1, virq 281
>> MSI2:           OKAY
>> root@k2g-evm:~# pcitest -m 3
>> [   61.271509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 273
>> [   61.278616] keystone-pcie 21801000.pcie: irq: bit 0, vector 2, virq 282
>> MSI3:           OKAY
>> root@k2g-evm:~# pcitest -m 4
>> [   62.791514] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 274
>> [   62.798619] keystone-pcie 21801000.pcie: irq: bit 0, vector 3, virq 283
>> MSI4:           OKAY
>> root@k2g-evm:~# pcitest -m 5
>> [   64.311513] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 275
>> [   64.318620] keystone-pcie 21801000.pcie: irq: bit 0, vector 4, virq 284
>> MSI5:           OKAY
>> root@k2g-evm:~# pcitest -m 6
>> [   66.351518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 276
>> [   66.358624] keystone-pcie 21801000.pcie: irq: bit 0, vector 5, virq 285
>> MSI6:           OKAY
>> root@k2g-evm:~# pcitest -m 7
>> [   67.811516] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 277
>> [   67.818623] keystone-pcie 21801000.pcie: irq: bit 0, vector 6, virq 286
>> MSI7:           OKAY
>> root@k2g-evm:~# pcitest -m 8
>> [   69.291518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 278
>> [   69.298625] keystone-pcie 21801000.pcie: irq: bit 0, vector 7, virq 287
>> MSI8:           OKAY
>> root@k2g-evm:~# pcitest -m 9
>> MSI9:           NOT OKAY
>> root@k2g-evm:~# pcitest -m 10
>> MSI10:          NOT OKAY
>>
>> So once I disable pcieportbus, I could get the first 8 interrupts correctly.
>> But after that there are no interrupts at all.
> 
> This issues is due to the previous interrupt not acknowledged and below fixes
> the issue.

Besides this issue that you fixed with this patch, is everything working well in
your setup?

> 
> diff --git a/drivers/pci/dwc/pcie-designware-host.c
> b/drivers/pci/dwc/pcie-designware-host.c
> index 2816bd98f66c..b25e788e9828 100644
> --- a/drivers/pci/dwc/pcie-designware-host.c
> +++ b/drivers/pci/dwc/pcie-designware-host.c
> @@ -58,8 +58,20 @@ static void dw_msi_unmask_irq(struct irq_data *d)
>         irq_chip_unmask_parent(d);
>  }
> 
> +static void dw_pci_ack_irq(struct irq_data *d)
> +{
> +       struct msi_desc *msi = irq_data_get_msi_desc(d);
> +       struct pcie_port *pp;
> +
> +       pp = (struct pcie_port *) msi_desc_to_pci_sysdata(msi);
> +
> +       if (pp->ops->msi_irq_ack)
> +               pp->ops->msi_irq_ack(d->irq, pp);
> +}
> +
>  static struct irq_chip dw_pcie_msi_irq_chip = {
>         .name = "PCI-MSI",
> +       .irq_ack = dw_pci_ack_irq,
>         .irq_mask = dw_msi_mask_irq,
>         .irq_unmask = dw_msi_unmask_irq,
>  };
> @@ -193,20 +205,8 @@ static void dw_pci_bottom_unmask(struct irq_data *data)
>         spin_unlock_irqrestore(&pp->lock, flags);
>  }
> 
> -static void dw_pci_bottom_ack(struct irq_data *d)
> -{
> -       struct msi_desc *msi = irq_data_get_msi_desc(d);
> -       struct pcie_port *pp;
> -
> -       pp = (struct pcie_port *) msi_desc_to_pci_sysdata(msi);
> -
> -       if (pp->ops->msi_irq_ack)
> -               pp->ops->msi_irq_ack(d->irq, pp);
> -}
> -
>  static struct irq_chip dw_pci_msi_bottom_irq_chip = {
>         .name = "DWPCI-MSI",
> -       .irq_ack                = dw_pci_bottom_ack,
>         .irq_compose_msi_msg = dw_pci_setup_msi_msg,
>         .irq_set_affinity = dw_pci_msi_set_affinity,
>         .irq_mask = dw_pci_bottom_mask,
> @@ -240,7 +240,7 @@ static int dw_pcie_irq_domain_alloc(struct irq_domain *domain,
>         for (i = 0; i < nr_irqs; i++)
>                 irq_domain_set_info(domain, virq + i, bit + i,
>                                     &dw_pci_msi_bottom_irq_chip,
> -                                   domain->host_data, handle_simple_irq,
> +                                   domain->host_data, handle_level_irq,
>                                     NULL, NULL);
> 
>         return 0;
> 
> Thanks
> Kishon
> 

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi Gustavo,

On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>> Hi,
>>
>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>> Hi Gustavo,
>>>
>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>> version of the patches that are running on our equipment, please check with yours
>>>>> Hi,
>>>>>
>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>> Hi Kishon,
>>>>>>
>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>
>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>
>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>> Once again, thanks.
>>>
>>> Thanks.
>>>
>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>> are some of my observations after applying your patch series.
>>>
>>> root@k2g-evm:~# cat /proc/interrupts
>>>            CPU0
>>>  17:          0     GICv2  29 Level     arch_timer
>>>  18:       1816     GICv2  30 Level     arch_timer
>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>  22:        792     GICv2 196 Edge      ttyS0
>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>  47:       7551     GICv2 128 Edge      mmc0
>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>> IPI0:          0  CPU wakeup interrupts
>>> IPI1:          0  Timer broadcast interrupts
>>> IPI2:          0  Rescheduling interrupts
>>> IPI3:          0  Function call interrupts
>>> IPI4:          0  CPU stop interrupts
>>> IPI5:          0  IRQ work interrupts
>>> IPI6:          0  completion interrupts
>>> Err:          0
>>>
>>> root@k2g-evm:~# pcitest -m 1
>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>> MSI1:           NOT OKAY
>>>
>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>> interrupt in 280.
>>
>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>> separately.
> 
> Ok great!
> 
>>>
>>> So I tried disabling the pcieportbus driver. These are my observations once I
>>> disable pcieportbus driver
>>>
>>> root@k2g-evm:~# cat /proc/interrupts
>>>            CPU0
>>>  17:          0     GICv2  29 Level     arch_timer
>>>  18:       2222     GICv2  30 Level     arch_timer
>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>  22:        821     GICv2 196 Edge      ttyS0
>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>  25:        909     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>  44:       2442     GICv2 240 Edge      2728000.edma_ccint
>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>  47:       7297     GICv2 128 Edge      mmc0
>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>> 280:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>> 281:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>> 282:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>> 283:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>> 284:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>> 285:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>> 286:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>> 287:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>> 288:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>> 289:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>> 290:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>> 291:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>> 292:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>> 293:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>> 294:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>> 295:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>> IPI0:          0  CPU wakeup interrupts
>>> IPI1:          0  Timer broadcast interrupts
>>> IPI2:          0  Rescheduling interrupts
>>> IPI3:          0  Function call interrupts
>>> IPI4:          0  CPU stop interrupts
>>> IPI5:          0  IRQ work interrupts
>>> IPI6:          0  completion interrupts
>>> Err:          0
>>>
>>> root@k2g-evm:~# pcitest -m 1
>>> [   57.791535] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>> [   57.798643] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>> MSI1:           OKAY
>>> root@k2g-evm:~# pcitest -m 2
>>> [   59.791509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 272
>>> [   59.798614] keystone-pcie 21801000.pcie: irq: bit 0, vector 1, virq 281
>>> MSI2:           OKAY
>>> root@k2g-evm:~# pcitest -m 3
>>> [   61.271509] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 273
>>> [   61.278616] keystone-pcie 21801000.pcie: irq: bit 0, vector 2, virq 282
>>> MSI3:           OKAY
>>> root@k2g-evm:~# pcitest -m 4
>>> [   62.791514] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 274
>>> [   62.798619] keystone-pcie 21801000.pcie: irq: bit 0, vector 3, virq 283
>>> MSI4:           OKAY
>>> root@k2g-evm:~# pcitest -m 5
>>> [   64.311513] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 275
>>> [   64.318620] keystone-pcie 21801000.pcie: irq: bit 0, vector 4, virq 284
>>> MSI5:           OKAY
>>> root@k2g-evm:~# pcitest -m 6
>>> [   66.351518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 276
>>> [   66.358624] keystone-pcie 21801000.pcie: irq: bit 0, vector 5, virq 285
>>> MSI6:           OKAY
>>> root@k2g-evm:~# pcitest -m 7
>>> [   67.811516] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 277
>>> [   67.818623] keystone-pcie 21801000.pcie: irq: bit 0, vector 6, virq 286
>>> MSI7:           OKAY
>>> root@k2g-evm:~# pcitest -m 8
>>> [   69.291518] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 278
>>> [   69.298625] keystone-pcie 21801000.pcie: irq: bit 0, vector 7, virq 287
>>> MSI8:           OKAY
>>> root@k2g-evm:~# pcitest -m 9
>>> MSI9:           NOT OKAY
>>> root@k2g-evm:~# pcitest -m 10
>>> MSI10:          NOT OKAY
>>>
>>> So once I disable pcieportbus, I could get the first 8 interrupts correctly.
>>> But after that there are no interrupts at all.
>>
>> This issues is due to the previous interrupt not acknowledged and below fixes
>> the issue.
> 
> Besides this issue that you fixed with this patch, is everything working well in
> your setup?

yeah, all the basic stuff seems to be working.

I also added a minor modification below to get rid of a warning [1] with
dra7xx. please see if you can include this in your patch

diff --git a/drivers/pci/dwc/pcie-designware-host.c
b/drivers/pci/dwc/pcie-designware-host.c
index 2816bd98f66c..d46a2205043e 100644
--- a/drivers/pci/dwc/pcie-designware-host.c
+++ b/drivers/pci/dwc/pcie-designware-host.c
@@ -445,9 +445,10 @@ int dw_pcie_host_init(struct pcie_port *pp)
                        if (ret)
                                goto error;

-                       irq_set_chained_handler_and_data(pci->pp.msi_irq,
-                                                        dw_chained_msi_isr,
-                                                        pci);
+                       if (pp->msi_irq)
+                               irq_set_chained_handler_and_data(pp->msi_irq,
+
dw_chained_msi_isr,
+                                                                pci);
                } else {
                        ret = pp->ops->msi_host_init(pci);
                        if (ret < 0)

Thanks
Kishon

[1] -> https://pastebin.ubuntu.com/26270254/
> 
>>
>> diff --git a/drivers/pci/dwc/pcie-designware-host.c
>> b/drivers/pci/dwc/pcie-designware-host.c
>> index 2816bd98f66c..b25e788e9828 100644
>> --- a/drivers/pci/dwc/pcie-designware-host.c
>> +++ b/drivers/pci/dwc/pcie-designware-host.c
>> @@ -58,8 +58,20 @@ static void dw_msi_unmask_irq(struct irq_data *d)
>>         irq_chip_unmask_parent(d);
>>  }
>>
>> +static void dw_pci_ack_irq(struct irq_data *d)
>> +{
>> +       struct msi_desc *msi = irq_data_get_msi_desc(d);
>> +       struct pcie_port *pp;
>> +
>> +       pp = (struct pcie_port *) msi_desc_to_pci_sysdata(msi);
>> +
>> +       if (pp->ops->msi_irq_ack)
>> +               pp->ops->msi_irq_ack(d->irq, pp);
>> +}
>> +
>>  static struct irq_chip dw_pcie_msi_irq_chip = {
>>         .name = "PCI-MSI",
>> +       .irq_ack = dw_pci_ack_irq,
>>         .irq_mask = dw_msi_mask_irq,
>>         .irq_unmask = dw_msi_unmask_irq,
>>  };
>> @@ -193,20 +205,8 @@ static void dw_pci_bottom_unmask(struct irq_data *data)
>>         spin_unlock_irqrestore(&pp->lock, flags);
>>  }
>>
>> -static void dw_pci_bottom_ack(struct irq_data *d)
>> -{
>> -       struct msi_desc *msi = irq_data_get_msi_desc(d);
>> -       struct pcie_port *pp;
>> -
>> -       pp = (struct pcie_port *) msi_desc_to_pci_sysdata(msi);
>> -
>> -       if (pp->ops->msi_irq_ack)
>> -               pp->ops->msi_irq_ack(d->irq, pp);
>> -}
>> -
>>  static struct irq_chip dw_pci_msi_bottom_irq_chip = {
>>         .name = "DWPCI-MSI",
>> -       .irq_ack                = dw_pci_bottom_ack,
>>         .irq_compose_msi_msg = dw_pci_setup_msi_msg,
>>         .irq_set_affinity = dw_pci_msi_set_affinity,
>>         .irq_mask = dw_pci_bottom_mask,
>> @@ -240,7 +240,7 @@ static int dw_pcie_irq_domain_alloc(struct irq_domain *domain,
>>         for (i = 0; i < nr_irqs; i++)
>>                 irq_domain_set_info(domain, virq + i, bit + i,
>>                                     &dw_pci_msi_bottom_irq_chip,
>> -                                   domain->host_data, handle_simple_irq,
>> +                                   domain->host_data, handle_level_irq,
>>                                     NULL, NULL);
>>
>>         return 0;
>>
>> Thanks
>> Kishon
>>
> 
> 

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi Gustavo,

On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>> Hi,
>>
>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>> Hi Gustavo,
>>>
>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>> version of the patches that are running on our equipment, please check with yours
>>>>> Hi,
>>>>>
>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>> Hi Kishon,
>>>>>>
>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>
>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>
>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>> Once again, thanks.
>>>
>>> Thanks.
>>>
>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>> are some of my observations after applying your patch series.
>>>
>>> root@k2g-evm:~# cat /proc/interrupts
>>>            CPU0
>>>  17:          0     GICv2  29 Level     arch_timer
>>>  18:       1816     GICv2  30 Level     arch_timer
>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>  22:        792     GICv2 196 Edge      ttyS0
>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>  47:       7551     GICv2 128 Edge      mmc0
>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>> IPI0:          0  CPU wakeup interrupts
>>> IPI1:          0  Timer broadcast interrupts
>>> IPI2:          0  Rescheduling interrupts
>>> IPI3:          0  Function call interrupts
>>> IPI4:          0  CPU stop interrupts
>>> IPI5:          0  IRQ work interrupts
>>> IPI6:          0  completion interrupts
>>> Err:          0
>>>
>>> root@k2g-evm:~# pcitest -m 1
>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>> MSI1:           NOT OKAY
>>>
>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>> interrupt in 280.
>>
>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>> separately.
> 
> Ok great!

This too looks like a bug with the new API. If the EP requests 16 interrupts,
the MSI vector (programmed in PCI_MSI_DATA) should be 0 and if 0 is not
available it should be 16
[From the spec: The Multiple Message Enable field (bits 6-4 of the Message
Control register) defines the number of low
order message data bits the function is permitted to modify to generate its
system software allocated vectors]. For 16 interrupts, Multiple Message Enable
will have a value of 4 which means the EP can modify the 4 low order bits to
raise the 16 interrupts.

With this series added, PCI_MSI_DATA is programmed to 0x1 which gets masked
while the EP tries to raise an interrupt and hence we are observing off by one
error.

Thanks
Kishon

Re: Re-activate task to add MSI-X to pcie-designware

[Gustavo Pimentel]

Hi Kishon,

On 28/12/2017 14:58, Kishon Vijay Abraham I wrote:
> Hi Gustavo,
> 
> On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
>> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>>> Hi,
>>>
>>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>>> Hi Gustavo,
>>>>
>>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>>> version of the patches that are running on our equipment, please check with yours
>>>>>> Hi,
>>>>>>
>>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>>> Hi Kishon,
>>>>>>>
>>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>>
>>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>>
>>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>>> Once again, thanks.
>>>>
>>>> Thanks.
>>>>
>>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>>> are some of my observations after applying your patch series.
>>>>
>>>> root@k2g-evm:~# cat /proc/interrupts
>>>>            CPU0
>>>>  17:          0     GICv2  29 Level     arch_timer
>>>>  18:       1816     GICv2  30 Level     arch_timer
>>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>>  22:        792     GICv2 196 Edge      ttyS0
>>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>>  47:       7551     GICv2 128 Edge      mmc0
>>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>>> IPI0:          0  CPU wakeup interrupts
>>>> IPI1:          0  Timer broadcast interrupts
>>>> IPI2:          0  Rescheduling interrupts
>>>> IPI3:          0  Function call interrupts
>>>> IPI4:          0  CPU stop interrupts
>>>> IPI5:          0  IRQ work interrupts
>>>> IPI6:          0  completion interrupts
>>>> Err:          0
>>>>
>>>> root@k2g-evm:~# pcitest -m 1
>>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>>> MSI1:           NOT OKAY
>>>>
>>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>>> interrupt in 280.
>>>
>>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>>> separately.
>>
>> Ok great!
> 
> This too looks like a bug with the new API. If the EP requests 16 interrupts,
> the MSI vector (programmed in PCI_MSI_DATA) should be 0 and if 0 is not
> available it should be 16
> [From the spec: The Multiple Message Enable field (bits 6-4 of the Message
> Control register) defines the number of low
> order message data bits the function is permitted to modify to generate its
> system software allocated vectors]. For 16 interrupts, Multiple Message Enable
> will have a value of 4 which means the EP can modify the 4 low order bits to
> raise the 16 interrupts.
> 
> With this series added, PCI_MSI_DATA is programmed to 0x1 which gets masked
> while the EP tries to raise an interrupt and hence we are observing off by one
> error.

The EP (USB 3.1 PCIe board) that I usually use only have 2 interrupts. I didn't
perform the tests like you did. I typically plug the USB flash drive and observe
the number of interrupts and perform some basic file operations on the USB flash
drive (copy/move/delete/rename).

Can you provide me information (source code, how to compile, how to use, that
kind of info) about the pcitest tool that you mentioned? I would like to use it
if possible, please.

Regards,
Gustavo

> 
> Thanks
> Kishon
> 

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi Gustavo,

On Friday 29 December 2017 03:18 PM, Gustavo Pimentel wrote:
> Hi Kishon,
> 
> On 28/12/2017 14:58, Kishon Vijay Abraham I wrote:
>> Hi Gustavo,
>>
>> On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
>>> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>>>> Hi,
>>>>
>>>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>>>> Hi Gustavo,
>>>>>
>>>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>>>> version of the patches that are running on our equipment, please check with yours
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>>>> Hi Kishon,
>>>>>>>>
>>>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>>>
>>>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>>>
>>>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>>>> Once again, thanks.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>>>> are some of my observations after applying your patch series.
>>>>>
>>>>> root@k2g-evm:~# cat /proc/interrupts
>>>>>            CPU0
>>>>>  17:          0     GICv2  29 Level     arch_timer
>>>>>  18:       1816     GICv2  30 Level     arch_timer
>>>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>>>  22:        792     GICv2 196 Edge      ttyS0
>>>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>>>  47:       7551     GICv2 128 Edge      mmc0
>>>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>>>> IPI0:          0  CPU wakeup interrupts
>>>>> IPI1:          0  Timer broadcast interrupts
>>>>> IPI2:          0  Rescheduling interrupts
>>>>> IPI3:          0  Function call interrupts
>>>>> IPI4:          0  CPU stop interrupts
>>>>> IPI5:          0  IRQ work interrupts
>>>>> IPI6:          0  completion interrupts
>>>>> Err:          0
>>>>>
>>>>> root@k2g-evm:~# pcitest -m 1
>>>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>>>> MSI1:           NOT OKAY
>>>>>
>>>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>>>> interrupt in 280.
>>>>
>>>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>>>> separately.
>>>
>>> Ok great!
>>
>> This too looks like a bug with the new API. If the EP requests 16 interrupts,
>> the MSI vector (programmed in PCI_MSI_DATA) should be 0 and if 0 is not
>> available it should be 16
>> [From the spec: The Multiple Message Enable field (bits 6-4 of the Message
>> Control register) defines the number of low
>> order message data bits the function is permitted to modify to generate its
>> system software allocated vectors]. For 16 interrupts, Multiple Message Enable
>> will have a value of 4 which means the EP can modify the 4 low order bits to
>> raise the 16 interrupts.
>>
>> With this series added, PCI_MSI_DATA is programmed to 0x1 which gets masked
>> while the EP tries to raise an interrupt and hence we are observing off by one
>> error.
> 
> The EP (USB 3.1 PCIe board) that I usually use only have 2 interrupts. I didn't
> perform the tests like you did. I typically plug the USB flash drive and observe
> the number of interrupts and perform some basic file operations on the USB flash
> drive (copy/move/delete/rename).

Do you have PCIEPORTBUS enabled? That takes 1 interrupt (assuming your host has
the required capabilities). You should be able to reproduce the issue with just
PCIEPORTBUS enabled then. (With your series added msg_data on USB device will
be 0x1 whereas it should be 0x2.

Before the patch series order_base_2 below helped in taking care of that.
pos0 = bitmap_find_free_region(pp->msi_irq_in_use, MAX_MSI_IRQS,
order_base_2(no_irqs));
> 
> Can you provide me information (source code, how to compile, how to use, that
> kind of info) about the pcitest tool that you mentioned? I would like to use it
> if possible, please.

Documentation/PCI/endpoint/ should have all the info..

Thanks
Kishon

Re: Re-activate task to add MSI-X to pcie-designware

[Gustavo Pimentel]

On 29/12/2017 10:05, Kishon Vijay Abraham I wrote:
> Hi Gustavo,
> 
> On Friday 29 December 2017 03:18 PM, Gustavo Pimentel wrote:
>> Hi Kishon,
>>
>> On 28/12/2017 14:58, Kishon Vijay Abraham I wrote:
>>> Hi Gustavo,
>>>
>>> On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
>>>> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>>>>> Hi,
>>>>>
>>>>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>>>>> Hi Gustavo,
>>>>>>
>>>>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>>>>> version of the patches that are running on our equipment, please check with yours
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>>>>> Hi Kishon,
>>>>>>>>>
>>>>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>>>>
>>>>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>>>>
>>>>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>>>>> Once again, thanks.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>>>>> are some of my observations after applying your patch series.
>>>>>>
>>>>>> root@k2g-evm:~# cat /proc/interrupts
>>>>>>            CPU0
>>>>>>  17:          0     GICv2  29 Level     arch_timer
>>>>>>  18:       1816     GICv2  30 Level     arch_timer
>>>>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>>>>  22:        792     GICv2 196 Edge      ttyS0
>>>>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>>>>  47:       7551     GICv2 128 Edge      mmc0
>>>>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>>>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>>>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>>>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>>>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>>>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>>>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>>>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>>>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>>>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>>>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>>>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>>>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>>>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>>>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>>>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>>>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>>>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>>>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>>>>> IPI0:          0  CPU wakeup interrupts
>>>>>> IPI1:          0  Timer broadcast interrupts
>>>>>> IPI2:          0  Rescheduling interrupts
>>>>>> IPI3:          0  Function call interrupts
>>>>>> IPI4:          0  CPU stop interrupts
>>>>>> IPI5:          0  IRQ work interrupts
>>>>>> IPI6:          0  completion interrupts
>>>>>> Err:          0
>>>>>>
>>>>>> root@k2g-evm:~# pcitest -m 1
>>>>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>>>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>>>>> MSI1:           NOT OKAY
>>>>>>
>>>>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>>>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>>>>> interrupt in 280.
>>>>>
>>>>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>>>>> separately.
>>>>
>>>> Ok great!
>>>
>>> This too looks like a bug with the new API. If the EP requests 16 interrupts,
>>> the MSI vector (programmed in PCI_MSI_DATA) should be 0 and if 0 is not
>>> available it should be 16
>>> [From the spec: The Multiple Message Enable field (bits 6-4 of the Message
>>> Control register) defines the number of low
>>> order message data bits the function is permitted to modify to generate its
>>> system software allocated vectors]. For 16 interrupts, Multiple Message Enable
>>> will have a value of 4 which means the EP can modify the 4 low order bits to
>>> raise the 16 interrupts.

Ok, I will try to replicate your test environment to fix this error, meanwhile I
will send a patch v4 to everyone in order to keep this alive and receive more
comments from everyone.

>>>
>>> With this series added, PCI_MSI_DATA is programmed to 0x1 which gets masked
>>> while the EP tries to raise an interrupt and hence we are observing off by one
>>> error.
>>
>> The EP (USB 3.1 PCIe board) that I usually use only have 2 interrupts. I didn't
>> perform the tests like you did. I typically plug the USB flash drive and observe
>> the number of interrupts and perform some basic file operations on the USB flash
>> drive (copy/move/delete/rename).
> 
> Do you have PCIEPORTBUS enabled? That takes 1 interrupt (assuming your host has
> the required capabilities). You should be able to reproduce the issue with just
> PCIEPORTBUS enabled then. (With your series added msg_data on USB device will
> be 0x1 whereas it should be 0x2.

I have to check with HW team maybe. I think I don't have access to that info
through lspci.

> 
> Before the patch series order_base_2 below helped in taking care of that.
> pos0 = bitmap_find_free_region(pp->msi_irq_in_use, MAX_MSI_IRQS,
> order_base_2(no_irqs));

I didn't understand this. Sorry. Can you explain me your idea?

>>
>> Can you provide me information (source code, how to compile, how to use, that
>> kind of info) about the pcitest tool that you mentioned? I would like to use it
>> if possible, please.
> 
> Documentation/PCI/endpoint/ should have all the info..

I have enable this settings in the kernel:
PCI_EPF_TEST
PCI_ENDPOINT_TEST
PCI_ENDPOINT_CONFIGFS

and also compile the pcitest.c file and generate a uImage for my ARC setup.

# ls /sys/class/pci_epc/
#

# ls /sys/kernel/config/
#

# ls /sys/bus/pci-epf/drivers
pci_epf_test


Probably it's missing a very basic thing, but i don't see what. Do you have any
idea?

> 
> Thanks
> Kishon
> 

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

Hi,

On Wednesday 03 January 2018 12:51 AM, Gustavo Pimentel wrote:
> On 29/12/2017 10:05, Kishon Vijay Abraham I wrote:
>> Hi Gustavo,
>>
>> On Friday 29 December 2017 03:18 PM, Gustavo Pimentel wrote:
>>> Hi Kishon,
>>>
>>> On 28/12/2017 14:58, Kishon Vijay Abraham I wrote:
>>>> Hi Gustavo,
>>>>
>>>> On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
>>>>> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>>>>>> Hi Gustavo,
>>>>>>>
>>>>>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>>>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>>>>>> version of the patches that are running on our equipment, please check with yours
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>>>>>> Hi Kishon,
>>>>>>>>>>
>>>>>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>>>>>
>>>>>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>>>>>
>>>>>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>>>>>> Once again, thanks.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>>>>>> are some of my observations after applying your patch series.
>>>>>>>
>>>>>>> root@k2g-evm:~# cat /proc/interrupts
>>>>>>>            CPU0
>>>>>>>  17:          0     GICv2  29 Level     arch_timer
>>>>>>>  18:       1816     GICv2  30 Level     arch_timer
>>>>>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>>>>>  22:        792     GICv2 196 Edge      ttyS0
>>>>>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>>>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>>>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>>>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>>>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>>>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>>>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>>>>>  47:       7551     GICv2 128 Edge      mmc0
>>>>>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>>>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>>>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>>>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>>>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>>>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>>>>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>>>>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>>>>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>>>>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>>>>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>>>>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>>>>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>>>>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>>>>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>>>>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>>>>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>>>>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>>>>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>>>>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>>>>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>>>>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>>>>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>>>>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>>>>>> IPI0:          0  CPU wakeup interrupts
>>>>>>> IPI1:          0  Timer broadcast interrupts
>>>>>>> IPI2:          0  Rescheduling interrupts
>>>>>>> IPI3:          0  Function call interrupts
>>>>>>> IPI4:          0  CPU stop interrupts
>>>>>>> IPI5:          0  IRQ work interrupts
>>>>>>> IPI6:          0  completion interrupts
>>>>>>> Err:          0
>>>>>>>
>>>>>>> root@k2g-evm:~# pcitest -m 1
>>>>>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>>>>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>>>>>> MSI1:           NOT OKAY
>>>>>>>
>>>>>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>>>>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>>>>>> interrupt in 280.
>>>>>>
>>>>>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>>>>>> separately.
>>>>>
>>>>> Ok great!
>>>>
>>>> This too looks like a bug with the new API. If the EP requests 16 interrupts,
>>>> the MSI vector (programmed in PCI_MSI_DATA) should be 0 and if 0 is not
>>>> available it should be 16
>>>> [From the spec: The Multiple Message Enable field (bits 6-4 of the Message
>>>> Control register) defines the number of low
>>>> order message data bits the function is permitted to modify to generate its
>>>> system software allocated vectors]. For 16 interrupts, Multiple Message Enable
>>>> will have a value of 4 which means the EP can modify the 4 low order bits to
>>>> raise the 16 interrupts.
> 
> Ok, I will try to replicate your test environment to fix this error, meanwhile I
> will send a patch v4 to everyone in order to keep this alive and receive more
> comments from everyone.
> 
>>>>
>>>> With this series added, PCI_MSI_DATA is programmed to 0x1 which gets masked
>>>> while the EP tries to raise an interrupt and hence we are observing off by one
>>>> error.
>>>
>>> The EP (USB 3.1 PCIe board) that I usually use only have 2 interrupts. I didn't
>>> perform the tests like you did. I typically plug the USB flash drive and observe
>>> the number of interrupts and perform some basic file operations on the USB flash
>>> drive (copy/move/delete/rename).
>>
>> Do you have PCIEPORTBUS enabled? That takes 1 interrupt (assuming your host has
>> the required capabilities). You should be able to reproduce the issue with just
>> PCIEPORTBUS enabled then. (With your series added msg_data on USB device will
>> be 0x1 whereas it should be 0x2.
> 
> I have to check with HW team maybe. I think I don't have access to that info
> through lspci.
> 
>>
>> Before the patch series order_base_2 below helped in taking care of that.
>> pos0 = bitmap_find_free_region(pp->msi_irq_in_use, MAX_MSI_IRQS,
>> order_base_2(no_irqs));
> 
> I didn't understand this. Sorry. Can you explain me your idea?

Never mind.
What I'm suggesting is msi_data (programmed in PCI_MSI_DATA) cannot be linear
across devices.

For example say you have 2 devices connected to the host, one requesting '2'
MSI interrupts and the other requesting '8' (MMC has a value of 3) msi interrupts.

The msi_data programmed in these devices right now (i.e after applying your
patches) are 0x0 and 0x2 which is not correct.

[From the spec: The Multiple Message Enable field (bits 6-4 of the Message
Control register) defines the number of low
order message data bits the function is permitted to modify to generate its
system software allocated vectors].

This means a device requesting 8 MSI interrupts can modify the lower 3 bits to
raise an interrupt. So if we program 0x2 in msi_data (starting msi vector
data), the device might not write a 0x2 to raise an interrupt. So msi_data can
be either 0x0 and if 0x0 is not available (since it's been allocated to some
other device) then msi_data should be 0x8.

Before your patch series, this issue was not present.
> 
>>>
>>> Can you provide me information (source code, how to compile, how to use, that
>>> kind of info) about the pcitest tool that you mentioned? I would like to use it
>>> if possible, please.
>>
>> Documentation/PCI/endpoint/ should have all the info..
> 
> I have enable this settings in the kernel:
> PCI_EPF_TEST
> PCI_ENDPOINT_TEST
> PCI_ENDPOINT_CONFIGFS
> 
> and also compile the pcitest.c file and generate a uImage for my ARC setup.
> 
> # ls /sys/class/pci_epc/
> #

you need to add endpoint controller support for your platform. You can either
refer pci-dra7xx.c or Niklas series adding EP mode support for ARTPEC-6 [1]

[1] -> https://lkml.org/lkml/2017/11/3/321

Thanks
Kishon

Re: Re-activate task to add MSI-X to pcie-designware

[Gustavo Pimentel]

On 29/12/2017 10:05, Kishon Vijay Abraham I wrote:
> Hi Gustavo,
> 
> On Friday 29 December 2017 03:18 PM, Gustavo Pimentel wrote:
>> Hi Kishon,
>>
>> On 28/12/2017 14:58, Kishon Vijay Abraham I wrote:
>>> Hi Gustavo,
>>>
>>> On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
>>>> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>>>>> Hi,
>>>>>
>>>>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>>>>> Hi Gustavo,
>>>>>>
>>>>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>>>>> version of the patches that are running on our equipment, please check with yours
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>>>>> Hi Kishon,
>>>>>>>>>
>>>>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>>>>
>>>>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>>>>
>>>>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>>>>> Once again, thanks.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>>>>> are some of my observations after applying your patch series.
>>>>>>
>>>>>> root@k2g-evm:~# cat /proc/interrupts
>>>>>>            CPU0
>>>>>>  17:          0     GICv2  29 Level     arch_timer
>>>>>>  18:       1816     GICv2  30 Level     arch_timer
>>>>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>>>>  22:        792     GICv2 196 Edge      ttyS0
>>>>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>>>>  47:       7551     GICv2 128 Edge      mmc0
>>>>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>>>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>>>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>>>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>>>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>>>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>>>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>>>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>>>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>>>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>>>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>>>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>>>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>>>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>>>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>>>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>>>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>>>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>>>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>>>>> IPI0:          0  CPU wakeup interrupts
>>>>>> IPI1:          0  Timer broadcast interrupts
>>>>>> IPI2:          0  Rescheduling interrupts
>>>>>> IPI3:          0  Function call interrupts
>>>>>> IPI4:          0  CPU stop interrupts
>>>>>> IPI5:          0  IRQ work interrupts
>>>>>> IPI6:          0  completion interrupts
>>>>>> Err:          0
>>>>>>
>>>>>> root@k2g-evm:~# pcitest -m 1
>>>>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>>>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>>>>> MSI1:           NOT OKAY
>>>>>>
>>>>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>>>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>>>>> interrupt in 280.
>>>>>
>>>>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>>>>> separately.
>>>>
>>>> Ok great!
>>>
>>> This too looks like a bug with the new API. If the EP requests 16 interrupts,
>>> the MSI vector (programmed in PCI_MSI_DATA) should be 0 and if 0 is not
>>> available it should be 16
>>> [From the spec: The Multiple Message Enable field (bits 6-4 of the Message
>>> Control register) defines the number of low
>>> order message data bits the function is permitted to modify to generate its
>>> system software allocated vectors]. For 16 interrupts, Multiple Message Enable
>>> will have a value of 4 which means the EP can modify the 4 low order bits to
>>> raise the 16 interrupts.
>>>
>>> With this series added, PCI_MSI_DATA is programmed to 0x1 which gets masked
>>> while the EP tries to raise an interrupt and hence we are observing off by one
>>> error.

So the problem that you saying could be on setting the correct value on the
PCI_MSI_DATA, right?
While I'm trying to support pcitest to be able in future to run the same tests
that you on my system, I have added some debug on msi.c file that could provide
me the same info that you have refer before in a short time.
With the same USB 3.1 PCIe board as a EP, I have generated 2 linux images (with
and without the patches series).

Without the patches
DEBUG:drivers/pci/msi.c:pci_msi_vec_count:881 msgctl = 0x86
DEBUG:drivers/pci/msi.c:msi_setup_entry:554 control = 0x86
DEBUG:drivers/pci/msi.c:msi_setup_entry:555 nvec = 0x1
DEBUG:drivers/pci/msi.c:msi_setup_entry:563 entry->msi_attrib.multi_cap = 0x3
DEBUG:drivers/pci/msi.c:msi_setup_entry:564 entry->msi_attrib.multiple = 0x0
DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:314 msgctl = 0x86
DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:317 entry->msi_attrib.multiple =
0x0, msgctl = 0x86

With the patches (I have commented the MSI_FLAG_PCI_MSIX flag)
DEBUG:drivers/pci/msi.c:pci_msi_vec_count:882 msgctl = 0x86
DEBUG:drivers/pci/msi.c:msi_setup_entry:555 control = 0x86
DEBUG:drivers/pci/msi.c:msi_setup_entry:556 nvec = 0x1
DEBUG:drivers/pci/msi.c:msi_setup_entry:564 entry->msi_attrib.multi_cap = 0x3
DEBUG:drivers/pci/msi.c:msi_setup_entry:565 entry->msi_attrib.multiple = 0x0
DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:315 msgctl = 0x86
DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:318 entry->msi_attrib.multiple =
0x0, msgctl = 0x86

The defined values before the patch are the same with the patch series. Maybe
this only happens with a bigger number of interrupts required. I will continue
to work on pcitest to perform the same test that you have done.

>>
>> The EP (USB 3.1 PCIe board) that I usually use only have 2 interrupts. I didn't
>> perform the tests like you did. I typically plug the USB flash drive and observe
>> the number of interrupts and perform some basic file operations on the USB flash
>> drive (copy/move/delete/rename).
> 
> Do you have PCIEPORTBUS enabled? That takes 1 interrupt (assuming your host has
> the required capabilities). You should be able to reproduce the issue with just
> PCIEPORTBUS enabled then. (With your series added msg_data on USB device will
> be 0x1 whereas it should be 0x2.

I checked now, I have PCIEPORTBUS enabled.

> 
> Before the patch series order_base_2 below helped in taking care of that.
> pos0 = bitmap_find_free_region(pp->msi_irq_in_use, MAX_MSI_IRQS,
> order_base_2(no_irqs));
>>
>> Can you provide me information (source code, how to compile, how to use, that
>> kind of info) about the pcitest tool that you mentioned? I would like to use it
>> if possible, please.
> 
> Documentation/PCI/endpoint/ should have all the info..

Ok. After this patch series, maybe we could improve the tool and add the MSI-X
support also. :)

> 
> Thanks
> Kishon
> 

Re: Re-activate task to add MSI-X to pcie-designware

[Kishon Vijay Abraham I]

On Tuesday 09 January 2018 04:02 PM, Gustavo Pimentel wrote:
> On 29/12/2017 10:05, Kishon Vijay Abraham I wrote:
>> Hi Gustavo,
>>
>> On Friday 29 December 2017 03:18 PM, Gustavo Pimentel wrote:
>>> Hi Kishon,
>>>
>>> On 28/12/2017 14:58, Kishon Vijay Abraham I wrote:
>>>> Hi Gustavo,
>>>>
>>>> On Wednesday 27 December 2017 07:55 PM, Gustavo Pimentel wrote:
>>>>> On 26/12/2017 12:57, Kishon Vijay Abraham I wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Saturday 23 December 2017 03:46 PM, Kishon Vijay Abraham I wrote:
>>>>>>> Hi Gustavo,
>>>>>>>
>>>>>>> On Thursday 21 December 2017 07:38 PM, Gustavo Pimentel wrote:
>>>>>>>> On 20/12/2017 13:03, Kishon Vijay Abraham I wrote:This is our most recent
>>>>>>>> version of the patches that are running on our equipment, please check with yours
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Monday 18 December 2017 09:31 PM, Gustavo Pimentel wrote:
>>>>>>>>>> Hi Kishon,
>>>>>>>>>>
>>>>>>>>>> I would like to collaborate with you on this subject, I have on my side João's
>>>>>>>>>> patches updated to the Bjorn's latest kernel version.
>>>>>>>>>
>>>>>>>>> cool, do you have a branch so that I can check what breaks in keystone and debug?
>>>>>>>>>
>>>>>>>> Unfortunately not, however I'll mailed you the patches immediately.
>>>>>>>> Once again, thanks.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> I tried raising MSI with pci_endpoint_test EP device (16 MSI interrupts). Here
>>>>>>> are some of my observations after applying your patch series.
>>>>>>>
>>>>>>> root@k2g-evm:~# cat /proc/interrupts
>>>>>>>            CPU0
>>>>>>>  17:          0     GICv2  29 Level     arch_timer
>>>>>>>  18:       1816     GICv2  30 Level     arch_timer
>>>>>>>  21:          0     GICv2  36 Edge      arm-pmu
>>>>>>>  22:        792     GICv2 196 Edge      ttyS0
>>>>>>>  23:          5     GICv2 120 Edge      2530000.i2c
>>>>>>>  24:          0     GICv2  33 Edge      soc:keystone_irq@26202a0
>>>>>>>  25:        901     GICv2 356 Level     2a00000.msgmgr rx_005_002
>>>>>>>  41:          0     GICv2 232 Edge      2700000.edma_ccint
>>>>>>>  43:          0     GICv2 249 Edge      2700000.edma_ccerrint
>>>>>>>  44:       2497     GICv2 240 Edge      2728000.edma_ccint
>>>>>>>  46:          0     GICv2 252 Edge      2728000.edma_ccerrint
>>>>>>>  47:       7551     GICv2 128 Edge      mmc0
>>>>>>>  48:          0     GICv2 160 Edge      2680000.keystone-dwc3, xhci-hcd:usb1
>>>>>>>  49:          0     GICv2 176 Edge      2580000.keystone-dwc3, xhci-hcd:usb3
>>>>>>>  50:          0     GICv2  96 Edge      21805400.spi
>>>>>>>  52:          0     GICv2 100 Edge      21805c00.spi
>>>>>>>  53:          0     GICv2 102 Edge      21806000.spi
>>>>>>>  54:          0     GICv2  92 Edge      pcie-error-irq
>>>>>>> 211:          0      GPIO  12 Edge    -davinci_gpio  23000000.mmc cd
>>>>>>> 280:          0   PCI-MSI   0 Edge      PCIe PME, aerdrv
>>>>>>> 281:          0   PCI-MSI 524288 Edge      pci-endpoint-test
>>>>>>> 282:          0   PCI-MSI 524289 Edge      pci-endpoint-test
>>>>>>> 283:          0   PCI-MSI 524290 Edge      pci-endpoint-test
>>>>>>> 284:          0   PCI-MSI 524291 Edge      pci-endpoint-test
>>>>>>> 285:          0   PCI-MSI 524292 Edge      pci-endpoint-test
>>>>>>> 286:          0   PCI-MSI 524293 Edge      pci-endpoint-test
>>>>>>> 287:          0   PCI-MSI 524294 Edge      pci-endpoint-test
>>>>>>> 288:          0   PCI-MSI 524295 Edge      pci-endpoint-test
>>>>>>> 289:          0   PCI-MSI 524296 Edge      pci-endpoint-test
>>>>>>> 290:          0   PCI-MSI 524297 Edge      pci-endpoint-test
>>>>>>> 291:          0   PCI-MSI 524298 Edge      pci-endpoint-test
>>>>>>> 292:          0   PCI-MSI 524299 Edge      pci-endpoint-test
>>>>>>> 293:          0   PCI-MSI 524300 Edge      pci-endpoint-test
>>>>>>> 294:          0   PCI-MSI 524301 Edge      pci-endpoint-test
>>>>>>> 295:          0   PCI-MSI 524302 Edge      pci-endpoint-test
>>>>>>> 296:          0   PCI-MSI 524303 Edge      pci-endpoint-test
>>>>>>> IPI0:          0  CPU wakeup interrupts
>>>>>>> IPI1:          0  Timer broadcast interrupts
>>>>>>> IPI2:          0  Rescheduling interrupts
>>>>>>> IPI3:          0  Function call interrupts
>>>>>>> IPI4:          0  CPU stop interrupts
>>>>>>> IPI5:          0  IRQ work interrupts
>>>>>>> IPI6:          0  completion interrupts
>>>>>>> Err:          0
>>>>>>>
>>>>>>> root@k2g-evm:~# pcitest -m 1
>>>>>>> [   45.966437] keystone-pcie 21801000.pcie: ks_pcie_msi_irq_handler, irq 271
>>>>>>> [   45.973544] keystone-pcie 21801000.pcie: irq: bit 0, vector 0, virq 280
>>>>>>> MSI1:           NOT OKAY
>>>>>>>
>>>>>>> Here there is an off by one error. "pcitest -m 1" should ideally have raised an
>>>>>>> interrupt in 281 but from the above debug print it looks like it's raising an
>>>>>>> interrupt in 280.
>>>>>>
>>>>>> The above error was due to an pci_endpoint bug. I'll send a patch to fix this
>>>>>> separately.
>>>>>
>>>>> Ok great!
>>>>
>>>> This too looks like a bug with the new API. If the EP requests 16 interrupts,
>>>> the MSI vector (programmed in PCI_MSI_DATA) should be 0 and if 0 is not
>>>> available it should be 16
>>>> [From the spec: The Multiple Message Enable field (bits 6-4 of the Message
>>>> Control register) defines the number of low
>>>> order message data bits the function is permitted to modify to generate its
>>>> system software allocated vectors]. For 16 interrupts, Multiple Message Enable
>>>> will have a value of 4 which means the EP can modify the 4 low order bits to
>>>> raise the 16 interrupts.
>>>>
>>>> With this series added, PCI_MSI_DATA is programmed to 0x1 which gets masked
>>>> while the EP tries to raise an interrupt and hence we are observing off by one
>>>> error.
> 
> So the problem that you saying could be on setting the correct value on the
> PCI_MSI_DATA, right?

correct.
> While I'm trying to support pcitest to be able in future to run the same tests
> that you on my system, I have added some debug on msi.c file that could provide
> me the same info that you have refer before in a short time.
> With the same USB 3.1 PCIe board as a EP, I have generated 2 linux images (with
> and without the patches series).
> 
> Without the patches
> DEBUG:drivers/pci/msi.c:pci_msi_vec_count:881 msgctl = 0x86
> DEBUG:drivers/pci/msi.c:msi_setup_entry:554 control = 0x86
> DEBUG:drivers/pci/msi.c:msi_setup_entry:555 nvec = 0x1
> DEBUG:drivers/pci/msi.c:msi_setup_entry:563 entry->msi_attrib.multi_cap = 0x3
> DEBUG:drivers/pci/msi.c:msi_setup_entry:564 entry->msi_attrib.multiple = 0x0
> DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:314 msgctl = 0x86
> DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:317 entry->msi_attrib.multiple =
> 0x0, msgctl = 0x86
> 
> With the patches (I have commented the MSI_FLAG_PCI_MSIX flag)
> DEBUG:drivers/pci/msi.c:pci_msi_vec_count:882 msgctl = 0x86
> DEBUG:drivers/pci/msi.c:msi_setup_entry:555 control = 0x86
> DEBUG:drivers/pci/msi.c:msi_setup_entry:556 nvec = 0x1
> DEBUG:drivers/pci/msi.c:msi_setup_entry:564 entry->msi_attrib.multi_cap = 0x3
> DEBUG:drivers/pci/msi.c:msi_setup_entry:565 entry->msi_attrib.multiple = 0x0
> DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:315 msgctl = 0x86
> DEBUG:drivers/pci/msi.c:__pci_write_msi_msg:318 entry->msi_attrib.multiple =
> 0x0, msgctl = 0x86

What is the msg data?
> 
> The defined values before the patch are the same with the patch series. Maybe
> this only happens with a bigger number of interrupts required. I will continue
> to work on pcitest to perform the same test that you have done.

The issue happens when 2 devices requesting for MSI.
> 
>>>
>>> The EP (USB 3.1 PCIe board) that I usually use only have 2 interrupts. I didn't
>>> perform the tests like you did. I typically plug the USB flash drive and observe
>>> the number of interrupts and perform some basic file operations on the USB flash
>>> drive (copy/move/delete/rename).
>>
>> Do you have PCIEPORTBUS enabled? That takes 1 interrupt (assuming your host has
>> the required capabilities). You should be able to reproduce the issue with just
>> PCIEPORTBUS enabled then. (With your series added msg_data on USB device will
>> be 0x1 whereas it should be 0x2.
> 
> I checked now, I have PCIEPORTBUS enabled.
> 
>>
>> Before the patch series order_base_2 below helped in taking care of that.
>> pos0 = bitmap_find_free_region(pp->msi_irq_in_use, MAX_MSI_IRQS,
>> order_base_2(no_irqs));
>>>
>>> Can you provide me information (source code, how to compile, how to use, that
>>> kind of info) about the pcitest tool that you mentioned? I would like to use it
>>> if possible, please.
>>
>> Documentation/PCI/endpoint/ should have all the info..
> 
> Ok. After this patch series, maybe we could improve the tool and add the MSI-X
> support also. :)

yeah.. that would be great!

Thanks
Kishon