* [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming @ 2019-05-29 12:37 ` Bharat Kumar Gogada 0 siblings, 0 replies; 12+ messages in thread From: Bharat Kumar Gogada @ 2019-05-29 12:37 UTC (permalink / raw) To: lorenzo.pieralisi, bhelgaas Cc: linux-pci, linux-arm-kernel, linux-kernel, rgummal, Bharat Kumar Gogada The current Multi MSI data programming fails if multiple end points requesting MSI and multi MSI are connected with switch, i.e the current multi MSI data being given is not considering the number of vectors being requested in case of multi MSI. Ex: Two EP's connected via switch, EP1 requesting single MSI first, EP2 requesting Multi MSI of count four. The current code gives MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due to which EP2 also sends interrupt with MSI data 0x0 which results in always invoking virq of EP1 due to which EP2 MSI interrupt never gets handled. Fix Multi MSI data programming with required alignment by using number of vectors being requested. Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe Host Controller") Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> --- V3: - Added example description of the issue --- drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c index 81538d7..8efcb8a 100644 --- a/drivers/pci/controller/pcie-xilinx-nwl.c +++ b/drivers/pci/controller/pcie-xilinx-nwl.c @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, int i; mutex_lock(&msi->lock); - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, + + /* + * Multi MSI count is requested in power of two + * Check if multi msi is requested + */ + if (nr_irqs % 2 == 0) + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, + nr_irqs, nr_irqs - 1); + else + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, nr_irqs, 0); if (bit >= INT_PCI_MSI_NR) { mutex_unlock(&msi->lock); -- 2.7.4 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming @ 2019-05-29 12:37 ` Bharat Kumar Gogada 0 siblings, 0 replies; 12+ messages in thread From: Bharat Kumar Gogada @ 2019-05-29 12:37 UTC (permalink / raw) To: lorenzo.pieralisi, bhelgaas Cc: Bharat Kumar Gogada, linux-pci, rgummal, linux-kernel, linux-arm-kernel The current Multi MSI data programming fails if multiple end points requesting MSI and multi MSI are connected with switch, i.e the current multi MSI data being given is not considering the number of vectors being requested in case of multi MSI. Ex: Two EP's connected via switch, EP1 requesting single MSI first, EP2 requesting Multi MSI of count four. The current code gives MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due to which EP2 also sends interrupt with MSI data 0x0 which results in always invoking virq of EP1 due to which EP2 MSI interrupt never gets handled. Fix Multi MSI data programming with required alignment by using number of vectors being requested. Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe Host Controller") Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> --- V3: - Added example description of the issue --- drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c index 81538d7..8efcb8a 100644 --- a/drivers/pci/controller/pcie-xilinx-nwl.c +++ b/drivers/pci/controller/pcie-xilinx-nwl.c @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, int i; mutex_lock(&msi->lock); - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, + + /* + * Multi MSI count is requested in power of two + * Check if multi msi is requested + */ + if (nr_irqs % 2 == 0) + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, + nr_irqs, nr_irqs - 1); + else + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, nr_irqs, 0); if (bit >= INT_PCI_MSI_NR) { mutex_unlock(&msi->lock); -- 2.7.4 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming 2019-05-29 12:37 ` Bharat Kumar Gogada @ 2019-05-31 16:09 ` Lorenzo Pieralisi -1 siblings, 0 replies; 12+ messages in thread From: Lorenzo Pieralisi @ 2019-05-31 16:09 UTC (permalink / raw) To: Bharat Kumar Gogada Cc: bhelgaas, linux-pci, linux-arm-kernel, linux-kernel, rgummal, marc.zyngier [+Marc] On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > The current Multi MSI data programming fails if multiple end points > requesting MSI and multi MSI are connected with switch, i.e the current > multi MSI data being given is not considering the number of vectors > being requested in case of multi MSI. > Ex: Two EP's connected via switch, EP1 requesting single MSI first, > EP2 requesting Multi MSI of count four. The current code gives > MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits > due to which EP2 also sends interrupt with MSI data 0x0 which results > in always invoking virq of EP1 due to which EP2 MSI interrupt never > gets handled. If this is a problem it is not the only driver where it should be fixed it seems. CC'ed Marc in case I have missed something in relation to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits (according to bits[6:4] in Message Control for MSI) in the MSI data, given that the data written is the hwirq number (in this specific MSI controller) it ought to be fixed. The commit log and patch should be rewritten (I will do that) but first I would like to understand if there are more drivers to be updated. Lorenzo > Fix Multi MSI data programming with required alignment by > using number of vectors being requested. > > Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe > Host Controller") > Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> > --- > V3: > - Added example description of the issue > --- > drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- > 1 file changed, 10 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c > index 81538d7..8efcb8a 100644 > --- a/drivers/pci/controller/pcie-xilinx-nwl.c > +++ b/drivers/pci/controller/pcie-xilinx-nwl.c > @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, > int i; > > mutex_lock(&msi->lock); > - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > + > + /* > + * Multi MSI count is requested in power of two > + * Check if multi msi is requested > + */ > + if (nr_irqs % 2 == 0) > + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > + nr_irqs, nr_irqs - 1); > + else > + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > nr_irqs, 0); > if (bit >= INT_PCI_MSI_NR) { > mutex_unlock(&msi->lock); > -- > 2.7.4 > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming @ 2019-05-31 16:09 ` Lorenzo Pieralisi 0 siblings, 0 replies; 12+ messages in thread From: Lorenzo Pieralisi @ 2019-05-31 16:09 UTC (permalink / raw) To: Bharat Kumar Gogada Cc: marc.zyngier, linux-pci, linux-kernel, rgummal, bhelgaas, linux-arm-kernel [+Marc] On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > The current Multi MSI data programming fails if multiple end points > requesting MSI and multi MSI are connected with switch, i.e the current > multi MSI data being given is not considering the number of vectors > being requested in case of multi MSI. > Ex: Two EP's connected via switch, EP1 requesting single MSI first, > EP2 requesting Multi MSI of count four. The current code gives > MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits > due to which EP2 also sends interrupt with MSI data 0x0 which results > in always invoking virq of EP1 due to which EP2 MSI interrupt never > gets handled. If this is a problem it is not the only driver where it should be fixed it seems. CC'ed Marc in case I have missed something in relation to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits (according to bits[6:4] in Message Control for MSI) in the MSI data, given that the data written is the hwirq number (in this specific MSI controller) it ought to be fixed. The commit log and patch should be rewritten (I will do that) but first I would like to understand if there are more drivers to be updated. Lorenzo > Fix Multi MSI data programming with required alignment by > using number of vectors being requested. > > Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe > Host Controller") > Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> > --- > V3: > - Added example description of the issue > --- > drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- > 1 file changed, 10 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c > index 81538d7..8efcb8a 100644 > --- a/drivers/pci/controller/pcie-xilinx-nwl.c > +++ b/drivers/pci/controller/pcie-xilinx-nwl.c > @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, > int i; > > mutex_lock(&msi->lock); > - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > + > + /* > + * Multi MSI count is requested in power of two > + * Check if multi msi is requested > + */ > + if (nr_irqs % 2 == 0) > + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > + nr_irqs, nr_irqs - 1); > + else > + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > nr_irqs, 0); > if (bit >= INT_PCI_MSI_NR) { > mutex_unlock(&msi->lock); > -- > 2.7.4 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming 2019-05-31 16:09 ` Lorenzo Pieralisi @ 2019-06-03 8:49 ` Marc Zyngier -1 siblings, 0 replies; 12+ messages in thread From: Marc Zyngier @ 2019-06-03 8:49 UTC (permalink / raw) To: Lorenzo Pieralisi, Bharat Kumar Gogada Cc: bhelgaas, linux-pci, linux-arm-kernel, linux-kernel, rgummal On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > [+Marc] > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: >> The current Multi MSI data programming fails if multiple end points >> requesting MSI and multi MSI are connected with switch, i.e the current >> multi MSI data being given is not considering the number of vectors >> being requested in case of multi MSI. >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, >> EP2 requesting Multi MSI of count four. The current code gives >> MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits >> due to which EP2 also sends interrupt with MSI data 0x0 which results >> in always invoking virq of EP1 due to which EP2 MSI interrupt never >> gets handled. > > If this is a problem it is not the only driver where it should be fixed > it seems. CC'ed Marc in case I have missed something in relation to MSI > IRQs but AFAIU it looks like HW is allowed to toggled bits (according to > bits[6:4] in Message Control for MSI) in the MSI data, given that the > data written is the hwirq number (in this specific MSI controller) > it ought to be fixed. Yeah, it looks like a number of MSI controllers could be quite broken in this particular area. > > The commit log and patch should be rewritten (I will do that) but > first I would like to understand if there are more drivers to be > updated. > > Lorenzo > >> Fix Multi MSI data programming with required alignment by >> using number of vectors being requested. >> >> Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe >> Host Controller") >> Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> >> --- >> V3: >> - Added example description of the issue >> --- >> drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- >> 1 file changed, 10 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c >> index 81538d7..8efcb8a 100644 >> --- a/drivers/pci/controller/pcie-xilinx-nwl.c >> +++ b/drivers/pci/controller/pcie-xilinx-nwl.c >> @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, >> int i; >> >> mutex_lock(&msi->lock); >> - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, >> + >> + /* >> + * Multi MSI count is requested in power of two >> + * Check if multi msi is requested >> + */ >> + if (nr_irqs % 2 == 0) >> + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, >> + nr_irqs, nr_irqs - 1); >> + else >> + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, >> nr_irqs, 0); >> if (bit >= INT_PCI_MSI_NR) { >> mutex_unlock(&msi->lock); >> -- >> 2.7.4 >> This doesn't look like the best fix. The only case where nr_irqs is not set to 1 is when using Multi-MSI, so the '% 2' case actually covers all cases. Now, and in the interest of consistency, other drivers use a construct such as: offset = bitmap_find_free_region(bitmap, bitmap_size, get_count_order(nr_irqs)); which has the advantage of dealing with the bitmap setting as well. I'd suggest something like this (completely untested): diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c index 3b031f00a94a..8b9b58909e7c 100644 --- a/drivers/pci/controller/pcie-xilinx-nwl.c +++ b/drivers/pci/controller/pcie-xilinx-nwl.c @@ -482,15 +482,13 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, int i; mutex_lock(&msi->lock); - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, - nr_irqs, 0); - if (bit >= INT_PCI_MSI_NR) { + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR, + get_count_order(nr_irqs)); + if (bit < 0) { mutex_unlock(&msi->lock); return -ENOSPC; } - bitmap_set(msi->bitmap, bit, nr_irqs); - for (i = 0; i < nr_irqs; i++) { irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip, domain->host_data, handle_simple_irq, @@ -508,7 +506,7 @@ static void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq, struct nwl_msi *msi = &pcie->msi; mutex_lock(&msi->lock); - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs); + bitmap_release_region(msi->bitmap, data->hwirq, get_count_order(nr_irqs)); mutex_unlock(&msi->lock); } Thanks, M. -- Jazz is not dead. It just smells funny... ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming @ 2019-06-03 8:49 ` Marc Zyngier 0 siblings, 0 replies; 12+ messages in thread From: Marc Zyngier @ 2019-06-03 8:49 UTC (permalink / raw) To: Lorenzo Pieralisi, Bharat Kumar Gogada Cc: bhelgaas, linux-pci, rgummal, linux-kernel, linux-arm-kernel On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > [+Marc] > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: >> The current Multi MSI data programming fails if multiple end points >> requesting MSI and multi MSI are connected with switch, i.e the current >> multi MSI data being given is not considering the number of vectors >> being requested in case of multi MSI. >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, >> EP2 requesting Multi MSI of count four. The current code gives >> MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits >> due to which EP2 also sends interrupt with MSI data 0x0 which results >> in always invoking virq of EP1 due to which EP2 MSI interrupt never >> gets handled. > > If this is a problem it is not the only driver where it should be fixed > it seems. CC'ed Marc in case I have missed something in relation to MSI > IRQs but AFAIU it looks like HW is allowed to toggled bits (according to > bits[6:4] in Message Control for MSI) in the MSI data, given that the > data written is the hwirq number (in this specific MSI controller) > it ought to be fixed. Yeah, it looks like a number of MSI controllers could be quite broken in this particular area. > > The commit log and patch should be rewritten (I will do that) but > first I would like to understand if there are more drivers to be > updated. > > Lorenzo > >> Fix Multi MSI data programming with required alignment by >> using number of vectors being requested. >> >> Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe >> Host Controller") >> Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> >> --- >> V3: >> - Added example description of the issue >> --- >> drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- >> 1 file changed, 10 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c >> index 81538d7..8efcb8a 100644 >> --- a/drivers/pci/controller/pcie-xilinx-nwl.c >> +++ b/drivers/pci/controller/pcie-xilinx-nwl.c >> @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, >> int i; >> >> mutex_lock(&msi->lock); >> - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, >> + >> + /* >> + * Multi MSI count is requested in power of two >> + * Check if multi msi is requested >> + */ >> + if (nr_irqs % 2 == 0) >> + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, >> + nr_irqs, nr_irqs - 1); >> + else >> + bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, >> nr_irqs, 0); >> if (bit >= INT_PCI_MSI_NR) { >> mutex_unlock(&msi->lock); >> -- >> 2.7.4 >> This doesn't look like the best fix. The only case where nr_irqs is not set to 1 is when using Multi-MSI, so the '% 2' case actually covers all cases. Now, and in the interest of consistency, other drivers use a construct such as: offset = bitmap_find_free_region(bitmap, bitmap_size, get_count_order(nr_irqs)); which has the advantage of dealing with the bitmap setting as well. I'd suggest something like this (completely untested): diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c index 3b031f00a94a..8b9b58909e7c 100644 --- a/drivers/pci/controller/pcie-xilinx-nwl.c +++ b/drivers/pci/controller/pcie-xilinx-nwl.c @@ -482,15 +482,13 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq, int i; mutex_lock(&msi->lock); - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, - nr_irqs, 0); - if (bit >= INT_PCI_MSI_NR) { + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR, + get_count_order(nr_irqs)); + if (bit < 0) { mutex_unlock(&msi->lock); return -ENOSPC; } - bitmap_set(msi->bitmap, bit, nr_irqs); - for (i = 0; i < nr_irqs; i++) { irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip, domain->host_data, handle_simple_irq, @@ -508,7 +506,7 @@ static void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq, struct nwl_msi *msi = &pcie->msi; mutex_lock(&msi->lock); - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs); + bitmap_release_region(msi->bitmap, data->hwirq, get_count_order(nr_irqs)); mutex_unlock(&msi->lock); } Thanks, M. -- Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 12+ messages in thread
* RE: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming 2019-06-03 8:49 ` Marc Zyngier @ 2019-06-06 4:49 ` Bharat Kumar Gogada -1 siblings, 0 replies; 12+ messages in thread From: Bharat Kumar Gogada @ 2019-06-06 4:49 UTC (permalink / raw) To: Marc Zyngier, lorenzo.pieralisi Cc: bhelgaas, linux-pci, linux-arm-kernel, linux-kernel, Ravikiran Gummaluri > On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > > [+Marc] > > > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > >> The current Multi MSI data programming fails if multiple end points > >> requesting MSI and multi MSI are connected with switch, i.e the > >> current multi MSI data being given is not considering the number of > >> vectors being requested in case of multi MSI. > >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, > >> EP2 requesting Multi MSI of count four. The current code gives MSI > >> data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due > >> to which EP2 also sends interrupt with MSI data 0x0 which results in > >> always invoking virq of EP1 due to which EP2 MSI interrupt never gets > >> handled. > > > > If this is a problem it is not the only driver where it should be > > fixed it seems. CC'ed Marc in case I have missed something in relation > > to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits > > (according to bits[6:4] in Message Control for MSI) in the MSI data, > > given that the data written is the hwirq number (in this specific MSI > > controller) it ought to be fixed. > > Yeah, it looks like a number of MSI controllers could be quite broken in this > particular area. > > > > > The commit log and patch should be rewritten (I will do that) but > > first I would like to understand if there are more drivers to be > > updated. > > > > Hi Lorenzo and Marc, thanks for your time. Marc, I'm yet to test the below suggested solution, GIC v2m and GIC v3 supports multi MSI, do we see above issue in these MSI controllers ? Regards, Bharat > > > >> Fix Multi MSI data programming with required alignment by using > >> number of vectors being requested. > >> > >> Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL > >> PCIe Host Controller") > >> Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> > >> --- > >> V3: > >> - Added example description of the issue > >> --- > >> drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- > >> 1 file changed, 10 insertions(+), 1 deletion(-) > >> > >> diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c > >> b/drivers/pci/controller/pcie-xilinx-nwl.c > >> index 81538d7..8efcb8a 100644 > >> --- a/drivers/pci/controller/pcie-xilinx-nwl.c > >> +++ b/drivers/pci/controller/pcie-xilinx-nwl.c > >> @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct > irq_domain *domain, unsigned int virq, > >> int i; > >> > >> mutex_lock(&msi->lock); > >> - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > >> + > >> + /* > >> + * Multi MSI count is requested in power of two > >> + * Check if multi msi is requested > >> + */ > >> + if (nr_irqs % 2 == 0) > >> + bit = bitmap_find_next_zero_area(msi->bitmap, > INT_PCI_MSI_NR, 0, > >> + nr_irqs, nr_irqs - 1); > >> + else > >> + bit = bitmap_find_next_zero_area(msi->bitmap, > INT_PCI_MSI_NR, 0, > >> nr_irqs, 0); > >> if (bit >= INT_PCI_MSI_NR) { > >> mutex_unlock(&msi->lock); > >> -- > >> 2.7.4 > >> > > This doesn't look like the best fix. The only case where nr_irqs is not > set to 1 is when using Multi-MSI, so the '% 2' case actually covers all > cases. Now, and in the interest of consistency, other drivers use a > construct such as: > > offset = bitmap_find_free_region(bitmap, bitmap_size, > get_count_order(nr_irqs)); > > which has the advantage of dealing with the bitmap setting as well. > > I'd suggest something like this (completely untested): > > diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c > b/drivers/pci/controller/pcie-xilinx-nwl.c > index 3b031f00a94a..8b9b58909e7c 100644 > --- a/drivers/pci/controller/pcie-xilinx-nwl.c > +++ b/drivers/pci/controller/pcie-xilinx-nwl.c > @@ -482,15 +482,13 @@ static int nwl_irq_domain_alloc(struct > irq_domain *domain, unsigned int virq, > int i; > > mutex_lock(&msi->lock); > - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > - nr_irqs, 0); > - if (bit >= INT_PCI_MSI_NR) { > + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR, > + get_count_order(nr_irqs)); > + if (bit < 0) { > mutex_unlock(&msi->lock); > return -ENOSPC; > } > > - bitmap_set(msi->bitmap, bit, nr_irqs); > - > for (i = 0; i < nr_irqs; i++) { > irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip, > domain->host_data, handle_simple_irq, > @@ -508,7 +506,7 @@ static void nwl_irq_domain_free(struct irq_domain > *domain, unsigned int virq, > struct nwl_msi *msi = &pcie->msi; > > mutex_lock(&msi->lock); > - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs); > + bitmap_release_region(msi->bitmap, data->hwirq, > get_count_order(nr_irqs)); > mutex_unlock(&msi->lock); > } > ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming @ 2019-06-06 4:49 ` Bharat Kumar Gogada 0 siblings, 0 replies; 12+ messages in thread From: Bharat Kumar Gogada @ 2019-06-06 4:49 UTC (permalink / raw) To: Marc Zyngier, lorenzo.pieralisi Cc: bhelgaas, linux-pci, Ravikiran Gummaluri, linux-kernel, linux-arm-kernel > On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > > [+Marc] > > > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > >> The current Multi MSI data programming fails if multiple end points > >> requesting MSI and multi MSI are connected with switch, i.e the > >> current multi MSI data being given is not considering the number of > >> vectors being requested in case of multi MSI. > >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, > >> EP2 requesting Multi MSI of count four. The current code gives MSI > >> data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due > >> to which EP2 also sends interrupt with MSI data 0x0 which results in > >> always invoking virq of EP1 due to which EP2 MSI interrupt never gets > >> handled. > > > > If this is a problem it is not the only driver where it should be > > fixed it seems. CC'ed Marc in case I have missed something in relation > > to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits > > (according to bits[6:4] in Message Control for MSI) in the MSI data, > > given that the data written is the hwirq number (in this specific MSI > > controller) it ought to be fixed. > > Yeah, it looks like a number of MSI controllers could be quite broken in this > particular area. > > > > > The commit log and patch should be rewritten (I will do that) but > > first I would like to understand if there are more drivers to be > > updated. > > > > Hi Lorenzo and Marc, thanks for your time. Marc, I'm yet to test the below suggested solution, GIC v2m and GIC v3 supports multi MSI, do we see above issue in these MSI controllers ? Regards, Bharat > > > >> Fix Multi MSI data programming with required alignment by using > >> number of vectors being requested. > >> > >> Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL > >> PCIe Host Controller") > >> Signed-off-by: Bharat Kumar Gogada <bharat.kumar.gogada@xilinx.com> > >> --- > >> V3: > >> - Added example description of the issue > >> --- > >> drivers/pci/controller/pcie-xilinx-nwl.c | 11 ++++++++++- > >> 1 file changed, 10 insertions(+), 1 deletion(-) > >> > >> diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c > >> b/drivers/pci/controller/pcie-xilinx-nwl.c > >> index 81538d7..8efcb8a 100644 > >> --- a/drivers/pci/controller/pcie-xilinx-nwl.c > >> +++ b/drivers/pci/controller/pcie-xilinx-nwl.c > >> @@ -483,7 +483,16 @@ static int nwl_irq_domain_alloc(struct > irq_domain *domain, unsigned int virq, > >> int i; > >> > >> mutex_lock(&msi->lock); > >> - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > >> + > >> + /* > >> + * Multi MSI count is requested in power of two > >> + * Check if multi msi is requested > >> + */ > >> + if (nr_irqs % 2 == 0) > >> + bit = bitmap_find_next_zero_area(msi->bitmap, > INT_PCI_MSI_NR, 0, > >> + nr_irqs, nr_irqs - 1); > >> + else > >> + bit = bitmap_find_next_zero_area(msi->bitmap, > INT_PCI_MSI_NR, 0, > >> nr_irqs, 0); > >> if (bit >= INT_PCI_MSI_NR) { > >> mutex_unlock(&msi->lock); > >> -- > >> 2.7.4 > >> > > This doesn't look like the best fix. The only case where nr_irqs is not > set to 1 is when using Multi-MSI, so the '% 2' case actually covers all > cases. Now, and in the interest of consistency, other drivers use a > construct such as: > > offset = bitmap_find_free_region(bitmap, bitmap_size, > get_count_order(nr_irqs)); > > which has the advantage of dealing with the bitmap setting as well. > > I'd suggest something like this (completely untested): > > diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c > b/drivers/pci/controller/pcie-xilinx-nwl.c > index 3b031f00a94a..8b9b58909e7c 100644 > --- a/drivers/pci/controller/pcie-xilinx-nwl.c > +++ b/drivers/pci/controller/pcie-xilinx-nwl.c > @@ -482,15 +482,13 @@ static int nwl_irq_domain_alloc(struct > irq_domain *domain, unsigned int virq, > int i; > > mutex_lock(&msi->lock); > - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0, > - nr_irqs, 0); > - if (bit >= INT_PCI_MSI_NR) { > + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR, > + get_count_order(nr_irqs)); > + if (bit < 0) { > mutex_unlock(&msi->lock); > return -ENOSPC; > } > > - bitmap_set(msi->bitmap, bit, nr_irqs); > - > for (i = 0; i < nr_irqs; i++) { > irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip, > domain->host_data, handle_simple_irq, > @@ -508,7 +506,7 @@ static void nwl_irq_domain_free(struct irq_domain > *domain, unsigned int virq, > struct nwl_msi *msi = &pcie->msi; > > mutex_lock(&msi->lock); > - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs); > + bitmap_release_region(msi->bitmap, data->hwirq, > get_count_order(nr_irqs)); > mutex_unlock(&msi->lock); > } > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming 2019-06-06 4:49 ` Bharat Kumar Gogada @ 2019-06-06 7:18 ` Marc Zyngier -1 siblings, 0 replies; 12+ messages in thread From: Marc Zyngier @ 2019-06-06 7:18 UTC (permalink / raw) To: Bharat Kumar Gogada Cc: lorenzo.pieralisi, bhelgaas, linux-pci, linux-arm-kernel, linux-kernel, Ravikiran Gummaluri On Thu, 06 Jun 2019 05:49:45 +0100, Bharat Kumar Gogada <bharatku@xilinx.com> wrote: > > > On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > > > [+Marc] > > > > > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > > >> The current Multi MSI data programming fails if multiple end points > > >> requesting MSI and multi MSI are connected with switch, i.e the > > >> current multi MSI data being given is not considering the number of > > >> vectors being requested in case of multi MSI. > > >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, > > >> EP2 requesting Multi MSI of count four. The current code gives MSI > > >> data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due > > >> to which EP2 also sends interrupt with MSI data 0x0 which results in > > >> always invoking virq of EP1 due to which EP2 MSI interrupt never gets > > >> handled. > > > > > > If this is a problem it is not the only driver where it should be > > > fixed it seems. CC'ed Marc in case I have missed something in relation > > > to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits > > > (according to bits[6:4] in Message Control for MSI) in the MSI data, > > > given that the data written is the hwirq number (in this specific MSI > > > controller) it ought to be fixed. > > > > Yeah, it looks like a number of MSI controllers could be quite broken in this > > particular area. > > > > > > > > The commit log and patch should be rewritten (I will do that) but > > > first I would like to understand if there are more drivers to be > > > updated. > > > > > > > Hi Lorenzo and Marc, thanks for your time. > Marc, I'm yet to test the below suggested solution, > GIC v2m and GIC v3 supports multi MSI, do we see above issue in > these MSI controllers ? To the best of my knowledge, these drivers do support MultiMSI correctly. GICv2m actually gained the support pretty recently (see de337ee30142). The GICv3 ITS never ha an issue with that, given that per device EventIDs are always 0-based. Thanks, M. -- Jazz is not dead, it just smells funny. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming @ 2019-06-06 7:18 ` Marc Zyngier 0 siblings, 0 replies; 12+ messages in thread From: Marc Zyngier @ 2019-06-06 7:18 UTC (permalink / raw) To: Bharat Kumar Gogada Cc: lorenzo.pieralisi, linux-pci, linux-kernel, Ravikiran Gummaluri, bhelgaas, linux-arm-kernel On Thu, 06 Jun 2019 05:49:45 +0100, Bharat Kumar Gogada <bharatku@xilinx.com> wrote: > > > On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > > > [+Marc] > > > > > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > > >> The current Multi MSI data programming fails if multiple end points > > >> requesting MSI and multi MSI are connected with switch, i.e the > > >> current multi MSI data being given is not considering the number of > > >> vectors being requested in case of multi MSI. > > >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, > > >> EP2 requesting Multi MSI of count four. The current code gives MSI > > >> data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due > > >> to which EP2 also sends interrupt with MSI data 0x0 which results in > > >> always invoking virq of EP1 due to which EP2 MSI interrupt never gets > > >> handled. > > > > > > If this is a problem it is not the only driver where it should be > > > fixed it seems. CC'ed Marc in case I have missed something in relation > > > to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits > > > (according to bits[6:4] in Message Control for MSI) in the MSI data, > > > given that the data written is the hwirq number (in this specific MSI > > > controller) it ought to be fixed. > > > > Yeah, it looks like a number of MSI controllers could be quite broken in this > > particular area. > > > > > > > > The commit log and patch should be rewritten (I will do that) but > > > first I would like to understand if there are more drivers to be > > > updated. > > > > > > > Hi Lorenzo and Marc, thanks for your time. > Marc, I'm yet to test the below suggested solution, > GIC v2m and GIC v3 supports multi MSI, do we see above issue in > these MSI controllers ? To the best of my knowledge, these drivers do support MultiMSI correctly. GICv2m actually gained the support pretty recently (see de337ee30142). The GICv3 ITS never ha an issue with that, given that per device EventIDs are always 0-based. Thanks, M. -- Jazz is not dead, it just smells funny. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming 2019-06-06 7:18 ` Marc Zyngier @ 2019-06-10 10:27 ` Lorenzo Pieralisi -1 siblings, 0 replies; 12+ messages in thread From: Lorenzo Pieralisi @ 2019-06-10 10:27 UTC (permalink / raw) To: Marc Zyngier Cc: Bharat Kumar Gogada, bhelgaas, linux-pci, linux-arm-kernel, linux-kernel, Ravikiran Gummaluri On Thu, Jun 06, 2019 at 08:18:25AM +0100, Marc Zyngier wrote: > On Thu, 06 Jun 2019 05:49:45 +0100, > Bharat Kumar Gogada <bharatku@xilinx.com> wrote: > > > > > On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > > > > [+Marc] > > > > > > > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > > > >> The current Multi MSI data programming fails if multiple end points > > > >> requesting MSI and multi MSI are connected with switch, i.e the > > > >> current multi MSI data being given is not considering the number of > > > >> vectors being requested in case of multi MSI. > > > >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, > > > >> EP2 requesting Multi MSI of count four. The current code gives MSI > > > >> data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due > > > >> to which EP2 also sends interrupt with MSI data 0x0 which results in > > > >> always invoking virq of EP1 due to which EP2 MSI interrupt never gets > > > >> handled. > > > > > > > > If this is a problem it is not the only driver where it should be > > > > fixed it seems. CC'ed Marc in case I have missed something in relation > > > > to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits > > > > (according to bits[6:4] in Message Control for MSI) in the MSI data, > > > > given that the data written is the hwirq number (in this specific MSI > > > > controller) it ought to be fixed. > > > > > > Yeah, it looks like a number of MSI controllers could be quite broken in this > > > particular area. > > > > > > > > > > > The commit log and patch should be rewritten (I will do that) but > > > > first I would like to understand if there are more drivers to be > > > > updated. > > > > > > > > > > Hi Lorenzo and Marc, thanks for your time. > > Marc, I'm yet to test the below suggested solution, > > GIC v2m and GIC v3 supports multi MSI, do we see above issue in > > these MSI controllers ? > > To the best of my knowledge, these drivers do support MultiMSI > correctly. GICv2m actually gained the support pretty recently (see > de337ee30142). The GICv3 ITS never ha an issue with that, given that > per device EventIDs are always 0-based. AFAIU I think the issues is only present in controllers that use the hwirq as MSI data and bitmap allocation that is not a power of two and that's what Marc suggested as fix. There is still some chasing to do to fix other MSI controllers in the kernel where this subtle issue went undetected (and the driver has the same bitmap allocation issues as this one). @Bharat, please test Marc's patch and post it on completion, I will rewrite your commit log because I want it to be clear so that we have a reference to the issue linked to the specs. Thanks, Lorenzo ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming @ 2019-06-10 10:27 ` Lorenzo Pieralisi 0 siblings, 0 replies; 12+ messages in thread From: Lorenzo Pieralisi @ 2019-06-10 10:27 UTC (permalink / raw) To: Marc Zyngier Cc: linux-pci, linux-kernel, Bharat Kumar Gogada, Ravikiran Gummaluri, bhelgaas, linux-arm-kernel On Thu, Jun 06, 2019 at 08:18:25AM +0100, Marc Zyngier wrote: > On Thu, 06 Jun 2019 05:49:45 +0100, > Bharat Kumar Gogada <bharatku@xilinx.com> wrote: > > > > > On 31/05/2019 17:09, Lorenzo Pieralisi wrote: > > > > [+Marc] > > > > > > > > On Wed, May 29, 2019 at 06:07:49PM +0530, Bharat Kumar Gogada wrote: > > > >> The current Multi MSI data programming fails if multiple end points > > > >> requesting MSI and multi MSI are connected with switch, i.e the > > > >> current multi MSI data being given is not considering the number of > > > >> vectors being requested in case of multi MSI. > > > >> Ex: Two EP's connected via switch, EP1 requesting single MSI first, > > > >> EP2 requesting Multi MSI of count four. The current code gives MSI > > > >> data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due > > > >> to which EP2 also sends interrupt with MSI data 0x0 which results in > > > >> always invoking virq of EP1 due to which EP2 MSI interrupt never gets > > > >> handled. > > > > > > > > If this is a problem it is not the only driver where it should be > > > > fixed it seems. CC'ed Marc in case I have missed something in relation > > > > to MSI IRQs but AFAIU it looks like HW is allowed to toggled bits > > > > (according to bits[6:4] in Message Control for MSI) in the MSI data, > > > > given that the data written is the hwirq number (in this specific MSI > > > > controller) it ought to be fixed. > > > > > > Yeah, it looks like a number of MSI controllers could be quite broken in this > > > particular area. > > > > > > > > > > > The commit log and patch should be rewritten (I will do that) but > > > > first I would like to understand if there are more drivers to be > > > > updated. > > > > > > > > > > Hi Lorenzo and Marc, thanks for your time. > > Marc, I'm yet to test the below suggested solution, > > GIC v2m and GIC v3 supports multi MSI, do we see above issue in > > these MSI controllers ? > > To the best of my knowledge, these drivers do support MultiMSI > correctly. GICv2m actually gained the support pretty recently (see > de337ee30142). The GICv3 ITS never ha an issue with that, given that > per device EventIDs are always 0-based. AFAIU I think the issues is only present in controllers that use the hwirq as MSI data and bitmap allocation that is not a power of two and that's what Marc suggested as fix. There is still some chasing to do to fix other MSI controllers in the kernel where this subtle issue went undetected (and the driver has the same bitmap allocation issues as this one). @Bharat, please test Marc's patch and post it on completion, I will rewrite your commit log because I want it to be clear so that we have a reference to the issue linked to the specs. Thanks, Lorenzo _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-06-10 10:27 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-05-29 12:37 [PATCH v3] PCI: xilinx-nwl: Fix Multi MSI data programming Bharat Kumar Gogada 2019-05-29 12:37 ` Bharat Kumar Gogada 2019-05-31 16:09 ` Lorenzo Pieralisi 2019-05-31 16:09 ` Lorenzo Pieralisi 2019-06-03 8:49 ` Marc Zyngier 2019-06-03 8:49 ` Marc Zyngier 2019-06-06 4:49 ` Bharat Kumar Gogada 2019-06-06 4:49 ` Bharat Kumar Gogada 2019-06-06 7:18 ` Marc Zyngier 2019-06-06 7:18 ` Marc Zyngier 2019-06-10 10:27 ` Lorenzo Pieralisi 2019-06-10 10:27 ` Lorenzo Pieralisi
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.