* Re: [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA [not found] ` <1535453838-12154-11-git-send-email-sunil.kovvuri@gmail.com> @ 2018-08-28 12:08 ` Arnd Bergmann 2018-08-28 12:42 ` Sunil Kovvuri 0 siblings, 1 reply; 6+ messages in thread From: Arnd Bergmann @ 2018-08-28 12:08 UTC (permalink / raw) To: sunil.kovvuri Cc: Jason Cooper, Marc Zyngier, linux-pci, Linux Kernel Mailing List, Linux ARM, Olof Johansson, Thomas Gleixner, linux-soc, gakula, sgoutham On Tue, Aug 28, 2018 at 12:58 PM <sunil.kovvuri@gmail.com> wrote: > > From: Geetha sowjanya <gakula@marvell.com> > > HW interprets RVU_AF_MSIXTR_BASE address as an IOVA, hence > create a IOMMU mapping for the physcial address configured by > firmware and reconfig RVU_AF_MSIXTR_BASE with IOVA. > > Signed-off-by: Geetha sowjanya <gakula@marvell.com> > Signed-off-by: Sunil Goutham <sgoutham@marvell.com> I think this needs some more explanation. What is the difference between the MSI-X support in this driver and every other one? Are you working around a hardware bug, or is there something odd in the implementation of your irqchip driver? Do you use a GIC to handle the MSI interrupts or something else? Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA 2018-08-28 12:08 ` [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA Arnd Bergmann @ 2018-08-28 12:42 ` Sunil Kovvuri 2018-08-28 12:57 ` Arnd Bergmann 0 siblings, 1 reply; 6+ messages in thread From: Sunil Kovvuri @ 2018-08-28 12:42 UTC (permalink / raw) To: Arnd Bergmann Cc: jason, Marc Zyngier, linux-pci, LKML, LAKML, olof, tglx, linux-soc, Geetha sowjanya, Sunil Goutham On Tue, Aug 28, 2018 at 5:39 PM Arnd Bergmann <arnd@arndb.de> wrote: > > On Tue, Aug 28, 2018 at 12:58 PM <sunil.kovvuri@gmail.com> wrote: > > > > From: Geetha sowjanya <gakula@marvell.com> > > > > HW interprets RVU_AF_MSIXTR_BASE address as an IOVA, hence > > create a IOMMU mapping for the physcial address configured by > > firmware and reconfig RVU_AF_MSIXTR_BASE with IOVA. > > > > Signed-off-by: Geetha sowjanya <gakula@marvell.com> > > Signed-off-by: Sunil Goutham <sgoutham@marvell.com> > > I think this needs some more explanation. What is the difference between > the MSI-X support in this driver and every other one? Are you working > around a hardware bug, or is there something odd in the implementation > of your irqchip driver? Do you use a GIC to handle the MSI interrupts > or something else? > > Arnd This admin function is a PCI device which is capable of provisioning HW blocks to other PCIe SRIOV devices in the system. Each HW block (eg memory buffer pools, NIC dewscriptors, crypto engines e.t.c) needs certain no of MSIX vectors. Admin function has a set of 32K MSIX vectors in memory (not on-chip) which based on HW block provisioning to a PCI device attaches the required number of vectors to that device. Some part of this configuration is done by low level firmware. RVU_AF_MSIXTR_BASE points to the memory region allocated for 32K MSIX vectors. If kernel is booted with IOMMU enabled and admin function device is attached to SMMU, HW will go through translation to access this MSIX vector memory region. Hence the mapping done in this patch. Thanks, Sunil. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA 2018-08-28 12:42 ` Sunil Kovvuri @ 2018-08-28 12:57 ` Arnd Bergmann 2018-08-28 13:17 ` Sunil Kovvuri 0 siblings, 1 reply; 6+ messages in thread From: Arnd Bergmann @ 2018-08-28 12:57 UTC (permalink / raw) To: Sunil Kovvuri Cc: Jason Cooper, Marc Zyngier, linux-pci, Linux Kernel Mailing List, Linux ARM, Olof Johansson, Thomas Gleixner, linux-soc, gakula, sgoutham On Tue, Aug 28, 2018 at 2:42 PM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote: > > On Tue, Aug 28, 2018 at 5:39 PM Arnd Bergmann <arnd@arndb.de> wrote: > > > > On Tue, Aug 28, 2018 at 12:58 PM <sunil.kovvuri@gmail.com> wrote: > > > > > > From: Geetha sowjanya <gakula@marvell.com> > > > > > > HW interprets RVU_AF_MSIXTR_BASE address as an IOVA, hence > > > create a IOMMU mapping for the physcial address configured by > > > firmware and reconfig RVU_AF_MSIXTR_BASE with IOVA. > > > > > > Signed-off-by: Geetha sowjanya <gakula@marvell.com> > > > Signed-off-by: Sunil Goutham <sgoutham@marvell.com> > > > > I think this needs some more explanation. What is the difference between > > the MSI-X support in this driver and every other one? Are you working > > around a hardware bug, or is there something odd in the implementation > > of your irqchip driver? Do you use a GIC to handle the MSI interrupts > > or something else? > > This admin function is a PCI device which is capable of provisioning > HW blocks to other PCIe SRIOV devices in the system. Each HW block > (eg memory buffer pools, NIC dewscriptors, crypto engines e.t.c) needs > certain no of MSIX vectors. Admin function has a set of 32K MSIX vectors > in memory (not on-chip) which based on HW block provisioning to a PCI device > attaches the required number of vectors to that device. Some part of this > configuration is done by low level firmware. > > RVU_AF_MSIXTR_BASE points to the memory region allocated for 32K MSIX > vectors. If kernel is booted with IOMMU enabled and admin function device > is attached to SMMU, HW will go through translation to access this MSIX > vector memory region. Hence the mapping done in this patch. Do you mean this is not a regular PCIe MSI-X interrupt to the GIC, but something internal to your device that gets routed through the IOMMU back into the device? I'm still confused. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA 2018-08-28 12:57 ` Arnd Bergmann @ 2018-08-28 13:17 ` Sunil Kovvuri 2018-08-30 13:53 ` Arnd Bergmann 0 siblings, 1 reply; 6+ messages in thread From: Sunil Kovvuri @ 2018-08-28 13:17 UTC (permalink / raw) To: Arnd Bergmann Cc: jason, Marc Zyngier, linux-pci, LKML, LAKML, olof, tglx, linux-soc, Geetha sowjanya, Sunil Goutham On Tue, Aug 28, 2018 at 6:27 PM Arnd Bergmann <arnd@arndb.de> wrote: > > On Tue, Aug 28, 2018 at 2:42 PM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote: > > > > On Tue, Aug 28, 2018 at 5:39 PM Arnd Bergmann <arnd@arndb.de> wrote: > > > > > > On Tue, Aug 28, 2018 at 12:58 PM <sunil.kovvuri@gmail.com> wrote: > > > > > > > > From: Geetha sowjanya <gakula@marvell.com> > > > > > > > > HW interprets RVU_AF_MSIXTR_BASE address as an IOVA, hence > > > > create a IOMMU mapping for the physcial address configured by > > > > firmware and reconfig RVU_AF_MSIXTR_BASE with IOVA. > > > > > > > > Signed-off-by: Geetha sowjanya <gakula@marvell.com> > > > > Signed-off-by: Sunil Goutham <sgoutham@marvell.com> > > > > > > I think this needs some more explanation. What is the difference between > > > the MSI-X support in this driver and every other one? Are you working > > > around a hardware bug, or is there something odd in the implementation > > > of your irqchip driver? Do you use a GIC to handle the MSI interrupts > > > or something else? > > > > This admin function is a PCI device which is capable of provisioning > > HW blocks to other PCIe SRIOV devices in the system. Each HW block > > (eg memory buffer pools, NIC dewscriptors, crypto engines e.t.c) needs > > certain no of MSIX vectors. Admin function has a set of 32K MSIX vectors > > in memory (not on-chip) which based on HW block provisioning to a PCI device > > attaches the required number of vectors to that device. Some part of this > > configuration is done by low level firmware. > > > > RVU_AF_MSIXTR_BASE points to the memory region allocated for 32K MSIX > > vectors. If kernel is booted with IOMMU enabled and admin function device > > is attached to SMMU, HW will go through translation to access this MSIX > > vector memory region. Hence the mapping done in this patch. > > Do you mean this is not a regular PCIe MSI-X interrupt to the GIC, but > something internal to your device that gets routed through the IOMMU > back into the device? > > I'm still confused. > > Arnd This is a regular PCIe MSI-X interrupt, the difference is that the bunch of PCI devices here doesn't have a fixed set of MSIX vectors. Admin function has a memory region with 32K MSIX vectors which it provisions to PCI devices based on the HW functional blocks attached to them. A PCI device which works as a ethernet device needs X number of vectors and a crypto device needs Y number of vectors. Since the admin function owns the whole MSIX vector region, HW uses this device's stream ID to access the vectors. Hence the IOMMU mapping. Once MSIX vectors are provisioned to a PCI device they work as normal MSIX interrupt like any other device. Thanks, Sunil. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA 2018-08-28 13:17 ` Sunil Kovvuri @ 2018-08-30 13:53 ` Arnd Bergmann 2018-08-30 18:39 ` Sunil Kovvuri 0 siblings, 1 reply; 6+ messages in thread From: Arnd Bergmann @ 2018-08-30 13:53 UTC (permalink / raw) To: Sunil Kovvuri Cc: Jason Cooper, Marc Zyngier, linux-pci, Linux Kernel Mailing List, Linux ARM, Olof Johansson, Thomas Gleixner, linux-soc, gakula, sgoutham On Tue, Aug 28, 2018 at 3:17 PM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote: > On Tue, Aug 28, 2018 at 6:27 PM Arnd Bergmann <arnd@arndb.de> wrote: > > On Tue, Aug 28, 2018 at 2:42 PM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote: > > > On Tue, Aug 28, 2018 at 5:39 PM Arnd Bergmann <arnd@arndb.de> wrote: > > > > On Tue, Aug 28, 2018 at 12:58 PM <sunil.kovvuri@gmail.com> wrote: > > > > > > This admin function is a PCI device which is capable of provisioning > > > HW blocks to other PCIe SRIOV devices in the system. Each HW block > > > (eg memory buffer pools, NIC dewscriptors, crypto engines e.t.c) needs > > > certain no of MSIX vectors. Admin function has a set of 32K MSIX vectors > > > in memory (not on-chip) which based on HW block provisioning to a PCI device > > > attaches the required number of vectors to that device. Some part of this > > > configuration is done by low level firmware. > > > > > > RVU_AF_MSIXTR_BASE points to the memory region allocated for 32K MSIX > > > vectors. If kernel is booted with IOMMU enabled and admin function device > > > is attached to SMMU, HW will go through translation to access this MSIX > > > vector memory region. Hence the mapping done in this patch. > > > > Do you mean this is not a regular PCIe MSI-X interrupt to the GIC, but > > something internal to your device that gets routed through the IOMMU > > back into the device? > > > > This is a regular PCIe MSI-X interrupt, the difference is that the > bunch of PCI devices > here doesn't have a fixed set of MSIX vectors. Admin function has a > memory region with > 32K MSIX vectors which it provisions to PCI devices based on the HW > functional blocks > attached to them. A PCI device which works as a ethernet device needs > X number of vectors > and a crypto device needs Y number of vectors. > > Since the admin function owns the whole MSIX vector region, HW uses > this device's stream ID > to access the vectors. Hence the IOMMU mapping. Once MSIX vectors are > provisioned to > a PCI device they work as normal MSIX interrupt like any other device. Ok, I think I got it now, just to confirm: the MSIX vectors you allocate in the admin device refer to memory backing the BAR that contains the MSI-X entries of the other functions, right? I was a bit confused here and assumed that you were mapping the MMIO area of an interrupt controller that receives the interupt transactions. Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA 2018-08-30 13:53 ` Arnd Bergmann @ 2018-08-30 18:39 ` Sunil Kovvuri 0 siblings, 0 replies; 6+ messages in thread From: Sunil Kovvuri @ 2018-08-30 18:39 UTC (permalink / raw) To: Arnd Bergmann Cc: jason, Marc Zyngier, linux-pci, LKML, LAKML, olof, tglx, linux-soc, Geetha sowjanya, Sunil Goutham On Thu, Aug 30, 2018 at 7:23 PM Arnd Bergmann <arnd@arndb.de> wrote: > > On Tue, Aug 28, 2018 at 3:17 PM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote: > > On Tue, Aug 28, 2018 at 6:27 PM Arnd Bergmann <arnd@arndb.de> wrote: > > > On Tue, Aug 28, 2018 at 2:42 PM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote: > > > > On Tue, Aug 28, 2018 at 5:39 PM Arnd Bergmann <arnd@arndb.de> wrote: > > > > > On Tue, Aug 28, 2018 at 12:58 PM <sunil.kovvuri@gmail.com> wrote: > > > > > > > > This admin function is a PCI device which is capable of provisioning > > > > HW blocks to other PCIe SRIOV devices in the system. Each HW block > > > > (eg memory buffer pools, NIC dewscriptors, crypto engines e.t.c) needs > > > > certain no of MSIX vectors. Admin function has a set of 32K MSIX vectors > > > > in memory (not on-chip) which based on HW block provisioning to a PCI device > > > > attaches the required number of vectors to that device. Some part of this > > > > configuration is done by low level firmware. > > > > > > > > RVU_AF_MSIXTR_BASE points to the memory region allocated for 32K MSIX > > > > vectors. If kernel is booted with IOMMU enabled and admin function device > > > > is attached to SMMU, HW will go through translation to access this MSIX > > > > vector memory region. Hence the mapping done in this patch. > > > > > > Do you mean this is not a regular PCIe MSI-X interrupt to the GIC, but > > > something internal to your device that gets routed through the IOMMU > > > back into the device? > > > > > > > This is a regular PCIe MSI-X interrupt, the difference is that the > > bunch of PCI devices > > here doesn't have a fixed set of MSIX vectors. Admin function has a > > memory region with > > 32K MSIX vectors which it provisions to PCI devices based on the HW > > functional blocks > > attached to them. A PCI device which works as a ethernet device needs > > X number of vectors > > and a crypto device needs Y number of vectors. > > > > Since the admin function owns the whole MSIX vector region, HW uses > > this device's stream ID > > to access the vectors. Hence the IOMMU mapping. Once MSIX vectors are > > provisioned to > > a PCI device they work as normal MSIX interrupt like any other device. > > Ok, I think I got it now, just to confirm: the MSIX vectors you allocate > in the admin device refer to memory backing the BAR that contains > the MSI-X entries of the other functions, right? Yes, that's correct. > > I was a bit confused here and assumed that you were mapping > the MMIO area of an interrupt controller that receives the interupt > transactions. > > Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-08-30 18:39 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <1535453838-12154-1-git-send-email-sunil.kovvuri@gmail.com>
[not found] ` <1535453838-12154-11-git-send-email-sunil.kovvuri@gmail.com>
2018-08-28 12:08 ` [PATCH 10/15] soc: octeontx2: Reconfig MSIX base with IOVA Arnd Bergmann
2018-08-28 12:42 ` Sunil Kovvuri
2018-08-28 12:57 ` Arnd Bergmann
2018-08-28 13:17 ` Sunil Kovvuri
2018-08-30 13:53 ` Arnd Bergmann
2018-08-30 18:39 ` Sunil Kovvuri
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).