Re: [PATCH] Ensure pci transactions coming from PLX NTB are handled when IOMMU is turned on

From: Bjorn Helgaas <helgaas@kernel.org>
To: Logan Gunthorpe <logang@deltatee.com>
Cc: Dmitry Safonov <0x7f454c46@gmail.com>,
	James Sewart <jamessewart@arista.com>,
	iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
	Dmitry Safonov <dima@arista.com>,
	linux-pci@vger.kernel.org,
	Alex Williamson <alex.williamson@redhat.com>
Subject: Re: [PATCH] Ensure pci transactions coming from PLX NTB are handled when IOMMU is turned on
Date: Wed, 20 Nov 2019 13:49:48 -0600	[thread overview]
Message-ID: <20191120194948.GA117003@google.com> (raw)
In-Reply-To: <c683ec65-40a9-6430-ae85-ce3934ed8af5@deltatee.com>

On Wed, Nov 20, 2019 at 12:30:48PM -0700, Logan Gunthorpe wrote:
> On 2019-11-20 10:48 a.m., Dmitry Safonov wrote:
> > On 11/5/19 12:17 PM, James Sewart wrote:
> >>
> >>> On 24 Oct 2019, at 13:52, James Sewart <jamessewart@arista.com> wrote:
> >>>
> >>> The PLX PEX NTB forwards DMA transactions using Requester ID's that don't exist as
> >>> PCI devices. The devfn for a transaction is used as an index into a lookup table
> >>> storing the origin of a transaction on the other side of the bridge.
> >>>
> >>> This patch aliases all possible devfn's to the NTB device so that any transaction
> >>> coming in is governed by the mappings for the NTB.
> >>>
> >>> Signed-Off-By: James Sewart <jamessewart@arista.com>
> >>> ---
> >>> drivers/pci/quirks.c | 22 ++++++++++++++++++++++
> >>> 1 file changed, 22 insertions(+)
> >>>
> >>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> >>> index 320255e5e8f8..647f546e427f 100644
> >>> --- a/drivers/pci/quirks.c
> >>> +++ b/drivers/pci/quirks.c
> >>> @@ -5315,6 +5315,28 @@ SWITCHTEC_QUIRK(0x8574);  /* PFXI 64XG3 */
> >>> SWITCHTEC_QUIRK(0x8575);  /* PFXI 80XG3 */
> >>> SWITCHTEC_QUIRK(0x8576);  /* PFXI 96XG3 */
> >>>
> >>> +/*
> >>> + * PLX NTB uses devfn proxy IDs to move TLPs between NT endpoints. These IDs
> >>> + * are used to forward responses to the originator on the other side of the
> >>> + * NTB. Alias all possible IDs to the NTB to permit access when the IOMMU is
> >>> + * turned on.
> >>> + */
> >>> +static void quirk_PLX_NTB_dma_alias(struct pci_dev *pdev)
> >>> +{
> >>> +	if (!pdev->dma_alias_mask)
> >>> +		pdev->dma_alias_mask = kcalloc(BITS_TO_LONGS(U8_MAX),
> >>> +					      sizeof(long), GFP_KERNEL);
> >>> +	if (!pdev->dma_alias_mask) {
> >>> +		dev_warn(&pdev->dev, "Unable to allocate DMA alias mask\n");
> >>> +		return;
> >>> +	}
> >>> +
> >>> +	// PLX NTB may use all 256 devfns
> >>> +	memset(pdev->dma_alias_mask, U8_MAX, (U8_MAX+1)/BITS_PER_BYTE);
> 
> I think it would be better to create a pci_add_dma_alias_range()
> function instead of directly accessing dma_alias_mask. We could then use
> that function to clean up quirk_switchtec_ntb_dma_alias() which is
> essentially doing the same thing.

Great idea!

Bjorn