From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933061AbeEWNd1 (ORCPT ); Wed, 23 May 2018 09:33:27 -0400 Received: from mail.kernel.org ([198.145.29.99]:46448 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932969AbeEWNdX (ORCPT ); Wed, 23 May 2018 09:33:23 -0400 Date: Wed, 23 May 2018 08:33:21 -0500 From: Bjorn Helgaas To: Logan Gunthorpe Cc: Doug Meyer , kurt.schwemmer@microsemi.com, linux-pci@vger.kernel.org, linux-ntb , Bjorn Helgaas , Jon Mason , "Jiang, Dave" , Allen Hubbe , linux-kernel@vger.kernel.org, Alex Williamson Subject: Re: [PATCH 2/2] NTB: PCI Quirk to Enable Switchtec NT Functionality with IOMMU On Message-ID: <20180523133321.GA150632@bhelgaas-glaptop.roam.corp.google.com> References: <1526558413-23113-1-git-send-email-dmeyer@gigaio.com> <1526558413-23113-3-git-send-email-dmeyer@gigaio.com> <20180517134521.GC19955@bhelgaas-glaptop.roam.corp.google.com> <9dd68ac0-5b9e-b257-49d2-20327c2ab282@deltatee.com> <20180522215126.GA22385@bhelgaas-glaptop.roam.corp.google.com> <598f5880-22b2-bbb4-4a37-5a60bae2348b@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <598f5880-22b2-bbb4-4a37-5a60bae2348b@deltatee.com> User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, May 22, 2018 at 04:23:13PM -0600, Logan Gunthorpe wrote: > On 22/05/18 03:51 PM, Bjorn Helgaas wrote: > > I don't think the question of when the aliases need to be added is > > quite closed. Logan said "it seems pci_add_dma_alias() must be called > > before the driver is initialized and therefore in a quirk", but that > > doesn't make clear *why* the alias needs to be added before the driver > > is initialized. The alias shouldn't be needed until the device does a > > DMA, and it shouldn't do that until after the driver initializes. > > No, Doug tried it in the driver first and it didn't work. The symbol is > also not exported which was probably done because it can't be used in > the driver. > > > I suspect the reason the existing quirks are in drivers/pci/quirks.c > > is because the IOMMU driver is in the host OS, but the host may not > > have a driver for the device if the device is passed through to a > > guest OS. In that case, the only way to add the alias is by using a > > quirk that is always built into the host OS. > > Digging into the code a bit, it's not because it must be done by the > Host OS but because it must be done before the IOMMU groups are created. > The IOMMU code registers a bus_notifier and creates the groups based on > the dma_alias mask when it receives the BUS_NOTIFY_ADD_DEVICE event. > This event is notified in device_add() just before a call to > bus_probe_device()[1]. Therefore, if a driver attempts to use > pci_add_dma_alias() as part of it's probe routine, it will be too late > as the IOMMU has already setup the groups based on the original version > of the dma_alias_mask. This (and Alex's) analysis is very useful and I'd like to capture it somehow, perhaps by expanding the poor pci_add_dma_alias() function comment I added with f0af9593372a ("PCI: Add pci_add_dma_alias() to abstract implementation"). The admonition to "call early" without any details about *how* early or *why* it needs to be called early is not really very useful. If we added your analysis, it would be a great help to anybody who reworks IOMMU groups in the future. > I suspect this is by design as the groups must be created before and any > dma_maps are done on the device and some drivers may create dma_maps > during probe. > > Logan > > [1] > https://elixir.bootlin.com/linux/v4.17-rc6/source/drivers/base/core.c#L1863