* [PATCH v2 0/1] Fix IOMMU setup for hotplugged devices on pseries
@ 2019-09-05 19:13 Shawn Anastasio
2019-09-05 19:13 ` [PATCH v2 1/1] powerpc/pci: Fix pcibios_setup_device() ordering Shawn Anastasio
0 siblings, 1 reply; 5+ messages in thread
From: Shawn Anastasio @ 2019-09-05 19:13 UTC (permalink / raw)
To: linux-pci, linuxppc-dev; +Cc: sbobroff, aik, lukas, oohall, bhelgaas
Changes from v2:
- Remove pcibios_fixup_dev()
- Remove pcibios_setup_bus_device() call in pcibios_fixup_bus() since
all device setup will now be handled in pcibios_bus_add_device()
On pseries QEMU guests, IOMMU setup for hotplugged PCI devices is currently
broken for all but the first device on a given bus. The culprit is an ordering
issue in the pseries hotplug path (via pci_rescan_bus()) which results in IOMMU
group assigment occuring before device registration in sysfs. This triggers
the following check in arch/powerpc/kernel/iommu.c:
/*
* The sysfs entries should be populated before
* binding IOMMU group. If sysfs entries isn't
* ready, we simply bail.
*/
if (!device_is_registered(dev))
return -ENOENT;
This fails for hotplugged devices since the pcibios_add_device() call in the
pseries hotplug path (in pci_device_add()) occurs before device_add().
Since the IOMMU groups are set up in pcibios_add_device(), this means that a
sysfs entry will not yet be present and it will fail.
There is a special case that allows the first hotplugged device on a bus to
succeed, though. The powerpc pcibios_add_device() implementation will skip
initializing the device if bus setup is not yet complete.
Later, the pci core will call pcibios_fixup_bus() which will perform setup
for the first (and only) device on the bus and since it has already been
registered in sysfs, the IOMMU setup will succeed.
The current solution is to move all device setup to pcibios_bus_add_device()
which will occur after all devices have been registered.
Shawn Anastasio (1):
powerpc/pci: Fix pcibios_setup_device() ordering
arch/powerpc/kernel/pci-common.c | 25 +++++++++----------------
1 file changed, 9 insertions(+), 16 deletions(-)
--
2.20.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2 1/1] powerpc/pci: Fix pcibios_setup_device() ordering
2019-09-05 19:13 [PATCH v2 0/1] Fix IOMMU setup for hotplugged devices on pseries Shawn Anastasio
@ 2019-09-05 19:13 ` Shawn Anastasio
2019-09-09 7:59 ` Alexey Kardashevskiy
0 siblings, 1 reply; 5+ messages in thread
From: Shawn Anastasio @ 2019-09-05 19:13 UTC (permalink / raw)
To: linux-pci, linuxppc-dev; +Cc: sbobroff, aik, lukas, oohall, bhelgaas
Move PCI device setup from pcibios_add_device() and pcibios_fixup_bus() to
pcibios_bus_add_device(). This ensures that platform-specific DMA and IOMMU
setup occurs after the device has been registered in sysfs, which is a
requirement for IOMMU group assignment to work
This fixes IOMMU group assignment for hotplugged devices on pseries, where
the existing behavior results in IOMMU assignment before registration.
Thanks to Lukas Wunner <lukas@wunner.de> for the suggestion.
Signed-off-by: Shawn Anastasio <shawn@anastas.io>
---
arch/powerpc/kernel/pci-common.c | 25 +++++++++----------------
1 file changed, 9 insertions(+), 16 deletions(-)
diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c
index f627e15bb43c..d119c77efb69 100644
--- a/arch/powerpc/kernel/pci-common.c
+++ b/arch/powerpc/kernel/pci-common.c
@@ -261,12 +261,6 @@ int pcibios_sriov_disable(struct pci_dev *pdev)
#endif /* CONFIG_PCI_IOV */
-void pcibios_bus_add_device(struct pci_dev *pdev)
-{
- if (ppc_md.pcibios_bus_add_device)
- ppc_md.pcibios_bus_add_device(pdev);
-}
-
static resource_size_t pcibios_io_size(const struct pci_controller *hose)
{
#ifdef CONFIG_PPC64
@@ -987,15 +981,17 @@ static void pcibios_setup_device(struct pci_dev *dev)
ppc_md.pci_irq_fixup(dev);
}
-int pcibios_add_device(struct pci_dev *dev)
+void pcibios_bus_add_device(struct pci_dev *pdev)
{
- /*
- * We can only call pcibios_setup_device() after bus setup is complete,
- * since some of the platform specific DMA setup code depends on it.
- */
- if (dev->bus->is_added)
- pcibios_setup_device(dev);
+ /* Perform platform-specific device setup */
+ pcibios_setup_device(pdev);
+
+ if (ppc_md.pcibios_bus_add_device)
+ ppc_md.pcibios_bus_add_device(pdev);
+}
+int pcibios_add_device(struct pci_dev *dev)
+{
#ifdef CONFIG_PCI_IOV
if (ppc_md.pcibios_fixup_sriov)
ppc_md.pcibios_fixup_sriov(dev);
@@ -1037,9 +1033,6 @@ void pcibios_fixup_bus(struct pci_bus *bus)
/* Now fixup the bus bus */
pcibios_setup_bus_self(bus);
-
- /* Now fixup devices on that bus */
- pcibios_setup_bus_devices(bus);
}
EXPORT_SYMBOL(pcibios_fixup_bus);
--
2.20.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH v2 1/1] powerpc/pci: Fix pcibios_setup_device() ordering
2019-09-05 19:13 ` [PATCH v2 1/1] powerpc/pci: Fix pcibios_setup_device() ordering Shawn Anastasio
@ 2019-09-09 7:59 ` Alexey Kardashevskiy
2019-09-27 18:33 ` Shawn Anastasio
0 siblings, 1 reply; 5+ messages in thread
From: Alexey Kardashevskiy @ 2019-09-09 7:59 UTC (permalink / raw)
To: Shawn Anastasio, linux-pci, linuxppc-dev
Cc: sbobroff, lukas, oohall, bhelgaas
On 06/09/2019 05:13, Shawn Anastasio wrote:
> Move PCI device setup from pcibios_add_device() and pcibios_fixup_bus() to
> pcibios_bus_add_device(). This ensures that platform-specific DMA and IOMMU
> setup occurs after the device has been registered in sysfs, which is a
> requirement for IOMMU group assignment to work
>
> This fixes IOMMU group assignment for hotplugged devices on pseries, where
> the existing behavior results in IOMMU assignment before registration.
Although this is a correct approach which we should proceed with, this
breaks adding of SRIOV VFs from pnv_tce_iommu_bus_notifier (and possibly
the bare metal PCI hotplug), I am trying to fix that now...
>
> Thanks to Lukas Wunner <lukas@wunner.de> for the suggestion.
>
> Signed-off-by: Shawn Anastasio <shawn@anastas.io>
> ---
> arch/powerpc/kernel/pci-common.c | 25 +++++++++----------------
> 1 file changed, 9 insertions(+), 16 deletions(-)
>
> diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c
> index f627e15bb43c..d119c77efb69 100644
> --- a/arch/powerpc/kernel/pci-common.c
> +++ b/arch/powerpc/kernel/pci-common.c
> @@ -261,12 +261,6 @@ int pcibios_sriov_disable(struct pci_dev *pdev)
>
> #endif /* CONFIG_PCI_IOV */
>
> -void pcibios_bus_add_device(struct pci_dev *pdev)
> -{
> - if (ppc_md.pcibios_bus_add_device)
> - ppc_md.pcibios_bus_add_device(pdev);
> -}
> -
> static resource_size_t pcibios_io_size(const struct pci_controller *hose)
> {
> #ifdef CONFIG_PPC64
> @@ -987,15 +981,17 @@ static void pcibios_setup_device(struct pci_dev *dev)
> ppc_md.pci_irq_fixup(dev);
> }
>
> -int pcibios_add_device(struct pci_dev *dev)
> +void pcibios_bus_add_device(struct pci_dev *pdev)
> {
> - /*
> - * We can only call pcibios_setup_device() after bus setup is complete,
> - * since some of the platform specific DMA setup code depends on it.
> - */
> - if (dev->bus->is_added)
> - pcibios_setup_device(dev);
> + /* Perform platform-specific device setup */
> + pcibios_setup_device(pdev);
> +
> + if (ppc_md.pcibios_bus_add_device)
> + ppc_md.pcibios_bus_add_device(pdev);
> +}
>
> +int pcibios_add_device(struct pci_dev *dev)
> +{
> #ifdef CONFIG_PCI_IOV
> if (ppc_md.pcibios_fixup_sriov)
> ppc_md.pcibios_fixup_sriov(dev);
> @@ -1037,9 +1033,6 @@ void pcibios_fixup_bus(struct pci_bus *bus)
>
> /* Now fixup the bus bus */
> pcibios_setup_bus_self(bus);
> -
> - /* Now fixup devices on that bus */
> - pcibios_setup_bus_devices(bus);
> }
> EXPORT_SYMBOL(pcibios_fixup_bus);
>
>
--
Alexey
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v2 1/1] powerpc/pci: Fix pcibios_setup_device() ordering
2019-09-09 7:59 ` Alexey Kardashevskiy
@ 2019-09-27 18:33 ` Shawn Anastasio
2019-09-28 12:18 ` Alexey Kardashevskiy
0 siblings, 1 reply; 5+ messages in thread
From: Shawn Anastasio @ 2019-09-27 18:33 UTC (permalink / raw)
To: Alexey Kardashevskiy, linux-pci, linuxppc-dev
Cc: sbobroff, lukas, oohall, bhelgaas
On 9/9/19 2:59 AM, Alexey Kardashevskiy wrote:
>
>
> On 06/09/2019 05:13, Shawn Anastasio wrote:
>> Move PCI device setup from pcibios_add_device() and pcibios_fixup_bus() to
>> pcibios_bus_add_device(). This ensures that platform-specific DMA and IOMMU
>> setup occurs after the device has been registered in sysfs, which is a
>> requirement for IOMMU group assignment to work
>>
>> This fixes IOMMU group assignment for hotplugged devices on pseries, where
>> the existing behavior results in IOMMU assignment before registration.
>
>
> Although this is a correct approach which we should proceed with, this
> breaks adding of SRIOV VFs from pnv_tce_iommu_bus_notifier (and possibly
> the bare metal PCI hotplug), I am trying to fix that now...
Were you able to make any progress? I can think of a couple of ways
to fix SRIOV, but they're not particularly elegant and involve
duplication.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2 1/1] powerpc/pci: Fix pcibios_setup_device() ordering
2019-09-27 18:33 ` Shawn Anastasio
@ 2019-09-28 12:18 ` Alexey Kardashevskiy
0 siblings, 0 replies; 5+ messages in thread
From: Alexey Kardashevskiy @ 2019-09-28 12:18 UTC (permalink / raw)
To: Shawn Anastasio, linux-pci, linuxppc-dev
Cc: sbobroff, lukas, oohall, bhelgaas
On 28/09/2019 04:33, Shawn Anastasio wrote:
> On 9/9/19 2:59 AM, Alexey Kardashevskiy wrote:
>>
>>
>> On 06/09/2019 05:13, Shawn Anastasio wrote:
>>> Move PCI device setup from pcibios_add_device() and pcibios_fixup_bus() to
>>> pcibios_bus_add_device(). This ensures that platform-specific DMA and IOMMU
>>> setup occurs after the device has been registered in sysfs, which is a
>>> requirement for IOMMU group assignment to work
>>>
>>> This fixes IOMMU group assignment for hotplugged devices on pseries, where
>>> the existing behavior results in IOMMU assignment before registration.
>>
>>
>> Although this is a correct approach which we should proceed with, this
>> breaks adding of SRIOV VFs from pnv_tce_iommu_bus_notifier (and possibly
>> the bare metal PCI hotplug), I am trying to fix that now...
>
> Were you able to make any progress? I can think of a couple of ways
> to fix SRIOV, but they're not particularly elegant and involve
> duplication.
A bigger change for ppc+pci is coming from Oliver (I guess with your patch as well) which will tackle this one too, soon.
--
Alexey
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-09-28 12:21 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-05 19:13 [PATCH v2 0/1] Fix IOMMU setup for hotplugged devices on pseries Shawn Anastasio
2019-09-05 19:13 ` [PATCH v2 1/1] powerpc/pci: Fix pcibios_setup_device() ordering Shawn Anastasio
2019-09-09 7:59 ` Alexey Kardashevskiy
2019-09-27 18:33 ` Shawn Anastasio
2019-09-28 12:18 ` Alexey Kardashevskiy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).