From: Michael Ellerman <patch-notifications@ellerman.id.au>
To: Oliver O'Halloran <oohall@gmail.com>, linuxppc-dev@lists.ozlabs.org
Cc: aik@ozlabs.ru, linux-pci@vger.kernel.org,
Oliver O'Halloran <oohall@gmail.com>,
shawn@anastas.io
Subject: Re: [PATCH v2 1/3] powernv/iov: Ensure the pdn for VFs always contains a valid PE number
Date: Tue, 7 Jan 2020 10:33:16 +1100 (AEDT) [thread overview]
Message-ID: <47sBdm0wtzz9sRR@ozlabs.org> (raw)
In-Reply-To: <20191028085424.12006-1-oohall@gmail.com>
On Mon, 2019-10-28 at 08:54:22 UTC, Oliver O'Halloran wrote:
> On pseries there is a bug with adding hotplugged devices to an IOMMU group.
> For a number of dumb reasons fixing that bug first requires re-working how
> VFs are configured on PowerNV. For background, on PowerNV we use the
> pcibios_sriov_enable() hook to do two things:
>
> 1. Create a pci_dn structure for each of the VFs, and
> 2. Configure the PHB's internal BARs so the MMIO range for each VF
> maps to a unique PE.
>
> Roughly speaking a PE is the hardware counterpart to a Linux IOMMU group
> since all the devices in a PE share the same IOMMU table. A PE also defines
> the set of devices that should be isolated in response to a PCI error (i.e.
> bad DMA, UR/CA, AER events, etc). When isolated all MMIO and DMA traffic to
> and from devicein the PE is blocked by the root complex until the PE is
> recovered by the OS.
>
> The requirement to block MMIO causes a giant headache because the P8 PHB
> generally uses a fixed mapping between MMIO addresses and PEs. As a result
> we need to delay configuring the IOMMU groups for device until after MMIO
> resources are assigned. For physical devices (i.e. non-VFs) the PE
> assignment is done in pcibios_setup_bridge() which is called immediately
> after the MMIO resources for downstream devices (and the bridge's windows)
> are assigned. For VFs the setup is more complicated because:
>
> a) pcibios_setup_bridge() is not called again when VFs are activated, and
> b) The pci_dev for VFs are created by generic code which runs after
> pcibios_sriov_enable() is called.
>
> The work around for this is a two step process:
>
> 1. A fixup in pcibios_add_device() is used to initialised the cached
> pe_number in pci_dn, then
> 2. A bus notifier then adds the device to the IOMMU group for the PE
> specified in pci_dn->pe_number.
>
> A side effect fixing the pseries bug mentioned in the first paragraph is
> moving the fixup out of pcibios_add_device() and into
> pcibios_bus_add_device(), which is called much later. This results in step
> 2. failing because pci_dn->pe_number won't be initialised when the bus
> notifier is run.
>
> We can fix this by removing the need for the fixup. The PE for a VF is
> known before the VF is even scanned so we can initialise pci_dn->pe_number
> pcibios_sriov_enable() instead. Unfortunately, moving the initialisation
> causes two problems:
>
> 1. We trip the WARN_ON() in the current fixup code, and
> 2. The EEH core clears pdn->pe_number when recovering a VF and relies
> on the fixup to correctly re-set it.
>
> The only justification for either of these is a comment in eeh_rmv_device()
> suggesting that pdn->pe_number *must* be set to IODA_INVALID_PE in order
> for the VF to be scanned. However, this comment appears to have no basis
> in reality. Both bugs can be fixed by just deleting the code.
>
> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> Signed-off-by: Oliver O'Halloran <oohall@gmail.com>
Series applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/3b5b9997b331e77ce967eba2c4bc80dc3134a7fe
cheers
prev parent reply other threads:[~2020-01-06 23:37 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-28 8:54 [PATCH v2 1/3] powernv/iov: Ensure the pdn for VFs always contains a valid PE number Oliver O'Halloran
2019-10-28 8:54 ` [PATCH v2 2/3] powerpc/pci: Fix pcibios_setup_device() ordering Oliver O'Halloran
2019-10-28 8:54 ` [PATCH v2 3/3] powerpc/pci: Remove pcibios_setup_bus_devices() Oliver O'Halloran
2019-12-03 23:45 ` [PATCH v2 1/3] powernv/iov: Ensure the pdn for VFs always contains a valid PE number Shawn Anastasio
2020-01-06 23:33 ` Michael Ellerman [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47sBdm0wtzz9sRR@ozlabs.org \
--to=patch-notifications@ellerman.id.au \
--cc=aik@ozlabs.ru \
--cc=linux-pci@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=oohall@gmail.com \
--cc=shawn@anastas.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).