archive mirror
 help / color / mirror / Atom feed
From: Bjorn Helgaas <>
To: Andrey Grodzovsky <>
Cc: "Deucher, Alexander" <>,
	"" <>,
	"" <>,
	"Koenig, Christian" <>,
	"Antonovitch, Anatoli" <>
Subject: Re: Avoid MMIO write access after hot unplug [WAS - Re: Question about supporting AMD eGPU hot plug case]
Date: Fri, 5 Feb 2021 09:38:09 -0600	[thread overview]
Message-ID: <20210205153809.GA179207@bjorn-Precision-5520> (raw)
In-Reply-To: <>

On Thu, Feb 04, 2021 at 11:03:10AM -0500, Andrey Grodzovsky wrote:
> + linux-pci mailing list and a bit of extra details bellow.
> On 2/2/21 12:51 PM, Andrey Grodzovsky wrote:
> > Bjorn, Sergey I would also like to use this opportunity to ask you a
> > question on a related topic - Hot unplug.
> > I've been working for a while on this (see latest patchset set here
> >
> > My question is specifically regarding this patch
> >
> > - the idea here is to
> > prevent any accesses to MMIO address ranges still mapped in kernel page

I think you're talking about a PCI BAR that is mapped into a user

It is impossible to reliably *prevent* MMIO accesses to a BAR on a
PCI device that has been unplugged.  There is always a window where
the CPU issues a load/store and the device is unplugged before the
load/store completes.

If a PCIe device is unplugged, an MMIO read to that BAR will complete
on PCIe with an uncorrectable fatal error.  When that happens there is
no valid data from the PCIe device, so the PCIe host bridge typically
fabricates ~0 data so the CPU load instruction can complete.

If you want to reliably recover from unplugs like this, I think you
have to check for that ~0 data at the important points, i.e., where
you make decisions based on the data.  Of course, ~0 may be valid data
in some cases.  You have to use your knowledge of the device
programming model to determine whether ~0 is possible, and if it is,
check in some other way, like another MMIO read, to tell whether the
read succeeded and returned ~0, or failed because of PCIe error and
returned ~0.

> > table by the driver AFTER the device is gone physically and from the
> > PCI  subsystem, post pci_driver.remove
> > call back execution. This happens because struct device (and struct
> > drm_device representing the graphic card) are still present because some
> > user clients which  are not aware
> > of hot removal still hold device file open and hence prevents device
> > refcount to drop to 0. The solution in this patch is brute force where
> > we try and find any place we access MMIO
> > mapped to kernel address space and guard against the write access with a
> > 'device-unplug' flag. This solution is obliviously racy because a device
> > can be unplugged right after checking the falg.
> > I had an idea to instead not release but to keep those ranges reserved
> > even after pci_driver.remove, until DRM device is destroyed when it's
> > refcount drops to 0 and by this to prevent new devices plugged in
> > and allocated some of the same MMIO address  range to get accidental
> > writes into their registers.
> > But, on dri-devel I was advised that this will upset the PCI subsystem
> > and so best to be avoided but I still would like another opinion from
> > PCI experts on whether this approach is indeed not possible ?
> > 
> > Andrey

       reply	other threads:[~2021-02-05 16:05 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <>
2021-02-05 15:38 ` Bjorn Helgaas [this message]
2021-02-05 16:08   ` Avoid MMIO write access after hot unplug [WAS - Re: Question about supporting AMD eGPU hot plug case] Andrey Grodzovsky
2021-02-05 17:59     ` Daniel Vetter
2021-02-05 19:09       ` Andrey Grodzovsky
2021-02-05 19:45     ` Bjorn Helgaas
2021-02-05 20:42       ` Andrey Grodzovsky
2021-02-05 20:49         ` Daniel Vetter
2021-02-05 21:24           ` Andrey Grodzovsky
2021-02-05 22:15             ` Daniel Vetter
2021-02-05 21:35         ` Keith Busch
2021-02-05 21:40           ` Andrey Grodzovsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210205153809.GA179207@bjorn-Precision-5520 \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).