From: Bjorn Helgaas <helgaas@kernel.org>
To: Huacai Chen <chenhuacai@gmail.com>
Cc: "Huacai Chen" <chenhuacai@loongson.cn>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Lorenzo Pieralisi" <lorenzo.pieralisi@arm.com>,
"Rob Herring" <robh@kernel.org>,
"Krzysztof Wilczyński" <kw@linux.com>,
linux-pci <linux-pci@vger.kernel.org>,
"Xuefeng Li" <lixuefeng@loongson.cn>,
"Jiaxun Yang" <jiaxun.yang@flygoat.com>
Subject: Re: [PATCH V13 5/6] PCI: Add quirk for LS7A to avoid reboot failure
Date: Fri, 17 Jun 2022 06:37:08 -0500 [thread overview]
Message-ID: <20220617113708.GA1177054@bhelgaas> (raw)
In-Reply-To: <CAAhV-H7G5BN4n-jRnfVYOikkk0pCt=ZLB0MW=iKu+s07gieKXg@mail.gmail.com>
On Fri, Jun 17, 2022 at 10:21:14AM +0800, Huacai Chen wrote:
> On Fri, Jun 17, 2022 at 6:57 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Thu, Jun 16, 2022 at 04:39:46PM +0800, Huacai Chen wrote:
> > > On Thu, Jun 9, 2022 at 3:31 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > On Wed, Jun 08, 2022 at 05:34:21PM +0800, Huacai Chen wrote:
> > > > > On Fri, Jun 3, 2022 at 12:29 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > On Thu, Jun 02, 2022 at 08:48:20PM +0800, Huacai Chen wrote:
> > > > > > > On Wed, Jun 1, 2022 at 7:35 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > > > On Sat, Apr 30, 2022 at 04:48:45PM +0800, Huacai Chen wrote:
> > > > > > > > > Commit cc27b735ad3a75574a ("PCI/portdrv: Turn off PCIe
> > > > > > > > > services during shutdown") causes poweroff/reboot
> > > > > > > > > failure on systems with LS7A chipset. We found that if
> > > > > > > > > we remove "pci_command &= ~PCI_COMMAND_MASTER;" in
> > > > > > > > > do_pci_disable_device(), it can work well. The hardware
> > > > > > > > > engineer says that the root cause is that CPU is still
> > > > > > > > > accessing PCIe devices while poweroff/reboot, and if we
> > > > > > > > > disable the Bus Master Bit at this time, the PCIe
> > > > > > > > > controller doesn't forward requests to downstream
> > > > > > > > > devices, and also doesn't send TIMEOUT to CPU, which
> > > > > > > > > causes CPU wait forever (hardware deadlock). This
> > > > > > > > > behavior is a PCIe protocol violation (Bus Master should
> > > > > > > > > not be involved in CPU MMIO transactions), and it will
> > > > > > > > > be fixed in new revisions of hardware (add timeout
> > > > > > > > > mechanism for CPU read request, whether or not Bus
> > > > > > > > > Master bit is cleared).
> > > > > > > >
> > > > > > > > LS7A might have bugs in that clearing Bus Master Enable
> > > > > > > > prevents the root port from forwarding Memory or I/O
> > > > > > > > requests in the downstream direction.
> > > > > > > >
> > > > > > > > But this feels like a bit of a band-aid because we don't
> > > > > > > > know exactly what those requests are. If we're removing
> > > > > > > > the Root Port, I assume we think we no longer need any
> > > > > > > > devices *below* the Root Port.
> > > > > > > >
> > > > > > > > If that's not the case, e.g., if we still need to produce
> > > > > > > > console output or save state to a device, we probably
> > > > > > > > should not be removing the Root Port at all.
> > > > > > >
> > > > > > > Do you mean it is better to skip the whole
> > > > > > > pcie_port_device_remove() instead of just removing the
> > > > > > > "clear bus master" operation for the buggy hardware?
> > > > > >
> > > > > > No, that's not what I want at all. That's just another
> > > > > > band-aid to avoid a problem without understanding what the
> > > > > > problem is.
> > > > > >
> > > > > > My point is that apparently we remove a Root Port (which means
> > > > > > we've already removed any devices under it), and then we try
> > > > > > to use a device below the Root Port. That seems broken. I
> > > > > > want to understand why we try to use a device after we've
> > > > > > removed it.
> > > > >
> > > > > I agree, and I think "why we try to use a device after remove
> > > > > it" is because the userspace programs don't know whether a
> > > > > device is "usable", so they just use it, at any time. Then it
> > > > > seems it is the responsibility of the device drivers to avoid
> > > > > the problem.
> > > >
> > > > How is userspace able to use a device after the device is removed?
> > > > E.g., if userspace does a read/write to a device that has been
> > > > removed, the syscall should return error, not touch the missing
> > > > device. If userspace mmaps a device, an access after the device
> > > > has been removed should fault, not do MMIO to the missing device.
> > >
> > > To give more details, let's take the graphics driver (e.g. amdgpu)
> > > as an example again. The userspace programs call printf() to display
> > > "shutting down xxx service" during shutdown/reboot. Or we can even
> > > simplify further, the kernel calls printk() to display something
> > > during shutdown/reboot. You know, printk() can happen at any time,
> > > even after we call pcie_port_device_remove() to disable the pcie
> > > port on the graphic card.
> >
> > I've been focusing on the *remove* path, but you said the problem
> > you're solving is with *poweroff/reboot*. pcie_portdrv_remove() is
> > used for both paths, but if there's a reason we need those paths to be
> > different, we might be able to split them.
>
> I'm very sorry for that. I have misunderstood before because I suppose
> the "remove path" is the pcie_portdrv_remove() function, but your
> meaning is the .remove() callback in pcie_portdriver. Am I right this
> time?
No need to be sorry, you clearly said from the beginning that this was
a shutdown issue, not a remove issue! I was just confused because the
.remove() and the .shutdown() callbacks are both
pcie_portdrv_remove(), so I was thinking "remove" even though you said
"poweroff".
> > For remove, we have to assume accesses to the device may already or
> > will soon fail. A driver that touches the device, or a device that
> > performs DMA, after its drv->remove() method has been called would be
> > seriously broken. The remove operation also unbinds the driver from
> > the device.
>
> Then what will happen about the "remove path"? If we still take the
> graphics driver as an example, "rmmod amdgpu" always fails with
> "device is busy" because the graphics card is always be used once
> after the driver is loaded. So the "remove path" has no chance to be
> executed.
Do you think this is a problem? It doesn't sound like a problem to
me, but I don't know anything about graphics drivers. I assume that
if a device is in use, the expected behavior is that we can't remove
the driver.
> But if we take a NIC driver as an example, "rmmod igb" can
> mostly succeed, and there will be no access on the device after
> removing, at least in our observation. I think there is nothing broken
> about the "remove path".
I agree.
Bjorn
next prev parent reply other threads:[~2022-06-17 11:37 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-30 8:48 [PATCH V13 0/6] PCI: Loongson pci improvements and quirks Huacai Chen
2022-04-30 8:48 ` [PATCH V13 1/6] PCI: loongson: Use generic 8/16/32-bit config ops on LS2K/LS7A Huacai Chen
2022-06-01 2:08 ` Bjorn Helgaas
2022-06-02 4:18 ` Huacai Chen
2022-04-30 8:48 ` [PATCH V13 2/6] PCI: loongson: Add ACPI init support Huacai Chen
2022-05-31 23:04 ` Bjorn Helgaas
2022-06-02 7:09 ` Huacai Chen
2022-04-30 8:48 ` [PATCH V13 3/6] PCI: loongson: Don't access unexisting devices Huacai Chen
2022-05-31 23:14 ` Bjorn Helgaas
2022-06-02 4:28 ` Huacai Chen
2022-06-02 16:23 ` Bjorn Helgaas
2022-06-02 20:00 ` Jiaxun Yang
2022-04-30 8:48 ` [PATCH V13 4/6] PCI: loongson: Improve the MRRS quirk for LS7A Huacai Chen
2022-06-01 2:22 ` Bjorn Helgaas
2022-06-01 11:59 ` Jiaxun Yang
2022-06-02 4:17 ` Huacai Chen
2022-06-02 16:20 ` Bjorn Helgaas
2022-06-03 12:13 ` Krzysztof Hałasa
2022-06-03 22:57 ` Jiaxun Yang
2022-06-04 0:07 ` Bjorn Helgaas
2022-06-08 8:29 ` Huacai Chen
2022-04-30 8:48 ` [PATCH V13 5/6] PCI: Add quirk for LS7A to avoid reboot failure Huacai Chen
2022-05-31 23:35 ` Bjorn Helgaas
2022-06-02 12:48 ` Huacai Chen
2022-06-02 16:29 ` Bjorn Helgaas
2022-06-08 9:34 ` Huacai Chen
2022-06-08 19:31 ` Bjorn Helgaas
2022-06-16 8:39 ` Huacai Chen
2022-06-16 22:57 ` Bjorn Helgaas
2022-06-17 2:21 ` Huacai Chen
2022-06-17 11:37 ` Bjorn Helgaas [this message]
2022-06-17 12:14 ` Huacai Chen
2022-04-30 8:48 ` [PATCH V13 6/6] PCI: Add quirk for multifunction devices of LS7A Huacai Chen
2022-06-01 2:07 ` Bjorn Helgaas
2022-06-01 7:36 ` Jianmin Lv
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220617113708.GA1177054@bhelgaas \
--to=helgaas@kernel.org \
--cc=bhelgaas@google.com \
--cc=chenhuacai@gmail.com \
--cc=chenhuacai@loongson.cn \
--cc=jiaxun.yang@flygoat.com \
--cc=kw@linux.com \
--cc=linux-pci@vger.kernel.org \
--cc=lixuefeng@loongson.cn \
--cc=lorenzo.pieralisi@arm.com \
--cc=robh@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).