Re: [PATCH] PCI: Avoid FLR for AMD Starship USB 3.0

[Marcos Scriven <marcos@scriven.org>]

On Thu, 28 May 2020 at 09:12, Marcos Scriven <marcos@scriven.org> wrote:
>
> On Wed, 27 May 2020 at 22:42, Deucher, Alexander
> <Alexander.Deucher@amd.com> wrote:
> >
> > [AMD Official Use Only - Internal Distribution Only]
> >
> > > -----Original Message-----
> > > From: Bjorn Helgaas <helgaas@kernel.org>
> > > Sent: Wednesday, May 27, 2020 5:32 PM
> > > To: Kevin Buettner <kevinb@redhat.com>
> > > Cc: linux-pci@vger.kernel.org; Bjorn Helgaas <bhelgaas@google.com>; Alex
> > > Williamson <alex.williamson@redhat.com>; Deucher, Alexander
> > > <Alexander.Deucher@amd.com>; Koenig, Christian
> > > <Christian.Koenig@amd.com>
> > > Subject: Re: [PATCH] PCI: Avoid FLR for AMD Starship USB 3.0
> > >
> > > [+cc Alex D, Christian -- do you guys have any contacts or insight into why we
> > > suddenly have three new AMD devices that advertise FLR support but it
> > > doesn't work?  Are we doing something wrong in Linux, or are these devices
> > > defective?
> >
> > +Nehal who handles our USB drivers.
> >
> > Nehal any ideas about FLR or whether it should be advertised?
> >
> > Alex
> >
>
> I had read somewhere that the IO die in the Ryzen/Threadripper
> packages are identical to the ones used in the motherboard chipsets.
>
> Since the latter do reset ok, it would seem a BIOS update of the AGESA
> may potentially fix the issue.
>
> Unfortunately, it's not something motherboard manufacturer's customer
> support people know how to deal with or pass back up the chain to AMD
> engineers. Actual use of this feature seems to be fairly niche.
>
> After I added the workaround for the USB and audio controllers on the
> 3rd-gen Ryzen, I tried contacting Kim Phillips (who I found as a
> kernel committer to x86/cpu/amd), but haven't heard back.
>
> It would be wonderful to know if this can potentially be fixed in CPU
> firmware, and whether there's any likelihood of it actually being
> distributed by motherboard manufacturers.
>
> Marcos
>
>
>

Dear Alex/Nehal

I wonder if you're able to comment please on whether FLR should be advertised?

Is there any chance this could be fixed at the bios/AGESA level, and
effectively rolled out?

Thanks

Marcos

> > >
> > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> > > kernel.org%2Fr%2F20200524003529.598434ff%40f31-
> > > 4.lan&amp;data=02%7C01%7Calexander.deucher%40amd.com%7Ccb77b56b
> > > 62ae47f60f8808d802855759%7C3dd8961fe4884e608e11a82d994e183d%7C0%
> > > 7C0%7C637262119015438912&amp;sdata=3z%2Btn%2Bv2pvUl3X0Tzk%2BLoi
> > > Mk06dLZCmgUOrsGf3kLpY%3D&amp;reserved=0
> > >   AMD Starship USB 3.0 host controller
> > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> > > kernel.org%2Fr%2FCAAri2DpkcuQZYbT6XsALhx2e6vRqPHwtbjHYeiH7MNp4z
> > > mt1RA%40mail.gmail.com&amp;data=02%7C01%7Calexander.deucher%40a
> > > md.com%7Ccb77b56b62ae47f60f8808d802855759%7C3dd8961fe4884e608e11
> > > a82d994e183d%7C0%7C0%7C637262119015438912&amp;sdata=69GsHB0HCp
> > > 6x0xW0tA%2FrAln0Vy0Yc9I8QSHowebdIxI%3D&amp;reserved=0
> > >   AMD Matisse HD Audio & USB 3.0 host controller ]
> > >
> > > On Sun, May 24, 2020 at 12:35:29AM -0700, Kevin Buettner wrote:
> > > > This commit adds an entry to the quirk_no_flr table for the AMD
> > > > Starship USB 3.0 host controller.
> > > >
> > > > Tested on a Micro-Star International Co., Ltd. MS-7C59/Creator TRX40
> > > > motherboard with an AMD Ryzen Threadripper 3970X.
> > > >
> > > > Without this patch, when attempting to assign (pass through) an AMD
> > > > Starship USB 3.0 host controller to a guest OS, the system becomes
> > > > increasingly unresponsive over the course of several minutes,
> > > > eventually requiring a hard reset.
> > > >
> > > > Shortly after attempting to start the guest, I see these messages:
> > > >
> > > > May 23 22:59:46 mesquite kernel: vfio-pci 0000:05:00.3: not ready
> > > > 1023ms after FLR; waiting May 23 22:59:48 mesquite kernel: vfio-pci
> > > > 0000:05:00.3: not ready 2047ms after FLR; waiting May 23 22:59:51
> > > > mesquite kernel: vfio-pci 0000:05:00.3: not ready 4095ms after FLR;
> > > > waiting May 23 22:59:56 mesquite kernel: vfio-pci 0000:05:00.3: not
> > > > ready 8191ms after FLR; waiting
> > > >
> > > > And then eventually:
> > > >
> > > > May 23 23:01:00 mesquite kernel: vfio-pci 0000:05:00.3: not ready
> > > > 65535ms after FLR; giving up May 23 23:01:05 mesquite kernel: INFO:
> > > > NMI handler (perf_event_nmi_handler) took too long to run: 0.000 msecs
> > > > May 23 23:01:06 mesquite kernel: perf: interrupt took too long (642744
> > > > > 2500), lowering kernel.perf_event_max_sample_rate to 1000 May 23
> > > > 23:01:07 mesquite kernel: INFO: NMI handler (perf_event_nmi_handler)
> > > > took too long to run: 82.270 msecs May 23 23:01:08 mesquite kernel: INFO:
> > > NMI handler (perf_event_nmi_handler) took too long to run: 680.608 msecs
> > > May 23 23:01:08 mesquite kernel: INFO: NMI handler
> > > (perf_event_nmi_handler) took too long to run: 100.952 msecs ...
> > > >  kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 22s!
> > > > [qemu-system-x86:7487] May 23 23:01:25 mesquite kernel: watchdog:
> > > BUG:
> > > > soft lockup - CPU#3 stuck for 22s! [qemu-system-x86:7487]
> > > >
> > > > The above log snippets were obtained using the aforementioned hardware
> > > > running Fedora 32 w/ kernel package kernel-5.6.13-300.fc32.x86_64.  My
> > > > fix was applied to a local copy of the F32 kernel package, then
> > > > rebuilt, etc.
> > > >
> > > > With this patch in place, the host kernel doesn't exhibit these
> > > > problems.  The guest OS (also Fedora 32) starts up and works as
> > > > expected with the passed-through USB host controller.
> > > >
> > > > Signed-off-by: Kevin Buettner <kevinb@redhat.com>
> > >
> > > Applied to pci/virtualization for v5.8, thanks!
> > >
> > > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index
> > > > 43a0c2ce635e..b1db58d00d2b 100644
> > > > --- a/drivers/pci/quirks.c
> > > > +++ b/drivers/pci/quirks.c
> > > > @@ -5133,6 +5133,7 @@
> > > DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x443,
> > > quirk_intel_qat_vf_cap);
> > > >   * FLR may cause the following to devices to hang:
> > > >   *
> > > >   * AMD Starship/Matisse HD Audio Controller 0x1487
> > > > + * AMD Starship USB 3.0 Host Controller 0x148c
> > > >   * AMD Matisse USB 3.0 Host Controller 0x149c
> > > >   * Intel 82579LM Gigabit Ethernet Controller 0x1502
> > > >   * Intel 82579V Gigabit Ethernet Controller 0x1503 @@ -5143,6 +5144,7
> > > > @@ static void quirk_no_flr(struct pci_dev *dev)
> > > >     dev->dev_flags |= PCI_DEV_FLAGS_NO_FLR_RESET;  }
> > > > DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_no_flr);
> > > > +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x148c,
> > > quirk_no_flr);
> > > >  DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c,
> > > quirk_no_flr);
> > > > DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1502,
> > > quirk_no_flr);
> > > > DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x1503,
> > > quirk_no_flr);
> > > >