RX 5500 XT: PCIe link speed stuck at Gen1 2.5GT/s by default

RX 5500 XT: PCIe link speed stuck at Gen1 2.5GT/s by default

[Bjorn Helgaas]

See https://gitlab.freedesktop.org/drm/amd/-/issues/1447

This report was against DRM, but I'm mentioning it here because we've
been talking about some link speed init issues lately, and AFAICT the
gitlab reports don't show up anywhere in lore.

Robert Hancock reported:

> I'm using a RX 5500 XT card on an Asus PRIME H270-PRO motherboard,
> Intel i5-7500 CPU, with kernel 5.10.9 under Fedora 33. I noticed
> that in Linux, "lspci -vv" always showed the GPU PCIe link running
> at 2.5GT/s link speed and never seemed to change regardless of the
> application being run, while in Windows, GPU-Z shows the link
> running at the max supported 8GT/s speed when under graphical load.
> 
> It seems like the driver thinks that 2.5GT/s is the max allowable
> speed, based on the pp_dpm_pcie file:
> 
>   > cd /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/
>   > cat pp_dpm_pcie
>   0: 2.5GT/s, x8 81Mhz *
>   1: 2.5GT/s, x8 619Mhz *
> 
> I'm assuming that something is going wrong with the PCIe link speed
> detection in the driver. Using the "amdgpu.pcie_gen_cap=0x70007"
> kernel command line option seems to result in the driver detecting
> the proper 8GT/s maximum speed.
> 
> lspci -vv output from booting without overriding the speed is
> attached.

RE: RX 5500 XT: PCIe link speed stuck at Gen1 2.5GT/s by default

[Deucher, Alexander]

[Public]

> -----Original Message-----
> From: Bjorn Helgaas <helgaas@kernel.org>
> Sent: Wednesday, March 30, 2022 7:45 AM
> To: linux-pci@vger.kernel.org
> Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Robert Hancock
> <robert.hancock@calian.com>; dri-devel@lists.freedesktop.org
> Subject: RX 5500 XT: PCIe link speed stuck at Gen1 2.5GT/s by default
> 
> See
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.f
> reedesktop.org%2Fdrm%2Famd%2F-
> %2Fissues%2F1447&amp;data=04%7C01%7Calexander.deucher%40amd.com
> %7C01b4e1eed4ac4c97ab0408da1242abe9%7C3dd8961fe4884e608e11a82
> d994e183d%7C0%7C0%7C637842374822556101%7CUnknown%7CTWFpbGZ
> sb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> 0%3D%7C3000&amp;sdata=elFgBq%2FoCfqIO98aaGeS3d1omBdJj%2BSH6HO
> 13oNaBWM%3D&amp;reserved=0
> 
> This report was against DRM, but I'm mentioning it here because we've been
> talking about some link speed init issues lately, and AFAICT the gitlab reports
> don't show up anywhere in lore.
> 

Just noticed this.  The GPU driver calls pcie_bandwidth_available() to get the speed and link widths of the link to root port.  The driver then limits the link speed and number of lanes to the max available on the link.  There's no reason to run the link faster than the slowest link in the chain.  For the most part this works fine because I think the PCIe spec envisions that the links will negotiate the fastest link available when the system comes up.  But it seems that some platforms don't always do this.  Maybe pcie_bandwidth_available() should return the max bandwidth based on the caps of the links rather than the current status.  That said, I'm not sure how you would differentiate between the platforms that came up in a slow link t save power vs platforms that came up in a slow link because there was some problem negotiating a faster link or who, if anyone would be responsible for making sure the links are upgraded when needed.

Alex

> Robert Hancock reported:
> 
> > I'm using a RX 5500 XT card on an Asus PRIME H270-PRO motherboard,
> > Intel i5-7500 CPU, with kernel 5.10.9 under Fedora 33. I noticed that
> > in Linux, "lspci -vv" always showed the GPU PCIe link running at
> > 2.5GT/s link speed and never seemed to change regardless of the
> > application being run, while in Windows, GPU-Z shows the link running
> > at the max supported 8GT/s speed when under graphical load.
> >
> > It seems like the driver thinks that 2.5GT/s is the max allowable
> > speed, based on the pp_dpm_pcie file:
> >
> >   > cd
> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:0
> 3:00.0/
> >   > cat pp_dpm_pcie
> >   0: 2.5GT/s, x8 81Mhz *
> >   1: 2.5GT/s, x8 619Mhz *
> >
> > I'm assuming that something is going wrong with the PCIe link speed
> > detection in the driver. Using the "amdgpu.pcie_gen_cap=0x70007"
> > kernel command line option seems to result in the driver detecting the
> > proper 8GT/s maximum speed.
> >
> > lspci -vv output from booting without overriding the speed is
> > attached.