Re: [PATCH] Limit AMD Radeon rebar hack to a single revision

["Christian König" <christian.koenig@amd.com>]

Hi Robin and Bjorn,

Am 26.10.21 um 23:49 schrieb Robin McCorkell:
> Thanks, v2 posted with the commit message fixes and correct tagging.
>
> My device isn't a Sapphire RX 5600 XT Pulse, it's an RX 5600M and OEM
> as far as I can tell. The condition in the original code was too broad
> and was catching devices like mine (or the devices of the other bug
> participants) where the hack was breaking things.

well that doesn't really make sense. As far as I know the problem is an 
invalid PCIe config space on the whole series of the ASIC, but I can 
double check with the hardware engineers once more.

What could be is that a later hardware revision doesn't have that bug 
any more and we don't need to apply the workaround. That is trivial to 
confirm with an verbose lspci output, but even then the workaround won't 
hurt in any way.

> On Tue, 26 Oct 2021 at 22:28, Bjorn Helgaas <helgaas@kernel.org> wrote:
>> [+cc Christian, Nirmoy, authors of 907830b0fc9e]
>>
>> Subject line should look like previous ones for this file, e.g.,
>>
>>    88769e64cf99 ("PCI: Add ACS quirk for Pericom PI7C9X2G switches")
>>    e3f4bd3462f6 ("PCI: Mark Atheros QCA6174 to avoid bus reset")
>>    60b78ed088eb ("PCI: Add AMD GPU multi-function power dependencies")
>>    8304a3a199ee ("PCI: Set dma-can-stall for HiSilicon chips")
>>    8c09e896cef8 ("PCI: Allow PASID on fake PCIe devices without TLP prefixes")
>>    32837d8a8f63 ("PCI: Add ACS quirks for Cavium multi-function devices")
>>    e0bff4322092 ("PCI: Increase D3 delay for AMD Renoir/Cezanne XHCI")
>>    ...
>>    907830b0fc9e ("PCI: Add a REBAR size quirk for Sapphire RX 5600 XT Pulse")
>>
>> Would be good to mention "Sapphire RX 5600 XT Pulse" explicitly since
>> that's what 907830b0fc9e said.
>>
>> On Tue, Oct 26, 2021 at 09:46:38PM +0100, Robin McCorkell wrote:
>>> A particular RX 5600 device requires a hack in the rebar logic, but the
>>> current branch is too general and catches other devices too, breaking
>>> them. This patch changes the branch to be more selective on the
>>> particular revision.
>>>
>>> See also: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fdrm%2Famd%2F-%2Fissues%2F1707&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C23c4b6057dc641fd111f08d998ca8666%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637708818349122129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=lhBo74fAemYtZKYSqkfRtvN1Jo48%2BnZtNIaAAFS2DfY%3D&amp;reserved=0
>> There's a lot of legwork in the bug report to bisect this, but no
>> explanation of what the root cause turned out to be.

Yes agree, that bisection is extremely unlikely to be correct.

See the driver allocates addresses from both ends of the BAR and works 
towards the middle.

So if something doesn't work correctly we see that immediately because 
the driver isn't even able to initialize the hardware.

What could be is that resizing the BAR reduces the overhead drastically. 
Resulting in some use cases to have a 10-20% performance benefit. When 
we have a timing bug somewhere else the patch could potentially make 
that much more likely to be seen.

Going to comment on the bug report as well.

>> 907830b0fc9e says RX 5600 XT Pulse advertises 256MB-1GB BAR 0 sizes but
>> actually supports up to 8GB.
>>
>> Does this patch mean your RX 5600 XT Pulse supports fewer sizes and
>> advertises them correctly?  And consequently we resize BAR 0 to
>> something that's too big, and something fails when we try to access
>> the part the isn't actually implemented by the device?

We do have devices which have a non power of two local memory, e.g. 3GiB 
or 6GiB. In those cases we resize the BAR to the next power of two and 
that still works.

>>
>> It would be useful to attach your "lspci -vv" output to the bug
>> report.
>>
>>> This patch fixes intermittent freezes on other RX 5600 devices where the
>>> hack is unnecessary. Credit to all contributors in the linked issue on
>>> the AMD bug tracker.
>> Thanks.  This would need a signed-off-by [1].
>>
>> We should also include a "Fixes:" line for the commit the problem was
>> bisected to, 907830b0fc9e ("PCI: Add a REBAR size quirk for Sapphire
>> RX 5600 XT Pulse"), if I understand correctly, e.g.,
>>
>>    Fixes: 907830b0fc9e ("PCI: Add a REBAR size quirk for Sapphire RX 5600 XT Pulse")
>>
>> And probably a stable tag, since 907830b0fc9e appeared in v5.12, e.g.,
>>
>>    Cc: stable@vger.kernel.org    # v5.12+
>>
>> If stable maintainers backported 907830b0fc9e to earlier kernels, as
>> it appears they have, it's up to them to watch for fixes to
>> 907830b0fc9e.

What would be very helpful as well is to CC the relevant AMD engineers 
which wrote the original patch and signed it off.

Regards,
Christian.

>>
>> [1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2FDocumentation%2Fprocess%2Fsubmitting-patches.rst%3Fid%3Dv5.14%23n365&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C23c4b6057dc641fd111f08d998ca8666%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637708818349132123%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=cY7hOAFEuEqQuwUOY50hXmS0BvyP9D%2BQNBNsxn%2Bgmtg%3D&amp;reserved=0
>>
>>> ---
>>>   drivers/pci/pci.c | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>>> index ce2ab62b64cf..1fe75243019e 100644
>>> --- a/drivers/pci/pci.c
>>> +++ b/drivers/pci/pci.c
>>> @@ -3647,7 +3647,7 @@ u32 pci_rebar_get_possible_sizes(struct pci_dev *pdev, int bar)
>>>
>>>        /* Sapphire RX 5600 XT Pulse has an invalid cap dword for BAR 0 */
>>>        if (pdev->vendor == PCI_VENDOR_ID_ATI && pdev->device == 0x731f &&
>>> -         bar == 0 && cap == 0x7000)
>>> +         pdev->revision == 0xC1 && bar == 0 && cap == 0x7000)
>> I'd like to get the AMD folks to chime in and confirm that revision
>> 0xC1 is the only one that requires this quirk.
>>
>>>                cap = 0x3f000;
>>>
>>>        return cap >> 4;
>>> --
>>> 2.31.1
>>>