Re: PCIE BAR resizing blocked by another BAR on same device?

From: Dag B <dag@bakke.com>
To: "Christian König" <christian.koenig@amd.com>,
	"Bjorn Helgaas" <helgaas@kernel.org>
Cc: linux-pci@vger.kernel.org
Subject: Re: PCIE BAR resizing blocked by another BAR on same device?
Date: Fri, 19 Apr 2024 00:54:59 +0200	[thread overview]
Message-ID: <8b6832cc-059a-4a9b-bc0c-af641ca3d487@bakke.com> (raw)
In-Reply-To: <ee971d86-5fe4-4e0e-9eb2-21272e793974@bakke.com>

On 18.04.2024 15:13, Dag B wrote:
>
> On 18.04.2024 14:24, Christian König wrote:
>> Am 18.04.24 um 12:42 schrieb Dag B:
>>>
>>> [SNIP]
>>>>
>>>>>>
>>>>>> Is there a good ELI13 resource explaining how resizable BAR works 
>>>>>> in Linux?
>>>>>>
>>>>>> My current kernel command-line contains: pci=assign-busses,realloc
>>>>
>>>> That's a really really bad idea. The "assign-busses" flag was 
>>>> introduced to get 20year old laptops to see their cardbus PCI devices.
>>>
>>> I threw a lot of mud at the wall to see what stuck. Removing it now 
>>> did not make a big difference.
>>>
>>> Removing realloc prevents the second TB3 GPU from being initialized, 
>>> so keeping that for now.
>>
>> That's really interesting. Why does it fail without that?
>>
>> It basically means that your BIOS is somehow broken and only the 
>> Linux PCI subsystem is able to assign resources correctly.
>>
>> Please provide the output of "sudo lspci -v" and "sudo lspci -tv" as 
>> file attachment (*not* inline in a mail!).
>
>
> In case I have expressed myself awkwardly, the realloc=off case 
> appears to make the device driver have issues with the second GPU.
>
>
> I have attached both outputs, for realloc=off.
>
> Not knowing what is considered acceptable message sizes on this m/l, I 
> uploaded the same for realloc=on, as well as output from dmesg for 
> both cases to:
>
> https://github.com/dagbdagb/p53
>
> If the m/l has mechanisms to archive attachments and replace them with 
> links, I'll redo the exercise in a follow-up email. I understand the 
> value of having the 'context' of the discussion readily available in 
> one place.
>
>
> Dag B

I now have one GPU enabled with the full-fat BAR. The other has issues 
assigning address space for the BARs with this config, and cannot be 
initialized.

pci=realloc=on,hpiosize=64K,hpmemsize=64M,hpmmioprefsize=64G,pcie_scan_all,hpbussize=0x33 

..results in:

     Capabilities: [bb0 v1] Physical Resizable BAR
         BAR 0: current size: 16MB, supported: 16MB
         BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 
1GB 2GB 4GB 8GB 16GB 32GB
         BAR 3: current size: 32MB, supported: 32MB

Still mostly throwing mud at the wall, but the hp* options do appear to 
make a difference. Would love to understand these options better.,

Dag B