Re: PCIE BAR resizing blocked by another BAR on same device?

From: "Ilpo Järvinen" <ilpo.jarvinen@linux.intel.com>
To: Dag B <dag@bakke.com>
Cc: "Christian König" <christian.koenig@amd.com>,
	"Bjorn Helgaas" <helgaas@kernel.org>,
	linux-pci@vger.kernel.org
Subject: Re: PCIE BAR resizing blocked by another BAR on same device?
Date: Fri, 19 Apr 2024 18:19:50 +0300 (EEST)	[thread overview]
Message-ID: <815337f1-920f-b2ad-7f28-b1b366eb23f5@linux.intel.com> (raw)
In-Reply-To: <ee971d86-5fe4-4e0e-9eb2-21272e793974@bakke.com>

[-- Attachment #1: Type: text/plain, Size: 2160 bytes --]

On Thu, 18 Apr 2024, Dag B wrote:
> On 18.04.2024 14:24, Christian König wrote:
> > Am 18.04.24 um 12:42 schrieb Dag B:
> > > 
> > > [SNIP]
> > > > 
> > > > > > 
> > > > > > Is there a good ELI13 resource explaining how resizable BAR works in
> > > > > > Linux?
> > > > > > 
> > > > > > My current kernel command-line contains: pci=assign-busses,realloc
> > > > 
> > > > That's a really really bad idea. The "assign-busses" flag was introduced
> > > > to get 20year old laptops to see their cardbus PCI devices.
> > > 
> > > I threw a lot of mud at the wall to see what stuck. Removing it now did
> > > not make a big difference.
> > > 
> > > Removing realloc prevents the second TB3 GPU from being initialized, so
> > > keeping that for now.
> > 
> > That's really interesting. Why does it fail without that?
> > 
> > It basically means that your BIOS is somehow broken and only the Linux PCI
> > subsystem is able to assign resources correctly.
> > 
> > Please provide the output of "sudo lspci -v" and "sudo lspci -tv" as file
> > attachment (*not* inline in a mail!).
> 
> 
> In case I have expressed myself awkwardly, the realloc=off case appears to
> make the device driver have issues with the second GPU.
> 
> 
> I have attached both outputs, for realloc=off.
> 
> Not knowing what is considered acceptable message sizes on this m/l, I
> uploaded the same for realloc=on, as well as output from dmesg for both cases
> to:
> 
> https://github.com/dagbdagb/p53
> 
> If the m/l has mechanisms to archive attachments and replace them with links,
> I'll redo the exercise in a follow-up email. I understand the value of having
> the 'context' of the discussion readily available in one place.

The mem BAR & bridge window configuration is identical between 
realloc=on/off.

The error seems to relate to io BAR:

[    2.782439] nvidia 0000:09:00.0: BAR 5 [io  0x0000-0x007f]: not claimed; can't enable device
[    2.783139] NVRM: pci_enable_device failed, aborting

With realloc=on, the entire IO window is disabled for the bridges and for 
some reason nvidia doesn't abort in that case.

-- 
 i.