Re: PCIE BAR resizing blocked by another BAR on same device?

[Dag B <dag@bakke.com>]

I have partial success!  And bits and pieces of an explanation for the 
lack of complete success.

I have a short question or two towards the end, for which the rest here 
is just setup.


On 19.04.2024 17:31, Christian König wrote:
> Am 19.04.24 um 17:19 schrieb Ilpo Järvinen:
>> On Thu, 18 Apr 2024, Dag B wrote:
>>> On 18.04.2024 14:24, Christian König wrote:
>>>> Am 18.04.24 um 12:42 schrieb Dag B:
>>>>> [SNIP]
>>>> Please provide the output of "sudo lspci -v" and "sudo lspci -tv" 
>>>> as file
>>>> attachment (*not* inline in a mail!).
>>>
>>>
>>> https://github.com/dagbdagb/p53
>>>
>>> If the m/l has mechanisms to archive attachments and replace them 
>>> with links,
>>> I'll redo the exercise in a follow-up email. I understand the value 
>>> of having
>>> the 'context' of the discussion readily available in one place.
>> The mem BAR & bridge window configuration is identical between
>> realloc=on/off.
>>
>> The error seems to relate to io BAR:
>>
>> [    2.782439] nvidia 0000:09:00.0: BAR 5 [io  0x0000-0x007f]: not 
>> claimed; can't enable device
>> [    2.783139] NVRM: pci_enable_device failed, aborting
>>
>> With realloc=on, the entire IO window is disabled for the bridges and 
>> for
>> some reason nvidia doesn't abort in that case.
>
> That actually makes a lot of sense.
>
> At least on AMD hardware the IO window is only used for VGA emulation 
> and I strongly suspect it's the same on the NVIDIA GPUs.
>
> So what basically happens is that the BIOS for some reason enables the 
> IO range on both GPUs while when Linux makes the re-alloc it disables 
> the ranges. Most likely because the Linux PCI code knows that they 
> should only be used if this device is the primary VGA device used 
> during boot.
>
> Now when pci_enable_device() is called the function checks if all 
> enabled BARs actually have resources and without realloc=on the I/O 
> BAR has nothing allocated and the function fails. While with 
> realloc=on the BAR is disabled.
>
Thank you for the explanation.


> Well, what a mess. @Dag I would just strongly suggest to see if you 
> can update the BIOS. What happens here is clearly incorrect.

Unlikely to be an option, I am afraid. Hardware too old.  :-/


> Regarding the resizing as far as I can see the BIOS allocates only a 
> single 1GiB window to the upstream bridge, that is most likely way to 
> small for anything than the default 256MiB BAR.
>
> Maybe try to force assign more address space to this bridge. IIRC one 
> of the kernel parameters could be used for that, but of hand I don't 
> remember the syntax.


Found and enabled said option. It works :-)   To reiterate:

- laptop with two TB3 ports.

- to each port, a TB3 eGPU dock with an Nvidia GPU. Currently, I get:

     Capabilities: [bb0 v1] Physical Resizable BAR
         BAR 0: current size: 16MB, supported: 16MB
         BAR 1: current size: 256MB, supported: 64MB 128MB 256MB 512MB 
1GB 2GB 4GB 8GB 16GB 32GB
         BAR 3: current size: 32MB, supported: 32MB

     Capabilities: [bb0 v1] Physical Resizable BAR
         BAR 0: current size: 16MB, supported: 16MB
         BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 
1GB 2GB 4GB 8GB 16GB 32GB
         BAR 3: current size: 32MB, supported: 32MB

- Kernel command-line currently contains: 
pci=realloc=on,hpiosize=1024,hpmmioprefsize=64G      (more than 64G does 
not work)

In short, both eGPUs work. But only one has a big BAR.  Seeing how the 
GPUs are hanging off TB3, system bandwidth to these GPUs will be limited 
in any case. A big BAR is not going to change that at all. But at this 
point I'd like to understand what linux can and cannot do w.r.t. 
resource allocations.


TB3 ports in lspci:

04:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge 
[Titan Ridge 4C 2018] (rev 06)
05:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge 
[Titan Ridge 4C 2018] (rev 06)
05:01.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge 
[Titan Ridge 4C 2018] (rev 06)
05:02.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge 
[Titan Ridge 4C 2018] (rev 06)
05:04.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge 
[Titan Ridge 4C 2018] (rev 06)


TB3 eGPU docks, as seen from boltctl:

  ● Razer Core X Chroma
    ├─ type:          peripheral
    ├─ name:          Core X Chroma
  ● Razer Core X Chroma #2
    ├─ type:          peripheral
    ├─ name:          Core X Chroma
  ● Razer Core X
    ├─ type:          peripheral
    ├─ name:          Core X

Yes, the first dock appears as two devices. It is part of the problem, 
it appears. lspci illustrates this as follows:

06:00.0 System peripheral: Intel Corporation JHL7540 Thunderbolt 3 NHI 
[Titan Ridge 4C 2018] (rev 06)
07:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 4C 2016] (rev 02)
08:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 4C 2016] (rev 02)
08:04.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 4C 2016] (rev 02)
09:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 
3090] (rev a1)
0a:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 4C 2016] (rev 02)
0b:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 4C 2016] (rev 02)
0b:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 4C 2016] (rev 02)
0b:02.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 4C 2016] (rev 02)
0c:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host 
Controller
0d:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host 
Controller
0e:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host 
Controller
2c:00.0 USB controller: Intel Corporation JHL7540 Thunderbolt 3 USB 
Controller [Titan Ridge 4C 2018] (rev 06)
2d:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 2C 2016] (rev 02)
2e:01.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C 
step) [Alpine Ridge 2C 2016] (rev 02)
2f:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 
3090] (rev a1)

or as a tree:

00:1c.0 root_port, slot 0, device present, speed 8GT/s, width x4
  └─04:00.0 upstream_port, Intel Corporation (8086) JHL7540 Thunderbolt 
3 Bridge [Titan Ridge 4C 2018] (15ea)
     ├─05:00.0 downstream_port, slot 0, device present, speed 2.5GT/s, 
width x4
     │  └─06:00.0 endpoint, Intel Corporation (8086) JHL7540 Thunderbolt 
3 NHI [Titan Ridge 4C 2018] (15eb)
     ├─05:01.0 downstream_port, slot 1, device present, speed 2.5GT/s, 
width x4
     │  └─07:00.0 upstream_port, Intel Corporation (8086) JHL6540 
Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (15d3)
     │     ├─08:01.0 downstream_port, slot 0, device present, speed 
2.5GT/s, width x4
     │     │  └─09:00.0 legacy_endpoint
     │     └─08:04.0 downstream_port, slot 4, device present, speed 
2.5GT/s, width x4
     │        └─0a:00.0 upstream_port, Intel Corporation (8086) JHL6540 
Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (15d3)
     │           ├─0b:00.0 downstream_port, slot 0, device present, 
speed 8GT/s, width x1
     │           │  └─0c:00.0 endpoint, ASMedia Technology Inc. (1b21) 
ASM1142 USB 3.1 Host Controller (1242)
     │           ├─0b:01.0 downstream_port, slot 0, device present, 
speed 8GT/s, width x1
     │           │  └─0d:00.0 endpoint, ASMedia Technology Inc. (1b21) 
ASM1142 USB 3.1 Host Controller (1242)
     │           └─0b:02.0 downstream_port, slot 0, device present, 
speed 8GT/s, width x1
     │              └─0e:00.0 endpoint, ASMedia Technology Inc. (1b21) 
ASM1142 USB 3.1 Host Controller (1242)
     ├─05:02.0 downstream_port, slot 0, device present, speed 2.5GT/s, 
width x4
     │  └─2c:00.0 endpoint, Intel Corporation (8086) JHL7540 Thunderbolt 
3 USB Controller [Titan Ridge 4C 2018] (15ec)
     └─05:04.0 downstream_port, slot 4, device present, speed 2.5GT/s, 
width x4
        └─2d:00.0 upstream_port, Intel Corporation (8086) JHL6340 
Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (15da)
           └─2e:01.0 downstream_port, slot 0, device present, speed 
2.5GT/s, width x4
              └─2f:00.0 legacy_endpoint


/proc/iomem looks like this:

4000000000-7fffffffff : PCI Bus 0000:00
   4000000000-400fffffff : 0000:00:02.0
   4010000000-6027ffffff : PCI Bus 0000:04
     4010000000-6027ffffff : PCI Bus 0000:05
       4010000000-5027ffffff : PCI Bus 0000:07
         4010000000-5027ffffff : PCI Bus 0000:08
           4010000000-4027ffffff : PCI Bus 0000:09
           4010000000-401fffffff : 0000:09:00.0
           4020000000-4021ffffff : 0000:09:00.0
           4028000000-5027ffffff : PCI Bus 0000:0a
           4028000000-5027ffffff : PCI Bus 0000:0b
           4028000000-457d4fffff : PCI Bus 0000:0c
           457d500000-4ad29fffff : PCI Bus 0000:0d
           4ad2a00000-5027efffff : PCI Bus 0000:0e
       5400000000-5fffffffff : PCI Bus 0000:2d
         5400000000-5fffffffff : PCI Bus 0000:2e
           5400000000-5fffffffff : PCI Bus 0000:2f
           5400000000-5401ffffff : 0000:2f:00.0
           5800000000-5fffffffff : 0000:2f:00.0

I read this as the stuff downstream of the laptop TB3 controller has the 
range 4010000000-6027ffffff.

The GPU assigned bus id 2f gets all of 5400000000-5fffffffff

The GPU assigned bus id 09  (hanging off 08.01) has to share 
4010000000-5027ffffff with stuff hanging off 08:04 (0a-0e).

0c, 0d and 0e are some USB ports and a (USB-attached) ethernet 
interface, on a separate card in (only) one of the TB3 docks. I can 
physically pull the card with the ports from the dock, but /proc/iomem 
will list the address allocation for 0c-0e *even if they now are missing 
from the lspci output*.


Unbinding 08.4 (and everything below it) and rebinding it to the 
pci-stub driver does not change how /proc/iomem looks either. Nor with a 
pci rescan.


So, the question(s):

Does linux, in this case, have a mechanism for shutting down 08:04 (and 
everything downstream) *and* freeing the reserved resources, such that 
09:00 gets to pick whatever it wants from 4010000000-5027ffffff ?

Or to pre-allocate resources for a particular pci id (08:01), such that 
08:04 simply fails? Which does not require writing actual C-code?

Getting a 'simpler' dock may be the easier way out. I may have crossed 
the line for simple solutions some time ago.    :-)


Thanks,

Dag B