* PCIE BAR resizing blocked by another BAR on same device?
@ 2024-04-17 13:16 Dag B
2024-04-17 15:13 ` Bjorn Helgaas
0 siblings, 1 reply; 10+ messages in thread
From: Dag B @ 2024-04-17 13:16 UTC (permalink / raw)
To: linux-pci
Hi.
In short, I have a GPU for which lspci reveals:
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 128MB, supported: 64MB 128MB 256MB 512MB
1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
In dmesg I see:
[ 0.517191] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 0.517238] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
I take it the location of BAR 3 right after BAR 1 explains why I get:
p53 # echo 9 > resource1_resize
-bash: echo: write error: No space left on device
Shrinking it and increasing it to the orginal size works.
Is there anything I can do with current kernel functionality to reserve
memory address space for the full-fat BAR 1? Or relocate it?
If not, is this something which *can* be worked around in the kernel?
And if so, does it belong with the PCI subsystem? Or the devicedriver
for the device in question?
Is there a good ELI13 resource explaining how resizable BAR works in Linux?
My current kernel command-line contains: pci=assign-busses,realloc
My GPU is attached via TB3 to a system for which resizable BAR is and
will remain a foreign concept in the BIOS.
dmesg excerpt below reflects resizing the BAR to 128MB and then back to
256MB.
[ 1730.091789] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: releasing
[ 1730.091875] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: releasing
[ 1730.092072] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: releasing
[ 1730.092151] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: assigned
[ 1730.092223] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 1730.092335] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 1730.092444] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 1730.092510] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 1730.092595] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 1730.092680] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 1730.092778] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 1730.092850] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 1730.092933] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 1730.093018] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]
[ 1759.817306] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: releasing
[ 1759.817394] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: releasing
[ 1759.817591] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: releasing
[ 1759.817668] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: assigned
[ 1759.817740] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: assigned
[ 1759.817853] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: assigned
[ 1759.817964] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 1759.818035] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 1759.818120] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 1759.818204] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 1759.818303] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 1759.818374] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 1759.818459] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 1759.818544] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]
[ 1769.797178] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: releasing
[ 1769.797241] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: releasing
[ 1769.797417] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: releasing
[ 1769.797473] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: can't assign; no space
[ 1769.797515] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: failed to assign
[ 1769.797557] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
can't assign; no space
[ 1769.797594] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
failed to assign
[ 1769.797630] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
can't assign; no space
[ 1769.797666] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
failed to assign
[ 1769.797703] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 1769.797761] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 1769.797829] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 1769.797895] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 1769.797972] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 1769.798027] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 1769.798089] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 1769.798155] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]
[ 1769.798270] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: assigned
[ 1769.798358] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: assigned
[ 2669.324929] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: releasing
[ 2669.324992] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: releasing
[ 2669.325164] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: releasing
[ 2669.325219] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: can't assign; no space
[ 2669.327023] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
64bit pref]: failed to assign
[ 2669.328798] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
can't assign; no space
[ 2669.330482] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
failed to assign
[ 2669.331104] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
can't assign; no space
[ 2669.331682] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
failed to assign
[ 2669.332258] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 2669.332855] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 2669.333444] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 2669.334130] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 2669.334821] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 2669.335460] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 2669.336063] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 2669.336657] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]
[ 2669.337442] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: assigned
[ 2669.338073] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: assigned
[ 2673.200263] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff
64bit pref]: releasing
[ 2673.201935] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff
64bit pref]: releasing
[ 2673.203801] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x600bffffff 64bit pref]: releasing
[ 2673.205461] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]: assigned
[ 2673.206197] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 2673.206800] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 2673.207534] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
[ 2673.208143] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
[ 2673.208734] pcieport 0000:06:00.0: bridge window [mem
0xa4000000-0xafffffff]
[ 2673.209323] pcieport 0000:06:00.0: bridge window [mem
0x6000000000-0x601fffffff 64bit pref]
[ 2673.209916] pcieport 0000:07:01.0: PCI bridge to [bus 08]
[ 2673.210526] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
[ 2673.211170] pcieport 0000:07:01.0: bridge window [mem
0xa4000000-0xa57fffff]
[ 2673.211755] pcieport 0000:07:01.0: bridge window [mem
0x6000000000-0x6017ffffff 64bit pref]
Unsure what else from dmesg or kernel config is relevant here. Can post
the full 100k here or somewhere else. Or excerpts.
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006f23bfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000006f23c000-0x000000006f23cfff]
reserved
[ 0.000000] BIOS-e820: [mem 0x000000006f23d000-0x000000007a792fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007a793000-0x000000007fa6cfff]
reserved
[ 0.000000] BIOS-e820: [mem 0x000000007fa6d000-0x000000007fca9fff]
ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x000000007fcaa000-0x000000007fd0efff]
ACPI data
[ 0.000000] BIOS-e820: [mem 0x000000007fd0f000-0x000000007fd0ffff] usable
[ 0.000000] BIOS-e820: [mem 0x000000007fd10000-0x0000000087ffffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000887fffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000088800000-0x000000008f7fffff]
reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff]
reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000106c7fffff] usable
[ 0.000000] efi: Not removing mem55: MMIO
range=[0xfe010000-0xfe010fff] (4KB) from e820 map
[ 0.000003] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000006] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.503260] e820: reserve RAM buffer [mem 0x0009f000-0x0009ffff]
[ 0.503262] e820: reserve RAM buffer [mem 0x6f23c000-0x6fffffff]
[ 0.503263] e820: reserve RAM buffer [mem 0x7a793000-0x7bffffff]
[ 0.503264] e820: reserve RAM buffer [mem 0x7fd10000-0x7fffffff]
[ 0.503265] e820: reserve RAM buffer [mem 0x88800000-0x8bffffff]
[ 0.503266] e820: reserve RAM buffer [mem 0x106c800000-0x106fffffff]
[ 0.255546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.445755] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
ClockPM Segments MSI HPX-Type3]
[ 0.447023] acpi PNP0A08:00: _OSC: platform does not support [AER]
[ 0.449507] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME
PCIeCapability LTR]
[ 0.449511] acpi PNP0A08:00: FADT indicates ASPM is unsupported,
using BIOS configuration
[ 0.522380] caller snb_uncore_imc_init_box+0x73/0xd0 mapping multiple
BARs
Thanks,
Dag B
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-17 13:16 PCIE BAR resizing blocked by another BAR on same device? Dag B
@ 2024-04-17 15:13 ` Bjorn Helgaas
2024-04-18 7:51 ` Christian König
0 siblings, 1 reply; 10+ messages in thread
From: Bjorn Helgaas @ 2024-04-17 15:13 UTC (permalink / raw)
To: Dag B, Christian König; +Cc: linux-pci
[+to Christian, resizable BAR expert]
On Wed, Apr 17, 2024 at 03:16:06PM +0200, Dag B wrote:
> Hi.
>
> In short, I have a GPU for which lspci reveals:
>
> Capabilities: [bb0 v1] Physical Resizable BAR
>
> BAR 0: current size: 16MB, supported: 16MB
> BAR 1: current size: 128MB, supported: 64MB 128MB 256MB 512MB 1GB
> 2GB 4GB 8GB 16GB 32GB
> BAR 3: current size: 32MB, supported: 32MB
>
> In dmesg I see:
>
> [ 0.517191] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 0.517238] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
>
> I take it the location of BAR 3 right after BAR 1 explains why I get:
>
> p53 # echo 9 > resource1_resize
> -bash: echo: write error: No space left on device
>
> Shrinking it and increasing it to the orginal size works.
>
>
> Is there anything I can do with current kernel functionality to reserve
> memory address space for the full-fat BAR 1? Or relocate it?
>
> If not, is this something which *can* be worked around in the kernel? And if
> so, does it belong with the PCI subsystem? Or the devicedriver for the
> device in question?
>
> Is there a good ELI13 resource explaining how resizable BAR works in Linux?
>
> My current kernel command-line contains: pci=assign-busses,realloc
>
> My GPU is attached via TB3 to a system for which resizable BAR is and will
> remain a foreign concept in the BIOS.
>
> dmesg excerpt below reflects resizing the BAR to 128MB and then back to
> 256MB.
>
> [ 1730.091789] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: releasing
> [ 1730.091875] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: releasing
> [ 1730.092072] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: releasing
> [ 1730.092151] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: assigned
> [ 1730.092223] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 1730.092335] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 1730.092444] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1730.092510] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 1730.092595] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1730.092680] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1730.092778] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1730.092850] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 1730.092933] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1730.093018] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]
> [ 1759.817306] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: releasing
> [ 1759.817394] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: releasing
> [ 1759.817591] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: releasing
> [ 1759.817668] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: assigned
> [ 1759.817740] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 1759.817853] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 1759.817964] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1759.818035] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 1759.818120] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1759.818204] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1759.818303] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1759.818374] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 1759.818459] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1759.818544] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 1769.797178] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 1769.797241] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 1769.797417] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 1769.797473] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: can't assign; no space
> [ 1769.797515] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: failed to assign
> [ 1769.797557] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> can't assign; no space
> [ 1769.797594] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> failed to assign
> [ 1769.797630] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> can't assign; no space
> [ 1769.797666] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> failed to assign
> [ 1769.797703] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1769.797761] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 1769.797829] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1769.797895] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1769.797972] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1769.798027] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 1769.798089] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1769.798155] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 1769.798270] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 1769.798358] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 2669.324929] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 2669.324992] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 2669.325164] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 2669.325219] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: can't assign; no space
> [ 2669.327023] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: failed to assign
> [ 2669.328798] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> can't assign; no space
> [ 2669.330482] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> failed to assign
> [ 2669.331104] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> can't assign; no space
> [ 2669.331682] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> failed to assign
> [ 2669.332258] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 2669.332855] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 2669.333444] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 2669.334130] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 2669.334821] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 2669.335460] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 2669.336063] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 2669.336657] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 2669.337442] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 2669.338073] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 2673.200263] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 2673.201935] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 2673.203801] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 2673.205461] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: assigned
> [ 2673.206197] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 2673.206800] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 2673.207534] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 2673.208143] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
> [ 2673.208734] pcieport 0000:06:00.0: bridge window [mem
> 0xa4000000-0xafffffff]
> [ 2673.209323] pcieport 0000:06:00.0: bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 2673.209916] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 2673.210526] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
> [ 2673.211170] pcieport 0000:07:01.0: bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 2673.211755] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]
>
>
> Unsure what else from dmesg or kernel config is relevant here. Can post the
> full 100k here or somewhere else. Or excerpts.
>
> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006f23bfff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000006f23c000-0x000000006f23cfff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x000000006f23d000-0x000000007a792fff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000007a793000-0x000000007fa6cfff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x000000007fa6d000-0x000000007fca9fff] ACPI
> NVS
> [ 0.000000] BIOS-e820: [mem 0x000000007fcaa000-0x000000007fd0efff] ACPI
> data
> [ 0.000000] BIOS-e820: [mem 0x000000007fd0f000-0x000000007fd0ffff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000007fd10000-0x0000000087ffffff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000887fffff] usable
> [ 0.000000] BIOS-e820: [mem 0x0000000088800000-0x000000008f7fffff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff]
> reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000106c7fffff] usable
> [ 0.000000] efi: Not removing mem55: MMIO range=[0xfe010000-0xfe010fff]
> (4KB) from e820 map
> [ 0.000003] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
> [ 0.000006] e820: remove [mem 0x000a0000-0x000fffff] usable
> [ 0.503260] e820: reserve RAM buffer [mem 0x0009f000-0x0009ffff]
> [ 0.503262] e820: reserve RAM buffer [mem 0x6f23c000-0x6fffffff]
> [ 0.503263] e820: reserve RAM buffer [mem 0x7a793000-0x7bffffff]
> [ 0.503264] e820: reserve RAM buffer [mem 0x7fd10000-0x7fffffff]
> [ 0.503265] e820: reserve RAM buffer [mem 0x88800000-0x8bffffff]
> [ 0.503266] e820: reserve RAM buffer [mem 0x106c800000-0x106fffffff]
>
>
> [ 0.255546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> [ 0.445755] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
> ClockPM Segments MSI HPX-Type3]
> [ 0.447023] acpi PNP0A08:00: _OSC: platform does not support [AER]
> [ 0.449507] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME
> PCIeCapability LTR]
> [ 0.449511] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using
> BIOS configuration
>
> [ 0.522380] caller snb_uncore_imc_init_box+0x73/0xd0 mapping multiple
> BARs
>
>
> Thanks,
>
>
> Dag B
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-17 15:13 ` Bjorn Helgaas
@ 2024-04-18 7:51 ` Christian König
2024-04-18 10:42 ` Dag B
0 siblings, 1 reply; 10+ messages in thread
From: Christian König @ 2024-04-18 7:51 UTC (permalink / raw)
To: Bjorn Helgaas, Dag B; +Cc: linux-pci
Hi Dag and Bjorn,
Am 17.04.24 um 17:13 schrieb Bjorn Helgaas:
> [+to Christian, resizable BAR expert]
>
> On Wed, Apr 17, 2024 at 03:16:06PM +0200, Dag B wrote:
>> Hi.
>>
>> In short, I have a GPU for which lspci reveals:
>>
>> Capabilities: [bb0 v1] Physical Resizable BAR
>>
>> BAR 0: current size: 16MB, supported: 16MB
>> BAR 1: current size: 128MB, supported: 64MB 128MB 256MB 512MB 1GB
>> 2GB 4GB 8GB 16GB 32GB
>> BAR 3: current size: 32MB, supported: 32MB
>>
>> In dmesg I see:
>>
>> [ 0.517191] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
>> pref]: assigned
>> [ 0.517238] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
>> pref]: assigned
>> [ 0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
>>
>> I take it the location of BAR 3 right after BAR 1 explains why I get:
>>
>> p53 # echo 9 > resource1_resize
>> -bash: echo: write error: No space left on device
>>
>> Shrinking it and increasing it to the orginal size works.
>>
>>
>> Is there anything I can do with current kernel functionality to reserve
>> memory address space for the full-fat BAR 1? Or relocate it?
No, sorry. The BARs have to be released and re-assigned all at the same
time for this to work correctly.
That's one of the reasons why we choose to do this in the driver during
the load process instead of the PCI subsystem.
The sysfs functionality is more or less just for testing.
>> If not, is this something which *can* be worked around in the kernel? And if
>> so, does it belong with the PCI subsystem? Or the devicedriver for the
>> device in question?
The device driver is the only place where you know all the hw specific
things necessary to release a device BAR and eventually move and resize
upstream bridges.
>>
>> Is there a good ELI13 resource explaining how resizable BAR works in Linux?
>>
>> My current kernel command-line contains: pci=assign-busses,realloc
That's a really really bad idea. The "assign-busses" flag was introduced
to get 20year old laptops to see their cardbus PCI devices.
On modern systems specifying this flag breaks a lot of ACPI handling
because we can no longer match ACPI devices to their PCI counterparts
because all the bus numbers have changed.
While it might get things working temporary if you have to use this
option then please replace your BIOS. It's not really a doable to use
this on modern hw.
>> My GPU is attached via TB3 to a system for which resizable BAR is and will
>> remain a foreign concept in the BIOS.
What happens if you hot remove and re-plug the TB3 after the system has
started?
Regards,
Christian.
>>
>> dmesg excerpt below reflects resizing the BAR to 128MB and then back to
>> 256MB.
>>
>> [ 1730.091789] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
>> pref]: releasing
>> [ 1730.091875] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
>> pref]: releasing
>> [ 1730.092072] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x6017ffffff 64bit pref]: releasing
>> [ 1730.092151] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x6017ffffff 64bit pref]: assigned
>> [ 1730.092223] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
>> pref]: assigned
>> [ 1730.092335] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
>> pref]: assigned
>> [ 1730.092444] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
>> [ 1730.092510] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
>> [ 1730.092595] pcieport 0000:06:00.0: bridge window [mem
>> 0xa4000000-0xafffffff]
>> [ 1730.092680] pcieport 0000:06:00.0: bridge window [mem
>> 0x6000000000-0x601fffffff 64bit pref]
>> [ 1730.092778] pcieport 0000:07:01.0: PCI bridge to [bus 08]
>> [ 1730.092850] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
>> [ 1730.092933] pcieport 0000:07:01.0: bridge window [mem
>> 0xa4000000-0xa57fffff]
>> [ 1730.093018] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x6017ffffff 64bit pref]
>> [ 1759.817306] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
>> pref]: releasing
>> [ 1759.817394] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
>> pref]: releasing
>> [ 1759.817591] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x6017ffffff 64bit pref]: releasing
>> [ 1759.817668] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x600bffffff 64bit pref]: assigned
>> [ 1759.817740] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
>> pref]: assigned
>> [ 1759.817853] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
>> pref]: assigned
>> [ 1759.817964] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
>> [ 1759.818035] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
>> [ 1759.818120] pcieport 0000:06:00.0: bridge window [mem
>> 0xa4000000-0xafffffff]
>> [ 1759.818204] pcieport 0000:06:00.0: bridge window [mem
>> 0x6000000000-0x601fffffff 64bit pref]
>> [ 1759.818303] pcieport 0000:07:01.0: PCI bridge to [bus 08]
>> [ 1759.818374] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
>> [ 1759.818459] pcieport 0000:07:01.0: bridge window [mem
>> 0xa4000000-0xa57fffff]
>> [ 1759.818544] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x600bffffff 64bit pref]
>> [ 1769.797178] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
>> pref]: releasing
>> [ 1769.797241] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
>> pref]: releasing
>> [ 1769.797417] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x600bffffff 64bit pref]: releasing
>> [ 1769.797473] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
>> 64bit pref]: can't assign; no space
>> [ 1769.797515] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
>> 64bit pref]: failed to assign
>> [ 1769.797557] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
>> can't assign; no space
>> [ 1769.797594] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
>> failed to assign
>> [ 1769.797630] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
>> can't assign; no space
>> [ 1769.797666] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
>> failed to assign
>> [ 1769.797703] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
>> [ 1769.797761] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
>> [ 1769.797829] pcieport 0000:06:00.0: bridge window [mem
>> 0xa4000000-0xafffffff]
>> [ 1769.797895] pcieport 0000:06:00.0: bridge window [mem
>> 0x6000000000-0x601fffffff 64bit pref]
>> [ 1769.797972] pcieport 0000:07:01.0: PCI bridge to [bus 08]
>> [ 1769.798027] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
>> [ 1769.798089] pcieport 0000:07:01.0: bridge window [mem
>> 0xa4000000-0xa57fffff]
>> [ 1769.798155] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x600bffffff 64bit pref]
>> [ 1769.798270] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
>> pref]: assigned
>> [ 1769.798358] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
>> pref]: assigned
>> [ 2669.324929] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
>> pref]: releasing
>> [ 2669.324992] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
>> pref]: releasing
>> [ 2669.325164] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x600bffffff 64bit pref]: releasing
>> [ 2669.325219] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
>> 64bit pref]: can't assign; no space
>> [ 2669.327023] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
>> 64bit pref]: failed to assign
>> [ 2669.328798] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
>> can't assign; no space
>> [ 2669.330482] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
>> failed to assign
>> [ 2669.331104] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
>> can't assign; no space
>> [ 2669.331682] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
>> failed to assign
>> [ 2669.332258] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
>> [ 2669.332855] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
>> [ 2669.333444] pcieport 0000:06:00.0: bridge window [mem
>> 0xa4000000-0xafffffff]
>> [ 2669.334130] pcieport 0000:06:00.0: bridge window [mem
>> 0x6000000000-0x601fffffff 64bit pref]
>> [ 2669.334821] pcieport 0000:07:01.0: PCI bridge to [bus 08]
>> [ 2669.335460] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
>> [ 2669.336063] pcieport 0000:07:01.0: bridge window [mem
>> 0xa4000000-0xa57fffff]
>> [ 2669.336657] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x600bffffff 64bit pref]
>> [ 2669.337442] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
>> pref]: assigned
>> [ 2669.338073] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
>> pref]: assigned
>> [ 2673.200263] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
>> pref]: releasing
>> [ 2673.201935] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
>> pref]: releasing
>> [ 2673.203801] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x600bffffff 64bit pref]: releasing
>> [ 2673.205461] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x6017ffffff 64bit pref]: assigned
>> [ 2673.206197] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
>> pref]: assigned
>> [ 2673.206800] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
>> pref]: assigned
>> [ 2673.207534] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
>> [ 2673.208143] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff]
>> [ 2673.208734] pcieport 0000:06:00.0: bridge window [mem
>> 0xa4000000-0xafffffff]
>> [ 2673.209323] pcieport 0000:06:00.0: bridge window [mem
>> 0x6000000000-0x601fffffff 64bit pref]
>> [ 2673.209916] pcieport 0000:07:01.0: PCI bridge to [bus 08]
>> [ 2673.210526] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff]
>> [ 2673.211170] pcieport 0000:07:01.0: bridge window [mem
>> 0xa4000000-0xa57fffff]
>> [ 2673.211755] pcieport 0000:07:01.0: bridge window [mem
>> 0x6000000000-0x6017ffffff 64bit pref]
>>
>>
>> Unsure what else from dmesg or kernel config is relevant here. Can post the
>> full 100k here or somewhere else. Or excerpts.
>>
>> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
>> [ 0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff]
>> reserved
>> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006f23bfff] usable
>> [ 0.000000] BIOS-e820: [mem 0x000000006f23c000-0x000000006f23cfff]
>> reserved
>> [ 0.000000] BIOS-e820: [mem 0x000000006f23d000-0x000000007a792fff] usable
>> [ 0.000000] BIOS-e820: [mem 0x000000007a793000-0x000000007fa6cfff]
>> reserved
>> [ 0.000000] BIOS-e820: [mem 0x000000007fa6d000-0x000000007fca9fff] ACPI
>> NVS
>> [ 0.000000] BIOS-e820: [mem 0x000000007fcaa000-0x000000007fd0efff] ACPI
>> data
>> [ 0.000000] BIOS-e820: [mem 0x000000007fd0f000-0x000000007fd0ffff] usable
>> [ 0.000000] BIOS-e820: [mem 0x000000007fd10000-0x0000000087ffffff]
>> reserved
>> [ 0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000887fffff] usable
>> [ 0.000000] BIOS-e820: [mem 0x0000000088800000-0x000000008f7fffff]
>> reserved
>> [ 0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff]
>> reserved
>> [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000106c7fffff] usable
>> [ 0.000000] efi: Not removing mem55: MMIO range=[0xfe010000-0xfe010fff]
>> (4KB) from e820 map
>> [ 0.000003] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
>> [ 0.000006] e820: remove [mem 0x000a0000-0x000fffff] usable
>> [ 0.503260] e820: reserve RAM buffer [mem 0x0009f000-0x0009ffff]
>> [ 0.503262] e820: reserve RAM buffer [mem 0x6f23c000-0x6fffffff]
>> [ 0.503263] e820: reserve RAM buffer [mem 0x7a793000-0x7bffffff]
>> [ 0.503264] e820: reserve RAM buffer [mem 0x7fd10000-0x7fffffff]
>> [ 0.503265] e820: reserve RAM buffer [mem 0x88800000-0x8bffffff]
>> [ 0.503266] e820: reserve RAM buffer [mem 0x106c800000-0x106fffffff]
>>
>>
>> [ 0.255546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>> [ 0.445755] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
>> ClockPM Segments MSI HPX-Type3]
>> [ 0.447023] acpi PNP0A08:00: _OSC: platform does not support [AER]
>> [ 0.449507] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME
>> PCIeCapability LTR]
>> [ 0.449511] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using
>> BIOS configuration
>>
>> [ 0.522380] caller snb_uncore_imc_init_box+0x73/0xd0 mapping multiple
>> BARs
>>
>>
>> Thanks,
>>
>>
>> Dag B
>>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-18 7:51 ` Christian König
@ 2024-04-18 10:42 ` Dag B
2024-04-18 12:24 ` Christian König
0 siblings, 1 reply; 10+ messages in thread
From: Dag B @ 2024-04-18 10:42 UTC (permalink / raw)
To: Christian König, Bjorn Helgaas; +Cc: linux-pci
On 18.04.2024 09:51, Christian König wrote:
> Hi Dag and Bjorn,
>
> Am 17.04.24 um 17:13 schrieb Bjorn Helgaas:
>> [+to Christian, resizable BAR expert]
>>
>> On Wed, Apr 17, 2024 at 03:16:06PM +0200, Dag B wrote:
>>> Hi.
>>>
>>> In short, I have a GPU for which lspci reveals:
>>>
>>> Capabilities: [bb0 v1] Physical Resizable BAR
>>>
>>> BAR 0: current size: 16MB, supported: 16MB
>>> BAR 1: current size: 128MB, supported: 64MB 128MB 256MB
>>> 512MB 1GB
>>> 2GB 4GB 8GB 16GB 32GB
>>> BAR 3: current size: 32MB, supported: 32MB
>>>
>>> In dmesg I see:
>>>
>>> [ 0.517191] pci 0000:08:00.0: BAR 1 [mem
>>> 0x6000000000-0x600fffffff 64bit
>>> pref]: assigned
>>> [ 0.517238] pci 0000:08:00.0: BAR 3 [mem
>>> 0x6010000000-0x6011ffffff 64bit
>>> pref]: assigned
>>> [ 0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]:
>>> assigned
>>>
>>> I take it the location of BAR 3 right after BAR 1 explains why I get:
>>>
>>> p53 # echo 9 > resource1_resize
>>> -bash: echo: write error: No space left on device
>>>
>>> Shrinking it and increasing it to the orginal size works.
>>>
>>>
>>> Is there anything I can do with current kernel functionality to reserve
>>> memory address space for the full-fat BAR 1? Or relocate it?
>
> No, sorry. The BARs have to be released and re-assigned all at the
> same time for this to work correctly.
>
> That's one of the reasons why we choose to do this in the driver
> during the load process instead of the PCI subsystem.
>
> The sysfs functionality is more or less just for testing.
>
>>> If not, is this something which *can* be worked around in the
>>> kernel? And if
>>> so, does it belong with the PCI subsystem? Or the devicedriver for the
>>> device in question?
>
> The device driver is the only place where you know all the hw specific
> things necessary to release a device BAR and eventually move and
> resize upstream bridges.
Noted.
>
>>>
>>> Is there a good ELI13 resource explaining how resizable BAR works in
>>> Linux?
>>>
>>> My current kernel command-line contains: pci=assign-busses,realloc
>
> That's a really really bad idea. The "assign-busses" flag was
> introduced to get 20year old laptops to see their cardbus PCI devices.
I threw a lot of mud at the wall to see what stuck. Removing it now did
not make a big difference.
Removing realloc prevents the second TB3 GPU from being initialized, so
keeping that for now.
>
>>> My GPU is attached via TB3 to a system for which resizable BAR is
>>> and will
>>> remain a foreign concept in the BIOS.
>
> What happens if you hot remove and re-plug the TB3 after the system
> has started?
>
Much the same as during initial boot. Both good and bad. See below.
Do any of the pci=hp* options have any significance/impact on what dmesg
says below?
Is IO address space moveable? Relevant kernel config/options impacting
this? Is it all in the hands of the device driver?
So, so many questions. And barely competent to ask them. Please forgive me.
Current kernel command-line snippet:
pci=realloc,hpiosize=16K,hpmemsize=64M,pcie_scan_all,hpbussize=0x33
I very much appreciate your input. Will try to get the attention of the
people responsible for the driver.
Thanks,
Dag B
p53 ~ # dmesg | grep 09:00.0
[ 0.471780] pci 0000:09:00.0: [10de:2204] type 00 class 0x030000 PCIe
Legacy Endpoint
[ 0.471816] pci 0000:09:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.471844] pci 0000:09:00.0: BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.471873] pci 0000:09:00.0: BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.471890] pci 0000:09:00.0: BAR 5 [io 0x0000-0x007f]
[ 0.471907] pci 0000:09:00.0: ROM [mem 0x00000000-0x0007ffff pref]
[ 0.472133] pci 0000:09:00.0: PME# supported from D0 D3hot
[ 0.472382] pci 0000:09:00.0: 8.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x4 link at 0000:05:01.0 (capable of 252.048
Gb/s with 16.0 GT/s PCIe x16 link)
[ 0.491866] pci 0000:09:00.0: vgaarb: bridge control possible
[ 0.491866] pci 0000:09:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none
[ 0.491866] pnp 00:03: disabling [io 0x002e-0x002f] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:03: disabling [io 0x004e-0x004f] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:03: disabling [io 0x0061] because it overlaps
0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:03: disabling [io 0x0063] because it overlaps
0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:03: disabling [io 0x0065] because it overlaps
0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:03: disabling [io 0x0067] because it overlaps
0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:03: disabling [io 0x0070] because it overlaps
0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0010-0x001f] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0024-0x0025] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0028-0x0029] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x002c-0x002d] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0030-0x0031] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0034-0x0035] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0038-0x0039] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x003c-0x003d] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0050-0x0053] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.491866] pnp 00:08: disabling [io 0x0072-0x0077] because it
overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
[ 0.493216] pnp 00:0b: disabling [mem 0x00000000-0x0009ffff] because
it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493220] pnp 00:0b: disabling [mem 0x000c0000-0x000c3fff disabled]
because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493225] pnp 00:0b: disabling [mem 0x000c8000-0x000cbfff disabled]
because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493230] pnp 00:0b: disabling [mem 0x000d0000-0x000d3fff disabled]
because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493234] pnp 00:0b: disabling [mem 0x000d8000-0x000dbfff disabled]
because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493238] pnp 00:0b: disabling [mem 0x000e0000-0x000e3fff] because
it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493242] pnp 00:0b: disabling [mem 0x000e8000-0x000ebfff] because
it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493247] pnp 00:0b: disabling [mem 0x000f0000-0x000fffff] because
it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493251] pnp 00:0b: disabling [mem 0x00100000-0x8f7fffff] because
it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
[ 0.493255] pnp 00:0b: disabling [mem 0x00000000-0x0009ffff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493260] pnp 00:0b: disabling [mem 0x000c0000-0x000c3fff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493265] pnp 00:0b: disabling [mem 0x000c8000-0x000cbfff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493270] pnp 00:0b: disabling [mem 0x000d0000-0x000d3fff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493274] pnp 00:0b: disabling [mem 0x000d8000-0x000dbfff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493279] pnp 00:0b: disabling [mem 0x000e0000-0x000e3fff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493283] pnp 00:0b: disabling [mem 0x000e8000-0x000ebfff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493288] pnp 00:0b: disabling [mem 0x000f0000-0x000fffff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493292] pnp 00:0b: disabling [mem 0x00100000-0x8f7fffff disabled]
because it overlaps 0000:09:00.0 BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 0.493297] pnp 00:0b: disabling [mem 0x00000000-0x0009ffff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493302] pnp 00:0b: disabling [mem 0x000c0000-0x000c3fff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493306] pnp 00:0b: disabling [mem 0x000c8000-0x000cbfff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493311] pnp 00:0b: disabling [mem 0x000d0000-0x000d3fff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493315] pnp 00:0b: disabling [mem 0x000d8000-0x000dbfff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493320] pnp 00:0b: disabling [mem 0x000e0000-0x000e3fff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493324] pnp 00:0b: disabling [mem 0x000e8000-0x000ebfff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493329] pnp 00:0b: disabling [mem 0x000f0000-0x000fffff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.493333] pnp 00:0b: disabling [mem 0x00100000-0x8f7fffff disabled]
because it overlaps 0000:09:00.0 BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 0.503894] pci 0000:09:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 0.503940] pci 0000:09:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 0.503963] pci 0000:09:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
[ 0.503972] pci 0000:09:00.0: ROM [mem 0xa5000000-0xa507ffff pref]:
assigned
[ 0.503984] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 0.503987] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 0.504331] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 0.504334] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 0.504704] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 0.504707] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 0.505073] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 0.505076] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 0.505441] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 0.505444] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 0.505810] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 0.505813] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 0.507057] pci 0000:09:00.1: D0 power state depends on 0000:09:00.0
[ 0.509437] pci 0000:09:00.0: Adding to iommu group 23
[ 2.833427] nvidia 0000:09:00.0: enabling device (0000 -> 0002)
[ 2.833519] nvidia 0000:09:00.0: vgaarb: VGA decodes changed:
olddecodes=io+mem,decodes=none:owns=none
[ 4.954613] [drm] Initialized nvidia-drm 0.0.0 20160202 for
0000:09:00.0 on minor 2
[ 228.414765] NVRM: GPU 0000:09:00.0: GPU has fallen off the bus.
[ 228.445633] pci 0000:09:00.0: Unable to change power state from
unknown to D0, device inaccessible
[ 233.991103] pci 0000:09:00.0: [10de:2204] type 00 class 0x030000 PCIe
Legacy Endpoint
[ 233.993053] pci 0000:09:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
[ 233.994986] pci 0000:09:00.0: BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 233.996854] pci 0000:09:00.0: BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 233.998727] pci 0000:09:00.0: BAR 5 [io 0x0000-0x007f]
[ 234.000585] pci 0000:09:00.0: ROM [mem 0x00000000-0x0007ffff pref]
[ 234.002720] pci 0000:09:00.0: PME# supported from D0 D3hot
[ 234.004889] pci 0000:09:00.0: 8.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x4 link at 0000:05:01.0 (capable of 252.048
Gb/s with 16.0 GT/s PCIe x16 link)
[ 234.007000] pci 0000:09:00.0: Adding to iommu group 23
[ 234.008925] pci 0000:09:00.0: vgaarb: bridge control possible
[ 234.010828] pci 0000:09:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none
[ 234.087850] pci 0000:09:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
64bit pref]: assigned
[ 234.089631] pci 0000:09:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
64bit pref]: assigned
[ 234.091492] pci 0000:09:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
[ 234.093241] pci 0000:09:00.0: ROM [mem 0xa5000000-0xa507ffff pref]:
assigned
[ 234.096831] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 234.098652] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 234.155043] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 234.156615] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 234.183809] nvidia 0000:09:00.0: enabling device (0000 -> 0002)
[ 234.185579] nvidia 0000:09:00.0: vgaarb: VGA decodes changed:
olddecodes=io+mem,decodes=none:owns=none
[ 234.310173] pci 0000:09:00.1: D0 power state depends on 0000:09:00.0
And doing the same for the 2nd GPU:
p53 ~ # dmesg | grep 2f:00.0
[ 1.215862] pci 0000:2f:00.0: [10de:2204] type 00 class 0x030000 PCIe
Legacy Endpoint
[ 1.215893] pci 0000:2f:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
[ 1.215918] pci 0000:2f:00.0: BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 1.215942] pci 0000:2f:00.0: BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 1.215956] pci 0000:2f:00.0: BAR 5 [io 0x0000-0x007f]
[ 1.215970] pci 0000:2f:00.0: ROM [mem 0x00000000-0x0007ffff pref]
[ 1.216765] pci 0000:2f:00.0: PME# supported from D0 D3hot
[ 1.217000] pci 0000:2f:00.0: 8.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x4 link at 0000:05:04.0 (capable of 252.048
Gb/s with 16.0 GT/s PCIe x16 link)
[ 1.217226] pci 0000:2f:00.0: Adding to iommu group 29
[ 1.217237] pci 0000:2f:00.0: vgaarb: bridge control possible
[ 1.217238] pci 0000:2f:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none
[ 1.218458] pci 0000:2f:00.0: BAR 1 [mem 0x6020000000-0x602fffffff
64bit pref]: assigned
[ 1.218481] pci 0000:2f:00.0: BAR 3 [mem 0x6030000000-0x6031ffffff
64bit pref]: assigned
[ 1.218501] pci 0000:2f:00.0: BAR 0 [mem 0xb1000000-0xb1ffffff]: assigned
[ 1.218507] pci 0000:2f:00.0: ROM [mem 0xb0800000-0xb087ffff pref]:
assigned
[ 1.218514] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 1.218514] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 1.218579] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 1.218580] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 1.219748] pci 0000:2f:00.1: D0 power state depends on 0000:2f:00.0
[ 2.883186] nvidia 0000:2f:00.0: enabling device (0000 -> 0002)
[ 2.883945] nvidia 0000:2f:00.0: vgaarb: VGA decodes changed:
olddecodes=io+mem,decodes=none:owns=none
[ 6.367931] [drm] Initialized nvidia-drm 0.0.0 20160202 for
0000:2f:00.0 on minor 3
[ 485.913085] NVRM: GPU 0000:2f:00.0: GPU has fallen off the bus.
[ 485.913727] NVRM: GPU 0000:2f:00.0: GPU serial number is PKWUQ0B9VFK0SG.
[ 485.938963] pci 0000:2f:00.0: Unable to change power state from
unknown to D0, device inaccessible
[ 489.941767] pci 0000:2f:00.0: [10de:2204] type 00 class 0x030000 PCIe
Legacy Endpoint
[ 489.944551] pci 0000:2f:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
[ 489.947287] pci 0000:2f:00.0: BAR 1 [mem 0x00000000-0x0fffffff 64bit
pref]
[ 489.950056] pci 0000:2f:00.0: BAR 3 [mem 0x00000000-0x01ffffff 64bit
pref]
[ 489.952835] pci 0000:2f:00.0: BAR 5 [io 0x0000-0x007f]
[ 489.955655] pci 0000:2f:00.0: ROM [mem 0x00000000-0x0007ffff pref]
[ 489.958721] pci 0000:2f:00.0: PME# supported from D0 D3hot
[ 489.961746] pci 0000:2f:00.0: 8.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x4 link at 0000:05:04.0 (capable of 252.048
Gb/s with 16.0 GT/s PCIe x16 link)
[ 489.964859] pci 0000:2f:00.0: Adding to iommu group 29
[ 489.967703] pci 0000:2f:00.0: vgaarb: bridge control possible
[ 489.970506] pci 0000:2f:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none
[ 490.025678] pci 0000:2f:00.0: BAR 1 [mem 0x6020000000-0x602fffffff
64bit pref]: assigned
[ 490.027887] pci 0000:2f:00.0: BAR 3 [mem 0x6030000000-0x6031ffffff
64bit pref]: assigned
[ 490.029918] pci 0000:2f:00.0: BAR 0 [mem 0xb1000000-0xb1ffffff]: assigned
[ 490.031940] pci 0000:2f:00.0: ROM [mem 0xb0800000-0xb087ffff pref]:
assigned
[ 490.036008] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 490.038096] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 490.075208] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't assign;
no space
[ 490.077005] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to assign
[ 490.099288] nvidia 0000:2f:00.0: enabling device (0000 -> 0002)
[ 490.101217] nvidia 0000:2f:00.0: vgaarb: VGA decodes changed:
olddecodes=io+mem,decodes=none:owns=none
[ 490.265952] pci 0000:2f:00.1: D0 power state depends on 0000:2f:00.0
BAR 5 is missing in the lspci output. Same for both.
lspci specifies 'Physical Resizable'. Is that implied for all BARs?
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 256MB, supported: 64MB 128MB 256MB 512MB
1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-18 10:42 ` Dag B
@ 2024-04-18 12:24 ` Christian König
2024-04-18 13:13 ` Dag B
0 siblings, 1 reply; 10+ messages in thread
From: Christian König @ 2024-04-18 12:24 UTC (permalink / raw)
To: Dag B, Bjorn Helgaas; +Cc: linux-pci
Am 18.04.24 um 12:42 schrieb Dag B:
>
> [SNIP]
>>
>>>>
>>>> Is there a good ELI13 resource explaining how resizable BAR works
>>>> in Linux?
>>>>
>>>> My current kernel command-line contains: pci=assign-busses,realloc
>>
>> That's a really really bad idea. The "assign-busses" flag was
>> introduced to get 20year old laptops to see their cardbus PCI devices.
>
> I threw a lot of mud at the wall to see what stuck. Removing it now
> did not make a big difference.
>
> Removing realloc prevents the second TB3 GPU from being initialized,
> so keeping that for now.
That's really interesting. Why does it fail without that?
It basically means that your BIOS is somehow broken and only the Linux
PCI subsystem is able to assign resources correctly.
Please provide the output of "sudo lspci -v" and "sudo lspci -tv" as
file attachment (*not* inline in a mail!).
>>>> My GPU is attached via TB3 to a system for which resizable BAR is
>>>> and will
>>>> remain a foreign concept in the BIOS.
>>
>> What happens if you hot remove and re-plug the TB3 after the system
>> has started?
>>
> Much the same as during initial boot. Both good and bad. See below.
>
>
> Do any of the pci=hp* options have any significance/impact on what
> dmesg says below?
No, the pci=hp* options are a hint for the PCI subsystem how much
address space to assign to each hot plugged bridge and device from the
upstream bridge.
But if your BIOS doesn't assign anything to the upstream bridge or your
don't have a window big enough on the root complex you are pretty much
busted from the beginning.
And that you get all those "pnp 00:0b: disabling" message also doesn't
makes the BIOS trustworthy.
>
> Is IO address space moveable?
No, the IO address space is usually only assigned when the device starts
with VGA emulation. And as long as VGA emulation is active you can't
move or resize anything.
What drivers usually do is to turn of the VGA emulation (which also
disables the IO address space) and then resize.
> Relevant kernel config/options impacting this? Is it all in the hands
> of the device driver?
Well the pci=hp* options are already all you need. But I don't think
they will help in this case here.
>
> So, so many questions. And barely competent to ask them. Please
> forgive me.
>
>
> Current kernel command-line snippet:
> pci=realloc,hpiosize=16K,hpmemsize=64M,pcie_scan_all,hpbussize=0x33
>
>
> I very much appreciate your input. Will try to get the attention of
> the people responsible for the driver.
>
>
> Thanks,
>
>
> Dag B
>
>
> p53 ~ # dmesg | grep 09:00.0
> [ 0.471780] pci 0000:09:00.0: [10de:2204] type 00 class 0x030000
> PCIe Legacy Endpoint
> [ 0.471816] pci 0000:09:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
> [ 0.471844] pci 0000:09:00.0: BAR 1 [mem 0x00000000-0x0fffffff
> 64bit pref]
> [ 0.471873] pci 0000:09:00.0: BAR 3 [mem 0x00000000-0x01ffffff
> 64bit pref]
> [ 0.471890] pci 0000:09:00.0: BAR 5 [io 0x0000-0x007f]
> [ 0.471907] pci 0000:09:00.0: ROM [mem 0x00000000-0x0007ffff pref]
> [ 0.472133] pci 0000:09:00.0: PME# supported from D0 D3hot
> [ 0.472382] pci 0000:09:00.0: 8.000 Gb/s available PCIe bandwidth,
> limited by 2.5 GT/s PCIe x4 link at 0000:05:01.0 (capable of 252.048
> Gb/s with 16.0 GT/s PCIe x16 link)
> [ 0.491866] pci 0000:09:00.0: vgaarb: bridge control possible
> [ 0.491866] pci 0000:09:00.0: vgaarb: VGA device added:
> decodes=io+mem,owns=none,locks=none
> [ 0.491866] pnp 00:03: disabling [io 0x002e-0x002f] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:03: disabling [io 0x004e-0x004f] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:03: disabling [io 0x0061] because it overlaps
> 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:03: disabling [io 0x0063] because it overlaps
> 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:03: disabling [io 0x0065] because it overlaps
> 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:03: disabling [io 0x0067] because it overlaps
> 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:03: disabling [io 0x0070] because it overlaps
> 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0010-0x001f] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0024-0x0025] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0028-0x0029] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x002c-0x002d] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0030-0x0031] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0034-0x0035] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0038-0x0039] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x003c-0x003d] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0050-0x0053] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.491866] pnp 00:08: disabling [io 0x0072-0x0077] because it
> overlaps 0000:09:00.0 BAR 5 [io 0x0000-0x007f]
> [ 0.493216] pnp 00:0b: disabling [mem 0x00000000-0x0009ffff]
> because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
> [ 0.493220] pnp 00:0b: disabling [mem 0x000c0000-0x000c3fff
> disabled] because it overlaps 0000:09:00.0 BAR 0 [mem
> 0x00000000-0x00ffffff]
> [ 0.493225] pnp 00:0b: disabling [mem 0x000c8000-0x000cbfff
> disabled] because it overlaps 0000:09:00.0 BAR 0 [mem
> 0x00000000-0x00ffffff]
> [ 0.493230] pnp 00:0b: disabling [mem 0x000d0000-0x000d3fff
> disabled] because it overlaps 0000:09:00.0 BAR 0 [mem
> 0x00000000-0x00ffffff]
> [ 0.493234] pnp 00:0b: disabling [mem 0x000d8000-0x000dbfff
> disabled] because it overlaps 0000:09:00.0 BAR 0 [mem
> 0x00000000-0x00ffffff]
> [ 0.493238] pnp 00:0b: disabling [mem 0x000e0000-0x000e3fff]
> because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
> [ 0.493242] pnp 00:0b: disabling [mem 0x000e8000-0x000ebfff]
> because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
> [ 0.493247] pnp 00:0b: disabling [mem 0x000f0000-0x000fffff]
> because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
> [ 0.493251] pnp 00:0b: disabling [mem 0x00100000-0x8f7fffff]
> because it overlaps 0000:09:00.0 BAR 0 [mem 0x00000000-0x00ffffff]
> [ 0.493255] pnp 00:0b: disabling [mem 0x00000000-0x0009ffff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493260] pnp 00:0b: disabling [mem 0x000c0000-0x000c3fff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493265] pnp 00:0b: disabling [mem 0x000c8000-0x000cbfff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493270] pnp 00:0b: disabling [mem 0x000d0000-0x000d3fff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493274] pnp 00:0b: disabling [mem 0x000d8000-0x000dbfff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493279] pnp 00:0b: disabling [mem 0x000e0000-0x000e3fff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493283] pnp 00:0b: disabling [mem 0x000e8000-0x000ebfff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493288] pnp 00:0b: disabling [mem 0x000f0000-0x000fffff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493292] pnp 00:0b: disabling [mem 0x00100000-0x8f7fffff
> disabled] because it overlaps 0000:09:00.0 BAR 1 [mem
> 0x00000000-0x0fffffff 64bit pref]
> [ 0.493297] pnp 00:0b: disabling [mem 0x00000000-0x0009ffff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493302] pnp 00:0b: disabling [mem 0x000c0000-0x000c3fff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493306] pnp 00:0b: disabling [mem 0x000c8000-0x000cbfff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493311] pnp 00:0b: disabling [mem 0x000d0000-0x000d3fff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493315] pnp 00:0b: disabling [mem 0x000d8000-0x000dbfff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493320] pnp 00:0b: disabling [mem 0x000e0000-0x000e3fff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493324] pnp 00:0b: disabling [mem 0x000e8000-0x000ebfff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493329] pnp 00:0b: disabling [mem 0x000f0000-0x000fffff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.493333] pnp 00:0b: disabling [mem 0x00100000-0x8f7fffff
> disabled] because it overlaps 0000:09:00.0 BAR 3 [mem
> 0x00000000-0x01ffffff 64bit pref]
> [ 0.503894] pci 0000:09:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
> 64bit pref]: assigned
> [ 0.503940] pci 0000:09:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
> 64bit pref]: assigned
> [ 0.503963] pci 0000:09:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]:
> assigned
> [ 0.503972] pci 0000:09:00.0: ROM [mem 0xa5000000-0xa507ffff pref]:
> assigned
> [ 0.503984] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 0.503987] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 0.504331] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 0.504334] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 0.504704] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 0.504707] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 0.505073] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 0.505076] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 0.505441] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 0.505444] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 0.505810] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 0.505813] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 0.507057] pci 0000:09:00.1: D0 power state depends on 0000:09:00.0
> [ 0.509437] pci 0000:09:00.0: Adding to iommu group 23
> [ 2.833427] nvidia 0000:09:00.0: enabling device (0000 -> 0002)
> [ 2.833519] nvidia 0000:09:00.0: vgaarb: VGA decodes changed:
> olddecodes=io+mem,decodes=none:owns=none
> [ 4.954613] [drm] Initialized nvidia-drm 0.0.0 20160202 for
> 0000:09:00.0 on minor 2
> [ 228.414765] NVRM: GPU 0000:09:00.0: GPU has fallen off the bus.
> [ 228.445633] pci 0000:09:00.0: Unable to change power state from
> unknown to D0, device inaccessible
> [ 233.991103] pci 0000:09:00.0: [10de:2204] type 00 class 0x030000
> PCIe Legacy Endpoint
> [ 233.993053] pci 0000:09:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
> [ 233.994986] pci 0000:09:00.0: BAR 1 [mem 0x00000000-0x0fffffff
> 64bit pref]
> [ 233.996854] pci 0000:09:00.0: BAR 3 [mem 0x00000000-0x01ffffff
> 64bit pref]
> [ 233.998727] pci 0000:09:00.0: BAR 5 [io 0x0000-0x007f]
> [ 234.000585] pci 0000:09:00.0: ROM [mem 0x00000000-0x0007ffff pref]
> [ 234.002720] pci 0000:09:00.0: PME# supported from D0 D3hot
> [ 234.004889] pci 0000:09:00.0: 8.000 Gb/s available PCIe bandwidth,
> limited by 2.5 GT/s PCIe x4 link at 0000:05:01.0 (capable of 252.048
> Gb/s with 16.0 GT/s PCIe x16 link)
> [ 234.007000] pci 0000:09:00.0: Adding to iommu group 23
> [ 234.008925] pci 0000:09:00.0: vgaarb: bridge control possible
> [ 234.010828] pci 0000:09:00.0: vgaarb: VGA device added:
> decodes=io+mem,owns=none,locks=none
> [ 234.087850] pci 0000:09:00.0: BAR 1 [mem 0x6000000000-0x600fffffff
> 64bit pref]: assigned
> [ 234.089631] pci 0000:09:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff
> 64bit pref]: assigned
> [ 234.091492] pci 0000:09:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]:
> assigned
> [ 234.093241] pci 0000:09:00.0: ROM [mem 0xa5000000-0xa507ffff pref]:
> assigned
> [ 234.096831] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 234.098652] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 234.155043] pci 0000:09:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 234.156615] pci 0000:09:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 234.183809] nvidia 0000:09:00.0: enabling device (0000 -> 0002)
> [ 234.185579] nvidia 0000:09:00.0: vgaarb: VGA decodes changed:
> olddecodes=io+mem,decodes=none:owns=none
> [ 234.310173] pci 0000:09:00.1: D0 power state depends on 0000:09:00.0
>
>
>
> And doing the same for the 2nd GPU:
>
> p53 ~ # dmesg | grep 2f:00.0
> [ 1.215862] pci 0000:2f:00.0: [10de:2204] type 00 class 0x030000
> PCIe Legacy Endpoint
> [ 1.215893] pci 0000:2f:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
> [ 1.215918] pci 0000:2f:00.0: BAR 1 [mem 0x00000000-0x0fffffff
> 64bit pref]
> [ 1.215942] pci 0000:2f:00.0: BAR 3 [mem 0x00000000-0x01ffffff
> 64bit pref]
> [ 1.215956] pci 0000:2f:00.0: BAR 5 [io 0x0000-0x007f]
> [ 1.215970] pci 0000:2f:00.0: ROM [mem 0x00000000-0x0007ffff pref]
> [ 1.216765] pci 0000:2f:00.0: PME# supported from D0 D3hot
> [ 1.217000] pci 0000:2f:00.0: 8.000 Gb/s available PCIe bandwidth,
> limited by 2.5 GT/s PCIe x4 link at 0000:05:04.0 (capable of 252.048
> Gb/s with 16.0 GT/s PCIe x16 link)
> [ 1.217226] pci 0000:2f:00.0: Adding to iommu group 29
> [ 1.217237] pci 0000:2f:00.0: vgaarb: bridge control possible
> [ 1.217238] pci 0000:2f:00.0: vgaarb: VGA device added:
> decodes=io+mem,owns=none,locks=none
> [ 1.218458] pci 0000:2f:00.0: BAR 1 [mem 0x6020000000-0x602fffffff
> 64bit pref]: assigned
> [ 1.218481] pci 0000:2f:00.0: BAR 3 [mem 0x6030000000-0x6031ffffff
> 64bit pref]: assigned
> [ 1.218501] pci 0000:2f:00.0: BAR 0 [mem 0xb1000000-0xb1ffffff]:
> assigned
> [ 1.218507] pci 0000:2f:00.0: ROM [mem 0xb0800000-0xb087ffff pref]:
> assigned
> [ 1.218514] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 1.218514] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 1.218579] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 1.218580] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 1.219748] pci 0000:2f:00.1: D0 power state depends on 0000:2f:00.0
> [ 2.883186] nvidia 0000:2f:00.0: enabling device (0000 -> 0002)
> [ 2.883945] nvidia 0000:2f:00.0: vgaarb: VGA decodes changed:
> olddecodes=io+mem,decodes=none:owns=none
> [ 6.367931] [drm] Initialized nvidia-drm 0.0.0 20160202 for
> 0000:2f:00.0 on minor 3
> [ 485.913085] NVRM: GPU 0000:2f:00.0: GPU has fallen off the bus.
> [ 485.913727] NVRM: GPU 0000:2f:00.0: GPU serial number is
> PKWUQ0B9VFK0SG.
> [ 485.938963] pci 0000:2f:00.0: Unable to change power state from
> unknown to D0, device inaccessible
> [ 489.941767] pci 0000:2f:00.0: [10de:2204] type 00 class 0x030000
> PCIe Legacy Endpoint
> [ 489.944551] pci 0000:2f:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
> [ 489.947287] pci 0000:2f:00.0: BAR 1 [mem 0x00000000-0x0fffffff
> 64bit pref]
> [ 489.950056] pci 0000:2f:00.0: BAR 3 [mem 0x00000000-0x01ffffff
> 64bit pref]
> [ 489.952835] pci 0000:2f:00.0: BAR 5 [io 0x0000-0x007f]
> [ 489.955655] pci 0000:2f:00.0: ROM [mem 0x00000000-0x0007ffff pref]
> [ 489.958721] pci 0000:2f:00.0: PME# supported from D0 D3hot
> [ 489.961746] pci 0000:2f:00.0: 8.000 Gb/s available PCIe bandwidth,
> limited by 2.5 GT/s PCIe x4 link at 0000:05:04.0 (capable of 252.048
> Gb/s with 16.0 GT/s PCIe x16 link)
> [ 489.964859] pci 0000:2f:00.0: Adding to iommu group 29
> [ 489.967703] pci 0000:2f:00.0: vgaarb: bridge control possible
> [ 489.970506] pci 0000:2f:00.0: vgaarb: VGA device added:
> decodes=io+mem,owns=none,locks=none
> [ 490.025678] pci 0000:2f:00.0: BAR 1 [mem 0x6020000000-0x602fffffff
> 64bit pref]: assigned
> [ 490.027887] pci 0000:2f:00.0: BAR 3 [mem 0x6030000000-0x6031ffffff
> 64bit pref]: assigned
> [ 490.029918] pci 0000:2f:00.0: BAR 0 [mem 0xb1000000-0xb1ffffff]:
> assigned
> [ 490.031940] pci 0000:2f:00.0: ROM [mem 0xb0800000-0xb087ffff pref]:
> assigned
> [ 490.036008] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 490.038096] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 490.075208] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: can't
> assign; no space
> [ 490.077005] pci 0000:2f:00.0: BAR 5 [io size 0x0080]: failed to
> assign
> [ 490.099288] nvidia 0000:2f:00.0: enabling device (0000 -> 0002)
> [ 490.101217] nvidia 0000:2f:00.0: vgaarb: VGA decodes changed:
> olddecodes=io+mem,decodes=none:owns=none
> [ 490.265952] pci 0000:2f:00.1: D0 power state depends on 0000:2f:00.0
>
>
> BAR 5 is missing in the lspci output. Same for both.
>
> lspci specifies 'Physical Resizable'. Is that implied for all BARs?
No, that means only BAR1 is resizeable.
Regards,
Christian.
>
> Capabilities: [bb0 v1] Physical Resizable BAR
> BAR 0: current size: 16MB, supported: 16MB
> BAR 1: current size: 256MB, supported: 64MB 128MB 256MB 512MB
> 1GB 2GB 4GB 8GB 16GB 32GB
> BAR 3: current size: 32MB, supported: 32MB
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-18 12:24 ` Christian König
@ 2024-04-18 13:13 ` Dag B
2024-04-18 22:54 ` Dag B
2024-04-19 15:19 ` Ilpo Järvinen
0 siblings, 2 replies; 10+ messages in thread
From: Dag B @ 2024-04-18 13:13 UTC (permalink / raw)
To: Christian König, Bjorn Helgaas; +Cc: linux-pci
[-- Attachment #1: Type: text/plain, Size: 1551 bytes --]
On 18.04.2024 14:24, Christian König wrote:
> Am 18.04.24 um 12:42 schrieb Dag B:
>>
>> [SNIP]
>>>
>>>>>
>>>>> Is there a good ELI13 resource explaining how resizable BAR works
>>>>> in Linux?
>>>>>
>>>>> My current kernel command-line contains: pci=assign-busses,realloc
>>>
>>> That's a really really bad idea. The "assign-busses" flag was
>>> introduced to get 20year old laptops to see their cardbus PCI devices.
>>
>> I threw a lot of mud at the wall to see what stuck. Removing it now
>> did not make a big difference.
>>
>> Removing realloc prevents the second TB3 GPU from being initialized,
>> so keeping that for now.
>
> That's really interesting. Why does it fail without that?
>
> It basically means that your BIOS is somehow broken and only the Linux
> PCI subsystem is able to assign resources correctly.
>
> Please provide the output of "sudo lspci -v" and "sudo lspci -tv" as
> file attachment (*not* inline in a mail!).
In case I have expressed myself awkwardly, the realloc=off case appears
to make the device driver have issues with the second GPU.
I have attached both outputs, for realloc=off.
Not knowing what is considered acceptable message sizes on this m/l, I
uploaded the same for realloc=on, as well as output from dmesg for both
cases to:
https://github.com/dagbdagb/p53
If the m/l has mechanisms to archive attachments and replace them with
links, I'll redo the exercise in a follow-up email. I understand the
value of having the 'context' of the discussion readily available in one
place.
Dag B
[-- Attachment #2: lspci-v-realloc-off.txt --]
[-- Type: text/plain, Size: 38914 bytes --]
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 0d)
Subsystem: Lenovo 8th Gen Core Processor Host Bridge/DRAM Registers
Flags: bus master, fast devsel, latency 0, IOMMU group 1
Capabilities: [e0] Vendor Specific Information: Len=10 <?>
Kernel driver in use: skl_uncore
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d) (prog-if 00 [Normal decode])
Subsystem: Lenovo 6th-10th Gen Core Processor PCIe Controller (x16)
Flags: bus master, fast devsel, latency 0, IRQ 122, IOMMU group 2
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 2000-2fff [size=4K] [16-bit]
Memory behind bridge: bd000000-be0fffff [size=17M] [32-bit]
Prefetchable memory behind bridge: 90000000-a1ffffff [size=288M] [32-bit]
Capabilities: [88] Subsystem: Lenovo 6th-10th Gen Core Processor PCIe Controller (x16)
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), IntMsgNum 0
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Capabilities: [d94] Secondary PCI Express
Kernel driver in use: pcieport
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-H GT2 [UHD Graphics 630] (rev 02) (prog-if 00 [VGA controller])
Subsystem: Lenovo CoffeeLake-H GT2 [UHD Graphics 630]
Flags: bus master, fast devsel, latency 0, IRQ 133, IOMMU group 0
Memory at 6040000000 (64-bit, non-prefetchable) [size=16M]
Memory at 4000000000 (64-bit, prefetchable) [size=256M]
I/O ports at 3000 [size=64]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express Root Complex Integrated Endpoint, IntMsgNum 0
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [d0] Power Management version 2
Capabilities: [100] Process Address Space ID (PASID)
Capabilities: [200] Address Translation Service (ATS)
Capabilities: [300] Page Request Interface (PRI)
Kernel driver in use: i915
00:04.0 Signal processing controller: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem (rev 0d)
Subsystem: Lenovo Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem
Flags: fast devsel, IRQ 16, IOMMU group 3
Memory at 6041100000 (64-bit, non-prefetchable) [size=32K]
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [d0] Power Management version 3
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel driver in use: proc_thermal
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
Subsystem: Lenovo Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
Flags: fast devsel, IRQ 255, IOMMU group 4
Memory at 6041115000 (64-bit, non-prefetchable) [disabled] [size=4K]
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [dc] Power Management version 2
Capabilities: [f0] PCI Advanced Features
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
Subsystem: Lenovo Cannon Lake PCH Thermal Controller
Flags: fast devsel, IRQ 16, IOMMU group 5
Memory at 6041114000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Kernel driver in use: intel_pch_thermal
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) (prog-if 30 [XHCI])
Subsystem: Lenovo Cannon Lake PCH USB 3.1 xHCI Host Controller
Flags: bus master, medium devsel, latency 0, IRQ 150, IOMMU group 6
Memory at be420000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=8/8 Maskable- 64bit+
Capabilities: [90] Vendor Specific Information: Len=14 <?>
Kernel driver in use: xhci_hcd
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
Subsystem: Lenovo Cannon Lake PCH Shared SRAM
Flags: fast devsel, IOMMU group 6
Memory at 604110c000 (64-bit, non-prefetchable) [disabled] [size=8K]
Memory at 6041113000 (64-bit, non-prefetchable) [disabled] [size=4K]
Capabilities: [80] Power Management version 3
00:15.0 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
Subsystem: Lenovo Cannon Lake PCH Serial IO I2C Controller
Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 7
Memory at 6041112000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
Kernel driver in use: intel-lpss
00:15.1 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
Subsystem: Lenovo Cannon Lake PCH Serial IO I2C Controller
Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 7
Memory at 6041111000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
Kernel driver in use: intel-lpss
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
Subsystem: Lenovo Cannon Lake PCH HECI Controller
Flags: bus master, fast devsel, latency 0, IRQ 135, IOMMU group 8
Memory at 6041110000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [a4] Vendor Specific Information: Len=14 <?>
Kernel driver in use: mei_me
00:16.3 Serial controller: Intel Corporation Cannon Lake PCH Active Management Technology - SOL (rev 10) (prog-if 02 [16550])
Subsystem: Lenovo Cannon Lake PCH Active Management Technology - SOL
Flags: 66MHz, fast devsel, IRQ 255, IOMMU group 8
I/O ports at 3060 [size=8]
Memory at be431000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [50] Power Management version 3
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0) (prog-if 00 [Normal decode])
Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Flags: bus master, fast devsel, latency 0, IRQ 123, IOMMU group 9
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: [disabled] [16-bit]
Memory behind bridge: be300000-be3fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [40] Express Root Port (Slot+), IntMsgNum 0
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Capabilities: [a0] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Access Control Services
Capabilities: [150] Precision Time Measurement
Capabilities: [200] L1 PM Substates
Capabilities: [220] Secondary PCI Express
Capabilities: [250] Downstream Port Containment
Kernel driver in use: pcieport
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0) (prog-if 00 [Normal decode])
Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Flags: bus master, fast devsel, latency 0, IRQ 124, IOMMU group 10
Bus: primary=00, secondary=04, subordinate=51, sec-latency=0
I/O behind bridge: 4000-4fff [size=4K] [16-bit]
Memory behind bridge: a4000000-bc1fffff [size=386M] [32-bit]
Prefetchable memory behind bridge: 6000000000-603fffffff [size=1G] [32-bit]
Capabilities: [40] Express Root Port (Slot+), IntMsgNum 0
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Capabilities: [a0] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Access Control Services
Capabilities: [150] Precision Time Measurement
Capabilities: [220] Secondary PCI Express
Capabilities: [250] Downstream Port Containment
Kernel driver in use: pcieport
00:1c.5 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0) (prog-if 00 [Normal decode])
Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Flags: bus master, fast devsel, latency 0, IRQ 125, IOMMU group 11
Bus: primary=00, secondary=52, subordinate=52, sec-latency=0
I/O behind bridge: [disabled] [16-bit]
Memory behind bridge: be200000-be2fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [40] Express Root Port (Slot+), IntMsgNum 0
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Capabilities: [a0] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Access Control Services
Capabilities: [150] Precision Time Measurement
Capabilities: [200] L1 PM Substates
Capabilities: [220] Secondary PCI Express
Capabilities: [250] Downstream Port Containment
Kernel driver in use: pcieport
00:1c.7 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #8 (rev f0) (prog-if 00 [Normal decode])
Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Flags: bus master, fast devsel, latency 0, IRQ 126, IOMMU group 12
Bus: primary=00, secondary=54, subordinate=54, sec-latency=0
I/O behind bridge: [disabled] [16-bit]
Memory behind bridge: be100000-be1fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [40] Express Root Port (Slot+), IntMsgNum 0
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Lenovo Cannon Lake PCH PCI Express Root Port
Capabilities: [a0] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Access Control Services
Capabilities: [150] Precision Time Measurement
Capabilities: [200] L1 PM Substates
Capabilities: [220] Secondary PCI Express
Capabilities: [250] Downstream Port Containment
Kernel driver in use: pcieport
00:1e.0 Communication controller: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller (rev 10)
Subsystem: Lenovo Cannon Lake PCH Serial IO UART Host Controller
Flags: bus master, fast devsel, latency 0, IRQ 20, IOMMU group 13
Memory at 604110f000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
Kernel driver in use: intel-lpss
00:1f.0 ISA bridge: Intel Corporation Cannon Lake LPC Controller (rev 10)
Subsystem: Lenovo Cannon Lake LPC Controller
Flags: bus master, medium devsel, latency 0, IOMMU group 14
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10) (prog-if 80)
Subsystem: Lenovo Cannon Lake PCH cAVS
Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 14
Memory at 6041108000 (64-bit, non-prefetchable) [size=16K]
Memory at 6041000000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [50] Power Management version 3
Capabilities: [80] Vendor Specific Information: Len=14 <?>
Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
Subsystem: Lenovo Cannon Lake PCH SMBus Controller
Flags: medium devsel, IRQ 16, IOMMU group 14
Memory at 604110e000 (64-bit, non-prefetchable) [size=256]
I/O ports at efa0 [size=32]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
Subsystem: Lenovo Cannon Lake PCH SPI Controller
Flags: fast devsel, IOMMU group 14
Memory at be430000 (32-bit, non-prefetchable) [size=4K]
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
Subsystem: Lenovo Ethernet Connection (7) I219-LM
Flags: bus master, fast devsel, latency 0, IRQ 137, IOMMU group 14
Memory at be400000 (32-bit, non-prefetchable) [size=128K]
Capabilities: [c8] Power Management version 3
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Kernel driver in use: e1000e
01:00.0 VGA compatible controller: NVIDIA Corporation TU106GLM [Quadro RTX 3000 Mobile / Max-Q] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Lenovo TU106GLM [Quadro RTX 3000 Mobile / Max-Q]
Flags: bus master, fast devsel, latency 0, IRQ 158, IOMMU group 2
Memory at bd000000 (32-bit, non-prefetchable) [size=16M]
Memory at 90000000 (64-bit, prefetchable) [size=256M]
Memory at a0000000 (64-bit, prefetchable) [size=32M]
I/O ports at 2000 [size=128]
Expansion ROM at be080000 [virtual] [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, IntMsgNum 0
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Capabilities: [bb0] Physical Resizable BAR
Kernel driver in use: nvidia
Kernel modules: nvidia_drm, nvidia
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd SSD 970 EVO
Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 15
Memory at be300000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [70] Express Endpoint, IntMsgNum 0
Capabilities: [b0] MSI-X: Enable+ Count=33 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158] Power Budgeting <?>
Capabilities: [168] Secondary PCI Express
Capabilities: [188] Latency Tolerance Reporting
Capabilities: [190] L1 PM Substates
Kernel driver in use: nvme
04:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06) (prog-if 00 [Normal decode])
Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 16
Bus: primary=04, secondary=05, subordinate=51, sec-latency=0
I/O behind bridge: 4000-4fff [size=4K] [16-bit]
Memory behind bridge: a4000000-bc1fffff [size=386M] [32-bit]
Prefetchable memory behind bridge: 6000000000-603fffffff [size=1G] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Capabilities: [c0] Express Upstream Port, IntMsgNum 0
Capabilities: [50] Capability ID 0x15 [0000]
Capabilities: [100] Device Serial Number 49-dd-bd-31-cb-00-aa-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] Secondary PCI Express
Capabilities: [800] Latency Tolerance Reporting
Capabilities: [a00] L1 PM Substates
Capabilities: [b00] Precision Time Measurement
Kernel driver in use: pcieport
05:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06) (prog-if 00 [Normal decode])
Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Flags: bus master, fast devsel, latency 0, IRQ 127, IOMMU group 17
Bus: primary=05, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: bc100000-bc1fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [50] Capability ID 0x15 [0000]
Capabilities: [100] Device Serial Number 49-dd-bd-31-cb-00-aa-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] Secondary PCI Express
Capabilities: [900] Access Control Services
Kernel driver in use: pcieport
05:01.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06) (prog-if 00 [Normal decode])
Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Flags: bus master, fast devsel, latency 0, IRQ 128, IOMMU group 18
Bus: primary=05, secondary=07, subordinate=2b, sec-latency=0
I/O behind bridge: 4000-4fff [size=4K] [16-bit]
Memory behind bridge: a4000000-afffffff [size=192M] [32-bit]
Prefetchable memory behind bridge: 6000000000-601fffffff [size=512M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [50] Capability ID 0x15 [0000]
Capabilities: [100] Device Serial Number 49-dd-bd-31-cb-00-aa-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] Secondary PCI Express
Capabilities: [900] Access Control Services
Kernel driver in use: pcieport
05:02.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06) (prog-if 00 [Normal decode])
Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Flags: bus master, fast devsel, latency 0, IRQ 129, IOMMU group 19
Bus: primary=05, secondary=2c, subordinate=2c, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b0000000-b00fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [50] Capability ID 0x15 [0000]
Capabilities: [100] Device Serial Number 49-dd-bd-31-cb-00-aa-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] Secondary PCI Express
Capabilities: [900] Access Control Services
Kernel driver in use: pcieport
05:04.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06) (prog-if 00 [Normal decode])
Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Flags: bus master, fast devsel, latency 0, IRQ 130, IOMMU group 20
Bus: primary=05, secondary=2d, subordinate=51, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b0100000-bc0fffff [size=192M] [32-bit]
Prefetchable memory behind bridge: 6020000000-603fffffff [size=512M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Lenovo JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [50] Capability ID 0x15 [0000]
Capabilities: [100] Device Serial Number 49-dd-bd-31-cb-00-aa-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] Secondary PCI Express
Capabilities: [900] Access Control Services
Kernel driver in use: pcieport
06:00.0 System peripheral: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] (rev 06)
Subsystem: Lenovo JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018]
Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 21
Memory at bc100000 (32-bit, non-prefetchable) [size=256K]
Memory at bc140000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [c0] Express Endpoint, IntMsgNum 0
Capabilities: [a0] MSI-X: Enable- Count=16 Masked-
Capabilities: [100] Device Serial Number 49-dd-bd-31-cb-00-aa-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Latency Tolerance Reporting
07:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Physical Slot: 1
Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 22
Bus: primary=07, secondary=08, subordinate=2b, sec-latency=0
I/O behind bridge: 4000-4fff [size=4K] [16-bit]
Memory behind bridge: a4000000-afffffff [size=192M] [32-bit]
Prefetchable memory behind bridge: 6000000000-601fffffff [size=512M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Capabilities: [c0] Express Upstream Port, IntMsgNum 0
Capabilities: [100] Device Serial Number c4-52-1d-21-0f-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [600] Latency Tolerance Reporting
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
08:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Flags: bus master, fast devsel, latency 0, IRQ 131, IOMMU group 23
Bus: primary=08, secondary=09, subordinate=09, sec-latency=0
I/O behind bridge: 4000-4fff [size=4K] [16-bit]
Memory behind bridge: a4000000-a57fffff [size=24M] [32-bit]
Prefetchable memory behind bridge: 6000000000-6017ffffff [size=384M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [100] Device Serial Number c4-52-1d-21-0f-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
08:04.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Flags: bus master, fast devsel, latency 0, IRQ 132, IOMMU group 24
Bus: primary=08, secondary=0a, subordinate=2b, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: a5800000-afffffff [size=168M] [32-bit]
Prefetchable memory behind bridge: 6018000000-601fffffff [size=128M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [100] Device Serial Number c4-52-1d-21-0f-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
09:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Gigabyte Technology Co., Ltd GA102 [GeForce RTX 3090]
Flags: fast devsel, IOMMU group 23
Memory at a4000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Memory at 6000000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at 6010000000 (64-bit, prefetchable) [disabled] [size=32M]
I/O ports at 4000 [disabled] [size=128]
Expansion ROM at a5000000 [virtual] [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, IntMsgNum 0
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Capabilities: [bb0] Physical Resizable BAR
Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
Capabilities: [d00] Lane Margining at the Receiver
Capabilities: [e00] Data Link Feature <?>
Kernel modules: nvidia_drm, nvidia
0a:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Physical Slot: 4-1
Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 24
Bus: primary=0a, secondary=0b, subordinate=0e, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: a5800000-afffffff [size=168M] [32-bit]
Prefetchable memory behind bridge: 6018000000-601fffffff [size=128M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Capabilities: [c0] Express Upstream Port, IntMsgNum 0
Capabilities: [100] Device Serial Number c4-52-1c-c9-6e-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [600] Latency Tolerance Reporting
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
0b:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Flags: bus master, fast devsel, latency 0, IRQ 167, IOMMU group 24
Bus: primary=0b, secondary=0c, subordinate=0c, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: a5800000-a8ffffff [size=56M] [32-bit]
Prefetchable memory behind bridge: 6018000000-601a9fffff [size=42M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [100] Device Serial Number c4-52-1c-c9-6e-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
0b:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Flags: bus master, fast devsel, latency 0, IRQ 168, IOMMU group 24
Bus: primary=0b, secondary=0d, subordinate=0d, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: a9000000-ac7fffff [size=56M] [32-bit]
Prefetchable memory behind bridge: 601aa00000-601d3fffff [size=42M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [100] Device Serial Number c4-52-1c-c9-6e-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
0b:02.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Flags: bus master, fast devsel, latency 0, IRQ 169, IOMMU group 24
Bus: primary=0b, secondary=0e, subordinate=0e, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: ac800000-afffffff [size=56M] [32-bit]
Prefetchable memory behind bridge: 601d400000-601fdfffff [size=42M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [100] Device Serial Number c4-52-1c-c9-6e-b7-d0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
0c:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller (prog-if 30 [XHCI])
Subsystem: Razer USA Ltd. ASM1142 USB 3.1 Host Controller
Flags: fast devsel, IRQ 17, IOMMU group 24
Memory at a5800000 (64-bit, non-prefetchable) [size=32K]
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [68] MSI-X: Enable+ Count=8 Masked-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, IntMsgNum 0
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Capabilities: [280] Secondary PCI Express
Capabilities: [300] Latency Tolerance Reporting
Kernel driver in use: xhci_hcd
0d:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller (prog-if 30 [XHCI])
Subsystem: Razer USA Ltd. ASM1142 USB 3.1 Host Controller
Flags: fast devsel, IRQ 18, IOMMU group 24
Memory at a9000000 (64-bit, non-prefetchable) [size=32K]
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [68] MSI-X: Enable+ Count=8 Masked-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, IntMsgNum 0
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Capabilities: [280] Secondary PCI Express
Capabilities: [300] Latency Tolerance Reporting
Kernel driver in use: xhci_hcd
0e:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller (prog-if 30 [XHCI])
Subsystem: Razer USA Ltd. ASM1142 USB 3.1 Host Controller
Flags: fast devsel, IRQ 19, IOMMU group 24
Memory at ac800000 (64-bit, non-prefetchable) [size=32K]
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [68] MSI-X: Enable+ Count=8 Masked-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, IntMsgNum 0
Capabilities: [100] Virtual Channel
Capabilities: [200] Advanced Error Reporting
Capabilities: [280] Secondary PCI Express
Capabilities: [300] Latency Tolerance Reporting
Kernel driver in use: xhci_hcd
2c:00.0 USB controller: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] (rev 06) (prog-if 30 [XHCI])
Subsystem: Lenovo JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018]
Flags: fast devsel, IRQ 159, IOMMU group 25
Memory at b0000000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=8/8 Maskable- 64bit+
Capabilities: [c0] Express Endpoint, IntMsgNum 0
Capabilities: [100] Device Serial Number 49-dd-bd-31-cb-00-aa-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
Capabilities: [700] Secondary PCI Express
Capabilities: [800] Latency Tolerance Reporting
Kernel driver in use: xhci_hcd
2d:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
Physical Slot: 4
Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 28
Bus: primary=2d, secondary=2e, subordinate=2f, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b0800000-bbffffff [size=184M] [32-bit]
Prefetchable memory behind bridge: 6020000000-603fffffff [size=512M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
Capabilities: [c0] Express Upstream Port, IntMsgNum 0
Capabilities: [100] Device Serial Number 3a-ca-84-32-99-c9-a0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [600] Latency Tolerance Reporting
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
2e:01.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02) (prog-if 00 [Normal decode])
Subsystem: Razer USA Ltd. JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
Flags: bus master, fast devsel, latency 0, IRQ 194, IOMMU group 29
Bus: primary=2e, secondary=2f, subordinate=2f, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b0800000-bbffffff [size=184M] [32-bit]
Prefetchable memory behind bridge: 6020000000-603fffffff [size=512M] [32-bit]
Capabilities: [80] Power Management version 3
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [ac] Subsystem: Razer USA Ltd. JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016]
Capabilities: [c0] Express Downstream Port (Slot+), IntMsgNum 0
Capabilities: [100] Device Serial Number 3a-ca-84-32-99-c9-a0-00
Capabilities: [200] Advanced Error Reporting
Capabilities: [300] Virtual Channel
Capabilities: [400] Power Budgeting <?>
Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=0d8 <?>
Capabilities: [700] Secondary PCI Express
Kernel driver in use: pcieport
2f:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Hewlett-Packard Company GA102 [GeForce RTX 3090]
Flags: bus master, fast devsel, latency 0, IRQ 195, IOMMU group 29
Memory at b1000000 (32-bit, non-prefetchable) [size=16M]
Memory at 6020000000 (64-bit, prefetchable) [size=512M]
Memory at b2000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at b0800000 [virtual] [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, IntMsgNum 0
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Capabilities: [bb0] Physical Resizable BAR
Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
Capabilities: [d00] Lane Margining at the Receiver
Capabilities: [e00] Data Link Feature <?>
Kernel driver in use: nvidia
Kernel modules: nvidia_drm, nvidia
52:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
Subsystem: Intel Corporation Wi-Fi 6 AX200
Flags: fast devsel, IRQ 255, IOMMU group 26
Memory at be200000 (64-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [c8] Power Management version 3
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [40] Express Endpoint, IntMsgNum 0
Capabilities: [80] MSI-X: Enable- Count=16 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [14c] Latency Tolerance Reporting
Capabilities: [154] L1 PM Substates
54:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader (rev 01)
Subsystem: Lenovo RTS525A PCI Express Card Reader
Flags: bus master, fast devsel, latency 0, IRQ 134, IOMMU group 27
Memory at be100000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [b0] Express Endpoint, IntMsgNum 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-00-00-01-00-4c-e0-00
Capabilities: [158] Latency Tolerance Reporting
Capabilities: [160] L1 PM Substates
Kernel driver in use: rtsx_pci
[-- Attachment #3: lspci-tv-realloc-off.txt --]
[-- Type: text/plain, Size: 2794 bytes --]
-[0000:00]-+-00.0 Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers
+-01.0-[01]----00.0 NVIDIA Corporation TU106GLM [Quadro RTX 3000 Mobile / Max-Q]
+-02.0 Intel Corporation CoffeeLake-H GT2 [UHD Graphics 630]
+-04.0 Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem
+-08.0 Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
+-12.0 Intel Corporation Cannon Lake PCH Thermal Controller
+-14.0 Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller
+-14.2 Intel Corporation Cannon Lake PCH Shared SRAM
+-15.0 Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0
+-15.1 Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1
+-16.0 Intel Corporation Cannon Lake PCH HECI Controller
+-16.3 Intel Corporation Cannon Lake PCH Active Management Technology - SOL
+-1b.0-[02]----00.0 Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
+-1c.0-[04-51]----00.0-[05-51]--+-00.0-[06]----00.0 Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018]
| +-01.0-[07-2b]----00.0-[08-2b]--+-01.0-[09]----00.0 NVIDIA Corporation GA102 [GeForce RTX 3090]
| | \-04.0-[0a-2b]----00.0-[0b-0e]--+-00.0-[0c]----00.0 ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
| | +-01.0-[0d]----00.0 ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
| | \-02.0-[0e]----00.0 ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
| +-02.0-[2c]----00.0 Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018]
| \-04.0-[2d-51]----00.0-[2e-2f]----01.0-[2f]----00.0 NVIDIA Corporation GA102 [GeForce RTX 3090]
+-1c.5-[52]----00.0 Intel Corporation Wi-Fi 6 AX200
+-1c.7-[54]----00.0 Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader
+-1e.0 Intel Corporation Cannon Lake PCH Serial IO UART Host Controller
+-1f.0 Intel Corporation Cannon Lake LPC Controller
+-1f.3 Intel Corporation Cannon Lake PCH cAVS
+-1f.4 Intel Corporation Cannon Lake PCH SMBus Controller
+-1f.5 Intel Corporation Cannon Lake PCH SPI Controller
\-1f.6 Intel Corporation Ethernet Connection (7) I219-LM
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-18 13:13 ` Dag B
@ 2024-04-18 22:54 ` Dag B
2024-04-19 15:19 ` Ilpo Järvinen
1 sibling, 0 replies; 10+ messages in thread
From: Dag B @ 2024-04-18 22:54 UTC (permalink / raw)
To: Christian König, Bjorn Helgaas; +Cc: linux-pci
On 18.04.2024 15:13, Dag B wrote:
>
> On 18.04.2024 14:24, Christian König wrote:
>> Am 18.04.24 um 12:42 schrieb Dag B:
>>>
>>> [SNIP]
>>>>
>>>>>>
>>>>>> Is there a good ELI13 resource explaining how resizable BAR works
>>>>>> in Linux?
>>>>>>
>>>>>> My current kernel command-line contains: pci=assign-busses,realloc
>>>>
>>>> That's a really really bad idea. The "assign-busses" flag was
>>>> introduced to get 20year old laptops to see their cardbus PCI devices.
>>>
>>> I threw a lot of mud at the wall to see what stuck. Removing it now
>>> did not make a big difference.
>>>
>>> Removing realloc prevents the second TB3 GPU from being initialized,
>>> so keeping that for now.
>>
>> That's really interesting. Why does it fail without that?
>>
>> It basically means that your BIOS is somehow broken and only the
>> Linux PCI subsystem is able to assign resources correctly.
>>
>> Please provide the output of "sudo lspci -v" and "sudo lspci -tv" as
>> file attachment (*not* inline in a mail!).
>
>
> In case I have expressed myself awkwardly, the realloc=off case
> appears to make the device driver have issues with the second GPU.
>
>
> I have attached both outputs, for realloc=off.
>
> Not knowing what is considered acceptable message sizes on this m/l, I
> uploaded the same for realloc=on, as well as output from dmesg for
> both cases to:
>
> https://github.com/dagbdagb/p53
>
> If the m/l has mechanisms to archive attachments and replace them with
> links, I'll redo the exercise in a follow-up email. I understand the
> value of having the 'context' of the discussion readily available in
> one place.
>
>
> Dag B
I now have one GPU enabled with the full-fat BAR. The other has issues
assigning address space for the BARs with this config, and cannot be
initialized.
pci=realloc=on,hpiosize=64K,hpmemsize=64M,hpmmioprefsize=64G,pcie_scan_all,hpbussize=0x33
..results in:
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB
1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
Still mostly throwing mud at the wall, but the hp* options do appear to
make a difference. Would love to understand these options better.,
Dag B
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-18 13:13 ` Dag B
2024-04-18 22:54 ` Dag B
@ 2024-04-19 15:19 ` Ilpo Järvinen
2024-04-19 15:31 ` Christian König
1 sibling, 1 reply; 10+ messages in thread
From: Ilpo Järvinen @ 2024-04-19 15:19 UTC (permalink / raw)
To: Dag B; +Cc: Christian König, Bjorn Helgaas, linux-pci
[-- Attachment #1: Type: text/plain, Size: 2160 bytes --]
On Thu, 18 Apr 2024, Dag B wrote:
> On 18.04.2024 14:24, Christian König wrote:
> > Am 18.04.24 um 12:42 schrieb Dag B:
> > >
> > > [SNIP]
> > > >
> > > > > >
> > > > > > Is there a good ELI13 resource explaining how resizable BAR works in
> > > > > > Linux?
> > > > > >
> > > > > > My current kernel command-line contains: pci=assign-busses,realloc
> > > >
> > > > That's a really really bad idea. The "assign-busses" flag was introduced
> > > > to get 20year old laptops to see their cardbus PCI devices.
> > >
> > > I threw a lot of mud at the wall to see what stuck. Removing it now did
> > > not make a big difference.
> > >
> > > Removing realloc prevents the second TB3 GPU from being initialized, so
> > > keeping that for now.
> >
> > That's really interesting. Why does it fail without that?
> >
> > It basically means that your BIOS is somehow broken and only the Linux PCI
> > subsystem is able to assign resources correctly.
> >
> > Please provide the output of "sudo lspci -v" and "sudo lspci -tv" as file
> > attachment (*not* inline in a mail!).
>
>
> In case I have expressed myself awkwardly, the realloc=off case appears to
> make the device driver have issues with the second GPU.
>
>
> I have attached both outputs, for realloc=off.
>
> Not knowing what is considered acceptable message sizes on this m/l, I
> uploaded the same for realloc=on, as well as output from dmesg for both cases
> to:
>
> https://github.com/dagbdagb/p53
>
> If the m/l has mechanisms to archive attachments and replace them with links,
> I'll redo the exercise in a follow-up email. I understand the value of having
> the 'context' of the discussion readily available in one place.
The mem BAR & bridge window configuration is identical between
realloc=on/off.
The error seems to relate to io BAR:
[ 2.782439] nvidia 0000:09:00.0: BAR 5 [io 0x0000-0x007f]: not claimed; can't enable device
[ 2.783139] NVRM: pci_enable_device failed, aborting
With realloc=on, the entire IO window is disabled for the bridges and for
some reason nvidia doesn't abort in that case.
--
i.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-19 15:19 ` Ilpo Järvinen
@ 2024-04-19 15:31 ` Christian König
2024-04-27 20:42 ` Dag B
0 siblings, 1 reply; 10+ messages in thread
From: Christian König @ 2024-04-19 15:31 UTC (permalink / raw)
To: Ilpo Järvinen, Dag B; +Cc: Bjorn Helgaas, linux-pci
Am 19.04.24 um 17:19 schrieb Ilpo Järvinen:
> On Thu, 18 Apr 2024, Dag B wrote:
>> On 18.04.2024 14:24, Christian König wrote:
>>> Am 18.04.24 um 12:42 schrieb Dag B:
>>>> [SNIP]
>>>>>>> Is there a good ELI13 resource explaining how resizable BAR works in
>>>>>>> Linux?
>>>>>>>
>>>>>>> My current kernel command-line contains: pci=assign-busses,realloc
>>>>> That's a really really bad idea. The "assign-busses" flag was introduced
>>>>> to get 20year old laptops to see their cardbus PCI devices.
>>>> I threw a lot of mud at the wall to see what stuck. Removing it now did
>>>> not make a big difference.
>>>>
>>>> Removing realloc prevents the second TB3 GPU from being initialized, so
>>>> keeping that for now.
>>> That's really interesting. Why does it fail without that?
>>>
>>> It basically means that your BIOS is somehow broken and only the Linux PCI
>>> subsystem is able to assign resources correctly.
>>>
>>> Please provide the output of "sudo lspci -v" and "sudo lspci -tv" as file
>>> attachment (*not* inline in a mail!).
>>
>> In case I have expressed myself awkwardly, the realloc=off case appears to
>> make the device driver have issues with the second GPU.
>>
>>
>> I have attached both outputs, for realloc=off.
>>
>> Not knowing what is considered acceptable message sizes on this m/l, I
>> uploaded the same for realloc=on, as well as output from dmesg for both cases
>> to:
>>
>> https://github.com/dagbdagb/p53
>>
>> If the m/l has mechanisms to archive attachments and replace them with links,
>> I'll redo the exercise in a follow-up email. I understand the value of having
>> the 'context' of the discussion readily available in one place.
> The mem BAR & bridge window configuration is identical between
> realloc=on/off.
>
> The error seems to relate to io BAR:
>
> [ 2.782439] nvidia 0000:09:00.0: BAR 5 [io 0x0000-0x007f]: not claimed; can't enable device
> [ 2.783139] NVRM: pci_enable_device failed, aborting
>
> With realloc=on, the entire IO window is disabled for the bridges and for
> some reason nvidia doesn't abort in that case.
That actually makes a lot of sense.
At least on AMD hardware the IO window is only used for VGA emulation
and I strongly suspect it's the same on the NVIDIA GPUs.
So what basically happens is that the BIOS for some reason enables the
IO range on both GPUs while when Linux makes the re-alloc it disables
the ranges. Most likely because the Linux PCI code knows that they
should only be used if this device is the primary VGA device used during
boot.
Now when pci_enable_device() is called the function checks if all
enabled BARs actually have resources and without realloc=on the I/O BAR
has nothing allocated and the function fails. While with realloc=on the
BAR is disabled.
Well, what a mess. @Dag I would just strongly suggest to see if you can
update the BIOS. What happens here is clearly incorrect.
Regarding the resizing as far as I can see the BIOS allocates only a
single 1GiB window to the upstream bridge, that is most likely way to
small for anything than the default 256MiB BAR.
Maybe try to force assign more address space to this bridge. IIRC one of
the kernel parameters could be used for that, but of hand I don't
remember the syntax.
Regards,
Christian.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: PCIE BAR resizing blocked by another BAR on same device?
2024-04-19 15:31 ` Christian König
@ 2024-04-27 20:42 ` Dag B
0 siblings, 0 replies; 10+ messages in thread
From: Dag B @ 2024-04-27 20:42 UTC (permalink / raw)
To: Christian König, Ilpo Järvinen; +Cc: Bjorn Helgaas, linux-pci
I have partial success! And bits and pieces of an explanation for the
lack of complete success.
I have a short question or two towards the end, for which the rest here
is just setup.
On 19.04.2024 17:31, Christian König wrote:
> Am 19.04.24 um 17:19 schrieb Ilpo Järvinen:
>> On Thu, 18 Apr 2024, Dag B wrote:
>>> On 18.04.2024 14:24, Christian König wrote:
>>>> Am 18.04.24 um 12:42 schrieb Dag B:
>>>>> [SNIP]
>>>> Please provide the output of "sudo lspci -v" and "sudo lspci -tv"
>>>> as file
>>>> attachment (*not* inline in a mail!).
>>>
>>>
>>> https://github.com/dagbdagb/p53
>>>
>>> If the m/l has mechanisms to archive attachments and replace them
>>> with links,
>>> I'll redo the exercise in a follow-up email. I understand the value
>>> of having
>>> the 'context' of the discussion readily available in one place.
>> The mem BAR & bridge window configuration is identical between
>> realloc=on/off.
>>
>> The error seems to relate to io BAR:
>>
>> [ 2.782439] nvidia 0000:09:00.0: BAR 5 [io 0x0000-0x007f]: not
>> claimed; can't enable device
>> [ 2.783139] NVRM: pci_enable_device failed, aborting
>>
>> With realloc=on, the entire IO window is disabled for the bridges and
>> for
>> some reason nvidia doesn't abort in that case.
>
> That actually makes a lot of sense.
>
> At least on AMD hardware the IO window is only used for VGA emulation
> and I strongly suspect it's the same on the NVIDIA GPUs.
>
> So what basically happens is that the BIOS for some reason enables the
> IO range on both GPUs while when Linux makes the re-alloc it disables
> the ranges. Most likely because the Linux PCI code knows that they
> should only be used if this device is the primary VGA device used
> during boot.
>
> Now when pci_enable_device() is called the function checks if all
> enabled BARs actually have resources and without realloc=on the I/O
> BAR has nothing allocated and the function fails. While with
> realloc=on the BAR is disabled.
>
Thank you for the explanation.
> Well, what a mess. @Dag I would just strongly suggest to see if you
> can update the BIOS. What happens here is clearly incorrect.
Unlikely to be an option, I am afraid. Hardware too old. :-/
> Regarding the resizing as far as I can see the BIOS allocates only a
> single 1GiB window to the upstream bridge, that is most likely way to
> small for anything than the default 256MiB BAR.
>
> Maybe try to force assign more address space to this bridge. IIRC one
> of the kernel parameters could be used for that, but of hand I don't
> remember the syntax.
Found and enabled said option. It works :-) To reiterate:
- laptop with two TB3 ports.
- to each port, a TB3 eGPU dock with an Nvidia GPU. Currently, I get:
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 256MB, supported: 64MB 128MB 256MB 512MB
1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
Capabilities: [bb0 v1] Physical Resizable BAR
BAR 0: current size: 16MB, supported: 16MB
BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB
1GB 2GB 4GB 8GB 16GB 32GB
BAR 3: current size: 32MB, supported: 32MB
- Kernel command-line currently contains:
pci=realloc=on,hpiosize=1024,hpmmioprefsize=64G (more than 64G does
not work)
In short, both eGPUs work. But only one has a big BAR. Seeing how the
GPUs are hanging off TB3, system bandwidth to these GPUs will be limited
in any case. A big BAR is not going to change that at all. But at this
point I'd like to understand what linux can and cannot do w.r.t.
resource allocations.
TB3 ports in lspci:
04:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge
[Titan Ridge 4C 2018] (rev 06)
05:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge
[Titan Ridge 4C 2018] (rev 06)
05:01.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge
[Titan Ridge 4C 2018] (rev 06)
05:02.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge
[Titan Ridge 4C 2018] (rev 06)
05:04.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge
[Titan Ridge 4C 2018] (rev 06)
TB3 eGPU docks, as seen from boltctl:
● Razer Core X Chroma
├─ type: peripheral
├─ name: Core X Chroma
● Razer Core X Chroma #2
├─ type: peripheral
├─ name: Core X Chroma
● Razer Core X
├─ type: peripheral
├─ name: Core X
Yes, the first dock appears as two devices. It is part of the problem,
it appears. lspci illustrates this as follows:
06:00.0 System peripheral: Intel Corporation JHL7540 Thunderbolt 3 NHI
[Titan Ridge 4C 2018] (rev 06)
07:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 4C 2016] (rev 02)
08:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 4C 2016] (rev 02)
08:04.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 4C 2016] (rev 02)
09:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX
3090] (rev a1)
0a:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 4C 2016] (rev 02)
0b:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 4C 2016] (rev 02)
0b:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 4C 2016] (rev 02)
0b:02.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 4C 2016] (rev 02)
0c:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host
Controller
0d:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host
Controller
0e:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host
Controller
2c:00.0 USB controller: Intel Corporation JHL7540 Thunderbolt 3 USB
Controller [Titan Ridge 4C 2018] (rev 06)
2d:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 2C 2016] (rev 02)
2e:01.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C
step) [Alpine Ridge 2C 2016] (rev 02)
2f:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX
3090] (rev a1)
or as a tree:
00:1c.0 root_port, slot 0, device present, speed 8GT/s, width x4
└─04:00.0 upstream_port, Intel Corporation (8086) JHL7540 Thunderbolt
3 Bridge [Titan Ridge 4C 2018] (15ea)
├─05:00.0 downstream_port, slot 0, device present, speed 2.5GT/s,
width x4
│ └─06:00.0 endpoint, Intel Corporation (8086) JHL7540 Thunderbolt
3 NHI [Titan Ridge 4C 2018] (15eb)
├─05:01.0 downstream_port, slot 1, device present, speed 2.5GT/s,
width x4
│ └─07:00.0 upstream_port, Intel Corporation (8086) JHL6540
Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (15d3)
│ ├─08:01.0 downstream_port, slot 0, device present, speed
2.5GT/s, width x4
│ │ └─09:00.0 legacy_endpoint
│ └─08:04.0 downstream_port, slot 4, device present, speed
2.5GT/s, width x4
│ └─0a:00.0 upstream_port, Intel Corporation (8086) JHL6540
Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (15d3)
│ ├─0b:00.0 downstream_port, slot 0, device present,
speed 8GT/s, width x1
│ │ └─0c:00.0 endpoint, ASMedia Technology Inc. (1b21)
ASM1142 USB 3.1 Host Controller (1242)
│ ├─0b:01.0 downstream_port, slot 0, device present,
speed 8GT/s, width x1
│ │ └─0d:00.0 endpoint, ASMedia Technology Inc. (1b21)
ASM1142 USB 3.1 Host Controller (1242)
│ └─0b:02.0 downstream_port, slot 0, device present,
speed 8GT/s, width x1
│ └─0e:00.0 endpoint, ASMedia Technology Inc. (1b21)
ASM1142 USB 3.1 Host Controller (1242)
├─05:02.0 downstream_port, slot 0, device present, speed 2.5GT/s,
width x4
│ └─2c:00.0 endpoint, Intel Corporation (8086) JHL7540 Thunderbolt
3 USB Controller [Titan Ridge 4C 2018] (15ec)
└─05:04.0 downstream_port, slot 4, device present, speed 2.5GT/s,
width x4
└─2d:00.0 upstream_port, Intel Corporation (8086) JHL6340
Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (15da)
└─2e:01.0 downstream_port, slot 0, device present, speed
2.5GT/s, width x4
└─2f:00.0 legacy_endpoint
/proc/iomem looks like this:
4000000000-7fffffffff : PCI Bus 0000:00
4000000000-400fffffff : 0000:00:02.0
4010000000-6027ffffff : PCI Bus 0000:04
4010000000-6027ffffff : PCI Bus 0000:05
4010000000-5027ffffff : PCI Bus 0000:07
4010000000-5027ffffff : PCI Bus 0000:08
4010000000-4027ffffff : PCI Bus 0000:09
4010000000-401fffffff : 0000:09:00.0
4020000000-4021ffffff : 0000:09:00.0
4028000000-5027ffffff : PCI Bus 0000:0a
4028000000-5027ffffff : PCI Bus 0000:0b
4028000000-457d4fffff : PCI Bus 0000:0c
457d500000-4ad29fffff : PCI Bus 0000:0d
4ad2a00000-5027efffff : PCI Bus 0000:0e
5400000000-5fffffffff : PCI Bus 0000:2d
5400000000-5fffffffff : PCI Bus 0000:2e
5400000000-5fffffffff : PCI Bus 0000:2f
5400000000-5401ffffff : 0000:2f:00.0
5800000000-5fffffffff : 0000:2f:00.0
I read this as the stuff downstream of the laptop TB3 controller has the
range 4010000000-6027ffffff.
The GPU assigned bus id 2f gets all of 5400000000-5fffffffff
The GPU assigned bus id 09 (hanging off 08.01) has to share
4010000000-5027ffffff with stuff hanging off 08:04 (0a-0e).
0c, 0d and 0e are some USB ports and a (USB-attached) ethernet
interface, on a separate card in (only) one of the TB3 docks. I can
physically pull the card with the ports from the dock, but /proc/iomem
will list the address allocation for 0c-0e *even if they now are missing
from the lspci output*.
Unbinding 08.4 (and everything below it) and rebinding it to the
pci-stub driver does not change how /proc/iomem looks either. Nor with a
pci rescan.
So, the question(s):
Does linux, in this case, have a mechanism for shutting down 08:04 (and
everything downstream) *and* freeing the reserved resources, such that
09:00 gets to pick whatever it wants from 4010000000-5027ffffff ?
Or to pre-allocate resources for a particular pci id (08:01), such that
08:04 simply fails? Which does not require writing actual C-code?
Getting a 'simpler' dock may be the easier way out. I may have crossed
the line for simple solutions some time ago. :-)
Thanks,
Dag B
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-04-27 21:18 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-17 13:16 PCIE BAR resizing blocked by another BAR on same device? Dag B
2024-04-17 15:13 ` Bjorn Helgaas
2024-04-18 7:51 ` Christian König
2024-04-18 10:42 ` Dag B
2024-04-18 12:24 ` Christian König
2024-04-18 13:13 ` Dag B
2024-04-18 22:54 ` Dag B
2024-04-19 15:19 ` Ilpo Järvinen
2024-04-19 15:31 ` Christian König
2024-04-27 20:42 ` Dag B
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).