PCI: bus resource allocation error

PCI: bus resource allocation error

[Yicong Yang]

Hi,

recently I met a problem with pci bus resource allocation. The allocation strategy
makes me confused and leads to a wrong allocation results.

There is a hisilicon network device with four functions under one root port. The
original bios resources allocation looks like:

7c:00.0 Root Port
     prefetchable memory behind bridge: 12000000-0x1210fffff 17M [64bit pref]
    7d:00.0
        bar0: 0x121000000-0x12100ffff 64k  [64bit pref]
        bar2: 0x120000000-0x1200fffff 1M   [64bit pref]
        bar7: 0x121010000-0x12103ffff 128K [64bit pref]
        bar9: 0x120100000-0x1203fffff 3M   [64bit pref]
    7d:00.1
        bar0: 0x121040000-0x12104ffff 64k  [64bit pref]
        bar2: 0x120400000-0x1204fffff 1M   [64bit pref]
        bar7: 0x121050000-0x12107ffff 128K [64bit pref]
        bar9: 0x120500000-0x1207fffff 3M   [64bit pref]
    7d:00.2
        bar0: 0x121080000-0x12108ffff 64k  [64bit pref]
        bar2: 0x120800000-0x1208fffff 1M   [64bit pref]
        bar7: 0x121090000-0x1210bffff 128K [64bit pref]
        bar9: 0x120900000-0x120bfffff 3M   [64bit pref]
    7d:00.3
        bar0: 0x1210c0000-0x1210cffff 64k  [64bit pref]
        bar2: 0x120c00000-0x120cfffff 1M   [64bit pref]
        bar7: 0x121010000-0x12103ffff 128K [64bit pref]
        bar9: 0x120d00000-0x120ffffff 3M   [64bit pref]

When I remove function 7d:00.3 and try to rescan the bus[7c], kernel prints the
error information.
[  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
[  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
[  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
[  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
[  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
[  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
[  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
[  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
[  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
                                                            ^^^^^^^^^^^^^^^^^^^^^^^   
[  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
[  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
                                                            ^^^^^^^^^^^^^^^^^^^^^^^
[  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
[  391.864261] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
[  391.872148] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
[  391.880035] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
[  391.887920] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]

When looking into the code, the functions called like:
    pci_rescan_bus()
        pci_assign_unassigned_bus_resources()
            __pci_bus_size_bridges()
                pbus_size_mem()

The function 7d:00.3 is added and enabled well as the required resources are satisfied.
As it request 64bit prefetchable resources, there is no reason to open bar14 for it.

When a new function is added, the framework trys to size the bridge memory
window for it. In __pci_bus_size_bridges(), firstly the framework trys to size bar15 for the
new added 5M resources as we require 64bit pref mem. But bar15 has *parent*
so pbus_size_mem() return failure with bar15 unchanged. Then the framework try to put
resources in bar14, 32bit mem window, and the bar14 is unused so it is sized to 5M and
pbus_size_mem() return success.
After bridge size settles down, the framework assign resources for each bar. *As the bios
doesn't reserve a 32bit mem window for the bridge*, bar14 assignment is failed and print
the error assigen information. When assigning 7d:00.3, the framework try to find a space
in bar15 firstly and succeed. Then the flow is terminated. The bar14 is even not touched.

Here comes the question:
    Why should we resize the bridge memory window when only one function is removed and
rescanned later? The bridge memory window should remain unchanged in such a situation.
    Is there a *certain condition* which requirs such operation?
    If all the functions are removed, we should use pci_rescan_bus_bridge_resize() to rescan and add
devices as the required memory size maybe changed.  Otherwise, i think we should try to assign
resources directly without sizing the bridge resource.
   
Thanks,
Yang

Re: PCI: bus resource allocation error

[Bjorn Helgaas]

[+cc Nicholas, who is working in this area]

On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
> Hi,
> 
> recently I met a problem with pci bus resource allocation. The allocation strategy
> makes me confused and leads to a wrong allocation results.
> 
> There is a hisilicon network device with four functions under one root port. The
> original bios resources allocation looks like:

What kernel is this?  Can you collect the complete dmesg log?

> 7c:00.0 Root Port
>      prefetchable memory behind bridge: 12000000-0x1210fffff 17M [64bit pref]
>     7d:00.0
>         bar0: 0x121000000-0x12100ffff 64k  [64bit pref]
>         bar2: 0x120000000-0x1200fffff 1M   [64bit pref]
>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>         bar9: 0x120100000-0x1203fffff 3M   [64bit pref]
>     7d:00.1
>         bar0: 0x121040000-0x12104ffff 64k  [64bit pref]
>         bar2: 0x120400000-0x1204fffff 1M   [64bit pref]
>         bar7: 0x121050000-0x12107ffff 128K [64bit pref]
>         bar9: 0x120500000-0x1207fffff 3M   [64bit pref]
>     7d:00.2
>         bar0: 0x121080000-0x12108ffff 64k  [64bit pref]
>         bar2: 0x120800000-0x1208fffff 1M   [64bit pref]
>         bar7: 0x121090000-0x1210bffff 128K [64bit pref]
>         bar9: 0x120900000-0x120bfffff 3M   [64bit pref]
>     7d:00.3
>         bar0: 0x1210c0000-0x1210cffff 64k  [64bit pref]
>         bar2: 0x120c00000-0x120cfffff 1M   [64bit pref]
>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>         bar9: 0x120d00000-0x120ffffff 3M   [64bit pref]
> 
> When I remove function 7d:00.3 and try to rescan the bus[7c], kernel prints the
> error information.
> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
> [  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
> [  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
> [  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
> [  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
> [  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^   
> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^
> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.864261] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
> [  391.872148] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
> [  391.880035] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
> [  391.887920] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
> 
> When looking into the code, the functions called like:
>     pci_rescan_bus()
>         pci_assign_unassigned_bus_resources()
>             __pci_bus_size_bridges()
>                 pbus_size_mem()
> 
> The function 7d:00.3 is added and enabled well as the required resources are satisfied.
> As it request 64bit prefetchable resources, there is no reason to open bar14 for it.
> 
> When a new function is added, the framework trys to size the bridge memory
> window for it. In __pci_bus_size_bridges(), firstly the framework trys to size bar15 for the
> new added 5M resources as we require 64bit pref mem. But bar15 has *parent*
> so pbus_size_mem() return failure with bar15 unchanged. Then the framework try to put
> resources in bar14, 32bit mem window, and the bar14 is unused so it is sized to 5M and
> pbus_size_mem() return success.
> After bridge size settles down, the framework assign resources for each bar. *As the bios
> doesn't reserve a 32bit mem window for the bridge*, bar14 assignment is failed and print
> the error assigen information. When assigning 7d:00.3, the framework try to find a space
> in bar15 firstly and succeed. Then the flow is terminated. The bar14 is even not touched.
> 
> Here comes the question:
>     Why should we resize the bridge memory window when only one function is removed and
> rescanned later? The bridge memory window should remain unchanged in such a situation.
>     Is there a *certain condition* which requirs such operation?
>     If all the functions are removed, we should use pci_rescan_bus_bridge_resize() to rescan and add
> devices as the required memory size maybe changed.  Otherwise, i think we should try to assign
> resources directly without sizing the bridge resource.
>    
> Thanks,
> Yang
> 

Re: PCI: bus resource allocation error

[Yicong Yang]

On 2020/1/9 12:27, Bjorn Helgaas wrote:
> [+cc Nicholas, who is working in this area]
>
> On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
>> Hi,
>>
>> recently I met a problem with pci bus resource allocation. The allocation strategy
>> makes me confused and leads to a wrong allocation results.
>>
>> There is a hisilicon network device with four functions under one root port. The
>> original bios resources allocation looks like:
> What kernel is this?  Can you collect the complete dmesg log?

The kernel version is 5.4.0.  the dmesg log is like:

[  496.598130] hns3 0000:7d:00.3 eth11: net stop
[  496.602702] hns3 0000:7d:00.3 eth11: link down
[  496.655963] pci 0000:7d:00.3: Removing from iommu group 49
[  508.453948] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
[  508.459943] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
[  508.467317] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
[  508.474720] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
[  508.482175] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
[  508.492485] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
[  508.499939] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
[  508.510351] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
[  508.520836] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
[  508.527513] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
[  508.534538] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
[  508.541216] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
[  508.548246] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
[  508.556132] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
[  508.564022] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
[  508.571909] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
[  508.579903] hns3 0000:7d:00.3: Adding to iommu group 49
[  508.585782] hns3 0000:7d:00.3: The firmware version is 1.9.23.6
[  508.594571] libphy: hisilicon MII bus: probed
[  508.694463] hns3 0000:7d:00.3: hclge driver initialization finished.
[  508.701446] RTL8211F Gigabit Ethernet mii-0000:7d:00.3:07: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=mii-0000:7d:00.3:07, irq=POLL)

>
>> 7c:00.0 Root Port
>>      prefetchable memory behind bridge: 12000000-0x1210fffff 17M [64bit pref]
>>     7d:00.0
>>         bar0: 0x121000000-0x12100ffff 64k  [64bit pref]
>>         bar2: 0x120000000-0x1200fffff 1M   [64bit pref]
>>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>>         bar9: 0x120100000-0x1203fffff 3M   [64bit pref]
>>     7d:00.1
>>         bar0: 0x121040000-0x12104ffff 64k  [64bit pref]
>>         bar2: 0x120400000-0x1204fffff 1M   [64bit pref]
>>         bar7: 0x121050000-0x12107ffff 128K [64bit pref]
>>         bar9: 0x120500000-0x1207fffff 3M   [64bit pref]
>>     7d:00.2
>>         bar0: 0x121080000-0x12108ffff 64k  [64bit pref]
>>         bar2: 0x120800000-0x1208fffff 1M   [64bit pref]
>>         bar7: 0x121090000-0x1210bffff 128K [64bit pref]
>>         bar9: 0x120900000-0x120bfffff 3M   [64bit pref]
>>     7d:00.3
>>         bar0: 0x1210c0000-0x1210cffff 64k  [64bit pref]
>>         bar2: 0x120c00000-0x120cfffff 1M   [64bit pref]
>>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>>         bar9: 0x120d00000-0x120ffffff 3M   [64bit pref]
>>
>> When I remove function 7d:00.3 and try to rescan the bus[7c], kernel prints the
>> error information.
>> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
>> [  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
>> [  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
>> [  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
>> [  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
>> [  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
>> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
>> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
>> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^   
>> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.864261] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
>> [  391.872148] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
>> [  391.880035] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
>> [  391.887920] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
>>
>> When looking into the code, the functions called like:
>>     pci_rescan_bus()
>>         pci_assign_unassigned_bus_resources()
>>             __pci_bus_size_bridges()
>>                 pbus_size_mem()
>>
>> The function 7d:00.3 is added and enabled well as the required resources are satisfied.
>> As it request 64bit prefetchable resources, there is no reason to open bar14 for it.
>>
>> When a new function is added, the framework trys to size the bridge memory
>> window for it. In __pci_bus_size_bridges(), firstly the framework trys to size bar15 for the
>> new added 5M resources as we require 64bit pref mem. But bar15 has *parent*
>> so pbus_size_mem() return failure with bar15 unchanged. Then the framework try to put
>> resources in bar14, 32bit mem window, and the bar14 is unused so it is sized to 5M and
>> pbus_size_mem() return success.
>> After bridge size settles down, the framework assign resources for each bar. *As the bios
>> doesn't reserve a 32bit mem window for the bridge*, bar14 assignment is failed and print
>> the error assigen information. When assigning 7d:00.3, the framework try to find a space
>> in bar15 firstly and succeed. Then the flow is terminated. The bar14 is even not touched.
>>
>> Here comes the question:
>>     Why should we resize the bridge memory window when only one function is removed and
>> rescanned later? The bridge memory window should remain unchanged in such a situation.
>>     Is there a *certain condition* which requirs such operation?
>>     If all the functions are removed, we should use pci_rescan_bus_bridge_resize() to rescan and add
>> devices as the required memory size maybe changed.  Otherwise, i think we should try to assign
>> resources directly without sizing the bridge resource.
>>    
>> Thanks,
>> Yang
>>
> .
>

Re: PCI: bus resource allocation error

[Bjorn Helgaas]

On Thu, Jan 09, 2020 at 06:31:57PM +0800, Yicong Yang wrote:
> On 2020/1/9 12:27, Bjorn Helgaas wrote:
> > [+cc Nicholas, who is working in this area]
> >
> > On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
> >> Hi,
> >>
> >> recently I met a problem with pci bus resource allocation. The allocation strategy
> >> makes me confused and leads to a wrong allocation results.
> >>
> >> There is a hisilicon network device with four functions under one root port. The
> >> original bios resources allocation looks like:
> > What kernel is this?  Can you collect the complete dmesg log?
> 
> The kernel version is 5.4.0.  

Good; at least we know this isn't related to Nicholas' new resource
code that's in -next right now.

> the dmesg log is like:

The below is not the complete dmesg log.  I don't know what your
system is, but the complete log might be in /var/log/dmesg, or maybe
you could capture it with the "ignore_loglevel" kernel parameter and a
serial console?

> [  496.598130] hns3 0000:7d:00.3 eth11: net stop
> ...

Re: PCI: bus resource allocation error

[Bjorn Helgaas]

On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
> Hi,
> 
> recently I met a problem with pci bus resource allocation. The allocation strategy
> makes me confused and leads to a wrong allocation results.
> 
> There is a hisilicon network device with four functions under one root port. The
> original bios resources allocation looks like:
> 
> 7c:00.0 Root Port
>      prefetchable memory behind bridge: 12000000-0x1210fffff 17M [64bit pref]
>     7d:00.0
>         bar0: 0x121000000-0x12100ffff 64k  [64bit pref]
>         bar2: 0x120000000-0x1200fffff 1M   [64bit pref]
>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>         bar9: 0x120100000-0x1203fffff 3M   [64bit pref]
>     7d:00.1
>         bar0: 0x121040000-0x12104ffff 64k  [64bit pref]
>         bar2: 0x120400000-0x1204fffff 1M   [64bit pref]
>         bar7: 0x121050000-0x12107ffff 128K [64bit pref]
>         bar9: 0x120500000-0x1207fffff 3M   [64bit pref]
>     7d:00.2
>         bar0: 0x121080000-0x12108ffff 64k  [64bit pref]
>         bar2: 0x120800000-0x1208fffff 1M   [64bit pref]
>         bar7: 0x121090000-0x1210bffff 128K [64bit pref]
>         bar9: 0x120900000-0x120bfffff 3M   [64bit pref]
>     7d:00.3
>         bar0: 0x1210c0000-0x1210cffff 64k  [64bit pref]
>         bar2: 0x120c00000-0x120cfffff 1M   [64bit pref]
>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>         bar9: 0x120d00000-0x120ffffff 3M   [64bit pref]

This looks like an incorrect assignment, i.e., possibly a BIOS defect:
7d:00.0 and 7d:00.3 are assigned the same space for bar7:

  7d:00.0 bar7: 0x121010000-0x12103ffff 128K [64bit pref]
  7d:00.3 bar7: 0x121010000-0x12103ffff 128K [64bit pref]

> When I remove function 7d:00.3 and try to rescan the bus[7c], kernel prints the
> error information.
> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
> [  391.776024] pci 0000:7d:00.3: bar0 reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
> [  391.783394] pci 0000:7d:00.3: bar2 reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
> [  391.790786] pci 0000:7d:00.3: bar7 reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
> [  391.798238] pci 0000:7d:00.3: bar7 VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
> [  391.808543] pci 0000:7d:00.3: bar9 reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^   
> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^
> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.864261] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
> [  391.872148] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
> [  391.880035] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
> [  391.887920] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]

What is the incorrect allocation here?  This looks the same as the
original assignment from BIOS, except that BAR 7 (the VF BAR 2 space)
no longer overlaps BAR 7 of 7d:00.0.

> When looking into the code, the functions called like:
>     pci_rescan_bus()
>         pci_assign_unassigned_bus_resources()
>             __pci_bus_size_bridges()
>                 pbus_size_mem()
> 
> The function 7d:00.3 is added and enabled well as the required resources are satisfied.
> As it request 64bit prefetchable resources, there is no reason to open bar14 for it.
> 
> When a new function is added, the framework trys to size the bridge memory
> window for it. In __pci_bus_size_bridges(), firstly the framework trys to size bar15 for the
> new added 5M resources as we require 64bit pref mem. But bar15 has *parent*
> so pbus_size_mem() return failure with bar15 unchanged. Then the framework try to put
> resources in bar14, 32bit mem window, and the bar14 is unused so it is sized to 5M and
> pbus_size_mem() return success.
> After bridge size settles down, the framework assign resources for each bar. *As the bios
> doesn't reserve a 32bit mem window for the bridge*, bar14 assignment is failed and print
> the error assigen information. When assigning 7d:00.3, the framework try to find a space
> in bar15 firstly and succeed. Then the flow is terminated. The bar14 is even not touched.
> 
> Here comes the question:
>     Why should we resize the bridge memory window when only one function is removed and
> rescanned later? The bridge memory window should remain unchanged in such a situation.

In this case you removed a function and re-added the same function
later, so it needs the same amount of resources.  In that case, I
agree, we probably shouldn't change the bridge window.  But I don't
think we *did* change the bridge window here.  Did I miss something?

I agree the messages about BAR 14 (the non-prefetchable window) are
confusing and we probably shouldn't have even tried to assign space
for it.

I guess I'm missing something, because other than the annoying BAR 14
messages, I don't see the actual problem here.

Bjorn

Re: PCI: bus resource allocation error

[Nicholas Johnson]

On Thu, Jan 09, 2020 at 03:55:17PM -0600, Bjorn Helgaas wrote:
> On Thu, Jan 09, 2020 at 06:31:57PM +0800, Yicong Yang wrote:
> > On 2020/1/9 12:27, Bjorn Helgaas wrote:
> > > [+cc Nicholas, who is working in this area]
> > >
> > > On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
> > >> Hi,
> > >>
> > >> recently I met a problem with pci bus resource allocation. The allocation strategy
> > >> makes me confused and leads to a wrong allocation results.
> > >>
> > >> There is a hisilicon network device with four functions under one root port. The
> > >> original bios resources allocation looks like:
> > > What kernel is this?  Can you collect the complete dmesg log?
> > 
> > The kernel version is 5.4.0.  
> 
> Good; at least we know this isn't related to Nicholas' new resource
> code that's in -next right now.

It is not in next - it is in the release candidates, right?
> 
> > the dmesg log is like:
> 
> The below is not the complete dmesg log.  I don't know what your
> system is, but the complete log might be in /var/log/dmesg, or maybe
> you could capture it with the "ignore_loglevel" kernel parameter and a
> serial console?
> 
> > [  496.598130] hns3 0000:7d:00.3 eth11: net stop
> > ...
Yicong, please provide the full outputs of: "dmesg" and "sudo lspci 
-xxxx" and "sudo cat /proc/iomem" in a location that is publicly 
accessible so that anybody reading this email can find it.

Kind regards,
Nicholas

Re: PCI: bus resource allocation error

[Bjorn Helgaas]

On Thu, Jan 09, 2020 at 11:55:18PM +0000, Nicholas Johnson wrote:
> On Thu, Jan 09, 2020 at 03:55:17PM -0600, Bjorn Helgaas wrote:
> > On Thu, Jan 09, 2020 at 06:31:57PM +0800, Yicong Yang wrote:
> > > On 2020/1/9 12:27, Bjorn Helgaas wrote:
> > > > [+cc Nicholas, who is working in this area]
> > > >
> > > > On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
> > > >> Hi,
> > > >>
> > > >> recently I met a problem with pci bus resource allocation. The allocation strategy
> > > >> makes me confused and leads to a wrong allocation results.
> > > >>
> > > >> There is a hisilicon network device with four functions under one root port. The
> > > >> original bios resources allocation looks like:
> > > > What kernel is this?  Can you collect the complete dmesg log?
> > > 
> > > The kernel version is 5.4.0.  
> > 
> > Good; at least we know this isn't related to Nicholas' new resource
> > code that's in -next right now.
> 
> It is not in next - it is in the release candidates, right?

There are a few things already in v5.5-rc5:

  c13704f5685d ("PCI: Avoid double hpmemsize MMIO window assignment")
  d7b8a217521c ("PCI: Add "pci=hpmmiosize" and "pci=hpmmioprefsize" parameters")

The following are currently in -next and should appear in v5.6-rc1:

  ddbbbb6cb825 ("PCI: Allow extend_bridge_window() to shrink resource if necessary")
  2b2108891303 ("PCI: Set resource size directly in extend_bridge_window()")
  7bd85f16152b ("PCI: Rename extend_bridge_window() parameter")
  5b55d9cf7d43 ("PCI: Consider alignment of hot-added bridges when distributing available resources")

I wanted to make sure Yicong was not testing -next.

Bjorn

Re: PCI: bus resource allocation error

[Yicong Yang]

On 2020/1/10 7:00, Bjorn Helgaas wrote:
> On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
>> Hi,
>>
>> recently I met a problem with pci bus resource allocation. The allocation strategy
>> makes me confused and leads to a wrong allocation results.
>>
>> There is a hisilicon network device with four functions under one root port. The
>> original bios resources allocation looks like:
>>
>> 7c:00.0 Root Port
>>      prefetchable memory behind bridge: 12000000-0x1210fffff 17M [64bit pref]
>>     7d:00.0
>>         bar0: 0x121000000-0x12100ffff 64k  [64bit pref]
>>         bar2: 0x120000000-0x1200fffff 1M   [64bit pref]
>>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>>         bar9: 0x120100000-0x1203fffff 3M   [64bit pref]
>>     7d:00.1
>>         bar0: 0x121040000-0x12104ffff 64k  [64bit pref]
>>         bar2: 0x120400000-0x1204fffff 1M   [64bit pref]
>>         bar7: 0x121050000-0x12107ffff 128K [64bit pref]
>>         bar9: 0x120500000-0x1207fffff 3M   [64bit pref]
>>     7d:00.2
>>         bar0: 0x121080000-0x12108ffff 64k  [64bit pref]
>>         bar2: 0x120800000-0x1208fffff 1M   [64bit pref]
>>         bar7: 0x121090000-0x1210bffff 128K [64bit pref]
>>         bar9: 0x120900000-0x120bfffff 3M   [64bit pref]
>>     7d:00.3
>>         bar0: 0x1210c0000-0x1210cffff 64k  [64bit pref]
>>         bar2: 0x120c00000-0x120cfffff 1M   [64bit pref]
>>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>>         bar9: 0x120d00000-0x120ffffff 3M   [64bit pref]
> This looks like an incorrect assignment, i.e., possibly a BIOS defect:
> 7d:00.0 and 7d:00.3 are assigned the same space for bar7:
It seems I've made a wrong type. Ignore it and see the dmesg.log/console.log.
>
>   7d:00.0 bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>   7d:00.3 bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>
>> When I remove function 7d:00.3 and try to rescan the bus[7c], kernel prints the
>> error information.
>> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
>> [  391.776024] pci 0000:7d:00.3: bar0 reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
>> [  391.783394] pci 0000:7d:00.3: bar2 reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
>> [  391.790786] pci 0000:7d:00.3: bar7 reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
>> [  391.798238] pci 0000:7d:00.3: bar7 VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
>> [  391.808543] pci 0000:7d:00.3: bar9 reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
>> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
>> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
>> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^   
>> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.864261] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
>> [  391.872148] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
>> [  391.880035] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
>> [  391.887920] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
> What is the incorrect allocation here?  This looks the same as the
> original assignment from BIOS, except that BAR 7 (the VF BAR 2 space)
> no longer overlaps BAR 7 of 7d:00.0.
The allocation of the endpoints is fine and works well. But the Bar14 of the root port is wrong open and
the assignment is failed. It shouldn't be opened as none of the subordinates use it, the bios doesn't
reserve a  32bit memory region for it as well.
>> When looking into the code, the functions called like:
>>     pci_rescan_bus()
>>         pci_assign_unassigned_bus_resources()
>>             __pci_bus_size_bridges()
>>                 pbus_size_mem()
>>
>> The function 7d:00.3 is added and enabled well as the required resources are satisfied.
>> As it request 64bit prefetchable resources, there is no reason to open bar14 for it.
>>
>> When a new function is added, the framework trys to size the bridge memory
>> window for it. In __pci_bus_size_bridges(), firstly the framework trys to size bar15 for the
>> new added 5M resources as we require 64bit pref mem. But bar15 has *parent*
>> so pbus_size_mem() return failure with bar15 unchanged. Then the framework try to put
>> resources in bar14, 32bit mem window, and the bar14 is unused so it is sized to 5M and
>> pbus_size_mem() return success.
>> After bridge size settles down, the framework assign resources for each bar. *As the bios
>> doesn't reserve a 32bit mem window for the bridge*, bar14 assignment is failed and print
>> the error assigen information. When assigning 7d:00.3, the framework try to find a space
>> in bar15 firstly and succeed. Then the flow is terminated. The bar14 is even not touched.
>>
>> Here comes the question:
>>     Why should we resize the bridge memory window when only one function is removed and
>> rescanned later? The bridge memory window should remain unchanged in such a situation.
> In this case you removed a function and re-added the same function
> later, so it needs the same amount of resources.  In that case, I
> agree, we probably shouldn't change the bridge window.  But I don't
> think we *did* change the bridge window here.  Did I miss something?
We call pbus_size_mem() to size the bridge bars in __pci_bus_size_bridges().
    __pci_bus_size_bridges()
        pbus_size_mem()    // try to size bar15 [64bit pref]
        pbus_size_mem()    // try to size bar14 [32bit]
In pbus_size_mem(), if the bar isn't spare (which means resource[bar]->parent is not NULL),
the function will return directly. Otherwise, the bar size is changed to contain the necessary
resource.
In this condition, as all of the subordinates are allocated in bar15. so the pbus_size_mem()
returned directly when sizing bar15, but change the size of bar14. which leads to the error
in later assignment of bar14. As a 32bit memory window is not reserved for this root port.

>
> I agree the messages about BAR 14 (the non-prefetchable window) are
> confusing and we probably shouldn't have even tried to assign space
> for it.
>
> I guess I'm missing something, because other than the annoying BAR 14
> messages, I don't see the actual problem here.
I would like to remove  __pci_bus_size_bridges() from pci_rescan_bus() routine, as it's no
necessary to resize the bridges window. we'll call pci_rescan_bus_bridge_resize() if all the devices
down the rp is removed and resize the bridge window if necessary.

But I don't know whether there is a condition that needs to resize the bridge window
when rescanning the bus?

> Bjorn
>
> .
>

Re: PCI: bus resource allocation error

[Yicong Yang]

Hi,

It seems the attachments are blocked by the server.
The necessary console output is below.
The kernel version is 5.4, centos release 7.6.  I didn't
change the PCI codes.

[root@localhost ~]# echo 8 > /proc/sys/kernel/printk
[root@localhost ~]# lspci -vvv -s 7c:00.0
7c:00.0 PCI bridge: Huawei Technologies Co., Ltd. HiSilicon PCI-PCI Bridge (rev 20) (prog-if 00 [Normal decode])
        Bus: primary=7c, secondary=7d, subordinate=7d, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: fff00000-000fffff
        Prefetchable memory behind bridge: 0000000120000000-00000001210fffff

[root@localhost ~]# lspci -vvv -s 7d:00.0
7d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE RDMA Network Controller (rev 21)
        Region 0: Memory at 121000000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at 120000000 (64-bit, prefetchable) [size=1M]
        Capabilities: [200 v1] Single Root I/O Virtualization (SR-IOV)
                Region 0: Memory at 0000000121010000 (64-bit, prefetchable)
                Region 2: Memory at 0000000120100000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0

[root@localhost ~]# lspci -vvv -s 7d:00.1
7d:00.1 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE Network Controller (rev 21)
        Region 0: Memory at 121040000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at 120400000 (64-bit, prefetchable) [size=1M]
        Capabilities: [200 v1] Single Root I/O Virtualization (SR-IOV)
                Region 0: Memory at 0000000121050000 (64-bit, prefetchable)
                Region 2: Memory at 0000000120500000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0

[root@localhost ~]# lspci -vvv -s 7d:00.2
7d:00.2 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE RDMA Network Controller (rev 21)
        Region 0: Memory at 121080000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at 120800000 (64-bit, prefetchable) [size=1M]
        Capabilities: [200 v1] Single Root I/O Virtualization (SR-IOV)
                Region 0: Memory at 0000000121090000 (64-bit, prefetchable)
                Region 2: Memory at 0000000120900000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0

[root@localhost ~]# lspci -vvv -s 7d:00.3
7d:00.3 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE Network Controller (rev 21)
        Region 0: Memory at 1210c0000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at 120c00000 (64-bit, prefetchable) [size=1M]
        Capabilities: [200 v1] Single Root I/O Virtualization (SR-IOV)
                Region 0: Memory at 00000001210d0000 (64-bit, prefetchable)
                Region 2: Memory at 0000000120d00000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
[root@localhost ~]# cat /proc/iomem
120000000-13fffffff : PCI Bus 0000:7c
  120000000-1210fffff : PCI Bus 0000:7d
    120000000-1200fffff : 0000:7d:00.0
      120000000-1200fffff : hclge
    120100000-1203fffff : 0000:7d:00.0
    120400000-1204fffff : 0000:7d:00.1
      120400000-1204fffff : hclge
    120500000-1207fffff : 0000:7d:00.1
    120800000-1208fffff : 0000:7d:00.2
      120800000-1208fffff : hclge
    120900000-120bfffff : 0000:7d:00.2
    120c00000-120cfffff : 0000:7d:00.3
      120c00000-120cfffff : hclge
    120d00000-120ffffff : 0000:7d:00.3
    121000000-12100ffff : 0000:7d:00.0
      121000000-12100ffff : hclge
    121010000-12103ffff : 0000:7d:00.0
    121040000-12104ffff : 0000:7d:00.1
      121040000-12104ffff : hclge
    121050000-12107ffff : 0000:7d:00.1
    121080000-12108ffff : 0000:7d:00.2
      121080000-12108ffff : hclge
    121090000-1210bffff : 0000:7d:00.2
    1210c0000-1210cffff : 0000:7d:00.3
      1210c0000-1210cffff : hclge
    1210d0000-1210fffff : 0000:7d:00.3
[root@localhost ~]# cd /sys/devices/pci0000\:7c/0000\:7c\:00.0/
[root@localhost 0000:7c:00.0]# echo 1 > 0000\:7d\:00.3/remove
[  687.008181] hns3 0000:7d:00.3 eth11: net stop
[  687.012769] hns3 0000:7d:00.3 eth11: link down
[  687.098528] pci 0000:7d:00.3: Removing from iommu group 40
[root@localhost 0000:7c:00.0]# echo 1 > ../pci_bus/0000\:7c/bus_rescan
[  705.747983] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
[  705.753983] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
[  705.761351] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
[  705.768755] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
[  705.776208] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
[  705.786515] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
[  705.793967] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
[  705.804377] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
[  705.814859] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
[  705.821532] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
[  705.828553] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
[  705.835228] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
[  705.842250] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
[  705.850135] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
[  705.858022] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
[  705.865909] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
[  705.873917] hns3 0000:7d:00.3: Adding to iommu group 40
[  705.879792] hns3 0000:7d:00.3: The firmware version is 1.9.23.6
[  705.887760] libphy: hisilicon MII bus: probed
[  705.948493] hns3 0000:7d:00.3: hclge driver initialization finished.
[  705.955523] RTL8211F Gigabit Ethernet mii-0000:7d:00.3:07: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=mii-0000:7d:00.3:07, irq=POLL)
[root@localhost 0000:7c:00.0]#


On 2020/1/9 11:35, Yicong Yang wrote:
> Hi,
>
> recently I met a problem with pci bus resource allocation. The allocation strategy
> makes me confused and leads to a wrong allocation results.
>
> There is a hisilicon network device with four functions under one root port. The
> original bios resources allocation looks like:
>
> 7c:00.0 Root Port
>      prefetchable memory behind bridge: 12000000-0x1210fffff 17M [64bit pref]
>     7d:00.0
>         bar0: 0x121000000-0x12100ffff 64k  [64bit pref]
>         bar2: 0x120000000-0x1200fffff 1M   [64bit pref]
>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>         bar9: 0x120100000-0x1203fffff 3M   [64bit pref]
>     7d:00.1
>         bar0: 0x121040000-0x12104ffff 64k  [64bit pref]
>         bar2: 0x120400000-0x1204fffff 1M   [64bit pref]
>         bar7: 0x121050000-0x12107ffff 128K [64bit pref]
>         bar9: 0x120500000-0x1207fffff 3M   [64bit pref]
>     7d:00.2
>         bar0: 0x121080000-0x12108ffff 64k  [64bit pref]
>         bar2: 0x120800000-0x1208fffff 1M   [64bit pref]
>         bar7: 0x121090000-0x1210bffff 128K [64bit pref]
>         bar9: 0x120900000-0x120bfffff 3M   [64bit pref]
>     7d:00.3
>         bar0: 0x1210c0000-0x1210cffff 64k  [64bit pref]
>         bar2: 0x120c00000-0x120cfffff 1M   [64bit pref]
>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>         bar9: 0x120d00000-0x120ffffff 3M   [64bit pref]
>
> When I remove function 7d:00.3 and try to rescan the bus[7c], kernel prints the
> error information.
> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
> [  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
> [  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
> [  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
> [  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
> [  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^   
> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^
> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.864261] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
> [  391.872148] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
> [  391.880035] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
> [  391.887920] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
>
> When looking into the code, the functions called like:
>     pci_rescan_bus()
>         pci_assign_unassigned_bus_resources()
>             __pci_bus_size_bridges()
>                 pbus_size_mem()
>
> The function 7d:00.3 is added and enabled well as the required resources are satisfied.
> As it request 64bit prefetchable resources, there is no reason to open bar14 for it.
>
> When a new function is added, the framework trys to size the bridge memory
> window for it. In __pci_bus_size_bridges(), firstly the framework trys to size bar15 for the
> new added 5M resources as we require 64bit pref mem. But bar15 has *parent*
> so pbus_size_mem() return failure with bar15 unchanged. Then the framework try to put
> resources in bar14, 32bit mem window, and the bar14 is unused so it is sized to 5M and
> pbus_size_mem() return success.
> After bridge size settles down, the framework assign resources for each bar. *As the bios
> doesn't reserve a 32bit mem window for the bridge*, bar14 assignment is failed and print
> the error assigen information. When assigning 7d:00.3, the framework try to find a space
> in bar15 firstly and succeed. Then the flow is terminated. The bar14 is even not touched.
>
> Here comes the question:
>     Why should we resize the bridge memory window when only one function is removed and
> rescanned later? The bridge memory window should remain unchanged in such a situation.
>     Is there a *certain condition* which requirs such operation?
>     If all the functions are removed, we should use pci_rescan_bus_bridge_resize() to rescan and add
> devices as the required memory size maybe changed.  Otherwise, i think we should try to assign
> resources directly without sizing the bridge resource.
>    
> Thanks,
> Yang
>
>
> .
>

Re: PCI: bus resource allocation error

[Nicholas Johnson]

Hi,

On Fri, Jan 10, 2020 at 03:33:27PM +0800, Yicong Yang wrote:
> Hi,
> 
> It seems the attachments are blocked by the server.
> The necessary console output is below.
> The kernel version is 5.4, centos release 7.6.  I didn't
> change the PCI codes.

It is very difficult for me to get the wider picture of your system 
without the full output of "sudo lspci -xxxx". Can you place them on 
PasteBin and send the links, rather than attaching them directly?

I can try to speculate based on what you sent, but I cannot be sure it 
will be enough. For example, I do not know if your computer has multiple 
root complexes, which have shown to complicate things.

Thanks!

Re: PCI: bus resource allocation error

[Yicong Yang]

Hi,

I cannot get the log on the original machine.
So I tested on another one and got the same error prints.
You can find the console output at
https://paste.ubuntu.com/p/5VHVnKWSty/
as well as the "lspci -xxxx" infomation.

Thanks,
Yang

On 2020/1/10 15:40, Nicholas Johnson wrote:
> Hi,
>
> On Fri, Jan 10, 2020 at 03:33:27PM +0800, Yicong Yang wrote:
>> Hi,
>>
>> It seems the attachments are blocked by the server.
>> The necessary console output is below.
>> The kernel version is 5.4, centos release 7.6.  I didn't
>> change the PCI codes.
> It is very difficult for me to get the wider picture of your system 
> without the full output of "sudo lspci -xxxx". Can you place them on 
> PasteBin and send the links, rather than attaching them directly?
>
> I can try to speculate based on what you sent, but I cannot be sure it 
> will be enough. For example, I do not know if your computer has multiple 
> root complexes, which have shown to complicate things.
>
> Thanks!
>
>

Re: PCI: bus resource allocation error

[Yicong Yang]

Hi Bjorn and Nicholas,

Would you mind looking at the this and help me with the issues?

I reproduced the issues on another machine and pasted the console log along with
the lspci info on https://paste.ubuntu.com/p/5VHVnKWSty/.

As it has been a long time since last mail, I briefly illustrate the issues below:

There are 4 functions of a network card under one root port as below:
 +-[0000:7c]---00.0-[7d]--+-00.0  Device 19e5:a222
 |                        +-00.1  Device 19e5:a222
 |                        +-00.2  Device 19e5:a222
 |                        \-00.3  Device 19e5:a221

When I remove one function and rescan the bus[7c], the kernel print the error
message as below:

[  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
[  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
[  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
[  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
[  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
[  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
[  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
[  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
[  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
or the machine in the pastebin prints:

[  790.671091] pci 0000:7d:00.3: Removing from iommu group 5
[  937.541937] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
[  937.541949] pci 0000:7d:00.3: reg 0x10: [mem 0x1221f0000-0x1221fffff 64bit pref]
[  937.541953] pci 0000:7d:00.3: reg 0x18: [mem 0x121f00000-0x121ffffff 64bit pref]
[  937.542113] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[  937.542116] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[  937.542120] pci 0000:7d:00.3: BAR 2: assigned [mem 0x121f00000-0x121ffffff 64bit pref]
[  937.542125] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1221f0000-0x1221fffff 64bit pref]
[  937.542253] hns3 0000:7d:00.3: Adding to iommu group 5

Both the function and the root ports work well, and the function get the resource it requested as before
remove. So from my perspective the message shouldn't be printed and should be eliminated.

I looked into the codes and got some informations:

when echo 1 > bus_rescan, kernel calls:
    pci_rescan_bus()
        pci_assign_unassigned_bus_resources()
            __pci_bus_size_bridges()
               /* first try to put the resource in 64 bit MMIO window(Bar 15). 
                * As it's not empty, function will return directly.
                */
               pbus_size_mem()

               /* Then try to put rest resource in 32-bit MMIO window(Bar 14)
                * As it's not occupied by any functions, the resources are
                * put here. As no io memory is reserved in the bios for bar14,
                * error message prints when allocated.
                */
               pbus_size_mem() /* problem is here */

In pbus_size_mem(), kernel try to size the bar to contain function resource.
If it's occupied by any function (judged by find_bus_resource_of_type() in
pbus_size_mem()), it'll return directly without any sizing. Otherwise the bar
will be sized to put the request resource in.

It's just a rescan process, the bar size shouldn't be sized or allocated by
calling __pci_bus_size_bridges(). As previous resource space in the bar
reserved and the function will demands no extra spaces after rescan.
The current process seems unreasonable.

Thanks,
Yicong Yang

Re: PCI: bus resource allocation error

[Nicholas Johnson]

On Tue, Feb 11, 2020 at 06:36:42PM +0800, Yicong Yang wrote:
> Hi Bjorn and Nicholas,
> 
> Would you mind looking at the this and help me with the issues?
Sorry for dropping off the radar, I have had a lot going on.

> 
> I reproduced the issues on another machine and pasted the console log along with
> the lspci info on https://paste.ubuntu.com/p/5VHVnKWSty/.
> 
> As it has been a long time since last mail, I briefly illustrate the issues below:
> 
> There are 4 functions of a network card under one root port as below:
>  +-[0000:7c]---00.0-[7d]--+-00.0  Device 19e5:a222
>  |                        +-00.1  Device 19e5:a222
>  |                        +-00.2  Device 19e5:a222
>  |                        \-00.3  Device 19e5:a221
> 
> When I remove one function and rescan the bus[7c], the kernel print the error
> message as below:
Why do you need to remove and rescan the bus? It is possible that we 
should be addressing the problem of why you have the need to do that in 
the first place, rather than why the rescan does not work.

> 
> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
> [  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
> [  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
> [  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
> [  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
> [  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
> or the machine in the pastebin prints:
> 
> [  790.671091] pci 0000:7d:00.3: Removing from iommu group 5
> [  937.541937] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
> [  937.541949] pci 0000:7d:00.3: reg 0x10: [mem 0x1221f0000-0x1221fffff 64bit pref]
> [  937.541953] pci 0000:7d:00.3: reg 0x18: [mem 0x121f00000-0x121ffffff 64bit pref]
> [  937.542113] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [  937.542116] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [  937.542120] pci 0000:7d:00.3: BAR 2: assigned [mem 0x121f00000-0x121ffffff 64bit pref]
> [  937.542125] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1221f0000-0x1221fffff 64bit pref]
> [  937.542253] hns3 0000:7d:00.3: Adding to iommu group 5
> 
> Both the function and the root ports work well, and the function get the resource it requested as before
> remove. So from my perspective the message shouldn't be printed and should be eliminated.
> 
> I looked into the codes and got some informations:
> 
> when echo 1 > bus_rescan, kernel calls:
>     pci_rescan_bus()
>         pci_assign_unassigned_bus_resources()
>             __pci_bus_size_bridges()
>                /* first try to put the resource in 64 bit MMIO window(Bar 15). 
>                 * As it's not empty, function will return directly.
>                 */
>                pbus_size_mem()
> 
>                /* Then try to put rest resource in 32-bit MMIO window(Bar 14)
>                 * As it's not occupied by any functions, the resources are
>                 * put here. As no io memory is reserved in the bios for bar14,
>                 * error message prints when allocated.
>                 */
>                pbus_size_mem() /* problem is here */
> 
> In pbus_size_mem(), kernel try to size the bar to contain function resource.
> If it's occupied by any function (judged by find_bus_resource_of_type() in
> pbus_size_mem()), it'll return directly without any sizing. Otherwise the bar
> will be sized to put the request resource in.
> 
> It's just a rescan process, the bar size shouldn't be sized or allocated by
> calling __pci_bus_size_bridges(). As previous resource space in the bar
> reserved and the function will demands no extra spaces after rescan.
> The current process seems unreasonable.
If the BIOS assigned the resources with different packing than what the 
kernel would do, then the rescan may not fit into the space. You can try 
pci=realloc,nocrs if you have not already. Your system looks like it is 
ARM64 so you cannot use pci=nocrs, unfortunately. The ideal situation is 
if the kernel throws away everything the BIOS did and does everything 
itself (assuming that this will not cause platform conflicts). But there 
is no way of doing this without modifying the kernel, and I am not sure 
how to do it fully.

However, I have come to find that the code in drivers/pci/setup-bus.c is 
very old and full of holes. It does require a re-write (in my opinion, 
and some other people have agreed to some extent), which I want to do, 
but will need much more experience before I can pull something like that 
off and get it accepted by others.

Unfortunately without the system in front of me and the ability to make 
lots of changes to the kernel very quickly to find a fix, I might not be 
able to figure it out.

> 
> Thanks,
> Yicong Yang
> 
> 

Regards,
Nicholas

Re: PCI: bus resource allocation error

[Bjorn Helgaas]

On Tue, Feb 11, 2020 at 01:43:16PM +0000, Nicholas Johnson wrote:
> If the BIOS assigned the resources with different packing than what the 
> kernel would do, then the rescan may not fit into the space. You can try 
> pci=realloc,nocrs if you have not already. Your system looks like it is 
> ARM64 so you cannot use pci=nocrs, unfortunately. 

"pci=nocrs" is a poor workaround for BIOS and Linux bugs.  It is
guaranteed to break hot-add on multi-host bridge systems because _CRS
is what tells us what resources go to each bridge.  Even on
single-bridge systems "pci=nocrs" is dangerous because it may cause
Linux to assign resources that aren't routed to PCI or are being used
by other devices.

It's fine for debugging, but it's never the right long-term answer.

> The ideal situation is 
> if the kernel throws away everything the BIOS did and does everything 
> itself (assuming that this will not cause platform conflicts).

I do not think this is the ideal situation.  If the assignment done by
BIOS works, I think Linux should leave it alone.

If the BIOS assignment *doesn't* work, or if we hot-add a device and
can't assign resources to it, sure, it makes sense for Linux to try to
reassign things.  But we should not throw away the BIOS assignments as
a matter of course.

Bjorn

Re: PCI: bus resource allocation error

[Yicong Yang]

Hi Nicholas,


On 2020/2/11 21:43, Nicholas Johnson wrote:
>> There are 4 functions of a network card under one root port as below:
>>  +-[0000:7c]---00.0-[7d]--+-00.0  Device 19e5:a222
>>  |                        +-00.1  Device 19e5:a222
>>  |                        +-00.2  Device 19e5:a222
>>  |                        \-00.3  Device 19e5:a221
>>
>> When I remove one function and rescan the bus[7c], the kernel print the error
>> message as below:
> Why do you need to remove and rescan the bus? It is possible that we 
> should be addressing the problem of why you have the need to do that in 
> the first place, rather than why the rescan does not work.

It is found in a test procedure coincidently. There is no certain condition to do it,
but the problem does exists.


>
>> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
>> [  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
>> [  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
>> [  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
>> [  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
>> [  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
>> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
>> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
>> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>>                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>>                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
>> or the machine in the pastebin prints:
>>
>> [  790.671091] pci 0000:7d:00.3: Removing from iommu group 5
>> [  937.541937] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
>> [  937.541949] pci 0000:7d:00.3: reg 0x10: [mem 0x1221f0000-0x1221fffff 64bit pref]
>> [  937.541953] pci 0000:7d:00.3: reg 0x18: [mem 0x121f00000-0x121ffffff 64bit pref]
>> [  937.542113] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  937.542116] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  937.542120] pci 0000:7d:00.3: BAR 2: assigned [mem 0x121f00000-0x121ffffff 64bit pref]
>> [  937.542125] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1221f0000-0x1221fffff 64bit pref]
>> [  937.542253] hns3 0000:7d:00.3: Adding to iommu group 5
>>
>> Both the function and the root ports work well, and the function get the resource it requested as before
>> remove. So from my perspective the message shouldn't be printed and should be eliminated.
>>
>> I looked into the codes and got some informations:
>>
>> when echo 1 > bus_rescan, kernel calls:
>>     pci_rescan_bus()
>>         pci_assign_unassigned_bus_resources()
>>             __pci_bus_size_bridges()
>>                /* first try to put the resource in 64 bit MMIO window(Bar 15). 
>>                 * As it's not empty, function will return directly.
>>                 */
>>                pbus_size_mem()
>>
>>                /* Then try to put rest resource in 32-bit MMIO window(Bar 14)
>>                 * As it's not occupied by any functions, the resources are
>>                 * put here. As no io memory is reserved in the bios for bar14,
>>                 * error message prints when allocated.
>>                 */
>>                pbus_size_mem() /* problem is here */
>>
>> In pbus_size_mem(), kernel try to size the bar to contain function resource.
>> If it's occupied by any function (judged by find_bus_resource_of_type() in
>> pbus_size_mem()), it'll return directly without any sizing. Otherwise the bar
>> will be sized to put the request resource in.
>>
>> It's just a rescan process, the bar size shouldn't be sized or allocated by
>> calling __pci_bus_size_bridges(). As previous resource space in the bar
>> reserved and the function will demands no extra spaces after rescan.
>> The current process seems unreasonable.
> If the BIOS assigned the resources with different packing than what the 
> kernel would do, then the rescan may not fit into the space. You can try 
> pci=realloc,nocrs if you have not already. Your system looks like it is 
> ARM64 so you cannot use pci=nocrs, unfortunately. The ideal situation is 
> if the kernel throws away everything the BIOS did and does everything 
> itself (assuming that this will not cause platform conflicts). But there 
> is no way of doing this without modifying the kernel, and I am not sure 
> how to do it fully.

pci=realloc affects when allocating the root bridge and root bus. So it doesn't
meet the condition here, as the root bus resource has been allocated and the problem
occurs when allocating and assigning resource on subordinate bus [7d]. And it does do
no help as I add pci=realloc when boot.

Luckily, I found your patch c13704f5685d ("PCI: Avoid double hpmemsize MMIO window assignment")
which have solved the problem. It's merged in v5.5 kernel so I didn't test it as the
issues happened on v5.4.

The find_free_bus_resource() is modified to find_bus_resource_of_type() in the patch, and it'll return the
first assigned resource matching the target type rather than previous NULL if not found. Then when
allocating target resource (64 bit pref) in 64 bit pref bar15 when first time call pbus_size_mem(),
we'll get success 0 rather than -ENOSPEC. Then we'll not try to put the target resource in
bar14 subsequently as previous did, which will avoid opening bar14 and wrong allocating. Thanks for you
good work!

But for me it still seems unnecessary to call __pci_bus_size_bridges() to try to size
the bus in a rescan routine, actually it seems redundant here. Is it possible to call
__pci_bus_assign_resources() in pci_rescan_bus() directly? However, I haven't look further
in it, just thoughts.

Thanks,
Yicong Yang

>
> However, I have come to find that the code in drivers/pci/setup-bus.c is 
> very old and full of holes. It does require a re-write (in my opinion, 
> and some other people have agreed to some extent), which I want to do, 
> but will need much more experience before I can pull something like that 
> off and get it accepted by others.
>
> Unfortunately without the system in front of me and the ability to make 
> lots of changes to the kernel very quickly to find a fix, I might not be 
> able to figure it out.
>
>> Thanks,
>> Yicong Yang
>>
>>
> Regards,
> Nicholas
>
> .
>