Re: PCI: bus resource allocation error

[Yicong Yang <yangyicong@hisilicon.com>]

On 2020/1/9 12:27, Bjorn Helgaas wrote:
> [+cc Nicholas, who is working in this area]
>
> On Thu, Jan 09, 2020 at 11:35:09AM +0800, Yicong Yang wrote:
>> Hi,
>>
>> recently I met a problem with pci bus resource allocation. The allocation strategy
>> makes me confused and leads to a wrong allocation results.
>>
>> There is a hisilicon network device with four functions under one root port. The
>> original bios resources allocation looks like:
> What kernel is this?  Can you collect the complete dmesg log?

The kernel version is 5.4.0.  the dmesg log is like:

[  496.598130] hns3 0000:7d:00.3 eth11: net stop
[  496.602702] hns3 0000:7d:00.3 eth11: link down
[  496.655963] pci 0000:7d:00.3: Removing from iommu group 49
[  508.453948] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
[  508.459943] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
[  508.467317] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
[  508.474720] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
[  508.482175] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
[  508.492485] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
[  508.499939] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
[  508.510351] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
[  508.520836] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
[  508.527513] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
[  508.534538] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
[  508.541216] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
[  508.548246] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
[  508.556132] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
[  508.564022] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
[  508.571909] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
[  508.579903] hns3 0000:7d:00.3: Adding to iommu group 49
[  508.585782] hns3 0000:7d:00.3: The firmware version is 1.9.23.6
[  508.594571] libphy: hisilicon MII bus: probed
[  508.694463] hns3 0000:7d:00.3: hclge driver initialization finished.
[  508.701446] RTL8211F Gigabit Ethernet mii-0000:7d:00.3:07: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=mii-0000:7d:00.3:07, irq=POLL)

>
>> 7c:00.0 Root Port
>>      prefetchable memory behind bridge: 12000000-0x1210fffff 17M [64bit pref]
>>     7d:00.0
>>         bar0: 0x121000000-0x12100ffff 64k  [64bit pref]
>>         bar2: 0x120000000-0x1200fffff 1M   [64bit pref]
>>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>>         bar9: 0x120100000-0x1203fffff 3M   [64bit pref]
>>     7d:00.1
>>         bar0: 0x121040000-0x12104ffff 64k  [64bit pref]
>>         bar2: 0x120400000-0x1204fffff 1M   [64bit pref]
>>         bar7: 0x121050000-0x12107ffff 128K [64bit pref]
>>         bar9: 0x120500000-0x1207fffff 3M   [64bit pref]
>>     7d:00.2
>>         bar0: 0x121080000-0x12108ffff 64k  [64bit pref]
>>         bar2: 0x120800000-0x1208fffff 1M   [64bit pref]
>>         bar7: 0x121090000-0x1210bffff 128K [64bit pref]
>>         bar9: 0x120900000-0x120bfffff 3M   [64bit pref]
>>     7d:00.3
>>         bar0: 0x1210c0000-0x1210cffff 64k  [64bit pref]
>>         bar2: 0x120c00000-0x120cfffff 1M   [64bit pref]
>>         bar7: 0x121010000-0x12103ffff 128K [64bit pref]
>>         bar9: 0x120d00000-0x120ffffff 3M   [64bit pref]
>>
>> When I remove function 7d:00.3 and try to rescan the bus[7c], kernel prints the
>> error information.
>> [  391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
>> [  391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
>> [  391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
>> [  391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
>> [  391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
>> [  391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
>> [  391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
>> [  391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
>> [  391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^   
>> [  391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
>>                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
>> [  391.864261] pci 0000:7d:00.3: BAR 2: assigned [mem 0x120c00000-0x120cfffff 64bit pref]
>> [  391.872148] pci 0000:7d:00.3: BAR 9: assigned [mem 0x120d00000-0x120ffffff 64bit pref]
>> [  391.880035] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1210c0000-0x1210cffff 64bit pref]
>> [  391.887920] pci 0000:7d:00.3: BAR 7: assigned [mem 0x1210d0000-0x1210fffff 64bit pref]
>>
>> When looking into the code, the functions called like:
>>     pci_rescan_bus()
>>         pci_assign_unassigned_bus_resources()
>>             __pci_bus_size_bridges()
>>                 pbus_size_mem()
>>
>> The function 7d:00.3 is added and enabled well as the required resources are satisfied.
>> As it request 64bit prefetchable resources, there is no reason to open bar14 for it.
>>
>> When a new function is added, the framework trys to size the bridge memory
>> window for it. In __pci_bus_size_bridges(), firstly the framework trys to size bar15 for the
>> new added 5M resources as we require 64bit pref mem. But bar15 has *parent*
>> so pbus_size_mem() return failure with bar15 unchanged. Then the framework try to put
>> resources in bar14, 32bit mem window, and the bar14 is unused so it is sized to 5M and
>> pbus_size_mem() return success.
>> After bridge size settles down, the framework assign resources for each bar. *As the bios
>> doesn't reserve a 32bit mem window for the bridge*, bar14 assignment is failed and print
>> the error assigen information. When assigning 7d:00.3, the framework try to find a space
>> in bar15 firstly and succeed. Then the flow is terminated. The bar14 is even not touched.
>>
>> Here comes the question:
>>     Why should we resize the bridge memory window when only one function is removed and
>> rescanned later? The bridge memory window should remain unchanged in such a situation.
>>     Is there a *certain condition* which requirs such operation?
>>     If all the functions are removed, we should use pci_rescan_bus_bridge_resize() to rescan and add
>> devices as the required memory size maybe changed.  Otherwise, i think we should try to assign
>> resources directly without sizing the bridge resource.
>>    
>> Thanks,
>> Yang
>>
> .
>