From: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
To: Yicong Yang <yangyicong@hisilicon.com>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
fangjian 00545541 <f.fangjian@huawei.com>
Subject: Re: PCI: bus resource allocation error
Date: Tue, 11 Feb 2020 13:43:16 +0000 [thread overview]
Message-ID: <SL2P216MB04433D05965A7D1C0E6A4BD680180@SL2P216MB0443.KORP216.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <2e588019-0a42-c386-7314-e1cf5dbc3371@hisilicon.com>
On Tue, Feb 11, 2020 at 06:36:42PM +0800, Yicong Yang wrote:
> Hi Bjorn and Nicholas,
>
> Would you mind looking at the this and help me with the issues?
Sorry for dropping off the radar, I have had a lot going on.
>
> I reproduced the issues on another machine and pasted the console log along with
> the lspci info on https://paste.ubuntu.com/p/5VHVnKWSty/.
>
> As it has been a long time since last mail, I briefly illustrate the issues below:
>
> There are 4 functions of a network card under one root port as below:
> +-[0000:7c]---00.0-[7d]--+-00.0 Device 19e5:a222
> | +-00.1 Device 19e5:a222
> | +-00.2 Device 19e5:a222
> | \-00.3 Device 19e5:a221
>
> When I remove one function and rescan the bus[7c], the kernel print the error
> message as below:
Why do you need to remove and rescan the bus? It is possible that we
should be addressing the problem of why you have the need to do that in
the first place, rather than why the rescan does not work.
>
> [ 391.770030] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
> [ 391.776024] pci 0000:7d:00.3: reg 0x10: [mem 0x1210c0000-0x1210cffff 64bit pref]
> [ 391.783394] pci 0000:7d:00.3: reg 0x18: [mem 0x120c00000-0x120cfffff 64bit pref]
> [ 391.790786] pci 0000:7d:00.3: reg 0x224: [mem 0x1210d0000-0x1210dffff 64bit pref]
> [ 391.798238] pci 0000:7d:00.3: VF(n) BAR0 space: [mem 0x1210d0000-0x1210fffff 64bit pref] (contains BAR0 for 3 VFs)
> [ 391.808543] pci 0000:7d:00.3: reg 0x22c: [mem 0x120d00000-0x120dfffff 64bit pref]
> [ 391.815994] pci 0000:7d:00.3: VF(n) BAR2 space: [mem 0x120d00000-0x120ffffff 64bit pref] (contains BAR2 for 3 VFs)
> [ 391.826391] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 300000 add_align 100000
> [ 391.836869] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00500000]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [ 391.843543] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00500000]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [ 391.850562] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [ 391.857237] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> or the machine in the pastebin prints:
>
> [ 790.671091] pci 0000:7d:00.3: Removing from iommu group 5
> [ 937.541937] pci 0000:7d:00.3: [19e5:a221] type 00 class 0x020000
> [ 937.541949] pci 0000:7d:00.3: reg 0x10: [mem 0x1221f0000-0x1221fffff 64bit pref]
> [ 937.541953] pci 0000:7d:00.3: reg 0x18: [mem 0x121f00000-0x121ffffff 64bit pref]
> [ 937.542113] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [ 937.542116] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> [ 937.542120] pci 0000:7d:00.3: BAR 2: assigned [mem 0x121f00000-0x121ffffff 64bit pref]
> [ 937.542125] pci 0000:7d:00.3: BAR 0: assigned [mem 0x1221f0000-0x1221fffff 64bit pref]
> [ 937.542253] hns3 0000:7d:00.3: Adding to iommu group 5
>
> Both the function and the root ports work well, and the function get the resource it requested as before
> remove. So from my perspective the message shouldn't be printed and should be eliminated.
>
> I looked into the codes and got some informations:
>
> when echo 1 > bus_rescan, kernel calls:
> pci_rescan_bus()
> pci_assign_unassigned_bus_resources()
> __pci_bus_size_bridges()
> /* first try to put the resource in 64 bit MMIO window(Bar 15).
> * As it's not empty, function will return directly.
> */
> pbus_size_mem()
>
> /* Then try to put rest resource in 32-bit MMIO window(Bar 14)
> * As it's not occupied by any functions, the resources are
> * put here. As no io memory is reserved in the bios for bar14,
> * error message prints when allocated.
> */
> pbus_size_mem() /* problem is here */
>
> In pbus_size_mem(), kernel try to size the bar to contain function resource.
> If it's occupied by any function (judged by find_bus_resource_of_type() in
> pbus_size_mem()), it'll return directly without any sizing. Otherwise the bar
> will be sized to put the request resource in.
>
> It's just a rescan process, the bar size shouldn't be sized or allocated by
> calling __pci_bus_size_bridges(). As previous resource space in the bar
> reserved and the function will demands no extra spaces after rescan.
> The current process seems unreasonable.
If the BIOS assigned the resources with different packing than what the
kernel would do, then the rescan may not fit into the space. You can try
pci=realloc,nocrs if you have not already. Your system looks like it is
ARM64 so you cannot use pci=nocrs, unfortunately. The ideal situation is
if the kernel throws away everything the BIOS did and does everything
itself (assuming that this will not cause platform conflicts). But there
is no way of doing this without modifying the kernel, and I am not sure
how to do it fully.
However, I have come to find that the code in drivers/pci/setup-bus.c is
very old and full of holes. It does require a re-write (in my opinion,
and some other people have agreed to some extent), which I want to do,
but will need much more experience before I can pull something like that
off and get it accepted by others.
Unfortunately without the system in front of me and the ability to make
lots of changes to the kernel very quickly to find a fix, I might not be
able to figure it out.
>
> Thanks,
> Yicong Yang
>
>
Regards,
Nicholas
next prev parent reply other threads:[~2020-02-11 13:43 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-09 3:35 PCI: bus resource allocation error Yicong Yang
2020-01-09 4:27 ` Bjorn Helgaas
2020-01-09 10:31 ` Yicong Yang
2020-01-09 21:55 ` Bjorn Helgaas
2020-01-09 23:55 ` Nicholas Johnson
2020-01-10 0:08 ` Bjorn Helgaas
2020-01-09 23:00 ` Bjorn Helgaas
2020-01-10 7:08 ` Yicong Yang
2020-01-10 7:33 ` Yicong Yang
2020-01-10 7:40 ` Nicholas Johnson
2020-01-14 8:25 ` Yicong Yang
2020-02-11 10:36 ` Yicong Yang
2020-02-11 13:43 ` Nicholas Johnson [this message]
2020-02-11 19:43 ` Bjorn Helgaas
2020-02-18 3:18 ` Yicong Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=SL2P216MB04433D05965A7D1C0E6A4BD680180@SL2P216MB0443.KORP216.PROD.OUTLOOK.COM \
--to=nicholas.johnson-opensource@outlook.com.au \
--cc=f.fangjian@huawei.com \
--cc=helgaas@kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=yangyicong@hisilicon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).