From: Bjorn Helgaas <helgaas@kernel.org>
To: Tomasz Pala <gotar@polanet.pl>
Cc: linux-pci@vger.kernel.org,
"Dan J Williams" <dan.j.williams@intel.com>,
"Kan Liang" <kan.liang@linux.intel.com>,
"Tony Luck" <tony.luck@intel.com>,
"David E Box" <david.e.box@intel.com>,
"Yunying Sun" <yunying.sun@intel.com>,
"Dave Jiang" <dave.jiang@intel.com>,
"Mika Westerberg" <mika.westerberg@linux.intel.com>,
"Giovanni Cabiddu" <giovanni.cabiddu@intel.com>,
"Herbert Xu" <herbert@gondor.apana.org.au>,
"Hans de Goede" <hdegoede@redhat.com>,
"Florent DELAHAYE" <linuxkernelml@undead.fr>,
"Konrad J Hambrick" <kjhambrick@gmail.com>,
"Matt Hansen" <2lprbe78@duck.com>,
"Nicholas Johnson" <nicholas.johnson-opensource@outlook.com.au>,
"Benoit Grégoire" <benoitg@coeus.ca>,
"Werner Sembach" <wse@tuxedocomputers.com>,
mumblingdrunkard@protonmail.com, linux-kernel@vger.kernel.org,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Sebastian Manciulea" <manciuleas@protonmail.com>
Subject: Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Date: Thu, 9 Nov 2023 12:44:05 -0600 [thread overview]
Message-ID: <20231109184405.GA488882@bhelgaas> (raw)
In-Reply-To: <20231103191858.GA163196@bhelgaas>
[+cc Sebastian]
On Fri, Nov 03, 2023 at 02:18:58PM -0500, Bjorn Helgaas wrote:
> On Thu, Oct 26, 2023 at 03:53:19PM -0500, Bjorn Helgaas wrote:
> > On Thu, Oct 12, 2023 at 05:33:47PM +0200, Tomasz Pala wrote:
> > > On Tue, Jan 10, 2023 at 12:02:43 -0600, Bjorn Helgaas wrote:
> > > > Normally we reject ECAM space unless it is reported as
> > > > reserved in the E820 table or via a PNP0C02 _CRS method (PCI
> > > > Firmware, r3.3, sec 4.1.2). This means PCI extended config
> > > > space (offsets 0x100-0xfff) may not be accessible.
> > > >
> > > > Some firmware doesn't report ECAM space via PNP0C02 _CRS
> > > > methods, but does mention it as an EfiMemoryMappedIO region
> > > > via EFI GetMemoryMap(), which is normally converted to an E820
> > > > entry by a bootloader or EFI stub.
> > > >
> > > > 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820
> > > > map"), removes E820 entries that correspond to
> > > > EfiMemoryMappedIO regions because some other firmware uses
> > > > EfiMemoryMappedIO for PCI host bridge windows, and the E820
> > > > entries prevent Linux from allocating BAR space for hot-added
> > > > devices.
> > > >
> > > > Allow use of ECAM for extended config space when the region is
> > > > covered by an EfiMemoryMappedIO region, even if it's not
> > > > included in E820 or PNP0C02 _CRS.
> > >
> > > I'm still having a problem initializing ixgbe NICs with pristine
> > > 6.5.7 kernel.
> >
> > Thanks very much for the report, and sorry for the inconvenience and
> > my delay in looking at it.
> >
> > > efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
> > > [mem 0x7f800000-0xfed1bfff] available for PCI devices
> > > PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
> > > [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
> > > PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
> > > ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
> > > ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
> > > ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
> > > ixgbe: probe of 0000:02:00.0 failed with error -16
> >
> > Something is wrong with our allocation scheme. Both the MMCONFIG
> > region and the ixgbe BAR 0 are at 0x80000000, which obviously cannot
> > work. Maybe the full dmesg log will have a clue about why we didn't
> > move ixgbe out of the way.
> >
> > > After disabling the code causing this (using always-false condition:
> > > if (size >= 256*1024 && 0) {
> > > ) in the chunk:
> > >
> > > https://lore.kernel.org/lkml/20221208190341.1560157-2-helgaas@kernel.org/
> > >
> > > the BAR starts at 0x90000000 (not 0x80000000):
> > >
> > > efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map
> > > [...]
> > > [mem 0x90000000-0xfed1bfff] available for PCI devices
> > > [...]
> > > PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
> > > PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry
> > >
> > > and everything seems to work again.
> > >
> > >
> > > I've got full system bootup logs from the upstream and worked around,
> > > but I'm not sure if this is OK to attach them (the CC list is long).
> >
> > Would you mind opening a new report at https://bugzilla.kernel.org,
> > attaching those logs, and responding here with the URL?
>
> Thanks for the report and the logs, which are attached at
> https://bugzilla.kernel.org/show_bug.cgi?id=218050
>
> I think the problem is that the MMCONFIG region is at
> [mem 0x80000000-0x8fffffff], and that is *also* included in one of the
> host bridge windows reported via _CRS:
>
> PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
> pci_bus 0000:00: root bus resource [mem 0x80000000-0xfbffffff window]
>
> I'll try to figure out how to deal with that. In the meantime, would
> you mind attaching the contents of /proc/iomem to the bugzilla? I
> think you have to cat it as root to get the actual values included.
I attached a debug patch to both bugzilla entries. If you could
attach the "acpidump" output and (if practical) boot a kernel with the
debug patch and attach the dmesg logs, that would be great.
Bjorn
next prev parent reply other threads:[~2023-11-09 18:44 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-10 18:02 [PATCH 0/2] PCI: Fix extended config space regression Bjorn Helgaas
2023-01-10 18:02 ` [PATCH 1/2] x86/pci: Simplify is_mmconf_reserved() messages Bjorn Helgaas
2023-01-10 18:07 ` Dan Williams
2023-01-10 18:12 ` Bjorn Helgaas
2023-01-10 18:02 ` [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space Bjorn Helgaas
2023-01-10 18:15 ` Rafael J. Wysocki
2023-01-10 18:29 ` Dan Williams
2023-01-10 19:30 ` Bjorn Helgaas
2023-10-12 15:33 ` Tomasz Pala
2023-10-16 17:31 ` Tomasz Pala
2023-10-26 20:53 ` Bjorn Helgaas
2023-11-03 19:18 ` Bjorn Helgaas
2023-11-09 18:44 ` Bjorn Helgaas [this message]
2023-11-18 14:21 ` Tomasz Pala
2023-11-20 16:29 ` Bjorn Helgaas
2023-11-21 15:24 ` Tomasz Pala
2023-11-21 18:19 ` Bjorn Helgaas
2023-11-08 17:47 ` Bjorn Helgaas
2023-11-08 20:20 ` Bjorn Helgaas
2023-12-06 11:54 ` Linux regression tracking #update (Thorsten Leemhuis)
2023-01-10 20:38 ` [PATCH 0/2] PCI: Fix extended config space regression Liang, Kan
2023-01-10 22:57 ` Luck, Tony
2023-01-11 12:10 ` Giovanni Cabiddu
2023-01-11 22:30 ` Bjorn Helgaas
2023-01-12 5:50 ` Sun, Yunying
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231109184405.GA488882@bhelgaas \
--to=helgaas@kernel.org \
--cc=2lprbe78@duck.com \
--cc=benoitg@coeus.ca \
--cc=bhelgaas@google.com \
--cc=dan.j.williams@intel.com \
--cc=dave.jiang@intel.com \
--cc=david.e.box@intel.com \
--cc=giovanni.cabiddu@intel.com \
--cc=gotar@polanet.pl \
--cc=hdegoede@redhat.com \
--cc=herbert@gondor.apana.org.au \
--cc=kan.liang@linux.intel.com \
--cc=kjhambrick@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linuxkernelml@undead.fr \
--cc=manciuleas@protonmail.com \
--cc=mika.westerberg@linux.intel.com \
--cc=mumblingdrunkard@protonmail.com \
--cc=nicholas.johnson-opensource@outlook.com.au \
--cc=tony.luck@intel.com \
--cc=wse@tuxedocomputers.com \
--cc=yunying.sun@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).