From: Bjorn Helgaas <helgaas@kernel.org>
To: "Billie Alsup (balsup)" <balsup@cisco.com>
Cc: Paul Menzel <pmenzel@molgen.mpg.de>,
Bjorn Helgaas <bhelgaas@google.com>,
Guohan Lu <lguohan@gmail.com>,
"Madhava Reddy Siddareddygari (msiddare)" <msiddare@cisco.com>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges
Date: Thu, 15 Jul 2021 13:56:49 -0500 [thread overview]
Message-ID: <20210715185649.GA1984276@bjorn-Precision-5520> (raw)
In-Reply-To: <545AA576-42A5-47A7-A08A-062582B1569A@cisco.com>
On Thu, Jul 15, 2021 at 06:12:25PM +0000, Billie Alsup (balsup) wrote:
> It took me a while to figure out that the "New Outlook" option
> doesn't actually allow sending plain text, so I have to switch to
> "Old Outlook" mode.
Since you've gone to that much trouble already, also note
http://vger.kernel.org/majordomo-info.html and
https://people.kernel.org/tglx/notes-about-netiquette
BTW, the attribution in the email you quoted below got corrupted in
such a way that it appears that I wrote the whole thing, instead of
what actually happened, which is that I wrote a half dozen lines of
response to your email. Linux uses old skool email conventions ;)
> It is not clear as to what parameters Linux would use to consider a
> window broken.
By "broken," I just mean things that are prohibited by the PCI spec,
like "region doesn't fit in a window of an upstream device" or
"non-prefetchable region placed in a prefetchable window."
> But if the kernel preserves some bridge window
> assignment, then it seems feasible for our BIOS to run this same
> algorithm (reading PLX persistent scratch registers to determine
> window sizes). I will raise this possibility with our own kernel
> team to discuss with the bios team. We can also look more closely
> at the resource_alignment options to see if that might suffice.
> Thanks for the information!
> From: Bjorn Helgaas <helgaas@kernel.org>
> Date: Thursday, July 15, 2021 at 10:14 AM
> To: "Billie Alsup (balsup)" <balsup@cisco.com>
> Cc: Paul Menzel <pmenzel@molgen.mpg.de>, Bjorn Helgaas <bhelgaas@google.com>, Guohan Lu <lguohan@gmail.com>, "Madhava Reddy Siddareddygari (msiddare)" <msiddare@cisco.com>, "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "David S. Miller" <davem@davemloft.net>, Jakub Kicinski <kuba@kernel.org>, "netdev@vger.kernel.org" <netdev@vger.kernel.org>, Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
> Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges
>
> On Thu, Jul 15, 2021 at 04:52:26PM +0000, Billie Alsup (balsup) wrote:
> We are aware of how Cisco device specific this code is, and hadn't
> intended to upstream it. This code was originally written for an
> older kernel version (4.8.28-WR9.0.0.26_cgl). I am not the original
> author; I just ported it into various SONiC linux kernels. We use
> ACPI with SONiC (although not on our non-SONiC products), so I
> thought I might be able to define such windows within the ACPI tree
> and have some generic code to read such configuration information
> from the ACPI tables,. However, initial attempts failed so I went
> with the existing approach. I believe we did look at the hpmmiosize
> parameter, but iirc it applied to each bridge, rather than being a
> pool of address space to dynamically parcel out as necessary.
>
> Right. I mentioned "pci=resource_alignment=" because it claims to be
> able to specify window sizes for specific bridges. But I haven't
> exercised that myself.
>
> There are multiple bridges involved in the hardware (there are 8
> hot-plug fabric cards, each with multiple PCI devices). Devices on
> the card are in multiple power zones, so all devices are not
> immediately visible to the pci scanning code. The top level bridge
> reserves close to 5G. The 2nd level (towards the fabric cards)
> reserve 4.5G. The 3rd level has 9 bridges each reserving 512M. The
> 4th level reserves 384M (with a 512M alignment restriction iirc).
> The 5th level reserves 384M (again with an alignment restriction).
> That defines the bridge hierarchy visible at boot. Things behind
> that 5th level are hot-plugged where there are two more bridge
> levels and 5 devices (1 requiring 2x64M blocks and 4 requiring
> 1x64M).
>
> I'm not sure if the Cisco kernel team has revisited the hpmmiosize
> and resource_alignment parameters since this initial implementation.
> Reading the description of Sergey's patches, he seems to be
> approaching the same problem from a different direction. It is
> unclear if such an approach is practical for our environment. It
> would require updates to all of our SONiC drivers to support
> stopping/remapping/restarting, and it is unclear if that is
> acceptable. It is certainly less preferable to pre-reserving the
> required space. For our embedded product, we know exactly what
> devices will be plugged in, and allowing that to be pre-programmed
> into the PLX eeprom gives us the flexibility we need.
>
> If you know up front what devices are possible and how much space they
> need, possibly your firmware could assign the bridge windows you need.
> Linux generally does not change window assignments unless they are
> broken.
>
> Bjorn
>
>
next prev parent reply other threads:[~2021-07-15 19:07 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <BYAPR11MB3527AEB1E4969C0833D1697ED9129@BYAPR11MB3527.namprd11.prod.outlook.com>
2021-07-15 17:13 ` [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges Bjorn Helgaas
2021-07-15 18:12 ` Billie Alsup (balsup)
2021-07-15 18:56 ` Bjorn Helgaas [this message]
2021-07-15 19:49 ` Billie Alsup (balsup)
2021-07-15 19:56 ` Bjorn Helgaas
2021-07-13 7:31 [PATCH] " Paul Menzel
2021-07-15 13:50 ` [RFC][PATCH] " Paul Menzel
2021-07-15 14:07 ` Bjorn Helgaas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210715185649.GA1984276@bjorn-Precision-5520 \
--to=helgaas@kernel.org \
--cc=balsup@cisco.com \
--cc=bhelgaas@google.com \
--cc=davem@davemloft.net \
--cc=kuba@kernel.org \
--cc=lguohan@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=msiddare@cisco.com \
--cc=netdev@vger.kernel.org \
--cc=pmenzel@molgen.mpg.de \
--cc=s.miroshnichenko@yadro.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).