From: Bjorn Helgaas <helgaas@kernel.org>
To: Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
Cc: linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
linux@yadro.com, Sam Bobroff <sbobroff@linux.ibm.com>,
Rajat Jain <rajatja@google.com>, Lukas Wunner <lukas@wunner.de>,
Oliver O'Halloran <oohall@gmail.com>,
David Laight <David.Laight@ACULAB.COM>
Subject: Re: [PATCH v5 03/23] PCI: hotplug: Add a flag for the movable BARs feature
Date: Tue, 15 Oct 2019 17:14:49 -0500 [thread overview]
Message-ID: <20191015221449.GA181069@google.com> (raw)
In-Reply-To: <92cf42bf-5044-329d-f1be-53b48801865f@yadro.com>
On Mon, Sep 30, 2019 at 03:59:25PM +0300, Sergey Miroshnichenko wrote:
> Hello Bjorn,
>
> On 9/28/19 1:02 AM, Bjorn Helgaas wrote:
> > On Fri, Aug 16, 2019 at 07:50:41PM +0300, Sergey Miroshnichenko wrote:
> > > When hot-adding a device, the bridge may have windows not big enough (or
> > > fragmented too much) for newly requested BARs to fit in. And expanding
> > > these bridge windows may be impossible because blocked by "neighboring"
> > > BARs and bridge windows.
> > >
> > > Still, it may be possible to allocate a memory region for new BARs with the
> > > following procedure:
> > >
> > > 1) notify all the drivers which support movable BARs to pause and release
> > > the BARs; the rest of the drivers are guaranteed that their devices will
> > > not get BARs moved;
> > >
> > > 2) release all the bridge windows except of root bridges;
> > >
> > > 3) try to recalculate new bridge windows that will fit all the BAR types:
> > > - fixed;
> > > - immovable;
> > > - movable;
> > > - newly requested by hot-added devices;
> > >
> > > 4) if the previous step fails, disable BARs for one of the hot-added
> > > devices and retry from step 3;
> > >
> > > 5) notify the drivers, so they remap BARs and resume.
> >
> > You don't do the actual recalculation in *this* patch, but since you
> > mention the procedure here, are we confident that we never make things
> > worse?
> >
> > It's possible that a hot-add will trigger this attempt to move things
> > around, and it's possible that we won't find space for the new device
> > even if we move things around. But are we certain that every device
> > that worked *before* the hot-add will still work *afterwards*?
> >
> > Much of the assignment was probably done by the BIOS using different
> > algorithms than Linux has, so I think there's some chance that the
> > BIOS did a better job and if we lose that BIOS assignment, we might
> > not be able to recreate it.
>
> If a hardware has some special constraints on BAR assignment that the
> kernel is not aware of yet, the movable BARs may break things after a
> hotplug event. So the feature must be disabled there (manually) until
> the kernel get support for that special needs.
I'm not talking about special constraints on BAR assignment. (I'm not
sure what those constraints would be -- AFAIK the constraints for a
spec-compliant device are all discoverable via the BAR size and type
(or the Enhanced Allocation capability)).
What I'm concerned about is the case where we boot with a working
assignment, we hot-add a device, we move things around to try to
accommodate the new device, and not only do we fail to find resources
for the new device, we also fail to find a working assignment for the
devices that were present at boot. We've moved things around from
what BIOS did, and since we use a different algorithm than the BIOS,
there's no guarantee that we'll be able to find the assignment BIOS
did.
> > I'm not sure why the PCI_CLASS_DISPLAY_VGA special case is there; can
> > you add a comment about why that's needed? Obviously we can't move
> > the 0xa0000 legacy frame buffer because I think devices are allowed to
> > claim that region even if no BAR describes it. But I would think
> > *other* BARs of VGA devices could be movable.
>
> Sure, I'll add a comment to the code.
>
> The issue that we are avoiding by that is the "nomodeset" command line
> argument, which prevents a video driver from being bound, so the BARs
> are seems to be used, but can't be moved, otherwise machines just hang
> after hotplug events. That was the only special ugly case we've
> spotted during testing. I'll check if it will be enough just to work
> around the 0xa0000.
"nomodeset" is not really documented and is a funny way to say "don't
bind video drivers that know about it", but OK. Thanks for checking
on the other BARs.
> > > +bool pci_movable_bars_enabled(void);
> >
> > I would really like it if this were simply
> >
> > extern bool pci_no_movable_bars;
> >
> > in drivers/pci/pci.h. It would default to false since it's
> > uninitialized, and "pci=no_movable_bars" would set it to true.
>
> I have a premonition of platforms that will not support the feature.
> Wouldn't be better to put this variable-flag to include/linux/pci.h ,
> so code in arch/* can set it, so they could work by default, without
> the command line argument?
In general I don't see why a platform wouldn't support this since
there really isn't anything platform-specific here. But if a platform
does need to disable it, having arch code set this flag sounds
reasonable. We shouldn't make it globally visible until we actually
need that, though.
> > We have similar "=off" and "=force" parameters for ASPM and other
> > things, and it makes the code really hard to analyze.
The "=off" and "=force" things are the biggest things I'd like to
avoid.
Bjorn
next prev parent reply other threads:[~2019-10-15 22:15 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-16 16:50 [PATCH v5 00/23] PCI: Allow BAR movement during hotplug Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 01/23] PCI: Fix race condition in pci_enable/disable_device() Sergey Miroshnichenko
2019-08-22 12:37 ` Marta Rybczynska
2019-09-27 21:59 ` Bjorn Helgaas
2019-09-30 8:53 ` Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 02/23] PCI: Enable bridge's I/O and MEM access for hotplugged devices Sergey Miroshnichenko
2019-09-27 22:01 ` Bjorn Helgaas
2019-08-16 16:50 ` [PATCH v5 03/23] PCI: hotplug: Add a flag for the movable BARs feature Sergey Miroshnichenko
2019-09-27 22:02 ` Bjorn Helgaas
2019-09-30 8:44 ` David Laight
2019-09-30 16:17 ` Sergey Miroshnichenko
2019-09-30 12:59 ` Sergey Miroshnichenko
2019-10-15 22:14 ` Bjorn Helgaas [this message]
2019-10-16 15:50 ` Sergey Miroshnichenko
2019-10-16 17:29 ` Bjorn Helgaas
2019-08-16 16:50 ` [PATCH v5 04/23] PCI: Define PCI-specific version of the release_child_resources() Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 05/23] PCI: hotplug: movable BARs: Fix reassigning the released bridge windows Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 06/23] PCI: hotplug: movable BARs: Recalculate all bridge windows during rescan Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 07/23] PCI: hotplug: movable BARs: Don't allow added devices to steal resources Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 08/23] PCI: Include fixed and immovable BARs into the bus size calculating Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 09/23] PCI: Prohibit assigning BARs and bridge windows to non-direct parents Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 10/23] PCI: hotplug: movable BARs: Try to assign unassigned resources only once Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 11/23] PCI: hotplug: movable BARs: Calculate immovable parts of bridge windows Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 12/23] PCI: hotplug: movable BARs: Compute limits for relocated " Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 13/23] PCI: Make sure bridge windows include their fixed BARs Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 14/23] PCI: Fix assigning the fixed prefetchable resources Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 15/23] PCI: hotplug: movable BARs: Assign fixed and immovable BARs before others Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 16/23] PCI: hotplug: movable BARs: Don't reserve IO/mem bus space Sergey Miroshnichenko
2019-09-04 5:42 ` Oliver O'Halloran
2019-09-04 11:22 ` Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 17/23] powerpc/pci: Fix crash with enabled movable BARs Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 18/23] powerpc/pci: Handle BAR movement Sergey Miroshnichenko
2019-09-04 5:37 ` Oliver O'Halloran
2019-09-06 16:24 ` Sergey Miroshnichenko
2019-09-09 14:02 ` Oliver O'Halloran
2019-08-16 16:50 ` [PATCH v5 19/23] PCI: hotplug: Configure MPS for hot-added bridges during bus rescan Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 20/23] PCI: hotplug: movable BARs: Enable the feature by default Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 21/23] nvme-pci: Handle movable BARs Sergey Miroshnichenko
2019-08-16 16:51 ` [PATCH v5 22/23] PCI/portdrv: Declare support of " Sergey Miroshnichenko
2019-08-16 16:51 ` [PATCH v5 23/23] PCI: pciehp: movable BARs: Trigger a domain rescan on hp events Sergey Miroshnichenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191015221449.GA181069@google.com \
--to=helgaas@kernel.org \
--cc=David.Laight@ACULAB.COM \
--cc=linux-pci@vger.kernel.org \
--cc=linux@yadro.com \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=lukas@wunner.de \
--cc=oohall@gmail.com \
--cc=rajatja@google.com \
--cc=s.miroshnichenko@yadro.com \
--cc=sbobroff@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).