Failure to HotPlug if devices not present at boot - no memory space

Failure to HotPlug if devices not present at boot - no memory space

[Barry Grussling]

Hello,

I am having some issues with PCIe hotplug on Linux.  Bottom
line is if PCIe endpoints are present and enabled on boot,
things work fine.  I can disable/enable endpoints and rescan
the PCI chain great and everything is found and loaded.

If PCIe endpoints are not present on boot and instead
are added later, I get "no space" and "failed to assign"
when I try to rescan the PCIe bus.

Booting with pci=realloc=on seems to have no effect.

I have verified I have CONFIG_PCI_REALLOC_ENABLE_AUTO=y
and CONFIG_PCI_IOV=y in my config.

The root complex is an ARM Cortex A9 SoC.  There is
a PLX PEX 8619 16-lane PCIe switch chip hooked up to
the root complex of the SoC.

Here is the enumeration from a device when all the PCIe
endpoints are present and enabled when the SoC boots:
pci 0000:00:00.0: BAR 8: assigned [mem 0xc0000000-0xc7dfffff]
pci 0000:01:00.0: BAR 8: assigned [mem 0xc0000000-0xc7bfffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0xc7c00000-0xc7c1ffff]
pci 0000:01:00.1: BAR 0: assigned [mem 0xc7c20000-0xc7c3ffff]
pci 0000:02:00.0: BAR 8: assigned [mem 0xc0000000-0xc27fffff]
pci 0000:02:02.0: BAR 8: assigned [mem 0xc2800000-0xc4ffffff]
pci 0000:02:0f.0: BAR 8: assigned [mem 0xc5000000-0xc77fffff]
pci 0000:02:03.0: BAR 8: assigned [mem 0xc7800000-0xc78fffff]
pci 0000:02:0b.0: BAR 8: assigned [mem 0xc7900000-0xc79fffff]
pci 0000:02:0d.0: BAR 8: assigned [mem 0xc7a00000-0xc7afffff]
pci 0000:03:00.0: BAR 1: assigned [mem 0xc0000000-0xc0ffffff]
pci 0000:03:00.0: BAR 2: assigned [mem 0xc1000000-0xc1ffffff]
pci 0000:03:00.0: BAR 0: assigned [mem 0xc2000000-0xc20fffff]
pci 0000:03:00.0: BAR 3: assigned [mem 0xc2100000-0xc21003ff]
pci 0000:03:00.0: BAR 4: assigned [mem 0xc2100400-0xc21007ff]
pci 0000:03:00.0: BAR 5: assigned [mem 0xc2100800-0xc2100bff]
pci 0000:02:00.0: PCI bridge to [bus 03]
pci 0000:02:00.0:   bridge window [mem 0xc0000000-0xc27fffff]
pci 0000:04:00.0: BAR 1: assigned [mem 0xc3000000-0xc3ffffff]
pci 0000:04:00.0: BAR 2: assigned [mem 0xc4000000-0xc4ffffff]
pci 0000:04:00.0: BAR 0: assigned [mem 0xc2800000-0xc28fffff]
pci 0000:04:00.0: BAR 3: assigned [mem 0xc2900000-0xc29003ff]
pci 0000:04:00.0: BAR 4: assigned [mem 0xc2900400-0xc29007ff]
pci 0000:04:00.0: BAR 5: assigned [mem 0xc2900800-0xc2900bff]
pci 0000:02:02.0: PCI bridge to [bus 04]
pci 0000:02:02.0:   bridge window [mem 0xc2800000-0xc4ffffff]
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:02:03.0:   bridge window [mem 0xc7800000-0xc78fffff]
pci 0000:02:0b.0: PCI bridge to [bus 06]
pci 0000:02:0b.0:   bridge window [mem 0xc7900000-0xc79fffff]
pci 0000:02:0d.0: PCI bridge to [bus 07]
pci 0000:02:0d.0:   bridge window [mem 0xc7a00000-0xc7afffff]
pci 0000:08:00.0: BAR 1: assigned [mem 0xc5000000-0xc5ffffff]
pci 0000:08:00.0: BAR 2: assigned [mem 0xc6000000-0xc6ffffff]
pci 0000:08:00.0: BAR 0: assigned [mem 0xc7000000-0xc70fffff]
pci 0000:08:00.0: BAR 3: assigned [mem 0xc7100000-0xc71003ff]
pci 0000:08:00.0: BAR 4: assigned [mem 0xc7100400-0xc71007ff]
pci 0000:08:00.0: BAR 5: assigned [mem 0xc7100800-0xc7100bff]
pci 0000:02:0f.0: PCI bridge to [bus 08]
pci 0000:02:0f.0:   bridge window [mem 0xc5000000-0xc77fffff]
pci 0000:01:00.0: PCI bridge to [bus 02-08]
pci 0000:01:00.0:   bridge window [mem 0xc0000000-0xc7bfffff]
pci 0000:00:00.0: PCI bridge to [bus 01-08]
pci 0000:00:00.0:   bridge window [mem 0xc0000000-0xc7dfffff]
pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
pci 0000:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:03:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
pci 0000:02:02.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:04:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
pci 0000:02:03.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:0b.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:0d.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:0f.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:08:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
pci 0000:01:00.1: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  256

In this case I can request a driver removal, power off the device,
power it back on, rescan the bus, and I get:
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:02:03.0:   bridge window [mem 0xc7800000-0xc78fffff]
pci 0000:02:0b.0: PCI bridge to [bus 06]
pci 0000:02:0b.0:   bridge window [mem 0xc7900000-0xc79fffff]
pci 0000:02:0d.0: PCI bridge to [bus 07]
pci 0000:02:0d.0:   bridge window [mem 0xc7a00000-0xc7afffff]
pci 0000:08:00.0: BAR 1: assigned [mem 0xc5000000-0xc5ffffff]
pci 0000:08:00.0: BAR 2: assigned [mem 0xc6000000-0xc6ffffff]
pci 0000:08:00.0: BAR 0: assigned [mem 0xc7000000-0xc70fffff]
pci 0000:08:00.0: BAR 3: assigned [mem 0xc7100000-0xc71003ff]
pci 0000:08:00.0: BAR 4: assigned [mem 0xc7100400-0xc71007ff]
pci 0000:08:00.0: BAR 5: assigned [mem 0xc7100800-0xc7100bff]
TestDrv 0000:08:00.0: Driver Loaded Successfully

Here is the boot enumeration when one endpoint (00:08:00.0) is
not present at SoC boot time:
pci 0000:00:00.0: BAR 8: assigned [mem 0xc0000000-0xc59fffff]
pci 0000:01:00.0: BAR 8: assigned [mem 0xc0000000-0xc57fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0xc5800000-0xc581ffff]
pci 0000:01:00.1: BAR 0: assigned [mem 0xc5820000-0xc583ffff]
pci 0000:02:00.0: BAR 8: assigned [mem 0xc0000000-0xc27fffff]
pci 0000:02:02.0: BAR 8: assigned [mem 0xc2800000-0xc4ffffff]
pci 0000:02:03.0: BAR 8: assigned [mem 0xc5000000-0xc50fffff]
pci 0000:02:0b.0: BAR 8: assigned [mem 0xc5100000-0xc51fffff]
pci 0000:02:0d.0: BAR 8: assigned [mem 0xc5200000-0xc52fffff]
pci 0000:02:0f.0: BAR 8: assigned [mem 0xc5300000-0xc53fffff]
pci 0000:03:00.0: BAR 1: assigned [mem 0xc0000000-0xc0ffffff]
pci 0000:03:00.0: BAR 2: assigned [mem 0xc1000000-0xc1ffffff]
pci 0000:03:00.0: BAR 0: assigned [mem 0xc2000000-0xc20fffff]
pci 0000:03:00.0: BAR 3: assigned [mem 0xc2100000-0xc21003ff]
pci 0000:03:00.0: BAR 4: assigned [mem 0xc2100400-0xc21007ff]
pci 0000:03:00.0: BAR 5: assigned [mem 0xc2100800-0xc2100bff]
pci 0000:02:00.0: PCI bridge to [bus 03]
pci 0000:02:00.0:   bridge window [mem 0xc0000000-0xc27fffff]
pci 0000:04:00.0: BAR 1: assigned [mem 0xc3000000-0xc3ffffff]
pci 0000:04:00.0: BAR 2: assigned [mem 0xc4000000-0xc4ffffff]
pci 0000:04:00.0: BAR 0: assigned [mem 0xc2800000-0xc28fffff]
pci 0000:04:00.0: BAR 3: assigned [mem 0xc2900000-0xc29003ff]
pci 0000:04:00.0: BAR 4: assigned [mem 0xc2900400-0xc29007ff]
pci 0000:04:00.0: BAR 5: assigned [mem 0xc2900800-0xc2900bff]
pci 0000:02:02.0: PCI bridge to [bus 04]
pci 0000:02:02.0:   bridge window [mem 0xc2800000-0xc4ffffff]
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:02:03.0:   bridge window [mem 0xc5000000-0xc50fffff]
pci 0000:02:0b.0: PCI bridge to [bus 06]
pci 0000:02:0b.0:   bridge window [mem 0xc5100000-0xc51fffff]
pci 0000:02:0d.0: PCI bridge to [bus 07]
pci 0000:02:0d.0:   bridge window [mem 0xc5200000-0xc52fffff]
pci 0000:02:0f.0: PCI bridge to [bus 08]
pci 0000:02:0f.0:   bridge window [mem 0xc5300000-0xc53fffff]
pci 0000:01:00.0: PCI bridge to [bus 02-08]
pci 0000:01:00.0:   bridge window [mem 0xc0000000-0xc57fffff]
pci 0000:00:00.0: PCI bridge to [bus 01-08]
pci 0000:00:00.0:   bridge window [mem 0xc0000000-0xc59fffff]
pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
pci 0000:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:03:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
pci 0000:02:02.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:04:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256
pci 0000:02:03.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:0b.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:0d.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:02:0f.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  128
pci 0000:01:00.1: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  256

If I then apply power to the endpoint at (00:08:00.0) and
attempt to scan the bus I get:
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:02:03.0:   bridge window [mem 0xc5000000-0xc50fffff]
pci 0000:02:0b.0: PCI bridge to [bus 06]
pci 0000:02:0b.0:   bridge window [mem 0xc5100000-0xc51fffff]
pci 0000:02:0d.0: PCI bridge to [bus 07]
pci 0000:02:0d.0:   bridge window [mem 0xc5200000-0xc52fffff]
pci 0000:08:00.0: BAR 1: no space for [mem size 0x01000000]
pci 0000:08:00.0: BAR 1: failed to assign [mem size 0x01000000]
pci 0000:08:00.0: BAR 2: no space for [mem size 0x01000000]
pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x01000000]
pci 0000:08:00.0: BAR 0: assigned [mem 0xc5300000-0xc53fffff]
pci 0000:08:00.0: BAR 3: no space for [mem size 0x00000400]
pci 0000:08:00.0: BAR 3: failed to assign [mem size 0x00000400]
pci 0000:08:00.0: BAR 4: no space for [mem size 0x00000400]
pci 0000:08:00.0: BAR 4: failed to assign [mem size 0x00000400]
pci 0000:08:00.0: BAR 5: no space for [mem size 0x00000400]
pci 0000:08:00.0: BAR 5: failed to assign [mem size 0x00000400]
TestDrv 0000:08:00.0: Unable to Map Bar 2!

Note the working system allocates 0xc0000000 to 0xc7dfffff to
the PCI subsystem but the failing boot only allocates 0xc0000000
to 0xc59fffff.

My Google-Fu turned up some posts about pci=realloc being needed.
I have booted this unit with and without that kernel parameter
and I can't see any difference (the diff output of my
commands is the same).  I am currently booting the unit
with "pci=realloc=on,pcie_bus_perf".

I did some reading in the kernel-parameters.txt and tried
booting with "pci=realloc=on,pcie_bus_perf,hpmemsize=256M"
hoping that would allow for a large PCI space but it
didn't seem to have any effect.

The kernel on the device is currently 3.17.  In an effort
to see if this was fixed in a future kernel I cherry-picked
all the changes to drivers/pci/bus.c up to and including
3a9ad0b PCI: Add pci_bus_addr_t

but that resulted in no difference.

I am rappidly crashing into the end of my PCI Express on Linux
knowledge.  Any help or guidance would be greatly appreciated.

Thanks,

Barry

Re: Failure to HotPlug if devices not present at boot - no memory space

[Bjorn Helgaas]

Hi Barry,

On Mon, Aug 3, 2015 at 10:28 AM, Barry Grussling <barry@grussling.com> wrote:
> Hello,
>
> I am having some issues with PCIe hotplug on Linux.  Bottom
> line is if PCIe endpoints are present and enabled on boot,
> things work fine.  I can disable/enable endpoints and rescan
> the PCI chain great and everything is found and loaded.
>
> If PCIe endpoints are not present on boot and instead
> are added later, I get "no space" and "failed to assign"
> when I try to rescan the PCIe bus.
>
> Booting with pci=realloc=on seems to have no effect.
>
> I have verified I have CONFIG_PCI_REALLOC_ENABLE_AUTO=y
> and CONFIG_PCI_IOV=y in my config.
>
> The root complex is an ARM Cortex A9 SoC.  There is
> a PLX PEX 8619 16-lane PCIe switch chip hooked up to
> the root complex of the SoC.

Can you attach complete dmesg logs and "lspci -vv" output for both
cases?  (You can attach them to a kernel.org bugzilla if they're too
big for email.)

> ...
> Note the working system allocates 0xc0000000 to 0xc7dfffff to
> the PCI subsystem but the failing boot only allocates 0xc0000000
> to 0xc59fffff.

The dmesg logs will tell us what space is allocated to the host
bridge.  That is generally done by firmware or the bootloader, and
Linux doesn't change it.  So if firmware only assigns 0xc0000000 to
0xc59fffff to the host bridge, the current Linux kernel will only
assign PCI devices inside that region; it will never expand the
region.

Bjorn

Re: Failure to HotPlug if devices not present at boot - no memory space

[Barry Grussling]

> Can you attach complete dmesg logs and "lspci -vv" output for both
> cases?  (You can attach them to a kernel.org bugzilla if they're too
> big for email.)
>

Can do:
https://bugzilla.kernel.org/show_bug.cgi?id=102231

> The dmesg logs will tell us what space is allocated to the host
> bridge.  That is generally done by firmware or the bootloader, and
> Linux doesn't change it.  So if firmware only assigns 0xc0000000 to
> 0xc59fffff to the host bridge, the current Linux kernel will only
> assign PCI devices inside that region; it will never expand the
> region.
Okay.  That makes sense.

We are using u-boot.  I don't believe our current u-boot build has any
PCIe knowledge in it though.

Our SoC/PCIe root complex is the Altera Cyclone V:
https://www.altera.com/en_US/pdfs/literature/ug/ug_c5_pcie_avmm.pdf

Thanks for your help!

Barry

Re: Failure to HotPlug if devices not present at boot - no memory space

[Bjorn Helgaas]

[+cc Yinghai]

On Mon, Aug 03, 2015 at 01:33:20PM -0700, Barry Grussling wrote:
> > Can you attach complete dmesg logs and "lspci -vv" output for both
> > cases?  (You can attach them to a kernel.org bugzilla if they're too
> > big for email.)
> >
> 
> Can do:
> https://bugzilla.kernel.org/show_bug.cgi?id=102231
> 
> > The dmesg logs will tell us what space is allocated to the host
> > bridge.  That is generally done by firmware or the bootloader, and
> > Linux doesn't change it.  So if firmware only assigns 0xc0000000 to
> > 0xc59fffff to the host bridge, the current Linux kernel will only
> > assign PCI devices inside that region; it will never expand the
> > region.
> Okay.  That makes sense.
> 
> We are using u-boot.  I don't believe our current u-boot build has any
> PCIe knowledge in it though.

You might have a host bridge driver that knows how to set up the
windows.  Your dmesg logs mention altera_hps2fpga_bridge, which
doesn't seem to be in the upstream kernel, so I don't know what it
does.

Both logs show this:

  PCI host bridge to bus 0000:00
  pci_bus 0000:00: root bus resource [mem 0xc0000000-0xcfffffff]
  pci_bus 0000:00: root bus resource [mem 0xd0000000-0xdfffffff pref]

which means the top-level apertures are the same regardless of whether
08:00.0 is present at boot.  That means the PCI core should be able to
assign space for the hot-added device, at least in principle.

The log where 08:00.0 is present at boot shows this:

  pci 0000:08:00.0: [1aa9:0013] type 00 class 0x020000
  pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000fffff] (size  1M)
  pci 0000:08:00.0: reg 0x14: [mem 0x00000000-0x00ffffff] (size 16M)
  pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0x00ffffff] (size 16M)
  pci 0000:08:00.0: reg 0x1c: [mem 0x00000000-0x000003ff] (size  1K)
  pci 0000:08:00.0: reg 0x20: [mem 0x00000000-0x000003ff] (size  1K)
  pci 0000:08:00.0: reg 0x24: [mem 0x00000000-0x000003ff] (size  1K)

so it needs a little over 33M of space.

In the other log ("08:00.0 not present at boot"), we actually *do*
enumerate an 08:00.0 device:

  pci 0000:08:00.0: [1d00:0000] type 00 class 0x00001d
  pci 0000:08:00.0: reg 0x30: [mem 0x00001800-0x00001fff pref]

I don't know what this device is (lspci says "non-VGA unclassified
device") and it only requests 2K of space for an option ROM.  And the
log doesn't include the part where you hot-add the real device --
that's the interesting information should be.

I see that you already boot with "pcie=hpmemsize=256M", but that
doesn't help in this case because it's only used for hotplug bridges,
and 02:0f.0 (the Downstream Port leading to bus 08) is not marked as
hotplug-capable:

  02:0f.0 PCI bridge: PLX Technology, Inc. PEX 8619
    Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
      SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-

I think if you could get the PCI_EXP_SLTCAP_HPC bit set for 02:0f.0,
and you booted with "pcie=hpmemsize=256M", it would likely work.  It's
possible it could be made to work even without that, but in general
it's hard to reserve the right amount of space for a bridge if you
don't know what might be plugged in later.

Bjorn

Re: Failure to HotPlug if devices not present at boot - no memory space

[Barry Grussling]

On Mon, Aug 3, 2015 at 3:27 PM, Bjorn Helgaas <bhelgaas@google.com> wrote:
>
> You might have a host bridge driver that knows how to set up the
> windows.  Your dmesg logs mention altera_hps2fpga_bridge, which
> doesn't seem to be in the upstream kernel, so I don't know what it
> does.

Sorry.  Forgot to tell you it is based on the driver available at:
http://rocketboards.org/foswiki/view/Projects/PCIeRootPortWithMSI

I figured/hoped it was a general PCIe lack of understanding on
my part and not something specifically related to the PCIe root complex
driver.

> In the other log ("08:00.0 not present at boot"), we actually *do*
> enumerate an 08:00.0 device:
>
>   pci 0000:08:00.0: [1d00:0000] type 00 class 0x00001d
>   pci 0000:08:00.0: reg 0x30: [mem 0x00001800-0x00001fff pref]

Yea.  Not sure what that is since there is no device physically present.
PLX must be reserving the spot or something.

> I don't know what this device is (lspci says "non-VGA unclassified
> device") and it only requests 2K of space for an option ROM.  And the
> log doesn't include the part where you hot-add the real device --
> that's the interesting information should be.

I added the dmesg for enumeration after hot-adding the device
to the bugzilla entry.  Don't want to omit the interesting
part.

> I see that you already boot with "pcie=hpmemsize=256M", but that
> doesn't help in this case because it's only used for hotplug bridges,
> and 02:0f.0 (the Downstream Port leading to bus 08) is not marked as
> hotplug-capable:
>
>   02:0f.0 PCI bridge: PLX Technology, Inc. PEX 8619
>     Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
>       SltCap:   AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
>
> I think if you could get the PCI_EXP_SLTCAP_HPC bit set for 02:0f.0,
> and you booted with "pcie=hpmemsize=256M", it would likely work.  It's
> possible it could be made to work even without that, but in general
> it's hard to reserve the right amount of space for a bridge if you
> don't know what might be plugged in later.

This makes sense to me.  I think I have enough information now to
try to massage the PLX port into being a hotplug port and maybe
everything will work.

Thanks for yor help Bjorn!  I have something to try now.

Barry

Re: Failure to HotPlug if devices not present at boot - no memory space

[Yinghai Lu]

On Mon, Aug 3, 2015 at 4:31 PM, Barry Grussling <barry@grussling.com> wrote:
> On Mon, Aug 3, 2015 at 3:27 PM, Bjorn Helgaas <bhelgaas@google.com> wrote:
>>
>> You might have a host bridge driver that knows how to set up the
>> windows.  Your dmesg logs mention altera_hps2fpga_bridge, which
>> doesn't seem to be in the upstream kernel, so I don't know what it
>> does.

Please try to use

echo 1 > /sys/devices/pci0000:00/000:02:0f.0/pci_bus/0000:08/rescan

it will increase mmio range for 02:0f.0.

If it does not work, you may need to try
git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
for-pci-v4.3-next

as it include one fix

PCI: Treat optional as must in first try for bridge rescan

https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/patch/?id=f83b7ff4676da783f4615cb5e60102cd5181eee9

Also you turn off the power between the removal and rescan, you may need to
boot with "pci=pcie_bus_perf" and use following patch that configure
the MPSS to make it consistent between bridge and device.

Subject: [PATCH] PCI: Configure MPS during rescan bus

When we disable link and renable link of pcie downstream port, MPSS of device
under the bridge could reset to 128.

Just call pcie_bus_configure_settings just like boot path and hotplug
path.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 drivers/pci/probe.c |   11 +++++++++++
 1 file changed, 11 insertions(+)

Index: linux-2.6/drivers/pci/probe.c
===================================================================
--- linux-2.6.orig/drivers/pci/probe.c
+++ linux-2.6/drivers/pci/probe.c
@@ -2409,6 +2409,8 @@ unsigned int pci_rescan_bus_bridge_resiz

     pci_assign_unassigned_bridge_resources(bridge);

+    pcie_bus_configure_settings(bus);
+
     pci_bus_add_devices(bus);

     return max;
@@ -2429,6 +2431,15 @@ unsigned int pci_rescan_bus(struct pci_b

     max = pci_scan_child_bus(bus);
     pci_assign_unassigned_bus_resources(bus);
+
+    if (pci_is_root_bus(bus)) {
+        struct pci_bus *child;
+
+                list_for_each_entry(child, &bus->children, node)
+            pcie_bus_configure_settings(child);
+    } else
+        pcie_bus_configure_settings(bus);
+
     pci_bus_add_devices(bus);

     return max;