Debugging incorrect mps settings detected

Debugging incorrect mps settings detected

[Keith Busch]

Hello,

Hot adding a pci device results in improper MPS settings detected,
reading 128 even though the upstream port is 256. This causes a warning
print to be logged suggesting I should report a bug. :)

Here's the dmesg of the hot plug event:

[ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27)
[ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802
[ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[ 2107.014986] pcieport 0000:05:07.0: bridge window [io  0x1000-0x0fff] to [bus 09] add_size 1000
[ 2107.014996] pcieport 0000:05:07.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
[ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
[ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff pref]
[ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff 64bit]
[ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09]
[ 2107.015037] pcieport 0000:05:07.0:   bridge window [mem 0xdb600000-0xdb7fffff]
[ 2107.015042] pcieport 0000:05:07.0:   bridge window [mem 0xd5200000-0xd52fffff 64bit pref]
[ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a bug
[ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform

The commit message that added the warning indicates it may be a BIOS
issue that incorrectly configures this, but BIOS isn't involved with
setting the MPS on a device hot add, am I right? Or is that wrong?
This is a Dell R620 server, if that matters, though we see this on other
platforms as well.

Using pci=pcie_bus_perf works around the issue, but that seems a bit
odd to suggest to users. Is setting MPS on hot-add something the PCI
driver layer ought to be doing, or should I be looking into the platform
BIOS instead?

Thanks!
Keith

Re: Debugging incorrect mps settings detected

[Bjorn Helgaas]

[+cc Yijing, Jon]

On Tue, Jun 3, 2014 at 2:04 PM, Keith Busch <keith.busch@intel.com> wrote:
> Hello,
>
> Hot adding a pci device results in improper MPS settings detected,
> reading 128 even though the upstream port is 256. This causes a warning
> print to be logged suggesting I should report a bug. :)
>
> Here's the dmesg of the hot plug event:
>
> [ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27)
> [ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802
> [ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
> [ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
> [ 2107.014986] pcieport 0000:05:07.0: bridge window [io  0x1000-0x0fff] to
> [bus 09] add_size 1000
> [ 2107.014996] pcieport 0000:05:07.0: res[13]=[io  0x1000-0x0fff]
> get_res_add_size add_size 1000
> [ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
> [ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
> [ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff
> pref]
> [ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff
> 64bit]
> [ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09]
> [ 2107.015037] pcieport 0000:05:07.0:   bridge window [mem
> 0xdb600000-0xdb7fffff]
> [ 2107.015042] pcieport 0000:05:07.0:   bridge window [mem
> 0xd5200000-0xd52fffff 64bit pref]
> [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream
> 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a
> bug
> [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform
>
> The commit message that added the warning indicates it may be a BIOS
> issue that incorrectly configures this, but BIOS isn't involved with
> setting the MPS on a device hot add, am I right? Or is that wrong?

The BIOS is not involved in setting MPS on a device hot-add.

> This is a Dell R620 server, if that matters, though we see this on other
> platforms as well.
>
> Using pci=pcie_bus_perf works around the issue, but that seems a bit
> odd to suggest to users. Is setting MPS on hot-add something the PCI
> driver layer ought to be doing, or should I be looking into the platform
> BIOS instead?

The PCI core should be doing something intelligent here.  If we can't
configure MPS safely, we should prevent the device from being enabled.
 But this is stuff the PCI core doesn't implement yet.

It's hard to reconfigure MPS after boot because it affects other
devices in the system, and I don't think we can always change the
settings of other devices while they're active.

I'm not sure that warning is correct.  It seems like it should suggest
pcie_bus_peer2peer, which is documented as guaranteeing that hot-added
devices will work.  But this is always confusing to me, and I could be
wrong about this.

Bjorn

Re: Debugging incorrect mps settings detected

[Yijing Wang]

accurate>> 0xd5200000-0xd52fffff 64bit pref]
>> [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream
>> 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a
>> bug
>> [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform
>>
>> The commit message that added the warning indicates it may be a BIOS
>> issue that incorrectly configures this, but BIOS isn't involved with
>> setting the MPS on a device hot add, am I right? Or is that wrong?
> 
> The BIOS is not involved in setting MPS on a device hot-add.
> 
>> This is a Dell R620 server, if that matters, though we see this on other
>> platforms as well.
>>
>> Using pci=pcie_bus_perf works around the issue, but that seems a bit
>> odd to suggest to users. Is setting MPS on hot-add something the PCI
>> driver layer ought to be doing, or should I be looking into the platform
>> BIOS instead?
> 
> The PCI core should be doing something intelligent here.  If we can't
> configure MPS safely, we should prevent the device from being enabled.
>  But this is stuff the PCI core doesn't implement yet.

Hi Bjorn,
   We had a long discussion about fix this issue.
Link: http://marc.info/?l=linux-pci&m=137714192204947&w=2

Now more BIOS configure device mps larger than 128B, So I think
we should consider to fix it before we have a perfect solution about mps setting.

> 
> It's hard to reconfigure MPS after boot because it affects other
> devices in the system, and I don't think we can always change the
> settings of other devices while they're active.

Yes, what we can only change is the device not active after hotplug.


> 
> I'm not sure that warning is correct.  It seems like it should suggest
> pcie_bus_peer2peer, which is documented as guaranteeing that hot-added
> devices will work.  But this is always confusing to me, and I could be
> wrong about this.

I think warning is proper, pcie_bus_safe is designed to fix the hotlplug issue.
pcie_bus_peer2perr guarantee that hotplug device work fine, because all mps setting
128B. Maybe we should change document a little, make it more clear.


> 
> Bjorn
> 
> 


-- 
Thanks!
Yijing

Re: Debugging incorrect mps settings detected

[Jon Mason]

On Tue, Jun 03, 2014 at 03:01:31PM -0600, Bjorn Helgaas wrote:
> [+cc Yijing, Jon]
> 
> On Tue, Jun 3, 2014 at 2:04 PM, Keith Busch <keith.busch@intel.com> wrote:
> > Hello,
> >
> > Hot adding a pci device results in improper MPS settings detected,
> > reading 128 even though the upstream port is 256. This causes a warning
> > print to be logged suggesting I should report a bug. :)
> >
> > Here's the dmesg of the hot plug event:
> >
> > [ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27)
> > [ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802
> > [ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
> > [ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
> > [ 2107.014986] pcieport 0000:05:07.0: bridge window [io  0x1000-0x0fff] to
> > [bus 09] add_size 1000
> > [ 2107.014996] pcieport 0000:05:07.0: res[13]=[io  0x1000-0x0fff]
> > get_res_add_size add_size 1000
> > [ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
> > [ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
> > [ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff
> > pref]
> > [ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff
> > 64bit]
> > [ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09]
> > [ 2107.015037] pcieport 0000:05:07.0:   bridge window [mem
> > 0xdb600000-0xdb7fffff]
> > [ 2107.015042] pcieport 0000:05:07.0:   bridge window [mem
> > 0xd5200000-0xd52fffff 64bit pref]
> > [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream
> > 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a
> > bug
> > [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform
> >
> > The commit message that added the warning indicates it may be a BIOS
> > issue that incorrectly configures this, but BIOS isn't involved with
> > setting the MPS on a device hot add, am I right? Or is that wrong?
> 
> The BIOS is not involved in setting MPS on a device hot-add.
> 
> > This is a Dell R620 server, if that matters, though we see this on other
> > platforms as well.
> >
> > Using pci=pcie_bus_perf works around the issue, but that seems a bit
> > odd to suggest to users. Is setting MPS on hot-add something the PCI
> > driver layer ought to be doing, or should I be looking into the platform
> > BIOS instead?
> 
> The PCI core should be doing something intelligent here.  If we can't
> configure MPS safely, we should prevent the device from being enabled.
>  But this is stuff the PCI core doesn't implement yet.

The tuning of the MPS is disabled by default, which is why the warning
is there.  Now, we can attempt to tune it regardless of whether tuning
is enabled (assuming this device is the only one under the root port),
we can fail to all the hotplug, or we can enable "safe" tuning by
default.  The first option seems a bit hacky.  The second will fail
too often, since the MPSS of the hotplugged device would have to match
the MPS setting of slot/fabric.  The third seems like the best option.

> It's hard to reconfigure MPS after boot because it affects other
> devices in the system, and I don't think we can always change the
> settings of other devices while they're active.

Unless the slot is under its own root port, otherwise the MPS will
have to be floored.  This is the behavior in the "safe" tuning.

I suggest that the "safe" tuning get made the default behavior and we
let it soak for a little while to shake out any issues that there may
be.

Thanks,
Jon

> I'm not sure that warning is correct.  It seems like it should suggest
> pcie_bus_peer2peer, which is documented as guaranteeing that hot-added
> devices will work.  But this is always confusing to me, and I could be
> wrong about this.
> 
> Bjorn

Re: Debugging incorrect mps settings detected

[Keith Busch]

On Wed, 4 Jun 2014, Jon Mason wrote:
> On Tue, Jun 03, 2014 at 03:01:31PM -0600, Bjorn Helgaas wrote:
>> [+cc Yijing, Jon]
>> The PCI core should be doing something intelligent here.  If we can't
>> configure MPS safely, we should prevent the device from being enabled.
>>  But this is stuff the PCI core doesn't implement yet.
>
> The tuning of the MPS is disabled by default, which is why the warning
> is there.  Now, we can attempt to tune it regardless of whether tuning
> is enabled (assuming this device is the only one under the root port),
> we can fail to all the hotplug, or we can enable "safe" tuning by
> default.  The first option seems a bit hacky.  The second will fail
> too often, since the MPSS of the hotplugged device would have to match
> the MPS setting of slot/fabric.  The third seems like the best option.
>
>> It's hard to reconfigure MPS after boot because it affects other
>> devices in the system, and I don't think we can always change the
>> settings of other devices while they're active.
>
> Unless the slot is under its own root port, otherwise the MPS will
> have to be floored.  This is the behavior in the "safe" tuning.
>
> I suggest that the "safe" tuning get made the default behavior and we
> let it soak for a little while to shake out any issues that there may
> be.

Thanks for all the info. I've tried different tunings on several different
platforms that were all failing before. The 'safe' tuning was the only
one successful on all platforms, so just providing a data point that
this appears to be a good direction to go.

RE: Debugging incorrect mps settings detected

[Jordan_Hargrave]

We are still seeing issues too with the maxpayload default setting being incorrect for hotplug PCIE systems on Linux.

The default setting doesn't work if any root port is configured with MPS>128.  Adding in a hotplug device will default to MPS=128 and will cause PCIE error.
Clamping to pci_bus_safe causes the whole system to run inefficiently at MPS=128

Windows and VMWare are already using the equivalent of pcie_bus_perf and not seeing problems. 

--jordan hargrave
Dell Enterprise Linux Engineering
________________________________________
From: linux-pci-owner@vger.kernel.org [linux-pci-owner@vger.kernel.org] On Behalf Of Bjorn Helgaas [bhelgaas@google.com]
Sent: Tuesday, June 03, 2014 4:01 PM
To: Keith Busch
Cc: linux-pci@vger.kernel.org; Yijing Wang; Jon Mason
Subject: Re: Debugging incorrect mps settings detected

[+cc Yijing, Jon]

On Tue, Jun 3, 2014 at 2:04 PM, Keith Busch <keith.busch@intel.com> wrote:
> Hello,
>
> Hot adding a pci device results in improper MPS settings detected,
> reading 128 even though the upstream port is 256. This causes a warning
> print to be logged suggesting I should report a bug. :)
>
> Here's the dmesg of the hot plug event:
>
> [ 2105.051941] pciehp 0000:05:07.0:pcie24: Card present on Slot(27)
> [ 2107.012811] pci 0000:09:00.0: [8086:0953] type 00 class 0x010802
> [ 2107.012838] pci 0000:09:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
> [ 2107.012873] pci 0000:09:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
> [ 2107.014986] pcieport 0000:05:07.0: bridge window [io  0x1000-0x0fff] to
> [bus 09] add_size 1000
> [ 2107.014996] pcieport 0000:05:07.0: res[13]=[io  0x1000-0x0fff]
> get_res_add_size add_size 1000
> [ 2107.015001] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
> [ 2107.015006] pcieport 0000:05:07.0: BAR 13: can't assign io (size 0x1000)
> [ 2107.015013] pci 0000:09:00.0: BAR 6: assigned [mem 0xd5200000-0xd520ffff
> pref]
> [ 2107.015018] pci 0000:09:00.0: BAR 0: assigned [mem 0xdb600000-0xdb603fff
> 64bit]
> [ 2107.015030] pcieport 0000:05:07.0: PCI bridge to [bus 09]
> [ 2107.015037] pcieport 0000:05:07.0:   bridge window [mem
> 0xdb600000-0xdb7fffff]
> [ 2107.015042] pcieport 0000:05:07.0:   bridge window [mem
> 0xd5200000-0xd52fffff 64bit pref]
> [ 2107.015060] pci 0000:09:00.0: Max Payload Size 128, but upstream
> 0000:05:07.0 set to 256; if necessary, use "pci=pcie_bus_safe" and report a
> bug
> [ 2107.015079] pci 0000:09:00.0: no hotplug settings from platform
>
> The commit message that added the warning indicates it may be a BIOS
> issue that incorrectly configures this, but BIOS isn't involved with
> setting the MPS on a device hot add, am I right? Or is that wrong?

The BIOS is not involved in setting MPS on a device hot-add.

> This is a Dell R620 server, if that matters, though we see this on other
> platforms as well.
>
> Using pci=pcie_bus_perf works around the issue, but that seems a bit
> odd to suggest to users. Is setting MPS on hot-add something the PCI
> driver layer ought to be doing, or should I be looking into the platform
> BIOS instead?

The PCI core should be doing something intelligent here.  If we can't
configure MPS safely, we should prevent the device from being enabled.
 But this is stuff the PCI core doesn't implement yet.

It's hard to reconfigure MPS after boot because it affects other
devices in the system, and I don't think we can always change the
settings of other devices while they're active.

I'm not sure that warning is correct.  It seems like it should suggest
pcie_bus_peer2peer, which is documented as guaranteeing that hot-added
devices will work.  But this is always confusing to me, and I could be
wrong about this.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html