PCI rescan : can't assign mem

PCI rescan : can't assign mem

[Peio]

I have a board with an PCI Express-to-PCI Bridge (Texas Instruments 
XIO2001).
Attached to the PCI bus, there are 2 FPGA PCI Target devices needing 32MB 
memory map each.
The PCI rescan "echo 1 > /sys/bus/pci/rescan" memory assign fails.

dmesg messages:
[  150.978654] pci 0000:02:03.0: [4150:0001] type 00 class 0x068000
[  150.978684] pci 0000:02:03.0: reg 0x10: [mem 0x00000000-0x01ffffff]
[  150.978885] pci 0000:02:04.0: [4150:000a] type 00 class 0x078000
[  150.978909] pci 0000:02:04.0: reg 0x10: [mem 0x00000000-0x01ffffff]
[  150.979158] pcieport 0000:00:1c.0: bridge window [mem 0x00100000-
0x000fffff 64bit pref] to [bus 01-02] add_size 200000
[  150.979166] pcieport 0000:00:1c.0: bridge window [mem 0x02000000-
0x05ffffff] to [bus 01-02] add_size 2000000
[  150.979183] pcieport 0000:00:1c.2: bridge window [mem 0x00100000-
0x000fffff 64bit pref] to [bus 04] add_size 200000
[  150.979190] i915 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has 
bogus alignment
[  150.979199] pcieport 0000:00:1c.0: res[14]=[mem 0x02000000-0x05ffffff] 
get_res_add_size add_size 2000000
[  150.979205] pcieport 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff 
64bit pref] get_res_add_size add_size 200000
[  150.979211] pcieport 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff 
64bit pref] get_res_add_size add_size 200000
[  150.979222] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size 
0x6000000)
[  150.979230] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979238] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979245] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size 
0x4000000)
[  150.979253] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979260] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size 
0x200000)
[  150.979266] pci 0000:01:00.0: BAR 14: can't assign mem (size 0x4000000)
[  150.979272] pci 0000:02:03.0: BAR 0: can't assign mem (size 0x2000000)
[  150.979277] pci 0000:02:04.0: BAR 0: can't assign mem (size 0x2000000)
[  150.979281] pci 0000:01:00.0: PCI bridge to [bus 02]

lspci:
00:00.0 Host bridge: Intel Corporation ValleyView SSA-CUnit (rev 11)
00:02.0 VGA compatible controller: Intel Corporation ValleyView Gen7 (rev 
11)
00:1a.0 Encryption controller: Intel Corporation ValleyView SEC (rev 11)
00:1b.0 Audio device: Intel Corporation ValleyView High Definition Audio 
Controller (rev 11)
00:1c.0 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev 
11)
00:1c.1 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev 
11)
00:1c.2 PCI bridge: Intel Corporation ValleyView PCI Express Root Port (rev 
11)
00:1d.0 USB controller: Intel Corporation ValleyView USB Enhanced Host 
Controller (rev 11)
00:1f.0 ISA bridge: Intel Corporation ValleyView Power Control Unit (rev 11)
00:1f.3 SMBus: Intel Corporation ValleyView SMBus Controller (rev 11)
01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
02:03.0 Bridge: ONA Electroerosion PCI32TLITE FILSTRUP1 PCI to VME Bridge 
Controller (rev 04)
02:04.0 Communication controller: ONA Electroerosion Device 000a
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
RTL8101E/RTL8102E PCI Express Fast Ethernet controller (rev 05)
04:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6101/6102 single-
port PATA133 interface (rev c0)

lspci -vt
-[0000:00]-+-00.0  Intel Corporation ValleyView SSA-CUnit
           +-02.0  Intel Corporation ValleyView Gen7
           +-1a.0  Intel Corporation ValleyView SEC
           +-1b.0  Intel Corporation ValleyView High Definition Audio 
Controller
           +-1c.0-[01-02]----00.0-[02]--+-03.0  ONA Electroerosion 
PCI32TLITE FILSTRUP1 PCI to VME Bridge Controller
           |                            \-04.0  ONA Electroerosion Device 
000a
           +-1c.1-[03]----00.0  Realtek Semiconductor Co., Ltd. 
RTL8101E/RTL8102E PCI Express Fast Ethernet controller
           +-1c.2-[04]----00.0  Marvell Technology Group Ltd. 88SE6101/6102 
single-port PATA133 interface
           +-1d.0  Intel Corporation ValleyView USB Enhanced Host Controller
           +-1f.0  Intel Corporation ValleyView Power Control Unit
           \-1f.3  Intel Corporation ValleyView SMBus Controller

Linux: Ubuntu 14.04
Kernel: 3.13.0-62-generic

Is possible to fix it?

 

Re: PCI rescan : can't assign mem

[Yinghai Lu]

On Mon, Aug 31, 2015 at 12:47 AM, Peio <peio@azkarate.eu> wrote:
> I have a board with an PCI Express-to-PCI Bridge (Texas Instruments
> XIO2001).
> Attached to the PCI bus, there are 2 FPGA PCI Target devices needing 32MB
> memory map each.
> The PCI rescan "echo 1 > /sys/bus/pci/rescan" memory assign fails.
>
> dmesg messages:
> [  150.978654] pci 0000:02:03.0: [4150:0001] type 00 class 0x068000
> [  150.978684] pci 0000:02:03.0: reg 0x10: [mem 0x00000000-0x01ffffff]
> [  150.978885] pci 0000:02:04.0: [4150:000a] type 00 class 0x078000
> [  150.978909] pci 0000:02:04.0: reg 0x10: [mem 0x00000000-0x01ffffff]
> [  150.979158] pcieport 0000:00:1c.0: bridge window [mem 0x00100000-
> 0x000fffff 64bit pref] to [bus 01-02] add_size 200000
> [  150.979166] pcieport 0000:00:1c.0: bridge window [mem 0x02000000-
> 0x05ffffff] to [bus 01-02] add_size 2000000
> [  150.979183] pcieport 0000:00:1c.2: bridge window [mem 0x00100000-
> 0x000fffff 64bit pref] to [bus 04] add_size 200000
> [  150.979190] i915 0000:00:02.0: BAR 6: [??? 0x00000000 flags 0x2] has
> bogus alignment
> [  150.979199] pcieport 0000:00:1c.0: res[14]=[mem 0x02000000-0x05ffffff]
> get_res_add_size add_size 2000000
> [  150.979205] pcieport 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff
> 64bit pref] get_res_add_size add_size 200000
> [  150.979211] pcieport 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff
> 64bit pref] get_res_add_size add_size 200000
> [  150.979222] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size
> 0x6000000)
> [  150.979230] pcieport 0000:00:1c.0: BAR 15: can't assign mem pref (size
> 0x200000)
> [  150.979238] pcieport 0000:00:1c.2: BAR 15: can't assign mem pref (size
> 0x200000)
> [  150.979245] pcieport 0000:00:1c.0: BAR 14: can't assign mem (size
> 0x4000000)

> Linux: Ubuntu 14.04
> Kernel: 3.13.0-62-generic

how about current upsteam kernel?

> Is possible to fix it?

Please whole post boot log with "debug ignore_loglevel".

Need _CRS from BIOS provide enough MMIO range for 00:1c.0

Yinghai

Re: PCI rescan : can't assign mem

[Peio]

Yinghai Lu <yinghai <at> kernel.org> writes:


> how about current upsteam kernel?

> > Is possible to fix it?
> 
> Please whole post boot log with "debug ignore_loglevel".
> 
> Need _CRS from BIOS provide enough MMIO range for 00:1c.0
> 
> Yinghai
> 

I prefer the stable kernel used by ubuntu 14.04.
My BIOS don't have MMIO range configuration options.
Log with "debug ignore_loglevel" sentto Yinghai (too long for post here).

Re: PCI rescan : can't assign mem

[Yinghai Lu]

On Tue, Sep 1, 2015 at 2:01 AM, Peio <peio@azkarate.eu> wrote:
> Yinghai Lu <yinghai <at> kernel.org> writes:
>
>
>> how about current upsteam kernel?
>
>> > Is possible to fix it?
>>
>> Please whole post boot log with "debug ignore_loglevel".
>>
>> Need _CRS from BIOS provide enough MMIO range for 00:1c.0
>>
> I prefer the stable kernel used by ubuntu 14.04.
> My BIOS don't have MMIO range configuration options.
> Log with "debug ignore_loglevel" sentto Yinghai (too long for post here).

root bus resource:

[    0.277955] PCI host bridge to bus 0000:00
[    0.277966] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.277975] pci_bus 0000:00: root bus resource [io  0x0000-0x006f]
[    0.277983] pci_bus 0000:00: root bus resource [io  0x0078-0x0cf7]
[    0.277991] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    0.278000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[    0.278008] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff]
[    0.278016] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000fffff]
[    0.278025] pci_bus 0000:00: root bus resource [mem 0x80000000-0x909ffffe]

and 00:02.0 use almost all

[    0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff]
[    0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff pref]

so we have no chance to allocate MMIO to 00:1c.0

[    0.359994] pci 0000:00:1c.0: bridge window [io  0x1000-0x0fff] to
[bus 01-02] add_size 1000
[    0.360009] pci 0000:00:1c.0: bridge window [mem
0x00100000-0x000fffff 64bit pref] to [bus 01-02] add_size 200000
[    0.360022] pci 0000:00:1c.0: bridge window [mem
0x00100000-0x000fffff] to [bus 01-02] add_size 200000
[    0.360045] pci 0000:00:1c.2: bridge window [mem
0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000
[    0.360061] pci 0000:00:1c.0: res[14]=[mem 0x00100000-0x000fffff]
get_res_add_size add_size 200000
[    0.360072] pci 0000:00:1c.0: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[    0.360084] pci 0000:00:1c.2: res[15]=[mem 0x00100000-0x000fffff
64bit pref] get_res_add_size add_size 200000
[    0.360095] pci 0000:00:1c.0: res[13]=[io  0x1000-0x0fff]
get_res_add_size add_size 1000
[    0.360111] pci 0000:00:1c.0: BAR 14: can't assign mem (size 0x200000)
[    0.360123] pci 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000)
[    0.360134] pci 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000)
[    0.360146] pci 0000:00:1c.0: BAR 13: assigned [io  0x4000-0x4fff]
[    0.360158] pci 0000:00:1c.2: BAR 15: can't assign mem pref (size 0x200000)
[    0.360169] pci 0000:00:1c.0: BAR 14: can't assign mem (size 0x200000)
[    0.360180] pci 0000:00:1c.0: BAR 15: can't assign mem pref (size 0x200000)
[    0.360189] pci 0000:01:00.0: PCI bridge to [bus 02]
[    0.360214] pci 0000:00:1c.0: PCI bridge to [bus 01-02]
[    0.360223] pci 0000:00:1c.0:   bridge window [io  0x4000-0x4fff]

Re: PCI rescan : can't assign mem

[Peio]

Yinghai Lu <yinghai <at> kernel.org> writes:

> 
> On Tue, Sep 1, 2015 at 2:01 AM, Peio <peio <at> azkarate.eu> wrote:
> > Yinghai Lu <yinghai <at> kernel.org> writes:

> root bus resource:
> 
> [    0.277955] PCI host bridge to bus 0000:00
> [    0.277966] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    0.277975] pci_bus 0000:00: root bus resource [io  0x0000-0x006f]
> [    0.277983] pci_bus 0000:00: root bus resource [io  0x0078-0x0cf7]
> [    0.277991] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
> [    0.278000] pci_bus 0000:00: root bus resource [mem 0x000a0000-
0x000bffff]
> [    0.278008] pci_bus 0000:00: root bus resource [mem 0x000c0000-
0x000dffff]
> [    0.278016] pci_bus 0000:00: root bus resource [mem 0x000e0000-
0x000fffff]
> [    0.278025] pci_bus 0000:00: root bus resource [mem 0x80000000-
0x909ffffe]
> 
> and 00:02.0 use almost all
> 
> [    0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff]
> [    0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff 
pref]
> 
> so we have no chance to allocate MMIO to 00:1c.0
> 

Any workaround?. 

Is the BIOS responsible for allocating memory resources in Linux?.
Perhaps changing some BIOS parameter ACPI related?.
If I change BIOS parameters I'll need a fresh install or the actual kernel 
reconigzes changes?.

Re: PCI rescan : can't assign mem

[Yinghai Lu]

On Wed, Sep 2, 2015 at 12:01 AM, Peio <peio@azkarate.eu> wrote:
> Yinghai Lu <yinghai <at> kernel.org> writes:
>> and 00:02.0 use almost all
>>
>> [    0.278221] pci 0000:00:02.0: reg 0x10: [mem 0x90000000-0x903fffff]
>> [    0.278239] pci 0000:00:02.0: reg 0x18: [mem 0x80000000-0x8fffffff
> pref]
>>
>> so we have no chance to allocate MMIO to 00:1c.0
>>
>
> Any workaround?.

boot with pci=nocrs ?

>
> Is the BIOS responsible for allocating memory resources in Linux?.

The resource range for root bus. As BIOS may need to set other registers to make
them effective.

> Perhaps changing some BIOS parameter ACPI related?.

Try to get BIOS update.

> If I change BIOS parameters I'll need a fresh install or the actual kernel
> reconigzes changes?.
should not.

Re: PCI rescan : can't assign mem

[Peio]

Yinghai Lu <yinghai <at> kernel.org> writes:


> boot with pci=nocrs ?
> 
> Try to get BIOS update.
> 
> > If I change BIOS parameters I'll need a fresh install or the actual kernel
> > reconigzes changes?.
> should not.
> 

I'll try pci=nocrs and BIOS update.
Thank you Yinghai.

Re: PCI rescan : can't assign mem [SOLVED]

[Peio]

Peio <peio <at> azkarate.eu> writes:

> 
> Yinghai Lu <yinghai <at> kernel.org> writes:
> 
> > boot with pci=nocrs ?
> > 
> > Try to get BIOS update.
> > 
> > > If I change BIOS parameters I'll need a fresh install or the actual 
kernel
> > > reconigzes changes?.
> > should not.
> > 
> 
> I'll try pci=nocrs and BIOS update.
> Thank you Yinghai.
> 
> 
I solved the issue.

1.- Boot kernel with pci=nocrs parameter (linux assigns lot of memory ro pci 
root bus).

2.- Remove upstream bridge : 
sudo echo 1 > /sys/bus/pci/devices/0000:00:1c.0/remove

3.- Rescan PCI bus
sudo echo 1 > /sys/bus/pci/rescan

Re: PCI rescan : can't assign mem [SOLVED]

[Bjorn Helgaas]

On Thu, Sep 3, 2015 at 4:45 AM, Peio <peio@azkarate.eu> wrote:
> Peio <peio <at> azkarate.eu> writes:
>
>>
>> Yinghai Lu <yinghai <at> kernel.org> writes:
>>
>> > boot with pci=nocrs ?
>> >
>> > Try to get BIOS update.
>> >
>> > > If I change BIOS parameters I'll need a fresh install or the actual
> kernel
>> > > reconigzes changes?.
>> > should not.
>> >
>>
>> I'll try pci=nocrs and BIOS update.
>> Thank you Yinghai.
>>
>>
> I solved the issue.
>
> 1.- Boot kernel with pci=nocrs parameter (linux assigns lot of memory ro pci
> root bus).
>
> 2.- Remove upstream bridge :
> sudo echo 1 > /sys/bus/pci/devices/0000:00:1c.0/remove
>
> 3.- Rescan PCI bus
> sudo echo 1 > /sys/bus/pci/rescan

This is a workaround, not a solution.  It is not safe to use
"pci=nocrs" in general because it can break things related to resource
assignment, including hot-add of new devices.

I haven't looked at the problem in detail, but if the BIOS gives us a
_CRS with host bridge windows that are too small to accommodate all
your devices, there's really nothing the kernel can do about it.  You
just have to regard it as a limitation of that platform.

Bjorn

Re: PCI rescan : can't assign mem [SOLVED]

[Peio]

Bjorn Helgaas <bhelgaas <at> google.com> writes:

> This is a workaround, not a solution.  It is not safe to use
> "pci=nocrs" in general because it can break things related to resource
> assignment, including hot-add of new devices.
> 
> I haven't looked at the problem in detail, but if the BIOS gives us a
> _CRS with host bridge windows that are too small to accommodate all
> your devices, there's really nothing the kernel can do about it.  You
> just have to regard it as a limitation of that platform.
> 
> Bjorn
> 
I'm sure itsn't a platform limitation, a BIOS issue assigning resources 
instead.

Anyway, It's a closed system, without hot-add or not own devices.

I suspect this workaround is fine to me. 

Re: PCI rescan : can't assign mem [SOLVED]

[Bjorn Helgaas]

On Fri, Sep 4, 2015 at 12:32 AM, Peio <peio@azkarate.eu> wrote:
> Bjorn Helgaas <bhelgaas <at> google.com> writes:
>
>> This is a workaround, not a solution.  It is not safe to use
>> "pci=nocrs" in general because it can break things related to resource
>> assignment, including hot-add of new devices.
>>
>> I haven't looked at the problem in detail, but if the BIOS gives us a
>> _CRS with host bridge windows that are too small to accommodate all
>> your devices, there's really nothing the kernel can do about it.  You
>> just have to regard it as a limitation of that platform.
>>
>> Bjorn
>>
> I'm sure itsn't a platform limitation, a BIOS issue assigning resources
> instead.

Yep.  I include the BIOS in "the platform" because it's something we
have no control over.

> Anyway, It's a closed system, without hot-add or not own devices.
>
> I suspect this workaround is fine to me.

OK.  I didn't phrase the risk very well.  When we use "pci=nocrs",
we're ignoring the host bridge apertures the BIOS tells us about.
That means Linux may assign resources outside of those apertures to
PCI devices.  They *might* work, but there's no guarantee that they
will.  This can affect devices present at boot as well as hot-added
ones.

If we were really smart, we might emit a warning when we assigned
resources outside of the apertures, but that's probably more work than
it's worth.

If the workaround works for you, that's great and you should keep
using it.  I just don't want to propagate the idea that "pci=nocrs" is
the general solution for anything.   Ideally, a user should not have
to use kernel parameters except for debugging or to work around
hardware or firmware problems.

Bjorn

Re: PCI rescan : can't assign mem [SOLVED]

[Peio]

Bjorn Helgaas <bhelgaas <at> google.com> writes:

> 
> On Fri, Sep 4, 2015 at 12:32 AM, Peio <peio <at> azkarate.eu> wrote:
> > Bjorn Helgaas <bhelgaas <at> google.com> writes:
> >-
> 
> 
> If the workaround works for you, that's great and you should keep
> using it.  I just don't want to propagate the idea that "pci=nocrs" is
> the general solution for anything.   Ideally, a user should not have
> to use kernel parameters except for debugging or to work around
> hardware or firmware problems.
> 
> Bjorn
> 

At the moment I'll use the workaround, but a better solution will be, update 
the BIOS to a version assigning more memory resources to pcie.
Thank you for your warning Bjorn.

Peio