Why does BIOS assign memory to 16 byte BAR

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Hi,

I'm observing that on x86 BIOS successfully assigns memory if an End point requests
BAR of size 16byte.

But as per Spec:
The minimum memory address range requested by a BAR is 128 bytes.

Why BIOS is successfully allocating region to 16 byte BAR requests?

Please let me know if my observation is wrong.

Regards,
Bharat



This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Hi,

I'm observing that on x86 BIOS successfully assigns memory if an End point requests
BAR of size 16byte.

But as per Spec:
The minimum memory address range requested by a BAR is 128 bytes.

Why BIOS is successfully allocating region to 16 byte BAR requests?

Please let me know if my observation is wrong.

Regards,
Bharat



This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Hi,

I'm observing that on x86 BIOS successfully assigns memory if an End point requests
BAR of size 16byte.

But as per Spec:
The minimum memory address range requested by a BAR is 128 bytes.

Why BIOS is successfully allocating region to 16 byte BAR requests?

Please let me know if my observation is wrong.

Regards,
Bharat



This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

Hi Bharat,

On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> Hi,
> 
> I'm observing that on x86 BIOS successfully assigns memory if an End point requests
> BAR of size 16byte.
> 
> But as per Spec:
> The minimum memory address range requested by a BAR is 128 bytes.

Can you provide the spec reference for this?  I don't see it in PCI
r3.0.

PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
which would correspond to a minimum size of 16 bytes.

> Why BIOS is successfully allocating region to 16 byte BAR requests?

If you want to know why the BIOS does this, you'd have to ask the BIOS
writer.

If you were to ask about *Linux*, I'd say we should not gratuitously
enforce things in the spec merely for the sake of being compliant with
the letter of the spec.  We should enforce things that are logically
required or that are required for compatibility, security, etc.  But
if the spec contained an arbitrary restriction  for no good reason,
and real-world hardware violated that restriction, I'd say Linux
should accommodate the hardware.

Bjorn

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

Hi Bharat,

On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> Hi,
> 
> I'm observing that on x86 BIOS successfully assigns memory if an End point requests
> BAR of size 16byte.
> 
> But as per Spec:
> The minimum memory address range requested by a BAR is 128 bytes.

Can you provide the spec reference for this?  I don't see it in PCI
r3.0.

PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
which would correspond to a minimum size of 16 bytes.

> Why BIOS is successfully allocating region to 16 byte BAR requests?

If you want to know why the BIOS does this, you'd have to ask the BIOS
writer.

If you were to ask about *Linux*, I'd say we should not gratuitously
enforce things in the spec merely for the sake of being compliant with
the letter of the spec.  We should enforce things that are logically
required or that are required for compatibility, security, etc.  But
if the spec contained an arbitrary restriction  for no good reason,
and real-world hardware violated that restriction, I'd say Linux
should accommodate the hardware.

Bjorn

Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

Hi Bharat,

On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> Hi,
> 
> I'm observing that on x86 BIOS successfully assigns memory if an End point requests
> BAR of size 16byte.
> 
> But as per Spec:
> The minimum memory address range requested by a BAR is 128 bytes.

Can you provide the spec reference for this?  I don't see it in PCI
r3.0.

PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
which would correspond to a minimum size of 16 bytes.

> Why BIOS is successfully allocating region to 16 byte BAR requests?

If you want to know why the BIOS does this, you'd have to ask the BIOS
writer.

If you were to ask about *Linux*, I'd say we should not gratuitously
enforce things in the spec merely for the sake of being compliant with
the letter of the spec.  We should enforce things that are logically
required or that are required for compatibility, security, etc.  But
if the spec contained an arbitrary restriction  for no good reason,
and real-world hardware violated that restriction, I'd say Linux
should accommodate the hardware.

Bjorn

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> Hi Bharat,
> 
> On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > Hi,
> > 
> > I'm observing that on x86 BIOS successfully assigns memory if an End point requests
> > BAR of size 16byte.
> > 
> > But as per Spec:
> > The minimum memory address range requested by a BAR is 128 bytes.
> 
> Can you provide the spec reference for this?  I don't see it in PCI
> r3.0.
>
> PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> which would correspond to a minimum size of 16 bytes.

The reference above is to the conventional PCI spec.  I happened to
trip over a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express
endpoint, "the minimum memory address range requested by a BAR is 128
bytes."

I don't think linux currently enforces this minimum.

Bjorn

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> Hi Bharat,
> 
> On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > Hi,
> > 
> > I'm observing that on x86 BIOS successfully assigns memory if an End point requests
> > BAR of size 16byte.
> > 
> > But as per Spec:
> > The minimum memory address range requested by a BAR is 128 bytes.
> 
> Can you provide the spec reference for this?  I don't see it in PCI
> r3.0.
>
> PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> which would correspond to a minimum size of 16 bytes.

The reference above is to the conventional PCI spec.  I happened to
trip over a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express
endpoint, "the minimum memory address range requested by a BAR is 128
bytes."

I don't think linux currently enforces this minimum.

Bjorn

Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> Hi Bharat,
> 
> On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > Hi,
> > 
> > I'm observing that on x86 BIOS successfully assigns memory if an End point requests
> > BAR of size 16byte.
> > 
> > But as per Spec:
> > The minimum memory address range requested by a BAR is 128 bytes.
> 
> Can you provide the spec reference for this?  I don't see it in PCI
> r3.0.
>
> PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> which would correspond to a minimum size of 16 bytes.

The reference above is to the conventional PCI spec.  I happened to
trip over a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express
endpoint, "the minimum memory address range requested by a BAR is 128
bytes."

I don't think linux currently enforces this minimum.

Bjorn

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: Re: Why does BIOS assign memory to 16 byte BAR
> 
> On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > Hi Bharat,
> >
> > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > Hi,
> > >
> > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > point requests BAR of size 16byte.
> > >
> > > But as per Spec:
> > > The minimum memory address range requested by a BAR is 128 bytes.
> >
> > Can you provide the spec reference for this?  I don't see it in PCI
> > r3.0.
> >
> > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > which would correspond to a minimum size of 16 bytes.
> 
> The reference above is to the conventional PCI spec.  I happened to trip over
> a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the minimum
> memory address range requested by a BAR is 128 bytes."
> 
> I don't think linux currently enforces this minimum.
> 

Hi Bjorn Thanks for the reply.

Here is what the issue we are seeing.

We have total memory for BAR's on our SoC of 256 MB.
When an End Point request individually 16 byte BAR's our root port assigns memory to BAR's successfully.

But if I have an End point which has 4 BAR's each 32 bit and request as following:
When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are running this as part of SIG compliance test case)
2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
We have 256 MB still)

We have only one End point connected to our root port.

Here is the log:
[    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
[    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
[    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus 00-ff]
[    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
[    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
[    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
[    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
[    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same bus?
[    2.319861] iommu: Adding device 0000:00:00.0 to group 1
[    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same bus?
[    2.320258] iommu: Adding device 0000:01:00.0 to group 1
[    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
[    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60000000]
[    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff pref]
[    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
[    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40000000]
[    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000 64bit]
[    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00100000 64bit]
[    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
[    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00000010]
[    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
[    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00000010]
[    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]

Please let me know, what might might be the issue.

Thanks & Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: Re: Why does BIOS assign memory to 16 byte BAR
>=20
> On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > Hi Bharat,
> >
> > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > Hi,
> > >
> > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > point requests BAR of size 16byte.
> > >
> > > But as per Spec:
> > > The minimum memory address range requested by a BAR is 128 bytes.
> >
> > Can you provide the spec reference for this?  I don't see it in PCI
> > r3.0.
> >
> > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > which would correspond to a minimum size of 16 bytes.
>=20
> The reference above is to the conventional PCI spec.  I happened to trip =
over
> a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the m=
inimum
> memory address range requested by a BAR is 128 bytes."
>=20
> I don't think linux currently enforces this minimum.
>=20

Hi Bjorn Thanks for the reply.

Here is what the issue we are seeing.

We have total memory for BAR's on our SoC of 256 MB.
When an End Point request individually 16 byte BAR's our root port assigns =
memory to BAR's successfully.

But if I have an End point which has 4 BAR's each 32 bit and request as fol=
lowing:
When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are runni=
ng this as part of SIG compliance test case)
2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not ge=
tting BAR's assigned. (Even though BAR space is available, since 1GB failed=
,
We have 256 MB still)

We have only one End point connected to our root port.

Here is the log:
[    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
[    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
[    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus 00-=
ff]
[    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
[    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
[    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
[    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-0xeffffff=
f]
[    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7fffff=
fff pref]
[    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same b=
us?
[    2.319861] iommu: Adding device 0000:00:00.0 to group 1
[    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same b=
us?
[    2.320258] iommu: Adding device 0000:01:00.0 to group 1
[    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
[    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x600000=
00]
[    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff=
 pref]
[    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
[    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x400000=
00]
[    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000 6=
4bit]
[    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x001000=
00 64bit]
[    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
[    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x000000=
10]
[    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
[    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x000000=
10]
[    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]

Please let me know, what might might be the issue.

Thanks & Regards,
Bharat

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: Re: Why does BIOS assign memory to 16 byte BAR
> 
> On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > Hi Bharat,
> >
> > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > Hi,
> > >
> > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > point requests BAR of size 16byte.
> > >
> > > But as per Spec:
> > > The minimum memory address range requested by a BAR is 128 bytes.
> >
> > Can you provide the spec reference for this?  I don't see it in PCI
> > r3.0.
> >
> > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > which would correspond to a minimum size of 16 bytes.
> 
> The reference above is to the conventional PCI spec.  I happened to trip over
> a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the minimum
> memory address range requested by a BAR is 128 bytes."
> 
> I don't think linux currently enforces this minimum.
> 

Hi Bjorn Thanks for the reply.

Here is what the issue we are seeing.

We have total memory for BAR's on our SoC of 256 MB.
When an End Point request individually 16 byte BAR's our root port assigns memory to BAR's successfully.

But if I have an End point which has 4 BAR's each 32 bit and request as following:
When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are running this as part of SIG compliance test case)
2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
We have 256 MB still)

We have only one End point connected to our root port.

Here is the log:
[    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
[    2.319332] PCI host bridge /amba/pcie at fd0e0000 ranges:
[    2.319349]   No bus range found for /amba/pcie at fd0e0000, using [bus 00-ff]
[    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
[    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
[    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
[    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
[    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same bus?
[    2.319861] iommu: Adding device 0000:00:00.0 to group 1
[    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same bus?
[    2.320258] iommu: Adding device 0000:01:00.0 to group 1
[    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
[    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60000000]
[    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff pref]
[    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
[    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40000000]
[    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000 64bit]
[    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00100000 64bit]
[    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
[    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00000010]
[    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
[    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00000010]
[    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]

Please let me know, what might might be the issue.

Thanks & Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> >
> > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > Hi Bharat,
> > >
> > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > Hi,
> > > >
> > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > point requests BAR of size 16byte.
> > > >
> > > > But as per Spec:
> > > > The minimum memory address range requested by a BAR is 128 bytes.
> > >
> > > Can you provide the spec reference for this?  I don't see it in PCI
> > > r3.0.
> > >
> > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > which would correspond to a minimum size of 16 bytes.
> >
> > The reference above is to the conventional PCI spec.  I happened to trip
> over
> > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> minimum
> > memory address range requested by a BAR is 128 bytes."
> >
> > I don't think linux currently enforces this minimum.
> >
> 
> Hi Bjorn Thanks for the reply.
> 
> Here is what the issue we are seeing.
> 
> We have total memory for BAR's on our SoC of 256 MB.
> When an End Point request individually 16 byte BAR's our root port assigns
> memory to BAR's successfully.
> 
> But if I have an End point which has 4 BAR's each 32 bit and request as
> following:
> When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> running this as part of SIG compliance test case)
> 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
> We have 256 MB still)
> 
> We have only one End point connected to our root port.
> 
> Here is the log:
> [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
> [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus 00-
> ff]
> [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
> [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> 0x7ffffffff pref]
> [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same bus?
> [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same bus?
> [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60000000]
> [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff
> pref]
> [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40000000]
> [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> 64bit]
> [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00100000
> 64bit]
> [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00000010]
> [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00000010]
> [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> 
> Please let me know, what might might be the issue.
> 

Adding to the above will kernel allocate other memory BARs to an End Point if 
one BAR assignment fails ?

What if the End Point has multiple function and say first function BAR assignment failed,
will the kernel assign BAR's to second function on same bus and device ?  

Thanks & Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> >
> > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > Hi Bharat,
> > >
> > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > Hi,
> > > >
> > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > point requests BAR of size 16byte.
> > > >
> > > > But as per Spec:
> > > > The minimum memory address range requested by a BAR is 128 bytes.
> > >
> > > Can you provide the spec reference for this?  I don't see it in PCI
> > > r3.0.
> > >
> > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > which would correspond to a minimum size of 16 bytes.
> >
> > The reference above is to the conventional PCI spec.  I happened to trip
> over
> > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> minimum
> > memory address range requested by a BAR is 128 bytes."
> >
> > I don't think linux currently enforces this minimum.
> >
> 
> Hi Bjorn Thanks for the reply.
> 
> Here is what the issue we are seeing.
> 
> We have total memory for BAR's on our SoC of 256 MB.
> When an End Point request individually 16 byte BAR's our root port assigns
> memory to BAR's successfully.
> 
> But if I have an End point which has 4 BAR's each 32 bit and request as
> following:
> When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> running this as part of SIG compliance test case)
> 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
> We have 256 MB still)
> 
> We have only one End point connected to our root port.
> 
> Here is the log:
> [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
> [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus 00-
> ff]
> [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
> [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> 0x7ffffffff pref]
> [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same bus?
> [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same bus?
> [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60000000]
> [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff
> pref]
> [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40000000]
> [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> 64bit]
> [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00100000
> 64bit]
> [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00000010]
> [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00000010]
> [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> 
> Please let me know, what might might be the issue.
> 

Adding to the above will kernel allocate other memory BARs to an End Point if 
one BAR assignment fails ?

What if the End Point has multiple function and say first function BAR assignment failed,
will the kernel assign BAR's to second function on same bus and device ?  

Thanks & Regards,
Bharat

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> >
> > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > Hi Bharat,
> > >
> > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > Hi,
> > > >
> > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > point requests BAR of size 16byte.
> > > >
> > > > But as per Spec:
> > > > The minimum memory address range requested by a BAR is 128 bytes.
> > >
> > > Can you provide the spec reference for this?  I don't see it in PCI
> > > r3.0.
> > >
> > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > which would correspond to a minimum size of 16 bytes.
> >
> > The reference above is to the conventional PCI spec.  I happened to trip
> over
> > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> minimum
> > memory address range requested by a BAR is 128 bytes."
> >
> > I don't think linux currently enforces this minimum.
> >
> 
> Hi Bjorn Thanks for the reply.
> 
> Here is what the issue we are seeing.
> 
> We have total memory for BAR's on our SoC of 256 MB.
> When an End Point request individually 16 byte BAR's our root port assigns
> memory to BAR's successfully.
> 
> But if I have an End point which has 4 BAR's each 32 bit and request as
> following:
> When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> running this as part of SIG compliance test case)
> 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
> We have 256 MB still)
> 
> We have only one End point connected to our root port.
> 
> Here is the log:
> [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> [    2.319332] PCI host bridge /amba/pcie at fd0e0000 ranges:
> [    2.319349]   No bus range found for /amba/pcie at fd0e0000, using [bus 00-
> ff]
> [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
> [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> 0x7ffffffff pref]
> [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same bus?
> [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same bus?
> [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60000000]
> [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff
> pref]
> [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40000000]
> [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> 64bit]
> [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00100000
> 64bit]
> [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00000010]
> [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00000010]
> [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> 
> Please let me know, what might might be the issue.
> 

Adding to the above will kernel allocate other memory BARs to an End Point if 
one BAR assignment fails ?

What if the End Point has multiple function and say first function BAR assignment failed,
will the kernel assign BAR's to second function on same bus and device ?  

Thanks & Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: RE: Why does BIOS assign memory to 16 byte BAR
> >
> > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > >
> > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > Hi Bharat,
> > > >
> > > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > > Hi,
> > > > >
> > > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > > point requests BAR of size 16byte.
> > > > >
> > > > > But as per Spec:
> > > > > The minimum memory address range requested by a BAR is 128
> bytes.
> > > >
> > > > Can you provide the spec reference for this?  I don't see it in PCI
> > > > r3.0.
> > > >
> > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > > which would correspond to a minimum size of 16 bytes.
> > >
> > > The reference above is to the conventional PCI spec.  I happened to trip
> > over
> > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> > minimum
> > > memory address range requested by a BAR is 128 bytes."
> > >
> > > I don't think linux currently enforces this minimum.
> > >
> >
> > Hi Bjorn Thanks for the reply.
> >
> > Here is what the issue we are seeing.
> >
> > We have total memory for BAR's on our SoC of 256 MB.
> > When an End Point request individually 16 byte BAR's our root port assigns
> > memory to BAR's successfully.
> >
> > But if I have an End point which has 4 BAR's each 32 bit and request as
> > following:
> > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> > running this as part of SIG compliance test case)
> > 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> > getting BAR's assigned. (Even though BAR space is available, since 1GB
> failed,
> > We have 256 MB still)
> >
> > We have only one End point connected to our root port.
> >
> > Here is the log:
> > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> > [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
> > [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus
> 00-
> > ff]
> > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
> 0xefffffff]
> > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> > 0x7ffffffff pref]
> > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60000000]
> > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
> 0xe01007ff
> > pref]
> > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40000000]
> > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> > 64bit]
> > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00100000
> > 64bit]
> > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00000010]
> > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00000010]
> > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> >
> > Please let me know, what might might be the issue.
> >
> 
> Adding to the above will kernel allocate other memory BARs to an End Point
> if
> one BAR assignment fails ?
> 
> What if the End Point has multiple function and say first function BAR
> assignment failed,
> will the kernel assign BAR's to second function on same bus and device ?
> 
After debugging in function pci_bus_alloc_from_region we have two resources,
One which is taken as parameter (let's say resA), other is from  pci_bus_for_each_resource (let's say resB).

The resA contains different start address from resB, I see that resB is obtained from device resource 
from bus->resource, but im unable to understand how resA (start address & size) gets its data. 

Can any one help me understand this so that I might know reason why 16byte BAR allocation failing after
1GB request.

Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
>=20
> > Subject: RE: Why does BIOS assign memory to 16 byte BAR
> >
> > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > >
> > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > Hi Bharat,
> > > >
> > > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote=
:
> > > > > Hi,
> > > > >
> > > > > I'm observing that on x86 BIOS successfully assigns memory if an =
End
> > > > > point requests BAR of size 16byte.
> > > > >
> > > > > But as per Spec:
> > > > > The minimum memory address range requested by a BAR is 128
> bytes.
> > > >
> > > > Can you provide the spec reference for this?  I don't see it in PCI
> > > > r3.0.
> > > >
> > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > > which would correspond to a minimum size of 16 bytes.
> > >
> > > The reference above is to the conventional PCI spec.  I happened to t=
rip
> > over
> > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "t=
he
> > minimum
> > > memory address range requested by a BAR is 128 bytes."
> > >
> > > I don't think linux currently enforces this minimum.
> > >
> >
> > Hi Bjorn Thanks for the reply.
> >
> > Here is what the issue we are seeing.
> >
> > We have total memory for BAR's on our SoC of 256 MB.
> > When an End Point request individually 16 byte BAR's our root port assi=
gns
> > memory to BAR's successfully.
> >
> > But if I have an End point which has 4 BAR's each 32 bit and request as
> > following:
> > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> > running this as part of SIG compliance test case)
> > 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are no=
t
> > getting BAR's assigned. (Even though BAR space is available, since 1GB
> failed,
> > We have 256 MB still)
> >
> > We have only one End point connected to our root port.
> >
> > Here is the log:
> > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> > [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
> > [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus
> 00-
> > ff]
> > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
> 0xefffffff]
> > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> > 0x7ffffffff pref]
> > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the sa=
me
> bus?
> > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the sa=
me
> bus?
> > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x600000=
00]
> > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60=
000000]
> > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
> 0xe01007ff
> > pref]
> > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x400000=
00]
> > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40=
000000]
> > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x001000=
00
> > 64bit]
> > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00=
100000
> > 64bit]
> > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x000000=
10]
> > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00=
000010]
> > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x000000=
10]
> > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00=
000010]
> > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> >
> > Please let me know, what might might be the issue.
> >
>=20
> Adding to the above will kernel allocate other memory BARs to an End Poin=
t
> if
> one BAR assignment fails ?
>=20
> What if the End Point has multiple function and say first function BAR
> assignment failed,
> will the kernel assign BAR's to second function on same bus and device ?
>=20
After debugging in function pci_bus_alloc_from_region we have two resources=
,
One which is taken as parameter (let's say resA), other is from  pci_bus_fo=
r_each_resource (let's say resB).

The resA contains different start address from resB, I see that resB is obt=
ained from device resource=20
from bus->resource, but im unable to understand how resA (start address & s=
ize) gets its data.=20

Can any one help me understand this so that I might know reason why 16byte =
BAR allocation failing after
1GB request.

Regards,
Bharat

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: RE: Why does BIOS assign memory to 16 byte BAR
> >
> > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > >
> > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > Hi Bharat,
> > > >
> > > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > > Hi,
> > > > >
> > > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > > point requests BAR of size 16byte.
> > > > >
> > > > > But as per Spec:
> > > > > The minimum memory address range requested by a BAR is 128
> bytes.
> > > >
> > > > Can you provide the spec reference for this?  I don't see it in PCI
> > > > r3.0.
> > > >
> > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > > which would correspond to a minimum size of 16 bytes.
> > >
> > > The reference above is to the conventional PCI spec.  I happened to trip
> > over
> > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> > minimum
> > > memory address range requested by a BAR is 128 bytes."
> > >
> > > I don't think linux currently enforces this minimum.
> > >
> >
> > Hi Bjorn Thanks for the reply.
> >
> > Here is what the issue we are seeing.
> >
> > We have total memory for BAR's on our SoC of 256 MB.
> > When an End Point request individually 16 byte BAR's our root port assigns
> > memory to BAR's successfully.
> >
> > But if I have an End point which has 4 BAR's each 32 bit and request as
> > following:
> > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> > running this as part of SIG compliance test case)
> > 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> > getting BAR's assigned. (Even though BAR space is available, since 1GB
> failed,
> > We have 256 MB still)
> >
> > We have only one End point connected to our root port.
> >
> > Here is the log:
> > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> > [    2.319332] PCI host bridge /amba/pcie at fd0e0000 ranges:
> > [    2.319349]   No bus range found for /amba/pcie at fd0e0000, using [bus
> 00-
> > ff]
> > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
> 0xefffffff]
> > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> > 0x7ffffffff pref]
> > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x60000000]
> > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
> 0xe01007ff
> > pref]
> > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x40000000]
> > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> > 64bit]
> > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size 0x00100000
> > 64bit]
> > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00000010]
> > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x00000010]
> > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> >
> > Please let me know, what might might be the issue.
> >
> 
> Adding to the above will kernel allocate other memory BARs to an End Point
> if
> one BAR assignment fails ?
> 
> What if the End Point has multiple function and say first function BAR
> assignment failed,
> will the kernel assign BAR's to second function on same bus and device ?
> 
After debugging in function pci_bus_alloc_from_region we have two resources,
One which is taken as parameter (let's say resA), other is from  pci_bus_for_each_resource (let's say resB).

The resA contains different start address from resB, I see that resB is obtained from device resource 
from bus->resource, but im unable to understand how resA (start address & size) gets its data. 

Can any one help me understand this so that I might know reason why 16byte BAR allocation failing after
1GB request.

Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Adding yinghai lu.

> > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > > >
> > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > > Hi Bharat,
> > > > >
> > > > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm observing that on x86 BIOS successfully assigns memory if
> > > > > > an End point requests BAR of size 16byte.
> > > > > >
> > > > > > But as per Spec:
> > > > > > The minimum memory address range requested by a BAR is 128
> > bytes.
> > > > >
> > > > > Can you provide the spec reference for this?  I don't see it in
> > > > > PCI r3.0.
> > > > >
> > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as
> > > > > writable, which would correspond to a minimum size of 16 bytes.
> > > >
> > > > The reference above is to the conventional PCI spec.  I happened
> > > > to trip
> > > over
> > > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint,
> > > > "the
> > > minimum
> > > > memory address range requested by a BAR is 128 bytes."
> > > >
> > > > I don't think linux currently enforces this minimum.
> > > >
> > >
> > > Hi Bjorn Thanks for the reply.
> > >
> > > Here is what the issue we are seeing.
> > >
> > > We have total memory for BAR's on our SoC of 256 MB.
> > > When an End Point request individually 16 byte BAR's our root port
> > > assigns memory to BAR's successfully.
> > >
> > > But if I have an End point which has 4 BAR's each 32 bit and request
> > > as
> > > following:
> > > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We
> > > are running this as part of SIG compliance test case) 2nd BAR
> > > requests 1MB and other 2 BAR's request 16byte, but these are not
> > > getting BAR's assigned. (Even though BAR space is available, since
> > > 1GB
> > failed,
> > > We have 256 MB still)
> > >
> > > We have only one End point connected to our root port.
> > >
> > > Here is the log:
> > > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> > > [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
> > > [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus
> > 00-
> > > ff]
> > > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> > > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> > > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> > > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> > > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> > > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
> > 0xefffffff]
> > > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> > > 0x7ffffffff pref]
> > > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> > > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> > > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> > > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size
> 0x60000000]
> > > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
> > 0xe01007ff
> > > pref]
> > > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> > > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size
> 0x40000000]
> > > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> > > 64bit]
> > > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size
> 0x00100000
> > > 64bit]
> > > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> > > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size
> 0x00000010]
> > > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> > > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size
> 0x00000010]
> > > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> > >
> > > Please let me know, what might might be the issue.
> > >
> >
> > Adding to the above will kernel allocate other memory BARs to an End
> > Point if one BAR assignment fails ?
> >
> > What if the End Point has multiple function and say first function BAR
> > assignment failed, will the kernel assign BAR's to second function on
> > same bus and device ?
> >
> After debugging in function pci_bus_alloc_from_region we have two
> resources, One which is taken as parameter (let's say resA), other is from
> pci_bus_for_each_resource (let's say resB).
> 
> The resA contains different start address from resB, I see that resB is
> obtained from device resource from bus->resource, but im unable to
> understand how resA (start address & size) gets its data.
> 
> Can any one help me understand this so that I might know reason why
> 16byte BAR allocation failing after 1GB request.
> 
> Regards,
> Bharat
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Adding yinghai lu.

> > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > > >
> > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > > Hi Bharat,
> > > > >
> > > > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm observing that on x86 BIOS successfully assigns memory if
> > > > > > an End point requests BAR of size 16byte.
> > > > > >
> > > > > > But as per Spec:
> > > > > > The minimum memory address range requested by a BAR is 128
> > bytes.
> > > > >
> > > > > Can you provide the spec reference for this?  I don't see it in
> > > > > PCI r3.0.
> > > > >
> > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as
> > > > > writable, which would correspond to a minimum size of 16 bytes.
> > > >
> > > > The reference above is to the conventional PCI spec.  I happened
> > > > to trip
> > > over
> > > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint,
> > > > "the
> > > minimum
> > > > memory address range requested by a BAR is 128 bytes."
> > > >
> > > > I don't think linux currently enforces this minimum.
> > > >
> > >
> > > Hi Bjorn Thanks for the reply.
> > >
> > > Here is what the issue we are seeing.
> > >
> > > We have total memory for BAR's on our SoC of 256 MB.
> > > When an End Point request individually 16 byte BAR's our root port
> > > assigns memory to BAR's successfully.
> > >
> > > But if I have an End point which has 4 BAR's each 32 bit and request
> > > as
> > > following:
> > > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We
> > > are running this as part of SIG compliance test case) 2nd BAR
> > > requests 1MB and other 2 BAR's request 16byte, but these are not
> > > getting BAR's assigned. (Even though BAR space is available, since
> > > 1GB
> > failed,
> > > We have 256 MB still)
> > >
> > > We have only one End point connected to our root port.
> > >
> > > Here is the log:
> > > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> > > [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
> > > [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus
> > 00-
> > > ff]
> > > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> > > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> > > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> > > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> > > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> > > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
> > 0xefffffff]
> > > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> > > 0x7ffffffff pref]
> > > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> > > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> > > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> > > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size
> 0x60000000]
> > > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
> > 0xe01007ff
> > > pref]
> > > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> > > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size
> 0x40000000]
> > > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> > > 64bit]
> > > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size
> 0x00100000
> > > 64bit]
> > > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> > > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size
> 0x00000010]
> > > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> > > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size
> 0x00000010]
> > > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> > >
> > > Please let me know, what might might be the issue.
> > >
> >
> > Adding to the above will kernel allocate other memory BARs to an End
> > Point if one BAR assignment fails ?
> >
> > What if the End Point has multiple function and say first function BAR
> > assignment failed, will the kernel assign BAR's to second function on
> > same bus and device ?
> >
> After debugging in function pci_bus_alloc_from_region we have two
> resources, One which is taken as parameter (let's say resA), other is from
> pci_bus_for_each_resource (let's say resB).
> 
> The resA contains different start address from resB, I see that resB is
> obtained from device resource from bus->resource, but im unable to
> understand how resA (start address & size) gets its data.
> 
> Can any one help me understand this so that I might know reason why
> 16byte BAR allocation failing after 1GB request.
> 
> Regards,
> Bharat
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Adding yinghai lu.

> > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > > >
> > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > > Hi Bharat,
> > > > >
> > > > > On Fri, Jul 22, 2016 at 09:24:22AM +0000, Bharat Kumar Gogada wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm observing that on x86 BIOS successfully assigns memory if
> > > > > > an End point requests BAR of size 16byte.
> > > > > >
> > > > > > But as per Spec:
> > > > > > The minimum memory address range requested by a BAR is 128
> > bytes.
> > > > >
> > > > > Can you provide the spec reference for this?  I don't see it in
> > > > > PCI r3.0.
> > > > >
> > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as
> > > > > writable, which would correspond to a minimum size of 16 bytes.
> > > >
> > > > The reference above is to the conventional PCI spec.  I happened
> > > > to trip
> > > over
> > > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint,
> > > > "the
> > > minimum
> > > > memory address range requested by a BAR is 128 bytes."
> > > >
> > > > I don't think linux currently enforces this minimum.
> > > >
> > >
> > > Hi Bjorn Thanks for the reply.
> > >
> > > Here is what the issue we are seeing.
> > >
> > > We have total memory for BAR's on our SoC of 256 MB.
> > > When an End Point request individually 16 byte BAR's our root port
> > > assigns memory to BAR's successfully.
> > >
> > > But if I have an End point which has 4 BAR's each 32 bit and request
> > > as
> > > following:
> > > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We
> > > are running this as part of SIG compliance test case) 2nd BAR
> > > requests 1MB and other 2 BAR's request 16byte, but these are not
> > > getting BAR's assigned. (Even though BAR space is available, since
> > > 1GB
> > failed,
> > > We have 256 MB still)
> > >
> > > We have only one End point connected to our root port.
> > >
> > > Here is the log:
> > > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
> > > [    2.319332] PCI host bridge /amba/pcie at fd0e0000 ranges:
> > > [    2.319349]   No bus range found for /amba/pcie at fd0e0000, using [bus
> > 00-
> > > ff]
> > > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
> > > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
> > > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
> > > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
> > > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
> > > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
> > 0xefffffff]
> > > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
> > > 0x7ffffffff pref]
> > > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
> > > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
> > > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
> > > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size
> 0x60000000]
> > > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
> > 0xe01007ff
> > > pref]
> > > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
> > > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size
> 0x40000000]
> > > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
> > > 64bit]
> > > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size
> 0x00100000
> > > 64bit]
> > > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
> > > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size
> 0x00000010]
> > > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
> > > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size
> 0x00000010]
> > > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
> > >
> > > Please let me know, what might might be the issue.
> > >
> >
> > Adding to the above will kernel allocate other memory BARs to an End
> > Point if one BAR assignment fails ?
> >
> > What if the End Point has multiple function and say first function BAR
> > assignment failed, will the kernel assign BAR's to second function on
> > same bus and device ?
> >
> After debugging in function pci_bus_alloc_from_region we have two
> resources, One which is taken as parameter (let's say resA), other is from
> pci_bus_for_each_resource (let's say resB).
> 
> The resA contains different start address from resB, I see that resB is
> obtained from device resource from bus->resource, but im unable to
> understand how resA (start address & size) gets its data.
> 
> Can any one help me understand this so that I might know reason why
> 16byte BAR allocation failing after 1GB request.
> 
> Regards,
> Bharat
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in the
> body of a message to majordomo at vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: Why does BIOS assign memory to 16 byte BAR

[Yinghai Lu]

On Tue, Jul 26, 2016 at 8:43 AM, Bharat Kumar Gogada
<bharat.kumar.gogada@xilinx.com> wrote:
>> > > We have only one End point connected to our root port.
>> > >
>> > > Here is the log:
>> > > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
>> > > [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
>> > > [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus
>> > 00-
>> > > ff]
>> > > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
>> > > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
>> > > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
>> > > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
>> > > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
>> > > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
>> > > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
>> > 0xefffffff]
>> > > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
>> > > 0x7ffffffff pref]
>> > > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
>> > > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
>> > > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
>> > > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size
>> 0x60000000]
>> > > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
>> > 0xe01007ff
>> > > pref]
>> > > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
>> > > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size
>> 0x40000000]
>> > > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
>> > > 64bit]
>> > > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size
>> 0x00100000
>> > > 64bit]
>> > > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
>> > > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size
>> 0x00000010]
>> > > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
>> > > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size
>> 0x00000010]
>> > > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
>> > >
>> >
>> > What if the End Point has multiple function and say first function BAR
>> > assignment failed, will the kernel assign BAR's to second function on
>> > same bus and device ?

Your system host bridge: has resource
pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
then one pci bridge:
pci 0000:00:00.0
then 0000:01:00.0 have four bars:
pci 0000:01:00.0: BAR 0:  [mem size 0x40000000]
pci 0000:01:00.0: BAR 4:  [mem size 0x00100000 64bit]
pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]


kernel need to get allocation for pci 0000:00:00.0 at first

but can not find big enough space.

pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
as it should come from [mem 0xe0100000-0xefffffff], and that is
less 1.5G.

so all children resource from pci 0000:01:00.0 all fail.


please check if you modify your FPGA code to make pci 0000:01:00.0

BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.

or you can check if can increase root bus mmio range

 MEM 0xe0100000..0xefffffff -> 0xe0100000
nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00

to have more than 1.5G.

Thanks

Yinghai

Re: Why does BIOS assign memory to 16 byte BAR

[Yinghai Lu]

On Tue, Jul 26, 2016 at 8:43 AM, Bharat Kumar Gogada
<bharat.kumar.gogada@xilinx.com> wrote:
>> > > We have only one End point connected to our root port.
>> > >
>> > > Here is the log:
>> > > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
>> > > [    2.319332] PCI host bridge /amba/pcie@fd0e0000 ranges:
>> > > [    2.319349]   No bus range found for /amba/pcie@fd0e0000, using [bus
>> > 00-
>> > > ff]
>> > > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
>> > > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
>> > > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
>> > > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
>> > > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
>> > > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
>> > > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
>> > 0xefffffff]
>> > > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
>> > > 0x7ffffffff pref]
>> > > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
>> > > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
>> > > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
>> > > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size
>> 0x60000000]
>> > > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
>> > 0xe01007ff
>> > > pref]
>> > > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
>> > > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size
>> 0x40000000]
>> > > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
>> > > 64bit]
>> > > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size
>> 0x00100000
>> > > 64bit]
>> > > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
>> > > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size
>> 0x00000010]
>> > > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
>> > > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size
>> 0x00000010]
>> > > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
>> > >
>> >
>> > What if the End Point has multiple function and say first function BAR
>> > assignment failed, will the kernel assign BAR's to second function on
>> > same bus and device ?

Your system host bridge: has resource
pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
then one pci bridge:
pci 0000:00:00.0
then 0000:01:00.0 have four bars:
pci 0000:01:00.0: BAR 0:  [mem size 0x40000000]
pci 0000:01:00.0: BAR 4:  [mem size 0x00100000 64bit]
pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]


kernel need to get allocation for pci 0000:00:00.0 at first

but can not find big enough space.

pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
as it should come from [mem 0xe0100000-0xefffffff], and that is
less 1.5G.

so all children resource from pci 0000:01:00.0 all fail.


please check if you modify your FPGA code to make pci 0000:01:00.0

BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.

or you can check if can increase root bus mmio range

 MEM 0xe0100000..0xefffffff -> 0xe0100000
nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00

to have more than 1.5G.

Thanks

Yinghai

Why does BIOS assign memory to 16 byte BAR

[Yinghai Lu]

On Tue, Jul 26, 2016 at 8:43 AM, Bharat Kumar Gogada
<bharat.kumar.gogada@xilinx.com> wrote:
>> > > We have only one End point connected to our root port.
>> > >
>> > > Here is the log:
>> > > [    2.319289] nwl-pcie fd0e0000.pcie: Link is UP
>> > > [    2.319332] PCI host bridge /amba/pcie at fd0e0000 ranges:
>> > > [    2.319349]   No bus range found for /amba/pcie at fd0e0000, using [bus
>> > 00-
>> > > ff]
>> > > [    2.319374]    IO 0xe0000000..0xe000ffff -> 0x00000000
>> > > [    2.319415]   MEM 0xe0100000..0xefffffff -> 0xe0100000
>> > > [    2.319431]   MEM 0x600000000..0x7ffffffff -> 0x600000000
>> > > [    2.319539] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
>> > > [    2.319557] pci_bus 0000:00: root bus resource [bus 00-ff]
>> > > [    2.319573] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
>> > > [    2.319589] pci_bus 0000:00: root bus resource [mem 0xe0100000-
>> > 0xefffffff]
>> > > [    2.319606] pci_bus 0000:00: root bus resource [mem 0x600000000-
>> > > 0x7ffffffff pref]
>> > > [    2.319845] pci 0000:00:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [    2.319861] iommu: Adding device 0000:00:00.0 to group 1
>> > > [    2.320243] pci 0000:01:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [    2.320258] iommu: Adding device 0000:01:00.0 to group 1
>> > > [    2.320313] pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
>> > > [    2.320331] pci 0000:00:00.0: BAR 8: failed to assign [mem size
>> 0x60000000]
>> > > [    2.320349] pci 0000:00:00.0: BAR 6: assigned [mem 0xe0100000-
>> > 0xe01007ff
>> > > pref]
>> > > [    2.320374] pci 0000:01:00.0: BAR 0: no space for [mem size 0x40000000]
>> > > [    2.320390] pci 0000:01:00.0: BAR 0: failed to assign [mem size
>> 0x40000000]
>> > > [    2.320407] pci 0000:01:00.0: BAR 4: no space for [mem size 0x00100000
>> > > 64bit]
>> > > [    2.320423] pci 0000:01:00.0: BAR 4: failed to assign [mem size
>> 0x00100000
>> > > 64bit]
>> > > [    2.320446] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00000010]
>> > > [    2.320461] pci 0000:01:00.0: BAR 2: failed to assign [mem size
>> 0x00000010]
>> > > [    2.320477] pci 0000:01:00.0: BAR 3: no space for [mem size 0x00000010]
>> > > [    2.320493] pci 0000:01:00.0: BAR 3: failed to assign [mem size
>> 0x00000010]
>> > > [    2.320509] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
>> > >
>> >
>> > What if the End Point has multiple function and say first function BAR
>> > assignment failed, will the kernel assign BAR's to second function on
>> > same bus and device ?

Your system host bridge: has resource
pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff]
pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
then one pci bridge:
pci 0000:00:00.0
then 0000:01:00.0 have four bars:
pci 0000:01:00.0: BAR 0:  [mem size 0x40000000]
pci 0000:01:00.0: BAR 4:  [mem size 0x00100000 64bit]
pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]


kernel need to get allocation for pci 0000:00:00.0 at first

but can not find big enough space.

pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000]
as it should come from [mem 0xe0100000-0xefffffff], and that is
less 1.5G.

so all children resource from pci 0000:01:00.0 all fail.


please check if you modify your FPGA code to make pci 0000:01:00.0

BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.

or you can check if can increase root bus mmio range

 MEM 0xe0100000..0xefffffff -> 0xe0100000
nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00

to have more than 1.5G.

Thanks

Yinghai

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Your system host bridge: has resource
> pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff] pci_bus
> 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref] then one pci
> bridge:
> pci 0000:00:00.0
> then 0000:01:00.0 have four bars:
> pci 0000:01:00.0: BAR 0:  [mem size 0x40000000] pci 0000:01:00.0: BAR 4:
> [mem size 0x00100000 64bit] pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
> pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]
> 
> 
> kernel need to get allocation for pci 0000:00:00.0 at first
> 
> but can not find big enough space.
> 
> pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000] as it should
> come from [mem 0xe0100000-0xefffffff], and that is less 1.5G.
> 
> so all children resource from pci 0000:01:00.0 all fail.
> 
> 
> please check if you modify your FPGA code to make pci 0000:01:00.0
> 
> BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> 
> or you can check if can increase root bus mmio range
> 
>  MEM 0xe0100000..0xefffffff -> 0xe0100000 nwl-pcie fd0e0000.pcie: PCI host
> bridge to bus 0000:00
> 
> to have more than 1.5G.
> 
Thanks Yinghai Lu.
We see that similar test is passing in x86 machine, where function one requesting 1GB BAR's is failing,
but function two requesting BAR's with 16byte is getting assigned BAR's.

To my knowledge on x86 BIOS assigns resources, or will kernel assign reosurces on x86 ?
If kernel does is there any difference between x86 and arm64 resource assignment logic ?

Thanks & Regards,
Bharat

  

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

PiBZb3VyIHN5c3RlbSBob3N0IGJyaWRnZTogaGFzIHJlc291cmNlDQo+IHBjaV9idXMgMDAwMDow
MDogcm9vdCBidXMgcmVzb3VyY2UgW21lbSAweGUwMTAwMDAwLTB4ZWZmZmZmZmZdIHBjaV9idXMN
Cj4gMDAwMDowMDogcm9vdCBidXMgcmVzb3VyY2UgW21lbSAweDYwMDAwMDAwMC0weDdmZmZmZmZm
ZiBwcmVmXSB0aGVuIG9uZSBwY2kNCj4gYnJpZGdlOg0KPiBwY2kgMDAwMDowMDowMC4wDQo+IHRo
ZW4gMDAwMDowMTowMC4wIGhhdmUgZm91ciBiYXJzOg0KPiBwY2kgMDAwMDowMTowMC4wOiBCQVIg
MDogIFttZW0gc2l6ZSAweDQwMDAwMDAwXSBwY2kgMDAwMDowMTowMC4wOiBCQVIgNDoNCj4gW21l
bSBzaXplIDB4MDAxMDAwMDAgNjRiaXRdIHBjaSAwMDAwOjAxOjAwLjA6IEJBUiAyOiAgW21lbSBz
aXplIDB4MDAwMDAwMTBdDQo+IHBjaSAwMDAwOjAxOjAwLjA6IEJBUiAzOiAgW21lbSBzaXplIDB4
MDAwMDAwMTBdDQo+IA0KPiANCj4ga2VybmVsIG5lZWQgdG8gZ2V0IGFsbG9jYXRpb24gZm9yIHBj
aSAwMDAwOjAwOjAwLjAgYXQgZmlyc3QNCj4gDQo+IGJ1dCBjYW4gbm90IGZpbmQgYmlnIGVub3Vn
aCBzcGFjZS4NCj4gDQo+IHBjaSAwMDAwOjAwOjAwLjA6IEJBUiA4OiBubyBzcGFjZSBmb3IgW21l
bSBzaXplIDB4NjAwMDAwMDBdIGFzIGl0IHNob3VsZA0KPiBjb21lIGZyb20gW21lbSAweGUwMTAw
MDAwLTB4ZWZmZmZmZmZdLCBhbmQgdGhhdCBpcyBsZXNzIDEuNUcuDQo+IA0KPiBzbyBhbGwgY2hp
bGRyZW4gcmVzb3VyY2UgZnJvbSBwY2kgMDAwMDowMTowMC4wIGFsbCBmYWlsLg0KPiANCj4gDQo+
IHBsZWFzZSBjaGVjayBpZiB5b3UgbW9kaWZ5IHlvdXIgRlBHQSBjb2RlIHRvIG1ha2UgcGNpIDAw
MDA6MDE6MDAuMA0KPiANCj4gQkFSIDAsIGFuZCBCQVIgNCB0byB1c2UgNjRiaXQgcHJlZiBpbnN0
ZWFkIG5vbi1wcmVmIG1taW8uDQo+IA0KPiBvciB5b3UgY2FuIGNoZWNrIGlmIGNhbiBpbmNyZWFz
ZSByb290IGJ1cyBtbWlvIHJhbmdlDQo+IA0KPiAgTUVNIDB4ZTAxMDAwMDAuLjB4ZWZmZmZmZmYg
LT4gMHhlMDEwMDAwMCBud2wtcGNpZSBmZDBlMDAwMC5wY2llOiBQQ0kgaG9zdA0KPiBicmlkZ2Ug
dG8gYnVzIDAwMDA6MDANCj4gDQo+IHRvIGhhdmUgbW9yZSB0aGFuIDEuNUcuDQo+IA0KVGhhbmtz
IFlpbmdoYWkgTHUuDQpXZSBzZWUgdGhhdCBzaW1pbGFyIHRlc3QgaXMgcGFzc2luZyBpbiB4ODYg
bWFjaGluZSwgd2hlcmUgZnVuY3Rpb24gb25lIHJlcXVlc3RpbmcgMUdCIEJBUidzIGlzIGZhaWxp
bmcsDQpidXQgZnVuY3Rpb24gdHdvIHJlcXVlc3RpbmcgQkFSJ3Mgd2l0aCAxNmJ5dGUgaXMgZ2V0
dGluZyBhc3NpZ25lZCBCQVIncy4NCg0KVG8gbXkga25vd2xlZGdlIG9uIHg4NiBCSU9TIGFzc2ln
bnMgcmVzb3VyY2VzLCBvciB3aWxsIGtlcm5lbCBhc3NpZ24gcmVvc3VyY2VzIG9uIHg4NiA/DQpJ
ZiBrZXJuZWwgZG9lcyBpcyB0aGVyZSBhbnkgZGlmZmVyZW5jZSBiZXR3ZWVuIHg4NiBhbmQgYXJt
NjQgcmVzb3VyY2UgYXNzaWdubWVudCBsb2dpYyA/DQoNClRoYW5rcyAmIFJlZ2FyZHMsDQpCaGFy
YXQNCg0KICANCg==

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Your system host bridge: has resource
> pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff] pci_bus
> 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref] then one pci
> bridge:
> pci 0000:00:00.0
> then 0000:01:00.0 have four bars:
> pci 0000:01:00.0: BAR 0:  [mem size 0x40000000] pci 0000:01:00.0: BAR 4:
> [mem size 0x00100000 64bit] pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
> pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]
> 
> 
> kernel need to get allocation for pci 0000:00:00.0 at first
> 
> but can not find big enough space.
> 
> pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000] as it should
> come from [mem 0xe0100000-0xefffffff], and that is less 1.5G.
> 
> so all children resource from pci 0000:01:00.0 all fail.
> 
> 
> please check if you modify your FPGA code to make pci 0000:01:00.0
> 
> BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> 
> or you can check if can increase root bus mmio range
> 
>  MEM 0xe0100000..0xefffffff -> 0xe0100000 nwl-pcie fd0e0000.pcie: PCI host
> bridge to bus 0000:00
> 
> to have more than 1.5G.
> 
Thanks Yinghai Lu.
We see that similar test is passing in x86 machine, where function one requesting 1GB BAR's is failing,
but function two requesting BAR's with 16byte is getting assigned BAR's.

To my knowledge on x86 BIOS assigns resources, or will kernel assign reosurces on x86 ?
If kernel does is there any difference between x86 and arm64 resource assignment logic ?

Thanks & Regards,
Bharat

  

Re: Why does BIOS assign memory to 16 byte BAR

[Lorenzo Pieralisi]

On Wed, Jul 27, 2016 at 06:33:29AM +0000, Bharat Kumar Gogada wrote:
> > Your system host bridge: has resource
> > pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff] pci_bus
> > 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref] then one pci
> > bridge:
> > pci 0000:00:00.0
> > then 0000:01:00.0 have four bars:
> > pci 0000:01:00.0: BAR 0:  [mem size 0x40000000] pci 0000:01:00.0: BAR 4:
> > [mem size 0x00100000 64bit] pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
> > pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]
> > 
> > 
> > kernel need to get allocation for pci 0000:00:00.0 at first
> > 
> > but can not find big enough space.
> > 
> > pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000] as it should
> > come from [mem 0xe0100000-0xefffffff], and that is less 1.5G.
> > 
> > so all children resource from pci 0000:01:00.0 all fail.
> > 
> > 
> > please check if you modify your FPGA code to make pci 0000:01:00.0
> > 
> > BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> > 
> > or you can check if can increase root bus mmio range
> > 
> >  MEM 0xe0100000..0xefffffff -> 0xe0100000 nwl-pcie fd0e0000.pcie: PCI host
> > bridge to bus 0000:00
> > 
> > to have more than 1.5G.
> > 
> Thanks Yinghai Lu.
> We see that similar test is passing in x86 machine, where function one
> requesting 1GB BAR's is failing, but function two requesting BAR's
> with 16byte is getting assigned BAR's.
> 
> To my knowledge on x86 BIOS assigns resources, or will kernel assign
> reosurces on x86 ?  If kernel does is there any difference between x86
> and arm64 resource assignment logic ?

We can't answer your question if you do not provide a full log
of x86 and ARM PCI configurations you are testing I am afraid.

It is also unclear to me what "a similar test is passing in x86
machine" means, in particular in relation to the HW configuration
you are testing on x86.

Yes, there are differences between x86 and ARM resources assignments,
x86 tries to claim PCI resources as set-up by BIOS and assign them
iff the claiming fails whereas on ARM (and that's done in the host
bridge driver) FW configuration is always discarded and the kernel
reassigns the whole PCI resource hierarchy entirely, but to help we
need more data as I said above.

I suspect the issue you are facing has to do as Yinghai mentioned
with the prefetchable memory window set-up, saying that "it works"
on x86 does not really help unless you provide data to debug it.

Thanks,
Lorenzo

Re: Why does BIOS assign memory to 16 byte BAR

[Lorenzo Pieralisi]

On Wed, Jul 27, 2016 at 06:33:29AM +0000, Bharat Kumar Gogada wrote:
> > Your system host bridge: has resource
> > pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff] pci_bus
> > 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref] then one pci
> > bridge:
> > pci 0000:00:00.0
> > then 0000:01:00.0 have four bars:
> > pci 0000:01:00.0: BAR 0:  [mem size 0x40000000] pci 0000:01:00.0: BAR 4:
> > [mem size 0x00100000 64bit] pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
> > pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]
> > 
> > 
> > kernel need to get allocation for pci 0000:00:00.0 at first
> > 
> > but can not find big enough space.
> > 
> > pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000] as it should
> > come from [mem 0xe0100000-0xefffffff], and that is less 1.5G.
> > 
> > so all children resource from pci 0000:01:00.0 all fail.
> > 
> > 
> > please check if you modify your FPGA code to make pci 0000:01:00.0
> > 
> > BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> > 
> > or you can check if can increase root bus mmio range
> > 
> >  MEM 0xe0100000..0xefffffff -> 0xe0100000 nwl-pcie fd0e0000.pcie: PCI host
> > bridge to bus 0000:00
> > 
> > to have more than 1.5G.
> > 
> Thanks Yinghai Lu.
> We see that similar test is passing in x86 machine, where function one
> requesting 1GB BAR's is failing, but function two requesting BAR's
> with 16byte is getting assigned BAR's.
> 
> To my knowledge on x86 BIOS assigns resources, or will kernel assign
> reosurces on x86 ?  If kernel does is there any difference between x86
> and arm64 resource assignment logic ?

We can't answer your question if you do not provide a full log
of x86 and ARM PCI configurations you are testing I am afraid.

It is also unclear to me what "a similar test is passing in x86
machine" means, in particular in relation to the HW configuration
you are testing on x86.

Yes, there are differences between x86 and ARM resources assignments,
x86 tries to claim PCI resources as set-up by BIOS and assign them
iff the claiming fails whereas on ARM (and that's done in the host
bridge driver) FW configuration is always discarded and the kernel
reassigns the whole PCI resource hierarchy entirely, but to help we
need more data as I said above.

I suspect the issue you are facing has to do as Yinghai mentioned
with the prefetchable memory window set-up, saying that "it works"
on x86 does not really help unless you provide data to debug it.

Thanks,
Lorenzo

Why does BIOS assign memory to 16 byte BAR

[Lorenzo Pieralisi]

On Wed, Jul 27, 2016 at 06:33:29AM +0000, Bharat Kumar Gogada wrote:
> > Your system host bridge: has resource
> > pci_bus 0000:00: root bus resource [mem 0xe0100000-0xefffffff] pci_bus
> > 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref] then one pci
> > bridge:
> > pci 0000:00:00.0
> > then 0000:01:00.0 have four bars:
> > pci 0000:01:00.0: BAR 0:  [mem size 0x40000000] pci 0000:01:00.0: BAR 4:
> > [mem size 0x00100000 64bit] pci 0000:01:00.0: BAR 2:  [mem size 0x00000010]
> > pci 0000:01:00.0: BAR 3:  [mem size 0x00000010]
> > 
> > 
> > kernel need to get allocation for pci 0000:00:00.0 at first
> > 
> > but can not find big enough space.
> > 
> > pci 0000:00:00.0: BAR 8: no space for [mem size 0x60000000] as it should
> > come from [mem 0xe0100000-0xefffffff], and that is less 1.5G.
> > 
> > so all children resource from pci 0000:01:00.0 all fail.
> > 
> > 
> > please check if you modify your FPGA code to make pci 0000:01:00.0
> > 
> > BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> > 
> > or you can check if can increase root bus mmio range
> > 
> >  MEM 0xe0100000..0xefffffff -> 0xe0100000 nwl-pcie fd0e0000.pcie: PCI host
> > bridge to bus 0000:00
> > 
> > to have more than 1.5G.
> > 
> Thanks Yinghai Lu.
> We see that similar test is passing in x86 machine, where function one
> requesting 1GB BAR's is failing, but function two requesting BAR's
> with 16byte is getting assigned BAR's.
> 
> To my knowledge on x86 BIOS assigns resources, or will kernel assign
> reosurces on x86 ?  If kernel does is there any difference between x86
> and arm64 resource assignment logic ?

We can't answer your question if you do not provide a full log
of x86 and ARM PCI configurations you are testing I am afraid.

It is also unclear to me what "a similar test is passing in x86
machine" means, in particular in relation to the HW configuration
you are testing on x86.

Yes, there are differences between x86 and ARM resources assignments,
x86 tries to claim PCI resources as set-up by BIOS and assign them
iff the claiming fails whereas on ARM (and that's done in the host
bridge driver) FW configuration is always discarded and the kernel
reassigns the whole PCI resource hierarchy entirely, but to help we
need more data as I said above.

I suspect the issue you are facing has to do as Yinghai mentioned
with the prefetchable memory window set-up, saying that "it works"
on x86 does not really help unless you provide data to debug it.

Thanks,
Lorenzo

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> > We see that similar test is passing in x86 machine, where function one
> > requesting 1GB BAR's is failing, but function two requesting BAR's
> > with 16byte is getting assigned BAR's.
> >
> > To my knowledge on x86 BIOS assigns resources, or will kernel assign
> > reosurces on x86 ?  If kernel does is there any difference between x86
> > and arm64 resource assignment logic ?
>
> We can't answer your question if you do not provide a full log of x86 and
> ARM PCI configurations you are testing I am afraid.
>
> It is also unclear to me what "a similar test is passing in x86 machine" means,
> in particular in relation to the HW configuration you are testing on x86.
>
> Yes, there are differences between x86 and ARM resources assignments,
> x86 tries to claim PCI resources as set-up by BIOS and assign them iff the
> claiming fails whereas on ARM (and that's done in the host bridge driver) FW
> configuration is always discarded and the kernel reassigns the whole PCI
> resource hierarchy entirely, but to help we need more data as I said above.
>

Thanks Lorenzo.
I will try to get x86 configuration details soon and post, as the setup is at different location.
On ARM we using pcie-xilinx-nwl.c configurations where we have where we have 240MB BAR space
as per device tree documentation. (The kernel log I posted for testing purpose is using different address spaces,
if tests were successful, we will soon send patches for device tree)

Similar test case means, on x86 where we have multifunction device, where 1st function requests 1GB memory BAR's
and 2nd function requests 16byte memory BAR's, for 1st function BAR's are not assigned, but for 2nd function 16byte BAR assignment
is successful.

But on ARM similar test case fails to assign BAR's for both first function and second function.

What is the reason on ARM we are handling resource assignment in hierarchy format?
Because on ARM, in the above case due to lack of resources first function BAR assignment fails  which is fine but
even though we have enough resources for 16byte BAR allocations for second function, we are not assigning.


Thanks & Regards,
Bharat


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> > We see that similar test is passing in x86 machine, where function one
> > requesting 1GB BAR's is failing, but function two requesting BAR's
> > with 16byte is getting assigned BAR's.
> >
> > To my knowledge on x86 BIOS assigns resources, or will kernel assign
> > reosurces on x86 ?  If kernel does is there any difference between x86
> > and arm64 resource assignment logic ?
>
> We can't answer your question if you do not provide a full log of x86 and
> ARM PCI configurations you are testing I am afraid.
>
> It is also unclear to me what "a similar test is passing in x86 machine" =
means,
> in particular in relation to the HW configuration you are testing on x86.
>
> Yes, there are differences between x86 and ARM resources assignments,
> x86 tries to claim PCI resources as set-up by BIOS and assign them iff th=
e
> claiming fails whereas on ARM (and that's done in the host bridge driver)=
 FW
> configuration is always discarded and the kernel reassigns the whole PCI
> resource hierarchy entirely, but to help we need more data as I said abov=
e.
>

Thanks Lorenzo.
I will try to get x86 configuration details soon and post, as the setup is =
at different location.
On ARM we using pcie-xilinx-nwl.c configurations where we have where we hav=
e 240MB BAR space
as per device tree documentation. (The kernel log I posted for testing purp=
ose is using different address spaces,
if tests were successful, we will soon send patches for device tree)

Similar test case means, on x86 where we have multifunction device, where 1=
st function requests 1GB memory BAR's
and 2nd function requests 16byte memory BAR's, for 1st function BAR's are n=
ot assigned, but for 2nd function 16byte BAR assignment
is successful.

But on ARM similar test case fails to assign BAR's for both first function =
and second function.

What is the reason on ARM we are handling resource assignment in hierarchy =
format?
Because on ARM, in the above case due to lack of resources first function B=
AR assignment fails  which is fine but
even though we have enough resources for 16byte BAR allocations for second =
function, we are not assigning.


Thanks & Regards,
Bharat


This email and any attachments are intended for the sole use of the named r=
ecipient(s) and contain(s) confidential information that may be proprietary=
, privileged or copyrighted under applicable law. If you are not the intend=
ed recipient, do not read, copy, or forward this email message or any attac=
hments. Delete this email message and any attachments immediately.

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> > We see that similar test is passing in x86 machine, where function one
> > requesting 1GB BAR's is failing, but function two requesting BAR's
> > with 16byte is getting assigned BAR's.
> >
> > To my knowledge on x86 BIOS assigns resources, or will kernel assign
> > reosurces on x86 ?  If kernel does is there any difference between x86
> > and arm64 resource assignment logic ?
>
> We can't answer your question if you do not provide a full log of x86 and
> ARM PCI configurations you are testing I am afraid.
>
> It is also unclear to me what "a similar test is passing in x86 machine" means,
> in particular in relation to the HW configuration you are testing on x86.
>
> Yes, there are differences between x86 and ARM resources assignments,
> x86 tries to claim PCI resources as set-up by BIOS and assign them iff the
> claiming fails whereas on ARM (and that's done in the host bridge driver) FW
> configuration is always discarded and the kernel reassigns the whole PCI
> resource hierarchy entirely, but to help we need more data as I said above.
>

Thanks Lorenzo.
I will try to get x86 configuration details soon and post, as the setup is at different location.
On ARM we using pcie-xilinx-nwl.c configurations where we have where we have 240MB BAR space
as per device tree documentation. (The kernel log I posted for testing purpose is using different address spaces,
if tests were successful, we will soon send patches for device tree)

Similar test case means, on x86 where we have multifunction device, where 1st function requests 1GB memory BAR's
and 2nd function requests 16byte memory BAR's, for 1st function BAR's are not assigned, but for 2nd function 16byte BAR assignment
is successful.

But on ARM similar test case fails to assign BAR's for both first function and second function.

What is the reason on ARM we are handling resource assignment in hierarchy format?
Because on ARM, in the above case due to lack of resources first function BAR assignment fails  which is fine but
even though we have enough resources for 16byte BAR allocations for second function, we are not assigning.


Thanks & Regards,
Bharat


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.