64bit BARs: prefetchable only?

64bit BARs: prefetchable only?

[Rajat Jain]

Hello,

I'm working on a system that can potentially have a large amount of PCI memory requirements (say 8GB or more) for the memory mapped PCI device registers. Since these are registers, I want non-prefetchable memory windows and not prefetchable ones.

Now, I'm taking a look at the PCI-to-PCI bridge architecture specifications, and I see that for bridges (downstream ports / root oorts etc), while there are 64 bits available to specify the Prefetchable memory base and limit, the (non-prefetchable) memory base and limit register are only 32 bit. I looked at various different specs related to PCI, but I could not find anything that could help me understand how to program a bridge so that it forwards a transaction to a 64bit BAR, using the Non-prefetchable windows.

My questions: 

1) Is there any way possible to have a non-prefetchable memory, greater than 4GB, programmed on a PCI bridge?

2) Is it possible to have to program a non-prefetchable window, may be small size (say 1MB), but starts at a PCI address ABOVE 4GB?

3) How do the 64 bit BARs behave. Does the PCI standard requires all 64 bit BARs to be only mapped using prefetchable regions?

I tried to find the above answers in the PCI spec, the PCI-to-PCI bridge spec, and the PCI express spec but could not find. I'd appreciate if some one could point out to any references.

Thanks,

Rajat

RE: 64bit BARs: prefetchable only?

[Jake Oshins]

> Hello,
> 
> I'm working on a system that can potentially have a large amount of PCI
> memory requirements (say 8GB or more) for the memory mapped PCI device
> registers. Since these are registers, I want non-prefetchable memory windows
> and not prefetchable ones.
> 
> Now, I'm taking a look at the PCI-to-PCI bridge architecture specifications, and I
> see that for bridges (downstream ports / root oorts etc), while there are 64 bits
> available to specify the Prefetchable memory base and limit, the (non-
> prefetchable) memory base and limit register are only 32 bit. I looked at
> various different specs related to PCI, but I could not find anything that could
> help me understand how to program a bridge so that it forwards a transaction
> to a 64bit BAR, using the Non-prefetchable windows.
> 
> My questions:
> 
> 1) Is there any way possible to have a non-prefetchable memory, greater than
> 4GB, programmed on a PCI bridge?
> 

No, there isn't, but it probably doesn't matter in practice.  You're almost certainly dealing with PCI Express here, and the concept of prefetchable doesn't apply to Express.  Bridges aren't allowed to buffer reads with Express, and so nothing is actually prefetched.  The prefetchable bit in the BAR is legacy from PCI, which did involve buffering in bridges.


> 2) Is it possible to have to program a non-prefetchable window, may be small
> size (say 1MB), but starts at a PCI address ABOVE 4GB?
> 

No, but see the answer above.

> 3) How do the 64 bit BARs behave. Does the PCI standard requires all 64 bit
> BARs to be only mapped using prefetchable regions?
> 

No, it doesn't.


Cheers,
Jake Oshins

RE: 64bit BARs: prefetchable only?

[Rajat Jain]

Hello Jake,

Thanks for your response. 

> -----Original Message-----
> From: Jake Oshins [mailto:jakeo@microsoft.com]
> Sent: Monday, February 09, 2015 1:25 PM
> To: Rajat Jain; linux-pci@vger.kernel.org
> Cc: Stu Grossman; Guenter Roeck
> Subject: RE: 64bit BARs: prefetchable only?
> 
> > Hello,
> >
> > I'm working on a system that can potentially have a large amount of
> > PCI memory requirements (say 8GB or more) for the memory mapped PCI
> > device registers. Since these are registers, I want non-prefetchable
> > memory windows and not prefetchable ones.
> >
> > Now, I'm taking a look at the PCI-to-PCI bridge architecture
> > specifications, and I see that for bridges (downstream ports / root
> > oorts etc), while there are 64 bits available to specify the
> > Prefetchable memory base and limit, the (non-
> > prefetchable) memory base and limit register are only 32 bit. I looked
> > at various different specs related to PCI, but I could not find
> > anything that could help me understand how to program a bridge so that
> > it forwards a transaction to a 64bit BAR, using the Non-prefetchable
> windows.
> >
> > My questions:
> >
> > 1) Is there any way possible to have a non-prefetchable memory,
> > greater than 4GB, programmed on a PCI bridge?
> >
> 
> No, there isn't, but it probably doesn't matter in practice.  You're almost
> certainly dealing with PCI Express here

Yes, I am.

> , and the concept of prefetchable
> doesn't apply to Express.  Bridges aren't allowed to buffer reads with
> Express, and so nothing is actually prefetched.

Can you please point me to the section in the PCI Express spec that mandates this? The only reference I could find in the spec to this was in an "implementation note" below:

========================================================================
"Implementation Note: Additional Guidance on the Prefetchable Bit in Memory Space BARs"
=======================================================================
...
<snip>
...
On PCI Express systems that meet the criteria enumerated below, setting the Prefetchable bit in a
candidate BAR will still permit correct operation even if the BAR's range includes some locations
that have read side-effects or cannot tolerate write merging. This is primarily due to the fact that
PCI Express Memory Reads always contain an explicit length, and PCI Express Switches never
prefetch or do byte merging. Generally only 64-bit BARs are good candidates, since only Legacy
Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most scalable platforms map
all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the Prefetchable bit
value.
...
<snip>
...
========================================================================

I see that this has been mentioned matter-of-factly but I'm assuming that this should probably be spelled out more loudly somewhere else?

>  The prefetchable bit in the
> BAR is legacy from PCI, which did involve buffering in bridges.
> 
> 
> > 2) Is it possible to have to program a non-prefetchable window, may be
> > small size (say 1MB), but starts at a PCI address ABOVE 4GB?
> >
> 
> No, but see the answer above.
> 
> > 3) How do the 64 bit BARs behave. Does the PCI standard requires all
> > 64 bit BARs to be only mapped using prefetchable regions?
> >
> 
> No, it doesn't.


I did not get this. How does it work if I map a 64 bit BAR using non-prefetchable region, if there is no way for the bridges to forward the traffic to it (The memory base and limit registers are only 32 bit)?

Thanks,

Rajat

> 
> 
> Cheers,
> Jake Oshins

RE: 64bit BARs: prefetchable only?

[Jake Oshins]

> -----Original Message-----
> From: Rajat Jain [mailto:rajatjain@juniper.net]
> Sent: Tuesday, February 10, 2015 1:52 PM
> To: Jake Oshins; linux-pci@vger.kernel.org
> Cc: Stu Grossman; Guenter Roeck
> Subject: RE: 64bit BARs: prefetchable only?
> 
> > , and the concept of prefetchable
> > doesn't apply to Express.  Bridges aren't allowed to buffer reads with
> > Express, and so nothing is actually prefetched.
> 
> Can you please point me to the section in the PCI Express spec that mandates
> this? The only reference I could find in the spec to this was in an
> "implementation note" below:
> 
> ================================================================
> ========
> "Implementation Note: Additional Guidance on the Prefetchable Bit in Memory
> Space BARs"
> ================================================================
> =======
> ...
> <snip>
> ...
> On PCI Express systems that meet the criteria enumerated below, setting the
> Prefetchable bit in a
> candidate BAR will still permit correct operation even if the BAR's range
> includes some locations
> that have read side-effects or cannot tolerate write merging. This is primarily
> due to the fact that
> PCI Express Memory Reads always contain an explicit length, and PCI Express
> Switches never
> prefetch or do byte merging. Generally only 64-bit BARs are good candidates,
> since only Legacy
> Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most
> scalable platforms map
> all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the
> Prefetchable bit
> value.
> ...
> <snip>
> ...
> ================================================================
> ========

This is exactly the section that I would have pointed you to.  PCI Express switches are called switches rather than bridges because they do no buffering.  When your endpoint has credits to put TLPs on the fabric, that means that those TLPs are going to pass all the way across the fabric.  Your endpoint will never have credits when this can't happen.  Thus there's no notion of prefetching.  If you want a more detailed discussion than the implementation note above, read the sections on how an endpoint is granted the right to send TLPs.

> I did not get this. How does it work if I map a 64 bit BAR using non-prefetchable
> region, if there is no way for the bridges to forward the traffic to it (The
> memory base and limit registers are only 32 bit)?
> 

If you map a 64-bit BAR into a non-prefetchable region, that will involve writing zeros to all the bits in the upper part of the 64-bit BAR, so that it falls within the bridge windows.  (This is strictly true only for endpoints which are not embedded in the root complex.  These can appear on the root bus, and thus they might have addresses above 4GB in their BARs.)

And yes, in my opinion, this makes the requirement that all BARs be 64-bit capable (on Express) a little silly.

-- Jake Oshins