Hi Bjorn,

On 10/2/2017 12:35 PM, Bjorn Helgaas wrote:
> On Fri, Sep 29, 2017 at 05:47:23PM -0700, Ray Jui wrote:
>> Hi Bjorn and all,
>>
>> One of our ARM64 based processor (NS2) has our system address
>> reversed for PCIe outbound transaction in the range of 0x0000_0000
>> and 0x1fff_ffff. If we use 1:1 mapping between the system address
>> and PCIe address, we end up having the PCIe address in the same
>> range.
>>
>> In this case, we noticed that certain USB-PCI adapters will decline
>> access to BARs mapped in this address range with "Unsupported
>> Request". In this case, if we change the PCIe address range to
>> something else like 0x8000_0000, it would work without an issue.
>>
>> This sounds like an issue with the USB PCI adapter that cannot allow
>> BARs to be used with specific PCIe address range. Is my
>> understanding correct?
> 
> I assume you're seeing "Unsupported Request" completions on the PCIe
> link?  Can you tell with an analyzer that the TLP actually appears on
> the link and it's the device that is responding with UR?  I'm just
> wondering if there's something in the RC that could be
> short-circuiting this.  But I guess you mention "certain USB-PCI
> adapters", which implies that these BAR addresses work for *some*
> devices, just not these particular USB-PCI ones.

Yes, UR is observed with a PCIe analyzer. See attached two JPGs for 
captures from PCIe analyzer (one uses PCIe address of 0x0000_0000 and 
the other uses 0x8000_0000).

And yes, this behavior is only seen with certain PCIe-USB adapters. Most 
other PCIe endpoint devices (NVMe SSDs, Ethernet dongle, PCIe switches, 
and etc) work fine with PICe address of 0x0000_0000.

> 
> AFAIK, there's no PCIe reason why the 0x0000_0000 - 0x1fff_ffff
> region shouldn't be used for BARs.
> 

Okay, thanks for helping to confirm that.

> It's surprising to me that a device would special-case addresses like
> that, but maybe it does.

If this is proven to be a shortcoming of some specific PCIe-USB adapters 
(and in fact it seems to be the case so far based on all of the data 
points). Does it make any sense to you to have a workaround done by the 
root complex, i.e., to avoid using 0x0000_0000 outbound region; instead, 
use something else, .e.g, 0x1000_0000

> 
> Bjorn
> 

Thanks,

Ray