Re: ARM Max Read Req Size and PCIE_BUS_PERFORMANCE stories

["Krzysztof Hałasa" <khalasa@piap.pl>]

Hi Bjorn,

Bjorn Helgaas <helgaas@kernel.org> writes:

> Super.  IIUC, i.MX6 is another DWC-based controller, so this looks
> like another case of the issue that afflicts amlogic, keystone,
> loongson (weirdly apparently *not* DWC-based), meson, and probably
> others.
>
> Some previous discussion here:
> https://lore.kernel.org/linux-pci/20210707155418.GA897940@bjorn-Precision-5520/

Ok. So I guess the fix for i.MX6 is a simple PCI fixup, limiting MRRS to
512 (while MPS = 128).

Now the Kconfig states that MRRS=MPS is the "best performance" case. Is
it really true? One could think requesting 512 bytes (memory read)
completed with 4 response packets 128 bytes each could be faster than
4 requests and 4 responses. Various docs suggest that lowering MRRS is
good for "interactivity", but not exactly for throughput.
Should I do some benchmarking?

> This sounds like a defect in the CPU/PCI host adapter.  If the device
> initiates a 4096-byte read and the CPU or whatever can't deal with it,
> the host adapter should break it up into whatever the CPU *can*
> handle.  I don't think this is the device's problem or the device
> driver's problem.

As I see it, this is not a CPU (ARM core, IXI bus etc) problem. This is
DWC PCIe thing, and the cause is apparently the size of its buffers.
That's why the max number of tags (which I assume is a number of
individual read requests) depends on the MRRS size.

The PCIe 3 docs state:
Max_Read_Request_Size – This field sets the maximum Read
Request size for the Function as a Requester. The Function
must not generate Read Requests with a size exceeding the set
value.

Not sure about the "set value", but perhaps a limit (lower than 4096) is
permitted - e.g. for the whole system or bus.

The i.MX6 errata list states:
"ERR003754 PCIe: 9000403702—AHB/AXI Bridge Master responds with UR
status instead of CA status for inbound MRd requesting greater than
CX_REMOTE_RD_REQ_SIZE

Description:
         The AHB/AXI Bridge RAM is sized at configuration time to support inbound read requests with a
         maximum size of CX_REMOTE_RD_REQ_SIZE. When this limit is violated the core responds
         with UR status, when it should respond with CA status."

The above suggests that aborting an oversized read request is ok. It
should be done with a CA (Completer Abort) code rather than UR
(Unsupported Request), but that's just a difference in the error code.

>> 2. should the PCI code limit MRRS to MPS by default?
>> 3. should the PCI code limit MRRS to the maximum safe value (512 on
>>    this CPU)?
>
> How do we learn the maximum safe value?  Is this something a native
> driver could read from PCIE_PL_MRCCR0 (see below)?

It will read "512" by default (unless maybe some boot loader etc.
changed it).
I think, for these particular SoCs, a fixed 512 would do.
But perhaps we could experiment with larger values, which need to be
*written* to this register before use.

I will think about it.
-- 
Krzysztof "Chris" Hałasa

Sieć Badawcza Łukasiewicz
Przemysłowy Instytut Automatyki i Pomiarów PIAP
Al. Jerozolimskie 202, 02-486 Warszawa