Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device

[Sinan Kaya <okaya@codeaurora.org>]

On 1/22/2018 5:51 PM, Bjorn Helgaas wrote:
> On Mon, Jan 22, 2018 at 05:04:03PM -0500, Sinan Kaya wrote:
>> On 1/22/2018 4:36 PM, Bjorn Helgaas wrote:

>>> Reducing MPS may be necessary if there are several devices in the
>>> hierarchy and one requires a smaller MPS than the others.  That
>>> obviously reduces the maximum read and write performance.
>>>
>>> Reducing the MRRS may be useful to prevent one device from hogging
>>> a link, but of course, it reduces read performance for that device
>>> because we need more read requests.
>>
>> Maybe, a picture could help.
>>
>>                root (MPS=256)
>>                  |
>>          ------------------
>>         /                  \
>>    bridge0 (MPS=256)      bridge1 (MPS=128)
>>       /                       \
>>     EP0 (MPS=256)            EP1 (MPS=128)
>>
>> If I understood this right, code allows the configuration above with
>> the performance mode so that MPS doesn't have to be uniform across
>> the tree. 
> 
> Yes.  In PERFORMANCE mode, we will set EP1's MRRS=128 and
> EP0's MRRS=256, just as you show.
> 
>> It just needs to be consistent between the root port and endpoints.
> 
> No, it doesn't need to be consistent.  In PERFORMANCE mode, we'll set
> the root's MPS=256 and EP1's MPS=128.
> 
> (I'm not actually 100% convinced that the PERFORMANCE mode approach of
> reducing MRRS is safe, necessary, and maintainable.  I suspect that in
> many of the interesting cases, the device we care about is the only
> one below a Root Port, and we can get the performance we need by
> maximizing MPS and MRRS for that Root Port and its children,
> independent of the rest of the system.)

Maybe, I started seeing more and more NVMe devices behind a switch every
day. That's why, I'm concerned.

> 
>> Why are we reducing MRRS in this case?
> 
> We have to set EP1's MRRS=128 so it will never receive a completion
> larger than 128 bytes.  If we set EP1's MRRS=256, it could receive
> 256-byte TLPs, which it would treat as malformed.  (We also assume no
> peer-to-peer DMA that targets EP1.)

What if we were to keep root port MPS as 128? and not touch the BIOS
configured MRRS (4k) ?

Everybody should be happy, right?

I know there is a rule to check the completions against MPS. Root port
could generate transactions that is a multiple of 128 bytes for reads.

Is there any rule against checking incoming writes?

> 
>> Are we assuming that root bus cannot handle more than 256 bytes and
>> bridge1 would be starved while root bus is passing the completions
>> to bridge0?
> 
> We don't have to assume.  Every device tells us via Dev Cap what size
> TLPs it can handle.  In your example, I assume the root's Dev Cap
> tells us it supports 256-byte TLPs.
> 
> PERFORMANCE mode reduces MRRS not because of a starvation issue, but
> because reducing EP1's MRRS allows EP0 to use a larger MPS.
> 


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.