Ice Lake PCIe Read TLPs with random RequesterIDs (RIDs)

Ice Lake PCIe Read TLPs with random RequesterIDs (RIDs)

[Eric Pilmore]

I am curious to know if anybody working with Ice Lake has seen this
behavior where the MemRD TLPs, from a "cpu(s)" issuing I/O Reads
to an I/O device, are carrying varying RequesterID (RID) values that
don't represent any enumerated devices (per lspci) in the system?
The particular system in use is a Supermicro box with 2 CPU sockets,
each socket comprised of a 16 core (2 thread/core) processor. I have
not seen any correlation between issuing the I/O Read operation from
a particular core and the RID that shows up in the respective MemRD
TLP. Normally, I would have expected the RID to have been either 0x0000
or the BDF of the respective Root Port into which the device being
read was plugged into.

I assume this behavior is intended to increase I/O bandwidth and reduce
latency by allowing even more outstanding I/O Reads than is normally
afforded by the (per-device) Tag field of a MemRD TLP.

I have been searching around, but having trouble finding specifics about
this behavior in any specs or micro-architecture documents. If anybody
knows of any pointers it would be greatly appreciated.

Thanks in advance!

Eric



-- 
Eric Pilmore
epilmore@gigaio.com
http://gigaio.com
Phone: (858) 775 2514

This e-mail message is intended only for the individual(s) to whom
it is addressed and may contain information that is privileged,
confidential, proprietary, or otherwise exempt from disclosure under
applicable law. If you believe you have received this message in
error, please advise the sender by return e-mail and delete it from
your mailbox.
Thank you.

Re: Ice Lake PCIe Read TLPs with random RequesterIDs (RIDs)

[Bjorn Helgaas]

[+cc Ashok]

On Wed, Sep 29, 2021 at 01:40:41PM -0700, Eric Pilmore wrote:
> I am curious to know if anybody working with Ice Lake has seen this
> behavior where the MemRD TLPs, from a "cpu(s)" issuing I/O Reads
> to an I/O device, are carrying varying RequesterID (RID) values that
> don't represent any enumerated devices (per lspci) in the system?
> The particular system in use is a Supermicro box with 2 CPU sockets,
> each socket comprised of a 16 core (2 thread/core) processor. I have
> not seen any correlation between issuing the I/O Read operation from
> a particular core and the RID that shows up in the respective MemRD
> TLP. Normally, I would have expected the RID to have been either 0x0000
> or the BDF of the respective Root Port into which the device being
> read was plugged into.
> 
> I assume this behavior is intended to increase I/O bandwidth and reduce
> latency by allowing even more outstanding I/O Reads than is normally
> afforded by the (per-device) Tag field of a MemRD TLP.
> 
> I have been searching around, but having trouble finding specifics about
> this behavior in any specs or micro-architecture documents. If anybody
> knows of any pointers it would be greatly appreciated.

I don't know anything about this.  Added Ashok in case he can shed any
light.