PCI: latency

PCI: latency

[Subhashini Rao Beerisetty]

 [ Please keep me in CC as I'm not subscribed to the list]

Hi all,

We are using the Linux OS on an x86_64 machine. I need to measure the
PCIe latency on my system, does kernel have any latency measurement
module for the PCIe bus?

Thanks,

Re: PCI: latency

[Greg KH]

On Fri, Dec 03, 2021 at 10:30:58PM +0530, Subhashini Rao Beerisetty wrote:
>  [ Please keep me in CC as I'm not subscribed to the list]
> 
> Hi all,
> 
> We are using the Linux OS on an x86_64 machine. I need to measure the
> PCIe latency on my system, does kernel have any latency measurement
> module for the PCIe bus?

How exactly are you going to define PCIe latency?

And what is this going to be used for?

thanks,

greg k-h

RE: latency

[David Laight]

From: Subhashini Rao Beerisetty <subhashbeerisetty@gmail.com>
> Sent: 03 December 2021 17:01
> 
>  [ Please keep me in CC as I'm not subscribed to the list]
> 
> Hi all,
> 
> We are using the Linux OS on an x86_64 machine. I need to measure the
> PCIe latency on my system, does kernel have any latency measurement
> module for the PCIe bus?

Slower than you expect :-)

Writes are asynchronous so really only limited by the actual speed
of the PCIe link and the rate the slave can process them.
So the actual latency of writes doesn't matter and the throughput
is reasonable.

Reads are much more problematic.
While the PCIe bus allows multiple outstanding read requests the
Intel x86 I've tested will only generate one outstanding request
for each cpu core.
So buffer reads are particularly slow.

The delays between on read completing and the next read TLP being
sent are (probably) negligible compared to the other delays.
So the latency of a read is just the time the two TLP take to
be transmitted over the wire (including delays for PCIe bridges)
plus the time the slave takes to generate the response TLP.
On the fpga slaves we are using that is (from memory) about 128
cycles of the 62.5MHz clock - ie absolutely ages.

For reads you definitely need to use the largest register size
possible - each read instruction (even misaligned ones) generates
exactly one read TLP.

If you are designing an interface for an fpga then consider using
writes from both sides for everything except bulk data.

You can (probably) measure the latency of your actual system using:
	x = rdtsc();
	v = readl();
	lfence;
	elapsed = rdtsc() - x;
However the TSC values depend on the current cpu frequency (which
will change 'randomly').
Or put the readl() into a loop and do enough that the high-res
system time delts makes sense.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: PCI: latency

[Bjorn Helgaas]

On Fri, Dec 03, 2021 at 10:30:58PM +0530, Subhashini Rao Beerisetty wrote:
>  [ Please keep me in CC as I'm not subscribed to the list]
> 
> Hi all,
> 
> We are using the Linux OS on an x86_64 machine. I need to measure the
> PCIe latency on my system, does kernel have any latency measurement
> module for the PCIe bus?

No, unfortunately not.  Maybe perf could help, but I don't know enough
to give you any pointers.

There are a couple tune/trace things like [1], but they're specific to
hardware that you're probably not using.

[1] https://lore.kernel.org/r/20211116090625.53702-1-yangyicong@hisilicon.com

Re: Latency.

[Uwaysi Bin Kareem]

There is a whole thread on phoronix, regarding this, where many people  
have similar sentiments.
Good to see others who understand the same. :)

http://phoronix.com/forums/showthread.php?71741-A-Low-Latency-Kernel-For-Linux-Gaming

Some people don`t seem to be noticing it. They are fewer in the thread  
though, but quite common.
Please also read my response to this:  
http://phoronix.com/forums/showthread.php?71741-A-Low-Latency-Kernel-For-Linux-Gaming&p=284229#post284229

Peace Be With You.

On Fri, 31 Aug 2012 00:50:28 +0200, Uwaysi Bin Kareem  
<uwaysi.bin.kareem@paradoxuncreated.com> wrote:

> I have done some research on latency. I have config`d a linux kernel to  
> run 0.3ms reliable latency with audiostreams, under normal  
> worksituations. (An audioapp, and maybe some small tasks in between).
>
> This also resulted in an extremely smooth gameplaying experience, like  
> an asm-programmed custom hardware arcade. (Why gamebox-developers isn`t  
> using this, is a mystery).
>
> Recently I also tried to come as close to that experience on windows,  
> and found that win32priorityseparation on 25, all processes on idle, to  
> avoid cpu2 stalling cpu1, and minimal drivers, services, and processes  
> gave a similar experience. Windows btw, also gives lower latency, if one  
> moves windows, which one can use/abuse in a script/hack.
>
> The feeling from low latency systems brings back the exhilaration of  
> custom hardware and assembly programming. It gives a different feel, and  
> I do believe it sets a high quality expectation to software and I wonder  
> if that is why the Amiga is said to have so much good software, and  
> responsible for it`s reputation.
>
> My windows-partition now runs as good as an Amiga, and I managed to make  
> it run even better, reminding me of singletasking systems like Mac OS.
>
> Games are just so much more fun with this. And the overall os is so much  
> more responsive.
>
> More optimized stuff like Wayland will ofcourse even improve things more.
>
> I do think that for "desktop" the focus should really be on low-latency  
> systems.
> If "desktop" and "server" are the two different profiles you usually  
> config for in linux, how about two different standard configs? Or are  
> these merging aswell, since I would think multi-cpu servers appreciate  
> low os-jitter aswell?
>
> Just some thoughts.
>
> Peace Be With You.

Latency.

[Uwaysi Bin Kareem]

I have done some research on latency. I have config`d a linux kernel to  
run 0.3ms reliable latency with audiostreams, under normal worksituations.  
(An audioapp, and maybe some small tasks in between).

This also resulted in an extremely smooth gameplaying experience, like an  
asm-programmed custom hardware arcade. (Why gamebox-developers isn`t using  
this, is a mystery).

Recently I also tried to come as close to that experience on windows, and  
found that win32priorityseparation on 25, all processes on idle, to avoid  
cpu2 stalling cpu1, and minimal drivers, services, and processes gave a  
similar experience. Windows btw, also gives lower latency, if one moves  
windows, which one can use/abuse in a script/hack.

The feeling from low latency systems brings back the exhilaration of  
custom hardware and assembly programming. It gives a different feel, and I  
do believe it sets a high quality expectation to software and I wonder if  
that is why the Amiga is said to have so much good software, and  
responsible for it`s reputation.

My windows-partition now runs as good as an Amiga, and I managed to make  
it run even better, reminding me of singletasking systems like Mac OS.

Games are just so much more fun with this. And the overall os is so much  
more responsive.

More optimized stuff like Wayland will ofcourse even improve things more.

I do think that for "desktop" the focus should really be on low-latency  
systems.
If "desktop" and "server" are the two different profiles you usually  
config for in linux, how about two different standard configs? Or are  
these merging aswell, since I would think multi-cpu servers appreciate low  
os-jitter aswell?

Just some thoughts.

Peace Be With You.