Re: Excessive network latency when using Realtek R8168/R8111 et al NIC

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2023-04-28 20:46:17 [+1000], Rod Webster wrote:
> Sebastian
Hi Rod,

please keep list in Cc: Avoiding top-posting is a quite nice.

> Thanks, the Real TIme PREEMPT_RT kernel is a prerequisite for
> Linuxcnc. Note in this discussion, I am only referring to X86 machines

Okay. Then it could be related to softirq rework which started in
v5.0.19-rt11 and updated in v5.9-rc6-rt9. 

Are you able to recompile a kernel?

> although we also install our software on ARM and others.
> linuxcnc-uspace is included in the debian Bookworm repositories and
> linux-image-rt  is one of its dependencies..
> There are really two issues we have experienced. These have only
> become a factor since the 5.X kernels. Only one seems to be relevant
> to the RT project.
> 1. Excessive real time latency/jitter in the Debian RT images. I have
> escalated this to them on their Bug 1034550. Ref
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034550

As mentioned, the images are provided for convince. Depending one the
problem, the bugs are forwarded sometimes here.

However as per you Debian bug report v4.19-RT is fine, v5.10-RT isn't
and v6.1-RT is better or is it fine?

> 2. Excessive real time network latency/jitter when using the
> PREEMPT_RT kernel when communicating with Intel ethernet hardware in
> an RT environment.

Where is the Intel ethernet coming from? You were saying it is the
realtek nic.

> It is the second issue I wish to bring to your attention.  eg.
> Excessive network latency. This appears to be more prevalent on
> Realtek hardware.
> Note that we document setting hardware-irq-coalesce-rx-usecs 0 on
> Intel NIC's is required. This setting has no effect on other makes of
> NIC's.
> 
> In relation to Realtek NIC's, I can only speak for Debian but by
> default they use the R8169 Kernel module. (CONFIG_R8169?) They also
> offer the R8168-dkms and R8125-dkms drivers. We have many data points
> from our users that the R8169 driver offers very poor  RT performance.
> But just installing a dkms driver is not enough on Debian to resolve
> the issue.

Is the vendor driver better than in the tree? This is not obvious from
what you are writing.

> We need to also build a RT kernel from the upstream sources and apply
> your RT patches.
> 
> So aside from the awful high latency/jitter implementation of  RT by
> Debian since the 5.10 kernel (Bookworm), the current strategy to use
> the R8169 as a generic driver is flawed. Debian says to report any use
> of the R8168-dkms driver and they will fix it.
> 
> A lot of small industrial PC's use Realtek NIC and with the pending
> release of Bookworm now Linuxcnc is in its repos, there will be a wide
> ranging impact.
> 
> Our FGPA ethernet hardware is capable of reporting maximum ethernet
> read and write times in timer ticks and is used in a point to point
> connection (no routers).
> 
> I hope you can investigate this matter.

I am surprised to hear the vendor driver is doing fine while the in-tree
driver is bad.

Two things you could do:
- enabling tracing to see what is causing the delays/ latency spike. You
  will need a trigger to notice the latency and then stop the trace at
  this point. If so, I could provide additional steps unless you can do
  it yourself.

- You could try to isolate the realtek driver on one CPU and moving
  everything else to another.

> end
> -Rod Webster
> 
> 
> 
> Rod Webster
> 
> VMN®
> 
> www.vmn.com.au

Sebastian