Re: Excessive network latency when using Realtek R8168/R8111 et al NIC

[Rod Webster <rod@vmn.com.au>]

[-- Attachment #1: Type: text/plain, Size: 2338 bytes --]

> Okay. Then it could be related to softirq rework which started in
> v5.0.19-rt11 and updated in v5.9-rc6-rt9.
v5.9 could also be significant as the R8125 driver support was added
in that version
I upgraded from Debian Buster to Bullseye (v5.10) on one PC for this reason.

> Are you able to recompile a kernel?
Yes,
I have attached a text file containing my 6.3 build steps.
Note we had to git pull the kernel source as it was not available via
http at that time
I have not tried with your 6.3-rc11 patch yet.

> However as per you Debian bug report v4.19-RT is fine, v5.10-RT isn't
> and v6.1-RT is better or is it fine?
I built v6.1-rc7-rt5 before Debian advanced to v6.1
It was reported to be an improvement and it solved the Debian latency
issue for some users.
v6.3 was reported as being as good as the 4.x kernes by our hardware vendor

> Where is the Intel ethernet coming from? You were saying it is the
> realtek nic.
Sorry, you mentioned coalesce but that's an Intel feature and does not
exist with Realtek.

> Is the vendor driver better than in the tree? This is not obvious from
> what you are writing.
I have been unable to compile the R8168 driver from source on the
Realtek website.
They are behind with supported kernels and the build process is
invariably broken for me.
On the Hardkernel Odroid forum, it has been reported that compilng the
R8125 from Realtek source offers 15-35% improvement over the in-tree
driver.
 I have one of their  PC's (H2+). It's currently running v5.10 and the
in-tree R8168 driver (which must be the Debian default).
 It is affected and I plan to upgrade to v6.3 with the R8125-dkms driver

> Two things you could do:
> - enabling tracing to see what is causing the delays/ latency spike. You
>   will need a trigger to notice the latency and then stop the trace at
>   this point. If so, I could provide additional steps unless you can do
>   it yourself.
Please advise additional steps to do this.
We might  be able to modify our hardware device driver to stop the
trace as it reports latency incursions if they occur.

> - You could try to isolate the realtek driver on one CPU and moving
>   everything else to another.
This sounds interesting. Please advise how.
On a 4 core machine we typically add isolcpus=2,3 for best RT performance.

Thanks for your support.

-end
-Rod Webster

[-- Attachment #2: Instructions.txt --]
[-- Type: text/plain, Size: 1046 bytes --]

apt update 
sudo apt install -y build-essential bin86 libncurses5-dev pkg-config libssl-dev dwarves bison flex gnupg libelf-dev libssl-dev wget qtbase5-dev debhelper rsync git
mkdir rtlinux 
cd rtlinux 
git clone -b v6.3-rc5 --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux-6.3-rc5 
wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.3/patch-6.3-rc5-rt8.patch.gz gunzip patch-6.3-rc5-rt8.patch.gz 
cp patch-6.3-rc5-rt8.patch linux-6.3-rc5 
cd linux-6.3-rc5 
cat patch-6.3-rc5-rt8.patch | patch -p1 
make xconfig 
#Set the following: General Settings/Preemption model = Fully Preemptible Kernel
#Save then exit (Click on General settings and select option on the right side.


# These keys override security and key settings to allow .debs to be built
scripts/config --disable SYSTEM_REVOCATION_KEYS 
scripts/config --disable DEBUG_INFO 
scripts/config --enable DEBUG_INFO_NONE 
scripts/config --set-str SYSTEM_TRUSTED_KEYS "" 
make -j$(nproc) deb-pkg LOCALVERSION=-linuxcnc