Re: Excessive network latency when using Realtek R8168/R8111 et al NIC

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2023-04-29 07:37:25 [+1000], Rod Webster wrote:
Sorry for the break.

> > Is the vendor driver better than in the tree? This is not obvious from
> > what you are writing.
> I have been unable to compile the R8168 driver from source on the
> Realtek website.
> They are behind with supported kernels and the build process is
> invariably broken for me.
> On the Hardkernel Odroid forum, it has been reported that compilng the
> R8125 from Realtek source offers 15-35% improvement over the in-tree
> driver.
>  I have one of their  PC's (H2+). It's currently running v5.10 and the
> in-tree R8168 driver (which must be the Debian default).
>  It is affected and I plan to upgrade to v6.3 with the R8125-dkms driver

That is interesting. 15%-35% improvement for the in-tree driver would be
certainly welcomed. :)

> > Two things you could do:
> > - enabling tracing to see what is causing the delays/ latency spike. You
> >   will need a trigger to notice the latency and then stop the trace at
> >   this point. If so, I could provide additional steps unless you can do
> >   it yourself.
> Please advise additional steps to do this.
> We might  be able to modify our hardware device driver to stop the
> trace as it reports latency incursions if they occur.

Either with a tool like trace-cmd or manual.
- trace-cmd
  - Enable events:
    trace-cmd start -e sched:* -e irq:* -e irq_vectors:*
  - Start the test
  - Once the error happens, stop the trace
    trace-cmd stop
  - Extract the trace
    trace-cmd extract
  - The command produces a trace.dat file. It can viewed in text via
    "trace-cmd report" or kernelshark.
  - Disable the events
    trace-cmd reset

- Manual
  - Enable events
    cd /sys/kernel/debug/tracing/
    echo 1 > events/sched/enable
    echo 1 > events/irq/enable
    echo 1 > events/irq_vectors/enable
  - Start the test
  - Once the error happens, stop the trace (assume you are still in the
    previous folder)
    echo 0 > tracing_on
  - Extract the trace
    cat trace > ~/trace.txt
  - The average editor can be used.
  - Disable the events (and reset).
    echo 0 > events/enable
    echo 1 > tracing_on

> > - You could try to isolate the realtek driver on one CPU and moving
> >   everything else to another.
> This sounds interesting. Please advise how.
> On a 4 core machine we typically add isolcpus=2,3 for best RT performance.

So this isolcpus CPU thingy is already going in that direction. If you
have two NICs, say one with RT traffic and one without, then the RT
traffic should go the RT CPUs while the non-RT traffic to the non-RT
CPUs. The isolcpus option works only for tasks. You should do the same
interrupts. See the irqaffinity option.
Lets first look at the trace and what the problem is and what can be
done…

> Thanks for your support.
> 
> -end
> -Rod Webster

Sebastian