Re: Excessive network latency when using Realtek R8168/R8111 et al NIC

[Marcelo Tosatti <mtosatti@redhat.com>]

On Mon, May 22, 2023 at 01:37:19PM -0700, Peter Wallace wrote:
> On Tue, 23 May 2023, Rod Webster wrote:
> 
> > Date: Tue, 23 May 2023 06:02:13 +1000
> > From: Rod Webster <rod@vmn.com.au>
> > To: Marcelo Tosatti <mtosatti@redhat.com>
> > Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
> >     linux-rt-users@vger.kernel.org
> > Subject: Re: Excessive network latency when using Realtek R8168/R8111 et al
> >     NIC
> > 
> > This stuff is hard! I just realised that rtapi_app is a red herring!
> > rtapi_app is Linuxcnc and there is nothing wrong with it. Its thread
> > is on a 1000us cycle so it seems it gets all its jobs done in 200us
> > and then sleeps for 800us which makes perfect sense!
> > 
> > The issue we have is deeper than that. I think we should be looking at
> > the NIC interrupt (but don't trust the novice!).
> > The network communication is consuming more than the 800us slack from
> > time to time. When that happens, our hardware sees the timing overrun
> > and increments an internal packet error count. If too many of these
> > happen in succession, the hardware decides the RT environment can't be
> > relied on, disables further communication and returns an "error
> > finishing read" to Linuxcnc to say it's given up.
> > 
> > Marcelo, we didn't resort to C. We were able to use a bash script and
> > use a linuxcnc tool called halcmd to query the hardware as shown here.
> > #!/usr/bin/bash
> > stat=0
> > while (($stat < 1))
> > do
> > stat=`halcmd getp hm2_7i96s.0.packet-error-total`
> > done
> > trace-cmd stop
> > 
> > I think we need to increase the stat threshold  so we get more samples
> > in our trace before stopping it. The current trace will only have one
> > instance.
> > Thanks for letting me see the issue more clearly.
> > 
> > 
> > Rod Webster
> > 
> 
> 
> I should note that at least for Intel MACs, the 6.3.1-rt13 and 6.4.0-rc2-rt1
> kernels seem to solve the issue. Not sure what changed but maximum read time
> is now in the 200.. 250 usec peak region (about 100 usec more than average)
> This is the peak read latency after about 3 days of videos, compiling and
> local network activity.
> 
> Sadly 6.4.0-rc3-rt2 has regressed slightly in network latency on my test
> systems
> 
> My test systems were all Intel CPUs with 4 cores, isolcpus=3 and the Ethernet
> IRQ pinned to CPU3
> 
> 
> Peter Wallace

Are you guys using the realtime profiles from Tuned?

Edit /etc/tuned/realtime-virtual-host-variables.conf,

Then run

tuned-adm profile realtime-virtual-host

Note this will perform steps to isolate the configured CPU's,
including unpinning all IRQs from the isolated CPUs,
(which you can fix after applying the profile).
enabling nohz_full=, rcu_nocbs=, etc (can check 
/usr/lib/tuned/realtime-virtual-host/tuned.conf and script.sh 
to see what what it (and its parent profiles) do).