Load on RTL8168g/8111g stalls network for multiple seconds

From: "Johannes Brandstätter" <jbrandst@2ds.eu>
To: netdev@vger.kernel.org
Subject: Load on RTL8168g/8111g stalls network for multiple seconds
Date: Mon, 07 Jun 2021 15:11:05 +0200	[thread overview]
Message-ID: <5b08afe02cb0baa7ae3e19fd0bc9d1cbe9ea89c9.camel@2ds.eu> (raw)

Hi,

just the other day I wanted to set up a bridge between an external 2.5G
RTL8156 USB Ethernet adapter (using r8152) and the built in dual
RTL8168g/8111g Ethernet chip (using r8169).
I compiled the kernel as of 5.13.0-rc4 because of the r8125 supporting
the RTL8156.
This was done using the Debian kernel config of 5.10.0-4 as a base and
left the rest as default.

So this setup was working the way I wanted it to, but unfortunately
when running iperf3 against the machine it would rather quickly stall
all communications on the internal RTL8168g.
I was still able to communicate fine over the external RTL8156 link
with the machine.
Even without the generated network load, it would occasionally become
stalled.

The only information I could really gather were that the rx_missed
counter was going up, and this kernel message some time after the stall
was happening:

[81853.129107] r8169 0000:02:00.0 enp2s0: rtl_rxtx_empty_cond == 0
(loop: 42, delay: 100).

Which has apparently to do with the wait for an empty fifo within the
r8169 driver.

Until that the machine (an UP² board) using the RTL8168g ran without
any issues for multiple years in different configurations.
Only bridging immediately showed the issue when given enough network
load.

After many hours of trying out different things, nothing of which
showed any difference whatsoever, I tried to replace the internal
RTL8168g with an additional external USB Ethernet adapter which I had
laying around, having a RTL8153 inside.

Once the RTL8168g was removed and the RTL8153 added to the bridge, I
was unable to reproduce the issue.
Of course I'd rather like to make use of the two internal Ethernet
ports if I somehow can.

So is there anything I could try to do?

I'm eyeing with a regression test next on the kernel's r8168 driver.
Though this is without me knowing if there ever was a working version.
As this is a rather large task, with only limited time I wanted to seek
out some help before I go down that route.

Maybe you could point me into the right direction, as to what to try
next.

Thanks and best regards,
Johannes