On Thu, May 19, 2022 at 7:41 PM David Laight <David.Laight@aculab.com> wrote:
>
> From: Paolo Abeni
> > Sent: 19 May 2022 14:29
> ....
> > If the packet processing is 'bursty', you can have idle time and still
> > hit now and the 'rx ring is [almost] full' condition. If pause frames
> > are enabled, that will cause the peer to stop sending frames: drop can
> > happen in the switch, and the local NIC will not notice (unless there
> > are counters avaialble for pause frames sent).
>
> The test program sending the data does spread it out.
> So it isn't sending 2000 packets with minimal IPG every 10ms.
> (I'm sending from 2 systems.)
>
> I don't know if pause frames are enabled (ethtool might suggest they are).
> But detecting whether they are sent is another matter.
>
> In any case sending pause frames doesn't fix anything.
> They are largely entirely useless unless you have a cable
> that directly connects two computers.
>
> > AFAICS the packet processing is bursty, because enqueuing packets to a
> > remote CPU in considerably faster then full network stack processing.
>
> I have taken restricted ftrace traces of the receiving system.
> Not often seen more than 4 frames processed in one napi callback
> Certainly didn't spot blocks of 100+ that you might expect
> to see if the driver code was the bottleneck.
>
> > Side note: on a not-to-obsolete H/W the kernel should be able to
> > process >1mpps per cpu.
>
> Yes, and, IIRC, a 33Mhz 486 can saturate 10MHz ethernet with
> small packets.
>
> In this case the cpu are almost twiddling their thumbs.
>   model name      : Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
>   stepping        : 2
>   microcode       : 0x43
>   cpu MHz         : 1300.000
> cpu 14 (the one taking the interrupts) is running at full speed.
>
> cpu doesn't seem to be the bottleneck.
> The problem seems to be the hardware not using all the buffers
> it has been given.
>

When this happens, can you provide the register dump that you can
obtain using ethtool -d ?

>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)