Re: [net-next 0/3] Per epoll context busy poll support

From: Jakub Kicinski <kuba@kernel.org>
To: Joe Damato <jdamato@fastly.com>
Cc: "Samudrala, Sridhar" <sridhar.samudrala@intel.com>,
	Eric Dumazet <edumazet@google.com>,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	chuck.lever@oracle.com, jlayton@kernel.org,
	linux-api@vger.kernel.org, brauner@kernel.org,
	davem@davemloft.net, alexander.duyck@gmail.com,
	Wei Wang <weiwan@google.com>,
	Amritha Nambiar <amritha.nambiar@intel.com>
Subject: Re: [net-next 0/3] Per epoll context busy poll support
Date: Fri, 2 Feb 2024 17:15:39 -0800	[thread overview]
Message-ID: <20240202171539.7347cb01@kernel.org> (raw)
In-Reply-To: <20240202202344.GA9283@fastly.com>

On Fri, 2 Feb 2024 12:23:44 -0800 Joe Damato wrote:
> > Did you see SO_PREFER_BUSY_POLL by any chance? (In combination with
> > gro_flush_timeout IIRC). We added it a while back with Bjorn, it seems
> > like a great idea to me at the time but I'm unclear if anyone uses it 
> > in production..  
> 
> I have seen it while reading the code, yes. I think maybe I missed
> something about its interaction with gro_flush_timeout. In my use case,
> the machine has no traffic until after the app is started.
> 
> In this case, I haven't needed to worry about regular NAPI monopolizing the
> CPU and preventing busy poll from working.
> 
> Maybe I am missing something more nuanced, though? I'll have another look
> at the code, just incase.

We reused the gro_flush_timeout as an existing "user doesn't care if
packets get delayed by this much in worst case" value. If you set
SO_PREFER_BUSY_POLL the next time you busy pool the NAPI will be marked 
as "already scheduled" and a timer is set (to gro_flush_timeout).
If NIC IRQ fires before gro_flush_timeout it gets ignored, because NAPI
is already marked as scheduled.
If you busy poll again the timer gets postponed for another
gro_flush_timeout nsec.
If timer fires we go back to normal NAPI processing.

The idea is that you set gro_flush_timeout to some high value, like 
10 msec, and expect your app to poll more often than every 10 msec. 

Then the normal NAPI processing will never kick in, and there will 
be only 1 NIC IRQ after which the HW IRQ remains masked.
With high coalescing timer you technically still get an IRQ every
so often and interrupt the app. Worst case (UDP flood) you may even
get into an overload where the app gets starved out completely..