Re: Kernel UDP behavior with missing destinations

[Willem de Bruijn <willemdebruijn.kernel@gmail.com>]

On Fri, May 17, 2019 at 8:57 AM David Laight <David.Laight@aculab.com> wrote:
>
> From: Willem de Bruijn
> > Sent: 17 May 2019 04:23
> > On Thu, May 16, 2019 at 8:27 PM Adam Urban <adam.urban@appleguru.org> wrote:
> > >
> > > And replying to your earlier comment about TTL, yes I think a TTL on
> > > arp_queues would be hugely helpful.
> > >
> > > In any environment where you are streaming time-sensitive UDP traffic,
> > > you really want the kernel to be tuned to immediately drop the
> > > outgoing packet if the destination isn't yet known/in the arp table
> > > already...
>
> I suspect we may suffer from the same problems when sending out a lot
> of RTP (think of sending 1000s of UDP messages to different addresses
> every 20ms).
> For various reasons the sends are done from a single raw socket (rather
> than 'connected' UDP sockets).
>
> > For packets that need to be sent immediately or not at all, you
> > probably do not want a TTL, but simply for the send call to fail
> > immediately with EAGAIN instead of queuing the packet for ARP
> > resolution at all. Which is approximated with unres_qlen 0.
> >
> > The relation between unres_qlen_bytes, arp_queue and SO_SNDBUF is
> > pretty straightforward in principal. Packets can be queued on the arp
> > queue until the byte limit is reached. Any packets on this queue still
> > have their memory counted towards their socket send budget. If a
> > packet is queued that causes to exceed the threshold, older packets
> > are freed and dropped as needed. Calculating the exact numbers is not
> > as straightforward, as, for instance, skb->truesize is a kernel
> > implementation detail.
>
> But 'fiddling' with the arp queue will affect all traffic.
> So you'd need it to be per socket option so that it is a property
> of the message by the time it reaches the arp code.

A per socket or even datagram do-not-queue signal would be
interesting. Where any queuing would instead result in send failure
(though this feedback signal does not work for secondary qdiscs).

The recent SCM_TXTIME cmsg has a deadline mode that might implement
this. In which case we would only have to check for it in the neighbor
layer.

>
> > The simple solution is just to overprovision the socket SO_SNDBUF. If
> > there are few sockets in the system that perform this role, that seems
> > perfectly fine.
>
> That depends on how often you are sending messages compared to the
> arp timeout. If you are sending 50 messages a second to each of 1000
> destinations the over provisioning of SO_SNDBUF would have to be extreme.
>
> FWIW we do sometimes see sendmsg() taking much longer than expected,
> but haven't get tracked down why.

I've observed this problem with health checks under particular ARP
settings as well.