Re: [PATCH net 1/4] net/udp_gso: Allow TX timestamp with UDP GSO

From: Fred Klassen <fklassen@appneta.com>
To: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
Cc: "David S. Miller" <davem@davemloft.net>,
	Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>,
	Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
	Shuah Khan <shuah@kernel.org>,
	Network Development <netdev@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	linux-kselftest@vger.kernel.org,
	Willem de Bruijn <willemb@google.com>
Subject: Re: [PATCH net 1/4] net/udp_gso: Allow TX timestamp with UDP GSO
Date: Fri, 24 May 2019 15:01:24 -0700	[thread overview]
Message-ID: <90E3853F-107D-45BA-93DC-D0BE8AC6FCBB@appneta.com> (raw)
In-Reply-To: <CAF=yD-Le-eTadOi7PL8WFEQCG=yLqb5gvKiks+s5Akeq8TenBQ@mail.gmail.com>

> On May 24, 2019, at 12:29 PM, Willem de Bruijn <willemdebruijn.kernel@gmail.com> wrote:
> 
> It is the last moment that a timestamp can be generated for the last
> byte, I don't see how that is "neither the start nor the end of a GSO
> packet”.

My misunderstanding. I thought TCP did last segment timestamping, not
last byte. In that case, your statements make sense.

>> It would be interesting if a practical case can be made for timestamping
>> the last segment. In my mind, I don’t see how that would be valuable.
> 
> It depends whether you are interested in measuring network latency or
> host transmit path latency.
> 
> For the latter, knowing the time from the start of the sendmsg call to
> the moment the last byte hits the wire is most relevant. Or in absence
> of (well defined) hardware support, the last byte being queued to the
> device is the next best thing.
> 
> It would make sense for this software implementation to follow
> established hardware behavior. But as far as I know, the exact time a
> hardware timestamp is taken is not consistent across devices, either.
> 
> For fine grained timestamped data, perhaps GSO is simply not a good
> mechanism. That said, it still has to queue a timestamp if requested.

I see your point. Makes sense to me.

>> When using hardware timestamping, I think you will find that nearly all
>> adapters only allow one timestamp at a time. Therefore only one
>> packet in a burst would get timestamped.
> 
> Can you elaborate? When the host queues N packets all with hardware
> timestamps requested, all N completions will have a timestamp? Or is
> that not guaranteed?
> 

It is not guaranteed. The best example is in ixgbe_main.c and search for
‘SKBTX_HW_TSTAMP’.  If there is a PTP TX timestamp in progress,
‘__IXGBE_PTP_TX_IN_PROGRESS’ is set and no other timestamps
are possible. The flag is cleared after transmit softirq, and only then
can another TX timestamp be taken.  

>> There are exceptions, for
>> example I am playing with a 100G Mellanox adapter that has
>> per-packet TX timestamping. However, I suspect that when I am
>> done testing, all I will see is timestamps that are representing wire
>> rate (e.g. 123nsec per 1500 byte packet).
>> 
>> Beyond testing the accuracy of a NIC’s timestamping capabilities, I
>> see very little value in doing per-segment timestamping.
> 
> Ack. Great detailed argument, thanks.

Thanks. I’m a timestamping nerd and have learned lots with this 
discussion.