From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56FACC072B5 for ; Fri, 24 May 2019 19:30:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2347B217F9 for ; Fri, 24 May 2019 19:30:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Omm51I/H" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732090AbfEXTaJ (ORCPT ); Fri, 24 May 2019 15:30:09 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:46671 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727115AbfEXTaJ (ORCPT ); Fri, 24 May 2019 15:30:09 -0400 Received: by mail-ed1-f67.google.com with SMTP id f37so15715042edb.13; Fri, 24 May 2019 12:30:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=bevgnpwWp43eLyKYVX1neQ0UpZhiFx6WSxipgBSsKRk=; b=Omm51I/HC8uqeMOORrTprPgd8EI0GEG2/vdjRflGw6kD4ao+xJC++uyCNJChLeTp7/ diqPKe29SGQhAYwnjtB5W1SVme/jv2KbE4Kx5hqEYDQRWo1gG6Eyzguv4X+8vshr/OiK CSFPo+j5TsK9FGba/4m6xMj76y3/NRFNYq+FsOza6sL2gfD2r+6DM9fkkMV/KzFw8tpb 80aQGS2QAltLMofWker9rQY+PLNZh1mGtqm4hpuVJsVZdAOpH37I5sxrXkhYDaUUyoEy NsXJbyV0TE4q8dIpWOgnePyDfOAOAeLbYZm/ceLzOekPUO1cRVhKsm5000Egk1nd9V9v rpoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=bevgnpwWp43eLyKYVX1neQ0UpZhiFx6WSxipgBSsKRk=; b=DEHslOi5EmzB0S3/wq4Dn0D3zVPahxUllkbNTA1pyQY5fFDt4qOFoVidLX8lejHeqr jXsz3CCyYsbpWdDRwoINZ6k9OISzi+/yNPiCR2HLanE/V9SLBmPzGWnmporQqAYCFiTT 71PITVOARfqlUWbuS9HBZVLMUYuHk06sNU0AQF7xAUhS97G0Yj5y4LPgGoOiPCfNTpBp j0LcakPC+neM2NJW6djs3y2Uyx4WbVmYqIX5w/0SnqLzMUa7mlKb3Vd1ALmHCCJJwidq aG0Ck1XOqQw4ZfvKUw6vU+nHumDUuFo8E7m6XaeitLhJsGjWCcR31oUax0yF1x9AaE9U S+WA== X-Gm-Message-State: APjAAAXOK0jKUCfmQPPm5YyiveG31+OlW8/N59racpQqzEzDkefBu486 kvFy4tIAxFcKp4L1uNIy7LlOinn1+uoIGMsmPjI= X-Google-Smtp-Source: APXvYqyS2Gr9Ox8h2x39qvakn/evhg5BERIyfc6ZfJWEkcvPeJmxxQycHb293J3FqUvkicWp3WNJTIyNkiHpZ5tELEo= X-Received: by 2002:aa7:ca54:: with SMTP id j20mr105844637edt.23.1558726206280; Fri, 24 May 2019 12:30:06 -0700 (PDT) MIME-Version: 1.0 References: <20190523210651.80902-1-fklassen@appneta.com> <20190523210651.80902-2-fklassen@appneta.com> In-Reply-To: From: Willem de Bruijn Date: Fri, 24 May 2019 15:29:29 -0400 Message-ID: Subject: Re: [PATCH net 1/4] net/udp_gso: Allow TX timestamp with UDP GSO To: Fred Klassen Cc: "David S. Miller" , Alexey Kuznetsov , Hideaki YOSHIFUJI , Shuah Khan , Network Development , LKML , linux-kselftest@vger.kernel.org, Willem de Bruijn Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, May 24, 2019 at 12:34 PM Fred Klassen wrote: > > > Interesting. TCP timestamping takes the opposite choice and does > > timestamp the last byte in the sendmsg request. > > > > I have a difficult time with the philosophy of TX timestamping the last > segment. The actual timestamp occurs just before the last segment > is sent. This is neither the start nor the end of a GSO packet, which > to me seems somewhat arbitrary. It is even more arbitrary when using > software TX tiimestamping. These are timestamps represent the > time that the packet is queued onto the NIC=E2=80=99s buffer, not actual > time leaving the wire. It is the last moment that a timestamp can be generated for the last byte, I don't see how that is "neither the start nor the end of a GSO packet". > Queuing to a ring buffer is usually much faster > than wire rates. Therefore, say the timestamp of the last 1500 byte > segment of a 64K GSO packet may in reality be representing a time > about half way through the burst. > > Since the timestamp of a TX packet occurs just before any data is sent, > I have found it most valuable to timestamp just before the first byte of > the packet or burst. Conversely, I find it most valuable to get an RX > timestamp after the last byte arrives. > > > It sounds like it depends on the workload. Perhaps this then needs to > > be configurable with an SOF_.. flag. > > > > It would be interesting if a practical case can be made for timestamping > the last segment. In my mind, I don=E2=80=99t see how that would be valua= ble. It depends whether you are interested in measuring network latency or host transmit path latency. For the latter, knowing the time from the start of the sendmsg call to the moment the last byte hits the wire is most relevant. Or in absence of (well defined) hardware support, the last byte being queued to the device is the next best thing. It would make sense for this software implementation to follow established hardware behavior. But as far as I know, the exact time a hardware timestamp is taken is not consistent across devices, either. For fine grained timestamped data, perhaps GSO is simply not a good mechanism. That said, it still has to queue a timestamp if requested. > > Another option would be to return a timestamp for every segment. But > > they would all return the same tskey. And it causes different behavior > > with and without hardware offload. > > When it comes to RX packets, getting per-packet (or per segment) > timestamps is invaluable. They represent actual wire times. However > my previous research into TX timestamping has led me to conclude > that there is no practical value when timestamping every packet of > a back-to-back burst. > > When using software TX timestamping, The inter-packet timestamps > are typically much faster than line rate. Whereas you may be sending > on a GigE link, you may measure 20Gbps. At higher rates, I have found > that the overhead of per-packet software timestamping can produce > gaps in packets. > > When using hardware timestamping, I think you will find that nearly all > adapters only allow one timestamp at a time. Therefore only one > packet in a burst would get timestamped. Can you elaborate? When the host queues N packets all with hardware timestamps requested, all N completions will have a timestamp? Or is that not guaranteed? > There are exceptions, for > example I am playing with a 100G Mellanox adapter that has > per-packet TX timestamping. However, I suspect that when I am > done testing, all I will see is timestamps that are representing wire > rate (e.g. 123nsec per 1500 byte packet). > > Beyond testing the accuracy of a NIC=E2=80=99s timestamping capabilities,= I > see very little value in doing per-segment timestamping. Ack. Great detailed argument, thanks.