From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 047B2C433E6 for ; Sun, 17 Jan 2021 08:00:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C27E623107 for ; Sun, 17 Jan 2021 08:00:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727997AbhAQH67 (ORCPT ); Sun, 17 Jan 2021 02:58:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727663AbhAQH6l (ORCPT ); Sun, 17 Jan 2021 02:58:41 -0500 Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E369C061757 for ; Sat, 16 Jan 2021 23:58:00 -0800 (PST) Received: by mail-ot1-x332.google.com with SMTP id f6so3973057ots.9 for ; Sat, 16 Jan 2021 23:58:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=VESx0q50uHif31i5KEl5hWIvCIRQQNFTAOMGoAM4sHk=; b=dowfzTW23mQmJth/93y3cwCyGyvAl+fFqQcvXECxcGq8jo5zaQkT722+738BfC0A14 Bsqd/ozm5BiNtKsHXJ5UDIa7Wfm31hW52j0wVk/7WFwHHDzt1eytW0ilYs1zsM5BMX7s t1nLNTzQDYtxUd/eiOB+foBYE4wfdrR00TKQgH+31X2W7aLhZPfvJR4go5ov7vm4hlh+ 814jWwD7JM8JKTkGOMTEvwqXRqGSJC9Z16hI/+aYhFdXlqRtovk9JT9kkHGl5VyDVRY2 DewWaIsAU0QFt3CgW6rxqv0Qz3cazUlkFC55+YDczF0e0pq3KuAN2Y+g7PvU+9NnN9hd t2zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=VESx0q50uHif31i5KEl5hWIvCIRQQNFTAOMGoAM4sHk=; b=TMreqASkfhNXg+LCCGGqXn0gaKGe/tbiK4M12DFwjXMQ3c6jtfqYeWIycLucj16A7t Yd2oZyiYuOOlEpz10YGUtLPlXTxsta3/txUqQ2S/ZpKUgBmZMP4EnMrM9qwvRiL76qlF QJcd7yHlHtGFrmrsGUipF2WMNLmMyBHB9NK/8GpFiVu6VLYJofJbtTwgjHc5CUqGF/43 FLhTFPch2q+XjFutZN/MIyKVT0a9/HVYuNh8/UWUBAgBJg219jpHTUHSyBk0RZIkhIHk yBCtkkm+XGidrWu4Pqt46ueCXwUex84i5mm8VX60I8OB65fRA0fPgJ23BekEJwfreWhT R5Sw== X-Gm-Message-State: AOAM531lLRwF5XSxWLGFvuIOIwPJqFD119CSFpU+qFlB6dgw8VgvAQp6 pKVb6OqrgHbdbYZrVNlaJYCgz8x90gfeSp0hJ5UCnw== X-Google-Smtp-Source: ABdhPJw72h9Qq08nM/YnqFoYtJLR5CwoCt5GmjOdfecQ7OjxFIthON9NM5A1BUKUsEtaY5OOl0Rzt3RdzKkeCVJhpec= X-Received: by 2002:a9d:4715:: with SMTP id a21mr14710269otf.220.1610870278976; Sat, 16 Jan 2021 23:57:58 -0800 (PST) MIME-Version: 1.0 References: <20210112194143.1494-1-yuri.benditovich@daynix.com> <78bbc518-4b73-4629-68fb-2713250f8967@redhat.com> <8ea218a8-a068-1ed9-929d-67ad30111c3c@redhat.com> In-Reply-To: <8ea218a8-a068-1ed9-929d-67ad30111c3c@redhat.com> From: Yuri Benditovich Date: Sun, 17 Jan 2021 09:57:47 +0200 Message-ID: Subject: Re: [RFC PATCH 0/7] Support for virtio-net hash reporting To: Jason Wang Cc: Willem de Bruijn , "David S. Miller" , Jakub Kicinski , "Michael S . Tsirkin" , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Randy Dunlap , "Gustavo A . R . Silva" , Herbert Xu , Steffen Klassert , Pablo Neira Ayuso , decui@microsoft.com, cai@lca.pw, Jakub Sitnicki , Marco Elver , Paolo Abeni , Network Development , linux-kernel , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, bpf , Yan Vugenfirer Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Thu, Jan 14, 2021 at 5:39 AM Jason Wang wrote: > > > On 2021/1/13 =E4=B8=8B=E5=8D=8810:33, Willem de Bruijn wrote: > > On Tue, Jan 12, 2021 at 11:11 PM Jason Wang wrote= : > >> > >> On 2021/1/13 =E4=B8=8A=E5=8D=887:47, Willem de Bruijn wrote: > >>> On Tue, Jan 12, 2021 at 3:29 PM Yuri Benditovich > >>> wrote: > >>>> On Tue, Jan 12, 2021 at 9:49 PM Yuri Benditovich > >>>> wrote: > >>>>> On Tue, Jan 12, 2021 at 9:41 PM Yuri Benditovich > >>>>> wrote: > >>>>>> Existing TUN module is able to use provided "steering eBPF" to > >>>>>> calculate per-packet hash and derive the destination queue to > >>>>>> place the packet to. The eBPF uses mapped configuration data > >>>>>> containing a key for hash calculation and indirection table > >>>>>> with array of queues' indices. > >>>>>> > >>>>>> This series of patches adds support for virtio-net hash reporting > >>>>>> feature as defined in virtio specification. It extends the TUN mod= ule > >>>>>> and the "steering eBPF" as follows: > >>>>>> > >>>>>> Extended steering eBPF calculates the hash value and hash type, ke= eps > >>>>>> hash value in the skb->hash and returns index of destination virtq= ueue > >>>>>> and the type of the hash. TUN module keeps returned hash type in > >>>>>> (currently unused) field of the skb. > >>>>>> skb->__unused renamed to 'hash_report_type'. > >>>>>> > >>>>>> When TUN module is called later to allocate and fill the virtio-ne= t > >>>>>> header and push it to destination virtqueue it populates the hash > >>>>>> and the hash type into virtio-net header. > >>>>>> > >>>>>> VHOST driver is made aware of respective virtio-net feature that > >>>>>> extends the virtio-net header to report the hash value and hash re= port > >>>>>> type. > >>>>> Comment from Willem de Bruijn: > >>>>> > >>>>> Skbuff fields are in short supply. I don't think we need to add one > >>>>> just for this narrow path entirely internal to the tun device. > >>>>> > >>>> We understand that and try to minimize the impact by using an alread= y > >>>> existing unused field of skb. > >>> Not anymore. It was repurposed as a flags field very recently. > >>> > >>> This use case is also very narrow in scope. And a very short path fro= m > >>> data producer to consumer. So I don't think it needs to claim scarce > >>> bits in the skb. > >>> > >>> tun_ebpf_select_queue stores the field, tun_put_user reads it and > >>> converts it to the virtio_net_hdr in the descriptor. > >>> > >>> tun_ebpf_select_queue is called from .ndo_select_queue. Storing the > >>> field in skb->cb is fragile, as in theory some code could overwrite > >>> that between field between ndo_select_queue and > >>> ndo_start_xmit/tun_net_xmit, from which point it is fully under tun > >>> control again. But in practice, I don't believe anything does. > >>> > >>> Alternatively an existing skb field that is used only on disjoint > >>> datapaths, such as ingress-only, could be viable. > >> > >> A question here. We had metadata support in XDP for cooperation betwee= n > >> eBPF programs. Do we have something similar in the skb? > >> > >> E.g in the RSS, if we want to pass some metadata information between > >> eBPF program and the logic that generates the vnet header (either hard > >> logic in the kernel or another eBPF program). Is there any way that ca= n > >> avoid the possible conflicts of qdiscs? > > Not that I am aware of. The closest thing is cb[]. > > > > It'll have to aliase a field like that, that is known unused for the gi= ven path. > > > Right, we need to make sure cb is not used by other ones. I'm not sure > how hard to achieve that consider Qemu installs the eBPF program but it > doesn't deal with networking configurations. > > > > > > One other approach that has been used within linear call stacks is out > > of band. Like percpu variables softnet_data.xmit.more and > > mirred_rec_level. But that is perhaps a bit overwrought for this use > > case. > > > Yes, and if we go that way then eBPF turns out to be a burden since we > need to invent helpers to access those auxiliary data structure. It > would be better then to hard-coded the RSS in the kernel. > > > > > >>>>> Instead, you could just run the flow_dissector in tun_put_user if t= he > >>>>> feature is negotiated. Indeed, the flow dissector seems more apt to= me > >>>>> than BPF here. Note that the flow dissector internally can be > >>>>> overridden by a BPF program if the admin so chooses. > >>>>> > >>>> When this set of patches is related to hash delivery in the virtio-n= et > >>>> packet in general, > >>>> it was prepared in context of RSS feature implementation as defined = in > >>>> virtio spec [1] > >>>> In case of RSS it is not enough to run the flow_dissector in tun_put= _user: > >>>> in tun_ebpf_select_queue the TUN calls eBPF to calculate the hash, > >>>> hash type and queue index > >>>> according to the (mapped) parameters (key, hash types, indirection > >>>> table) received from the guest. > >>> TUNSETSTEERINGEBPF was added to support more diverse queue selection > >>> than the default in case of multiqueue tun. Not sure what the exact > >>> use cases are. > >>> > >>> But RSS is exactly the purpose of the flow dissector. It is used for > >>> that purpose in the software variant RPS. The flow dissector > >>> implements a superset of the RSS spec, and certainly computes a > >>> four-tuple for TCP/IPv6. In the case of RPS, it is skipped if the NIC > >>> has already computed a 4-tuple hash. > >>> > >>> What it does not give is a type indication, such as > >>> VIRTIO_NET_HASH_TYPE_TCPv6. I don't understand how this would be used= . > >>> In datapaths where the NIC has already computed the four-tuple hash > >>> and stored it in skb->hash --the common case for servers--, That type > >>> field is the only reason to have to compute again. > >> > >> The problem is there's no guarantee that the packet comes from the NIC= , > >> it could be a simple VM2VM or host2VM packet. > >> > >> And even if the packet is coming from the NIC that calculates the hash > >> there's no guarantee that it's the has that guest want (guest may use > >> different RSS keys). > > Ah yes, of course. > > > > I would still revisit the need to store a detailed hash_type along with > > the hash, as as far I can tell that conveys no actionable information > > to the guest. > > > Yes, need to figure out its usage. According to [1], it only mention > that storing has type is a charge of driver. Maybe Yuri can answer this. > For the case of Windows VM we can't know how exactly the network stack uses provided hash data (including hash type). But: different releases of Windows enable different hash types (for example UDP hash is enabled only on Server 2016 and up). Indeed the Windows requires a little more from the network adapter/driver than Linux does. The addition of RSS support to virtio specification takes in account the widest set of requirements (i.e. Windows one), our initial impression is that this should be enough also for Linux. The NDIS specification in part of RSS is _mandatory_ and there are certification tests that check that the driver provides the hash data as expected. All the high-performance network adapters have such RSS functionality in the hardware. With pre-RSS QEMU (i.e. where the virtio-net device does not indicate the RSS support) the virtio-net driver for Windows does all the job related to RSS: - hash calculation - hash/hash_type delivery - reporting each packet on the correct CPU according to RSS settings With RSS support in QEMU all the packets always come on a proper CPU and the driver never needs to reschedule them. The driver still need to calculate the hash and report it to Windows. In this case we do the same job twice: the d= evice (QEMU or eBPF) does calculate the hash and get proper queue/CPU to deliver the packet. But the hash is not delivered by the device, so the driver need= s to recalculate it and report to the Windows. If we add HASH_REPORT support (current set of patches) and the device indicates this feature we can avoid hash recalculation in the driver assuming we receive the correct hash value and hash type. Otherwise the driver can't know which exactly hash the device has calculated. Please let me know if I did not answer the question. > Thanks > > [1] > https://docs.microsoft.com/en-us/windows-hardware/drivers/network/indicat= ing-rss-receive-data > > > > >