From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F34DDC433E0 for ; Wed, 13 Jan 2021 14:41:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AC80A23436 for ; Wed, 13 Jan 2021 14:41:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727346AbhAMOkw (ORCPT ); Wed, 13 Jan 2021 09:40:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725843AbhAMOkv (ORCPT ); Wed, 13 Jan 2021 09:40:51 -0500 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E901C061575 for ; Wed, 13 Jan 2021 06:40:11 -0800 (PST) Received: by mail-pj1-x1030.google.com with SMTP id b5so1333104pjl.0 for ; Wed, 13 Jan 2021 06:40:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=2ujGnSKYEthvi0rZ2iClFRGUsHpgEMsu727w+G030VY=; b=KUPL716HUHhkSt/RjEEjpWlYVDKM/abejwsbcqehL7DCSQm+pJqoncUorlVw/OV54k YViU+8HHMxFP67B4XdlZeuFHhKN8LFgxxyAuzbMhI/TONKoL8kkMF1z2PUJzY2Z+Gzd3 6FVdE8sdiBLfF+nzsTAeDoSVodGoWs6pbHJdyHrDWIya8fLkDBpMubiZhda8l6YVYSQL TQ6gMvrlDp0E9Pb02g8KV/58eBqEYu+BKtSUdEXFQfnPiTe80AhbbzV2fj9dw+KuF5nx E8XWoEDWqy7jLDC+GnO6H3xkacv1NVRiFqCnTzYxIU/DjZQs+qOL0hVLJ+Us+11Zm6ot B0ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=2ujGnSKYEthvi0rZ2iClFRGUsHpgEMsu727w+G030VY=; b=I+YWYUYkQd9oVP9V+C5h7cfdIGKGsV/4kknae+G5Uz7U1UN5ny8CV8sSdVngYmU43n 8ICKOZEEpUrBQH5ZueQxbP5035D2+J6EiumF/eHjwj80SSIsJeed0p3ffk5O5xT0ZjOW ECyvSqvQNUomApTnU/TSlnNtSNqNx+y13jBVMDV2HE1vh+c4bIUk0wuYS4dLMxNTWdl5 T5Rml2s7x0TUl/Mm+4GfzC1CVytTLDmsH7g8BpYc2ZLKFPH5iyvIGhg/5iP2nWg6QqHg 8US9vopC7s4Z9IP3rSWdSf/WJN5Z7iFcQd2VGOWL5E+/F1OcZqpMj7iAxMHwOKsVa3/r +/+g== X-Gm-Message-State: AOAM5310NWpUdPkEP3C9x5H008nG+hnyS9Jbp82PlJPScdttgw+Nfb47 9xg1oQUXB0M8+CyHVNBOUJDRVw5rtZ0= X-Google-Smtp-Source: ABdhPJwT1Zm0UPK4ODuO//ALggvJHiXbeQCZz14nbGGBW8ZLBLBdmCdeB12XvlBqYuyH2zw6VhQyqQ== X-Received: by 2002:a17:90a:aa8d:: with SMTP id l13mr1206923pjq.0.1610548810650; Wed, 13 Jan 2021 06:40:10 -0800 (PST) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com. [209.85.214.181]) by smtp.gmail.com with ESMTPSA id ob6sm2979563pjb.30.2021.01.13.06.40.10 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 13 Jan 2021 06:40:10 -0800 (PST) Received: by mail-pl1-f181.google.com with SMTP id g3so1189027plp.2 for ; Wed, 13 Jan 2021 06:40:10 -0800 (PST) X-Received: by 2002:a67:bd0a:: with SMTP id y10mr2141127vsq.28.1610548441862; Wed, 13 Jan 2021 06:34:01 -0800 (PST) MIME-Version: 1.0 References: <20210112194143.1494-1-yuri.benditovich@daynix.com> <78bbc518-4b73-4629-68fb-2713250f8967@redhat.com> In-Reply-To: <78bbc518-4b73-4629-68fb-2713250f8967@redhat.com> From: Willem de Bruijn Date: Wed, 13 Jan 2021 09:33:25 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC PATCH 0/7] Support for virtio-net hash reporting To: Jason Wang Cc: Willem de Bruijn , Yuri Benditovich , "David S. Miller" , Jakub Kicinski , "Michael S . Tsirkin" , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , rdunlap@infradead.org, "Gustavo A . R . Silva" , Herbert Xu , Steffen Klassert , Pablo Neira Ayuso , decui@microsoft.com, cai@lca.pw, Jakub Sitnicki , Marco Elver , Paolo Abeni , Network Development , linux-kernel , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, bpf , Yan Vugenfirer Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jan 12, 2021 at 11:11 PM Jason Wang wrote: > > > On 2021/1/13 =E4=B8=8A=E5=8D=887:47, Willem de Bruijn wrote: > > On Tue, Jan 12, 2021 at 3:29 PM Yuri Benditovich > > wrote: > >> On Tue, Jan 12, 2021 at 9:49 PM Yuri Benditovich > >> wrote: > >>> On Tue, Jan 12, 2021 at 9:41 PM Yuri Benditovich > >>> wrote: > >>>> Existing TUN module is able to use provided "steering eBPF" to > >>>> calculate per-packet hash and derive the destination queue to > >>>> place the packet to. The eBPF uses mapped configuration data > >>>> containing a key for hash calculation and indirection table > >>>> with array of queues' indices. > >>>> > >>>> This series of patches adds support for virtio-net hash reporting > >>>> feature as defined in virtio specification. It extends the TUN modul= e > >>>> and the "steering eBPF" as follows: > >>>> > >>>> Extended steering eBPF calculates the hash value and hash type, keep= s > >>>> hash value in the skb->hash and returns index of destination virtque= ue > >>>> and the type of the hash. TUN module keeps returned hash type in > >>>> (currently unused) field of the skb. > >>>> skb->__unused renamed to 'hash_report_type'. > >>>> > >>>> When TUN module is called later to allocate and fill the virtio-net > >>>> header and push it to destination virtqueue it populates the hash > >>>> and the hash type into virtio-net header. > >>>> > >>>> VHOST driver is made aware of respective virtio-net feature that > >>>> extends the virtio-net header to report the hash value and hash repo= rt > >>>> type. > >>> Comment from Willem de Bruijn: > >>> > >>> Skbuff fields are in short supply. I don't think we need to add one > >>> just for this narrow path entirely internal to the tun device. > >>> > >> We understand that and try to minimize the impact by using an already > >> existing unused field of skb. > > Not anymore. It was repurposed as a flags field very recently. > > > > This use case is also very narrow in scope. And a very short path from > > data producer to consumer. So I don't think it needs to claim scarce > > bits in the skb. > > > > tun_ebpf_select_queue stores the field, tun_put_user reads it and > > converts it to the virtio_net_hdr in the descriptor. > > > > tun_ebpf_select_queue is called from .ndo_select_queue. Storing the > > field in skb->cb is fragile, as in theory some code could overwrite > > that between field between ndo_select_queue and > > ndo_start_xmit/tun_net_xmit, from which point it is fully under tun > > control again. But in practice, I don't believe anything does. > > > > Alternatively an existing skb field that is used only on disjoint > > datapaths, such as ingress-only, could be viable. > > > A question here. We had metadata support in XDP for cooperation between > eBPF programs. Do we have something similar in the skb? > > E.g in the RSS, if we want to pass some metadata information between > eBPF program and the logic that generates the vnet header (either hard > logic in the kernel or another eBPF program). Is there any way that can > avoid the possible conflicts of qdiscs? Not that I am aware of. The closest thing is cb[]. It'll have to aliase a field like that, that is known unused for the given = path. One other approach that has been used within linear call stacks is out of band. Like percpu variables softnet_data.xmit.more and mirred_rec_level. But that is perhaps a bit overwrought for this use case. > > > >>> Instead, you could just run the flow_dissector in tun_put_user if the > >>> feature is negotiated. Indeed, the flow dissector seems more apt to m= e > >>> than BPF here. Note that the flow dissector internally can be > >>> overridden by a BPF program if the admin so chooses. > >>> > >> When this set of patches is related to hash delivery in the virtio-net > >> packet in general, > >> it was prepared in context of RSS feature implementation as defined in > >> virtio spec [1] > >> In case of RSS it is not enough to run the flow_dissector in tun_put_u= ser: > >> in tun_ebpf_select_queue the TUN calls eBPF to calculate the hash, > >> hash type and queue index > >> according to the (mapped) parameters (key, hash types, indirection > >> table) received from the guest. > > TUNSETSTEERINGEBPF was added to support more diverse queue selection > > than the default in case of multiqueue tun. Not sure what the exact > > use cases are. > > > > But RSS is exactly the purpose of the flow dissector. It is used for > > that purpose in the software variant RPS. The flow dissector > > implements a superset of the RSS spec, and certainly computes a > > four-tuple for TCP/IPv6. In the case of RPS, it is skipped if the NIC > > has already computed a 4-tuple hash. > > > > What it does not give is a type indication, such as > > VIRTIO_NET_HASH_TYPE_TCPv6. I don't understand how this would be used. > > In datapaths where the NIC has already computed the four-tuple hash > > and stored it in skb->hash --the common case for servers--, That type > > field is the only reason to have to compute again. > > > The problem is there's no guarantee that the packet comes from the NIC, > it could be a simple VM2VM or host2VM packet. > > And even if the packet is coming from the NIC that calculates the hash > there's no guarantee that it's the has that guest want (guest may use > different RSS keys). Ah yes, of course. I would still revisit the need to store a detailed hash_type along with the hash, as as far I can tell that conveys no actionable information to the guest. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EB38C433E0 for ; Wed, 13 Jan 2021 14:40:19 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A86722343B for ; Wed, 13 Jan 2021 14:40:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A86722343B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=virtualization-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 35DCA2E0E3; Wed, 13 Jan 2021 14:40:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Syx4aw4HiypI; Wed, 13 Jan 2021 14:40:17 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id 2A5ED2D059; Wed, 13 Jan 2021 14:40:17 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0A59CC088B; Wed, 13 Jan 2021 14:40:17 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9FDEDC013A for ; Wed, 13 Jan 2021 14:40:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 898AF84070 for ; Wed, 13 Jan 2021 14:40:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tX7geV3nzv6S for ; Wed, 13 Jan 2021 14:40:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 5EECA841AC for ; Wed, 13 Jan 2021 14:40:11 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id y8so1174779plp.8 for ; Wed, 13 Jan 2021 06:40:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=2ujGnSKYEthvi0rZ2iClFRGUsHpgEMsu727w+G030VY=; b=KUPL716HUHhkSt/RjEEjpWlYVDKM/abejwsbcqehL7DCSQm+pJqoncUorlVw/OV54k YViU+8HHMxFP67B4XdlZeuFHhKN8LFgxxyAuzbMhI/TONKoL8kkMF1z2PUJzY2Z+Gzd3 6FVdE8sdiBLfF+nzsTAeDoSVodGoWs6pbHJdyHrDWIya8fLkDBpMubiZhda8l6YVYSQL TQ6gMvrlDp0E9Pb02g8KV/58eBqEYu+BKtSUdEXFQfnPiTe80AhbbzV2fj9dw+KuF5nx E8XWoEDWqy7jLDC+GnO6H3xkacv1NVRiFqCnTzYxIU/DjZQs+qOL0hVLJ+Us+11Zm6ot B0ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=2ujGnSKYEthvi0rZ2iClFRGUsHpgEMsu727w+G030VY=; b=PLnhTVPEqMHoiB3oKyKLCrERhJUC9D2jLO0xsAe/4xpmoSPsGa6NszpswN5N094yf/ L4v+lDhRKekr8dGkeFxIIG2a/Pt24Wk3AwPUzAHo7kaUg4ZQcn/hUeobzAcLD5tfDNZf ih26FzyyTdRuImXgUBVU5j1csBrGJY2a/GXCgPn6VaeD5BfVay3W7SVUzDpcVjjpxibC gMoKwIGm4zpmW7uEtHOWssWMk02+r1orRbALt8EDPiRSSARLT5R0wLRW+9409iqBfICZ sjQlRXnhNenJZ7bCEX1l5AlY+HghI3Gw65XlPrqlSyJDbY9hGmWluw7FTbLHERzTs6zl J0LA== X-Gm-Message-State: AOAM533FAQ7mL/4qwWKqk79F8TrsE5QiWVDJT7kPmST50C1GZDY+MCfg EVbX3PT7zSnKaZcspkoHDJ923YddgeU= X-Google-Smtp-Source: ABdhPJwN4FfSjFH93GkGBWxS+czVke7jQTBXXJJWKTWPrtC7QhbAjm2b5y86ieX+9gPJfbNEIc8r3Q== X-Received: by 2002:a17:90a:74cd:: with SMTP id p13mr2701552pjl.25.1610548810338; Wed, 13 Jan 2021 06:40:10 -0800 (PST) Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com. [209.85.214.174]) by smtp.gmail.com with ESMTPSA id k18sm3119389pjz.26.2021.01.13.06.40.09 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 13 Jan 2021 06:40:10 -0800 (PST) Received: by mail-pl1-f174.google.com with SMTP id t6so1189183plq.1 for ; Wed, 13 Jan 2021 06:40:09 -0800 (PST) X-Received: by 2002:a67:bd0a:: with SMTP id y10mr2141127vsq.28.1610548441862; Wed, 13 Jan 2021 06:34:01 -0800 (PST) MIME-Version: 1.0 References: <20210112194143.1494-1-yuri.benditovich@daynix.com> <78bbc518-4b73-4629-68fb-2713250f8967@redhat.com> In-Reply-To: <78bbc518-4b73-4629-68fb-2713250f8967@redhat.com> From: Willem de Bruijn Date: Wed, 13 Jan 2021 09:33:25 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC PATCH 0/7] Support for virtio-net hash reporting To: Jason Wang Cc: Song Liu , kvm@vger.kernel.org, "Michael S . Tsirkin" , John Fastabend , Alexei Starovoitov , linux-kernel , Yan Vugenfirer , Steffen Klassert , Willem de Bruijn , Herbert Xu , Daniel Borkmann , decui@microsoft.com, Andrii Nakryiko , Yonghong Song , Paolo Abeni , Pablo Neira Ayuso , Marco Elver , KP Singh , Yuri Benditovich , cai@lca.pw, Jakub Kicinski , virtualization@lists.linux-foundation.org, Jakub Sitnicki , Network Development , rdunlap@infradead.org, "Gustavo A . R . Silva" , "David S. Miller" , bpf , Martin KaFai Lau X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" T24gVHVlLCBKYW4gMTIsIDIwMjEgYXQgMTE6MTEgUE0gSmFzb24gV2FuZyA8amFzb3dhbmdAcmVk aGF0LmNvbT4gd3JvdGU6Cj4KPgo+IE9uIDIwMjEvMS8xMyDkuIrljYg3OjQ3LCBXaWxsZW0gZGUg QnJ1aWpuIHdyb3RlOgo+ID4gT24gVHVlLCBKYW4gMTIsIDIwMjEgYXQgMzoyOSBQTSBZdXJpIEJl bmRpdG92aWNoCj4gPiA8eXVyaS5iZW5kaXRvdmljaEBkYXluaXguY29tPiB3cm90ZToKPiA+PiBP biBUdWUsIEphbiAxMiwgMjAyMSBhdCA5OjQ5IFBNIFl1cmkgQmVuZGl0b3ZpY2gKPiA+PiA8eXVy aS5iZW5kaXRvdmljaEBkYXluaXguY29tPiB3cm90ZToKPiA+Pj4gT24gVHVlLCBKYW4gMTIsIDIw MjEgYXQgOTo0MSBQTSBZdXJpIEJlbmRpdG92aWNoCj4gPj4+IDx5dXJpLmJlbmRpdG92aWNoQGRh eW5peC5jb20+IHdyb3RlOgo+ID4+Pj4gRXhpc3RpbmcgVFVOIG1vZHVsZSBpcyBhYmxlIHRvIHVz ZSBwcm92aWRlZCAic3RlZXJpbmcgZUJQRiIgdG8KPiA+Pj4+IGNhbGN1bGF0ZSBwZXItcGFja2V0 IGhhc2ggYW5kIGRlcml2ZSB0aGUgZGVzdGluYXRpb24gcXVldWUgdG8KPiA+Pj4+IHBsYWNlIHRo ZSBwYWNrZXQgdG8uIFRoZSBlQlBGIHVzZXMgbWFwcGVkIGNvbmZpZ3VyYXRpb24gZGF0YQo+ID4+ Pj4gY29udGFpbmluZyBhIGtleSBmb3IgaGFzaCBjYWxjdWxhdGlvbiBhbmQgaW5kaXJlY3Rpb24g dGFibGUKPiA+Pj4+IHdpdGggYXJyYXkgb2YgcXVldWVzJyBpbmRpY2VzLgo+ID4+Pj4KPiA+Pj4+ IFRoaXMgc2VyaWVzIG9mIHBhdGNoZXMgYWRkcyBzdXBwb3J0IGZvciB2aXJ0aW8tbmV0IGhhc2gg cmVwb3J0aW5nCj4gPj4+PiBmZWF0dXJlIGFzIGRlZmluZWQgaW4gdmlydGlvIHNwZWNpZmljYXRp b24uIEl0IGV4dGVuZHMgdGhlIFRVTiBtb2R1bGUKPiA+Pj4+IGFuZCB0aGUgInN0ZWVyaW5nIGVC UEYiIGFzIGZvbGxvd3M6Cj4gPj4+Pgo+ID4+Pj4gRXh0ZW5kZWQgc3RlZXJpbmcgZUJQRiBjYWxj dWxhdGVzIHRoZSBoYXNoIHZhbHVlIGFuZCBoYXNoIHR5cGUsIGtlZXBzCj4gPj4+PiBoYXNoIHZh bHVlIGluIHRoZSBza2ItPmhhc2ggYW5kIHJldHVybnMgaW5kZXggb2YgZGVzdGluYXRpb24gdmly dHF1ZXVlCj4gPj4+PiBhbmQgdGhlIHR5cGUgb2YgdGhlIGhhc2guIFRVTiBtb2R1bGUga2VlcHMg cmV0dXJuZWQgaGFzaCB0eXBlIGluCj4gPj4+PiAoY3VycmVudGx5IHVudXNlZCkgZmllbGQgb2Yg dGhlIHNrYi4KPiA+Pj4+IHNrYi0+X191bnVzZWQgcmVuYW1lZCB0byAnaGFzaF9yZXBvcnRfdHlw ZScuCj4gPj4+Pgo+ID4+Pj4gV2hlbiBUVU4gbW9kdWxlIGlzIGNhbGxlZCBsYXRlciB0byBhbGxv Y2F0ZSBhbmQgZmlsbCB0aGUgdmlydGlvLW5ldAo+ID4+Pj4gaGVhZGVyIGFuZCBwdXNoIGl0IHRv IGRlc3RpbmF0aW9uIHZpcnRxdWV1ZSBpdCBwb3B1bGF0ZXMgdGhlIGhhc2gKPiA+Pj4+IGFuZCB0 aGUgaGFzaCB0eXBlIGludG8gdmlydGlvLW5ldCBoZWFkZXIuCj4gPj4+Pgo+ID4+Pj4gVkhPU1Qg ZHJpdmVyIGlzIG1hZGUgYXdhcmUgb2YgcmVzcGVjdGl2ZSB2aXJ0aW8tbmV0IGZlYXR1cmUgdGhh dAo+ID4+Pj4gZXh0ZW5kcyB0aGUgdmlydGlvLW5ldCBoZWFkZXIgdG8gcmVwb3J0IHRoZSBoYXNo IHZhbHVlIGFuZCBoYXNoIHJlcG9ydAo+ID4+Pj4gdHlwZS4KPiA+Pj4gQ29tbWVudCBmcm9tIFdp bGxlbSBkZSBCcnVpam46Cj4gPj4+Cj4gPj4+IFNrYnVmZiBmaWVsZHMgYXJlIGluIHNob3J0IHN1 cHBseS4gSSBkb24ndCB0aGluayB3ZSBuZWVkIHRvIGFkZCBvbmUKPiA+Pj4ganVzdCBmb3IgdGhp cyBuYXJyb3cgcGF0aCBlbnRpcmVseSBpbnRlcm5hbCB0byB0aGUgdHVuIGRldmljZS4KPiA+Pj4K PiA+PiBXZSB1bmRlcnN0YW5kIHRoYXQgYW5kIHRyeSB0byBtaW5pbWl6ZSB0aGUgaW1wYWN0IGJ5 IHVzaW5nIGFuIGFscmVhZHkKPiA+PiBleGlzdGluZyB1bnVzZWQgZmllbGQgb2Ygc2tiLgo+ID4g Tm90IGFueW1vcmUuIEl0IHdhcyByZXB1cnBvc2VkIGFzIGEgZmxhZ3MgZmllbGQgdmVyeSByZWNl bnRseS4KPiA+Cj4gPiBUaGlzIHVzZSBjYXNlIGlzIGFsc28gdmVyeSBuYXJyb3cgaW4gc2NvcGUu IEFuZCBhIHZlcnkgc2hvcnQgcGF0aCBmcm9tCj4gPiBkYXRhIHByb2R1Y2VyIHRvIGNvbnN1bWVy LiBTbyBJIGRvbid0IHRoaW5rIGl0IG5lZWRzIHRvIGNsYWltIHNjYXJjZQo+ID4gYml0cyBpbiB0 aGUgc2tiLgo+ID4KPiA+IHR1bl9lYnBmX3NlbGVjdF9xdWV1ZSBzdG9yZXMgdGhlIGZpZWxkLCB0 dW5fcHV0X3VzZXIgcmVhZHMgaXQgYW5kCj4gPiBjb252ZXJ0cyBpdCB0byB0aGUgdmlydGlvX25l dF9oZHIgaW4gdGhlIGRlc2NyaXB0b3IuCj4gPgo+ID4gdHVuX2VicGZfc2VsZWN0X3F1ZXVlIGlz IGNhbGxlZCBmcm9tIC5uZG9fc2VsZWN0X3F1ZXVlLiAgU3RvcmluZyB0aGUKPiA+IGZpZWxkIGlu IHNrYi0+Y2IgaXMgZnJhZ2lsZSwgYXMgaW4gdGhlb3J5IHNvbWUgY29kZSBjb3VsZCBvdmVyd3Jp dGUKPiA+IHRoYXQgYmV0d2VlbiBmaWVsZCBiZXR3ZWVuIG5kb19zZWxlY3RfcXVldWUgYW5kCj4g PiBuZG9fc3RhcnRfeG1pdC90dW5fbmV0X3htaXQsIGZyb20gd2hpY2ggcG9pbnQgaXQgaXMgZnVs bHkgdW5kZXIgdHVuCj4gPiBjb250cm9sIGFnYWluLiBCdXQgaW4gcHJhY3RpY2UsIEkgZG9uJ3Qg YmVsaWV2ZSBhbnl0aGluZyBkb2VzLgo+ID4KPiA+IEFsdGVybmF0aXZlbHkgYW4gZXhpc3Rpbmcg c2tiIGZpZWxkIHRoYXQgaXMgdXNlZCBvbmx5IG9uIGRpc2pvaW50Cj4gPiBkYXRhcGF0aHMsIHN1 Y2ggYXMgaW5ncmVzcy1vbmx5LCBjb3VsZCBiZSB2aWFibGUuCj4KPgo+IEEgcXVlc3Rpb24gaGVy ZS4gV2UgaGFkIG1ldGFkYXRhIHN1cHBvcnQgaW4gWERQIGZvciBjb29wZXJhdGlvbiBiZXR3ZWVu Cj4gZUJQRiBwcm9ncmFtcy4gRG8gd2UgaGF2ZSBzb21ldGhpbmcgc2ltaWxhciBpbiB0aGUgc2ti Pwo+Cj4gRS5nIGluIHRoZSBSU1MsIGlmIHdlIHdhbnQgdG8gcGFzcyBzb21lIG1ldGFkYXRhIGlu Zm9ybWF0aW9uIGJldHdlZW4KPiBlQlBGIHByb2dyYW0gYW5kIHRoZSBsb2dpYyB0aGF0IGdlbmVy YXRlcyB0aGUgdm5ldCBoZWFkZXIgKGVpdGhlciBoYXJkCj4gbG9naWMgaW4gdGhlIGtlcm5lbCBv ciBhbm90aGVyIGVCUEYgcHJvZ3JhbSkuIElzIHRoZXJlIGFueSB3YXkgdGhhdCBjYW4KPiBhdm9p ZCB0aGUgcG9zc2libGUgY29uZmxpY3RzIG9mIHFkaXNjcz8KCk5vdCB0aGF0IEkgYW0gYXdhcmUg b2YuIFRoZSBjbG9zZXN0IHRoaW5nIGlzIGNiW10uCgpJdCdsbCBoYXZlIHRvIGFsaWFzZSBhIGZp ZWxkIGxpa2UgdGhhdCwgdGhhdCBpcyBrbm93biB1bnVzZWQgZm9yIHRoZSBnaXZlbiBwYXRoLgoK T25lIG90aGVyIGFwcHJvYWNoIHRoYXQgaGFzIGJlZW4gdXNlZCB3aXRoaW4gbGluZWFyIGNhbGwg c3RhY2tzIGlzIG91dApvZiBiYW5kLiBMaWtlIHBlcmNwdSB2YXJpYWJsZXMgc29mdG5ldF9kYXRh LnhtaXQubW9yZSBhbmQKbWlycmVkX3JlY19sZXZlbC4gQnV0IHRoYXQgaXMgcGVyaGFwcyBhIGJp dCBvdmVyd3JvdWdodCBmb3IgdGhpcyB1c2UKY2FzZS4KCj4gPgo+ID4+PiBJbnN0ZWFkLCB5b3Ug Y291bGQganVzdCBydW4gdGhlIGZsb3dfZGlzc2VjdG9yIGluIHR1bl9wdXRfdXNlciBpZiB0aGUK PiA+Pj4gZmVhdHVyZSBpcyBuZWdvdGlhdGVkLiBJbmRlZWQsIHRoZSBmbG93IGRpc3NlY3RvciBz ZWVtcyBtb3JlIGFwdCB0byBtZQo+ID4+PiB0aGFuIEJQRiBoZXJlLiBOb3RlIHRoYXQgdGhlIGZs b3cgZGlzc2VjdG9yIGludGVybmFsbHkgY2FuIGJlCj4gPj4+IG92ZXJyaWRkZW4gYnkgYSBCUEYg cHJvZ3JhbSBpZiB0aGUgYWRtaW4gc28gY2hvb3Nlcy4KPiA+Pj4KPiA+PiBXaGVuIHRoaXMgc2V0 IG9mIHBhdGNoZXMgaXMgcmVsYXRlZCB0byBoYXNoIGRlbGl2ZXJ5IGluIHRoZSB2aXJ0aW8tbmV0 Cj4gPj4gcGFja2V0IGluIGdlbmVyYWwsCj4gPj4gaXQgd2FzIHByZXBhcmVkIGluIGNvbnRleHQg b2YgUlNTIGZlYXR1cmUgaW1wbGVtZW50YXRpb24gYXMgZGVmaW5lZCBpbgo+ID4+IHZpcnRpbyBz cGVjIFsxXQo+ID4+IEluIGNhc2Ugb2YgUlNTIGl0IGlzIG5vdCBlbm91Z2ggdG8gcnVuIHRoZSBm bG93X2Rpc3NlY3RvciBpbiB0dW5fcHV0X3VzZXI6Cj4gPj4gaW4gdHVuX2VicGZfc2VsZWN0X3F1 ZXVlIHRoZSBUVU4gY2FsbHMgZUJQRiB0byBjYWxjdWxhdGUgdGhlIGhhc2gsCj4gPj4gaGFzaCB0 eXBlIGFuZCBxdWV1ZSBpbmRleAo+ID4+IGFjY29yZGluZyB0byB0aGUgKG1hcHBlZCkgcGFyYW1l dGVycyAoa2V5LCBoYXNoIHR5cGVzLCBpbmRpcmVjdGlvbgo+ID4+IHRhYmxlKSByZWNlaXZlZCBm cm9tIHRoZSBndWVzdC4KPiA+IFRVTlNFVFNURUVSSU5HRUJQRiB3YXMgYWRkZWQgdG8gc3VwcG9y dCBtb3JlIGRpdmVyc2UgcXVldWUgc2VsZWN0aW9uCj4gPiB0aGFuIHRoZSBkZWZhdWx0IGluIGNh c2Ugb2YgbXVsdGlxdWV1ZSB0dW4uIE5vdCBzdXJlIHdoYXQgdGhlIGV4YWN0Cj4gPiB1c2UgY2Fz ZXMgYXJlLgo+ID4KPiA+IEJ1dCBSU1MgaXMgZXhhY3RseSB0aGUgcHVycG9zZSBvZiB0aGUgZmxv dyBkaXNzZWN0b3IuIEl0IGlzIHVzZWQgZm9yCj4gPiB0aGF0IHB1cnBvc2UgaW4gdGhlIHNvZnR3 YXJlIHZhcmlhbnQgUlBTLiBUaGUgZmxvdyBkaXNzZWN0b3IKPiA+IGltcGxlbWVudHMgYSBzdXBl cnNldCBvZiB0aGUgUlNTIHNwZWMsIGFuZCBjZXJ0YWlubHkgY29tcHV0ZXMgYQo+ID4gZm91ci10 dXBsZSBmb3IgVENQL0lQdjYuIEluIHRoZSBjYXNlIG9mIFJQUywgaXQgaXMgc2tpcHBlZCBpZiB0 aGUgTklDCj4gPiBoYXMgYWxyZWFkeSBjb21wdXRlZCBhIDQtdHVwbGUgaGFzaC4KPiA+Cj4gPiBX aGF0IGl0IGRvZXMgbm90IGdpdmUgaXMgYSB0eXBlIGluZGljYXRpb24sIHN1Y2ggYXMKPiA+IFZJ UlRJT19ORVRfSEFTSF9UWVBFX1RDUHY2LiBJIGRvbid0IHVuZGVyc3RhbmQgaG93IHRoaXMgd291 bGQgYmUgdXNlZC4KPiA+IEluIGRhdGFwYXRocyB3aGVyZSB0aGUgTklDIGhhcyBhbHJlYWR5IGNv bXB1dGVkIHRoZSBmb3VyLXR1cGxlIGhhc2gKPiA+IGFuZCBzdG9yZWQgaXQgaW4gc2tiLT5oYXNo IC0tdGhlIGNvbW1vbiBjYXNlIGZvciBzZXJ2ZXJzLS0sIFRoYXQgdHlwZQo+ID4gZmllbGQgaXMg dGhlIG9ubHkgcmVhc29uIHRvIGhhdmUgdG8gY29tcHV0ZSBhZ2Fpbi4KPgo+Cj4gVGhlIHByb2Js ZW0gaXMgdGhlcmUncyBubyBndWFyYW50ZWUgdGhhdCB0aGUgcGFja2V0IGNvbWVzIGZyb20gdGhl IE5JQywKPiBpdCBjb3VsZCBiZSBhIHNpbXBsZSBWTTJWTSBvciBob3N0MlZNIHBhY2tldC4KPgo+ IEFuZCBldmVuIGlmIHRoZSBwYWNrZXQgaXMgY29taW5nIGZyb20gdGhlIE5JQyB0aGF0IGNhbGN1 bGF0ZXMgdGhlIGhhc2gKPiB0aGVyZSdzIG5vIGd1YXJhbnRlZSB0aGF0IGl0J3MgdGhlIGhhcyB0 aGF0IGd1ZXN0IHdhbnQgKGd1ZXN0IG1heSB1c2UKPiBkaWZmZXJlbnQgUlNTIGtleXMpLgoKQWgg eWVzLCBvZiBjb3Vyc2UuCgpJIHdvdWxkIHN0aWxsIHJldmlzaXQgdGhlIG5lZWQgdG8gc3RvcmUg YSBkZXRhaWxlZCBoYXNoX3R5cGUgYWxvbmcgd2l0aAp0aGUgaGFzaCwgYXMgYXMgZmFyIEkgY2Fu IHRlbGwgdGhhdCBjb252ZXlzIG5vIGFjdGlvbmFibGUgaW5mb3JtYXRpb24KdG8gdGhlIGd1ZXN0 LgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpWaXJ0dWFs aXphdGlvbiBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRhdGlv bi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8v dmlydHVhbGl6YXRpb24=