From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 587D7C433EF for ; Tue, 8 Mar 2022 07:34:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 066DA408E9; Tue, 8 Mar 2022 07:34:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RLfBswCPjQ-C; Tue, 8 Mar 2022 07:34:38 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id BCFF9408E2; Tue, 8 Mar 2022 07:34:37 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 86C6CC001D; Tue, 8 Mar 2022 07:34:37 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9A764C000B for ; Tue, 8 Mar 2022 07:34:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 8940A813F1 for ; Tue, 8 Mar 2022 07:34:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lGtaVB2XqvxS for ; Tue, 8 Mar 2022 07:34:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id EDD38813F0 for ; Tue, 8 Mar 2022 07:34:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1646724873; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dfKE5Kp7qwEgqHd/rfsj2BcHX1IWW69+V0DA8sWS2/s=; b=PmSeyMJQo53odAGtJE1fJt0hHqtRNFI8l9Mh4QLiaUgsWqsd7fXIByVEUQVYV5gEiPjT/p 5WWfYmoxmwqGC5zqM1D70j4HZb5r67TlzqVypn1MTqwj96BrhdeS8xXjKDUAPrtI8oUUHn q5UBW4InCGrC6vV9+LIyhskre55S17o= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-618-8FtPXA51M7G2LMtK9MdcSw-1; Tue, 08 Mar 2022 02:34:32 -0500 X-MC-Unique: 8FtPXA51M7G2LMtK9MdcSw-1 Received: by mail-lf1-f71.google.com with SMTP id y23-20020a0565123f1700b00448221b91e5so1628478lfa.13 for ; Mon, 07 Mar 2022 23:34:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=dfKE5Kp7qwEgqHd/rfsj2BcHX1IWW69+V0DA8sWS2/s=; b=sIom5UUsD9eu+h6m9NHYA28FZ930Y2bp2XKtbmuAHH38YdcIyXFmfzcBl60dVAsInd ewHIEpE8JlzztpxT045vV3GfHcdH3b6ZuS9wfJ7PFLVg9t+JnfW6ha+ViYlGMi9XetAR vzL696vETROyrjhvVsezubiXJHVPCKZLmYCWAKpDNHQysFwNLD/hdM97WCTfCLCR1+Qx ZeSNSr68zdkNPG7Qhl2ljZff7YWiMV+ySHGQP+V6km+T4ZQX2D89eRYaKNUDNeqwS0Q4 B85IEMEKixri+F0YOI+A8ZtZj0MuNciRzc1PPiJ+9HvwRmSNq2AVJ6QqlkZNzrevP0sL cGUg== X-Gm-Message-State: AOAM531EOvX5AXTkbxv/N2rkYA5YcmwDptrdHFQP3WnmvNdvu9wAYlkK b1AXIRnoxM9AY83rbiQGdYye3y3Ui58ThwAhfrpfWkiHMy1Qnp4kHJiX2YHLXB19erSJRalOl4x 47yquOO9UQrUYKrpj3X3wLHCUtEzuDRNV/91K47txpUR8LRvnbD9ROcbcvQ== X-Received: by 2002:a05:6512:3890:b0:448:21bd:28d with SMTP id n16-20020a056512389000b0044821bd028dmr9841720lft.376.1646724869532; Mon, 07 Mar 2022 23:34:29 -0800 (PST) X-Google-Smtp-Source: ABdhPJweEt8oq3dJWo9keGtJcmLQBKP6tmkQblYJ9WoLzYvOuHGtpuFBBadzif55GpUgMxivqoE/gWxiXR+IWX280Uo= X-Received: by 2002:a05:6512:3890:b0:448:21bd:28d with SMTP id n16-20020a056512389000b0044821bd028dmr9841700lft.376.1646724869192; Mon, 07 Mar 2022 23:34:29 -0800 (PST) MIME-Version: 1.0 References: <20220307153334.3854134-1-eperezma@redhat.com> <14d4fde4-6ea5-4805-b684-c33f6b448565@redhat.com> <20220308020348-mutt-send-email-mst@kernel.org> <20220308022300-mutt-send-email-mst@kernel.org> In-Reply-To: <20220308022300-mutt-send-email-mst@kernel.org> From: Jason Wang Date: Tue, 8 Mar 2022 15:34:17 +0800 Message-ID: Subject: Re: [PATCH v5 00/15] vDPA shadow virtqueue To: "Michael S. Tsirkin" Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: qemu-devel , virtualization , Eli Cohen , Eric Blake , Eduardo Habkost , Cindy Lu , "Fangyi \(Eric\)" , Markus Armbruster , yebiaoxiang@huawei.com, =?UTF-8?Q?Eugenio_P=C3=A9rez?= , Liuxiangdong , Laurent Vivier , Parav Pandit , Richard Henderson , Gautam Dawar , Xiao W Wang , Stefan Hajnoczi , Harpreet Singh Anand , Lingshan X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" T24gVHVlLCBNYXIgOCwgMjAyMiBhdCAzOjI4IFBNIE1pY2hhZWwgUy4gVHNpcmtpbiA8bXN0QHJl ZGhhdC5jb20+IHdyb3RlOgo+Cj4gT24gVHVlLCBNYXIgMDgsIDIwMjIgYXQgMDM6MTQ6MzVQTSAr MDgwMCwgSmFzb24gV2FuZyB3cm90ZToKPiA+IE9uIFR1ZSwgTWFyIDgsIDIwMjIgYXQgMzoxMSBQ TSBNaWNoYWVsIFMuIFRzaXJraW4gPG1zdEByZWRoYXQuY29tPiB3cm90ZToKPiA+ID4KPiA+ID4g T24gVHVlLCBNYXIgMDgsIDIwMjIgYXQgMDI6MDM6MzJQTSArMDgwMCwgSmFzb24gV2FuZyB3cm90 ZToKPiA+ID4gPgo+ID4gPiA+IOWcqCAyMDIyLzMvNyDkuIvljYgxMTozMywgRXVnZW5pbyBQw6ly ZXog5YaZ6YGTOgo+ID4gPiA+ID4gVGhpcyBzZXJpZXMgZW5hYmxlIHNoYWRvdyB2aXJ0cXVldWUg KFNWUSkgZm9yIHZob3N0LXZkcGEgZGV2aWNlcy4gVGhpcwo+ID4gPiA+ID4gaXMgaW50ZW5kZWQg YXMgYSBuZXcgbWV0aG9kIG9mIHRyYWNraW5nIHRoZSBtZW1vcnkgdGhlIGRldmljZXMgdG91Y2gK PiA+ID4gPiA+IGR1cmluZyBhIG1pZ3JhdGlvbiBwcm9jZXNzOiBJbnN0ZWFkIG9mIHJlbGF5IG9u IHZob3N0IGRldmljZSdzIGRpcnR5Cj4gPiA+ID4gPiBsb2dnaW5nIGNhcGFiaWxpdHksIFNWUSBp bnRlcmNlcHRzIHRoZSBWUSBkYXRhcGxhbmUgZm9yd2FyZGluZyB0aGUKPiA+ID4gPiA+IGRlc2Ny aXB0b3JzIGJldHdlZW4gVk0gYW5kIGRldmljZS4gVGhpcyB3YXkgcWVtdSBpcyB0aGUgZWZmZWN0 aXZlCj4gPiA+ID4gPiB3cml0ZXIgb2YgZ3Vlc3RzIG1lbW9yeSwgbGlrZSBpbiBxZW11J3Mgdmly dGlvIGRldmljZSBvcGVyYXRpb24uCj4gPiA+ID4gPgo+ID4gPiA+ID4gV2hlbiBTVlEgaXMgZW5h YmxlZCBxZW11IG9mZmVycyBhIG5ldyB2aXJ0dWFsIGFkZHJlc3Mgc3BhY2UgdG8gdGhlCj4gPiA+ ID4gPiBkZXZpY2UgdG8gcmVhZCBhbmQgd3JpdGUgaW50bywgYW5kIGl0IG1hcHMgbmV3IHZyaW5n cyBhbmQgdGhlIGd1ZXN0Cj4gPiA+ID4gPiBtZW1vcnkgaW4gaXQuIFNWUSBhbHNvIGludGVyY2Vw dHMga2lja3MgYW5kIGNhbGxzIGJldHdlZW4gdGhlIGRldmljZQo+ID4gPiA+ID4gYW5kIHRoZSBn dWVzdC4gVXNlZCBidWZmZXJzIHJlbGF5IHdvdWxkIGNhdXNlIGRpcnR5IG1lbW9yeSBiZWluZwo+ ID4gPiA+ID4gdHJhY2tlZC4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBUaGlzIGVmZmVjdGl2ZWx5IG1l YW5zIHRoYXQgdkRQQSBkZXZpY2UgcGFzc3Rocm91Z2ggaXMgaW50ZXJjZXB0ZWQgYnkKPiA+ID4g PiA+IHFlbXUuIFdoaWxlIFNWUSBzaG91bGQgb25seSBiZSBlbmFibGVkIGF0IG1pZ3JhdGlvbiB0 aW1lLCB0aGUgc3dpdGNoaW5nCj4gPiA+ID4gPiBmcm9tIHJlZ3VsYXIgbW9kZSB0byBTVlEgbW9k ZSBpcyBsZWZ0IGZvciBhIGZ1dHVyZSBzZXJpZXMuCj4gPiA+ID4gPgo+ID4gPiA+ID4gSXQgaXMg YmFzZWQgb24gdGhlIGlkZWFzIG9mIERQREsgU1cgYXNzaXN0ZWQgTE0sIGluIHRoZSBzZXJpZXMg b2YKPiA+ID4gPiA+IERQREsncyBodHRwczovL3BhdGNod29yay5kcGRrLm9yZy9jb3Zlci80ODM3 MC8gLiBIb3dldmVyLCB0aGVzZSBkb2VzCj4gPiA+ID4gPiBub3QgbWFwIHRoZSBzaGFkb3cgdnEg aW4gZ3Vlc3QncyBWQSwgYnV0IGluIHFlbXUncy4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBGb3IgcWVt dSB0byB1c2Ugc2hhZG93IHZpcnRxdWV1ZXMgdGhlIGd1ZXN0IHZpcnRpbyBkcml2ZXIgbXVzdCBu b3QgdXNlCj4gPiA+ID4gPiBmZWF0dXJlcyBsaWtlIGV2ZW50X2lkeC4KPiA+ID4gPiA+Cj4gPiA+ ID4gPiBTVlEgbmVlZHMgdG8gYmUgZW5hYmxlZCB3aXRoIGNtZGxpbmU6Cj4gPiA+ID4gPgo+ID4g PiA+ID4gLW5ldGRldiB0eXBlPXZob3N0LXZkcGEsdmhvc3RkZXY9dmhvc3QtdmRwYS0wLGlkPXZo b3N0LXZkcGEwLHN2cT1vbgo+ID4gPgo+ID4gPiBBIHN0YWJsZSBBUEkgZm9yIGFuIGluY29tcGxl dGUgZmVhdHVyZSBpcyBhIHByb2JsZW0gaW1oby4KPiA+Cj4gPiBJdCBzaG91bGQgYmUgIngtc3Zx Ii4KPgo+Cj4gV2VsbCBsb29rIGF0IHBhdGNoIDE1LgoKSXQncyBhIGJ1ZyB0aGF0IG5lZWRzIHRv IGJlIGZpeGVkLgoKPgo+ID4gPgo+ID4gPgo+ID4gPiA+ID4KPiA+ID4gPiA+IFRoZSBmaXJzdCB0 aHJlZSBwYXRjaGVzIGVuYWJsZXMgbm90aWZpY2F0aW9ucyBmb3J3YXJkaW5nIHdpdGgKPiA+ID4g PiA+IGFzc2lzdGFuY2Ugb2YgcWVtdS4gSXQncyBlYXN5IHRvIGVuYWJsZSBvbmx5IHRoaXMgaWYg dGhlIHJlbGV2YW50Cj4gPiA+ID4gPiBjbWRsaW5lIHBhcnQgb2YgdGhlIGxhc3QgcGF0Y2ggaXMg YXBwbGllZCBvbiB0b3Agb2YgdGhlc2UuCj4gPiA+ID4gPgo+ID4gPiA+ID4gTmV4dCBmb3VyIHBh dGNoZXMgaW1wbGVtZW50IHRoZSBhY3R1YWwgYnVmZmVyIGZvcndhcmRpbmcuIEhvd2V2ZXIsCj4g PiA+ID4gPiBhZGRyZXNzIGFyZSBub3QgdHJhbnNsYXRlZCBmcm9tIEhWQSBzbyB0aGV5IHdpbGwg bmVlZCBhIGhvc3QgZGV2aWNlIHdpdGgKPiA+ID4gPiA+IGFuIGlvbW11IGFsbG93aW5nIHRoZW0g dG8gYWNjZXNzIGFsbCBvZiB0aGUgSFZBIHJhbmdlLgo+ID4gPiA+ID4KPiA+ID4gPiA+IFRoZSBs YXN0IHBhcnQgb2YgdGhlIHNlcmllcyB1c2VzIHByb3Blcmx5IHRoZSBob3N0IGlvbW11LCBzbyBx ZW11Cj4gPiA+ID4gPiBjcmVhdGVzIGEgbmV3IGlvdmEgYWRkcmVzcyBzcGFjZSBpbiB0aGUgZGV2 aWNlJ3MgcmFuZ2UgYW5kIHRyYW5zbGF0ZXMKPiA+ID4gPiA+IHRoZSBidWZmZXJzIGluIGl0LiBG aW5hbGx5LCBpdCBhZGRzIHRoZSBjbWRsaW5lIHBhcmFtZXRlci4KPiA+ID4gPiA+Cj4gPiA+ID4g PiBTb21lIHNpbXBsZSBwZXJmb3JtYW5jZSB0ZXN0cyB3aXRoIG5ldHBlcmYgd2VyZSBkb25lLiBU aGV5IHVzZWQgYSBuZXN0ZWQKPiA+ID4gPiA+IGd1ZXN0IHdpdGggdnBfdmRwYSwgdmhvc3Qta2Vy bmVsIGF0IEwwIGhvc3QuIFN0YXJ0aW5nIHdpdGggbm8gc3ZxIGFuZCBhCj4gPiA+ID4gPiBiYXNl bGluZSBhdmVyYWdlIG9mIH45MDA5Ljk2TWJwczoKPiA+ID4gPiA+IFJlY3YgICBTZW5kICAgIFNl bmQKPiA+ID4gPiA+IFNvY2tldCBTb2NrZXQgIE1lc3NhZ2UgIEVsYXBzZWQKPiA+ID4gPiA+IFNp emUgICBTaXplICAgIFNpemUgICAgIFRpbWUgICAgIFRocm91Z2hwdXQKPiA+ID4gPiA+IGJ5dGVz ICBieXRlcyAgIGJ5dGVzICAgIHNlY3MuICAgIDEwXjZiaXRzL3NlYwo+ID4gPiA+ID4gMTMxMDcy ICAxNjM4NCAgMTYzODQgICAgMzAuMDEgICAgOTA2MS4wMwo+ID4gPiA+ID4gMTMxMDcyICAxNjM4 NCAgMTYzODQgICAgMzAuMDEgICAgODk2Mi45NAo+ID4gPiA+ID4gMTMxMDcyICAxNjM4NCAgMTYz ODQgICAgMzAuMDEgICAgOTAwNS45Mgo+ID4gPiA+ID4KPiA+ID4gPiA+IFRvIGVuYWJsZSBTVlEg YnVmZmVycyBmb3J3YXJkaW5nIHJlZHVjZSB0aHJvdWdocHV0IHRvIGFib3V0Cj4gPiA+ID4gPiBS ZWN2ICAgU2VuZCAgICBTZW5kCj4gPiA+ID4gPiBTb2NrZXQgU29ja2V0ICBNZXNzYWdlICBFbGFw c2VkCj4gPiA+ID4gPiBTaXplICAgU2l6ZSAgICBTaXplICAgICBUaW1lICAgICBUaHJvdWdocHV0 Cj4gPiA+ID4gPiBieXRlcyAgYnl0ZXMgICBieXRlcyAgICBzZWNzLiAgICAxMF42Yml0cy9zZWMK PiA+ID4gPiA+IDEzMTA3MiAgMTYzODQgIDE2Mzg0ICAgIDMwLjAxICAgIDc2ODkuNzIKPiA+ID4g PiA+IDEzMTA3MiAgMTYzODQgIDE2Mzg0ICAgIDMwLjAwICAgIDc3NTIuMDcKPiA+ID4gPiA+IDEz MTA3MiAgMTYzODQgIDE2Mzg0ICAgIDMwLjAxICAgIDc3NTAuMzAKPiA+ID4gPiA+Cj4gPiA+ID4g PiBIb3dldmVyLCBtYW55IHBlcmZvcm1hbmNlIGltcHJvdmVtZW50cyB3ZXJlIGxlZnQgb3V0IG9m IHRoaXMgc2VyaWVzIGZvcgo+ID4gPiA+ID4gc2ltcGxpY2l0eSwgc28gZGlmZmVyZW5jZSBzaG91 bGQgc2hyaW5rIGluIHRoZSBmdXR1cmUuCj4gPiA+ID4gPgo+ID4gPiA+ID4gQ29tbWVudHMgYXJl IHdlbGNvbWUuCj4gPiA+ID4KPiA+ID4gPgo+ID4gPiA+IEhpIE1pY2hhZWw6Cj4gPiA+ID4KPiA+ ID4gPiBXaGF0IGRvIHlvdSB0aGluayBvZiB0aGlzIHNlcmllcz8gSXQgbG9va3MgZ29vZCB0byBt ZSBhcyBhIHN0YXJ0LiBUaGUKPiA+ID4gPiBmZWF0dXJlIGNvdWxkIG9ubHkgYmUgZW5hYmxlZCBh cyBhIGRlZGljYXRlZCBwYXJhbWV0ZXIuIElmIHlvdSdyZSBvaywgSSdkCj4gPiA+ID4gdHJ5IHRv IG1ha2UgaXQgZm9yIDcuMC4KPiA+ID4gPgo+ID4gPiA+IFRoYW5rcwo+ID4gPgo+ID4gPiBXZWxs IHRoYXQncyBjdXR0aW5nIGl0IGF3ZnVsbHkgY2xvc2UsIGFuZCBpdCdzIG5vdCByZWFsbHkgdXNl ZnVsCj4gPiA+IGF0IHRoZSBjdXJyZW50IHN0YWdlLCBpcyBpdD8KPiA+Cj4gPiBUaGlzIGFsbG93 cyB2RFBBIHRvIGJlIG1pZ3JhdGVkIHdoZW4gdXNpbmcgIngtc3ZxPW9uIi4KPiA+IEJ1dCBhbnlo b3cgaXQncwo+ID4gZXhwZXJpbWVudGFsLgo+Cj4gaXQncyBsZXNzIGV4cGVyaW1lbnRhbCB0aGFu IGluY29tcGxldGUuIEl0IHNlZW1zIHByZXR0eSBjbGVhcmx5IG5vdAo+IHRoZSB3YXkgaXQgd2ls bCB3b3JrIGRvd24gdGhlIHJvYWQsIHdlIGRvbid0IHdhbnQgc3ZxIGludm9sdmVkCj4gYXQgYWxs IHRpbWVzLgoKUmlnaHQsIGJ1dCBTVlEgY291bGQgYmUgdXNlZCBmb3Igb3RoZXIgcGxhY2VzIGUu ZyBwcm92aWRpbmcgbWlncmF0aW9uCmNvbXBhdGliaWxpdHkgd2hlbiB0aGUgZGVzdGluYXRpb24g bGFja3Mgc29tZSBmZWF0dXJlcy4KCj4KPiA+ID4KPiA+ID4gVGhlIElPVkEgdHJpY2sgZG9lcyBu b3QgZmVlbCBjb21wbGV0ZSBlaXRoZXIuCj4gPgo+ID4gSSBkb24ndCBnZXQgaGVyZS4gV2UgZG9u J3QgdXNlIGFueSBJT1ZBIHRyaWNrIGFzIERQREsgKGl0IHJlc2VydmUgSU9WQQo+ID4gZm9yIHNo YWRvdyB2cSkgZGlkLiBTbyB3ZSB3b24ndCBzdWZmZXIgZnJvbSB0aGUgaXNzdWVzIG9mIERQREsu Cj4gPgo+ID4gVGhhbmtzCj4KPiBNYXliZSBJIG1pc3VuZHJzdGFuZCBob3cgdGhpcyBhbGwgd29y a3MuCj4gSSByZWZlciB0byBhbGwgdGhlIGlvdmFfdHJlZV9hbGxvY19tYXAgdGhpbmdzLgoKSXQn cyBhIHNpbXBsZSBJT1ZBIGFsbG9jYXRlciBhY3R1YWxseS4gQW55dGhpbmcgd3Jvbmcgd2l0aCB0 aGF0PwoKSSdtIGZpbmUgd2l0aCBtYWtpbmcgaXQgZm9yIHRoZSBmdXR1cmUgcmVsZWFzZS4KClRo YW5rcwoKPgo+ID4gPgo+ID4gPiA+Cj4gPiA+ID4gPgo+ID4gPiA+ID4gVE9ETyBvbiBmdXR1cmUg c2VyaWVzOgo+ID4gPiA+ID4gKiBFdmVudCwgaW5kaXJlY3QsIHBhY2tlZCwgYW5kIG90aGVycyBm ZWF0dXJlcyBvZiB2aXJ0aW8uCj4gPiA+ID4gPiAqIFRvIHN1cHBvcnQgZGlmZmVyZW50IHNldCBv ZiBmZWF0dXJlcyBiZXR3ZWVuIHRoZSBkZXZpY2U8LT5TVlEgYW5kIHRoZQo+ID4gPiA+ID4gICAg U1ZRPC0+Z3Vlc3QgY29tbXVuaWNhdGlvbi4KPiA+ID4gPiA+ICogU3VwcG9ydCBvZiBkZXZpY2Ug aG9zdCBub3RpZmllciBtZW1vcnkgcmVnaW9ucy4KPiA+ID4gPiA+ICogVG8gc2VwcGFyYXRlIGJ1 ZmZlcnMgZm9yd2FyZGluZyBpbiBpdHMgb3duIEFJTyBjb250ZXh0LCBzbyB3ZSBjYW4KPiA+ID4g PiA+ICAgIHRocm93IG1vcmUgdGhyZWFkcyB0byB0aGF0IHRhc2sgYW5kIHdlIGRvbid0IG5lZWQg dG8gc3RvcCB0aGUgbWFpbgo+ID4gPiA+ID4gICAgZXZlbnQgbG9vcC4KPiA+ID4gPiA+ICogU3Vw cG9ydCBtdWx0aXF1ZXVlIHZpcnRpby1uZXQgdmRwYS4KPiA+ID4gPiA+ICogUHJvcGVyIGRvY3Vt ZW50YXRpb24uCj4gPiA+ID4gPgo+ID4gPiA+ID4gQ2hhbmdlcyBmcm9tIHY0Ogo+ID4gPiA+ID4g KiBJdGVyYXRlIGlvdmEtPmh2YSB0cmVlIGluc3RlYWQgb24gbWFpbnRhaW4gb3duIHRyZWUgc28g d2Ugc3VwcG9ydCBIVkEKPiA+ID4gPiA+ICAgIG92ZXJsYXBzLgo+ID4gPiA+ID4gKiBGaXg6IEVy cm5vIGNvbXBsZXRpb24gYXQgZmFpbHVyZS4KPiA+ID4gPiA+ICogUmVuYW1lIHgtc3ZxIHRvIHN2 cSwgc28gY2hhbmdlcyB0byBzdGFibGUgZG9lcyBub3QgYWZmZWN0IGNtZGxpbmUgcGFyYW1ldGVy Lgo+ID4gPiA+ID4KPiA+ID4gPiA+IENoYW5nZXMgZnJvbSB2MzoKPiA+ID4gPiA+ICogQWRkIEB1 bnN0YWJsZSBmZWF0dXJlIHRvIE5ldGRldlZob3N0VkRQQU9wdGlvbnMueC1zdnEuCj4gPiA+ID4g PiAqIEZpeCB1bmNvbXBsZXRlIG1hcHBpbmcgKGJ5IDEgYnl0ZSkgb2YgbWVtb3J5IHJlZ2lvbnMg aWYgc3ZxIGlzIGVuYWJsZWQuCj4gPiA+ID4gPiB2MyBsaW5rOgo+ID4gPiA+ID4gaHR0cHM6Ly9s b3JlLmtlcm5lbC5vcmcvcWVtdS1kZXZlbC8yMDIyMDMwMjIwMzAxMi4zNDc2ODM1LTEtZXBlcmV6 bWFAcmVkaGF0LmNvbS8KPiA+ID4gPiA+Cj4gPiA+ID4gPiBDaGFuZ2VzIGZyb20gdjI6Cj4gPiA+ ID4gPiAqIExlc3MgYXNzZXJ0aW9ucyBhbmQgbW9yZSBlcnJvciBoYW5kbGluZyBpbiBpb3ZhIHRy ZWUgY29kZS4KPiA+ID4gPiA+ICogQmV0dGVyIGRvY3VtZW50YXRpb24sIGJvdGggZml4aW5nIGVy cm9ycyBhbmQgbWFraW5nIEBwYXJhbTogZm9ybWF0Cj4gPiA+ID4gPiAqIEhvbW9nZW5laXplIFNW USBhdmFpbF9pZHhfc2hhZG93IGFuZCBzaGFkb3dfdXNlZF9pZHggdG8gbWFrZSBzaGFkb3cgYQo+ ID4gPiA+ID4gICAgcHJlZml4IGF0IGJvdGggdGltZXMuCj4gPiA+ID4gPiAqIEZpeDogRm8gbm90 IHVzZSBWaXJ0UXVldWVFbGVtZW50LT5sZW4gZmllbGQsIHRyYWNrIHNlcGFyYXRlZGx5Lgo+ID4g PiA+ID4gKiBTcGxpdCB2aG9zdF9zdnFfe2VuYWJsZSxkaXNhYmxlfV9ub3RpZmljYXRpb24sIHNv IHRoZSBjb2RlIGxvb2tzIG1vcmUKPiA+ID4gPiA+ICAgIGxpa2UgdGhlIGtlcm5lbCBkcml2ZXIg Y29kZS4KPiA+ID4gPiA+ICogU21hbGwgaW1wcm92ZW1lbnRzLgo+ID4gPiA+ID4gdjIgbGluazoK PiA+ID4gPiA+IGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2FsbC9DQUphcXlXZlhIRTBDNTRSXy1P aXdKempDMGdQcGtFM2VYMEw4QmVlWlhHbTFFUllQdEFAbWFpbC5nbWFpbC5jb20vCj4gPiA+ID4g Pgo+ID4gPiA+ID4gQ2hhbmdlcyBmcm9tIHYxOgo+ID4gPiA+ID4gKiBGZWF0dXJlIHNldCBhdCBk ZXZpY2UtPlNWUSBpcyBub3cgdGhlIHNhbWUgYXMgU1ZRLT5ndWVzdC4KPiA+ID4gPiA+ICogU2l6 ZSBvZiBTVlEgaXMgbm90IG1heCBhdmFpbGFibGUgZGV2aWNlIHNpemUgYW55bW9yZSwgYnV0IGd1 ZXN0J3MKPiA+ID4gPiA+ICAgIG5lZ290aWF0ZWQuCj4gPiA+ID4gPiAqIEFkZCBWSE9TVF9GSUxF X1VOQklORCBraWNrIGFuZCBjYWxsIGZkIHRyZWF0bWVudC4KPiA+ID4gPiA+ICogTWFrZSBTVlEg YSBwdWJsaWMgc3RydWN0Cj4gPiA+ID4gPiAqIENvbWUgYmFjayB0byBwcmV2aW91cyBhcHByb2Fj aCB0byBpb3ZhLXRyZWUKPiA+ID4gPiA+ICogU29tZSBhc3NlcnRpb25zIGFyZSBub3cgZmFpbCBw YXRocy4gU29tZSBlcnJvcnMgYXJlIG5vdyBsb2dfZ3Vlc3QuCj4gPiA+ID4gPiAqIE9ubHkgbWFz ayBfRl9MT0cgZmVhdHVyZSBhdCB2ZHBhX3NldF9mZWF0dXJlcyBzdnEgZW5hYmxlIHBhdGguCj4g PiA+ID4gPiAqIFJlZmFjdG9yIHNvbWUgZXJyb3JzIGFuZCBtZXNzYWdlcy4gQWRkIG1pc3Npbmcg ZXJyb3IgdW53aW5kaW5ncy4KPiA+ID4gPiA+ICogQWRkIG1lbW9yeSBiYXJyaWVyIGF0IF9GX05P X05PVElGWSBzZXQuCj4gPiA+ID4gPiAqIFN0b3AgY2hlY2tpbmcgZm9yIGZlYXR1cmVzIGZsYWdz IG91dCBvZiB0cmFuc3BvcnQgcmFuZ2UuCj4gPiA+ID4gPiB2MSBsaW5rOgo+ID4gPiA+ID4gaHR0 cHM6Ly9sb3JlLmtlcm5lbC5vcmcvdmlydHVhbGl6YXRpb24vN2Q4NmM3MTUtNmQ3MS04YTI3LTkx ZjUtOGQ0N2I3MWUzMjAxQHJlZGhhdC5jb20vCj4gPiA+ID4gPgo+ID4gPiA+ID4gQ2hhbmdlcyBm cm9tIHY0IFJGQzoKPiA+ID4gPiA+ICogU3VwcG9ydCBvZiBhbGxvY2F0aW5nIC8gZnJlZWluZyBp b3ZhIHJhbmdlcyBpbiBJT1ZBIHRyZWUuIEV4dGVuZGluZwo+ID4gPiA+ID4gICAgYWxyZWFkeSBw cmVzZW50IGlvdmEtdHJlZSBmb3IgdGhhdC4KPiA+ID4gPiA+ICogUHJvcGVyIHZhbGlkYXRpb24g b2YgZ3Vlc3QgZmVhdHVyZXMuIE5vdyBTVlEgY2FuIG5lZ290aWF0ZSBhCj4gPiA+ID4gPiAgICBk aWZmZXJlbnQgc2V0IG9mIGZlYXR1cmVzIHdpdGggdGhlIGRldmljZSB3aGVuIGVuYWJsZWQuCj4g PiA+ID4gPiAqIFN1cHBvcnQgb2YgaG9zdCBub3RpZmllcnMgbWVtb3J5IHJlZ2lvbnMKPiA+ID4g PiA+ICogSGFuZGxpbmcgb2YgU1ZRIGZ1bGwgcXVldWUgaW4gY2FzZSBndWVzdCdzIGRlc2NyaXB0 b3JzIHNwYW4gdG8KPiA+ID4gPiA+ICAgIGRpZmZlcmVudCBtZW1vcnkgcmVnaW9ucyAocWVtdSdz IFZBIGNodW5rcykuCj4gPiA+ID4gPiAqIEZsdXNoIHBlbmRpbmcgdXNlZCBidWZmZXJzIGF0IGVu ZCBvZiBTVlEgb3BlcmF0aW9uLgo+ID4gPiA+ID4gKiBRTVAgY29tbWFuZCBub3cgbG9va3MgYnkg TmV0Q2xpZW50U3RhdGUgbmFtZS4gT3RoZXIgZGV2aWNlcyB3aWxsIG5lZWQKPiA+ID4gPiA+ICAg IHRvIGltcGxlbWVudCBpdCdzIHdheSB0byBlbmFibGUgdmRwYS4KPiA+ID4gPiA+ICogUmVuYW1l IFFNUCBjb21tYW5kIHRvIHNldCwgc28gaXQgbG9va3MgbW9yZSBsaWtlIGEgd2F5IG9mIHdvcmtp bmcKPiA+ID4gPiA+ICogQmV0dGVyIHVzZSBvZiBxZW11IGVycm9yIHN5c3RlbQo+ID4gPiA+ID4g KiBNYWtlIGEgZmV3IGFzc2VydGlvbnMgcHJvcGVyIGVycm9yLWhhbmRsaW5nIHBhdGhzLgo+ID4g PiA+ID4gKiBBZGQgbW9yZSBkb2N1bWVudGF0aW9uCj4gPiA+ID4gPiAqIExlc3MgY291cGxpbmcg b2YgdmlydGlvIC8gdmhvc3QsIHRoYXQgY291bGQgY2F1c2UgZnJpY3Rpb24gb24gY2hhbmdlcwo+ ID4gPiA+ID4gKiBBZGRyZXNzZWQgbWFueSBvdGhlciBzbWFsbCBjb21tZW50cyBhbmQgc21hbGwg Zml4ZXMuCj4gPiA+ID4gPgo+ID4gPiA+ID4gQ2hhbmdlcyBmcm9tIHYzIFJGQzoKPiA+ID4gPiA+ ICAgICogTW92ZSBldmVyeXRoaW5nIHRvIHZob3N0LXZkcGEgYmFja2VuZC4gQSBiaWcgY2hhbmdl LCB0aGlzIGFsbG93ZWQKPiA+ID4gPiA+ICAgICAgc29tZSBjbGVhbnVwIGJ1dCBtb3JlIGNvZGUg aGFzIGJlZW4gYWRkZWQgaW4gb3RoZXIgcGxhY2VzLgo+ID4gPiA+ID4gICAgKiBNb3JlIHVzZSBv ZiBnbGliIHV0aWxpdGllcywgZXNwZWNpYWxseSB0byBtYW5hZ2UgbWVtb3J5Lgo+ID4gPiA+ID4g djMgbGluazoKPiA+ID4gPiA+IGh0dHBzOi8vbGlzdHMubm9uZ251Lm9yZy9hcmNoaXZlL2h0bWwv cWVtdS1kZXZlbC8yMDIxLTA1L21zZzA2MDMyLmh0bWwKPiA+ID4gPiA+Cj4gPiA+ID4gPiBDaGFu Z2VzIGZyb20gdjIgUkZDOgo+ID4gPiA+ID4gICAgKiBBZGRpbmcgdmhvc3QtdmRwYSBkZXZpY2Vz IHN1cHBvcnQKPiA+ID4gPiA+ICAgICogRml4ZWQgc29tZSBtZW1vcnkgbGVha3MgcG9pbnRlZCBi eSBkaWZmZXJlbnQgY29tbWVudHMKPiA+ID4gPiA+IHYyIGxpbms6Cj4gPiA+ID4gPiBodHRwczov L2xpc3RzLm5vbmdudS5vcmcvYXJjaGl2ZS9odG1sL3FlbXUtZGV2ZWwvMjAyMS0wMy9tc2cwNTYw MC5odG1sCj4gPiA+ID4gPgo+ID4gPiA+ID4gQ2hhbmdlcyBmcm9tIHYxIFJGQzoKPiA+ID4gPiA+ ICAgICogVXNlIFFNUCBpbnN0ZWFkIG9mIG1pZ3JhdGlvbiB0byBzdGFydCBTVlEgbW9kZS4KPiA+ ID4gPiA+ICAgICogT25seSBhY2NlcHRpbmcgSU9NTVUgZGV2aWNlcywgY2xvc2VyIGJlaGF2aW9y IHdpdGggdGFyZ2V0IGRldmljZXMKPiA+ID4gPiA+ICAgICAgKHZEUEEpCj4gPiA+ID4gPiAgICAq IEZpeCBpbnZhbGlkIG1hc2tpbmcvdW5tYXNraW5nIG9mIHZob3N0IGNhbGwgZmQuCj4gPiA+ID4g PiAgICAqIFVzZSBvZiBwcm9wZXIgbWV0aG9kcyBmb3Igc3luY2hyb25pemF0aW9uLgo+ID4gPiA+ ID4gICAgKiBObyBuZWVkIHRvIG1vZGlmeSBWaXJ0SU8gZGV2aWNlIGNvZGUsIGFsbCBvZiB0aGUg Y2hhbmdlcyBhcmUKPiA+ID4gPiA+ICAgICAgY29udGFpbmVkIGluIHZob3N0IGNvZGUuCj4gPiA+ ID4gPiAgICAqIERlbGV0ZSBzdXBlcmZsdW91cyBjb2RlLgo+ID4gPiA+ID4gICAgKiBBbiBpbnRl cm1lZGlhdGUgUkZDIHdhcyBzZW50IHdpdGggb25seSB0aGUgbm90aWZpY2F0aW9ucyBmb3J3YXJk aW5nCj4gPiA+ID4gPiAgICAgIGNoYW5nZXMuIEl0IGNhbiBiZSBzZWVuIGluCj4gPiA+ID4gPiAg ICAgIGh0dHBzOi8vcGF0Y2hldy5vcmcvUUVNVS8yMDIxMDEyOTIwNTQxNS44NzYyOTAtMS1lcGVy ZXptYUByZWRoYXQuY29tLwo+ID4gPiA+ID4gdjEgbGluazoKPiA+ID4gPiA+IGh0dHBzOi8vbGlz dHMuZ251Lm9yZy9hcmNoaXZlL2h0bWwvcWVtdS1kZXZlbC8yMDIwLTExL21zZzA1MzcyLmh0bWwK PiA+ID4gPiA+Cj4gPiA+ID4gPiBFdWdlbmlvIFDDqXJleiAoMjApOgo+ID4gPiA+ID4gICAgICAg IHZpcnRpbzogQWRkIFZJUlRJT19GX1FVRVVFX1NUQVRFCj4gPiA+ID4gPiAgICAgICAgdmlydGlv LW5ldDogSG9ub3IgVklSVElPX0NPTkZJR19TX0RFVklDRV9TVE9QUEVECj4gPiA+ID4gPiAgICAg ICAgdmlydGlvOiBBZGQgdmlydGlvX3F1ZXVlX2lzX2hvc3Rfbm90aWZpZXJfZW5hYmxlZAo+ID4g PiA+ID4gICAgICAgIHZob3N0OiBNYWtlIHZob3N0X3ZpcnRxdWV1ZV97c3RhcnQsc3RvcH0gcHVi bGljCj4gPiA+ID4gPiAgICAgICAgdmhvc3Q6IEFkZCB4LXZob3N0LWVuYWJsZS1zaGFkb3ctdnEg cW1wCj4gPiA+ID4gPiAgICAgICAgdmhvc3Q6IEFkZCBWaG9zdFNoYWRvd1ZpcnRxdWV1ZQo+ID4g PiA+ID4gICAgICAgIHZkcGE6IFJlZ2lzdGVyIHZkcGEgZGV2aWNlcyBpbiBhIGxpc3QKPiA+ID4g PiA+ICAgICAgICB2aG9zdDogUm91dGUgZ3Vlc3QtPmhvc3Qgbm90aWZpY2F0aW9uIHRocm91Z2gg c2hhZG93IHZpcnRxdWV1ZQo+ID4gPiA+ID4gICAgICAgIEFkZCB2aG9zdF9zdnFfZ2V0X3N2cV9j YWxsX25vdGlmaWVyCj4gPiA+ID4gPiAgICAgICAgQWRkIHZob3N0X3N2cV9zZXRfZ3Vlc3RfY2Fs bF9ub3RpZmllcgo+ID4gPiA+ID4gICAgICAgIHZkcGE6IFNhdmUgY2FsbF9mZCBpbiB2aG9zdC12 ZHBhCj4gPiA+ID4gPiAgICAgICAgdmhvc3QtdmRwYTogVGFrZSBpbnRvIGFjY291bnQgU1ZRIGlu IHZob3N0X3ZkcGFfc2V0X3ZyaW5nX2NhbGwKPiA+ID4gPiA+ICAgICAgICB2aG9zdDogUm91dGUg aG9zdC0+Z3Vlc3Qgbm90aWZpY2F0aW9uIHRocm91Z2ggc2hhZG93IHZpcnRxdWV1ZQo+ID4gPiA+ ID4gICAgICAgIHZpcnRpbzogQWRkIHZob3N0X3NoYWRvd192cV9nZXRfdnJpbmdfYWRkcgo+ID4g PiA+ID4gICAgICAgIHZkcGE6IFNhdmUgaG9zdCBhbmQgZ3Vlc3QgZmVhdHVyZXMKPiA+ID4gPiA+ ICAgICAgICB2aG9zdDogQWRkIHZob3N0X3N2cV92YWxpZF9kZXZpY2VfZmVhdHVyZXMgdG8gc2hh ZG93IHZxCj4gPiA+ID4gPiAgICAgICAgdmhvc3Q6IFNoYWRvdyB2aXJ0cXVldWUgYnVmZmVycyBm b3J3YXJkaW5nCj4gPiA+ID4gPiAgICAgICAgdmhvc3Q6IEFkZCBWaG9zdElPVkFUcmVlCj4gPiA+ ID4gPiAgICAgICAgdmhvc3Q6IFVzZSBhIHRyZWUgdG8gc3RvcmUgbWVtb3J5IG1hcHBpbmdzCj4g PiA+ID4gPiAgICAgICAgdmRwYTogQWRkIGN1c3RvbSBJT1RMQiB0cmFuc2xhdGlvbnMgdG8gU1ZR Cj4gPiA+ID4gPgo+ID4gPiA+ID4gRXVnZW5pbyBQw6lyZXogKDE1KToKPiA+ID4gPiA+ICAgIHZo b3N0OiBBZGQgVmhvc3RTaGFkb3dWaXJ0cXVldWUKPiA+ID4gPiA+ICAgIHZob3N0OiBBZGQgU2hh ZG93IFZpcnRRdWV1ZSBraWNrIGZvcndhcmRpbmcgY2FwYWJpbGl0aWVzCj4gPiA+ID4gPiAgICB2 aG9zdDogQWRkIFNoYWRvdyBWaXJ0UXVldWUgY2FsbCBmb3J3YXJkaW5nIGNhcGFiaWxpdGllcwo+ ID4gPiA+ID4gICAgdmhvc3Q6IEFkZCB2aG9zdF9zdnFfdmFsaWRfZmVhdHVyZXMgdG8gc2hhZG93 IHZxCj4gPiA+ID4gPiAgICB2aXJ0aW86IEFkZCB2aG9zdF9zdnFfZ2V0X3ZyaW5nX2FkZHIKPiA+ ID4gPiA+ICAgIHZkcGE6IGFkYXB0IHZob3N0X29wcyBjYWxsYmFja3MgdG8gc3ZxCj4gPiA+ID4g PiAgICB2aG9zdDogU2hhZG93IHZpcnRxdWV1ZSBidWZmZXJzIGZvcndhcmRpbmcKPiA+ID4gPiA+ ICAgIHV0aWw6IEFkZCBpb3ZhX3RyZWVfYWxsb2NfbWFwCj4gPiA+ID4gPiAgICB1dGlsOiBhZGQg aW92YV90cmVlX2ZpbmRfaW92YQo+ID4gPiA+ID4gICAgdmhvc3Q6IEFkZCBWaG9zdElPVkFUcmVl Cj4gPiA+ID4gPiAgICB2ZHBhOiBBZGQgY3VzdG9tIElPVExCIHRyYW5zbGF0aW9ucyB0byBTVlEK PiA+ID4gPiA+ICAgIHZkcGE6IEFkYXB0IHZob3N0X3ZkcGFfZ2V0X3ZyaW5nX2Jhc2UgdG8gU1ZR Cj4gPiA+ID4gPiAgICB2ZHBhOiBOZXZlciBzZXQgbG9nX2Jhc2UgYWRkciBpZiBTVlEgaXMgZW5h YmxlZAo+ID4gPiA+ID4gICAgdmRwYTogRXhwb3NlIFZIT1NUX0ZfTE9HX0FMTCBvbiBTVlEKPiA+ ID4gPiA+ICAgIHZkcGE6IEFkZCB4LXN2cSB0byBOZXRkZXZWaG9zdFZEUEFPcHRpb25zCj4gPiA+ ID4gPgo+ID4gPiA+ID4gICBxYXBpL25ldC5qc29uICAgICAgICAgICAgICAgICAgICAgIHwgICA4 ICstCj4gPiA+ID4gPiAgIGh3L3ZpcnRpby92aG9zdC1pb3ZhLXRyZWUuaCAgICAgICAgfCAgMjcg KysKPiA+ID4gPiA+ICAgaHcvdmlydGlvL3Zob3N0LXNoYWRvdy12aXJ0cXVldWUuaCB8ICA4NyAr KysrCj4gPiA+ID4gPiAgIGluY2x1ZGUvaHcvdmlydGlvL3Zob3N0LXZkcGEuaCAgICAgfCAgIDgg Kwo+ID4gPiA+ID4gICBpbmNsdWRlL3FlbXUvaW92YS10cmVlLmggICAgICAgICAgIHwgIDM4ICst Cj4gPiA+ID4gPiAgIGh3L3ZpcnRpby92aG9zdC1pb3ZhLXRyZWUuYyAgICAgICAgfCAxMTAgKysr KysKPiA+ID4gPiA+ICAgaHcvdmlydGlvL3Zob3N0LXNoYWRvdy12aXJ0cXVldWUuYyB8IDYzNyAr KysrKysrKysrKysrKysrKysrKysrKysrKysrKwo+ID4gPiA+ID4gICBody92aXJ0aW8vdmhvc3Qt dmRwYS5jICAgICAgICAgICAgIHwgNTI1ICsrKysrKysrKysrKysrKysrKysrKysrLQo+ID4gPiA+ ID4gICBuZXQvdmhvc3QtdmRwYS5jICAgICAgICAgICAgICAgICAgIHwgIDQ4ICsrLQo+ID4gPiA+ ID4gICB1dGlsL2lvdmEtdHJlZS5jICAgICAgICAgICAgICAgICAgIHwgMTY5ICsrKysrKysrCj4g PiA+ID4gPiAgIGh3L3ZpcnRpby9tZXNvbi5idWlsZCAgICAgICAgICAgICAgfCAgIDIgKy0KPiA+ ID4gPiA+ICAgMTEgZmlsZXMgY2hhbmdlZCwgMTYzMyBpbnNlcnRpb25zKCspLCAyNiBkZWxldGlv bnMoLSkKPiA+ID4gPiA+ICAgY3JlYXRlIG1vZGUgMTAwNjQ0IGh3L3ZpcnRpby92aG9zdC1pb3Zh LXRyZWUuaAo+ID4gPiA+ID4gICBjcmVhdGUgbW9kZSAxMDA2NDQgaHcvdmlydGlvL3Zob3N0LXNo YWRvdy12aXJ0cXVldWUuaAo+ID4gPiA+ID4gICBjcmVhdGUgbW9kZSAxMDA2NDQgaHcvdmlydGlv L3Zob3N0LWlvdmEtdHJlZS5jCj4gPiA+ID4gPiAgIGNyZWF0ZSBtb2RlIDEwMDY0NCBody92aXJ0 aW8vdmhvc3Qtc2hhZG93LXZpcnRxdWV1ZS5jCj4gPiA+ID4gPgo+ID4gPgo+CgpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpWaXJ0dWFsaXphdGlvbiBtYWls aW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKaHR0cHM6 Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8vdmlydHVhbGl6YXRp b24= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5FF2C433F5 for ; Tue, 8 Mar 2022 07:52:17 +0000 (UTC) Received: from localhost ([::1]:39660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nRUdk-000467-U5 for qemu-devel@archiver.kernel.org; Tue, 08 Mar 2022 02:52:16 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47016) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nRUMg-0001X5-3x for qemu-devel@nongnu.org; Tue, 08 Mar 2022 02:34:39 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:29328) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nRUMd-0008CD-Lt for qemu-devel@nongnu.org; Tue, 08 Mar 2022 02:34:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1646724874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dfKE5Kp7qwEgqHd/rfsj2BcHX1IWW69+V0DA8sWS2/s=; b=aRp7P1SD/LUDrx/M70fCLakBbI4YRJE8eIPQqNeselaWYgc7NrNbxJ5WoEwd1Rk1PFiyz0 JW+SazDC7xTl7S5gCs1pTpLFM5XqyVwAOFLkgo2Wz0OW5nV0DqnnCSrFekXuV4vFspMN95 jOdv7gWeQvYZ/YLokEuWw73L6ElR8N0= Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-488-T65b1JtTMcS2h3HEXbokvw-1; Tue, 08 Mar 2022 02:34:31 -0500 X-MC-Unique: T65b1JtTMcS2h3HEXbokvw-1 Received: by mail-lj1-f200.google.com with SMTP id 20-20020a2e0914000000b0024635d136ddso7468728ljj.22 for ; Mon, 07 Mar 2022 23:34:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=dfKE5Kp7qwEgqHd/rfsj2BcHX1IWW69+V0DA8sWS2/s=; b=45VBUjrk8j21qXdye1pktKdDjEPYGfP+qSnWUiq12oBOEj2s85tvu4IWmFGFKvtXVu 8D9/MVnrr2jNJASu0M3HqQ48Gqu6Q94TkbfsrRFt9aTD7KT1ujKZugQlj5WZNded8Tx9 LEGm+t5yqdeDTN54TWR2GXZwT1eiHStIuK+kVNCG6fpyskNRSjgIPy+XHoj/8SoxxjL8 AGk9fuEydENLHLX7dpOfFHQWopdc5XSvAxJA63sUEGhND663wZ4cDAgPRW4DkHF+/uo5 yvpxOQg1y4q1cKZ5YS7UURKJiFFXnn8hNN2FZjRtwyp+Z/cfP4SLjNOx4lZMUt7ZvcxQ gt4A== X-Gm-Message-State: AOAM531D7N3t5TRqwkyn+4AoEORAzIUX0zogUEXnxEb5y+PN5vSMryWZ 4hBXGqjr4Sa+hesKFfsSEERW5dN4AjH0vI6JuuwiUoGueYXTUpaRldIXbIJbTfqt9OhX7bLuOiy Y+b/I/z5OIGx/zLG13IYLFI7N4log/Wo= X-Received: by 2002:a05:6512:3890:b0:448:21bd:28d with SMTP id n16-20020a056512389000b0044821bd028dmr9841733lft.376.1646724869601; Mon, 07 Mar 2022 23:34:29 -0800 (PST) X-Google-Smtp-Source: ABdhPJweEt8oq3dJWo9keGtJcmLQBKP6tmkQblYJ9WoLzYvOuHGtpuFBBadzif55GpUgMxivqoE/gWxiXR+IWX280Uo= X-Received: by 2002:a05:6512:3890:b0:448:21bd:28d with SMTP id n16-20020a056512389000b0044821bd028dmr9841700lft.376.1646724869192; Mon, 07 Mar 2022 23:34:29 -0800 (PST) MIME-Version: 1.0 References: <20220307153334.3854134-1-eperezma@redhat.com> <14d4fde4-6ea5-4805-b684-c33f6b448565@redhat.com> <20220308020348-mutt-send-email-mst@kernel.org> <20220308022300-mutt-send-email-mst@kernel.org> In-Reply-To: <20220308022300-mutt-send-email-mst@kernel.org> From: Jason Wang Date: Tue, 8 Mar 2022 15:34:17 +0800 Message-ID: Subject: Re: [PATCH v5 00/15] vDPA shadow virtqueue To: "Michael S. Tsirkin" Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=170.10.133.124; envelope-from=jasowang@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel , Peter Xu , virtualization , Eli Cohen , Eric Blake , Eduardo Habkost , Cindy Lu , "Fangyi \(Eric\)" , Markus Armbruster , yebiaoxiang@huawei.com, =?UTF-8?Q?Eugenio_P=C3=A9rez?= , Liuxiangdong , Stefano Garzarella , Laurent Vivier , Parav Pandit , Richard Henderson , Gautam Dawar , Xiao W Wang , Stefan Hajnoczi , Juan Quintela , Harpreet Singh Anand , Lingshan Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Tue, Mar 8, 2022 at 3:28 PM Michael S. Tsirkin wrote: > > On Tue, Mar 08, 2022 at 03:14:35PM +0800, Jason Wang wrote: > > On Tue, Mar 8, 2022 at 3:11 PM Michael S. Tsirkin wrot= e: > > > > > > On Tue, Mar 08, 2022 at 02:03:32PM +0800, Jason Wang wrote: > > > > > > > > =E5=9C=A8 2022/3/7 =E4=B8=8B=E5=8D=8811:33, Eugenio P=C3=A9rez =E5= =86=99=E9=81=93: > > > > > This series enable shadow virtqueue (SVQ) for vhost-vdpa devices.= This > > > > > is intended as a new method of tracking the memory the devices to= uch > > > > > during a migration process: Instead of relay on vhost device's di= rty > > > > > logging capability, SVQ intercepts the VQ dataplane forwarding th= e > > > > > descriptors between VM and device. This way qemu is the effective > > > > > writer of guests memory, like in qemu's virtio device operation. > > > > > > > > > > When SVQ is enabled qemu offers a new virtual address space to th= e > > > > > device to read and write into, and it maps new vrings and the gue= st > > > > > memory in it. SVQ also intercepts kicks and calls between the dev= ice > > > > > and the guest. Used buffers relay would cause dirty memory being > > > > > tracked. > > > > > > > > > > This effectively means that vDPA device passthrough is intercepte= d by > > > > > qemu. While SVQ should only be enabled at migration time, the swi= tching > > > > > from regular mode to SVQ mode is left for a future series. > > > > > > > > > > It is based on the ideas of DPDK SW assisted LM, in the series of > > > > > DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these d= oes > > > > > not map the shadow vq in guest's VA, but in qemu's. > > > > > > > > > > For qemu to use shadow virtqueues the guest virtio driver must no= t use > > > > > features like event_idx. > > > > > > > > > > SVQ needs to be enabled with cmdline: > > > > > > > > > > -netdev type=3Dvhost-vdpa,vhostdev=3Dvhost-vdpa-0,id=3Dvhost-vdpa= 0,svq=3Don > > > > > > A stable API for an incomplete feature is a problem imho. > > > > It should be "x-svq". > > > Well look at patch 15. It's a bug that needs to be fixed. > > > > > > > > > > > > > > > > > The first three patches enables notifications forwarding with > > > > > assistance of qemu. It's easy to enable only this if the relevant > > > > > cmdline part of the last patch is applied on top of these. > > > > > > > > > > Next four patches implement the actual buffer forwarding. However= , > > > > > address are not translated from HVA so they will need a host devi= ce with > > > > > an iommu allowing them to access all of the HVA range. > > > > > > > > > > The last part of the series uses properly the host iommu, so qemu > > > > > creates a new iova address space in the device's range and transl= ates > > > > > the buffers in it. Finally, it adds the cmdline parameter. > > > > > > > > > > Some simple performance tests with netperf were done. They used a= nested > > > > > guest with vp_vdpa, vhost-kernel at L0 host. Starting with no svq= and a > > > > > baseline average of ~9009.96Mbps: > > > > > Recv Send Send > > > > > Socket Socket Message Elapsed > > > > > Size Size Size Time Throughput > > > > > bytes bytes bytes secs. 10^6bits/sec > > > > > 131072 16384 16384 30.01 9061.03 > > > > > 131072 16384 16384 30.01 8962.94 > > > > > 131072 16384 16384 30.01 9005.92 > > > > > > > > > > To enable SVQ buffers forwarding reduce throughput to about > > > > > Recv Send Send > > > > > Socket Socket Message Elapsed > > > > > Size Size Size Time Throughput > > > > > bytes bytes bytes secs. 10^6bits/sec > > > > > 131072 16384 16384 30.01 7689.72 > > > > > 131072 16384 16384 30.00 7752.07 > > > > > 131072 16384 16384 30.01 7750.30 > > > > > > > > > > However, many performance improvements were left out of this seri= es for > > > > > simplicity, so difference should shrink in the future. > > > > > > > > > > Comments are welcome. > > > > > > > > > > > > Hi Michael: > > > > > > > > What do you think of this series? It looks good to me as a start. T= he > > > > feature could only be enabled as a dedicated parameter. If you're o= k, I'd > > > > try to make it for 7.0. > > > > > > > > Thanks > > > > > > Well that's cutting it awfully close, and it's not really useful > > > at the current stage, is it? > > > > This allows vDPA to be migrated when using "x-svq=3Don". > > But anyhow it's > > experimental. > > it's less experimental than incomplete. It seems pretty clearly not > the way it will work down the road, we don't want svq involved > at all times. Right, but SVQ could be used for other places e.g providing migration compatibility when the destination lacks some features. > > > > > > > The IOVA trick does not feel complete either. > > > > I don't get here. We don't use any IOVA trick as DPDK (it reserve IOVA > > for shadow vq) did. So we won't suffer from the issues of DPDK. > > > > Thanks > > Maybe I misundrstand how this all works. > I refer to all the iova_tree_alloc_map things. It's a simple IOVA allocater actually. Anything wrong with that? I'm fine with making it for the future release. Thanks > > > > > > > > > > > > > > > > > > TODO on future series: > > > > > * Event, indirect, packed, and others features of virtio. > > > > > * To support different set of features between the device<->SVQ a= nd the > > > > > SVQ<->guest communication. > > > > > * Support of device host notifier memory regions. > > > > > * To sepparate buffers forwarding in its own AIO context, so we c= an > > > > > throw more threads to that task and we don't need to stop the = main > > > > > event loop. > > > > > * Support multiqueue virtio-net vdpa. > > > > > * Proper documentation. > > > > > > > > > > Changes from v4: > > > > > * Iterate iova->hva tree instead on maintain own tree so we suppo= rt HVA > > > > > overlaps. > > > > > * Fix: Errno completion at failure. > > > > > * Rename x-svq to svq, so changes to stable does not affect cmdli= ne parameter. > > > > > > > > > > Changes from v3: > > > > > * Add @unstable feature to NetdevVhostVDPAOptions.x-svq. > > > > > * Fix uncomplete mapping (by 1 byte) of memory regions if svq is = enabled. > > > > > v3 link: > > > > > https://lore.kernel.org/qemu-devel/20220302203012.3476835-1-epere= zma@redhat.com/ > > > > > > > > > > Changes from v2: > > > > > * Less assertions and more error handling in iova tree code. > > > > > * Better documentation, both fixing errors and making @param: for= mat > > > > > * Homogeneize SVQ avail_idx_shadow and shadow_used_idx to make sh= adow a > > > > > prefix at both times. > > > > > * Fix: Fo not use VirtQueueElement->len field, track separatedly. > > > > > * Split vhost_svq_{enable,disable}_notification, so the code look= s more > > > > > like the kernel driver code. > > > > > * Small improvements. > > > > > v2 link: > > > > > https://lore.kernel.org/all/CAJaqyWfXHE0C54R_-OiwJzjC0gPpkE3eX0L8= BeeZXGm1ERYPtA@mail.gmail.com/ > > > > > > > > > > Changes from v1: > > > > > * Feature set at device->SVQ is now the same as SVQ->guest. > > > > > * Size of SVQ is not max available device size anymore, but guest= 's > > > > > negotiated. > > > > > * Add VHOST_FILE_UNBIND kick and call fd treatment. > > > > > * Make SVQ a public struct > > > > > * Come back to previous approach to iova-tree > > > > > * Some assertions are now fail paths. Some errors are now log_gue= st. > > > > > * Only mask _F_LOG feature at vdpa_set_features svq enable path. > > > > > * Refactor some errors and messages. Add missing error unwindings= . > > > > > * Add memory barrier at _F_NO_NOTIFY set. > > > > > * Stop checking for features flags out of transport range. > > > > > v1 link: > > > > > https://lore.kernel.org/virtualization/7d86c715-6d71-8a27-91f5-8d= 47b71e3201@redhat.com/ > > > > > > > > > > Changes from v4 RFC: > > > > > * Support of allocating / freeing iova ranges in IOVA tree. Exten= ding > > > > > already present iova-tree for that. > > > > > * Proper validation of guest features. Now SVQ can negotiate a > > > > > different set of features with the device when enabled. > > > > > * Support of host notifiers memory regions > > > > > * Handling of SVQ full queue in case guest's descriptors span to > > > > > different memory regions (qemu's VA chunks). > > > > > * Flush pending used buffers at end of SVQ operation. > > > > > * QMP command now looks by NetClientState name. Other devices wil= l need > > > > > to implement it's way to enable vdpa. > > > > > * Rename QMP command to set, so it looks more like a way of worki= ng > > > > > * Better use of qemu error system > > > > > * Make a few assertions proper error-handling paths. > > > > > * Add more documentation > > > > > * Less coupling of virtio / vhost, that could cause friction on c= hanges > > > > > * Addressed many other small comments and small fixes. > > > > > > > > > > Changes from v3 RFC: > > > > > * Move everything to vhost-vdpa backend. A big change, this al= lowed > > > > > some cleanup but more code has been added in other places. > > > > > * More use of glib utilities, especially to manage memory. > > > > > v3 link: > > > > > https://lists.nongnu.org/archive/html/qemu-devel/2021-05/msg06032= .html > > > > > > > > > > Changes from v2 RFC: > > > > > * Adding vhost-vdpa devices support > > > > > * Fixed some memory leaks pointed by different comments > > > > > v2 link: > > > > > https://lists.nongnu.org/archive/html/qemu-devel/2021-03/msg05600= .html > > > > > > > > > > Changes from v1 RFC: > > > > > * Use QMP instead of migration to start SVQ mode. > > > > > * Only accepting IOMMU devices, closer behavior with target de= vices > > > > > (vDPA) > > > > > * Fix invalid masking/unmasking of vhost call fd. > > > > > * Use of proper methods for synchronization. > > > > > * No need to modify VirtIO device code, all of the changes are > > > > > contained in vhost code. > > > > > * Delete superfluous code. > > > > > * An intermediate RFC was sent with only the notifications for= warding > > > > > changes. It can be seen in > > > > > https://patchew.org/QEMU/20210129205415.876290-1-eperezma@re= dhat.com/ > > > > > v1 link: > > > > > https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05372.ht= ml > > > > > > > > > > Eugenio P=C3=A9rez (20): > > > > > virtio: Add VIRTIO_F_QUEUE_STATE > > > > > virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED > > > > > virtio: Add virtio_queue_is_host_notifier_enabled > > > > > vhost: Make vhost_virtqueue_{start,stop} public > > > > > vhost: Add x-vhost-enable-shadow-vq qmp > > > > > vhost: Add VhostShadowVirtqueue > > > > > vdpa: Register vdpa devices in a list > > > > > vhost: Route guest->host notification through shadow virtq= ueue > > > > > Add vhost_svq_get_svq_call_notifier > > > > > Add vhost_svq_set_guest_call_notifier > > > > > vdpa: Save call_fd in vhost-vdpa > > > > > vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_= call > > > > > vhost: Route host->guest notification through shadow virtq= ueue > > > > > virtio: Add vhost_shadow_vq_get_vring_addr > > > > > vdpa: Save host and guest features > > > > > vhost: Add vhost_svq_valid_device_features to shadow vq > > > > > vhost: Shadow virtqueue buffers forwarding > > > > > vhost: Add VhostIOVATree > > > > > vhost: Use a tree to store memory mappings > > > > > vdpa: Add custom IOTLB translations to SVQ > > > > > > > > > > Eugenio P=C3=A9rez (15): > > > > > vhost: Add VhostShadowVirtqueue > > > > > vhost: Add Shadow VirtQueue kick forwarding capabilities > > > > > vhost: Add Shadow VirtQueue call forwarding capabilities > > > > > vhost: Add vhost_svq_valid_features to shadow vq > > > > > virtio: Add vhost_svq_get_vring_addr > > > > > vdpa: adapt vhost_ops callbacks to svq > > > > > vhost: Shadow virtqueue buffers forwarding > > > > > util: Add iova_tree_alloc_map > > > > > util: add iova_tree_find_iova > > > > > vhost: Add VhostIOVATree > > > > > vdpa: Add custom IOTLB translations to SVQ > > > > > vdpa: Adapt vhost_vdpa_get_vring_base to SVQ > > > > > vdpa: Never set log_base addr if SVQ is enabled > > > > > vdpa: Expose VHOST_F_LOG_ALL on SVQ > > > > > vdpa: Add x-svq to NetdevVhostVDPAOptions > > > > > > > > > > qapi/net.json | 8 +- > > > > > hw/virtio/vhost-iova-tree.h | 27 ++ > > > > > hw/virtio/vhost-shadow-virtqueue.h | 87 ++++ > > > > > include/hw/virtio/vhost-vdpa.h | 8 + > > > > > include/qemu/iova-tree.h | 38 +- > > > > > hw/virtio/vhost-iova-tree.c | 110 +++++ > > > > > hw/virtio/vhost-shadow-virtqueue.c | 637 ++++++++++++++++++++++= +++++++ > > > > > hw/virtio/vhost-vdpa.c | 525 ++++++++++++++++++++++= +- > > > > > net/vhost-vdpa.c | 48 ++- > > > > > util/iova-tree.c | 169 ++++++++ > > > > > hw/virtio/meson.build | 2 +- > > > > > 11 files changed, 1633 insertions(+), 26 deletions(-) > > > > > create mode 100644 hw/virtio/vhost-iova-tree.h > > > > > create mode 100644 hw/virtio/vhost-shadow-virtqueue.h > > > > > create mode 100644 hw/virtio/vhost-iova-tree.c > > > > > create mode 100644 hw/virtio/vhost-shadow-virtqueue.c > > > > > > > > >