From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3A02ECDFAA for ; Tue, 17 Jul 2018 00:46:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8A5CA20841 for ; Tue, 17 Jul 2018 00:46:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8A5CA20841 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730667AbeGQBPY (ORCPT ); Mon, 16 Jul 2018 21:15:24 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:33224 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729827AbeGQBPX (ORCPT ); Mon, 16 Jul 2018 21:15:23 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 20D90401EF04; Tue, 17 Jul 2018 00:45:30 +0000 (UTC) Received: from [10.72.12.85] (ovpn-12-85.pek2.redhat.com [10.72.12.85]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CF2F1111E40C; Tue, 17 Jul 2018 00:45:19 +0000 (UTC) Subject: Re: [PATCH net-next V2 0/8] Packed virtqueue support for vhost To: "Michael S. Tsirkin" Cc: kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, wexu@redhat.com, jfreimann@redhat.com, tiwei.bie@intel.com, maxime.coquelin@redhat.com References: <1531711691-6769-1-git-send-email-jasowang@redhat.com> <20180716113720-mutt-send-email-mst@kernel.org> <33f4643f-f226-0389-1f4f-607c289db94e@redhat.com> <20180716154102-mutt-send-email-mst@kernel.org> From: Jason Wang Message-ID: <5ba5c927-a0b4-f399-7a88-b90763765142@redhat.com> Date: Tue, 17 Jul 2018 08:45:16 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20180716154102-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 17 Jul 2018 00:45:30 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 17 Jul 2018 00:45:30 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'jasowang@redhat.com' RCPT:'' Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2018年07月16日 20:49, Michael S. Tsirkin wrote: > On Mon, Jul 16, 2018 at 05:46:33PM +0800, Jason Wang wrote: >> >> On 2018年07月16日 16:39, Michael S. Tsirkin wrote: >>> On Mon, Jul 16, 2018 at 11:28:03AM +0800, Jason Wang wrote: >>>> Hi all: >>>> >>>> This series implements packed virtqueues. The code were tested with >>>> Tiwei's guest driver series at https://patchwork.ozlabs.org/cover/942297/ >>>> >>>> >>>> Pktgen test for both RX and TX does not show obvious difference with >>>> split virtqueues. The main bottleneck is the guest Linux driver, since >>>> it can not stress vhost for a 100% CPU utilization. A full TCP >>>> benchmark is ongoing. Will test virtio-net pmd as well when it was >>>> ready. >>> Well the question then is why we should bother merging this >>> if this doesn't give a performance gain. >> We meet bottlenecks at other places. I can only test Linux driver which has >> lots of overheads e.g interrupts. And perf show only a small fraction of >> time were spent on e.g virtqueue manipulation. I hope virtio-net pmd can >> give us different result but we don't have one ready for testing now. (Jen's >> V4 have bugs thus can not work with this series). > Can't linux busy poll? For vhost busy polling, there's no difference since guest can not give vhost enough stress. For guest busy polling, it does not work for the packets generated by pktgen. > And how about testing loopback with XDP? No difference, I even shortcut both the tun_get_user() on host and netif_receive_skb() in guest. >>> Do you see >>> a gain in CPU utilization maybe? >> Unfortunately not. >> >>> If not - let's wait for that TCP benchmark result? >> We can, but you know TCP_STREAM result is sometime misleading. >> >> A brunch of other patches of mine were rebased on this and then blocked on >> this series. Consider we don't meet regression, maybe we can merge this >> first and try optimizations or fixups on top? >> >> Thanks > I'm not sure I understand this approach. Packed ring is just an optimization. > What value is there in merging it if it does not help speed? If you want to support migration from dpdk or vDPA backend. And we still have the chance to see the performance with virito-net pmd in the future. If this does not make sense for you, I will leave this series until we can get results from virtio-net pmd (or find a way that packed virtqueue outperform). And I will start to post other optimizations on vhost. Thanks > >>>> Notes: >>>> - This version depends on Tiwei's series at https://patchwork.ozlabs.org/cover/942297/ >>>> >>>> This version were tested with: >>>> >>>> - Zerocopy (Out of Order) support >>>> - vIOMMU support >>>> - mergeable buffer on/off >>>> - busy polling on/off >>>> - vsock (nc-vsock) >>>> >>>> Changes from V1: >>>> - drop uapi patch and use Tiwei's >>>> - split the enablement of packed virtqueue into a separate patch >>>> >>>> Changes from RFC V5: >>>> >>>> - save unnecessary barriers during vhost_add_used_packed_n() >>>> - more compact math for event idx >>>> - fix failure of SET_VRING_BASE when avail_wrap_counter is true >>>> - fix not copy avail_wrap_counter during GET_VRING_BASE >>>> - introduce SET_VRING_USED_BASE/GET_VRING_USED_BASE for syncing last_used_idx >>>> - rename used_wrap_counter to last_used_wrap_counter >>>> - rebase to net-next >>>> >>>> Changes from RFC V4: >>>> >>>> - fix signalled_used index recording >>>> - track avail index correctly >>>> - various minor fixes >>>> >>>> Changes from RFC V3: >>>> >>>> - Fix math on event idx checking >>>> - Sync last avail wrap counter through GET/SET_VRING_BASE >>>> - remove desc_event prefix in the driver/device structure >>>> >>>> Changes from RFC V2: >>>> >>>> - do not use & in checking desc_event_flags >>>> - off should be most significant bit >>>> - remove the workaround of mergeable buffer for dpdk prototype >>>> - id should be in the last descriptor in the chain >>>> - keep _F_WRITE for write descriptor when adding used >>>> - device flags updating should use ADDR_USED type >>>> - return error on unexpected unavail descriptor in a chain >>>> - return false in vhost_ve_avail_empty is descriptor is available >>>> - track last seen avail_wrap_counter >>>> - correctly examine available descriptor in get_indirect_packed() >>>> - vhost_idx_diff should return u16 instead of bool >>>> >>>> Changes from RFC V1: >>>> >>>> - Refactor vhost used elem code to avoid open coding on used elem >>>> - Event suppression support (compile test only). >>>> - Indirect descriptor support (compile test only). >>>> - Zerocopy support. >>>> - vIOMMU support. >>>> - SCSI/VSOCK support (compile test only). >>>> - Fix several bugs >>>> >>>> Jason Wang (8): >>>> vhost: move get_rx_bufs to vhost.c >>>> vhost: hide used ring layout from device >>>> vhost: do not use vring_used_elem >>>> vhost_net: do not explicitly manipulate vhost_used_elem >>>> vhost: vhost_put_user() can accept metadata type >>>> vhost: packed ring support >>>> vhost: event suppression for packed ring >>>> vhost: enable packed virtqueues >>>> >>>> drivers/vhost/net.c | 143 ++----- >>>> drivers/vhost/scsi.c | 62 +-- >>>> drivers/vhost/vhost.c | 994 ++++++++++++++++++++++++++++++++++++++++----- >>>> drivers/vhost/vhost.h | 55 ++- >>>> drivers/vhost/vsock.c | 42 +- >>>> include/uapi/linux/vhost.h | 7 + >>>> 6 files changed, 1035 insertions(+), 268 deletions(-) >>>> >>>> -- >>>> 2.7.4 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Wang Subject: Re: [PATCH net-next V2 0/8] Packed virtqueue support for vhost Date: Tue, 17 Jul 2018 08:45:16 +0800 Message-ID: <5ba5c927-a0b4-f399-7a88-b90763765142@redhat.com> References: <1531711691-6769-1-git-send-email-jasowang@redhat.com> <20180716113720-mutt-send-email-mst@kernel.org> <33f4643f-f226-0389-1f4f-607c289db94e@redhat.com> <20180716154102-mutt-send-email-mst@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Cc: kvm@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, maxime.coquelin@redhat.com, wexu@redhat.com To: "Michael S. Tsirkin" Return-path: In-Reply-To: <20180716154102-mutt-send-email-mst@kernel.org> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org List-Id: netdev.vger.kernel.org CgpPbiAyMDE45bm0MDfmnIgxNuaXpSAyMDo0OSwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+ IE9uIE1vbiwgSnVsIDE2LCAyMDE4IGF0IDA1OjQ2OjMzUE0gKzA4MDAsIEphc29uIFdhbmcgd3Jv dGU6Cj4+Cj4+IE9uIDIwMTjlubQwN+aciDE25pelIDE2OjM5LCBNaWNoYWVsIFMuIFRzaXJraW4g d3JvdGU6Cj4+PiBPbiBNb24sIEp1bCAxNiwgMjAxOCBhdCAxMToyODowM0FNICswODAwLCBKYXNv biBXYW5nIHdyb3RlOgo+Pj4+IEhpIGFsbDoKPj4+Pgo+Pj4+IFRoaXMgc2VyaWVzIGltcGxlbWVu dHMgcGFja2VkIHZpcnRxdWV1ZXMuIFRoZSBjb2RlIHdlcmUgdGVzdGVkIHdpdGgKPj4+PiBUaXdl aSdzIGd1ZXN0IGRyaXZlciBzZXJpZXMgYXQgaHR0cHM6Ly9wYXRjaHdvcmsub3psYWJzLm9yZy9j b3Zlci85NDIyOTcvCj4+Pj4KPj4+Pgo+Pj4+IFBrdGdlbiB0ZXN0IGZvciBib3RoIFJYIGFuZCBU WCBkb2VzIG5vdCBzaG93IG9idmlvdXMgZGlmZmVyZW5jZSB3aXRoCj4+Pj4gc3BsaXQgdmlydHF1 ZXVlcy4gVGhlIG1haW4gYm90dGxlbmVjayBpcyB0aGUgZ3Vlc3QgTGludXggZHJpdmVyLCBzaW5j ZQo+Pj4+IGl0IGNhbiBub3Qgc3RyZXNzIHZob3N0IGZvciBhIDEwMCUgQ1BVIHV0aWxpemF0aW9u LiBBIGZ1bGwgVENQCj4+Pj4gYmVuY2htYXJrIGlzIG9uZ29pbmcuIFdpbGwgdGVzdCB2aXJ0aW8t bmV0IHBtZCBhcyB3ZWxsIHdoZW4gaXQgd2FzCj4+Pj4gcmVhZHkuCj4+PiBXZWxsIHRoZSBxdWVz dGlvbiB0aGVuIGlzIHdoeSB3ZSBzaG91bGQgYm90aGVyIG1lcmdpbmcgdGhpcwo+Pj4gaWYgdGhp cyBkb2Vzbid0IGdpdmUgYSBwZXJmb3JtYW5jZSBnYWluLgo+PiBXZSBtZWV0IGJvdHRsZW5lY2tz IGF0IG90aGVyIHBsYWNlcy4gSSBjYW4gb25seSB0ZXN0IExpbnV4IGRyaXZlciB3aGljaCBoYXMK Pj4gbG90cyBvZiBvdmVyaGVhZHMgZS5nIGludGVycnVwdHMuIEFuZCBwZXJmIHNob3cgb25seSBh IHNtYWxsIGZyYWN0aW9uIG9mCj4+IHRpbWUgd2VyZSBzcGVudCBvbiBlLmcgdmlydHF1ZXVlIG1h bmlwdWxhdGlvbi4gSSBob3BlIHZpcnRpby1uZXQgcG1kIGNhbgo+PiBnaXZlIHVzIGRpZmZlcmVu dCByZXN1bHQgYnV0IHdlIGRvbid0IGhhdmUgb25lIHJlYWR5IGZvciB0ZXN0aW5nIG5vdy4gKEpl bidzCj4+IFY0IGhhdmUgYnVncyB0aHVzIGNhbiBub3Qgd29yayB3aXRoIHRoaXMgc2VyaWVzKS4K PiBDYW4ndCBsaW51eCBidXN5IHBvbGw/CgpGb3Igdmhvc3QgYnVzeSBwb2xsaW5nLCB0aGVyZSdz IG5vIGRpZmZlcmVuY2Ugc2luY2UgZ3Vlc3QgY2FuIG5vdCBnaXZlIAp2aG9zdCBlbm91Z2ggc3Ry ZXNzLiBGb3IgZ3Vlc3QgYnVzeSBwb2xsaW5nLCBpdCBkb2VzIG5vdCB3b3JrIGZvciB0aGUgCnBh Y2tldHMgZ2VuZXJhdGVkIGJ5IHBrdGdlbi4KCj4gQW5kIGhvdyBhYm91dCB0ZXN0aW5nIGxvb3Bi YWNrIHdpdGggWERQPwoKTm8gZGlmZmVyZW5jZSwgSSBldmVuIHNob3J0Y3V0IGJvdGggdGhlIHR1 bl9nZXRfdXNlcigpIG9uIGhvc3QgYW5kIApuZXRpZl9yZWNlaXZlX3NrYigpIGluIGd1ZXN0LgoK Pj4+ICAgIERvIHlvdSBzZWUKPj4+IGEgZ2FpbiBpbiBDUFUgdXRpbGl6YXRpb24gbWF5YmU/Cj4+ IFVuZm9ydHVuYXRlbHkgbm90Lgo+Pgo+Pj4gSWYgbm90IC0gbGV0J3Mgd2FpdCBmb3IgdGhhdCBU Q1AgYmVuY2htYXJrIHJlc3VsdD8KPj4gV2UgY2FuLCBidXQgeW91IGtub3cgVENQX1NUUkVBTSBy ZXN1bHQgaXMgc29tZXRpbWUgbWlzbGVhZGluZy4KPj4KPj4gQSBicnVuY2ggb2Ygb3RoZXIgcGF0 Y2hlcyBvZiBtaW5lIHdlcmUgcmViYXNlZCBvbiB0aGlzIGFuZCB0aGVuIGJsb2NrZWQgb24KPj4g dGhpcyBzZXJpZXMuIENvbnNpZGVyIHdlIGRvbid0IG1lZXQgcmVncmVzc2lvbiwgbWF5YmUgd2Ug Y2FuIG1lcmdlIHRoaXMKPj4gZmlyc3QgYW5kIHRyeSBvcHRpbWl6YXRpb25zIG9yIGZpeHVwcyBv biB0b3A/Cj4+Cj4+IFRoYW5rcwo+IEknbSBub3Qgc3VyZSBJIHVuZGVyc3RhbmQgdGhpcyBhcHBy b2FjaC4gUGFja2VkIHJpbmcgaXMganVzdCBhbiBvcHRpbWl6YXRpb24uCj4gV2hhdCB2YWx1ZSBp cyB0aGVyZSBpbiBtZXJnaW5nIGl0IGlmIGl0IGRvZXMgbm90IGhlbHAgc3BlZWQ/CgpJZiB5b3Ug d2FudCB0byBzdXBwb3J0IG1pZ3JhdGlvbiBmcm9tIGRwZGsgb3IgdkRQQSBiYWNrZW5kLiBBbmQg d2Ugc3RpbGwgCmhhdmUgdGhlIGNoYW5jZSB0byBzZWUgdGhlIHBlcmZvcm1hbmNlIHdpdGggdmly aXRvLW5ldCBwbWQgaW4gdGhlIApmdXR1cmUuIElmIHRoaXMgZG9lcyBub3QgbWFrZSBzZW5zZSBm b3IgeW91LCBJIHdpbGwgbGVhdmUgdGhpcyBzZXJpZXMgCnVudGlsIHdlIGNhbiBnZXQgcmVzdWx0 cyBmcm9tIHZpcnRpby1uZXQgcG1kIChvciBmaW5kIGEgd2F5IHRoYXQgcGFja2VkIAp2aXJ0cXVl dWUgb3V0cGVyZm9ybSkuIEFuZCBJIHdpbGwgc3RhcnQgdG8gcG9zdCBvdGhlciBvcHRpbWl6YXRp b25zIG9uIAp2aG9zdC4KClRoYW5rcwoKPgo+Pj4+IE5vdGVzOgo+Pj4+IC0gVGhpcyB2ZXJzaW9u IGRlcGVuZHMgb24gVGl3ZWkncyBzZXJpZXMgYXQgaHR0cHM6Ly9wYXRjaHdvcmsub3psYWJzLm9y Zy9jb3Zlci85NDIyOTcvCj4+Pj4KPj4+PiBUaGlzIHZlcnNpb24gd2VyZSB0ZXN0ZWQgd2l0aDoK Pj4+Pgo+Pj4+IC0gWmVyb2NvcHkgKE91dCBvZiBPcmRlcikgc3VwcG9ydAo+Pj4+IC0gdklPTU1V IHN1cHBvcnQKPj4+PiAtIG1lcmdlYWJsZSBidWZmZXIgb24vb2ZmCj4+Pj4gLSBidXN5IHBvbGxp bmcgb24vb2ZmCj4+Pj4gLSB2c29jayAobmMtdnNvY2spCj4+Pj4KPj4+PiBDaGFuZ2VzIGZyb20g VjE6Cj4+Pj4gLSBkcm9wIHVhcGkgcGF0Y2ggYW5kIHVzZSBUaXdlaSdzCj4+Pj4gLSBzcGxpdCB0 aGUgZW5hYmxlbWVudCBvZiBwYWNrZWQgdmlydHF1ZXVlIGludG8gYSBzZXBhcmF0ZSBwYXRjaAo+ Pj4+Cj4+Pj4gQ2hhbmdlcyBmcm9tIFJGQyBWNToKPj4+Pgo+Pj4+IC0gc2F2ZSB1bm5lY2Vzc2Fy eSBiYXJyaWVycyBkdXJpbmcgdmhvc3RfYWRkX3VzZWRfcGFja2VkX24oKQo+Pj4+IC0gbW9yZSBj b21wYWN0IG1hdGggZm9yIGV2ZW50IGlkeAo+Pj4+IC0gZml4IGZhaWx1cmUgb2YgU0VUX1ZSSU5H X0JBU0Ugd2hlbiBhdmFpbF93cmFwX2NvdW50ZXIgaXMgdHJ1ZQo+Pj4+IC0gZml4IG5vdCBjb3B5 IGF2YWlsX3dyYXBfY291bnRlciBkdXJpbmcgR0VUX1ZSSU5HX0JBU0UKPj4+PiAtIGludHJvZHVj ZSBTRVRfVlJJTkdfVVNFRF9CQVNFL0dFVF9WUklOR19VU0VEX0JBU0UgZm9yIHN5bmNpbmcgbGFz dF91c2VkX2lkeAo+Pj4+IC0gcmVuYW1lIHVzZWRfd3JhcF9jb3VudGVyIHRvIGxhc3RfdXNlZF93 cmFwX2NvdW50ZXIKPj4+PiAtIHJlYmFzZSB0byBuZXQtbmV4dAo+Pj4+Cj4+Pj4gQ2hhbmdlcyBm cm9tIFJGQyBWNDoKPj4+Pgo+Pj4+IC0gZml4IHNpZ25hbGxlZF91c2VkIGluZGV4IHJlY29yZGlu Zwo+Pj4+IC0gdHJhY2sgYXZhaWwgaW5kZXggY29ycmVjdGx5Cj4+Pj4gLSB2YXJpb3VzIG1pbm9y IGZpeGVzCj4+Pj4KPj4+PiBDaGFuZ2VzIGZyb20gUkZDIFYzOgo+Pj4+Cj4+Pj4gLSBGaXggbWF0 aCBvbiBldmVudCBpZHggY2hlY2tpbmcKPj4+PiAtIFN5bmMgbGFzdCBhdmFpbCB3cmFwIGNvdW50 ZXIgdGhyb3VnaCBHRVQvU0VUX1ZSSU5HX0JBU0UKPj4+PiAtIHJlbW92ZSBkZXNjX2V2ZW50IHBy ZWZpeCBpbiB0aGUgZHJpdmVyL2RldmljZSBzdHJ1Y3R1cmUKPj4+Pgo+Pj4+IENoYW5nZXMgZnJv bSBSRkMgVjI6Cj4+Pj4KPj4+PiAtIGRvIG5vdCB1c2UgJiBpbiBjaGVja2luZyBkZXNjX2V2ZW50 X2ZsYWdzCj4+Pj4gLSBvZmYgc2hvdWxkIGJlIG1vc3Qgc2lnbmlmaWNhbnQgYml0Cj4+Pj4gLSBy ZW1vdmUgdGhlIHdvcmthcm91bmQgb2YgbWVyZ2VhYmxlIGJ1ZmZlciBmb3IgZHBkayBwcm90b3R5 cGUKPj4+PiAtIGlkIHNob3VsZCBiZSBpbiB0aGUgbGFzdCBkZXNjcmlwdG9yIGluIHRoZSBjaGFp bgo+Pj4+IC0ga2VlcCBfRl9XUklURSBmb3Igd3JpdGUgZGVzY3JpcHRvciB3aGVuIGFkZGluZyB1 c2VkCj4+Pj4gLSBkZXZpY2UgZmxhZ3MgdXBkYXRpbmcgc2hvdWxkIHVzZSBBRERSX1VTRUQgdHlw ZQo+Pj4+IC0gcmV0dXJuIGVycm9yIG9uIHVuZXhwZWN0ZWQgdW5hdmFpbCBkZXNjcmlwdG9yIGlu IGEgY2hhaW4KPj4+PiAtIHJldHVybiBmYWxzZSBpbiB2aG9zdF92ZV9hdmFpbF9lbXB0eSBpcyBk ZXNjcmlwdG9yIGlzIGF2YWlsYWJsZQo+Pj4+IC0gdHJhY2sgbGFzdCBzZWVuIGF2YWlsX3dyYXBf Y291bnRlcgo+Pj4+IC0gY29ycmVjdGx5IGV4YW1pbmUgYXZhaWxhYmxlIGRlc2NyaXB0b3IgaW4g Z2V0X2luZGlyZWN0X3BhY2tlZCgpCj4+Pj4gLSB2aG9zdF9pZHhfZGlmZiBzaG91bGQgcmV0dXJu IHUxNiBpbnN0ZWFkIG9mIGJvb2wKPj4+Pgo+Pj4+IENoYW5nZXMgZnJvbSBSRkMgVjE6Cj4+Pj4K Pj4+PiAtIFJlZmFjdG9yIHZob3N0IHVzZWQgZWxlbSBjb2RlIHRvIGF2b2lkIG9wZW4gY29kaW5n IG9uIHVzZWQgZWxlbQo+Pj4+IC0gRXZlbnQgc3VwcHJlc3Npb24gc3VwcG9ydCAoY29tcGlsZSB0 ZXN0IG9ubHkpLgo+Pj4+IC0gSW5kaXJlY3QgZGVzY3JpcHRvciBzdXBwb3J0IChjb21waWxlIHRl c3Qgb25seSkuCj4+Pj4gLSBaZXJvY29weSBzdXBwb3J0Lgo+Pj4+IC0gdklPTU1VIHN1cHBvcnQu Cj4+Pj4gLSBTQ1NJL1ZTT0NLIHN1cHBvcnQgKGNvbXBpbGUgdGVzdCBvbmx5KS4KPj4+PiAtIEZp eCBzZXZlcmFsIGJ1Z3MKPj4+Pgo+Pj4+IEphc29uIFdhbmcgKDgpOgo+Pj4+ICAgICB2aG9zdDog bW92ZSBnZXRfcnhfYnVmcyB0byB2aG9zdC5jCj4+Pj4gICAgIHZob3N0OiBoaWRlIHVzZWQgcmlu ZyBsYXlvdXQgZnJvbSBkZXZpY2UKPj4+PiAgICAgdmhvc3Q6IGRvIG5vdCB1c2UgdnJpbmdfdXNl ZF9lbGVtCj4+Pj4gICAgIHZob3N0X25ldDogZG8gbm90IGV4cGxpY2l0bHkgbWFuaXB1bGF0ZSB2 aG9zdF91c2VkX2VsZW0KPj4+PiAgICAgdmhvc3Q6IHZob3N0X3B1dF91c2VyKCkgY2FuIGFjY2Vw dCBtZXRhZGF0YSB0eXBlCj4+Pj4gICAgIHZob3N0OiBwYWNrZWQgcmluZyBzdXBwb3J0Cj4+Pj4g ICAgIHZob3N0OiBldmVudCBzdXBwcmVzc2lvbiBmb3IgcGFja2VkIHJpbmcKPj4+PiAgICAgdmhv c3Q6IGVuYWJsZSBwYWNrZWQgdmlydHF1ZXVlcwo+Pj4+Cj4+Pj4gICAgZHJpdmVycy92aG9zdC9u ZXQuYyAgICAgICAgfCAxNDMgKystLS0tLQo+Pj4+ICAgIGRyaXZlcnMvdmhvc3Qvc2NzaS5jICAg ICAgIHwgIDYyICstLQo+Pj4+ICAgIGRyaXZlcnMvdmhvc3Qvdmhvc3QuYyAgICAgIHwgOTk0ICsr KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKystLS0tLQo+Pj4+ICAgIGRyaXZl cnMvdmhvc3Qvdmhvc3QuaCAgICAgIHwgIDU1ICsrLQo+Pj4+ICAgIGRyaXZlcnMvdmhvc3QvdnNv Y2suYyAgICAgIHwgIDQyICstCj4+Pj4gICAgaW5jbHVkZS91YXBpL2xpbnV4L3Zob3N0LmggfCAg IDcgKwo+Pj4+ICAgIDYgZmlsZXMgY2hhbmdlZCwgMTAzNSBpbnNlcnRpb25zKCspLCAyNjggZGVs ZXRpb25zKC0pCj4+Pj4KPj4+PiAtLSAKPj4+PiAyLjcuNAoKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KVmlydHVhbGl6YXRpb24gbWFpbGluZyBsaXN0ClZp cnR1YWxpemF0aW9uQGxpc3RzLmxpbnV4LWZvdW5kYXRpb24ub3JnCmh0dHBzOi8vbGlzdHMubGlu dXhmb3VuZGF0aW9uLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3ZpcnR1YWxpemF0aW9u