From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1DFAC6377D for ; Thu, 22 Jul 2021 09:05:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CAADD60C41 for ; Thu, 22 Jul 2021 09:05:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230491AbhGVIYj (ORCPT ); Thu, 22 Jul 2021 04:24:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:27203 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230419AbhGVIYg (ORCPT ); Thu, 22 Jul 2021 04:24:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626944710; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=613H4wAGuEnZg548mMLIz3DNJ0VmDXrroK5bj+IIg9Q=; b=Z6amGgMpJWyGu9uXdYbWbvnlys+Fd9XZTxqSadFj9GGEfWOGvzOW8FE/EWmOMHsiWDFCZb KPQGYNvvz5GLLPoH+lvPKEfNtOa5scWyWfuIYnt7bjxmvRgi5iHveAB/K5ZxvMr8rNP1tF FbrOt/R7r4nOtXWPXWhsfThZJz+QJw0= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-77-PJp9Dz2MOqy34PjioK4HeA-1; Thu, 22 Jul 2021 05:05:08 -0400 X-MC-Unique: PJp9Dz2MOqy34PjioK4HeA-1 Received: by mail-pl1-f199.google.com with SMTP id p9-20020a170902e749b029012b7acb0e05so2406439plf.23 for ; Thu, 22 Jul 2021 02:05:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=613H4wAGuEnZg548mMLIz3DNJ0VmDXrroK5bj+IIg9Q=; b=AjoXa+ITVxeR+pYAj/+Ls0sOtQQRMwV0UVbnB52GO5Y3l+EsRxiQfGWeGE2hnM9OFq ltegPMaT4ZSMX3XD2GSQTrs6nl1Umkin6KfI5TL6e0GGDtaN75Lcvz4lEK71fm1rRNps FOk6j3wnDbnOBYA3Y/fdTIKZI6R+TjKoVuS9zUH0yedlAAPq39vreNxk8zgKX86NGuII 0CNeX7pqw33DCTgzPzZeIcq3lbEd4DJAqXEEup0P2LSwZiOz8c/PKGyN1Gxu+HrhXhMz DHaMucBtw98GHFK5RkzUxibzBuS9IPwRtSmrpxI9zz80xyLC6BrRdcLweDJemxaNTgK+ a3vg== X-Gm-Message-State: AOAM532fAfW96+iEhFukKjk2mQ7hB3VzlWjBxGqjXo4YrX6thNabzIt+ XaZsVJrIbqgWPs29b9++qrHj5kEpOR4tWMLa+yMGfCpmcXKt8IjF4LqRaaGdW9xDXM883P0MvHW sMXkpC9GW52vF0ReUIeUSPt0= X-Received: by 2002:a63:1658:: with SMTP id 24mr12874153pgw.307.1626944707783; Thu, 22 Jul 2021 02:05:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy0+x8zNhuoIB4bhgvnGT/Lauq7S1tovNJXVfSEHqXJ8wvxOVkqIxFpT/VtggSDtTXoyT+32Q== X-Received: by 2002:a63:1658:: with SMTP id 24mr12874128pgw.307.1626944707458; Thu, 22 Jul 2021 02:05:07 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id t3sm4781930pfd.153.2021.07.22.02.05.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 22 Jul 2021 02:05:07 -0700 (PDT) Subject: Re: [RFC 0/3] cpuidle: add poll_source API and virtio vq polling To: Stefan Hajnoczi Cc: linux-kernel@vger.kernel.org, Daniel Lezcano , Stefano Garzarella , Ming Lei , "Michael S . Tsirkin" , Marcelo Tosatti , Jens Axboe , linux-block@vger.kernel.org, "Rafael J. Wysocki" , virtualization@lists.linux-foundation.org, linux-pm@vger.kernel.org, Christoph Hellwig References: <20210713161906.457857-1-stefanha@redhat.com> <1008dee4-fce1-2462-1520-f5432bc89a07@redhat.com> From: Jason Wang Message-ID: <869a993d-a1b0-1c39-d081-4cdd2b71041f@redhat.com> Date: Thu, 22 Jul 2021 17:04:57 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org 在 2021/7/21 下午5:41, Stefan Hajnoczi 写道: > On Wed, Jul 21, 2021 at 11:29:55AM +0800, Jason Wang wrote: >> 在 2021/7/14 上午12:19, Stefan Hajnoczi 写道: >>> These patches are not polished yet but I would like request feedback on this >>> approach and share performance results with you. >>> >>> Idle CPUs tentatively enter a busy wait loop before halting when the cpuidle >>> haltpoll driver is enabled inside a virtual machine. This reduces wakeup >>> latency for events that occur soon after the vCPU becomes idle. >>> >>> This patch series extends the cpuidle busy wait loop with the new poll_source >>> API so drivers can participate in polling. Such polling-aware drivers disable >>> their device's irq during the busy wait loop to avoid the cost of interrupts. >>> This reduces latency further than regular cpuidle haltpoll, which still relies >>> on irqs. >>> >>> Virtio drivers are modified to use the poll_source API so all virtio device >>> types get this feature. The following virtio-blk fio benchmark results show the >>> improvement: >>> >>> IOPS (numjobs=4, iodepth=1, 4 virtqueues) >>> before poll_source io_poll >>> 4k randread 167102 186049 (+11%) 186654 (+11%) >>> 4k randwrite 162204 181214 (+11%) 181850 (+12%) >>> 4k randrw 159520 177071 (+11%) 177928 (+11%) >>> >>> The comparison against io_poll shows that cpuidle poll_source achieves >>> equivalent performance to the block layer's io_poll feature (which I >>> implemented in a separate patch series [1]). >>> >>> The advantage of poll_source is that applications do not need to explicitly set >>> the RWF_HIPRI I/O request flag. The poll_source approach is attractive because >>> few applications actually use RWF_HIPRI and it takes advantage of CPU cycles we >>> would have spent in cpuidle haltpoll anyway. >>> >>> The current series does not improve virtio-net. I haven't investigated deeply, >>> but it is possible that NAPI and poll_source do not combine. See the final >>> patch for a starting point on making the two work together. >>> >>> I have not tried this on bare metal but it might help there too. The cost of >>> disabling a device's irq must be less than the savings from avoiding irq >>> handling for this optimization to make sense. >>> >>> [1] https://lore.kernel.org/linux-block/20210520141305.355961-1-stefanha@redhat.com/ >> >> Hi Stefan: >> >> Some questions: >> >> 1) What's the advantages of introducing polling at virtio level instead of >> doing it at each subsystems? Polling in virtio level may only work well if >> all (or most) of the devices are virtio > I'm not sure I understand the question. cpuidle haltpoll benefits all > devices today, except it incurs interrupt latency. The poll_source API > eliminates the interrupt latency for drivers that can disable device > interrupts cheaply. > > This patch adds poll_source to core virtio code so that all virtio > drivers get this feature for free. No driver-specific changes are > needed. > > If you mean networking, block layer, etc by "subsystems" then there's > nothing those subsystems can do to help. Whether poll_source can be used > depends on the specific driver, not the subsystem. If you consider > drivers/virtio/ a subsystem, then that's exactly what the patch series > is doing. I meant, if we choose to use idle poll, we have some several choices: 1) bus level (e.g the virtio) 2) subsystem level (e.g the networking and block) I'm not sure which one is better. > >> 2) What's the advantages of using cpuidle instead of using a thread (and >> leverage the scheduler)? > In order to combine with the existing cpuidle infrastructure. No new > polling loop is introduced and no additional CPU cycles are spent on > polling. > > If cpuidle itself is converted to threads then poll_source would > automatically operate in a thread too, but this patch series doesn't > change how the core cpuidle code works. So networking subsystem can use NAPI busy polling in the process context which means it can be leveraged by the scheduler. I'm not sure it's a good idea to poll drivers for a specific bus in the general cpu idle layer. Another questions, are those numbers measured by APICV capable machine? Virtio-net turns on the tx interrupts since 2 years ago. And we don't see too much difference when measured with a APICV host. > >> 3) Any reason it's virtio_pci specific not a general virtio one? > Good idea, it is possible to move the virtio_pci changes into virtio.c. > > Other transports can't use this feature yet though. Only virtio_pci > supports vq irq affinity. But the code can be generic and if other > transports ever support vq irq affinity they'll get it for free. Yes. Thanks > >> (Btw, do we need to cc scheduler guys?) > I'm not sure. This patch series doesn't change how cpuidle interacts > with the scheduler. The cpuidle maintainers can pull in more people, if > necessary. > > Stefan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6CFDC6379A for ; Thu, 22 Jul 2021 09:05:20 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5AB2261370 for ; Thu, 22 Jul 2021 09:05:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5AB2261370 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=virtualization-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1C4F16066E; Thu, 22 Jul 2021 09:05:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9T7py704qRdz; Thu, 22 Jul 2021 09:05:19 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 92D8B60657; Thu, 22 Jul 2021 09:05:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5FA2EC001A; Thu, 22 Jul 2021 09:05:18 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id E5E21C000E for ; Thu, 22 Jul 2021 09:05:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id E0E4D60657 for ; Thu, 22 Jul 2021 09:05:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LQi3enXaRLXQ for ; Thu, 22 Jul 2021 09:05:16 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id AB9A86062D for ; Thu, 22 Jul 2021 09:05:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626944715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=613H4wAGuEnZg548mMLIz3DNJ0VmDXrroK5bj+IIg9Q=; b=aS+hBmnykPE37TnJjdcuJw/YnzXpE2V4OuFQvdpoPiy8LnpIRt5TrT0b4IHWzjUVe3Rvnr H5V5gRkI4ypHj4ALT0hJpdh5wHPabZWUw0u1htmQU1vwyjOnU4BR1bbVhARSUTHAnLUn/x b9YlbHq0czXXRnxvI3WZQHgxrLjlUUU= Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-150-I13iGMCEPQiJmM1g-6BN6g-1; Thu, 22 Jul 2021 05:05:08 -0400 X-MC-Unique: I13iGMCEPQiJmM1g-6BN6g-1 Received: by mail-pl1-f200.google.com with SMTP id a6-20020a1709027d86b02901019f88b046so2407154plm.21 for ; Thu, 22 Jul 2021 02:05:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=613H4wAGuEnZg548mMLIz3DNJ0VmDXrroK5bj+IIg9Q=; b=SujJbR3n9U2YjLzOU2hVexP/tfSC2GV90jA91l1rkeyhoP6qbKhSgNziCTPw4RPl5b xu1o9hXfzW8Rsdvg7/chEbQqzfXfkF88QlmDhHr+DkGfK+173Asu2mXsQ6k41ABU3f5k Tkf/KkBnzlv4YhenyEhAMLmo7GR1vMieiVp4gq6qezSXD6wjgGNMo3m+0xpv5YkJfU0f P+hEev/3RRVaxvoZ1AkDy+vPC5V63E3Rxl9Kk4Hd2CB9o8SA3+OQqTIz98LTnsZK0UE1 BHoz7HWvznmzsKuljcihrKuo/g+jSbkQgJMrB8niaNvyIuD/MqVkARigo/2TZ/egeFmO 1xVA== X-Gm-Message-State: AOAM532ewcDzZzC94HJVG4vlA/dfUfQlOfKByRtr5fIrxj2Vaxuddgxs 6xYK/DVQ6Qr/hgueBRzjVASKukNfgyK8ZgYd5KQIONzNdWMKLtk0NuU8yDy3ceZVT6qzcxC1e3E EnRu3tkqf2tjuc6nJ3L8DsaGUQCNVtIo2YWiBl2nrCQ== X-Received: by 2002:a63:1658:: with SMTP id 24mr12874154pgw.307.1626944707783; Thu, 22 Jul 2021 02:05:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy0+x8zNhuoIB4bhgvnGT/Lauq7S1tovNJXVfSEHqXJ8wvxOVkqIxFpT/VtggSDtTXoyT+32Q== X-Received: by 2002:a63:1658:: with SMTP id 24mr12874128pgw.307.1626944707458; Thu, 22 Jul 2021 02:05:07 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id t3sm4781930pfd.153.2021.07.22.02.05.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 22 Jul 2021 02:05:07 -0700 (PDT) Subject: Re: [RFC 0/3] cpuidle: add poll_source API and virtio vq polling To: Stefan Hajnoczi References: <20210713161906.457857-1-stefanha@redhat.com> <1008dee4-fce1-2462-1520-f5432bc89a07@redhat.com> From: Jason Wang Message-ID: <869a993d-a1b0-1c39-d081-4cdd2b71041f@redhat.com> Date: Thu, 22 Jul 2021 17:04:57 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: Jens Axboe , Christoph Hellwig , Daniel Lezcano , "Michael S . Tsirkin" , linux-pm@vger.kernel.org, "Rafael J. Wysocki" , linux-kernel@vger.kernel.org, Ming Lei , linux-block@vger.kernel.org, virtualization@lists.linux-foundation.org X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" CuWcqCAyMDIxLzcvMjEg5LiL5Y2INTo0MSwgU3RlZmFuIEhham5vY3ppIOWGmemBkzoKPiBPbiBX ZWQsIEp1bCAyMSwgMjAyMSBhdCAxMToyOTo1NUFNICswODAwLCBKYXNvbiBXYW5nIHdyb3RlOgo+ PiDlnKggMjAyMS83LzE0IOS4iuWNiDEyOjE5LCBTdGVmYW4gSGFqbm9jemkg5YaZ6YGTOgo+Pj4g VGhlc2UgcGF0Y2hlcyBhcmUgbm90IHBvbGlzaGVkIHlldCBidXQgSSB3b3VsZCBsaWtlIHJlcXVl c3QgZmVlZGJhY2sgb24gdGhpcwo+Pj4gYXBwcm9hY2ggYW5kIHNoYXJlIHBlcmZvcm1hbmNlIHJl c3VsdHMgd2l0aCB5b3UuCj4+Pgo+Pj4gSWRsZSBDUFVzIHRlbnRhdGl2ZWx5IGVudGVyIGEgYnVz eSB3YWl0IGxvb3AgYmVmb3JlIGhhbHRpbmcgd2hlbiB0aGUgY3B1aWRsZQo+Pj4gaGFsdHBvbGwg ZHJpdmVyIGlzIGVuYWJsZWQgaW5zaWRlIGEgdmlydHVhbCBtYWNoaW5lLiBUaGlzIHJlZHVjZXMg d2FrZXVwCj4+PiBsYXRlbmN5IGZvciBldmVudHMgdGhhdCBvY2N1ciBzb29uIGFmdGVyIHRoZSB2 Q1BVIGJlY29tZXMgaWRsZS4KPj4+Cj4+PiBUaGlzIHBhdGNoIHNlcmllcyBleHRlbmRzIHRoZSBj cHVpZGxlIGJ1c3kgd2FpdCBsb29wIHdpdGggdGhlIG5ldyBwb2xsX3NvdXJjZQo+Pj4gQVBJIHNv IGRyaXZlcnMgY2FuIHBhcnRpY2lwYXRlIGluIHBvbGxpbmcuIFN1Y2ggcG9sbGluZy1hd2FyZSBk cml2ZXJzIGRpc2FibGUKPj4+IHRoZWlyIGRldmljZSdzIGlycSBkdXJpbmcgdGhlIGJ1c3kgd2Fp dCBsb29wIHRvIGF2b2lkIHRoZSBjb3N0IG9mIGludGVycnVwdHMuCj4+PiBUaGlzIHJlZHVjZXMg bGF0ZW5jeSBmdXJ0aGVyIHRoYW4gcmVndWxhciBjcHVpZGxlIGhhbHRwb2xsLCB3aGljaCBzdGls bCByZWxpZXMKPj4+IG9uIGlycXMuCj4+Pgo+Pj4gVmlydGlvIGRyaXZlcnMgYXJlIG1vZGlmaWVk IHRvIHVzZSB0aGUgcG9sbF9zb3VyY2UgQVBJIHNvIGFsbCB2aXJ0aW8gZGV2aWNlCj4+PiB0eXBl cyBnZXQgdGhpcyBmZWF0dXJlLiBUaGUgZm9sbG93aW5nIHZpcnRpby1ibGsgZmlvIGJlbmNobWFy ayByZXN1bHRzIHNob3cgdGhlCj4+PiBpbXByb3ZlbWVudDoKPj4+Cj4+PiAgICAgICAgICAgICAg ICBJT1BTIChudW1qb2JzPTQsIGlvZGVwdGg9MSwgNCB2aXJ0cXVldWVzKQo+Pj4gICAgICAgICAg ICAgICAgICBiZWZvcmUgICBwb2xsX3NvdXJjZSAgICAgIGlvX3BvbGwKPj4+IDRrIHJhbmRyZWFk ICAgIDE2NzEwMiAgMTg2MDQ5ICgrMTElKSAgMTg2NjU0ICgrMTElKQo+Pj4gNGsgcmFuZHdyaXRl ICAgMTYyMjA0ICAxODEyMTQgKCsxMSUpICAxODE4NTAgKCsxMiUpCj4+PiA0ayByYW5kcncgICAg ICAxNTk1MjAgIDE3NzA3MSAoKzExJSkgIDE3NzkyOCAoKzExJSkKPj4+Cj4+PiBUaGUgY29tcGFy aXNvbiBhZ2FpbnN0IGlvX3BvbGwgc2hvd3MgdGhhdCBjcHVpZGxlIHBvbGxfc291cmNlIGFjaGll dmVzCj4+PiBlcXVpdmFsZW50IHBlcmZvcm1hbmNlIHRvIHRoZSBibG9jayBsYXllcidzIGlvX3Bv bGwgZmVhdHVyZSAod2hpY2ggSQo+Pj4gaW1wbGVtZW50ZWQgaW4gYSBzZXBhcmF0ZSBwYXRjaCBz ZXJpZXMgWzFdKS4KPj4+Cj4+PiBUaGUgYWR2YW50YWdlIG9mIHBvbGxfc291cmNlIGlzIHRoYXQg YXBwbGljYXRpb25zIGRvIG5vdCBuZWVkIHRvIGV4cGxpY2l0bHkgc2V0Cj4+PiB0aGUgUldGX0hJ UFJJIEkvTyByZXF1ZXN0IGZsYWcuIFRoZSBwb2xsX3NvdXJjZSBhcHByb2FjaCBpcyBhdHRyYWN0 aXZlIGJlY2F1c2UKPj4+IGZldyBhcHBsaWNhdGlvbnMgYWN0dWFsbHkgdXNlIFJXRl9ISVBSSSBh bmQgaXQgdGFrZXMgYWR2YW50YWdlIG9mIENQVSBjeWNsZXMgd2UKPj4+IHdvdWxkIGhhdmUgc3Bl bnQgaW4gY3B1aWRsZSBoYWx0cG9sbCBhbnl3YXkuCj4+Pgo+Pj4gVGhlIGN1cnJlbnQgc2VyaWVz IGRvZXMgbm90IGltcHJvdmUgdmlydGlvLW5ldC4gSSBoYXZlbid0IGludmVzdGlnYXRlZCBkZWVw bHksCj4+PiBidXQgaXQgaXMgcG9zc2libGUgdGhhdCBOQVBJIGFuZCBwb2xsX3NvdXJjZSBkbyBu b3QgY29tYmluZS4gU2VlIHRoZSBmaW5hbAo+Pj4gcGF0Y2ggZm9yIGEgc3RhcnRpbmcgcG9pbnQg b24gbWFraW5nIHRoZSB0d28gd29yayB0b2dldGhlci4KPj4+Cj4+PiBJIGhhdmUgbm90IHRyaWVk IHRoaXMgb24gYmFyZSBtZXRhbCBidXQgaXQgbWlnaHQgaGVscCB0aGVyZSB0b28uIFRoZSBjb3N0 IG9mCj4+PiBkaXNhYmxpbmcgYSBkZXZpY2UncyBpcnEgbXVzdCBiZSBsZXNzIHRoYW4gdGhlIHNh dmluZ3MgZnJvbSBhdm9pZGluZyBpcnEKPj4+IGhhbmRsaW5nIGZvciB0aGlzIG9wdGltaXphdGlv biB0byBtYWtlIHNlbnNlLgo+Pj4KPj4+IFsxXSBodHRwczovL2xvcmUua2VybmVsLm9yZy9saW51 eC1ibG9jay8yMDIxMDUyMDE0MTMwNS4zNTU5NjEtMS1zdGVmYW5oYUByZWRoYXQuY29tLwo+Pgo+ PiBIaSBTdGVmYW46Cj4+Cj4+IFNvbWUgcXVlc3Rpb25zOgo+Pgo+PiAxKSBXaGF0J3MgdGhlIGFk dmFudGFnZXMgb2YgaW50cm9kdWNpbmcgcG9sbGluZyBhdCB2aXJ0aW8gbGV2ZWwgaW5zdGVhZCBv Zgo+PiBkb2luZyBpdCBhdCBlYWNoIHN1YnN5c3RlbXM/IFBvbGxpbmcgaW4gdmlydGlvIGxldmVs IG1heSBvbmx5IHdvcmsgd2VsbCBpZgo+PiBhbGwgKG9yIG1vc3QpIG9mIHRoZSBkZXZpY2VzIGFy ZSB2aXJ0aW8KPiBJJ20gbm90IHN1cmUgSSB1bmRlcnN0YW5kIHRoZSBxdWVzdGlvbi4gY3B1aWRs ZSBoYWx0cG9sbCBiZW5lZml0cyBhbGwKPiBkZXZpY2VzIHRvZGF5LCBleGNlcHQgaXQgaW5jdXJz IGludGVycnVwdCBsYXRlbmN5LiBUaGUgcG9sbF9zb3VyY2UgQVBJCj4gZWxpbWluYXRlcyB0aGUg aW50ZXJydXB0IGxhdGVuY3kgZm9yIGRyaXZlcnMgdGhhdCBjYW4gZGlzYWJsZSBkZXZpY2UKPiBp bnRlcnJ1cHRzIGNoZWFwbHkuCj4KPiBUaGlzIHBhdGNoIGFkZHMgcG9sbF9zb3VyY2UgdG8gY29y ZSB2aXJ0aW8gY29kZSBzbyB0aGF0IGFsbCB2aXJ0aW8KPiBkcml2ZXJzIGdldCB0aGlzIGZlYXR1 cmUgZm9yIGZyZWUuIE5vIGRyaXZlci1zcGVjaWZpYyBjaGFuZ2VzIGFyZQo+IG5lZWRlZC4KPgo+ IElmIHlvdSBtZWFuIG5ldHdvcmtpbmcsIGJsb2NrIGxheWVyLCBldGMgYnkgInN1YnN5c3RlbXMi IHRoZW4gdGhlcmUncwo+IG5vdGhpbmcgdGhvc2Ugc3Vic3lzdGVtcyBjYW4gZG8gdG8gaGVscC4g V2hldGhlciBwb2xsX3NvdXJjZSBjYW4gYmUgdXNlZAo+IGRlcGVuZHMgb24gdGhlIHNwZWNpZmlj IGRyaXZlciwgbm90IHRoZSBzdWJzeXN0ZW0uIElmIHlvdSBjb25zaWRlcgo+IGRyaXZlcnMvdmly dGlvLyBhIHN1YnN5c3RlbSwgdGhlbiB0aGF0J3MgZXhhY3RseSB3aGF0IHRoZSBwYXRjaCBzZXJp ZXMKPiBpcyBkb2luZy4KCgpJIG1lYW50LCBpZiB3ZSBjaG9vc2UgdG8gdXNlIGlkbGUgcG9sbCwg d2UgaGF2ZSBzb21lIHNldmVyYWwgY2hvaWNlczoKCjEpIGJ1cyBsZXZlbCAoZS5nIHRoZSB2aXJ0 aW8pCjIpIHN1YnN5c3RlbSBsZXZlbCAoZS5nIHRoZSBuZXR3b3JraW5nIGFuZCBibG9jaykKCkkn bSBub3Qgc3VyZSB3aGljaCBvbmUgaXMgYmV0dGVyLgoKCj4KPj4gMikgV2hhdCdzIHRoZSBhZHZh bnRhZ2VzIG9mIHVzaW5nIGNwdWlkbGUgaW5zdGVhZCBvZiB1c2luZyBhIHRocmVhZCAoYW5kCj4+ IGxldmVyYWdlIHRoZSBzY2hlZHVsZXIpPwo+IEluIG9yZGVyIHRvIGNvbWJpbmUgd2l0aCB0aGUg ZXhpc3RpbmcgY3B1aWRsZSBpbmZyYXN0cnVjdHVyZS4gTm8gbmV3Cj4gcG9sbGluZyBsb29wIGlz IGludHJvZHVjZWQgYW5kIG5vIGFkZGl0aW9uYWwgQ1BVIGN5Y2xlcyBhcmUgc3BlbnQgb24KPiBw b2xsaW5nLgo+Cj4gSWYgY3B1aWRsZSBpdHNlbGYgaXMgY29udmVydGVkIHRvIHRocmVhZHMgdGhl biBwb2xsX3NvdXJjZSB3b3VsZAo+IGF1dG9tYXRpY2FsbHkgb3BlcmF0ZSBpbiBhIHRocmVhZCB0 b28sIGJ1dCB0aGlzIHBhdGNoIHNlcmllcyBkb2Vzbid0Cj4gY2hhbmdlIGhvdyB0aGUgY29yZSBj cHVpZGxlIGNvZGUgd29ya3MuCgoKU28gbmV0d29ya2luZyBzdWJzeXN0ZW0gY2FuIHVzZSBOQVBJ IGJ1c3kgcG9sbGluZyBpbiB0aGUgcHJvY2VzcyBjb250ZXh0IAp3aGljaCBtZWFucyBpdCBjYW4g YmUgbGV2ZXJhZ2VkIGJ5IHRoZSBzY2hlZHVsZXIuCgpJJ20gbm90IHN1cmUgaXQncyBhIGdvb2Qg aWRlYSB0byBwb2xsIGRyaXZlcnMgZm9yIGEgc3BlY2lmaWMgYnVzIGluIHRoZSAKZ2VuZXJhbCBj cHUgaWRsZSBsYXllci4KCkFub3RoZXIgcXVlc3Rpb25zLCBhcmUgdGhvc2UgbnVtYmVycyBtZWFz dXJlZCBieSBBUElDViBjYXBhYmxlIG1hY2hpbmU/CgpWaXJ0aW8tbmV0IHR1cm5zIG9uIHRoZSB0 eCBpbnRlcnJ1cHRzIHNpbmNlIDIgeWVhcnMgYWdvLiBBbmQgd2UgZG9uJ3QgCnNlZSB0b28gbXVj aCBkaWZmZXJlbmNlIHdoZW4gbWVhc3VyZWQgd2l0aCBhIEFQSUNWIGhvc3QuCgoKPgo+PiAzKSBB bnkgcmVhc29uIGl0J3MgdmlydGlvX3BjaSBzcGVjaWZpYyBub3QgYSBnZW5lcmFsIHZpcnRpbyBv bmU/Cj4gR29vZCBpZGVhLCBpdCBpcyBwb3NzaWJsZSB0byBtb3ZlIHRoZSB2aXJ0aW9fcGNpIGNo YW5nZXMgaW50byB2aXJ0aW8uYy4KPgo+IE90aGVyIHRyYW5zcG9ydHMgY2FuJ3QgdXNlIHRoaXMg ZmVhdHVyZSB5ZXQgdGhvdWdoLiBPbmx5IHZpcnRpb19wY2kKPiBzdXBwb3J0cyB2cSBpcnEgYWZm aW5pdHkuIEJ1dCB0aGUgY29kZSBjYW4gYmUgZ2VuZXJpYyBhbmQgaWYgb3RoZXIKPiB0cmFuc3Bv cnRzIGV2ZXIgc3VwcG9ydCB2cSBpcnEgYWZmaW5pdHkgdGhleSdsbCBnZXQgaXQgZm9yIGZyZWUu CgoKWWVzLgoKVGhhbmtzCgoKPgo+PiAoQnR3LCBkbyB3ZSBuZWVkIHRvIGNjIHNjaGVkdWxlciBn dXlzPykKPiBJJ20gbm90IHN1cmUuIFRoaXMgcGF0Y2ggc2VyaWVzIGRvZXNuJ3QgY2hhbmdlIGhv dyBjcHVpZGxlIGludGVyYWN0cwo+IHdpdGggdGhlIHNjaGVkdWxlci4gVGhlIGNwdWlkbGUgbWFp bnRhaW5lcnMgY2FuIHB1bGwgaW4gbW9yZSBwZW9wbGUsIGlmCj4gbmVjZXNzYXJ5Lgo+Cj4gU3Rl ZmFuCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpWaXJ0 dWFsaXphdGlvbiBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRh dGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGlu Zm8vdmlydHVhbGl6YXRpb24=