From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F3E9EB64DA for ; Mon, 10 Jul 2023 09:25:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229924AbjGJJZp (ORCPT ); Mon, 10 Jul 2023 05:25:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229493AbjGJJZm (ORCPT ); Mon, 10 Jul 2023 05:25:42 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id B67448E for ; Mon, 10 Jul 2023 02:25:40 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 96F6B2B; Mon, 10 Jul 2023 02:26:22 -0700 (PDT) Received: from [10.57.77.63] (unknown [10.57.77.63]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5ACE53F740; Mon, 10 Jul 2023 02:25:38 -0700 (PDT) Message-ID: <501d1a6a-c9cf-94c3-b773-c648b944b30f@arm.com> Date: Mon, 10 Jul 2023 10:25:36 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance To: "Huang, Ying" Cc: David Hildenbrand , Matthew Wilcox , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Johannes Weiner , Alexander Zhu References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> <87y1jofoyi.fsf@yhuang6-desk2.ccr.corp.intel.com> <6c2f3127-9334-85ba-48f6-83a9c87abde0@arm.com> <874jmcf7kh.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Ryan Roberts In-Reply-To: <874jmcf7kh.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/07/2023 10:18, Huang, Ying wrote: > Ryan Roberts writes: > >> On 10/07/2023 04:03, Huang, Ying wrote: >>> Ryan Roberts writes: >>> >>>> On 07/07/2023 15:07, David Hildenbrand wrote: >>>>> On 07.07.23 15:57, Matthew Wilcox wrote: >>>>>> On Fri, Jul 07, 2023 at 01:29:02PM +0200, David Hildenbrand wrote: >>>>>>> On 07.07.23 11:52, Ryan Roberts wrote: >>>>>>>> On 07/07/2023 09:01, Huang, Ying wrote: >>>>>>>>> Although we can use smaller page order for FLEXIBLE_THP, it's hard to >>>>>>>>> avoid internal fragmentation completely.  So, I think that finally we >>>>>>>>> will need to provide a mechanism for the users to opt out, e.g., >>>>>>>>> something like "always madvise never" via >>>>>>>>> /sys/kernel/mm/transparent_hugepage/enabled.  I'm not sure whether it's >>>>>>>>> a good idea to reuse the existing interface of THP. >>>>>>>> >>>>>>>> I wouldn't want to tie this to the existing interface, simply because that >>>>>>>> implies that we would want to follow the "always" and "madvise" advice too; >>>>>>>> That >>>>>>>> means that on a thp=madvise system (which is certainly the case for android and >>>>>>>> other client systems) we would have to disable large anon folios for VMAs that >>>>>>>> haven't explicitly opted in. That breaks the intention that this should be an >>>>>>>> invisible performance boost. I think it's important to set the policy for >>>>>>>> use of >>>>>>> >>>>>>> It will never ever be a completely invisible performance boost, just like >>>>>>> ordinary THP. >>>>>>> >>>>>>> Using the exact same existing toggle is the right thing to do. If someone >>>>>>> specify "never" or "madvise", then do exactly that. >>>>>>> >>>>>>> It might make sense to have more modes or additional toggles, but >>>>>>> "madvise=never" means no memory waste. >>>>>> >>>>>> I hate the existing mechanisms.  They are an abdication of our >>>>>> responsibility, and an attempt to blame the user (be it the sysadmin >>>>>> or the programmer) of our code for using it wrongly.  We should not >>>>>> replicate this mistake. >>>>> >>>>> I don't agree regarding the programmer responsibility. In some cases the >>>>> programmer really doesn't want to get more memory populated than requested -- >>>>> and knows exactly why setting MADV_NOHUGEPAGE is the right thing to do. >>>>> >>>>> Regarding the madvise=never/madvise/always (sys admin decision), memory waste >>>>> (and nailing down bugs or working around them in customer setups) have been very >>>>> good reasons to let the admin have a word. >>>>> >>>>>> >>>>>> Our code should be auto-tuning.  I posted a long, detailed outline here: >>>>>> https://lore.kernel.org/linux-mm/Y%2FU8bQd15aUO97vS@casper.infradead.org/ >>>>>> >>>>> >>>>> Well, "auto-tuning" also should be perfect for everybody, but once reality >>>>> strikes you know it isn't. >>>>> >>>>> If people don't feel like using THP, let them have a word. The "madvise" config >>>>> option is probably more controversial. But the "always vs. never" absolutely >>>>> makes sense to me. >>>>> >>>>>>> I remember I raised it already in the past, but you *absolutely* have to >>>>>>> respect the MADV_NOHUGEPAGE flag. There is user space out there (for >>>>>>> example, userfaultfd) that doesn't want the kernel to populate any >>>>>>> additional page tables. So if you have to respect that already, then also >>>>>>> respect MADV_HUGEPAGE, simple. >>>>>> >>>>>> Possibly having uffd enabled on a VMA should disable using large folios, >>>>> >>>>> There are cases where we enable uffd *after* already touching memory (postcopy >>>>> live migration in QEMU being the famous example). That doesn't fly. >>>>> >>>>>> I can get behind that.  But the notion that userspace knows what it's >>>>>> doing ... hahaha.  Just ignore the madvise flags.  Userspace doesn't >>>>>> know what it's doing. >>>>> >>>>> If user space sets MADV_NOHUGEPAGE, it exactly knows what it is doing ... in >>>>> some cases. And these include cases I care about messing with sparse VM memory :) >>>>> >>>>> I have strong opinions against populating more than required when user space set >>>>> MADV_NOHUGEPAGE. >>>> >>>> I can see your point about honouring MADV_NOHUGEPAGE, so think that it is >>>> reasonable to fallback to allocating an order-0 page in a VMA that has it set. >>>> The app has gone out of its way to explicitly set it, after all. >>>> >>>> I think the correct behaviour for the global thp controls (cmdline and sysfs) >>>> are less obvious though. I could get on board with disabling large anon folios >>>> globally when thp="never". But for other situations, I would prefer to keep >>>> large anon folios enabled (treat "madvise" as "always"), >>> >>> If we have some mechanism to auto-tune the large folios usage, for >>> example, detect the internal fragmentation and split the large folio, >>> then we can use thp="always" as default configuration. If my memory >>> were correct, this is what Johannes and Alexander is working on. >> >> Could you point me to that work? I'd like to understand what the mechanism is. >> The other half of my work aims to use arm64's pte "contiguous bit" to tell the >> HW that a span of PTEs share the same mapping and is therefore coalesced into a >> single TLB entry. The side effect of this, however, is that we only have a >> single access and dirty bit for the whole contpte extent. So I'd like to avoid >> any mechanism that relies on getting access/dirty at the base page granularity >> for a large folio. > > Please take a look at the THP shrinker patchset, > > https://lore.kernel.org/linux-mm/cover.1667454613.git.alexlzhu@fb.com/ Thanks! > >>> >>>> with the argument that >>>> their order is much smaller than traditional THP and therefore the internal >>>> fragmentation is significantly reduced. >>> >>> Do you have any data for this? >> >> Some; its partly based on intuition that the smaller the allocation unit, the >> smaller the internal fragmentation. And partly on peak memory usage data I've >> collected for the benchmarks I'm running, comparing baseline-4k kernel with >> baseline-16k and baseline-64 kernels along with a 4k kernel that supports large >> anon folios (I appreciate that's not exactly what we are talking about here, and >> it's not exactly an extensive set of results!): >> >> >> Kernel Compliation with 8 Jobs: >> | kernel | peak | >> |:--------------|-------:| >> | baseline-4k | 0.0% | >> | anonfolio | 0.1% | >> | baseline-16k | 6.3% | >> | baseline-64k | 28.1% | >> >> >> Kernel Compliation with 80 Jobs: >> | kernel | peak | >> |:--------------|-------:| >> | baseline-4k | 0.0% | >> | anonfolio | 1.7% | >> | baseline-16k | 2.6% | >> | baseline-64k | 12.3% | >> > > Why is anonfolio better than baseline-64k if you always allocate 64k > anonymous folio? Because page cache uses 64k in baseline-64k? No, because the VMA boundaries are aligned to 4K and not 64K. Large Anon Folios only allocates a 64K folio if it does not breach the bounds of the VMA (and if it doesn't overlap other allocated PTEs). > > We may need to test some workloads with sparse access patterns too. Yes, I agree if you have a workload with a pathalogical memory access pattern where it writes to addresses with a stride of 64K, all contained in a single VMA, then you will end up allocating 16x the memory. This is obviously an unrealistic extreme though. > > Best Regards, > Huang, Ying > >>> >>>> I really don't want to end up with user >>>> space ever having to opt-in (with MADV_HUGEPAGE) to see the benefits of large >>>> anon folios. >>>> >>>> I still feel that it would be better for the thp and large anon folio controls >>>> to be independent though - what's the argument for tying them together? >>>> > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C79C0EB64D9 for ; Mon, 10 Jul 2023 09:26:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rh//I6daP9kmMz2QLUgQT8ZuBIN8/zvjap25+dU/xmo=; b=neFMn7Ghh9Kz9k hJZZMLX0nH7mkslXKujrQ9nt76ChJlCnyOQdFAh+b5PmTZ1WtqHctVfcvL2Lt7ZPzfXGjItyv4UE2 GN4NWyGX9ePefgZZE95VVCpJVs4wbLzxms+wRgmvDnO3HQxXLLHWMtA0ANdntZnKaduD2e2KSn6PE Ux97hQr0e1BD0Xt3O9nfGIcEzYYBNn2UIDINrYNtjWUgZaWnBGtnLA/zWG/DPcabaWHrq3WhNgfJh uSEF74FXYPdXJCW4QjrpRJzactulU5Urn6fFZRqfi6uCwvncS/JsXIZyWPdHE4Kt7pDA76qYVhWST Sk6fVKshpIoMC534crFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qIn9O-00AzpT-19; Mon, 10 Jul 2023 09:25:46 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qIn9K-00AzoW-1Y for linux-arm-kernel@lists.infradead.org; Mon, 10 Jul 2023 09:25:44 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 96F6B2B; Mon, 10 Jul 2023 02:26:22 -0700 (PDT) Received: from [10.57.77.63] (unknown [10.57.77.63]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5ACE53F740; Mon, 10 Jul 2023 02:25:38 -0700 (PDT) Message-ID: <501d1a6a-c9cf-94c3-b773-c648b944b30f@arm.com> Date: Mon, 10 Jul 2023 10:25:36 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance To: "Huang, Ying" Cc: David Hildenbrand , Matthew Wilcox , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Johannes Weiner , Alexander Zhu References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> <87y1jofoyi.fsf@yhuang6-desk2.ccr.corp.intel.com> <6c2f3127-9334-85ba-48f6-83a9c87abde0@arm.com> <874jmcf7kh.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Ryan Roberts In-Reply-To: <874jmcf7kh.fsf@yhuang6-desk2.ccr.corp.intel.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230710_022542_615727_4E01D2B9 X-CRM114-Status: GOOD ( 38.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMTAvMDcvMjAyMyAxMDoxOCwgSHVhbmcsIFlpbmcgd3JvdGU6Cj4gUnlhbiBSb2JlcnRzIDxy eWFuLnJvYmVydHNAYXJtLmNvbT4gd3JpdGVzOgo+IAo+PiBPbiAxMC8wNy8yMDIzIDA0OjAzLCBI dWFuZywgWWluZyB3cm90ZToKPj4+IFJ5YW4gUm9iZXJ0cyA8cnlhbi5yb2JlcnRzQGFybS5jb20+ IHdyaXRlczoKPj4+Cj4+Pj4gT24gMDcvMDcvMjAyMyAxNTowNywgRGF2aWQgSGlsZGVuYnJhbmQg d3JvdGU6Cj4+Pj4+IE9uIDA3LjA3LjIzIDE1OjU3LCBNYXR0aGV3IFdpbGNveCB3cm90ZToKPj4+ Pj4+IE9uIEZyaSwgSnVsIDA3LCAyMDIzIGF0IDAxOjI5OjAyUE0gKzAyMDAsIERhdmlkIEhpbGRl bmJyYW5kIHdyb3RlOgo+Pj4+Pj4+IE9uIDA3LjA3LjIzIDExOjUyLCBSeWFuIFJvYmVydHMgd3Jv dGU6Cj4+Pj4+Pj4+IE9uIDA3LzA3LzIwMjMgMDk6MDEsIEh1YW5nLCBZaW5nIHdyb3RlOgo+Pj4+ Pj4+Pj4gQWx0aG91Z2ggd2UgY2FuIHVzZSBzbWFsbGVyIHBhZ2Ugb3JkZXIgZm9yIEZMRVhJQkxF X1RIUCwgaXQncyBoYXJkIHRvCj4+Pj4+Pj4+PiBhdm9pZCBpbnRlcm5hbCBmcmFnbWVudGF0aW9u IGNvbXBsZXRlbHkuwqAgU28sIEkgdGhpbmsgdGhhdCBmaW5hbGx5IHdlCj4+Pj4+Pj4+PiB3aWxs IG5lZWQgdG8gcHJvdmlkZSBhIG1lY2hhbmlzbSBmb3IgdGhlIHVzZXJzIHRvIG9wdCBvdXQsIGUu Zy4sCj4+Pj4+Pj4+PiBzb21ldGhpbmcgbGlrZSAiYWx3YXlzIG1hZHZpc2UgbmV2ZXIiIHZpYQo+ Pj4+Pj4+Pj4gL3N5cy9rZXJuZWwvbW0vdHJhbnNwYXJlbnRfaHVnZXBhZ2UvZW5hYmxlZC7CoCBJ J20gbm90IHN1cmUgd2hldGhlciBpdCdzCj4+Pj4+Pj4+PiBhIGdvb2QgaWRlYSB0byByZXVzZSB0 aGUgZXhpc3RpbmcgaW50ZXJmYWNlIG9mIFRIUC4KPj4+Pj4+Pj4KPj4+Pj4+Pj4gSSB3b3VsZG4n dCB3YW50IHRvIHRpZSB0aGlzIHRvIHRoZSBleGlzdGluZyBpbnRlcmZhY2UsIHNpbXBseSBiZWNh dXNlIHRoYXQKPj4+Pj4+Pj4gaW1wbGllcyB0aGF0IHdlIHdvdWxkIHdhbnQgdG8gZm9sbG93IHRo ZSAiYWx3YXlzIiBhbmQgIm1hZHZpc2UiIGFkdmljZSB0b287Cj4+Pj4+Pj4+IFRoYXQKPj4+Pj4+ Pj4gbWVhbnMgdGhhdCBvbiBhIHRocD1tYWR2aXNlIHN5c3RlbSAod2hpY2ggaXMgY2VydGFpbmx5 IHRoZSBjYXNlIGZvciBhbmRyb2lkIGFuZAo+Pj4+Pj4+PiBvdGhlciBjbGllbnQgc3lzdGVtcykg d2Ugd291bGQgaGF2ZSB0byBkaXNhYmxlIGxhcmdlIGFub24gZm9saW9zIGZvciBWTUFzIHRoYXQK Pj4+Pj4+Pj4gaGF2ZW4ndCBleHBsaWNpdGx5IG9wdGVkIGluLiBUaGF0IGJyZWFrcyB0aGUgaW50 ZW50aW9uIHRoYXQgdGhpcyBzaG91bGQgYmUgYW4KPj4+Pj4+Pj4gaW52aXNpYmxlIHBlcmZvcm1h bmNlIGJvb3N0LiBJIHRoaW5rIGl0J3MgaW1wb3J0YW50IHRvIHNldCB0aGUgcG9saWN5IGZvcgo+ Pj4+Pj4+PiB1c2Ugb2YKPj4+Pj4+Pgo+Pj4+Pj4+IEl0IHdpbGwgbmV2ZXIgZXZlciBiZSBhIGNv bXBsZXRlbHkgaW52aXNpYmxlIHBlcmZvcm1hbmNlIGJvb3N0LCBqdXN0IGxpa2UKPj4+Pj4+PiBv cmRpbmFyeSBUSFAuCj4+Pj4+Pj4KPj4+Pj4+PiBVc2luZyB0aGUgZXhhY3Qgc2FtZSBleGlzdGlu ZyB0b2dnbGUgaXMgdGhlIHJpZ2h0IHRoaW5nIHRvIGRvLiBJZiBzb21lb25lCj4+Pj4+Pj4gc3Bl Y2lmeSAibmV2ZXIiIG9yICJtYWR2aXNlIiwgdGhlbiBkbyBleGFjdGx5IHRoYXQuCj4+Pj4+Pj4K Pj4+Pj4+PiBJdCBtaWdodCBtYWtlIHNlbnNlIHRvIGhhdmUgbW9yZSBtb2RlcyBvciBhZGRpdGlv bmFsIHRvZ2dsZXMsIGJ1dAo+Pj4+Pj4+ICJtYWR2aXNlPW5ldmVyIiBtZWFucyBubyBtZW1vcnkg d2FzdGUuCj4+Pj4+Pgo+Pj4+Pj4gSSBoYXRlIHRoZSBleGlzdGluZyBtZWNoYW5pc21zLsKgIFRo ZXkgYXJlIGFuIGFiZGljYXRpb24gb2Ygb3VyCj4+Pj4+PiByZXNwb25zaWJpbGl0eSwgYW5kIGFu IGF0dGVtcHQgdG8gYmxhbWUgdGhlIHVzZXIgKGJlIGl0IHRoZSBzeXNhZG1pbgo+Pj4+Pj4gb3Ig dGhlIHByb2dyYW1tZXIpIG9mIG91ciBjb2RlIGZvciB1c2luZyBpdCB3cm9uZ2x5LsKgIFdlIHNo b3VsZCBub3QKPj4+Pj4+IHJlcGxpY2F0ZSB0aGlzIG1pc3Rha2UuCj4+Pj4+Cj4+Pj4+IEkgZG9u J3QgYWdyZWUgcmVnYXJkaW5nIHRoZSBwcm9ncmFtbWVyIHJlc3BvbnNpYmlsaXR5LiBJbiBzb21l IGNhc2VzIHRoZQo+Pj4+PiBwcm9ncmFtbWVyIHJlYWxseSBkb2Vzbid0IHdhbnQgdG8gZ2V0IG1v cmUgbWVtb3J5IHBvcHVsYXRlZCB0aGFuIHJlcXVlc3RlZCAtLQo+Pj4+PiBhbmQga25vd3MgZXhh Y3RseSB3aHkgc2V0dGluZyBNQURWX05PSFVHRVBBR0UgaXMgdGhlIHJpZ2h0IHRoaW5nIHRvIGRv Lgo+Pj4+Pgo+Pj4+PiBSZWdhcmRpbmcgdGhlIG1hZHZpc2U9bmV2ZXIvbWFkdmlzZS9hbHdheXMg KHN5cyBhZG1pbiBkZWNpc2lvbiksIG1lbW9yeSB3YXN0ZQo+Pj4+PiAoYW5kIG5haWxpbmcgZG93 biBidWdzIG9yIHdvcmtpbmcgYXJvdW5kIHRoZW0gaW4gY3VzdG9tZXIgc2V0dXBzKSBoYXZlIGJl ZW4gdmVyeQo+Pj4+PiBnb29kIHJlYXNvbnMgdG8gbGV0IHRoZSBhZG1pbiBoYXZlIGEgd29yZC4K Pj4+Pj4KPj4+Pj4+Cj4+Pj4+PiBPdXIgY29kZSBzaG91bGQgYmUgYXV0by10dW5pbmcuwqAgSSBw b3N0ZWQgYSBsb25nLCBkZXRhaWxlZCBvdXRsaW5lIGhlcmU6Cj4+Pj4+PiBodHRwczovL2xvcmUu a2VybmVsLm9yZy9saW51eC1tbS9ZJTJGVThiUWQxNWFVTzk3dlNAY2FzcGVyLmluZnJhZGVhZC5v cmcvCj4+Pj4+Pgo+Pj4+Pgo+Pj4+PiBXZWxsLCAiYXV0by10dW5pbmciIGFsc28gc2hvdWxkIGJl IHBlcmZlY3QgZm9yIGV2ZXJ5Ym9keSwgYnV0IG9uY2UgcmVhbGl0eQo+Pj4+PiBzdHJpa2VzIHlv dSBrbm93IGl0IGlzbid0Lgo+Pj4+Pgo+Pj4+PiBJZiBwZW9wbGUgZG9uJ3QgZmVlbCBsaWtlIHVz aW5nIFRIUCwgbGV0IHRoZW0gaGF2ZSBhIHdvcmQuIFRoZSAibWFkdmlzZSIgY29uZmlnCj4+Pj4+ IG9wdGlvbiBpcyBwcm9iYWJseSBtb3JlIGNvbnRyb3ZlcnNpYWwuIEJ1dCB0aGUgImFsd2F5cyB2 cy4gbmV2ZXIiIGFic29sdXRlbHkKPj4+Pj4gbWFrZXMgc2Vuc2UgdG8gbWUuCj4+Pj4+Cj4+Pj4+ Pj4gSSByZW1lbWJlciBJIHJhaXNlZCBpdCBhbHJlYWR5IGluIHRoZSBwYXN0LCBidXQgeW91ICph YnNvbHV0ZWx5KiBoYXZlIHRvCj4+Pj4+Pj4gcmVzcGVjdCB0aGUgTUFEVl9OT0hVR0VQQUdFIGZs YWcuIFRoZXJlIGlzIHVzZXIgc3BhY2Ugb3V0IHRoZXJlIChmb3IKPj4+Pj4+PiBleGFtcGxlLCB1 c2VyZmF1bHRmZCkgdGhhdCBkb2Vzbid0IHdhbnQgdGhlIGtlcm5lbCB0byBwb3B1bGF0ZSBhbnkK Pj4+Pj4+PiBhZGRpdGlvbmFsIHBhZ2UgdGFibGVzLiBTbyBpZiB5b3UgaGF2ZSB0byByZXNwZWN0 IHRoYXQgYWxyZWFkeSwgdGhlbiBhbHNvCj4+Pj4+Pj4gcmVzcGVjdCBNQURWX0hVR0VQQUdFLCBz aW1wbGUuCj4+Pj4+Pgo+Pj4+Pj4gUG9zc2libHkgaGF2aW5nIHVmZmQgZW5hYmxlZCBvbiBhIFZN QSBzaG91bGQgZGlzYWJsZSB1c2luZyBsYXJnZSBmb2xpb3MsCj4+Pj4+Cj4+Pj4+IFRoZXJlIGFy ZSBjYXNlcyB3aGVyZSB3ZSBlbmFibGUgdWZmZCAqYWZ0ZXIqIGFscmVhZHkgdG91Y2hpbmcgbWVt b3J5IChwb3N0Y29weQo+Pj4+PiBsaXZlIG1pZ3JhdGlvbiBpbiBRRU1VIGJlaW5nIHRoZSBmYW1v dXMgZXhhbXBsZSkuIFRoYXQgZG9lc24ndCBmbHkuCj4+Pj4+Cj4+Pj4+PiBJIGNhbiBnZXQgYmVo aW5kIHRoYXQuwqAgQnV0IHRoZSBub3Rpb24gdGhhdCB1c2Vyc3BhY2Uga25vd3Mgd2hhdCBpdCdz Cj4+Pj4+PiBkb2luZyAuLi4gaGFoYWhhLsKgIEp1c3QgaWdub3JlIHRoZSBtYWR2aXNlIGZsYWdz LsKgIFVzZXJzcGFjZSBkb2Vzbid0Cj4+Pj4+PiBrbm93IHdoYXQgaXQncyBkb2luZy4KPj4+Pj4K Pj4+Pj4gSWYgdXNlciBzcGFjZSBzZXRzIE1BRFZfTk9IVUdFUEFHRSwgaXQgZXhhY3RseSBrbm93 cyB3aGF0IGl0IGlzIGRvaW5nIC4uLiBpbgo+Pj4+PiBzb21lIGNhc2VzLiBBbmQgdGhlc2UgaW5j bHVkZSBjYXNlcyBJIGNhcmUgYWJvdXQgbWVzc2luZyB3aXRoIHNwYXJzZSBWTSBtZW1vcnkgOikK Pj4+Pj4KPj4+Pj4gSSBoYXZlIHN0cm9uZyBvcGluaW9ucyBhZ2FpbnN0IHBvcHVsYXRpbmcgbW9y ZSB0aGFuIHJlcXVpcmVkIHdoZW4gdXNlciBzcGFjZSBzZXQKPj4+Pj4gTUFEVl9OT0hVR0VQQUdF Lgo+Pj4+Cj4+Pj4gSSBjYW4gc2VlIHlvdXIgcG9pbnQgYWJvdXQgaG9ub3VyaW5nIE1BRFZfTk9I VUdFUEFHRSwgc28gdGhpbmsgdGhhdCBpdCBpcwo+Pj4+IHJlYXNvbmFibGUgdG8gZmFsbGJhY2sg dG8gYWxsb2NhdGluZyBhbiBvcmRlci0wIHBhZ2UgaW4gYSBWTUEgdGhhdCBoYXMgaXQgc2V0Lgo+ Pj4+IFRoZSBhcHAgaGFzIGdvbmUgb3V0IG9mIGl0cyB3YXkgdG8gZXhwbGljaXRseSBzZXQgaXQs IGFmdGVyIGFsbC4KPj4+Pgo+Pj4+IEkgdGhpbmsgdGhlIGNvcnJlY3QgYmVoYXZpb3VyIGZvciB0 aGUgZ2xvYmFsIHRocCBjb250cm9scyAoY21kbGluZSBhbmQgc3lzZnMpCj4+Pj4gYXJlIGxlc3Mg b2J2aW91cyB0aG91Z2guIEkgY291bGQgZ2V0IG9uIGJvYXJkIHdpdGggZGlzYWJsaW5nIGxhcmdl IGFub24gZm9saW9zCj4+Pj4gZ2xvYmFsbHkgd2hlbiB0aHA9Im5ldmVyIi4gQnV0IGZvciBvdGhl ciBzaXR1YXRpb25zLCBJIHdvdWxkIHByZWZlciB0byBrZWVwCj4+Pj4gbGFyZ2UgYW5vbiBmb2xp b3MgZW5hYmxlZCAodHJlYXQgIm1hZHZpc2UiIGFzICJhbHdheXMiKSwKPj4+Cj4+PiBJZiB3ZSBo YXZlIHNvbWUgbWVjaGFuaXNtIHRvIGF1dG8tdHVuZSB0aGUgbGFyZ2UgZm9saW9zIHVzYWdlLCBm b3IKPj4+IGV4YW1wbGUsIGRldGVjdCB0aGUgaW50ZXJuYWwgZnJhZ21lbnRhdGlvbiBhbmQgc3Bs aXQgdGhlIGxhcmdlIGZvbGlvLAo+Pj4gdGhlbiB3ZSBjYW4gdXNlIHRocD0iYWx3YXlzIiBhcyBk ZWZhdWx0IGNvbmZpZ3VyYXRpb24uICBJZiBteSBtZW1vcnkKPj4+IHdlcmUgY29ycmVjdCwgdGhp cyBpcyB3aGF0IEpvaGFubmVzIGFuZCBBbGV4YW5kZXIgaXMgd29ya2luZyBvbi4KPj4KPj4gQ291 bGQgeW91IHBvaW50IG1lIHRvIHRoYXQgd29yaz8gSSdkIGxpa2UgdG8gdW5kZXJzdGFuZCB3aGF0 IHRoZSBtZWNoYW5pc20gaXMuCj4+IFRoZSBvdGhlciBoYWxmIG9mIG15IHdvcmsgYWltcyB0byB1 c2UgYXJtNjQncyBwdGUgImNvbnRpZ3VvdXMgYml0IiB0byB0ZWxsIHRoZQo+PiBIVyB0aGF0IGEg c3BhbiBvZiBQVEVzIHNoYXJlIHRoZSBzYW1lIG1hcHBpbmcgYW5kIGlzIHRoZXJlZm9yZSBjb2Fs ZXNjZWQgaW50byBhCj4+IHNpbmdsZSBUTEIgZW50cnkuIFRoZSBzaWRlIGVmZmVjdCBvZiB0aGlz LCBob3dldmVyLCBpcyB0aGF0IHdlIG9ubHkgaGF2ZSBhCj4+IHNpbmdsZSBhY2Nlc3MgYW5kIGRp cnR5IGJpdCBmb3IgdGhlIHdob2xlIGNvbnRwdGUgZXh0ZW50LiBTbyBJJ2QgbGlrZSB0byBhdm9p ZAo+PiBhbnkgbWVjaGFuaXNtIHRoYXQgcmVsaWVzIG9uIGdldHRpbmcgYWNjZXNzL2RpcnR5IGF0 IHRoZSBiYXNlIHBhZ2UgZ3JhbnVsYXJpdHkKPj4gZm9yIGEgbGFyZ2UgZm9saW8uCj4gCj4gUGxl YXNlIHRha2UgYSBsb29rIGF0IHRoZSBUSFAgc2hyaW5rZXIgcGF0Y2hzZXQsCj4gCj4gaHR0cHM6 Ly9sb3JlLmtlcm5lbC5vcmcvbGludXgtbW0vY292ZXIuMTY2NzQ1NDYxMy5naXQuYWxleGx6aHVA ZmIuY29tLwoKVGhhbmtzIQoKPiAKPj4+Cj4+Pj4gd2l0aCB0aGUgYXJndW1lbnQgdGhhdAo+Pj4+ IHRoZWlyIG9yZGVyIGlzIG11Y2ggc21hbGxlciB0aGFuIHRyYWRpdGlvbmFsIFRIUCBhbmQgdGhl cmVmb3JlIHRoZSBpbnRlcm5hbAo+Pj4+IGZyYWdtZW50YXRpb24gaXMgc2lnbmlmaWNhbnRseSBy ZWR1Y2VkLgo+Pj4KPj4+IERvIHlvdSBoYXZlIGFueSBkYXRhIGZvciB0aGlzPwo+Pgo+PiBTb21l OyBpdHMgcGFydGx5IGJhc2VkIG9uIGludHVpdGlvbiB0aGF0IHRoZSBzbWFsbGVyIHRoZSBhbGxv Y2F0aW9uIHVuaXQsIHRoZQo+PiBzbWFsbGVyIHRoZSBpbnRlcm5hbCBmcmFnbWVudGF0aW9uLiBB bmQgcGFydGx5IG9uIHBlYWsgbWVtb3J5IHVzYWdlIGRhdGEgSSd2ZQo+PiBjb2xsZWN0ZWQgZm9y IHRoZSBiZW5jaG1hcmtzIEknbSBydW5uaW5nLCBjb21wYXJpbmcgYmFzZWxpbmUtNGsga2VybmVs IHdpdGgKPj4gYmFzZWxpbmUtMTZrIGFuZCBiYXNlbGluZS02NCBrZXJuZWxzIGFsb25nIHdpdGgg YSA0ayBrZXJuZWwgdGhhdCBzdXBwb3J0cyBsYXJnZQo+PiBhbm9uIGZvbGlvcyAoSSBhcHByZWNp YXRlIHRoYXQncyBub3QgZXhhY3RseSB3aGF0IHdlIGFyZSB0YWxraW5nIGFib3V0IGhlcmUsIGFu ZAo+PiBpdCdzIG5vdCBleGFjdGx5IGFuIGV4dGVuc2l2ZSBzZXQgb2YgcmVzdWx0cyEpOgo+Pgo+ Pgo+PiBLZXJuZWwgQ29tcGxpYXRpb24gd2l0aCA4IEpvYnM6Cj4+IHwga2VybmVsICAgICAgICB8 ICAgcGVhayB8Cj4+IHw6LS0tLS0tLS0tLS0tLS18LS0tLS0tLTp8Cj4+IHwgYmFzZWxpbmUtNGsg ICB8ICAgMC4wJSB8Cj4+IHwgYW5vbmZvbGlvICAgICB8ICAgMC4xJSB8Cj4+IHwgYmFzZWxpbmUt MTZrICB8ICAgNi4zJSB8Cj4+IHwgYmFzZWxpbmUtNjRrICB8ICAyOC4xJSB8Cj4+Cj4+Cj4+IEtl cm5lbCBDb21wbGlhdGlvbiB3aXRoIDgwIEpvYnM6Cj4+IHwga2VybmVsICAgICAgICB8ICAgcGVh ayB8Cj4+IHw6LS0tLS0tLS0tLS0tLS18LS0tLS0tLTp8Cj4+IHwgYmFzZWxpbmUtNGsgICB8ICAg MC4wJSB8Cj4+IHwgYW5vbmZvbGlvICAgICB8ICAgMS43JSB8Cj4+IHwgYmFzZWxpbmUtMTZrICB8 ICAgMi42JSB8Cj4+IHwgYmFzZWxpbmUtNjRrICB8ICAxMi4zJSB8Cj4+Cj4gCj4gV2h5IGlzIGFu b25mb2xpbyBiZXR0ZXIgdGhhbiBiYXNlbGluZS02NGsgaWYgeW91IGFsd2F5cyBhbGxvY2F0ZSA2 NGsKPiBhbm9ueW1vdXMgZm9saW8/ICBCZWNhdXNlIHBhZ2UgY2FjaGUgdXNlcyA2NGsgaW4gYmFz ZWxpbmUtNjRrPwoKTm8sIGJlY2F1c2UgdGhlIFZNQSBib3VuZGFyaWVzIGFyZSBhbGlnbmVkIHRv IDRLIGFuZCBub3QgNjRLLiBMYXJnZSBBbm9uIEZvbGlvcwpvbmx5IGFsbG9jYXRlcyBhIDY0SyBm b2xpbyBpZiBpdCBkb2VzIG5vdCBicmVhY2ggdGhlIGJvdW5kcyBvZiB0aGUgVk1BIChhbmQgaWYK aXQgZG9lc24ndCBvdmVybGFwIG90aGVyIGFsbG9jYXRlZCBQVEVzKS4KCj4gCj4gV2UgbWF5IG5l ZWQgdG8gdGVzdCBzb21lIHdvcmtsb2FkcyB3aXRoIHNwYXJzZSBhY2Nlc3MgcGF0dGVybnMgdG9v LgoKWWVzLCBJIGFncmVlIGlmIHlvdSBoYXZlIGEgd29ya2xvYWQgd2l0aCBhIHBhdGhhbG9naWNh bCBtZW1vcnkgYWNjZXNzIHBhdHRlcm4Kd2hlcmUgaXQgd3JpdGVzIHRvIGFkZHJlc3NlcyB3aXRo IGEgc3RyaWRlIG9mIDY0SywgYWxsIGNvbnRhaW5lZCBpbiBhIHNpbmdsZQpWTUEsIHRoZW4geW91 IHdpbGwgZW5kIHVwIGFsbG9jYXRpbmcgMTZ4IHRoZSBtZW1vcnkuIFRoaXMgaXMgb2J2aW91c2x5 IGFuCnVucmVhbGlzdGljIGV4dHJlbWUgdGhvdWdoLgoKPiAKPiBCZXN0IFJlZ2FyZHMsCj4gSHVh bmcsIFlpbmcKPiAKPj4+Cj4+Pj4gSSByZWFsbHkgZG9uJ3Qgd2FudCB0byBlbmQgdXAgd2l0aCB1 c2VyCj4+Pj4gc3BhY2UgZXZlciBoYXZpbmcgdG8gb3B0LWluICh3aXRoIE1BRFZfSFVHRVBBR0Up IHRvIHNlZSB0aGUgYmVuZWZpdHMgb2YgbGFyZ2UKPj4+PiBhbm9uIGZvbGlvcy4KPj4+Pgo+Pj4+ IEkgc3RpbGwgZmVlbCB0aGF0IGl0IHdvdWxkIGJlIGJldHRlciBmb3IgdGhlIHRocCBhbmQgbGFy Z2UgYW5vbiBmb2xpbyBjb250cm9scwo+Pj4+IHRvIGJlIGluZGVwZW5kZW50IHRob3VnaCAtIHdo YXQncyB0aGUgYXJndW1lbnQgZm9yIHR5aW5nIHRoZW0gdG9nZXRoZXI/Cj4+Pj4KPiAKCgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2Vy bmVsIG1haWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0 cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVs Cg==