From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0C68EB64D9 for ; Mon, 10 Jul 2023 09:20:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231552AbjGJJUX (ORCPT ); Mon, 10 Jul 2023 05:20:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230257AbjGJJUV (ORCPT ); Mon, 10 Jul 2023 05:20:21 -0400 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0128B8E for ; Mon, 10 Jul 2023 02:20:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688980819; x=1720516819; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version:content-transfer-encoding; bh=ly/MDRV2EOLHcR5c4dEoNmyijTn1dRwSPnIkbL/BJH0=; b=lNrYko/wa+aaadU0DFKPgPX3FcNyWk3SrgzxRmosiFO5LXWYjZT94iDX hMhzvlwL/3aIMRp68PP3LV/q4jmfN8U48lLUXVujX6S7+0r32JJl5UGPs 3xtVQ33AQf351eDuBnhtu03j7fmTC0C9js0+kmiY52kcbxjhI3oq9BsNT f1o5Qd/Mn47ZUtdpvH6TTfSmTVGzYtBfe3wVdCeLkmCbEY20rLuMNOG9V QyZMr8Po0vq9+rQjc6S9fnK5XvSILYPowMzlhOU+m65err5CdgyHk06N0 UNqtJPVp9cf4kSLbvZYsw6Q0LyPBClvFwIywpK8RpQZmsOWvEQSnaqn7q w==; X-IronPort-AV: E=McAfee;i="6600,9927,10766"; a="427988494" X-IronPort-AV: E=Sophos;i="6.01,194,1684825200"; d="scan'208";a="427988494" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jul 2023 02:20:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10766"; a="670880190" X-IronPort-AV: E=Sophos;i="6.01,194,1684825200"; d="scan'208";a="670880190" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jul 2023 02:20:15 -0700 From: "Huang, Ying" To: Ryan Roberts Cc: David Hildenbrand , Matthew Wilcox , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , , , , Johannes Weiner , Alexander Zhu Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> <87y1jofoyi.fsf@yhuang6-desk2.ccr.corp.intel.com> <6c2f3127-9334-85ba-48f6-83a9c87abde0@arm.com> Date: Mon, 10 Jul 2023 17:18:38 +0800 In-Reply-To: <6c2f3127-9334-85ba-48f6-83a9c87abde0@arm.com> (Ryan Roberts's message of "Mon, 10 Jul 2023 09:55:14 +0100") Message-ID: <874jmcf7kh.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Ryan Roberts writes: > On 10/07/2023 04:03, Huang, Ying wrote: >> Ryan Roberts writes: >>=20 >>> On 07/07/2023 15:07, David Hildenbrand wrote: >>>> On 07.07.23 15:57, Matthew Wilcox wrote: >>>>> On Fri, Jul 07, 2023 at 01:29:02PM +0200, David Hildenbrand wrote: >>>>>> On 07.07.23 11:52, Ryan Roberts wrote: >>>>>>> On 07/07/2023 09:01, Huang, Ying wrote: >>>>>>>> Although we can use smaller page order for FLEXIBLE_THP, it's hard= to >>>>>>>> avoid internal fragmentation completely.=C2=A0 So, I think that fi= nally we >>>>>>>> will need to provide a mechanism for the users to opt out, e.g., >>>>>>>> something like "always madvise never" via >>>>>>>> /sys/kernel/mm/transparent_hugepage/enabled.=C2=A0 I'm not sure wh= ether it's >>>>>>>> a good idea to reuse the existing interface of THP. >>>>>>> >>>>>>> I wouldn't want to tie this to the existing interface, simply becau= se that >>>>>>> implies that we would want to follow the "always" and "madvise" adv= ice too; >>>>>>> That >>>>>>> means that on a thp=3Dmadvise system (which is certainly the case f= or android and >>>>>>> other client systems) we would have to disable large anon folios fo= r VMAs that >>>>>>> haven't explicitly opted in. That breaks the intention that this sh= ould be an >>>>>>> invisible performance boost. I think it's important to set the poli= cy for >>>>>>> use of >>>>>> >>>>>> It will never ever be a completely invisible performance boost, just= like >>>>>> ordinary THP. >>>>>> >>>>>> Using the exact same existing toggle is the right thing to do. If so= meone >>>>>> specify "never" or "madvise", then do exactly that. >>>>>> >>>>>> It might make sense to have more modes or additional toggles, but >>>>>> "madvise=3Dnever" means no memory waste. >>>>> >>>>> I hate the existing mechanisms.=C2=A0 They are an abdication of our >>>>> responsibility, and an attempt to blame the user (be it the sysadmin >>>>> or the programmer) of our code for using it wrongly.=C2=A0 We should = not >>>>> replicate this mistake. >>>> >>>> I don't agree regarding the programmer responsibility. In some cases t= he >>>> programmer really doesn't want to get more memory populated than reque= sted -- >>>> and knows exactly why setting MADV_NOHUGEPAGE is the right thing to do. >>>> >>>> Regarding the madvise=3Dnever/madvise/always (sys admin decision), mem= ory waste >>>> (and nailing down bugs or working around them in customer setups) have= been very >>>> good reasons to let the admin have a word. >>>> >>>>> >>>>> Our code should be auto-tuning.=C2=A0 I posted a long, detailed outli= ne here: >>>>> https://lore.kernel.org/linux-mm/Y%2FU8bQd15aUO97vS@casper.infradead.= org/ >>>>> >>>> >>>> Well, "auto-tuning" also should be perfect for everybody, but once rea= lity >>>> strikes you know it isn't. >>>> >>>> If people don't feel like using THP, let them have a word. The "madvis= e" config >>>> option is probably more controversial. But the "always vs. never" abso= lutely >>>> makes sense to me. >>>> >>>>>> I remember I raised it already in the past, but you *absolutely* hav= e to >>>>>> respect the MADV_NOHUGEPAGE flag. There is user space out there (for >>>>>> example, userfaultfd) that doesn't want the kernel to populate any >>>>>> additional page tables. So if you have to respect that already, then= also >>>>>> respect MADV_HUGEPAGE, simple. >>>>> >>>>> Possibly having uffd enabled on a VMA should disable using large foli= os, >>>> >>>> There are cases where we enable uffd *after* already touching memory (= postcopy >>>> live migration in QEMU being the famous example). That doesn't fly. >>>> >>>>> I can get behind that.=C2=A0 But the notion that userspace knows what= it's >>>>> doing ... hahaha.=C2=A0 Just ignore the madvise flags.=C2=A0 Userspac= e doesn't >>>>> know what it's doing. >>>> >>>> If user space sets MADV_NOHUGEPAGE, it exactly knows what it is doing = ... in >>>> some cases. And these include cases I care about messing with sparse V= M memory :) >>>> >>>> I have strong opinions against populating more than required when user= space set >>>> MADV_NOHUGEPAGE. >>> >>> I can see your point about honouring MADV_NOHUGEPAGE, so think that it = is >>> reasonable to fallback to allocating an order-0 page in a VMA that has = it set. >>> The app has gone out of its way to explicitly set it, after all. >>> >>> I think the correct behaviour for the global thp controls (cmdline and = sysfs) >>> are less obvious though. I could get on board with disabling large anon= folios >>> globally when thp=3D"never". But for other situations, I would prefer t= o keep >>> large anon folios enabled (treat "madvise" as "always"), >>=20 >> If we have some mechanism to auto-tune the large folios usage, for >> example, detect the internal fragmentation and split the large folio, >> then we can use thp=3D"always" as default configuration. If my memory >> were correct, this is what Johannes and Alexander is working on. > > Could you point me to that work? I'd like to understand what the mechanis= m is. > The other half of my work aims to use arm64's pte "contiguous bit" to tel= l the > HW that a span of PTEs share the same mapping and is therefore coalesced = into a > single TLB entry. The side effect of this, however, is that we only have a > single access and dirty bit for the whole contpte extent. So I'd like to = avoid > any mechanism that relies on getting access/dirty at the base page granul= arity > for a large folio. Please take a look at the THP shrinker patchset, https://lore.kernel.org/linux-mm/cover.1667454613.git.alexlzhu@fb.com/ >>=20 >>> with the argument that >>> their order is much smaller than traditional THP and therefore the inte= rnal >>> fragmentation is significantly reduced. >>=20 >> Do you have any data for this? > > Some; its partly based on intuition that the smaller the allocation unit,= the > smaller the internal fragmentation. And partly on peak memory usage data = I've > collected for the benchmarks I'm running, comparing baseline-4k kernel wi= th > baseline-16k and baseline-64 kernels along with a 4k kernel that supports= large > anon folios (I appreciate that's not exactly what we are talking about he= re, and > it's not exactly an extensive set of results!): > > > Kernel Compliation with 8 Jobs: > | kernel | peak | > |:--------------|-------:| > | baseline-4k | 0.0% | > | anonfolio | 0.1% | > | baseline-16k | 6.3% | > | baseline-64k | 28.1% | > > > Kernel Compliation with 80 Jobs: > | kernel | peak | > |:--------------|-------:| > | baseline-4k | 0.0% | > | anonfolio | 1.7% | > | baseline-16k | 2.6% | > | baseline-64k | 12.3% | > Why is anonfolio better than baseline-64k if you always allocate 64k anonymous folio? Because page cache uses 64k in baseline-64k? We may need to test some workloads with sparse access patterns too. Best Regards, Huang, Ying >>=20 >>> I really don't want to end up with user >>> space ever having to opt-in (with MADV_HUGEPAGE) to see the benefits of= large >>> anon folios. >>> >>> I still feel that it would be better for the thp and large anon folio c= ontrols >>> to be independent though - what's the argument for tying them together? >>> From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DDD21EB64D9 for ; Mon, 10 Jul 2023 09:20:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:In-Reply-To: Date:References:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=N4tTtwvWnKIMvTs44c/sxkJGEo6hTkkBl8GIV/Eg/E8=; b=YEQbnip2QSRiKL u7CmuSSLrivTJ9SkdZKaq4lGD8BOAYAiYkvBeKwOmdbkM17vvTBwyHXtp567csBZImXvxpKuHKnzf /0xpqRQjaitzqqejavoY3lLsFn+TRDI9LsIsHVVL1PuI6n1ikK7J9rIK/yHXH870gl7Y4apqeEP08 14sNonjZoaL+bj4QgOn2MS7mFgQbG1sequ7Q2cWVXVgG/O2/myRGTRvlNraqW5478BlIO8gWX/Rf6 HohDD/5cQEHvM/dMaOOfpDidwHaxAmO4aGYq2LchYdxAPB2kMW92LoFu3wlN6/agzI2CmvnrwKS8t kNqbM0B2XmPcXSpsPa/Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qIn4C-00AzGK-1F; Mon, 10 Jul 2023 09:20:24 +0000 Received: from mga06b.intel.com ([134.134.136.31] helo=mga06.intel.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qIn48-00AzFE-10 for linux-arm-kernel@lists.infradead.org; Mon, 10 Jul 2023 09:20:22 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688980820; x=1720516820; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version:content-transfer-encoding; bh=ly/MDRV2EOLHcR5c4dEoNmyijTn1dRwSPnIkbL/BJH0=; b=UiCBxgWV4LlKJQAuty6LiBjicGIRHN/IfEUK0lRXLjSYiXObBVo1nG+S C4HktOcIWkVOk9bAYLsXRK+1mRno3h0Il+bglDs+GD0PsdcD6prF1U4Um 51P9O0P+7WCp30ovE0Y3auPIZsWWO3E7ONh5o92A24V7+v3bserS6qoz/ 414rCiNSdp+3W5iQ5ir0r57+Prv1bskwQwJqLbWXMwVRtGO08rgm+RONS 4nqi6/MKCdTnV3N6W11rHW57snIxfoA1yFD8Dv3GNHRjxaIY0VqbsAi0J ApI1Sesvi+z9YAXAJB11MSpkE7yNnBycYOD/SHMrYlFg8mV8SNgLjwUey Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10766"; a="427988500" X-IronPort-AV: E=Sophos;i="6.01,194,1684825200"; d="scan'208";a="427988500" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jul 2023 02:20:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10766"; a="670880190" X-IronPort-AV: E=Sophos;i="6.01,194,1684825200"; d="scan'208";a="670880190" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jul 2023 02:20:15 -0700 From: "Huang, Ying" To: Ryan Roberts Cc: David Hildenbrand , Matthew Wilcox , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , , , , Johannes Weiner , Alexander Zhu Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> <87y1jofoyi.fsf@yhuang6-desk2.ccr.corp.intel.com> <6c2f3127-9334-85ba-48f6-83a9c87abde0@arm.com> Date: Mon, 10 Jul 2023 17:18:38 +0800 In-Reply-To: <6c2f3127-9334-85ba-48f6-83a9c87abde0@arm.com> (Ryan Roberts's message of "Mon, 10 Jul 2023 09:55:14 +0100") Message-ID: <874jmcf7kh.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230710_022020_397612_8F8CD170 X-CRM114-Status: GOOD ( 37.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org UnlhbiBSb2JlcnRzIDxyeWFuLnJvYmVydHNAYXJtLmNvbT4gd3JpdGVzOgoKPiBPbiAxMC8wNy8y MDIzIDA0OjAzLCBIdWFuZywgWWluZyB3cm90ZToKPj4gUnlhbiBSb2JlcnRzIDxyeWFuLnJvYmVy dHNAYXJtLmNvbT4gd3JpdGVzOgo+PiAKPj4+IE9uIDA3LzA3LzIwMjMgMTU6MDcsIERhdmlkIEhp bGRlbmJyYW5kIHdyb3RlOgo+Pj4+IE9uIDA3LjA3LjIzIDE1OjU3LCBNYXR0aGV3IFdpbGNveCB3 cm90ZToKPj4+Pj4gT24gRnJpLCBKdWwgMDcsIDIwMjMgYXQgMDE6Mjk6MDJQTSArMDIwMCwgRGF2 aWQgSGlsZGVuYnJhbmQgd3JvdGU6Cj4+Pj4+PiBPbiAwNy4wNy4yMyAxMTo1MiwgUnlhbiBSb2Jl cnRzIHdyb3RlOgo+Pj4+Pj4+IE9uIDA3LzA3LzIwMjMgMDk6MDEsIEh1YW5nLCBZaW5nIHdyb3Rl Ogo+Pj4+Pj4+PiBBbHRob3VnaCB3ZSBjYW4gdXNlIHNtYWxsZXIgcGFnZSBvcmRlciBmb3IgRkxF WElCTEVfVEhQLCBpdCdzIGhhcmQgdG8KPj4+Pj4+Pj4gYXZvaWQgaW50ZXJuYWwgZnJhZ21lbnRh dGlvbiBjb21wbGV0ZWx5LsKgIFNvLCBJIHRoaW5rIHRoYXQgZmluYWxseSB3ZQo+Pj4+Pj4+PiB3 aWxsIG5lZWQgdG8gcHJvdmlkZSBhIG1lY2hhbmlzbSBmb3IgdGhlIHVzZXJzIHRvIG9wdCBvdXQs IGUuZy4sCj4+Pj4+Pj4+IHNvbWV0aGluZyBsaWtlICJhbHdheXMgbWFkdmlzZSBuZXZlciIgdmlh Cj4+Pj4+Pj4+IC9zeXMva2VybmVsL21tL3RyYW5zcGFyZW50X2h1Z2VwYWdlL2VuYWJsZWQuwqAg SSdtIG5vdCBzdXJlIHdoZXRoZXIgaXQncwo+Pj4+Pj4+PiBhIGdvb2QgaWRlYSB0byByZXVzZSB0 aGUgZXhpc3RpbmcgaW50ZXJmYWNlIG9mIFRIUC4KPj4+Pj4+Pgo+Pj4+Pj4+IEkgd291bGRuJ3Qg d2FudCB0byB0aWUgdGhpcyB0byB0aGUgZXhpc3RpbmcgaW50ZXJmYWNlLCBzaW1wbHkgYmVjYXVz ZSB0aGF0Cj4+Pj4+Pj4gaW1wbGllcyB0aGF0IHdlIHdvdWxkIHdhbnQgdG8gZm9sbG93IHRoZSAi YWx3YXlzIiBhbmQgIm1hZHZpc2UiIGFkdmljZSB0b287Cj4+Pj4+Pj4gVGhhdAo+Pj4+Pj4+IG1l YW5zIHRoYXQgb24gYSB0aHA9bWFkdmlzZSBzeXN0ZW0gKHdoaWNoIGlzIGNlcnRhaW5seSB0aGUg Y2FzZSBmb3IgYW5kcm9pZCBhbmQKPj4+Pj4+PiBvdGhlciBjbGllbnQgc3lzdGVtcykgd2Ugd291 bGQgaGF2ZSB0byBkaXNhYmxlIGxhcmdlIGFub24gZm9saW9zIGZvciBWTUFzIHRoYXQKPj4+Pj4+ PiBoYXZlbid0IGV4cGxpY2l0bHkgb3B0ZWQgaW4uIFRoYXQgYnJlYWtzIHRoZSBpbnRlbnRpb24g dGhhdCB0aGlzIHNob3VsZCBiZSBhbgo+Pj4+Pj4+IGludmlzaWJsZSBwZXJmb3JtYW5jZSBib29z dC4gSSB0aGluayBpdCdzIGltcG9ydGFudCB0byBzZXQgdGhlIHBvbGljeSBmb3IKPj4+Pj4+PiB1 c2Ugb2YKPj4+Pj4+Cj4+Pj4+PiBJdCB3aWxsIG5ldmVyIGV2ZXIgYmUgYSBjb21wbGV0ZWx5IGlu dmlzaWJsZSBwZXJmb3JtYW5jZSBib29zdCwganVzdCBsaWtlCj4+Pj4+PiBvcmRpbmFyeSBUSFAu Cj4+Pj4+Pgo+Pj4+Pj4gVXNpbmcgdGhlIGV4YWN0IHNhbWUgZXhpc3RpbmcgdG9nZ2xlIGlzIHRo ZSByaWdodCB0aGluZyB0byBkby4gSWYgc29tZW9uZQo+Pj4+Pj4gc3BlY2lmeSAibmV2ZXIiIG9y ICJtYWR2aXNlIiwgdGhlbiBkbyBleGFjdGx5IHRoYXQuCj4+Pj4+Pgo+Pj4+Pj4gSXQgbWlnaHQg bWFrZSBzZW5zZSB0byBoYXZlIG1vcmUgbW9kZXMgb3IgYWRkaXRpb25hbCB0b2dnbGVzLCBidXQK Pj4+Pj4+ICJtYWR2aXNlPW5ldmVyIiBtZWFucyBubyBtZW1vcnkgd2FzdGUuCj4+Pj4+Cj4+Pj4+ IEkgaGF0ZSB0aGUgZXhpc3RpbmcgbWVjaGFuaXNtcy7CoCBUaGV5IGFyZSBhbiBhYmRpY2F0aW9u IG9mIG91cgo+Pj4+PiByZXNwb25zaWJpbGl0eSwgYW5kIGFuIGF0dGVtcHQgdG8gYmxhbWUgdGhl IHVzZXIgKGJlIGl0IHRoZSBzeXNhZG1pbgo+Pj4+PiBvciB0aGUgcHJvZ3JhbW1lcikgb2Ygb3Vy IGNvZGUgZm9yIHVzaW5nIGl0IHdyb25nbHkuwqAgV2Ugc2hvdWxkIG5vdAo+Pj4+PiByZXBsaWNh dGUgdGhpcyBtaXN0YWtlLgo+Pj4+Cj4+Pj4gSSBkb24ndCBhZ3JlZSByZWdhcmRpbmcgdGhlIHBy b2dyYW1tZXIgcmVzcG9uc2liaWxpdHkuIEluIHNvbWUgY2FzZXMgdGhlCj4+Pj4gcHJvZ3JhbW1l ciByZWFsbHkgZG9lc24ndCB3YW50IHRvIGdldCBtb3JlIG1lbW9yeSBwb3B1bGF0ZWQgdGhhbiBy ZXF1ZXN0ZWQgLS0KPj4+PiBhbmQga25vd3MgZXhhY3RseSB3aHkgc2V0dGluZyBNQURWX05PSFVH RVBBR0UgaXMgdGhlIHJpZ2h0IHRoaW5nIHRvIGRvLgo+Pj4+Cj4+Pj4gUmVnYXJkaW5nIHRoZSBt YWR2aXNlPW5ldmVyL21hZHZpc2UvYWx3YXlzIChzeXMgYWRtaW4gZGVjaXNpb24pLCBtZW1vcnkg d2FzdGUKPj4+PiAoYW5kIG5haWxpbmcgZG93biBidWdzIG9yIHdvcmtpbmcgYXJvdW5kIHRoZW0g aW4gY3VzdG9tZXIgc2V0dXBzKSBoYXZlIGJlZW4gdmVyeQo+Pj4+IGdvb2QgcmVhc29ucyB0byBs ZXQgdGhlIGFkbWluIGhhdmUgYSB3b3JkLgo+Pj4+Cj4+Pj4+Cj4+Pj4+IE91ciBjb2RlIHNob3Vs ZCBiZSBhdXRvLXR1bmluZy7CoCBJIHBvc3RlZCBhIGxvbmcsIGRldGFpbGVkIG91dGxpbmUgaGVy ZToKPj4+Pj4gaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvbGludXgtbW0vWSUyRlU4YlFkMTVhVU85 N3ZTQGNhc3Blci5pbmZyYWRlYWQub3JnLwo+Pj4+Pgo+Pj4+Cj4+Pj4gV2VsbCwgImF1dG8tdHVu aW5nIiBhbHNvIHNob3VsZCBiZSBwZXJmZWN0IGZvciBldmVyeWJvZHksIGJ1dCBvbmNlIHJlYWxp dHkKPj4+PiBzdHJpa2VzIHlvdSBrbm93IGl0IGlzbid0Lgo+Pj4+Cj4+Pj4gSWYgcGVvcGxlIGRv bid0IGZlZWwgbGlrZSB1c2luZyBUSFAsIGxldCB0aGVtIGhhdmUgYSB3b3JkLiBUaGUgIm1hZHZp c2UiIGNvbmZpZwo+Pj4+IG9wdGlvbiBpcyBwcm9iYWJseSBtb3JlIGNvbnRyb3ZlcnNpYWwuIEJ1 dCB0aGUgImFsd2F5cyB2cy4gbmV2ZXIiIGFic29sdXRlbHkKPj4+PiBtYWtlcyBzZW5zZSB0byBt ZS4KPj4+Pgo+Pj4+Pj4gSSByZW1lbWJlciBJIHJhaXNlZCBpdCBhbHJlYWR5IGluIHRoZSBwYXN0 LCBidXQgeW91ICphYnNvbHV0ZWx5KiBoYXZlIHRvCj4+Pj4+PiByZXNwZWN0IHRoZSBNQURWX05P SFVHRVBBR0UgZmxhZy4gVGhlcmUgaXMgdXNlciBzcGFjZSBvdXQgdGhlcmUgKGZvcgo+Pj4+Pj4g ZXhhbXBsZSwgdXNlcmZhdWx0ZmQpIHRoYXQgZG9lc24ndCB3YW50IHRoZSBrZXJuZWwgdG8gcG9w dWxhdGUgYW55Cj4+Pj4+PiBhZGRpdGlvbmFsIHBhZ2UgdGFibGVzLiBTbyBpZiB5b3UgaGF2ZSB0 byByZXNwZWN0IHRoYXQgYWxyZWFkeSwgdGhlbiBhbHNvCj4+Pj4+PiByZXNwZWN0IE1BRFZfSFVH RVBBR0UsIHNpbXBsZS4KPj4+Pj4KPj4+Pj4gUG9zc2libHkgaGF2aW5nIHVmZmQgZW5hYmxlZCBv biBhIFZNQSBzaG91bGQgZGlzYWJsZSB1c2luZyBsYXJnZSBmb2xpb3MsCj4+Pj4KPj4+PiBUaGVy ZSBhcmUgY2FzZXMgd2hlcmUgd2UgZW5hYmxlIHVmZmQgKmFmdGVyKiBhbHJlYWR5IHRvdWNoaW5n IG1lbW9yeSAocG9zdGNvcHkKPj4+PiBsaXZlIG1pZ3JhdGlvbiBpbiBRRU1VIGJlaW5nIHRoZSBm YW1vdXMgZXhhbXBsZSkuIFRoYXQgZG9lc24ndCBmbHkuCj4+Pj4KPj4+Pj4gSSBjYW4gZ2V0IGJl aGluZCB0aGF0LsKgIEJ1dCB0aGUgbm90aW9uIHRoYXQgdXNlcnNwYWNlIGtub3dzIHdoYXQgaXQn cwo+Pj4+PiBkb2luZyAuLi4gaGFoYWhhLsKgIEp1c3QgaWdub3JlIHRoZSBtYWR2aXNlIGZsYWdz LsKgIFVzZXJzcGFjZSBkb2Vzbid0Cj4+Pj4+IGtub3cgd2hhdCBpdCdzIGRvaW5nLgo+Pj4+Cj4+ Pj4gSWYgdXNlciBzcGFjZSBzZXRzIE1BRFZfTk9IVUdFUEFHRSwgaXQgZXhhY3RseSBrbm93cyB3 aGF0IGl0IGlzIGRvaW5nIC4uLiBpbgo+Pj4+IHNvbWUgY2FzZXMuIEFuZCB0aGVzZSBpbmNsdWRl IGNhc2VzIEkgY2FyZSBhYm91dCBtZXNzaW5nIHdpdGggc3BhcnNlIFZNIG1lbW9yeSA6KQo+Pj4+ Cj4+Pj4gSSBoYXZlIHN0cm9uZyBvcGluaW9ucyBhZ2FpbnN0IHBvcHVsYXRpbmcgbW9yZSB0aGFu IHJlcXVpcmVkIHdoZW4gdXNlciBzcGFjZSBzZXQKPj4+PiBNQURWX05PSFVHRVBBR0UuCj4+Pgo+ Pj4gSSBjYW4gc2VlIHlvdXIgcG9pbnQgYWJvdXQgaG9ub3VyaW5nIE1BRFZfTk9IVUdFUEFHRSwg c28gdGhpbmsgdGhhdCBpdCBpcwo+Pj4gcmVhc29uYWJsZSB0byBmYWxsYmFjayB0byBhbGxvY2F0 aW5nIGFuIG9yZGVyLTAgcGFnZSBpbiBhIFZNQSB0aGF0IGhhcyBpdCBzZXQuCj4+PiBUaGUgYXBw IGhhcyBnb25lIG91dCBvZiBpdHMgd2F5IHRvIGV4cGxpY2l0bHkgc2V0IGl0LCBhZnRlciBhbGwu Cj4+Pgo+Pj4gSSB0aGluayB0aGUgY29ycmVjdCBiZWhhdmlvdXIgZm9yIHRoZSBnbG9iYWwgdGhw IGNvbnRyb2xzIChjbWRsaW5lIGFuZCBzeXNmcykKPj4+IGFyZSBsZXNzIG9idmlvdXMgdGhvdWdo LiBJIGNvdWxkIGdldCBvbiBib2FyZCB3aXRoIGRpc2FibGluZyBsYXJnZSBhbm9uIGZvbGlvcwo+ Pj4gZ2xvYmFsbHkgd2hlbiB0aHA9Im5ldmVyIi4gQnV0IGZvciBvdGhlciBzaXR1YXRpb25zLCBJ IHdvdWxkIHByZWZlciB0byBrZWVwCj4+PiBsYXJnZSBhbm9uIGZvbGlvcyBlbmFibGVkICh0cmVh dCAibWFkdmlzZSIgYXMgImFsd2F5cyIpLAo+PiAKPj4gSWYgd2UgaGF2ZSBzb21lIG1lY2hhbmlz bSB0byBhdXRvLXR1bmUgdGhlIGxhcmdlIGZvbGlvcyB1c2FnZSwgZm9yCj4+IGV4YW1wbGUsIGRl dGVjdCB0aGUgaW50ZXJuYWwgZnJhZ21lbnRhdGlvbiBhbmQgc3BsaXQgdGhlIGxhcmdlIGZvbGlv LAo+PiB0aGVuIHdlIGNhbiB1c2UgdGhwPSJhbHdheXMiIGFzIGRlZmF1bHQgY29uZmlndXJhdGlv bi4gIElmIG15IG1lbW9yeQo+PiB3ZXJlIGNvcnJlY3QsIHRoaXMgaXMgd2hhdCBKb2hhbm5lcyBh bmQgQWxleGFuZGVyIGlzIHdvcmtpbmcgb24uCj4KPiBDb3VsZCB5b3UgcG9pbnQgbWUgdG8gdGhh dCB3b3JrPyBJJ2QgbGlrZSB0byB1bmRlcnN0YW5kIHdoYXQgdGhlIG1lY2hhbmlzbSBpcy4KPiBU aGUgb3RoZXIgaGFsZiBvZiBteSB3b3JrIGFpbXMgdG8gdXNlIGFybTY0J3MgcHRlICJjb250aWd1 b3VzIGJpdCIgdG8gdGVsbCB0aGUKPiBIVyB0aGF0IGEgc3BhbiBvZiBQVEVzIHNoYXJlIHRoZSBz YW1lIG1hcHBpbmcgYW5kIGlzIHRoZXJlZm9yZSBjb2FsZXNjZWQgaW50byBhCj4gc2luZ2xlIFRM QiBlbnRyeS4gVGhlIHNpZGUgZWZmZWN0IG9mIHRoaXMsIGhvd2V2ZXIsIGlzIHRoYXQgd2Ugb25s eSBoYXZlIGEKPiBzaW5nbGUgYWNjZXNzIGFuZCBkaXJ0eSBiaXQgZm9yIHRoZSB3aG9sZSBjb250 cHRlIGV4dGVudC4gU28gSSdkIGxpa2UgdG8gYXZvaWQKPiBhbnkgbWVjaGFuaXNtIHRoYXQgcmVs aWVzIG9uIGdldHRpbmcgYWNjZXNzL2RpcnR5IGF0IHRoZSBiYXNlIHBhZ2UgZ3JhbnVsYXJpdHkK PiBmb3IgYSBsYXJnZSBmb2xpby4KClBsZWFzZSB0YWtlIGEgbG9vayBhdCB0aGUgVEhQIHNocmlu a2VyIHBhdGNoc2V0LAoKaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvbGludXgtbW0vY292ZXIuMTY2 NzQ1NDYxMy5naXQuYWxleGx6aHVAZmIuY29tLwoKPj4gCj4+PiB3aXRoIHRoZSBhcmd1bWVudCB0 aGF0Cj4+PiB0aGVpciBvcmRlciBpcyBtdWNoIHNtYWxsZXIgdGhhbiB0cmFkaXRpb25hbCBUSFAg YW5kIHRoZXJlZm9yZSB0aGUgaW50ZXJuYWwKPj4+IGZyYWdtZW50YXRpb24gaXMgc2lnbmlmaWNh bnRseSByZWR1Y2VkLgo+PiAKPj4gRG8geW91IGhhdmUgYW55IGRhdGEgZm9yIHRoaXM/Cj4KPiBT b21lOyBpdHMgcGFydGx5IGJhc2VkIG9uIGludHVpdGlvbiB0aGF0IHRoZSBzbWFsbGVyIHRoZSBh bGxvY2F0aW9uIHVuaXQsIHRoZQo+IHNtYWxsZXIgdGhlIGludGVybmFsIGZyYWdtZW50YXRpb24u IEFuZCBwYXJ0bHkgb24gcGVhayBtZW1vcnkgdXNhZ2UgZGF0YSBJJ3ZlCj4gY29sbGVjdGVkIGZv ciB0aGUgYmVuY2htYXJrcyBJJ20gcnVubmluZywgY29tcGFyaW5nIGJhc2VsaW5lLTRrIGtlcm5l bCB3aXRoCj4gYmFzZWxpbmUtMTZrIGFuZCBiYXNlbGluZS02NCBrZXJuZWxzIGFsb25nIHdpdGgg YSA0ayBrZXJuZWwgdGhhdCBzdXBwb3J0cyBsYXJnZQo+IGFub24gZm9saW9zIChJIGFwcHJlY2lh dGUgdGhhdCdzIG5vdCBleGFjdGx5IHdoYXQgd2UgYXJlIHRhbGtpbmcgYWJvdXQgaGVyZSwgYW5k Cj4gaXQncyBub3QgZXhhY3RseSBhbiBleHRlbnNpdmUgc2V0IG9mIHJlc3VsdHMhKToKPgo+Cj4g S2VybmVsIENvbXBsaWF0aW9uIHdpdGggOCBKb2JzOgo+IHwga2VybmVsICAgICAgICB8ICAgcGVh ayB8Cj4gfDotLS0tLS0tLS0tLS0tLXwtLS0tLS0tOnwKPiB8IGJhc2VsaW5lLTRrICAgfCAgIDAu MCUgfAo+IHwgYW5vbmZvbGlvICAgICB8ICAgMC4xJSB8Cj4gfCBiYXNlbGluZS0xNmsgIHwgICA2 LjMlIHwKPiB8IGJhc2VsaW5lLTY0ayAgfCAgMjguMSUgfAo+Cj4KPiBLZXJuZWwgQ29tcGxpYXRp b24gd2l0aCA4MCBKb2JzOgo+IHwga2VybmVsICAgICAgICB8ICAgcGVhayB8Cj4gfDotLS0tLS0t LS0tLS0tLXwtLS0tLS0tOnwKPiB8IGJhc2VsaW5lLTRrICAgfCAgIDAuMCUgfAo+IHwgYW5vbmZv bGlvICAgICB8ICAgMS43JSB8Cj4gfCBiYXNlbGluZS0xNmsgIHwgICAyLjYlIHwKPiB8IGJhc2Vs aW5lLTY0ayAgfCAgMTIuMyUgfAo+CgpXaHkgaXMgYW5vbmZvbGlvIGJldHRlciB0aGFuIGJhc2Vs aW5lLTY0ayBpZiB5b3UgYWx3YXlzIGFsbG9jYXRlIDY0awphbm9ueW1vdXMgZm9saW8/ICBCZWNh dXNlIHBhZ2UgY2FjaGUgdXNlcyA2NGsgaW4gYmFzZWxpbmUtNjRrPwoKV2UgbWF5IG5lZWQgdG8g dGVzdCBzb21lIHdvcmtsb2FkcyB3aXRoIHNwYXJzZSBhY2Nlc3MgcGF0dGVybnMgdG9vLgoKQmVz dCBSZWdhcmRzLApIdWFuZywgWWluZwoKPj4gCj4+PiBJIHJlYWxseSBkb24ndCB3YW50IHRvIGVu ZCB1cCB3aXRoIHVzZXIKPj4+IHNwYWNlIGV2ZXIgaGF2aW5nIHRvIG9wdC1pbiAod2l0aCBNQURW X0hVR0VQQUdFKSB0byBzZWUgdGhlIGJlbmVmaXRzIG9mIGxhcmdlCj4+PiBhbm9uIGZvbGlvcy4K Pj4+Cj4+PiBJIHN0aWxsIGZlZWwgdGhhdCBpdCB3b3VsZCBiZSBiZXR0ZXIgZm9yIHRoZSB0aHAg YW5kIGxhcmdlIGFub24gZm9saW8gY29udHJvbHMKPj4+IHRvIGJlIGluZGVwZW5kZW50IHRob3Vn aCAtIHdoYXQncyB0aGUgYXJndW1lbnQgZm9yIHR5aW5nIHRoZW0gdG9nZXRoZXI/Cj4+PgoKCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1r ZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpo dHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJu ZWwK