From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86F93EB64D9 for ; Fri, 7 Jul 2023 16:07:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232778AbjGGQHN (ORCPT ); Fri, 7 Jul 2023 12:07:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232729AbjGGQHL (ORCPT ); Fri, 7 Jul 2023 12:07:11 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28B921BF4 for ; Fri, 7 Jul 2023 09:06:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688745985; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kwPjc8vcrtj61iKL+PFdekq3QvvaLbWA6Qsa4S/CLJA=; b=bqcB+LYL5KjUnIegbgjvcmL+uRkgMMlu+08OGAq3jdds2J4fyxMvUCaiuJTLmQjV5b0tbL y6eGVT76b9+gdYhb5TSsL4QeQqAZIEC9dfNQgdg6RYpJX/x1lG1tcJ4MPOH/SO9Yd+uUuY 7bH+RJqr+JIe2+1JmrgJlykxVip6VEA= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-147-H1gmjeaKMByG-OSWQHGW5A-1; Fri, 07 Jul 2023 12:06:24 -0400 X-MC-Unique: H1gmjeaKMByG-OSWQHGW5A-1 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-314326f6e23so1043671f8f.2 for ; Fri, 07 Jul 2023 09:06:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688745983; x=1691337983; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kwPjc8vcrtj61iKL+PFdekq3QvvaLbWA6Qsa4S/CLJA=; b=JdJR0PimfsPfAlUJh3K6GDhguP62qKQjhMxqCLUVl1VOIv/dL8FfHDdZxfrn1+GTl0 K5BTBLhdlKD5vqN8wq4j7Im2b/bRzDWKDorVT++ufByTUEFWspyQvgaKc2tJnz6dJ7WI JEMYeLiQChHgnILofvEvw7L4eVYmoID31ilTbjEJO9n0dYbGTYylBJDPKhaMfwk6cdGk 8lxxVn2/XI48tby9YXBv2BdjR4s+tPNYLPMcI6Vrl8t/TTf9fWfcqCnsOj5btk4szbJk TIjyToJTrBeGUjrzQPj2iuUQCqvje250KGdiBDk5nN/gipEym4bgmqNlPRc0yBi0BFDI xAXQ== X-Gm-Message-State: ABy/qLYmWIKg0gboxOlHpvLauvz9DwKBapceDyX/E2zMo282W9UlvkSb fxnKbGv5HuW0NwpNqgjA0DfHdv7jEUkTr1/ul3SbjPE1IrKDHa8DDYvextRzh+1siO4290mogM8 1BUNw2mdYP8jgrD4o/LCCvKRk X-Received: by 2002:a5d:58c5:0:b0:314:5f6f:68ce with SMTP id o5-20020a5d58c5000000b003145f6f68cemr3020442wrf.66.1688745982877; Fri, 07 Jul 2023 09:06:22 -0700 (PDT) X-Google-Smtp-Source: APBJJlH5cYvipoXxcLVPWrgX1VBfttdwQWGQsXJzVcd1hL47gcf10idF/3InhZjed94f6o29TxN3Jg== X-Received: by 2002:a5d:58c5:0:b0:314:5f6f:68ce with SMTP id o5-20020a5d58c5000000b003145f6f68cemr3020411wrf.66.1688745982463; Fri, 07 Jul 2023 09:06:22 -0700 (PDT) Received: from ?IPV6:2003:d8:2f04:3c00:248f:bf5b:b03e:aac7? (p200300d82f043c00248fbf5bb03eaac7.dip0.t-ipconnect.de. [2003:d8:2f04:3c00:248f:bf5b:b03e:aac7]) by smtp.gmail.com with ESMTPSA id v8-20020a5d5908000000b0031437ec7ec1sm4858220wrd.2.2023.07.07.09.06.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 07 Jul 2023 09:06:21 -0700 (PDT) Message-ID: Date: Fri, 7 Jul 2023 18:06:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Content-Language: en-US To: Ryan Roberts , Matthew Wilcox Cc: "Huang, Ying" , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance In-Reply-To: <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07.07.23 17:13, Ryan Roberts wrote: > On 07/07/2023 15:07, David Hildenbrand wrote: >> On 07.07.23 15:57, Matthew Wilcox wrote: >>> On Fri, Jul 07, 2023 at 01:29:02PM +0200, David Hildenbrand wrote: >>>> On 07.07.23 11:52, Ryan Roberts wrote: >>>>> On 07/07/2023 09:01, Huang, Ying wrote: >>>>>> Although we can use smaller page order for FLEXIBLE_THP, it's hard to >>>>>> avoid internal fragmentation completely.  So, I think that finally we >>>>>> will need to provide a mechanism for the users to opt out, e.g., >>>>>> something like "always madvise never" via >>>>>> /sys/kernel/mm/transparent_hugepage/enabled.  I'm not sure whether it's >>>>>> a good idea to reuse the existing interface of THP. >>>>> >>>>> I wouldn't want to tie this to the existing interface, simply because that >>>>> implies that we would want to follow the "always" and "madvise" advice too; >>>>> That >>>>> means that on a thp=madvise system (which is certainly the case for android and >>>>> other client systems) we would have to disable large anon folios for VMAs that >>>>> haven't explicitly opted in. That breaks the intention that this should be an >>>>> invisible performance boost. I think it's important to set the policy for >>>>> use of >>>> >>>> It will never ever be a completely invisible performance boost, just like >>>> ordinary THP. >>>> >>>> Using the exact same existing toggle is the right thing to do. If someone >>>> specify "never" or "madvise", then do exactly that. >>>> >>>> It might make sense to have more modes or additional toggles, but >>>> "madvise=never" means no memory waste. >>> >>> I hate the existing mechanisms.  They are an abdication of our >>> responsibility, and an attempt to blame the user (be it the sysadmin >>> or the programmer) of our code for using it wrongly.  We should not >>> replicate this mistake. >> >> I don't agree regarding the programmer responsibility. In some cases the >> programmer really doesn't want to get more memory populated than requested -- >> and knows exactly why setting MADV_NOHUGEPAGE is the right thing to do. >> >> Regarding the madvise=never/madvise/always (sys admin decision), memory waste >> (and nailing down bugs or working around them in customer setups) have been very >> good reasons to let the admin have a word. >> >>> >>> Our code should be auto-tuning.  I posted a long, detailed outline here: >>> https://lore.kernel.org/linux-mm/Y%2FU8bQd15aUO97vS@casper.infradead.org/ >>> >> >> Well, "auto-tuning" also should be perfect for everybody, but once reality >> strikes you know it isn't. >> >> If people don't feel like using THP, let them have a word. The "madvise" config >> option is probably more controversial. But the "always vs. never" absolutely >> makes sense to me. >> >>>> I remember I raised it already in the past, but you *absolutely* have to >>>> respect the MADV_NOHUGEPAGE flag. There is user space out there (for >>>> example, userfaultfd) that doesn't want the kernel to populate any >>>> additional page tables. So if you have to respect that already, then also >>>> respect MADV_HUGEPAGE, simple. >>> >>> Possibly having uffd enabled on a VMA should disable using large folios, >> >> There are cases where we enable uffd *after* already touching memory (postcopy >> live migration in QEMU being the famous example). That doesn't fly. >> >>> I can get behind that.  But the notion that userspace knows what it's >>> doing ... hahaha.  Just ignore the madvise flags.  Userspace doesn't >>> know what it's doing. >> >> If user space sets MADV_NOHUGEPAGE, it exactly knows what it is doing ... in >> some cases. And these include cases I care about messing with sparse VM memory :) >> >> I have strong opinions against populating more than required when user space set >> MADV_NOHUGEPAGE. > > I can see your point about honouring MADV_NOHUGEPAGE, so think that it is > reasonable to fallback to allocating an order-0 page in a VMA that has it set. > The app has gone out of its way to explicitly set it, after all. > > I think the correct behaviour for the global thp controls (cmdline and sysfs) > are less obvious though. I could get on board with disabling large anon folios > globally when thp="never". But for other situations, I would prefer to keep > large anon folios enabled (treat "madvise" as "always"), with the argument that > their order is much smaller than traditional THP and therefore the internal > fragmentation is significantly reduced. I really don't want to end up with user > space ever having to opt-in (with MADV_HUGEPAGE) to see the benefits of large > anon folios. I was briefly playing with a nasty idea of an additional "madvise-pmd" option (that could be the new default), that would use PMD THP only in madvise'd regions, and ordinary everywhere else. But let's disregard that for now. I think there is a bigger issue (below). > > I still feel that it would be better for the thp and large anon folio controls > to be independent though - what's the argument for tying them together? Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD THP on aarch64 (4k kernel), how are they any different? Just the way they are mapped ... It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls, but how is "2MiB vs. 2 MiB" different? Having that said, I think we have to make up our mind how much control we want to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not trivial: memory waste is a real issue on some systems where we limit THP to madvise(). Just throwing it out for discussing: What about keeping the "all / madvise / never" semantics (and MADV_NOHUGEPAGE ...) but having an additional config knob that specifies in which cases we *still* allow flexible THP even though the system was configured for "madvise". I can't come up with a good name for that, but something like "max_auto_size=64k" could be something reasonable to set. We could have an arch+hw specific default. (we all hate config options, I know, but I think there are good reasons to have such bare-minimum ones) -- Cheers, David / dhildenb From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8A73DEB64D9 for ; Fri, 7 Jul 2023 16:07:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:Subject:From:References:Cc:To: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hPc2jjRjS2b1/xigBVK7G8vM5mvl/OU7UyuRmDq2iR4=; b=0W8StBkvL6uG5i Rx5vX794W3FBoceYIjhgMiTCQJ7m2qzkXuQg8EzgW/b0D9Z2lr8gW6bjyA+1UsCAXAzC7A6RISIGx hqoqTj8WZS0wKweoFX3MdvcpBR/iMuhWJDa9tZuatcBc+WbbvJS9edHcULEQHT2dQn9odlTOC/yJY NhvYDL8YaAe+xOfazCe4hbk3NdKoDJT9YRT1Y+a8mYoMiaLQxUR+5zAgFG0xfCIzVCzoZKDyvTTBG RPYG8FUfhuUr6pw3l0DH6IhGgVIpnkS/bpi7A7/vYtEgGu9/2qMHP+GytGPXz38GfoC3SdAf8N7oD GxGsnzfep7gh9e9N366Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qHnyb-0058mo-1u; Fri, 07 Jul 2023 16:06:33 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qHnyY-0058lV-0U for linux-arm-kernel@lists.infradead.org; Fri, 07 Jul 2023 16:06:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688745985; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kwPjc8vcrtj61iKL+PFdekq3QvvaLbWA6Qsa4S/CLJA=; b=bqcB+LYL5KjUnIegbgjvcmL+uRkgMMlu+08OGAq3jdds2J4fyxMvUCaiuJTLmQjV5b0tbL y6eGVT76b9+gdYhb5TSsL4QeQqAZIEC9dfNQgdg6RYpJX/x1lG1tcJ4MPOH/SO9Yd+uUuY 7bH+RJqr+JIe2+1JmrgJlykxVip6VEA= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-589-x91lFtFZO3OJ-hkQMc-Deg-1; Fri, 07 Jul 2023 12:06:24 -0400 X-MC-Unique: x91lFtFZO3OJ-hkQMc-Deg-1 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-3fbefe1b402so11823965e9.1 for ; Fri, 07 Jul 2023 09:06:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688745983; x=1691337983; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kwPjc8vcrtj61iKL+PFdekq3QvvaLbWA6Qsa4S/CLJA=; b=ThTvNl0eDmDBRBoUF4UxtAySpg2im2UeiL3Div7R4nXYlJ3WXY1zo+JG90SwbYCICX mzaviJeAO9SdH3wn7+8uzwSG3ojG34bjjSJkLscUbzb3NhxrDWwfZgs5Vo0NOWaYKIrZ DmL67QSOUMwyObVuwuzlA8uigs7BvB/bXf/nBUP8p/wLLHEFod/i+fiDz7PTIjpiRMvv yfTXH5m0VqNUL13emKgAvZeTyRB9E+IHeDsuGSeU0n5/KZerNdT978o9lgOkfKMLii1R bfA9nfKAzWN9BVK9ZzbahrTQeuLt/H03QHzxF9RN/ggZqNLcpyujORtIjl9ScmeD5qmE ih2Q== X-Gm-Message-State: ABy/qLYMOY5htWqESZKVhrmy/3do5pfcrzGmvxiR0Ljj7OWXW7aM18Ub CkRf1noLl3J/nGiVflKLU/FtIzzeO/bzjhPfVgV2GcE8y3ADrA+zrrRL7tlaNjPddZMh91PqhSr wZOY6gfxz3RfVQ2V5g7V/BuE9heEuGZgY4JQ= X-Received: by 2002:a5d:58c5:0:b0:314:5f6f:68ce with SMTP id o5-20020a5d58c5000000b003145f6f68cemr3020448wrf.66.1688745982878; Fri, 07 Jul 2023 09:06:22 -0700 (PDT) X-Google-Smtp-Source: APBJJlH5cYvipoXxcLVPWrgX1VBfttdwQWGQsXJzVcd1hL47gcf10idF/3InhZjed94f6o29TxN3Jg== X-Received: by 2002:a5d:58c5:0:b0:314:5f6f:68ce with SMTP id o5-20020a5d58c5000000b003145f6f68cemr3020411wrf.66.1688745982463; Fri, 07 Jul 2023 09:06:22 -0700 (PDT) Received: from ?IPV6:2003:d8:2f04:3c00:248f:bf5b:b03e:aac7? (p200300d82f043c00248fbf5bb03eaac7.dip0.t-ipconnect.de. [2003:d8:2f04:3c00:248f:bf5b:b03e:aac7]) by smtp.gmail.com with ESMTPSA id v8-20020a5d5908000000b0031437ec7ec1sm4858220wrd.2.2023.07.07.09.06.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 07 Jul 2023 09:06:21 -0700 (PDT) Message-ID: Date: Fri, 7 Jul 2023 18:06:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: Ryan Roberts , Matthew Wilcox Cc: "Huang, Ying" , Andrew Morton , "Kirill A. Shutemov" , Yin Fengwei , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230703135330.1865927-1-ryan.roberts@arm.com> <20230703135330.1865927-5-ryan.roberts@arm.com> <87edlkgnfa.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e60630-5e9d-c8df-ab79-cb0767de680e@arm.com> <524bacd2-4a47-2b8b-6685-c46e31a01631@redhat.com> <1e406f04-78ef-6573-e1f1-f0d0e0d5246a@redhat.com> <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance In-Reply-To: <9dd036a8-9ba3-0cc4-b791-cb3178237728@arm.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230707_090630_271649_1EC5819F X-CRM114-Status: GOOD ( 41.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMDcuMDcuMjMgMTc6MTMsIFJ5YW4gUm9iZXJ0cyB3cm90ZToKPiBPbiAwNy8wNy8yMDIzIDE1 OjA3LCBEYXZpZCBIaWxkZW5icmFuZCB3cm90ZToKPj4gT24gMDcuMDcuMjMgMTU6NTcsIE1hdHRo ZXcgV2lsY294IHdyb3RlOgo+Pj4gT24gRnJpLCBKdWwgMDcsIDIwMjMgYXQgMDE6Mjk6MDJQTSAr MDIwMCwgRGF2aWQgSGlsZGVuYnJhbmQgd3JvdGU6Cj4+Pj4gT24gMDcuMDcuMjMgMTE6NTIsIFJ5 YW4gUm9iZXJ0cyB3cm90ZToKPj4+Pj4gT24gMDcvMDcvMjAyMyAwOTowMSwgSHVhbmcsIFlpbmcg d3JvdGU6Cj4+Pj4+PiBBbHRob3VnaCB3ZSBjYW4gdXNlIHNtYWxsZXIgcGFnZSBvcmRlciBmb3Ig RkxFWElCTEVfVEhQLCBpdCdzIGhhcmQgdG8KPj4+Pj4+IGF2b2lkIGludGVybmFsIGZyYWdtZW50 YXRpb24gY29tcGxldGVseS7CoCBTbywgSSB0aGluayB0aGF0IGZpbmFsbHkgd2UKPj4+Pj4+IHdp bGwgbmVlZCB0byBwcm92aWRlIGEgbWVjaGFuaXNtIGZvciB0aGUgdXNlcnMgdG8gb3B0IG91dCwg ZS5nLiwKPj4+Pj4+IHNvbWV0aGluZyBsaWtlICJhbHdheXMgbWFkdmlzZSBuZXZlciIgdmlhCj4+ Pj4+PiAvc3lzL2tlcm5lbC9tbS90cmFuc3BhcmVudF9odWdlcGFnZS9lbmFibGVkLsKgIEknbSBu b3Qgc3VyZSB3aGV0aGVyIGl0J3MKPj4+Pj4+IGEgZ29vZCBpZGVhIHRvIHJldXNlIHRoZSBleGlz dGluZyBpbnRlcmZhY2Ugb2YgVEhQLgo+Pj4+Pgo+Pj4+PiBJIHdvdWxkbid0IHdhbnQgdG8gdGll IHRoaXMgdG8gdGhlIGV4aXN0aW5nIGludGVyZmFjZSwgc2ltcGx5IGJlY2F1c2UgdGhhdAo+Pj4+ PiBpbXBsaWVzIHRoYXQgd2Ugd291bGQgd2FudCB0byBmb2xsb3cgdGhlICJhbHdheXMiIGFuZCAi bWFkdmlzZSIgYWR2aWNlIHRvbzsKPj4+Pj4gVGhhdAo+Pj4+PiBtZWFucyB0aGF0IG9uIGEgdGhw PW1hZHZpc2Ugc3lzdGVtICh3aGljaCBpcyBjZXJ0YWlubHkgdGhlIGNhc2UgZm9yIGFuZHJvaWQg YW5kCj4+Pj4+IG90aGVyIGNsaWVudCBzeXN0ZW1zKSB3ZSB3b3VsZCBoYXZlIHRvIGRpc2FibGUg bGFyZ2UgYW5vbiBmb2xpb3MgZm9yIFZNQXMgdGhhdAo+Pj4+PiBoYXZlbid0IGV4cGxpY2l0bHkg b3B0ZWQgaW4uIFRoYXQgYnJlYWtzIHRoZSBpbnRlbnRpb24gdGhhdCB0aGlzIHNob3VsZCBiZSBh bgo+Pj4+PiBpbnZpc2libGUgcGVyZm9ybWFuY2UgYm9vc3QuIEkgdGhpbmsgaXQncyBpbXBvcnRh bnQgdG8gc2V0IHRoZSBwb2xpY3kgZm9yCj4+Pj4+IHVzZSBvZgo+Pj4+Cj4+Pj4gSXQgd2lsbCBu ZXZlciBldmVyIGJlIGEgY29tcGxldGVseSBpbnZpc2libGUgcGVyZm9ybWFuY2UgYm9vc3QsIGp1 c3QgbGlrZQo+Pj4+IG9yZGluYXJ5IFRIUC4KPj4+Pgo+Pj4+IFVzaW5nIHRoZSBleGFjdCBzYW1l IGV4aXN0aW5nIHRvZ2dsZSBpcyB0aGUgcmlnaHQgdGhpbmcgdG8gZG8uIElmIHNvbWVvbmUKPj4+ PiBzcGVjaWZ5ICJuZXZlciIgb3IgIm1hZHZpc2UiLCB0aGVuIGRvIGV4YWN0bHkgdGhhdC4KPj4+ Pgo+Pj4+IEl0IG1pZ2h0IG1ha2Ugc2Vuc2UgdG8gaGF2ZSBtb3JlIG1vZGVzIG9yIGFkZGl0aW9u YWwgdG9nZ2xlcywgYnV0Cj4+Pj4gIm1hZHZpc2U9bmV2ZXIiIG1lYW5zIG5vIG1lbW9yeSB3YXN0 ZS4KPj4+Cj4+PiBJIGhhdGUgdGhlIGV4aXN0aW5nIG1lY2hhbmlzbXMuwqAgVGhleSBhcmUgYW4g YWJkaWNhdGlvbiBvZiBvdXIKPj4+IHJlc3BvbnNpYmlsaXR5LCBhbmQgYW4gYXR0ZW1wdCB0byBi bGFtZSB0aGUgdXNlciAoYmUgaXQgdGhlIHN5c2FkbWluCj4+PiBvciB0aGUgcHJvZ3JhbW1lcikg b2Ygb3VyIGNvZGUgZm9yIHVzaW5nIGl0IHdyb25nbHkuwqAgV2Ugc2hvdWxkIG5vdAo+Pj4gcmVw bGljYXRlIHRoaXMgbWlzdGFrZS4KPj4KPj4gSSBkb24ndCBhZ3JlZSByZWdhcmRpbmcgdGhlIHBy b2dyYW1tZXIgcmVzcG9uc2liaWxpdHkuIEluIHNvbWUgY2FzZXMgdGhlCj4+IHByb2dyYW1tZXIg cmVhbGx5IGRvZXNuJ3Qgd2FudCB0byBnZXQgbW9yZSBtZW1vcnkgcG9wdWxhdGVkIHRoYW4gcmVx dWVzdGVkIC0tCj4+IGFuZCBrbm93cyBleGFjdGx5IHdoeSBzZXR0aW5nIE1BRFZfTk9IVUdFUEFH RSBpcyB0aGUgcmlnaHQgdGhpbmcgdG8gZG8uCj4+Cj4+IFJlZ2FyZGluZyB0aGUgbWFkdmlzZT1u ZXZlci9tYWR2aXNlL2Fsd2F5cyAoc3lzIGFkbWluIGRlY2lzaW9uKSwgbWVtb3J5IHdhc3RlCj4+ IChhbmQgbmFpbGluZyBkb3duIGJ1Z3Mgb3Igd29ya2luZyBhcm91bmQgdGhlbSBpbiBjdXN0b21l ciBzZXR1cHMpIGhhdmUgYmVlbiB2ZXJ5Cj4+IGdvb2QgcmVhc29ucyB0byBsZXQgdGhlIGFkbWlu IGhhdmUgYSB3b3JkLgo+Pgo+Pj4KPj4+IE91ciBjb2RlIHNob3VsZCBiZSBhdXRvLXR1bmluZy7C oCBJIHBvc3RlZCBhIGxvbmcsIGRldGFpbGVkIG91dGxpbmUgaGVyZToKPj4+IGh0dHBzOi8vbG9y ZS5rZXJuZWwub3JnL2xpbnV4LW1tL1klMkZVOGJRZDE1YVVPOTd2U0BjYXNwZXIuaW5mcmFkZWFk Lm9yZy8KPj4+Cj4+Cj4+IFdlbGwsICJhdXRvLXR1bmluZyIgYWxzbyBzaG91bGQgYmUgcGVyZmVj dCBmb3IgZXZlcnlib2R5LCBidXQgb25jZSByZWFsaXR5Cj4+IHN0cmlrZXMgeW91IGtub3cgaXQg aXNuJ3QuCj4+Cj4+IElmIHBlb3BsZSBkb24ndCBmZWVsIGxpa2UgdXNpbmcgVEhQLCBsZXQgdGhl bSBoYXZlIGEgd29yZC4gVGhlICJtYWR2aXNlIiBjb25maWcKPj4gb3B0aW9uIGlzIHByb2JhYmx5 IG1vcmUgY29udHJvdmVyc2lhbC4gQnV0IHRoZSAiYWx3YXlzIHZzLiBuZXZlciIgYWJzb2x1dGVs eQo+PiBtYWtlcyBzZW5zZSB0byBtZS4KPj4KPj4+PiBJIHJlbWVtYmVyIEkgcmFpc2VkIGl0IGFs cmVhZHkgaW4gdGhlIHBhc3QsIGJ1dCB5b3UgKmFic29sdXRlbHkqIGhhdmUgdG8KPj4+PiByZXNw ZWN0IHRoZSBNQURWX05PSFVHRVBBR0UgZmxhZy4gVGhlcmUgaXMgdXNlciBzcGFjZSBvdXQgdGhl cmUgKGZvcgo+Pj4+IGV4YW1wbGUsIHVzZXJmYXVsdGZkKSB0aGF0IGRvZXNuJ3Qgd2FudCB0aGUg a2VybmVsIHRvIHBvcHVsYXRlIGFueQo+Pj4+IGFkZGl0aW9uYWwgcGFnZSB0YWJsZXMuIFNvIGlm IHlvdSBoYXZlIHRvIHJlc3BlY3QgdGhhdCBhbHJlYWR5LCB0aGVuIGFsc28KPj4+PiByZXNwZWN0 IE1BRFZfSFVHRVBBR0UsIHNpbXBsZS4KPj4+Cj4+PiBQb3NzaWJseSBoYXZpbmcgdWZmZCBlbmFi bGVkIG9uIGEgVk1BIHNob3VsZCBkaXNhYmxlIHVzaW5nIGxhcmdlIGZvbGlvcywKPj4KPj4gVGhl cmUgYXJlIGNhc2VzIHdoZXJlIHdlIGVuYWJsZSB1ZmZkICphZnRlciogYWxyZWFkeSB0b3VjaGlu ZyBtZW1vcnkgKHBvc3Rjb3B5Cj4+IGxpdmUgbWlncmF0aW9uIGluIFFFTVUgYmVpbmcgdGhlIGZh bW91cyBleGFtcGxlKS4gVGhhdCBkb2Vzbid0IGZseS4KPj4KPj4+IEkgY2FuIGdldCBiZWhpbmQg dGhhdC7CoCBCdXQgdGhlIG5vdGlvbiB0aGF0IHVzZXJzcGFjZSBrbm93cyB3aGF0IGl0J3MKPj4+ IGRvaW5nIC4uLiBoYWhhaGEuwqAgSnVzdCBpZ25vcmUgdGhlIG1hZHZpc2UgZmxhZ3MuwqAgVXNl cnNwYWNlIGRvZXNuJ3QKPj4+IGtub3cgd2hhdCBpdCdzIGRvaW5nLgo+Pgo+PiBJZiB1c2VyIHNw YWNlIHNldHMgTUFEVl9OT0hVR0VQQUdFLCBpdCBleGFjdGx5IGtub3dzIHdoYXQgaXQgaXMgZG9p bmcgLi4uIGluCj4+IHNvbWUgY2FzZXMuIEFuZCB0aGVzZSBpbmNsdWRlIGNhc2VzIEkgY2FyZSBh Ym91dCBtZXNzaW5nIHdpdGggc3BhcnNlIFZNIG1lbW9yeSA6KQo+Pgo+PiBJIGhhdmUgc3Ryb25n IG9waW5pb25zIGFnYWluc3QgcG9wdWxhdGluZyBtb3JlIHRoYW4gcmVxdWlyZWQgd2hlbiB1c2Vy IHNwYWNlIHNldAo+PiBNQURWX05PSFVHRVBBR0UuCj4gCj4gSSBjYW4gc2VlIHlvdXIgcG9pbnQg YWJvdXQgaG9ub3VyaW5nIE1BRFZfTk9IVUdFUEFHRSwgc28gdGhpbmsgdGhhdCBpdCBpcwo+IHJl YXNvbmFibGUgdG8gZmFsbGJhY2sgdG8gYWxsb2NhdGluZyBhbiBvcmRlci0wIHBhZ2UgaW4gYSBW TUEgdGhhdCBoYXMgaXQgc2V0Lgo+IFRoZSBhcHAgaGFzIGdvbmUgb3V0IG9mIGl0cyB3YXkgdG8g ZXhwbGljaXRseSBzZXQgaXQsIGFmdGVyIGFsbC4KPiAKPiBJIHRoaW5rIHRoZSBjb3JyZWN0IGJl aGF2aW91ciBmb3IgdGhlIGdsb2JhbCB0aHAgY29udHJvbHMgKGNtZGxpbmUgYW5kIHN5c2ZzKQo+ IGFyZSBsZXNzIG9idmlvdXMgdGhvdWdoLiBJIGNvdWxkIGdldCBvbiBib2FyZCB3aXRoIGRpc2Fi bGluZyBsYXJnZSBhbm9uIGZvbGlvcwo+IGdsb2JhbGx5IHdoZW4gdGhwPSJuZXZlciIuIEJ1dCBm b3Igb3RoZXIgc2l0dWF0aW9ucywgSSB3b3VsZCBwcmVmZXIgdG8ga2VlcAo+IGxhcmdlIGFub24g Zm9saW9zIGVuYWJsZWQgKHRyZWF0ICJtYWR2aXNlIiBhcyAiYWx3YXlzIiksIHdpdGggdGhlIGFy Z3VtZW50IHRoYXQKPiB0aGVpciBvcmRlciBpcyBtdWNoIHNtYWxsZXIgdGhhbiB0cmFkaXRpb25h bCBUSFAgYW5kIHRoZXJlZm9yZSB0aGUgaW50ZXJuYWwKPiBmcmFnbWVudGF0aW9uIGlzIHNpZ25p ZmljYW50bHkgcmVkdWNlZC4gSSByZWFsbHkgZG9uJ3Qgd2FudCB0byBlbmQgdXAgd2l0aCB1c2Vy Cj4gc3BhY2UgZXZlciBoYXZpbmcgdG8gb3B0LWluICh3aXRoIE1BRFZfSFVHRVBBR0UpIHRvIHNl ZSB0aGUgYmVuZWZpdHMgb2YgbGFyZ2UKPiBhbm9uIGZvbGlvcy4KCkkgd2FzIGJyaWVmbHkgcGxh eWluZyB3aXRoIGEgbmFzdHkgaWRlYSBvZiBhbiBhZGRpdGlvbmFsICJtYWR2aXNlLXBtZCIgCm9w dGlvbiAodGhhdCBjb3VsZCBiZSB0aGUgbmV3IGRlZmF1bHQpLCB0aGF0IHdvdWxkIHVzZSBQTUQg VEhQIG9ubHkgaW4gCm1hZHZpc2UnZCByZWdpb25zLCBhbmQgb3JkaW5hcnkgZXZlcnl3aGVyZSBl bHNlLiBCdXQgbGV0J3MgZGlzcmVnYXJkIAp0aGF0IGZvciBub3cuIEkgdGhpbmsgdGhlcmUgaXMg YSBiaWdnZXIgaXNzdWUgKGJlbG93KS4KCj4gCj4gSSBzdGlsbCBmZWVsIHRoYXQgaXQgd291bGQg YmUgYmV0dGVyIGZvciB0aGUgdGhwIGFuZCBsYXJnZSBhbm9uIGZvbGlvIGNvbnRyb2xzCj4gdG8g YmUgaW5kZXBlbmRlbnQgdGhvdWdoIC0gd2hhdCdzIHRoZSBhcmd1bWVudCBmb3IgdHlpbmcgdGhl bSB0b2dldGhlcj8KClRoaW5raW5nIGFib3V0IGRlc2lyZWQgMiBNaUIgZmxleGlibGUgVEhQIG9u IGFhcmNoNjQgKDY0ayBrZXJuZWwpIHZzLCAyIApNaUIgUE1EIFRIUCBvbiBhYXJjaDY0ICg0ayBr ZXJuZWwpLCBob3cgYXJlIHRoZXkgYW55IGRpZmZlcmVudD8gSnVzdCB0aGUgCndheSB0aGV5IGFy ZSBtYXBwZWQgLi4uCgpJdCdzIGVhc3kgdG8gc2F5ICI2NGsgdnMuIDIgTWlCIiBpcyBhIGRpZmZl cmVuY2UgYW5kIHdlIHdhbnQgc2VwYXJhdGUgCmNvbnRyb2xzLCBidXQgaG93IGlzICIyTWlCIHZz LiAyIE1pQiIgZGlmZmVyZW50PwoKSGF2aW5nIHRoYXQgc2FpZCwgSSB0aGluayB3ZSBoYXZlIHRv IG1ha2UgdXAgb3VyIG1pbmQgaG93IG11Y2ggY29udHJvbCAKd2Ugd2FudCB0byBnaXZlIHVzZXIg c3BhY2UuIEFnYWluLCB0aGUgIjJNaUIgdnMuIDIgTWlCIiBjYXNlIG5pY2VseSAKc2hvd3MgdGhh dCBpdCdzIG5vdCB0cml2aWFsOiBtZW1vcnkgd2FzdGUgaXMgYSByZWFsIGlzc3VlIG9uIHNvbWUg CnN5c3RlbXMgd2hlcmUgd2UgbGltaXQgVEhQIHRvIG1hZHZpc2UoKS4KCgpKdXN0IHRocm93aW5n IGl0IG91dCBmb3IgZGlzY3Vzc2luZzoKCldoYXQgYWJvdXQga2VlcGluZyB0aGUgImFsbCAvIG1h ZHZpc2UgLyBuZXZlciIgc2VtYW50aWNzIChhbmQgCk1BRFZfTk9IVUdFUEFHRSAuLi4pIGJ1dCBo YXZpbmcgYW4gYWRkaXRpb25hbCBjb25maWcga25vYiB0aGF0IHNwZWNpZmllcyAKaW4gd2hpY2gg Y2FzZXMgd2UgKnN0aWxsKiBhbGxvdyBmbGV4aWJsZSBUSFAgZXZlbiB0aG91Z2ggdGhlIHN5c3Rl bSB3YXMgCmNvbmZpZ3VyZWQgZm9yICJtYWR2aXNlIi4KCkkgY2FuJ3QgY29tZSB1cCB3aXRoIGEg Z29vZCBuYW1lIGZvciB0aGF0LCBidXQgc29tZXRoaW5nIGxpa2UgCiJtYXhfYXV0b19zaXplPTY0 ayIgY291bGQgYmUgc29tZXRoaW5nIHJlYXNvbmFibGUgdG8gc2V0LiBXZSBjb3VsZCBoYXZlIAph biBhcmNoK2h3IHNwZWNpZmljIGRlZmF1bHQuCgood2UgYWxsIGhhdGUgY29uZmlnIG9wdGlvbnMs IEkga25vdywgYnV0IEkgdGhpbmsgdGhlcmUgYXJlIGdvb2QgcmVhc29ucyAKdG8gaGF2ZSBzdWNo IGJhcmUtbWluaW11bSBvbmVzKQoKLS0gCkNoZWVycywKCkRhdmlkIC8gZGhpbGRlbmIKCgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2Vy bmVsIG1haWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0 cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVs Cg==