From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D2C4C433E7 for ; Sat, 17 Oct 2020 09:35:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF41C2074A for ; Sat, 17 Oct 2020 09:35:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="P7Mw4N70" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2437793AbgJQJfT (ORCPT ); Sat, 17 Oct 2020 05:35:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:44005 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2406377AbgJQJfS (ORCPT ); Sat, 17 Oct 2020 05:35:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1602927316; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9/htpQ+v+VSa/IpeLT5PQ55xosbr/PLDzK95J2wOkMY=; b=P7Mw4N70hrBaLS/1C3qDhVolN5t2C3qifU5n3Er415G07SMXIzdmhS2ua1f09CxjMFwGOn Iq3yECkNdiJ/Gbwr3tpP+Jnvxrd8ynIWEIIfSBfRwN1ZOBB801KJbOY4J1H8I8FzKQ9rdq 6fyQIOLvmVhb2juhXTb6W7i1CPsCPqg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-571-sJD8wFJWM0qZdR5kwaQyfg-1; Sat, 17 Oct 2020 05:35:14 -0400 X-MC-Unique: sJD8wFJWM0qZdR5kwaQyfg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 858BF185B384; Sat, 17 Oct 2020 09:35:10 +0000 (UTC) Received: from [10.36.112.50] (ovpn-112-50.ams2.redhat.com [10.36.112.50]) by smtp.corp.redhat.com (Postfix) with ESMTP id 34A6027BB3; Sat, 17 Oct 2020 09:35:05 +0000 (UTC) Subject: Re: arm64: dropping prevent_bootmem_remove_notifier To: Sudarshan Rajagopalan , Anshuman Khandual , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Suren Baghdasaryan , pratikp@codeaurora.org, Gavin Shan , Mark Rutland , Logan Gunthorpe , Andrew Morton , Steven Price , Muchun Song References: From: David Hildenbrand Organization: Red Hat GmbH Message-ID: <630d58e0-cddc-bd39-1f2f-cb933740a133@redhat.com> Date: Sat, 17 Oct 2020 11:35:05 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.3.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 17.10.20 01:11, Sudarshan Rajagopalan wrote: > > Hello Anshuman, > David here, in general, if your driver offlines+removes random memory, it is doing something *very* wrong and dangerous. You shouldn't ever be offlining+removing memory unless a) you own that boot memory after boot. E.g., the ACPI driver owns DIMMs after a reboot. b) you added that memory via add_memory() and friends. Even trusting that offline memory can be used by your driver is wrong. Just imagine you racing with actual memory hot(un)plug, you'll be in *big* trouble. For example, 1. You offlined memory and assume you can use it. A DIMM can simply get unplugged. you're doomed. 2. You offlined+removed memory and assume you can use it. A DIMM can simply get unplugged and the whole machine would crash. Or imagine your driver running on a system that has virtio-mem, which will try to remove/offline+remove memory that was added by virtio-mem/ is under its control. Long story short: don't do it. There is *one* instance in Linux where we currently allow it for legacy reasons. It is powernv/memtrace code that offlines+removes boot memory. But here we are guaranteed to run in an environment (HW) without any actual memory hot(un)plug. I guess you're going to say "but in our environment we don't have ..." - this is not a valid argument to change such generic things upstream / introducing such hacks. > In the patch that enables memory hot-remove (commit bbd6ec605c0f > ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier > put in place that prevents boot memory from being offlined and removed. > Also commit text mentions that boot memory on arm64 cannot be removed. > We wanted to understand more about the reasoning for this. X86 and other > archs doesn’t seem to do this prevention. There’s also comment in the > code that this notifier could be dropped in future if and when boot > memory can be removed. The issue is that with *actual* memory hotunplug (for what the whole machinery should be used for), that memory/DIMM will be gone. And as you cannot fixup the initial memmap, if you were to reboot that machine, you would simply crash immediately. On x86, you can have that easily: hotplug DIMMs on bare metal and reboot. The DIMMs will be exposed via e820 during boot, so they are "early", although if done right (movable_node, movable_core and similar), they can get hotunplugged later. Important in environments where you want to hotunplug whole nodes. But has HW on x86 will properly adjust the initial memmap / e820, there is no such issue as on arm64. > > The current logic is that only “new” memory blocks which are hot-added > can later be offlined and removed. The memory that system booted up with > cannot be offlined and removed. But there could be many usercases such > as inter-VM memory sharing where a primary VM could offline and > hot-remove a block/section of memory and lend it to secondary VM where > it could hot-add it. And after usecase is done, the reverse happens That use case is using the wrong mechanisms. It shouldn't be offlining+removing memory. Read below. > where secondary VM hot-removes and gives it back to primary which can > hot-add it back. In such cases, the present logic for arm64 doesn’t > allow this hot-remove in primary to happen. > > Also, on systems with movable zone that sort of guarantees pages to be > migrated and isolated so that blocks can be offlined, this logic also > defeats the purpose of having a movable zone which system can rely on > memory hot-plugging, which say virt-io mem also relies on for fully > plugged memory blocks. The MOVABLE_ZONE is *not* just for better guarantees when trying to hotunplug memory. It also increases the number of THP/huge pages. And that part works just fine. > > So we’re trying to understand the reasoning for such a prevention put in > place for arm64 arch alone. > > One possible way to solve this is by marking the required sections as > “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. > This puts these sections in the context of “memory hotpluggable” which > can be offlined-removed and added-onlined which are part of boot RAM > itself and doesn’t need any extra blocks to be hot added. This way of > marking certain sections as “non-early” could be exported so that module > drivers can set the required number of sections as “memory Oh please no. No driver should be doing that. That's just hacking around the root issue: you're not supposed to do that. > hotpluggable”. This could have certain checks put in place to see which > sections are allowed, example only movable zone sections can be marked > as “non-early”. > I assume what your use case wants to achieve is, starting VMs with large, contiguous memory backings, not wasting memory for the memmap in the hypervisor. The "traditional" way of doing that is using the "mem=" boot parameter, and starting VMs with memory within the "never exposed to Linux" part. While that in general works, I consider it an ugly hack. And it doesn't really allow the hypervisor the reuse unexposed memory. The obvious way for a driver to *allocate* memory (because that's what you want to do!) is using alloc_contig_range(). I know, that there are no guarantees. So you could be using CMA, ... but then, you still have the memmap consuming memory in your hypervisor. What you could try instead is 1. Using hugetlbfs with huge (2MB) / gigantic (1GB) (on x86) pages for backing your guest. 2. To free up the memmap, you could then go into the direction proposed by Muchun Song [1]. That's then a clean way for a driver to allocate/use memory without abusing memory hot(un)plug infrastructure, minimizing the memmap consumption. [1] https://lkml.kernel.org/r/20200915125947.26204-1-songmuchun@bytedance.com -- Thanks, David / dhildenb From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FF7CC433DF for ; Sat, 17 Oct 2020 09:37:34 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C919020720 for ; Sat, 17 Oct 2020 09:37:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="S+dyuALe"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FTrUrVzG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C919020720 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=lU+ScINbElS+IcRDsBz+ALJuDDMsA91DBoMngcaLCQI=; b=S+dyuALeTBUZQ8EWuGweohFyD xmjz6PQ7DF24/p/vtlEsBuEWEsWIPCrAbOTXmpqEwen4Vuo3IuDvJuXhFedTlK4gkvPfgq39C9SHZ pcy4w+UmbDfrDEVNSfCtRN9ZjWHz4AD6Dqo+ZXgoL1bYQePpUnRM4AUYNTG1D/u8gf0ppvgCf41pE cz05ZXEaod8ruuQWExH/LoCYgwm2gzQjDhlRIFmJF82eEe1p3Y9rXaZkp7/Nii6uI1mot3jYcgqMo KJRbnKY9ZcsX4gBOO/3Fg9RL6Gzpp5yaB2gzT5SQZr5HTdWIeuxuIGUeAbnVIQ2ELNGTT705vI2Qy 8bWviqw8w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kTicZ-0008Ao-0R; Sat, 17 Oct 2020 09:35:27 +0000 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kTicS-0008AR-CQ for linux-arm-kernel@lists.infradead.org; Sat, 17 Oct 2020 09:35:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1602927318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9/htpQ+v+VSa/IpeLT5PQ55xosbr/PLDzK95J2wOkMY=; b=FTrUrVzGaBMO1ZMtAdazzqZ8vMSL6bzbfdx7pS3AQFDobUskJWoxMJKMhFQpSxcvXlGjQT ht4pWe/vv4iSr3akvW3NImaOYvBYCw+cI0ZMug5grDhQEpnASgKwMTY2TWFInqspmQcccN vBRlYFM0E8mvozer9u5Rfn6tZEr2cy8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-571-sJD8wFJWM0qZdR5kwaQyfg-1; Sat, 17 Oct 2020 05:35:14 -0400 X-MC-Unique: sJD8wFJWM0qZdR5kwaQyfg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 858BF185B384; Sat, 17 Oct 2020 09:35:10 +0000 (UTC) Received: from [10.36.112.50] (ovpn-112-50.ams2.redhat.com [10.36.112.50]) by smtp.corp.redhat.com (Postfix) with ESMTP id 34A6027BB3; Sat, 17 Oct 2020 09:35:05 +0000 (UTC) Subject: Re: arm64: dropping prevent_bootmem_remove_notifier To: Sudarshan Rajagopalan , Anshuman Khandual , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: From: David Hildenbrand Organization: Red Hat GmbH Message-ID: <630d58e0-cddc-bd39-1f2f-cb933740a133@redhat.com> Date: Sat, 17 Oct 2020 11:35:05 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.3.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201017_053520_561583_92BED461 X-CRM114-Status: GOOD ( 35.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Gavin Shan , Logan Gunthorpe , Steven Price , Suren Baghdasaryan , Muchun Song , Andrew Morton , pratikp@codeaurora.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMTcuMTAuMjAgMDE6MTEsIFN1ZGFyc2hhbiBSYWphZ29wYWxhbiB3cm90ZToKPiAKPiBIZWxs byBBbnNodW1hbiwKPiAKRGF2aWQgaGVyZSwKCmluIGdlbmVyYWwsIGlmIHlvdXIgZHJpdmVyIG9m ZmxpbmVzK3JlbW92ZXMgcmFuZG9tIG1lbW9yeSwgaXQgaXMgZG9pbmcKc29tZXRoaW5nICp2ZXJ5 KiB3cm9uZyBhbmQgZGFuZ2Vyb3VzLiBZb3Ugc2hvdWxkbid0IGV2ZXIgYmUKb2ZmbGluaW5nK3Jl bW92aW5nIG1lbW9yeSB1bmxlc3MKYSkgeW91IG93biB0aGF0IGJvb3QgbWVtb3J5IGFmdGVyIGJv b3QuIEUuZy4sIHRoZSBBQ1BJIGRyaXZlciBvd25zIERJTU1zCmFmdGVyIGEgcmVib290LgpiKSB5 b3UgYWRkZWQgdGhhdCBtZW1vcnkgdmlhIGFkZF9tZW1vcnkoKSBhbmQgZnJpZW5kcy4KCkV2ZW4g dHJ1c3RpbmcgdGhhdCBvZmZsaW5lIG1lbW9yeSBjYW4gYmUgdXNlZCBieSB5b3VyIGRyaXZlciBp cyB3cm9uZy4KCkp1c3QgaW1hZ2luZSB5b3UgcmFjaW5nIHdpdGggYWN0dWFsIG1lbW9yeSBob3Qo dW4pcGx1ZywgeW91J2xsIGJlIGluCipiaWcqIHRyb3VibGUuIEZvciBleGFtcGxlLAoKMS4gWW91 IG9mZmxpbmVkIG1lbW9yeSBhbmQgYXNzdW1lIHlvdSBjYW4gdXNlIGl0LiBBIERJTU0gY2FuIHNp bXBseSBnZXQKdW5wbHVnZ2VkLiB5b3UncmUgZG9vbWVkLgoyLiBZb3Ugb2ZmbGluZWQrcmVtb3Zl ZCBtZW1vcnkgYW5kIGFzc3VtZSB5b3UgY2FuIHVzZSBpdC4gQSBESU1NIGNhbgpzaW1wbHkgZ2V0 IHVucGx1Z2dlZCBhbmQgdGhlIHdob2xlIG1hY2hpbmUgd291bGQgY3Jhc2guCgpPciBpbWFnaW5l IHlvdXIgZHJpdmVyIHJ1bm5pbmcgb24gYSBzeXN0ZW0gdGhhdCBoYXMgdmlydGlvLW1lbSwgd2hp Y2gKd2lsbCB0cnkgdG8gcmVtb3ZlL29mZmxpbmUrcmVtb3ZlIG1lbW9yeSB0aGF0IHdhcyBhZGRl ZCBieSB2aXJ0aW8tbWVtLwppcyB1bmRlciBpdHMgY29udHJvbC4KCkxvbmcgc3Rvcnkgc2hvcnQ6 IGRvbid0IGRvIGl0LgoKVGhlcmUgaXMgKm9uZSogaW5zdGFuY2UgaW4gTGludXggd2hlcmUgd2Ug Y3VycmVudGx5IGFsbG93IGl0IGZvciBsZWdhY3kKcmVhc29ucy4gSXQgaXMgcG93ZXJudi9tZW10 cmFjZSBjb2RlIHRoYXQgb2ZmbGluZXMrcmVtb3ZlcyBib290IG1lbW9yeS4KQnV0IGhlcmUgd2Ug YXJlIGd1YXJhbnRlZWQgdG8gcnVuIGluIGFuIGVudmlyb25tZW50IChIVykgd2l0aG91dCBhbnkK YWN0dWFsIG1lbW9yeSBob3QodW4pcGx1Zy4KCkkgZ3Vlc3MgeW91J3JlIGdvaW5nIHRvIHNheSAi YnV0IGluIG91ciBlbnZpcm9ubWVudCB3ZSBkb24ndCBoYXZlIC4uLiIgLQp0aGlzIGlzIG5vdCBh IHZhbGlkIGFyZ3VtZW50IHRvIGNoYW5nZSBzdWNoIGdlbmVyaWMgdGhpbmdzIHVwc3RyZWFtIC8K aW50cm9kdWNpbmcgc3VjaCBoYWNrcy4KCj4gSW4gdGhlIHBhdGNoIHRoYXQgZW5hYmxlcyBtZW1v cnkgaG90LXJlbW92ZSAoY29tbWl0IGJiZDZlYzYwNWMwZiAKPiAoImFybTY0L21tOiBFbmFibGUg bWVtb3J5IGhvdCByZW1vdmUiKSkgZm9yIGFybTY0LCB0aGVyZeKAmXMgYSBub3RpZmllciAKPiBw dXQgaW4gcGxhY2UgdGhhdCBwcmV2ZW50cyBib290IG1lbW9yeSBmcm9tIGJlaW5nIG9mZmxpbmVk IGFuZCByZW1vdmVkLiAKPiBBbHNvIGNvbW1pdCB0ZXh0IG1lbnRpb25zIHRoYXQgYm9vdCBtZW1v cnkgb24gYXJtNjQgY2Fubm90IGJlIHJlbW92ZWQuIAo+IFdlIHdhbnRlZCB0byB1bmRlcnN0YW5k IG1vcmUgYWJvdXQgdGhlIHJlYXNvbmluZyBmb3IgdGhpcy4gWDg2IGFuZCBvdGhlciAKPiBhcmNo cyBkb2VzbuKAmXQgc2VlbSB0byBkbyB0aGlzIHByZXZlbnRpb24uIFRoZXJl4oCZcyBhbHNvIGNv bW1lbnQgaW4gdGhlIAo+IGNvZGUgdGhhdCB0aGlzIG5vdGlmaWVyIGNvdWxkIGJlIGRyb3BwZWQg aW4gZnV0dXJlIGlmIGFuZCB3aGVuIGJvb3QgCj4gbWVtb3J5IGNhbiBiZSByZW1vdmVkLgoKVGhl IGlzc3VlIGlzIHRoYXQgd2l0aCAqYWN0dWFsKiBtZW1vcnkgaG90dW5wbHVnIChmb3Igd2hhdCB0 aGUgd2hvbGUKbWFjaGluZXJ5IHNob3VsZCBiZSB1c2VkIGZvciksIHRoYXQgbWVtb3J5L0RJTU0g d2lsbCBiZSBnb25lLiBBbmQgYXMgeW91CmNhbm5vdCBmaXh1cCB0aGUgaW5pdGlhbCBtZW1tYXAs IGlmIHlvdSB3ZXJlIHRvIHJlYm9vdCB0aGF0IG1hY2hpbmUsIHlvdQp3b3VsZCBzaW1wbHkgY3Jh c2ggaW1tZWRpYXRlbHkuCgpPbiB4ODYsIHlvdSBjYW4gaGF2ZSB0aGF0IGVhc2lseTogaG90cGx1 ZyBESU1NcyBvbiBiYXJlIG1ldGFsIGFuZApyZWJvb3QuIFRoZSBESU1NcyB3aWxsIGJlIGV4cG9z ZWQgdmlhIGU4MjAgZHVyaW5nIGJvb3QsIHNvIHRoZXkgYXJlCiJlYXJseSIsIGFsdGhvdWdoIGlm IGRvbmUgcmlnaHQgKG1vdmFibGVfbm9kZSwgbW92YWJsZV9jb3JlIGFuZApzaW1pbGFyKSwgdGhl eSBjYW4gZ2V0IGhvdHVucGx1Z2dlZCBsYXRlci4gSW1wb3J0YW50IGluIGVudmlyb25tZW50cwp3 aGVyZSB5b3Ugd2FudCB0byBob3R1bnBsdWcgd2hvbGUgbm9kZXMuIEJ1dCBoYXMgSFcgb24geDg2 IHdpbGwgcHJvcGVybHkKYWRqdXN0IHRoZSBpbml0aWFsIG1lbW1hcCAvIGU4MjAsIHRoZXJlIGlz IG5vIHN1Y2ggaXNzdWUgYXMgb24gYXJtNjQuCgo+IAo+IFRoZSBjdXJyZW50IGxvZ2ljIGlzIHRo YXQgb25seSDigJxuZXfigJ0gbWVtb3J5IGJsb2NrcyB3aGljaCBhcmUgaG90LWFkZGVkIAo+IGNh biBsYXRlciBiZSBvZmZsaW5lZCBhbmQgcmVtb3ZlZC4gVGhlIG1lbW9yeSB0aGF0IHN5c3RlbSBi b290ZWQgdXAgd2l0aCAKPiBjYW5ub3QgYmUgb2ZmbGluZWQgYW5kIHJlbW92ZWQuIEJ1dCB0aGVy ZSBjb3VsZCBiZSBtYW55IHVzZXJjYXNlcyBzdWNoIAo+IGFzIGludGVyLVZNIG1lbW9yeSBzaGFy aW5nIHdoZXJlIGEgcHJpbWFyeSBWTSBjb3VsZCBvZmZsaW5lIGFuZCAKPiBob3QtcmVtb3ZlIGEg YmxvY2svc2VjdGlvbiBvZiBtZW1vcnkgYW5kIGxlbmQgaXQgdG8gc2Vjb25kYXJ5IFZNIHdoZXJl IAo+IGl0IGNvdWxkIGhvdC1hZGQgaXQuIEFuZCBhZnRlciB1c2VjYXNlIGlzIGRvbmUsIHRoZSBy ZXZlcnNlIGhhcHBlbnMgCgpUaGF0IHVzZSBjYXNlIGlzIHVzaW5nIHRoZSB3cm9uZyBtZWNoYW5p c21zLiBJdCBzaG91bGRuJ3QgYmUKb2ZmbGluaW5nK3JlbW92aW5nIG1lbW9yeS4gUmVhZCBiZWxv dy4KCj4gd2hlcmUgc2Vjb25kYXJ5IFZNIGhvdC1yZW1vdmVzIGFuZCBnaXZlcyBpdCBiYWNrIHRv IHByaW1hcnkgd2hpY2ggY2FuIAo+IGhvdC1hZGQgaXQgYmFjay4gSW4gc3VjaCBjYXNlcywgdGhl IHByZXNlbnQgbG9naWMgZm9yIGFybTY0IGRvZXNu4oCZdCAKPiBhbGxvdyB0aGlzIGhvdC1yZW1v dmUgaW4gcHJpbWFyeSB0byBoYXBwZW4uCj4gCj4gQWxzbywgb24gc3lzdGVtcyB3aXRoIG1vdmFi bGUgem9uZSB0aGF0IHNvcnQgb2YgZ3VhcmFudGVlcyBwYWdlcyB0byBiZSAKPiBtaWdyYXRlZCBh bmQgaXNvbGF0ZWQgc28gdGhhdCBibG9ja3MgY2FuIGJlIG9mZmxpbmVkLCB0aGlzIGxvZ2ljIGFs c28gCj4gZGVmZWF0cyB0aGUgcHVycG9zZSBvZiBoYXZpbmcgYSBtb3ZhYmxlIHpvbmUgd2hpY2gg c3lzdGVtIGNhbiByZWx5IG9uIAo+IG1lbW9yeSBob3QtcGx1Z2dpbmcsIHdoaWNoIHNheSB2aXJ0 LWlvIG1lbSBhbHNvIHJlbGllcyBvbiBmb3IgZnVsbHkgCj4gcGx1Z2dlZCBtZW1vcnkgYmxvY2tz LgoKVGhlIE1PVkFCTEVfWk9ORSBpcyAqbm90KiBqdXN0IGZvciBiZXR0ZXIgZ3VhcmFudGVlcyB3 aGVuIHRyeWluZyB0bwpob3R1bnBsdWcgbWVtb3J5LiBJdCBhbHNvIGluY3JlYXNlcyB0aGUgbnVt YmVyIG9mIFRIUC9odWdlIHBhZ2VzLiBBbmQKdGhhdCBwYXJ0IHdvcmtzIGp1c3QgZmluZS4KCj4g Cj4gU28gd2XigJlyZSB0cnlpbmcgdG8gdW5kZXJzdGFuZCB0aGUgcmVhc29uaW5nIGZvciBzdWNo IGEgcHJldmVudGlvbiBwdXQgaW4gCj4gcGxhY2UgZm9yIGFybTY0IGFyY2ggYWxvbmUuCj4gCj4g T25lIHBvc3NpYmxlIHdheSB0byBzb2x2ZSB0aGlzIGlzIGJ5IG1hcmtpbmcgdGhlIHJlcXVpcmVk IHNlY3Rpb25zIGFzIAo+IOKAnG5vbi1lYXJseeKAnSBieSByZW1vdmluZyB0aGUgU0VDVElPTl9J U19FQVJMWSBiaXQgaW4gaXRzIHNlY3Rpb25fbWVtX21hcC4gCj4gVGhpcyBwdXRzIHRoZXNlIHNl Y3Rpb25zIGluIHRoZSBjb250ZXh0IG9mIOKAnG1lbW9yeSBob3RwbHVnZ2FibGXigJ0gd2hpY2gg Cj4gY2FuIGJlIG9mZmxpbmVkLXJlbW92ZWQgYW5kIGFkZGVkLW9ubGluZWQgd2hpY2ggYXJlIHBh cnQgb2YgYm9vdCBSQU0gCj4gaXRzZWxmIGFuZCBkb2VzbuKAmXQgbmVlZCBhbnkgZXh0cmEgYmxv Y2tzIHRvIGJlIGhvdCBhZGRlZC4gVGhpcyB3YXkgb2YgCj4gbWFya2luZyBjZXJ0YWluIHNlY3Rp b25zIGFzIOKAnG5vbi1lYXJseeKAnSBjb3VsZCBiZSBleHBvcnRlZCBzbyB0aGF0IG1vZHVsZSAK PiBkcml2ZXJzIGNhbiBzZXQgdGhlIHJlcXVpcmVkIG51bWJlciBvZiBzZWN0aW9ucyBhcyDigJxt ZW1vcnkgCgpPaCBwbGVhc2Ugbm8uIE5vIGRyaXZlciBzaG91bGQgYmUgZG9pbmcgdGhhdC4gVGhh dCdzIGp1c3QgaGFja2luZyBhcm91bmQKdGhlIHJvb3QgaXNzdWU6IHlvdSdyZSBub3Qgc3VwcG9z ZWQgdG8gZG8gdGhhdC4KCj4gaG90cGx1Z2dhYmxl4oCdLiBUaGlzIGNvdWxkIGhhdmUgY2VydGFp biBjaGVja3MgcHV0IGluIHBsYWNlIHRvIHNlZSB3aGljaCAKPiBzZWN0aW9ucyBhcmUgYWxsb3dl ZCwgZXhhbXBsZSBvbmx5IG1vdmFibGUgem9uZSBzZWN0aW9ucyBjYW4gYmUgbWFya2VkIAo+IGFz IOKAnG5vbi1lYXJseeKAnS4KPiAKCkkgYXNzdW1lIHdoYXQgeW91ciB1c2UgY2FzZSB3YW50cyB0 byBhY2hpZXZlIGlzLCBzdGFydGluZyBWTXMgd2l0aApsYXJnZSwgY29udGlndW91cyBtZW1vcnkg YmFja2luZ3MsIG5vdCB3YXN0aW5nIG1lbW9yeSBmb3IgdGhlIG1lbW1hcCBpbgp0aGUgaHlwZXJ2 aXNvci4KClRoZSAidHJhZGl0aW9uYWwiIHdheSBvZiBkb2luZyB0aGF0IGlzIHVzaW5nIHRoZSAi bWVtPSIgYm9vdCBwYXJhbWV0ZXIsCmFuZCBzdGFydGluZyBWTXMgd2l0aCBtZW1vcnkgd2l0aGlu IHRoZSAibmV2ZXIgZXhwb3NlZCB0byBMaW51eCIgcGFydC4KV2hpbGUgdGhhdCBpbiBnZW5lcmFs IHdvcmtzLCBJIGNvbnNpZGVyIGl0IGFuIHVnbHkgaGFjay4gQW5kIGl0IGRvZXNuJ3QKcmVhbGx5 IGFsbG93IHRoZSBoeXBlcnZpc29yIHRoZSByZXVzZSB1bmV4cG9zZWQgbWVtb3J5LgoKVGhlIG9i dmlvdXMgd2F5IGZvciBhIGRyaXZlciB0byAqYWxsb2NhdGUqIG1lbW9yeSAoYmVjYXVzZSB0aGF0 J3Mgd2hhdAp5b3Ugd2FudCB0byBkbyEpIGlzIHVzaW5nIGFsbG9jX2NvbnRpZ19yYW5nZSgpLiBJ IGtub3csIHRoYXQgdGhlcmUgYXJlCm5vIGd1YXJhbnRlZXMuIFNvIHlvdSBjb3VsZCBiZSB1c2lu ZyBDTUEsIC4uLiBidXQgdGhlbiwgeW91IHN0aWxsIGhhdmUKdGhlIG1lbW1hcCBjb25zdW1pbmcg bWVtb3J5IGluIHlvdXIgaHlwZXJ2aXNvci4KCldoYXQgeW91IGNvdWxkIHRyeSBpbnN0ZWFkIGlz CgoxLiBVc2luZyBodWdldGxiZnMgd2l0aCBodWdlICgyTUIpIC8gZ2lnYW50aWMgKDFHQikgKG9u IHg4NikgcGFnZXMgZm9yCmJhY2tpbmcgeW91ciBndWVzdC4KMi4gVG8gZnJlZSB1cCB0aGUgbWVt bWFwLCB5b3UgY291bGQgdGhlbiBnbyBpbnRvIHRoZSBkaXJlY3Rpb24gcHJvcG9zZWQKYnkgTXVj aHVuIFNvbmcgWzFdLgoKVGhhdCdzIHRoZW4gYSBjbGVhbiB3YXkgZm9yIGEgZHJpdmVyIHRvIGFs bG9jYXRlL3VzZSBtZW1vcnkgd2l0aG91dAphYnVzaW5nIG1lbW9yeSBob3QodW4pcGx1ZyBpbmZy YXN0cnVjdHVyZSwgbWluaW1pemluZyB0aGUgbWVtbWFwCmNvbnN1bXB0aW9uLgoKClsxXQpodHRw czovL2xrbWwua2VybmVsLm9yZy9yLzIwMjAwOTE1MTI1OTQ3LjI2MjA0LTEtc29uZ211Y2h1bkBi eXRlZGFuY2UuY29tCgotLSAKVGhhbmtzLAoKRGF2aWQgLyBkaGlsZGVuYgoKCl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFp bGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlz dHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK