From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A26BEC433E7 for ; Mon, 19 Oct 2020 06:31:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 582282225A for ; Mon, 19 Oct 2020 06:31:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727001AbgJSGbD (ORCPT ); Mon, 19 Oct 2020 02:31:03 -0400 Received: from foss.arm.com ([217.140.110.172]:50214 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726626AbgJSGbD (ORCPT ); Mon, 19 Oct 2020 02:31:03 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 51154D6E; Sun, 18 Oct 2020 23:31:02 -0700 (PDT) Received: from [10.163.77.151] (unknown [10.163.77.151]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 918243F719; Sun, 18 Oct 2020 23:30:58 -0700 (PDT) Subject: Re: arm64: dropping prevent_bootmem_remove_notifier To: David Hildenbrand , Sudarshan Rajagopalan , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Suren Baghdasaryan , pratikp@codeaurora.org, Gavin Shan , Mark Rutland , Logan Gunthorpe , Andrew Morton , Steven Price , Muchun Song References: <630d58e0-cddc-bd39-1f2f-cb933740a133@redhat.com> From: Anshuman Khandual Message-ID: <764f2d7b-8e11-c911-a93f-6682d86d20ab@arm.com> Date: Mon, 19 Oct 2020 12:00:30 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <630d58e0-cddc-bd39-1f2f-cb933740a133@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/17/2020 03:05 PM, David Hildenbrand wrote: > On 17.10.20 01:11, Sudarshan Rajagopalan wrote: >> >> Hello Anshuman, >> > David here, > > in general, if your driver offlines+removes random memory, it is doing > something *very* wrong and dangerous. You shouldn't ever be > offlining+removing memory unless > a) you own that boot memory after boot. E.g., the ACPI driver owns DIMMs > after a reboot. > b) you added that memory via add_memory() and friends. Right. > > Even trusting that offline memory can be used by your driver is wrong. Right. > > Just imagine you racing with actual memory hot(un)plug, you'll be in > *big* trouble. For example, > > 1. You offlined memory and assume you can use it. A DIMM can simply get > unplugged. you're doomed. > 2. You offlined+removed memory and assume you can use it. A DIMM can > simply get unplugged and the whole machine would crash. > > Or imagine your driver running on a system that has virtio-mem, which > will try to remove/offline+remove memory that was added by virtio-mem/ > is under its control. > > Long story short: don't do it. > > There is *one* instance in Linux where we currently allow it for legacy > reasons. It is powernv/memtrace code that offlines+removes boot memory. > But here we are guaranteed to run in an environment (HW) without any > actual memory hot(un)plug. > > I guess you're going to say "but in our environment we don't have ..." - > this is not a valid argument to change such generic things upstream / > introducing such hacks. Agreed. > >> In the patch that enables memory hot-remove (commit bbd6ec605c0f >> ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier >> put in place that prevents boot memory from being offlined and removed. >> Also commit text mentions that boot memory on arm64 cannot be removed. >> We wanted to understand more about the reasoning for this. X86 and other >> archs doesn’t seem to do this prevention. There’s also comment in the >> code that this notifier could be dropped in future if and when boot >> memory can be removed. > > The issue is that with *actual* memory hotunplug (for what the whole > machinery should be used for), that memory/DIMM will be gone. And as you > cannot fixup the initial memmap, if you were to reboot that machine, you > would simply crash immediately. Right. > > On x86, you can have that easily: hotplug DIMMs on bare metal and > reboot. The DIMMs will be exposed via e820 during boot, so they are > "early", although if done right (movable_node, movable_core and > similar), they can get hotunplugged later. Important in environments > where you want to hotunplug whole nodes. But has HW on x86 will properly > adjust the initial memmap / e820, there is no such issue as on arm64. That is the primary problem. > >> >> The current logic is that only “new” memory blocks which are hot-added >> can later be offlined and removed. The memory that system booted up with >> cannot be offlined and removed. But there could be many usercases such >> as inter-VM memory sharing where a primary VM could offline and >> hot-remove a block/section of memory and lend it to secondary VM where >> it could hot-add it. And after usecase is done, the reverse happens > > That use case is using the wrong mechanisms. It shouldn't be > offlining+removing memory. Read below. > >> where secondary VM hot-removes and gives it back to primary which can >> hot-add it back. In such cases, the present logic for arm64 doesn’t >> allow this hot-remove in primary to happen. >> >> Also, on systems with movable zone that sort of guarantees pages to be >> migrated and isolated so that blocks can be offlined, this logic also >> defeats the purpose of having a movable zone which system can rely on >> memory hot-plugging, which say virt-io mem also relies on for fully >> plugged memory blocks. > > The MOVABLE_ZONE is *not* just for better guarantees when trying to > hotunplug memory. It also increases the number of THP/huge pages. And > that part works just fine. Right. > >> >> So we’re trying to understand the reasoning for such a prevention put in >> place for arm64 arch alone. >> >> One possible way to solve this is by marking the required sections as >> “non-early” by removing the SECTION_IS_EARLY bit in its section_mem_map. >> This puts these sections in the context of “memory hotpluggable” which >> can be offlined-removed and added-onlined which are part of boot RAM >> itself and doesn’t need any extra blocks to be hot added. This way of >> marking certain sections as “non-early” could be exported so that module >> drivers can set the required number of sections as “memory > > Oh please no. No driver should be doing that. That's just hacking around > the root issue: you're not supposed to do that. > >> hotpluggable”. This could have certain checks put in place to see which >> sections are allowed, example only movable zone sections can be marked >> as “non-early”. >> > > I assume what your use case wants to achieve is, starting VMs with > large, contiguous memory backings, not wasting memory for the memmap in > the hypervisor. > > The "traditional" way of doing that is using the "mem=" boot parameter, > and starting VMs with memory within the "never exposed to Linux" part. I suggested something similar. Memory not exposed to Linux via this method can be hot added or removed later on. > While that in general works, I consider it an ugly hack. And it doesn't > really allow the hypervisor the reuse unexposed memory. > > The obvious way for a driver to *allocate* memory (because that's what > you want to do!) is using alloc_contig_range(). I know, that there are > no guarantees. So you could be using CMA, ... but then, you still have > the memmap consuming memory in your hypervisor. > > What you could try instead is > > 1. Using hugetlbfs with huge (2MB) / gigantic (1GB) (on x86) pages for > backing your guest. > 2. To free up the memmap, you could then go into the direction proposed > by Muchun Song [1]. > > That's then a clean way for a driver to allocate/use memory without > abusing memory hot(un)plug infrastructure, minimizing the memmap > consumption. > > > [1] > https://lkml.kernel.org/r/20200915125947.26204-1-songmuchun@bytedance.com > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4F3BC433DF for ; Mon, 19 Oct 2020 06:33:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7A3FF2225A for ; Mon, 19 Oct 2020 06:33:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="qQSXe/h2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7A3FF2225A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UGJsb1AXVL05j3ijJjjGgBt2AAn5VSuZeQOo5iOJC8c=; b=qQSXe/h2RNYKWQUhcOR8hSINd 830tIegjZ3uwpQjmG+5J9uYYp68hSnvocVPDtHxaTJdPlKooCw8zBJH5H98bbGwDoqxllAIVUEPVL L/6qZeEtRT4j+Jqi0xntU617iVrfX+Xhe9K0SOiPM7sVrHTG4Oayn8bt1GGa1AI1jkFhODam6m4Z4 gVNN+/bnEY642Hppiw6qTRJTljxmnV2vyhE+9EoX/mxjR+pnl5SKnTuSRHJrFS41bKIICYfO8qO2g H7rdhU0yqLGJlRuNEXgTR3VDi8/LZqbPX6QagYl6BviBQ39R1tnH27WU60Q4oXDJekLFm4TqyriLO HgnCCEdhA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kUOhI-0001ms-95; Mon, 19 Oct 2020 06:31:08 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kUOhF-0001mS-NS for linux-arm-kernel@lists.infradead.org; Mon, 19 Oct 2020 06:31:06 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 51154D6E; Sun, 18 Oct 2020 23:31:02 -0700 (PDT) Received: from [10.163.77.151] (unknown [10.163.77.151]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 918243F719; Sun, 18 Oct 2020 23:30:58 -0700 (PDT) Subject: Re: arm64: dropping prevent_bootmem_remove_notifier To: David Hildenbrand , Sudarshan Rajagopalan , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <630d58e0-cddc-bd39-1f2f-cb933740a133@redhat.com> From: Anshuman Khandual Message-ID: <764f2d7b-8e11-c911-a93f-6682d86d20ab@arm.com> Date: Mon, 19 Oct 2020 12:00:30 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <630d58e0-cddc-bd39-1f2f-cb933740a133@redhat.com> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201019_023105_860180_86C2229F X-CRM114-Status: GOOD ( 43.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Gavin Shan , Logan Gunthorpe , Steven Price , Suren Baghdasaryan , Muchun Song , Andrew Morton , pratikp@codeaurora.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CgpPbiAxMC8xNy8yMDIwIDAzOjA1IFBNLCBEYXZpZCBIaWxkZW5icmFuZCB3cm90ZToKPiBPbiAx Ny4xMC4yMCAwMToxMSwgU3VkYXJzaGFuIFJhamFnb3BhbGFuIHdyb3RlOgo+Pgo+PiBIZWxsbyBB bnNodW1hbiwKPj4KPiBEYXZpZCBoZXJlLAo+IAo+IGluIGdlbmVyYWwsIGlmIHlvdXIgZHJpdmVy IG9mZmxpbmVzK3JlbW92ZXMgcmFuZG9tIG1lbW9yeSwgaXQgaXMgZG9pbmcKPiBzb21ldGhpbmcg KnZlcnkqIHdyb25nIGFuZCBkYW5nZXJvdXMuIFlvdSBzaG91bGRuJ3QgZXZlciBiZQo+IG9mZmxp bmluZytyZW1vdmluZyBtZW1vcnkgdW5sZXNzCj4gYSkgeW91IG93biB0aGF0IGJvb3QgbWVtb3J5 IGFmdGVyIGJvb3QuIEUuZy4sIHRoZSBBQ1BJIGRyaXZlciBvd25zIERJTU1zCj4gYWZ0ZXIgYSBy ZWJvb3QuCj4gYikgeW91IGFkZGVkIHRoYXQgbWVtb3J5IHZpYSBhZGRfbWVtb3J5KCkgYW5kIGZy aWVuZHMuCgpSaWdodC4KCj4gCj4gRXZlbiB0cnVzdGluZyB0aGF0IG9mZmxpbmUgbWVtb3J5IGNh biBiZSB1c2VkIGJ5IHlvdXIgZHJpdmVyIGlzIHdyb25nLgoKUmlnaHQuCgo+IAo+IEp1c3QgaW1h Z2luZSB5b3UgcmFjaW5nIHdpdGggYWN0dWFsIG1lbW9yeSBob3QodW4pcGx1ZywgeW91J2xsIGJl IGluCj4gKmJpZyogdHJvdWJsZS4gRm9yIGV4YW1wbGUsCj4gCj4gMS4gWW91IG9mZmxpbmVkIG1l bW9yeSBhbmQgYXNzdW1lIHlvdSBjYW4gdXNlIGl0LiBBIERJTU0gY2FuIHNpbXBseSBnZXQKPiB1 bnBsdWdnZWQuIHlvdSdyZSBkb29tZWQuCj4gMi4gWW91IG9mZmxpbmVkK3JlbW92ZWQgbWVtb3J5 IGFuZCBhc3N1bWUgeW91IGNhbiB1c2UgaXQuIEEgRElNTSBjYW4KPiBzaW1wbHkgZ2V0IHVucGx1 Z2dlZCBhbmQgdGhlIHdob2xlIG1hY2hpbmUgd291bGQgY3Jhc2guCj4gCj4gT3IgaW1hZ2luZSB5 b3VyIGRyaXZlciBydW5uaW5nIG9uIGEgc3lzdGVtIHRoYXQgaGFzIHZpcnRpby1tZW0sIHdoaWNo Cj4gd2lsbCB0cnkgdG8gcmVtb3ZlL29mZmxpbmUrcmVtb3ZlIG1lbW9yeSB0aGF0IHdhcyBhZGRl ZCBieSB2aXJ0aW8tbWVtLwo+IGlzIHVuZGVyIGl0cyBjb250cm9sLgo+IAo+IExvbmcgc3Rvcnkg c2hvcnQ6IGRvbid0IGRvIGl0Lgo+IAo+IFRoZXJlIGlzICpvbmUqIGluc3RhbmNlIGluIExpbnV4 IHdoZXJlIHdlIGN1cnJlbnRseSBhbGxvdyBpdCBmb3IgbGVnYWN5Cj4gcmVhc29ucy4gSXQgaXMg cG93ZXJudi9tZW10cmFjZSBjb2RlIHRoYXQgb2ZmbGluZXMrcmVtb3ZlcyBib290IG1lbW9yeS4K PiBCdXQgaGVyZSB3ZSBhcmUgZ3VhcmFudGVlZCB0byBydW4gaW4gYW4gZW52aXJvbm1lbnQgKEhX KSB3aXRob3V0IGFueQo+IGFjdHVhbCBtZW1vcnkgaG90KHVuKXBsdWcuCj4gCj4gSSBndWVzcyB5 b3UncmUgZ29pbmcgdG8gc2F5ICJidXQgaW4gb3VyIGVudmlyb25tZW50IHdlIGRvbid0IGhhdmUg Li4uIiAtCj4gdGhpcyBpcyBub3QgYSB2YWxpZCBhcmd1bWVudCB0byBjaGFuZ2Ugc3VjaCBnZW5l cmljIHRoaW5ncyB1cHN0cmVhbSAvCj4gaW50cm9kdWNpbmcgc3VjaCBoYWNrcy4KCkFncmVlZC4K Cj4gCj4+IEluIHRoZSBwYXRjaCB0aGF0IGVuYWJsZXMgbWVtb3J5IGhvdC1yZW1vdmUgKGNvbW1p dCBiYmQ2ZWM2MDVjMGYgCj4+ICgiYXJtNjQvbW06IEVuYWJsZSBtZW1vcnkgaG90IHJlbW92ZSIp KSBmb3IgYXJtNjQsIHRoZXJl4oCZcyBhIG5vdGlmaWVyIAo+PiBwdXQgaW4gcGxhY2UgdGhhdCBw cmV2ZW50cyBib290IG1lbW9yeSBmcm9tIGJlaW5nIG9mZmxpbmVkIGFuZCByZW1vdmVkLiAKPj4g QWxzbyBjb21taXQgdGV4dCBtZW50aW9ucyB0aGF0IGJvb3QgbWVtb3J5IG9uIGFybTY0IGNhbm5v dCBiZSByZW1vdmVkLiAKPj4gV2Ugd2FudGVkIHRvIHVuZGVyc3RhbmQgbW9yZSBhYm91dCB0aGUg cmVhc29uaW5nIGZvciB0aGlzLiBYODYgYW5kIG90aGVyIAo+PiBhcmNocyBkb2VzbuKAmXQgc2Vl bSB0byBkbyB0aGlzIHByZXZlbnRpb24uIFRoZXJl4oCZcyBhbHNvIGNvbW1lbnQgaW4gdGhlIAo+ PiBjb2RlIHRoYXQgdGhpcyBub3RpZmllciBjb3VsZCBiZSBkcm9wcGVkIGluIGZ1dHVyZSBpZiBh bmQgd2hlbiBib290IAo+PiBtZW1vcnkgY2FuIGJlIHJlbW92ZWQuCj4gCj4gVGhlIGlzc3VlIGlz IHRoYXQgd2l0aCAqYWN0dWFsKiBtZW1vcnkgaG90dW5wbHVnIChmb3Igd2hhdCB0aGUgd2hvbGUK PiBtYWNoaW5lcnkgc2hvdWxkIGJlIHVzZWQgZm9yKSwgdGhhdCBtZW1vcnkvRElNTSB3aWxsIGJl IGdvbmUuIEFuZCBhcyB5b3UKPiBjYW5ub3QgZml4dXAgdGhlIGluaXRpYWwgbWVtbWFwLCBpZiB5 b3Ugd2VyZSB0byByZWJvb3QgdGhhdCBtYWNoaW5lLCB5b3UKPiB3b3VsZCBzaW1wbHkgY3Jhc2gg aW1tZWRpYXRlbHkuCgpSaWdodC4KCj4gCj4gT24geDg2LCB5b3UgY2FuIGhhdmUgdGhhdCBlYXNp bHk6IGhvdHBsdWcgRElNTXMgb24gYmFyZSBtZXRhbCBhbmQKPiByZWJvb3QuIFRoZSBESU1NcyB3 aWxsIGJlIGV4cG9zZWQgdmlhIGU4MjAgZHVyaW5nIGJvb3QsIHNvIHRoZXkgYXJlCj4gImVhcmx5 IiwgYWx0aG91Z2ggaWYgZG9uZSByaWdodCAobW92YWJsZV9ub2RlLCBtb3ZhYmxlX2NvcmUgYW5k Cj4gc2ltaWxhciksIHRoZXkgY2FuIGdldCBob3R1bnBsdWdnZWQgbGF0ZXIuIEltcG9ydGFudCBp biBlbnZpcm9ubWVudHMKPiB3aGVyZSB5b3Ugd2FudCB0byBob3R1bnBsdWcgd2hvbGUgbm9kZXMu IEJ1dCBoYXMgSFcgb24geDg2IHdpbGwgcHJvcGVybHkKPiBhZGp1c3QgdGhlIGluaXRpYWwgbWVt bWFwIC8gZTgyMCwgdGhlcmUgaXMgbm8gc3VjaCBpc3N1ZSBhcyBvbiBhcm02NC4KClRoYXQgaXMg dGhlIHByaW1hcnkgcHJvYmxlbS4KCj4gCj4+Cj4+IFRoZSBjdXJyZW50IGxvZ2ljIGlzIHRoYXQg b25seSDigJxuZXfigJ0gbWVtb3J5IGJsb2NrcyB3aGljaCBhcmUgaG90LWFkZGVkIAo+PiBjYW4g bGF0ZXIgYmUgb2ZmbGluZWQgYW5kIHJlbW92ZWQuIFRoZSBtZW1vcnkgdGhhdCBzeXN0ZW0gYm9v dGVkIHVwIHdpdGggCj4+IGNhbm5vdCBiZSBvZmZsaW5lZCBhbmQgcmVtb3ZlZC4gQnV0IHRoZXJl IGNvdWxkIGJlIG1hbnkgdXNlcmNhc2VzIHN1Y2ggCj4+IGFzIGludGVyLVZNIG1lbW9yeSBzaGFy aW5nIHdoZXJlIGEgcHJpbWFyeSBWTSBjb3VsZCBvZmZsaW5lIGFuZCAKPj4gaG90LXJlbW92ZSBh IGJsb2NrL3NlY3Rpb24gb2YgbWVtb3J5IGFuZCBsZW5kIGl0IHRvIHNlY29uZGFyeSBWTSB3aGVy ZSAKPj4gaXQgY291bGQgaG90LWFkZCBpdC4gQW5kIGFmdGVyIHVzZWNhc2UgaXMgZG9uZSwgdGhl IHJldmVyc2UgaGFwcGVucyAKPiAKPiBUaGF0IHVzZSBjYXNlIGlzIHVzaW5nIHRoZSB3cm9uZyBt ZWNoYW5pc21zLiBJdCBzaG91bGRuJ3QgYmUKPiBvZmZsaW5pbmcrcmVtb3ZpbmcgbWVtb3J5LiBS ZWFkIGJlbG93Lgo+IAo+PiB3aGVyZSBzZWNvbmRhcnkgVk0gaG90LXJlbW92ZXMgYW5kIGdpdmVz IGl0IGJhY2sgdG8gcHJpbWFyeSB3aGljaCBjYW4gCj4+IGhvdC1hZGQgaXQgYmFjay4gSW4gc3Vj aCBjYXNlcywgdGhlIHByZXNlbnQgbG9naWMgZm9yIGFybTY0IGRvZXNu4oCZdCAKPj4gYWxsb3cg dGhpcyBob3QtcmVtb3ZlIGluIHByaW1hcnkgdG8gaGFwcGVuLgo+Pgo+PiBBbHNvLCBvbiBzeXN0 ZW1zIHdpdGggbW92YWJsZSB6b25lIHRoYXQgc29ydCBvZiBndWFyYW50ZWVzIHBhZ2VzIHRvIGJl IAo+PiBtaWdyYXRlZCBhbmQgaXNvbGF0ZWQgc28gdGhhdCBibG9ja3MgY2FuIGJlIG9mZmxpbmVk LCB0aGlzIGxvZ2ljIGFsc28gCj4+IGRlZmVhdHMgdGhlIHB1cnBvc2Ugb2YgaGF2aW5nIGEgbW92 YWJsZSB6b25lIHdoaWNoIHN5c3RlbSBjYW4gcmVseSBvbiAKPj4gbWVtb3J5IGhvdC1wbHVnZ2lu Zywgd2hpY2ggc2F5IHZpcnQtaW8gbWVtIGFsc28gcmVsaWVzIG9uIGZvciBmdWxseSAKPj4gcGx1 Z2dlZCBtZW1vcnkgYmxvY2tzLgo+IAo+IFRoZSBNT1ZBQkxFX1pPTkUgaXMgKm5vdCoganVzdCBm b3IgYmV0dGVyIGd1YXJhbnRlZXMgd2hlbiB0cnlpbmcgdG8KPiBob3R1bnBsdWcgbWVtb3J5LiBJ dCBhbHNvIGluY3JlYXNlcyB0aGUgbnVtYmVyIG9mIFRIUC9odWdlIHBhZ2VzLiBBbmQKPiB0aGF0 IHBhcnQgd29ya3MganVzdCBmaW5lLgoKUmlnaHQuCgo+IAo+Pgo+PiBTbyB3ZeKAmXJlIHRyeWlu ZyB0byB1bmRlcnN0YW5kIHRoZSByZWFzb25pbmcgZm9yIHN1Y2ggYSBwcmV2ZW50aW9uIHB1dCBp biAKPj4gcGxhY2UgZm9yIGFybTY0IGFyY2ggYWxvbmUuCj4+Cj4+IE9uZSBwb3NzaWJsZSB3YXkg dG8gc29sdmUgdGhpcyBpcyBieSBtYXJraW5nIHRoZSByZXF1aXJlZCBzZWN0aW9ucyBhcyAKPj4g 4oCcbm9uLWVhcmx54oCdIGJ5IHJlbW92aW5nIHRoZSBTRUNUSU9OX0lTX0VBUkxZIGJpdCBpbiBp dHMgc2VjdGlvbl9tZW1fbWFwLiAKPj4gVGhpcyBwdXRzIHRoZXNlIHNlY3Rpb25zIGluIHRoZSBj b250ZXh0IG9mIOKAnG1lbW9yeSBob3RwbHVnZ2FibGXigJ0gd2hpY2ggCj4+IGNhbiBiZSBvZmZs aW5lZC1yZW1vdmVkIGFuZCBhZGRlZC1vbmxpbmVkIHdoaWNoIGFyZSBwYXJ0IG9mIGJvb3QgUkFN IAo+PiBpdHNlbGYgYW5kIGRvZXNu4oCZdCBuZWVkIGFueSBleHRyYSBibG9ja3MgdG8gYmUgaG90 IGFkZGVkLiBUaGlzIHdheSBvZiAKPj4gbWFya2luZyBjZXJ0YWluIHNlY3Rpb25zIGFzIOKAnG5v bi1lYXJseeKAnSBjb3VsZCBiZSBleHBvcnRlZCBzbyB0aGF0IG1vZHVsZSAKPj4gZHJpdmVycyBj YW4gc2V0IHRoZSByZXF1aXJlZCBudW1iZXIgb2Ygc2VjdGlvbnMgYXMg4oCcbWVtb3J5IAo+IAo+ IE9oIHBsZWFzZSBuby4gTm8gZHJpdmVyIHNob3VsZCBiZSBkb2luZyB0aGF0LiBUaGF0J3MganVz dCBoYWNraW5nIGFyb3VuZAo+IHRoZSByb290IGlzc3VlOiB5b3UncmUgbm90IHN1cHBvc2VkIHRv IGRvIHRoYXQuCj4gCj4+IGhvdHBsdWdnYWJsZeKAnS4gVGhpcyBjb3VsZCBoYXZlIGNlcnRhaW4g Y2hlY2tzIHB1dCBpbiBwbGFjZSB0byBzZWUgd2hpY2ggCj4+IHNlY3Rpb25zIGFyZSBhbGxvd2Vk LCBleGFtcGxlIG9ubHkgbW92YWJsZSB6b25lIHNlY3Rpb25zIGNhbiBiZSBtYXJrZWQgCj4+IGFz IOKAnG5vbi1lYXJseeKAnS4KPj4KPiAKPiBJIGFzc3VtZSB3aGF0IHlvdXIgdXNlIGNhc2Ugd2Fu dHMgdG8gYWNoaWV2ZSBpcywgc3RhcnRpbmcgVk1zIHdpdGgKPiBsYXJnZSwgY29udGlndW91cyBt ZW1vcnkgYmFja2luZ3MsIG5vdCB3YXN0aW5nIG1lbW9yeSBmb3IgdGhlIG1lbW1hcCBpbgo+IHRo ZSBoeXBlcnZpc29yLgo+IAo+IFRoZSAidHJhZGl0aW9uYWwiIHdheSBvZiBkb2luZyB0aGF0IGlz IHVzaW5nIHRoZSAibWVtPSIgYm9vdCBwYXJhbWV0ZXIsCj4gYW5kIHN0YXJ0aW5nIFZNcyB3aXRo IG1lbW9yeSB3aXRoaW4gdGhlICJuZXZlciBleHBvc2VkIHRvIExpbnV4IiBwYXJ0LgoKSSBzdWdn ZXN0ZWQgc29tZXRoaW5nIHNpbWlsYXIuIE1lbW9yeSBub3QgZXhwb3NlZCB0byBMaW51eCB2aWEg dGhpcyBtZXRob2QKY2FuIGJlIGhvdCBhZGRlZCBvciByZW1vdmVkIGxhdGVyIG9uLgoKPiBXaGls ZSB0aGF0IGluIGdlbmVyYWwgd29ya3MsIEkgY29uc2lkZXIgaXQgYW4gdWdseSBoYWNrLiBBbmQg aXQgZG9lc24ndAo+IHJlYWxseSBhbGxvdyB0aGUgaHlwZXJ2aXNvciB0aGUgcmV1c2UgdW5leHBv c2VkIG1lbW9yeS4KPiAKPiBUaGUgb2J2aW91cyB3YXkgZm9yIGEgZHJpdmVyIHRvICphbGxvY2F0 ZSogbWVtb3J5IChiZWNhdXNlIHRoYXQncyB3aGF0Cj4geW91IHdhbnQgdG8gZG8hKSBpcyB1c2lu ZyBhbGxvY19jb250aWdfcmFuZ2UoKS4gSSBrbm93LCB0aGF0IHRoZXJlIGFyZQo+IG5vIGd1YXJh bnRlZXMuIFNvIHlvdSBjb3VsZCBiZSB1c2luZyBDTUEsIC4uLiBidXQgdGhlbiwgeW91IHN0aWxs IGhhdmUKPiB0aGUgbWVtbWFwIGNvbnN1bWluZyBtZW1vcnkgaW4geW91ciBoeXBlcnZpc29yLgo+ IAo+IFdoYXQgeW91IGNvdWxkIHRyeSBpbnN0ZWFkIGlzCj4gCj4gMS4gVXNpbmcgaHVnZXRsYmZz IHdpdGggaHVnZSAoMk1CKSAvIGdpZ2FudGljICgxR0IpIChvbiB4ODYpIHBhZ2VzIGZvcgo+IGJh Y2tpbmcgeW91ciBndWVzdC4KPiAyLiBUbyBmcmVlIHVwIHRoZSBtZW1tYXAsIHlvdSBjb3VsZCB0 aGVuIGdvIGludG8gdGhlIGRpcmVjdGlvbiBwcm9wb3NlZAo+IGJ5IE11Y2h1biBTb25nIFsxXS4K PiAKPiBUaGF0J3MgdGhlbiBhIGNsZWFuIHdheSBmb3IgYSBkcml2ZXIgdG8gYWxsb2NhdGUvdXNl IG1lbW9yeSB3aXRob3V0Cj4gYWJ1c2luZyBtZW1vcnkgaG90KHVuKXBsdWcgaW5mcmFzdHJ1Y3R1 cmUsIG1pbmltaXppbmcgdGhlIG1lbW1hcAo+IGNvbnN1bXB0aW9uLgo+IAo+IAo+IFsxXQo+IGh0 dHBzOi8vbGttbC5rZXJuZWwub3JnL3IvMjAyMDA5MTUxMjU5NDcuMjYyMDQtMS1zb25nbXVjaHVu QGJ5dGVkYW5jZS5jb20KPiAKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxA bGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xp c3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK