From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E33DEC2D0A3 for ; Thu, 29 Oct 2020 21:03:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 72B8120790 for ; Thu, 29 Oct 2020 21:03:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=mg.codeaurora.org header.i=@mg.codeaurora.org header.b="BV/v+ieE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726435AbgJ2VDi (ORCPT ); Thu, 29 Oct 2020 17:03:38 -0400 Received: from m42-4.mailgun.net ([69.72.42.4]:41931 "EHLO m42-4.mailgun.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726090AbgJ2VDi (ORCPT ); Thu, 29 Oct 2020 17:03:38 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1604005416; h=Message-ID: References: In-Reply-To: Subject: Cc: To: From: Date: Content-Transfer-Encoding: Content-Type: MIME-Version: Sender; bh=wrMdI+4RaPfZPeWzQXW90cx5Z685EYq4/9tTEJkTtvc=; b=BV/v+ieETU2Vzk06dQb/3sEQ7d0pb7V1KkfPiQArBoSpE5JpJVhfs3vZZkMwmrQPUncmf6v2 d3NIlooPMuBb5AzY+Q7WCWf1/Daa+BcXrqBXWRgIoBu/apkZs6kQ6t5UvtGmu9PohiAu9G+i 8wl8jJvssUg8KAByPpZv5osAe/s= X-Mailgun-Sending-Ip: 69.72.42.4 X-Mailgun-Sid: WyI0MWYwYSIsICJsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnIiwgImJlOWU0YSJd Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n05.prod.us-east-1.postgun.com with SMTP id 5f9b2dd3dec7e4448e2748fc (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Thu, 29 Oct 2020 21:02:11 GMT Sender: sudaraja=codeaurora.org@mg.codeaurora.org Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 0A3D4C43382; Thu, 29 Oct 2020 21:02:11 +0000 (UTC) Received: from mail.codeaurora.org (localhost.localdomain [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: sudaraja) by smtp.codeaurora.org (Postfix) with ESMTPSA id 9A0C5C433CB; Thu, 29 Oct 2020 21:02:09 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Thu, 29 Oct 2020 14:02:09 -0700 From: Sudarshan Rajagopalan To: Anshuman Khandual Cc: Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Suren Baghdasaryan , pratikp@codeaurora.org, Gavin Shan , Mark Rutland , Logan Gunthorpe , David Hildenbrand , Andrew Morton , Steven Price Subject: Re: arm64: dropping prevent_bootmem_remove_notifier In-Reply-To: References: Message-ID: <57952529a81c6d5ba2521c203dfc54b5@codeaurora.org> X-Sender: sudaraja@codeaurora.org User-Agent: Roundcube Webmail/1.3.9 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Anshuman, David, Thanks for all the detailed explanations for the reasoning to have bootmem protected from being removed. Also, I do agree drivers being able to mark memory sections isn't the right thing to do. We went ahead with the approach of using "mem=" as you suggested to limit the bootmem and add remaining blocks using add_memory_driver_managed() so that driver has ownership of these blocks. We do have some follow-up questions regarding this - will initiate a discussion soon. On 2020-10-18 22:37, Anshuman Khandual wrote: > Hello Sudarshan, > > On 10/17/2020 04:41 AM, Sudarshan Rajagopalan wrote: >> >> Hello Anshuman, >> >> In the patch that enables memory hot-remove (commit bbd6ec605c0f >> ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier >> put in place that prevents boot memory from being offlined and >> removed. Also commit text mentions that boot memory on arm64 cannot be >> removed. We wanted to understand more about the reasoning for this. >> X86 and other archs doesn’t seem to do this prevention. There’s also >> comment in the code that this notifier could be dropped in future if >> and when boot memory can be removed. > > Right and till then the notifier cannot be dropped. There was a lot of > discussions > around this topic during multiple iterations of memory hot remove > series. Hence, I > would just request you to please go through them first. This list here > is from one > such series (https://lwn.net/Articles/809179/) but might not be > exhaustive. > > ----------------- > On arm64 platform, it is essential to ensure that the boot time > discovered > memory couldn't be hot-removed so that, > > 1. FW data structures used across kexec are idempotent > e.g. the EFI memory map. > > 2. linear map or vmemmap would not have to be dynamically split, and > can > map boot memory at a large granularity > > 3. Avoid penalizing paths that have to walk page tables, where we can > be > certain that the memory is not hot-removable > ----------------- > > The primary reason being kexec which would need substantial rework > otherwise. > >> >> The current logic is that only “new” memory blocks which are hot-added >> can later be offlined and removed. The memory that system booted up >> with cannot be offlined and removed. But there could be many usercases >> such as inter-VM memory sharing where a primary VM could offline and >> hot-remove a block/section of memory and lend it to secondary VM where >> it could hot-add it. And after usecase is done, the reverse happens >> where secondary VM hot-removes and gives it back to primary which can >> hot-add it back. In such cases, the present logic for arm64 doesn’t >> allow this hot-remove in primary to happen. > > That is not true. Each VM could just boot with a minimum boot memory > which can > not be offlined or removed but then a possible larger portion of memory > can be > hot added during the boot process itself, making them available for any > future > inter VM sharing purpose. Hence this problem could easily be solved in > the user > space itself. > >> >> Also, on systems with movable zone that sort of guarantees pages to be >> migrated and isolated so that blocks can be offlined, this logic also >> defeats the purpose of having a movable zone which system can rely on >> memory hot-plugging, which say virt-io mem also relies on for fully >> plugged memory blocks. > ZONE_MOVABLE does not really guarantee migration, isolation and > removal. There > are reasons an offline request might just fail. I agree that those > reasons are > normally not platform related but core memory gives platform an > opportunity to > decline an offlining request via a notifier. Hence ZONE_MOVABLE offline > can be > denied. Semantics wise we are still okay. > > This might look bit inconsistent that > movablecore/kernelcore/movable_node with > firmware sending in 'hot pluggable' memory (IIRC arm64 does not really > support > this yet), the system might end up with ZONE_MOVABLE marked boot memory > which > cannot be offlined or removed. But an offline notifier action is > orthogonal. > Hence did not block those kernel command line paths that creates > ZONE_MOVABLE > during boot to preserve existing behavior. > >> >> I understand that some region of boot RAM shouldn’t be allowed to be >> removed, but such regions won’t be allowed to be offlined in first >> place since pages cannot be migrated and isolated, example reserved >> pages. >> >> So we’re trying to understand the reasoning for such a prevention put >> in place for arm64 arch alone. > > Primary reason being kexec. During kexec on arm64, next kernel's memory > map is > derived from firmware and not from current running kernel. So the next > kernel > will crash if it would access memory that might have been removed in > running > kernel. Until kexec on arm64 changes substantially and takes into > account the > real available memory on the current kernel, boot memory cannot be > removed. > >> >> One possible way to solve this is by marking the required sections as >> “non-early” by removing the SECTION_IS_EARLY bit in its >> section_mem_map. > > That is too intrusive from core memory perspective. > > This puts these sections in the context of “memory hotpluggable” > which can be offlined-removed and added-onlined which are part of boot > RAM itself and doesn’t need any extra blocks to be hot added. This way > of marking certain sections as “non-early” could be exported so that > module drivers can set the required number of sections as “memory > hotpluggable”. This could have certain checks put in place to see > which sections are allowed, example only movable zone sections can be > marked as “non-early”. > > Giving modules the right to mark memory hotpluggable ? That is too > intrusive > and would still not solve the problem with kexec. > >> >> Your thoughts on this? We are also looking for different ways to solve >> the problem without having to completely dropping this notifier, but >> just putting out the concern here about the notifier logic that is >> breaking our usecase which is a generic memory sharing usecase using >> memory hotplug feature. > > Completely preventing boot memory offline and removal is essential for > kexec > to work as expected afterwards. As suggested previously, splitting the > VM > memory into boot and non boot chunks during init can help work around > this > restriction effectively in userspace itself and would not require any > kernel > changes. > > - Anshuman Sudarshan -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD189C2D0A3 for ; Thu, 29 Oct 2020 21:04:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1107F20790 for ; Thu, 29 Oct 2020 21:04:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="E9f/KTaA"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=mg.codeaurora.org header.i=@mg.codeaurora.org header.b="a+uP7lAc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1107F20790 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-ID:References:In-Reply-To:Subject:To:From: Date:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Idt+n7veVG0ssXwGZ0FR5ulq9irQ/YmPNYUNTT5uQk0=; b=E9f/KTaAb4AmbO7C+GZTGJncT 4lW90os6530myWqZZIGg38mAKpL8sEZZOXGzxRFja1XZhSnXx8DoEVDHILI8cwfW69av51kjVL6z5 UUdpnjEa7qvB6pcN8W/lF/nfMv6RR59rUkoQmHm4eASwtqfmwUfTMerAo+IYQqb8KbJwnN7ze7ssW RQH3xbuOcCbhiumSieVfQ97BIU0906jeAzfEvLzisfT7s+noTx4qQevf4yRNr3xDn1IrvZ417QYKL g50Bs1m3otrpq0ub1npQsqN6hflymKOfygL0O+lAe68qdi2WcmcIVVziy8u15QcYyPuErg0l4zxmq uBnijT7/A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYF4C-00028X-A4; Thu, 29 Oct 2020 21:02:40 +0000 Received: from z5.mailgun.us ([104.130.96.5]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYF47-00027S-8E for linux-arm-kernel@lists.infradead.org; Thu, 29 Oct 2020 21:02:38 +0000 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1604005357; h=Message-ID: References: In-Reply-To: Subject: Cc: To: From: Date: Content-Transfer-Encoding: Content-Type: MIME-Version: Sender; bh=wrMdI+4RaPfZPeWzQXW90cx5Z685EYq4/9tTEJkTtvc=; b=a+uP7lAc9K2WaUURkbRN4l31Kcj9gwYZAhrvy8BJkh9jxMDlkcGY1AS7DW78hZwILpR1qsFc CFgUaRnuPJl0QdqWawTqOuuOq+I1RgRqsuYCScM5olZsNhRS8JZ+TLIjqHpc/NnwckYpasiu 1BX2unL0BsUFysXgDlB/9zrAUGE= X-Mailgun-Sending-Ip: 104.130.96.5 X-Mailgun-Sid: WyJiYzAxZiIsICJsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmciLCAiYmU5ZTRhIl0= Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n01.prod.us-west-2.postgun.com with SMTP id 5f9b2dd3c3d7c9858a595e80 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Thu, 29 Oct 2020 21:02:11 GMT Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 0CD74C43385; Thu, 29 Oct 2020 21:02:11 +0000 (UTC) Received: from mail.codeaurora.org (localhost.localdomain [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: sudaraja) by smtp.codeaurora.org (Postfix) with ESMTPSA id 9A0C5C433CB; Thu, 29 Oct 2020 21:02:09 +0000 (UTC) MIME-Version: 1.0 Date: Thu, 29 Oct 2020 14:02:09 -0700 From: Sudarshan Rajagopalan To: Anshuman Khandual Subject: Re: arm64: dropping prevent_bootmem_remove_notifier In-Reply-To: References: Message-ID: <57952529a81c6d5ba2521c203dfc54b5@codeaurora.org> X-Sender: sudaraja@codeaurora.org User-Agent: Roundcube Webmail/1.3.9 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201029_170237_204773_17DC5D03 X-CRM114-Status: GOOD ( 41.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Gavin Shan , Will Deacon , David Hildenbrand , Catalin Marinas , linux-kernel@vger.kernel.org, Steven Price , Logan Gunthorpe , Andrew Morton , Suren Baghdasaryan , linux-arm-kernel@lists.infradead.org, pratikp@codeaurora.org Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CgpIaSBBbnNodW1hbiwgRGF2aWQsCgpUaGFua3MgZm9yIGFsbCB0aGUgZGV0YWlsZWQgZXhwbGFu YXRpb25zIGZvciB0aGUgcmVhc29uaW5nIHRvIGhhdmUgCmJvb3RtZW0gcHJvdGVjdGVkIGZyb20g YmVpbmcgcmVtb3ZlZC4gQWxzbywgSSBkbyBhZ3JlZSBkcml2ZXJzIGJlaW5nIAphYmxlIHRvIG1h cmsgbWVtb3J5IHNlY3Rpb25zIGlzbid0IHRoZSByaWdodCB0aGluZyB0byBkby4KCldlIHdlbnQg YWhlYWQgd2l0aCB0aGUgYXBwcm9hY2ggb2YgdXNpbmcgIm1lbT0iIGFzIHlvdSBzdWdnZXN0ZWQg dG8gCmxpbWl0IHRoZSBib290bWVtIGFuZCBhZGQgcmVtYWluaW5nIGJsb2NrcyB1c2luZyAKYWRk X21lbW9yeV9kcml2ZXJfbWFuYWdlZCgpIHNvIHRoYXQgZHJpdmVyIGhhcyBvd25lcnNoaXAgb2Yg dGhlc2UgCmJsb2Nrcy4KCldlIGRvIGhhdmUgc29tZSBmb2xsb3ctdXAgcXVlc3Rpb25zIHJlZ2Fy ZGluZyB0aGlzIC0gd2lsbCBpbml0aWF0ZSBhIApkaXNjdXNzaW9uIHNvb24uCgoKT24gMjAyMC0x MC0xOCAyMjozNywgQW5zaHVtYW4gS2hhbmR1YWwgd3JvdGU6Cj4gSGVsbG8gU3VkYXJzaGFuLAo+ IAo+IE9uIDEwLzE3LzIwMjAgMDQ6NDEgQU0sIFN1ZGFyc2hhbiBSYWphZ29wYWxhbiB3cm90ZToK Pj4gCj4+IEhlbGxvIEFuc2h1bWFuLAo+PiAKPj4gSW4gdGhlIHBhdGNoIHRoYXQgZW5hYmxlcyBt ZW1vcnkgaG90LXJlbW92ZSAoY29tbWl0IGJiZDZlYzYwNWMwZiAKPj4gKCJhcm02NC9tbTogRW5h YmxlIG1lbW9yeSBob3QgcmVtb3ZlIikpIGZvciBhcm02NCwgdGhlcmXigJlzIGEgbm90aWZpZXIg Cj4+IHB1dCBpbiBwbGFjZSB0aGF0IHByZXZlbnRzIGJvb3QgbWVtb3J5IGZyb20gYmVpbmcgb2Zm bGluZWQgYW5kIAo+PiByZW1vdmVkLiBBbHNvIGNvbW1pdCB0ZXh0IG1lbnRpb25zIHRoYXQgYm9v dCBtZW1vcnkgb24gYXJtNjQgY2Fubm90IGJlIAo+PiByZW1vdmVkLiBXZSB3YW50ZWQgdG8gdW5k ZXJzdGFuZCBtb3JlIGFib3V0IHRoZSByZWFzb25pbmcgZm9yIHRoaXMuIAo+PiBYODYgYW5kIG90 aGVyIGFyY2hzIGRvZXNu4oCZdCBzZWVtIHRvIGRvIHRoaXMgcHJldmVudGlvbi4gVGhlcmXigJlz IGFsc28gCj4+IGNvbW1lbnQgaW4gdGhlIGNvZGUgdGhhdCB0aGlzIG5vdGlmaWVyIGNvdWxkIGJl IGRyb3BwZWQgaW4gZnV0dXJlIGlmIAo+PiBhbmQgd2hlbiBib290IG1lbW9yeSBjYW4gYmUgcmVt b3ZlZC4KPiAKPiBSaWdodCBhbmQgdGlsbCB0aGVuIHRoZSBub3RpZmllciBjYW5ub3QgYmUgZHJv cHBlZC4gVGhlcmUgd2FzIGEgbG90IG9mCj4gZGlzY3Vzc2lvbnMKPiBhcm91bmQgdGhpcyB0b3Bp YyBkdXJpbmcgbXVsdGlwbGUgaXRlcmF0aW9ucyBvZiBtZW1vcnkgaG90IHJlbW92ZQo+IHNlcmll cy4gSGVuY2UsIEkKPiB3b3VsZCBqdXN0IHJlcXVlc3QgeW91IHRvIHBsZWFzZSBnbyB0aHJvdWdo IHRoZW0gZmlyc3QuIFRoaXMgbGlzdCBoZXJlCj4gaXMgZnJvbSBvbmUKPiBzdWNoIHNlcmllcyAo aHR0cHM6Ly9sd24ubmV0L0FydGljbGVzLzgwOTE3OS8pIGJ1dCBtaWdodCBub3QgYmUgCj4gZXho YXVzdGl2ZS4KPiAKPiAtLS0tLS0tLS0tLS0tLS0tLQo+IE9uIGFybTY0IHBsYXRmb3JtLCBpdCBp cyBlc3NlbnRpYWwgdG8gZW5zdXJlIHRoYXQgdGhlIGJvb3QgdGltZSAKPiBkaXNjb3ZlcmVkCj4g bWVtb3J5IGNvdWxkbid0IGJlIGhvdC1yZW1vdmVkIHNvIHRoYXQsCj4gCj4gMS4gRlcgZGF0YSBz dHJ1Y3R1cmVzIHVzZWQgYWNyb3NzIGtleGVjIGFyZSBpZGVtcG90ZW50Cj4gICAgZS5nLiB0aGUg RUZJIG1lbW9yeSBtYXAuCj4gCj4gMi4gbGluZWFyIG1hcCBvciB2bWVtbWFwIHdvdWxkIG5vdCBo YXZlIHRvIGJlIGR5bmFtaWNhbGx5IHNwbGl0LCBhbmQgCj4gY2FuCj4gICAgbWFwIGJvb3QgbWVt b3J5IGF0IGEgbGFyZ2UgZ3JhbnVsYXJpdHkKPiAKPiAzLiBBdm9pZCBwZW5hbGl6aW5nIHBhdGhz IHRoYXQgaGF2ZSB0byB3YWxrIHBhZ2UgdGFibGVzLCB3aGVyZSB3ZSBjYW4gCj4gYmUKPiAgICBj ZXJ0YWluIHRoYXQgdGhlIG1lbW9yeSBpcyBub3QgaG90LXJlbW92YWJsZQo+IC0tLS0tLS0tLS0t LS0tLS0tCj4gCj4gVGhlIHByaW1hcnkgcmVhc29uIGJlaW5nIGtleGVjIHdoaWNoIHdvdWxkIG5l ZWQgc3Vic3RhbnRpYWwgcmV3b3JrIAo+IG90aGVyd2lzZS4KPiAKPj4gCj4+IFRoZSBjdXJyZW50 IGxvZ2ljIGlzIHRoYXQgb25seSDigJxuZXfigJ0gbWVtb3J5IGJsb2NrcyB3aGljaCBhcmUgaG90 LWFkZGVkIAo+PiBjYW4gbGF0ZXIgYmUgb2ZmbGluZWQgYW5kIHJlbW92ZWQuIFRoZSBtZW1vcnkg dGhhdCBzeXN0ZW0gYm9vdGVkIHVwIAo+PiB3aXRoIGNhbm5vdCBiZSBvZmZsaW5lZCBhbmQgcmVt b3ZlZC4gQnV0IHRoZXJlIGNvdWxkIGJlIG1hbnkgdXNlcmNhc2VzIAo+PiBzdWNoIGFzIGludGVy LVZNIG1lbW9yeSBzaGFyaW5nIHdoZXJlIGEgcHJpbWFyeSBWTSBjb3VsZCBvZmZsaW5lIGFuZCAK Pj4gaG90LXJlbW92ZSBhIGJsb2NrL3NlY3Rpb24gb2YgbWVtb3J5IGFuZCBsZW5kIGl0IHRvIHNl Y29uZGFyeSBWTSB3aGVyZSAKPj4gaXQgY291bGQgaG90LWFkZCBpdC4gQW5kIGFmdGVyIHVzZWNh c2UgaXMgZG9uZSwgdGhlIHJldmVyc2UgaGFwcGVucyAKPj4gd2hlcmUgc2Vjb25kYXJ5IFZNIGhv dC1yZW1vdmVzIGFuZCBnaXZlcyBpdCBiYWNrIHRvIHByaW1hcnkgd2hpY2ggY2FuIAo+PiBob3Qt YWRkIGl0IGJhY2suIEluIHN1Y2ggY2FzZXMsIHRoZSBwcmVzZW50IGxvZ2ljIGZvciBhcm02NCBk b2VzbuKAmXQgCj4+IGFsbG93IHRoaXMgaG90LXJlbW92ZSBpbiBwcmltYXJ5IHRvIGhhcHBlbi4K PiAKPiBUaGF0IGlzIG5vdCB0cnVlLiBFYWNoIFZNIGNvdWxkIGp1c3QgYm9vdCB3aXRoIGEgbWlu aW11bSBib290IG1lbW9yeSAKPiB3aGljaCBjYW4KPiBub3QgYmUgb2ZmbGluZWQgb3IgcmVtb3Zl ZCBidXQgdGhlbiBhIHBvc3NpYmxlIGxhcmdlciBwb3J0aW9uIG9mIG1lbW9yeSAKPiBjYW4gYmUK PiBob3QgYWRkZWQgZHVyaW5nIHRoZSBib290IHByb2Nlc3MgaXRzZWxmLCBtYWtpbmcgdGhlbSBh dmFpbGFibGUgZm9yIGFueSAKPiBmdXR1cmUKPiBpbnRlciBWTSBzaGFyaW5nIHB1cnBvc2UuIEhl bmNlIHRoaXMgcHJvYmxlbSBjb3VsZCBlYXNpbHkgYmUgc29sdmVkIGluIAo+IHRoZSB1c2VyCj4g c3BhY2UgaXRzZWxmLgo+IAo+PiAKPj4gQWxzbywgb24gc3lzdGVtcyB3aXRoIG1vdmFibGUgem9u ZSB0aGF0IHNvcnQgb2YgZ3VhcmFudGVlcyBwYWdlcyB0byBiZSAKPj4gbWlncmF0ZWQgYW5kIGlz b2xhdGVkIHNvIHRoYXQgYmxvY2tzIGNhbiBiZSBvZmZsaW5lZCwgdGhpcyBsb2dpYyBhbHNvIAo+ PiBkZWZlYXRzIHRoZSBwdXJwb3NlIG9mIGhhdmluZyBhIG1vdmFibGUgem9uZSB3aGljaCBzeXN0 ZW0gY2FuIHJlbHkgb24gCj4+IG1lbW9yeSBob3QtcGx1Z2dpbmcsIHdoaWNoIHNheSB2aXJ0LWlv IG1lbSBhbHNvIHJlbGllcyBvbiBmb3IgZnVsbHkgCj4+IHBsdWdnZWQgbWVtb3J5IGJsb2Nrcy4K PiBaT05FX01PVkFCTEUgZG9lcyBub3QgcmVhbGx5IGd1YXJhbnRlZSBtaWdyYXRpb24sIGlzb2xh dGlvbiBhbmQgCj4gcmVtb3ZhbC4gVGhlcmUKPiBhcmUgcmVhc29ucyBhbiBvZmZsaW5lIHJlcXVl c3QgbWlnaHQganVzdCBmYWlsLiBJIGFncmVlIHRoYXQgdGhvc2UgCj4gcmVhc29ucyBhcmUKPiBu b3JtYWxseSBub3QgcGxhdGZvcm0gcmVsYXRlZCBidXQgY29yZSBtZW1vcnkgZ2l2ZXMgcGxhdGZv cm0gYW4gCj4gb3Bwb3J0dW5pdHkgdG8KPiBkZWNsaW5lIGFuIG9mZmxpbmluZyByZXF1ZXN0IHZp YSBhIG5vdGlmaWVyLiBIZW5jZSBaT05FX01PVkFCTEUgb2ZmbGluZSAKPiBjYW4gYmUKPiBkZW5p ZWQuIFNlbWFudGljcyB3aXNlIHdlIGFyZSBzdGlsbCBva2F5Lgo+IAo+IFRoaXMgbWlnaHQgbG9v ayBiaXQgaW5jb25zaXN0ZW50IHRoYXQgCj4gbW92YWJsZWNvcmUva2VybmVsY29yZS9tb3ZhYmxl X25vZGUgd2l0aAo+IGZpcm13YXJlIHNlbmRpbmcgaW4gJ2hvdCBwbHVnZ2FibGUnIG1lbW9yeSAo SUlSQyBhcm02NCBkb2VzIG5vdCByZWFsbHkgCj4gc3VwcG9ydAo+IHRoaXMgeWV0KSwgdGhlIHN5 c3RlbSBtaWdodCBlbmQgdXAgd2l0aCBaT05FX01PVkFCTEUgbWFya2VkIGJvb3QgbWVtb3J5IAo+ IHdoaWNoCj4gY2Fubm90IGJlIG9mZmxpbmVkIG9yIHJlbW92ZWQuIEJ1dCBhbiBvZmZsaW5lIG5v dGlmaWVyIGFjdGlvbiBpcyAKPiBvcnRob2dvbmFsLgo+IEhlbmNlIGRpZCBub3QgYmxvY2sgdGhv c2Uga2VybmVsIGNvbW1hbmQgbGluZSBwYXRocyB0aGF0IGNyZWF0ZXMgCj4gWk9ORV9NT1ZBQkxF Cj4gZHVyaW5nIGJvb3QgdG8gcHJlc2VydmUgZXhpc3RpbmcgYmVoYXZpb3IuCj4gCj4+IAo+PiBJ IHVuZGVyc3RhbmQgdGhhdCBzb21lIHJlZ2lvbiBvZiBib290IFJBTSBzaG91bGRu4oCZdCBiZSBh bGxvd2VkIHRvIGJlIAo+PiByZW1vdmVkLCBidXQgc3VjaCByZWdpb25zIHdvbuKAmXQgYmUgYWxs b3dlZCB0byBiZSBvZmZsaW5lZCBpbiBmaXJzdCAKPj4gcGxhY2Ugc2luY2UgcGFnZXMgY2Fubm90 IGJlIG1pZ3JhdGVkIGFuZCBpc29sYXRlZCwgZXhhbXBsZSByZXNlcnZlZCAKPj4gcGFnZXMuCj4+ IAo+PiBTbyB3ZeKAmXJlIHRyeWluZyB0byB1bmRlcnN0YW5kIHRoZSByZWFzb25pbmcgZm9yIHN1 Y2ggYSBwcmV2ZW50aW9uIHB1dCAKPj4gaW4gcGxhY2UgZm9yIGFybTY0IGFyY2ggYWxvbmUuCj4g Cj4gUHJpbWFyeSByZWFzb24gYmVpbmcga2V4ZWMuIER1cmluZyBrZXhlYyBvbiBhcm02NCwgbmV4 dCBrZXJuZWwncyBtZW1vcnkgCj4gbWFwIGlzCj4gZGVyaXZlZCBmcm9tIGZpcm13YXJlIGFuZCBu b3QgZnJvbSBjdXJyZW50IHJ1bm5pbmcga2VybmVsLiBTbyB0aGUgbmV4dCAKPiBrZXJuZWwKPiB3 aWxsIGNyYXNoIGlmIGl0IHdvdWxkIGFjY2VzcyBtZW1vcnkgdGhhdCBtaWdodCBoYXZlIGJlZW4g cmVtb3ZlZCBpbiAKPiBydW5uaW5nCj4ga2VybmVsLiBVbnRpbCBrZXhlYyBvbiBhcm02NCBjaGFu Z2VzIHN1YnN0YW50aWFsbHkgYW5kIHRha2VzIGludG8gCj4gYWNjb3VudCB0aGUKPiByZWFsIGF2 YWlsYWJsZSBtZW1vcnkgb24gdGhlIGN1cnJlbnQga2VybmVsLCBib290IG1lbW9yeSBjYW5ub3Qg YmUgCj4gcmVtb3ZlZC4KPiAKPj4gCj4+IE9uZSBwb3NzaWJsZSB3YXkgdG8gc29sdmUgdGhpcyBp cyBieSBtYXJraW5nIHRoZSByZXF1aXJlZCBzZWN0aW9ucyBhcyAKPj4g4oCcbm9uLWVhcmx54oCd IGJ5IHJlbW92aW5nIHRoZSBTRUNUSU9OX0lTX0VBUkxZIGJpdCBpbiBpdHMgCj4+IHNlY3Rpb25f bWVtX21hcC4KPiAKPiBUaGF0IGlzIHRvbyBpbnRydXNpdmUgZnJvbSBjb3JlIG1lbW9yeSBwZXJz cGVjdGl2ZS4KPiAKPiAgVGhpcyBwdXRzIHRoZXNlIHNlY3Rpb25zIGluIHRoZSBjb250ZXh0IG9m IOKAnG1lbW9yeSBob3RwbHVnZ2FibGXigJ0KPiB3aGljaCBjYW4gYmUgb2ZmbGluZWQtcmVtb3Zl ZCBhbmQgYWRkZWQtb25saW5lZCB3aGljaCBhcmUgcGFydCBvZiBib290Cj4gUkFNIGl0c2VsZiBh bmQgZG9lc27igJl0IG5lZWQgYW55IGV4dHJhIGJsb2NrcyB0byBiZSBob3QgYWRkZWQuIFRoaXMg d2F5Cj4gb2YgbWFya2luZyBjZXJ0YWluIHNlY3Rpb25zIGFzIOKAnG5vbi1lYXJseeKAnSBjb3Vs ZCBiZSBleHBvcnRlZCBzbyB0aGF0Cj4gbW9kdWxlIGRyaXZlcnMgY2FuIHNldCB0aGUgcmVxdWly ZWQgbnVtYmVyIG9mIHNlY3Rpb25zIGFzIOKAnG1lbW9yeQo+IGhvdHBsdWdnYWJsZeKAnS4gVGhp cyBjb3VsZCBoYXZlIGNlcnRhaW4gY2hlY2tzIHB1dCBpbiBwbGFjZSB0byBzZWUKPiB3aGljaCBz ZWN0aW9ucyBhcmUgYWxsb3dlZCwgZXhhbXBsZSBvbmx5IG1vdmFibGUgem9uZSBzZWN0aW9ucyBj YW4gYmUKPiBtYXJrZWQgYXMg4oCcbm9uLWVhcmx54oCdLgo+IAo+IEdpdmluZyBtb2R1bGVzIHRo ZSByaWdodCB0byBtYXJrIG1lbW9yeSBob3RwbHVnZ2FibGUgPyBUaGF0IGlzIHRvbyAKPiBpbnRy dXNpdmUKPiBhbmQgd291bGQgc3RpbGwgbm90IHNvbHZlIHRoZSBwcm9ibGVtIHdpdGgga2V4ZWMu Cj4gCj4+IAo+PiBZb3VyIHRob3VnaHRzIG9uIHRoaXM/IFdlIGFyZSBhbHNvIGxvb2tpbmcgZm9y IGRpZmZlcmVudCB3YXlzIHRvIHNvbHZlIAo+PiB0aGUgcHJvYmxlbSB3aXRob3V0IGhhdmluZyB0 byBjb21wbGV0ZWx5IGRyb3BwaW5nIHRoaXMgbm90aWZpZXIsIGJ1dCAKPj4ganVzdCBwdXR0aW5n IG91dCB0aGUgY29uY2VybiBoZXJlIGFib3V0IHRoZSBub3RpZmllciBsb2dpYyB0aGF0IGlzIAo+ PiBicmVha2luZyBvdXIgdXNlY2FzZSB3aGljaCBpcyBhIGdlbmVyaWMgbWVtb3J5IHNoYXJpbmcg dXNlY2FzZSB1c2luZyAKPj4gbWVtb3J5IGhvdHBsdWcgZmVhdHVyZS4KPiAKPiBDb21wbGV0ZWx5 IHByZXZlbnRpbmcgYm9vdCBtZW1vcnkgb2ZmbGluZSBhbmQgcmVtb3ZhbCBpcyBlc3NlbnRpYWwg Zm9yIAo+IGtleGVjCj4gdG8gd29yayBhcyBleHBlY3RlZCBhZnRlcndhcmRzLiBBcyBzdWdnZXN0 ZWQgcHJldmlvdXNseSwgc3BsaXR0aW5nIHRoZSAKPiBWTQo+IG1lbW9yeSBpbnRvIGJvb3QgYW5k IG5vbiBib290IGNodW5rcyBkdXJpbmcgaW5pdCBjYW4gaGVscCB3b3JrIGFyb3VuZCAKPiB0aGlz Cj4gcmVzdHJpY3Rpb24gZWZmZWN0aXZlbHkgaW4gdXNlcnNwYWNlIGl0c2VsZiBhbmQgd291bGQg bm90IHJlcXVpcmUgYW55IAo+IGtlcm5lbAo+IGNoYW5nZXMuCj4gCj4gLSBBbnNodW1hbgoKClN1 ZGFyc2hhbgoKLS0KUXVhbGNvbW0gSW5ub3ZhdGlvbiBDZW50ZXIsIEluYy4gaXMgYSBtZW1iZXIg b2YgQ29kZSBBdXJvcmEgRm9ydW0sIGEgCkxpbnV4IEZvdW5kYXRpb24gQ29sbGFib3JhdGl2ZSBQ cm9qZWN0CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwps aW51eC1hcm0ta2VybmVsIG1haWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJh ZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51 eC1hcm0ta2VybmVsCg==