From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41EE3C433F5 for ; Mon, 10 Jan 2022 11:27:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244733AbiAJL1D (ORCPT ); Mon, 10 Jan 2022 06:27:03 -0500 Received: from foss.arm.com ([217.140.110.172]:33070 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244686AbiAJL0p (ORCPT ); Mon, 10 Jan 2022 06:26:45 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 869FB2B; Mon, 10 Jan 2022 03:26:35 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.11.128]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E39EB3F5A1; Mon, 10 Jan 2022 03:26:33 -0800 (PST) Date: Mon, 10 Jan 2022 11:26:29 +0000 From: Mark Rutland To: He Ying Cc: catalin.marinas@arm.com, will@kernel.org, marcan@marcan.st, maz@kernel.org, joey.gouly@arm.com, pcc@google.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] arm64: Make CONFIG_ARM64_PSEUDO_NMI macro wrap all the pseudo-NMI code Message-ID: References: <20220107085536.214501-1-heying24@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jan 10, 2022 at 11:00:43AM +0800, He Ying wrote: > Hi Mark, > > I'm just back from the weekend and sorry for the delayed reply. > > > 在 2022/1/7 21:19, Mark Rutland 写道: > > On Fri, Jan 07, 2022 at 03:55:36AM -0500, He Ying wrote: > > > Our product has been updating its kernel from 4.4 to 5.10 recently and > > > found a performance issue. We do a bussiness test called ARP test, which > > > tests the latency for a ping-pong packets traffic with a certain payload. > > > The result is as following. > > > > > > - 4.4 kernel: avg = ~20s > > > - 5.10 kernel (CONFIG_ARM64_PSEUDO_NMI is not set): avg = ~40s > > Have you tested with a recent mainline kernel, e.g. v5.15? > > Actuallly no, that's because this test is only available for the product > environment and > > we don't have an available 5.15 kernel for it yet. Ok; do you see anything comparable with any tests available to upstream developers? e.g. hackbench or `perf bench sched` ? > > Is this test publicly available, and can you say which hardrware (e.g. which > > CPU implementation) you're testing with? > > Actually no. The test is only available for our product environment now. We > are testing > > with hisilicon 1213 (4 ARM Cortex-A72 cores). Thanks for the CPU info; there are a number of other Cortex-A72 platforms out there, so it might be possible to reproduce the behaviour elsewhere. > > > I have been just learning arm64 pseudo-NMI code and have a question, > > > why is the related code not wrapped by CONFIG_ARM64_PSEUDO_NMI? > > The code in question is all patched via alternatives, and when > > CONFIG_ARM64_PSEUDO_NMI is not selected, the code was expected to only have the > > overhead of the regular DAIF manipulation. > I don't understand alernatives very well and I'll apreciate it if you can > explain it a bit more. Code using alternatives is patched at runtime, so for: ALTERNATIVE(x, y, cap) ... the `x` instructions will be inline in the code, and the `y` instructions will be placed in a separate linker section. If the `cap` capability is detected, the `y` instructions will be copied over the `x` instructions. So for: | asm volatile(ALTERNATIVE( | "msr daifclr, #3 // arch_local_irq_enable", | __msr_s(SYS_ICC_PMR_EL1, "%0"), | ARM64_HAS_IRQ_PRIO_MASKING) | : | : "r" ((unsigned long) GIC_PRIO_IRQON) | : "memory"); ... where ARM64_HAS_IRQ_PRIO_MASKING is not detected, the inline instructions will be: | msr daifclr, #3 // arch_local_irq_enable A separate linker section will contain: ... and a separate linker section will contain: | __msr_s(SYS_ICC_PMR_EL1, "%0" ... and some metadata will be recorded in a `struct alt_instr`, which will be unused and will take some space, but should have no runtime overhead -- the runtime cost should only be the `msr daifclr, #3` instruction. > > > I wonder if this brings some performance regression. > > > > > > First, I make this patch and then do the test again. Here's the result. > > > > > > - 5.10 kernel with this patch not applied: avg = ~40s > > > - 5.10 kernel with this patch applied: avg = ~23s > > > > > > Amazing! Note that all kernel is built with CONFIG_ARM64_PSEUDO_NMI not > > > set. It seems the pseudo-NMI feature actually brings some overhead to > > > performance event if CONFIG_ARM64_PSEUDO_NMI is not set. > > I'm surprised the overhead is so significant; as above this is all patched in > > and so the overhead when this is disabled is expected to be *extremely* small. > > > > For example, wjen CONFIG_ARM64_PSEUDO_NMI, in arch_local_irq_enable(): > > > > * The portion under the system_has_prio_mask_debugging() test will be removed > > entirely by the compiler, as this internally checks > > IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI). > > > > * The assembly will be left as a write to DAIFClr. The only additional cost > > should be that of generating GIC_PRIO_IRQON into a register. > > > > * The pmr_sync() will be removed entirely by the compiler as is defined > > conditionally dependent on CONFIG_ARM64_PSEUDO_NMI. > > > > I can't spot an obvious issue with that or ther other cases. In the common case > > those add no new instructions, and in the worst case they only add NOPs. > > Thanks for your detailed explaination! Actually I can't understand the > result exactly. > > I build two 5.10 kernel images with this patch applied or not and objdump > them. Indeed, the disassembles of 'arch_local_irq_restore' are the same. Do > you have any ideas how we can find the root cause why this patch improves the > performance so much? > > However, the test result is trustworthy because we do it many times and the > result is always repeatable. Due to the large numbers, I suspect this must be due to a specific fast-path, and it's possible that this is due to secondary factors (e.g. alignment of code) rather than the pseudo-NMK code itself. We need to narrow down *where* time is being spent. Since it appears that this is related to the local IRQ state management, it isn't likely that we can determine that reliably with the PMU. Given that, I think the first step is to reproduce the result elsewhere, for which we will need some plublicly available test-case. > > > Furthermore, I find the feature also brings some overhead to vmlinux size. > > > I build 5.10 kernel with this patch applied or not while > > > CONFIG_ARM64_PSEUDO_NMI is not set. > > > > > > - 5.10 kernel with this patch not applied: vmlinux size is 384060600 Bytes. > > > - 5.10 kernel with this patch applied: vmlinux size is 383842936 Bytes. > > > > > > That means arm64 pseudo-NMI feature may bring ~200KB overhead to > > > vmlinux size. > > I suspect that's just the (unused) alternatives, and we could improve that by > > passing the config into the alternative blocks. > > Do you mean the sections generated by the alternatives? I don't understand > alernatives very well and I'll apreciate it if you can explain it a bit > more. Yes; I meant the sections generated to hold the alternatives code and the alternatives metadata. > > > Above all, arm64 pseudo-NMI feature brings some overhead to vmlinux size > > > and performance even if config is not set. To avoid it, add macro control > > > all around the related code. > > > > > > Signed-off-by: He Ying > > > --- > > > arch/arm64/include/asm/irqflags.h | 38 +++++++++++++++++++++++++++++-- > > > arch/arm64/kernel/entry.S | 4 ++++ > > > 2 files changed, 40 insertions(+), 2 deletions(-) > > > > > > diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h > > > index b57b9b1e4344..82f771b41cf5 100644 > > > --- a/arch/arm64/include/asm/irqflags.h > > > +++ b/arch/arm64/include/asm/irqflags.h > > > @@ -26,6 +26,7 @@ > > > */ > > > static inline void arch_local_irq_enable(void) > > > { > > > +#ifdef CONFIG_ARM64_PSEUDO_NMI > > > if (system_has_prio_mask_debugging()) { > > > u32 pmr = read_sysreg_s(SYS_ICC_PMR_EL1); > > > @@ -41,10 +42,18 @@ static inline void arch_local_irq_enable(void) > > > : "memory"); > > > pmr_sync(); > > > +#else > > > + asm volatile( > > > + "msr daifclr, #3 // arch_local_irq_enable" > > > + : > > > + : > > > + : "memory"); > > > +#endif > > I'm happy to rework this to improve matters, but I am very much not happy with > > duplicating the logic for the !PSEUDO_NMI case. Adding more ifdeffery and > > copies of that is not acceptable. > I agree. Adding these ifdeffery is a bit ugly. Let's see if there are some > better ways. > > > > Instead, can you please try changing the alternative to also take the config, > > e.g. here have: > > > > | asm volatile(ALTERNATIVE( > > | "msr daifclr, #3 // arch_local_irq_enable", > > | __msr_s(SYS_ICC_PMR_EL1, "%0"), > > | ARM64_HAS_IRQ_PRIO_MASKING, > > | CONFIG_ARM64_PSEUDO_NMI) > > | : > > | : "r" ((unsigned long) GIC_PRIO_IRQON) > > | : "memory"); > > > > ... and see if that makes a significant difference? > > > > Likewise for the other casees. > > OK, I'll try it. But I have some questions. Here's the comment of > ALERNATIVE: > > /* >  * Usage: asm(ALTERNATIVE(oldinstr, newinstr, feature)); >  * >  * Usage: asm(ALTERNATIVE(oldinstr, newinstr, feature, CONFIG_FOO)); >  * N.B. If CONFIG_FOO is specified, but not selected, the whole block >  *      will be omitted, including oldinstr. >  */ > > If CONFIG_FOO is not selected, the whole block will be omitted including > oldinstr. > > But we still want the oldinstr in this situation. Do I misunderstand > something? Sorry; you are right, and my suggestion was broken. I'll need to have a think about this; we might be able to rework this to use a static key instead, but IIRC last time I tried there were issues with that approach. I have some (old) work-in-progress patches at: https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/daif-rework I suspect I won't have time to renew that in the near future, but an approach like that might be worthwhile. Thanks, Mark. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8978C433F5 for ; Mon, 10 Jan 2022 11:28:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=91pPOQbGjzNPB/A3eoKfdjDeCTyGYjAYY7Kpi1ScwgQ=; b=sraTw3AkPWj0uZ PjHqhKRz/elVXxQgQqw0PS1f1lqiqjY6QKtAzXthxdMmc0yEylzu+i3BcnnxNcIQl5zMZhHPT8IDZ FE1ce/KBMaihCZYP6lokOXgi1ukUst62Xy1ddcbjrDAVx3KqLqhFIL0pauASB//dwSqhHpC2I85ao NCkH4+3asBlX/w/xHAQc2Wazc7GfTCPG9VoDa5tu8NcGlxPupt2i/mcgfZOP6970b0hL7yUxfJRQQ NKQFjKwjSdAobr2CK/Up0DJazYScLgCufJ/xcuDgDEW1MB5SZaPgnfXzLHxXzVtA4/hXEU/6FRm+O sONpfhMvAPs8Hef2ZH0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6soy-00B5iu-Cd; Mon, 10 Jan 2022 11:26:40 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1n6sou-00B5iB-D5 for linux-arm-kernel@lists.infradead.org; Mon, 10 Jan 2022 11:26:38 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 869FB2B; Mon, 10 Jan 2022 03:26:35 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.11.128]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E39EB3F5A1; Mon, 10 Jan 2022 03:26:33 -0800 (PST) Date: Mon, 10 Jan 2022 11:26:29 +0000 From: Mark Rutland To: He Ying Cc: catalin.marinas@arm.com, will@kernel.org, marcan@marcan.st, maz@kernel.org, joey.gouly@arm.com, pcc@google.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] arm64: Make CONFIG_ARM64_PSEUDO_NMI macro wrap all the pseudo-NMI code Message-ID: References: <20220107085536.214501-1-heying24@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220110_032636_578904_583EE628 X-CRM114-Status: GOOD ( 68.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gTW9uLCBKYW4gMTAsIDIwMjIgYXQgMTE6MDA6NDNBTSArMDgwMCwgSGUgWWluZyB3cm90ZToK PiBIaSBNYXJrLAo+IAo+IEknbSBqdXN0IGJhY2sgZnJvbSB0aGUgd2Vla2VuZCBhbmQgc29ycnkg Zm9yIHRoZSBkZWxheWVkIHJlcGx5Lgo+IAo+IAo+IOWcqCAyMDIyLzEvNyAyMToxOSwgTWFyayBS dXRsYW5kIOWGmemBkzoKPiA+IE9uIEZyaSwgSmFuIDA3LCAyMDIyIGF0IDAzOjU1OjM2QU0gLTA1 MDAsIEhlIFlpbmcgd3JvdGU6Cj4gPiA+IE91ciBwcm9kdWN0IGhhcyBiZWVuIHVwZGF0aW5nIGl0 cyBrZXJuZWwgZnJvbSA0LjQgdG8gNS4xMCByZWNlbnRseSBhbmQKPiA+ID4gZm91bmQgYSBwZXJm b3JtYW5jZSBpc3N1ZS4gV2UgZG8gYSBidXNzaW5lc3MgdGVzdCBjYWxsZWQgQVJQIHRlc3QsIHdo aWNoCj4gPiA+IHRlc3RzIHRoZSBsYXRlbmN5IGZvciBhIHBpbmctcG9uZyBwYWNrZXRzIHRyYWZm aWMgd2l0aCBhIGNlcnRhaW4gcGF5bG9hZC4KPiA+ID4gVGhlIHJlc3VsdCBpcyBhcyBmb2xsb3dp bmcuCj4gPiA+IAo+ID4gPiAgIC0gNC40IGtlcm5lbDogYXZnID0gfjIwcwo+ID4gPiAgIC0gNS4x MCBrZXJuZWwgKENPTkZJR19BUk02NF9QU0VVRE9fTk1JIGlzIG5vdCBzZXQpOiBhdmcgPSB+NDBz Cj4gPiBIYXZlIHlvdSB0ZXN0ZWQgd2l0aCBhIHJlY2VudCBtYWlubGluZSBrZXJuZWwsIGUuZy4g djUuMTU/Cj4gCj4gQWN0dWFsbGx5IG5vLCB0aGF0J3MgYmVjYXVzZSB0aGlzIHRlc3QgaXMgb25s eSBhdmFpbGFibGUgZm9yIHRoZSBwcm9kdWN0Cj4gZW52aXJvbm1lbnQgYW5kCj4gCj4gd2UgZG9u J3QgaGF2ZSBhbiBhdmFpbGFibGUgNS4xNSBrZXJuZWwgZm9yIGl0IHlldC4KCk9rOyBkbyB5b3Ug c2VlIGFueXRoaW5nIGNvbXBhcmFibGUgd2l0aCBhbnkgdGVzdHMgYXZhaWxhYmxlIHRvIHVwc3Ry ZWFtCmRldmVsb3BlcnM/IGUuZy4gaGFja2JlbmNoIG9yIGBwZXJmIGJlbmNoIHNjaGVkYCA/Cgo+ ID4gSXMgdGhpcyB0ZXN0IHB1YmxpY2x5IGF2YWlsYWJsZSwgYW5kIGNhbiB5b3Ugc2F5IHdoaWNo IGhhcmRyd2FyZSAoZS5nLiB3aGljaAo+ID4gQ1BVIGltcGxlbWVudGF0aW9uKSB5b3UncmUgdGVz dGluZyB3aXRoPwo+IAo+IEFjdHVhbGx5IG5vLiBUaGUgdGVzdCBpcyBvbmx5IGF2YWlsYWJsZSBm b3Igb3VyIHByb2R1Y3QgZW52aXJvbm1lbnQgbm93LiBXZQo+IGFyZSB0ZXN0aW5nCj4gCj4gd2l0 aCBoaXNpbGljb24gMTIxMyAoNCBBUk0gQ29ydGV4LUE3MiBjb3JlcykuCgpUaGFua3MgZm9yIHRo ZSBDUFUgaW5mbzsgdGhlcmUgYXJlIGEgbnVtYmVyIG9mIG90aGVyIENvcnRleC1BNzIgcGxhdGZv cm1zIG91dAp0aGVyZSwgc28gaXQgbWlnaHQgYmUgcG9zc2libGUgdG8gcmVwcm9kdWNlIHRoZSBi ZWhhdmlvdXIgZWxzZXdoZXJlLgoKPiA+ID4gSSBoYXZlIGJlZW4ganVzdCBsZWFybmluZyBhcm02 NCBwc2V1ZG8tTk1JIGNvZGUgYW5kIGhhdmUgYSBxdWVzdGlvbiwKPiA+ID4gd2h5IGlzIHRoZSBy ZWxhdGVkIGNvZGUgbm90IHdyYXBwZWQgYnkgQ09ORklHX0FSTTY0X1BTRVVET19OTUk/Cj4gPiBU aGUgY29kZSBpbiBxdWVzdGlvbiBpcyBhbGwgcGF0Y2hlZCB2aWEgYWx0ZXJuYXRpdmVzLCBhbmQg d2hlbgo+ID4gQ09ORklHX0FSTTY0X1BTRVVET19OTUkgaXMgbm90IHNlbGVjdGVkLCB0aGUgY29k ZSB3YXMgZXhwZWN0ZWQgdG8gb25seSBoYXZlIHRoZQo+ID4gb3ZlcmhlYWQgb2YgdGhlIHJlZ3Vs YXIgREFJRiBtYW5pcHVsYXRpb24uCj4gSSBkb24ndCB1bmRlcnN0YW5kIGFsZXJuYXRpdmVzIHZl cnkgd2VsbCBhbmQgSSdsbCBhcHJlY2lhdGUgaXQgaWYgeW91IGNhbgo+IGV4cGxhaW4gaXQgYSBi aXQgbW9yZS4KCkNvZGUgdXNpbmcgYWx0ZXJuYXRpdmVzIGlzIHBhdGNoZWQgYXQgcnVudGltZSwg c28gZm9yOgoKCUFMVEVSTkFUSVZFKHgsIHksIGNhcCkKCi4uLiB0aGUgYHhgIGluc3RydWN0aW9u cyB3aWxsIGJlIGlubGluZSBpbiB0aGUgY29kZSwgYW5kIHRoZSBgeWAgaW5zdHJ1Y3Rpb25zCndp bGwgYmUgcGxhY2VkIGluIGEgc2VwYXJhdGUgbGlua2VyIHNlY3Rpb24uIElmIHRoZSBgY2FwYCBj YXBhYmlsaXR5IGlzCmRldGVjdGVkLCB0aGUgYHlgIGluc3RydWN0aW9ucyB3aWxsIGJlIGNvcGll ZCBvdmVyIHRoZSBgeGAgaW5zdHJ1Y3Rpb25zLgoKU28gZm9yOgoKfCBhc20gdm9sYXRpbGUoQUxU RVJOQVRJVkUoCnwgICAgICAgICAibXNyICAgIGRhaWZjbHIsICMzICAgICAgICAgICAgIC8vIGFy Y2hfbG9jYWxfaXJxX2VuYWJsZSIsCnwgICAgICAgICBfX21zcl9zKFNZU19JQ0NfUE1SX0VMMSwg IiUwIiksCnwgICAgICAgICBBUk02NF9IQVNfSVJRX1BSSU9fTUFTS0lORykKfCAgICAgICAgIDoK fCAgICAgICAgIDogInIiICgodW5zaWduZWQgbG9uZykgR0lDX1BSSU9fSVJRT04pCnwgICAgICAg ICA6ICJtZW1vcnkiKTsKCi4uLiB3aGVyZSBBUk02NF9IQVNfSVJRX1BSSU9fTUFTS0lORyBpcyBu b3QgZGV0ZWN0ZWQsIHRoZSBpbmxpbmUgaW5zdHJ1Y3Rpb25zCndpbGwgYmU6Cgp8IG1zciAgICBk YWlmY2xyLCAjMyAgICAgICAgICAgICAvLyBhcmNoX2xvY2FsX2lycV9lbmFibGUgCgpBIHNlcGFy YXRlIGxpbmtlciBzZWN0aW9uIHdpbGwgY29udGFpbjoKLi4uIGFuZCBhIHNlcGFyYXRlIGxpbmtl ciBzZWN0aW9uIHdpbGwgY29udGFpbjoKCnwgX19tc3JfcyhTWVNfSUNDX1BNUl9FTDEsICIlMCIK Ci4uLiBhbmQgc29tZSBtZXRhZGF0YSB3aWxsIGJlIHJlY29yZGVkIGluIGEgYHN0cnVjdCBhbHRf aW5zdHJgLCB3aGljaCB3aWxsIGJlCnVudXNlZCBhbmQgd2lsbCB0YWtlIHNvbWUgc3BhY2UsIGJ1 dCBzaG91bGQgaGF2ZSBubyBydW50aW1lIG92ZXJoZWFkIC0tIHRoZQpydW50aW1lIGNvc3Qgc2hv dWxkIG9ubHkgYmUgdGhlIGBtc3IgZGFpZmNsciwgIzNgIGluc3RydWN0aW9uLgoKPiA+ID4gSSB3 b25kZXIgaWYgdGhpcyBicmluZ3Mgc29tZSBwZXJmb3JtYW5jZSByZWdyZXNzaW9uLgo+ID4gPiAK PiA+ID4gRmlyc3QsIEkgbWFrZSB0aGlzIHBhdGNoIGFuZCB0aGVuIGRvIHRoZSB0ZXN0IGFnYWlu LiBIZXJlJ3MgdGhlIHJlc3VsdC4KPiA+ID4gCj4gPiA+ICAgLSA1LjEwIGtlcm5lbCB3aXRoIHRo aXMgcGF0Y2ggbm90IGFwcGxpZWQ6IGF2ZyA9IH40MHMKPiA+ID4gICAtIDUuMTAga2VybmVsIHdp dGggdGhpcyBwYXRjaCBhcHBsaWVkOiBhdmcgPSB+MjNzCj4gPiA+IAo+ID4gPiBBbWF6aW5nISBO b3RlIHRoYXQgYWxsIGtlcm5lbCBpcyBidWlsdCB3aXRoIENPTkZJR19BUk02NF9QU0VVRE9fTk1J IG5vdAo+ID4gPiBzZXQuIEl0IHNlZW1zIHRoZSBwc2V1ZG8tTk1JIGZlYXR1cmUgYWN0dWFsbHkg YnJpbmdzIHNvbWUgb3ZlcmhlYWQgdG8KPiA+ID4gcGVyZm9ybWFuY2UgZXZlbnQgaWYgQ09ORklH X0FSTTY0X1BTRVVET19OTUkgaXMgbm90IHNldC4KPiA+IEknbSBzdXJwcmlzZWQgdGhlIG92ZXJo ZWFkIGlzIHNvIHNpZ25pZmljYW50OyBhcyBhYm92ZSB0aGlzIGlzIGFsbCBwYXRjaGVkIGluCj4g PiBhbmQgc28gdGhlIG92ZXJoZWFkIHdoZW4gdGhpcyBpcyBkaXNhYmxlZCBpcyBleHBlY3RlZCB0 byBiZSAqZXh0cmVtZWx5KiBzbWFsbC4KPiA+IAo+ID4gRm9yIGV4YW1wbGUsIHdqZW4gQ09ORklH X0FSTTY0X1BTRVVET19OTUksIGluIGFyY2hfbG9jYWxfaXJxX2VuYWJsZSgpOgo+ID4gCj4gPiAq IFRoZSBwb3J0aW9uIHVuZGVyIHRoZSBzeXN0ZW1faGFzX3ByaW9fbWFza19kZWJ1Z2dpbmcoKSB0 ZXN0IHdpbGwgYmUgcmVtb3ZlZAo+ID4gICAgZW50aXJlbHkgYnkgdGhlIGNvbXBpbGVyLCBhcyB0 aGlzIGludGVybmFsbHkgY2hlY2tzCj4gPiAgICBJU19FTkFCTEVEKENPTkZJR19BUk02NF9QU0VV RE9fTk1JKS4KPiA+IAo+ID4gKiBUaGUgYXNzZW1ibHkgd2lsbCBiZSBsZWZ0IGFzIGEgd3JpdGUg dG8gREFJRkNsci4gVGhlIG9ubHkgYWRkaXRpb25hbCBjb3N0Cj4gPiAgICBzaG91bGQgYmUgdGhh dCBvZiBnZW5lcmF0aW5nIEdJQ19QUklPX0lSUU9OIGludG8gYSByZWdpc3Rlci4KPiA+IAo+ID4g KiBUaGUgcG1yX3N5bmMoKSB3aWxsIGJlIHJlbW92ZWQgZW50aXJlbHkgYnkgdGhlIGNvbXBpbGVy IGFzIGlzIGRlZmluZWQKPiA+ICAgIGNvbmRpdGlvbmFsbHkgZGVwZW5kZW50IG9uIENPTkZJR19B Uk02NF9QU0VVRE9fTk1JLgo+ID4gCj4gPiBJIGNhbid0IHNwb3QgYW4gb2J2aW91cyBpc3N1ZSB3 aXRoIHRoYXQgb3IgdGhlciBvdGhlciBjYXNlcy4gSW4gdGhlIGNvbW1vbiBjYXNlCj4gPiB0aG9z ZSBhZGQgbm8gbmV3IGluc3RydWN0aW9ucywgYW5kIGluIHRoZSB3b3JzdCBjYXNlIHRoZXkgb25s eSBhZGQgTk9Qcy4KPiAKPiBUaGFua3MgZm9yIHlvdXIgZGV0YWlsZWQgZXhwbGFpbmF0aW9uISBB Y3R1YWxseSBJIGNhbid0IHVuZGVyc3RhbmQgdGhlCj4gcmVzdWx0IGV4YWN0bHkuCj4gCj4gSSBi dWlsZCB0d28gNS4xMCBrZXJuZWwgaW1hZ2VzIHdpdGggdGhpcyBwYXRjaCBhcHBsaWVkIG9yIG5v dCBhbmQgb2JqZHVtcAo+IHRoZW0uIEluZGVlZCwgdGhlIGRpc2Fzc2VtYmxlcyBvZiAnYXJjaF9s b2NhbF9pcnFfcmVzdG9yZScgYXJlIHRoZSBzYW1lLiBEbwo+IHlvdSBoYXZlIGFueSBpZGVhcyBo b3cgd2UgY2FuIGZpbmQgdGhlIHJvb3QgY2F1c2Ugd2h5IHRoaXMgcGF0Y2ggaW1wcm92ZXMgdGhl Cj4gcGVyZm9ybWFuY2Ugc28gbXVjaD8KPiAKPiBIb3dldmVyLCB0aGUgdGVzdCByZXN1bHQgaXMg dHJ1c3R3b3J0aHkgYmVjYXVzZSB3ZSBkbyBpdCBtYW55IHRpbWVzIGFuZCB0aGUKPiByZXN1bHQg aXMgYWx3YXlzIHJlcGVhdGFibGUuCgpEdWUgdG8gdGhlIGxhcmdlIG51bWJlcnMsIEkgc3VzcGVj dCB0aGlzIG11c3QgYmUgZHVlIHRvIGEgc3BlY2lmaWMgZmFzdC1wYXRoLAphbmQgaXQncyBwb3Nz aWJsZSB0aGF0IHRoaXMgaXMgZHVlIHRvIHNlY29uZGFyeSBmYWN0b3JzIChlLmcuIGFsaWdubWVu dCBvZgpjb2RlKSByYXRoZXIgdGhhbiB0aGUgcHNldWRvLU5NSyBjb2RlIGl0c2VsZi4KCldlIG5l ZWQgdG8gbmFycm93IGRvd24gKndoZXJlKiB0aW1lIGlzIGJlaW5nIHNwZW50LiBTaW5jZSBpdCBh cHBlYXJzIHRoYXQgdGhpcwppcyByZWxhdGVkIHRvIHRoZSBsb2NhbCBJUlEgc3RhdGUgbWFuYWdl bWVudCwgaXQgaXNuJ3QgbGlrZWx5IHRoYXQgd2UgY2FuCmRldGVybWluZSB0aGF0IHJlbGlhYmx5 IHdpdGggdGhlIFBNVS4gR2l2ZW4gdGhhdCwgSSB0aGluayB0aGUgZmlyc3Qgc3RlcCBpcyB0bwpy ZXByb2R1Y2UgdGhlIHJlc3VsdCBlbHNld2hlcmUsIGZvciB3aGljaCB3ZSB3aWxsIG5lZWQgc29t ZSBwbHVibGljbHkgYXZhaWxhYmxlCnRlc3QtY2FzZS4KCj4gPiA+IEZ1cnRoZXJtb3JlLCBJIGZp bmQgdGhlIGZlYXR1cmUgYWxzbyBicmluZ3Mgc29tZSBvdmVyaGVhZCB0byB2bWxpbnV4IHNpemUu Cj4gPiA+IEkgYnVpbGQgNS4xMCBrZXJuZWwgd2l0aCB0aGlzIHBhdGNoIGFwcGxpZWQgb3Igbm90 IHdoaWxlCj4gPiA+IENPTkZJR19BUk02NF9QU0VVRE9fTk1JIGlzIG5vdCBzZXQuCj4gPiA+IAo+ ID4gPiAgIC0gNS4xMCBrZXJuZWwgd2l0aCB0aGlzIHBhdGNoIG5vdCBhcHBsaWVkOiB2bWxpbnV4 IHNpemUgaXMgMzg0MDYwNjAwIEJ5dGVzLgo+ID4gPiAgIC0gNS4xMCBrZXJuZWwgd2l0aCB0aGlz IHBhdGNoIGFwcGxpZWQ6IHZtbGludXggc2l6ZSBpcyAzODM4NDI5MzYgQnl0ZXMuCj4gPiA+IAo+ ID4gPiBUaGF0IG1lYW5zIGFybTY0IHBzZXVkby1OTUkgZmVhdHVyZSBtYXkgYnJpbmcgfjIwMEtC IG92ZXJoZWFkIHRvCj4gPiA+IHZtbGludXggc2l6ZS4KPiA+IEkgc3VzcGVjdCB0aGF0J3MganVz dCB0aGUgKHVudXNlZCkgYWx0ZXJuYXRpdmVzLCBhbmQgd2UgY291bGQgaW1wcm92ZSB0aGF0IGJ5 Cj4gPiBwYXNzaW5nIHRoZSBjb25maWcgaW50byB0aGUgYWx0ZXJuYXRpdmUgYmxvY2tzLgo+IAo+ IERvIHlvdSBtZWFuIHRoZSBzZWN0aW9ucyBnZW5lcmF0ZWQgYnkgdGhlIGFsdGVybmF0aXZlcz8g SSBkb24ndCB1bmRlcnN0YW5kCj4gYWxlcm5hdGl2ZXMgdmVyeSB3ZWxsIGFuZCBJJ2xsIGFwcmVj aWF0ZSBpdCBpZiB5b3UgY2FuIGV4cGxhaW4gaXQgYSBiaXQKPiBtb3JlLgoKWWVzOyBJIG1lYW50 IHRoZSBzZWN0aW9ucyBnZW5lcmF0ZWQgdG8gaG9sZCB0aGUgYWx0ZXJuYXRpdmVzIGNvZGUgYW5k IHRoZQphbHRlcm5hdGl2ZXMgbWV0YWRhdGEuCgoKPiA+ID4gQWJvdmUgYWxsLCBhcm02NCBwc2V1 ZG8tTk1JIGZlYXR1cmUgYnJpbmdzIHNvbWUgb3ZlcmhlYWQgdG8gdm1saW51eCBzaXplCj4gPiA+ IGFuZCBwZXJmb3JtYW5jZSBldmVuIGlmIGNvbmZpZyBpcyBub3Qgc2V0LiBUbyBhdm9pZCBpdCwg YWRkIG1hY3JvIGNvbnRyb2wKPiA+ID4gYWxsIGFyb3VuZCB0aGUgcmVsYXRlZCBjb2RlLgo+ID4g PiAKPiA+ID4gU2lnbmVkLW9mZi1ieTogSGUgWWluZyA8aGV5aW5nMjRAaHVhd2VpLmNvbT4KPiA+ ID4gLS0tCj4gPiA+ICAgYXJjaC9hcm02NC9pbmNsdWRlL2FzbS9pcnFmbGFncy5oIHwgMzggKysr KysrKysrKysrKysrKysrKysrKysrKysrKystLQo+ID4gPiAgIGFyY2gvYXJtNjQva2VybmVsL2Vu dHJ5LlMgICAgICAgICB8ICA0ICsrKysKPiA+ID4gICAyIGZpbGVzIGNoYW5nZWQsIDQwIGluc2Vy dGlvbnMoKyksIDIgZGVsZXRpb25zKC0pCj4gPiA+IAo+ID4gPiBkaWZmIC0tZ2l0IGEvYXJjaC9h cm02NC9pbmNsdWRlL2FzbS9pcnFmbGFncy5oIGIvYXJjaC9hcm02NC9pbmNsdWRlL2FzbS9pcnFm bGFncy5oCj4gPiA+IGluZGV4IGI1N2I5YjFlNDM0NC4uODJmNzcxYjQxY2Y1IDEwMDY0NAo+ID4g PiAtLS0gYS9hcmNoL2FybTY0L2luY2x1ZGUvYXNtL2lycWZsYWdzLmgKPiA+ID4gKysrIGIvYXJj aC9hcm02NC9pbmNsdWRlL2FzbS9pcnFmbGFncy5oCj4gPiA+IEBAIC0yNiw2ICsyNiw3IEBACj4g PiA+ICAgICovCj4gPiA+ICAgc3RhdGljIGlubGluZSB2b2lkIGFyY2hfbG9jYWxfaXJxX2VuYWJs ZSh2b2lkKQo+ID4gPiAgIHsKPiA+ID4gKyNpZmRlZiBDT05GSUdfQVJNNjRfUFNFVURPX05NSQo+ ID4gPiAgIAlpZiAoc3lzdGVtX2hhc19wcmlvX21hc2tfZGVidWdnaW5nKCkpIHsKPiA+ID4gICAJ CXUzMiBwbXIgPSByZWFkX3N5c3JlZ19zKFNZU19JQ0NfUE1SX0VMMSk7Cj4gPiA+IEBAIC00MSwx MCArNDIsMTggQEAgc3RhdGljIGlubGluZSB2b2lkIGFyY2hfbG9jYWxfaXJxX2VuYWJsZSh2b2lk KQo+ID4gPiAgIAkJOiAibWVtb3J5Iik7Cj4gPiA+ICAgCXBtcl9zeW5jKCk7Cj4gPiA+ICsjZWxz ZQo+ID4gPiArCWFzbSB2b2xhdGlsZSgKPiA+ID4gKwkJIm1zcglkYWlmY2xyLCAjMwkJLy8gYXJj aF9sb2NhbF9pcnFfZW5hYmxlIgo+ID4gPiArCQk6Cj4gPiA+ICsJCToKPiA+ID4gKwkJOiAibWVt b3J5Iik7Cj4gPiA+ICsjZW5kaWYKPiA+IEknbSBoYXBweSB0byByZXdvcmsgdGhpcyB0byBpbXBy b3ZlIG1hdHRlcnMsIGJ1dCBJIGFtIHZlcnkgbXVjaCBub3QgaGFwcHkgd2l0aAo+ID4gZHVwbGlj YXRpbmcgdGhlIGxvZ2ljIGZvciB0aGUgIVBTRVVET19OTUkgY2FzZS4gQWRkaW5nIG1vcmUgaWZk ZWZmZXJ5IGFuZAo+ID4gY29waWVzIG9mIHRoYXQgaXMgbm90IGFjY2VwdGFibGUuCj4gSSBhZ3Jl ZS4gQWRkaW5nIHRoZXNlIGlmZGVmZmVyeSBpcyBhIGJpdCB1Z2x5LiBMZXQncyBzZWUgaWYgdGhl cmUgYXJlIHNvbWUKPiBiZXR0ZXIgd2F5cy4KPiA+IAo+ID4gSW5zdGVhZCwgY2FuIHlvdSBwbGVh c2UgdHJ5IGNoYW5naW5nIHRoZSBhbHRlcm5hdGl2ZSB0byBhbHNvIHRha2UgdGhlIGNvbmZpZywK PiA+IGUuZy4gaGVyZSBoYXZlOgo+ID4gCj4gPiB8ICAgICAgIGFzbSB2b2xhdGlsZShBTFRFUk5B VElWRSgKPiA+IHwgICAgICAgICAgICAgICAibXNyICAgIGRhaWZjbHIsICMzICAgICAgICAgICAg IC8vIGFyY2hfbG9jYWxfaXJxX2VuYWJsZSIsCj4gPiB8ICAgICAgICAgICAgICAgX19tc3JfcyhT WVNfSUNDX1BNUl9FTDEsICIlMCIpLAo+ID4gfCAgICAgICAgICAgICAgIEFSTTY0X0hBU19JUlFf UFJJT19NQVNLSU5HLAo+ID4gfCAgICAgICAgICAgICAgIENPTkZJR19BUk02NF9QU0VVRE9fTk1J KQo+ID4gfCAgICAgICAgICAgICAgIDoKPiA+IHwgICAgICAgICAgICAgICA6ICJyIiAoKHVuc2ln bmVkIGxvbmcpIEdJQ19QUklPX0lSUU9OKQo+ID4gfCAgICAgICAgICAgICAgIDogIm1lbW9yeSIp Owo+ID4gCj4gPiAuLi4gYW5kIHNlZSBpZiB0aGF0IG1ha2VzIGEgc2lnbmlmaWNhbnQgZGlmZmVy ZW5jZT8KPiA+IAo+ID4gTGlrZXdpc2UgZm9yIHRoZSBvdGhlciBjYXNlZXMuCj4gCj4gT0ssIEkn bGwgdHJ5IGl0LiBCdXQgSSBoYXZlIHNvbWUgcXVlc3Rpb25zLiBIZXJlJ3MgdGhlIGNvbW1lbnQg b2YKPiBBTEVSTkFUSVZFOgo+IAo+IC8qCj4gwqAqIFVzYWdlOiBhc20oQUxURVJOQVRJVkUob2xk aW5zdHIsIG5ld2luc3RyLCBmZWF0dXJlKSk7Cj4gwqAqCj4gwqAqIFVzYWdlOiBhc20oQUxURVJO QVRJVkUob2xkaW5zdHIsIG5ld2luc3RyLCBmZWF0dXJlLCBDT05GSUdfRk9PKSk7Cj4gwqAqIE4u Qi4gSWYgQ09ORklHX0ZPTyBpcyBzcGVjaWZpZWQsIGJ1dCBub3Qgc2VsZWN0ZWQsIHRoZSB3aG9s ZSBibG9jawo+IMKgKsKgwqDCoMKgwqAgd2lsbCBiZSBvbWl0dGVkLCBpbmNsdWRpbmcgb2xkaW5z dHIuCj4gwqAqLwo+IAo+IElmIENPTkZJR19GT08gaXMgbm90IHNlbGVjdGVkLCB0aGUgd2hvbGUg YmxvY2sgd2lsbCBiZSBvbWl0dGVkIGluY2x1ZGluZwo+IG9sZGluc3RyLgo+IAo+IEJ1dCB3ZSBz dGlsbCB3YW50IHRoZSBvbGRpbnN0ciBpbiB0aGlzIHNpdHVhdGlvbi4gRG8gSSBtaXN1bmRlcnN0 YW5kCj4gc29tZXRoaW5nPwoKU29ycnk7IHlvdSBhcmUgcmlnaHQsIGFuZCBteSBzdWdnZXN0aW9u IHdhcyBicm9rZW4uCgpJJ2xsIG5lZWQgdG8gaGF2ZSBhIHRoaW5rIGFib3V0IHRoaXM7IHdlIG1p Z2h0IGJlIGFibGUgdG8gcmV3b3JrIHRoaXMgdG8gdXNlIGEKc3RhdGljIGtleSBpbnN0ZWFkLCBi dXQgSUlSQyBsYXN0IHRpbWUgSSB0cmllZCB0aGVyZSB3ZXJlIGlzc3VlcyB3aXRoIHRoYXQKYXBw cm9hY2guIEkgaGF2ZSBzb21lIChvbGQpIHdvcmstaW4tcHJvZ3Jlc3MgcGF0Y2hlcyBhdDoKCiAg aHR0cHM6Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvbWFyay9saW51 eC5naXQvbG9nLz9oPWFybTY0L2RhaWYtcmV3b3JrCgpJIHN1c3BlY3QgSSB3b24ndCBoYXZlIHRp bWUgdG8gcmVuZXcgdGhhdCBpbiB0aGUgbmVhciBmdXR1cmUsIGJ1dCBhbiBhcHByb2FjaApsaWtl IHRoYXQgbWlnaHQgYmUgd29ydGh3aGlsZS4KClRoYW5rcywKTWFyay4KCl9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGlu ZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMu aW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK