From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2835EC4332F for ; Thu, 15 Dec 2022 02:09:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229639AbiLOCJZ (ORCPT ); Wed, 14 Dec 2022 21:09:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229469AbiLOCJV (ORCPT ); Wed, 14 Dec 2022 21:09:21 -0500 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BCE536C42 for ; Wed, 14 Dec 2022 18:09:20 -0800 (PST) Received: by mail-qt1-x82e.google.com with SMTP id s9so4122453qtx.6 for ; Wed, 14 Dec 2022 18:09:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bluespec-com.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:from:to:cc:subject:date:message-id:reply-to; bh=4eDacVPeRCxSHKdGOkoS+IOwT3PA0un8lmYN5+fjPhw=; b=y7057BeXP8Mk6u2/+Eoxa5be8UoreoDppdS/r/OZICVloTQdOTwaMkQbq3qcq3ZOgS IhIc9GjFqKsHtXrK4K+zhdOUbdCmlIs+s6lLZk5zd5HuGWlS8c8mWM9f1vas+tE16DtT 7gnd9H/ihmf30zvyBXYUU7O8mhs2tfby4PkfVK09y+fXpJ2w0RiehMNk3voHmeYDDuab eyVpNKXt7TLSUFjaJs15VZer4Pb1tfWqaCPejg1d9QHfE5XhXKm0d0k2OkKOvPLzyR3T rCXCe84Bc23wJ1L+Q1kJTjNYjKuCRAodJb38tiQQ5ec3uWE0uk/HCZo0Z0xPeZ/Cl4Ag 3GlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4eDacVPeRCxSHKdGOkoS+IOwT3PA0un8lmYN5+fjPhw=; b=2g2pkbvo5Kb7LtInK5P6Ae3r21OMpPiuuNKJGtFqhUY7PwOUPxHN3kg0+/Au8h8mAa 20NOPOZTknx/XekDrMRqdGObIFIqYhMpWlFvExoY6ad5cjH7pLSy0mNLT2a9c5AHI6Pk gye+HswJvGIg96afOGRxYhkxu5uXNEZ6oY9tsP7bxhYKnVSTjnmA56J/CVVaSHJiuJFT oDOU5e3UUg6zm+rvcwbWBax7UQDHnwUKuTluFE/Yy44DCU9W4HGFxQrg8f9fUyuwmBmn pM351ZmQYud/aawsB9fcgV8FvNjxE20d3v8tqt2ASMMw6Ct6fT0tdiGxDGPbBN1qnA3T hJNA== X-Gm-Message-State: ANoB5pnMYYnqFQnjtHeYIhe6pRc3bB85sQgi4dgFLxnkQRqdoJSLSbsU FhOwGiUDfo5m6zK0Epjs1u/h X-Google-Smtp-Source: AA0mqf5I2BrheF1m6gxgerTJOGj7W6D5Syqpq0MvS6v7R0LmC1jB9uD9VC1LFaYtkT4QXreRi+4OLA== X-Received: by 2002:a05:622a:1a21:b0:3a7:e7e7:566 with SMTP id f33-20020a05622a1a2100b003a7e7e70566mr45474629qtb.66.1671070159263; Wed, 14 Dec 2022 18:09:19 -0800 (PST) Received: from bruce.bluespec.com ([38.70.11.124]) by smtp.gmail.com with ESMTPSA id h22-20020ac85496000000b0035cd6a4ba3csm2658868qtq.39.2022.12.14.18.09.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Dec 2022 18:09:18 -0800 (PST) Date: Wed, 14 Dec 2022 21:09:13 -0500 From: Darius Rad To: Vineet Gupta Cc: Palmer Dabbelt , fweimer@redhat.com, Andrew Waterman , stillson@rivosinc.com, Paul Walmsley , anup@brainfault.org, atishp@atishpatra.org, guoren@kernel.org, Conor Dooley , greentime.hu@sifive.com, vincent.chen@sifive.com, andy.chiu@sifive.com, arnd@kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, bjorn@kernel.org, libc-alpha@sourceware.org, christoph.muellner@vrull.eu, Aaron Durbin , linux@rivosinc.com Subject: Re: RISCV Vector unit disabled by default for new task (was Re: [PATCH v12 17/17] riscv: prctl to enable vector commands) Message-ID: Mail-Followup-To: Vineet Gupta , Palmer Dabbelt , fweimer@redhat.com, Andrew Waterman , stillson@rivosinc.com, Paul Walmsley , anup@brainfault.org, atishp@atishpatra.org, guoren@kernel.org, Conor Dooley , greentime.hu@sifive.com, vincent.chen@sifive.com, andy.chiu@sifive.com, arnd@kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, bjorn@kernel.org, libc-alpha@sourceware.org, christoph.muellner@vrull.eu, Aaron Durbin , linux@rivosinc.com References: <8fe9cfaf-2cbc-8de6-3928-067de9113bfc@rivosinc.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <8fe9cfaf-2cbc-8de6-3928-067de9113bfc@rivosinc.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Dec 14, 2022 at 12:07:03PM -0800, Vineet Gupta wrote: > On 12/13/22 08:43, Darius Rad wrote: > > On Fri, Dec 09, 2022 at 11:42:19AM -0800, Vineet Gupta wrote: > > > But keeping the V unit disabled by default and using prctl as a gatekeeper > > > to enable it feels unnecessary and tedious. > > > Here's my reasoning below (I'm collating comments from prior msgs as well). > > Please reference the previous discussion [1] which has covered topics that > > have not been discussed recently. > > > > [1] https://lists.infradead.org/pipermail/linux-riscv/2021-September/thread.html#8361 > > I sure read thru that thread, and many more :-) to get context. > The highlight is we should something because AVX/AMX do so (or failed to do > so). > But on the flip side ARM SVE is not disabling this by default. > Your other concerns seems to be potential power implications for leaving it > on and sharing of V unit across harts (see more on that below) > Maybe leaving it on all the time will be motivation for hw designers to be > more considerate of the idle power draw. > That is not quite the same take away I had from the AMX discussion. I would also disagree that ARM SVE is not disabling things by default, although the behavior is somewhat different. The significant point that I see from that discussion, which also applies to SVE, and also applies to RISC-V vector, is that the extension is necessarily dependant upon a functional unit that is reasonably large with respect to the size of the processor and has a significant amount of additional architectural state. The argument there is that AMX/SVE/RVV is a significant system resource and should be treated as such: by having the kernel track usage of it and by having a process specifically request access to it. For any of AMX/SVE/RVV, use of the extension operates as follows: 1. A process requests access to the extension, 2. The kernel allocates memory for the additional state for this process, 3. The kernel enables the extension for the process, and finally 4. The process is able to use the instructions. I don't recall the exact details, but my understanding is that AMX is going to use an x86 specific mechanism and require and explict request from user space to request access to AMX. For SVE, it is in fact disabled by default in the kernel. When a thread executes the first SVE instruction, it will cause an exception, the kernel will allocate memory for SVE state and enable TIF_SVE. Further use of SVE instructions will proceed without exceptions. Although SVE is disabled by default, it is enabled automatically. Since this is done automatically during an exception handler, there is no opportunity for memory allocation errors to be reported, as there are in the AMX case. For RVV, I do not recall ever seeing Linux patches that automatically enable vector. I have seen it enabled unconditionally, or with a manual enable (i.e., prctl). It is possible to write a program that does not ever use AMX, and when that program is run, the process will not incur the power or memory overhead of AMX. It is also possible to do that with SVE. This is simply not possible if RISC-V will, by default for every process, enable and allocate state memory for vector. So my thought would be what is the motivation for being even less flexible than SVE, if you feel that the AMX mechanism is too onerous? > > > > > 2. People want the prctl gatekeeping for ability to gracefully handle memory > > > allocation failure for the extra V-state within kernel. But that is only > > > additional 4K (for typical 128 wide V regs) per task. > > But vector state scales up to as much as 256k. Are you suggesting that > > there is no possibility that future systems would support more than > > VLEN=128? > > I mentioned "typical". And below also said that memory allocation concerns > are moot, since fork/execve failures due to failing to allocate would take > care of those anyways. > But again, what if one were using such an admittedly atypical system? Why should such a user be compelled to take a memory hit for every process, even if they specifically go out of their way to avoid using vector instructions? > > > If that is failing, > > > the system is not doing well anyways. Besides it is not an issue at all > > > since ENOMEM in clone/execve for the additional space should handle the > > > failure anyways. Only very sophisticated apps would downgrade from executing > > > V to Scalar code if the prctl failed. > > This seems unlikely. As vector support does not exist in any present > > hardware, and the vector extension is only optional in the RISC-V profiles > > that include it, I would think that it is almost certain that any > > application that supports V would have a fallback path for when the V > > extension is not available. > > For specialized cases sure we would expect fat binaries with IFUNC based > dispatches (glibc mem*/str* are obvious examples). But with newer compilers > autovec is increasing becoming default even at medium optimization levels > such as -O2. So V code littered all over is just a matter of time for the > profiles/variants which allow V. For less capable systems w/o V this is all > but moot discussion since kernel itself need not be built with V enabled. > To me, that seems like a lot of speculation, and certainly not worth relying on to limit functionality. > > > Another motivation for requiring that user space request use of the vector > > extension is that the vector unit may be shared between multiple harts > > and/or have power or performance implications in the system. By requiring > > that user space request access, it allows the system to decline that > > access, and user space can handle this gracefully. > > But in this specific example won't the prctl cause more pain. So 2 > concurrent processes on 2 different harts with shared V unit. One sends > prctl to enable and other wants to disable, what would the kernel do. Will > it just be whoever ends up running later wins. Granted I'm not too familiar > with how such a cross-hart sharing would work in a Vector instructions being > part of ISA  (vs. Vector accelerator with job push/pull approach) > I think you are misunderstanding both the mechanism and the potential implementation. The mechanism is not a switch per hart, it is a switch per process (or thread). It indicates to the kernel that this process (or thread) is using the resource, and the kernel will allocate memory, enable instructions, etc., as needed when that processes is scheduled on a hart. The theoretical implementation is something like what was done with floating point in the AMD Bulldozer architecture. All the sharing of resources is done automatically in hardware, but obviously performance measurements could demonstrate that each thread/hart does not have its own execution unit. I would imagine that in such an implementation, the vector unit would not be able to, for example, enter a low power mode unless both harts that depend on it are not using it. I would also imagine that single thread performance for vector would be optimized if only one thread were using the vector unit at a time. If the vector unit is enabled unconditionally by default for all processes, this is not possible. // darius From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72F0DC4332F for ; Thu, 15 Dec 2022 02:09:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IUHtvRVUTky0rTN6pDzC2ABKVCjxCeLZ8ymjzdhcHMI=; b=BFr7U6fxdOVvXq F6jw/uEhxwHa2EwLlvle4WgLlywPEgeAvTu9lkmXe4nTDbq6on3TrkvwmZoHz0C5PtENHapPi7NPZ M3YTmCy3OnHhRZA25eutnClYPsAenXMAcvN9gINJeQD4abhEfj4B7C1kCN4eH21uu9WR/oSZK73Ia INy5lXgsB+i++dwW3U6v5Qg+Kmuzzvw69/p1+tiofp38ApJ+BZPvmdojETnXRPgdVUxYzY8HE7om0 CGvXXTzjn2/reB0yERWV2OWfr4/e7xtFuMlAm9HpG+H9QqaWGUjXhlFWpFJPduojhlUWtry5/oqOO sm/RcnnYXFW72GIMXvrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1p5dgb-005cOJ-LD; Thu, 15 Dec 2022 02:09:25 +0000 Received: from mail-qt1-x82e.google.com ([2607:f8b0:4864:20::82e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1p5dgX-005cMi-K7 for linux-riscv@lists.infradead.org; Thu, 15 Dec 2022 02:09:23 +0000 Received: by mail-qt1-x82e.google.com with SMTP id ay32so4089224qtb.11 for ; Wed, 14 Dec 2022 18:09:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bluespec-com.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:from:to:cc:subject:date:message-id:reply-to; bh=4eDacVPeRCxSHKdGOkoS+IOwT3PA0un8lmYN5+fjPhw=; b=y7057BeXP8Mk6u2/+Eoxa5be8UoreoDppdS/r/OZICVloTQdOTwaMkQbq3qcq3ZOgS IhIc9GjFqKsHtXrK4K+zhdOUbdCmlIs+s6lLZk5zd5HuGWlS8c8mWM9f1vas+tE16DtT 7gnd9H/ihmf30zvyBXYUU7O8mhs2tfby4PkfVK09y+fXpJ2w0RiehMNk3voHmeYDDuab eyVpNKXt7TLSUFjaJs15VZer4Pb1tfWqaCPejg1d9QHfE5XhXKm0d0k2OkKOvPLzyR3T rCXCe84Bc23wJ1L+Q1kJTjNYjKuCRAodJb38tiQQ5ec3uWE0uk/HCZo0Z0xPeZ/Cl4Ag 3GlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4eDacVPeRCxSHKdGOkoS+IOwT3PA0un8lmYN5+fjPhw=; b=OGoJkVExM2tAAnwKOHOmZPmhuBJ/rAaBixd6aHRd8uBQh2CLlhWfrMP9NV0TwYmr5T ACq6YicvmlkBZeVYVJLEJCLOxfCHkJrPe2lg61ICG58UBablZG/BwHIUAq6gyJmLGQjA vxLvqKPogAfMtoKZ6NM59cID752gSkJLtHjY4p8VlO+vMW/pdKBRTW1RglbOpseB4FDC nBncnNuJF/LGCyr6TuwgQQxMT2B1BjxcLYIi8dYKmUfE8zI13Sw4+h6dcK4TYa1H5ruZ SBDKwM+jSFLmat0rgLLr+YnFqy7gXBe/umQQS88LxVGnn3ytsJUNXGBGOJ4IeOyQuk+c jpzg== X-Gm-Message-State: ANoB5pnNceVp3JtEGkwKg3lJzNTgLbCKEt1FyiL6vVMxK0dFCzNI9XA1 CBhzz9/DYk/oUeK4c9cP+pMAcCP40sDfqAuvVw== X-Google-Smtp-Source: AA0mqf5I2BrheF1m6gxgerTJOGj7W6D5Syqpq0MvS6v7R0LmC1jB9uD9VC1LFaYtkT4QXreRi+4OLA== X-Received: by 2002:a05:622a:1a21:b0:3a7:e7e7:566 with SMTP id f33-20020a05622a1a2100b003a7e7e70566mr45474629qtb.66.1671070159263; Wed, 14 Dec 2022 18:09:19 -0800 (PST) Received: from bruce.bluespec.com ([38.70.11.124]) by smtp.gmail.com with ESMTPSA id h22-20020ac85496000000b0035cd6a4ba3csm2658868qtq.39.2022.12.14.18.09.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Dec 2022 18:09:18 -0800 (PST) Date: Wed, 14 Dec 2022 21:09:13 -0500 From: Darius Rad To: Vineet Gupta Cc: Palmer Dabbelt , fweimer@redhat.com, Andrew Waterman , stillson@rivosinc.com, Paul Walmsley , anup@brainfault.org, atishp@atishpatra.org, guoren@kernel.org, Conor Dooley , greentime.hu@sifive.com, vincent.chen@sifive.com, andy.chiu@sifive.com, arnd@kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, bjorn@kernel.org, libc-alpha@sourceware.org, christoph.muellner@vrull.eu, Aaron Durbin , linux@rivosinc.com Subject: Re: RISCV Vector unit disabled by default for new task (was Re: [PATCH v12 17/17] riscv: prctl to enable vector commands) Message-ID: Mail-Followup-To: Vineet Gupta , Palmer Dabbelt , fweimer@redhat.com, Andrew Waterman , stillson@rivosinc.com, Paul Walmsley , anup@brainfault.org, atishp@atishpatra.org, guoren@kernel.org, Conor Dooley , greentime.hu@sifive.com, vincent.chen@sifive.com, andy.chiu@sifive.com, arnd@kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, bjorn@kernel.org, libc-alpha@sourceware.org, christoph.muellner@vrull.eu, Aaron Durbin , linux@rivosinc.com References: <8fe9cfaf-2cbc-8de6-3928-067de9113bfc@rivosinc.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <8fe9cfaf-2cbc-8de6-3928-067de9113bfc@rivosinc.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221214_180921_908053_25DDCA3A X-CRM114-Status: GOOD ( 51.43 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gV2VkLCBEZWMgMTQsIDIwMjIgYXQgMTI6MDc6MDNQTSAtMDgwMCwgVmluZWV0IEd1cHRhIHdy b3RlOgo+IE9uIDEyLzEzLzIyIDA4OjQzLCBEYXJpdXMgUmFkIHdyb3RlOgo+ID4gT24gRnJpLCBE ZWMgMDksIDIwMjIgYXQgMTE6NDI6MTlBTSAtMDgwMCwgVmluZWV0IEd1cHRhIHdyb3RlOgo+ID4g PiBCdXQga2VlcGluZyB0aGUgViB1bml0IGRpc2FibGVkIGJ5IGRlZmF1bHQgYW5kIHVzaW5nIHBy Y3RsIGFzIGEgZ2F0ZWtlZXBlcgo+ID4gPiB0byBlbmFibGUgaXQgZmVlbHMgdW5uZWNlc3Nhcnkg YW5kIHRlZGlvdXMuCj4gPiA+IEhlcmUncyBteSByZWFzb25pbmcgYmVsb3cgKEknbSBjb2xsYXRp bmcgY29tbWVudHMgZnJvbSBwcmlvciBtc2dzIGFzIHdlbGwpLgo+ID4gUGxlYXNlIHJlZmVyZW5j ZSB0aGUgcHJldmlvdXMgZGlzY3Vzc2lvbiBbMV0gd2hpY2ggaGFzIGNvdmVyZWQgdG9waWNzIHRo YXQKPiA+IGhhdmUgbm90IGJlZW4gZGlzY3Vzc2VkIHJlY2VudGx5Lgo+ID4gCj4gPiBbMV0gaHR0 cHM6Ly9saXN0cy5pbmZyYWRlYWQub3JnL3BpcGVybWFpbC9saW51eC1yaXNjdi8yMDIxLVNlcHRl bWJlci90aHJlYWQuaHRtbCM4MzYxCj4gCj4gSSBzdXJlIHJlYWQgdGhydSB0aGF0IHRocmVhZCwg YW5kIG1hbnkgbW9yZSA6LSkgdG8gZ2V0IGNvbnRleHQuCj4gVGhlIGhpZ2hsaWdodCBpcyB3ZSBz aG91bGQgc29tZXRoaW5nIGJlY2F1c2UgQVZYL0FNWCBkbyBzbyAob3IgZmFpbGVkIHRvIGRvCj4g c28pLgo+IEJ1dCBvbiB0aGUgZmxpcCBzaWRlIEFSTSBTVkUgaXMgbm90IGRpc2FibGluZyB0aGlz IGJ5IGRlZmF1bHQuCj4gWW91ciBvdGhlciBjb25jZXJucyBzZWVtcyB0byBiZSBwb3RlbnRpYWwg cG93ZXIgaW1wbGljYXRpb25zIGZvciBsZWF2aW5nIGl0Cj4gb24gYW5kIHNoYXJpbmcgb2YgViB1 bml0IGFjcm9zcyBoYXJ0cyAoc2VlIG1vcmUgb24gdGhhdCBiZWxvdykKPiBNYXliZSBsZWF2aW5n IGl0IG9uIGFsbCB0aGUgdGltZSB3aWxsIGJlIG1vdGl2YXRpb24gZm9yIGh3IGRlc2lnbmVycyB0 byBiZQo+IG1vcmUgY29uc2lkZXJhdGUgb2YgdGhlIGlkbGUgcG93ZXIgZHJhdy4KPiAKClRoYXQg aXMgbm90IHF1aXRlIHRoZSBzYW1lIHRha2UgYXdheSBJIGhhZCBmcm9tIHRoZSBBTVggZGlzY3Vz c2lvbi4gIEkKd291bGQgYWxzbyBkaXNhZ3JlZSB0aGF0IEFSTSBTVkUgaXMgbm90IGRpc2FibGlu ZyB0aGluZ3MgYnkgZGVmYXVsdCwKYWx0aG91Z2ggdGhlIGJlaGF2aW9yIGlzIHNvbWV3aGF0IGRp ZmZlcmVudC4KClRoZSBzaWduaWZpY2FudCBwb2ludCB0aGF0IEkgc2VlIGZyb20gdGhhdCBkaXNj dXNzaW9uLCB3aGljaCBhbHNvIGFwcGxpZXMKdG8gU1ZFLCBhbmQgYWxzbyBhcHBsaWVzIHRvIFJJ U0MtViB2ZWN0b3IsIGlzIHRoYXQgdGhlIGV4dGVuc2lvbiBpcwpuZWNlc3NhcmlseSBkZXBlbmRh bnQgdXBvbiBhIGZ1bmN0aW9uYWwgdW5pdCB0aGF0IGlzIHJlYXNvbmFibHkgbGFyZ2Ugd2l0aApy ZXNwZWN0IHRvIHRoZSBzaXplIG9mIHRoZSBwcm9jZXNzb3IgYW5kIGhhcyBhIHNpZ25pZmljYW50 IGFtb3VudCBvZgphZGRpdGlvbmFsIGFyY2hpdGVjdHVyYWwgc3RhdGUuICBUaGUgYXJndW1lbnQg dGhlcmUgaXMgdGhhdCBBTVgvU1ZFL1JWViBpcwphIHNpZ25pZmljYW50IHN5c3RlbSByZXNvdXJj ZSBhbmQgc2hvdWxkIGJlIHRyZWF0ZWQgYXMgc3VjaDogYnkgaGF2aW5nIHRoZQprZXJuZWwgdHJh Y2sgdXNhZ2Ugb2YgaXQgYW5kIGJ5IGhhdmluZyBhIHByb2Nlc3Mgc3BlY2lmaWNhbGx5IHJlcXVl c3QKYWNjZXNzIHRvIGl0LgoKRm9yIGFueSBvZiBBTVgvU1ZFL1JWViwgdXNlIG9mIHRoZSBleHRl bnNpb24gb3BlcmF0ZXMgYXMgZm9sbG93czoKCiAgMS4gQSBwcm9jZXNzIHJlcXVlc3RzIGFjY2Vz cyB0byB0aGUgZXh0ZW5zaW9uLAoKICAyLiBUaGUga2VybmVsIGFsbG9jYXRlcyBtZW1vcnkgZm9y IHRoZSBhZGRpdGlvbmFsIHN0YXRlIGZvciB0aGlzIHByb2Nlc3MsCgogIDMuIFRoZSBrZXJuZWwg ZW5hYmxlcyB0aGUgZXh0ZW5zaW9uIGZvciB0aGUgcHJvY2VzcywgYW5kIGZpbmFsbHkKCiAgNC4g VGhlIHByb2Nlc3MgaXMgYWJsZSB0byB1c2UgdGhlIGluc3RydWN0aW9ucy4KCkkgZG9uJ3QgcmVj YWxsIHRoZSBleGFjdCBkZXRhaWxzLCBidXQgbXkgdW5kZXJzdGFuZGluZyBpcyB0aGF0IEFNWCBp cyBnb2luZwp0byB1c2UgYW4geDg2IHNwZWNpZmljIG1lY2hhbmlzbSBhbmQgcmVxdWlyZSBhbmQg ZXhwbGljdCByZXF1ZXN0IGZyb20gdXNlcgpzcGFjZSB0byByZXF1ZXN0IGFjY2VzcyB0byBBTVgu CgpGb3IgU1ZFLCBpdCBpcyBpbiBmYWN0IGRpc2FibGVkIGJ5IGRlZmF1bHQgaW4gdGhlIGtlcm5l bC4gIFdoZW4gYSB0aHJlYWQKZXhlY3V0ZXMgdGhlIGZpcnN0IFNWRSBpbnN0cnVjdGlvbiwgaXQg d2lsbCBjYXVzZSBhbiBleGNlcHRpb24sIHRoZSBrZXJuZWwKd2lsbCBhbGxvY2F0ZSBtZW1vcnkg Zm9yIFNWRSBzdGF0ZSBhbmQgZW5hYmxlIFRJRl9TVkUuICBGdXJ0aGVyIHVzZSBvZiBTVkUKaW5z dHJ1Y3Rpb25zIHdpbGwgcHJvY2VlZCB3aXRob3V0IGV4Y2VwdGlvbnMuICBBbHRob3VnaCBTVkUg aXMgZGlzYWJsZWQgYnkKZGVmYXVsdCwgaXQgaXMgZW5hYmxlZCBhdXRvbWF0aWNhbGx5LiAgU2lu Y2UgdGhpcyBpcyBkb25lIGF1dG9tYXRpY2FsbHkKZHVyaW5nIGFuIGV4Y2VwdGlvbiBoYW5kbGVy LCB0aGVyZSBpcyBubyBvcHBvcnR1bml0eSBmb3IgbWVtb3J5IGFsbG9jYXRpb24KZXJyb3JzIHRv IGJlIHJlcG9ydGVkLCBhcyB0aGVyZSBhcmUgaW4gdGhlIEFNWCBjYXNlLgoKRm9yIFJWViwgSSBk byBub3QgcmVjYWxsIGV2ZXIgc2VlaW5nIExpbnV4IHBhdGNoZXMgdGhhdCBhdXRvbWF0aWNhbGx5 IGVuYWJsZQp2ZWN0b3IuICBJIGhhdmUgc2VlbiBpdCBlbmFibGVkIHVuY29uZGl0aW9uYWxseSwg b3Igd2l0aCBhIG1hbnVhbCBlbmFibGUKKGkuZS4sIHByY3RsKS4KCkl0IGlzIHBvc3NpYmxlIHRv IHdyaXRlIGEgcHJvZ3JhbSB0aGF0IGRvZXMgbm90IGV2ZXIgdXNlIEFNWCwgYW5kIHdoZW4gdGhh dApwcm9ncmFtIGlzIHJ1biwgdGhlIHByb2Nlc3Mgd2lsbCBub3QgaW5jdXIgdGhlIHBvd2VyIG9y IG1lbW9yeSBvdmVyaGVhZCBvZgpBTVguICBJdCBpcyBhbHNvIHBvc3NpYmxlIHRvIGRvIHRoYXQg d2l0aCBTVkUuICBUaGlzIGlzIHNpbXBseSBub3QgcG9zc2libGUKaWYgUklTQy1WIHdpbGwsIGJ5 IGRlZmF1bHQgZm9yIGV2ZXJ5IHByb2Nlc3MsIGVuYWJsZSBhbmQgYWxsb2NhdGUgc3RhdGUKbWVt b3J5IGZvciB2ZWN0b3IuCgpTbyBteSB0aG91Z2h0IHdvdWxkIGJlIHdoYXQgaXMgdGhlIG1vdGl2 YXRpb24gZm9yIGJlaW5nIGV2ZW4gbGVzcyBmbGV4aWJsZQp0aGFuIFNWRSwgaWYgeW91IGZlZWwg dGhhdCB0aGUgQU1YIG1lY2hhbmlzbSBpcyB0b28gb25lcm91cz8KCj4gPiAKPiA+ID4gMi4gUGVv cGxlIHdhbnQgdGhlIHByY3RsIGdhdGVrZWVwaW5nIGZvciBhYmlsaXR5IHRvIGdyYWNlZnVsbHkg aGFuZGxlIG1lbW9yeQo+ID4gPiBhbGxvY2F0aW9uIGZhaWx1cmUgZm9yIHRoZSBleHRyYSBWLXN0 YXRlIHdpdGhpbiBrZXJuZWwuIEJ1dCB0aGF0IGlzIG9ubHkKPiA+ID4gYWRkaXRpb25hbCA0SyAo Zm9yIHR5cGljYWwgMTI4IHdpZGUgViByZWdzKSBwZXIgdGFzay4KPiA+IEJ1dCB2ZWN0b3Igc3Rh dGUgc2NhbGVzIHVwIHRvIGFzIG11Y2ggYXMgMjU2ay4gIEFyZSB5b3Ugc3VnZ2VzdGluZyB0aGF0 Cj4gPiB0aGVyZSBpcyBubyBwb3NzaWJpbGl0eSB0aGF0IGZ1dHVyZSBzeXN0ZW1zIHdvdWxkIHN1 cHBvcnQgbW9yZSB0aGFuCj4gPiBWTEVOPTEyOD8KPiAKPiBJIG1lbnRpb25lZCAidHlwaWNhbCIu IEFuZCBiZWxvdyBhbHNvIHNhaWQgdGhhdCBtZW1vcnkgYWxsb2NhdGlvbiBjb25jZXJucwo+IGFy ZSBtb290LCBzaW5jZSBmb3JrL2V4ZWN2ZSBmYWlsdXJlcyBkdWUgdG8gZmFpbGluZyB0byBhbGxv Y2F0ZSB3b3VsZCB0YWtlCj4gY2FyZSBvZiB0aG9zZSBhbnl3YXlzLgo+IAoKQnV0IGFnYWluLCB3 aGF0IGlmIG9uZSB3ZXJlIHVzaW5nIHN1Y2ggYW4gYWRtaXR0ZWRseSBhdHlwaWNhbCBzeXN0ZW0/ ICBXaHkKc2hvdWxkIHN1Y2ggYSB1c2VyIGJlIGNvbXBlbGxlZCB0byB0YWtlIGEgbWVtb3J5IGhp dCBmb3IgZXZlcnkgcHJvY2VzcywKZXZlbiBpZiB0aGV5IHNwZWNpZmljYWxseSBnbyBvdXQgb2Yg dGhlaXIgd2F5IHRvIGF2b2lkIHVzaW5nIHZlY3RvcgppbnN0cnVjdGlvbnM/Cgo+ID4gPiBJZiB0 aGF0IGlzIGZhaWxpbmcsCj4gPiA+IHRoZSBzeXN0ZW0gaXMgbm90IGRvaW5nIHdlbGwgYW55d2F5 cy4gQmVzaWRlcyBpdCBpcyBub3QgYW4gaXNzdWUgYXQgYWxsCj4gPiA+IHNpbmNlIEVOT01FTSBp biBjbG9uZS9leGVjdmUgZm9yIHRoZSBhZGRpdGlvbmFsIHNwYWNlIHNob3VsZCBoYW5kbGUgdGhl Cj4gPiA+IGZhaWx1cmUgYW55d2F5cy4gT25seSB2ZXJ5IHNvcGhpc3RpY2F0ZWQgYXBwcyB3b3Vs ZCBkb3duZ3JhZGUgZnJvbSBleGVjdXRpbmcKPiA+ID4gViB0byBTY2FsYXIgY29kZSBpZiB0aGUg cHJjdGwgZmFpbGVkLgo+ID4gVGhpcyBzZWVtcyB1bmxpa2VseS4gIEFzIHZlY3RvciBzdXBwb3J0 IGRvZXMgbm90IGV4aXN0IGluIGFueSBwcmVzZW50Cj4gPiBoYXJkd2FyZSwgYW5kIHRoZSB2ZWN0 b3IgZXh0ZW5zaW9uIGlzIG9ubHkgb3B0aW9uYWwgaW4gdGhlIFJJU0MtViBwcm9maWxlcwo+ID4g dGhhdCBpbmNsdWRlIGl0LCBJIHdvdWxkIHRoaW5rIHRoYXQgaXQgaXMgYWxtb3N0IGNlcnRhaW4g dGhhdCBhbnkKPiA+IGFwcGxpY2F0aW9uIHRoYXQgc3VwcG9ydHMgViB3b3VsZCBoYXZlIGEgZmFs bGJhY2sgcGF0aCBmb3Igd2hlbiB0aGUgVgo+ID4gZXh0ZW5zaW9uIGlzIG5vdCBhdmFpbGFibGUu Cj4gCj4gRm9yIHNwZWNpYWxpemVkIGNhc2VzIHN1cmUgd2Ugd291bGQgZXhwZWN0IGZhdCBiaW5h cmllcyB3aXRoIElGVU5DIGJhc2VkCj4gZGlzcGF0Y2hlcyAoZ2xpYmMgbWVtKi9zdHIqIGFyZSBv YnZpb3VzIGV4YW1wbGVzKS4gQnV0IHdpdGggbmV3ZXIgY29tcGlsZXJzCj4gYXV0b3ZlYyBpcyBp bmNyZWFzaW5nIGJlY29taW5nIGRlZmF1bHQgZXZlbiBhdCBtZWRpdW0gb3B0aW1pemF0aW9uIGxl dmVscwo+IHN1Y2ggYXMgLU8yLiBTbyBWIGNvZGUgbGl0dGVyZWQgYWxsIG92ZXIgaXMganVzdCBh IG1hdHRlciBvZiB0aW1lIGZvciB0aGUKPiBwcm9maWxlcy92YXJpYW50cyB3aGljaCBhbGxvdyBW LiBGb3IgbGVzcyBjYXBhYmxlIHN5c3RlbXMgdy9vIFYgdGhpcyBpcyBhbGwKPiBidXQgbW9vdCBk aXNjdXNzaW9uIHNpbmNlIGtlcm5lbCBpdHNlbGYgbmVlZCBub3QgYmUgYnVpbHQgd2l0aCBWIGVu YWJsZWQuCj4gCgpUbyBtZSwgdGhhdCBzZWVtcyBsaWtlIGEgbG90IG9mIHNwZWN1bGF0aW9uLCBh bmQgY2VydGFpbmx5IG5vdCB3b3J0aApyZWx5aW5nIG9uIHRvIGxpbWl0IGZ1bmN0aW9uYWxpdHku Cgo+IAo+ID4gQW5vdGhlciBtb3RpdmF0aW9uIGZvciByZXF1aXJpbmcgdGhhdCB1c2VyIHNwYWNl IHJlcXVlc3QgdXNlIG9mIHRoZSB2ZWN0b3IKPiA+IGV4dGVuc2lvbiBpcyB0aGF0IHRoZSB2ZWN0 b3IgdW5pdCBtYXkgYmUgc2hhcmVkIGJldHdlZW4gbXVsdGlwbGUgaGFydHMKPiA+IGFuZC9vciBo YXZlIHBvd2VyIG9yIHBlcmZvcm1hbmNlIGltcGxpY2F0aW9ucyBpbiB0aGUgc3lzdGVtLiAgQnkg cmVxdWlyaW5nCj4gPiB0aGF0IHVzZXIgc3BhY2UgcmVxdWVzdCBhY2Nlc3MsIGl0IGFsbG93cyB0 aGUgc3lzdGVtIHRvIGRlY2xpbmUgdGhhdAo+ID4gYWNjZXNzLCBhbmQgdXNlciBzcGFjZSBjYW4g aGFuZGxlIHRoaXMgZ3JhY2VmdWxseS4KPiAKPiBCdXQgaW4gdGhpcyBzcGVjaWZpYyBleGFtcGxl IHdvbid0IHRoZSBwcmN0bCBjYXVzZSBtb3JlIHBhaW4uIFNvIDIKPiBjb25jdXJyZW50IHByb2Nl c3NlcyBvbiAyIGRpZmZlcmVudCBoYXJ0cyB3aXRoIHNoYXJlZCBWIHVuaXQuIE9uZSBzZW5kcwo+ IHByY3RsIHRvIGVuYWJsZSBhbmQgb3RoZXIgd2FudHMgdG8gZGlzYWJsZSwgd2hhdCB3b3VsZCB0 aGUga2VybmVsIGRvLiBXaWxsCj4gaXQganVzdCBiZSB3aG9ldmVyIGVuZHMgdXAgcnVubmluZyBs YXRlciB3aW5zLiBHcmFudGVkIEknbSBub3QgdG9vIGZhbWlsaWFyCj4gd2l0aCBob3cgc3VjaCBh IGNyb3NzLWhhcnQgc2hhcmluZyB3b3VsZCB3b3JrIGluIGEgVmVjdG9yIGluc3RydWN0aW9ucyBi ZWluZwo+IHBhcnQgb2YgSVNBwqAgKHZzLiBWZWN0b3IgYWNjZWxlcmF0b3Igd2l0aCBqb2IgcHVz aC9wdWxsIGFwcHJvYWNoKQo+IAoKSSB0aGluayB5b3UgYXJlIG1pc3VuZGVyc3RhbmRpbmcgYm90 aCB0aGUgbWVjaGFuaXNtIGFuZCB0aGUgcG90ZW50aWFsCmltcGxlbWVudGF0aW9uLgoKVGhlIG1l Y2hhbmlzbSBpcyBub3QgYSBzd2l0Y2ggcGVyIGhhcnQsIGl0IGlzIGEgc3dpdGNoIHBlciBwcm9j ZXNzIChvcgp0aHJlYWQpLiAgSXQgaW5kaWNhdGVzIHRvIHRoZSBrZXJuZWwgdGhhdCB0aGlzIHBy b2Nlc3MgKG9yIHRocmVhZCkgaXMgdXNpbmcKdGhlIHJlc291cmNlLCBhbmQgdGhlIGtlcm5lbCB3 aWxsIGFsbG9jYXRlIG1lbW9yeSwgZW5hYmxlIGluc3RydWN0aW9ucywKZXRjLiwgYXMgbmVlZGVk IHdoZW4gdGhhdCBwcm9jZXNzZXMgaXMgc2NoZWR1bGVkIG9uIGEgaGFydC4KClRoZSB0aGVvcmV0 aWNhbCBpbXBsZW1lbnRhdGlvbiBpcyBzb21ldGhpbmcgbGlrZSB3aGF0IHdhcyBkb25lIHdpdGgK ZmxvYXRpbmcgcG9pbnQgaW4gdGhlIEFNRCBCdWxsZG96ZXIgYXJjaGl0ZWN0dXJlLiAgQWxsIHRo ZSBzaGFyaW5nIG9mCnJlc291cmNlcyBpcyBkb25lIGF1dG9tYXRpY2FsbHkgaW4gaGFyZHdhcmUs IGJ1dCBvYnZpb3VzbHkgcGVyZm9ybWFuY2UKbWVhc3VyZW1lbnRzIGNvdWxkIGRlbW9uc3RyYXRl IHRoYXQgZWFjaCB0aHJlYWQvaGFydCBkb2VzIG5vdCBoYXZlIGl0cyBvd24KZXhlY3V0aW9uIHVu aXQuICBJIHdvdWxkIGltYWdpbmUgdGhhdCBpbiBzdWNoIGFuIGltcGxlbWVudGF0aW9uLCB0aGUg dmVjdG9yCnVuaXQgd291bGQgbm90IGJlIGFibGUgdG8sIGZvciBleGFtcGxlLCBlbnRlciBhIGxv dyBwb3dlciBtb2RlIHVubGVzcyBib3RoCmhhcnRzIHRoYXQgZGVwZW5kIG9uIGl0IGFyZSBub3Qg dXNpbmcgaXQuICBJIHdvdWxkIGFsc28gaW1hZ2luZSB0aGF0IHNpbmdsZQp0aHJlYWQgcGVyZm9y bWFuY2UgZm9yIHZlY3RvciB3b3VsZCBiZSBvcHRpbWl6ZWQgaWYgb25seSBvbmUgdGhyZWFkIHdl cmUKdXNpbmcgdGhlIHZlY3RvciB1bml0IGF0IGEgdGltZS4gIElmIHRoZSB2ZWN0b3IgdW5pdCBp cyBlbmFibGVkCnVuY29uZGl0aW9uYWxseSBieSBkZWZhdWx0IGZvciBhbGwgcHJvY2Vzc2VzLCB0 aGlzIGlzIG5vdCBwb3NzaWJsZS4KCi8vIGRhcml1cwoKCl9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LXJpc2N2IG1haWxpbmcgbGlzdApsaW51eC1y aXNjdkBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxt YW4vbGlzdGluZm8vbGludXgtcmlzY3YK