From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BC27C433E0 for ; Wed, 24 Feb 2021 08:49:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 482D960233 for ; Wed, 24 Feb 2021 08:49:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233234AbhBXItE (ORCPT ); Wed, 24 Feb 2021 03:49:04 -0500 Received: from ste-pvt-msa1.bahnhof.se ([213.80.101.70]:2119 "EHLO ste-pvt-msa1.bahnhof.se" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234163AbhBXIst (ORCPT ); Wed, 24 Feb 2021 03:48:49 -0500 Received: from localhost (localhost [127.0.0.1]) by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTP id B4E0E3F3A1; Wed, 24 Feb 2021 09:47:58 +0100 (CET) Authentication-Results: ste-pvt-msa1.bahnhof.se; dkim=pass (1024-bit key; unprotected) header.d=shipmail.org header.i=@shipmail.org header.b=cMx2eW2G; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at bahnhof.se Received: from ste-pvt-msa1.bahnhof.se ([127.0.0.1]) by localhost (ste-pvt-msa1.bahnhof.se [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xdBZ-0obx0Pt; Wed, 24 Feb 2021 09:47:57 +0100 (CET) Received: by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTPA id 850F03F311; Wed, 24 Feb 2021 09:47:53 +0100 (CET) Received: from [192.168.0.209] (h-205-35.A357.priv.bahnhof.se [155.4.205.35]) by mail1.shipmail.org (Postfix) with ESMTPSA id DD979360057; Wed, 24 Feb 2021 09:47:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=shipmail.org; s=mail; t=1614156472; bh=mM8r7vHLddSvuXQyaRkaso8763XSLKZHpHc7GB+CbQM=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=cMx2eW2G1SkQmupq9oGww2E8UZd1m2KVhl7ad7vV9aIjMXcDvp/J8k/4FaGyKqw23 OfEqT1ZM9evWy18fh15GVfJAiMWL+3jnfTrvyXtB/DHVeDzBKNTcfQELfL4djUhFTQ RGh3mfZS5vwd/2+i/DP+4B9MSEwu5LwJTcCQYQLI= Subject: Re: [Linaro-mm-sig] [PATCH] dma-fence: Document recoverable page fault implications To: Daniel Vetter , DRI Development Cc: Felix Kuehling , Maarten Lankhorst , linaro-mm-sig@lists.linaro.org, Jerome Glisse , =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= , Daniel Vetter , =?UTF-8?Q?Christian_K=c3=b6nig?= , linux-media@vger.kernel.org References: <20210203152921.2429937-1-daniel.vetter@ffwll.ch> From: =?UTF-8?Q?Thomas_Hellstr=c3=b6m_=28Intel=29?= Message-ID: <81df5b1c-2838-49d8-4ae4-bab4f55b411a@shipmail.org> Date: Wed, 24 Feb 2021 09:47:52 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: <20210203152921.2429937-1-daniel.vetter@ffwll.ch> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org On 2/3/21 4:29 PM, Daniel Vetter wrote: > Recently there was a fairly long thread about recoreable hardware page > faults, how they can deadlock, and what to do about that. > > While the discussion is still fresh I figured good time to try and > document the conclusions a bit. This documentation section explains > what's the potential problem, and the remedies we've discussed, > roughly ordered from best to worst. > > v2: Linus -> Linux typoe (Dave) > > v3: > - Make it clear drivers only need to implement one option (Christian) > - Make it clearer that implicit sync is out the window with exclusive > fences (Christian) > - Add the fairly theoretical option of segementing the memory (either > statically or through dynamic checks at runtime for which piece of > memory is managed how) and explain why it's not a great idea (Felix) > > References: https://lore.kernel.org/dri-devel/20210107030127.20393-1-Felix.Kuehling@amd.com/ > Cc: Dave Airlie > Cc: Maarten Lankhorst > Cc: Thomas Hellström > Cc: "Christian König" > Cc: Jerome Glisse > Cc: Felix Kuehling > Signed-off-by: Daniel Vetter > Cc: Sumit Semwal > Cc: linux-media@vger.kernel.org > Cc: linaro-mm-sig@lists.linaro.org > --- > Documentation/driver-api/dma-buf.rst | 76 ++++++++++++++++++++++++++++ > 1 file changed, 76 insertions(+) > > diff --git a/Documentation/driver-api/dma-buf.rst b/Documentation/driver-api/dma-buf.rst > index a2133d69872c..7f37ec30d9fd 100644 > --- a/Documentation/driver-api/dma-buf.rst > +++ b/Documentation/driver-api/dma-buf.rst > @@ -257,3 +257,79 @@ fences in the kernel. This means: > userspace is allowed to use userspace fencing or long running compute > workloads. This also means no implicit fencing for shared buffers in these > cases. > + > +Recoverable Hardware Page Faults Implications > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Modern hardware supports recoverable page faults, which has a lot of > +implications for DMA fences. > + > +First, a pending page fault obviously holds up the work that's running on the > +accelerator and a memory allocation is usually required to resolve the fault. > +But memory allocations are not allowed to gate completion of DMA fences, which > +means any workload using recoverable page faults cannot use DMA fences for > +synchronization. Synchronization fences controlled by userspace must be used > +instead. > + > +On GPUs this poses a problem, because current desktop compositor protocols on > +Linux rely on DMA fences, which means without an entirely new userspace stack > +built on top of userspace fences, they cannot benefit from recoverable page > +faults. Specifically this means implicit synchronization will not be possible. > +The exception is when page faults are only used as migration hints and never to > +on-demand fill a memory request. For now this means recoverable page > +faults on GPUs are limited to pure compute workloads. > + > +Furthermore GPUs usually have shared resources between the 3D rendering and > +compute side, like compute units or command submission engines. If both a 3D > +job with a DMA fence and a compute workload using recoverable page faults are > +pending they could deadlock: > + > +- The 3D workload might need to wait for the compute job to finish and release > + hardware resources first. > + > +- The compute workload might be stuck in a page fault, because the memory > + allocation is waiting for the DMA fence of the 3D workload to complete. > + > +There are a few options to prevent this problem, one of which drivers need to > +ensure: > + > +- Compute workloads can always be preempted, even when a page fault is pending > + and not yet repaired. Not all hardware supports this. > + > +- DMA fence workloads and workloads which need page fault handling have > + independent hardware resources to guarantee forward progress. This could be > + achieved through e.g. through dedicated engines and minimal compute unit > + reservations for DMA fence workloads. > + > +- The reservation approach could be further refined by only reserving the > + hardware resources for DMA fence workloads when they are in-flight. This must > + cover the time from when the DMA fence is visible to other threads up to > + moment when fence is completed through dma_fence_signal(). > + > +- As a last resort, if the hardware provides no useful reservation mechanics, > + all workloads must be flushed from the GPU when switching between jobs > + requiring DMA fences or jobs requiring page fault handling: This means all DMA > + fences must complete before a compute job with page fault handling can be > + inserted into the scheduler queue. And vice versa, before a DMA fence can be > + made visible anywhere in the system, all compute workloads must be preempted > + to guarantee all pending GPU page faults are flushed. > + > +- Only a fairly theoretical option would be to untangle these dependencies when > + allocating memory to repair hardware page faults, either through separate > + memory blocks or runtime tracking of the full dependency graph of all DMA > + fences. This results very wide impact on the kernel, since resolving the page > + on the CPU side can itself involve a page fault. It is much more feasible and > + robust to limit the impact of handling hardware page faults to the specific > + driver. > + > +Note that workloads that run on independent hardware like copy engines or other > +GPUs do not have any impact. This allows us to keep using DMA fences internally > +in the kernel even for resolving hardware page faults, e.g. by using copy > +engines to clear or copy memory needed to resolve the page fault. > + > +In some ways this page fault problem is a special case of the `Infinite DMA > +Fences` discussions: Infinite fences from compute workloads are allowed to > +depend on DMA fences, but not the other way around. And not even the page fault > +problem is new, because some other CPU thread in userspace might > +hit a page fault which holds up a userspace fence - supporting page faults on > +GPUs doesn't anything fundamentally new. To me, in general this looks good. One thing, though is that for a first time reader it might not be totally clear what's special with a compute workload. Perhaps some clarification? Also since the current cross-driver dma_fence locking order is 1) dma_resv -> 2) memory_allocation / reclaim -> 3) dma_fence_wait/critical And the locking order required for recoverable pagefault is a) dma_resv -> b) fence_wait/critical -> c) memory_allocation / reclaim (Possibly with a) and b) interchanged above, Is it possible to service a recoverable pagefault without taking the dma_resv lock?) It's clear that the fence critical section in b) is not compatible with the dma_fence wait in 3) and thus the memory restrictions are needed. But I think given the memory allocation restrictions for recoverable pagefaults I guess at some point we must ask ourselves why are they necessary and what's the price to be paid for getting rid of them, and document also that. *If* it's the case that it all boils down to the 2) -> 3) locking order above, and that's mandated *only* by the dma_fence wait in the userptr mmu notifiers, I think these restrictions are a pretty high price to pay. Wouldn't it be possible now to replace that fence wait with either page pinning (which now is coherent since 5.9) or preempt-ctx fences + unpinned pages if available and thus invert the 2) -> 3) locking order? Thanks, Thomas From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAE96C433E0 for ; Wed, 24 Feb 2021 08:48:03 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 23B6060233 for ; Wed, 24 Feb 2021 08:48:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 23B6060233 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shipmail.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 557AB6E8C6; Wed, 24 Feb 2021 08:48:02 +0000 (UTC) Received: from ste-pvt-msa1.bahnhof.se (ste-pvt-msa1.bahnhof.se [213.80.101.70]) by gabe.freedesktop.org (Postfix) with ESMTPS id 74C9C6E8C6 for ; Wed, 24 Feb 2021 08:48:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTP id B4E0E3F3A1; Wed, 24 Feb 2021 09:47:58 +0100 (CET) Authentication-Results: ste-pvt-msa1.bahnhof.se; dkim=pass (1024-bit key; unprotected) header.d=shipmail.org header.i=@shipmail.org header.b=cMx2eW2G; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at bahnhof.se Received: from ste-pvt-msa1.bahnhof.se ([127.0.0.1]) by localhost (ste-pvt-msa1.bahnhof.se [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xdBZ-0obx0Pt; Wed, 24 Feb 2021 09:47:57 +0100 (CET) Received: by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTPA id 850F03F311; Wed, 24 Feb 2021 09:47:53 +0100 (CET) Received: from [192.168.0.209] (h-205-35.A357.priv.bahnhof.se [155.4.205.35]) by mail1.shipmail.org (Postfix) with ESMTPSA id DD979360057; Wed, 24 Feb 2021 09:47:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=shipmail.org; s=mail; t=1614156472; bh=mM8r7vHLddSvuXQyaRkaso8763XSLKZHpHc7GB+CbQM=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=cMx2eW2G1SkQmupq9oGww2E8UZd1m2KVhl7ad7vV9aIjMXcDvp/J8k/4FaGyKqw23 OfEqT1ZM9evWy18fh15GVfJAiMWL+3jnfTrvyXtB/DHVeDzBKNTcfQELfL4djUhFTQ RGh3mfZS5vwd/2+i/DP+4B9MSEwu5LwJTcCQYQLI= Subject: Re: [Linaro-mm-sig] [PATCH] dma-fence: Document recoverable page fault implications To: Daniel Vetter , DRI Development References: <20210203152921.2429937-1-daniel.vetter@ffwll.ch> From: =?UTF-8?Q?Thomas_Hellstr=c3=b6m_=28Intel=29?= Message-ID: <81df5b1c-2838-49d8-4ae4-bab4f55b411a@shipmail.org> Date: Wed, 24 Feb 2021 09:47:52 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: <20210203152921.2429937-1-daniel.vetter@ffwll.ch> Content-Language: en-US X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Felix Kuehling , linaro-mm-sig@lists.linaro.org, Jerome Glisse , =?UTF-8?Q?Thomas_Hellstr=c3=b6m?= , Daniel Vetter , =?UTF-8?Q?Christian_K=c3=b6nig?= , linux-media@vger.kernel.org Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Ck9uIDIvMy8yMSA0OjI5IFBNLCBEYW5pZWwgVmV0dGVyIHdyb3RlOgo+IFJlY2VudGx5IHRoZXJl IHdhcyBhIGZhaXJseSBsb25nIHRocmVhZCBhYm91dCByZWNvcmVhYmxlIGhhcmR3YXJlIHBhZ2UK PiBmYXVsdHMsIGhvdyB0aGV5IGNhbiBkZWFkbG9jaywgYW5kIHdoYXQgdG8gZG8gYWJvdXQgdGhh dC4KPgo+IFdoaWxlIHRoZSBkaXNjdXNzaW9uIGlzIHN0aWxsIGZyZXNoIEkgZmlndXJlZCBnb29k IHRpbWUgdG8gdHJ5IGFuZAo+IGRvY3VtZW50IHRoZSBjb25jbHVzaW9ucyBhIGJpdC4gVGhpcyBk b2N1bWVudGF0aW9uIHNlY3Rpb24gZXhwbGFpbnMKPiB3aGF0J3MgdGhlIHBvdGVudGlhbCBwcm9i bGVtLCBhbmQgdGhlIHJlbWVkaWVzIHdlJ3ZlIGRpc2N1c3NlZCwKPiByb3VnaGx5IG9yZGVyZWQg ZnJvbSBiZXN0IHRvIHdvcnN0Lgo+Cj4gdjI6IExpbnVzIC0+IExpbnV4IHR5cG9lIChEYXZlKQo+ Cj4gdjM6Cj4gLSBNYWtlIGl0IGNsZWFyIGRyaXZlcnMgb25seSBuZWVkIHRvIGltcGxlbWVudCBv bmUgb3B0aW9uIChDaHJpc3RpYW4pCj4gLSBNYWtlIGl0IGNsZWFyZXIgdGhhdCBpbXBsaWNpdCBz eW5jIGlzIG91dCB0aGUgd2luZG93IHdpdGggZXhjbHVzaXZlCj4gICAgZmVuY2VzIChDaHJpc3Rp YW4pCj4gLSBBZGQgdGhlIGZhaXJseSB0aGVvcmV0aWNhbCBvcHRpb24gb2Ygc2VnZW1lbnRpbmcg dGhlIG1lbW9yeSAoZWl0aGVyCj4gICAgc3RhdGljYWxseSBvciB0aHJvdWdoIGR5bmFtaWMgY2hl Y2tzIGF0IHJ1bnRpbWUgZm9yIHdoaWNoIHBpZWNlIG9mCj4gICAgbWVtb3J5IGlzIG1hbmFnZWQg aG93KSBhbmQgZXhwbGFpbiB3aHkgaXQncyBub3QgYSBncmVhdCBpZGVhIChGZWxpeCkKPgo+IFJl ZmVyZW5jZXM6IGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2RyaS1kZXZlbC8yMDIxMDEwNzAzMDEy Ny4yMDM5My0xLUZlbGl4Lkt1ZWhsaW5nQGFtZC5jb20vCj4gQ2M6IERhdmUgQWlybGllIDxhaXJs aWVkQGdtYWlsLmNvbT4KPiBDYzogTWFhcnRlbiBMYW5raG9yc3QgPG1hYXJ0ZW4ubGFua2hvcnN0 QGxpbnV4LmludGVsLmNvbT4KPiBDYzogVGhvbWFzIEhlbGxzdHLDtm0gPHRob21hcy5oZWxsc3Ry b21AaW50ZWwuY29tPgo+IENjOiAiQ2hyaXN0aWFuIEvDtm5pZyIgPGNocmlzdGlhbi5rb2VuaWdA YW1kLmNvbT4KPiBDYzogSmVyb21lIEdsaXNzZSA8amdsaXNzZUByZWRoYXQuY29tPgo+IENjOiBG ZWxpeCBLdWVobGluZyA8ZmVsaXgua3VlaGxpbmdAYW1kLmNvbT4KPiBTaWduZWQtb2ZmLWJ5OiBE YW5pZWwgVmV0dGVyIDxkYW5pZWwudmV0dGVyQGludGVsLmNvbT4KPiBDYzogU3VtaXQgU2Vtd2Fs IDxzdW1pdC5zZW13YWxAbGluYXJvLm9yZz4KPiBDYzogbGludXgtbWVkaWFAdmdlci5rZXJuZWwu b3JnCj4gQ2M6IGxpbmFyby1tbS1zaWdAbGlzdHMubGluYXJvLm9yZwo+IC0tLQo+ICAgRG9jdW1l bnRhdGlvbi9kcml2ZXItYXBpL2RtYS1idWYucnN0IHwgNzYgKysrKysrKysrKysrKysrKysrKysr KysrKysrKwo+ICAgMSBmaWxlIGNoYW5nZWQsIDc2IGluc2VydGlvbnMoKykKPgo+IGRpZmYgLS1n aXQgYS9Eb2N1bWVudGF0aW9uL2RyaXZlci1hcGkvZG1hLWJ1Zi5yc3QgYi9Eb2N1bWVudGF0aW9u L2RyaXZlci1hcGkvZG1hLWJ1Zi5yc3QKPiBpbmRleCBhMjEzM2Q2OTg3MmMuLjdmMzdlYzMwZDlm ZCAxMDA2NDQKPiAtLS0gYS9Eb2N1bWVudGF0aW9uL2RyaXZlci1hcGkvZG1hLWJ1Zi5yc3QKPiAr KysgYi9Eb2N1bWVudGF0aW9uL2RyaXZlci1hcGkvZG1hLWJ1Zi5yc3QKPiBAQCAtMjU3LDMgKzI1 Nyw3OSBAQCBmZW5jZXMgaW4gdGhlIGtlcm5lbC4gVGhpcyBtZWFuczoKPiAgICAgdXNlcnNwYWNl IGlzIGFsbG93ZWQgdG8gdXNlIHVzZXJzcGFjZSBmZW5jaW5nIG9yIGxvbmcgcnVubmluZyBjb21w dXRlCj4gICAgIHdvcmtsb2Fkcy4gVGhpcyBhbHNvIG1lYW5zIG5vIGltcGxpY2l0IGZlbmNpbmcg Zm9yIHNoYXJlZCBidWZmZXJzIGluIHRoZXNlCj4gICAgIGNhc2VzLgo+ICsKPiArUmVjb3ZlcmFi bGUgSGFyZHdhcmUgUGFnZSBGYXVsdHMgSW1wbGljYXRpb25zCj4gK35+fn5+fn5+fn5+fn5+fn5+ fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fgo+ICsKPiArTW9kZXJuIGhhcmR3YXJlIHN1cHBv cnRzIHJlY292ZXJhYmxlIHBhZ2UgZmF1bHRzLCB3aGljaCBoYXMgYSBsb3Qgb2YKPiAraW1wbGlj YXRpb25zIGZvciBETUEgZmVuY2VzLgo+ICsKPiArRmlyc3QsIGEgcGVuZGluZyBwYWdlIGZhdWx0 IG9idmlvdXNseSBob2xkcyB1cCB0aGUgd29yayB0aGF0J3MgcnVubmluZyBvbiB0aGUKPiArYWNj ZWxlcmF0b3IgYW5kIGEgbWVtb3J5IGFsbG9jYXRpb24gaXMgdXN1YWxseSByZXF1aXJlZCB0byBy ZXNvbHZlIHRoZSBmYXVsdC4KPiArQnV0IG1lbW9yeSBhbGxvY2F0aW9ucyBhcmUgbm90IGFsbG93 ZWQgdG8gZ2F0ZSBjb21wbGV0aW9uIG9mIERNQSBmZW5jZXMsIHdoaWNoCj4gK21lYW5zIGFueSB3 b3JrbG9hZCB1c2luZyByZWNvdmVyYWJsZSBwYWdlIGZhdWx0cyBjYW5ub3QgdXNlIERNQSBmZW5j ZXMgZm9yCj4gK3N5bmNocm9uaXphdGlvbi4gU3luY2hyb25pemF0aW9uIGZlbmNlcyBjb250cm9s bGVkIGJ5IHVzZXJzcGFjZSBtdXN0IGJlIHVzZWQKPiAraW5zdGVhZC4KPiArCj4gK09uIEdQVXMg dGhpcyBwb3NlcyBhIHByb2JsZW0sIGJlY2F1c2UgY3VycmVudCBkZXNrdG9wIGNvbXBvc2l0b3Ig cHJvdG9jb2xzIG9uCj4gK0xpbnV4IHJlbHkgb24gRE1BIGZlbmNlcywgd2hpY2ggbWVhbnMgd2l0 aG91dCBhbiBlbnRpcmVseSBuZXcgdXNlcnNwYWNlIHN0YWNrCj4gK2J1aWx0IG9uIHRvcCBvZiB1 c2Vyc3BhY2UgZmVuY2VzLCB0aGV5IGNhbm5vdCBiZW5lZml0IGZyb20gcmVjb3ZlcmFibGUgcGFn ZQo+ICtmYXVsdHMuIFNwZWNpZmljYWxseSB0aGlzIG1lYW5zIGltcGxpY2l0IHN5bmNocm9uaXph dGlvbiB3aWxsIG5vdCBiZSBwb3NzaWJsZS4KPiArVGhlIGV4Y2VwdGlvbiBpcyB3aGVuIHBhZ2Ug ZmF1bHRzIGFyZSBvbmx5IHVzZWQgYXMgbWlncmF0aW9uIGhpbnRzIGFuZCBuZXZlciB0bwo+ICtv bi1kZW1hbmQgZmlsbCBhIG1lbW9yeSByZXF1ZXN0LiBGb3Igbm93IHRoaXMgbWVhbnMgcmVjb3Zl cmFibGUgcGFnZQo+ICtmYXVsdHMgb24gR1BVcyBhcmUgbGltaXRlZCB0byBwdXJlIGNvbXB1dGUg d29ya2xvYWRzLgo+ICsKPiArRnVydGhlcm1vcmUgR1BVcyB1c3VhbGx5IGhhdmUgc2hhcmVkIHJl c291cmNlcyBiZXR3ZWVuIHRoZSAzRCByZW5kZXJpbmcgYW5kCj4gK2NvbXB1dGUgc2lkZSwgbGlr ZSBjb21wdXRlIHVuaXRzIG9yIGNvbW1hbmQgc3VibWlzc2lvbiBlbmdpbmVzLiBJZiBib3RoIGEg M0QKPiAram9iIHdpdGggYSBETUEgZmVuY2UgYW5kIGEgY29tcHV0ZSB3b3JrbG9hZCB1c2luZyBy ZWNvdmVyYWJsZSBwYWdlIGZhdWx0cyBhcmUKPiArcGVuZGluZyB0aGV5IGNvdWxkIGRlYWRsb2Nr Ogo+ICsKPiArLSBUaGUgM0Qgd29ya2xvYWQgbWlnaHQgbmVlZCB0byB3YWl0IGZvciB0aGUgY29t cHV0ZSBqb2IgdG8gZmluaXNoIGFuZCByZWxlYXNlCj4gKyAgaGFyZHdhcmUgcmVzb3VyY2VzIGZp cnN0Lgo+ICsKPiArLSBUaGUgY29tcHV0ZSB3b3JrbG9hZCBtaWdodCBiZSBzdHVjayBpbiBhIHBh Z2UgZmF1bHQsIGJlY2F1c2UgdGhlIG1lbW9yeQo+ICsgIGFsbG9jYXRpb24gaXMgd2FpdGluZyBm b3IgdGhlIERNQSBmZW5jZSBvZiB0aGUgM0Qgd29ya2xvYWQgdG8gY29tcGxldGUuCj4gKwo+ICtU aGVyZSBhcmUgYSBmZXcgb3B0aW9ucyB0byBwcmV2ZW50IHRoaXMgcHJvYmxlbSwgb25lIG9mIHdo aWNoIGRyaXZlcnMgbmVlZCB0bwo+ICtlbnN1cmU6Cj4gKwo+ICstIENvbXB1dGUgd29ya2xvYWRz IGNhbiBhbHdheXMgYmUgcHJlZW1wdGVkLCBldmVuIHdoZW4gYSBwYWdlIGZhdWx0IGlzIHBlbmRp bmcKPiArICBhbmQgbm90IHlldCByZXBhaXJlZC4gTm90IGFsbCBoYXJkd2FyZSBzdXBwb3J0cyB0 aGlzLgo+ICsKPiArLSBETUEgZmVuY2Ugd29ya2xvYWRzIGFuZCB3b3JrbG9hZHMgd2hpY2ggbmVl ZCBwYWdlIGZhdWx0IGhhbmRsaW5nIGhhdmUKPiArICBpbmRlcGVuZGVudCBoYXJkd2FyZSByZXNv dXJjZXMgdG8gZ3VhcmFudGVlIGZvcndhcmQgcHJvZ3Jlc3MuIFRoaXMgY291bGQgYmUKPiArICBh Y2hpZXZlZCB0aHJvdWdoIGUuZy4gdGhyb3VnaCBkZWRpY2F0ZWQgZW5naW5lcyBhbmQgbWluaW1h bCBjb21wdXRlIHVuaXQKPiArICByZXNlcnZhdGlvbnMgZm9yIERNQSBmZW5jZSB3b3JrbG9hZHMu Cj4gKwo+ICstIFRoZSByZXNlcnZhdGlvbiBhcHByb2FjaCBjb3VsZCBiZSBmdXJ0aGVyIHJlZmlu ZWQgYnkgb25seSByZXNlcnZpbmcgdGhlCj4gKyAgaGFyZHdhcmUgcmVzb3VyY2VzIGZvciBETUEg ZmVuY2Ugd29ya2xvYWRzIHdoZW4gdGhleSBhcmUgaW4tZmxpZ2h0LiBUaGlzIG11c3QKPiArICBj b3ZlciB0aGUgdGltZSBmcm9tIHdoZW4gdGhlIERNQSBmZW5jZSBpcyB2aXNpYmxlIHRvIG90aGVy IHRocmVhZHMgdXAgdG8KPiArICBtb21lbnQgd2hlbiBmZW5jZSBpcyBjb21wbGV0ZWQgdGhyb3Vn aCBkbWFfZmVuY2Vfc2lnbmFsKCkuCj4gKwo+ICstIEFzIGEgbGFzdCByZXNvcnQsIGlmIHRoZSBo YXJkd2FyZSBwcm92aWRlcyBubyB1c2VmdWwgcmVzZXJ2YXRpb24gbWVjaGFuaWNzLAo+ICsgIGFs bCB3b3JrbG9hZHMgbXVzdCBiZSBmbHVzaGVkIGZyb20gdGhlIEdQVSB3aGVuIHN3aXRjaGluZyBi ZXR3ZWVuIGpvYnMKPiArICByZXF1aXJpbmcgRE1BIGZlbmNlcyBvciBqb2JzIHJlcXVpcmluZyBw YWdlIGZhdWx0IGhhbmRsaW5nOiBUaGlzIG1lYW5zIGFsbCBETUEKPiArICBmZW5jZXMgbXVzdCBj b21wbGV0ZSBiZWZvcmUgYSBjb21wdXRlIGpvYiB3aXRoIHBhZ2UgZmF1bHQgaGFuZGxpbmcgY2Fu IGJlCj4gKyAgaW5zZXJ0ZWQgaW50byB0aGUgc2NoZWR1bGVyIHF1ZXVlLiBBbmQgdmljZSB2ZXJz YSwgYmVmb3JlIGEgRE1BIGZlbmNlIGNhbiBiZQo+ICsgIG1hZGUgdmlzaWJsZSBhbnl3aGVyZSBp biB0aGUgc3lzdGVtLCBhbGwgY29tcHV0ZSB3b3JrbG9hZHMgbXVzdCBiZSBwcmVlbXB0ZWQKPiAr ICB0byBndWFyYW50ZWUgYWxsIHBlbmRpbmcgR1BVIHBhZ2UgZmF1bHRzIGFyZSBmbHVzaGVkLgo+ ICsKPiArLSBPbmx5IGEgZmFpcmx5IHRoZW9yZXRpY2FsIG9wdGlvbiB3b3VsZCBiZSB0byB1bnRh bmdsZSB0aGVzZSBkZXBlbmRlbmNpZXMgd2hlbgo+ICsgIGFsbG9jYXRpbmcgbWVtb3J5IHRvIHJl cGFpciBoYXJkd2FyZSBwYWdlIGZhdWx0cywgZWl0aGVyIHRocm91Z2ggc2VwYXJhdGUKPiArICBt ZW1vcnkgYmxvY2tzIG9yIHJ1bnRpbWUgdHJhY2tpbmcgb2YgdGhlIGZ1bGwgZGVwZW5kZW5jeSBn cmFwaCBvZiBhbGwgRE1BCj4gKyAgZmVuY2VzLiBUaGlzIHJlc3VsdHMgdmVyeSB3aWRlIGltcGFj dCBvbiB0aGUga2VybmVsLCBzaW5jZSByZXNvbHZpbmcgdGhlIHBhZ2UKPiArICBvbiB0aGUgQ1BV IHNpZGUgY2FuIGl0c2VsZiBpbnZvbHZlIGEgcGFnZSBmYXVsdC4gSXQgaXMgbXVjaCBtb3JlIGZl YXNpYmxlIGFuZAo+ICsgIHJvYnVzdCB0byBsaW1pdCB0aGUgaW1wYWN0IG9mIGhhbmRsaW5nIGhh cmR3YXJlIHBhZ2UgZmF1bHRzIHRvIHRoZSBzcGVjaWZpYwo+ICsgIGRyaXZlci4KPiArCj4gK05v dGUgdGhhdCB3b3JrbG9hZHMgdGhhdCBydW4gb24gaW5kZXBlbmRlbnQgaGFyZHdhcmUgbGlrZSBj b3B5IGVuZ2luZXMgb3Igb3RoZXIKPiArR1BVcyBkbyBub3QgaGF2ZSBhbnkgaW1wYWN0LiBUaGlz IGFsbG93cyB1cyB0byBrZWVwIHVzaW5nIERNQSBmZW5jZXMgaW50ZXJuYWxseQo+ICtpbiB0aGUg a2VybmVsIGV2ZW4gZm9yIHJlc29sdmluZyBoYXJkd2FyZSBwYWdlIGZhdWx0cywgZS5nLiBieSB1 c2luZyBjb3B5Cj4gK2VuZ2luZXMgdG8gY2xlYXIgb3IgY29weSBtZW1vcnkgbmVlZGVkIHRvIHJl c29sdmUgdGhlIHBhZ2UgZmF1bHQuCj4gKwo+ICtJbiBzb21lIHdheXMgdGhpcyBwYWdlIGZhdWx0 IHByb2JsZW0gaXMgYSBzcGVjaWFsIGNhc2Ugb2YgdGhlIGBJbmZpbml0ZSBETUEKPiArRmVuY2Vz YCBkaXNjdXNzaW9uczogSW5maW5pdGUgZmVuY2VzIGZyb20gY29tcHV0ZSB3b3JrbG9hZHMgYXJl IGFsbG93ZWQgdG8KPiArZGVwZW5kIG9uIERNQSBmZW5jZXMsIGJ1dCBub3QgdGhlIG90aGVyIHdh eSBhcm91bmQuIEFuZCBub3QgZXZlbiB0aGUgcGFnZSBmYXVsdAo+ICtwcm9ibGVtIGlzIG5ldywg YmVjYXVzZSBzb21lIG90aGVyIENQVSB0aHJlYWQgaW4gdXNlcnNwYWNlIG1pZ2h0Cj4gK2hpdCBh IHBhZ2UgZmF1bHQgd2hpY2ggaG9sZHMgdXAgYSB1c2Vyc3BhY2UgZmVuY2UgLSBzdXBwb3J0aW5n IHBhZ2UgZmF1bHRzIG9uCj4gK0dQVXMgZG9lc24ndCBhbnl0aGluZyBmdW5kYW1lbnRhbGx5IG5l dy4KClRvIG1lLCBpbiBnZW5lcmFsIHRoaXMgbG9va3MgZ29vZC4gT25lIHRoaW5nLCB0aG91Z2gg aXMgdGhhdCBmb3IgYSBmaXJzdCAKdGltZSByZWFkZXIgaXQgbWlnaHQgbm90IGJlIHRvdGFsbHkg Y2xlYXIgd2hhdCdzIHNwZWNpYWwgd2l0aCBhIGNvbXB1dGUgCndvcmtsb2FkLiBQZXJoYXBzIHNv bWUgY2xhcmlmaWNhdGlvbj8KCkFsc28gc2luY2UgdGhlIGN1cnJlbnQgY3Jvc3MtZHJpdmVyIGRt YV9mZW5jZSBsb2NraW5nIG9yZGVyIGlzCgoxKSBkbWFfcmVzdiAtPgoyKSBtZW1vcnlfYWxsb2Nh dGlvbiAvIHJlY2xhaW0gLT4KMykgZG1hX2ZlbmNlX3dhaXQvY3JpdGljYWwKCkFuZCB0aGUgbG9j a2luZyBvcmRlciByZXF1aXJlZCBmb3IgcmVjb3ZlcmFibGUgcGFnZWZhdWx0IGlzCgphKSBkbWFf cmVzdiAtPgpiKSBmZW5jZV93YWl0L2NyaXRpY2FsIC0+CmMpIG1lbW9yeV9hbGxvY2F0aW9uIC8g cmVjbGFpbQoKKFBvc3NpYmx5IHdpdGggYSkgYW5kIGIpIGludGVyY2hhbmdlZCBhYm92ZSwgSXMg aXQgcG9zc2libGUgdG8gc2VydmljZSBhIApyZWNvdmVyYWJsZSBwYWdlZmF1bHQgd2l0aG91dCB0 YWtpbmcgdGhlIGRtYV9yZXN2IGxvY2s/KQoKSXQncyBjbGVhciB0aGF0IHRoZSBmZW5jZSBjcml0 aWNhbCBzZWN0aW9uIGluIGIpIGlzIG5vdCBjb21wYXRpYmxlIHdpdGggCnRoZSBkbWFfZmVuY2Ug d2FpdCBpbiAzKSBhbmQgdGh1cyB0aGUgbWVtb3J5IHJlc3RyaWN0aW9ucyBhcmUgbmVlZGVkLiAK QnV0IEkgdGhpbmsgZ2l2ZW4gdGhlIG1lbW9yeSBhbGxvY2F0aW9uIHJlc3RyaWN0aW9ucyBmb3Ig cmVjb3ZlcmFibGUgCnBhZ2VmYXVsdHMgSSBndWVzcyBhdCBzb21lIHBvaW50IHdlIG11c3QgYXNr IG91cnNlbHZlcyB3aHkgYXJlIHRoZXkgCm5lY2Vzc2FyeSBhbmQgd2hhdCdzIHRoZSBwcmljZSB0 byBiZSBwYWlkIGZvciBnZXR0aW5nIHJpZCBvZiB0aGVtLCBhbmQgCmRvY3VtZW50IGFsc28gdGhh dC4gKklmKiBpdCdzIHRoZSBjYXNlIHRoYXQgaXQgYWxsIGJvaWxzIGRvd24gdG8gdGhlIDIpIAot PiAzKSBsb2NraW5nIG9yZGVyIGFib3ZlLCBhbmQgdGhhdCdzIG1hbmRhdGVkICpvbmx5KiBieSB0 aGUgZG1hX2ZlbmNlIAp3YWl0IGluIHRoZSB1c2VycHRyIG1tdSBub3RpZmllcnMsIEkgdGhpbmsg dGhlc2UgcmVzdHJpY3Rpb25zIGFyZSBhIApwcmV0dHkgaGlnaCBwcmljZSB0byBwYXkuIFdvdWxk bid0IGl0IGJlIHBvc3NpYmxlIG5vdyB0byByZXBsYWNlIHRoYXQgCmZlbmNlIHdhaXQgd2l0aCBl aXRoZXIgcGFnZSBwaW5uaW5nICh3aGljaCBub3cgaXMgY29oZXJlbnQgc2luY2UgNS45KSBvciAK cHJlZW1wdC1jdHggZmVuY2VzICsgdW5waW5uZWQgcGFnZXMgaWYgYXZhaWxhYmxlIGFuZCB0aHVz IGludmVydCB0aGUgMikgCi0+IDMpIGxvY2tpbmcgb3JkZXI/CgpUaGFua3MsClRob21hcwoKCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmRyaS1kZXZlbCBt YWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3Rz LmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1kZXZlbAo=