From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D871AC433E2 for ; Tue, 15 Sep 2020 10:59:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D9B6206DC for ; Tue, 15 Sep 2020 10:59:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kapsi.fi header.i=@kapsi.fi header.b="hwBuVZbs" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726119AbgIOK66 (ORCPT ); Tue, 15 Sep 2020 06:58:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726102AbgIOK6M (ORCPT ); Tue, 15 Sep 2020 06:58:12 -0400 Received: from mail.kapsi.fi (mail.kapsi.fi [IPv6:2001:67c:1be8::25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 460CEC06174A for ; Tue, 15 Sep 2020 03:58:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=xP8x+rJt/TDG2zJOATSck7z/xoasmq8xmsbnci0vekA=; b=hwBuVZbsJMj7BYuovOczCmWmJO +stzJG5cE13CJx9193Mr1QqrLnyIYFr2dEBTNvH4uS4OfvxU/d70aeA3FWpW/OG3S5az3GX1oqWXf Gy3l6+SOIMchyV7itseuSvP9I8VAnHy0cgi7qVnFPEAiTJ4TPmHovyatcKq2uChQLEwNKW4lKlPQb bPsBCRtOyhAR3vYn4xTxonvGxpw4Cf/Kk+V1JDib+7E1XX8bpmAMoL0wt3Ia+hVdhEm6PNGB2530b trYEYdAotwlfHVeLzwnFUVq8L4yY/uQriT+AByVq3MoY41ZFU/5DG8rNBkmlkgs6iYBnWqfTU0aWq y2teu5yQ==; Received: from dsl-hkibng22-54f986-236.dhcp.inet.fi ([84.249.134.236] helo=[192.168.1.10]) by mail.kapsi.fi with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1kI8et-0001i2-0o; Tue, 15 Sep 2020 13:57:59 +0300 Subject: Re: [RFC PATCH v2 10/17] WIP: gpu: host1x: Add no-recovery mode To: Dmitry Osipenko , Mikko Perttunen , thierry.reding@gmail.com, jonathanh@nvidia.com, airlied@linux.ie, daniel@ffwll.ch Cc: linux-tegra@vger.kernel.org, dri-devel@lists.freedesktop.org, talho@nvidia.com, bhuntsman@nvidia.com References: <20200905103420.3021852-1-mperttunen@nvidia.com> <20200905103420.3021852-11-mperttunen@nvidia.com> <7d7a29e8-3151-12ea-da66-d8a479edda84@gmail.com> <07f933b3-10d9-0318-e2f3-6dfd5b4903ac@gmail.com> <28f18a23-b588-004d-4945-91b7a593607a@kapsi.fi> <3f80aff2-23ce-9b1f-d242-e46e974fbeed@gmail.com> <56c956a3-af14-559b-8022-2228a65e82a6@kapsi.fi> From: Mikko Perttunen Message-ID: <3e8abd77-6d33-98fa-7df0-17b9c10596eb@kapsi.fi> Date: Tue, 15 Sep 2020 13:57:46 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-SA-Exim-Connect-IP: 84.249.134.236 X-SA-Exim-Mail-From: cyndis@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false Sender: linux-tegra-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-tegra@vger.kernel.org On 9/13/20 9:37 PM, Dmitry Osipenko wrote: > 13.09.2020 12:51, Mikko Perttunen пишет: > ... >>> All waits that are internal to a job should only wait for relative sync >>> point increments. > >>> In the grate-kernel every job uses unique-and-clean sync point (which is >>> also internal to the kernel driver) and a relative wait [1] is used for >>> the job's internal sync point increments [2][3][4], and thus, kernel >>> driver simply jumps over a hung job by updating DMAGET to point at the >>> start of a next job. >> >> Issues I have with this approach: >> >> * Both this and my approach have the requirement for userspace, that if >> a job hangs, the userspace must ensure all external waiters have timed >> out / been stopped before the syncpoint can be freed, as if the >> syncpoint gets reused before then, false waiter completions can happen. >> >> So freeing the syncpoint must be exposed to userspace. The kernel cannot >> do this since there may be waiters that the kernel is not aware of. My >> proposal only has one syncpoint, which I feel makes this part simpler, too. >> >> * I believe this proposal requires allocating a syncpoint for each >> externally visible syncpoint increment that the job does. This can use >> up quite a few syncpoints, and it makes syncpoints a dynamically >> allocated resource with unbounded allocation latency. This is a problem >> for safety-related systems. > > Maybe we could have a special type of a "shared" sync point that is > allocated per-hardware engine? Then shared SP won't be a scarce resource > and job won't depend on it. The kernel or userspace driver may take care > of recovering the counter value of a shared SP when job hangs or do > whatever else is needed without affecting the job's sync point. Having a shared syncpoint opens up possibilities for interference between jobs (if we're not using the firewall, the HW cannot distinguish between jobs on the same channel), and doesn't work if there are multiple channels using the same engine, which we want to do for newer chips (for performance and virtualization reasons). Even then, even if we need to allocate one syncpoint per job, the issue seems to be there. > > Primarily I'm not feeling very happy about retaining the job's sync > point recovery code because it was broken the last time I touched it and > grate-kernel works fine without it. I'm not planning to retain it any longer than necessary, which is until the staging interface is removed. Technically I can already remove it now -- that would cause any users of the staging interface to potentially behave weirdly if a job times out, but maybe we don't care about that all that much? > >> * If a job fails on a "virtual channel" (userctx), I think it's a >> reasonable expectation that further jobs on that "virtual channel" will >> not execute, and I think implementing that model is simpler than doing >> recovery. > > Couldn't jobs just use explicit fencing? Then a second job won't be > executed if first job hangs and explicit dependency is expressed. I'm > not sure that concept of a "virtual channel" is applicable to drm-scheduler. I assume what you mean is that each job incrementing a syncpoint would first wait for the preceding job incrementing that syncpoint to complete, by waiting for the preceding job's fence value. I would consider what I do in this patch to be an optimization of that. Let's say we detect a timed out job and just skip that job in the CDMA pushbuffer (but do not CPU-increment syncpoints), then at every subsequent job using that syncpoint, we will be detecting a timeout and skipping it eventually. With the "NOPping" in this patch we just pre-emptively cancel those jobs so that we don't have to spend time waiting for timeouts in the future. Functionally these should be the same, though. The wait-for-preceding-job-to-complete thing should already be there in form of the "serialize" operation if the jobs use the same syncpoint. So, if DRM scheduler's current operation is just skipping the timing out job and continuing from the next job, that's functionally fine. But we could improve DRM scheduler to allow for also cancelling future jobs that we know will time out. That would be in essence "virtual channel" support. Userspace still has options -- if it puts in other prefences, timeouts will happen as usual. If it wants to have multiple "threads" of execution where a timeout on one doesn't affect the others, it can use different syncpoints for them. > > I'll need to see a full-featured driver implementation and the test > cases that cover all the problems that you're worried about because I'm > not aware about all the T124+ needs and seeing code should help. Maybe > in the end yours approach will be the best, but for now it's not clear :) > My primary goal is simplicity of programming model and implementation. Regarding the resource management concerns, I can of course create a test case that allocates a lot of resources, but what I'm afraid about is that once we put this into a big system, with several VMs with their own resource reservations (including syncpoints), and the GPU and camera subsystems using hundreds of syncpoints, dynamic usage of those resources will create uncertainty in the system, and bug reports. And of course, if we want to make a safety-related system, you also need to document before-hand how you are ensuring that e.g. job submission (including syncpoint allocation if that is dynamic) happens under x microseconds. I don't think the model used in the grate host1x driver is bad, and I think the code there and its integration with the existing kernel frameworks are beautiful, and that is definitely a goal for the mainline driver as well. But I think we can make things even simpler overall and more reliable. Mikko From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EA1DC2D0E0 for ; Tue, 15 Sep 2020 10:58:06 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A5B2F206DC for ; Tue, 15 Sep 2020 10:58:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kapsi.fi header.i=@kapsi.fi header.b="hwBuVZbs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A5B2F206DC Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kapsi.fi Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0A94989F08; Tue, 15 Sep 2020 10:58:04 +0000 (UTC) Received: from mail.kapsi.fi (mail.kapsi.fi [IPv6:2001:67c:1be8::25]) by gabe.freedesktop.org (Postfix) with ESMTPS id 43C5089F08 for ; Tue, 15 Sep 2020 10:58:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=xP8x+rJt/TDG2zJOATSck7z/xoasmq8xmsbnci0vekA=; b=hwBuVZbsJMj7BYuovOczCmWmJO +stzJG5cE13CJx9193Mr1QqrLnyIYFr2dEBTNvH4uS4OfvxU/d70aeA3FWpW/OG3S5az3GX1oqWXf Gy3l6+SOIMchyV7itseuSvP9I8VAnHy0cgi7qVnFPEAiTJ4TPmHovyatcKq2uChQLEwNKW4lKlPQb bPsBCRtOyhAR3vYn4xTxonvGxpw4Cf/Kk+V1JDib+7E1XX8bpmAMoL0wt3Ia+hVdhEm6PNGB2530b trYEYdAotwlfHVeLzwnFUVq8L4yY/uQriT+AByVq3MoY41ZFU/5DG8rNBkmlkgs6iYBnWqfTU0aWq y2teu5yQ==; Received: from dsl-hkibng22-54f986-236.dhcp.inet.fi ([84.249.134.236] helo=[192.168.1.10]) by mail.kapsi.fi with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1kI8et-0001i2-0o; Tue, 15 Sep 2020 13:57:59 +0300 Subject: Re: [RFC PATCH v2 10/17] WIP: gpu: host1x: Add no-recovery mode To: Dmitry Osipenko , Mikko Perttunen , thierry.reding@gmail.com, jonathanh@nvidia.com, airlied@linux.ie, daniel@ffwll.ch References: <20200905103420.3021852-1-mperttunen@nvidia.com> <20200905103420.3021852-11-mperttunen@nvidia.com> <7d7a29e8-3151-12ea-da66-d8a479edda84@gmail.com> <07f933b3-10d9-0318-e2f3-6dfd5b4903ac@gmail.com> <28f18a23-b588-004d-4945-91b7a593607a@kapsi.fi> <3f80aff2-23ce-9b1f-d242-e46e974fbeed@gmail.com> <56c956a3-af14-559b-8022-2228a65e82a6@kapsi.fi> From: Mikko Perttunen Message-ID: <3e8abd77-6d33-98fa-7df0-17b9c10596eb@kapsi.fi> Date: Tue, 15 Sep 2020 13:57:46 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-SA-Exim-Connect-IP: 84.249.134.236 X-SA-Exim-Mail-From: cyndis@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-tegra@vger.kernel.org, talho@nvidia.com, bhuntsman@nvidia.com, dri-devel@lists.freedesktop.org Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" T24gOS8xMy8yMCA5OjM3IFBNLCBEbWl0cnkgT3NpcGVua28gd3JvdGU6Cj4gMTMuMDkuMjAyMCAx Mjo1MSwgTWlra28gUGVydHR1bmVuINC/0LjRiNC10YI6Cj4gLi4uCj4+PiBBbGwgd2FpdHMgdGhh dCBhcmUgaW50ZXJuYWwgdG8gYSBqb2Igc2hvdWxkIG9ubHkgd2FpdCBmb3IgcmVsYXRpdmUgc3lu Ywo+Pj4gcG9pbnQgaW5jcmVtZW50cy4gPgo+Pj4gSW4gdGhlIGdyYXRlLWtlcm5lbCBldmVyeSBq b2IgdXNlcyB1bmlxdWUtYW5kLWNsZWFuIHN5bmMgcG9pbnQgKHdoaWNoIGlzCj4+PiBhbHNvIGlu dGVybmFsIHRvIHRoZSBrZXJuZWwgZHJpdmVyKSBhbmQgYSByZWxhdGl2ZSB3YWl0IFsxXSBpcyB1 c2VkIGZvcgo+Pj4gdGhlIGpvYidzIGludGVybmFsIHN5bmMgcG9pbnQgaW5jcmVtZW50cyBbMl1b M11bNF0sIGFuZCB0aHVzLCBrZXJuZWwKPj4+IGRyaXZlciBzaW1wbHkganVtcHMgb3ZlciBhIGh1 bmcgam9iIGJ5IHVwZGF0aW5nIERNQUdFVCB0byBwb2ludCBhdCB0aGUKPj4+IHN0YXJ0IG9mIGEg bmV4dCBqb2IuCj4+Cj4+IElzc3VlcyBJIGhhdmUgd2l0aCB0aGlzIGFwcHJvYWNoOgo+Pgo+PiAq IEJvdGggdGhpcyBhbmQgbXkgYXBwcm9hY2ggaGF2ZSB0aGUgcmVxdWlyZW1lbnQgZm9yIHVzZXJz cGFjZSwgdGhhdCBpZgo+PiBhIGpvYiBoYW5ncywgdGhlIHVzZXJzcGFjZSBtdXN0IGVuc3VyZSBh bGwgZXh0ZXJuYWwgd2FpdGVycyBoYXZlIHRpbWVkCj4+IG91dCAvIGJlZW4gc3RvcHBlZCBiZWZv cmUgdGhlIHN5bmNwb2ludCBjYW4gYmUgZnJlZWQsIGFzIGlmIHRoZQo+PiBzeW5jcG9pbnQgZ2V0 cyByZXVzZWQgYmVmb3JlIHRoZW4sIGZhbHNlIHdhaXRlciBjb21wbGV0aW9ucyBjYW4gaGFwcGVu Lgo+Pgo+PiBTbyBmcmVlaW5nIHRoZSBzeW5jcG9pbnQgbXVzdCBiZSBleHBvc2VkIHRvIHVzZXJz cGFjZS4gVGhlIGtlcm5lbCBjYW5ub3QKPj4gZG8gdGhpcyBzaW5jZSB0aGVyZSBtYXkgYmUgd2Fp dGVycyB0aGF0IHRoZSBrZXJuZWwgaXMgbm90IGF3YXJlIG9mLiBNeQo+PiBwcm9wb3NhbCBvbmx5 IGhhcyBvbmUgc3luY3BvaW50LCB3aGljaCBJIGZlZWwgbWFrZXMgdGhpcyBwYXJ0IHNpbXBsZXIs IHRvby4KPj4KPj4gKiBJIGJlbGlldmUgdGhpcyBwcm9wb3NhbCByZXF1aXJlcyBhbGxvY2F0aW5n IGEgc3luY3BvaW50IGZvciBlYWNoCj4+IGV4dGVybmFsbHkgdmlzaWJsZSBzeW5jcG9pbnQgaW5j cmVtZW50IHRoYXQgdGhlIGpvYiBkb2VzLiBUaGlzIGNhbiB1c2UKPj4gdXAgcXVpdGUgYSBmZXcg c3luY3BvaW50cywgYW5kIGl0IG1ha2VzIHN5bmNwb2ludHMgYSBkeW5hbWljYWxseQo+PiBhbGxv Y2F0ZWQgcmVzb3VyY2Ugd2l0aCB1bmJvdW5kZWQgYWxsb2NhdGlvbiBsYXRlbmN5LiBUaGlzIGlz IGEgcHJvYmxlbQo+PiBmb3Igc2FmZXR5LXJlbGF0ZWQgc3lzdGVtcy4KPiAKPiBNYXliZSB3ZSBj b3VsZCBoYXZlIGEgc3BlY2lhbCB0eXBlIG9mIGEgInNoYXJlZCIgc3luYyBwb2ludCB0aGF0IGlz Cj4gYWxsb2NhdGVkIHBlci1oYXJkd2FyZSBlbmdpbmU/IFRoZW4gc2hhcmVkIFNQIHdvbid0IGJl IGEgc2NhcmNlIHJlc291cmNlCj4gYW5kIGpvYiB3b24ndCBkZXBlbmQgb24gaXQuIFRoZSBrZXJu ZWwgb3IgdXNlcnNwYWNlIGRyaXZlciBtYXkgdGFrZSBjYXJlCj4gb2YgcmVjb3ZlcmluZyB0aGUg Y291bnRlciB2YWx1ZSBvZiBhIHNoYXJlZCBTUCB3aGVuIGpvYiBoYW5ncyBvciBkbwo+IHdoYXRl dmVyIGVsc2UgaXMgbmVlZGVkIHdpdGhvdXQgYWZmZWN0aW5nIHRoZSBqb2IncyBzeW5jIHBvaW50 LgoKSGF2aW5nIGEgc2hhcmVkIHN5bmNwb2ludCBvcGVucyB1cCBwb3NzaWJpbGl0aWVzIGZvciBp bnRlcmZlcmVuY2UgCmJldHdlZW4gam9icyAoaWYgd2UncmUgbm90IHVzaW5nIHRoZSBmaXJld2Fs bCwgdGhlIEhXIGNhbm5vdCBkaXN0aW5ndWlzaCAKYmV0d2VlbiBqb2JzIG9uIHRoZSBzYW1lIGNo YW5uZWwpLCBhbmQgZG9lc24ndCB3b3JrIGlmIHRoZXJlIGFyZSAKbXVsdGlwbGUgY2hhbm5lbHMg dXNpbmcgdGhlIHNhbWUgZW5naW5lLCB3aGljaCB3ZSB3YW50IHRvIGRvIGZvciBuZXdlciAKY2hp cHMgKGZvciBwZXJmb3JtYW5jZSBhbmQgdmlydHVhbGl6YXRpb24gcmVhc29ucykuCgpFdmVuIHRo ZW4sIGV2ZW4gaWYgd2UgbmVlZCB0byBhbGxvY2F0ZSBvbmUgc3luY3BvaW50IHBlciBqb2IsIHRo ZSBpc3N1ZSAKc2VlbXMgdG8gYmUgdGhlcmUuCgo+IAo+IFByaW1hcmlseSBJJ20gbm90IGZlZWxp bmcgdmVyeSBoYXBweSBhYm91dCByZXRhaW5pbmcgdGhlIGpvYidzIHN5bmMKPiBwb2ludCByZWNv dmVyeSBjb2RlIGJlY2F1c2UgaXQgd2FzIGJyb2tlbiB0aGUgbGFzdCB0aW1lIEkgdG91Y2hlZCBp dCBhbmQKPiBncmF0ZS1rZXJuZWwgd29ya3MgZmluZSB3aXRob3V0IGl0LgoKSSdtIG5vdCBwbGFu bmluZyB0byByZXRhaW4gaXQgYW55IGxvbmdlciB0aGFuIG5lY2Vzc2FyeSwgd2hpY2ggaXMgdW50 aWwgCnRoZSBzdGFnaW5nIGludGVyZmFjZSBpcyByZW1vdmVkLiBUZWNobmljYWxseSBJIGNhbiBh bHJlYWR5IHJlbW92ZSBpdCAKbm93IC0tIHRoYXQgd291bGQgY2F1c2UgYW55IHVzZXJzIG9mIHRo ZSBzdGFnaW5nIGludGVyZmFjZSB0byAKcG90ZW50aWFsbHkgYmVoYXZlIHdlaXJkbHkgaWYgYSBq b2IgdGltZXMgb3V0LCBidXQgbWF5YmUgd2UgZG9uJ3QgY2FyZSAKYWJvdXQgdGhhdCBhbGwgdGhh dCBtdWNoPwoKPiAKPj4gKiBJZiBhIGpvYiBmYWlscyBvbiBhICJ2aXJ0dWFsIGNoYW5uZWwiICh1 c2VyY3R4KSwgSSB0aGluayBpdCdzIGEKPj4gcmVhc29uYWJsZSBleHBlY3RhdGlvbiB0aGF0IGZ1 cnRoZXIgam9icyBvbiB0aGF0ICJ2aXJ0dWFsIGNoYW5uZWwiIHdpbGwKPj4gbm90IGV4ZWN1dGUs IGFuZCBJIHRoaW5rIGltcGxlbWVudGluZyB0aGF0IG1vZGVsIGlzIHNpbXBsZXIgdGhhbiBkb2lu Zwo+PiByZWNvdmVyeS4KPiAKPiBDb3VsZG4ndCBqb2JzIGp1c3QgdXNlIGV4cGxpY2l0IGZlbmNp bmc/IFRoZW4gYSBzZWNvbmQgam9iIHdvbid0IGJlCj4gZXhlY3V0ZWQgaWYgZmlyc3Qgam9iIGhh bmdzIGFuZCBleHBsaWNpdCBkZXBlbmRlbmN5IGlzIGV4cHJlc3NlZC4gSSdtCj4gbm90IHN1cmUg dGhhdCBjb25jZXB0IG9mIGEgInZpcnR1YWwgY2hhbm5lbCIgaXMgYXBwbGljYWJsZSB0byBkcm0t c2NoZWR1bGVyLgoKSSBhc3N1bWUgd2hhdCB5b3UgbWVhbiBpcyB0aGF0IGVhY2ggam9iIGluY3Jl bWVudGluZyBhIHN5bmNwb2ludCB3b3VsZCAKZmlyc3Qgd2FpdCBmb3IgdGhlIHByZWNlZGluZyBq b2IgaW5jcmVtZW50aW5nIHRoYXQgc3luY3BvaW50IHRvIApjb21wbGV0ZSwgYnkgd2FpdGluZyBm b3IgdGhlIHByZWNlZGluZyBqb2IncyBmZW5jZSB2YWx1ZS4KCkkgd291bGQgY29uc2lkZXIgd2hh dCBJIGRvIGluIHRoaXMgcGF0Y2ggdG8gYmUgYW4gb3B0aW1pemF0aW9uIG9mIHRoYXQuIApMZXQn cyBzYXkgd2UgZGV0ZWN0IGEgdGltZWQgb3V0IGpvYiBhbmQganVzdCBza2lwIHRoYXQgam9iIGlu IHRoZSBDRE1BIApwdXNoYnVmZmVyIChidXQgZG8gbm90IENQVS1pbmNyZW1lbnQgc3luY3BvaW50 cyksIHRoZW4gYXQgZXZlcnkgCnN1YnNlcXVlbnQgam9iIHVzaW5nIHRoYXQgc3luY3BvaW50LCB3 ZSB3aWxsIGJlIGRldGVjdGluZyBhIHRpbWVvdXQgYW5kIApza2lwcGluZyBpdCBldmVudHVhbGx5 LiBXaXRoIHRoZSAiTk9QcGluZyIgaW4gdGhpcyBwYXRjaCB3ZSBqdXN0IApwcmUtZW1wdGl2ZWx5 IGNhbmNlbCB0aG9zZSBqb2JzIHNvIHRoYXQgd2UgZG9uJ3QgaGF2ZSB0byBzcGVuZCB0aW1lIAp3 YWl0aW5nIGZvciB0aW1lb3V0cyBpbiB0aGUgZnV0dXJlLiBGdW5jdGlvbmFsbHkgdGhlc2Ugc2hv dWxkIGJlIHRoZSAKc2FtZSwgdGhvdWdoLgoKVGhlIHdhaXQtZm9yLXByZWNlZGluZy1qb2ItdG8t Y29tcGxldGUgdGhpbmcgc2hvdWxkIGFscmVhZHkgYmUgdGhlcmUgaW4gCmZvcm0gb2YgdGhlICJz ZXJpYWxpemUiIG9wZXJhdGlvbiBpZiB0aGUgam9icyB1c2UgdGhlIHNhbWUgc3luY3BvaW50LgoK U28sIGlmIERSTSBzY2hlZHVsZXIncyBjdXJyZW50IG9wZXJhdGlvbiBpcyBqdXN0IHNraXBwaW5n IHRoZSB0aW1pbmcgb3V0IApqb2IgYW5kIGNvbnRpbnVpbmcgZnJvbSB0aGUgbmV4dCBqb2IsIHRo YXQncyBmdW5jdGlvbmFsbHkgZmluZS4gQnV0IHdlIApjb3VsZCBpbXByb3ZlIERSTSBzY2hlZHVs ZXIgdG8gYWxsb3cgZm9yIGFsc28gY2FuY2VsbGluZyBmdXR1cmUgam9icyAKdGhhdCB3ZSBrbm93 IHdpbGwgdGltZSBvdXQuIFRoYXQgd291bGQgYmUgaW4gZXNzZW5jZSAidmlydHVhbCBjaGFubmVs IiAKc3VwcG9ydC4KClVzZXJzcGFjZSBzdGlsbCBoYXMgb3B0aW9ucyAtLSBpZiBpdCBwdXRzIGlu IG90aGVyIHByZWZlbmNlcywgdGltZW91dHMgCndpbGwgaGFwcGVuIGFzIHVzdWFsLiBJZiBpdCB3 YW50cyB0byBoYXZlIG11bHRpcGxlICJ0aHJlYWRzIiBvZiAKZXhlY3V0aW9uIHdoZXJlIGEgdGlt ZW91dCBvbiBvbmUgZG9lc24ndCBhZmZlY3QgdGhlIG90aGVycywgaXQgY2FuIHVzZSAKZGlmZmVy ZW50IHN5bmNwb2ludHMgZm9yIHRoZW0uCgo+IAo+IEknbGwgbmVlZCB0byBzZWUgYSBmdWxsLWZl YXR1cmVkIGRyaXZlciBpbXBsZW1lbnRhdGlvbiBhbmQgdGhlIHRlc3QKPiBjYXNlcyB0aGF0IGNv dmVyIGFsbCB0aGUgcHJvYmxlbXMgdGhhdCB5b3UncmUgd29ycmllZCBhYm91dCBiZWNhdXNlIEkn bQo+IG5vdCBhd2FyZSBhYm91dCBhbGwgdGhlIFQxMjQrIG5lZWRzIGFuZCBzZWVpbmcgY29kZSBz aG91bGQgaGVscC4gTWF5YmUKPiBpbiB0aGUgZW5kIHlvdXJzIGFwcHJvYWNoIHdpbGwgYmUgdGhl IGJlc3QsIGJ1dCBmb3Igbm93IGl0J3Mgbm90IGNsZWFyIDopCj4gCgpNeSBwcmltYXJ5IGdvYWwg aXMgc2ltcGxpY2l0eSBvZiBwcm9ncmFtbWluZyBtb2RlbCBhbmQgaW1wbGVtZW50YXRpb24uIApS ZWdhcmRpbmcgdGhlIHJlc291cmNlIG1hbmFnZW1lbnQgY29uY2VybnMsIEkgY2FuIG9mIGNvdXJz ZSBjcmVhdGUgYSAKdGVzdCBjYXNlIHRoYXQgYWxsb2NhdGVzIGEgbG90IG9mIHJlc291cmNlcywg YnV0IHdoYXQgSSdtIGFmcmFpZCBhYm91dCAKaXMgdGhhdCBvbmNlIHdlIHB1dCB0aGlzIGludG8g YSBiaWcgc3lzdGVtLCB3aXRoIHNldmVyYWwgVk1zIHdpdGggdGhlaXIgCm93biByZXNvdXJjZSBy ZXNlcnZhdGlvbnMgKGluY2x1ZGluZyBzeW5jcG9pbnRzKSwgYW5kIHRoZSBHUFUgYW5kIGNhbWVy YSAKc3Vic3lzdGVtcyB1c2luZyBodW5kcmVkcyBvZiBzeW5jcG9pbnRzLCBkeW5hbWljIHVzYWdl IG9mIHRob3NlIApyZXNvdXJjZXMgd2lsbCBjcmVhdGUgdW5jZXJ0YWludHkgaW4gdGhlIHN5c3Rl bSwgYW5kIGJ1ZyByZXBvcnRzLgoKQW5kIG9mIGNvdXJzZSwgaWYgd2Ugd2FudCB0byBtYWtlIGEg c2FmZXR5LXJlbGF0ZWQgc3lzdGVtLCB5b3UgYWxzbyBuZWVkIAp0byBkb2N1bWVudCBiZWZvcmUt aGFuZCBob3cgeW91IGFyZSBlbnN1cmluZyB0aGF0IGUuZy4gam9iIHN1Ym1pc3Npb24gCihpbmNs dWRpbmcgc3luY3BvaW50IGFsbG9jYXRpb24gaWYgdGhhdCBpcyBkeW5hbWljKSBoYXBwZW5zIHVu ZGVyIHggCm1pY3Jvc2Vjb25kcy4KCkkgZG9uJ3QgdGhpbmsgdGhlIG1vZGVsIHVzZWQgaW4gdGhl IGdyYXRlIGhvc3QxeCBkcml2ZXIgaXMgYmFkLCBhbmQgSSAKdGhpbmsgdGhlIGNvZGUgdGhlcmUg YW5kIGl0cyBpbnRlZ3JhdGlvbiB3aXRoIHRoZSBleGlzdGluZyBrZXJuZWwgCmZyYW1ld29ya3Mg YXJlIGJlYXV0aWZ1bCwgYW5kIHRoYXQgaXMgZGVmaW5pdGVseSBhIGdvYWwgZm9yIHRoZSBtYWlu bGluZSAKZHJpdmVyIGFzIHdlbGwuIEJ1dCBJIHRoaW5rIHdlIGNhbiBtYWtlIHRoaW5ncyBldmVu IHNpbXBsZXIgb3ZlcmFsbCBhbmQgCm1vcmUgcmVsaWFibGUuCgpNaWtrbwpfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0 CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3Rv cC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWwK