From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A478C07E9B for ; Fri, 9 Jul 2021 08:00:29 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 00FCC613C3 for ; Fri, 9 Jul 2021 08:00:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00FCC613C3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id AAF286E9D6; Fri, 9 Jul 2021 08:00:27 +0000 (UTC) Received: from mail-oi1-x22a.google.com (mail-oi1-x22a.google.com [IPv6:2607:f8b0:4864:20::22a]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1CFC56E9D4 for ; Fri, 9 Jul 2021 08:00:26 +0000 (UTC) Received: by mail-oi1-x22a.google.com with SMTP id u66so5437634oif.13 for ; Fri, 09 Jul 2021 01:00:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=4Tm6rbhq6rt9ho8Bl6eVvSWWbDirLBWL3poqengHkQM=; b=cWUbVFQufEddmU6NXf5u7WC6fcnvndpqVjZeZXeZCcSYX2cktBcUxUBwg089EwCYHi /Uu4s/ipVTMHCBSYXYMZrL+YnRvU8Mhf4EaJefC3korjw2voNxEbQstCk/sxVQ2slul0 A5/rfvrOaX7XI91SNvVtG903grhYSoWEDhMyM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=4Tm6rbhq6rt9ho8Bl6eVvSWWbDirLBWL3poqengHkQM=; b=ZpjTssovDVe46RVoR3xFV9QeZt/88ZpNh6qv+EJI12nR4yWOmTMj/gAmQZlrm3ksqW R2kpKy3D68vFdCpBLywnKwwRasyaCLeXrR7/YxkX92Cx95GentUSWQWFKQ3c8c0/DkGQ Z5/dvNt+HGSTg1DYRKgDV+ACZvKsyLxs9cbZt80i34FGf3stfoaU7kWBTpkwx2csCl86 67py/HVwl2aVhSuUE5xvEB1CPpkEZiIB9UuhZ7Iuq+b7xDltNOFsgCBDEM2nBA9VcjDW L5pYxHWL5V6AZkKvRekH1fMazf9S4sk1C9Gy/zeRg53xheGPvx06gMWP5Cg89mmwk3qn 6K9Q== X-Gm-Message-State: AOAM530ulJheq1KM5Ms6EO3vKmEsAAz5SNpO0iKXizK/s4f4ST4Q3E7V fibNEZDukKedFLoBF89yz3G3jCcc2QbwU9rbhgtaxg== X-Google-Smtp-Source: ABdhPJwpYZhP/rk5oEw1WXb6I1u345ELPS2Mya0mKaMsr52a+aV56mu3CTUMBnYv6GhYXipU5gPS4xgH0qXe9Ie58LQ= X-Received: by 2002:aca:f491:: with SMTP id s139mr21163595oih.128.1625817625401; Fri, 09 Jul 2021 01:00:25 -0700 (PDT) MIME-Version: 1.0 References: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> <20210708173754.3877540-2-daniel.vetter@ffwll.ch> <5588d1c3-833b-bd95-69e6-a103f2e8affa@amd.com> <871a4619-8a17-134f-9d9c-40a522473946@amd.com> In-Reply-To: <871a4619-8a17-134f-9d9c-40a522473946@amd.com> From: Daniel Vetter Date: Fri, 9 Jul 2021 10:00:14 +0200 Message-ID: Subject: Re: [PATCH v3 01/20] drm/sched: entity->rq selection cannot fail To: =?UTF-8?Q?Christian_K=C3=B6nig?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jack Zhang , Intel Graphics Development , DRI Development , Steven Price , Luben Tuikov , Boris Brezillon , Daniel Vetter Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Fri, Jul 9, 2021 at 9:23 AM Christian K=C3=B6nig wrote: > Am 09.07.21 um 09:14 schrieb Daniel Vetter: > > On Fri, Jul 9, 2021 at 8:53 AM Christian K=C3=B6nig wrote: > >> Am 08.07.21 um 19:37 schrieb Daniel Vetter: > >>> If it does, someone managed to set up a sched_entity without > >>> schedulers, which is just a driver bug. > >> NAK, it is perfectly valid for rq selection to fail. > > There isn't a better way to explain stuff to someone who's new to the > > code and tries to improve it with docs than to NAK stuff with > > incomplete explanations? > > Well as far as I understand it a NAK means that the author has missed > something important and needs to re-iterate. It comes around as very screaming at least to me (all uppercase and all that) and personally I only associate it with unchecked angry kernel maintainers on lkml celebrating their status and putting down some noobs for shits and giggles. I think here on dri-devel you're the only one doing it regularly. > It's just to say that we absolutely can't merge a patch or something > will break. Well yeah I know that when a patch breaks something I can't merge it. For drm-intel we also documented that clearly, but for drm-misc it's not spelled out. I'll fix that. > >> See drm_sched_pick_best(): > >> > >> if (!sched->ready) { > >> DRM_WARN("scheduler %s is not ready, skippin= g", > >> sched->name); > >> continue; > >> } > >> > >> This can happen when a device reset fails for some engine. > > Well yeah I didn't expect amdgpu to just change this directly, so I > > didn't find it. Getting an ENOENT on a hw failure instead of an EIO is > > a bit interesting semantics I guess, also what happens with the jobs > > which raced against the scheduler not being ready? I'm not seeing any > > checks for ready in the main scheduler logic so this at least looks > > somewhat accidental as a side effect, also no other driver than amdgpu > > communitcates that reset failed back to drm/sched like this. They seem > > to just not, and I guess timeout on the next request will get us into > > an endless reset loop? > > Correct. Key point is that there aren't any jobs which are currently > scheduled. > > When the ready flag is changed the scheduler is paused, e.g. the main > thread is not running any more. > > I'm pretty sure that all of this is horrible racy, but nobody really > looked into the design from a higher level as far as I know. Yeah the scheduler thread is fine because it's stopped, but it also doesn't look at sched->ready, so it can't race. What does race is new submissions, and if they stuff something into the queue then I'm wondering what happens to that. Also what happens to the requests already in the queue. Eventually I guess userspace notices the ENOENT, tears down the context, and the kernel then also tears down the context and cleans up the mess. But it's rather inglorious until it collapses down to a coherent state again I think. Or is there something with the scheduler restart flow which is guaranteed to catch these, and we're maybe just missing a bunch of barriers? Either way I think a proper interface to terminally wedge a sched would be good, so that at least we can pass back something meaningful like -EIO. And also tell "the gpu died" apart from "the driver author tore down the scheduler while it was still in use", which I think we really should catch with some WARN_ON. Anyway for the immediate issue of "don't break amdgpu" I think I'll reshuffle the split between job_init and job_arm again, and add a big comment to job_init that it can fail with ENOENT, and why, and what kind of interface would be more proper. i915 will need the terminally wedged flow too so I'll probably have to look into this, but that will need some proper thought. Cheers, Daniel > > Christian. > > > > > -Daniel > > > > > >> Regards, > >> Christian. > >> > >>> We BUG_ON() here because in the next patch drm_sched_job_init() will > >>> be split up, with drm_sched_job_arm() never failing. And that's the > >>> part where the rq selection will end up in. > >>> > >>> Note that if having an empty sched_list set on an entity is indeed a > >>> valid use-case, we can keep that check in job_init even after the spl= it > >>> into job_init/arm. > >>> > >>> Signed-off-by: Daniel Vetter > >>> Cc: "Christian K=C3=B6nig" > >>> Cc: Luben Tuikov > >>> Cc: Daniel Vetter > >>> Cc: Steven Price > >>> Cc: Andrey Grodzovsky > >>> Cc: Boris Brezillon > >>> Cc: Jack Zhang > >>> --- > >>> drivers/gpu/drm/scheduler/sched_entity.c | 2 +- > >>> drivers/gpu/drm/scheduler/sched_main.c | 3 +-- > >>> 2 files changed, 2 insertions(+), 3 deletions(-) > >>> > >>> diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/d= rm/scheduler/sched_entity.c > >>> index 79554aa4dbb1..6fc116ee7302 100644 > >>> --- a/drivers/gpu/drm/scheduler/sched_entity.c > >>> +++ b/drivers/gpu/drm/scheduler/sched_entity.c > >>> @@ -45,7 +45,7 @@ > >>> * @guilty: atomic_t set to 1 when a job on this queue > >>> * is found to be guilty causing a timeout > >>> * > >>> - * Note: the sched_list should have at least one element to schedule > >>> + * Note: the sched_list must have at least one element to schedule > >>> * the entity > >>> * > >>> * Returns 0 on success or a negative error code on failure. > >>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm= /scheduler/sched_main.c > >>> index 33c414d55fab..01dd47154181 100644 > >>> --- a/drivers/gpu/drm/scheduler/sched_main.c > >>> +++ b/drivers/gpu/drm/scheduler/sched_main.c > >>> @@ -586,8 +586,7 @@ int drm_sched_job_init(struct drm_sched_job *job, > >>> struct drm_gpu_scheduler *sched; > >>> > >>> drm_sched_entity_select_rq(entity); > >>> - if (!entity->rq) > >>> - return -ENOENT; > >>> + BUG_ON(!entity->rq); > >>> > >>> sched =3D entity->rq->sched; > >>> > > > -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32674C07E99 for ; Fri, 9 Jul 2021 08:00:28 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E099D613C8 for ; Fri, 9 Jul 2021 08:00:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E099D613C8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8843C6E9D4; Fri, 9 Jul 2021 08:00:27 +0000 (UTC) Received: from mail-oi1-x22f.google.com (mail-oi1-x22f.google.com [IPv6:2607:f8b0:4864:20::22f]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1520A6E9D2 for ; Fri, 9 Jul 2021 08:00:26 +0000 (UTC) Received: by mail-oi1-x22f.google.com with SMTP id l21so11260395oig.3 for ; Fri, 09 Jul 2021 01:00:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=4Tm6rbhq6rt9ho8Bl6eVvSWWbDirLBWL3poqengHkQM=; b=cWUbVFQufEddmU6NXf5u7WC6fcnvndpqVjZeZXeZCcSYX2cktBcUxUBwg089EwCYHi /Uu4s/ipVTMHCBSYXYMZrL+YnRvU8Mhf4EaJefC3korjw2voNxEbQstCk/sxVQ2slul0 A5/rfvrOaX7XI91SNvVtG903grhYSoWEDhMyM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=4Tm6rbhq6rt9ho8Bl6eVvSWWbDirLBWL3poqengHkQM=; b=dW26d5b5x3KKVHdVjPIo+EktBQpsv9sWRiZCDeEn35qjSf2EjL0eVbmInyuTWSyJU6 UnLHYHU5MueanscQWOadjR7eLaXk62yGZn9gqDP8QyERusPe06Z4zdQhZwxPNyRrjXqf 87W11yZ+3iGXjyLngufa4nEPCW+INnFMSs7BGQT407SZWj7dy7lAXv5zXoJ1iuBDpOsG jL/9K3Eynxx8eWbdtxPmNEBctTGjw5X3Hv2qkOAGnAHKTZVSPthEtk58IqQRbxl38G0e ItvjjAv/7am02FJbUUmAcHDECib5VALmn5ub/ZutVCeuIsnEA6vANDAPSqq8QtkP7oUb fwdg== X-Gm-Message-State: AOAM530IbMBF3zAZpmMS1cJ7vZImqZJn7Lx3s+4+wVQp1mREAm5Cad3+ HAuuLaghLAHa046LCUVsHoPS8TxZdFGk/jw8EknWgg== X-Google-Smtp-Source: ABdhPJwpYZhP/rk5oEw1WXb6I1u345ELPS2Mya0mKaMsr52a+aV56mu3CTUMBnYv6GhYXipU5gPS4xgH0qXe9Ie58LQ= X-Received: by 2002:aca:f491:: with SMTP id s139mr21163595oih.128.1625817625401; Fri, 09 Jul 2021 01:00:25 -0700 (PDT) MIME-Version: 1.0 References: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> <20210708173754.3877540-2-daniel.vetter@ffwll.ch> <5588d1c3-833b-bd95-69e6-a103f2e8affa@amd.com> <871a4619-8a17-134f-9d9c-40a522473946@amd.com> In-Reply-To: <871a4619-8a17-134f-9d9c-40a522473946@amd.com> From: Daniel Vetter Date: Fri, 9 Jul 2021 10:00:14 +0200 Message-ID: To: =?UTF-8?Q?Christian_K=C3=B6nig?= Subject: Re: [Intel-gfx] [PATCH v3 01/20] drm/sched: entity->rq selection cannot fail X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrey Grodzovsky , Jack Zhang , Intel Graphics Development , DRI Development , Steven Price , Luben Tuikov , Daniel Vetter Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" T24gRnJpLCBKdWwgOSwgMjAyMSBhdCA5OjIzIEFNIENocmlzdGlhbiBLw7ZuaWcgPGNocmlzdGlh bi5rb2VuaWdAYW1kLmNvbT4gd3JvdGU6Cj4gQW0gMDkuMDcuMjEgdW0gMDk6MTQgc2NocmllYiBE YW5pZWwgVmV0dGVyOgo+ID4gT24gRnJpLCBKdWwgOSwgMjAyMSBhdCA4OjUzIEFNIENocmlzdGlh biBLw7ZuaWcgPGNocmlzdGlhbi5rb2VuaWdAYW1kLmNvbT4gd3JvdGU6Cj4gPj4gQW0gMDguMDcu MjEgdW0gMTk6Mzcgc2NocmllYiBEYW5pZWwgVmV0dGVyOgo+ID4+PiBJZiBpdCBkb2VzLCBzb21l b25lIG1hbmFnZWQgdG8gc2V0IHVwIGEgc2NoZWRfZW50aXR5IHdpdGhvdXQKPiA+Pj4gc2NoZWR1 bGVycywgd2hpY2ggaXMganVzdCBhIGRyaXZlciBidWcuCj4gPj4gTkFLLCBpdCBpcyBwZXJmZWN0 bHkgdmFsaWQgZm9yIHJxIHNlbGVjdGlvbiB0byBmYWlsLgo+ID4gVGhlcmUgaXNuJ3QgYSBiZXR0 ZXIgd2F5IHRvIGV4cGxhaW4gc3R1ZmYgdG8gc29tZW9uZSB3aG8ncyBuZXcgdG8gdGhlCj4gPiBj b2RlIGFuZCB0cmllcyB0byBpbXByb3ZlIGl0IHdpdGggZG9jcyB0aGFuIHRvIE5BSyBzdHVmZiB3 aXRoCj4gPiBpbmNvbXBsZXRlIGV4cGxhbmF0aW9ucz8KPgo+IFdlbGwgYXMgZmFyIGFzIEkgdW5k ZXJzdGFuZCBpdCBhIE5BSyBtZWFucyB0aGF0IHRoZSBhdXRob3IgaGFzIG1pc3NlZAo+IHNvbWV0 aGluZyBpbXBvcnRhbnQgYW5kIG5lZWRzIHRvIHJlLWl0ZXJhdGUuCgpJdCBjb21lcyBhcm91bmQg YXMgdmVyeSBzY3JlYW1pbmcgYXQgbGVhc3QgdG8gbWUgKGFsbCB1cHBlcmNhc2UgYW5kCmFsbCB0 aGF0KSBhbmQgcGVyc29uYWxseSBJIG9ubHkgYXNzb2NpYXRlIGl0IHdpdGggdW5jaGVja2VkIGFu Z3J5Cmtlcm5lbCBtYWludGFpbmVycyBvbiBsa21sIGNlbGVicmF0aW5nIHRoZWlyIHN0YXR1cyBh bmQgcHV0dGluZyBkb3duCnNvbWUgbm9vYnMgZm9yIHNoaXRzIGFuZCBnaWdnbGVzLiBJIHRoaW5r IGhlcmUgb24gZHJpLWRldmVsIHlvdSdyZSB0aGUKb25seSBvbmUgZG9pbmcgaXQgcmVndWxhcmx5 LgoKPiBJdCdzIGp1c3QgdG8gc2F5IHRoYXQgd2UgYWJzb2x1dGVseSBjYW4ndCBtZXJnZSBhIHBh dGNoIG9yIHNvbWV0aGluZwo+IHdpbGwgYnJlYWsuCgpXZWxsIHllYWggSSBrbm93IHRoYXQgd2hl biBhIHBhdGNoIGJyZWFrcyBzb21ldGhpbmcgSSBjYW4ndCBtZXJnZSBpdC4KRm9yIGRybS1pbnRl bCB3ZSBhbHNvIGRvY3VtZW50ZWQgdGhhdCBjbGVhcmx5LCBidXQgZm9yIGRybS1taXNjIGl0J3MK bm90IHNwZWxsZWQgb3V0LiBJJ2xsIGZpeCB0aGF0LgoKPiA+PiBTZWUgZHJtX3NjaGVkX3BpY2tf YmVzdCgpOgo+ID4+Cj4gPj4gICAgICAgICAgICAgICAgICAgaWYgKCFzY2hlZC0+cmVhZHkpIHsK PiA+PiAgICAgICAgICAgICAgICAgICAgICAgICAgIERSTV9XQVJOKCJzY2hlZHVsZXIgJXMgaXMg bm90IHJlYWR5LCBza2lwcGluZyIsCj4gPj4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBzY2hlZC0+bmFtZSk7Cj4gPj4gICAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51 ZTsKPiA+PiAgICAgICAgICAgICAgICAgICB9Cj4gPj4KPiA+PiBUaGlzIGNhbiBoYXBwZW4gd2hl biBhIGRldmljZSByZXNldCBmYWlscyBmb3Igc29tZSBlbmdpbmUuCj4gPiBXZWxsIHllYWggSSBk aWRuJ3QgZXhwZWN0IGFtZGdwdSB0byBqdXN0IGNoYW5nZSB0aGlzIGRpcmVjdGx5LCBzbyBJCj4g PiBkaWRuJ3QgZmluZCBpdC4gR2V0dGluZyBhbiBFTk9FTlQgb24gYSBodyBmYWlsdXJlIGluc3Rl YWQgb2YgYW4gRUlPIGlzCj4gPiBhIGJpdCBpbnRlcmVzdGluZyBzZW1hbnRpY3MgSSBndWVzcywg YWxzbyB3aGF0IGhhcHBlbnMgd2l0aCB0aGUgam9icwo+ID4gd2hpY2ggcmFjZWQgYWdhaW5zdCB0 aGUgc2NoZWR1bGVyIG5vdCBiZWluZyByZWFkeT8gSSdtIG5vdCBzZWVpbmcgYW55Cj4gPiBjaGVj a3MgZm9yIHJlYWR5IGluIHRoZSBtYWluIHNjaGVkdWxlciBsb2dpYyBzbyB0aGlzIGF0IGxlYXN0 IGxvb2tzCj4gPiBzb21ld2hhdCBhY2NpZGVudGFsIGFzIGEgc2lkZSBlZmZlY3QsIGFsc28gbm8g b3RoZXIgZHJpdmVyIHRoYW4gYW1kZ3B1Cj4gPiBjb21tdW5pdGNhdGVzIHRoYXQgcmVzZXQgZmFp bGVkIGJhY2sgdG8gZHJtL3NjaGVkIGxpa2UgdGhpcy4gVGhleSBzZWVtCj4gPiB0byBqdXN0IG5v dCwgYW5kIEkgZ3Vlc3MgdGltZW91dCBvbiB0aGUgbmV4dCByZXF1ZXN0IHdpbGwgZ2V0IHVzIGlu dG8KPiA+IGFuIGVuZGxlc3MgcmVzZXQgbG9vcD8KPgo+IENvcnJlY3QuIEtleSBwb2ludCBpcyB0 aGF0IHRoZXJlIGFyZW4ndCBhbnkgam9icyB3aGljaCBhcmUgY3VycmVudGx5Cj4gc2NoZWR1bGVk Lgo+Cj4gV2hlbiB0aGUgcmVhZHkgZmxhZyBpcyBjaGFuZ2VkIHRoZSBzY2hlZHVsZXIgaXMgcGF1 c2VkLCBlLmcuIHRoZSBtYWluCj4gdGhyZWFkIGlzIG5vdCBydW5uaW5nIGFueSBtb3JlLgo+Cj4g SSdtIHByZXR0eSBzdXJlIHRoYXQgYWxsIG9mIHRoaXMgaXMgaG9ycmlibGUgcmFjeSwgYnV0IG5v Ym9keSByZWFsbHkKPiBsb29rZWQgaW50byB0aGUgZGVzaWduIGZyb20gYSBoaWdoZXIgbGV2ZWwg YXMgZmFyIGFzIEkga25vdy4KClllYWggdGhlIHNjaGVkdWxlciB0aHJlYWQgaXMgZmluZSBiZWNh dXNlIGl0J3Mgc3RvcHBlZCwgYnV0IGl0IGFsc28KZG9lc24ndCBsb29rIGF0IHNjaGVkLT5yZWFk eSwgc28gaXQgY2FuJ3QgcmFjZS4gV2hhdCBkb2VzIHJhY2UgaXMgbmV3CnN1Ym1pc3Npb25zLCBh bmQgaWYgdGhleSBzdHVmZiBzb21ldGhpbmcgaW50byB0aGUgcXVldWUgdGhlbiBJJ20Kd29uZGVy aW5nIHdoYXQgaGFwcGVucyB0byB0aGF0LiBBbHNvIHdoYXQgaGFwcGVucyB0byB0aGUgcmVxdWVz dHMKYWxyZWFkeSBpbiB0aGUgcXVldWUuCgpFdmVudHVhbGx5IEkgZ3Vlc3MgdXNlcnNwYWNlIG5v dGljZXMgdGhlIEVOT0VOVCwgdGVhcnMgZG93biB0aGUKY29udGV4dCwgYW5kIHRoZSBrZXJuZWwg dGhlbiBhbHNvIHRlYXJzIGRvd24gdGhlIGNvbnRleHQgYW5kIGNsZWFucyB1cAp0aGUgbWVzcy4g QnV0IGl0J3MgcmF0aGVyIGluZ2xvcmlvdXMgdW50aWwgaXQgY29sbGFwc2VzIGRvd24gdG8gYQpj b2hlcmVudCBzdGF0ZSBhZ2FpbiBJIHRoaW5rLgoKT3IgaXMgdGhlcmUgc29tZXRoaW5nIHdpdGgg dGhlIHNjaGVkdWxlciByZXN0YXJ0IGZsb3cgd2hpY2ggaXMKZ3VhcmFudGVlZCB0byBjYXRjaCB0 aGVzZSwgYW5kIHdlJ3JlIG1heWJlIGp1c3QgbWlzc2luZyBhIGJ1bmNoIG9mCmJhcnJpZXJzPwoK RWl0aGVyIHdheSBJIHRoaW5rIGEgcHJvcGVyIGludGVyZmFjZSB0byB0ZXJtaW5hbGx5IHdlZGdl IGEgc2NoZWQKd291bGQgYmUgZ29vZCwgc28gdGhhdCBhdCBsZWFzdCB3ZSBjYW4gcGFzcyBiYWNr IHNvbWV0aGluZyBtZWFuaW5nZnVsCmxpa2UgLUVJTy4gQW5kIGFsc28gdGVsbCAidGhlIGdwdSBk aWVkIiBhcGFydCBmcm9tICJ0aGUgZHJpdmVyIGF1dGhvcgp0b3JlIGRvd24gdGhlIHNjaGVkdWxl ciB3aGlsZSBpdCB3YXMgc3RpbGwgaW4gdXNlIiwgd2hpY2ggSSB0aGluayB3ZQpyZWFsbHkgc2hv dWxkIGNhdGNoIHdpdGggc29tZSBXQVJOX09OLgoKQW55d2F5IGZvciB0aGUgaW1tZWRpYXRlIGlz c3VlIG9mICJkb24ndCBicmVhayBhbWRncHUiIEkgdGhpbmsgSSdsbApyZXNodWZmbGUgdGhlIHNw bGl0IGJldHdlZW4gam9iX2luaXQgYW5kIGpvYl9hcm0gYWdhaW4sIGFuZCBhZGQgYSBiaWcKY29t bWVudCB0byBqb2JfaW5pdCB0aGF0IGl0IGNhbiBmYWlsIHdpdGggRU5PRU5ULCBhbmQgd2h5LCBh bmQgd2hhdApraW5kIG9mIGludGVyZmFjZSB3b3VsZCBiZSBtb3JlIHByb3Blci4gaTkxNSB3aWxs IG5lZWQgdGhlIHRlcm1pbmFsbHkKd2VkZ2VkIGZsb3cgdG9vIHNvIEknbGwgcHJvYmFibHkgaGF2 ZSB0byBsb29rIGludG8gdGhpcywgYnV0IHRoYXQgd2lsbApuZWVkIHNvbWUgcHJvcGVyIHRob3Vn aHQuCgpDaGVlcnMsIERhbmllbAoKCj4KPiBDaHJpc3RpYW4uCj4KPgo+Cj4gPiAtRGFuaWVsCj4g Pgo+ID4KPiA+PiBSZWdhcmRzLAo+ID4+IENocmlzdGlhbi4KPiA+Pgo+ID4+PiBXZSBCVUdfT04o KSBoZXJlIGJlY2F1c2UgaW4gdGhlIG5leHQgcGF0Y2ggZHJtX3NjaGVkX2pvYl9pbml0KCkgd2ls bAo+ID4+PiBiZSBzcGxpdCB1cCwgd2l0aCBkcm1fc2NoZWRfam9iX2FybSgpIG5ldmVyIGZhaWxp bmcuIEFuZCB0aGF0J3MgdGhlCj4gPj4+IHBhcnQgd2hlcmUgdGhlIHJxIHNlbGVjdGlvbiB3aWxs IGVuZCB1cCBpbi4KPiA+Pj4KPiA+Pj4gTm90ZSB0aGF0IGlmIGhhdmluZyBhbiBlbXB0eSBzY2hl ZF9saXN0IHNldCBvbiBhbiBlbnRpdHkgaXMgaW5kZWVkIGEKPiA+Pj4gdmFsaWQgdXNlLWNhc2Us IHdlIGNhbiBrZWVwIHRoYXQgY2hlY2sgaW4gam9iX2luaXQgZXZlbiBhZnRlciB0aGUgc3BsaXQK PiA+Pj4gaW50byBqb2JfaW5pdC9hcm0uCj4gPj4+Cj4gPj4+IFNpZ25lZC1vZmYtYnk6IERhbmll bCBWZXR0ZXIgPGRhbmllbC52ZXR0ZXJAaW50ZWwuY29tPgo+ID4+PiBDYzogIkNocmlzdGlhbiBL w7ZuaWciIDxjaHJpc3RpYW4ua29lbmlnQGFtZC5jb20+Cj4gPj4+IENjOiBMdWJlbiBUdWlrb3Yg PGx1YmVuLnR1aWtvdkBhbWQuY29tPgo+ID4+PiBDYzogRGFuaWVsIFZldHRlciA8ZGFuaWVsLnZl dHRlckBmZndsbC5jaD4KPiA+Pj4gQ2M6IFN0ZXZlbiBQcmljZSA8c3RldmVuLnByaWNlQGFybS5j b20+Cj4gPj4+IENjOiBBbmRyZXkgR3JvZHpvdnNreSA8YW5kcmV5Lmdyb2R6b3Zza3lAYW1kLmNv bT4KPiA+Pj4gQ2M6IEJvcmlzIEJyZXppbGxvbiA8Ym9yaXMuYnJlemlsbG9uQGNvbGxhYm9yYS5j b20+Cj4gPj4+IENjOiBKYWNrIFpoYW5nIDxKYWNrLlpoYW5nMUBhbWQuY29tPgo+ID4+PiAtLS0K PiA+Pj4gICAgZHJpdmVycy9ncHUvZHJtL3NjaGVkdWxlci9zY2hlZF9lbnRpdHkuYyB8IDIgKy0K PiA+Pj4gICAgZHJpdmVycy9ncHUvZHJtL3NjaGVkdWxlci9zY2hlZF9tYWluLmMgICB8IDMgKy0t Cj4gPj4+ICAgIDIgZmlsZXMgY2hhbmdlZCwgMiBpbnNlcnRpb25zKCspLCAzIGRlbGV0aW9ucygt KQo+ID4+Pgo+ID4+PiBkaWZmIC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL3NjaGVkdWxlci9zY2hl ZF9lbnRpdHkuYyBiL2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfZW50aXR5LmMKPiA+ Pj4gaW5kZXggNzk1NTRhYTRkYmIxLi42ZmMxMTZlZTczMDIgMTAwNjQ0Cj4gPj4+IC0tLSBhL2Ry aXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfZW50aXR5LmMKPiA+Pj4gKysrIGIvZHJpdmVy cy9ncHUvZHJtL3NjaGVkdWxlci9zY2hlZF9lbnRpdHkuYwo+ID4+PiBAQCAtNDUsNyArNDUsNyBA QAo+ID4+PiAgICAgKiBAZ3VpbHR5OiBhdG9taWNfdCBzZXQgdG8gMSB3aGVuIGEgam9iIG9uIHRo aXMgcXVldWUKPiA+Pj4gICAgICogICAgICAgICAgaXMgZm91bmQgdG8gYmUgZ3VpbHR5IGNhdXNp bmcgYSB0aW1lb3V0Cj4gPj4+ICAgICAqCj4gPj4+IC0gKiBOb3RlOiB0aGUgc2NoZWRfbGlzdCBz aG91bGQgaGF2ZSBhdCBsZWFzdCBvbmUgZWxlbWVudCB0byBzY2hlZHVsZQo+ID4+PiArICogTm90 ZTogdGhlIHNjaGVkX2xpc3QgbXVzdCBoYXZlIGF0IGxlYXN0IG9uZSBlbGVtZW50IHRvIHNjaGVk dWxlCj4gPj4+ICAgICAqICAgICAgIHRoZSBlbnRpdHkKPiA+Pj4gICAgICoKPiA+Pj4gICAgICog UmV0dXJucyAwIG9uIHN1Y2Nlc3Mgb3IgYSBuZWdhdGl2ZSBlcnJvciBjb2RlIG9uIGZhaWx1cmUu Cj4gPj4+IGRpZmYgLS1naXQgYS9kcml2ZXJzL2dwdS9kcm0vc2NoZWR1bGVyL3NjaGVkX21haW4u YyBiL2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfbWFpbi5jCj4gPj4+IGluZGV4IDMz YzQxNGQ1NWZhYi4uMDFkZDQ3MTU0MTgxIDEwMDY0NAo+ID4+PiAtLS0gYS9kcml2ZXJzL2dwdS9k cm0vc2NoZWR1bGVyL3NjaGVkX21haW4uYwo+ID4+PiArKysgYi9kcml2ZXJzL2dwdS9kcm0vc2No ZWR1bGVyL3NjaGVkX21haW4uYwo+ID4+PiBAQCAtNTg2LDggKzU4Niw3IEBAIGludCBkcm1fc2No ZWRfam9iX2luaXQoc3RydWN0IGRybV9zY2hlZF9qb2IgKmpvYiwKPiA+Pj4gICAgICAgIHN0cnVj dCBkcm1fZ3B1X3NjaGVkdWxlciAqc2NoZWQ7Cj4gPj4+Cj4gPj4+ICAgICAgICBkcm1fc2NoZWRf ZW50aXR5X3NlbGVjdF9ycShlbnRpdHkpOwo+ID4+PiAtICAgICBpZiAoIWVudGl0eS0+cnEpCj4g Pj4+IC0gICAgICAgICAgICAgcmV0dXJuIC1FTk9FTlQ7Cj4gPj4+ICsgICAgIEJVR19PTighZW50 aXR5LT5ycSk7Cj4gPj4+Cj4gPj4+ICAgICAgICBzY2hlZCA9IGVudGl0eS0+cnEtPnNjaGVkOwo+ ID4+Pgo+ID4KPgoKCi0tCkRhbmllbCBWZXR0ZXIKU29mdHdhcmUgRW5naW5lZXIsIEludGVsIENv cnBvcmF0aW9uCmh0dHA6Ly9ibG9nLmZmd2xsLmNoCl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fCkludGVsLWdmeCBtYWlsaW5nIGxpc3QKSW50ZWwtZ2Z4QGxp c3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFu L2xpc3RpbmZvL2ludGVsLWdmeAo=