From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99546C07E96 for ; Thu, 8 Jul 2021 17:38:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8747E61920 for ; Thu, 8 Jul 2021 17:38:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229795AbhGHRkz (ORCPT ); Thu, 8 Jul 2021 13:40:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229650AbhGHRky (ORCPT ); Thu, 8 Jul 2021 13:40:54 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AAE0C061574 for ; Thu, 8 Jul 2021 10:38:12 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id f9so2862143wrq.11 for ; Thu, 08 Jul 2021 10:38:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6BU0hbbARRWppYffO9U0YO5ajiPUOHOT/nRCLYKYZXQ=; b=kr6ZcMmVDAflvEIUeFnswWB6ZiIkPVa9z9/fI0obC8Vb7Pvopx5YS5WJZv4bdsq/oW mr4fL2pW/582V40IZxBd9PE34PFCk+il+cZxaEgWH/cz9pqjeUDY2JxD38dbImpgy4Zo mJFuuypwHTPEUoZ4wTD1efLmhYU+C3NlBt/J0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6BU0hbbARRWppYffO9U0YO5ajiPUOHOT/nRCLYKYZXQ=; b=ou4LU0jcCANqlPT9qIeSZyPP5Fp5reEuY78wVhXLqV18TyB9ZTnVuiJ3TuQt3wzcPd uJcz7csTIQiTpDrwbTiOT8RctCJSuA/UWVYYeBfqffXTL9Q0d1jy+jtHeFYJ4gziTgSV qY9T6qzpgfrMONb4fKbWCx55sSDz7FlKpzj0hybvS0nTyOpkiqd28tOJwuxPaCaTjIKj 9feSNyr3yk6nJp5K14QRCq7+oP3vbzJdbaLwVPcVS46WCX0q0Qh+Sf2AcpPCwYtyb1kt as69rd8ZaJspMxzP13fPgnrz2YUhjIvFHMWnpCR3GQihwHhjEbTcsgInPWhO9jzhtRK4 aiXg== X-Gm-Message-State: AOAM5316+YqM1sfAqmwekbRtH19TZ1tqkaM7vC4Sq5hTfmbmrtnmja3+ IQ7+Eg7b9ROmLU6hFY8eFwXKVA== X-Google-Smtp-Source: ABdhPJy29DgncW3AGG5DBDPcJZXx3UPlvMJDuIqCzadTpGFlGvNjl+8mP5gWdAnwFH/LQzvEyC98jw== X-Received: by 2002:a5d:4252:: with SMTP id s18mr35789899wrr.88.1625765890822; Thu, 08 Jul 2021 10:38:10 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id l4sm2521853wme.26.2021.07.08.10.38.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jul 2021 10:38:10 -0700 (PDT) From: Daniel Vetter To: DRI Development Cc: Intel Graphics Development , Daniel Vetter , Steven Price , Daniel Vetter , David Airlie , Daniel Vetter , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Andrey Grodzovsky , Lee Jones , Nirmoy Das , Boris Brezillon , Luben Tuikov , Alex Deucher , Jack Zhang , linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org Subject: [PATCH v3 04/20] drm/sched: Add dependency tracking Date: Thu, 8 Jul 2021 19:37:38 +0200 Message-Id: <20210708173754.3877540-5-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> References: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Instead of just a callback we can just glue in the gem helpers that panfrost, v3d and lima currently use. There's really not that many ways to skin this cat. On the naming bikeshed: The idea for using _await_ to denote adding dependencies to a job comes from i915, where that's used quite extensively all over the place, in lots of datastructures. v2/3: Rebased. Reviewed-by: Steven Price (v1) Signed-off-by: Daniel Vetter Cc: David Airlie Cc: Daniel Vetter Cc: Sumit Semwal Cc: "Christian König" Cc: Andrey Grodzovsky Cc: Lee Jones Cc: Nirmoy Das Cc: Boris Brezillon Cc: Luben Tuikov Cc: Alex Deucher Cc: Jack Zhang Cc: linux-media@vger.kernel.org Cc: linaro-mm-sig@lists.linaro.org --- drivers/gpu/drm/scheduler/sched_entity.c | 18 +++- drivers/gpu/drm/scheduler/sched_main.c | 103 +++++++++++++++++++++++ include/drm/gpu_scheduler.h | 31 ++++++- 3 files changed, 146 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 4e1124ed80e0..c7e6d29c9a33 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -211,6 +211,19 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, job->sched->ops->free_job(job); } +static struct dma_fence * +drm_sched_job_dependency(struct drm_sched_job *job, + struct drm_sched_entity *entity) +{ + if (!xa_empty(&job->dependencies)) + return xa_erase(&job->dependencies, job->last_dependency++); + + if (job->sched->ops->dependency) + return job->sched->ops->dependency(job, entity); + + return NULL; +} + /** * drm_sched_entity_kill_jobs - Make sure all remaining jobs are killed * @@ -229,7 +242,7 @@ static void drm_sched_entity_kill_jobs(struct drm_sched_entity *entity) struct drm_sched_fence *s_fence = job->s_fence; /* Wait for all dependencies to avoid data corruptions */ - while ((f = job->sched->ops->dependency(job, entity))) + while ((f = drm_sched_job_dependency(job, entity))) dma_fence_wait(f, false); drm_sched_fence_scheduled(s_fence); @@ -419,7 +432,6 @@ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity) */ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) { - struct drm_gpu_scheduler *sched = entity->rq->sched; struct drm_sched_job *sched_job; sched_job = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); @@ -427,7 +439,7 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) return NULL; while ((entity->dependency = - sched->ops->dependency(sched_job, entity))) { + drm_sched_job_dependency(sched_job, entity))) { trace_drm_sched_job_wait_dep(sched_job, entity->dependency); if (drm_sched_entity_add_dependency_cb(entity)) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 7e94754eb34c..ad62f1d2991c 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -594,6 +594,8 @@ int drm_sched_job_init(struct drm_sched_job *job, INIT_LIST_HEAD(&job->list); + xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC); + return 0; } EXPORT_SYMBOL(drm_sched_job_init); @@ -631,6 +633,98 @@ void drm_sched_job_arm(struct drm_sched_job *job) } EXPORT_SYMBOL(drm_sched_job_arm); +/** + * drm_sched_job_await_fence - adds the fence as a job dependency + * @job: scheduler job to add the dependencies to + * @fence: the dma_fence to add to the list of dependencies. + * + * Note that @fence is consumed in both the success and error cases. + * + * Returns: + * 0 on success, or an error on failing to expand the array. + */ +int drm_sched_job_await_fence(struct drm_sched_job *job, + struct dma_fence *fence) +{ + struct dma_fence *entry; + unsigned long index; + u32 id = 0; + int ret; + + if (!fence) + return 0; + + /* Deduplicate if we already depend on a fence from the same context. + * This lets the size of the array of deps scale with the number of + * engines involved, rather than the number of BOs. + */ + xa_for_each(&job->dependencies, index, entry) { + if (entry->context != fence->context) + continue; + + if (dma_fence_is_later(fence, entry)) { + dma_fence_put(entry); + xa_store(&job->dependencies, index, fence, GFP_KERNEL); + } else { + dma_fence_put(fence); + } + return 0; + } + + ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); + if (ret != 0) + dma_fence_put(fence); + + return ret; +} +EXPORT_SYMBOL(drm_sched_job_await_fence); + +/** + * drm_sched_job_await_implicit - adds implicit dependencies as job dependencies + * @job: scheduler job to add the dependencies to + * @obj: the gem object to add new dependencies from. + * @write: whether the job might write the object (so we need to depend on + * shared fences in the reservation object). + * + * This should be called after drm_gem_lock_reservations() on your array of + * GEM objects used in the job but before updating the reservations with your + * own fences. + * + * Returns: + * 0 on success, or an error on failing to expand the array. + */ +int drm_sched_job_await_implicit(struct drm_sched_job *job, + struct drm_gem_object *obj, + bool write) +{ + int ret; + struct dma_fence **fences; + unsigned int i, fence_count; + + if (!write) { + struct dma_fence *fence = dma_resv_get_excl_unlocked(obj->resv); + + return drm_sched_job_await_fence(job, fence); + } + + ret = dma_resv_get_fences(obj->resv, NULL, &fence_count, &fences); + if (ret || !fence_count) + return ret; + + for (i = 0; i < fence_count; i++) { + ret = drm_sched_job_await_fence(job, fences[i]); + if (ret) + break; + } + + for (; i < fence_count; i++) + dma_fence_put(fences[i]); + kfree(fences); + return ret; +} +EXPORT_SYMBOL(drm_sched_job_await_implicit); + + /** * drm_sched_job_cleanup - clean up scheduler job resources * @job: scheduler job to clean up @@ -646,6 +740,9 @@ EXPORT_SYMBOL(drm_sched_job_arm); */ void drm_sched_job_cleanup(struct drm_sched_job *job) { + struct dma_fence *fence; + unsigned long index; + if (kref_read(&job->s_fence->finished.refcount)) { /* drm_sched_job_arm() has been called */ dma_fence_put(&job->s_fence->finished); @@ -655,6 +752,12 @@ void drm_sched_job_cleanup(struct drm_sched_job *job) } job->s_fence = NULL; + + xa_for_each(&job->dependencies, index, fence) { + dma_fence_put(fence); + } + xa_destroy(&job->dependencies); + } EXPORT_SYMBOL(drm_sched_job_cleanup); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 83afc3aa8e2f..74fb321dbc44 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -27,9 +27,12 @@ #include #include #include +#include #define MAX_WAIT_SCHED_ENTITY_Q_EMPTY msecs_to_jiffies(1000) +struct drm_gem_object; + struct drm_gpu_scheduler; struct drm_sched_rq; @@ -198,6 +201,16 @@ struct drm_sched_job { enum drm_sched_priority s_priority; struct drm_sched_entity *entity; struct dma_fence_cb cb; + /** + * @dependencies: + * + * Contains the dependencies as struct dma_fence for this job, see + * drm_sched_job_await_fence() and drm_sched_job_await_implicit(). + */ + struct xarray dependencies; + + /** @last_dependency: tracks @dependencies as they signal */ + unsigned long last_dependency; }; static inline bool drm_sched_invalidate_job(struct drm_sched_job *s_job, @@ -220,9 +233,14 @@ enum drm_gpu_sched_stat { */ struct drm_sched_backend_ops { /** - * @dependency: Called when the scheduler is considering scheduling - * this job next, to get another struct dma_fence for this job to - * block on. Once it returns NULL, run_job() may be called. + * @dependency: + * + * Called when the scheduler is considering scheduling this job next, to + * get another struct dma_fence for this job to block on. Once it + * returns NULL, run_job() may be called. + * + * If a driver exclusively uses drm_sched_job_await_fence() and + * drm_sched_job_await_implicit() this can be ommitted and left as NULL. */ struct dma_fence *(*dependency)(struct drm_sched_job *sched_job, struct drm_sched_entity *s_entity); @@ -349,6 +367,13 @@ int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, void *owner); void drm_sched_job_arm(struct drm_sched_job *job); +int drm_sched_job_await_fence(struct drm_sched_job *job, + struct dma_fence *fence); +int drm_sched_job_await_implicit(struct drm_sched_job *job, + struct drm_gem_object *obj, + bool write); + + void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list); -- 2.32.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AB1AC07E9C for ; Thu, 8 Jul 2021 17:38:26 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D885B6191D for ; Thu, 8 Jul 2021 17:38:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D885B6191D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8D5566E953; Thu, 8 Jul 2021 17:38:13 +0000 (UTC) Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1B1046E949 for ; Thu, 8 Jul 2021 17:38:12 +0000 (UTC) Received: by mail-wr1-x429.google.com with SMTP id f17so8524521wrt.6 for ; Thu, 08 Jul 2021 10:38:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6BU0hbbARRWppYffO9U0YO5ajiPUOHOT/nRCLYKYZXQ=; b=kr6ZcMmVDAflvEIUeFnswWB6ZiIkPVa9z9/fI0obC8Vb7Pvopx5YS5WJZv4bdsq/oW mr4fL2pW/582V40IZxBd9PE34PFCk+il+cZxaEgWH/cz9pqjeUDY2JxD38dbImpgy4Zo mJFuuypwHTPEUoZ4wTD1efLmhYU+C3NlBt/J0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6BU0hbbARRWppYffO9U0YO5ajiPUOHOT/nRCLYKYZXQ=; b=mOPFYNObvw1zkp8BiFSQp9J4c2XRNlHOrEz5E/R6re1mMtdhbarfmxTmm+Zykzo7Ed NYGozDWZIML6fWf+QXMsB+/SmZBetcA/9URkM5t6aBmcexLp3WXzGa5SNnBLIiguKx9o XlxZ/Zt4KgcBWmD861iIAkPRAW4v7rWMfIIrdB0ST7k1gd4YBsguAhvQ1ybgpoxggvK/ qlD/STtEWK00oKOM5eLtvkdWltFDSNP9IMg95EXwpoFlFDsab1yW57XMGu6tnt5tdJa/ 2151kH0MseqLA+Tdbqe7x0IULXmQ9OLzMMrxWRMPtmfP/FWHn27S2xoBQHSxTpm4mRTw PYbw== X-Gm-Message-State: AOAM533Erdqs9KNSL685J15UyP3tu1O7dtPeNDV5uNFyQG2XiTo6Ky+E c021Ax316roK83bwVmXnGZDzyyJJIk4= X-Google-Smtp-Source: ABdhPJy29DgncW3AGG5DBDPcJZXx3UPlvMJDuIqCzadTpGFlGvNjl+8mP5gWdAnwFH/LQzvEyC98jw== X-Received: by 2002:a5d:4252:: with SMTP id s18mr35789899wrr.88.1625765890822; Thu, 08 Jul 2021 10:38:10 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id l4sm2521853wme.26.2021.07.08.10.38.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jul 2021 10:38:10 -0700 (PDT) From: Daniel Vetter To: DRI Development Subject: [PATCH v3 04/20] drm/sched: Add dependency tracking Date: Thu, 8 Jul 2021 19:37:38 +0200 Message-Id: <20210708173754.3877540-5-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> References: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jack Zhang , =?UTF-8?q?Christian=20K=C3=B6nig?= , David Airlie , Daniel Vetter , Intel Graphics Development , Steven Price , linaro-mm-sig@lists.linaro.org, Boris Brezillon , Alex Deucher , Daniel Vetter , linux-media@vger.kernel.org, Lee Jones , Luben Tuikov , Nirmoy Das Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Instead of just a callback we can just glue in the gem helpers that panfrost, v3d and lima currently use. There's really not that many ways to skin this cat. On the naming bikeshed: The idea for using _await_ to denote adding dependencies to a job comes from i915, where that's used quite extensively all over the place, in lots of datastructures. v2/3: Rebased. Reviewed-by: Steven Price (v1) Signed-off-by: Daniel Vetter Cc: David Airlie Cc: Daniel Vetter Cc: Sumit Semwal Cc: "Christian König" Cc: Andrey Grodzovsky Cc: Lee Jones Cc: Nirmoy Das Cc: Boris Brezillon Cc: Luben Tuikov Cc: Alex Deucher Cc: Jack Zhang Cc: linux-media@vger.kernel.org Cc: linaro-mm-sig@lists.linaro.org --- drivers/gpu/drm/scheduler/sched_entity.c | 18 +++- drivers/gpu/drm/scheduler/sched_main.c | 103 +++++++++++++++++++++++ include/drm/gpu_scheduler.h | 31 ++++++- 3 files changed, 146 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 4e1124ed80e0..c7e6d29c9a33 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -211,6 +211,19 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, job->sched->ops->free_job(job); } +static struct dma_fence * +drm_sched_job_dependency(struct drm_sched_job *job, + struct drm_sched_entity *entity) +{ + if (!xa_empty(&job->dependencies)) + return xa_erase(&job->dependencies, job->last_dependency++); + + if (job->sched->ops->dependency) + return job->sched->ops->dependency(job, entity); + + return NULL; +} + /** * drm_sched_entity_kill_jobs - Make sure all remaining jobs are killed * @@ -229,7 +242,7 @@ static void drm_sched_entity_kill_jobs(struct drm_sched_entity *entity) struct drm_sched_fence *s_fence = job->s_fence; /* Wait for all dependencies to avoid data corruptions */ - while ((f = job->sched->ops->dependency(job, entity))) + while ((f = drm_sched_job_dependency(job, entity))) dma_fence_wait(f, false); drm_sched_fence_scheduled(s_fence); @@ -419,7 +432,6 @@ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity) */ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) { - struct drm_gpu_scheduler *sched = entity->rq->sched; struct drm_sched_job *sched_job; sched_job = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); @@ -427,7 +439,7 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) return NULL; while ((entity->dependency = - sched->ops->dependency(sched_job, entity))) { + drm_sched_job_dependency(sched_job, entity))) { trace_drm_sched_job_wait_dep(sched_job, entity->dependency); if (drm_sched_entity_add_dependency_cb(entity)) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 7e94754eb34c..ad62f1d2991c 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -594,6 +594,8 @@ int drm_sched_job_init(struct drm_sched_job *job, INIT_LIST_HEAD(&job->list); + xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC); + return 0; } EXPORT_SYMBOL(drm_sched_job_init); @@ -631,6 +633,98 @@ void drm_sched_job_arm(struct drm_sched_job *job) } EXPORT_SYMBOL(drm_sched_job_arm); +/** + * drm_sched_job_await_fence - adds the fence as a job dependency + * @job: scheduler job to add the dependencies to + * @fence: the dma_fence to add to the list of dependencies. + * + * Note that @fence is consumed in both the success and error cases. + * + * Returns: + * 0 on success, or an error on failing to expand the array. + */ +int drm_sched_job_await_fence(struct drm_sched_job *job, + struct dma_fence *fence) +{ + struct dma_fence *entry; + unsigned long index; + u32 id = 0; + int ret; + + if (!fence) + return 0; + + /* Deduplicate if we already depend on a fence from the same context. + * This lets the size of the array of deps scale with the number of + * engines involved, rather than the number of BOs. + */ + xa_for_each(&job->dependencies, index, entry) { + if (entry->context != fence->context) + continue; + + if (dma_fence_is_later(fence, entry)) { + dma_fence_put(entry); + xa_store(&job->dependencies, index, fence, GFP_KERNEL); + } else { + dma_fence_put(fence); + } + return 0; + } + + ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); + if (ret != 0) + dma_fence_put(fence); + + return ret; +} +EXPORT_SYMBOL(drm_sched_job_await_fence); + +/** + * drm_sched_job_await_implicit - adds implicit dependencies as job dependencies + * @job: scheduler job to add the dependencies to + * @obj: the gem object to add new dependencies from. + * @write: whether the job might write the object (so we need to depend on + * shared fences in the reservation object). + * + * This should be called after drm_gem_lock_reservations() on your array of + * GEM objects used in the job but before updating the reservations with your + * own fences. + * + * Returns: + * 0 on success, or an error on failing to expand the array. + */ +int drm_sched_job_await_implicit(struct drm_sched_job *job, + struct drm_gem_object *obj, + bool write) +{ + int ret; + struct dma_fence **fences; + unsigned int i, fence_count; + + if (!write) { + struct dma_fence *fence = dma_resv_get_excl_unlocked(obj->resv); + + return drm_sched_job_await_fence(job, fence); + } + + ret = dma_resv_get_fences(obj->resv, NULL, &fence_count, &fences); + if (ret || !fence_count) + return ret; + + for (i = 0; i < fence_count; i++) { + ret = drm_sched_job_await_fence(job, fences[i]); + if (ret) + break; + } + + for (; i < fence_count; i++) + dma_fence_put(fences[i]); + kfree(fences); + return ret; +} +EXPORT_SYMBOL(drm_sched_job_await_implicit); + + /** * drm_sched_job_cleanup - clean up scheduler job resources * @job: scheduler job to clean up @@ -646,6 +740,9 @@ EXPORT_SYMBOL(drm_sched_job_arm); */ void drm_sched_job_cleanup(struct drm_sched_job *job) { + struct dma_fence *fence; + unsigned long index; + if (kref_read(&job->s_fence->finished.refcount)) { /* drm_sched_job_arm() has been called */ dma_fence_put(&job->s_fence->finished); @@ -655,6 +752,12 @@ void drm_sched_job_cleanup(struct drm_sched_job *job) } job->s_fence = NULL; + + xa_for_each(&job->dependencies, index, fence) { + dma_fence_put(fence); + } + xa_destroy(&job->dependencies); + } EXPORT_SYMBOL(drm_sched_job_cleanup); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 83afc3aa8e2f..74fb321dbc44 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -27,9 +27,12 @@ #include #include #include +#include #define MAX_WAIT_SCHED_ENTITY_Q_EMPTY msecs_to_jiffies(1000) +struct drm_gem_object; + struct drm_gpu_scheduler; struct drm_sched_rq; @@ -198,6 +201,16 @@ struct drm_sched_job { enum drm_sched_priority s_priority; struct drm_sched_entity *entity; struct dma_fence_cb cb; + /** + * @dependencies: + * + * Contains the dependencies as struct dma_fence for this job, see + * drm_sched_job_await_fence() and drm_sched_job_await_implicit(). + */ + struct xarray dependencies; + + /** @last_dependency: tracks @dependencies as they signal */ + unsigned long last_dependency; }; static inline bool drm_sched_invalidate_job(struct drm_sched_job *s_job, @@ -220,9 +233,14 @@ enum drm_gpu_sched_stat { */ struct drm_sched_backend_ops { /** - * @dependency: Called when the scheduler is considering scheduling - * this job next, to get another struct dma_fence for this job to - * block on. Once it returns NULL, run_job() may be called. + * @dependency: + * + * Called when the scheduler is considering scheduling this job next, to + * get another struct dma_fence for this job to block on. Once it + * returns NULL, run_job() may be called. + * + * If a driver exclusively uses drm_sched_job_await_fence() and + * drm_sched_job_await_implicit() this can be ommitted and left as NULL. */ struct dma_fence *(*dependency)(struct drm_sched_job *sched_job, struct drm_sched_entity *s_entity); @@ -349,6 +367,13 @@ int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, void *owner); void drm_sched_job_arm(struct drm_sched_job *job); +int drm_sched_job_await_fence(struct drm_sched_job *job, + struct dma_fence *fence); +int drm_sched_job_await_implicit(struct drm_sched_job *job, + struct drm_gem_object *obj, + bool write); + + void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list); -- 2.32.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91301C07E96 for ; Thu, 8 Jul 2021 17:38:30 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6160F61879 for ; Thu, 8 Jul 2021 17:38:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6160F61879 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DAF5A6E95B; Thu, 8 Jul 2021 17:38:14 +0000 (UTC) Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by gabe.freedesktop.org (Postfix) with ESMTPS id 323046E952 for ; Thu, 8 Jul 2021 17:38:12 +0000 (UTC) Received: by mail-wr1-x42d.google.com with SMTP id l7so7588192wrv.7 for ; Thu, 08 Jul 2021 10:38:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6BU0hbbARRWppYffO9U0YO5ajiPUOHOT/nRCLYKYZXQ=; b=kr6ZcMmVDAflvEIUeFnswWB6ZiIkPVa9z9/fI0obC8Vb7Pvopx5YS5WJZv4bdsq/oW mr4fL2pW/582V40IZxBd9PE34PFCk+il+cZxaEgWH/cz9pqjeUDY2JxD38dbImpgy4Zo mJFuuypwHTPEUoZ4wTD1efLmhYU+C3NlBt/J0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6BU0hbbARRWppYffO9U0YO5ajiPUOHOT/nRCLYKYZXQ=; b=ov32Um08MXclXhlfBJmOzHyB/ksBDmxmKzEq/Uj7Wm0NNm6ZH2NGLTP7Z+fmTiKGLB 7zXlI0u4oVl+HrEXNYWqbpXO3fAuUMg6O4pD/mkfqs6Kic60SNnpleKulL7HHy7xmbun 7VugezsjYTXuEw8kPNsDYGsPiydAQpbrHgSOGIzihaMiVb5+M7QvWdtHgH0z5Ryrh0mH 1Af6RhZkQkNgsvR9eXRXOKpso4MVCv78sd8CkSGMZAt/hSDWQv+X3kfZGVuaR64NWsKS MWr+u8lF5qAR9DzuehrT9linaKi+czcEzSWBi7H8XfdGXwij43xGNyRJnqZ50Tnns+7n UAwQ== X-Gm-Message-State: AOAM533fJ/1SgU7kMlUpRNyLB1yxlUdwOyc/wf7FUEY3FuxyMrsW9oJk R69oRaRdHjkLC88JqVB8JPLgMMe8MoQ= X-Google-Smtp-Source: ABdhPJy29DgncW3AGG5DBDPcJZXx3UPlvMJDuIqCzadTpGFlGvNjl+8mP5gWdAnwFH/LQzvEyC98jw== X-Received: by 2002:a5d:4252:: with SMTP id s18mr35789899wrr.88.1625765890822; Thu, 08 Jul 2021 10:38:10 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id l4sm2521853wme.26.2021.07.08.10.38.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jul 2021 10:38:10 -0700 (PDT) From: Daniel Vetter To: DRI Development Date: Thu, 8 Jul 2021 19:37:38 +0200 Message-Id: <20210708173754.3877540-5-daniel.vetter@ffwll.ch> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> References: <20210708173754.3877540-1-daniel.vetter@ffwll.ch> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v3 04/20] drm/sched: Add dependency tracking X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrey Grodzovsky , Jack Zhang , =?UTF-8?q?Christian=20K=C3=B6nig?= , David Airlie , Daniel Vetter , Intel Graphics Development , Steven Price , linaro-mm-sig@lists.linaro.org, Alex Deucher , Daniel Vetter , linux-media@vger.kernel.org, Lee Jones , Sumit Semwal , Luben Tuikov , Nirmoy Das Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" SW5zdGVhZCBvZiBqdXN0IGEgY2FsbGJhY2sgd2UgY2FuIGp1c3QgZ2x1ZSBpbiB0aGUgZ2VtIGhl bHBlcnMgdGhhdApwYW5mcm9zdCwgdjNkIGFuZCBsaW1hIGN1cnJlbnRseSB1c2UuIFRoZXJlJ3Mg cmVhbGx5IG5vdCB0aGF0IG1hbnkKd2F5cyB0byBza2luIHRoaXMgY2F0LgoKT24gdGhlIG5hbWlu ZyBiaWtlc2hlZDogVGhlIGlkZWEgZm9yIHVzaW5nIF9hd2FpdF8gdG8gZGVub3RlIGFkZGluZwpk ZXBlbmRlbmNpZXMgdG8gYSBqb2IgY29tZXMgZnJvbSBpOTE1LCB3aGVyZSB0aGF0J3MgdXNlZCBx dWl0ZQpleHRlbnNpdmVseSBhbGwgb3ZlciB0aGUgcGxhY2UsIGluIGxvdHMgb2YgZGF0YXN0cnVj dHVyZXMuCgp2Mi8zOiBSZWJhc2VkLgoKUmV2aWV3ZWQtYnk6IFN0ZXZlbiBQcmljZSA8c3RldmVu LnByaWNlQGFybS5jb20+ICh2MSkKU2lnbmVkLW9mZi1ieTogRGFuaWVsIFZldHRlciA8ZGFuaWVs LnZldHRlckBpbnRlbC5jb20+CkNjOiBEYXZpZCBBaXJsaWUgPGFpcmxpZWRAbGludXguaWU+CkNj OiBEYW5pZWwgVmV0dGVyIDxkYW5pZWxAZmZ3bGwuY2g+CkNjOiBTdW1pdCBTZW13YWwgPHN1bWl0 LnNlbXdhbEBsaW5hcm8ub3JnPgpDYzogIkNocmlzdGlhbiBLw7ZuaWciIDxjaHJpc3RpYW4ua29l bmlnQGFtZC5jb20+CkNjOiBBbmRyZXkgR3JvZHpvdnNreSA8YW5kcmV5Lmdyb2R6b3Zza3lAYW1k LmNvbT4KQ2M6IExlZSBKb25lcyA8bGVlLmpvbmVzQGxpbmFyby5vcmc+CkNjOiBOaXJtb3kgRGFz IDxuaXJtb3kuYWllbWRAZ21haWwuY29tPgpDYzogQm9yaXMgQnJlemlsbG9uIDxib3Jpcy5icmV6 aWxsb25AY29sbGFib3JhLmNvbT4KQ2M6IEx1YmVuIFR1aWtvdiA8bHViZW4udHVpa292QGFtZC5j b20+CkNjOiBBbGV4IERldWNoZXIgPGFsZXhhbmRlci5kZXVjaGVyQGFtZC5jb20+CkNjOiBKYWNr IFpoYW5nIDxKYWNrLlpoYW5nMUBhbWQuY29tPgpDYzogbGludXgtbWVkaWFAdmdlci5rZXJuZWwu b3JnCkNjOiBsaW5hcm8tbW0tc2lnQGxpc3RzLmxpbmFyby5vcmcKLS0tCiBkcml2ZXJzL2dwdS9k cm0vc2NoZWR1bGVyL3NjaGVkX2VudGl0eS5jIHwgIDE4ICsrKy0KIGRyaXZlcnMvZ3B1L2RybS9z Y2hlZHVsZXIvc2NoZWRfbWFpbi5jICAgfCAxMDMgKysrKysrKysrKysrKysrKysrKysrKysKIGlu Y2x1ZGUvZHJtL2dwdV9zY2hlZHVsZXIuaCAgICAgICAgICAgICAgfCAgMzEgKysrKysrLQogMyBm aWxlcyBjaGFuZ2VkLCAxNDYgaW5zZXJ0aW9ucygrKSwgNiBkZWxldGlvbnMoLSkKCmRpZmYgLS1n aXQgYS9kcml2ZXJzL2dwdS9kcm0vc2NoZWR1bGVyL3NjaGVkX2VudGl0eS5jIGIvZHJpdmVycy9n cHUvZHJtL3NjaGVkdWxlci9zY2hlZF9lbnRpdHkuYwppbmRleCA0ZTExMjRlZDgwZTAuLmM3ZTZk MjljOWEzMyAxMDA2NDQKLS0tIGEvZHJpdmVycy9ncHUvZHJtL3NjaGVkdWxlci9zY2hlZF9lbnRp dHkuYworKysgYi9kcml2ZXJzL2dwdS9kcm0vc2NoZWR1bGVyL3NjaGVkX2VudGl0eS5jCkBAIC0y MTEsNiArMjExLDE5IEBAIHN0YXRpYyB2b2lkIGRybV9zY2hlZF9lbnRpdHlfa2lsbF9qb2JzX2Ni KHN0cnVjdCBkbWFfZmVuY2UgKmYsCiAJam9iLT5zY2hlZC0+b3BzLT5mcmVlX2pvYihqb2IpOwog fQogCitzdGF0aWMgc3RydWN0IGRtYV9mZW5jZSAqCitkcm1fc2NoZWRfam9iX2RlcGVuZGVuY3ko c3RydWN0IGRybV9zY2hlZF9qb2IgKmpvYiwKKwkJCSBzdHJ1Y3QgZHJtX3NjaGVkX2VudGl0eSAq ZW50aXR5KQoreworCWlmICgheGFfZW1wdHkoJmpvYi0+ZGVwZW5kZW5jaWVzKSkKKwkJcmV0dXJu IHhhX2VyYXNlKCZqb2ItPmRlcGVuZGVuY2llcywgam9iLT5sYXN0X2RlcGVuZGVuY3krKyk7CisK KwlpZiAoam9iLT5zY2hlZC0+b3BzLT5kZXBlbmRlbmN5KQorCQlyZXR1cm4gam9iLT5zY2hlZC0+ b3BzLT5kZXBlbmRlbmN5KGpvYiwgZW50aXR5KTsKKworCXJldHVybiBOVUxMOworfQorCiAvKioK ICAqIGRybV9zY2hlZF9lbnRpdHlfa2lsbF9qb2JzIC0gTWFrZSBzdXJlIGFsbCByZW1haW5pbmcg am9icyBhcmUga2lsbGVkCiAgKgpAQCAtMjI5LDcgKzI0Miw3IEBAIHN0YXRpYyB2b2lkIGRybV9z Y2hlZF9lbnRpdHlfa2lsbF9qb2JzKHN0cnVjdCBkcm1fc2NoZWRfZW50aXR5ICplbnRpdHkpCiAJ CXN0cnVjdCBkcm1fc2NoZWRfZmVuY2UgKnNfZmVuY2UgPSBqb2ItPnNfZmVuY2U7CiAKIAkJLyog V2FpdCBmb3IgYWxsIGRlcGVuZGVuY2llcyB0byBhdm9pZCBkYXRhIGNvcnJ1cHRpb25zICovCi0J CXdoaWxlICgoZiA9IGpvYi0+c2NoZWQtPm9wcy0+ZGVwZW5kZW5jeShqb2IsIGVudGl0eSkpKQor CQl3aGlsZSAoKGYgPSBkcm1fc2NoZWRfam9iX2RlcGVuZGVuY3koam9iLCBlbnRpdHkpKSkKIAkJ CWRtYV9mZW5jZV93YWl0KGYsIGZhbHNlKTsKIAogCQlkcm1fc2NoZWRfZmVuY2Vfc2NoZWR1bGVk KHNfZmVuY2UpOwpAQCAtNDE5LDcgKzQzMiw2IEBAIHN0YXRpYyBib29sIGRybV9zY2hlZF9lbnRp dHlfYWRkX2RlcGVuZGVuY3lfY2Ioc3RydWN0IGRybV9zY2hlZF9lbnRpdHkgKmVudGl0eSkKICAq Lwogc3RydWN0IGRybV9zY2hlZF9qb2IgKmRybV9zY2hlZF9lbnRpdHlfcG9wX2pvYihzdHJ1Y3Qg ZHJtX3NjaGVkX2VudGl0eSAqZW50aXR5KQogewotCXN0cnVjdCBkcm1fZ3B1X3NjaGVkdWxlciAq c2NoZWQgPSBlbnRpdHktPnJxLT5zY2hlZDsKIAlzdHJ1Y3QgZHJtX3NjaGVkX2pvYiAqc2NoZWRf am9iOwogCiAJc2NoZWRfam9iID0gdG9fZHJtX3NjaGVkX2pvYihzcHNjX3F1ZXVlX3BlZWsoJmVu dGl0eS0+am9iX3F1ZXVlKSk7CkBAIC00MjcsNyArNDM5LDcgQEAgc3RydWN0IGRybV9zY2hlZF9q b2IgKmRybV9zY2hlZF9lbnRpdHlfcG9wX2pvYihzdHJ1Y3QgZHJtX3NjaGVkX2VudGl0eSAqZW50 aXR5KQogCQlyZXR1cm4gTlVMTDsKIAogCXdoaWxlICgoZW50aXR5LT5kZXBlbmRlbmN5ID0KLQkJ CXNjaGVkLT5vcHMtPmRlcGVuZGVuY3koc2NoZWRfam9iLCBlbnRpdHkpKSkgeworCQkJZHJtX3Nj aGVkX2pvYl9kZXBlbmRlbmN5KHNjaGVkX2pvYiwgZW50aXR5KSkpIHsKIAkJdHJhY2VfZHJtX3Nj aGVkX2pvYl93YWl0X2RlcChzY2hlZF9qb2IsIGVudGl0eS0+ZGVwZW5kZW5jeSk7CiAKIAkJaWYg KGRybV9zY2hlZF9lbnRpdHlfYWRkX2RlcGVuZGVuY3lfY2IoZW50aXR5KSkKZGlmZiAtLWdpdCBh L2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfbWFpbi5jIGIvZHJpdmVycy9ncHUvZHJt L3NjaGVkdWxlci9zY2hlZF9tYWluLmMKaW5kZXggN2U5NDc1NGViMzRjLi5hZDYyZjFkMjk5MWMg MTAwNjQ0Ci0tLSBhL2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfbWFpbi5jCisrKyBi L2RyaXZlcnMvZ3B1L2RybS9zY2hlZHVsZXIvc2NoZWRfbWFpbi5jCkBAIC01OTQsNiArNTk0LDgg QEAgaW50IGRybV9zY2hlZF9qb2JfaW5pdChzdHJ1Y3QgZHJtX3NjaGVkX2pvYiAqam9iLAogCiAJ SU5JVF9MSVNUX0hFQUQoJmpvYi0+bGlzdCk7CiAKKwl4YV9pbml0X2ZsYWdzKCZqb2ItPmRlcGVu ZGVuY2llcywgWEFfRkxBR1NfQUxMT0MpOworCiAJcmV0dXJuIDA7CiB9CiBFWFBPUlRfU1lNQk9M KGRybV9zY2hlZF9qb2JfaW5pdCk7CkBAIC02MzEsNiArNjMzLDk4IEBAIHZvaWQgZHJtX3NjaGVk X2pvYl9hcm0oc3RydWN0IGRybV9zY2hlZF9qb2IgKmpvYikKIH0KIEVYUE9SVF9TWU1CT0woZHJt X3NjaGVkX2pvYl9hcm0pOwogCisvKioKKyAqIGRybV9zY2hlZF9qb2JfYXdhaXRfZmVuY2UgLSBh ZGRzIHRoZSBmZW5jZSBhcyBhIGpvYiBkZXBlbmRlbmN5CisgKiBAam9iOiBzY2hlZHVsZXIgam9i IHRvIGFkZCB0aGUgZGVwZW5kZW5jaWVzIHRvCisgKiBAZmVuY2U6IHRoZSBkbWFfZmVuY2UgdG8g YWRkIHRvIHRoZSBsaXN0IG9mIGRlcGVuZGVuY2llcy4KKyAqCisgKiBOb3RlIHRoYXQgQGZlbmNl IGlzIGNvbnN1bWVkIGluIGJvdGggdGhlIHN1Y2Nlc3MgYW5kIGVycm9yIGNhc2VzLgorICoKKyAq IFJldHVybnM6CisgKiAwIG9uIHN1Y2Nlc3MsIG9yIGFuIGVycm9yIG9uIGZhaWxpbmcgdG8gZXhw YW5kIHRoZSBhcnJheS4KKyAqLworaW50IGRybV9zY2hlZF9qb2JfYXdhaXRfZmVuY2Uoc3RydWN0 IGRybV9zY2hlZF9qb2IgKmpvYiwKKwkJCSAgICAgIHN0cnVjdCBkbWFfZmVuY2UgKmZlbmNlKQor eworCXN0cnVjdCBkbWFfZmVuY2UgKmVudHJ5OworCXVuc2lnbmVkIGxvbmcgaW5kZXg7CisJdTMy IGlkID0gMDsKKwlpbnQgcmV0OworCisJaWYgKCFmZW5jZSkKKwkJcmV0dXJuIDA7CisKKwkvKiBE ZWR1cGxpY2F0ZSBpZiB3ZSBhbHJlYWR5IGRlcGVuZCBvbiBhIGZlbmNlIGZyb20gdGhlIHNhbWUg Y29udGV4dC4KKwkgKiBUaGlzIGxldHMgdGhlIHNpemUgb2YgdGhlIGFycmF5IG9mIGRlcHMgc2Nh bGUgd2l0aCB0aGUgbnVtYmVyIG9mCisJICogZW5naW5lcyBpbnZvbHZlZCwgcmF0aGVyIHRoYW4g dGhlIG51bWJlciBvZiBCT3MuCisJICovCisJeGFfZm9yX2VhY2goJmpvYi0+ZGVwZW5kZW5jaWVz LCBpbmRleCwgZW50cnkpIHsKKwkJaWYgKGVudHJ5LT5jb250ZXh0ICE9IGZlbmNlLT5jb250ZXh0 KQorCQkJY29udGludWU7CisKKwkJaWYgKGRtYV9mZW5jZV9pc19sYXRlcihmZW5jZSwgZW50cnkp KSB7CisJCQlkbWFfZmVuY2VfcHV0KGVudHJ5KTsKKwkJCXhhX3N0b3JlKCZqb2ItPmRlcGVuZGVu Y2llcywgaW5kZXgsIGZlbmNlLCBHRlBfS0VSTkVMKTsKKwkJfSBlbHNlIHsKKwkJCWRtYV9mZW5j ZV9wdXQoZmVuY2UpOworCQl9CisJCXJldHVybiAwOworCX0KKworCXJldCA9IHhhX2FsbG9jKCZq b2ItPmRlcGVuZGVuY2llcywgJmlkLCBmZW5jZSwgeGFfbGltaXRfMzJiLCBHRlBfS0VSTkVMKTsK KwlpZiAocmV0ICE9IDApCisJCWRtYV9mZW5jZV9wdXQoZmVuY2UpOworCisJcmV0dXJuIHJldDsK K30KK0VYUE9SVF9TWU1CT0woZHJtX3NjaGVkX2pvYl9hd2FpdF9mZW5jZSk7CisKKy8qKgorICog ZHJtX3NjaGVkX2pvYl9hd2FpdF9pbXBsaWNpdCAtIGFkZHMgaW1wbGljaXQgZGVwZW5kZW5jaWVz IGFzIGpvYiBkZXBlbmRlbmNpZXMKKyAqIEBqb2I6IHNjaGVkdWxlciBqb2IgdG8gYWRkIHRoZSBk ZXBlbmRlbmNpZXMgdG8KKyAqIEBvYmo6IHRoZSBnZW0gb2JqZWN0IHRvIGFkZCBuZXcgZGVwZW5k ZW5jaWVzIGZyb20uCisgKiBAd3JpdGU6IHdoZXRoZXIgdGhlIGpvYiBtaWdodCB3cml0ZSB0aGUg b2JqZWN0IChzbyB3ZSBuZWVkIHRvIGRlcGVuZCBvbgorICogc2hhcmVkIGZlbmNlcyBpbiB0aGUg cmVzZXJ2YXRpb24gb2JqZWN0KS4KKyAqCisgKiBUaGlzIHNob3VsZCBiZSBjYWxsZWQgYWZ0ZXIg ZHJtX2dlbV9sb2NrX3Jlc2VydmF0aW9ucygpIG9uIHlvdXIgYXJyYXkgb2YKKyAqIEdFTSBvYmpl Y3RzIHVzZWQgaW4gdGhlIGpvYiBidXQgYmVmb3JlIHVwZGF0aW5nIHRoZSByZXNlcnZhdGlvbnMg d2l0aCB5b3VyCisgKiBvd24gZmVuY2VzLgorICoKKyAqIFJldHVybnM6CisgKiAwIG9uIHN1Y2Nl c3MsIG9yIGFuIGVycm9yIG9uIGZhaWxpbmcgdG8gZXhwYW5kIHRoZSBhcnJheS4KKyAqLworaW50 IGRybV9zY2hlZF9qb2JfYXdhaXRfaW1wbGljaXQoc3RydWN0IGRybV9zY2hlZF9qb2IgKmpvYiwK KwkJCQkgc3RydWN0IGRybV9nZW1fb2JqZWN0ICpvYmosCisJCQkJIGJvb2wgd3JpdGUpCit7CisJ aW50IHJldDsKKwlzdHJ1Y3QgZG1hX2ZlbmNlICoqZmVuY2VzOworCXVuc2lnbmVkIGludCBpLCBm ZW5jZV9jb3VudDsKKworCWlmICghd3JpdGUpIHsKKwkJc3RydWN0IGRtYV9mZW5jZSAqZmVuY2Ug PSBkbWFfcmVzdl9nZXRfZXhjbF91bmxvY2tlZChvYmotPnJlc3YpOworCisJCXJldHVybiBkcm1f c2NoZWRfam9iX2F3YWl0X2ZlbmNlKGpvYiwgZmVuY2UpOworCX0KKworCXJldCA9IGRtYV9yZXN2 X2dldF9mZW5jZXMob2JqLT5yZXN2LCBOVUxMLCAmZmVuY2VfY291bnQsICZmZW5jZXMpOworCWlm IChyZXQgfHwgIWZlbmNlX2NvdW50KQorCQlyZXR1cm4gcmV0OworCisJZm9yIChpID0gMDsgaSA8 IGZlbmNlX2NvdW50OyBpKyspIHsKKwkJcmV0ID0gZHJtX3NjaGVkX2pvYl9hd2FpdF9mZW5jZShq b2IsIGZlbmNlc1tpXSk7CisJCWlmIChyZXQpCisJCQlicmVhazsKKwl9CisKKwlmb3IgKDsgaSA8 IGZlbmNlX2NvdW50OyBpKyspCisJCWRtYV9mZW5jZV9wdXQoZmVuY2VzW2ldKTsKKwlrZnJlZShm ZW5jZXMpOworCXJldHVybiByZXQ7Cit9CitFWFBPUlRfU1lNQk9MKGRybV9zY2hlZF9qb2JfYXdh aXRfaW1wbGljaXQpOworCisKIC8qKgogICogZHJtX3NjaGVkX2pvYl9jbGVhbnVwIC0gY2xlYW4g dXAgc2NoZWR1bGVyIGpvYiByZXNvdXJjZXMKICAqIEBqb2I6IHNjaGVkdWxlciBqb2IgdG8gY2xl YW4gdXAKQEAgLTY0Niw2ICs3NDAsOSBAQCBFWFBPUlRfU1lNQk9MKGRybV9zY2hlZF9qb2JfYXJt KTsKICAqLwogdm9pZCBkcm1fc2NoZWRfam9iX2NsZWFudXAoc3RydWN0IGRybV9zY2hlZF9qb2Ig KmpvYikKIHsKKwlzdHJ1Y3QgZG1hX2ZlbmNlICpmZW5jZTsKKwl1bnNpZ25lZCBsb25nIGluZGV4 OworCiAJaWYgKGtyZWZfcmVhZCgmam9iLT5zX2ZlbmNlLT5maW5pc2hlZC5yZWZjb3VudCkpIHsK IAkJLyogZHJtX3NjaGVkX2pvYl9hcm0oKSBoYXMgYmVlbiBjYWxsZWQgKi8KIAkJZG1hX2ZlbmNl X3B1dCgmam9iLT5zX2ZlbmNlLT5maW5pc2hlZCk7CkBAIC02NTUsNiArNzUyLDEyIEBAIHZvaWQg ZHJtX3NjaGVkX2pvYl9jbGVhbnVwKHN0cnVjdCBkcm1fc2NoZWRfam9iICpqb2IpCiAJfQogCiAJ am9iLT5zX2ZlbmNlID0gTlVMTDsKKworCXhhX2Zvcl9lYWNoKCZqb2ItPmRlcGVuZGVuY2llcywg aW5kZXgsIGZlbmNlKSB7CisJCWRtYV9mZW5jZV9wdXQoZmVuY2UpOworCX0KKwl4YV9kZXN0cm95 KCZqb2ItPmRlcGVuZGVuY2llcyk7CisKIH0KIEVYUE9SVF9TWU1CT0woZHJtX3NjaGVkX2pvYl9j bGVhbnVwKTsKIApkaWZmIC0tZ2l0IGEvaW5jbHVkZS9kcm0vZ3B1X3NjaGVkdWxlci5oIGIvaW5j bHVkZS9kcm0vZ3B1X3NjaGVkdWxlci5oCmluZGV4IDgzYWZjM2FhOGUyZi4uNzRmYjMyMWRiYzQ0 IDEwMDY0NAotLS0gYS9pbmNsdWRlL2RybS9ncHVfc2NoZWR1bGVyLmgKKysrIGIvaW5jbHVkZS9k cm0vZ3B1X3NjaGVkdWxlci5oCkBAIC0yNyw5ICsyNywxMiBAQAogI2luY2x1ZGUgPGRybS9zcHNj X3F1ZXVlLmg+CiAjaW5jbHVkZSA8bGludXgvZG1hLWZlbmNlLmg+CiAjaW5jbHVkZSA8bGludXgv Y29tcGxldGlvbi5oPgorI2luY2x1ZGUgPGxpbnV4L3hhcnJheS5oPgogCiAjZGVmaW5lIE1BWF9X QUlUX1NDSEVEX0VOVElUWV9RX0VNUFRZIG1zZWNzX3RvX2ppZmZpZXMoMTAwMCkKIAorc3RydWN0 IGRybV9nZW1fb2JqZWN0OworCiBzdHJ1Y3QgZHJtX2dwdV9zY2hlZHVsZXI7CiBzdHJ1Y3QgZHJt X3NjaGVkX3JxOwogCkBAIC0xOTgsNiArMjAxLDE2IEBAIHN0cnVjdCBkcm1fc2NoZWRfam9iIHsK IAllbnVtIGRybV9zY2hlZF9wcmlvcml0eQkJc19wcmlvcml0eTsKIAlzdHJ1Y3QgZHJtX3NjaGVk X2VudGl0eSAgICAgICAgICplbnRpdHk7CiAJc3RydWN0IGRtYV9mZW5jZV9jYgkJY2I7CisJLyoq CisJICogQGRlcGVuZGVuY2llczoKKwkgKgorCSAqIENvbnRhaW5zIHRoZSBkZXBlbmRlbmNpZXMg YXMgc3RydWN0IGRtYV9mZW5jZSBmb3IgdGhpcyBqb2IsIHNlZQorCSAqIGRybV9zY2hlZF9qb2Jf YXdhaXRfZmVuY2UoKSBhbmQgZHJtX3NjaGVkX2pvYl9hd2FpdF9pbXBsaWNpdCgpLgorCSAqLwor CXN0cnVjdCB4YXJyYXkJCQlkZXBlbmRlbmNpZXM7CisKKwkvKiogQGxhc3RfZGVwZW5kZW5jeTog dHJhY2tzIEBkZXBlbmRlbmNpZXMgYXMgdGhleSBzaWduYWwgKi8KKwl1bnNpZ25lZCBsb25nCQkJ bGFzdF9kZXBlbmRlbmN5OwogfTsKIAogc3RhdGljIGlubGluZSBib29sIGRybV9zY2hlZF9pbnZh bGlkYXRlX2pvYihzdHJ1Y3QgZHJtX3NjaGVkX2pvYiAqc19qb2IsCkBAIC0yMjAsOSArMjMzLDE0 IEBAIGVudW0gZHJtX2dwdV9zY2hlZF9zdGF0IHsKICAqLwogc3RydWN0IGRybV9zY2hlZF9iYWNr ZW5kX29wcyB7CiAJLyoqCi0gICAgICAgICAqIEBkZXBlbmRlbmN5OiBDYWxsZWQgd2hlbiB0aGUg c2NoZWR1bGVyIGlzIGNvbnNpZGVyaW5nIHNjaGVkdWxpbmcKLSAgICAgICAgICogdGhpcyBqb2Ig bmV4dCwgdG8gZ2V0IGFub3RoZXIgc3RydWN0IGRtYV9mZW5jZSBmb3IgdGhpcyBqb2IgdG8KLQkg KiBibG9jayBvbi4gIE9uY2UgaXQgcmV0dXJucyBOVUxMLCBydW5fam9iKCkgbWF5IGJlIGNhbGxl ZC4KKwkgKiBAZGVwZW5kZW5jeToKKwkgKgorCSAqIENhbGxlZCB3aGVuIHRoZSBzY2hlZHVsZXIg aXMgY29uc2lkZXJpbmcgc2NoZWR1bGluZyB0aGlzIGpvYiBuZXh0LCB0bworCSAqIGdldCBhbm90 aGVyIHN0cnVjdCBkbWFfZmVuY2UgZm9yIHRoaXMgam9iIHRvIGJsb2NrIG9uLiAgT25jZSBpdAor CSAqIHJldHVybnMgTlVMTCwgcnVuX2pvYigpIG1heSBiZSBjYWxsZWQuCisJICoKKwkgKiBJZiBh IGRyaXZlciBleGNsdXNpdmVseSB1c2VzIGRybV9zY2hlZF9qb2JfYXdhaXRfZmVuY2UoKSBhbmQK KwkgKiBkcm1fc2NoZWRfam9iX2F3YWl0X2ltcGxpY2l0KCkgdGhpcyBjYW4gYmUgb21taXR0ZWQg YW5kIGxlZnQgYXMgTlVMTC4KIAkgKi8KIAlzdHJ1Y3QgZG1hX2ZlbmNlICooKmRlcGVuZGVuY3kp KHN0cnVjdCBkcm1fc2NoZWRfam9iICpzY2hlZF9qb2IsCiAJCQkJCXN0cnVjdCBkcm1fc2NoZWRf ZW50aXR5ICpzX2VudGl0eSk7CkBAIC0zNDksNiArMzY3LDEzIEBAIGludCBkcm1fc2NoZWRfam9i X2luaXQoc3RydWN0IGRybV9zY2hlZF9qb2IgKmpvYiwKIAkJICAgICAgIHN0cnVjdCBkcm1fc2No ZWRfZW50aXR5ICplbnRpdHksCiAJCSAgICAgICB2b2lkICpvd25lcik7CiB2b2lkIGRybV9zY2hl ZF9qb2JfYXJtKHN0cnVjdCBkcm1fc2NoZWRfam9iICpqb2IpOworaW50IGRybV9zY2hlZF9qb2Jf YXdhaXRfZmVuY2Uoc3RydWN0IGRybV9zY2hlZF9qb2IgKmpvYiwKKwkJCSAgICAgIHN0cnVjdCBk bWFfZmVuY2UgKmZlbmNlKTsKK2ludCBkcm1fc2NoZWRfam9iX2F3YWl0X2ltcGxpY2l0KHN0cnVj dCBkcm1fc2NoZWRfam9iICpqb2IsCisJCQkJIHN0cnVjdCBkcm1fZ2VtX29iamVjdCAqb2JqLAor CQkJCSBib29sIHdyaXRlKTsKKworCiB2b2lkIGRybV9zY2hlZF9lbnRpdHlfbW9kaWZ5X3NjaGVk KHN0cnVjdCBkcm1fc2NoZWRfZW50aXR5ICplbnRpdHksCiAJCQkJICAgIHN0cnVjdCBkcm1fZ3B1 X3NjaGVkdWxlciAqKnNjaGVkX2xpc3QsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHVuc2lnbmVkIGludCBudW1fc2NoZWRfbGlzdCk7Ci0tIAoyLjMyLjAKCl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkludGVsLWdmeCBtYWlsaW5nIGxp c3QKSW50ZWwtZ2Z4QGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3RzLmZyZWVkZXNr dG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ludGVsLWdmeAo=