From: Chris Wilson <chris@chris-wilson.co.uk> To: linux-kernel@vger.kernel.org Cc: Chris Wilson <chris@chris-wilson.co.uk>, Sumit Semwal <sumit.semwal@linaro.org>, Shuah Khan <shuahkh@osg.samsung.com>, Tejun Heo <tj@kernel.org>, Daniel Vetter <daniel.vetter@ffwll.ch>, Andrew Morton <akpm@linux-foundation.org>, Ingo Molnar <mingo@kernel.org>, Kees Cook <keescook@chromium.org>, Thomas Gleixner <tglx@linutronix.de>, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>, Dan Williams <dan.j.williams@intel.com>, Andrey Ryabinin <aryabinin@virtuozzo.com>, Davidlohr Bueso <dave@stgolabs.net>, Nikolay Aleksandrov <nikolay@cumulusnetworks.com>, "David S. Miller" <davem@davemloft.net>, "Peter Zijlstra (Intel)" <peterz@infradead.org>, Rasmus Villemoes <linux@rasmusvillemoes.dk>, Andy Shevchenko <andriy.shevchenko@linux.intel.com>, Dmitry Vyukov <dvyukov@google.com>, Alexander Potapenko <glider@google.com>, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org Subject: [PATCH 8/9] async: Add execution barriers Date: Fri, 24 Jun 2016 10:08:52 +0100 [thread overview] Message-ID: <1466759333-4703-9-git-send-email-chris@chris-wilson.co.uk> (raw) In-Reply-To: <1466759333-4703-1-git-send-email-chris@chris-wilson.co.uk> A frequent mode of operation is fanning out N tasks to execute in parallel, collating results, fanning out M tasks, rinse and repeat. This is also common to the notion of the async/sync kernel domain split. A barrier provides a mechanism by which all work queued after the barrier must wait (i.e. not be scheduled) until all work queued before the barrier is completed. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Sumit Semwal <sumit.semwal@linaro.org> Cc: Shuah Khan <shuahkh@osg.samsung.com> Cc: Tejun Heo <tj@kernel.org> Cc: Daniel Vetter <daniel.vetter@ffwll.ch> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Ingo Molnar <mingo@kernel.org> Cc: Kees Cook <keescook@chromium.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: Nikolay Aleksandrov <nikolay@cumulusnetworks.com> Cc: "David S. Miller" <davem@davemloft.net> Cc: "Peter Zijlstra (Intel)" <peterz@infradead.org> Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk> Cc: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: linux-kernel@vger.kernel.org Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Cc: linaro-mm-sig@lists.linaro.org --- include/linux/async.h | 4 ++++ kernel/async.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/include/linux/async.h b/include/linux/async.h index 45d6c8323b60..64a090e3f24f 100644 --- a/include/linux/async.h +++ b/include/linux/async.h @@ -26,6 +26,7 @@ struct async_work { struct async_domain { struct list_head pending; + struct kfence *barrier; unsigned registered:1; }; @@ -59,6 +60,9 @@ extern void async_synchronize_cookie(async_cookie_t cookie); extern void async_synchronize_cookie_domain(async_cookie_t cookie, struct async_domain *domain); +extern void async_barrier(void); +extern void async_barrier_domain(struct async_domain *domain); + extern bool current_is_async(void); extern struct async_work * diff --git a/kernel/async.c b/kernel/async.c index 83007c39a113..a22945f4b4c4 100644 --- a/kernel/async.c +++ b/kernel/async.c @@ -724,6 +724,12 @@ struct async_work *async_work_create(async_func_t func, void *data, gfp_t gfp) } EXPORT_SYMBOL_GPL(async_work_create); +static void async_barrier_delete(struct async_domain *domain) +{ + kfence_put(domain->barrier); + domain->barrier = NULL; +} + async_cookie_t queue_async_work(struct async_domain *domain, struct async_work *work, gfp_t gfp) @@ -744,6 +750,9 @@ async_cookie_t queue_async_work(struct async_domain *domain, async_pending_count++; spin_unlock_irqrestore(&async_lock, flags); + if (!kfence_add(&entry->base.fence, domain->barrier, gfp)) + async_barrier_delete(domain); + /* mark that this task has queued an async job, used by module init */ current->flags |= PF_USED_ASYNC; @@ -811,6 +820,55 @@ async_cookie_t async_schedule_domain(async_func_t func, void *data, } EXPORT_SYMBOL_GPL(async_schedule_domain); +static struct kfence * __async_barrier_create(struct async_domain *domain) +{ + struct kfence *fence; + struct async_entry *entry; + unsigned long flags; + int ret; + + fence = kfence_create(GFP_KERNEL); + if (!fence) + goto out_sync; + + ret = 0; + spin_lock_irqsave(&async_lock, flags); + list_for_each_entry(entry, &domain->pending, pending_link[0]) { + ret |= kfence_add(fence, &entry->base.fence, GFP_ATOMIC); + if (ret < 0) + break; + } + spin_unlock_irqrestore(&async_lock, flags); + if (ret <= 0) + goto out_put; + + kfence_add(fence, domain->barrier, GFP_KERNEL); + kfence_signal(fence); + return fence; + +out_put: + kfence_signal(fence); + kfence_put(fence); +out_sync: + async_synchronize_full_domain(domain); + return NULL; +} + +void async_barrier(void) +{ + async_barrier_domain(&async_dfl_domain); +} +EXPORT_SYMBOL_GPL(async_barrier); + +void async_barrier_domain(struct async_domain *domain) +{ + struct kfence *barrier = __async_barrier_create(domain); + + kfence_put(domain->barrier); + domain->barrier = barrier; +} +EXPORT_SYMBOL_GPL(async_barrier_domain); + /** * async_synchronize_full - synchronize all asynchronous function calls * @@ -834,6 +892,8 @@ EXPORT_SYMBOL_GPL(async_synchronize_full); void async_unregister_domain(struct async_domain *domain) { WARN_ON(!list_empty(&domain->pending)); + + async_barrier_delete(domain); domain->registered = 0; } EXPORT_SYMBOL_GPL(async_unregister_domain); -- 2.8.1
WARNING: multiple messages have this Message-ID (diff)
From: Chris Wilson <chris@chris-wilson.co.uk> To: linux-kernel@vger.kernel.org Cc: "Peter Zijlstra (Intel)" <peterz@infradead.org>, Daniel Vetter <daniel.vetter@ffwll.ch>, Rasmus Villemoes <linux@rasmusvillemoes.dk>, dri-devel@lists.freedesktop.org, Alexander Potapenko <glider@google.com>, Ingo Molnar <mingo@kernel.org>, Davidlohr Bueso <dave@stgolabs.net>, Shuah Khan <shuahkh@osg.samsung.com>, Andrey Ryabinin <aryabinin@virtuozzo.com>, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>, linux-media@vger.kernel.org, Kees Cook <keescook@chromium.org>, Nikolay Aleksandrov <nikolay@cumulusnetworks.com>, linaro-mm-sig@lists.linaro.org, Dan Williams <dan.j.williams@intel.com>, Andy Shevchenko <andriy.shevchenko@linux.intel.com>, Dmitry Vyukov <dvyukov@google.com>, Thomas Gleixner <tglx@linutronix.de>, Tejun Heo <tj@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, "David S. Miller" <davem@davemloft.net> Subject: [PATCH 8/9] async: Add execution barriers Date: Fri, 24 Jun 2016 10:08:52 +0100 [thread overview] Message-ID: <1466759333-4703-9-git-send-email-chris@chris-wilson.co.uk> (raw) In-Reply-To: <1466759333-4703-1-git-send-email-chris@chris-wilson.co.uk> A frequent mode of operation is fanning out N tasks to execute in parallel, collating results, fanning out M tasks, rinse and repeat. This is also common to the notion of the async/sync kernel domain split. A barrier provides a mechanism by which all work queued after the barrier must wait (i.e. not be scheduled) until all work queued before the barrier is completed. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Sumit Semwal <sumit.semwal@linaro.org> Cc: Shuah Khan <shuahkh@osg.samsung.com> Cc: Tejun Heo <tj@kernel.org> Cc: Daniel Vetter <daniel.vetter@ffwll.ch> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Ingo Molnar <mingo@kernel.org> Cc: Kees Cook <keescook@chromium.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: Nikolay Aleksandrov <nikolay@cumulusnetworks.com> Cc: "David S. Miller" <davem@davemloft.net> Cc: "Peter Zijlstra (Intel)" <peterz@infradead.org> Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk> Cc: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: linux-kernel@vger.kernel.org Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Cc: linaro-mm-sig@lists.linaro.org --- include/linux/async.h | 4 ++++ kernel/async.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/include/linux/async.h b/include/linux/async.h index 45d6c8323b60..64a090e3f24f 100644 --- a/include/linux/async.h +++ b/include/linux/async.h @@ -26,6 +26,7 @@ struct async_work { struct async_domain { struct list_head pending; + struct kfence *barrier; unsigned registered:1; }; @@ -59,6 +60,9 @@ extern void async_synchronize_cookie(async_cookie_t cookie); extern void async_synchronize_cookie_domain(async_cookie_t cookie, struct async_domain *domain); +extern void async_barrier(void); +extern void async_barrier_domain(struct async_domain *domain); + extern bool current_is_async(void); extern struct async_work * diff --git a/kernel/async.c b/kernel/async.c index 83007c39a113..a22945f4b4c4 100644 --- a/kernel/async.c +++ b/kernel/async.c @@ -724,6 +724,12 @@ struct async_work *async_work_create(async_func_t func, void *data, gfp_t gfp) } EXPORT_SYMBOL_GPL(async_work_create); +static void async_barrier_delete(struct async_domain *domain) +{ + kfence_put(domain->barrier); + domain->barrier = NULL; +} + async_cookie_t queue_async_work(struct async_domain *domain, struct async_work *work, gfp_t gfp) @@ -744,6 +750,9 @@ async_cookie_t queue_async_work(struct async_domain *domain, async_pending_count++; spin_unlock_irqrestore(&async_lock, flags); + if (!kfence_add(&entry->base.fence, domain->barrier, gfp)) + async_barrier_delete(domain); + /* mark that this task has queued an async job, used by module init */ current->flags |= PF_USED_ASYNC; @@ -811,6 +820,55 @@ async_cookie_t async_schedule_domain(async_func_t func, void *data, } EXPORT_SYMBOL_GPL(async_schedule_domain); +static struct kfence * __async_barrier_create(struct async_domain *domain) +{ + struct kfence *fence; + struct async_entry *entry; + unsigned long flags; + int ret; + + fence = kfence_create(GFP_KERNEL); + if (!fence) + goto out_sync; + + ret = 0; + spin_lock_irqsave(&async_lock, flags); + list_for_each_entry(entry, &domain->pending, pending_link[0]) { + ret |= kfence_add(fence, &entry->base.fence, GFP_ATOMIC); + if (ret < 0) + break; + } + spin_unlock_irqrestore(&async_lock, flags); + if (ret <= 0) + goto out_put; + + kfence_add(fence, domain->barrier, GFP_KERNEL); + kfence_signal(fence); + return fence; + +out_put: + kfence_signal(fence); + kfence_put(fence); +out_sync: + async_synchronize_full_domain(domain); + return NULL; +} + +void async_barrier(void) +{ + async_barrier_domain(&async_dfl_domain); +} +EXPORT_SYMBOL_GPL(async_barrier); + +void async_barrier_domain(struct async_domain *domain) +{ + struct kfence *barrier = __async_barrier_create(domain); + + kfence_put(domain->barrier); + domain->barrier = barrier; +} +EXPORT_SYMBOL_GPL(async_barrier_domain); + /** * async_synchronize_full - synchronize all asynchronous function calls * @@ -834,6 +892,8 @@ EXPORT_SYMBOL_GPL(async_synchronize_full); void async_unregister_domain(struct async_domain *domain) { WARN_ON(!list_empty(&domain->pending)); + + async_barrier_delete(domain); domain->registered = 0; } EXPORT_SYMBOL_GPL(async_unregister_domain); -- 2.8.1 _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
next prev parent reply other threads:[~2016-06-24 9:11 UTC|newest] Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-06-24 9:08 Introduce fences for N:M completion variables Chris Wilson 2016-06-24 9:08 ` [PATCH 1/9] lib: Add kselftests for async-domains Chris Wilson 2016-06-24 9:08 ` [PATCH 2/9] async: Introduce kfence, a N:M completion mechanism Chris Wilson 2016-06-24 9:08 ` Chris Wilson 2016-07-13 9:38 ` Peter Zijlstra 2016-07-13 9:38 ` Peter Zijlstra 2016-07-13 10:20 ` Chris Wilson 2016-07-13 10:20 ` Chris Wilson 2016-07-13 11:02 ` Daniel Vetter 2016-07-13 11:02 ` Daniel Vetter 2016-07-13 10:26 ` Peter Zijlstra 2016-07-13 10:26 ` Peter Zijlstra 2016-06-24 9:08 ` [PATCH 3/9] async: Extend kfence to allow struct embedding Chris Wilson 2016-06-24 9:08 ` Chris Wilson 2016-07-13 10:31 ` Peter Zijlstra 2016-06-24 9:08 ` [PATCH 4/9] async: Extend kfences for listening on DMA fences Chris Wilson 2016-06-24 9:08 ` Chris Wilson 2016-06-24 9:08 ` [PATCH 5/9] async: Wrap hrtimer to provide a time source for a kfence Chris Wilson 2016-06-24 9:08 ` Chris Wilson 2016-07-13 10:32 ` Peter Zijlstra 2016-06-24 9:08 ` [PATCH 6/9] async: Add a convenience wrapper for waiting on implicit dma-buf Chris Wilson 2016-06-24 9:08 ` Chris Wilson 2016-06-24 9:08 ` [PATCH 7/9] async: Add support for explicit fine-grained barriers Chris Wilson 2016-06-24 9:08 ` Chris Wilson 2016-06-24 9:08 ` Chris Wilson [this message] 2016-06-24 9:08 ` [PATCH 8/9] async: Add execution barriers Chris Wilson 2016-06-24 9:08 ` [PATCH 9/9] async: Introduce a dependency resolver for parallel execution Chris Wilson 2016-06-24 9:08 ` Chris Wilson 2016-07-17 12:58 ` Introduce fences for N:M completion variables [v2] Chris Wilson 2016-07-17 12:58 ` Chris Wilson 2016-07-17 12:58 ` [PATCH v2 1/7] kfence: Introduce kfence, a N:M completion mechanism Chris Wilson 2016-07-17 12:58 ` Chris Wilson 2016-07-17 12:58 ` [PATCH v2 2/7] kfence: Wrap hrtimer to provide a time source for a kfence Chris Wilson 2016-07-17 12:58 ` Chris Wilson 2016-07-17 12:58 ` [PATCH v2 3/7] kfence: Extend kfences for listening on DMA fences Chris Wilson 2016-07-17 12:58 ` Chris Wilson 2016-07-17 12:58 ` [PATCH v2 4/7] async: Add kselftests for async-domains Chris Wilson 2016-07-17 12:58 ` Chris Wilson 2016-07-17 12:58 ` [PATCH v2 5/7] async: Add support for explicit fine-grained barriers Chris Wilson 2016-07-17 12:58 ` Chris Wilson 2016-07-17 12:58 ` [PATCH v2 6/7] async: Add execution barriers Chris Wilson 2016-07-17 12:58 ` Chris Wilson 2016-07-17 12:58 ` [PATCH v2 7/7] async: Introduce a dependency resolver for parallel execution Chris Wilson 2016-07-17 12:58 ` Chris Wilson
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1466759333-4703-9-git-send-email-chris@chris-wilson.co.uk \ --to=chris@chris-wilson.co.uk \ --cc=akpm@linux-foundation.org \ --cc=andriy.shevchenko@linux.intel.com \ --cc=aryabinin@virtuozzo.com \ --cc=dan.j.williams@intel.com \ --cc=daniel.vetter@ffwll.ch \ --cc=dave@stgolabs.net \ --cc=davem@davemloft.net \ --cc=dri-devel@lists.freedesktop.org \ --cc=dvyukov@google.com \ --cc=glider@google.com \ --cc=keescook@chromium.org \ --cc=linaro-mm-sig@lists.linaro.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-media@vger.kernel.org \ --cc=linux@rasmusvillemoes.dk \ --cc=mingo@kernel.org \ --cc=nikolay@cumulusnetworks.com \ --cc=paulmck@linux.vnet.ibm.com \ --cc=peterz@infradead.org \ --cc=shuahkh@osg.samsung.com \ --cc=sumit.semwal@linaro.org \ --cc=tglx@linutronix.de \ --cc=tj@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.