From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
To: LKML <linux-kernel@vger.kernel.org>, linux-mm@kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>,
"Paul E . McKenney" <paulmck@kernel.org>,
"Theodore Y . Ts'o" <tytso@mit.edu>,
Matthew Wilcox <willy@infradead.org>,
Joel Fernandes <joel@joelfernandes.org>,
RCU <rcu@vger.kernel.org>, Uladzislau Rezki <urezki@gmail.com>,
Oleksiy Avramchenko <oleksiy.avramchenko@sonymobile.com>
Subject: [PATCH v2 14/16] rcu: Support reclaim for head-less object
Date: Mon, 25 May 2020 23:47:58 +0200 [thread overview]
Message-ID: <20200525214800.93072-15-urezki@gmail.com> (raw)
In-Reply-To: <20200525214800.93072-1-urezki@gmail.com>
Update the kvfree_call_rcu() function with head-less support.
This allows RCU to reclaim objects without an embedded rcu_head.
tree-RCU:
We introduce two chains of arrays to store SLAB-backed and vmalloc
pointers, each. Storage in either of these arrays does not require
embedding an rcu_head within the object.
Maintaining the arrays may become impossible due to high memory
pressure. For such cases there is an emergency path. Objects with
rcu_head inside are just queued on a backup rcu_head list. Later on
that list is drained. As for the head-less variant, as the current
context can sleep, the following emergency measures are applied:
a) Synchronously wait until a grace period has elapsed.
b) Call kvfree().
tiny-RCU:
For double argument calls, there are no new changes in behavior. For
single argument call, kvfree() is directly inlined on the current
stack after a synchronize_rcu() call. Note that for tiny-RCU, any
call to synchronize_rcu() is actually a quiescent state, therefore
it does nothing.
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Co-developed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
include/linux/rcutiny.h | 18 ++++++++++++++++-
kernel/rcu/tree.c | 45 +++++++++++++++++++++++++++++++++++++++--
2 files changed, 60 insertions(+), 3 deletions(-)
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 00bbd0e328c8..e7e4ec5938af 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -31,9 +31,25 @@ static inline void synchronize_rcu_expedited(void)
synchronize_rcu();
}
+/*
+ * Add one more declaration of kvfree() here. It is
+ * not so straight forward to just include <linux/mm.h>
+ * where it is defined due to getting many compile
+ * errors caused by that include.
+ */
+extern void kvfree(const void *addr);
+
static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
- call_rcu(head, func);
+ if (head) {
+ call_rcu(head, func);
+ return;
+ }
+
+ // kvfree_rcu(one_arg) call.
+ might_sleep();
+ synchronize_rcu();
+ kvfree((void *) func);
}
void rcu_qs(void);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index da29e6078392..2459830a3851 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3180,6 +3180,13 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
if (IS_ENABLED(CONFIG_PREEMPT_RT))
return false;
+ /*
+ * NOTE: For one argument of kvfree_rcu() we can
+ * drop the lock and get the page in sleepable
+ * context. That would allow to maintain an array
+ * for the CONFIG_PREEMPT_RT as well if no cached
+ * pages are available.
+ */
bnode = (struct kvfree_rcu_bulk_data *)
__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
}
@@ -3219,16 +3226,33 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
unsigned long flags;
struct kfree_rcu_cpu *krcp;
+ bool success;
void *ptr;
+ if (head) {
+ ptr = (void *) head - (unsigned long) func;
+ } else {
+ /*
+ * Please note there is a limitation for the head-less
+ * variant, that is why there is a clear rule for such
+ * objects: it can be used from might_sleep() context
+ * only. For other places please embed an rcu_head to
+ * your data.
+ */
+ might_sleep();
+ ptr = (unsigned long *) func;
+ }
+
krcp = krc_this_cpu_lock(&flags);
- ptr = (void *)head - (unsigned long)func;
// Queue the object but don't yet schedule the batch.
if (debug_rcu_head_queue(ptr)) {
// Probable double kfree_rcu(), just leak.
WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
__func__, head);
+
+ // Mark as success and leave.
+ success = true;
goto unlock_return;
}
@@ -3236,10 +3260,16 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
* Under high memory pressure GFP_NOWAIT can fail,
* in that case the emergency path is maintained.
*/
- if (unlikely(!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr))) {
+ success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr);
+ if (!success) {
+ if (head == NULL)
+ // Inline if kvfree_rcu(one_arg) call.
+ goto unlock_return;
+
head->func = func;
head->next = krcp->head;
krcp->head = head;
+ success = true;
}
WRITE_ONCE(krcp->count, krcp->count + 1);
@@ -3253,6 +3283,17 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
unlock_return:
krc_this_cpu_unlock(krcp, flags);
+
+ /*
+ * Inline kvfree() after synchronize_rcu(). We can do
+ * it from might_sleep() context only, so the current
+ * CPU can pass the QS state.
+ */
+ if (!success) {
+ debug_rcu_head_unqueue((struct rcu_head *) ptr);
+ synchronize_rcu();
+ kvfree(ptr);
+ }
}
EXPORT_SYMBOL_GPL(kvfree_call_rcu);
--
2.20.1
next prev parent reply other threads:[~2020-05-25 21:48 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-25 21:47 [PATCH v2 00/16] Introduce kvfree_rcu(1 or 2 arguments) Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 01/16] rcu/tree: Keep kfree_rcu() awake during lock contention Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 02/16] rcu/tree: Skip entry into the page allocator for PREEMPT_RT Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 03/16] rcu/tree: Repeat the monitor if any free channel is busy Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 04/16] rcu/tree: Make debug_objects logic independent of rcu_head Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 05/16] rcu/tree: Simplify KFREE_BULK_MAX_ENTR macro Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 06/16] rcu/tree: Move kfree_rcu_cpu locking/unlocking to separate functions Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 07/16] rcu/tree: Use static initializer for krc.lock Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 08/16] rcu/tree: cache specified number of objects Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 09/16] rcu/tree: Maintain separate array for vmalloc ptrs Uladzislau Rezki (Sony)
2020-06-17 23:46 ` Paul E. McKenney
2020-06-18 0:52 ` Matthew Wilcox
2020-06-18 3:18 ` Paul E. McKenney
2020-06-18 17:35 ` Uladzislau Rezki
2020-06-18 17:57 ` Paul E. McKenney
2020-06-18 18:34 ` Uladzislau Rezki
2020-06-18 19:03 ` Paul E. McKenney
2020-06-18 20:35 ` Uladzislau Rezki
2020-06-18 20:38 ` Matthew Wilcox
2020-06-18 21:17 ` Uladzislau Rezki
2020-06-18 21:34 ` Paul E. McKenney
2020-06-19 15:46 ` Uladzislau Rezki
2020-06-19 16:25 ` Paul E. McKenney
2020-06-22 19:04 ` Uladzislau Rezki
2020-06-22 19:53 ` Paul E. McKenney
2020-06-30 17:46 ` Uladzislau Rezki
2020-06-18 17:30 ` Uladzislau Rezki
2020-06-18 17:35 ` Matthew Wilcox
2020-06-18 20:03 ` Uladzislau Rezki
2020-06-18 17:25 ` Uladzislau Rezki
2020-06-18 17:32 ` Paul E. McKenney
2020-06-18 17:56 ` Uladzislau Rezki
2020-06-18 18:15 ` Matthew Wilcox
2020-06-18 18:23 ` Uladzislau Rezki
2020-06-18 18:37 ` Matthew Wilcox
2020-06-18 18:48 ` Uladzislau Rezki
2020-05-25 21:47 ` [PATCH v2 10/16] rcu/tiny: support vmalloc in tiny-RCU Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 11/16] rcu: Rename *_kfree_callback/*_kfree_rcu_offset/kfree_call_* Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 12/16] mm/list_lru.c: Rename kvfree_rcu() to local variant Uladzislau Rezki (Sony)
2020-05-25 21:47 ` [PATCH v2 13/16] rcu: Introduce 2 arg kvfree_rcu() interface Uladzislau Rezki (Sony)
2020-05-25 21:47 ` Uladzislau Rezki (Sony) [this message]
2020-05-25 21:47 ` [PATCH v2 15/16] rcu: Introduce single argument " Uladzislau Rezki (Sony)
2020-05-25 21:48 ` [PATCH v2 16/16] lib/test_vmalloc.c: Add test cases for kvfree_rcu() Uladzislau Rezki (Sony)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200525214800.93072-15-urezki@gmail.com \
--to=urezki@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=joel@joelfernandes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=oleksiy.avramchenko@sonymobile.com \
--cc=paulmck@kernel.org \
--cc=rcu@vger.kernel.org \
--cc=tytso@mit.edu \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).