From: Vlastimil Babka <vbabka@suse.cz>
To: "Joel Fernandes (Google)" <joel@joelfernandes.org>, rcu@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, rushikesh.s.kadam@intel.com,
urezki@gmail.com, neeraj.iitr10@gmail.com, frederic@kernel.org,
paulmck@kernel.org, rostedt@goodmis.org, vineeth@bitbyteword.org,
boqun.feng@gmail.com
Subject: Re: [PATCH v5 02/18] mm/sl[au]b: rearrange struct slab fields to allow larger rcu_head
Date: Fri, 2 Sep 2022 11:26:44 +0200 [thread overview]
Message-ID: <e84c90f2-d76f-83d4-d40b-403f894eda33@suse.cz> (raw)
In-Reply-To: <20220901221720.1105021-3-joel@joelfernandes.org>
On 9/2/22 00:17, Joel Fernandes (Google) wrote:
> From: Vlastimil Babka <vbabka@suse.cz>
>
> Joel reports [1] that increasing the rcu_head size for debugging
> purposes used to work before struct slab was split from struct page, but
> now runs into the various SLAB_MATCH() sanity checks of the layout.
>
> This is because the rcu_head in struct page is in union with large
> sub-structures and has space to grow without exceeding their size, while
> in struct slab (for SLAB and SLUB) it's in union only with a list_head.
>
> On closer inspection (and after the previous patch) we can put all
> fields except slab_cache to a union with rcu_head, as slab_cache is
> sufficient for the rcu freeing callbacks to work and the rest can be
> overwritten by rcu_head without causing issues.
>
> This is only somewhat complicated by the need to keep SLUB's
> freelist+counters aligned for cmpxchg_double. As a result the fields
> need to be reordered so that slab_cache is first (after page flags) and
> the union with rcu_head follows. For consistency, do that for SLAB as
> well, although not necessary there.
>
> As a result, the rcu_head field in struct page and struct slab is no
> longer at the same offset, but that doesn't matter as there is no
> casting that would rely on that in the slab freeing callbacks, so we can
> just drop the respective SLAB_MATCH() check.
>
> Also we need to update the SLAB_MATCH() for compound_head to reflect the
> new ordering.
>
> While at it, also add a static_assert to check the alignment needed for
> cmpxchg_double so mistakes are found sooner than a runtime GPF.
>
> [1] https://lore.kernel.org/all/85afd876-d8bb-0804-b2c5-48ed3055e702@joelfernandes.org/
>
> Reported-by: Joel Fernandes <joel@joelfernandes.org>
> Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
I've added patches 01 and 02 to slab tree for -next exposure before Joel's
full series posting, but it should be also ok if rcu tree carries them with
the whole patchset. I can then drop them from slab tree (there are no
dependencies with other stuff there) so we don't introduce duplicite commits
needlessly, just give me a heads up.
> ---
> mm/slab.h | 54 ++++++++++++++++++++++++++++++++----------------------
> 1 file changed, 32 insertions(+), 22 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index 4ec82bec15ec..2c248864ea91 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -11,37 +11,43 @@ struct slab {
>
> #if defined(CONFIG_SLAB)
>
> + struct kmem_cache *slab_cache;
> union {
> - struct list_head slab_list;
> + struct {
> + struct list_head slab_list;
> + void *freelist; /* array of free object indexes */
> + void *s_mem; /* first object */
> + };
> struct rcu_head rcu_head;
> };
> - struct kmem_cache *slab_cache;
> - void *freelist; /* array of free object indexes */
> - void *s_mem; /* first object */
> unsigned int active;
>
> #elif defined(CONFIG_SLUB)
>
> - union {
> - struct list_head slab_list;
> - struct rcu_head rcu_head;
> -#ifdef CONFIG_SLUB_CPU_PARTIAL
> - struct {
> - struct slab *next;
> - int slabs; /* Nr of slabs left */
> - };
> -#endif
> - };
> struct kmem_cache *slab_cache;
> - /* Double-word boundary */
> - void *freelist; /* first free object */
> union {
> - unsigned long counters;
> struct {
> - unsigned inuse:16;
> - unsigned objects:15;
> - unsigned frozen:1;
> + union {
> + struct list_head slab_list;
> +#ifdef CONFIG_SLUB_CPU_PARTIAL
> + struct {
> + struct slab *next;
> + int slabs; /* Nr of slabs left */
> + };
> +#endif
> + };
> + /* Double-word boundary */
> + void *freelist; /* first free object */
> + union {
> + unsigned long counters;
> + struct {
> + unsigned inuse:16;
> + unsigned objects:15;
> + unsigned frozen:1;
> + };
> + };
> };
> + struct rcu_head rcu_head;
> };
> unsigned int __unused;
>
> @@ -66,9 +72,10 @@ struct slab {
> #define SLAB_MATCH(pg, sl) \
> static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl))
> SLAB_MATCH(flags, __page_flags);
> -SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */
> #ifndef CONFIG_SLOB
> -SLAB_MATCH(rcu_head, rcu_head);
> +SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */
> +#else
> +SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */
> #endif
> SLAB_MATCH(_refcount, __page_refcount);
> #ifdef CONFIG_MEMCG
> @@ -76,6 +83,9 @@ SLAB_MATCH(memcg_data, memcg_data);
> #endif
> #undef SLAB_MATCH
> static_assert(sizeof(struct slab) <= sizeof(struct page));
> +#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && defined(CONFIG_SLUB)
> +static_assert(IS_ALIGNED(offsetof(struct slab, freelist), 16));
> +#endif
>
> /**
> * folio_slab - Converts from folio to slab.
next prev parent reply other threads:[~2022-09-02 9:26 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-01 22:17 [PATCH v5 00/18] Implement call_rcu_lazy() and miscellaneous fixes Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 01/18] mm/slub: perform free consistency checks before call_rcu Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 02/18] mm/sl[au]b: rearrange struct slab fields to allow larger rcu_head Joel Fernandes (Google)
2022-09-02 9:26 ` Vlastimil Babka [this message]
2022-09-02 9:30 ` Vlastimil Babka
2022-09-02 15:09 ` Joel Fernandes
2022-09-03 13:53 ` Paul E. McKenney
2022-09-01 22:17 ` [PATCH v5 03/18] rcu/tree: Use READ_ONCE() for lockless read of rnp->qsmask Joel Fernandes (Google)
2022-09-06 22:26 ` Boqun Feng
2022-09-06 22:31 ` Joel Fernandes
2022-09-01 22:17 ` [PATCH v5 04/18] rcu: Fix late wakeup when flush of bypass cblist happens Joel Fernandes (Google)
2022-09-02 11:35 ` Frederic Weisbecker
2022-09-02 23:58 ` Joel Fernandes
2022-09-03 15:10 ` Paul E. McKenney
2022-09-04 21:13 ` Frederic Weisbecker
2022-09-03 14:04 ` Paul E. McKenney
2022-09-03 14:05 ` Joel Fernandes
2022-09-06 3:07 ` Joel Fernandes
2022-09-06 9:48 ` Frederic Weisbecker
2022-09-07 2:43 ` Joel Fernandes
2022-09-01 22:17 ` [PATCH v5 05/18] rcu: Move trace_rcu_callback() before bypassing Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 06/18] rcu: Introduce call_rcu_lazy() API implementation Joel Fernandes (Google)
2022-09-02 15:21 ` Frederic Weisbecker
2022-09-02 23:09 ` Joel Fernandes
2022-09-05 12:59 ` Frederic Weisbecker
2022-09-05 20:18 ` Joel Fernandes
2022-09-05 20:32 ` Joel Fernandes
2022-09-06 8:55 ` Paul E. McKenney
2022-09-06 16:16 ` Joel Fernandes
2022-09-06 17:05 ` Paul E. McKenney
2022-09-03 22:00 ` Joel Fernandes
2022-09-04 21:01 ` Frederic Weisbecker
2022-09-05 20:20 ` Joel Fernandes
2022-09-06 3:05 ` Joel Fernandes
2022-09-06 15:17 ` Frederic Weisbecker
2022-09-06 16:15 ` Joel Fernandes
2022-09-06 16:31 ` Joel Fernandes
2022-09-06 16:38 ` Joel Fernandes
2022-09-06 16:43 ` Joel Fernandes
2022-09-06 19:11 ` Frederic Weisbecker
2022-09-07 2:56 ` Joel Fernandes
2022-09-07 9:56 ` Frederic Weisbecker
2022-09-07 10:03 ` Frederic Weisbecker
2022-09-07 14:01 ` Joel Fernandes
2022-09-07 0:06 ` Joel Fernandes
2022-09-07 9:40 ` Frederic Weisbecker
2022-09-07 13:44 ` Joel Fernandes
2022-09-07 15:38 ` Frederic Weisbecker
2022-09-07 15:39 ` Joel Fernandes
2022-09-21 23:52 ` Joel Fernandes
2022-09-06 18:16 ` Uladzislau Rezki
2022-09-06 18:21 ` Joel Fernandes
2022-09-07 8:52 ` Uladzislau Rezki
2022-09-07 15:23 ` Joel Fernandes
2022-09-03 14:03 ` Paul E. McKenney
2022-09-03 14:05 ` Joel Fernandes
2022-09-01 22:17 ` [PATCH v5 07/18] rcu: shrinker for lazy rcu Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 08/18] rcu: Add per-CB tracing for queuing, flush and invocation Joel Fernandes (Google)
2022-09-02 16:48 ` kernel test robot
2022-09-03 12:39 ` Paul E. McKenney
2022-09-03 14:07 ` Joel Fernandes
2022-09-02 19:01 ` kernel test robot
2022-09-01 22:17 ` [PATCH v5 09/18] rcuscale: Add laziness and kfree tests Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 10/18] rcutorture: Add test code for call_rcu_lazy() Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 11/18] fs: Move call_rcu() to call_rcu_lazy() in some paths Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 12/18] cred: Move call_rcu() to call_rcu_lazy() Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 13/18] security: " Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 14/18] net/core: " Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 15/18] kernel: Move various core kernel usages " Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 16/18] lib: Move call_rcu() " Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 17/18] i915: " Joel Fernandes (Google)
2022-09-01 22:17 ` [PATCH v5 18/18] fork: Move thread_stack_free_rcu() " Joel Fernandes (Google)
2022-09-03 15:22 ` [PATCH v5 00/18] Implement call_rcu_lazy() and miscellaneous fixes Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e84c90f2-d76f-83d4-d40b-403f894eda33@suse.cz \
--to=vbabka@suse.cz \
--cc=boqun.feng@gmail.com \
--cc=frederic@kernel.org \
--cc=joel@joelfernandes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=neeraj.iitr10@gmail.com \
--cc=paulmck@kernel.org \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=rushikesh.s.kadam@intel.com \
--cc=urezki@gmail.com \
--cc=vineeth@bitbyteword.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).