From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABAD7C28D13 for ; Thu, 25 Aug 2022 08:41:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22D37940008; Thu, 25 Aug 2022 04:41:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DD22940007; Thu, 25 Aug 2022 04:41:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A4FC940008; Thu, 25 Aug 2022 04:41:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EAF9A940007 for ; Thu, 25 Aug 2022 04:41:19 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BF4551219A5 for ; Thu, 25 Aug 2022 08:41:19 +0000 (UTC) X-FDA: 79837470678.28.C3A3A85 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf12.hostedemail.com (Postfix) with ESMTP id 4221240007 for ; Thu, 25 Aug 2022 08:41:18 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A5F371F9CC; Thu, 25 Aug 2022 08:41:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1661416876; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EzfDIHo5G6kGpiVBHYttSfthwWZbK7h4goasSNOw+t8=; b=OIE9b6A3OpNr7tg6YWQpzofkZOTXZb8Y4tlvRTL3N03ZAhpXe/OxY0lwLhpAdX3rr9ALsZ clrk0KWi15jAKszFXiPSeyUBtZ6xFriNuZcuzxVEAMeNQGxNbxDotRjhqxJiS3AELZIY+m zPTaqg3ir8gpqFid1vW3GolGyZGHLzo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1661416876; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EzfDIHo5G6kGpiVBHYttSfthwWZbK7h4goasSNOw+t8=; b=v9z5WICtAs6zRhGFzlCBkomsiNBeGBIhEutiJjqxBfVu/O6pyfNfFgSv5wdlqITOQBVR5b GP/ogIdnvJfbcSCg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 7114813517; Thu, 25 Aug 2022 08:41:16 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id IQchGqw1B2MRMwAAMHmgww (envelope-from ); Thu, 25 Aug 2022 08:41:16 +0000 Message-ID: <2903a7a4-7ef1-92e0-05df-ef7cf2fa65b1@suse.cz> Date: Thu, 25 Aug 2022 10:41:16 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.1.2 Subject: Re: [PATCH 6/5] slub: Make PREEMPT_RT support less convoluted Content-Language: en-US To: Sebastian Andrzej Siewior Cc: Rongwei Wang , Christoph Lameter , Joonsoo Kim , David Rientjes , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, Thomas Gleixner , Mike Galbraith , Andrew Morton References: <20220823170400.26546-1-vbabka@suse.cz> From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661416878; a=rsa-sha256; cv=none; b=vedJU2hjqlY5YkbnvAEFFWm0i6qRoopnnpYsbPZsw4noixBS4/XJPatEQVNAIX+m6MkhMC RrgpXIIKjSyUNOHBPGEB1VbOjkbLjVPaRbYAAMoOWQdYzNeOsZnkU7oHo05hiLq2XrRp/v Buj2Yu7CFnoLZT3DNq+Vs4GM2ULFD1Q= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=OIE9b6A3; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=v9z5WICt; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661416878; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EzfDIHo5G6kGpiVBHYttSfthwWZbK7h4goasSNOw+t8=; b=ogdCOevnz0xsJ9+povx8QIdsxM38VDfe+xC6qUqDh2l4G1hgCd9H5rpc6MfgOsOzagdtJ8 HOSquPoWRV2ZyzuG3Vkd69S02hV5nT4pa3l2HD+GqxJl64fx8bP57QUesLFKMqLvpWFqOK +fRYxvcq3zSvnX6zHSKBJ5+wORtHO1U= X-Rspam-User: Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=OIE9b6A3; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=v9z5WICt; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam12 X-Stat-Signature: 11s4mc45d5sc5nimtwmmw8twzbrkuwyk X-Rspamd-Queue-Id: 4221240007 X-HE-Tag: 1661416878-982391 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 8/25/22 09:51, Sebastian Andrzej Siewior wrote: > From: Thomas Gleixner > > The slub code already has a few helpers depending on PREEMPT_RT. Add a few > more and get rid of the CONFIG_PREEMPT_RT conditionals all over the place. > > No functional change. > > Signed-off-by: Thomas Gleixner > Cc: Andrew Morton > Cc: Christoph Lameter > Cc: David Rientjes > Cc: Joonsoo Kim > Cc: Pekka Enberg > Cc: Vlastimil Babka > Cc: linux-mm@kvack.org > Signed-off-by: Sebastian Andrzej Siewior > Acked-by: Peter Zijlstra (Intel) > --- > > Vlastimil, does it work for you to include this patch in your series? It > depends now on your series :) It has this USE_LOCKLESS_FAST_PATH() Linus > asked about so we should be good. Sure, I'll add it, thanks! > > mm/slub.c | 56 ++++++++++++++++++++++++-------------------------------- > 1 file changed, 24 insertions(+), 32 deletions(-) > > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -104,9 +104,11 @@ > * except the stat counters. This is a percpu structure manipulated only by > * the local cpu, so the lock protects against being preempted or interrupted > * by an irq. Fast path operations rely on lockless operations instead. > - * On PREEMPT_RT, the local lock does not actually disable irqs (and thus > - * prevent the lockless operations), so fastpath operations also need to take > - * the lock and are no longer lockless. > + * > + * On PREEMPT_RT, the local lock neither disables interrupts nor preemption > + * which means the lockless fastpath cannot be used as it might interfere with > + * an in-progress slow path operations. In this case the local lock is always > + * taken but it still utilizes the freelist for the common operations. > * > * lockless fastpaths > * > @@ -167,8 +169,9 @@ > * function call even on !PREEMPT_RT, use inline preempt_disable() there. > */ > #ifndef CONFIG_PREEMPT_RT > -#define slub_get_cpu_ptr(var) get_cpu_ptr(var) > -#define slub_put_cpu_ptr(var) put_cpu_ptr(var) > +#define slub_get_cpu_ptr(var) get_cpu_ptr(var) > +#define slub_put_cpu_ptr(var) put_cpu_ptr(var) > +#define USE_LOCKLESS_FAST_PATH() (true) > #else > #define slub_get_cpu_ptr(var) \ > ({ \ > @@ -180,6 +183,7 @@ do { \ > (void)(var); \ > migrate_enable(); \ > } while (0) > +#define USE_LOCKLESS_FAST_PATH() (false) > #endif > > #ifdef CONFIG_SLUB_DEBUG > @@ -474,7 +478,7 @@ static inline bool __cmpxchg_double_slab > void *freelist_new, unsigned long counters_new, > const char *n) > { > - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) > + if (USE_LOCKLESS_FAST_PATH()) > lockdep_assert_irqs_disabled(); > #if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ > defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) > @@ -3287,14 +3291,8 @@ static __always_inline void *slab_alloc_ > > object = c->freelist; > slab = c->slab; > - /* > - * We cannot use the lockless fastpath on PREEMPT_RT because if a > - * slowpath has taken the local_lock_irqsave(), it is not protected > - * against a fast path operation in an irq handler. So we need to take > - * the slow path which uses local_lock. It is still relatively fast if > - * there is a suitable cpu freelist. > - */ > - if (IS_ENABLED(CONFIG_PREEMPT_RT) || > + > + if (!USE_LOCKLESS_FAST_PATH() || > unlikely(!object || !slab || !node_match(slab, node))) { > object = __slab_alloc(s, gfpflags, node, addr, c); > } else { > @@ -3554,6 +3552,7 @@ static __always_inline void do_slab_free > void *tail_obj = tail ? : head; > struct kmem_cache_cpu *c; > unsigned long tid; > + void **freelist; > > redo: > /* > @@ -3568,9 +3567,13 @@ static __always_inline void do_slab_free > /* Same with comment on barrier() in slab_alloc_node() */ > barrier(); > > - if (likely(slab == c->slab)) { > -#ifndef CONFIG_PREEMPT_RT > - void **freelist = READ_ONCE(c->freelist); > + if (unlikely(slab != c->slab)) { > + __slab_free(s, slab, head, tail_obj, cnt, addr); > + return; > + } > + > + if (USE_LOCKLESS_FAST_PATH()) { > + freelist = READ_ONCE(c->freelist); > > set_freepointer(s, tail_obj, freelist); > > @@ -3582,16 +3585,8 @@ static __always_inline void do_slab_free > note_cmpxchg_failure("slab_free", s, tid); > goto redo; > } > -#else /* CONFIG_PREEMPT_RT */ > - /* > - * We cannot use the lockless fastpath on PREEMPT_RT because if > - * a slowpath has taken the local_lock_irqsave(), it is not > - * protected against a fast path operation in an irq handler. So > - * we need to take the local_lock. We shouldn't simply defer to > - * __slab_free() as that wouldn't use the cpu freelist at all. > - */ > - void **freelist; > - > + } else { > + /* Update the free list under the local lock */ > local_lock(&s->cpu_slab->lock); > c = this_cpu_ptr(s->cpu_slab); > if (unlikely(slab != c->slab)) { > @@ -3606,11 +3601,8 @@ static __always_inline void do_slab_free > c->tid = next_tid(tid); > > local_unlock(&s->cpu_slab->lock); > -#endif > - stat(s, FREE_FASTPATH); > - } else > - __slab_free(s, slab, head, tail_obj, cnt, addr); > - > + } > + stat(s, FREE_FASTPATH); > } > > static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab,