From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6754EC35296 for ; Tue, 10 May 2022 13:56:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245436AbiEJN5i (ORCPT ); Tue, 10 May 2022 09:57:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245164AbiEJNig (ORCPT ); Tue, 10 May 2022 09:38:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A1CF261972; Tue, 10 May 2022 06:27:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 159FC615C8; Tue, 10 May 2022 13:27:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E7396C385A6; Tue, 10 May 2022 13:27:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1652189270; bh=0XmO/YAfbsrugQJMEEhgrFYI+ZaVod6XCk3WLtjn0xA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UyX9MzcYJzMh0AntbjGX+5fLaDYcSGXunysi52um0ZqiK9BaWJfHfIR+pZ0LkbAQO jHcSPc49qnIcxL2mA1Z1dE3kJ6Idhx2GSCObtAn03SPSfK+nLVzzySqE1P3n9KEPtm lByoQMgRADE6U+CkK0XtXOgF99awezip9XWiveGQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Valentin Schneider , Sebastian Andrzej Siewior , Frederic Weisbecker , Peter Zijlstra , Josh Triplett , Joel Fernandes , Boqun Feng , Neeraj Upadhyay , Uladzislau Rezki , Thomas Gleixner , "Paul E. McKenney" Subject: [PATCH 5.10 65/70] rcu: Apply callbacks processing time limit only on softirq Date: Tue, 10 May 2022 15:08:24 +0200 Message-Id: <20220510130734.768705317@linuxfoundation.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220510130732.861729621@linuxfoundation.org> References: <20220510130732.861729621@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Frederic Weisbecker commit a554ba288845fd3f6f12311fd76a51694233458a upstream. Time limit only makes sense when callbacks are serviced in softirq mode because: _ In case we need to get back to the scheduler, cond_resched_tasks_rcu_qs() is called after each callback. _ In case some other softirq vector needs the CPU, the call to local_bh_enable() before cond_resched_tasks_rcu_qs() takes care about them via a call to do_softirq(). Therefore, make sure the time limit only applies to softirq mode. Reviewed-by: Valentin Schneider Tested-by: Valentin Schneider Tested-by: Sebastian Andrzej Siewior Signed-off-by: Frederic Weisbecker Cc: Valentin Schneider Cc: Peter Zijlstra Cc: Sebastian Andrzej Siewior Cc: Josh Triplett Cc: Joel Fernandes Cc: Boqun Feng Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Thomas Gleixner Signed-off-by: Paul E. McKenney [UR: backport to 5.10-stable] Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Greg Kroah-Hartman --- kernel/rcu/tree.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2456,7 +2456,7 @@ static void rcu_do_batch(struct rcu_data div = READ_ONCE(rcu_divisor); div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); - if (unlikely(bl > 100)) { + if (in_serving_softirq() && unlikely(bl > 100)) { long rrn = READ_ONCE(rcu_resched_ns); rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn; @@ -2494,6 +2494,18 @@ static void rcu_do_batch(struct rcu_data if (-rcl.len >= bl && (need_resched() || (!is_idle_task(current) && !rcu_is_callbacks_kthread()))) break; + + /* + * Make sure we don't spend too much time here and deprive other + * softirq vectors of CPU cycles. + */ + if (unlikely(tlimit)) { + /* only call local_clock() every 32 callbacks */ + if (likely((-rcl.len & 31) || local_clock() < tlimit)) + continue; + /* Exceeded the time limit, so leave. */ + break; + } } else { local_bh_enable(); lockdep_assert_irqs_enabled(); @@ -2501,18 +2513,6 @@ static void rcu_do_batch(struct rcu_data lockdep_assert_irqs_enabled(); local_bh_disable(); } - - /* - * Make sure we don't spend too much time here and deprive other - * softirq vectors of CPU cycles. - */ - if (unlikely(tlimit)) { - /* only call local_clock() every 32 callbacks */ - if (likely((-rcl.len & 31) || local_clock() < tlimit)) - continue; - /* Exceeded the time limit, so leave. */ - break; - } } local_irq_save(flags);