From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10580C433FE for ; Mon, 11 Oct 2021 14:52:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F0FE660EB6 for ; Mon, 11 Oct 2021 14:52:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244191AbhJKOyi (ORCPT ); Mon, 11 Oct 2021 10:54:38 -0400 Received: from mail.kernel.org ([198.145.29.99]:33748 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238405AbhJKOya (ORCPT ); Mon, 11 Oct 2021 10:54:30 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id D920F60ED4; Mon, 11 Oct 2021 14:52:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1633963950; bh=6F4fXCCvsvFVTEE2HFP3ezyGetHLyMVJDY1zgytRX/A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DhBBBGqW1rw4z+QnZygi7Zg5NF95ufXmswE46VtlCgP44kwsziK04wupiZJ5YjuxU atPWxl3Gkxu1pfe8J1CEO2h8/KhI7vRuLb6NxnJGYgN8NVMvwGKM2ztv0qnBVwwM1/ YwoYpLPu8Z9V+5JMnIYDAuNJW9ZAEc8tHTqoc0vFgs/KjHztJyh5pGDx5/KAAT7rFi IQoFZ8UACLqcfmAWy9nqMzOgtcApUPAlTbtiotaxwp+1C5k8YDwPhZXefbEG5dm8ri YEQ15Me/tWqPYEYg8PEkOP7ejw+h3vK2EamL3WGhUNYJERGWkm2slhBfLyxrDN2CDe e561Ud0v4ZgAg== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Sebastian Andrzej Siewior , Valentin Schneider , Peter Zijlstra , Uladzislau Rezki , Thomas Gleixner , Valentin Schneider , Boqun Feng , Neeraj Upadhyay , Josh Triplett , Joel Fernandes , rcu@vger.kernel.org Subject: [PATCH 10/11] rcu: Apply callbacks processing time limit only on softirq Date: Mon, 11 Oct 2021 16:51:39 +0200 Message-Id: <20211011145140.359412-11-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211011145140.359412-1-frederic@kernel.org> References: <20211011145140.359412-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Time limit only makes sense when callbacks are serviced in softirq mode because: _ In case we need to get back to the scheduler, cond_resched_tasks_rcu_qs() is called after each callback. _ In case some other softirq vector needs the CPU, the call to local_bh_enable() before cond_resched_tasks_rcu_qs() takes care about them via a call to do_softirq(). Therefore, make sure the time limit only applies to softirq mode. Reviewed-by: Valentin Schneider Tested-by: Valentin Schneider Tested-by: Sebastian Andrzej Siewior Signed-off-by: Frederic Weisbecker Cc: Valentin Schneider Cc: Peter Zijlstra Cc: Sebastian Andrzej Siewior Cc: Josh Triplett Cc: Joel Fernandes Cc: Boqun Feng Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Thomas Gleixner --- kernel/rcu/tree.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7a655d93a28a..0ca4f4ec724d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2498,7 +2498,7 @@ static void rcu_do_batch(struct rcu_data *rdp) div = READ_ONCE(rcu_divisor); div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); - if (unlikely(bl > 100)) { + if (in_serving_softirq() && unlikely(bl > 100)) { long rrn = READ_ONCE(rcu_resched_ns); rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn; @@ -2538,6 +2538,17 @@ static void rcu_do_batch(struct rcu_data *rdp) if (in_serving_softirq()) { if (count >= bl && (need_resched() || !is_idle_task(current))) break; + /* + * Make sure we don't spend too much time here and deprive other + * softirq vectors of CPU cycles. + */ + if (unlikely(tlimit)) { + /* only call local_clock() every 32 callbacks */ + if (likely((count & 31) || local_clock() < tlimit)) + continue; + /* Exceeded the time limit, so leave. */ + break; + } } else { local_bh_enable(); lockdep_assert_irqs_enabled(); @@ -2545,18 +2556,6 @@ static void rcu_do_batch(struct rcu_data *rdp) lockdep_assert_irqs_enabled(); local_bh_disable(); } - - /* - * Make sure we don't spend too much time here and deprive other - * softirq vectors of CPU cycles. - */ - if (unlikely(tlimit)) { - /* only call local_clock() every 32 callbacks */ - if (likely((count & 31) || local_clock() < tlimit)) - continue; - /* Exceeded the time limit, so leave. */ - break; - } } rcu_nocb_lock_irqsave(rdp, flags); -- 2.25.1