From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1EF6C388F3 for ; Tue, 10 May 2022 13:55:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245482AbiEJN5u (ORCPT ); Tue, 10 May 2022 09:57:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245163AbiEJNig (ORCPT ); Tue, 10 May 2022 09:38:36 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09353262643; Tue, 10 May 2022 06:27:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B5B33B81DA2; Tue, 10 May 2022 13:27:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EE139C385C6; Tue, 10 May 2022 13:27:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1652189267; bh=NMFFJtZLPoq1knCC/w0d1kwtFNpn/m+HxWT5aO4P7g0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iFzrkqqWcFHuqGknCvnPOwA0OMAtzq+rgdMjObePyDRbBqrnWG6bgRXucIkRKtyo0 WImRhGo6kv3kCCxN5G+B0jLVTnaUcvpHBNAL08RsTAOOW4UbUEvMxwKOQXxiZ/Rr1K ey36orMIPitqIVvVLETCESkONDR+2VYYZYX+g2CQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Valentin Schneider , Sebastian Andrzej Siewior , Frederic Weisbecker , Peter Zijlstra , Josh Triplett , Joel Fernandes , Boqun Feng , Neeraj Upadhyay , Uladzislau Rezki , Thomas Gleixner , "Paul E. McKenney" Subject: [PATCH 5.10 64/70] rcu: Fix callbacks processing time limit retaining cond_resched() Date: Tue, 10 May 2022 15:08:23 +0200 Message-Id: <20220510130734.740079392@linuxfoundation.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220510130732.861729621@linuxfoundation.org> References: <20220510130732.861729621@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Frederic Weisbecker commit 3e61e95e2d095e308616cba4ffb640f95a480e01 upstream. The callbacks processing time limit makes sure we are not exceeding a given amount of time executing the queue. However its "continue" clause bypasses the cond_resched() call on rcuc and NOCB kthreads, delaying it until we reach the limit, which can be very long... Make sure the scheduler has a higher priority than the time limit. Reviewed-by: Valentin Schneider Tested-by: Valentin Schneider Tested-by: Sebastian Andrzej Siewior Signed-off-by: Frederic Weisbecker Cc: Valentin Schneider Cc: Peter Zijlstra Cc: Sebastian Andrzej Siewior Cc: Josh Triplett Cc: Joel Fernandes Cc: Boqun Feng Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Thomas Gleixner Signed-off-by: Paul E. McKenney [UR: backport to 5.10-stable + commit update] Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Greg Kroah-Hartman --- kernel/rcu/tree.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2490,10 +2490,22 @@ static void rcu_do_batch(struct rcu_data * Stop only if limit reached and CPU has something to do. * Note: The rcl structure counts down from zero. */ - if (-rcl.len >= bl && !offloaded && - (need_resched() || - (!is_idle_task(current) && !rcu_is_callbacks_kthread()))) - break; + if (in_serving_softirq()) { + if (-rcl.len >= bl && (need_resched() || + (!is_idle_task(current) && !rcu_is_callbacks_kthread()))) + break; + } else { + local_bh_enable(); + lockdep_assert_irqs_enabled(); + cond_resched_tasks_rcu_qs(); + lockdep_assert_irqs_enabled(); + local_bh_disable(); + } + + /* + * Make sure we don't spend too much time here and deprive other + * softirq vectors of CPU cycles. + */ if (unlikely(tlimit)) { /* only call local_clock() every 32 callbacks */ if (likely((-rcl.len & 31) || local_clock() < tlimit)) @@ -2501,14 +2513,6 @@ static void rcu_do_batch(struct rcu_data /* Exceeded the time limit, so leave. */ break; } - if (offloaded) { - WARN_ON_ONCE(in_serving_softirq()); - local_bh_enable(); - lockdep_assert_irqs_enabled(); - cond_resched_tasks_rcu_qs(); - lockdep_assert_irqs_enabled(); - local_bh_disable(); - } } local_irq_save(flags);