From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54AA8C83F18 for ; Mon, 28 Aug 2023 11:55:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232466AbjH1Lz3 (ORCPT ); Mon, 28 Aug 2023 07:55:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231825AbjH1LzE (ORCPT ); Mon, 28 Aug 2023 07:55:04 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE47811A; Mon, 28 Aug 2023 04:55:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8CEAF6395A; Mon, 28 Aug 2023 11:55:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D0E14C433C8; Mon, 28 Aug 2023 11:55:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1693223701; bh=/8tAnL+tnbTvj3j/wSQn2yohVsi210nGoYiyMuVlcY0=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=py/HHypFgxYMTvU/b1ByweLOSW13BoBw4KW9kfXxlRpbHRc88BeX9oHunbW8X3yTw zjjXxZdyA3PkmpsHSgIU/ztzFQ0o8nl/ARKXpOiDSNjo7MxXKRrBYXVSYE+Heqt6/j OWh/9iqK3xK/7M7SjzTO0NEokMbE+hWgwIl3VPlacmc41DslM7tzoDE/6bpCUbGRQL 1Z+yhnvS2jilUsuzShVkZf3gH1kz3KIjTu8z6lQH3SWX9GRmPvtglchgsUF8gxDMaP vWwDgZVnTzs254K9+ApVpnMDm/vxENSftvEAqq00w5HuhJXIFOPC74uY1rr8/07Uj1 aXFk9uDRP2fiw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 80603CE0930; Mon, 28 Aug 2023 04:54:59 -0700 (PDT) Date: Mon, 28 Aug 2023 04:54:59 -0700 From: "Paul E. McKenney" To: Huacai Chen Cc: Joel Fernandes , Thomas Gleixner , Z qiang , Huacai Chen , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Boqun Feng , Ingo Molnar , John Stultz , Stephen Boyd , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Sergey Senozhatsky , rcu@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Binbin Zhou Subject: Re: [PATCH V4 2/2] rcu: Update jiffies in rcu_cpu_stall_reset() Message-ID: <75b1d69e-b93f-43ba-8289-9465b9fa39a8@paulmck-laptop> Reply-To: paulmck@kernel.org References: <8792da20-a58e-4cc0-b3d2-231d5ade2242@paulmck-laptop> <24e34f50-32d2-4b67-8ec0-1034c984d035@paulmck-laptop> <20230825232807.GA97898@google.com> <2681134d-cc88-49a0-a1bc-4ec0816288f6@paulmck-laptop> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Aug 28, 2023 at 07:30:43PM +0800, Huacai Chen wrote: > Hi, Paul and Joel, > > On Mon, Aug 28, 2023 at 6:47 PM Paul E. McKenney wrote: > > > > On Sun, Aug 27, 2023 at 06:11:40PM -0400, Joel Fernandes wrote: > > > On Sun, Aug 27, 2023 at 1:51 AM Huacai Chen wrote: > > > [..] > > > > > > > > The only way I know of to avoid these sorts of false positives is for > > > > > > > > the user to manually suppress all timeouts (perhaps using a kernel-boot > > > > > > > > parameter for your early-boot case), do the gdb work, and then unsuppress > > > > > > > > all stalls. Even that won't work for networking, because the other > > > > > > > > system's clock will be running throughout. > > > > > > > > > > > > > > > > In other words, from what I know now, there is no perfect solution. > > > > > > > > Therefore, there are sharp limits to the complexity of any solution that > > > > > > > > I will be willing to accept. > > > > > > > I think the simplest solution is (I hope Joel will not angry): > > > > > > > > > > > > Not angry at all, just want to help. ;-). The problem is the 300*HZ solution > > > > > > will also effect the VM workloads which also do a similar reset. Allow me few > > > > > > days to see if I can take a shot at fixing it slightly differently. I am > > > > > > trying Paul's idea of setting jiffies at a later time. I think it is doable. > > > > > > I think the advantage of doing this is it will make stall detection more > > > > > > robust in this face of these gaps in jiffie update. And that solution does > > > > > > not even need us to rely on ktime (and all the issues that come with that). > > > > > > > > > > > > > > > > I wrote a patch similar to Paul's idea and sent it out for review, the > > > > > advantage being it purely is based on jiffies. Could you try it out > > > > > and let me know? > > > > If you can cc my gmail , that could be better. > > > > > > Sure, will do. > > > > > > > I have read your patch, maybe the counter (nr_fqs_jiffies_stall) > > > > should be atomic_t and we should use atomic operation to decrement its > > > > value. Because rcu_gp_fqs() can be run concurrently, and we may miss > > > > the (nr_fqs == 1) condition. > > > > > > I don't think so. There is only 1 place where RMW operation happens > > > and rcu_gp_fqs() is called only from the GP kthread. So a concurrent > > > RMW (and hence a lost update) is not possible. > > > > Huacai, is your concern that the gdb user might have created a script > > (for example, printing a variable or two, then automatically continuing), > > so that breakpoints could happen in quick successsion, such that the > > second breakpoint might run concurrently with rcu_gp_fqs()? > > > > If this can really happen, the point that Joel makes is a good one, namely > > that rcu_gp_fqs() is single-threaded and (absent rcutorture) runs only > > once every few jiffies. And gdb breakpoints, even with scripting, should > > also be rather rare. So if this is an issue, a global lock should do the > > trick, perhaps even one of the existing locks in the rcu_state structure. > > The result should then be just as performant/scalable and a lot simpler > > than use of atomics. > > Sorry, I made a mistake. Yes, there is no concurrent issue, and this > approach probably works. But I have another problem: how to ensure > that there is a jiffies update in three calls to rcu_gp_fqs()? Or in > other word, is three also a magic number here? Each of the three calls to rcu_gp_fqs() involves a wakeup of and a context switch to RCU's grace-period kthread, each of which should be sufficient to update jiffies if initially in an out-of-date-jiffies state. The three is to some extent magic, the idea being to avoid a situation where an currently running FQS reenables stall warnings immediately after gdb disables them. Obviously, if your testing shows that some other value works better, please do let us know so that we can update! But we have to start somewhere. > And I rechecked the commit message of a80be428fbc1f1f3bc9e ("rcu: Do > not disable GP stall detection in rcu_cpu_stall_reset()"). I don't > know why Sergey said that the original code disables stall-detection > forever, in fact it only disables the detection in the current GP. Well, it does disable stall detection forever in the case where the current grace period lasts forever, which if I recall correctly was the case that Sergey was encountering. Thanx, Paul > Huacai > > > > > > Could you test the patch for the issue you are seeing and provide your > > > Tested-by tag? Thanks, > > > > Either way, testing would of course be very good! ;-) > > > > Thanx, Paul