From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8840FC388F9 for ; Thu, 12 Nov 2020 00:11:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BBDFE2076E for ; Thu, 12 Nov 2020 00:11:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="yuHOwueL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BBDFE2076E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 183B06B0036; Wed, 11 Nov 2020 19:11:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E6F96B005D; Wed, 11 Nov 2020 19:11:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC8AC6B0068; Wed, 11 Nov 2020 19:11:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0189.hostedemail.com [216.40.44.189]) by kanga.kvack.org (Postfix) with ESMTP id B26346B0036 for ; Wed, 11 Nov 2020 19:11:32 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 497D03626 for ; Thu, 12 Nov 2020 00:11:32 +0000 (UTC) X-FDA: 77473837224.18.silk06_5c07ea927301 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 23574100ED3BE for ; Thu, 12 Nov 2020 00:11:32 +0000 (UTC) X-HE-Tag: silk06_5c07ea927301 X-Filterd-Recvd-Size: 6067 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Thu, 12 Nov 2020 00:11:31 +0000 (UTC) Received: from paulmck-ThinkPad-P72.home (50-39-104-11.bvtn.or.frontiernet.net [50.39.104.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 61F942072C; Thu, 12 Nov 2020 00:11:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1605139890; bh=UUizPUi+5E2gGhUXhujEXQx6cSOfSopZTLC533nokJc=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=yuHOwueLPRh7XYhg7N1PKfNbkkla0wA87gc4bZSptoORGlluEdo8F9NTTrSZWIzMo fHJ3yLY04gmvNPf6eJmTpLkx2nSegrutBXIErzzSaaC/0mcw/APa0Ex+jXitD37e7N IaCO7rVaSLzEPVRe268SIRGIEi5tJ14Tj4E2Nfuw= Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id F0BFF352287B; Wed, 11 Nov 2020 16:11:29 -0800 (PST) Date: Wed, 11 Nov 2020 16:11:29 -0800 From: "Paul E. McKenney" To: Marco Elver Cc: Steven Rostedt , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Mark Rutland , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, peterz@infradead.org Subject: Re: [PATCH] kfence: Avoid stalling work queue task without allocations Message-ID: <20201112001129.GD3249@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org References: <20201110135320.3309507-1-elver@google.com> <20201111133813.GA81547@elver.google.com> <20201111130543.27d29462@gandalf.local.home> <20201111182333.GA3249@paulmck-ThinkPad-P72> <20201111183430.GN517454@elver.google.com> <20201111192123.GB3249@paulmck-ThinkPad-P72> <20201111202153.GT517454@elver.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201111202153.GT517454@elver.google.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Nov 11, 2020 at 09:21:53PM +0100, Marco Elver wrote: > On Wed, Nov 11, 2020 at 11:21AM -0800, Paul E. McKenney wrote: > [...] > > > > rcu: Don't invoke try_invoke_on_locked_down_task() with irqs disabled > > > > > > Sadly, no, next-20201110 already included that one, and that's what I > > > tested and got me all those warnings above. > > > > Hey, I had to ask! The only uncertainty I seee is the acquisition of > > the lock in rcu_iw_handler(), for which I add a lockdep check in the > > (untested) patch below. The other thing I could do is sprinkle such > > checks through the stall-warning code on the assumption that something > > RCU is calling is enabling interrupts. > > > > Other thoughts? > > > > Thanx, Paul > > > > ------------------------------------------------------------------------ > > > > diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h > > index 70d48c5..3d67650 100644 > > --- a/kernel/rcu/tree_stall.h > > +++ b/kernel/rcu/tree_stall.h > > @@ -189,6 +189,7 @@ static void rcu_iw_handler(struct irq_work *iwp) > > > > rdp = container_of(iwp, struct rcu_data, rcu_iw); > > rnp = rdp->mynode; > > + lockdep_assert_irqs_disabled(); > > raw_spin_lock_rcu_node(rnp); > > if (!WARN_ON_ONCE(!rdp->rcu_iw_pending)) { > > rdp->rcu_iw_gp_seq = rnp->gp_seq; > > This assert didn't fire yet, I just get more of the below. I'll keep > rerunning, but am not too hopeful... Is bisection a possibility? Failing that, please see the updated patch below. This adds a few more calls to lockdep_assert_irqs_disabled(), but perhaps more helpfully dumps the current stack of the CPU that the RCU grace-period kthread wants to run on in the case where this kthread has been starved of CPU. Thanx, Paul ------------------------------------------------------------------------ diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 70d48c5..d203ea0 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -189,6 +189,7 @@ static void rcu_iw_handler(struct irq_work *iwp) rdp = container_of(iwp, struct rcu_data, rcu_iw); rnp = rdp->mynode; + lockdep_assert_irqs_disabled(); raw_spin_lock_rcu_node(rnp); if (!WARN_ON_ONCE(!rdp->rcu_iw_pending)) { rdp->rcu_iw_gp_seq = rnp->gp_seq; @@ -449,21 +450,32 @@ static void print_cpu_stall_info(int cpu) /* Complain about starvation of grace-period kthread. */ static void rcu_check_gp_kthread_starvation(void) { + int cpu; struct task_struct *gpk = rcu_state.gp_kthread; unsigned long j; if (rcu_is_gp_kthread_starving(&j)) { + cpu = gpk ? task_cpu(gpk) : -1; pr_err("%s kthread starved for %ld jiffies! g%ld f%#x %s(%d) ->state=%#lx ->cpu=%d\n", rcu_state.name, j, (long)rcu_seq_current(&rcu_state.gp_seq), data_race(rcu_state.gp_flags), gp_state_getname(rcu_state.gp_state), rcu_state.gp_state, - gpk ? gpk->state : ~0, gpk ? task_cpu(gpk) : -1); + gpk ? gpk->state : ~0, cpu); if (gpk) { pr_err("\tUnless %s kthread gets sufficient CPU time, OOM is now expected behavior.\n", rcu_state.name); pr_err("RCU grace-period kthread stack dump:\n"); + lockdep_assert_irqs_disabled(); sched_show_task(gpk); + lockdep_assert_irqs_disabled(); + if (cpu >= 0) { + pr_err("Stack dump where RCU grace-period kthread last ran:\n"); + if (!trigger_single_cpu_backtrace(cpu)) + dump_cpu_task(cpu); + } + lockdep_assert_irqs_disabled(); wake_up_process(gpk); + lockdep_assert_irqs_disabled(); } } }