From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755352AbcCPQVu (ORCPT ); Wed, 16 Mar 2016 12:21:50 -0400 Received: from skprod3.natinst.com ([130.164.80.24]:55899 "EHLO ni.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753166AbcCPQVq (ORCPT ); Wed, 16 Mar 2016 12:21:46 -0400 Date: Wed, 16 Mar 2016 11:21:29 -0500 From: Josh Cartwright To: Paul Gortmaker Cc: Steven Rostedt , LKML , linux-rt-users , Thomas Gleixner , Carsten Emde , John Kacur , Sebastian Andrzej Siewior Subject: Re: [ANNOUNCE] 3.14.64-rt67 Message-ID: <20160316162129.GH28102@jcartwri.amer.corp.natinst.com> References: <20160314114957.45d956d7@gandalf.local.home> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-MIMETrack: Itemize by SMTP Server on US-AUS-MGWOut1/AUS/H/NIC(Release 8.5.3FP6 HF1218|December 12, 2014) at 03/16/2016 11:21:30 AM, Serialize by Router on US-AUS-MGWOut1/AUS/H/NIC(Release 8.5.3FP6 HF1218|December 12, 2014) at 03/16/2016 11:21:30 AM, Serialize complete at 03/16/2016 11:21:30 AM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-03-16_04:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 15, 2016 at 10:50:31PM -0400, Paul Gortmaker wrote: > On Tue, Mar 15, 2016 at 7:25 PM, Paul Gortmaker > wrote: > > On Tue, Mar 15, 2016 at 5:45 PM, Paul Gortmaker > > wrote: > >> On Mon, Mar 14, 2016 at 11:49 AM, Steven Rostedt wrote: > >>> > >>> Dear RT Folks, > >>> > >>> 3.14 release on PI(E) Day! > >>> > >>> I'm pleased to announce the 3.14.64-rt67 stable release. > >> > >> Testing this with what is largely a x86-64 defconfig but with RT_FULL, > >> I now see: > >> > >> root@dell760-paul:~# dmesg|grep NOH > >> [ 8.605854] NOHZ: local_softirq_pending 100 > >> [ 8.732677] NOHZ: local_softirq_pending 100 > >> [ 8.852729] NOHZ: local_softirq_pending 100 > >> [ 8.963964] NOHZ: local_softirq_pending 100 > >> [ 9.061892] NOHZ: local_softirq_pending 100 > >> [ 9.184921] NOHZ: local_softirq_pending 100 > >> [ 9.370958] NOHZ: local_softirq_pending 100 > >> [ 9.657811] NOHZ: local_softirq_pending 100 > >> [ 9.942631] NOHZ: local_softirq_pending 100 > >> [ 10.783710] NOHZ: local_softirq_pending 100 > >> root@dell760-paul:~# > >> > >> ...early in boot (we cap them after ~10 msgs). > >> > >> I think 100 is RCU if I did my bit counting properly; remind > >> me to submit a patch that uses the human readable names. As mentioned on IRC, 100 is HRTIMER_SOFTIRQ, which I think makes more sense...at least it meshes better with the commit you identified as being problematic. > >> > >> I had a good hunch which commit was responsible but I did > >> a check of it and the one directly underneath it to be sure, > >> and the latter boots w/o any pending messages. > >> > >> git log --oneline v3.14-rt ^v3.14.64 > >> [...] > >> 0a80a6849f19 latencyhist: disable jump-labels > >> a884ef48e1ca net: provide a way to delegate processing a softirq to ksoftirqd > >> 780d7ca2fdb0 softirq: split timer softirqs out of ksoftirqd <------ > >> *** fail *** Looking at the places where HRTIMER_SOFTIRQ is raised, it did look like there is at least one case where a hrtimer started from process context would cause HRTIMER_SOFTIRQ to be set pending, but the associated ktimersoftirq/N not woken, which seems problematic. Josh diff --git a/kernel/softirq.c b/kernel/softirq.c index 7abfdab..d91f378 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -749,7 +749,7 @@ void raise_softirq_irqoff(unsigned int nr) * */ if (!current->softirq_nestcnt) - wakeup_softirqd(); + wakeup_proper_softirq(nr); } static inline int ksoftirqd_softirq_pending(void)