From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754817AbdKASLb (ORCPT ); Wed, 1 Nov 2017 14:11:31 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52632 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754613AbdKASL3 (ORCPT ); Wed, 1 Nov 2017 14:11:29 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 2B884356CB Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dzickus@redhat.com Date: Wed, 1 Nov 2017 14:11:27 -0400 From: Don Zickus To: Guenter Roeck Cc: Thomas Gleixner , Peter Zijlstra , linux-kernel@vger.kernel.org, Ingo Molnar Subject: Re: Crashes in perf_event_ctx_lock_nested Message-ID: <20171101181126.j727fqjmdthjz4xk@redhat.com> References: <20171030224512.GA13592@roeck-us.net> <20171031134850.ynix2zqypmca2mtt@hirez.programming.kicks-ass.net> <20171031221107.GA12133@roeck-us.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171031221107.GA12133@roeck-us.net> User-Agent: NeoMutt/20171027 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 01 Nov 2017 18:11:29 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Oct 31, 2017 at 03:11:07PM -0700, Guenter Roeck wrote: > On Tue, Oct 31, 2017 at 10:32:00PM +0100, Thomas Gleixner wrote: > > [ ...] > > > So we have to revert > > > > a33d44843d45 ("watchdog/hardlockup/perf: Simplify deferred event destroy") > > > > Patch attached. > > > > Tested-by: Guenter Roeck > > There is still a problem. When running > > echo 6 > /proc/sys/kernel/watchdog_thresh > echo 5 > /proc/sys/kernel/watchdog_thresh > > repeatedly, the message > > NMI watchdog: Enabled. Permanently consumes one hw-PMU counter. > > stops after a while (after ~10-30 iterations, with fluctuations). > After adding trace messages into hardlockup_detector_perf_disable() > and hardlockup_detector_perf_enable(), I see: > > hardlockup_detector_perf_disable: disable(0): Number of CPUs: 3 > hardlockup_detector_perf_disable: disable(1): Number of CPUs: 2 > hardlockup_detector_perf_disable: disable(2): Number of CPUs: 1 > hardlockup_detector_perf_disable: disable(3): Number of CPUs: 0 > ... > hardlockup_detector_perf_disable: disable(0): Number of CPUs: 2 > hardlockup_detector_perf_disable: disable(1): Number of CPUs: 1 > hardlockup_detector_perf_disable: disable(2): Number of CPUs: 0 > hardlockup_detector_perf_disable: disable(3): Number of CPUs: -1 > ... > hardlockup_detector_perf_enable: enable(1): Number of CPUs: -6 > hardlockup_detector_perf_enable: enable(3): Number of CPUs: -5 > hardlockup_detector_perf_enable: enable(2): Number of CPUs: -4 > hardlockup_detector_perf_enable: enable(0): Number of CPUs: -3 > > Maybe watchdog_cpus needs to be atomic ? I switched it to atomic and it solves that problem. The functionality isn't broken currently, just the informational message. Patch attached to try. Cheers, Don ---8<-------------- diff --git a/kernel/watchdog_hld.c b/kernel/watchdog_hld.c index a7f137c1933a..8ee4da223b35 100644 --- a/kernel/watchdog_hld.c +++ b/kernel/watchdog_hld.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -25,7 +26,7 @@ static DEFINE_PER_CPU(struct perf_event *, dead_event); static struct cpumask dead_events_mask; static unsigned long hardlockup_allcpu_dumped; -static unsigned int watchdog_cpus; +static atomic_t watchdog_cpus = ATOMIC_INIT(0); void arch_touch_nmi_watchdog(void) { @@ -189,7 +190,8 @@ void hardlockup_detector_perf_enable(void) if (hardlockup_detector_event_create()) return; - if (!watchdog_cpus++) + /* use original value for check */ + if (!atomic_fetch_inc(&watchdog_cpus)) pr_info("Enabled. Permanently consumes one hw-PMU counter.\n"); perf_event_enable(this_cpu_read(watchdog_ev)); @@ -207,7 +209,7 @@ void hardlockup_detector_perf_disable(void) this_cpu_write(watchdog_ev, NULL); this_cpu_write(dead_event, event); cpumask_set_cpu(smp_processor_id(), &dead_events_mask); - watchdog_cpus--; + atomic_dec(&watchdog_cpus); } }