From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756359AbcG0Nzd (ORCPT ); Wed, 27 Jul 2016 09:55:33 -0400 Received: from resqmta-ch2-05v.sys.comcast.net ([69.252.207.37]:37792 "EHLO resqmta-ch2-05v.sys.comcast.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755912AbcG0Nzb (ORCPT ); Wed, 27 Jul 2016 09:55:31 -0400 Date: Wed, 27 Jul 2016 08:55:28 -0500 (CDT) From: Christoph Lameter X-X-Sender: cl@east.gentwo.org To: Chris Metcalf cc: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Viresh Kumar , Catalin Marinas , Will Deacon , Andy Lutomirski , Daniel Lezcano , linux-doc@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org Subject: clocksource_watchdog causing scheduling of timers every second (was [v13] support "task_isolation" mode) In-Reply-To: Message-ID: References: <1468529299-27929-1-git-send-email-cmetcalf@mellanox.com> <7a3f66f7-5011-7d59-2e0e-f57e4e42e6b6@mellanox.com> Content-Type: text/plain; charset=US-ASCII X-CMAE-Envelope: MS4wfJFm7Q0eKpQZrkR8LtLsMlEkfHw9mz7UCcVMUpHTs1t5FMVxkY+hdfP0QdrrzdGkt4fuSfN1oTKleyGK257vZ4DGlvrWUGMIBBFCkmqyAYjvMZkJ2kTR WSxXwYgWLHriW6jA8Ivodg2UJxznxBdkmfQhhDfI87BbaWzpxguIwBQHf2e7vbFS8HnwXPrlqA8mf9oP4m20O+AJdUljmhjj6Qy59IX89GjfNB6sEV0oH3bE rA9N8/N7DDFy8j+bcJsCDJqBzcz2YvMMaP1zY/ixdUDGzCJhv+vlSmREAyGheeWIu2fz8KIArqU3/z4WXPCWu/IAncme+RbBjs4i3DdEXlrL4ruf5EL4C14O 00z2JUvwPHbAHbu77M51Bb1DktOrKCcjOVpOhnMLIcQ+vmVPl778Zs/6arHXVNAsG6/aDk35FADRZgo/bQ2KKIuFE5WE49bb7oqYMn86LHGY5XAWS6PHJqMb u2Wt/Win2Q1/1xLct1jwVNwoN+GS1KrfZxgb0sDMgZOzLS3nIh2uzixVmys/yToUO8DJpov9JUjX8fU4AcD8ypK6z4Sy2RdyJQdeI8WkwEoJg+cS/YCjxldu SOx00vvj3/Auw0O0xgxjZ8r89jhAp830uZpKZsafwitqHr6By6t/usUT1fYDZtfMu3Q7MZzTS+hJespemrGeicWF4YNLZFdOQLp+lNDvunA2QIJO4NUSFojn SjAeEBdTrrgkVtykhfb4BjXGPgKfozcfYsiM1WHApPLDzaEVzY8cZw== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 25 Jul 2016, Christoph Lameter wrote: > Guess so. I will have a look at this when I get some time again. Ok so the problem is the clocksource_watchdog() function in kernel/time/clocksource.c. This function is active if CONFIG_CLOCKSOURCE_WATCHDOG is defined. It will check the timesources of each processor for being within bounds and then reschedule itself on the next one. The purpose of the function seems to be to determine *if* a clocksource is unstable. It does not mean that the clocksource *is* unstable. The critical piece of code is this: /* * Cycle through CPUs to check if the CPUs stay synchronized * to each other. */ next_cpu = cpumask_next(raw_smp_processor_id(), cpu_online_mask); if (next_cpu >= nr_cpu_ids) next_cpu = cpumask_first(cpu_online_mask); watchdog_timer.expires += WATCHDOG_INTERVAL; add_timer_on(&watchdog_timer, next_cpu); Should we just cycle through the cpus that are not isolated? Otherwise we need to have some means to check the clocksources for accuracy remotely (probably impossible for TSC etc). The WATCHDOG_INTERVAL is 1 second so this causes an interrupt every second. Note that we are running with the patch that removes the 1 HZ mininum time tick. With an older kernel code base (redhat) we can keep the kernel quiet for minutes. The clocksource watchdog causes timers to fire again. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Lameter Subject: clocksource_watchdog causing scheduling of timers every second (was [v13] support "task_isolation" mode) Date: Wed, 27 Jul 2016 08:55:28 -0500 (CDT) Message-ID: References: <1468529299-27929-1-git-send-email-cmetcalf@mellanox.com> <7a3f66f7-5011-7d59-2e0e-f57e4e42e6b6@mellanox.com> Content-Type: text/plain; charset=US-ASCII Return-path: In-Reply-To: Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Chris Metcalf Cc: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Viresh Kumar , Catalin Marinas , Will Deacon , Andy Lutomirski , Daniel Lezcano , linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-api@vger.kernel.org On Mon, 25 Jul 2016, Christoph Lameter wrote: > Guess so. I will have a look at this when I get some time again. Ok so the problem is the clocksource_watchdog() function in kernel/time/clocksource.c. This function is active if CONFIG_CLOCKSOURCE_WATCHDOG is defined. It will check the timesources of each processor for being within bounds and then reschedule itself on the next one. The purpose of the function seems to be to determine *if* a clocksource is unstable. It does not mean that the clocksource *is* unstable. The critical piece of code is this: /* * Cycle through CPUs to check if the CPUs stay synchronized * to each other. */ next_cpu = cpumask_next(raw_smp_processor_id(), cpu_online_mask); if (next_cpu >= nr_cpu_ids) next_cpu = cpumask_first(cpu_online_mask); watchdog_timer.expires += WATCHDOG_INTERVAL; add_timer_on(&watchdog_timer, next_cpu); Should we just cycle through the cpus that are not isolated? Otherwise we need to have some means to check the clocksources for accuracy remotely (probably impossible for TSC etc). The WATCHDOG_INTERVAL is 1 second so this causes an interrupt every second. Note that we are running with the patch that removes the 1 HZ mininum time tick. With an older kernel code base (redhat) we can keep the kernel quiet for minutes. The clocksource watchdog causes timers to fire again.