From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1031953AbbKDSpI (ORCPT ); Wed, 4 Nov 2015 13:45:08 -0500 Received: from shelob.surriel.com ([74.92.59.67]:46446 "EHLO shelob.surriel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1031497AbbKDSpD (ORCPT ); Wed, 4 Nov 2015 13:45:03 -0500 X-Greylist: delayed 1183 seconds by postgrey-1.27 at vger.kernel.org; Wed, 04 Nov 2015 13:45:03 EST Date: Wed, 4 Nov 2015 13:25:15 -0500 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, mingo@kernel.org, mgorman@suse.de, jstancek@redhat.com Subject: [PATCH] sched,numa cap pte scanning overhead to 3% of run time Message-ID: <20151104132515.07e41b75@annuminas.surriel.com> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.26; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org There is a fundamental mismatch between the runtime based NUMA scanning at the task level, and the wall clock time NUMA scanning at the mm level. On a severely overloaded system, with very large processes, this mismatch can cause the system to spend all of its time in change_prot_numa(). This can happen if the task spends at least two ticks in change_prot_numa(), and only gets two ticks of CPU time in the real time between two scan intervals of the mm. This patch ensures that if the system is so busy that the task got rescheduled during change_prot_numa(), we never spend more than 3% of run time scanning PTEs. This patch does nothing if the CPU is not overloaded at all, and the task is not rescheduled during change_prot_numa(). All of the above only works if we fix the math underflow issue in task_numa_tick, so do that as well (Jan Stancek). Signed-off-by: Rik van Riel Reported-and-tested-by: Jan Stancek --- kernel/sched/fair.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 824aa9f501a3..e9b9ac424a76 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2155,6 +2155,7 @@ void task_numa_work(struct callback_head *work) unsigned long migrate, next_scan, now = jiffies; struct task_struct *p = current; struct mm_struct *mm = p->mm; + u64 runtime = p->se.sum_exec_runtime; struct vm_area_struct *vma; unsigned long start, end; unsigned long nr_pte_updates = 0; @@ -2277,6 +2278,20 @@ void task_numa_work(struct callback_head *work) else reset_ptenuma_scan(p); up_read(&mm->mmap_sem); + + /* + * There is a fundamental mismatch between the runtime based + * NUMA scanning at the task level, and the wall clock time + * NUMA scanning at the mm level. On a severely overloaded + * system, with very large processes, this mismatch can cause + * the system to spend all of its time in change_prot_numa(). + * Limit NUMA PTE scanning to 3% of the task's run time, if + * we spent so much time scanning we got rescheduled. + */ + if (unlikely(p->se.sum_exec_runtime != runtime)) { + u64 diff = p->se.sum_exec_runtime - runtime; + p->node_stamp += 32 * diff; + } } /* @@ -2302,7 +2317,7 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr) now = curr->se.sum_exec_runtime; period = (u64)curr->numa_scan_period * NSEC_PER_MSEC; - if (now - curr->node_stamp > period) { + if (now > curr->node_stamp + period) { if (!curr->node_stamp) curr->numa_scan_period = task_scan_min(curr); curr->node_stamp += period;