From: Oleg Nesterov <oleg@redhat.com>
To: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Fox <afox@redhat.com>,
Stephen Johnston <sjohnsto@redhat.com>,
linux-kernel@vger.kernel.org,
Stanislaw Gruszka <sgruszka@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>
Subject: [PATCH v2] sched/cputime: make scale_stime() more precise
Date: Mon, 27 Jan 2020 13:28:17 +0100 [thread overview]
Message-ID: <20200127122817.GA10957@redhat.com> (raw)
In-Reply-To: <20190718131834.GA22211@redhat.com>
People report that utime and stime from /proc/<pid>/stat become very
wrong when the numbers are big enough, especially if you watch these
counters incrementally.
Say, if the monitored process runs 100 days 50/50 in user/kernel mode
it looks as if it runs 20 minutes entirely in kernel mode, then 20
minutes in user mode. See the test-case which tries to demonstrate this
behaviour:
https://lore.kernel.org/lkml/20200124154215.GA14714@redhat.com/
The new implementation does the additional div64_u64_rem() but according
to my naive measurements it is faster on x86_64, much faster if rtime/etc
are big enough. See
https://lore.kernel.org/lkml/20200123130541.GA30620@redhat.com/
Signed-off-by: Oleg Nesterov <oleg@redhat.com>
---
kernel/sched/cputime.c | 65 +++++++++++++++++++++++++-------------------------
1 file changed, 32 insertions(+), 33 deletions(-)
diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c
index d43318a..ae1ea09 100644
--- a/kernel/sched/cputime.c
+++ b/kernel/sched/cputime.c
@@ -528,42 +528,41 @@ void account_idle_ticks(unsigned long ticks)
*/
static u64 scale_stime(u64 stime, u64 rtime, u64 total)
{
- u64 scaled;
+ u64 res = 0, div, rem;
+ int shift;
- for (;;) {
- /* Make sure "rtime" is the bigger of stime/rtime */
- if (stime > rtime)
- swap(rtime, stime);
-
- /* Make sure 'total' fits in 32 bits */
- if (total >> 32)
- goto drop_precision;
-
- /* Does rtime (and thus stime) fit in 32 bits? */
- if (!(rtime >> 32))
- break;
-
- /* Can we just balance rtime/stime rather than dropping bits? */
- if (stime >> 31)
- goto drop_precision;
-
- /* We can grow stime and shrink rtime and try to make them both fit */
- stime <<= 1;
- rtime >>= 1;
- continue;
-
-drop_precision:
- /* We drop from rtime, it has more bits than stime */
- rtime >>= 1;
- total >>= 1;
+ /* can stime * rtime overflow ? */
+ if (ilog2(stime) + ilog2(rtime) > 62) {
+ /*
+ * (rtime * stime) / total is equal to
+ *
+ * (rtime / total) * stime +
+ * (rtime % total) * stime / total
+ *
+ * if nothing overflows. Can the 1st multiplication
+ * overflow? Yes, but we do not care: this can only
+ * happen if the end result can't fit in u64 anyway.
+ *
+ * So the code below does
+ *
+ * res = (rtime / total) * stime;
+ * rtime = rtime % total;
+ */
+ div = div64_u64_rem(rtime, total, &rem);
+ res = div * stime;
+ rtime = rem;
+
+ shift = ilog2(stime) + ilog2(rtime) - 62;
+ if (shift > 0) {
+ /* drop precision */
+ rtime >>= shift;
+ total >>= shift;
+ if (!total)
+ return res;
+ }
}
- /*
- * Make sure gcc understands that this is a 32x32->64 multiply,
- * followed by a 64/32->64 divide.
- */
- scaled = div_u64((u64) (u32) stime * (u64) (u32) rtime, (u32)total);
- return scaled;
+ return res + div64_u64(stime * rtime, total);
}
/*
--
2.5.0
next prev parent reply other threads:[~2020-01-27 12:28 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-18 13:18 [PATCH] sched/cputime: make scale_stime() more precise Oleg Nesterov
2019-07-18 13:21 ` Oleg Nesterov
2019-07-18 14:55 ` Oleg Nesterov
2019-07-19 11:03 ` Peter Zijlstra
2019-07-19 13:47 ` Peter Zijlstra
2019-07-19 14:37 ` Oleg Nesterov
2019-07-22 19:56 ` Peter Zijlstra
2019-07-23 14:00 ` Oleg Nesterov
2019-07-23 14:29 ` Oleg Nesterov
2019-07-19 14:03 ` Oleg Nesterov
2019-07-22 19:45 ` Peter Zijlstra
2019-07-22 10:52 ` Stanislaw Gruszka
2019-07-22 20:00 ` Peter Zijlstra
2019-07-23 9:37 ` Stanislaw Gruszka
2020-01-22 16:46 ` Oleg Nesterov
2020-01-23 13:05 ` Oleg Nesterov
2020-01-24 15:42 ` Oleg Nesterov
2020-01-27 12:28 ` Oleg Nesterov [this message]
2020-05-15 17:24 ` [PATCH v2] " Oleg Nesterov
2020-05-19 17:25 ` Peter Zijlstra
2020-05-19 18:33 ` Linus Torvalds
2020-05-19 18:42 ` Peter Zijlstra
2020-05-19 19:11 ` Peter Zijlstra
2020-05-19 19:51 ` Linus Torvalds
2020-05-20 15:24 ` Oleg Nesterov
2020-05-20 15:36 ` Peter Zijlstra
2020-05-20 20:10 ` Peter Zijlstra
2020-05-21 13:26 ` Oleg Nesterov
2020-06-16 12:21 ` [tip: sched/core] sched/cputime: Improve cputime_adjust() tip-bot2 for Oleg Nesterov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200127122817.GA10957@redhat.com \
--to=oleg@redhat.com \
--cc=afox@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=sgruszka@redhat.com \
--cc=sjohnsto@redhat.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).