From: Luiz Capitulino <lcapitulino@redhat.com>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Wanpeng Li <kernellwp@gmail.com>, Mike Galbraith <efault@gmx.de>,
Rik van Riel <riel@redhat.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [BUG nohz]: wrong user and system time accounting
Date: Tue, 4 Apr 2017 13:36:38 -0400 [thread overview]
Message-ID: <20170404133638.178b29f7@redhat.com> (raw)
In-Reply-To: <20170403150613.77541109@redhat.com>
On Mon, 3 Apr 2017 15:06:13 -0400
Luiz Capitulino <lcapitulino@redhat.com> wrote:
> On Mon, 3 Apr 2017 17:23:17 +0200
> Frederic Weisbecker <fweisbec@gmail.com> wrote:
>
> > Do you observe aligned ticks with trace events (hrtimer_expire_entry)?
> >
> > You might want to enforce the global clock to trace that:
> >
> > echo "global" > /sys/kernel/debug/tracing/trace_clock
>
> I've used the same trace points & debugging code I've been using to debug
> this issue, and this what I'm seeing:
>
> stress-25757 [002] 2742.717507: function: enter_from_user_mode <-- apic_timer_interrupt
> stress-25757 [002] 2742.717508: function: __context_tracking_exit <-- enter_from_user_mode
> stress-25757 [002] 2742.717508: bprint: vtime_delta: diff=0 (now=4297409970 vtime_snap=4297409970)
> stress-25757 [002] 2742.717509: function: smp_apic_timer_interrupt <-- apic_timer_interrupt
> stress-25757 [002] 2742.717509: function: irq_enter <-- smp_apic_timer_interrupt
> stress-25757 [002] 2742.717510: hrtimer_expire_entry: hrtimer=0xffffc900039fbe58 function=hrtimer_wakeup now=2742674000776
> stress-25757 [002] 2742.717514: function: irq_exit <-- smp_apic_timer_interrupt
> cyclictest-25760 [002] 2742.717518: function: vtime_account_system <-- vtime_common_task_switch
> cyclictest-25760 [002] 2742.717518: bprint: vtime_delta: diff=1000000 (now=4297409971 vtime_snap=4297409970)
> cyclictest-25760 [002] 2742.717519: function: __vtime_account_system <-- vtime_account_system
> cyclictest-25760 [002] 2742.717519: bprint: get_vtime_delta: vtime_snap=4297409970 now=4297409971
> cyclictest-25760 [002] 2742.717520: function: account_system_time <-- __vtime_account_system
> cyclictest-25760 [002] 2742.717520: bprint: account_system_time: cputime=961981
> cyclictest-25760 [002] 2742.717521: function: __context_tracking_enter <-- do_syscall_64
> cyclictest-25760 [002] 2742.717522: function: vtime_user_enter <-- __context_tracking_enter
> cyclictest-25760 [002] 2742.717522: bprint: vtime_delta: diff=0 (now=4297409971 vtime_snap=4297409971)
>
> CPU2 shows 98% system time while the other CPUs (from CPU3 to CPU7)
> show 98% user time (they're all running the same workload).
On further debugging this, I realized that I had overlooked something:
the timer interrupt in this trace is not the tick, but cyclictest's timer
(remember that the test-case consists of pinning cyclictest and a task
hogging the CPU to the same CPU).
I'm running cyclictest with -i 200. If I increase this to -i 1000, then
I seem unable to reproduce the issue (caution: even with -i 200 it
doesn't always happen. But it does usually happen after I restart the
test-case a few times. However, I've never been able to reproduce
with -i 1000).
Now, if it's really cyclictest that's causing the timer interrupts to
get aligned, I guess this might not have a solution? (note: I haven't
been able to reproduce this on bare-metal).
next prev parent reply other threads:[~2017-04-04 18:03 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-23 20:55 [BUG nohz]: wrong user and system time accounting Luiz Capitulino
2017-03-24 0:56 ` Rik van Riel
2017-03-24 1:05 ` Luiz Capitulino
2017-03-24 1:08 ` Rik van Riel
2017-03-24 1:39 ` Luiz Capitulino
2017-03-27 5:33 ` lkml
2017-03-24 1:52 ` Wanpeng Li
2017-03-24 3:56 ` Luiz Capitulino
2017-03-27 1:56 ` Wanpeng Li
2017-03-27 17:35 ` Rik van Riel
2017-03-28 7:19 ` Wanpeng Li
[not found] ` <20170328132406.7d23579c@redhat.com>
[not found] ` <20170328161454.4a5d9e8b@redhat.com>
2017-03-28 21:01 ` Rik van Riel
2017-03-28 21:26 ` Luiz Capitulino
2017-03-29 9:56 ` Wanpeng Li
2017-03-29 12:56 ` Frederic Weisbecker
2017-03-28 21:24 ` Rik van Riel
2017-03-28 21:30 ` Luiz Capitulino
[not found] ` <20170329131656.1d6cb743@redhat.com>
2017-03-29 20:08 ` Rik van Riel
2017-03-29 22:54 ` Frederic Weisbecker
2017-03-30 12:57 ` Rik van Riel
2017-03-30 1:58 ` Wanpeng Li
2017-03-30 12:40 ` Frederic Weisbecker
2017-03-30 13:19 ` Mike Galbraith
2017-03-30 4:27 ` Mike Galbraith
2017-03-30 6:47 ` Wanpeng Li
2017-03-30 11:52 ` Wanpeng Li
2017-03-30 12:33 ` Mike Galbraith
2017-03-30 13:38 ` Frederic Weisbecker
2017-03-30 13:59 ` Wanpeng Li
2017-03-30 14:18 ` Frederic Weisbecker
2017-03-30 21:25 ` Luiz Capitulino
2017-03-31 20:09 ` Luiz Capitulino
2017-03-31 23:24 ` Frederic Weisbecker
2017-04-01 3:11 ` Luiz Capitulino
2017-04-03 15:23 ` Frederic Weisbecker
2017-04-03 19:06 ` Luiz Capitulino
2017-04-04 17:36 ` Luiz Capitulino [this message]
2017-04-05 14:26 ` Rik van Riel
2017-04-11 11:03 ` Wanpeng Li
2017-04-11 11:36 ` Peter Zijlstra
2017-04-11 11:43 ` Wanpeng Li
2017-04-11 14:22 ` Thomas Gleixner
2017-04-12 13:18 ` Frederic Weisbecker
2017-04-12 14:57 ` Thomas Gleixner
2017-04-12 15:14 ` Frederic Weisbecker
2017-04-13 4:31 ` Wanpeng Li
2017-04-13 13:32 ` Frederic Weisbecker
2017-05-02 10:01 ` Wanpeng Li
2017-05-15 8:17 ` Wanpeng Li
2017-06-29 17:22 ` Frederic Weisbecker
2017-03-30 12:51 ` Frederic Weisbecker
2017-03-30 13:02 ` Rik van Riel
2017-03-30 13:35 ` Mike Galbraith
2017-04-03 14:40 ` Frederic Weisbecker
2017-04-04 7:32 ` Mike Galbraith
2017-03-30 13:44 ` Frederic Weisbecker
[not found] ` <20170329221700.GB23895@lerouge>
2017-03-29 22:46 ` Wanpeng Li
2017-03-30 2:14 ` Luiz Capitulino
2017-03-30 12:27 ` Wanpeng Li
2017-03-27 18:38 ` Luiz Capitulino
2017-03-28 5:28 ` Wanpeng Li
2017-03-28 13:44 ` Luiz Capitulino
2017-03-29 13:04 ` Frederic Weisbecker
2017-03-29 13:14 ` Rik van Riel
2017-03-29 13:23 ` Luiz Capitulino
2017-03-29 21:12 ` Frederic Weisbecker
2017-03-30 1:48 ` Luiz Capitulino
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170404133638.178b29f7@redhat.com \
--to=lcapitulino@redhat.com \
--cc=efault@gmx.de \
--cc=fweisbec@gmail.com \
--cc=kernellwp@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=riel@redhat.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).