Re: [RFC][PATCH 00/32] Nohz cpusets v2 (adaptive tickless kernel)

[Gilad Ben-Yossef <gilad@benyossef.com>]

On Wed, Mar 21, 2012 at 3:58 PM, Frederic Weisbecker <fweisbec@gmail.com> wrote:
> Hi all,
>
> A summary of what this is about can be found here:
>  https://lkml.org/lkml/2011/8/15/245
>
> There are still a lot of things to handle. Especially about
> what is done by scheduler_tick() but we also need to:
>
> - completely handle cputime accounting (need to find every "reader"
> of cputime and flush cputimes for all of them).
> -handle  perf
> - handle irqtime finegrained accounting
> - handle ilb load balancing
> - etc...
>

I gave the new version a spin (x86 8 way VM) and it looks cool.

I did get the following warning once, but couldn't recreate it:

[   31.812741] ------------[ cut here ]------------
[   31.812741] WARNING: at
/home/giladb/Workspace/linux/kernel/time/tick-sched.c:706
tick_nohz_account_ticks+0x7c/0x90()
[   31.812741] Hardware name: Bochs
[   31.812741] Modules linked in:
[   31.812741] Pid: 1006, comm: sh Not tainted 3.3.0-rc7+ #167
[   31.812741] Call Trace:
[   31.812741]  [<c102a3ad>] warn_slowpath_common+0x6d/0xa0
[   31.812741]  [<c106be0c>] ? tick_nohz_account_ticks+0x7c/0x90
[   31.812741]  [<c106be0c>] ? tick_nohz_account_ticks+0x7c/0x90
[   31.812741]  [<c102a3fd>] warn_slowpath_null+0x1d/0x20
[   31.812741]  [<c106be0c>] tick_nohz_account_ticks+0x7c/0x90
[   31.812741]  [<c106be5f>] tick_nohz_flush_current_times+0x3f/0x80
[   31.812741]  [<c106bf8d>] tick_nohz_restart_adaptive+0xd/0x30
[   31.812741]  [<c106c02e>] tick_nohz_check_adaptive+0x3e/0x50
[   31.812741]  [<c1018180>] smp_cpuset_update_nohz_interrupt+0x20/0x30
[   31.812741]  [<c1639c6a>] cpuset_update_nohz_interrupt+0x2a/0x30
[   31.812741]  [<c16395fd>] ? _raw_spin_unlock_irq+0xd/0x30
[   31.812741]  [<c10575c6>] finish_task_switch+0x46/0xa0
[   31.812741]  [<c1638558>] __schedule+0x398/0x910
[   31.812741]  [<c10ef2f1>] ? deactivate_slab+0x611/0x730
[   31.812741]  [<c1120777>] ? __find_get_block+0x97/0x1a0
[   31.812741]  [<c1221214>] ? cpumask_next_and+0x24/0xa0
[   31.812741]  [<c10558cb>] ? get_parent_ip+0xb/0x40
[   31.812741]  [<c1638b50>] schedule+0x30/0x50
[   31.812741]  [<c16379b5>] schedule_hrtimeout_range_clock+0xf5/0x110
[   31.812741]  [<c10558cb>] ? get_parent_ip+0xb/0x40
[   31.812741]  [<c10586db>] ? sub_preempt_count+0x7b/0xb0
[   31.812741]  [<c1639633>] ? _raw_spin_unlock_irqrestore+0x13/0x40
[   31.812741]  [<c1054140>] ? __wake_up+0x40/0x50
[   31.812741]  [<c1294d1f>] ? put_ldisc+0x3f/0xa0
[   31.812741]  [<c16379e2>] schedule_hrtimeout_range+0x12/0x20
[   31.812741]  [<c1107969>] poll_schedule_timeout+0x39/0x60
[   31.812741]  [<c1108020>] do_sys_poll+0x400/0x490
[   31.812741]  [<c1054d15>] ? cpuacct_charge+0x65/0x70
[   31.812741]  [<c1107a20>] ? poll_freewait+0x70/0x70
[   31.812741]  [<c1107af0>] ? __pollwait+0xd0/0xd0
[   31.812741]  [<c1107af0>] ? __pollwait+0xd0/0xd0
[   31.812741]  [<c10094a3>] ? native_sched_clock+0x33/0xe0
[   31.812741]  [<c105a0e2>] ? sched_clock_local+0xb2/0x190
[   31.812741]  [<c1054d15>] ? cpuacct_charge+0x65/0x70
[   31.812741]  [<c105b376>] ? update_curr+0x1a6/0x2a0
[   31.812741]  [<c105a2f9>] ? sched_clock_cpu+0x139/0x190
[   31.812741]  [<c105a0e2>] ? sched_clock_local+0xb2/0x190
[   31.812741]  [<c104dd43>] ? hrtimer_forward+0x163/0x1b0
[   31.812741]  [<c10644e2>] ? ktime_get+0x62/0x100
[   31.812741]  [<c1018b56>] ? lapic_next_event+0x16/0x20
[   31.812741]  [<c1069df2>] ? clockevents_program_event+0xc2/0x170
[   31.812741]  [<c106b514>] ? tick_program_event+0x24/0x30
[   31.812741]  [<c104cd1d>] ? hrtimer_interrupt+0x1ad/0x2e0
[   31.812741]  [<c1095128>] ? rcu_pending+0x58/0x70
[   31.812741]  [<c1030a3d>] ? irq_exit+0x6d/0x80
[   31.812741]  [<c1019363>] ? smp_apic_timer_interrupt+0x53/0x90
[   31.812741]  [<c11e0128>] ? avc_has_perm_noaudit+0xc8/0x360
[   31.812741]  [<c163a3b6>] ? apic_timer_interrupt+0x2a/0x30
[   31.812741]  [<c128f31e>] ? tty_ioctl+0x47e/0xa30
[   31.812741]  [<c11e0d66>] ? inode_has_perm+0x36/0x50
[   31.812741]  [<c11e13e8>] ? file_has_perm+0xa8/0xb0
[   31.812741]  [<c128eea0>] ? tty_check_change+0xe0/0xe0
[   31.812741]  [<c1106763>] ? do_vfs_ioctl+0x83/0x570
[   31.812741]  [<c11e4e46>] ? selinux_file_ioctl+0x56/0x110
[   31.812741]  [<c1108224>] sys_poll+0x54/0xb0
[   31.812741]  [<c1639b29>] syscall_call+0x7/0xb
[   31.812741] ---[ end trace 1d7d659b4aead681 ]---

With the two patches I'll attach to the next replies to this message,
I've been able to get a task running
on an isolated CPU with 0 timer interrupts.

In my case, I also had to disable the clocksource watchdog, but only
because TSC is not stable on my VM.
This is really not a nohz/cpuset problem.

There is one source of interference to cpu isolation this causes,
which is the cputime flush IPI. Every time you
run a command in the shell you get 3 - 4 IPIs sent to the nohz cpuset
to flush the cputimes so that thread group
times get computed correctly. That's not very nice :-)

I've tried disabling the IPI send, just to see how it goes and as far
as I've been able to tell you get bare metal like
environment for a 100% cpu bound code with no interrupts. Of course.
ps/top then show 0% cpu utilization for
that task since without the IPI the times it spends on the CPU is not
registered... that is a small price to pay
in my eyes for bare metal performance on Linux, but what do I know? :-)

Overall, way cool. Please keep it up !

GIlad

-- 
Gilad Ben-Yossef
Chief Coffee Drinker
gilad@benyossef.com
Israel Cell: +972-52-8260388
US Cell: +1-973-8260388
http://benyossef.com

"If you take a class in large-scale robotics, can you end up in a
situation where the homework eats your dog?"
 -- Jean-Baptiste Queru