Scheduler patches: 6x performance increase when system is under heavy load

Scheduler patches: 6x performance increase when system is under heavy load

[Alexandre-Xavier Labonté-Lamoureux]

Hi,

There's a research paper[1] called "The Linux Scheduler: a Decade of
Wasted Cores". It shows how the Linux Kernel scheduler is broken and
they provided fixes[2] for some of the know issues that it has. The
changes are illustrated in a presentation[3] that was shown during a
talk.

I patched my kernel with these patches using a simple tool[4]. I
compiled the Linux kernel on it and I was flabbergasted to see an
estimated six times performance increase. It was blazing fast and I
couldn't believe it.

Here are my results (using "time make -j32" on my VM that has 4 cores):

Kernel 4.8.14
  real 26m56.151s
  user 79m52.472s
  sys 7m42.964s

Same kernel, but patched:
  real 4m25.238s
  user 13m52.932s
  sys 1m25.820s

I hope you guys will look into this. If these changes were introduced
into the kernel, it would be great and benefit to a lot of people.
There may be some regression. People have said that it improved the
throughput of the kernel, but on some application such as games, there
are some performance spikes due to overscheduling. This already
happens in the vanilla kernel, but it's worsened a bit. Power
management will also be a thing to look into.

I hope the scheduler engineers will fix the kernel. The research paper
has been out for a while and nothing has been done yet. Personally, I
think six times is huge as someone who compiles huge projects often
and the regressions if any are not apparent to me. Other people should
try it too to test if it's good.

Regards,
Alexandre-Xavier Labonté-Lamoureux

[1] https://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf
[2] https://github.com/Freeaqingme/wastedcores/tree/linux-4.5/patches
[3] http://www.i3s.unice.fr/~jplozi/wastedcores/files/extended_talk.pdf
[4] https://github.com/Turbine1991/build_ubuntu_kernel_wastedcores

Re: Scheduler patches: 6x performance increase when system is under heavy load

[Peter Zijlstra]

On Sun, Dec 11, 2016 at 04:41:51PM -0500, Alexandre-Xavier Labonté-Lamoureux wrote:
> 
> Here are my results (using "time make -j32" on my VM that has 4 cores):
> 
> Kernel 4.8.14
>   real 26m56.151s
>   user 79m52.472s
>   sys 7m42.964s
> 
> Same kernel, but patched:
>   real 4m25.238s
>   user 13m52.932s
>   sys 1m25.820s
> 
> I hope you guys will look into this. 

Which of the 4 patches does this? 

Also, what hypervisor are you using and what does the output of booting
with "sched_debug" look like?

Lastly, can you reproduce on real hardware?

Re: Scheduler patches: 6x performance increase when system is under heavy load

[Alexandre-Xavier Labonté-Lamoureux]

> Which of the 4 patches does this?

I used all the 4 patches at the same time. Each patch fixes a
different bug. Would you like me to try each of them individually?
Were you already aware of each of these bugs?

> Also, what hypervisor are you using and what does the output of booting
> with "sched_debug" look like?

I was running the distro in VirualBox on Fedora. Here's the info from
/proc/sched_debug:
https://justpaste.it/11dhb
dmesg: https://justpaste.it/11dhr

> Lastly, can you reproduce on real hardware?

No. On real hardware, I tested in Ubuntu on an i7-4790 3.60GHz CPU
without disabling HT and I saw no difference between CFS, the patched
kernel and MuQSS. If I get to know a reason why one would be better
than the other, I'd take the time to test it on more hardware. I'm
curious how I got such a performance improvement in my VM.

On Tue, Dec 13, 2016 at 8:40 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Sun, Dec 11, 2016 at 04:41:51PM -0500, Alexandre-Xavier Labonté-Lamoureux wrote:
>>
>> Here are my results (using "time make -j32" on my VM that has 4 cores):
>>
>> Kernel 4.8.14
>>   real 26m56.151s
>>   user 79m52.472s
>>   sys 7m42.964s
>>
>> Same kernel, but patched:
>>   real 4m25.238s
>>   user 13m52.932s
>>   sys 1m25.820s
>>
>> I hope you guys will look into this.
>
> Which of the 4 patches does this?
>
> Also, what hypervisor are you using and what does the output of booting
> with "sched_debug" look like?
>
> Lastly, can you reproduce on real hardware?

Re: Scheduler patches: 6x performance increase when system is under heavy load

[Peter Zijlstra]

Sorry for the delay, got side-tracked for a bit..

On Wed, Dec 14, 2016 at 12:15:25AM -0500, Alexandre-Xavier Labonté-Lamoureux wrote:
> > Which of the 4 patches does this?
> 
> I used all the 4 patches at the same time. Each patch fixes a
> different bug. Would you like me to try each of them individually?

Yes.

> Were you already aware of each of these bugs?

I had seen the paper and the patches. One of the issues has been fixed,
one is a non-issue and we had ideas about at least one other and I
cannot quite remember what the 4th was.

> > Also, what hypervisor are you using and what does the output of booting
> > with "sched_debug" look like?
> 
> I was running the distro in VirualBox on Fedora. Here's the info from
> /proc/sched_debug:
> https://justpaste.it/11dhb
> dmesg: https://justpaste.it/11dhr
> 

Ah, I meant dmesg with the "sched_debug" boot cmdline option (and
probably: "debug ignore_loglevel" added too) of the unmodified kernel.

> > Lastly, can you reproduce on real hardware?
> 
> No. On real hardware, I tested in Ubuntu on an i7-4790 3.60GHz CPU
> without disabling HT and I saw no difference between CFS, the patched
> kernel and MuQSS. If I get to know a reason why one would be better
> than the other, I'd take the time to test it on more hardware. I'm
> curious how I got such a performance improvement in my VM.

OK, I'll also have to look at VirtualBox sources to see if they use
paravirt locking bits, not having support for that could make a
significant difference I suppose.