[PATCH] Make credit2 the default scheduler

[PATCH] Make credit2 the default scheduler

[George Dunlap]

Credit2 was declared "supported" in 4.8, and as of 4.10 had two other
critical features implemented (soft affinity / NUMA and caps).

Why change the default?

The code is better: more predictable, less jitter, easier to determine
how modifications will affect overall behavior, easier in the future
to make load-balancing behavior more subtle (e.g., taking into account
the cost of powering up extra cores, &c).

Overall performance compared to Credit1 is somewhat of a mixed bag.
Unfortunately most of what I have are tests using XenServer's internal
perf testing system, so I can't share the raw data (via links anyway).

Here is a summary of data from an internal e-mail Dario sent in the
past:

* DVDbench: On underloaded systems, credit2 outperformed credit1 by
about 4%.  On overloaded systems, credit2 underperformed by about 3%.

* On a range of tests (unixbench, lmbench, &c), credit and credit2
perform within 5% of each other (up and down).

* Credit2 fairly consistently beats credit for TCP-style workloads.

* Credit2 is sometimes equal to, sometimes 5-15% worse than, credit for
synthetic CPU workloads (e.g., Dhrystone).

* On LoginVSI, credit2 fairly consistently outperforms credit by about 10%.

Credit2, like credit, has a number of workloads / setups for which
performance could be improved.  Personally I think networking and
partially-loaded systems is going to be more representative of what
Xen is actually used for; so I think credit2 is on the whole the
better scheduler to use by default.  And in any case, making those
improvements on credit2 will be easier than on credit.

Signed-off-by: George Dunlap <george.dunlap@citrix.com>
---
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Jan Beulich <jbeulich@suse.com>
CC: Tim Deegan <tim@xen.org>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Konrad Wilk <konrad.wilk@oracle.com>
CC: Dario Faggioli <dfaggioli@suse.com>
---
 xen/common/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index 1a6d6281c1..3a1c585d25 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -215,7 +215,7 @@ config SCHED_NULL
 
 choice
 	prompt "Default Scheduler?"
-	default SCHED_CREDIT_DEFAULT
+	default SCHED_CREDIT2_DEFAULT
 
 	config SCHED_CREDIT_DEFAULT
 		bool "Credit Scheduler" if SCHED_CREDIT
@@ -236,7 +236,7 @@ config SCHED_DEFAULT
 	default "rtds" if SCHED_RTDS_DEFAULT
 	default "arinc653" if SCHED_ARINC653_DEFAULT
 	default "null" if SCHED_NULL_DEFAULT
-	default "credit"
+	default "credit2"
 
 endmenu
 
-- 
2.18.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1493 bytes --]

On Thu, 2018-09-13 at 17:38 +0100, George Dunlap wrote:
> Credit2 was declared "supported" in 4.8, and as of 4.10 had two other
> critical features implemented (soft affinity / NUMA and caps).
> 
> [..]
>
> Credit2, like credit, has a number of workloads / setups for which
> performance could be improved.  Personally I think networking and
> partially-loaded systems is going to be more representative of what
> Xen is actually used for; so I think credit2 is on the whole the
> better scheduler to use by default.  And in any case, making those
> improvements on credit2 will be easier than on credit.
> 
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>
After all the effort we've spent on this, I'm really, really happy to
see this (trying to) happen. Thanks for sending the patch. :-)

I fully agree with and second George's reasoning, and feel 100% like
providing my:

Acked-by: Dario Faggioli <dfaggioli@suse.com>

Of course, although we thing the scheduler is ready, the more
testing/profiling/benchmarking/... we get, the better.

So, everyone, don't be shy at throwing any kind of things against it,
and report what happens, so we can fix the problem (or revert to
Credit!) for 4.12

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[Jan Beulich]

>>> On 13.09.18 at 18:51, <dfaggioli@suse.com> wrote:
> On Thu, 2018-09-13 at 17:38 +0100, George Dunlap wrote:
>> Credit2 was declared "supported" in 4.8, and as of 4.10 had two other
>> critical features implemented (soft affinity / NUMA and caps).
>> 
>> [..]
>>
>> Credit2, like credit, has a number of workloads / setups for which
>> performance could be improved.  Personally I think networking and
>> partially-loaded systems is going to be more representative of what
>> Xen is actually used for; so I think credit2 is on the whole the
>> better scheduler to use by default.  And in any case, making those
>> improvements on credit2 will be easier than on credit.
>> 
>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>
> After all the effort we've spent on this, I'm really, really happy to
> see this (trying to) happen. Thanks for sending the patch. :-)
> 
> I fully agree with and second George's reasoning, and feel 100% like
> providing my:
> 
> Acked-by: Dario Faggioli <dfaggioli@suse.com>

And that's despite "sched=credit2 crashes system when using
cpupools"? While I agree that we shouldn't delay the switch for
much longer, in particular with there already being a fix available
from you I think that one should go in before the one here.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[Wei Liu]

On Fri, Sep 14, 2018 at 02:45:35AM -0600, Jan Beulich wrote:
> >>> On 13.09.18 at 18:51, <dfaggioli@suse.com> wrote:
> > On Thu, 2018-09-13 at 17:38 +0100, George Dunlap wrote:
> >> Credit2 was declared "supported" in 4.8, and as of 4.10 had two other
> >> critical features implemented (soft affinity / NUMA and caps).
> >> 
> >> [..]
> >>
> >> Credit2, like credit, has a number of workloads / setups for which
> >> performance could be improved.  Personally I think networking and
> >> partially-loaded systems is going to be more representative of what
> >> Xen is actually used for; so I think credit2 is on the whole the
> >> better scheduler to use by default.  And in any case, making those
> >> improvements on credit2 will be easier than on credit.
> >> 
> >> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
> >>
> > After all the effort we've spent on this, I'm really, really happy to
> > see this (trying to) happen. Thanks for sending the patch. :-)
> > 
> > I fully agree with and second George's reasoning, and feel 100% like
> > providing my:
> > 
> > Acked-by: Dario Faggioli <dfaggioli@suse.com>
> 
> And that's despite "sched=credit2 crashes system when using
> cpupools"? While I agree that we shouldn't delay the switch for

Urgh, this is a really nasty bug.

> much longer, in particular with there already being a fix available
> from you I think that one should go in before the one here.

+1.

Wei.

> 
> Jan
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1603 bytes --]

On Fri, 2018-09-14 at 09:45 +0100, Jan Beulich wrote:
> > > > On 13.09.18 at 18:51, <dfaggioli@suse.com> wrote:
> > I fully agree with and second George's reasoning, and feel 100%
> > like
> > providing my:
> > 
> > Acked-by: Dario Faggioli <dfaggioli@suse.com>
> 
> And that's despite "sched=credit2 crashes system when using
> cpupools"? 
>
Well, considering that, as you say, there's a fix on the mailing list
already, yes.

And I'm guessing George must feel kind of the same, as he sent this
patch, before reviewing/committing the other.

In fact, the sooner we switch, the earlier OSSTest will start use
Credit2 for everything, and show us whether or not there are other
issues. And, speaking about that specific bug, OSSTest does not [sic!]
test cpupools, so it's not even like it'll block pushes.

That being said...

> While I agree that we shouldn't delay the switch for
> much longer, in particular with there already being a fix available
> from you I think that one should go in before the one here.
> 
... I'm fine if maintainers/committers feels like waiting for the other
patch to go in. The order in which they're committed, is something I
have no control on. I don't think it's strictly necessary for this
patch and the fix to hit the repo in a particular order. But if others
do, no problem.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[Steven Haigh]

[-- Attachment #1.1: Type: text/plain, Size: 2130 bytes --]

On Friday, 14 September 2018 6:45:35 PM AEST Jan Beulich wrote:
> >>> On 13.09.18 at 18:51, <dfaggioli@suse.com> wrote:
> > On Thu, 2018-09-13 at 17:38 +0100, George Dunlap wrote:
> >> Credit2 was declared "supported" in 4.8, and as of 4.10 had two other
> >> critical features implemented (soft affinity / NUMA and caps).
> >> 
> >> [..]
> >> 
> >> Credit2, like credit, has a number of workloads / setups for which
> >> performance could be improved.  Personally I think networking and
> >> partially-loaded systems is going to be more representative of what
> >> Xen is actually used for; so I think credit2 is on the whole the
> >> better scheduler to use by default.  And in any case, making those
> >> improvements on credit2 will be easier than on credit.
> >> 
> >> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
> > 
> > After all the effort we've spent on this, I'm really, really happy to
> > see this (trying to) happen. Thanks for sending the patch. :-)
> > 
> > I fully agree with and second George's reasoning, and feel 100% like
> > providing my:
> > 
> > Acked-by: Dario Faggioli <dfaggioli@suse.com>
> 
> And that's despite "sched=credit2 crashes system when using
> cpupools"? While I agree that we shouldn't delay the switch for
> much longer, in particular with there already being a fix available
> from you I think that one should go in before the one here.

Even though my opinion probably isn't very heavy on this matter, I've used 
credit2 exclusively for a considerable time. If you're talking the issue I 
think you're talking about, then I discovered it when doing stuff that most 
people probably wouldn't bother with - evidenced that I hadn't done it before 
either.

I take peoples word on the performance +/- of a few percent here and there - 
so if its easier to maintain and better code, then yeah - it makes sense to 
move on with it. I certainly haven't found any normal use cases that would 
lead me to object to this.

-- 
Steven Haigh

📧 netwiz@crc.id.au       💻 https://www.crc.id.au
📞 +61 (3) 9001 6090    📱 0412 935 897

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[George Dunlap]

On 09/14/2018 10:02 AM, Wei Liu wrote:
> On Fri, Sep 14, 2018 at 02:45:35AM -0600, Jan Beulich wrote:
>>>>> On 13.09.18 at 18:51, <dfaggioli@suse.com> wrote:
>>> On Thu, 2018-09-13 at 17:38 +0100, George Dunlap wrote:
>>>> Credit2 was declared "supported" in 4.8, and as of 4.10 had two other
>>>> critical features implemented (soft affinity / NUMA and caps).
>>>>
>>>> [..]
>>>>
>>>> Credit2, like credit, has a number of workloads / setups for which
>>>> performance could be improved.  Personally I think networking and
>>>> partially-loaded systems is going to be more representative of what
>>>> Xen is actually used for; so I think credit2 is on the whole the
>>>> better scheduler to use by default.  And in any case, making those
>>>> improvements on credit2 will be easier than on credit.
>>>>
>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>>
>>> After all the effort we've spent on this, I'm really, really happy to
>>> see this (trying to) happen. Thanks for sending the patch. :-)
>>>
>>> I fully agree with and second George's reasoning, and feel 100% like
>>> providing my:
>>>
>>> Acked-by: Dario Faggioli <dfaggioli@suse.com>
>>
>> And that's despite "sched=credit2 crashes system when using
>> cpupools"? While I agree that we shouldn't delay the switch for
> 
> Urgh, this is a really nasty bug.
> 
>> much longer, in particular with there already being a fix available
>> from you I think that one should go in before the one here.
> 
> +1.

The other fix is in.  I'll wait a few days and push this change, if
there are no objections.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1861 bytes --]

On Sat, 2018-09-15 at 00:21 +1000, Steven Haigh wrote:
> On Friday, 14 September 2018 6:45:35 PM AEST Jan Beulich wrote:
> > > > > 
> > And that's despite "sched=credit2 crashes system when using
> > cpupools"? While I agree that we shouldn't delay the switch for
> > much longer, in particular with there already being a fix available
> > from you I think that one should go in before the one here.
> 
> Even though my opinion probably isn't very heavy on this matter, I've
> used 
> credit2 exclusively for a considerable time. 
>
Well, this is really interesting and useful to know. Can I ask what
your typical workload is (if any), and how are things going?

> If you're talking the issue I 
> think you're talking about, then I discovered it when doing stuff
> that most 
> people probably wouldn't bother with - evidenced that I hadn't done
> it before 
> either.
> 
Actually, we do expect the default scheduler not to crash if one
creates a cpupool.

Not that there hasn't been similar bug in Credit1, while it was the
default (check `git log' :-/). But what this all means is that we need
to do better at testing these things, e.g., finally adding cpupool and
CPU online/offline testing to OSSTest.

Anyway, the bugfix is in now. :-)

> I take peoples word on the performance +/- of a few percent here and
> there - 
> so if its easier to maintain and better code, then yeah - it makes
> sense to 
> move on with it. I certainly haven't found any normal use cases that
> would 
> lead me to object to this.
> 
Great, and thanks again for the feedback!

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[George Dunlap]

On 09/14/2018 01:29 PM, Dario Faggioli wrote:
> On Fri, 2018-09-14 at 09:45 +0100, Jan Beulich wrote:
>>>>> On 13.09.18 at 18:51, <dfaggioli@suse.com> wrote:
>>> I fully agree with and second George's reasoning, and feel 100%
>>> like
>>> providing my:
>>>
>>> Acked-by: Dario Faggioli <dfaggioli@suse.com>
>>
>> And that's despite "sched=credit2 crashes system when using
>> cpupools"? 
>>
> Well, considering that, as you say, there's a fix on the mailing list
> already, yes.
> 
> And I'm guessing George must feel kind of the same, as he sent this
> patch, before reviewing/committing the other.

Well, also because our team had just done our retrospective and noted
the state of credit2 (and I think Ian brought it up on IRC).  It was
either "do it now" or "put it on a list to be done soon"; and my "list
of things to be done soon" is already too long. :-)

> In fact, the sooner we switch, the earlier OSSTest will start use
> Credit2 for everything, and show us whether or not there are other
> issues. And, speaking about that specific bug, OSSTest does not [sic!]
> test cpupools, so it's not even like it'll block pushes.
> 
> That being said...
> 
>> While I agree that we shouldn't delay the switch for
>> much longer, in particular with there already being a fix available
>> from you I think that one should go in before the one here.
>>
> ... I'm fine if maintainers/committers feels like waiting for the other
> patch to go in. The order in which they're committed, is something I
> have no control on. I don't think it's strictly necessary for this
> patch and the fix to hit the repo in a particular order. But if others
> do, no problem.

I feel the same way: I don't see any reason to delay, but I don't think
it's worth pressing the point when it's just a few days.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[Steven Haigh]

On 2018-09-19 19:38, Dario Faggioli wrote:
> On Sat, 2018-09-15 at 00:21 +1000, Steven Haigh wrote:
>> On Friday, 14 September 2018 6:45:35 PM AEST Jan Beulich wrote:
>> > > > >
>> > And that's despite "sched=credit2 crashes system when using
>> > cpupools"? While I agree that we shouldn't delay the switch for
>> > much longer, in particular with there already being a fix available
>> > from you I think that one should go in before the one here.
>> 
>> Even though my opinion probably isn't very heavy on this matter, I've
>> used
>> credit2 exclusively for a considerable time.
>> 
> Well, this is really interesting and useful to know. Can I ask what
> your typical workload is (if any), and how are things going?

I don't really have a 'typical' workload. There's mail servers, web 
servers, DNS, shell boxes, all kinds of varied stuff. I haven't had any 
noticeable performance issues that I could look at and say "X is 
different".

>> If you're talking the issue I
>> think you're talking about, then I discovered it when doing stuff
>> that most
>> people probably wouldn't bother with - evidenced that I hadn't done
>> it before
>> either.
>> 
> Actually, we do expect the default scheduler not to crash if one
> creates a cpupool.
> 
> Not that there hasn't been similar bug in Credit1, while it was the
> default (check `git log' :-/). But what this all means is that we need
> to do better at testing these things, e.g., finally adding cpupool and
> CPU online/offline testing to OSSTest.
> 
> Anyway, the bugfix is in now. :-)

Agreed. I haven't had a chance to test that patch of yet - as I need to 
reconfigure the IPMI BMC to recover from a hard crash. Its tucked away 
in a rack out of sight and mind - so this isn't quite straight forward - 
but possible if I know I'm going to try and crash it :)

>> I take peoples word on the performance +/- of a few percent here and
>> there -
>> so if its easier to maintain and better code, then yeah - it makes
>> sense to
>> move on with it. I certainly haven't found any normal use cases that
>> would
>> lead me to object to this.
>> 
> Great, and thanks again for the feedback!
> 
> Regards,
> Dario

-- 
Steven Haigh

? netwiz@crc.id.au     ? http://www.crc.id.au
? +61 (3) 9001 6090    ? 0412 935 897

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] Make credit2 the default scheduler

[George Dunlap]

On Thu, Sep 13, 2018 at 5:39 PM George Dunlap <george.dunlap@citrix.com> wrote:
>
> Credit2 was declared "supported" in 4.8, and as of 4.10 had two other
> critical features implemented (soft affinity / NUMA and caps).
>
> Why change the default?
>
> The code is better: more predictable, less jitter, easier to determine
> how modifications will affect overall behavior, easier in the future
> to make load-balancing behavior more subtle (e.g., taking into account
> the cost of powering up extra cores, &c).
>
> Overall performance compared to Credit1 is somewhat of a mixed bag.
> Unfortunately most of what I have are tests using XenServer's internal
> perf testing system, so I can't share the raw data (via links anyway).
>
> Here is a summary of data from an internal e-mail Dario sent in the
> past:
>
> * DVDbench: On underloaded systems, credit2 outperformed credit1 by
> about 4%.  On overloaded systems, credit2 underperformed by about 3%.
>
> * On a range of tests (unixbench, lmbench, &c), credit and credit2
> perform within 5% of each other (up and down).
>
> * Credit2 fairly consistently beats credit for TCP-style workloads.
>
> * Credit2 is sometimes equal to, sometimes 5-15% worse than, credit for
> synthetic CPU workloads (e.g., Dhrystone).
>
> * On LoginVSI, credit2 fairly consistently outperforms credit by about 10%.
>
> Credit2, like credit, has a number of workloads / setups for which
> performance could be improved.  Personally I think networking and
> partially-loaded systems is going to be more representative of what
> Xen is actually used for; so I think credit2 is on the whole the
> better scheduler to use by default.  And in any case, making those
> improvements on credit2 will be easier than on credit.
>
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>

The fix for the migration issue has passed the push gate; I plan on
re-applying this change again (un-reverting the revert) Monday unless
I hear any objections.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel