[PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[George Dunlap]

Commit 3b4adba ("tools/libxl: include scheduler parameters in the
output of xl list -l") added scheduling parameters to the set of
information collected by libxl_retrieve_domain_configuration(), in
order to report that information in `xl list -l`.

Unfortunately, libxl_retrieve_domain_configuration() is also called by
the migration / save code, and the results passed to the restore /
receive code.  This meant scheduler parameters were inadvertently
added to the migration stream, without proper consideration for how to
handle corner cases.  The result was that if migrating from a host
running one scheduler to a host running a different scheduler, the
migration would fail with an error like the following:

libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3

Luckily there's a fairly straightforward way to set parameters in a
"best-effort" fashion.  libxl provides a single struct containing the
parameters of all schedulers, as well as a parameter specifying which
scheduler.  Parameters not used by a given scheduler are ignored.
Additionally, the struct contains a parameter to specify the
scheduler.  If you specify a specific scheduler,
libxl_domain_sched_params_set() will fail if there's a different
scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
the value of the current scheduler for that domain.

In domcreate_stream_done(), before calling libxl__build_post(), set
the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
scheduler parameters from the previous instantiation on a best-effort
basis.

Signed-off-by: George Dunlap <george.dunlap@citrix.com>
---
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Jan Beulich <jbeulich@suse.com>
CC: Dario Faggioli <dfaggioli@suse.com>
CC: Juergen Gross <jgross@suse.com>
---
 tools/libxl/libxl_create.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index dcfde7787e..caf79d4620 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1237,6 +1237,15 @@ static void domcreate_stream_done(libxl__egc *egc,
         ret = ERROR_INVAL;
         goto out;
     }
+
+    /* 
+     * The scheduler on the sending domain may be different than the
+     * scheduler running here.  Setting the scheduler to UNKNOWN will
+     * cause the code to take to take whatever parameters are
+     * available in that scheduler, while discarding the rest.
+     */
+    info->sched_params.sched = LIBXL_SCHEDULER_UNKNOWN;
+    
     ret = libxl__build_post(gc, domid, info, state, vments, localents);
     if (ret)
         goto out;
-- 
2.19.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[George Dunlap]

On 10/02/2018 04:49 PM, George Dunlap wrote:
> Commit 3b4adba ("tools/libxl: include scheduler parameters in the
> output of xl list -l") added scheduling parameters to the set of
> information collected by libxl_retrieve_domain_configuration(), in
> order to report that information in `xl list -l`.
> 
> Unfortunately, libxl_retrieve_domain_configuration() is also called by
> the migration / save code, and the results passed to the restore /
> receive code.  This meant scheduler parameters were inadvertently
> added to the migration stream, without proper consideration for how to
> handle corner cases.  The result was that if migrating from a host
> running one scheduler to a host running a different scheduler, the
> migration would fail with an error like the following:
> 
> libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
> libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
> 
> Luckily there's a fairly straightforward way to set parameters in a
> "best-effort" fashion.  libxl provides a single struct containing the
> parameters of all schedulers, as well as a parameter specifying which
> scheduler.  Parameters not used by a given scheduler are ignored.
> Additionally, the struct contains a parameter to specify the
> scheduler.  If you specify a specific scheduler,
> libxl_domain_sched_params_set() will fail if there's a different
> scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
> the value of the current scheduler for that domain.
> 
> In domcreate_stream_done(), before calling libxl__build_post(), set
> the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
> scheduler parameters from the previous instantiation on a best-effort
> basis.
> 
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>

Sorry, forgot to mention this should be considered 'RFC' until I can
arrange to test it.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[George Dunlap]

On 10/02/2018 04:49 PM, George Dunlap wrote:
> Commit 3b4adba ("tools/libxl: include scheduler parameters in the
> output of xl list -l") added scheduling parameters to the set of
> information collected by libxl_retrieve_domain_configuration(), in
> order to report that information in `xl list -l`.
> 
> Unfortunately, libxl_retrieve_domain_configuration() is also called by
> the migration / save code, and the results passed to the restore /
> receive code.  This meant scheduler parameters were inadvertently
> added to the migration stream, without proper consideration for how to
> handle corner cases.  The result was that if migrating from a host
> running one scheduler to a host running a different scheduler, the
> migration would fail with an error like the following:
> 
> libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
> libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
> 
> Luckily there's a fairly straightforward way to set parameters in a
> "best-effort" fashion.  libxl provides a single struct containing the
> parameters of all schedulers, as well as a parameter specifying which
> scheduler.  Parameters not used by a given scheduler are ignored.
> Additionally, the struct contains a parameter to specify the
> scheduler.  If you specify a specific scheduler,
> libxl_domain_sched_params_set() will fail if there's a different
> scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
> the value of the current scheduler for that domain.
> 
> In domcreate_stream_done(), before calling libxl__build_post(), set
> the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
> scheduler parameters from the previous instantiation on a best-effort
> basis.
> 
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>

I've tested this with save/restore now, and it works fine (including
changing the weight of the VM from default and having the updated weight
show up in the other scheduler on restore).

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[Juergen Gross]

On 02/10/2018 17:49, George Dunlap wrote:
> Commit 3b4adba ("tools/libxl: include scheduler parameters in the
> output of xl list -l") added scheduling parameters to the set of
> information collected by libxl_retrieve_domain_configuration(), in
> order to report that information in `xl list -l`.
> 
> Unfortunately, libxl_retrieve_domain_configuration() is also called by
> the migration / save code, and the results passed to the restore /
> receive code.  This meant scheduler parameters were inadvertently
> added to the migration stream, without proper consideration for how to
> handle corner cases.  The result was that if migrating from a host
> running one scheduler to a host running a different scheduler, the
> migration would fail with an error like the following:
> 
> libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
> libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
> 
> Luckily there's a fairly straightforward way to set parameters in a
> "best-effort" fashion.  libxl provides a single struct containing the
> parameters of all schedulers, as well as a parameter specifying which
> scheduler.  Parameters not used by a given scheduler are ignored.
> Additionally, the struct contains a parameter to specify the
> scheduler.  If you specify a specific scheduler,
> libxl_domain_sched_params_set() will fail if there's a different
> scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
> the value of the current scheduler for that domain.
> 
> In domcreate_stream_done(), before calling libxl__build_post(), set
> the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
> scheduler parameters from the previous instantiation on a best-effort
> basis.
> 
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>

Acked-by: Juergen Gross <jgross@suse.com>


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2416 bytes --]

On Wed, 2018-10-03 at 12:53 +0200, Juergen Gross wrote:
> On 02/10/2018 17:49, George Dunlap wrote:
> > Commit 3b4adba ("tools/libxl: include scheduler parameters in the
> > output of xl list -l") added scheduling parameters to the set of
> > information collected by libxl_retrieve_domain_configuration(), in
> > order to report that information in `xl list -l`.
> > 
> > Unfortunately, libxl_retrieve_domain_configuration() is also called
> > by
> > the migration / save code, and the results passed to the restore /
> > receive code.  This meant scheduler parameters were inadvertently
> > added to the migration stream, without proper consideration for how
> > to
> > handle corner cases.  The result was that if migrating from a host
> > running one scheduler to a host running a different scheduler, the
> > migration would fail with an error like the following:
> > 
> > libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain
> > 1:Getting domain sched credit: Invalid argument
> > libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain
> > 1:cannot (re-)build domain: -3
> > 
> > Luckily there's a fairly straightforward way to set parameters in a
> > "best-effort" fashion.  libxl provides a single struct containing
> > the
> > parameters of all schedulers, as well as a parameter specifying
> > which
> > scheduler.  Parameters not used by a given scheduler are ignored.
> > Additionally, the struct contains a parameter to specify the
> > scheduler.  If you specify a specific scheduler,
> > libxl_domain_sched_params_set() will fail if there's a different
> > scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will
> > use
> > the value of the current scheduler for that domain.
> > 
> > In domcreate_stream_done(), before calling libxl__build_post(), set
> > the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
> > scheduler parameters from the previous instantiation on a best-
> > effort
> > basis.
> > 
> > Signed-off-by: George Dunlap <george.dunlap@citrix.com>
> 
> Acked-by: Juergen Gross <jgross@suse.com>
> 
Reviewed-by: Dario Faggioli <dfaggioli@suse.com>

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2638 bytes --]

On Wed, 2018-10-03 at 12:53 +0200, Juergen Gross wrote:
> On 02/10/2018 17:49, George Dunlap wrote:
> > Commit 3b4adba ("tools/libxl: include scheduler parameters in the
> > output of xl list -l") added scheduling parameters to the set of
> > information collected by libxl_retrieve_domain_configuration(), in
> > order to report that information in `xl list -l`.
> > 
> > Unfortunately, libxl_retrieve_domain_configuration() is also called
> > by
> > the migration / save code, and the results passed to the restore /
> > receive code.  This meant scheduler parameters were inadvertently
> > added to the migration stream, without proper consideration for how
> > to
> > handle corner cases.  The result was that if migrating from a host
> > running one scheduler to a host running a different scheduler, the
> > migration would fail with an error like the following:
> > 
> > libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain
> > 1:Getting domain sched credit: Invalid argument
> > libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain
> > 1:cannot (re-)build domain: -3
> > 
> > Luckily there's a fairly straightforward way to set parameters in a
> > "best-effort" fashion.  libxl provides a single struct containing
> > the
> > parameters of all schedulers, as well as a parameter specifying
> > which
> > scheduler.  Parameters not used by a given scheduler are ignored.
> > Additionally, the struct contains a parameter to specify the
> > scheduler.  If you specify a specific scheduler,
> > libxl_domain_sched_params_set() will fail if there's a different
> > scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will
> > use
> > the value of the current scheduler for that domain.
> > 
> > In domcreate_stream_done(), before calling libxl__build_post(), set
> > the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
> > scheduler parameters from the previous instantiation on a best-
> > effort
> > basis.
> > 
> > Signed-off-by: George Dunlap <george.dunlap@citrix.com>
> 
> Acked-by: Juergen Gross <jgross@suse.com>
> 
Wei, Ian, ping?

We've reverted Credit2 as default scheduler, not because of a Credit2
own issue, but because of the bug fixed by this patch.

I think it would be both fair and appropriate to re-instate it (i.e.,
Credit2 as default), as soon as possible.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[Wei Liu]

On Tue, Oct 02, 2018 at 04:49:26PM +0100, George Dunlap wrote:
> Commit 3b4adba ("tools/libxl: include scheduler parameters in the
> output of xl list -l") added scheduling parameters to the set of
> information collected by libxl_retrieve_domain_configuration(), in
> order to report that information in `xl list -l`.
> 
> Unfortunately, libxl_retrieve_domain_configuration() is also called by
> the migration / save code, and the results passed to the restore /
> receive code.  This meant scheduler parameters were inadvertently
> added to the migration stream, without proper consideration for how to
> handle corner cases.  The result was that if migrating from a host
> running one scheduler to a host running a different scheduler, the
> migration would fail with an error like the following:
> 
> libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
> libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
> 
> Luckily there's a fairly straightforward way to set parameters in a
> "best-effort" fashion.  libxl provides a single struct containing the
> parameters of all schedulers, as well as a parameter specifying which
> scheduler.  Parameters not used by a given scheduler are ignored.
> Additionally, the struct contains a parameter to specify the
> scheduler.  If you specify a specific scheduler,
> libxl_domain_sched_params_set() will fail if there's a different
> scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
> the value of the current scheduler for that domain.
> 
> In domcreate_stream_done(), before calling libxl__build_post(), set
> the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
> scheduler parameters from the previous instantiation on a best-effort
> basis.
> 
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
> ---
> CC: Ian Jackson <ian.jackson@citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
> CC: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Jan Beulich <jbeulich@suse.com>
> CC: Dario Faggioli <dfaggioli@suse.com>
> CC: Juergen Gross <jgross@suse.com>
> ---
>  tools/libxl/libxl_create.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index dcfde7787e..caf79d4620 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -1237,6 +1237,15 @@ static void domcreate_stream_done(libxl__egc *egc,
>          ret = ERROR_INVAL;
>          goto out;
>      }
> +
> +    /* 
> +     * The scheduler on the sending domain may be different than the
> +     * scheduler running here.  Setting the scheduler to UNKNOWN will
> +     * cause the code to take to take whatever parameters are
> +     * available in that scheduler, while discarding the rest.
> +     */
> +    info->sched_params.sched = LIBXL_SCHEDULER_UNKNOWN;

But this function is called by normal domain creation as well. What if
libxl user does set the parameter?

Wei.

> +    
>      ret = libxl__build_post(gc, domid, info, state, vments, localents);
>      if (ret)
>          goto out;
> -- 
> 2.19.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[George Dunlap]

On 10/09/2018 04:12 PM, Wei Liu wrote:
> On Tue, Oct 02, 2018 at 04:49:26PM +0100, George Dunlap wrote:
>> Commit 3b4adba ("tools/libxl: include scheduler parameters in the
>> output of xl list -l") added scheduling parameters to the set of
>> information collected by libxl_retrieve_domain_configuration(), in
>> order to report that information in `xl list -l`.
>>
>> Unfortunately, libxl_retrieve_domain_configuration() is also called by
>> the migration / save code, and the results passed to the restore /
>> receive code.  This meant scheduler parameters were inadvertently
>> added to the migration stream, without proper consideration for how to
>> handle corner cases.  The result was that if migrating from a host
>> running one scheduler to a host running a different scheduler, the
>> migration would fail with an error like the following:
>>
>> libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
>> libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
>>
>> Luckily there's a fairly straightforward way to set parameters in a
>> "best-effort" fashion.  libxl provides a single struct containing the
>> parameters of all schedulers, as well as a parameter specifying which
>> scheduler.  Parameters not used by a given scheduler are ignored.
>> Additionally, the struct contains a parameter to specify the
>> scheduler.  If you specify a specific scheduler,
>> libxl_domain_sched_params_set() will fail if there's a different
>> scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
>> the value of the current scheduler for that domain.
>>
>> In domcreate_stream_done(), before calling libxl__build_post(), set
>> the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
>> scheduler parameters from the previous instantiation on a best-effort
>> basis.
>>
>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>> ---
>> CC: Ian Jackson <ian.jackson@citrix.com>
>> CC: Wei Liu <wei.liu2@citrix.com>
>> CC: Andrew Cooper <andrew.cooper3@citrix.com>
>> CC: Jan Beulich <jbeulich@suse.com>
>> CC: Dario Faggioli <dfaggioli@suse.com>
>> CC: Juergen Gross <jgross@suse.com>
>> ---
>>  tools/libxl/libxl_create.c | 9 +++++++++
>>  1 file changed, 9 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index dcfde7787e..caf79d4620 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -1237,6 +1237,15 @@ static void domcreate_stream_done(libxl__egc *egc,
>>          ret = ERROR_INVAL;
>>          goto out;
>>      }
>> +
>> +    /* 
>> +     * The scheduler on the sending domain may be different than the
>> +     * scheduler running here.  Setting the scheduler to UNKNOWN will
>> +     * cause the code to take to take whatever parameters are
>> +     * available in that scheduler, while discarding the rest.
>> +     */
>> +    info->sched_params.sched = LIBXL_SCHEDULER_UNKNOWN;
> 
> But this function is called by normal domain creation as well. What if
> libxl user does set the parameter?

Is it?  I thought the `stream` meant it was only called when doing
migrate / resume.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[Wei Liu]

On Tue, Oct 09, 2018 at 04:16:48PM +0100, George Dunlap wrote:
> On 10/09/2018 04:12 PM, Wei Liu wrote:
> > On Tue, Oct 02, 2018 at 04:49:26PM +0100, George Dunlap wrote:
> >> Commit 3b4adba ("tools/libxl: include scheduler parameters in the
> >> output of xl list -l") added scheduling parameters to the set of
> >> information collected by libxl_retrieve_domain_configuration(), in
> >> order to report that information in `xl list -l`.
> >>
> >> Unfortunately, libxl_retrieve_domain_configuration() is also called by
> >> the migration / save code, and the results passed to the restore /
> >> receive code.  This meant scheduler parameters were inadvertently
> >> added to the migration stream, without proper consideration for how to
> >> handle corner cases.  The result was that if migrating from a host
> >> running one scheduler to a host running a different scheduler, the
> >> migration would fail with an error like the following:
> >>
> >> libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
> >> libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
> >>
> >> Luckily there's a fairly straightforward way to set parameters in a
> >> "best-effort" fashion.  libxl provides a single struct containing the
> >> parameters of all schedulers, as well as a parameter specifying which
> >> scheduler.  Parameters not used by a given scheduler are ignored.
> >> Additionally, the struct contains a parameter to specify the
> >> scheduler.  If you specify a specific scheduler,
> >> libxl_domain_sched_params_set() will fail if there's a different
> >> scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
> >> the value of the current scheduler for that domain.
> >>
> >> In domcreate_stream_done(), before calling libxl__build_post(), set
> >> the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
> >> scheduler parameters from the previous instantiation on a best-effort
> >> basis.
> >>
> >> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
> >> ---
> >> CC: Ian Jackson <ian.jackson@citrix.com>
> >> CC: Wei Liu <wei.liu2@citrix.com>
> >> CC: Andrew Cooper <andrew.cooper3@citrix.com>
> >> CC: Jan Beulich <jbeulich@suse.com>
> >> CC: Dario Faggioli <dfaggioli@suse.com>
> >> CC: Juergen Gross <jgross@suse.com>
> >> ---
> >>  tools/libxl/libxl_create.c | 9 +++++++++
> >>  1 file changed, 9 insertions(+)
> >>
> >> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> >> index dcfde7787e..caf79d4620 100644
> >> --- a/tools/libxl/libxl_create.c
> >> +++ b/tools/libxl/libxl_create.c
> >> @@ -1237,6 +1237,15 @@ static void domcreate_stream_done(libxl__egc *egc,
> >>          ret = ERROR_INVAL;
> >>          goto out;
> >>      }
> >> +
> >> +    /* 
> >> +     * The scheduler on the sending domain may be different than the
> >> +     * scheduler running here.  Setting the scheduler to UNKNOWN will
> >> +     * cause the code to take to take whatever parameters are
> >> +     * available in that scheduler, while discarding the rest.
> >> +     */
> >> +    info->sched_params.sched = LIBXL_SCHEDULER_UNKNOWN;
> > 
> > But this function is called by normal domain creation as well. What if
> > libxl user does set the parameter?
> 
> Is it?  I thought the `stream` meant it was only called when doing
> migrate / resume.

Yes, you're right. Feel free to add my ack.

Wei.

> 
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] libxl: Restore scheduling parameters after migrate in best-effort fashion

[George Dunlap]

On 10/09/2018 04:21 PM, Wei Liu wrote:
> On Tue, Oct 09, 2018 at 04:16:48PM +0100, George Dunlap wrote:
>> On 10/09/2018 04:12 PM, Wei Liu wrote:
>>> On Tue, Oct 02, 2018 at 04:49:26PM +0100, George Dunlap wrote:
>>>> Commit 3b4adba ("tools/libxl: include scheduler parameters in the
>>>> output of xl list -l") added scheduling parameters to the set of
>>>> information collected by libxl_retrieve_domain_configuration(), in
>>>> order to report that information in `xl list -l`.
>>>>
>>>> Unfortunately, libxl_retrieve_domain_configuration() is also called by
>>>> the migration / save code, and the results passed to the restore /
>>>> receive code.  This meant scheduler parameters were inadvertently
>>>> added to the migration stream, without proper consideration for how to
>>>> handle corner cases.  The result was that if migrating from a host
>>>> running one scheduler to a host running a different scheduler, the
>>>> migration would fail with an error like the following:
>>>>
>>>> libxl: error: libxl_sched.c:232:sched_credit_domain_set: Domain 1:Getting domain sched credit: Invalid argument
>>>> libxl: error: libxl_create.c:1275:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
>>>>
>>>> Luckily there's a fairly straightforward way to set parameters in a
>>>> "best-effort" fashion.  libxl provides a single struct containing the
>>>> parameters of all schedulers, as well as a parameter specifying which
>>>> scheduler.  Parameters not used by a given scheduler are ignored.
>>>> Additionally, the struct contains a parameter to specify the
>>>> scheduler.  If you specify a specific scheduler,
>>>> libxl_domain_sched_params_set() will fail if there's a different
>>>> scheduler.  However, if you pass LIBXL_SCHEDULER_UNKNOWN, it will use
>>>> the value of the current scheduler for that domain.
>>>>
>>>> In domcreate_stream_done(), before calling libxl__build_post(), set
>>>> the scheduler to LIBXL_SCHEDULER_UNKNOWN.  This will propagate
>>>> scheduler parameters from the previous instantiation on a best-effort
>>>> basis.
>>>>
>>>> Signed-off-by: George Dunlap <george.dunlap@citrix.com>
>>>> ---
>>>> CC: Ian Jackson <ian.jackson@citrix.com>
>>>> CC: Wei Liu <wei.liu2@citrix.com>
>>>> CC: Andrew Cooper <andrew.cooper3@citrix.com>
>>>> CC: Jan Beulich <jbeulich@suse.com>
>>>> CC: Dario Faggioli <dfaggioli@suse.com>
>>>> CC: Juergen Gross <jgross@suse.com>
>>>> ---
>>>>  tools/libxl/libxl_create.c | 9 +++++++++
>>>>  1 file changed, 9 insertions(+)
>>>>
>>>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>>>> index dcfde7787e..caf79d4620 100644
>>>> --- a/tools/libxl/libxl_create.c
>>>> +++ b/tools/libxl/libxl_create.c
>>>> @@ -1237,6 +1237,15 @@ static void domcreate_stream_done(libxl__egc *egc,
>>>>          ret = ERROR_INVAL;
>>>>          goto out;
>>>>      }
>>>> +
>>>> +    /* 
>>>> +     * The scheduler on the sending domain may be different than the
>>>> +     * scheduler running here.  Setting the scheduler to UNKNOWN will
>>>> +     * cause the code to take to take whatever parameters are
>>>> +     * available in that scheduler, while discarding the rest.
>>>> +     */
>>>> +    info->sched_params.sched = LIBXL_SCHEDULER_UNKNOWN;
>>>
>>> But this function is called by normal domain creation as well. What if
>>> libxl user does set the parameter?
>>
>> Is it?  I thought the `stream` meant it was only called when doing
>> migrate / resume.
> 
> Yes, you're right. Feel free to add my ack.

Great, thanks.  I'll re-push the credit2 default patch after this gets
pushed to master.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel