[PATCH v3 0/4] xen/tools: sched: Credit1: improve handling of vCPU migration delay

[PATCH v3 0/4] xen/tools: sched: Credit1: improve handling of vCPU migration delay

[Dario Faggioli]

Hi,

Version 3 of this series.

v2:
 https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg02177.html
v1:
 https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg02029.html

I think I've addressed all the review comments (basically, the time conversion issues spotted by George).
Sorry it took a bit, was otherwise engaged.

Updated branches:

 git://xenbits.xen.org/people/dariof/xen.git rel/sched/credit/vcpu_migr_delay_percpool-v3
 http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/sched/credit/vcpu_migr_delay_percpool-v3
Or:
 https://github.com/fdario/xen/tree/rel/sched/credit/vcpu_migr_delay_percpool-v3

Travis is all green :-)
 https://travis-ci.org/fdario/xen/builds/353917909

Regards,
Dario
---
Dario Faggioli (4):
      xen: sched/credit1: make vcpu_migration_delay per-cpupool
      tools: libxl/xl: allow to get/set Credit1's vcpu_migration_delay
      tools: xenpm: continue to support {set,get}-vcpu-migration-delay
      xen/libxc: suppress direct access to Credit1's migration delay

 docs/man/xl.pod.1.in          |   11 +++++++++
 tools/libxc/include/xenctrl.h |    2 --
 tools/libxc/xc_pm.c           |   30 ------------------------
 tools/libxl/libxl.h           |    7 ++++++
 tools/libxl/libxl_sched.c     |    9 +++++++
 tools/libxl/libxl_types.idl   |    1 +
 tools/misc/xenpm.c            |   22 ++++++++++++++----
 tools/xl/xl_cmdtable.c        |    1 +
 tools/xl/xl_sched.c           |   23 +++++++++++++-----
 xen/common/sched_credit.c     |   51 +++++++++++++++++++++++------------------
 xen/drivers/acpi/pmstat.c     |   12 ----------
 xen/include/public/sysctl.h   |   10 ++++++--
 xen/include/xen/sched.h       |    3 --
 13 files changed, 98 insertions(+), 84 deletions(-)
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH v3 1/4] xen: sched/credit1: make vcpu_migration_delay per-cpupool

[Dario Faggioli]

Right now, vCPU migration delay is controlled by
the vcpu_migration_delay boot parameter. This means
the same value will always be used for every instance
of Credit1, in any cpupool that will be created.

Also, in order to get and set such value, a special
purpose libxc interface is defined, and used by the
xenpm tool. And this is problematic if Xen is built
without Credit1 support.

This commit adds a vcpu_migr_delay field inside
struct csched_private, so that we can get/set the
migration delay indepently for each Credit1 instance,
in different cpupools.

Getting and setting now happens via XEN_SYSCTL_SCHEDOP_*,
which is much better suited for this parameter.

The value of the boot time parameter is used for
initializing the vcpu_migr_delay field of the private
structure of all the scheduler instances, when they're
created.

While there, save reading NOW() and doing any s_time_t
operation, when the migration delay of a scheduler is
zero (as it is, by default), in
__csched_vcpu_is_cache_hot().

Finally, note that, from this commit on, using `xenpm
{set,get}-vcpu-migration-delay' will have no effect
any longer. A subsequent commit will re-enable it, for
the sake of backwards-compatibility.

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes from v1:
* improved the changelog, as suggested;
* add the _US suffix to XEN_SYSCTL_CSCHED_MGR_DLY_MAX;
* add the _us suffix vcpu_migration_delay too;
* fix wrong time conversions;
* drop redundant and wrong checks for [params]->vcpu_migration_delay to be 0.
---
 xen/common/sched_credit.c   |   45 +++++++++++++++++++++++++++++--------------
 xen/include/public/sysctl.h |    6 ++++++
 2 files changed, 36 insertions(+), 15 deletions(-)

diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 7c40ee2d00..1f4da65d98 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -214,7 +214,7 @@ struct csched_private {
 
     /* Period of master and tick in milliseconds */
     unsigned int tick_period_us, ticks_per_tslice;
-    s_time_t ratelimit, tslice;
+    s_time_t ratelimit, tslice, vcpu_migr_delay;
 
     struct list_head active_sdom;
     uint32_t weight;
@@ -677,24 +677,24 @@ __csched_vcpu_check(struct vcpu *vc)
  * implicit overheads such as cache-warming. 1ms (1000) has been measured
  * as a good value.
  */
-static unsigned int vcpu_migration_delay;
-integer_param("vcpu_migration_delay", vcpu_migration_delay);
+static unsigned int vcpu_migration_delay_us;
+integer_param("vcpu_migration_delay", vcpu_migration_delay_us);
 
 void set_vcpu_migration_delay(unsigned int delay)
 {
-    vcpu_migration_delay = delay;
+    vcpu_migration_delay_us = delay;
 }
 
 unsigned int get_vcpu_migration_delay(void)
 {
-    return vcpu_migration_delay;
+    return vcpu_migration_delay_us;
 }
 
-static inline int
-__csched_vcpu_is_cache_hot(struct vcpu *v)
+static inline bool
+__csched_vcpu_is_cache_hot(const struct csched_private *prv, struct vcpu *v)
 {
-    int hot = ((NOW() - v->last_run_time) <
-               ((uint64_t)vcpu_migration_delay * 1000u));
+    bool hot = prv->vcpu_migr_delay &&
+               (NOW() - v->last_run_time) < prv->vcpu_migr_delay;
 
     if ( hot )
         SCHED_STAT_CRANK(vcpu_hot);
@@ -703,7 +703,8 @@ __csched_vcpu_is_cache_hot(struct vcpu *v)
 }
 
 static inline int
-__csched_vcpu_is_migrateable(struct vcpu *vc, int dest_cpu, cpumask_t *mask)
+__csched_vcpu_is_migrateable(const struct csched_private *prv, struct vcpu *vc,
+                             int dest_cpu, cpumask_t *mask)
 {
     /*
      * Don't pick up work that's hot on peer PCPU, or that can't (or
@@ -714,7 +715,7 @@ __csched_vcpu_is_migrateable(struct vcpu *vc, int dest_cpu, cpumask_t *mask)
      */
     ASSERT(!vc->is_running);
 
-    return !__csched_vcpu_is_cache_hot(vc) &&
+    return !__csched_vcpu_is_cache_hot(prv, vc) &&
            cpumask_test_cpu(dest_cpu, mask);
 }
 
@@ -1251,7 +1252,8 @@ csched_sys_cntl(const struct scheduler *ops,
              || (params->ratelimit_us
                  && (params->ratelimit_us > XEN_SYSCTL_SCHED_RATELIMIT_MAX
                      || params->ratelimit_us < XEN_SYSCTL_SCHED_RATELIMIT_MIN))
-             || MICROSECS(params->ratelimit_us) > MILLISECS(params->tslice_ms) )
+             || MICROSECS(params->ratelimit_us) > MILLISECS(params->tslice_ms)
+             || params->vcpu_migr_delay_us > XEN_SYSCTL_CSCHED_MGR_DLY_MAX_US )
                 goto out;
 
         spin_lock_irqsave(&prv->lock, flags);
@@ -1261,12 +1263,14 @@ csched_sys_cntl(const struct scheduler *ops,
         else if ( prv->ratelimit && !params->ratelimit_us )
             printk(XENLOG_INFO "Disabling context switch rate limiting\n");
         prv->ratelimit = MICROSECS(params->ratelimit_us);
+        prv->vcpu_migr_delay = MICROSECS(params->vcpu_migr_delay_us);
         spin_unlock_irqrestore(&prv->lock, flags);
 
         /* FALLTHRU */
     case XEN_SYSCTL_SCHEDOP_getinfo:
         params->tslice_ms = prv->tslice / MILLISECS(1);
         params->ratelimit_us = prv->ratelimit / MICROSECS(1);
+        params->vcpu_migr_delay_us = prv->vcpu_migr_delay / MICROSECS(1);
         rc = 0;
         break;
     }
@@ -1585,6 +1589,7 @@ csched_tick(void *_cpu)
 static struct csched_vcpu *
 csched_runq_steal(int peer_cpu, int cpu, int pri, int balance_step)
 {
+    const struct csched_private * const prv = CSCHED_PRIV(per_cpu(scheduler, cpu));
     const struct csched_pcpu * const peer_pcpu = CSCHED_PCPU(peer_cpu);
     struct csched_vcpu *speer;
     struct list_head *iter;
@@ -1634,7 +1639,7 @@ csched_runq_steal(int peer_cpu, int cpu, int pri, int balance_step)
             continue;
 
         affinity_balance_cpumask(vc, balance_step, cpumask_scratch);
-        if ( __csched_vcpu_is_migrateable(vc, cpu, cpumask_scratch) )
+        if ( __csched_vcpu_is_migrateable(prv, vc, cpu, cpumask_scratch) )
         {
             /* We got a candidate. Grab it! */
             TRACE_3D(TRC_CSCHED_STOLEN_VCPU, peer_cpu,
@@ -2091,7 +2096,7 @@ csched_dump(const struct scheduler *ops)
            "\tratelimit          = %"PRI_stime"us\n"
            "\tcredits per msec   = %d\n"
            "\tticks per tslice   = %d\n"
-           "\tmigration delay    = %uus\n",
+           "\tmigration delay    = %"PRI_stime"us\n",
            prv->ncpus,
            prv->master,
            prv->credit,
@@ -2103,7 +2108,7 @@ csched_dump(const struct scheduler *ops)
            prv->ratelimit / MICROSECS(1),
            CSCHED_CREDITS_PER_MSEC,
            prv->ticks_per_tslice,
-           vcpu_migration_delay);
+           prv->vcpu_migr_delay/ MICROSECS(1));
 
     cpumask_scnprintf(idlers_buf, sizeof(idlers_buf), prv->idlers);
     printk("idlers: %s\n", idlers_buf);
@@ -2186,6 +2191,16 @@ csched_init(struct scheduler *ops)
     }
     else
         prv->ratelimit = MICROSECS(sched_ratelimit_us);
+
+    if ( vcpu_migration_delay_us > XEN_SYSCTL_CSCHED_MGR_DLY_MAX_US )
+    {
+        vcpu_migration_delay_us = 0;
+        printk("WARNING: vcpu_migration_delay outside of valid range [0,%d]us.\n"
+               "Resetting to default: %u\n",
+               XEN_SYSCTL_CSCHED_MGR_DLY_MAX_US, vcpu_migration_delay_us);
+    }
+    prv->vcpu_migr_delay = MICROSECS(vcpu_migration_delay_us);
+
     return 0;
 }
 
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 3669e32524..8ba644d6f0 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -601,6 +601,12 @@ struct xen_sysctl_credit_schedule {
 #define XEN_SYSCTL_CSCHED_TSLICE_MIN 1
     unsigned tslice_ms;
     unsigned ratelimit_us;
+    /*
+     * How long we consider a vCPU to be cache-hot on the
+     * CPU where it has run (max 100ms, in microseconds)
+    */
+#define XEN_SYSCTL_CSCHED_MGR_DLY_MAX_US (100 * 1000)
+    unsigned vcpu_migr_delay_us;
 };
 
 struct xen_sysctl_credit2_schedule {


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH v3 2/4] tools: libxl/xl: allow to get/set Credit1's vcpu_migration_delay

[Dario Faggioli]

Make it possible to get and set a (Credit1) scheduler's
vCPU migration delay via the SCHEDOP sysctl, from both
libxl and xl (no change needed in libxc).

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes from v2:
* drop a redundant scinfo->vcpu_migr_delay_us != 0 check

Changes from v1:
* add missing 'break', fix using wrong variable in xl_sched.c.
---
 docs/man/xl.pod.1.in        |   11 +++++++++++
 tools/libxl/libxl.h         |    7 +++++++
 tools/libxl/libxl_sched.c   |    9 +++++++++
 tools/libxl/libxl_types.idl |    1 +
 tools/xl/xl_cmdtable.c      |    1 +
 tools/xl/xl_sched.c         |   23 ++++++++++++++++-------
 6 files changed, 45 insertions(+), 7 deletions(-)

diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 7fd35c9ae7..48da2052cc 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -1046,6 +1046,17 @@ we will allow a higher-priority VM to pre-empt it.  The default value
 is 1000 microseconds (1ms).  Valid range is 100 to 500000 (500ms).
 The ratelimit length must be lower than the timeslice length.
 
+=item B<-m DELAY>, B<--migration_delay_us=DELAY>
+
+Migration delay specifies for how long a vCPU, after it stopped running should
+be considered "cache-hot". Basically, if less than DELAY us passed since when
+the vCPU was executing on a CPU, it is likely that most of the vCPU's working
+set is still in the CPU's cache, and therefore the vCPU is not migrated.
+
+Default is 0. Maximum is 100 ms. This can be effective at preventing vCPUs
+to bounce among CPUs too quickly, but, at the same time, the scheduler stops
+being fully work-conserving.
+
 =back
 
 B<COMBINATION>
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index eca0ea2c50..edd244278a 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -299,6 +299,13 @@
  */
 #define LIBXL_HAVE_SCHED_CREDIT2_PARAMS 1
 
+/*
+ * LIBXL_HAVE_SCHED_CREDIT_MIGR_DELAY indicates that there is a field
+ * in libxl_sched_credit_params called vcpu_migr_delay_us which controls
+ * the resistance of the vCPUs of the cpupool to migrations among pCPUs.
+ */
+#define LIBXL_HAVE_SCHED_CREDIT_MIGR_DELAY
+
 /*
  * LIBXL_HAVE_VIRIDIAN_CRASH_CTL indicates that the 'crash_ctl' value
  * is present in the viridian enlightenment enumeration.
diff --git a/tools/libxl/libxl_sched.c b/tools/libxl/libxl_sched.c
index 512788f736..7c53dc60e6 100644
--- a/tools/libxl/libxl_sched.c
+++ b/tools/libxl/libxl_sched.c
@@ -291,6 +291,7 @@ int libxl_sched_credit_params_get(libxl_ctx *ctx, uint32_t poolid,
 
     scinfo->tslice_ms = sparam.tslice_ms;
     scinfo->ratelimit_us = sparam.ratelimit_us;
+    scinfo->vcpu_migr_delay_us = sparam.vcpu_migr_delay_us;
 
     rc = 0;
  out:
@@ -321,9 +322,16 @@ int libxl_sched_credit_params_set(libxl_ctx *ctx, uint32_t poolid,
         rc = ERROR_INVAL;
         goto out;
     }
+    if (scinfo->vcpu_migr_delay_us > XEN_SYSCTL_CSCHED_MGR_DLY_MAX_US) {
+        LOG(ERROR, "vcpu migration delay should be >= 0 and <= %dus",
+            XEN_SYSCTL_CSCHED_MGR_DLY_MAX_US);
+        rc = ERROR_INVAL;
+        goto out;
+    }
 
     sparam.tslice_ms = scinfo->tslice_ms;
     sparam.ratelimit_us = scinfo->ratelimit_us;
+    sparam.vcpu_migr_delay_us = scinfo->vcpu_migr_delay_us;
 
     r = xc_sched_credit_params_set(ctx->xch, poolid, &sparam);
     if ( r < 0 ) {
@@ -334,6 +342,7 @@ int libxl_sched_credit_params_set(libxl_ctx *ctx, uint32_t poolid,
 
     scinfo->tslice_ms = sparam.tslice_ms;
     scinfo->ratelimit_us = sparam.ratelimit_us;
+    scinfo->vcpu_migr_delay_us = sparam.vcpu_migr_delay_us;
 
     rc = 0;
  out:
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 35038120ca..dbb287d6fe 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -973,6 +973,7 @@ libxl_pcitopology = Struct("pcitopology", [
 libxl_sched_credit_params = Struct("sched_credit_params", [
     ("tslice_ms", integer),
     ("ratelimit_us", integer),
+    ("vcpu_migr_delay_us", integer),
     ], dispose_fn=None)
 
 libxl_sched_credit2_params = Struct("sched_credit2_params", [
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 6d894394ca..bf2ced8140 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -257,6 +257,7 @@ struct cmd_spec cmd_table[] = {
       "-s         --schedparam           Query / modify scheduler parameters\n"
       "-t TSLICE, --tslice_ms=TSLICE     Set the timeslice, in milliseconds\n"
       "-r RLIMIT, --ratelimit_us=RLIMIT  Set the scheduling rate limit, in microseconds\n"
+      "-m DLY, --migration_delay_us=DLY  Set the migration delay, in microseconds\n"
       "-p CPUPOOL, --cpupool=CPUPOOL     Restrict output to CPUPOOL"
     },
     { "sched-credit2",
diff --git a/tools/xl/xl_sched.c b/tools/xl/xl_sched.c
index 7965ccbca0..73cd7040cd 100644
--- a/tools/xl/xl_sched.c
+++ b/tools/xl/xl_sched.c
@@ -172,10 +172,11 @@ static int sched_credit_pool_output(uint32_t poolid)
         printf("Cpupool %s: [sched params unavailable]\n",
                poolname);
     } else {
-        printf("Cpupool %s: tslice=%dms ratelimit=%dus\n",
+        printf("Cpupool %s: tslice=%dms ratelimit=%dus migration-delay=%dus\n",
                poolname,
                scparam.tslice_ms,
-               scparam.ratelimit_us);
+               scparam.ratelimit_us,
+               scparam.vcpu_migr_delay_us);
     }
     free(poolname);
     return 0;
@@ -469,10 +470,10 @@ int main_sched_credit(int argc, char **argv)
     const char *dom = NULL;
     const char *cpupool = NULL;
     int weight = 256, cap = 0;
-    int tslice = 0, ratelimit = 0;
+    int tslice = 0, ratelimit = 0, migrdelay = 0;
     bool opt_w = false, opt_c = false;
     bool opt_t = false, opt_r = false;
-    bool opt_s = false;
+    bool opt_s = false, opt_m = false;
     int opt, rc;
     static struct option opts[] = {
         {"domain", 1, 0, 'd'},
@@ -481,11 +482,12 @@ int main_sched_credit(int argc, char **argv)
         {"schedparam", 0, 0, 's'},
         {"tslice_ms", 1, 0, 't'},
         {"ratelimit_us", 1, 0, 'r'},
+        {"migration_delay_us", 1, 0, 'm'},
         {"cpupool", 1, 0, 'p'},
         COMMON_LONG_OPTS
     };
 
-    SWITCH_FOREACH_OPT(opt, "d:w:c:p:t:r:s", opts, "sched-credit", 0) {
+    SWITCH_FOREACH_OPT(opt, "d:w:c:p:t:r:m:s", opts, "sched-credit", 0) {
     case 'd':
         dom = optarg;
         break;
@@ -505,6 +507,10 @@ int main_sched_credit(int argc, char **argv)
         ratelimit = strtol(optarg, NULL, 10);
         opt_r = true;
         break;
+    case 'm':
+        migrdelay = strtol(optarg, NULL, 10);
+        opt_m = true;
+        break;
     case 's':
         opt_s = true;
         break;
@@ -522,7 +528,7 @@ int main_sched_credit(int argc, char **argv)
         fprintf(stderr, "Must specify a domain.\n");
         return EXIT_FAILURE;
     }
-    if (!opt_s && (opt_t || opt_r)) {
+    if (!opt_s && (opt_t || opt_r || opt_m)) {
         fprintf(stderr, "Must specify schedparam to set schedule "
                 "parameter values.\n");
         return EXIT_FAILURE;
@@ -541,7 +547,7 @@ int main_sched_credit(int argc, char **argv)
             }
         }
 
-        if (!opt_t && !opt_r) { /* Output scheduling parameters */
+        if (!opt_t && !opt_r && !opt_m) { /* Output scheduling parameters */
             if (sched_credit_pool_output(poolid))
                 return EXIT_FAILURE;
         } else { /* Set scheduling parameters*/
@@ -554,6 +560,9 @@ int main_sched_credit(int argc, char **argv)
             if (opt_r)
                 scparam.ratelimit_us = ratelimit;
 
+            if (opt_m)
+                scparam.vcpu_migr_delay_us = migrdelay;
+
             if (sched_credit_params_set(poolid, &scparam))
                 return EXIT_FAILURE;
         }


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH v3 3/4] tools: xenpm: continue to support {set, get}-vcpu-migration-delay

[Dario Faggioli]

Now that it is possible to get and set the migration
delay via the SCHEDOP sysctl, use that in xenpm, instead
of the special purpose libxc interface (which will be
removed in a following commit).

The sysctl, however, requires a cpupool-id argument,
for knowing on which scheduler it is operating on. In
this case, since we don't want to alter xenpm's command
line interface, we always use '0', which means xenpm
will always act on the default cpupool ('Pool-0').

>From this commit on, `xenpm {set,get}-vcpu-migration-delay'
commands work again. But that is only for the sake of
backward compatibility, and their use is deprecated, in
favour of 'xl sched-credit -s [-c <poolid>] -m <delay>'.

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes from v2:
* send the warning to stderr, rather than to stdout.
---
 tools/misc/xenpm.c |   22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/tools/misc/xenpm.c b/tools/misc/xenpm.c
index 762311e5a5..0e1968a23d 100644
--- a/tools/misc/xenpm.c
+++ b/tools/misc/xenpm.c
@@ -1071,14 +1071,24 @@ void set_sched_smt_func(int argc, char *argv[])
 
 void set_vcpu_migration_delay_func(int argc, char *argv[])
 {
+    struct xen_sysctl_credit_schedule sparam;
     int value;
 
+    fprintf(stderr, "WARNING: using xenpm for this purpose is deprecated."
+           " Check out `xl sched-credit -s -m DELAY'\n");
+
     if ( argc != 1 || (value = atoi(argv[0])) < 0 ) {
         fprintf(stderr, "Missing or invalid argument(s)\n");
         exit(EINVAL);
     }
 
-    if ( !xc_set_vcpu_migration_delay(xc_handle, value) )
+    if ( xc_sched_credit_params_get(xc_handle, 0, &sparam) < 0 ) {
+        fprintf(stderr, "getting Credit scheduler parameters failed\n");
+        exit(EINVAL);
+    }
+    sparam.vcpu_migr_delay_us = value;
+
+    if ( !xc_sched_credit_params_set(xc_handle, 0, &sparam) )
         printf("set vcpu migration delay to %d us succeeded\n", value);
     else
         fprintf(stderr, "set vcpu migration delay failed (%d - %s)\n",
@@ -1087,13 +1097,17 @@ void set_vcpu_migration_delay_func(int argc, char *argv[])
 
 void get_vcpu_migration_delay_func(int argc, char *argv[])
 {
-    uint32_t value;
+    struct xen_sysctl_credit_schedule sparam;
+
+    fprintf(stderr, "WARNING: using xenpm for this purpose is deprecated."
+           " Check out `xl sched-credit -s'\n");
 
     if ( argc )
         fprintf(stderr, "Ignoring argument(s)\n");
 
-    if ( !xc_get_vcpu_migration_delay(xc_handle, &value) )
-        printf("Scheduler vcpu migration delay is %d us\n", value);
+    if ( !xc_sched_credit_params_get(xc_handle, 0, &sparam) )
+        printf("Scheduler vcpu migration delay is %d us\n",
+               sparam.vcpu_migr_delay_us);
     else
         fprintf(stderr,
                 "Failed to get scheduler vcpu migration delay (%d - %s)\n",


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH v3 4/4] xen/libxc: suppress direct access to Credit1's migration delay

[Dario Faggioli]

Removes special purpose access to Credit1 vCPU
migration delay parameter.

This fixes a build breakage, occuring when Xen
is configured with SCHED_CREDIT=n.

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: Tim Deegan <tim@xen.org>
Cc: George Dunlap <george.dunlap@eu.citrix.com>
---
Changes from v1:
* bumped the interface version, as requested.
---
 tools/libxc/include/xenctrl.h |    2 --
 tools/libxc/xc_pm.c           |   30 ------------------------------
 xen/common/sched_credit.c     |   10 ----------
 xen/drivers/acpi/pmstat.c     |   12 ------------
 xen/include/public/sysctl.h   |    4 +---
 xen/include/xen/sched.h       |    3 ---
 6 files changed, 1 insertion(+), 60 deletions(-)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 543abfcb34..058e832c47 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -1920,8 +1920,6 @@ int xc_set_cpufreq_para(xc_interface *xch, int cpuid,
 int xc_get_cpufreq_avgfreq(xc_interface *xch, int cpuid, int *avg_freq);
 
 int xc_set_sched_opt_smt(xc_interface *xch, uint32_t value);
-int xc_set_vcpu_migration_delay(xc_interface *xch, uint32_t value);
-int xc_get_vcpu_migration_delay(xc_interface *xch, uint32_t *value);
 
 int xc_get_cpuidle_max_cstate(xc_interface *xch, uint32_t *value);
 int xc_set_cpuidle_max_cstate(xc_interface *xch, uint32_t value);
diff --git a/tools/libxc/xc_pm.c b/tools/libxc/xc_pm.c
index ae917bc630..67e2418e3f 100644
--- a/tools/libxc/xc_pm.c
+++ b/tools/libxc/xc_pm.c
@@ -367,36 +367,6 @@ int xc_set_sched_opt_smt(xc_interface *xch, uint32_t value)
    return rc;
 }
 
-int xc_set_vcpu_migration_delay(xc_interface *xch, uint32_t value)
-{
-   int rc;
-   DECLARE_SYSCTL;
-
-   sysctl.cmd = XEN_SYSCTL_pm_op;
-   sysctl.u.pm_op.cmd = XEN_SYSCTL_pm_op_set_vcpu_migration_delay;
-   sysctl.u.pm_op.cpuid = 0;
-   sysctl.u.pm_op.u.set_vcpu_migration_delay = value;
-   rc = do_sysctl(xch, &sysctl);
-
-   return rc;
-}
-
-int xc_get_vcpu_migration_delay(xc_interface *xch, uint32_t *value)
-{
-   int rc;
-   DECLARE_SYSCTL;
-
-   sysctl.cmd = XEN_SYSCTL_pm_op;
-   sysctl.u.pm_op.cmd = XEN_SYSCTL_pm_op_get_vcpu_migration_delay;
-   sysctl.u.pm_op.cpuid = 0;
-   rc = do_sysctl(xch, &sysctl);
-
-   if (!rc && value)
-       *value = sysctl.u.pm_op.u.get_vcpu_migration_delay;
-
-   return rc;
-}
-
 int xc_get_cpuidle_max_cstate(xc_interface *xch, uint32_t *value)
 {
     int rc;
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 1f4da65d98..a618d303b4 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -680,16 +680,6 @@ __csched_vcpu_check(struct vcpu *vc)
 static unsigned int vcpu_migration_delay_us;
 integer_param("vcpu_migration_delay", vcpu_migration_delay_us);
 
-void set_vcpu_migration_delay(unsigned int delay)
-{
-    vcpu_migration_delay_us = delay;
-}
-
-unsigned int get_vcpu_migration_delay(void)
-{
-    return vcpu_migration_delay_us;
-}
-
 static inline bool
 __csched_vcpu_is_cache_hot(const struct csched_private *prv, struct vcpu *v)
 {
diff --git a/xen/drivers/acpi/pmstat.c b/xen/drivers/acpi/pmstat.c
index 2a6c4c7444..a8fc52a35f 100644
--- a/xen/drivers/acpi/pmstat.c
+++ b/xen/drivers/acpi/pmstat.c
@@ -449,18 +449,6 @@ int do_pm_op(struct xen_sysctl_pm_op *op)
         break;
     }
 
-    case XEN_SYSCTL_pm_op_set_vcpu_migration_delay:
-    {
-        set_vcpu_migration_delay(op->u.set_vcpu_migration_delay);
-        break;
-    }
-
-    case XEN_SYSCTL_pm_op_get_vcpu_migration_delay:
-    {
-        op->u.get_vcpu_migration_delay = get_vcpu_migration_delay();
-        break;
-    }
-
     case XEN_SYSCTL_pm_op_get_max_cstate:
     {
         op->u.get_max_cstate = acpi_get_cstate_limit();
diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h
index 8ba644d6f0..839c1b9f25 100644
--- a/xen/include/public/sysctl.h
+++ b/xen/include/public/sysctl.h
@@ -36,7 +36,7 @@
 #include "physdev.h"
 #include "tmem.h"
 
-#define XEN_SYSCTL_INTERFACE_VERSION 0x00000010
+#define XEN_SYSCTL_INTERFACE_VERSION 0x00000011
 
 /*
  * Read console content from Xen buffer ring.
@@ -351,8 +351,6 @@ struct xen_sysctl_pm_op {
         uint32_t                    set_sched_opt_smt;
         uint32_t                    get_max_cstate;
         uint32_t                    set_max_cstate;
-        uint32_t                    get_vcpu_migration_delay;
-        uint32_t                    set_vcpu_migration_delay;
     } u;
 };
 
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 39f938644a..a82ee08dd6 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -903,9 +903,6 @@ static inline bool is_vcpu_online(const struct vcpu *v)
     return !test_bit(_VPF_down, &v->pause_flags);
 }
 
-void set_vcpu_migration_delay(unsigned int delay);
-unsigned int get_vcpu_migration_delay(void);
-
 extern bool sched_smt_power_savings;
 
 extern enum cpufreq_controller {


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3 4/4] xen/libxc: suppress direct access to Credit1's migration delay

[Jan Beulich]

>>> On 15.03.18 at 18:51, <dfaggioli@suse.com> wrote:
> Removes special purpose access to Credit1 vCPU
> migration delay parameter.
> 
> This fixes a build breakage, occuring when Xen
> is configured with SCHED_CREDIT=n.
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> Acked-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: Jan Beulich <jbeulich@suse.com>
(for the parts its applicable to)

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3 0/4] xen/tools: sched: Credit1: improve handling of vCPU migration delay

[Wei Liu]

On Thu, Mar 15, 2018 at 06:51:14PM +0100, Dario Faggioli wrote:
> Hi,
> 
> Version 3 of this series.
> 
> v2:
>  https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg02177.html
> v1:
>  https://lists.xenproject.org/archives/html/xen-devel/2018-02/msg02029.html
> 
> I think I've addressed all the review comments (basically, the time conversion issues spotted by George).
> Sorry it took a bit, was otherwise engaged.
> 
> Updated branches:
> 
>  git://xenbits.xen.org/people/dariof/xen.git rel/sched/credit/vcpu_migr_delay_percpool-v3
>  http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/sched/credit/vcpu_migr_delay_percpool-v3
> Or:
>  https://github.com/fdario/xen/tree/rel/sched/credit/vcpu_migr_delay_percpool-v3
> 
> Travis is all green :-)
>  https://travis-ci.org/fdario/xen/builds/353917909
> 
> Regards,
> Dario
> ---
> Dario Faggioli (4):
>       xen: sched/credit1: make vcpu_migration_delay per-cpupool
>       tools: libxl/xl: allow to get/set Credit1's vcpu_migration_delay
>       tools: xenpm: continue to support {set,get}-vcpu-migration-delay
>       xen/libxc: suppress direct access to Credit1's migration delay

FAOD I will leave committing this series to whoever acks patch 1.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3 2/4] tools: libxl/xl: allow to get/set Credit1's vcpu_migration_delay

[George Dunlap]

On 03/15/2018 05:51 PM, Dario Faggioli wrote:
> Make it possible to get and set a (Credit1) scheduler's
> vCPU migration delay via the SCHEDOP sysctl, from both
> libxl and xl (no change needed in libxc).
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> Acked-by: Wei Liu <wei.liu2@citrix.com>

Reviewed-by: George Dunlap <george.dunlap@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3 4/4] xen/libxc: suppress direct access to Credit1's migration delay

[George Dunlap]

On 03/15/2018 05:51 PM, Dario Faggioli wrote:
> Removes special purpose access to Credit1 vCPU
> migration delay parameter.
> 
> This fixes a build breakage, occuring when Xen
> is configured with SCHED_CREDIT=n.
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> Acked-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: George Dunlap <george.dunlap@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3 1/4] xen: sched/credit1: make vcpu_migration_delay per-cpupool

[George Dunlap]

On 03/15/2018 05:51 PM, Dario Faggioli wrote:
> Right now, vCPU migration delay is controlled by
> the vcpu_migration_delay boot parameter. This means
> the same value will always be used for every instance
> of Credit1, in any cpupool that will be created.
> 
> Also, in order to get and set such value, a special
> purpose libxc interface is defined, and used by the
> xenpm tool. And this is problematic if Xen is built
> without Credit1 support.
> 
> This commit adds a vcpu_migr_delay field inside
> struct csched_private, so that we can get/set the
> migration delay indepently for each Credit1 instance,
> in different cpupools.
> 
> Getting and setting now happens via XEN_SYSCTL_SCHEDOP_*,
> which is much better suited for this parameter.
> 
> The value of the boot time parameter is used for
> initializing the vcpu_migr_delay field of the private
> structure of all the scheduler instances, when they're
> created.
> 
> While there, save reading NOW() and doing any s_time_t
> operation, when the migration delay of a scheduler is
> zero (as it is, by default), in
> __csched_vcpu_is_cache_hot().
> 
> Finally, note that, from this commit on, using `xenpm
> {set,get}-vcpu-migration-delay' will have no effect
> any longer. A subsequent commit will re-enable it, for
> the sake of backwards-compatibility.
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>

Looks good, thanks!

Reviewed-by: George Dunlap <george.dunlap@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel