[PATCH v3 0/2] sched: Credit1 and Credit2 improvements... but *NO* soft-affinity for Credit2!

[PATCH v3 0/2] sched: Credit1 and Credit2 improvements... but *NO* soft-affinity for Credit2!

[Dario Faggioli]

And here they are, the last two remaining patches...

Dario
---
Dario Faggioli (2):
      xen: credit2: implement yield()
      xen: tracing: add trace records for schedule and rate-limiting.

 xen/common/sched_credit.c    |   32 +++++++++++++++
 xen/common/sched_credit2.c   |   92 +++++++++++++++++++++++++++++++++++-------
 xen/common/sched_rt.c        |   15 +++++++
 xen/common/schedule.c        |    2 +
 xen/include/xen/perfc_defn.h |    1 
 5 files changed, 127 insertions(+), 15 deletions(-)
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 1/2] xen: credit2: implement yield()

[Dario Faggioli]

When a vcpu explicitly yields it is usually giving
us an advice of "let someone else run and come back
to me in a bit."

Credit2 isn't, so far, doing anything when a vcpu
yields, which means an yield is basically a NOP (well,
actually, it's pure overhead, as it causes the scheduler
kick in, but the result is --at least 99% of the time--
that the very same vcpu that yielded continues to run).

With this patch, when a vcpu yields, we go and try
picking the next vcpu on the runqueue that can run on
the pcpu where the yielding vcpu is running. Of course,
if we don't find any other vcpu that wants and can run
there, the yielding vcpu will continue.

Also, add an yield performance counter, and fix the
style of a couple of comments.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Anshul Makkar <anshul.makkar@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes from v2:
 * implement yield as a "switch", rather than as a "knob", as suggested
   during review.

Changes from v1:
 * add _us to the parameter name, as suggested during review;
 * get rid of the minimum value for the yield bias;
 * apply the idle bias via subtraction of credits to the yielding vcpu, rather
   than via addition to all the others;
 * merge the Credit2 bits of what was patch 7 here, as suggested during review.
---
 xen/common/sched_credit2.c   |   54 +++++++++++++++++++++++++++++++-----------
 xen/common/schedule.c        |    2 ++
 xen/include/xen/perfc_defn.h |    1 +
 3 files changed, 43 insertions(+), 14 deletions(-)

diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index 72e31b5..c0646e9 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -181,7 +181,13 @@
  */
 #define __CSFLAG_runq_migrate_request 3
 #define CSFLAG_runq_migrate_request (1<<__CSFLAG_runq_migrate_request)
-
+/*
+ * CSFLAG_vcpu_yield: this vcpu was running, and has called vcpu_yield(). The
+ * scheduler is invoked to see if we can give the cpu to someone else, and
+ * get back to the yielding vcpu in a while.
+ */
+#define __CSFLAG_vcpu_yield 4
+#define CSFLAG_vcpu_yield (1<<__CSFLAG_vcpu_yield)
 
 static unsigned int __read_mostly opt_migrate_resist = 500;
 integer_param("sched_credit2_migrate_resist", opt_migrate_resist);
@@ -1431,6 +1437,14 @@ out:
 }
 
 static void
+csched2_vcpu_yield(const struct scheduler *ops, struct vcpu *v)
+{
+    struct csched2_vcpu * const svc = CSCHED2_VCPU(v);
+
+    __set_bit(__CSFLAG_vcpu_yield, &svc->flags);
+}
+
+static void
 csched2_context_saved(const struct scheduler *ops, struct vcpu *vc)
 {
     struct csched2_vcpu * const svc = CSCHED2_VCPU(vc);
@@ -2250,26 +2264,31 @@ runq_candidate(struct csched2_runqueue_data *rqd,
     struct list_head *iter;
     struct csched2_vcpu *snext = NULL;
     struct csched2_private *prv = CSCHED2_PRIV(per_cpu(scheduler, cpu));
+    bool yield = __test_and_clear_bit(__CSFLAG_vcpu_yield, &scurr->flags);
 
     *skipped = 0;
 
-    /* Default to current if runnable, idle otherwise */
-    if ( vcpu_runnable(scurr->vcpu) )
-        snext = scurr;
-    else
-        snext = CSCHED2_VCPU(idle_vcpu[cpu]);
-
     /*
      * Return the current vcpu if it has executed for less than ratelimit.
      * Adjuststment for the selected vcpu's credit and decision
      * for how long it will run will be taken in csched2_runtime.
+     *
+     * Note that, if scurr is yielding, we don't let rate limiting kick in.
+     * In fact, it may be the case that scurr is about to spin, and there's
+     * no point forcing it to do so until rate limiting expires.
      */
-    if ( prv->ratelimit_us && !is_idle_vcpu(scurr->vcpu) &&
+    if ( !yield && prv->ratelimit_us && !is_idle_vcpu(scurr->vcpu) &&
          vcpu_runnable(scurr->vcpu) &&
          (now - scurr->vcpu->runstate.state_entry_time) <
           MICROSECS(prv->ratelimit_us) )
         return scurr;
 
+    /* Default to current if runnable, idle otherwise */
+    if ( vcpu_runnable(scurr->vcpu) )
+        snext = scurr;
+    else
+        snext = CSCHED2_VCPU(idle_vcpu[cpu]);
+
     list_for_each( iter, &rqd->runq )
     {
         struct csched2_vcpu * svc = list_entry(iter, struct csched2_vcpu, runq_elem);
@@ -2293,8 +2312,10 @@ runq_candidate(struct csched2_runqueue_data *rqd,
             continue;
         }
 
-        /* If this is on a different processor, don't pull it unless
-         * its credit is at least CSCHED2_MIGRATE_RESIST higher. */
+        /*
+         * If this is on a different processor, don't pull it unless
+         * its credit is at least CSCHED2_MIGRATE_RESIST higher.
+         */
         if ( svc->vcpu->processor != cpu
              && snext->credit + CSCHED2_MIGRATE_RESIST > svc->credit )
         {
@@ -2303,9 +2324,12 @@ runq_candidate(struct csched2_runqueue_data *rqd,
             continue;
         }
 
-        /* If the next one on the list has more credit than current
-         * (or idle, if current is not runnable), choose it. */
-        if ( svc->credit > snext->credit )
+        /*
+         * If the next one on the list has more credit than current
+         * (or idle, if current is not runnable), or if current is
+         * yielding, choose it.
+         */
+        if ( yield || svc->credit > snext->credit )
             snext = svc;
 
         /* In any case, if we got this far, break. */
@@ -2391,7 +2415,8 @@ csched2_schedule(
      */
     if ( tasklet_work_scheduled )
     {
-        trace_var(TRC_CSCHED2_SCHED_TASKLET, 1, 0,  NULL);
+        __clear_bit(__CSFLAG_vcpu_yield, &scurr->flags);
+        trace_var(TRC_CSCHED2_SCHED_TASKLET, 1, 0, NULL);
         snext = CSCHED2_VCPU(idle_vcpu[cpu]);
     }
     else
@@ -2975,6 +3000,7 @@ static const struct scheduler sched_credit2_def = {
 
     .sleep          = csched2_vcpu_sleep,
     .wake           = csched2_vcpu_wake,
+    .yield          = csched2_vcpu_yield,
 
     .adjust         = csched2_dom_cntl,
     .adjust_global  = csched2_sys_cntl,
diff --git a/xen/common/schedule.c b/xen/common/schedule.c
index 104d203..5b444c4 100644
--- a/xen/common/schedule.c
+++ b/xen/common/schedule.c
@@ -947,6 +947,8 @@ long vcpu_yield(void)
     SCHED_OP(VCPU2OP(v), yield, v);
     vcpu_schedule_unlock_irq(lock, v);
 
+    SCHED_STAT_CRANK(vcpu_yield);
+
     TRACE_2D(TRC_SCHED_YIELD, current->domain->domain_id, current->vcpu_id);
     raise_softirq(SCHEDULE_SOFTIRQ);
     return 0;
diff --git a/xen/include/xen/perfc_defn.h b/xen/include/xen/perfc_defn.h
index 4a835b8..900fddd 100644
--- a/xen/include/xen/perfc_defn.h
+++ b/xen/include/xen/perfc_defn.h
@@ -23,6 +23,7 @@ PERFCOUNTER(vcpu_alloc,             "sched: vcpu_alloc")
 PERFCOUNTER(vcpu_insert,            "sched: vcpu_insert")
 PERFCOUNTER(vcpu_remove,            "sched: vcpu_remove")
 PERFCOUNTER(vcpu_sleep,             "sched: vcpu_sleep")
+PERFCOUNTER(vcpu_yield,             "sched: vcpu_yield")
 PERFCOUNTER(vcpu_wake_running,      "sched: vcpu_wake_running")
 PERFCOUNTER(vcpu_wake_onrunq,       "sched: vcpu_wake_onrunq")
 PERFCOUNTER(vcpu_wake_runnable,     "sched: vcpu_wake_runnable")


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 2/2] xen: tracing: add trace records for schedule and rate-limiting.

[Dario Faggioli]

As far as {csched, csched2, rt}_schedule() are concerned,
an "empty" event, would already make it easier to read and
understand a trace.

But while there, add a few useful information, like
if the cpu that is going through the scheduler has
been tickled or not, if it is currently idle, etc
(they vary, on a per-scheduler basis).

For Credit1 and Credit2, add a record about when
rate-limiting kicks in too.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Meng Xu <mengxu@cis.upenn.edu>
Cc: Anshul Makkar <anshul.makkar@citrix.com>
---
Changes from v1:
 * corrected the schedule record for sched_rt.c, as pointed out during review;
 * pack the Credit1 records as well, as requested during review.
---
 xen/common/sched_credit.c  |   32 ++++++++++++++++++++++++++++++++
 xen/common/sched_credit2.c |   38 +++++++++++++++++++++++++++++++++++++-
 xen/common/sched_rt.c      |   15 +++++++++++++++
 3 files changed, 84 insertions(+), 1 deletion(-)

diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 5700763..fc3a321 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -133,6 +133,8 @@
 #define TRC_CSCHED_TICKLE        TRC_SCHED_CLASS_EVT(CSCHED, 6)
 #define TRC_CSCHED_BOOST_START   TRC_SCHED_CLASS_EVT(CSCHED, 7)
 #define TRC_CSCHED_BOOST_END     TRC_SCHED_CLASS_EVT(CSCHED, 8)
+#define TRC_CSCHED_SCHEDULE      TRC_SCHED_CLASS_EVT(CSCHED, 9)
+#define TRC_CSCHED_RATELIMIT     TRC_SCHED_CLASS_EVT(CSCHED, 10)
 
 
 /*
@@ -1774,6 +1776,23 @@ csched_schedule(
     SCHED_STAT_CRANK(schedule);
     CSCHED_VCPU_CHECK(current);
 
+    /*
+     * Here in Credit1 code, we usually just call TRACE_nD() helpers, and
+     * don't care about packing. But scheduling happens very often, so it
+     * actually is important that the record is as small as possible.
+     */
+    if ( unlikely(tb_init_done) )
+    {
+        struct {
+            unsigned cpu:16, tasklet:8, idle:8;
+        } d;
+        d.cpu = cpu;
+        d.tasklet = tasklet_work_scheduled;
+        d.idle = is_idle_vcpu(current);
+        __trace_var(TRC_CSCHED_SCHEDULE, 1, sizeof(d),
+                    (unsigned char *)&d);
+    }
+
     runtime = now - current->runstate.state_entry_time;
     if ( runtime < 0 ) /* Does this ever happen? */
         runtime = 0;
@@ -1829,6 +1848,19 @@ csched_schedule(
         tslice = MICROSECS(prv->ratelimit_us) - runtime;
         if ( unlikely(runtime < CSCHED_MIN_TIMER) )
             tslice = CSCHED_MIN_TIMER;
+        if ( unlikely(tb_init_done) )
+        {
+            struct {
+                unsigned vcpu:16, dom:16;
+                unsigned runtime;
+            } d;
+            d.dom = scurr->vcpu->domain->domain_id;
+            d.vcpu = scurr->vcpu->vcpu_id;
+            d.runtime = runtime;
+            __trace_var(TRC_CSCHED_RATELIMIT, 1, sizeof(d),
+                        (unsigned char *)&d);
+        }
+
         ret.migrated = 0;
         goto out;
     }
diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
index c0646e9..6b98319 100644
--- a/xen/common/sched_credit2.c
+++ b/xen/common/sched_credit2.c
@@ -55,6 +55,8 @@
 #define TRC_CSCHED2_LOAD_BALANCE     TRC_SCHED_CLASS_EVT(CSCHED2, 17)
 #define TRC_CSCHED2_PICKED_CPU       TRC_SCHED_CLASS_EVT(CSCHED2, 19)
 #define TRC_CSCHED2_RUNQ_CANDIDATE   TRC_SCHED_CLASS_EVT(CSCHED2, 20)
+#define TRC_CSCHED2_SCHEDULE         TRC_SCHED_CLASS_EVT(CSCHED2, 21)
+#define TRC_CSCHED2_RATELIMIT        TRC_SCHED_CLASS_EVT(CSCHED2, 22)
 
 /*
  * WARNING: This is still in an experimental phase.  Status and work can be found at the
@@ -2281,7 +2283,22 @@ runq_candidate(struct csched2_runqueue_data *rqd,
          vcpu_runnable(scurr->vcpu) &&
          (now - scurr->vcpu->runstate.state_entry_time) <
           MICROSECS(prv->ratelimit_us) )
+    {
+        if ( unlikely(tb_init_done) )
+        {
+            struct {
+                unsigned vcpu:16, dom:16;
+                unsigned runtime;
+            } d;
+            d.dom = scurr->vcpu->domain->domain_id;
+            d.vcpu = scurr->vcpu->vcpu_id;
+            d.runtime = now - scurr->vcpu->runstate.state_entry_time;
+            __trace_var(TRC_CSCHED2_RATELIMIT, 1,
+                        sizeof(d),
+                        (unsigned char *)&d);
+        }
         return scurr;
+    }
 
     /* Default to current if runnable, idle otherwise */
     if ( vcpu_runnable(scurr->vcpu) )
@@ -2371,6 +2388,7 @@ csched2_schedule(
     struct csched2_vcpu *snext = NULL;
     unsigned int skipped_vcpus = 0;
     struct task_slice ret;
+    bool_t tickled;
 
     SCHED_STAT_CRANK(schedule);
     CSCHED2_VCPU_CHECK(current);
@@ -2385,13 +2403,31 @@ csched2_schedule(
     BUG_ON(!is_idle_vcpu(scurr->vcpu) && scurr->rqd != rqd);
 
     /* Clear "tickled" bit now that we've been scheduled */
-    if ( cpumask_test_cpu(cpu, &rqd->tickled) )
+    tickled = cpumask_test_cpu(cpu, &rqd->tickled);
+    if ( tickled )
     {
         __cpumask_clear_cpu(cpu, &rqd->tickled);
         cpumask_andnot(cpumask_scratch, &rqd->idle, &rqd->tickled);
         smt_idle_mask_set(cpu, cpumask_scratch, &rqd->smt_idle);
     }
 
+    if ( unlikely(tb_init_done) )
+    {
+        struct {
+            unsigned cpu:16, rq_id:16;
+            unsigned tasklet:8, idle:8, smt_idle:8, tickled:8;
+        } d;
+        d.cpu = cpu;
+        d.rq_id = c2r(ops, cpu);
+        d.tasklet = tasklet_work_scheduled;
+        d.idle = is_idle_vcpu(current);
+        d.smt_idle = cpumask_test_cpu(cpu, &rqd->smt_idle);
+        d.tickled = tickled;
+        __trace_var(TRC_CSCHED2_SCHEDULE, 1,
+                    sizeof(d),
+                    (unsigned char *)&d);
+    }
+
     /* Update credits */
     burn_credits(rqd, scurr, now);
 
diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 41c61a7..d95f798 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -160,6 +160,7 @@
 #define TRC_RTDS_BUDGET_BURN      TRC_SCHED_CLASS_EVT(RTDS, 3)
 #define TRC_RTDS_BUDGET_REPLENISH TRC_SCHED_CLASS_EVT(RTDS, 4)
 #define TRC_RTDS_SCHED_TASKLET    TRC_SCHED_CLASS_EVT(RTDS, 5)
+#define TRC_RTDS_SCHEDULE         TRC_SCHED_CLASS_EVT(RTDS, 6)
 
 static void repl_timer_handler(void *data);
 
@@ -1035,6 +1036,20 @@ rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_sched
     struct rt_vcpu *snext = NULL;
     struct task_slice ret = { .migrated = 0 };
 
+    /* TRACE */
+    {
+        struct __packed {
+            unsigned cpu:16, tasklet:8, tickled:4, idle:4;
+        } d;
+        d.cpu = cpu;
+        d.tasklet = tasklet_work_scheduled;
+        d.tickled = cpumask_test_cpu(cpu, &prv->tickled);
+        d.idle = is_idle_vcpu(current);
+        trace_var(TRC_RTDS_SCHEDULE, 1,
+                  sizeof(d),
+                  (unsigned char *)&d);
+    }
+
     /* clear ticked bit now that we've been scheduled */
     cpumask_clear_cpu(cpu, &prv->tickled);
 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 1/2] xen: credit2: implement yield()

[George Dunlap]

On 30/09/16 15:21, Dario Faggioli wrote:
> When a vcpu explicitly yields it is usually giving
> us an advice of "let someone else run and come back
> to me in a bit."
> 
> Credit2 isn't, so far, doing anything when a vcpu
> yields, which means an yield is basically a NOP (well,
> actually, it's pure overhead, as it causes the scheduler
> kick in, but the result is --at least 99% of the time--
> that the very same vcpu that yielded continues to run).
> 
> With this patch, when a vcpu yields, we go and try
> picking the next vcpu on the runqueue that can run on
> the pcpu where the yielding vcpu is running. Of course,
> if we don't find any other vcpu that wants and can run
> there, the yielding vcpu will continue.
> 
> Also, add an yield performance counter, and fix the
> style of a couple of comments.
> 
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>

Thanks!

Reviewed-by: George Dunlap <george.dunlap@citrix.com>

> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> Cc: Anshul Makkar <anshul.makkar@citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> Changes from v2:
>  * implement yield as a "switch", rather than as a "knob", as suggested
>    during review.
> 
> Changes from v1:
>  * add _us to the parameter name, as suggested during review;
>  * get rid of the minimum value for the yield bias;
>  * apply the idle bias via subtraction of credits to the yielding vcpu, rather
>    than via addition to all the others;
>  * merge the Credit2 bits of what was patch 7 here, as suggested during review.
> ---
>  xen/common/sched_credit2.c   |   54 +++++++++++++++++++++++++++++++-----------
>  xen/common/schedule.c        |    2 ++
>  xen/include/xen/perfc_defn.h |    1 +
>  3 files changed, 43 insertions(+), 14 deletions(-)
> 
> diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
> index 72e31b5..c0646e9 100644
> --- a/xen/common/sched_credit2.c
> +++ b/xen/common/sched_credit2.c
> @@ -181,7 +181,13 @@
>   */
>  #define __CSFLAG_runq_migrate_request 3
>  #define CSFLAG_runq_migrate_request (1<<__CSFLAG_runq_migrate_request)
> -
> +/*
> + * CSFLAG_vcpu_yield: this vcpu was running, and has called vcpu_yield(). The
> + * scheduler is invoked to see if we can give the cpu to someone else, and
> + * get back to the yielding vcpu in a while.
> + */
> +#define __CSFLAG_vcpu_yield 4
> +#define CSFLAG_vcpu_yield (1<<__CSFLAG_vcpu_yield)
>  
>  static unsigned int __read_mostly opt_migrate_resist = 500;
>  integer_param("sched_credit2_migrate_resist", opt_migrate_resist);
> @@ -1431,6 +1437,14 @@ out:
>  }
>  
>  static void
> +csched2_vcpu_yield(const struct scheduler *ops, struct vcpu *v)
> +{
> +    struct csched2_vcpu * const svc = CSCHED2_VCPU(v);
> +
> +    __set_bit(__CSFLAG_vcpu_yield, &svc->flags);
> +}
> +
> +static void
>  csched2_context_saved(const struct scheduler *ops, struct vcpu *vc)
>  {
>      struct csched2_vcpu * const svc = CSCHED2_VCPU(vc);
> @@ -2250,26 +2264,31 @@ runq_candidate(struct csched2_runqueue_data *rqd,
>      struct list_head *iter;
>      struct csched2_vcpu *snext = NULL;
>      struct csched2_private *prv = CSCHED2_PRIV(per_cpu(scheduler, cpu));
> +    bool yield = __test_and_clear_bit(__CSFLAG_vcpu_yield, &scurr->flags);
>  
>      *skipped = 0;
>  
> -    /* Default to current if runnable, idle otherwise */
> -    if ( vcpu_runnable(scurr->vcpu) )
> -        snext = scurr;
> -    else
> -        snext = CSCHED2_VCPU(idle_vcpu[cpu]);
> -
>      /*
>       * Return the current vcpu if it has executed for less than ratelimit.
>       * Adjuststment for the selected vcpu's credit and decision
>       * for how long it will run will be taken in csched2_runtime.
> +     *
> +     * Note that, if scurr is yielding, we don't let rate limiting kick in.
> +     * In fact, it may be the case that scurr is about to spin, and there's
> +     * no point forcing it to do so until rate limiting expires.
>       */
> -    if ( prv->ratelimit_us && !is_idle_vcpu(scurr->vcpu) &&
> +    if ( !yield && prv->ratelimit_us && !is_idle_vcpu(scurr->vcpu) &&
>           vcpu_runnable(scurr->vcpu) &&
>           (now - scurr->vcpu->runstate.state_entry_time) <
>            MICROSECS(prv->ratelimit_us) )
>          return scurr;
>  
> +    /* Default to current if runnable, idle otherwise */
> +    if ( vcpu_runnable(scurr->vcpu) )
> +        snext = scurr;
> +    else
> +        snext = CSCHED2_VCPU(idle_vcpu[cpu]);
> +
>      list_for_each( iter, &rqd->runq )
>      {
>          struct csched2_vcpu * svc = list_entry(iter, struct csched2_vcpu, runq_elem);
> @@ -2293,8 +2312,10 @@ runq_candidate(struct csched2_runqueue_data *rqd,
>              continue;
>          }
>  
> -        /* If this is on a different processor, don't pull it unless
> -         * its credit is at least CSCHED2_MIGRATE_RESIST higher. */
> +        /*
> +         * If this is on a different processor, don't pull it unless
> +         * its credit is at least CSCHED2_MIGRATE_RESIST higher.
> +         */
>          if ( svc->vcpu->processor != cpu
>               && snext->credit + CSCHED2_MIGRATE_RESIST > svc->credit )
>          {
> @@ -2303,9 +2324,12 @@ runq_candidate(struct csched2_runqueue_data *rqd,
>              continue;
>          }
>  
> -        /* If the next one on the list has more credit than current
> -         * (or idle, if current is not runnable), choose it. */
> -        if ( svc->credit > snext->credit )
> +        /*
> +         * If the next one on the list has more credit than current
> +         * (or idle, if current is not runnable), or if current is
> +         * yielding, choose it.
> +         */
> +        if ( yield || svc->credit > snext->credit )
>              snext = svc;
>  
>          /* In any case, if we got this far, break. */
> @@ -2391,7 +2415,8 @@ csched2_schedule(
>       */
>      if ( tasklet_work_scheduled )
>      {
> -        trace_var(TRC_CSCHED2_SCHED_TASKLET, 1, 0,  NULL);
> +        __clear_bit(__CSFLAG_vcpu_yield, &scurr->flags);
> +        trace_var(TRC_CSCHED2_SCHED_TASKLET, 1, 0, NULL);
>          snext = CSCHED2_VCPU(idle_vcpu[cpu]);
>      }
>      else
> @@ -2975,6 +3000,7 @@ static const struct scheduler sched_credit2_def = {
>  
>      .sleep          = csched2_vcpu_sleep,
>      .wake           = csched2_vcpu_wake,
> +    .yield          = csched2_vcpu_yield,
>  
>      .adjust         = csched2_dom_cntl,
>      .adjust_global  = csched2_sys_cntl,
> diff --git a/xen/common/schedule.c b/xen/common/schedule.c
> index 104d203..5b444c4 100644
> --- a/xen/common/schedule.c
> +++ b/xen/common/schedule.c
> @@ -947,6 +947,8 @@ long vcpu_yield(void)
>      SCHED_OP(VCPU2OP(v), yield, v);
>      vcpu_schedule_unlock_irq(lock, v);
>  
> +    SCHED_STAT_CRANK(vcpu_yield);
> +
>      TRACE_2D(TRC_SCHED_YIELD, current->domain->domain_id, current->vcpu_id);
>      raise_softirq(SCHEDULE_SOFTIRQ);
>      return 0;
> diff --git a/xen/include/xen/perfc_defn.h b/xen/include/xen/perfc_defn.h
> index 4a835b8..900fddd 100644
> --- a/xen/include/xen/perfc_defn.h
> +++ b/xen/include/xen/perfc_defn.h
> @@ -23,6 +23,7 @@ PERFCOUNTER(vcpu_alloc,             "sched: vcpu_alloc")
>  PERFCOUNTER(vcpu_insert,            "sched: vcpu_insert")
>  PERFCOUNTER(vcpu_remove,            "sched: vcpu_remove")
>  PERFCOUNTER(vcpu_sleep,             "sched: vcpu_sleep")
> +PERFCOUNTER(vcpu_yield,             "sched: vcpu_yield")
>  PERFCOUNTER(vcpu_wake_running,      "sched: vcpu_wake_running")
>  PERFCOUNTER(vcpu_wake_onrunq,       "sched: vcpu_wake_onrunq")
>  PERFCOUNTER(vcpu_wake_runnable,     "sched: vcpu_wake_runnable")
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 0/2] sched: Credit1 and Credit2 improvements... but *NO* soft-affinity for Credit2!

[George Dunlap]

On 30/09/16 15:21, Dario Faggioli wrote:
> And here they are, the last two remaining patches...
> 
> Dario
> ---
> Dario Faggioli (2):
>       xen: credit2: implement yield()
>       xen: tracing: add trace records for schedule and rate-limiting.

Applied, thanks.

 -George

> 
>  xen/common/sched_credit.c    |   32 +++++++++++++++
>  xen/common/sched_credit2.c   |   92 +++++++++++++++++++++++++++++++++++-------
>  xen/common/sched_rt.c        |   15 +++++++
>  xen/common/schedule.c        |    2 +
>  xen/include/xen/perfc_defn.h |    1 
>  5 files changed, 127 insertions(+), 15 deletions(-)
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 2/2] xen: tracing: add trace records for schedule and rate-limiting.

[Meng Xu]

On Fri, Sep 30, 2016 at 10:21 AM, Dario Faggioli
<dario.faggioli@citrix.com> wrote:
> As far as {csched, csched2, rt}_schedule() are concerned,
> an "empty" event, would already make it easier to read and
> understand a trace.
>
> But while there, add a few useful information, like
> if the cpu that is going through the scheduler has
> been tickled or not, if it is currently idle, etc
> (they vary, on a per-scheduler basis).
>
> For Credit1 and Credit2, add a record about when
> rate-limiting kicks in too.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> Cc: Meng Xu <mengxu@cis.upenn.edu>
> Cc: Anshul Makkar <anshul.makkar@citrix.com>
> ---
> Changes from v1:
>  * corrected the schedule record for sched_rt.c, as pointed out during review;
>  * pack the Credit1 records as well, as requested during review.
> ---
>  xen/common/sched_credit.c  |   32 ++++++++++++++++++++++++++++++++
>  xen/common/sched_credit2.c |   38 +++++++++++++++++++++++++++++++++++++-
>  xen/common/sched_rt.c      |   15 +++++++++++++++
>  3 files changed, 84 insertions(+), 1 deletion(-)

As to xen/common/sched_rt.c,
Reviewed-by: Meng Xu <mengxu@cis.upenn.edu>

Thanks,

Meng

-- 
------------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel