* [PATCH v4] xen: credit1: avoid boosting vCPUs being "just" migrated
@ 2016-02-24 17:42 Dario Faggioli
2016-02-29 14:34 ` George Dunlap
0 siblings, 1 reply; 2+ messages in thread
From: Dario Faggioli @ 2016-02-24 17:42 UTC (permalink / raw)
To: xen-devel; +Cc: George Dunlap, Jan Beulich
Moving a vCPU to a different pCPU means offlining it and
then waking it up, on the new pCPU. Credit1 grants BOOST
priority to vCPUs that wakes up, with the aim of improving
I/O latency. The net effect of this all is that vCPUs get
boosted when migrating, which shouldn't happen.
For instance, this causes scheduling anomalies and,
potentially, performance problems, as reported here:
http://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html
This patch fixes this by noting down (by means of a flag)
the fact that the vCPU is about to undergo a migration.
This way we can tell, later, during a wakeup, whether the
vCPU is migrating or unblocking, and decide whether or
not to apply the boosting.
Note that it is important that atomic-safe bit operations
are used when manipulating vCPUs' flags. Take the chance
and add a comment about this.
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
Cc: George Dunlap <george.dunlap@eu.citrix.com>
Cc: Jan Beulich <JBeulich@suse.com>
---
Changes from v3:
* use a local boolean variable to track if the vcpu is migrating,
to make the code clearer and easier to read, as suggested during
review.
Changes from v2:
* test_and_clear() is necessary when accessing svc->flags;
* added a comment about such need at the top, where the flags
are defined.
Changes from v1:
* rewritten, following suggestion got during review: there
are no wakeup flags any longer, and all is done in sched_credit.c
by setting a flag in csched_cpu_pick() and testing (and
cleating) it in csched_vcpu_wake().
---
xen/common/sched_credit.c | 28 ++++++++++++++++++++++++----
1 file changed, 24 insertions(+), 4 deletions(-)
diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
index 0af5f69..305889a 100644
--- a/xen/common/sched_credit.c
+++ b/xen/common/sched_credit.c
@@ -63,9 +63,14 @@
/*
* Flags
+ *
+ * Note that svc->flags (where these flags live) is protected by an
+ * inconsistent set of locks. Therefore atomic-safe bit operations must
+ * be used for accessing it.
*/
#define CSCHED_FLAG_VCPU_PARKED 0x0 /* VCPU over capped credits */
#define CSCHED_FLAG_VCPU_YIELD 0x1 /* VCPU yielding */
+#define CSCHED_FLAG_VCPU_MIGRATING 0x2 /* VCPU may have moved to a new pcpu */
/*
@@ -786,6 +791,16 @@ _csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc, bool_t commit)
static int
csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc)
{
+ struct csched_vcpu *svc = CSCHED_VCPU(vc);
+
+ /*
+ * We have been called by vcpu_migrate() (in schedule.c), as part
+ * of the process of seeing if vc can be migrated to another pcpu.
+ * We make a note about this in svc->flags so that later, in
+ * csched_vcpu_wake() (still called from vcpu_migrate()) we won't
+ * get boosted, which we don't deserve as we are "only" migrating.
+ */
+ set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags);
return _csched_cpu_pick(ops, vc, 1);
}
@@ -985,6 +1000,7 @@ static void
csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc)
{
struct csched_vcpu * const svc = CSCHED_VCPU(vc);
+ bool_t migrating;
BUG_ON( is_idle_vcpu(vc) );
@@ -1020,11 +1036,15 @@ csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc)
* more CPU resource intensive VCPUs without impacting overall
* system fairness.
*
- * The one exception is for VCPUs of capped domains unpausing
- * after earning credits they had overspent. We don't boost
- * those.
+ * There are two cases, when we don't want to boost:
+ * - VCPUs that are waking up after a migration, rather than
+ * after having block;
+ * - VCPUs of capped domains unpausing after earning credits
+ * they had overspent.
*/
- if ( svc->pri == CSCHED_PRI_TS_UNDER &&
+ migrating = test_and_clear_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags);
+
+ if ( !migrating && svc->pri == CSCHED_PRI_TS_UNDER &&
!test_bit(CSCHED_FLAG_VCPU_PARKED, &svc->flags) )
{
TRACE_2D(TRC_CSCHED_BOOST_START, vc->domain->domain_id, vc->vcpu_id);
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH v4] xen: credit1: avoid boosting vCPUs being "just" migrated
2016-02-24 17:42 [PATCH v4] xen: credit1: avoid boosting vCPUs being "just" migrated Dario Faggioli
@ 2016-02-29 14:34 ` George Dunlap
0 siblings, 0 replies; 2+ messages in thread
From: George Dunlap @ 2016-02-29 14:34 UTC (permalink / raw)
To: Dario Faggioli, xen-devel; +Cc: George Dunlap, Jan Beulich
On 24/02/16 17:42, Dario Faggioli wrote:
> Moving a vCPU to a different pCPU means offlining it and
> then waking it up, on the new pCPU. Credit1 grants BOOST
> priority to vCPUs that wakes up, with the aim of improving
> I/O latency. The net effect of this all is that vCPUs get
> boosted when migrating, which shouldn't happen.
>
> For instance, this causes scheduling anomalies and,
> potentially, performance problems, as reported here:
> http://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html
>
> This patch fixes this by noting down (by means of a flag)
> the fact that the vCPU is about to undergo a migration.
> This way we can tell, later, during a wakeup, whether the
> vCPU is migrating or unblocking, and decide whether or
> not to apply the boosting.
>
> Note that it is important that atomic-safe bit operations
> are used when manipulating vCPUs' flags. Take the chance
> and add a comment about this.
>
> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
Reviewed-by: George Dunlap <george.dunlap@citrix.com>
Thanks, sorry for the delay. :-)
> ---
> Cc: George Dunlap <george.dunlap@eu.citrix.com>
> Cc: Jan Beulich <JBeulich@suse.com>
> ---
> Changes from v3:
> * use a local boolean variable to track if the vcpu is migrating,
> to make the code clearer and easier to read, as suggested during
> review.
>
> Changes from v2:
> * test_and_clear() is necessary when accessing svc->flags;
> * added a comment about such need at the top, where the flags
> are defined.
>
> Changes from v1:
> * rewritten, following suggestion got during review: there
> are no wakeup flags any longer, and all is done in sched_credit.c
> by setting a flag in csched_cpu_pick() and testing (and
> cleating) it in csched_vcpu_wake().
> ---
> xen/common/sched_credit.c | 28 ++++++++++++++++++++++++----
> 1 file changed, 24 insertions(+), 4 deletions(-)
>
> diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c
> index 0af5f69..305889a 100644
> --- a/xen/common/sched_credit.c
> +++ b/xen/common/sched_credit.c
> @@ -63,9 +63,14 @@
>
> /*
> * Flags
> + *
> + * Note that svc->flags (where these flags live) is protected by an
> + * inconsistent set of locks. Therefore atomic-safe bit operations must
> + * be used for accessing it.
> */
> #define CSCHED_FLAG_VCPU_PARKED 0x0 /* VCPU over capped credits */
> #define CSCHED_FLAG_VCPU_YIELD 0x1 /* VCPU yielding */
> +#define CSCHED_FLAG_VCPU_MIGRATING 0x2 /* VCPU may have moved to a new pcpu */
>
>
> /*
> @@ -786,6 +791,16 @@ _csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc, bool_t commit)
> static int
> csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc)
> {
> + struct csched_vcpu *svc = CSCHED_VCPU(vc);
> +
> + /*
> + * We have been called by vcpu_migrate() (in schedule.c), as part
> + * of the process of seeing if vc can be migrated to another pcpu.
> + * We make a note about this in svc->flags so that later, in
> + * csched_vcpu_wake() (still called from vcpu_migrate()) we won't
> + * get boosted, which we don't deserve as we are "only" migrating.
> + */
> + set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags);
> return _csched_cpu_pick(ops, vc, 1);
> }
>
> @@ -985,6 +1000,7 @@ static void
> csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc)
> {
> struct csched_vcpu * const svc = CSCHED_VCPU(vc);
> + bool_t migrating;
>
> BUG_ON( is_idle_vcpu(vc) );
>
> @@ -1020,11 +1036,15 @@ csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc)
> * more CPU resource intensive VCPUs without impacting overall
> * system fairness.
> *
> - * The one exception is for VCPUs of capped domains unpausing
> - * after earning credits they had overspent. We don't boost
> - * those.
> + * There are two cases, when we don't want to boost:
> + * - VCPUs that are waking up after a migration, rather than
> + * after having block;
> + * - VCPUs of capped domains unpausing after earning credits
> + * they had overspent.
> */
> - if ( svc->pri == CSCHED_PRI_TS_UNDER &&
> + migrating = test_and_clear_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags);
> +
> + if ( !migrating && svc->pri == CSCHED_PRI_TS_UNDER &&
> !test_bit(CSCHED_FLAG_VCPU_PARKED, &svc->flags) )
> {
> TRACE_2D(TRC_CSCHED_BOOST_START, vc->domain->domain_id, vc->vcpu_id);
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2016-02-29 14:34 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-24 17:42 [PATCH v4] xen: credit1: avoid boosting vCPUs being "just" migrated Dario Faggioli
2016-02-29 14:34 ` George Dunlap
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.