* [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel
Cc: Lars Kurth, Julien Grall, Stefano Stabellini, Ian Jackson, committers
Sometimes we find ourselves seriously lacking the capacity to run
particular job(s). The result can be that the whole system stands
mostly idle while a small proportion of the resources runs flat out
with a giant queue.
In this series we arrange for osstest to be able to spot this
happening, and automatically rebalance load by give up earlier on the
jobs which are overly-contended.
There are some tuning parameters, of course. To summarise, I have
chosen here to treat jobs as starved if (for example):
We have completed 90% of the flight, and the remaining 10%
is projected to take 5x as long as the first 90%.
(The "90%" is by number of jobs.) See the patch
starvation: Infrastructure for jobs which are delaying their flights
for details of the heuristic and its parameters.
When situations like this persist it will still be good to manually
balance the load by adjusting the job mix in submitted flights. This
is because the starvation will not necessarily drop the same job in
subsequent flights on the same "branch", so starvation will impair the
regression detection.
Ian Jackson (21):
ts-hosts-allocate-Executive: with -U, just append to the same logfile
selecthost: Honour new $none_ok optional parameter
ts-logs-capture: Do not try to capture logs of hosts not allocated
alloc_resources: Support special abandonment values
starvation: Teach sg-report-flight about starved step state
starvation: Teach archaeologists about starved job state
starvation: Teach ms-flights-summary about job state starved
starvation: Teach sg-execute-flight about job state starved
step handling: Preserve step states set by ts-* scripts
TestSupport: Make "broken" print the actual job state
JobDB::Executive: step_*: fix log messages to talk about "steps"
starvation: Permit step_finish to set the state `starved'
TestSupport: Make "broken" set the step state too
tcl/JobDB-Executive: Do not squash "starved" status
starvation: Propagate starved job status into dependent jobs
ts-host-allocate-Executive: Break out $now and add a newline
starvation: Use "starved" for hostalloc_maxwait_max
starvation: Infrastructure for jobs which are delaying their flights
starvation: Abandon jobs which are unreasonably delaying their flight
hostalloc_maxwait_max: Use starvation most_optimistic
starvation: Better logging/debugging output
Osstest/Executive.pm | 95 ++++++++++++++++++++++++++---
Osstest/JobDB/Executive.pm | 8 ++-
Osstest/TestSupport.pm | 24 ++++++--
mg-hostalloc-starvation-demo | 53 ++++++++++++++++
ms-flights-summary | 9 +--
sg-execute-flight | 2 +-
sg-report-flight | 17 +++++-
tcl/JobDB-Executive.tcl | 6 +-
ts-hosts-allocate-Executive | 142 ++++++++++++++++++++++++++++++++++++++++---
ts-logs-capture | 7 ++-
10 files changed, 328 insertions(+), 35 deletions(-)
create mode 100755 mg-hostalloc-starvation-demo
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel
Cc: Lars Kurth, Julien Grall, Stefano Stabellini, Ian Jackson, committers
Sometimes we find ourselves seriously lacking the capacity to run
particular job(s). The result can be that the whole system stands
mostly idle while a small proportion of the resources runs flat out
with a giant queue.
In this series we arrange for osstest to be able to spot this
happening, and automatically rebalance load by give up earlier on the
jobs which are overly-contended.
There are some tuning parameters, of course. To summarise, I have
chosen here to treat jobs as starved if (for example):
We have completed 90% of the flight, and the remaining 10%
is projected to take 5x as long as the first 90%.
(The "90%" is by number of jobs.) See the patch
starvation: Infrastructure for jobs which are delaying their flights
for details of the heuristic and its parameters.
When situations like this persist it will still be good to manually
balance the load by adjusting the job mix in submitted flights. This
is because the starvation will not necessarily drop the same job in
subsequent flights on the same "branch", so starvation will impair the
regression detection.
Ian Jackson (21):
ts-hosts-allocate-Executive: with -U, just append to the same logfile
selecthost: Honour new $none_ok optional parameter
ts-logs-capture: Do not try to capture logs of hosts not allocated
alloc_resources: Support special abandonment values
starvation: Teach sg-report-flight about starved step state
starvation: Teach archaeologists about starved job state
starvation: Teach ms-flights-summary about job state starved
starvation: Teach sg-execute-flight about job state starved
step handling: Preserve step states set by ts-* scripts
TestSupport: Make "broken" print the actual job state
JobDB::Executive: step_*: fix log messages to talk about "steps"
starvation: Permit step_finish to set the state `starved'
TestSupport: Make "broken" set the step state too
tcl/JobDB-Executive: Do not squash "starved" status
starvation: Propagate starved job status into dependent jobs
ts-host-allocate-Executive: Break out $now and add a newline
starvation: Use "starved" for hostalloc_maxwait_max
starvation: Infrastructure for jobs which are delaying their flights
starvation: Abandon jobs which are unreasonably delaying their flight
hostalloc_maxwait_max: Use starvation most_optimistic
starvation: Better logging/debugging output
Osstest/Executive.pm | 95 ++++++++++++++++++++++++++---
Osstest/JobDB/Executive.pm | 8 ++-
Osstest/TestSupport.pm | 24 ++++++--
mg-hostalloc-starvation-demo | 53 ++++++++++++++++
ms-flights-summary | 9 +--
sg-execute-flight | 2 +-
sg-report-flight | 17 +++++-
tcl/JobDB-Executive.tcl | 6 +-
ts-hosts-allocate-Executive | 142 ++++++++++++++++++++++++++++++++++++++++---
ts-logs-capture | 7 ++-
10 files changed, 328 insertions(+), 35 deletions(-)
create mode 100755 mg-hostalloc-starvation-demo
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 01/21] ts-hosts-allocate-Executive: with -U, just append to the same logfile
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This is much more convenient for ad-hoc use.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index dcfc70f2..3da138b1 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -75,16 +75,18 @@ sub setup () {
$taskid= findtask();
- my $logbase = "hosts-allocate.debug".($compressdebug?".gz":"");
- my $logfh = open_unique_stashfile \$logbase;
+ my $logbase = "hosts-allocate.debug";
if ($compressdebug) {
+ $logbase .= ".gz";
+ my $logfh = open_unique_stashfile \$logbase;
my $logchild = open DEBUG, "|-"; defined $logchild or die $!;
if (!$logchild) {
open STDOUT, ">&", $logfh or die $!;
exec "gzip" or die $!;
}
} else {
- open DEBUG, ">&", $logfh or die $!;
+ my $logfh = "$stash/$logbase";
+ open DEBUG, ">>", $logfh or die $!;
}
DEBUG->autoflush(1);
logm("host allocation debug log in $logbase");
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 01/21] ts-hosts-allocate-Executive: with -U, just append to the same logfile
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This is much more convenient for ad-hoc use.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index dcfc70f2..3da138b1 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -75,16 +75,18 @@ sub setup () {
$taskid= findtask();
- my $logbase = "hosts-allocate.debug".($compressdebug?".gz":"");
- my $logfh = open_unique_stashfile \$logbase;
+ my $logbase = "hosts-allocate.debug";
if ($compressdebug) {
+ $logbase .= ".gz";
+ my $logfh = open_unique_stashfile \$logbase;
my $logchild = open DEBUG, "|-"; defined $logchild or die $!;
if (!$logchild) {
open STDOUT, ">&", $logfh or die $!;
exec "gzip" or die $!;
}
} else {
- open DEBUG, ">&", $logfh or die $!;
+ my $logfh = "$stash/$logbase";
+ open DEBUG, ">>", $logfh or die $!;
}
DEBUG->autoflush(1);
logm("host allocation debug log in $logbase");
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 02/21] selecthost: Honour new $none_ok optional parameter
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
No functional change with existing callers.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/TestSupport.pm | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ec867e4f..38c17d60 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1125,9 +1125,9 @@ sub power_state ($$;$) {
#---------- host selection and properties ----------
-sub selecthost ($);
-sub selecthost ($) {
- my ($ident) = @_;
+sub selecthost ($;$);
+sub selecthost ($;$) {
+ my ($ident, $none_ok) = @_;
# must be run outside transaction
# $ident is <identspec>
@@ -1154,7 +1154,10 @@ sub selecthost ($) {
$r{$ident}= $name;
} else {
$name= $r{$ident};
- die "no specified $ident" unless defined $name;
+ if (!defined $name) {
+ return undef if $none_ok;
+ die "no specified $ident";
+ }
}
my $ho= {
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 02/21] selecthost: Honour new $none_ok optional parameter
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
No functional change with existing callers.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/TestSupport.pm | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ec867e4f..38c17d60 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1125,9 +1125,9 @@ sub power_state ($$;$) {
#---------- host selection and properties ----------
-sub selecthost ($);
-sub selecthost ($) {
- my ($ident) = @_;
+sub selecthost ($;$);
+sub selecthost ($;$) {
+ my ($ident, $none_ok) = @_;
# must be run outside transaction
# $ident is <identspec>
@@ -1154,7 +1154,10 @@ sub selecthost ($) {
$r{$ident}= $name;
} else {
$name= $r{$ident};
- die "no specified $ident" unless defined $name;
+ if (!defined $name) {
+ return undef if $none_ok;
+ die "no specified $ident";
+ }
}
my $ho= {
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 03/21] ts-logs-capture: Do not try to capture logs of hosts not allocated
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Little significant change with existing non-broken flights etc.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ts-logs-capture | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/ts-logs-capture b/ts-logs-capture
index a429bb76..83234f6f 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -32,7 +32,12 @@ if (!$mjobdb->jobdb_enable_log_capture) {
our ($whhost) = @ARGV;
$whhost ||= 'host';
-our $ho= selecthost($whhost);
+
+our $ho= selecthost($whhost, 1);
+if (!$ho) {
+ logm("host $whhost not specified, not capturing logs");
+ exit 0;
+}
our @guests;
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 03/21] ts-logs-capture: Do not try to capture logs of hosts not allocated
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Little significant change with existing non-broken flights etc.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ts-logs-capture | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/ts-logs-capture b/ts-logs-capture
index a429bb76..83234f6f 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -32,7 +32,12 @@ if (!$mjobdb->jobdb_enable_log_capture) {
our ($whhost) = @ARGV;
$whhost ||= 'host';
-our $ho= selecthost($whhost);
+
+our $ho= selecthost($whhost, 1);
+if (!$ho) {
+ logm("host $whhost not specified, not capturing logs");
+ exit 0;
+}
our @guests;
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 04/21] alloc_resources: Support special abandonment values
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This gives a way for the caller's $resourcecall to signal something
interesting, back to its main loop. This is useful for calling
broken, for example: that can't be done within $resourcecall because
$resourcecall operates within the allocation db transaction (which
ought to be rolled back...)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/Executive.pm | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 0d8502b5..b27b8d25 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -737,7 +737,8 @@ sub alloc_resources {
#
# values of $ok
# 0 rollback, wait and try again
- # 1 commit, completed ok
+ # 1 commit, completed ok } return the
+ # >1 rollback and abandon } value of $ok
# $bookinglist should be undef or a hash for making a booking
#
# $resourcecall should not look at tasks.live
@@ -859,10 +860,14 @@ sub alloc_resources {
# $resourcecall itself failed: bomb out (don't retry)
$ok=-1;
}
- return db_retry_abort() unless $ok>0;
+ return db_retry_abort() unless $ok==1;
}]);
- if ($bookinglist && $ok!=-1) {
+ if ($ok<0 || $ok>1) {
+ return 1;
+ }
+
+ if ($bookinglist) {
my %prstart;
foreach my $book (@{ $bookinglist->{Bookings} }) {
my $pr = $book->{Reso};
@@ -890,8 +895,6 @@ sub alloc_resources {
if ($ok==1) {
print $qserv "thought-done\n" or die $!;
- } elsif ($ok<0) {
- return 1;
} else { # 0
logm("resource allocation: deferring");
print $qserv "thought-wait\n" or die $!;
@@ -935,8 +938,12 @@ END
$ok= 0;
}
}
- die unless $ok==1;
- logm("resource allocation: successful.");
+ if ($ok==1) {
+ logm("resource allocation: successful.");
+ } else {
+ logm("resource allocation: resourcecall returned ($ok,...)");
+ }
+ return $ok;
}
sub resource_check_allocated ($$) {
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 04/21] alloc_resources: Support special abandonment values
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This gives a way for the caller's $resourcecall to signal something
interesting, back to its main loop. This is useful for calling
broken, for example: that can't be done within $resourcecall because
$resourcecall operates within the allocation db transaction (which
ought to be rolled back...)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/Executive.pm | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 0d8502b5..b27b8d25 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -737,7 +737,8 @@ sub alloc_resources {
#
# values of $ok
# 0 rollback, wait and try again
- # 1 commit, completed ok
+ # 1 commit, completed ok } return the
+ # >1 rollback and abandon } value of $ok
# $bookinglist should be undef or a hash for making a booking
#
# $resourcecall should not look at tasks.live
@@ -859,10 +860,14 @@ sub alloc_resources {
# $resourcecall itself failed: bomb out (don't retry)
$ok=-1;
}
- return db_retry_abort() unless $ok>0;
+ return db_retry_abort() unless $ok==1;
}]);
- if ($bookinglist && $ok!=-1) {
+ if ($ok<0 || $ok>1) {
+ return 1;
+ }
+
+ if ($bookinglist) {
my %prstart;
foreach my $book (@{ $bookinglist->{Bookings} }) {
my $pr = $book->{Reso};
@@ -890,8 +895,6 @@ sub alloc_resources {
if ($ok==1) {
print $qserv "thought-done\n" or die $!;
- } elsif ($ok<0) {
- return 1;
} else { # 0
logm("resource allocation: deferring");
print $qserv "thought-wait\n" or die $!;
@@ -935,8 +938,12 @@ END
$ok= 0;
}
}
- die unless $ok==1;
- logm("resource allocation: successful.");
+ if ($ok==1) {
+ logm("resource allocation: successful.");
+ } else {
+ logm("resource allocation: resourcecall returned ($ok,...)");
+ }
+ return $ok;
}
sub resource_check_allocated ($$) {
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 05/21] starvation: Teach sg-report-flight about starved step state
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We are going to use this for situations where the resources to run the
test weren't available. In general we are going to treat this as not
a regression.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-flight | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sg-report-flight b/sg-report-flight
index a1adf952..2677c479 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -805,9 +805,9 @@ END
next;
}
- if ($st eq 'blocked') {
- print MRO "blocked $s->{job} $s->{testid}\n";
- print DEBUG " blocked, call it justifiable\n";
+ if ($st eq 'blocked' or $st eq 'starved') {
+ print MRO "$st $s->{job} $s->{testid}\n";
+ print DEBUG " $st, call it justifiable\n";
$failv->{Summary}= 'n/a';
$failv->{Blocker}= '';
next;
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 05/21] starvation: Teach sg-report-flight about starved step state
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We are going to use this for situations where the resources to run the
test weren't available. In general we are going to treat this as not
a regression.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-flight | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sg-report-flight b/sg-report-flight
index a1adf952..2677c479 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -805,9 +805,9 @@ END
next;
}
- if ($st eq 'blocked') {
- print MRO "blocked $s->{job} $s->{testid}\n";
- print DEBUG " blocked, call it justifiable\n";
+ if ($st eq 'blocked' or $st eq 'starved') {
+ print MRO "$st $s->{job} $s->{testid}\n";
+ print DEBUG " $st, call it justifiable\n";
$failv->{Summary}= 'n/a';
$failv->{Blocker}= '';
next;
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 06/21] starvation: Teach archaeologists about starved job state
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
sg-report-flight is a bit awkward. It thinks mostly about step
status, not job status. So, when justifying, if we don't find a step,
and the job state is starved, we treat the step as starved.
If there are only starved steps, then we don't have evidence that this
is a regression, because the test wasn't run in the baseline.
If there are other steps we look at those instead.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/Executive.pm | 4 ++--
sg-report-flight | 11 +++++++++++
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index b27b8d25..a9f9ac78 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -362,8 +362,8 @@ END
Content => (join " | ", @content),
ColourAttr => "bgcolor=\"$failcolour\"",
};
- } elsif ($status eq 'blocked') {
- return $single->("blocked", $purple),
+ } elsif ($status eq 'blocked' or $status eq 'starved') {
+ return $single->($status, $purple),
} elsif ($status eq 'queued') {
return $single->("($status)", $grey_dark),
} elsif ($status eq 'preparing') {
diff --git a/sg-report-flight b/sg-report-flight
index 2677c479..6c481f6f 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -160,6 +160,10 @@ sub findaflight ($$$$$) {
return undef;
}
+ my $jcheckq= db_prepare(<<END);
+ SELECT status FROM jobs WHERE flight=? AND job=?
+END
+
my $checkq= db_prepare(<<END);
SELECT status FROM steps WHERE flight=? AND job=? AND testid=?
AND status!='skip'
@@ -317,6 +321,11 @@ END
if (!defined $whynot && defined $job) {
$checkq->execute($tflight, $job, $testid);
($chkst) = $checkq->fetchrow_array();
+ if (!defined $chkst) {
+ $jcheckq->execute($tflight, $job);
+ my ($jchkst) = $jcheckq->fetchrow_array();
+ $chkst = $jchkst if $jchkst eq 'starved';
+ }
}
$whynot= $sub->($tflight, $chkst) if !defined $whynot;
if (!defined $whynot) {
@@ -861,6 +870,8 @@ END
my $tblocker = $st eq 'fail' ? 'regression' : $st;
my $tpriority = $j->{job} =~ m/^build/ ? 600 : 100;
$rtup= [ $tpriority, $tblocker, "REGR. vs.", $chkflight ];
+ } elsif ($chkst eq 'starved') { # actually, the job was 'starved'
+ $rtup= [ 50, '', "starved in", $chkflight ];
} else {
$rtup= [ 170, '', "$chkst in", $chkflight ];
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 06/21] starvation: Teach archaeologists about starved job state
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
sg-report-flight is a bit awkward. It thinks mostly about step
status, not job status. So, when justifying, if we don't find a step,
and the job state is starved, we treat the step as starved.
If there are only starved steps, then we don't have evidence that this
is a regression, because the test wasn't run in the baseline.
If there are other steps we look at those instead.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/Executive.pm | 4 ++--
sg-report-flight | 11 +++++++++++
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index b27b8d25..a9f9ac78 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -362,8 +362,8 @@ END
Content => (join " | ", @content),
ColourAttr => "bgcolor=\"$failcolour\"",
};
- } elsif ($status eq 'blocked') {
- return $single->("blocked", $purple),
+ } elsif ($status eq 'blocked' or $status eq 'starved') {
+ return $single->($status, $purple),
} elsif ($status eq 'queued') {
return $single->("($status)", $grey_dark),
} elsif ($status eq 'preparing') {
diff --git a/sg-report-flight b/sg-report-flight
index 2677c479..6c481f6f 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -160,6 +160,10 @@ sub findaflight ($$$$$) {
return undef;
}
+ my $jcheckq= db_prepare(<<END);
+ SELECT status FROM jobs WHERE flight=? AND job=?
+END
+
my $checkq= db_prepare(<<END);
SELECT status FROM steps WHERE flight=? AND job=? AND testid=?
AND status!='skip'
@@ -317,6 +321,11 @@ END
if (!defined $whynot && defined $job) {
$checkq->execute($tflight, $job, $testid);
($chkst) = $checkq->fetchrow_array();
+ if (!defined $chkst) {
+ $jcheckq->execute($tflight, $job);
+ my ($jchkst) = $jcheckq->fetchrow_array();
+ $chkst = $jchkst if $jchkst eq 'starved';
+ }
}
$whynot= $sub->($tflight, $chkst) if !defined $whynot;
if (!defined $whynot) {
@@ -861,6 +870,8 @@ END
my $tblocker = $st eq 'fail' ? 'regression' : $st;
my $tpriority = $j->{job} =~ m/^build/ ? 600 : 100;
$rtup= [ $tpriority, $tblocker, "REGR. vs.", $chkflight ];
+ } elsif ($chkst eq 'starved') { # actually, the job was 'starved'
+ $rtup= [ 50, '', "starved in", $chkflight ];
} else {
$rtup= [ 170, '', "$chkst in", $chkflight ];
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 07/21] starvation: Teach ms-flights-summary about job state starved
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This just affects sorting etc. in the summary display.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ms-flights-summary | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/ms-flights-summary b/ms-flights-summary
index 9d15dd14..8293d4f6 100755
--- a/ms-flights-summary
+++ b/ms-flights-summary
@@ -145,10 +145,11 @@ sub sort_stats($) {
preparing => 2,
'prep.alloc.' => 3,
blocked => 4,
- running => 5,
- pass => 6,
- fail => 7,
- broken => 8,
+ starved => 5,
+ running => 6,
+ pass => 7,
+ fail => 8,
+ broken => 9,
);
return sort { ($so{$a}//0) <=> ($so{$b}//0) } (keys %{$stats});
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 07/21] starvation: Teach ms-flights-summary about job state starved
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This just affects sorting etc. in the summary display.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ms-flights-summary | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/ms-flights-summary b/ms-flights-summary
index 9d15dd14..8293d4f6 100755
--- a/ms-flights-summary
+++ b/ms-flights-summary
@@ -145,10 +145,11 @@ sub sort_stats($) {
preparing => 2,
'prep.alloc.' => 3,
blocked => 4,
- running => 5,
- pass => 6,
- fail => 7,
- broken => 8,
+ starved => 5,
+ running => 6,
+ pass => 7,
+ fail => 8,
+ broken => 9,
);
return sort { ($so{$a}//0) <=> ($so{$b}//0) } (keys %{$stats});
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 08/21] starvation: Teach sg-execute-flight about job state starved
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
It needs to not mind if a job reports as `starved', even if sg-run-job
exited non-zero.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-execute-flight | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sg-execute-flight b/sg-execute-flight
index aed9823d..02f63316 100755
--- a/sg-execute-flight
+++ b/sg-execute-flight
@@ -189,7 +189,7 @@ proc main_iteration {} {
if {!$found} { error "job not in database $flight $job" }
switch -exact -- $jobinfo(status) {
- pass - fail - broken - blocked - truncated {
+ pass - fail - broken - blocked - truncated - starved {
}
queued - preparing - running {
jobdb::job-set-status-unlocked $flight $job broken
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 08/21] starvation: Teach sg-execute-flight about job state starved
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
It needs to not mind if a job reports as `starved', even if sg-run-job
exited non-zero.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-execute-flight | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sg-execute-flight b/sg-execute-flight
index aed9823d..02f63316 100755
--- a/sg-execute-flight
+++ b/sg-execute-flight
@@ -189,7 +189,7 @@ proc main_iteration {} {
if {!$found} { error "job not in database $flight $job" }
switch -exact -- $jobinfo(status) {
- pass - fail - broken - blocked - truncated {
+ pass - fail - broken - blocked - truncated - starved {
}
queued - preparing - running {
jobdb::job-set-status-unlocked $flight $job broken
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 09/21] step handling: Preserve step states set by ts-* scripts
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
sg-run-job would unconditionally set the step state to the value it
calculated, which would usually be `pass' or `fail' or
`broken' (according to the recipe).
Relax this interface somewhat to allow a test script to set the step
status itself: specifically, do not overwrite an existing status of
aborted broken starved
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tcl/JobDB-Executive.tcl | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 70728953..57ded025 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -316,11 +316,14 @@ proc spawn-step-commit {flight job stepno testid} {
proc step-set-status {flight job stepno st} {
transaction flights {
- db-update-1 "
+ db-execute "
UPDATE steps
SET status='$st',
finished=[clock seconds]
WHERE flight=$flight AND job='$job' AND stepno=$stepno
+ AND status<>'aborted'
+ AND status<>'broken'
+ AND status<>'starved'
"
set pause 0
db-execute-array stopinfo "
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 09/21] step handling: Preserve step states set by ts-* scripts
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
sg-run-job would unconditionally set the step state to the value it
calculated, which would usually be `pass' or `fail' or
`broken' (according to the recipe).
Relax this interface somewhat to allow a test script to set the step
status itself: specifically, do not overwrite an existing status of
aborted broken starved
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tcl/JobDB-Executive.tcl | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 70728953..57ded025 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -316,11 +316,14 @@ proc spawn-step-commit {flight job stepno testid} {
proc step-set-status {flight job stepno st} {
transaction flights {
- db-update-1 "
+ db-execute "
UPDATE steps
SET status='$st',
finished=[clock seconds]
WHERE flight=$flight AND job='$job' AND stepno=$stepno
+ AND status<>'aborted'
+ AND status<>'broken'
+ AND status<>'starved'
"
set pause 0
db-execute-array stopinfo "
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 10/21] TestSupport: Make "broken" print the actual job state
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
No functional change except to a message.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/TestSupport.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 38c17d60..e218ff35 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -242,7 +242,7 @@ sub broken ($;$) {
AND (status='queued' OR status='running' OR status='preparing')
END
});
- die "BROKEN: $m; ". ($affected>0 ? "marked $flight.$job $newst"
+ die uc($newst).": $m; ". ($affected>0 ? "marked $flight.$job $newst"
: "($flight.$job not marked $newst)");
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 10/21] TestSupport: Make "broken" print the actual job state
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
No functional change except to a message.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/TestSupport.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 38c17d60..e218ff35 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -242,7 +242,7 @@ sub broken ($;$) {
AND (status='queued' OR status='running' OR status='preparing')
END
});
- die "BROKEN: $m; ". ($affected>0 ? "marked $flight.$job $newst"
+ die uc($newst).": $m; ". ($affected>0 ? "marked $flight.$job $newst"
: "($flight.$job not marked $newst)");
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 11/21] JobDB::Executive: step_*: fix log messages to talk about "steps"
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
These functions are about to be sometimes called for non-substep
steps.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/JobDB/Executive.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index ea349052..3a680c9d 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -281,7 +281,7 @@ END
$logfile // "$stepno.$script.log",
$testid,time);
});
- logm("---------- substep $stepno $testid running ----------");
+ logm("---------- step $stepno $testid running ----------");
}
sub step_finish ($$) { #method
@@ -311,7 +311,7 @@ END
unless $oldstatus eq 'running';
$setq->execute($stepstatus,time, $flight,$job,$testid, $stepno);
});
- logm("---------- substep $stepno $testid $stepstatus ----------");
+ logm("---------- step $stepno $testid $stepstatus ----------");
}
sub host_check_allocated ($$) { #method
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 11/21] JobDB::Executive: step_*: fix log messages to talk about "steps"
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
These functions are about to be sometimes called for non-substep
steps.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/JobDB/Executive.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index ea349052..3a680c9d 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -281,7 +281,7 @@ END
$logfile // "$stepno.$script.log",
$testid,time);
});
- logm("---------- substep $stepno $testid running ----------");
+ logm("---------- step $stepno $testid running ----------");
}
sub step_finish ($$) { #method
@@ -311,7 +311,7 @@ END
unless $oldstatus eq 'running';
$setq->execute($stepstatus,time, $flight,$job,$testid, $stepno);
});
- logm("---------- substep $stepno $testid $stepstatus ----------");
+ logm("---------- step $stepno $testid $stepstatus ----------");
}
sub host_check_allocated ($$) { #method
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 12/21] starvation: Permit step_finish to set the state `starved'
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/JobDB/Executive.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 3a680c9d..2e11c8f3 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -287,7 +287,7 @@ END
sub step_finish ($$) { #method
my ($jd,$testid,$stepstatus) = @_;
die "$flight.$job $testid $stepstatus" unless grep { $stepstatus eq $_ }
- qw(pass fail skip blocked broken);
+ qw(pass fail skip blocked broken starved);
my $checkq = $dbh_tests->prepare(<<END);
SELECT stepno, status
FROM steps
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 12/21] starvation: Permit step_finish to set the state `starved'
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/JobDB/Executive.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 3a680c9d..2e11c8f3 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -287,7 +287,7 @@ END
sub step_finish ($$) { #method
my ($jd,$testid,$stepstatus) = @_;
die "$flight.$job $testid $stepstatus" unless grep { $stepstatus eq $_ }
- qw(pass fail skip blocked broken);
+ qw(pass fail skip blocked broken starved);
my $checkq = $dbh_tests->prepare(<<END);
SELECT stepno, status
FROM steps
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 13/21] TestSupport: Make "broken" set the step state too
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/TestSupport.pm | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e218ff35..8e20244d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -235,6 +235,17 @@ sub broken ($;$) {
# must be run outside transaction
my $affected;
$newst= 'broken' unless defined $newst;
+ eval {
+ # This is not always effective, but when we really care is
+ # situations where $newst is not `broken' (eg `starved') and
+ # we are running in Executive mode under sg-run-job, and then
+ # this works. If we are running under sg-run-job, sg-run-job
+ # will set the step status if we don't manage it.
+ my $testid = $ENV{'OSSTEST_TESTID'}
+ || die "OSSTEST_TESTID not set";
+ $mjobdb->step_finish($testid, $newst);
+ };
+ warn "failed to set step status $newst: $@" if $@;
db_retry($flight,'running', $dbh_tests,[qw(flights)], sub {
$affected= $dbh_tests->do(<<END, {}, $newst, $flight, $job);
UPDATE jobs SET status=?
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 13/21] TestSupport: Make "broken" set the step state too
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/TestSupport.pm | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e218ff35..8e20244d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -235,6 +235,17 @@ sub broken ($;$) {
# must be run outside transaction
my $affected;
$newst= 'broken' unless defined $newst;
+ eval {
+ # This is not always effective, but when we really care is
+ # situations where $newst is not `broken' (eg `starved') and
+ # we are running in Executive mode under sg-run-job, and then
+ # this works. If we are running under sg-run-job, sg-run-job
+ # will set the step status if we don't manage it.
+ my $testid = $ENV{'OSSTEST_TESTID'}
+ || die "OSSTEST_TESTID not set";
+ $mjobdb->step_finish($testid, $newst);
+ };
+ warn "failed to set step status $newst: $@" if $@;
db_retry($flight,'running', $dbh_tests,[qw(flights)], sub {
$affected= $dbh_tests->do(<<END, {}, $newst, $flight, $job);
UPDATE jobs SET status=?
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 14/21] tcl/JobDB-Executive: Do not squash "starved" status
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
ts-hosts-allocate is going to set the job status to `starved'
sometimes, and then die. `starved' needs to be added to the list of
job statuses that sg-run-job leaves alone.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tcl/JobDB-Executive.tcl | 1 +
1 file changed, 1 insertion(+)
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 57ded025..4f3409a9 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -55,6 +55,7 @@ proc job-set-status-unlocked {flight job st} {
UPDATE jobs SET status='$st'
WHERE flight=$flight AND job='$job'
AND status<>'aborted' AND status<>'broken'
+ AND status<>'starved'
"
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 14/21] tcl/JobDB-Executive: Do not squash "starved" status
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
ts-hosts-allocate is going to set the job status to `starved'
sometimes, and then die. `starved' needs to be added to the list of
job statuses that sg-run-job leaves alone.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
tcl/JobDB-Executive.tcl | 1 +
1 file changed, 1 insertion(+)
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 57ded025..4f3409a9 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -55,6 +55,7 @@ proc job-set-status-unlocked {flight job st} {
UPDATE jobs SET status='$st'
WHERE flight=$flight AND job='$job'
AND status<>'aborted' AND status<>'broken'
+ AND status<>'starved'
"
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 15/21] starvation: Propagate starved job status into dependent jobs
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
If a build job is starved, then the same status
(No jobs are marked `starved' yet.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/JobDB/Executive.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 2e11c8f3..be5588fc 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -380,6 +380,8 @@ END
} else {
die "job $oflight.$ojob ($why) queued (we are $ourstatus)";
}
+ } elsif ($jstatus eq 'starved') {
+ broken("job $oflight.$ojob ($why) $jstatus", $jstatus);
} else {
broken("job $oflight.$ojob ($why) $jstatus", 'blocked');
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 15/21] starvation: Propagate starved job status into dependent jobs
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
If a build job is starved, then the same status
(No jobs are marked `starved' yet.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/JobDB/Executive.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 2e11c8f3..be5588fc 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -380,6 +380,8 @@ END
} else {
die "job $oflight.$ojob ($why) queued (we are $ourstatus)";
}
+ } elsif ($jstatus eq 'starved') {
+ broken("job $oflight.$ojob ($why) $jstatus", $jstatus);
} else {
broken("job $oflight.$ojob ($why) $jstatus", 'blocked');
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 16/21] ts-host-allocate-Executive: Break out $now and add a newline
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work, no functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 3da138b1..3425c8ce 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -756,12 +756,14 @@ sub attempt_allocation {
logm("host allocation: successful, reporting to planner.");
} else {
logm("host allocation: planned start in $best->{Start} seconds.");
+
my $maxwait = $r{hostalloc_maxwait_max};
+ my $now = time // die $!;
if (defined $maxwait) {
# We quit if:
# * we have been waiting at least $maxwait/2
# * we estimate it will take at least $maxwait overall
- my $wait_sofar = (time // die $!) - $alloc_start_time;
+ my $wait_sofar = $now - $alloc_start_time;
die "timed out: $wait_sofar, $best->{Start}, $maxwait"
if $wait_sofar > $maxwait/2
&& $wait_sofar + $best->{Start} > $maxwait;
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 16/21] ts-host-allocate-Executive: Break out $now and add a newline
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work, no functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 3da138b1..3425c8ce 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -756,12 +756,14 @@ sub attempt_allocation {
logm("host allocation: successful, reporting to planner.");
} else {
logm("host allocation: planned start in $best->{Start} seconds.");
+
my $maxwait = $r{hostalloc_maxwait_max};
+ my $now = time // die $!;
if (defined $maxwait) {
# We quit if:
# * we have been waiting at least $maxwait/2
# * we estimate it will take at least $maxwait overall
- my $wait_sofar = (time // die $!) - $alloc_start_time;
+ my $wait_sofar = $now - $alloc_start_time;
die "timed out: $wait_sofar, $best->{Start}, $maxwait"
if $wait_sofar > $maxwait/2
&& $wait_sofar + $best->{Start} > $maxwait;
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 17/21] starvation: Use "starved" for hostalloc_maxwait_max
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Previously this was "broken".
We mustn't just call `broken' inside attempt_allocation because that
runs in a db transaction. Instead, we arrange that attempt_allocation
returns 2, which threads its way back out to the return value from
alloc_resources, and then call broken there.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 3425c8ce..8c9ddaf7 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -683,12 +683,17 @@ sub alloc_hosts () {
? -10000
: -10 * @hids;
- alloc_resources(WaitStart =>
+ my $ok = alloc_resources(WaitStart =>
($ENV{OSSTEST_RESOURCE_WAITSTART} || $fi->{started}),
WaitStartAdjust => $waitstartadjust,
DebugFh => \*DEBUG,
\&attempt_allocation);
+ if ($ok==2) {
+ broken "starved or timed out", 'starved';
+ }
+ die $ok unless $ok==1;
+
foreach my $hid (@hids) {
my $sel= $hid->{Selected};
die $hid->{Ident} unless defined $sel->{resname};
@@ -764,9 +769,11 @@ sub attempt_allocation {
# * we have been waiting at least $maxwait/2
# * we estimate it will take at least $maxwait overall
my $wait_sofar = $now - $alloc_start_time;
- die "timed out: $wait_sofar, $best->{Start}, $maxwait"
- if $wait_sofar > $maxwait/2
- && $wait_sofar + $best->{Start} > $maxwait;
+ if ($wait_sofar > $maxwait/2
+ && $wait_sofar + $best->{Start} > $maxwait) {
+ logm "timed out: $wait_sofar, $best->{Start}, $maxwait";
+ return 2;
+ }
}
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 17/21] starvation: Use "starved" for hostalloc_maxwait_max
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Previously this was "broken".
We mustn't just call `broken' inside attempt_allocation because that
runs in a db transaction. Instead, we arrange that attempt_allocation
returns 2, which threads its way back out to the return value from
alloc_resources, and then call broken there.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 3425c8ce..8c9ddaf7 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -683,12 +683,17 @@ sub alloc_hosts () {
? -10000
: -10 * @hids;
- alloc_resources(WaitStart =>
+ my $ok = alloc_resources(WaitStart =>
($ENV{OSSTEST_RESOURCE_WAITSTART} || $fi->{started}),
WaitStartAdjust => $waitstartadjust,
DebugFh => \*DEBUG,
\&attempt_allocation);
+ if ($ok==2) {
+ broken "starved or timed out", 'starved';
+ }
+ die $ok unless $ok==1;
+
foreach my $hid (@hids) {
my $sel= $hid->{Selected};
die $hid->{Ident} unless defined $sel->{resname};
@@ -764,9 +769,11 @@ sub attempt_allocation {
# * we have been waiting at least $maxwait/2
# * we estimate it will take at least $maxwait overall
my $wait_sofar = $now - $alloc_start_time;
- die "timed out: $wait_sofar, $best->{Start}, $maxwait"
- if $wait_sofar > $maxwait/2
- && $wait_sofar + $best->{Start} > $maxwait;
+ if ($wait_sofar > $maxwait/2
+ && $wait_sofar + $best->{Start} > $maxwait) {
+ logm "timed out: $wait_sofar, $best->{Start}, $maxwait";
+ return 2;
+ }
}
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 18/21] starvation: Infrastructure for jobs which are delaying their flights
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Provide hostalloc_starvation_* in Osstest::Executive, and a comment
saying what we are going to do. And provide a demo utility which
prints the effect of some particular runvar value on a range of
situations.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/Executive.pm | 70 ++++++++++++++++++++++++++++++++++++++++++++
mg-hostalloc-starvation-demo | 53 +++++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
create mode 100755 mg-hostalloc-starvation-demo
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index a9f9ac78..e741f529 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -22,6 +22,7 @@ use warnings;
use Osstest;
+use Carp;
use POSIX;
use IO::File;
use File::Copy;
@@ -55,6 +56,8 @@ BEGIN {
alloc_resources alloc_resources_rollback_begin_work
resource_check_allocated
executive_resource_shared_mark_ready
+ hostalloc_starvation_parse_runvar
+ hostalloc_starvation_calculate_X
duration_estimator
db_pg_dsn opendb opendb_state
db_schema_updates_applied db_schema_updates_intree
@@ -1053,6 +1056,73 @@ END
logm("$restype $resname shared $sharetype marked ready");
}
+# hostalloc_maxwait_starvation
+#
+# $r{hostalloc_maxwait_starvation} is either "never" or
+# comma-separated list of assignments:
+# <var>=<value>
+# where <var> is C, A, L or Xh and <value> is
+# a floating point number.
+#
+# terms:
+# calculated or observed values:
+# X wait factor, ratio between amount of time
+# to allow for slow jobs, compared to other jobs
+# W number of jobs waiting - strictly, jobs in states
+# preparing queued running
+# D number of jobs done - strictly, other states
+# tuning parameters:
+# Xt X when D=9 W=1, ie cancel one job out of ten
+# Xh X when D=1 W=1, ie cancel one job out of two
+# I length of time this expected delay must have
+# persisted before we abandon this job
+# C,A tuning parameters relating X to W and D (see below)
+#
+# we calculate:
+# X = C + L * W^A / D^B
+# where C, A are specified
+# and L, B are calculated from C, A, Xh, Xt
+#
+# and then we are willing to allow each job to take
+# X * (time spent for done jobs) + I
+#
+# default values are
+# C = 1.3
+# A = 1
+# Xt = 5
+# Xh = 200
+# I = 3600 (seconds)
+
+sub hostalloc_starvation_parse_runvar ($) {
+ my ($r_hostalloc_maxwait_starvation) = @_;
+ # returns \%p hashref to be fed to hostalloc_starvation_calculate_X
+ # caller may inspect elements of %p
+ # if was `never' then !%p
+ $r_hostalloc_maxwait_starvation //= '';
+ return { } if $r_hostalloc_maxwait_starvation eq 'never';
+ my %p = ( C => 1.3, A => 1, Xt => 5, Xh => 200, I => 3600 );
+ foreach (split /,/, $r_hostalloc_maxwait_starvation) {
+ m/=/ or die "$r_hostalloc_maxwait_starvation ?";
+ exists($p{$`}) or die "$r_hostalloc_maxwait_starvation $` ?";
+ $p{$`} = $' + 0.0;
+ }
+ # Xh = C + L * 1^A / 1^B => L = Xh - C
+ # Xt = C + L * 1^A / 9^B
+ # Xt = C + (Xh - C) * 1^A / 9^B => B = log_9[ Xh - C) / (Xt - C) ]
+ $p{L} = $p{Xh} - $p{C};
+ $p{B} = log(($p{Xh} - $p{C}) / ($p{Xt} - $p{C})) / log(9);
+ logm("hostalloc_maxwait_starvation: ".
+ join ' ', map { "$_=$p{$_}" } sort keys %p);
+ return \%p;
+}
+
+sub hostalloc_starvation_calculate_X ($$$) {
+ my ($p, $W, $D) = @_; # => X, undef meaning "forever"
+ return undef unless %$p and $D;
+ confess unless $W > 0 && $D > 0;
+ return $p->{C} + $p->{L} * $W**$p->{A} / $D**$p->{B};
+}
+
#---------- duration estimator ----------
sub duration_estimator ($$;$$) {
diff --git a/mg-hostalloc-starvation-demo b/mg-hostalloc-starvation-demo
new file mode 100755
index 00000000..7944cc24
--- /dev/null
+++ b/mg-hostalloc-starvation-demo
@@ -0,0 +1,53 @@
+#!/usr/bin/perl -w
+
+use strict;
+BEGIN { unshift @INC, qw(.); }
+
+use strict;
+use POSIX;
+use Osstest::Executive;
+use Data::Dumper;
+
+die unless @ARGV>=1;
+
+our ($rv) = $ARGV[0]; # $r{hostalloc_starvation};
+shift @ARGV;
+
+my $p = hostalloc_starvation_parse_runvar($rv);
+
+my $l = 8;
+
+my @ts = qw(1 2 3 5 10 20 33 66 100 200 333);
+
+foreach my $w (qw(332 99 32 9 3 2 1)) {
+ printf "W=%3d |", $w;
+ foreach my $t (@ts) {
+ my $d = $t - $w;
+ print " | ";
+ if ($d <= 0) {
+ printf " %${l}s", '';
+ } else {
+ my $x = hostalloc_starvation_calculate_X($p, $w, $d);
+ printf "%3d%% ", 100.0*$w/$t;
+ if (defined $x) {
+ printf "%${l}.2f", $x;
+ } else {
+ printf "%${l}s", 'never';
+ }
+ }
+ }
+ print "\n";
+}
+
+print "\n";
+print " ";
+foreach my $t (@ts) {
+ printf " | %${l}s", sprintf "T=%3d", $t;
+}
+print "\n";
+
+foreach (@ARGV) {
+ m/,/ or die $!;
+ my $x = hostalloc_starvation_calculate_X($p, $', $`);
+ printf "D=%3d W=%3d X=%${l}.2f\n", $`, $', $x;
+}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 18/21] starvation: Infrastructure for jobs which are delaying their flights
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Provide hostalloc_starvation_* in Osstest::Executive, and a comment
saying what we are going to do. And provide a demo utility which
prints the effect of some particular runvar value on a range of
situations.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/Executive.pm | 70 ++++++++++++++++++++++++++++++++++++++++++++
mg-hostalloc-starvation-demo | 53 +++++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+)
create mode 100755 mg-hostalloc-starvation-demo
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index a9f9ac78..e741f529 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -22,6 +22,7 @@ use warnings;
use Osstest;
+use Carp;
use POSIX;
use IO::File;
use File::Copy;
@@ -55,6 +56,8 @@ BEGIN {
alloc_resources alloc_resources_rollback_begin_work
resource_check_allocated
executive_resource_shared_mark_ready
+ hostalloc_starvation_parse_runvar
+ hostalloc_starvation_calculate_X
duration_estimator
db_pg_dsn opendb opendb_state
db_schema_updates_applied db_schema_updates_intree
@@ -1053,6 +1056,73 @@ END
logm("$restype $resname shared $sharetype marked ready");
}
+# hostalloc_maxwait_starvation
+#
+# $r{hostalloc_maxwait_starvation} is either "never" or
+# comma-separated list of assignments:
+# <var>=<value>
+# where <var> is C, A, L or Xh and <value> is
+# a floating point number.
+#
+# terms:
+# calculated or observed values:
+# X wait factor, ratio between amount of time
+# to allow for slow jobs, compared to other jobs
+# W number of jobs waiting - strictly, jobs in states
+# preparing queued running
+# D number of jobs done - strictly, other states
+# tuning parameters:
+# Xt X when D=9 W=1, ie cancel one job out of ten
+# Xh X when D=1 W=1, ie cancel one job out of two
+# I length of time this expected delay must have
+# persisted before we abandon this job
+# C,A tuning parameters relating X to W and D (see below)
+#
+# we calculate:
+# X = C + L * W^A / D^B
+# where C, A are specified
+# and L, B are calculated from C, A, Xh, Xt
+#
+# and then we are willing to allow each job to take
+# X * (time spent for done jobs) + I
+#
+# default values are
+# C = 1.3
+# A = 1
+# Xt = 5
+# Xh = 200
+# I = 3600 (seconds)
+
+sub hostalloc_starvation_parse_runvar ($) {
+ my ($r_hostalloc_maxwait_starvation) = @_;
+ # returns \%p hashref to be fed to hostalloc_starvation_calculate_X
+ # caller may inspect elements of %p
+ # if was `never' then !%p
+ $r_hostalloc_maxwait_starvation //= '';
+ return { } if $r_hostalloc_maxwait_starvation eq 'never';
+ my %p = ( C => 1.3, A => 1, Xt => 5, Xh => 200, I => 3600 );
+ foreach (split /,/, $r_hostalloc_maxwait_starvation) {
+ m/=/ or die "$r_hostalloc_maxwait_starvation ?";
+ exists($p{$`}) or die "$r_hostalloc_maxwait_starvation $` ?";
+ $p{$`} = $' + 0.0;
+ }
+ # Xh = C + L * 1^A / 1^B => L = Xh - C
+ # Xt = C + L * 1^A / 9^B
+ # Xt = C + (Xh - C) * 1^A / 9^B => B = log_9[ Xh - C) / (Xt - C) ]
+ $p{L} = $p{Xh} - $p{C};
+ $p{B} = log(($p{Xh} - $p{C}) / ($p{Xt} - $p{C})) / log(9);
+ logm("hostalloc_maxwait_starvation: ".
+ join ' ', map { "$_=$p{$_}" } sort keys %p);
+ return \%p;
+}
+
+sub hostalloc_starvation_calculate_X ($$$) {
+ my ($p, $W, $D) = @_; # => X, undef meaning "forever"
+ return undef unless %$p and $D;
+ confess unless $W > 0 && $D > 0;
+ return $p->{C} + $p->{L} * $W**$p->{A} / $D**$p->{B};
+}
+
#---------- duration estimator ----------
sub duration_estimator ($$;$$) {
diff --git a/mg-hostalloc-starvation-demo b/mg-hostalloc-starvation-demo
new file mode 100755
index 00000000..7944cc24
--- /dev/null
+++ b/mg-hostalloc-starvation-demo
@@ -0,0 +1,53 @@
+#!/usr/bin/perl -w
+
+use strict;
+BEGIN { unshift @INC, qw(.); }
+
+use strict;
+use POSIX;
+use Osstest::Executive;
+use Data::Dumper;
+
+die unless @ARGV>=1;
+
+our ($rv) = $ARGV[0]; # $r{hostalloc_starvation};
+shift @ARGV;
+
+my $p = hostalloc_starvation_parse_runvar($rv);
+
+my $l = 8;
+
+my @ts = qw(1 2 3 5 10 20 33 66 100 200 333);
+
+foreach my $w (qw(332 99 32 9 3 2 1)) {
+ printf "W=%3d |", $w;
+ foreach my $t (@ts) {
+ my $d = $t - $w;
+ print " | ";
+ if ($d <= 0) {
+ printf " %${l}s", '';
+ } else {
+ my $x = hostalloc_starvation_calculate_X($p, $w, $d);
+ printf "%3d%% ", 100.0*$w/$t;
+ if (defined $x) {
+ printf "%${l}.2f", $x;
+ } else {
+ printf "%${l}s", 'never';
+ }
+ }
+ }
+ print "\n";
+}
+
+print "\n";
+print " ";
+foreach my $t (@ts) {
+ printf " | %${l}s", sprintf "T=%3d", $t;
+}
+print "\n";
+
+foreach (@ARGV) {
+ m/,/ or die $!;
+ my $x = hostalloc_starvation_calculate_X($p, $', $`);
+ printf "D=%3d W=%3d X=%${l}.2f\n", $`, $', $x;
+}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 19/21] starvation: Abandon jobs which are unreasonably delaying their flight
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Sometimes, due to a shortage of available resources, a flight might be
delayed because a handful of jobs are waiting much longer than the
rest. Add a heuristic which causes these jobs to be abandoned.
We consider ourselves starving if we are starving now, based on the
most optimistic start time seen in the last I.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 105 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 8c9ddaf7..7ea3c4af 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -62,6 +62,8 @@ our %magictaskid;
our $fi;
our $jobinfo;
our $harness_rev;
+our $starvation_p;
+our @abs_start_estimates;
#---------- general utilities, setup, etc. ----------
@@ -114,12 +116,16 @@ END
}
$alloc_start_time = time // die $!;
+
+ $starvation_p =
+ hostalloc_starvation_parse_runvar($r{hostalloc_maxwait_starvation});
}
#---------- prepared sql statements ----------
# all users of these must ->finish them afterwards, to avoid db deadlock
our ($flagscheckq, $equivflagscheckq, $duration_estimator, $resprop_q,
+ $starvation_q,
$alloc_findres_q, $alloc_shared_q, $alloc_sharing_slot_q,
$claim_share_reuse_q, $claim_maxshare_q, $claim_rmshares_q,
$claim_noshares_q, $claim_rmshare_q, $claim_setres_q,
@@ -146,6 +152,15 @@ END
AND name = ?
END
+ $starvation_q= $dbh_tests->prepare(<<END);
+ SELECT job, jobs.status, max(steps.finished)
+ FROM jobs
+ LEFT JOIN steps
+ USING (flight,job)
+ WHERE flight= ?
+ GROUP BY job, jobs.status
+END
+
# for allocation
$alloc_findres_q= $dbh_tests->prepare(<<END);
@@ -712,6 +727,88 @@ sub alloc_hosts () {
logm("host allocation: all successful and recorded.");
}
+sub most_optimistic ($$$) {
+ my ($best, $now, $period) = @_;
+ # Records that we have now estimated $best->{Start}.
+ # Returns the most optimistic absolute start time "in the last
+ # $period". Returns undef if we don't have good data yet.
+
+ push @abs_start_estimates, { At => $now, Got => $best->{Start} + $now };
+
+ # Actually, what we do is prune all but the last entry from before
+ # $period, and we expect at least 4 estimates. That ensures that
+ # the answer involves at least one estimate at least $period ago.
+ # Ie what we actually return is
+ # Consider the most recent estimate which is at least $period
+ # ago (the "oldest relevant"), and all subsequent estimates.
+ # Answer is the most optimistic start time of all of those,
+ # provided there are at least 4 of them.
+ my $is_old = sub { return $_[0]{At} <= $now - $period; };
+ my $need_estimates = 4;
+ while (@abs_start_estimates > $need_estimates &&
+ $is_old->($abs_start_estimates[1])) {
+ # estimates[1] is at least $period ago and more recent
+ # than $estimates[0], so $estimates[0] cannot be the
+ # oldest relevant and is indeed older than the oldest
+ # relevant.
+ shift @abs_start_estimates;
+ }
+
+ my $pr = sub {
+ my ($e) = @_;
+ printf(DEBUG ' %s (@%s)',
+ $e->{Got} - $now,
+ $e->{At} - $now);
+ };
+
+ print DEBUG "most_optimistic: all:";
+ my $optimist;
+ foreach (@abs_start_estimates) {
+ $pr->($_);
+ $optimist = $_ if !$optimist || $_->{Got} < $optimist->{Got};
+ }
+ print DEBUG "\n";
+ printf(DEBUG "most_optimistic: (period=%s):", $period);
+ $pr->($optimist);
+ print DEBUG "\n";
+
+ return undef unless @abs_start_estimates >= $need_estimates;
+
+ return $optimist->{Got};
+}
+
+sub starving ($) {
+ my ($best_start_abs) = @_;
+ return (0, 'runvar says never give up') unless %$starvation_p;
+ return (0, 'no estimate') unless defined $best_start_abs;
+ $starvation_q->execute($flight);
+ my $d=0;
+ my $w=0;
+ my $maxfin=0;
+ while (my ($j,$st,$fin) = $starvation_q->fetchrow_array()) {
+ if ($st eq 'preparing' ||
+ $st eq 'queued' ||
+ $st eq 'running') {
+ $w++;
+ } else {
+ $d++;
+ return (0, "job $j status $st but no step finished time!")
+ unless defined $fin;
+ $maxfin = $fin if $fin > $maxfin;
+ }
+ }
+ # we quit if the total time from the start of the flight
+ # to our expected finish is more than the total time so
+ # far (for the completed jobs) by the margin X and I
+ my $X = hostalloc_starvation_calculate_X($starvation_p, $w, $d);
+ return (0, 'X=inf') unless defined $X;
+ my $total_d = $maxfin - $fi->{started};
+ my $projected_me = $best_start_abs - $fi->{started};
+ my $m = "D=$d W=$w X=$X maxfin=$maxfin";
+ my $bad = $projected_me > $X * $total_d + $starvation_p->{I};
+ return ($bad, $m);
+}
+
sub attempt_allocation {
my $mayalloc;
($plan, $mayalloc) = @_;
@@ -772,6 +869,14 @@ sub attempt_allocation {
if ($wait_sofar > $maxwait/2
&& $wait_sofar + $best->{Start} > $maxwait) {
logm "timed out: $wait_sofar, $best->{Start}, $maxwait";
+ } elsif (%$starvation_p) {
+ my $est_abs = most_optimistic($best, $now, $starvation_p->{I});
+ my ($starving, $m) = starving($est_abs);
+ $starvation_q->finish();
+ if (!$starving) {
+ print DEBUG "not starving: $m\n";
+ } else {
+ logm "starving ($m)";
return 2;
}
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 19/21] starvation: Abandon jobs which are unreasonably delaying their flight
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Sometimes, due to a shortage of available resources, a flight might be
delayed because a handful of jobs are waiting much longer than the
rest. Add a heuristic which causes these jobs to be abandoned.
We consider ourselves starving if we are starving now, based on the
most optimistic start time seen in the last I.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 105 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 8c9ddaf7..7ea3c4af 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -62,6 +62,8 @@ our %magictaskid;
our $fi;
our $jobinfo;
our $harness_rev;
+our $starvation_p;
+our @abs_start_estimates;
#---------- general utilities, setup, etc. ----------
@@ -114,12 +116,16 @@ END
}
$alloc_start_time = time // die $!;
+
+ $starvation_p =
+ hostalloc_starvation_parse_runvar($r{hostalloc_maxwait_starvation});
}
#---------- prepared sql statements ----------
# all users of these must ->finish them afterwards, to avoid db deadlock
our ($flagscheckq, $equivflagscheckq, $duration_estimator, $resprop_q,
+ $starvation_q,
$alloc_findres_q, $alloc_shared_q, $alloc_sharing_slot_q,
$claim_share_reuse_q, $claim_maxshare_q, $claim_rmshares_q,
$claim_noshares_q, $claim_rmshare_q, $claim_setres_q,
@@ -146,6 +152,15 @@ END
AND name = ?
END
+ $starvation_q= $dbh_tests->prepare(<<END);
+ SELECT job, jobs.status, max(steps.finished)
+ FROM jobs
+ LEFT JOIN steps
+ USING (flight,job)
+ WHERE flight= ?
+ GROUP BY job, jobs.status
+END
+
# for allocation
$alloc_findres_q= $dbh_tests->prepare(<<END);
@@ -712,6 +727,88 @@ sub alloc_hosts () {
logm("host allocation: all successful and recorded.");
}
+sub most_optimistic ($$$) {
+ my ($best, $now, $period) = @_;
+ # Records that we have now estimated $best->{Start}.
+ # Returns the most optimistic absolute start time "in the last
+ # $period". Returns undef if we don't have good data yet.
+
+ push @abs_start_estimates, { At => $now, Got => $best->{Start} + $now };
+
+ # Actually, what we do is prune all but the last entry from before
+ # $period, and we expect at least 4 estimates. That ensures that
+ # the answer involves at least one estimate at least $period ago.
+ # Ie what we actually return is
+ # Consider the most recent estimate which is at least $period
+ # ago (the "oldest relevant"), and all subsequent estimates.
+ # Answer is the most optimistic start time of all of those,
+ # provided there are at least 4 of them.
+ my $is_old = sub { return $_[0]{At} <= $now - $period; };
+ my $need_estimates = 4;
+ while (@abs_start_estimates > $need_estimates &&
+ $is_old->($abs_start_estimates[1])) {
+ # estimates[1] is at least $period ago and more recent
+ # than $estimates[0], so $estimates[0] cannot be the
+ # oldest relevant and is indeed older than the oldest
+ # relevant.
+ shift @abs_start_estimates;
+ }
+
+ my $pr = sub {
+ my ($e) = @_;
+ printf(DEBUG ' %s (@%s)',
+ $e->{Got} - $now,
+ $e->{At} - $now);
+ };
+
+ print DEBUG "most_optimistic: all:";
+ my $optimist;
+ foreach (@abs_start_estimates) {
+ $pr->($_);
+ $optimist = $_ if !$optimist || $_->{Got} < $optimist->{Got};
+ }
+ print DEBUG "\n";
+ printf(DEBUG "most_optimistic: (period=%s):", $period);
+ $pr->($optimist);
+ print DEBUG "\n";
+
+ return undef unless @abs_start_estimates >= $need_estimates;
+
+ return $optimist->{Got};
+}
+
+sub starving ($) {
+ my ($best_start_abs) = @_;
+ return (0, 'runvar says never give up') unless %$starvation_p;
+ return (0, 'no estimate') unless defined $best_start_abs;
+ $starvation_q->execute($flight);
+ my $d=0;
+ my $w=0;
+ my $maxfin=0;
+ while (my ($j,$st,$fin) = $starvation_q->fetchrow_array()) {
+ if ($st eq 'preparing' ||
+ $st eq 'queued' ||
+ $st eq 'running') {
+ $w++;
+ } else {
+ $d++;
+ return (0, "job $j status $st but no step finished time!")
+ unless defined $fin;
+ $maxfin = $fin if $fin > $maxfin;
+ }
+ }
+ # we quit if the total time from the start of the flight
+ # to our expected finish is more than the total time so
+ # far (for the completed jobs) by the margin X and I
+ my $X = hostalloc_starvation_calculate_X($starvation_p, $w, $d);
+ return (0, 'X=inf') unless defined $X;
+ my $total_d = $maxfin - $fi->{started};
+ my $projected_me = $best_start_abs - $fi->{started};
+ my $m = "D=$d W=$w X=$X maxfin=$maxfin";
+ my $bad = $projected_me > $X * $total_d + $starvation_p->{I};
+ return ($bad, $m);
+}
+
sub attempt_allocation {
my $mayalloc;
($plan, $mayalloc) = @_;
@@ -772,6 +869,14 @@ sub attempt_allocation {
if ($wait_sofar > $maxwait/2
&& $wait_sofar + $best->{Start} > $maxwait) {
logm "timed out: $wait_sofar, $best->{Start}, $maxwait";
+ } elsif (%$starvation_p) {
+ my $est_abs = most_optimistic($best, $now, $starvation_p->{I});
+ my ($starving, $m) = starving($est_abs);
+ $starvation_q->finish();
+ if (!$starving) {
+ print DEBUG "not starving: $m\n";
+ } else {
+ logm "starving ($m)";
return 2;
}
}
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 20/21] hostalloc_maxwait_max: Use starvation most_optimistic
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Now that we have a more sophisticated estimate of our likely scheduled
time, use it here too.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 7ea3c4af..b75927c1 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -865,10 +865,15 @@ sub attempt_allocation {
# We quit if:
# * we have been waiting at least $maxwait/2
# * we estimate it will take at least $maxwait overall
- my $wait_sofar = $now - $alloc_start_time;
- if ($wait_sofar > $maxwait/2
- && $wait_sofar + $best->{Start} > $maxwait) {
- logm "timed out: $wait_sofar, $best->{Start}, $maxwait";
+ my $est_abs = most_optimistic($best, $now, $maxwait/2);
+ if (defined $est_abs) {
+ my $wait_sofar = $now - $alloc_start_time;
+ my $wait_exp = $est_abs - $alloc_start_time;
+ if ($wait_exp > $maxwait) {
+ logm "timed out: $wait_exp > $maxwait ($wait_sofar)",
+ return 2;
+ }
+ }
} elsif (%$starvation_p) {
my $est_abs = most_optimistic($best, $now, $starvation_p->{I});
my ($starving, $m) = starving($est_abs);
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 20/21] hostalloc_maxwait_max: Use starvation most_optimistic
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Now that we have a more sophisticated estimate of our likely scheduled
time, use it here too.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 7ea3c4af..b75927c1 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -865,10 +865,15 @@ sub attempt_allocation {
# We quit if:
# * we have been waiting at least $maxwait/2
# * we estimate it will take at least $maxwait overall
- my $wait_sofar = $now - $alloc_start_time;
- if ($wait_sofar > $maxwait/2
- && $wait_sofar + $best->{Start} > $maxwait) {
- logm "timed out: $wait_sofar, $best->{Start}, $maxwait";
+ my $est_abs = most_optimistic($best, $now, $maxwait/2);
+ if (defined $est_abs) {
+ my $wait_sofar = $now - $alloc_start_time;
+ my $wait_exp = $est_abs - $alloc_start_time;
+ if ($wait_exp > $maxwait) {
+ logm "timed out: $wait_exp > $maxwait ($wait_sofar)",
+ return 2;
+ }
+ }
} elsif (%$starvation_p) {
my $est_abs = most_optimistic($best, $now, $starvation_p->{I});
my ($starving, $m) = starving($est_abs);
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [OSSTEST PATCH 21/21] starvation: Better logging/debugging output
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index b75927c1..64c4c9f9 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -777,8 +777,8 @@ sub most_optimistic ($$$) {
return $optimist->{Got};
}
-sub starving ($) {
- my ($best_start_abs) = @_;
+sub starving ($$) {
+ my ($best_start_abs, $now) = @_;
return (0, 'runvar says never give up') unless %$starvation_p;
return (0, 'no estimate') unless defined $best_start_abs;
$starvation_q->execute($flight);
@@ -801,11 +801,16 @@ sub starving ($) {
# to our expected finish is more than the total time so
# far (for the completed jobs) by the margin X and I
my $X = hostalloc_starvation_calculate_X($starvation_p, $w, $d);
- return (0, 'X=inf') unless defined $X;
+ return (0, "D=$d W=$w X=inf") unless defined $X;
my $total_d = $maxfin - $fi->{started};
my $projected_me = $best_start_abs - $fi->{started};
- my $m = "D=$d W=$w X=$X maxfin=$maxfin";
- my $bad = $projected_me > $X * $total_d + $starvation_p->{I};
+ my $lim = $X * $total_d + $starvation_p->{I};
+ my $Xcmp = ($projected_me - $starvation_p->{I}) / $total_d;
+ my $m = sprintf
+ "D=%d W=%d X=%.3f t_D=%s t_me=%s t_lim=%.3f X'=%.4f (fi.s=%s)",
+ $d, $w, $X, $total_d, $projected_me, $lim, $Xcmp,
+ $fi->{started} - $now;
+ my $bad = $projected_me > $lim;
return ($bad, $m);
}
@@ -876,7 +881,7 @@ sub attempt_allocation {
}
} elsif (%$starvation_p) {
my $est_abs = most_optimistic($best, $now, $starvation_p->{I});
- my ($starving, $m) = starving($est_abs);
+ my ($starving, $m) = starving($est_abs, $now);
$starvation_q->finish();
if (!$starving) {
print DEBUG "not starving: $m\n";
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [Xen-devel] [OSSTEST PATCH 21/21] starvation: Better logging/debugging output
@ 2019-04-18 16:31 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-18 16:31 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
ts-hosts-allocate-Executive | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index b75927c1..64c4c9f9 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -777,8 +777,8 @@ sub most_optimistic ($$$) {
return $optimist->{Got};
}
-sub starving ($) {
- my ($best_start_abs) = @_;
+sub starving ($$) {
+ my ($best_start_abs, $now) = @_;
return (0, 'runvar says never give up') unless %$starvation_p;
return (0, 'no estimate') unless defined $best_start_abs;
$starvation_q->execute($flight);
@@ -801,11 +801,16 @@ sub starving ($) {
# to our expected finish is more than the total time so
# far (for the completed jobs) by the margin X and I
my $X = hostalloc_starvation_calculate_X($starvation_p, $w, $d);
- return (0, 'X=inf') unless defined $X;
+ return (0, "D=$d W=$w X=inf") unless defined $X;
my $total_d = $maxfin - $fi->{started};
my $projected_me = $best_start_abs - $fi->{started};
- my $m = "D=$d W=$w X=$X maxfin=$maxfin";
- my $bad = $projected_me > $X * $total_d + $starvation_p->{I};
+ my $lim = $X * $total_d + $starvation_p->{I};
+ my $Xcmp = ($projected_me - $starvation_p->{I}) / $total_d;
+ my $m = sprintf
+ "D=%d W=%d X=%.3f t_D=%s t_me=%s t_lim=%.3f X'=%.4f (fi.s=%s)",
+ $d, $w, $X, $total_d, $projected_me, $lim, $Xcmp,
+ $fi->{started} - $now;
+ my $bad = $projected_me > $lim;
return ($bad, $m);
}
@@ -876,7 +881,7 @@ sub attempt_allocation {
}
} elsif (%$starvation_p) {
my $est_abs = most_optimistic($best, $now, $starvation_p->{I});
- my ($starving, $m) = starving($est_abs);
+ my ($starving, $m) = starving($est_abs, $now);
$starvation_q->finish();
if (!$starving) {
print DEBUG "not starving: $m\n";
--
2.11.0
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Re: [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-26 21:16 ` Julien Grall
0 siblings, 0 replies; 50+ messages in thread
From: Julien Grall @ 2019-04-26 21:16 UTC (permalink / raw)
To: Ian Jackson, xen-devel; +Cc: Lars Kurth, Stefano Stabellini, committers
Hi Ian,
On 4/18/19 5:31 PM, Ian Jackson wrote:
> Sometimes we find ourselves seriously lacking the capacity to run
> particular job(s). The result can be that the whole system stands
> mostly idle while a small proportion of the resources runs flat out
> with a giant queue.
>
> In this series we arrange for osstest to be able to spot this
> happening, and automatically rebalance load by give up earlier on the
> jobs which are overly-contended.
>
> There are some tuning parameters, of course. To summarise, I have
> chosen here to treat jobs as starved if (for example):
> We have completed 90% of the flight, and the remaining 10%
> is projected to take 5x as long as the first 90%.
> (The "90%" is by number of jobs.) See the patch
> starvation: Infrastructure for jobs which are delaying their flights
> for details of the heuristic and its parameters.
>
> When situations like this persist it will still be good to manually
> balance the load by adjusting the job mix in submitted flights. This
> is because the starvation will not necessarily drop the same job in
> subsequent flights on the same "branch", so starvation will impair the
> regression detection.
As we discussed on IRC, I understand this will have an impact on Arm32
testing. Do you have an estimate how likely the tests will be skipped?
I am wondering whether we should discuss to reduce the number of
testings done on Arm32. We did that in the past on Arm64 when we were
struggle with broken laxton0/laxton1.
Cheers,
>
> Ian Jackson (21):
> ts-hosts-allocate-Executive: with -U, just append to the same logfile
> selecthost: Honour new $none_ok optional parameter
> ts-logs-capture: Do not try to capture logs of hosts not allocated
> alloc_resources: Support special abandonment values
> starvation: Teach sg-report-flight about starved step state
> starvation: Teach archaeologists about starved job state
> starvation: Teach ms-flights-summary about job state starved
> starvation: Teach sg-execute-flight about job state starved
> step handling: Preserve step states set by ts-* scripts
> TestSupport: Make "broken" print the actual job state
> JobDB::Executive: step_*: fix log messages to talk about "steps"
> starvation: Permit step_finish to set the state `starved'
> TestSupport: Make "broken" set the step state too
> tcl/JobDB-Executive: Do not squash "starved" status
> starvation: Propagate starved job status into dependent jobs
> ts-host-allocate-Executive: Break out $now and add a newline
> starvation: Use "starved" for hostalloc_maxwait_max
> starvation: Infrastructure for jobs which are delaying their flights
> starvation: Abandon jobs which are unreasonably delaying their flight
> hostalloc_maxwait_max: Use starvation most_optimistic
> starvation: Better logging/debugging output
>
> Osstest/Executive.pm | 95 ++++++++++++++++++++++++++---
> Osstest/JobDB/Executive.pm | 8 ++-
> Osstest/TestSupport.pm | 24 ++++++--
> mg-hostalloc-starvation-demo | 53 ++++++++++++++++
> ms-flights-summary | 9 +--
> sg-execute-flight | 2 +-
> sg-report-flight | 17 +++++-
> tcl/JobDB-Executive.tcl | 6 +-
> ts-hosts-allocate-Executive | 142 ++++++++++++++++++++++++++++++++++++++++---
> ts-logs-capture | 7 ++-
> 10 files changed, 328 insertions(+), 35 deletions(-)
> create mode 100755 mg-hostalloc-starvation-demo
>
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Xen-devel] [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-26 21:16 ` Julien Grall
0 siblings, 0 replies; 50+ messages in thread
From: Julien Grall @ 2019-04-26 21:16 UTC (permalink / raw)
To: Ian Jackson, xen-devel; +Cc: Lars Kurth, Stefano Stabellini, committers
Hi Ian,
On 4/18/19 5:31 PM, Ian Jackson wrote:
> Sometimes we find ourselves seriously lacking the capacity to run
> particular job(s). The result can be that the whole system stands
> mostly idle while a small proportion of the resources runs flat out
> with a giant queue.
>
> In this series we arrange for osstest to be able to spot this
> happening, and automatically rebalance load by give up earlier on the
> jobs which are overly-contended.
>
> There are some tuning parameters, of course. To summarise, I have
> chosen here to treat jobs as starved if (for example):
> We have completed 90% of the flight, and the remaining 10%
> is projected to take 5x as long as the first 90%.
> (The "90%" is by number of jobs.) See the patch
> starvation: Infrastructure for jobs which are delaying their flights
> for details of the heuristic and its parameters.
>
> When situations like this persist it will still be good to manually
> balance the load by adjusting the job mix in submitted flights. This
> is because the starvation will not necessarily drop the same job in
> subsequent flights on the same "branch", so starvation will impair the
> regression detection.
As we discussed on IRC, I understand this will have an impact on Arm32
testing. Do you have an estimate how likely the tests will be skipped?
I am wondering whether we should discuss to reduce the number of
testings done on Arm32. We did that in the past on Arm64 when we were
struggle with broken laxton0/laxton1.
Cheers,
>
> Ian Jackson (21):
> ts-hosts-allocate-Executive: with -U, just append to the same logfile
> selecthost: Honour new $none_ok optional parameter
> ts-logs-capture: Do not try to capture logs of hosts not allocated
> alloc_resources: Support special abandonment values
> starvation: Teach sg-report-flight about starved step state
> starvation: Teach archaeologists about starved job state
> starvation: Teach ms-flights-summary about job state starved
> starvation: Teach sg-execute-flight about job state starved
> step handling: Preserve step states set by ts-* scripts
> TestSupport: Make "broken" print the actual job state
> JobDB::Executive: step_*: fix log messages to talk about "steps"
> starvation: Permit step_finish to set the state `starved'
> TestSupport: Make "broken" set the step state too
> tcl/JobDB-Executive: Do not squash "starved" status
> starvation: Propagate starved job status into dependent jobs
> ts-host-allocate-Executive: Break out $now and add a newline
> starvation: Use "starved" for hostalloc_maxwait_max
> starvation: Infrastructure for jobs which are delaying their flights
> starvation: Abandon jobs which are unreasonably delaying their flight
> hostalloc_maxwait_max: Use starvation most_optimistic
> starvation: Better logging/debugging output
>
> Osstest/Executive.pm | 95 ++++++++++++++++++++++++++---
> Osstest/JobDB/Executive.pm | 8 ++-
> Osstest/TestSupport.pm | 24 ++++++--
> mg-hostalloc-starvation-demo | 53 ++++++++++++++++
> ms-flights-summary | 9 +--
> sg-execute-flight | 2 +-
> sg-report-flight | 17 +++++-
> tcl/JobDB-Executive.tcl | 6 +-
> ts-hosts-allocate-Executive | 142 ++++++++++++++++++++++++++++++++++++++++---
> ts-logs-capture | 7 ++-
> 10 files changed, 328 insertions(+), 35 deletions(-)
> create mode 100755 mg-hostalloc-starvation-demo
>
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-29 14:46 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-29 14:46 UTC (permalink / raw)
To: Julien Grall; +Cc: xen-devel, Stefano Stabellini, committers, Lars Kurth
Julien Grall writes ("Re: [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight"):
> As we discussed on IRC, I understand this will have an impact on Arm32
> testing. Do you have an estimate how likely the tests will be skipped?
Many, maybe most. Very likely the smoke tests will continue to run.
> I am wondering whether we should discuss to reduce the number of
> testings done on Arm32. We did that in the past on Arm64 when we were
> struggle with broken laxton0/laxton1.
That is a sensible suggestion but before we do that kind of manual
rebalancing I would like to try moving at least the kernel builds, so
they run as amd64 cross builds. I think that will free up a lot of
capacity.
In the meantime, are you happy with me pushing this series to osstest
pretest at some point when convenient ?
Regards,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Xen-devel] [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-29 14:46 ` Ian Jackson
0 siblings, 0 replies; 50+ messages in thread
From: Ian Jackson @ 2019-04-29 14:46 UTC (permalink / raw)
To: Julien Grall; +Cc: xen-devel, Stefano Stabellini, committers, Lars Kurth
Julien Grall writes ("Re: [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight"):
> As we discussed on IRC, I understand this will have an impact on Arm32
> testing. Do you have an estimate how likely the tests will be skipped?
Many, maybe most. Very likely the smoke tests will continue to run.
> I am wondering whether we should discuss to reduce the number of
> testings done on Arm32. We did that in the past on Arm64 when we were
> struggle with broken laxton0/laxton1.
That is a sensible suggestion but before we do that kind of manual
rebalancing I would like to try moving at least the kernel builds, so
they run as amd64 cross builds. I think that will free up a lot of
capacity.
In the meantime, are you happy with me pushing this series to osstest
pretest at some point when convenient ?
Regards,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-30 14:26 ` Julien Grall
0 siblings, 0 replies; 50+ messages in thread
From: Julien Grall @ 2019-04-30 14:26 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Stefano Stabellini, committers, Lars Kurth
Hi Ian,
On 29/04/2019 15:46, Ian Jackson wrote:
> Julien Grall writes ("Re: [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight"):
>> As we discussed on IRC, I understand this will have an impact on Arm32
>> testing. Do you have an estimate how likely the tests will be skipped?
>
> Many, maybe most. Very likely the smoke tests will continue to run.
>
>> I am wondering whether we should discuss to reduce the number of
>> testings done on Arm32. We did that in the past on Arm64 when we were
>> struggle with broken laxton0/laxton1.
>
> That is a sensible suggestion but before we do that kind of manual
> rebalancing I would like to try moving at least the kernel builds, so
> they run as amd64 cross builds. I think that will free up a lot of
> capacity.
>
> In the meantime, are you happy with me pushing this series to osstest > pretest at some point when convenient ?
I am happy with that. Let's see how much tests will get dropped on Arm32 :).
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Xen-devel] [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight
@ 2019-04-30 14:26 ` Julien Grall
0 siblings, 0 replies; 50+ messages in thread
From: Julien Grall @ 2019-04-30 14:26 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Stefano Stabellini, committers, Lars Kurth
Hi Ian,
On 29/04/2019 15:46, Ian Jackson wrote:
> Julien Grall writes ("Re: [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight"):
>> As we discussed on IRC, I understand this will have an impact on Arm32
>> testing. Do you have an estimate how likely the tests will be skipped?
>
> Many, maybe most. Very likely the smoke tests will continue to run.
>
>> I am wondering whether we should discuss to reduce the number of
>> testings done on Arm32. We did that in the past on Arm64 when we were
>> struggle with broken laxton0/laxton1.
>
> That is a sensible suggestion but before we do that kind of manual
> rebalancing I would like to try moving at least the kernel builds, so
> they run as amd64 cross builds. I think that will free up a lot of
> capacity.
>
> In the meantime, are you happy with me pushing this series to osstest > pretest at some point when convenient ?
I am happy with that. Let's see how much tests will get dropped on Arm32 :).
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 50+ messages in thread
end of thread, other threads:[~2019-04-30 14:26 UTC | newest]
Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-18 16:31 [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 01/21] ts-hosts-allocate-Executive: with -U, just append to the same logfile Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 02/21] selecthost: Honour new $none_ok optional parameter Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 03/21] ts-logs-capture: Do not try to capture logs of hosts not allocated Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 04/21] alloc_resources: Support special abandonment values Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 05/21] starvation: Teach sg-report-flight about starved step state Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 06/21] starvation: Teach archaeologists about starved job state Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 07/21] starvation: Teach ms-flights-summary about job state starved Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 08/21] starvation: Teach sg-execute-flight " Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 09/21] step handling: Preserve step states set by ts-* scripts Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 10/21] TestSupport: Make "broken" print the actual job state Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 11/21] JobDB::Executive: step_*: fix log messages to talk about "steps" Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 12/21] starvation: Permit step_finish to set the state `starved' Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 13/21] TestSupport: Make "broken" set the step state too Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 14/21] tcl/JobDB-Executive: Do not squash "starved" status Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 15/21] starvation: Propagate starved job status into dependent jobs Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 16/21] ts-host-allocate-Executive: Break out $now and add a newline Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 17/21] starvation: Use "starved" for hostalloc_maxwait_max Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 18/21] starvation: Infrastructure for jobs which are delaying their flights Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 19/21] starvation: Abandon jobs which are unreasonably delaying their flight Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 20/21] hostalloc_maxwait_max: Use starvation most_optimistic Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-18 16:31 ` [OSSTEST PATCH 21/21] starvation: Better logging/debugging output Ian Jackson
2019-04-18 16:31 ` [Xen-devel] " Ian Jackson
2019-04-26 21:16 ` [OSSTEST PATCH 00/21] Abandon jobs which are unreasonably delaying their flight Julien Grall
2019-04-26 21:16 ` [Xen-devel] " Julien Grall
2019-04-29 14:46 ` Ian Jackson
2019-04-29 14:46 ` [Xen-devel] " Ian Jackson
2019-04-30 14:26 ` Julien Grall
2019-04-30 14:26 ` [Xen-devel] " Julien Grall
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.