From: Ian Jackson <iwj@xenproject.org>
To: xen-devel@lists.xenproject.org
Cc: Ian Jackson <ian.jackson@eu.citrix.com>,
Ian Jackson <Ian.Jackson@eu.citrix.com>
Subject: [OSSTEST PATCH 52/82] host allocation: Group jobs by their reuse parameters
Date: Wed, 7 Oct 2020 18:59:54 +0100 [thread overview]
Message-ID: <20201007180024.7932-53-iwj@xenproject.org> (raw)
In-Reply-To: <20201007180024.7932-1-iwj@xenproject.org>
From: Ian Jackson <ian.jackson@eu.citrix.com>
This promotes reuse by arranging that jobs that can reuse a host get
to run consecutively.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Osstest/JobDB/Executive.pm | 47 +++++++++++++++++++++++++++++++++++++
Osstest/JobDB/Standalone.pm | 2 ++
ts-host-reuse | 1 +
ts-hosts-allocate-Executive | 4 ++++
4 files changed, 54 insertions(+)
diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 30629572..8c235d45 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -389,6 +389,53 @@ END
}
}
+sub jobdb_set_hosts_infraprioritygroup ($$$$;$) { # method
+ my ($mo, $flight, $job, $group_key, $rref) = @_;
+ # Sets the runvar hosts_infraprioritygroup in $flight,$job
+ # The runvar values are NUM:GROUPKEY
+ # such that each GROUPKEY always has the same NUM, within the flight
+ # $rref is \%r (for use within a ts-*) or undef
+
+ my $vn = 'hosts_infraprioritygroup';
+
+ my $queryq = $dbh_tests->prepare(<<END);
+ SELECT job, val,
+ (job = ?) AS thisjob
+ FROM runvars
+ WHERE flight=?
+ AND name=?
+ ORDER BY thisjob DESC
+END
+ my $insertq = $dbh_tests->prepare(<<END);
+ INSERT INTO runvars (flight,job, name,val, synth)
+ VALUES (?, ?, ?, ?, 't')
+END
+
+ my $resulting;
+ db_retry($dbh_tests,[],sub {
+ my $use = 1;
+ $resulting = undef;
+ $queryq->execute($job, $flight, $vn);
+ while (my ($tjob, $tval, $thisjob) = $queryq->fetchrow_array()) {
+ if ($thisjob) {
+ logm("$vn: job is already in group $tval");
+ return;
+ }
+ $tval =~ m/^(\d+)\:/ or die "$flight $job $tval ?";
+ if ($' eq $group_key) {
+ $use = $1;
+ last;
+ } elsif ($1 >= $use) {
+ $use = $1 + 1;
+ }
+ }
+ $resulting = "$use:$group_key";
+ logm("$vn: inserting job into group $resulting");
+ $insertq->execute($flight,$job,$vn, $resulting);
+ });
+ $rref->{$vn} = $resulting if $rref && defined $resulting;
+}
+
sub jobdb_flight_started_for_log_capture ($$) { #method
my ($mo, $flight) = @_;
my $started= $dbh_tests->selectrow_array(<<END);
diff --git a/Osstest/JobDB/Standalone.pm b/Osstest/JobDB/Standalone.pm
index 4f320ccf..1db4dc78 100644
--- a/Osstest/JobDB/Standalone.pm
+++ b/Osstest/JobDB/Standalone.pm
@@ -118,6 +118,8 @@ sub jobdb_resource_shared_mark_ready { } #method
sub jobdb_check_other_job { } #method
+sub jobdb_set_hosts_infraprioritygroup { } # method
+
sub jobdb_flight_started_for_log_capture ($$) { #method
my ($mo, $flight) = @_;
return time - 60*60; # just the most recent serial log then,
diff --git a/ts-host-reuse b/ts-host-reuse
index 5bdb07d1..29abe987 100755
--- a/ts-host-reuse
+++ b/ts-host-reuse
@@ -141,6 +141,7 @@ sub act_prealloc () {
compute_test_sharetype();
$ho = selecthost($whhost, undef, 1);
set_runtime_hostflag($ho->{Ident}, "reuse-$sharetype");
+ $mjobdb->jobdb_set_hosts_infraprioritygroup($flight, $job, $sharetype, \%r);
}
sub act_start_test () {
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index fc107c08..a50f8bf3 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -733,9 +733,13 @@ sub alloc_hosts () {
? -10000
: -10 * @hids;
+ my $infrapriority =
+ ($r{hosts_infraprioritygroup} // '') =~ m/^(\d+):/ ? $1 : undef;
+
my $ok = alloc_resources(WaitStart =>
($ENV{OSSTEST_RESOURCE_WAITSTART} || $fi->{started}),
WaitStartAdjust => $waitstartadjust,
+ InfraPriority => $infrapriority,
DebugFh => \*DEBUG,
\&attempt_allocation);
--
2.20.1
next prev parent reply other threads:[~2020-10-07 18:29 UTC|newest]
Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-07 17:59 [OSSTEST PATCH 00/82] Reuse test hosts Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 01/82] ms-queuedaemon: Update for newer Tcl's socket channel ids Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 02/82] Executive.pm planner: fix typo Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 03/82] README.planner: Document magic job hostflags Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 04/82] sg-run-job: Minor whitespace (formatting) changes Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 05/82] sg-run-job: Improve some internal API docs Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 06/82] show_abs_time: Represent undef $timet as <undef> Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 07/82] ts-hosts-allocate-Executive: Add a comment about a warning Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 08/82] host reuse: ms-planner: Bring some variables forward Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 09/82] host reuse: ms-planner: Do not show reuse as shared in the plan Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 10/82] hsot reuse: ms-planner: Abbreviate reporting of test shares Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 11/82] cr-publish-flight-logs: Fix abs_time calls Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 12/82] target setup refactoring: Move target_kernkind_check Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 13/82] target setup refactoring: Move target_kernkind_console_inittab Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 14/82] target setup refactoring: Merge target_kernkind_* Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 15/82] target setup refactoring: Add a doc comment Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 16/82] abolish "kernkind"; desupport non-pvops kernels Ian Jackson
2020-10-13 9:29 ` Wei Liu
2020-10-07 17:59 ` [OSSTEST PATCH 17/82] Debian: osstest-erase-other-disks: Slightly guard against races Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 18/82] host allocation: Remove some unnecessary definedness tests Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 19/82] host allocation: Executive: Honour $xparams{InfraPriority} Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 20/82] ts-hosts-allocate-Executive: Fix handling of failed preps for same sharing Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 21/82] sg-run-job: support +! for *only* adding things to TESTID Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 22/82] sg-run-job: Use +! in per-host-ts implementation Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 23/82] db_retry: Make the sleeps random and increasing Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 24/82] host allocation: *_shared_mark_ready: Allow other states Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 25/82] host allocation: *_shared_mark_ready: Make $sharetype check optional Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 26/82] host allocation: selecthost: allow sort-of-selection of prospective hosts Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 27/82] host allocation: *_shared_mark_ready: allow alternative $oldtypes Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 28/82] host allocation: Support new reuse-* magic hostflag Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 29/82] host allocation: *_shared_mark_ready: Only prod when $newstate is ready Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 30/82] plan_search: Break out $share_compat_ok Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 31/82] plan_search: Improve debugging of $share_compat_ok->() Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 32/82] plan_search: Use plan's Wear information rather than tracking it ourselves Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 33/82] plan search: Move $share_compat_ok further up the file Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 34/82] plan_search: Track last sharing state to determine $share_reuse Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 35/82] resource reporting: Print username when listing "rogue tasks" Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 36/82] resource reporting, nfc: Break out report_rogue_task_description Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 37/82] ts-hosts-allocate-Executive: Better message for hosts abandoned mid-test Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 38/82] ts-host-reuse: New script, to do reuse state changes Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 39/82] ts-host-reuse: tolerate unremoveable lv Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 40/82] ts-host-reuse: Do not depend on bios Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 41/82] ts-host-reuse: Add some missing runvars to the host sharing control Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 42/82] host allocation: selecthost(): Support @IDENT for reuse Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 43/82] ts-hosts-allocate-Executive print sharing info in debug output Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 44/82] sg-run-job: New @ iffail tag for prep tasks Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 45/82] sg-run-job: Detect improper use of @ iffail with run-ts Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 46/82] shared/reuse: Use @ for ts-xen-build-prep Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 47/82] shared/reuse: Use @ for ts-host-install Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 48/82] shared/reuse: Use @ for freebsd host prep Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 49/82] host reuse: sg-run-job: per-host prep: Use @ for per-host-ts Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 50/82] shared/reuse: Rely on @ for ts-host-ping-check Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 51/82] host reuse: Reuse test hosts within a flight Ian Jackson
2020-10-07 17:59 ` Ian Jackson [this message]
2020-10-07 17:59 ` [OSSTEST PATCH 53/82] host reuse: Jiggle the infra-priority a bit, " Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 54/82] host reuse: Use literal for the hosts_infraprioritygroup runvar Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 55/82] host reuse: Bump host share reuse bonus Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 56/82] subst_netboot_template: Do not use all of %r Ian Jackson
2020-10-07 17:59 ` [OSSTEST PATCH 57/82] TestSupport: Provide runvar_is_synth Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 58/82] runvar access: Introduce access control machinery Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 59/82] runvar access: Provide runvar_glob Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 60/82] runvar access: Use runvar_glob for dmrestrict runvar search Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 61/82] runvar access: Introduce sharing_for_build Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 62/82] runvar access: Introduce effects_gone_before_share_reuse Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 63/82] test host reuse: Switch to principled sharing scope runvar scheme Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 64/82] hsot reuse: Hash the share type Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 65/82] hsot reuse: Make share type hash more easily greppable Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 66/82] host lifecycle: Machinery, db, for tracking relevant events Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 67/82] host lifecycle: Fix detection of concurrent jobs Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 68/82] host lifecycle: Prevent referential integrity violation Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 69/82] host lifecycle: Record lifecycle in db and runvar Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 70/82] sg-report-flight: Refactor runvar access Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 71/82] resource reporting, nfc: split a here document Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 72/82] resource reporting: Report host reuse/sharing in job report Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 73/82] host reuse: sg-run-job: Reanme post-test-ok parameter Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 74/82] host reuse: ts-host-reuse: Prepare for argument handling Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 75/82] host reuse: New protocol between sg-run-job and ts-host-reuse Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 76/82] sg-report-flight: Reformat slightly Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 77/82] sg-report-flight: Improvements to other job (share/reuse) reporting Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 78/82] flight other job reporting: Put nulls last in the report Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 79/82] flight other job reporting: Further improvements to ordering Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 80/82] tsreadconfig: Change misleading "setting" message Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 81/82] sg-report-flight: Sharing reports: more task finished info Ian Jackson
2020-10-07 18:00 ` [OSSTEST PATCH 82/82] sg-report-flight: Word-wrapping improvements to job and step names Ian Jackson
2020-10-07 18:22 ` [OSSTEST PATCH 00/82] Reuse test hosts Ian Jackson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201007180024.7932-53-iwj@xenproject.org \
--to=iwj@xenproject.org \
--cc=ian.jackson@eu.citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).