* [Fuego] merge create/delete job scripts into ftc
@ 2017-03-27 5:49 Daniel Sangorrin
2017-03-27 5:49 ` [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs Daniel Sangorrin
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Daniel Sangorrin @ 2017-03-27 5:49 UTC (permalink / raw)
To: fuego
Hi,
The following 2 patches merge the create/delete job
scripts from fuego-ro/scripts into ftc.
[PATCH] ftc: add subcommands add-jobs and rm-jobs
[PATCH] fuego: remove create/delete job scripts
The new add-jobs subcommand is not using python-jenkins
yet, and it's still based on a testplan (if not testplan
is provided it will try to use testplan_default). In
the future I will add an option to specify the spec
instead of a testplan for the user to be able to add
tests one by one.
The rm-jobs subcommand has a bit more functionality that
the original fuego-delete-jobs script. Now you can select
which jobs to delete more specifically by using wildcards
such as docker.testplan_*.Benchmark.*stress.
Thanks,
Daniel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs
2017-03-27 5:49 [Fuego] merge create/delete job scripts into ftc Daniel Sangorrin
@ 2017-03-27 5:49 ` Daniel Sangorrin
2017-03-28 21:27 ` Bird, Timothy
2017-03-27 5:49 ` [Fuego] [PATCH] fuego: remove create/delete job scripts Daniel Sangorrin
2017-03-28 21:12 ` [Fuego] merge create/delete job scripts into ftc Bird, Timothy
2 siblings, 1 reply; 7+ messages in thread
From: Daniel Sangorrin @ 2017-03-27 5:49 UTC (permalink / raw)
To: fuego
This completes the merge of toshiba scripts from
fuego-ro/scripts into ftc.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/scripts/ftc | 280 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 274 insertions(+), 6 deletions(-)
diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 574d6a4..59be6d1 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -84,11 +84,24 @@ TARGET_ENV_VAR="FTC_TARGET"
# format for command_help mapping with: key=name, value=(summary, long description)
command_help = {
-"rm-nodes":("Removes nodes from Jenkins.",
- """Usage: ftc rm-nodes [<target1> <target2> ...]
- Use list-nodes to see the existing nodes.
+"add-jobs":("Adds jobs to Jenkins.",
+ """Usage: ftc add-jobs -b <target> [-p <testplan> | -t <testcase>] [-d <foo.dist>]
+ Example: sudo -u jenkins ftc add-jobs -b docker -p testplan_docker
+ Example: sudo -u jenkins ftc add-jobs -b docker -t Benchmark.Dhrystone
+ Use list-plans to see the available test plans.
- If no target is provided all existing nodes will be removed."""),
+ This interface may change in the future."""),
+
+"rm-jobs":("Removes jobs from Jenkins.",
+ """Usage: ftc rm-jobs <target>.<testplan>.<testcase>
+ Use list-jobs to see the existing jobs. A wildcard can be used to
+ specify which jobs to remove (just make sure you have 4 words):
+ Example: sudo -u jenkins ftc rm-jobs docker.testplan_docker.*.*
+ Example: sudo -u jenkins ftc rm-jobs docker.*.F*.*stress
+ Multiple combinations of the <target>.<testplan>.<testcase> pattern
+ can be passed as well.
+
+ If no option is provided all existing jobs will be removed."""),
"add-nodes":("Adds new nodes to Jenkins.",
"""Usage: ftc add-nodes [-f] <target1> <target2> ...
@@ -98,6 +111,12 @@ command_help = {
Use -f for "force" mode. This tries to remove the node before adding it"""),
+"rm-nodes":("Removes nodes from Jenkins.",
+ """Usage: ftc rm-nodes [<target1> <target2> ...]
+ Use list-nodes to see the existing nodes.
+
+ If no target is provided all existing nodes will be removed."""),
+
"list-targets":("Show a list of available target boards.",
"""Usage: ftc list-targets [-q]
Prints target board names and summary information, if any.
@@ -847,6 +866,240 @@ def get_includes(include_filename, conf):
inc_vars = parse_shell_file(inc_path, conf)
return inc_vars
+def create_job(board, distrib, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks):
+ # flot only necessary for Benchmarks
+ if str(test).split('.')[0] == 'Benchmark':
+ flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
+ else:
+ flot_link = ''
+
+ # prepare links for descriptionsetter. Put a link to testlog.txt by default
+ template_link = '<a href="/fuego/userContent/fuego.logs/%s/${NODE_NAME}.${BUILD_ID}.${BUILD_ID}/%%s">%%s</a>' % test
+ html_links = template_link % ('testlog.txt', 'log')
+ if jobextralinks != 'None':
+ for key, value in jobextralinks.iteritems():
+ html_links = html_links + ' ' + template_link % (value, key)
+
+ tmp = "/tmp/fuego_tmp_job"
+ fd = open(tmp, "w+")
+ fd.write("""<?xml version='1.0' encoding='UTF-8'?>
+<project>
+ <actions/>
+ <description></description>
+ <keepDependencies>false</keepDependencies>
+ <properties/>
+ <scm class="hudson.scm.NullSCM"/>
+ <assignedNode>{board}</assignedNode>
+ <canRoam>false</canRoam>
+ <disabled>false</disabled>
+ <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
+ <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
+ <triggers/>
+ <concurrentBuild>false</concurrentBuild>
+ <customWorkspace>$FUEGO_RW/buildzone</customWorkspace>
+ <builders>
+ <hudson.tasks.Shell>
+ <command>export Reboot={reboot}
+export Rebuild={rebuild}
+export Target_PreCleanup={precleanup}
+export Target_PostCleanup={postcleanup}
+export TESTDIR={testdir}
+export TESTNAME={testname}
+export DISTRIB="{distrib}"
+export TESTPLAN="{testplan}"
+timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
+</command>
+ </hudson.tasks.Shell>
+ </builders>
+ <publishers>
+ {flot_link}
+ <hudson.plugins.descriptionsetter.DescriptionSetterPublisher plugin="description-setter@1.10">
+ <regexp></regexp>
+ <regexpForFailed></regexpForFailed>
+ <description>{html_links}</description>
+ <setForMatrix>false</setForMatrix>
+ </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
+ </publishers>
+ <buildWrappers/>
+</project>
+""".format(board=board, reboot=str(reboot), rebuild=str(rebuild), precleanup=str(precleanup), postcleanup=str(postcleanup),
+ distrib=distrib, testdir=str(test), testname=str(test).split('.')[1],
+ testplan=testplan, timeout=timeout, flot_link=flot_link, html_links=html_links))
+ fd.close()
+
+ print("Creating job " + test)
+ try:
+ subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' +
+ board + '.' + testplan + '.' + str(test) + ' < ' + tmp, shell=True)
+ subprocess.call('rm -f ' + tmp, shell=True)
+ except Exception as e:
+ print("Job already exists")
+ print(e)
+ sys.exit(1)
+
+def create_batch_job(board, testplan, alltests):
+ tmp = "/tmp/fuego_tmp_batch"
+ prefixed_alltests = [board + '.' + testplan + '.' + test for test in alltests]
+ fd = open(tmp, "w+")
+ fd.write("""<?xml version='1.0' encoding='UTF-8'?>
+<project>
+ <actions/>
+ <description></description>
+ <keepDependencies>false</keepDependencies>
+ <properties/>
+ <scm class="hudson.scm.NullSCM"/>
+ <assignedNode>{board}</assignedNode>
+ <canRoam>false</canRoam>
+ <disabled>false</disabled>
+ <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
+ <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
+ <triggers/>
+ <concurrentBuild>false</concurrentBuild>
+ <builders/>
+ <publishers>
+ <hudson.tasks.BuildTrigger>
+ <childProjects>{alltests}</childProjects>
+ <threshold>
+ <name>SUCCESS</name>
+ <ordinal>0</ordinal>
+ <color>BLUE</color>
+ <completeBuild>true</completeBuild>
+ </threshold>
+ </hudson.tasks.BuildTrigger>
+ </publishers>
+ <buildWrappers/>
+</project>
+""".format(board=board, alltests=','.join(prefixed_alltests)))
+ fd.close()
+
+ print("Creating batch job ")
+ try:
+ subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' +
+ board + '.' + testplan + '.batch < ' + tmp, shell=True)
+ subprocess.call('rm -f ' + tmp, shell=True)
+ except Exception as e:
+ print("Job already exists")
+ print(e)
+ sys.exit(1)
+
+def parse_testplan(testplan):
+ abspath = '/fuego-core/engine/overlays/testplans/' + testplan + '.json'
+ alltests = []
+ alltimeouts = []
+ allextralinks = []
+ allreboots = []
+ allrebuilds = []
+ allprecleanups = []
+ allpostcleanups = []
+
+ with open(abspath, "r") as f:
+ plan = json.load(f)
+ for test in plan['tests']:
+ alltests.append(test['testName'])
+ if 'timeout' in test:
+ alltimeouts.append(test['timeout'])
+ else:
+ alltimeouts.append('20m')
+ if 'reboot' in test:
+ allreboots.append(test['reboot'])
+ else:
+ allreboots.append('false')
+ if 'rebuild' in test:
+ allrebuilds.append(test['rebuild'])
+ else:
+ allrebuilds.append('true')
+ if 'precleanup' in test:
+ allprecleanups.append(test['precleanup'])
+ else:
+ allprecleanups.append('true')
+ if 'postcleanup' in test:
+ allpostcleanups.append(test['postcleanup'])
+ else:
+ allpostcleanups.append('true')
+ if 'extralinks' in test:
+ allextralinks.append(test['extralinks'])
+ else:
+ allextralinks.append('None')
+ return alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks
+
+def do_add_jobs(conf, options):
+ if '-b' in options:
+ board = options[options.index('-b') + 1]
+ target_list = get_fuego_targets(conf).keys()
+ if board not in target_list:
+ raise Exception('Board %s not found.' % board)
+ options.remove('-b')
+ options.remove(board)
+ else:
+ raise Exception('No board name supplied.')
+
+ if '-p' in options:
+ testplan = options[options.index('-p') + 1]
+ plan_list = get_plans(conf).keys()
+ if testplan not in plan_list:
+ raise Exception('Testplan %s not found.' % testplan)
+ options.remove('-p')
+ options.remove(testplan)
+ test = None
+ elif '-t' in options:
+ test = options[options.index('-t') + 1]
+ test_list = get_tests(conf).keys()
+ if test not in test_list:
+ raise Exception('Test %s not found.' % test)
+ options.remove('-t')
+ options.remove(test)
+ # TODO: use spec instead of testplan_default
+ testplan = 'testplan_default'
+ else:
+ raise Exception('No testplan or testcase supplied.')
+
+ if '-d' in options:
+ distrib = options[options.index('-d') + 1]
+ # TODO: distrib should be in the board's configuration
+ options.remove('-d')
+ options.remove(distrib)
+ else:
+ distrib = 'nosyslogd.dist'
+
+ if test:
+ timeout = '20m'
+ reboot = 'false'
+ rebuild = 'true'
+ precleanup = 'true'
+ postcleanup = 'true'
+ jobextralinks = 'None'
+ create_job(board, distrib, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
+ else:
+ alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks = parse_testplan(testplan)
+ for test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks):
+ create_job(board, distrib, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
+ create_batch_job(board, testplan, alltests)
+ sys.exit(0)
+
+def do_rm_jobs(conf, options):
+ if len(options) == 0:
+ # TODO: warn user that all jobs will be removed
+ jobs = [job['name'] for job in server.get_jobs()]
+ for job in jobs:
+ server.delete_job(job)
+ else:
+ for opt in options:
+ jobs = [job['name'] for job in server.get_jobs()]
+ if len(opt.split('.')) != 4:
+ raise Exception('%s has not 4 words.' % opt)
+ pattern = '^' + opt
+ pattern = pattern.replace('.', '\.')
+ pattern = pattern.replace('*', '\w*')
+ pattern = pattern + '$'
+ regex = re.compile(pattern)
+ matches = [string for string in jobs if re.match(regex, string)]
+ if len(matches) == 0:
+ raise Exception('No jobs matched %s.' % opt)
+ else:
+ for match in matches:
+ server.delete_job(match)
+ sys.exit(0)
+
def do_add_nodes(conf, options):
global server
@@ -886,6 +1139,7 @@ def do_rm_nodes(conf, options):
nodes = [node['name'] for node in server.get_nodes()]
if not options:
+ # TODO: add warning or ask for -f
options = nodes
else:
for board in options:
@@ -2658,15 +2912,29 @@ def main():
# read config
conf = config_class(config_dir + os.sep + CONFIG_FILE)
+ if command=="add-jobs":
+ # adds Jenkins jobs
+ try:
+ do_add_jobs(conf, options)
+ except Exception as e:
+ sys.exit(str(e) + '\n' + command_help['add-jobs'][1])
+
+ if command=="rm-jobs":
+ # removes Jenkins jobs
+ try:
+ do_rm_jobs(conf, options)
+ except Exception as e:
+ sys.exit(str(e) + '\n' + command_help['rm-jobs'][1])
+
if command=="add-nodes":
- # adds a Jenkins node
+ # adds Jenkins nodes
try:
do_add_nodes(conf, options)
except Exception as e:
sys.exit(str(e) + '\n' + command_help['add-nodes'][1])
if command=="rm-nodes":
- # removes a Jenkins node
+ # removes Jenkins nodes
try:
do_rm_nodes(conf, options)
except Exception as e:
--
2.7.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Fuego] [PATCH] fuego: remove create/delete job scripts
2017-03-27 5:49 [Fuego] merge create/delete job scripts into ftc Daniel Sangorrin
2017-03-27 5:49 ` [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs Daniel Sangorrin
@ 2017-03-27 5:49 ` Daniel Sangorrin
2017-03-28 21:13 ` Bird, Timothy
2017-03-28 21:12 ` [Fuego] merge create/delete job scripts into ftc Bird, Timothy
2 siblings, 1 reply; 7+ messages in thread
From: Daniel Sangorrin @ 2017-03-27 5:49 UTC (permalink / raw)
To: fuego
This scripts are now integrated into the ftc tool.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
README | 8 +-
fuego-ro/scripts/jobs/fuego-create-jobs | 199 --------------------------------
fuego-ro/scripts/jobs/fuego-delete-jobs | 32 -----
3 files changed, 5 insertions(+), 234 deletions(-)
delete mode 100755 fuego-ro/scripts/jobs/fuego-create-jobs
delete mode 100755 fuego-ro/scripts/jobs/fuego-delete-jobs
diff --git a/README b/README
index 329da20..398a38e 100644
--- a/README
+++ b/README
@@ -80,10 +80,12 @@ You can use the docker container as a fuego target in order to confirm that
the installation was successful.
# sudo -u jenkins ftc add-nodes docker
-# fuego-create-jobs --board docker --testplan testplan_docker --distrib nosyslogd.dist
+# sudo -u jenkins ftc add-jobs -b docker -p testplan_docker
-Build the job "docker.testplan_docker.batch", it will trigger the rest of
-the jobs. Use fuego-delete-jobs to remove all jobs.
+On the Jenkins interface, build the "docker.testplan_docker.batch" job.
+It will trigger the rest of the jobs. To remove the jobs use:
+
+# sudo -u jenkins ftc rm-jobs docker.testplan_docker.*.*
License
-------------------------------------------------------------------------------
diff --git a/fuego-ro/scripts/jobs/fuego-create-jobs b/fuego-ro/scripts/jobs/fuego-create-jobs
deleted file mode 100755
index b4df00e..0000000
--- a/fuego-ro/scripts/jobs/fuego-create-jobs
+++ /dev/null
@@ -1,199 +0,0 @@
-#!/usr/bin/python
-#
-# fuego-create-jobs - a tool to create a fuego jobs
-#
-# Copyright 2016 TOSHIBA Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of version 2 of the GNU General Public License as
-# published by the Free Software Foundation. The GNU General Public
-# License is available online at: http://www.gnu.org/copyleft/gpl.html
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# Author: Daniel Sangorrin <daniel.sangorrin (at) toshiba.co.jp>
-#
-import subprocess
-import sys
-import argparse
-import json
-
-def create_job(test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks, args):
- # flot only necessary for Benchmarks
- if str(test).split('.')[0] == 'Benchmark':
- flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
- else:
- flot_link = ''
- # prepare links for descriptionsetter. Put a link to log by default
- template_link = '<a href="/fuego/userContent/fuego.logs/%s/${NODE_NAME}.${BUILD_ID}.${BUILD_ID}/%%s">%%s</a>' % test
- html_links = template_link % ('testlog.txt', 'log')
- if jobextralinks != 'None':
- for key, value in jobextralinks.iteritems():
- html_links = html_links + ' ' + template_link % (value, key)
-
- tmp = "/tmp/fuego_tmp_job"
- fd = open(tmp, "w+")
- fd.write("""<?xml version='1.0' encoding='UTF-8'?>
-<project>
- <actions/>
- <description></description>
- <keepDependencies>false</keepDependencies>
- <properties/>
- <scm class="hudson.scm.NullSCM"/>
- <assignedNode>{board}</assignedNode>
- <canRoam>false</canRoam>
- <disabled>false</disabled>
- <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
- <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
- <triggers/>
- <concurrentBuild>false</concurrentBuild>
- <customWorkspace>$FUEGO_RW/buildzone</customWorkspace>
- <builders>
- <hudson.tasks.Shell>
- <command>export Reboot={reboot}
-export Rebuild={rebuild}
-export Target_PreCleanup={precleanup}
-export Target_PostCleanup={postcleanup}
-export TESTDIR={testdir}
-export TESTNAME={testname}
-export DISTRIB="{distrib}"
-export TESTPLAN="{testplan}"
-timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
-</command>
- </hudson.tasks.Shell>
- </builders>
- <publishers>
- {flot_link}
- <hudson.plugins.descriptionsetter.DescriptionSetterPublisher plugin="description-setter@1.10">
- <regexp></regexp>
- <regexpForFailed></regexpForFailed>
- <description>{html_links}</description>
- <setForMatrix>false</setForMatrix>
- </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
- </publishers>
- <buildWrappers/>
-</project>
-""".format(board=args.board, reboot=str(reboot), rebuild=str(rebuild), precleanup=str(precleanup), postcleanup=str(postcleanup),
- distrib=args.distrib, testdir=str(test), testname=str(test).split('.')[1],
- testplan=args.testplan, timeout=timeout, flot_link=flot_link, html_links=html_links))
- fd.close()
-
- print("Creating job " + test)
- try:
- subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' + str(args.board) + '.' + str(args.testplan) + '.' + str(test) + ' < ' + tmp, shell=True)
- subprocess.call('rm -f ' + tmp, shell=True)
- except Exception as e:
- print("Job already exists")
- print(e)
- sys.exit(1)
-
-def create_batch_job(alltests, args):
- tmp = "/tmp/fuego_tmp_batch"
- prefixed_alltests = [str(args.board) + '.' + str(args.testplan) + '.' + test for test in alltests]
- fd = open(tmp, "w+")
- fd.write("""<?xml version='1.0' encoding='UTF-8'?>
-<project>
- <actions/>
- <description></description>
- <keepDependencies>false</keepDependencies>
- <properties/>
- <scm class="hudson.scm.NullSCM"/>
- <assignedNode>{board}</assignedNode>
- <canRoam>false</canRoam>
- <disabled>false</disabled>
- <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
- <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
- <triggers/>
- <concurrentBuild>false</concurrentBuild>
- <builders/>
- <publishers>
- <hudson.tasks.BuildTrigger>
- <childProjects>{alltests}</childProjects>
- <threshold>
- <name>SUCCESS</name>
- <ordinal>0</ordinal>
- <color>BLUE</color>
- <completeBuild>true</completeBuild>
- </threshold>
- </hudson.tasks.BuildTrigger>
- </publishers>
- <buildWrappers/>
-</project>
-""".format(board=args.board, alltests=','.join(prefixed_alltests)))
- fd.close()
-
- print("Creating batch job ")
- try:
- subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' + str(args.board) + '.' + str(args.testplan) + '.batch < ' + tmp, shell=True)
- subprocess.call('rm -f ' + tmp, shell=True)
- except Exception as e:
- print("Job already exists")
- print(e)
- sys.exit(1)
-
-def parse_testplan(testplan):
- abspath = '/fuego-core/engine/overlays/testplans/' + testplan + '.json'
- alltests = []
- alltimeouts = []
- allextralinks = []
- allreboots = []
- allrebuilds = []
- allprecleanups = []
- allpostcleanups = []
-
- with open(abspath, "r") as f:
- plan = json.load(f)
- for test in plan['tests']:
- alltests.append(test['testName'])
- if 'timeout' in test:
- alltimeouts.append(test['timeout'])
- else:
- alltimeouts.append('20m')
- if 'reboot' in test:
- allreboots.append(test['reboot'])
- else:
- allreboots.append('false')
- if 'rebuild' in test:
- allrebuilds.append(test['rebuild'])
- else:
- allrebuilds.append('true')
- if 'precleanup' in test:
- allprecleanups.append(test['precleanup'])
- else:
- allprecleanups.append('true')
- if 'postcleanup' in test:
- allpostcleanups.append(test['postcleanup'])
- else:
- allpostcleanups.append('true')
- if 'extralinks' in test:
- allextralinks.append(test['extralinks'])
- else:
- allextralinks.append('None')
- return alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks
-
-if __name__=="__main__":
- parser = argparse.ArgumentParser(description='Create a fuego jobs for a node')
- parser.add_argument('-b', '--board', type=str, help='board name (e.g.: bbb for bbb.board)', required=True)
- parser.add_argument('-t', '--testplan', default='testplan_default', type=str, help='testplan (e.g.: testplan_docker for testplan_docker.json)')
- parser.add_argument('-d', '--distrib', default='nosyslogd.dist', type=str, help='DISTRIB file (e.g.: nosyslogd.dist)')
- parser.add_argument('-c', '--testcase', type=str, help='Testcase (e.g.: Benchmark.Dhrystone)')
- args = parser.parse_args()
-
- alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks = parse_testplan(args.testplan)
-
- if args.testcase is not None:
- try:
- i = alltests.index(args.testcase)
- except Exception as e:
- print(e)
- print("ERROR: " + args.testcase + " not in " + args.testplan)
- print(alltests)
- sys.exit(1)
- create_job(alltests[i], alltimeouts[i], allreboots[i], allrebuilds[i], allprecleanups[i], allpostcleanups[i], allextralinks[i], args)
- else:
- for test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks):
- create_job(test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks, args)
- create_batch_job(alltests, args)
diff --git a/fuego-ro/scripts/jobs/fuego-delete-jobs b/fuego-ro/scripts/jobs/fuego-delete-jobs
deleted file mode 100755
index ffe26ce..0000000
--- a/fuego-ro/scripts/jobs/fuego-delete-jobs
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/usr/bin/python
-#
-# fuego-create-jobs - a tool to create a fuego jobs
-#
-# Copyright 2016 TOSHIBA Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of version 2 of the GNU General Public License as
-# published by the Free Software Foundation. The GNU General Public
-# License is available online at: http://www.gnu.org/copyleft/gpl.html
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# Author: Daniel Sangorrin <daniel.sangorrin (at) toshiba.co.jp>
-#
-import subprocess
-import sys
-
-if __name__=="__main__":
- print("Deleting jobs")
- try:
- jobs = subprocess.check_output('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego list-jobs', shell=True).splitlines()
- for job in jobs:
- subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego delete-job ' + str(job), shell=True)
- sys.stdout.write(".")
- sys.stdout.flush()
- except Exception as e:
- print(e)
- sys.exit(1)
--
2.7.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Fuego] merge create/delete job scripts into ftc
2017-03-27 5:49 [Fuego] merge create/delete job scripts into ftc Daniel Sangorrin
2017-03-27 5:49 ` [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs Daniel Sangorrin
2017-03-27 5:49 ` [Fuego] [PATCH] fuego: remove create/delete job scripts Daniel Sangorrin
@ 2017-03-28 21:12 ` Bird, Timothy
2 siblings, 0 replies; 7+ messages in thread
From: Bird, Timothy @ 2017-03-28 21:12 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
> -----Original Message-----
> From: Daniel Sangorrin on Sunday, March 26, 2017 10:49 PM
> Hi,
>
> The following 2 patches merge the create/delete job
> scripts from fuego-ro/scripts into ftc.
>
> [PATCH] ftc: add subcommands add-jobs and rm-jobs
> [PATCH] fuego: remove create/delete job scripts
>
> The new add-jobs subcommand is not using python-jenkins
> yet, and it's still based on a testplan (if not testplan
> is provided it will try to use testplan_default). In
> the future I will add an option to specify the spec
> instead of a testplan for the user to be able to add
> tests one by one.
>
> The rm-jobs subcommand has a bit more functionality that
> the original fuego-delete-jobs script. Now you can select
> which jobs to delete more specifically by using wildcards
> such as docker.testplan_*.Benchmark.*stress.
Sounds very nice. I'm reviewing now and will give feedback on the
individual patches. I'm going to integrate these, but I may make some
modifications. So watch for those. I'll not modifications in the patch feedback
on subsequent e-mails.
-- Tim
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Fuego] [PATCH] fuego: remove create/delete job scripts
2017-03-27 5:49 ` [Fuego] [PATCH] fuego: remove create/delete job scripts Daniel Sangorrin
@ 2017-03-28 21:13 ` Bird, Timothy
0 siblings, 0 replies; 7+ messages in thread
From: Bird, Timothy @ 2017-03-28 21:13 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
> -----Original Message-----
> From: Daniel Sangorrin on Sunday, March 26, 2017 10:49 PM
>
> This scripts are now integrated into the ftc tool.
>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> README | 8 +-
> fuego-ro/scripts/jobs/fuego-create-jobs | 199 --------------------------------
> fuego-ro/scripts/jobs/fuego-delete-jobs | 32 -----
> 3 files changed, 5 insertions(+), 234 deletions(-)
> delete mode 100755 fuego-ro/scripts/jobs/fuego-create-jobs
> delete mode 100755 fuego-ro/scripts/jobs/fuego-delete-jobs
>
> diff --git a/README b/README
> index 329da20..398a38e 100644
> --- a/README
> +++ b/README
> @@ -80,10 +80,12 @@ You can use the docker container as a fuego target in
> order to confirm that
> the installation was successful.
>
> # sudo -u jenkins ftc add-nodes docker
> -# fuego-create-jobs --board docker --testplan testplan_docker --distrib
> nosyslogd.dist
> +# sudo -u jenkins ftc add-jobs -b docker -p testplan_docker
>
> -Build the job "docker.testplan_docker.batch", it will trigger the rest of
> -the jobs. Use fuego-delete-jobs to remove all jobs.
> +On the Jenkins interface, build the "docker.testplan_docker.batch" job.
> +It will trigger the rest of the jobs. To remove the jobs use:
> +
> +# sudo -u jenkins ftc rm-jobs docker.testplan_docker.*.*
>
> License
> -------------------------------------------------------------------------------
> diff --git a/fuego-ro/scripts/jobs/fuego-create-jobs b/fuego-
> ro/scripts/jobs/fuego-create-jobs
> deleted file mode 100755
> index b4df00e..0000000
> --- a/fuego-ro/scripts/jobs/fuego-create-jobs
> +++ /dev/null
> @@ -1,199 +0,0 @@
> -#!/usr/bin/python
> -#
> -# fuego-create-jobs - a tool to create a fuego jobs
> -#
> -# Copyright 2016 TOSHIBA Corporation
> -#
> -# This program is free software; you can redistribute it and/or modify
> -# it under the terms of version 2 of the GNU General Public License as
> -# published by the Free Software Foundation. The GNU General Public
> -# License is available online at: http://www.gnu.org/copyleft/gpl.html
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# Author: Daniel Sangorrin <daniel.sangorrin (at) toshiba.co.jp>
> -#
> -import subprocess
> -import sys
> -import argparse
> -import json
> -
> -def create_job(test, timeout, reboot, rebuild, precleanup, postcleanup,
> jobextralinks, args):
> - # flot only necessary for Benchmarks
> - if str(test).split('.')[0] == 'Benchmark':
> - flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
> - else:
> - flot_link = ''
> - # prepare links for descriptionsetter. Put a link to log by default
> - template_link = '<a
> href="/fuego/userContent/fuego.logs/%s/${NODE_NAME}.${BUILD_I
> D}.${BUILD_ID}/%%s">%%s</a>' % test
> - html_links = template_link % ('testlog.txt', 'log')
> - if jobextralinks != 'None':
> - for key, value in jobextralinks.iteritems():
> - html_links = html_links + ' ' + template_link % (value, key)
> -
> - tmp = "/tmp/fuego_tmp_job"
> - fd = open(tmp, "w+")
> - fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> -<project>
> - <actions/>
> - <description></description>
> - <keepDependencies>false</keepDependencies>
> - <properties/>
> - <scm class="hudson.scm.NullSCM"/>
> - <assignedNode>{board}</assignedNode>
> - <canRoam>false</canRoam>
> - <disabled>false</disabled>
> -
> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstream
> Building>
> -
> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuildin
> g>
> - <triggers/>
> - <concurrentBuild>false</concurrentBuild>
> - <customWorkspace>$FUEGO_RW/buildzone</customWorkspace>
> - <builders>
> - <hudson.tasks.Shell>
> - <command>export Reboot={reboot}
> -export Rebuild={rebuild}
> -export Target_PreCleanup={precleanup}
> -export Target_PostCleanup={postcleanup}
> -export TESTDIR={testdir}
> -export TESTNAME={testname}
> -export DISTRIB="{distrib}"
> -export TESTPLAN="{testplan}"
> -timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
> -</command>
> - </hudson.tasks.Shell>
> - </builders>
> - <publishers>
> - {flot_link}
> - <hudson.plugins.descriptionsetter.DescriptionSetterPublisher
> plugin="description-setter@1.10">
> - <regexp></regexp>
> - <regexpForFailed></regexpForFailed>
> - <description>{html_links}</description>
> - <setForMatrix>false</setForMatrix>
> - </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
> - </publishers>
> - <buildWrappers/>
> -</project>
> -""".format(board=args.board, reboot=str(reboot), rebuild=str(rebuild),
> precleanup=str(precleanup), postcleanup=str(postcleanup),
> - distrib=args.distrib, testdir=str(test), testname=str(test).split('.')[1],
> - testplan=args.testplan, timeout=timeout, flot_link=flot_link,
> html_links=html_links))
> - fd.close()
> -
> - print("Creating job " + test)
> - try:
> - subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-
> cli.jar -s http://localhost:8080/fuego create-job ' + str(args.board) + '.' +
> str(args.testplan) + '.' + str(test) + ' < ' + tmp, shell=True)
> - subprocess.call('rm -f ' + tmp, shell=True)
> - except Exception as e:
> - print("Job already exists")
> - print(e)
> - sys.exit(1)
> -
> -def create_batch_job(alltests, args):
> - tmp = "/tmp/fuego_tmp_batch"
> - prefixed_alltests = [str(args.board) + '.' + str(args.testplan) + '.' + test for
> test in alltests]
> - fd = open(tmp, "w+")
> - fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> -<project>
> - <actions/>
> - <description></description>
> - <keepDependencies>false</keepDependencies>
> - <properties/>
> - <scm class="hudson.scm.NullSCM"/>
> - <assignedNode>{board}</assignedNode>
> - <canRoam>false</canRoam>
> - <disabled>false</disabled>
> -
> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstream
> Building>
> -
> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuildin
> g>
> - <triggers/>
> - <concurrentBuild>false</concurrentBuild>
> - <builders/>
> - <publishers>
> - <hudson.tasks.BuildTrigger>
> - <childProjects>{alltests}</childProjects>
> - <threshold>
> - <name>SUCCESS</name>
> - <ordinal>0</ordinal>
> - <color>BLUE</color>
> - <completeBuild>true</completeBuild>
> - </threshold>
> - </hudson.tasks.BuildTrigger>
> - </publishers>
> - <buildWrappers/>
> -</project>
> -""".format(board=args.board, alltests=','.join(prefixed_alltests)))
> - fd.close()
> -
> - print("Creating batch job ")
> - try:
> - subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-
> cli.jar -s http://localhost:8080/fuego create-job ' + str(args.board) + '.' +
> str(args.testplan) + '.batch < ' + tmp, shell=True)
> - subprocess.call('rm -f ' + tmp, shell=True)
> - except Exception as e:
> - print("Job already exists")
> - print(e)
> - sys.exit(1)
> -
> -def parse_testplan(testplan):
> - abspath = '/fuego-core/engine/overlays/testplans/' + testplan + '.json'
> - alltests = []
> - alltimeouts = []
> - allextralinks = []
> - allreboots = []
> - allrebuilds = []
> - allprecleanups = []
> - allpostcleanups = []
> -
> - with open(abspath, "r") as f:
> - plan = json.load(f)
> - for test in plan['tests']:
> - alltests.append(test['testName'])
> - if 'timeout' in test:
> - alltimeouts.append(test['timeout'])
> - else:
> - alltimeouts.append('20m')
> - if 'reboot' in test:
> - allreboots.append(test['reboot'])
> - else:
> - allreboots.append('false')
> - if 'rebuild' in test:
> - allrebuilds.append(test['rebuild'])
> - else:
> - allrebuilds.append('true')
> - if 'precleanup' in test:
> - allprecleanups.append(test['precleanup'])
> - else:
> - allprecleanups.append('true')
> - if 'postcleanup' in test:
> - allpostcleanups.append(test['postcleanup'])
> - else:
> - allpostcleanups.append('true')
> - if 'extralinks' in test:
> - allextralinks.append(test['extralinks'])
> - else:
> - allextralinks.append('None')
> - return alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks
> -
> -if __name__=="__main__":
> - parser = argparse.ArgumentParser(description='Create a fuego jobs for a
> node')
> - parser.add_argument('-b', '--board', type=str, help='board name (e.g.:
> bbb for bbb.board)', required=True)
> - parser.add_argument('-t', '--testplan', default='testplan_default',
> type=str, help='testplan (e.g.: testplan_docker for testplan_docker.json)')
> - parser.add_argument('-d', '--distrib', default='nosyslogd.dist', type=str,
> help='DISTRIB file (e.g.: nosyslogd.dist)')
> - parser.add_argument('-c', '--testcase', type=str, help='Testcase (e.g.:
> Benchmark.Dhrystone)')
> - args = parser.parse_args()
> -
> - alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks = parse_testplan(args.testplan)
> -
> - if args.testcase is not None:
> - try:
> - i = alltests.index(args.testcase)
> - except Exception as e:
> - print(e)
> - print("ERROR: " + args.testcase + " not in " + args.testplan)
> - print(alltests)
> - sys.exit(1)
> - create_job(alltests[i], alltimeouts[i], allreboots[i], allrebuilds[i],
> allprecleanups[i], allpostcleanups[i], allextralinks[i], args)
> - else:
> - for test, timeout, reboot, rebuild, precleanup, postcleanup,
> jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds,
> allprecleanups, allpostcleanups, allextralinks):
> - create_job(test, timeout, reboot, rebuild, precleanup, postcleanup,
> jobextralinks, args)
> - create_batch_job(alltests, args)
> diff --git a/fuego-ro/scripts/jobs/fuego-delete-jobs b/fuego-
> ro/scripts/jobs/fuego-delete-jobs
> deleted file mode 100755
> index ffe26ce..0000000
> --- a/fuego-ro/scripts/jobs/fuego-delete-jobs
> +++ /dev/null
> @@ -1,32 +0,0 @@
> -#!/usr/bin/python
> -#
> -# fuego-create-jobs - a tool to create a fuego jobs
> -#
> -# Copyright 2016 TOSHIBA Corporation
> -#
> -# This program is free software; you can redistribute it and/or modify
> -# it under the terms of version 2 of the GNU General Public License as
> -# published by the Free Software Foundation. The GNU General Public
> -# License is available online at: http://www.gnu.org/copyleft/gpl.html
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# Author: Daniel Sangorrin <daniel.sangorrin (at) toshiba.co.jp>
> -#
> -import subprocess
> -import sys
> -
> -if __name__=="__main__":
> - print("Deleting jobs")
> - try:
> - jobs = subprocess.check_output('java -jar
> /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s
> http://localhost:8080/fuego list-jobs', shell=True).splitlines()
> - for job in jobs:
> - subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-
> cli.jar -s http://localhost:8080/fuego delete-job ' + str(job), shell=True)
> - sys.stdout.write(".")
> - sys.stdout.flush()
> - except Exception as e:
> - print(e)
> - sys.exit(1)
> --
> 2.7.4
Acked-by: Tim Bird
I saw this and integrated into my 'next'.
-- Tim
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs
2017-03-27 5:49 ` [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs Daniel Sangorrin
@ 2017-03-28 21:27 ` Bird, Timothy
2017-03-28 23:44 ` Daniel Sangorrin
0 siblings, 1 reply; 7+ messages in thread
From: Bird, Timothy @ 2017-03-28 21:27 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
> -----Original Message-----
> From Daniel Sangorrin on Sunday, March 26, 2017 10:49 PM
> This completes the merge of toshiba scripts from
> fuego-ro/scripts into ftc.
>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> engine/scripts/ftc | 280
> +++++++++++++++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 274 insertions(+), 6 deletions(-)
>
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index 574d6a4..59be6d1 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -84,11 +84,24 @@ TARGET_ENV_VAR="FTC_TARGET"
>
> # format for command_help mapping with: key=name, value=(summary,
> long description)
> command_help = {
> -"rm-nodes":("Removes nodes from Jenkins.",
> - """Usage: ftc rm-nodes [<target1> <target2> ...]
> - Use list-nodes to see the existing nodes.
> +"add-jobs":("Adds jobs to Jenkins.",
> + """Usage: ftc add-jobs -b <target> [-p <testplan> | -t <testcase>] [-d
> <foo.dist>]
> + Example: sudo -u jenkins ftc add-jobs -b docker -p testplan_docker
> + Example: sudo -u jenkins ftc add-jobs -b docker -t Benchmark.Dhrystone
> + Use list-plans to see the available test plans.
>
> - If no target is provided all existing nodes will be removed."""),
> + This interface may change in the future."""),
> +
> +"rm-jobs":("Removes jobs from Jenkins.",
> + """Usage: ftc rm-jobs <target>.<testplan>.<testcase>
> + Use list-jobs to see the existing jobs. A wildcard can be used to
> + specify which jobs to remove (just make sure you have 4 words):
> + Example: sudo -u jenkins ftc rm-jobs docker.testplan_docker.*.*
> + Example: sudo -u jenkins ftc rm-jobs docker.*.F*.*stress
> + Multiple combinations of the <target>.<testplan>.<testcase> pattern
> + can be passed as well.
> +
> + If no option is provided all existing jobs will be removed."""),
>
> "add-nodes":("Adds new nodes to Jenkins.",
> """Usage: ftc add-nodes [-f] <target1> <target2> ...
> @@ -98,6 +111,12 @@ command_help = {
>
> Use -f for "force" mode. This tries to remove the node before adding it"""),
>
> +"rm-nodes":("Removes nodes from Jenkins.",
> + """Usage: ftc rm-nodes [<target1> <target2> ...]
> + Use list-nodes to see the existing nodes.
> +
> + If no target is provided all existing nodes will be removed."""),
> +
> "list-targets":("Show a list of available target boards.",
> """Usage: ftc list-targets [-q]
> Prints target board names and summary information, if any.
> @@ -847,6 +866,240 @@ def get_includes(include_filename, conf):
> inc_vars = parse_shell_file(inc_path, conf)
> return inc_vars
>
> +def create_job(board, distrib, testplan, test, timeout, reboot, rebuild,
> precleanup, postcleanup, jobextralinks):
> + # flot only necessary for Benchmarks
> + if str(test).split('.')[0] == 'Benchmark':
> + flot_link = '<flotile.FlotPublisher plugin="flot@1.0-
> SNAPSHOT"/>'
Cool. I presume this gets to the flot plot automatically.
> + else:
> + flot_link = ''
> +
> + # prepare links for descriptionsetter. Put a link to testlog.txt by
> default
> + template_link = '<a
> href="/fuego/userContent/fuego.logs/%s/${NODE_NAME}.${BUILD_I
> D}.${BUILD_ID}/%%s">%%s</a>' % test
> + html_links = template_link % ('testlog.txt', 'log')
> + if jobextralinks != 'None':
> + for key, value in jobextralinks.iteritems():
> + html_links = html_links + ' ' + template_link % (value,
> key)
> +
> + tmp = "/tmp/fuego_tmp_job"
> + fd = open(tmp, "w+")
> + fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> +<project>
> + <actions/>
> + <description></description>
> + <keepDependencies>false</keepDependencies>
> + <properties/>
> + <scm class="hudson.scm.NullSCM"/>
> + <assignedNode>{board}</assignedNode>
> + <canRoam>false</canRoam>
> + <disabled>false</disabled>
> +
> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstream
> Building>
> +
> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuildin
> g>
> + <triggers/>
> + <concurrentBuild>false</concurrentBuild>
> + <customWorkspace>$FUEGO_RW/buildzone</customWorkspace>
> + <builders>
> + <hudson.tasks.Shell>
> + <command>export Reboot={reboot}
> +export Rebuild={rebuild}
> +export Target_PreCleanup={precleanup}
> +export Target_PostCleanup={postcleanup}
> +export TESTDIR={testdir}
> +export TESTNAME={testname}
> +export DISTRIB="{distrib}"
> +export TESTPLAN="{testplan}"
> +timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
> +</command>
> + </hudson.tasks.Shell>
> + </builders>
> + <publishers>
> + {flot_link}
> + <hudson.plugins.descriptionsetter.DescriptionSetterPublisher
> plugin="description-setter@1.10">
> + <regexp></regexp>
> + <regexpForFailed></regexpForFailed>
> + <description>{html_links}</description>
> + <setForMatrix>false</setForMatrix>
> + </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
> + </publishers>
> + <buildWrappers/>
> +</project>
> +""".format(board=board, reboot=str(reboot), rebuild=str(rebuild),
> precleanup=str(precleanup), postcleanup=str(postcleanup),
> + distrib=distrib, testdir=str(test), testname=str(test).split('.')[1],
> + testplan=testplan, timeout=timeout, flot_link=flot_link,
> html_links=html_links))
I'll take this as is. I think it would be good to have the config.xml template
declared outside the function (not inline). That way, if we need to support
different strings for different Jenkins versions, it will be easier to select an
appropriate one. Also, I'm going to experiment with grabbing these values
from locals().
> + fd.close()
> +
> + print("Creating job " + test)
> + try:
> + subprocess.call('java -jar /var/cache/jenkins/war/WEB-
> INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' +
> + board + '.' + testplan + '.' + str(test) + ' < ' + tmp,
> shell=True)
> + subprocess.call('rm -f ' + tmp, shell=True)
> + except Exception as e:
> + print("Job already exists")
> + print(e)
> + sys.exit(1)
> +
> +def create_batch_job(board, testplan, alltests):
> + tmp = "/tmp/fuego_tmp_batch"
> + prefixed_alltests = [board + '.' + testplan + '.' + test for test in alltests]
> + fd = open(tmp, "w+")
> + fd.write("""<?xml version='1.0' encoding='UTF-8'?>
> +<project>
> + <actions/>
> + <description></description>
> + <keepDependencies>false</keepDependencies>
> + <properties/>
> + <scm class="hudson.scm.NullSCM"/>
> + <assignedNode>{board}</assignedNode>
> + <canRoam>false</canRoam>
> + <disabled>false</disabled>
> +
> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstream
> Building>
> +
> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuildin
> g>
> + <triggers/>
> + <concurrentBuild>false</concurrentBuild>
> + <builders/>
> + <publishers>
> + <hudson.tasks.BuildTrigger>
> + <childProjects>{alltests}</childProjects>
> + <threshold>
> + <name>SUCCESS</name>
> + <ordinal>0</ordinal>
> + <color>BLUE</color>
> + <completeBuild>true</completeBuild>
> + </threshold>
> + </hudson.tasks.BuildTrigger>
> + </publishers>
> + <buildWrappers/>
> +</project>
> +""".format(board=board, alltests=','.join(prefixed_alltests)))
Same comment here as above - would be good to put this template
outside the function.
> + fd.close()
> +
> + print("Creating batch job ")
> + try:
> + subprocess.call('java -jar /var/cache/jenkins/war/WEB-
> INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' +
> + board + '.' + testplan + '.batch < ' + tmp, shell=True)
> + subprocess.call('rm -f ' + tmp, shell=True)
> + except Exception as e:
> + print("Job already exists")
> + print(e)
> + sys.exit(1)
> +
> +def parse_testplan(testplan):
> + abspath = '/fuego-core/engine/overlays/testplans/' + testplan +
> '.json'
> + alltests = []
> + alltimeouts = []
> + allextralinks = []
> + allreboots = []
> + allrebuilds = []
> + allprecleanups = []
> + allpostcleanups = []
> +
> + with open(abspath, "r") as f:
> + plan = json.load(f)
> + for test in plan['tests']:
> + alltests.append(test['testName'])
> + if 'timeout' in test:
> + alltimeouts.append(test['timeout'])
> + else:
> + alltimeouts.append('20m')
> + if 'reboot' in test:
> + allreboots.append(test['reboot'])
> + else:
> + allreboots.append('false')
> + if 'rebuild' in test:
> + allrebuilds.append(test['rebuild'])
> + else:
> + allrebuilds.append('true')
> + if 'precleanup' in test:
> + allprecleanups.append(test['precleanup'])
> + else:
> + allprecleanups.append('true')
> + if 'postcleanup' in test:
> + allpostcleanups.append(test['postcleanup'])
> + else:
> + allpostcleanups.append('true')
> + if 'extralinks' in test:
> + allextralinks.append(test['extralinks'])
> + else:
> + allextralinks.append('None')
> + return alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks
I'm not fond of these parallel arrays. IMHO it would be better to have a list
of instances. But I'm leaving this for now.
> +
> +def do_add_jobs(conf, options):
> + if '-b' in options:
> + board = options[options.index('-b') + 1]
> + target_list = get_fuego_targets(conf).keys()
> + if board not in target_list:
> + raise Exception('Board %s not found.' % board)
> + options.remove('-b')
> + options.remove(board)
> + else:
> + raise Exception('No board name supplied.')
> +
> + if '-p' in options:
> + testplan = options[options.index('-p') + 1]
> + plan_list = get_plans(conf).keys()
> + if testplan not in plan_list:
> + raise Exception('Testplan %s not found.' % testplan)
> + options.remove('-p')
> + options.remove(testplan)
> + test = None
> + elif '-t' in options:
> + test = options[options.index('-t') + 1]
> + test_list = get_tests(conf).keys()
> + if test not in test_list:
> + raise Exception('Test %s not found.' % test)
> + options.remove('-t')
> + options.remove(test)
> + # TODO: use spec instead of testplan_default
> + testplan = 'testplan_default'
> + else:
> + raise Exception('No testplan or testcase supplied.')
> +
> + if '-d' in options:
> + distrib = options[options.index('-d') + 1]
> + # TODO: distrib should be in the board's configuration
> + options.remove('-d')
> + options.remove(distrib)
> + else:
> + distrib = 'nosyslogd.dist'
> +
> + if test:
> + timeout = '20m'
> + reboot = 'false'
> + rebuild = 'true'
> + precleanup = 'true'
> + postcleanup = 'true'
> + jobextralinks = 'None'
> + create_job(board, distrib, testplan, test, timeout, reboot,
> rebuild, precleanup, postcleanup, jobextralinks)
> + else:
> + alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks = parse_testplan(testplan)
> + for test, timeout, reboot, rebuild, precleanup, postcleanup,
> jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds,
> allprecleanups, allpostcleanups, allextralinks):
> + create_job(board, distrib, testplan, test, timeout,
> reboot, rebuild, precleanup, postcleanup, jobextralinks)
> + create_batch_job(board, testplan, alltests)
> + sys.exit(0)
> +
> +def do_rm_jobs(conf, options):
> + if len(options) == 0:
> + # TODO: warn user that all jobs will be removed
> + jobs = [job['name'] for job in server.get_jobs()]
> + for job in jobs:
> + server.delete_job(job)
> + else:
> + for opt in options:
> + jobs = [job['name'] for job in server.get_jobs()]
> + if len(opt.split('.')) != 4:
> + raise Exception('%s has not 4 words.' % opt)
"has not" -> "does not have"
> + pattern = '^' + opt
> + pattern = pattern.replace('.', '\.')
> + pattern = pattern.replace('*', '\w*')
> + pattern = pattern + '$'
> + regex = re.compile(pattern)
> + matches = [string for string in jobs if re.match(regex,
> string)]
> + if len(matches) == 0:
> + raise Exception('No jobs matched %s.' % opt)
> + else:
> + for match in matches:
> + server.delete_job(match)
> + sys.exit(0)
> +
> def do_add_nodes(conf, options):
> global server
>
> @@ -886,6 +1139,7 @@ def do_rm_nodes(conf, options):
>
> nodes = [node['name'] for node in server.get_nodes()]
> if not options:
> + # TODO: add warning or ask for -f
> options = nodes
> else:
> for board in options:
> @@ -2658,15 +2912,29 @@ def main():
> # read config
> conf = config_class(config_dir + os.sep + CONFIG_FILE)
>
> + if command=="add-jobs":
> + # adds Jenkins jobs
> + try:
> + do_add_jobs(conf, options)
> + except Exception as e:
> + sys.exit(str(e) + '\n' + command_help['add-jobs'][1])
> +
> + if command=="rm-jobs":
> + # removes Jenkins jobs
> + try:
> + do_rm_jobs(conf, options)
> + except Exception as e:
> + sys.exit(str(e) + '\n' + command_help['rm-jobs'][1])
> +
> if command=="add-nodes":
> - # adds a Jenkins node
> + # adds Jenkins nodes
> try:
> do_add_nodes(conf, options)
> except Exception as e:
> sys.exit(str(e) + '\n' + command_help['add-nodes'][1])
>
> if command=="rm-nodes":
> - # removes a Jenkins node
> + # removes Jenkins nodes
> try:
> do_rm_nodes(conf, options)
> except Exception as e:
> --
> 2.7.4
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs
2017-03-28 21:27 ` Bird, Timothy
@ 2017-03-28 23:44 ` Daniel Sangorrin
0 siblings, 0 replies; 7+ messages in thread
From: Daniel Sangorrin @ 2017-03-28 23:44 UTC (permalink / raw)
To: 'Bird, Timothy', fuego
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > + # flot only necessary for Benchmarks
> > + if str(test).split('.')[0] == 'Benchmark':
> > + flot_link = '<flotile.FlotPublisher plugin="flot@1.0-
> > SNAPSHOT"/>'
> Cool. I presume this gets to the flot plot automatically.
That's right. In the future, I will check the yaml file instead of the TESTDIR.
description
> > + <setForMatrix>false</setForMatrix>
> > + </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
> > + </publishers>
> > + <buildWrappers/>
> > +</project>
> > +""".format(board=board, reboot=str(reboot), rebuild=str(rebuild),
> > precleanup=str(precleanup), postcleanup=str(postcleanup),
> > + distrib=distrib, testdir=str(test), testname=str(test).split('.')[1],
> > + testplan=testplan, timeout=timeout, flot_link=flot_link,
> > html_links=html_links))
>
> I'll take this as is. I think it would be good to have the config.xml template
> declared outside the function (not inline). That way, if we need to support
> different strings for different Jenkins versions, it will be easier to select an
> appropriate one. Also, I'm going to experiment with grabbing these values
> from locals().
Thanks. Actually, I was planning to switch to python-jenkins in the next step hoping
they would solve the Jenkins versioning for us. For now, I just wanted to get it
working and then polish up from there.
> > + return alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> > allpostcleanups, allextralinks
>
> I'm not fond of these parallel arrays. IMHO it would be better to have a list
> of instances. But I'm leaving this for now.
Yes, this needs refactoring. I will fix it in the next revision. Sorry about that.
Thanks,
Daniel
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-03-28 23:44 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-27 5:49 [Fuego] merge create/delete job scripts into ftc Daniel Sangorrin
2017-03-27 5:49 ` [Fuego] [PATCH] ftc: add subcommands add-jobs and rm-jobs Daniel Sangorrin
2017-03-28 21:27 ` Bird, Timothy
2017-03-28 23:44 ` Daniel Sangorrin
2017-03-27 5:49 ` [Fuego] [PATCH] fuego: remove create/delete job scripts Daniel Sangorrin
2017-03-28 21:13 ` Bird, Timothy
2017-03-28 21:12 ` [Fuego] merge create/delete job scripts into ftc Bird, Timothy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.