[Fuego] testplans and job inter-dependency

[Fuego] testplans and job inter-dependency

[Daniel Sangorrin]

Hello Tim,

I prepared a couple of patches that fix testplans' job
execution order, and allow triggering a new job once
the current one is finished.

[PATCH 1/2] testplans: execute tests in order
[PATCH 2/2] add-jobs: add trigger parameter to specify next job

I put a mention on the commit logs but I wanted to discuss a couple
of questions:
- currently in my patches, the next job is triggered no matter
  if the current job fails or succeeds. That behavior can be
  easily changed. I want to know if you prefer to have a
  new interface to switch the behavior, or just hardwire it to
  a predefined one (e.g. "always continue" or "stop running jobs 
  if one fails").
  I can see cases where you want to stop (eg: if the kernel
  build fails, then the next tests are worthless). On the other
  hand, i can also see cases where you want to go on (e.g.
  when the jobs in the testplan do not really depend on each
  other). 
  If you like the "behavior switch" interface better, then
  I could add a parameter (parallel=True?) to each test in a testplan
  to define that behavior and have a default behavior when
  omitted. Any preferences for the default behavior?
- the second question is related to the interface used when
  adding jobs without a testplan. The current interface 
  (--trigger myjob) only allows you to define one job to
  trigger, but i can add more in the future. The question
  is: if i do --trigger job1,job2,job3, should all those
  jobs be triggered in order (one after the other) or not.
  If we want a "behavior switch" here, the interface could
  become a bit convoluted. For example:
   --trigger job1->job2->job3,job4,job5->job6
  This could be used to define jobs that can run in parallel,
  and jobs that must run in a specific order.

Your feedback is very welcomed.

Thanks,
Daniel
    


  

[Fuego] [PATCH 1/2] testplans: execute tests in order

[Daniel Sangorrin]

Before this patch, a batch job was triggering all of the jobs
in a testplan in parallel. For that reason, execution order
of the jobs could not be guaranteed.

This patch creates a "linked chain" of jobs where the batch job just
triggers the first job in the testplan, and then the first job
triggers the second one in turn.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc | 53 ++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 40 insertions(+), 13 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 6dc3d6d..6f605db 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -1291,7 +1291,7 @@ def link_key(item):
         return "zzz+"+key
 
 
-def create_job(board, test):
+def create_job(board, test, next_job=None):
     flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
 
     # prepare links for the descriptionsetter plugin
@@ -1400,13 +1400,34 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
       <descriptionForFailed>{fail_links}</descriptionForFailed>
       <setForMatrix>false</setForMatrix>
     </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
-    </publishers>
-    <buildWrappers/>
-</project>
 """.format(board=board, reboot=test.reboot, rebuild=test.rebuild,
         precleanup=test.precleanup, postcleanup=test.postcleanup,
         testdir=test.name, testspec=test.spec, timeout=test.timeout,
         flot_link=flot_link, success_links=success_links, fail_links=fail_links))
+
+    if next_job is None:
+        fd.write("""    </publishers>
+    <buildWrappers/>
+</project>
+""")
+    else:
+        # [Note] To trigger the next job only if this job is succesful use
+        # <name>SUCCESS</name>
+        # <ordinal>0</ordinal>
+        # <color>BLUE</color>
+        fd.write("""    <hudson.tasks.BuildTrigger>
+      <childProjects>{next_job}</childProjects>
+      <threshold>
+        <name>FAILURE</name>
+        <ordinal>2</ordinal>
+        <color>RED</color>
+        <completeBuild>true</completeBuild>
+      </threshold>
+    </hudson.tasks.BuildTrigger>
+    </publishers>
+    <buildWrappers/>
+</project>
+""".format(next_job=next_job))
     fd.close()
 
     job_name = board + "." + test.spec + "." + test.name
@@ -1421,9 +1442,8 @@ ftc run-test -b $NODE_NAME -t {testdir} -s {testspec} \\
         sys.exit(1)
 
 
-def create_batch_job(board, testplan, plan_tests):
+def create_batch_job(board_name, testplan, next_job):
     tmp = "/tmp/fuego_tmp_batch"
-    job_list = [board+'.'+test.spec+'.'+test.name for test in plan_tests]
 
     fd = open(tmp, "w+")
     fd.write("""<?xml version='1.0' encoding='UTF-8'?>
@@ -1433,7 +1453,7 @@ def create_batch_job(board, testplan, plan_tests):
   <keepDependencies>false</keepDependencies>
   <properties/>
   <scm class="hudson.scm.NullSCM"/>
-  <assignedNode>{board}</assignedNode>
+  <assignedNode>{board_name}</assignedNode>
   <canRoam>false</canRoam>
   <disabled>false</disabled>
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
@@ -1443,7 +1463,7 @@ def create_batch_job(board, testplan, plan_tests):
   <builders/>
   <publishers>
     <hudson.tasks.BuildTrigger>
-      <childProjects>{alljobs}</childProjects>
+      <childProjects>{next_job}</childProjects>
       <threshold>
         <name>SUCCESS</name>
         <ordinal>0</ordinal>
@@ -1454,13 +1474,13 @@ def create_batch_job(board, testplan, plan_tests):
   </publishers>
   <buildWrappers/>
 </project>
-""".format(board=board, alljobs=','.join(job_list)))
+""".format(board_name=board_name, next_job=next_job))
 
     fd.close()
 
     print("Creating batch job ")
     try:
-        job_name = board+'.'+testplan+'.batch'
+        job_name = board_name+'.'+testplan+'.batch'
         subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' +
              job_name + '< ' + tmp, shell=True)
         os.unlink(tmp)
@@ -1628,9 +1648,16 @@ def do_add_jobs(conf, options):
     else:
         plan_tests = parse_testplan(testplan, test_dict)
         for board_name in board_names:
-            for test in plan_tests:
-                create_job(board_name, test)
-            create_batch_job(board_name, testplan, plan_tests)
+            for index, test in enumerate(plan_tests):
+                if index == len(plan_tests)-1:
+                    next_job = None
+                else:
+                    next_test = plan_tests[index + 1]
+                    next_job = board_name + "." + next_test.spec + "." + next_test.name
+                create_job(board_name, test, next_job)
+            next_test =  plan_tests[0]
+            next_job = board_name + "." + next_test.spec + "." + next_test.name
+            create_batch_job(board_name, testplan, next_job)
     sys.exit(0)
 
 
-- 
2.7.4

[Fuego] [PATCH 2/2] add-jobs: add trigger parameter to specify next job

[Daniel Sangorrin]

This patch adds the option --trigger <next job> to the
add-jobs interface. It allows triggering a new job once it
finishes.

Currently, the next job is triggered not matter whether
the job fails or not (I wrote a comment that shows how
to trigger it only if the job succeeds). Probably, we
need to add a new flag to specify that behavior.

Another thing that could be added is the ability to trigger
more jobs.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 6f605db..007e5a8 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -102,7 +102,7 @@ command_help = {
 "add-jobs": ("Adds jobs to Jenkins.",
     """Usage: ftc add-jobs -b <board>[,board2...] [-p <testplan> | -t <testcase> -s <testspec>]
        [--timeout <timeout>] [--rebuild <true|false>] [--reboot <true|false>]
-       [--precleanup <true|false>] [--postcleanup <true|false>]
+       [--precleanup <true|false>] [--postcleanup <true|false>] [--trigger <next_job>]
   Example: ftc add-jobs -b docker -p testplan_docker
   Example: ftc add-jobs -b docker -t Benchmark.Dhrystone --timeout 5m --rebuild false
   board,testplan,testpec: use list-board/plans/specs to see the available ones.
@@ -111,6 +111,7 @@ command_help = {
   reboot: if true reboot the board before the test.
   precleanup: if true clean the board's test folder before the test.
   postcleanup: if true clean the board's test folder after the test.
+  trigger: job to trigger next
 
   Note that you can specify more than one board using a comma-separated
   list for the <board> argument. e.g.
@@ -1608,6 +1609,8 @@ def do_add_jobs(conf, options):
 
     if '-p' in options and '-t' in options:
         error_out('Testplan (-p) and test (-t) options are mutually exclusive.')
+    elif '-p' in options and '--trigger' in options:
+        error_out('Testplan (-p) and --trigger options are mutually exclusive.')
     elif '-p' in options:
         try:
             testplan = options[options.index('-p') + 1]
@@ -1641,10 +1644,23 @@ def do_add_jobs(conf, options):
         options.remove(spec)
         test_dict["spec"] = spec
 
+    if '--trigger' in options:
+        try:
+            next_job = options[options.index('--trigger') + 1]
+        except IndexError:
+            error_out('job not provided after --trigger.')
+        jobs = [job['name'] for job in server.get_jobs()]
+        if next_job not in jobs:
+            raise Exception('Job %s not found.' % next_job)
+        options.remove('--trigger')
+        options.remove(next_job)
+    else:
+        next_job = None
+
     if test_name:
         test = test_class(test_dict)
         for board_name in board_names:
-            create_job(board_name, test)
+            create_job(board_name, test, next_job)
     else:
         plan_tests = parse_testplan(testplan, test_dict)
         for board_name in board_names:
-- 
2.7.4

Re: [Fuego] testplans and job inter-dependency

[Tim.Bird]

> -----Original Message-----
> From:  Daniel Sangorrin on Friday, November 09, 2018 12:27 AM
> 
> Hello Tim,
> 
> I prepared a couple of patches that fix testplans' job
> execution order, and allow triggering a new job once
> the current one is finished.
> 
> [PATCH 1/2] testplans: execute tests in order
> [PATCH 2/2] add-jobs: add trigger parameter to specify next job

OK - I looked at the patches, but I'm going to put my questions here.

I like the idea of being able to specify that a job can invoke another
job, or be dependent on the previous invocation of another job.

However, what happens if a job is part of more than one test plan?
It looks to me like the 'next_job' is a permanent part of a job definition.
So if Functional.hello_world is created as part of testplan_default,
I can't also use it as part of testplan_smoketest, or launch it standalone.
It seems like launching the job standalone would invoke the entire
sequence of next_jobs.

Also, this approach of using Jenkins is declarative in nature, which means
there it's difficult to do the dependencies conditionally, or to do complex
sequences where more than one job is launched (maybe in parallel)
when another job completes.

I haven't looked in detail at Jenkins pipelines, but my limited understanding
is that they essentially allow you to specify the sequence of operations (ie tests to run) 
procedurally in a high-level language of your choice.

I think I would prefer to see testplans in Fuego converted to this type of
system.

What if we composed test sequences using a Fuego test, with arbitrary
code in the test_run() function of fuegotest.sh

Here's an example:
make a directory:
fuego-core/engine/tests/Testplan.smoketest
with file: fuego_test.sh
that contained:
function test_run {
   ftc run-test -b ${NODE_NAME} -s default -t Benchmark.Dhrystone || true
   ftc run-test -b ${NODE_NAME} -s default -t Benchmark.dbench4 || true
   ftc run-test -b ${NODE_NAME} -s default -t Benchmark.hackbench || true
   ...
   ftc run-test --timeout 15m -b ${NODE_NAME} -s default -t Benchmark.OpenSSL || true
   ...
   ftc run-test -b ${NODE_NAME} -s default -t Functional.hello_world || true
}

Then we modify ftc and the core scripts to support Testplan as a legal kind
of test (along with "Benchmark" and "Functional") - which is just for composing other tests.

I have been thinking of adding an "Action" kind of test, that would be used for other
"lab" operations like deployment of the software under test, or setup of lab equipment
testing, or generating reports, or generating the binary cache of test program packages.

There are issues with using "ftc build-job" instead of "ftc run-test", since I don't think
Jenkins will executed nested tests (due to Jenkins starting only 1 test per executor,
and Fuego defining only one executor per node (or board).

This strategy allows us to run plans like we run tests, with something like:
ftc run-test -b minnowboard -t Testplan.smoketest

(Actually, calling them "Testplan" is not really required, but I don't want to call
them either "Functional" or "Benchmark", and I might use the "Testplan" prefix
to avoid invoking other test phases.)

Note that since the test invocations are procedural, you can do them conditionally
on the outcome of a previous test, or start them in parallel, or decide whether
to stop when a particular one fails.

> I put a mention on the commit logs but I wanted to discuss a couple
> of questions:
> - currently in my patches, the next job is triggered no matter
>   if the current job fails or succeeds. That behavior can be
>   easily changed. I want to know if you prefer to have a
>   new interface to switch the behavior, or just hardwire it to
>   a predefined one (e.g. "always continue" or "stop running jobs
>   if one fails").
I would want the behavior to match exactly what we have now,
for the first iteration of the feature.

>   I can see cases where you want to stop (eg: if the kernel
>   build fails, then the next tests are worthless). On the other
>   hand, i can also see cases where you want to go on (e.g.
>   when the jobs in the testplan do not really depend on each
>   other).
>   If you like the "behavior switch" interface better, then
>   I could add a parameter (parallel=True?) to each test in a testplan
>   to define that behavior and have a default behavior when
>   omitted. Any preferences for the default behavior?
> - the second question is related to the interface used when
>   adding jobs without a testplan. The current interface
>   (--trigger myjob) only allows you to define one job to
>   trigger, but i can add more in the future. The question
>   is: if i do --trigger job1,job2,job3, should all those
>   jobs be triggered in order (one after the other) or not.
>   If we want a "behavior switch" here, the interface could
>   become a bit convoluted. For example:
>    --trigger job1->job2->job3,job4,job5->job6
>   This could be used to define jobs that can run in parallel,
>   and jobs that must run in a specific order.
> 
> Your feedback is very welcomed.

Let me know what you think of my counter-idea to convert testplans
into procedurally defined lists of tests to execute, inside a test structure
like the other tests (away from the declarative syntax we have now).

  -- Tim

Re: [Fuego] testplans and job inter-dependency

[Daniel Sangorrin]

Hi Tim, 

Thanks for your feedback!! See my comments below.

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > -----Original Message-----
> > From:  Daniel Sangorrin on Friday, November 09, 2018 12:27 AM
> >
> > Hello Tim,
> >
> > I prepared a couple of patches that fix testplans' job
> > execution order, and allow triggering a new job once
> > the current one is finished.
> >
> > [PATCH 1/2] testplans: execute tests in order
> > [PATCH 2/2] add-jobs: add trigger parameter to specify next job
> 
> OK - I looked at the patches, but I'm going to put my questions here.
> 
> I like the idea of being able to specify that a job can invoke another
> job, or be dependent on the previous invocation of another job.
> 
> However, what happens if a job is part of more than one test plan?
> It looks to me like the 'next_job' is a permanent part of a job definition.
> So if Functional.hello_world is created as part of testplan_default,
> I can't also use it as part of testplan_smoketest, or launch it standalone.
> It seems like launching the job standalone would invoke the entire
> sequence of next_jobs.

That's something I hadn't thought about!

One possible, although dirty, way to avoid such "collisions" would be to
add the testplan name to the job names.
For example: mytestplan.bbb.100M.Benchmark.Dhrystone.

Having said that, it wouldn't be as powerful as what you describe below.

> Also, this approach of using Jenkins is declarative in nature, which means
> there it's difficult to do the dependencies conditionally, or to do complex
> sequences where more than one job is launched (maybe in parallel)
> when another job completes.
> 
> I haven't looked in detail at Jenkins pipelines, but my limited understanding
> is that they essentially allow you to specify the sequence of operations (ie tests to
> run)
> procedurally in a high-level language of your choice.

The procedural pipelines that you mention are probably the ones called "scripted style" which can be
written in groovy (maybe in other languages too, i don't know).
https://medium.com/@Lenkovits/jenkins-pipelines-and-their-dirty-secrets-1-9e535cd603f4

I think that Jenkins has now declarative pipelines where you can also do conditional execution (when, post, etc..):
https://jenkins.io/blog/2018/04/09/whats-in-declarative/
https://jenkins.io/doc/pipeline/tour/running-multiple-steps/#finishing-up

Unfortunately, we can't rely on them if we want to stay independent of Jenkins in the future.

> I think I would prefer to see testplans in Fuego converted to this type of
> system.
> 
> What if we composed test sequences using a Fuego test, with arbitrary
> code in the test_run() function of fuegotest.sh
> 
> Here's an example:
> make a directory:
> fuego-core/engine/tests/Testplan.smoketest
> with file: fuego_test.sh
> that contained:
> function test_run {
>    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.Dhrystone || true
>    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.dbench4 || true
>    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.hackbench || true
>    ...
>    ftc run-test --timeout 15m -b ${NODE_NAME} -s default -t Benchmark.OpenSSL
> || true
>    ...
>    ftc run-test -b ${NODE_NAME} -s default -t Functional.hello_world || true
> }
> 
> Then we modify ftc and the core scripts to support Testplan as a legal kind
> of test (along with "Benchmark" and "Functional") - which is just for composing
> other tests.
> 
> I have been thinking of adding an "Action" kind of test, that would be used for other
> "lab" operations like deployment of the software under test, or setup of lab
> equipment
> testing, or generating reports, or generating the binary cache of test program
> packages.
> 
> There are issues with using "ftc build-job" instead of "ftc run-test", since I don't think
> Jenkins will executed nested tests (due to Jenkins starting only 1 test per executor,
> and Fuego defining only one executor per node (or board).
> 
> This strategy allows us to run plans like we run tests, with something like:
> ftc run-test -b minnowboard -t Testplan.smoketest
> 
> (Actually, calling them "Testplan" is not really required, but I don't want to call
> them either "Functional" or "Benchmark", and I might use the "Testplan" prefix
> to avoid invoking other test phases.)
> 
> Note that since the test invocations are procedural, you can do them conditionally
> on the outcome of a previous test, or start them in parallel, or decide whether
> to stop when a particular one fails.

It sounds like a really powerful and flexible method. I like it because it is easier to understand than a declarative abstraction layer.
Should we still keep the current testplan files as a way to add jobs to jenkins in a declarative style?
By the way, are we going to keep relying on Jenkins for board access serialization? I thought you were going to create a board/resource reservation system.

> > I put a mention on the commit logs but I wanted to discuss a couple
> > of questions:
> > - currently in my patches, the next job is triggered no matter
> >   if the current job fails or succeeds. That behavior can be
> >   easily changed. I want to know if you prefer to have a
> >   new interface to switch the behavior, or just hardwire it to
> >   a predefined one (e.g. "always continue" or "stop running jobs
> >   if one fails").
> I would want the behavior to match exactly what we have now,
> for the first iteration of the feature.
> 
> >   I can see cases where you want to stop (eg: if the kernel
> >   build fails, then the next tests are worthless). On the other
> >   hand, i can also see cases where you want to go on (e.g.
> >   when the jobs in the testplan do not really depend on each
> >   other).
> >   If you like the "behavior switch" interface better, then
> >   I could add a parameter (parallel=True?) to each test in a testplan
> >   to define that behavior and have a default behavior when
> >   omitted. Any preferences for the default behavior?
> > - the second question is related to the interface used when
> >   adding jobs without a testplan. The current interface
> >   (--trigger myjob) only allows you to define one job to
> >   trigger, but i can add more in the future. The question
> >   is: if i do --trigger job1,job2,job3, should all those
> >   jobs be triggered in order (one after the other) or not.
> >   If we want a "behavior switch" here, the interface could
> >   become a bit convoluted. For example:
> >    --trigger job1->job2->job3,job4,job5->job6
> >   This could be used to define jobs that can run in parallel,
> >   and jobs that must run in a specific order.
> >
> > Your feedback is very welcomed.
> 
> Let me know what you think of my counter-idea to convert testplans
> into procedurally defined lists of tests to execute, inside a test structure
> like the other tests (away from the declarative syntax we have now).

The idea is very powerful, I really like it.
My only questions are:
- Should we completely remove the testplan files? or keep them around as an easy way of adding a set of jobs to Jenkins.
- If we do keep them, how about adding the testplan's name to the job name to avoid collisions between jobs from different testplans?

Thanks,
Daniel

Re: [Fuego] testplans and job inter-dependency

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin 
> 
> Hi Tim,
> 
> Thanks for your feedback!! See my comments below.
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > > -----Original Message-----
> > > From:  Daniel Sangorrin on Friday, November 09, 2018 12:27 AM
> > >
> > > Hello Tim,
> > >
> > > I prepared a couple of patches that fix testplans' job
> > > execution order, and allow triggering a new job once
> > > the current one is finished.
> > >
> > > [PATCH 1/2] testplans: execute tests in order
> > > [PATCH 2/2] add-jobs: add trigger parameter to specify next job
> >
> > OK - I looked at the patches, but I'm going to put my questions here.
> >
> > I like the idea of being able to specify that a job can invoke another
> > job, or be dependent on the previous invocation of another job.
> >
> > However, what happens if a job is part of more than one test plan?
> > It looks to me like the 'next_job' is a permanent part of a job definition.
> > So if Functional.hello_world is created as part of testplan_default,
> > I can't also use it as part of testplan_smoketest, or launch it standalone.
> > It seems like launching the job standalone would invoke the entire
> > sequence of next_jobs.
> 
> That's something I hadn't thought about!
> 
> One possible, although dirty, way to avoid such "collisions" would be to
> add the testplan name to the job names.
> For example: mytestplan.bbb.100M.Benchmark.Dhrystone.
The names are already long.  I'd like to come up with a different
mechanism for this if possible.
> 
> Having said that, it wouldn't be as powerful as what you describe below.
My way is powerful, but it does have tradeoffs.

> 
> > Also, this approach of using Jenkins is declarative in nature, which means
> > there it's difficult to do the dependencies conditionally, or to do complex
> > sequences where more than one job is launched (maybe in parallel)
> > when another job completes.
> >
> > I haven't looked in detail at Jenkins pipelines, but my limited understanding
> > is that they essentially allow you to specify the sequence of operations (ie
> tests to
> > run)
> > procedurally in a high-level language of your choice.
> 
> The procedural pipelines that you mention are probably the ones called
> "scripted style" which can be
> written in groovy (maybe in other languages too, i don't know).
> https://medium.com/@Lenkovits/jenkins-pipelines-and-their-dirty-secrets-
> 1-9e535cd603f4
> 
> I think that Jenkins has now declarative pipelines where you can also do
> conditional execution (when, post, etc..):
> https://jenkins.io/blog/2018/04/09/whats-in-declarative/
> https://jenkins.io/doc/pipeline/tour/running-multiple-steps/#finishing-up

Ugh.  My head is spinning with all their declarative syntax.  I think these
types of things are easy to read but nearly impossible to write without
constantly referring to a manual.  And sometimes the manual is very
hard to find or non-existent.  Well, maybe the same would be true
of a scripted, or procedural style. Either one requires knowledge that
is very specific to the test framework.

In terms of automated operations on the testplans themselves (aside from
running them), declarative style has the benefit that it's regular enough
for certain operations like automatic scanning.  This can be useful
for things like customizing the testplans.

I stumbled across this today:
https://git.linaro.org/qa/test-definitions.git/tree/plans/test-plan-overlay-example.yaml

I have always thought the best way to customize a testplan is just to copy it and
edit the parts you need to change.  But apparently LAVA allows you to
separate your modifications into a separate file.  I guess which approach would be better
would depend on how often the 'base' testplans were modified.  I don't
think they are modified that often, so I'm not sure it's worth the sacrifice
in writability and expressiveness to go with declarative style.

Note that LAVA's declarative style is much easier to read than Jenkins', IMHO.
(Maybe because they have less features.)
See https://git.linaro.org/qa/test-definitions.git/tree/plans/qcomlt/smoke.yaml
for a relatively simple example.

> Unfortunately, we can't rely on them if we want to stay independent of
> Jenkins in the future.

Indeed, my preference would be to do something outside of Jenkins.

> 
> > I think I would prefer to see testplans in Fuego converted to this type of
> > system.
> >
> > What if we composed test sequences using a Fuego test, with arbitrary
> > code in the test_run() function of fuegotest.sh
> >
> > Here's an example:
> > make a directory:
> > fuego-core/engine/tests/Testplan.smoketest
> > with file: fuego_test.sh
> > that contained:
> > function test_run {
> >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.Dhrystone || true
> >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.dbench4 || true
> >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.hackbench || true
> >    ...
> >    ftc run-test --timeout 15m -b ${NODE_NAME} -s default -t
> Benchmark.OpenSSL
> > || true
> >    ...
> >    ftc run-test -b ${NODE_NAME} -s default -t Functional.hello_world ||
> true
> > }
> >
> > Then we modify ftc and the core scripts to support Testplan as a legal kind
> > of test (along with "Benchmark" and "Functional") - which is just for
> composing
> > other tests.
> >
> > I have been thinking of adding an "Action" kind of test, that would be used
> for other
> > "lab" operations like deployment of the software under test, or setup of
> lab
> > equipment
> > testing, or generating reports, or generating the binary cache of test
> program
> > packages.
> >
> > There are issues with using "ftc build-job" instead of "ftc run-test", since I
> don't think
> > Jenkins will executed nested tests (due to Jenkins starting only 1 test per
> executor,
> > and Fuego defining only one executor per node (or board).
> >
> > This strategy allows us to run plans like we run tests, with something like:
> > ftc run-test -b minnowboard -t Testplan.smoketest
> >
> > (Actually, calling them "Testplan" is not really required, but I don't want to
> call
> > them either "Functional" or "Benchmark", and I might use the "Testplan"
> prefix
> > to avoid invoking other test phases.)
> >
> > Note that since the test invocations are procedural, you can do them
> conditionally
> > on the outcome of a previous test, or start them in parallel, or decide
> whether
> > to stop when a particular one fails.
> 
> It sounds like a really powerful and flexible method. I like it because it is
> easier to understand than a declarative abstraction layer.
> Should we still keep the current testplan files as a way to add jobs to jenkins
> in a declarative style?
Yes.  I try not to get rid of anything that people might be using.
We might deprecate it, but I don't see a need to remove it in the short term.

> By the way, are we going to keep relying on Jenkins for board access
> serialization? I thought you were going to create a board/resource
> reservation system.
Well, I have a way to reserve a board and release a reservation.

See 'ftc reserve-resource' and 'ftc release-resource'

A bit more is needed before it's fully functional for board serialization.
I need to add functionality to wait for a board to be released (essentially
a wait queue for a board).  We're creeping up on a "test scheduler"
feature, but I definitely don't want it to get more complicated before
the 1.4 release.  I don't want to introduce full board serialization
at the ftc layer before 1.4, because I suspect
there will be issues with not releasing the lock on test aborts initiated
by Jenkins.  Basically I want to introduce it when I have time to do some
testing.

> 
> > > I put a mention on the commit logs but I wanted to discuss a couple
> > > of questions:
> > > - currently in my patches, the next job is triggered no matter
> > >   if the current job fails or succeeds. That behavior can be
> > >   easily changed. I want to know if you prefer to have a
> > >   new interface to switch the behavior, or just hardwire it to
> > >   a predefined one (e.g. "always continue" or "stop running jobs
> > >   if one fails").
> > I would want the behavior to match exactly what we have now,
> > for the first iteration of the feature.
> >
> > >   I can see cases where you want to stop (eg: if the kernel
> > >   build fails, then the next tests are worthless). On the other
> > >   hand, i can also see cases where you want to go on (e.g.
> > >   when the jobs in the testplan do not really depend on each
> > >   other).
> > >   If you like the "behavior switch" interface better, then
> > >   I could add a parameter (parallel=True?) to each test in a testplan
> > >   to define that behavior and have a default behavior when
> > >   omitted. Any preferences for the default behavior?
> > > - the second question is related to the interface used when
> > >   adding jobs without a testplan. The current interface
> > >   (--trigger myjob) only allows you to define one job to
> > >   trigger, but i can add more in the future. The question
> > >   is: if i do --trigger job1,job2,job3, should all those
> > >   jobs be triggered in order (one after the other) or not.
> > >   If we want a "behavior switch" here, the interface could
> > >   become a bit convoluted. For example:
> > >    --trigger job1->job2->job3,job4,job5->job6
> > >   This could be used to define jobs that can run in parallel,
> > >   and jobs that must run in a specific order.
> > >
> > > Your feedback is very welcomed.
> >
> > Let me know what you think of my counter-idea to convert testplans
> > into procedurally defined lists of tests to execute, inside a test structure
> > like the other tests (away from the declarative syntax we have now).
> 
> The idea is very powerful, I really like it.
> My only questions are:
> - Should we completely remove the testplan files? or keep them around as
> an easy way of adding a set of jobs to Jenkins.
Keep them.

> - If we do keep them, how about adding the testplan's name to the job name
> to avoid collisions between jobs from different testplans?
I'd rather not.

I don't want to put off this feature (serializing the testplan jobs using a different
mechanism than we've got today).  But I'm not sure how to proceed with the least
impact.  Your changes were pretty small, since they leveraged a Jenkins feature.
To build up an alternative in Fuego core will take more time and thought.

Alternative ideas?  Either fuego core or Jenkins has to launch subsequent jobs.
If it's Jenkins, then we'll (obviously) have to use a Jenkins feature. If it's fuego core,
we'll likely have to implement some new code, while trying to leverage as much
existing functionality as we can.

On a related topic, I want to add a feature to track the batch-id for a run.  That is,
when tests are executed as part of the same batch, I want them to have all to have
the same batch-id in their run.json file.  This is so that we can query the results and
generate reports with this batch-id.  With the "test plan is a fuego test" design
I think this is easy:
function test_run {
    export FUEGO_BATCH_ID=$(get_next_batch_id)
    ...
}
and a couple of changes in ftc to save this in the run.json file.
  -- Tim

Re: [Fuego] testplans and job inter-dependency

[Daniel Sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
[...]
> > Having said that, it wouldn't be as powerful as what you describe below.
> My way is powerful, but it does have tradeoffs.

One of them is that you are not using jobs, and therefore our current visualization tools based on Jenkins will not work.

> > > Also, this approach of using Jenkins is declarative in nature, which means
> > > there it's difficult to do the dependencies conditionally, or to do complex
> > > sequences where more than one job is launched (maybe in parallel)
> > > when another job completes.
> > >
> > > I haven't looked in detail at Jenkins pipelines, but my limited understanding
> > > is that they essentially allow you to specify the sequence of operations (ie
> > tests to
> > > run)
> > > procedurally in a high-level language of your choice.
> >
> > The procedural pipelines that you mention are probably the ones called
> > "scripted style" which can be
> > written in groovy (maybe in other languages too, i don't know).
> > https://medium.com/@Lenkovits/jenkins-pipelines-and-their-dirty-secrets-
> > 1-9e535cd603f4
> >
> > I think that Jenkins has now declarative pipelines where you can also do
> > conditional execution (when, post, etc..):
> > https://jenkins.io/blog/2018/04/09/whats-in-declarative/
> > https://jenkins.io/doc/pipeline/tour/running-multiple-steps/#finishing-up
> 
> Ugh.  My head is spinning with all their declarative syntax.  I think these
> types of things are easy to read but nearly impossible to write without
> constantly referring to a manual.  And sometimes the manual is very
> hard to find or non-existent.  Well, maybe the same would be true
> of a scripted, or procedural style. Either one requires knowledge that
> is very specific to the test framework.
> 
> In terms of automated operations on the testplans themselves (aside from
> running them), declarative style has the benefit that it's regular enough
> for certain operations like automatic scanning.  This can be useful
> for things like customizing the testplans.
> 
> I stumbled across this today:
> https://git.linaro.org/qa/test-definitions.git/tree/plans/test-plan-overlay-example
> .yaml
> 
> I have always thought the best way to customize a testplan is just to copy it and
> edit the parts you need to change.  But apparently LAVA allows you to
> separate your modifications into a separate file.  I guess which approach would be
> better
> would depend on how often the 'base' testplans were modified.  I don't
> think they are modified that often, so I'm not sure it's worth the sacrifice
> in writability and expressiveness to go with declarative style.
> 
> Note that LAVA's declarative style is much easier to read than Jenkins', IMHO.
> (Maybe because they have less features.)
> See https://git.linaro.org/qa/test-definitions.git/tree/plans/qcomlt/smoke.yaml
> for a relatively simple example.

LAVA uses an elegant yaml abstraction.
However, I think Fuego wants to focus on usability. Even Fuego beginners should be able to easily modify an existing testplan without having to read through a lot of documentation.

> > Unfortunately, we can't rely on them if we want to stay independent of
> > Jenkins in the future.
> 
> Indeed, my preference would be to do something outside of Jenkins.
> 
> >
> > > I think I would prefer to see testplans in Fuego converted to this type of
> > > system.
> > >
> > > What if we composed test sequences using a Fuego test, with arbitrary
> > > code in the test_run() function of fuegotest.sh
> > >
> > > Here's an example:
> > > make a directory:
> > > fuego-core/engine/tests/Testplan.smoketest
> > > with file: fuego_test.sh
> > > that contained:
> > > function test_run {
> > >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.Dhrystone || true
> > >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.dbench4 || true
> > >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.hackbench || true
> > >    ...
> > >    ftc run-test --timeout 15m -b ${NODE_NAME} -s default -t
> > Benchmark.OpenSSL
> > > || true
> > >    ...
> > >    ftc run-test -b ${NODE_NAME} -s default -t Functional.hello_world ||
> > true
> > > }
> > >
> > > Then we modify ftc and the core scripts to support Testplan as a legal kind
> > > of test (along with "Benchmark" and "Functional") - which is just for
> > composing
> > > other tests.
> > >
> > > I have been thinking of adding an "Action" kind of test, that would be used
> > for other
> > > "lab" operations like deployment of the software under test, or setup of
> > lab
> > > equipment
> > > testing, or generating reports, or generating the binary cache of test
> > program
> > > packages.
> > >
> > > There are issues with using "ftc build-job" instead of "ftc run-test", since I
> > don't think
> > > Jenkins will executed nested tests (due to Jenkins starting only 1 test per
> > executor,
> > > and Fuego defining only one executor per node (or board).
> > >
> > > This strategy allows us to run plans like we run tests, with something like:
> > > ftc run-test -b minnowboard -t Testplan.smoketest
> > >
> > > (Actually, calling them "Testplan" is not really required, but I don't want to
> > call
> > > them either "Functional" or "Benchmark", and I might use the "Testplan"
> > prefix
> > > to avoid invoking other test phases.)
> > >
> > > Note that since the test invocations are procedural, you can do them
> > conditionally
> > > on the outcome of a previous test, or start them in parallel, or decide
> > whether
> > > to stop when a particular one fails.
> >
> > It sounds like a really powerful and flexible method. I like it because it is
> > easier to understand than a declarative abstraction layer.
> > Should we still keep the current testplan files as a way to add jobs to jenkins
> > in a declarative style?
> Yes.  I try not to get rid of anything that people might be using.
> We might deprecate it, but I don't see a need to remove it in the short term.
> 
> > By the way, are we going to keep relying on Jenkins for board access
> > serialization? I thought you were going to create a board/resource
> > reservation system.
> Well, I have a way to reserve a board and release a reservation.
> 
> See 'ftc reserve-resource' and 'ftc release-resource'

Cool. Shall we use that on the testplans? (e.g.: when executing 2 testplans on the same board at the same time, each testplan must first reserve the board)

> 
> A bit more is needed before it's fully functional for board serialization.
> I need to add functionality to wait for a board to be released (essentially
> a wait queue for a board).  We're creeping up on a "test scheduler"
> feature, but I definitely don't want it to get more complicated before
> the 1.4 release.  I don't want to introduce full board serialization
> at the ftc layer before 1.4, because I suspect
> there will be issues with not releasing the lock on test aborts initiated
> by Jenkins.  Basically I want to introduce it when I have time to do some
> testing.

OK, then I guess the testplans functionality should also come after 1.4.

> 
> >
> > > > I put a mention on the commit logs but I wanted to discuss a couple
> > > > of questions:
> > > > - currently in my patches, the next job is triggered no matter
> > > >   if the current job fails or succeeds. That behavior can be
> > > >   easily changed. I want to know if you prefer to have a
> > > >   new interface to switch the behavior, or just hardwire it to
> > > >   a predefined one (e.g. "always continue" or "stop running jobs
> > > >   if one fails").
> > > I would want the behavior to match exactly what we have now,
> > > for the first iteration of the feature.
> > >
> > > >   I can see cases where you want to stop (eg: if the kernel
> > > >   build fails, then the next tests are worthless). On the other
> > > >   hand, i can also see cases where you want to go on (e.g.
> > > >   when the jobs in the testplan do not really depend on each
> > > >   other).
> > > >   If you like the "behavior switch" interface better, then
> > > >   I could add a parameter (parallel=True?) to each test in a testplan
> > > >   to define that behavior and have a default behavior when
> > > >   omitted. Any preferences for the default behavior?
> > > > - the second question is related to the interface used when
> > > >   adding jobs without a testplan. The current interface
> > > >   (--trigger myjob) only allows you to define one job to
> > > >   trigger, but i can add more in the future. The question
> > > >   is: if i do --trigger job1,job2,job3, should all those
> > > >   jobs be triggered in order (one after the other) or not.
> > > >   If we want a "behavior switch" here, the interface could
> > > >   become a bit convoluted. For example:
> > > >    --trigger job1->job2->job3,job4,job5->job6
> > > >   This could be used to define jobs that can run in parallel,
> > > >   and jobs that must run in a specific order.
> > > >
> > > > Your feedback is very welcomed.
> > >
> > > Let me know what you think of my counter-idea to convert testplans
> > > into procedurally defined lists of tests to execute, inside a test structure
> > > like the other tests (away from the declarative syntax we have now).
> >
> > The idea is very powerful, I really like it.
> > My only questions are:
> > - Should we completely remove the testplan files? or keep them around as
> > an easy way of adding a set of jobs to Jenkins.
> Keep them.
> 
> > - If we do keep them, how about adding the testplan's name to the job name
> > to avoid collisions between jobs from different testplans?
> I'd rather not.
> I don't want to put off this feature (serializing the testplan jobs using a different
> mechanism than we've got today).  But I'm not sure how to proceed with the least
> impact.  Your changes were pretty small, since they leveraged a Jenkins feature.
> To build up an alternative in Fuego core will take more time and thought.
> 
> Alternative ideas?  Either fuego core or Jenkins has to launch subsequent jobs.
> If it's Jenkins, then we'll (obviously) have to use a Jenkins feature. If it's fuego core,
> we'll likely have to implement some new code, while trying to leverage as much
> existing functionality as we can.

The thing is that we still do not have a better visualization tool than Jenkins. For that reason, using my approach has the benefit of reusing our current visualization infrastructure.
#When I have time i will try to port Squad support back into Fuego to solve that.

> On a related topic, I want to add a feature to track the batch-id for a run.  That is,
> when tests are executed as part of the same batch, I want them to have all to have
> the same batch-id in their run.json file.  This is so that we can query the results and
> generate reports with this batch-id.  With the "test plan is a fuego test" design
> I think this is easy:
> function test_run {
>     export FUEGO_BATCH_ID=$(get_next_batch_id)
>     ...
> }
> and a couple of changes in ftc to save this in the run.json file.

I will explore the posibility of exporting FUEGO_BATCH_ID between the trigger job and the triggered job.
If that was possible, we could solve the "collisions" problem something like this:

if $FUEGO_BATCH_ID == "testplan_lts"
   start the next job in the testplan lts
if ...

Regards,
Daniel

Re: [Fuego] testplans and job inter-dependency

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin 
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> [...]
> > > Having said that, it wouldn't be as powerful as what you describe below.
> > My way is powerful, but it does have tradeoffs.
> 
> One of them is that you are not using jobs, and therefore our current
> visualization tools based on Jenkins will not work.
This is true.

However, if a Jenkins job exists, even if you do 'ftc run-test' outside of Jenkins,
ftc attempts to populate the Jenkins directories with suitable Jenkins
build files.  I haven't figured out how to get Jenkins to re-read the build directories
and update the build status for jobs without manually telling it to reload its
configuration from disk, but this seems to work OK.

I have done the following:
* create a job for a board-test combination
* build the job in Jenkins
* "ftc run-test" that board-test combination from the command line
* tell Jenkins to reload configuration from disk
   * or, inside the container do: service jenkins stop; service jenkins start
And the data for the outside job is in the visualization.

There's a fixthis in ftc to figure out how to have ftc cause the reload.

Also, there are still some thorny issues with build numbers when you do 'ftc run-test'
outside of Jenkins before creating a job for a board/test combination.  ftc has already
used some build numbers in that case, and it confuses Jenkins, which wants to start
the job with build number 1.  Ideally we would create the correct set of Jenkins
build files at the time of job creation, but that's probably difficult.

...
> > Note that LAVA's declarative style is much easier to read than Jenkins',
> IMHO.
> > (Maybe because they have less features.)
> > See https://git.linaro.org/qa/test-
> definitions.git/tree/plans/qcomlt/smoke.yaml
> > for a relatively simple example.
> 
> LAVA uses an elegant yaml abstraction.
> However, I think Fuego wants to focus on usability. Even Fuego beginners
> should be able to easily modify an existing testplan without having to read
> through a lot of documentation.
Agreed.

...
> > Well, I have a way to reserve a board and release a reservation.
> >
> > See 'ftc reserve-resource' and 'ftc release-resource'
> 
> Cool. Shall we use that on the testplans? (e.g.: when executing 2 testplans on
> the same board at the same time, each testplan must first reserve the board)
Not until after 1.4.

> 
> >
> > A bit more is needed before it's fully functional for board serialization.
> > I need to add functionality to wait for a board to be released (essentially
> > a wait queue for a board).  We're creeping up on a "test scheduler"
> > feature, but I definitely don't want it to get more complicated before
> > the 1.4 release.  I don't want to introduce full board serialization
> > at the ftc layer before 1.4, because I suspect
> > there will be issues with not releasing the lock on test aborts initiated
> > by Jenkins.  Basically I want to introduce it when I have time to do some
> > testing.
> 
> OK, then I guess the testplans functionality should also come after 1.4.

Unfortunately, yes.
 
...
> 
> The thing is that we still do not have a better visualization tool than Jenkins.
> For that reason, using my approach has the benefit of reusing our current
> visualization infrastructure.
> #When I have time i will try to port Squad support back into Fuego to solve
> that.
> 
> > On a related topic, I want to add a feature to track the batch-id for a run.
> That is,
> > when tests are executed as part of the same batch, I want them to have all
> to have
> > the same batch-id in their run.json file.  This is so that we can query the
> results and
> > generate reports with this batch-id.  With the "test plan is a fuego test"
> design
> > I think this is easy:
> > function test_run {
> >     export FUEGO_BATCH_ID=$(get_next_batch_id)
> >     ...
> > }
> > and a couple of changes in ftc to save this in the run.json file.
> 
> I will explore the posibility of exporting FUEGO_BATCH_ID between the
> trigger job and the triggered job.
> If that was possible, we could solve the "collisions" problem something like
> this:
> 
> if $FUEGO_BATCH_ID == "testplan_lts"
>    start the next job in the testplan lts
> if ...

Interesting.  I was thinking the batch-id would be something unique, like a timestamp.
But if it included the testplan name, then we could have ftc trigger the next job.

We could experiment with 'ftc build-job' in something like 'Functional.plan_lts/fuego_test.sh,
without changing anything in fuego-core. (We would still have to overcome the issue
of nested use of a Jenkins node).

What's easier to fix - causing reload of Jenkins build data, or working around the concurrency
lock on a Jenkins node?
 -- Tim