[Fuego] question on test plans in fuego

[Fuego] question on test plans in fuego

[Töws, Heinrich]

Hey altogether,

I have a question regarding test plans ..

Under "overlays/testplans/" (fuego-core) I see a lot of them.

What I discovered is that when I trigger a test plan batch job the tests 
are not scheduled in the some order as they are defined within the 
respective JSON file.

Why is that so? I'd like them to be scheduled in a specific order - is 
that possible?

Thanks.

-- 
Best regards,

Heinrich Toews

Re: [Fuego] question on test plans in fuego

[Bird, Tim]

> -----Original Message-----
> From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Töws, Heinrich
> 
> Hey altogether,
> 
> I have a question regarding test plans ..
> 
> Under "overlays/testplans/" (fuego-core) I see a lot of them.
> 
> What I discovered is that when I trigger a test plan batch job the tests
> are not scheduled in the some order as they are defined within the
> respective JSON file.
> 
> Why is that so? I'd like them to be scheduled in a specific order - is
> that possible?


Heinrich,

You have caught Fuego in the middle of a change.  For various reasons,
including the ordering problem you mention, we were not very happy 
with the old testplan system.  We have been working on a replacement
system called batch jobs.

The testplans in overlays/testplans are deprecated.  We have replacement
jobs for some of the testplans, but not all of them.

For a comparison to see how we now prefer to do things, compare
overlays/testplans/
with fuego-core/test/Functional.batch_smoketest/fuego_test.sh

In the batch job, fuego_test.sh has an embedded testplan, which is
used to populate the Jenkins interface with the jobs, and to provide
default parameters for the jobs.  However, the tests are run in sequence
inside the test's test_run() function.  This preserves the order of tests.

So, please use batch tests instead of testplans.  When you add a Jenkins
job for a batch test, then Fuego will add the dependent jobs for that test
automatically.  This needs to be updated in the documentation.
Sorry about that.

I hope this helps.

 -- Tim

P.S. One downside to the new "batch job" approach is that (from the
point of view of Jenkins) the batch test will have control of the board for
the duration of all sub-tests.  This might be too long.
We are still working on a solution to that issue.

> Thanks.
> 
> --
> Best regards,
> 
> Heinrich Toews
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] question on test plans in fuego

[Bird, Tim]

Oops - omitted a line, see below

> -----Original Message-----
> From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Bird, Tim
> 
> > -----Original Message-----
> > From: Fuego <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Töws, Heinrich
> >
> > Hey altogether,
> >
> > I have a question regarding test plans ..
> >
> > Under "overlays/testplans/" (fuego-core) I see a lot of them.
> >
> > What I discovered is that when I trigger a test plan batch job the tests
> > are not scheduled in the some order as they are defined within the
> > respective JSON file.
> >
> > Why is that so? I'd like them to be scheduled in a specific order - is
> > that possible?
> 
> 
> Heinrich,
> 
> You have caught Fuego in the middle of a change.  For various reasons,
> including the ordering problem you mention, we were not very happy
> with the old testplan system.  We have been working on a replacement
> system called batch jobs.
> 
> The testplans in overlays/testplans are deprecated.  We have replacement
> jobs for some of the testplans, but not all of them.
> 
> For a comparison to see how we now prefer to do things, compare
> overlays/testplans/

That should be:
fuego-core/overlays/testplans/testplan_smoketest.json

> with fuego-core/test/Functional.batch_smoketest/fuego_test.sh
> 

...
 -- Tim